CN110099112B - Data storage method, device, medium and terminal equipment based on point-to-point network - Google Patents
Data storage method, device, medium and terminal equipment based on point-to-point network Download PDFInfo
- Publication number
- CN110099112B CN110099112B CN201910347818.3A CN201910347818A CN110099112B CN 110099112 B CN110099112 B CN 110099112B CN 201910347818 A CN201910347818 A CN 201910347818A CN 110099112 B CN110099112 B CN 110099112B
- Authority
- CN
- China
- Prior art keywords
- data
- node
- point
- hash
- storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1458—Management of the backup or restore process
- G06F11/1464—Management of the backup or restore process for networked environments
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/104—Peer-to-peer [P2P] networks
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/104—Peer-to-peer [P2P] networks
- H04L67/1074—Peer-to-peer [P2P] networks for supporting data block transmission mechanisms
- H04L67/1078—Resource delivery mechanisms
- H04L67/108—Resource delivery mechanisms characterised by resources being split in blocks or fragments
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention belongs to the technical field of computers, and particularly relates to a data storage method and device based on a point-to-point network, a computer readable storage medium and terminal equipment. In the method, a source node divides data to be stored into each data fragment and determines the backup number of each data fragment; the source node determines a level 1 storage node corresponding to the s-th data fragment and sends a data storage request to the level 1 storage node; the nth-level storage node determines an nth + 1-level storage node corresponding to the s-th data fragment and sends a data storage request to the nth + 1-level storage node; after receiving the data storage request, the n + 1-th-level storage node downloads and stores the s-th data fragment from each target node, adds the node identification of the n + 1-th-level storage node into the storage node set, and subtracts a counting unit from the counting value until the counting value is 0, thereby reducing the occupation of the source node bandwidth resources.
Description
Technical Field
The invention belongs to the technical field of computers, and particularly relates to a data storage method and device based on a point-to-point network, a computer readable storage medium and terminal equipment.
Background
A Peer-to-Peer (P2P) network is a network architecture mainly composed of a large number of Peer nodes, where different nodes are related to each other and can transmit resources to each other. In the process of resource transmission, each node can acquire required resources from other nodes with resources in the peer-to-peer network and distribute the downloaded resources to each node needing the resources in the peer-to-peer network, so that the transmission efficiency of the resources in the network is greatly improved.
In the storage in the peer-to-peer network, in order to ensure the security of data, the redundancy of data needs to be increased, the same data is often distributed into multiple copies to the peer-to-peer network, and as a storage space consumer, repeated transmission of one copy of data consumes a large amount of bandwidth, resulting in waste of bandwidth resources.
Disclosure of Invention
In view of this, embodiments of the present invention provide a data storage method and apparatus based on a peer-to-peer network, a computer-readable storage medium, and a terminal device, so as to solve the problems that an existing office application is relatively complicated to operate and user experience is poor.
A first aspect of an embodiment of the present invention provides a data storage method based on a peer-to-peer network, where the method is applied to a peer-to-peer network including two or more nodes, and the method includes:
dividing data to be stored into each data fragment by a source node, and determining the backup number of each data fragment, wherein the source node is any one node in the point-to-point network;
the source node determines a level 1 storage node corresponding to an s-th data fragment in the point-to-point network and sends a data storage request to the level 1 storage node, wherein the data storage request comprises a count value and a storage node set, the count value is equal to the backup number of the s-th data fragment, the storage node set is empty, s is more than or equal to 1 and less than or equal to SN, and SN is the number of the data fragments;
after receiving the data storage request, the 1 st-level storage node downloads and stores the s-th data fragment from the source node, adds the node identification of the 1 st-level storage node into the storage node set, and subtracts a counting unit from the counting value;
the nth-level storage node determines an nth + 1-level storage node corresponding to the s-th data fragment in the point-to-point network and sends a data storage request to the nth + 1-level storage node, wherein n is a positive integer;
after receiving the data storage request, the n + 1-th-level storage node downloads and stores the s-th data fragment from each target node, adds the node identifier of the n + 1-th-level storage node into the storage node set, and subtracts a counting unit from the counting value until the counting value is 0, wherein the target node is a node corresponding to each node identifier in the storage node set of the data storage request.
A second aspect of embodiments of the present invention provides a data storage apparatus, which may include:
the data fragment dividing module is used for dividing the data to be stored into various data fragments;
the backup number determining module is used for determining the backup number of each data fragment;
the first determining module is used for determining a level 1 storage node corresponding to an s-th data fragment in the peer-to-peer network and sending a data storage request to the level 1 storage node, wherein the data storage request comprises a count value and a storage node set, the count value is equal to the backup number of the s-th data fragment, the storage node set is empty, s is more than or equal to 1 and less than or equal to SN, and SN is the number of the data fragments;
the first processing module is used for downloading and storing the s-th data fragment from the source node after receiving the data storage request, adding a local node identifier into the storage node set, and subtracting a counting unit from the counting value;
the second determining module is used for determining a next-level storage node corresponding to the s-th data fragment in the point-to-point network and sending a data storage request to the next-level storage node;
and the second processing module is used for downloading and storing the s-th data fragment from each target node after receiving the data storage request, adding a local node identifier into the storage node set, and subtracting a counting unit from the counting value, wherein the target node is a node corresponding to each node identifier in the storage node set of the data storage request.
A third aspect of embodiments of the present invention provides a computer-readable storage medium storing computer-readable instructions that, when executed by a processor, implement the functions of the data storage apparatus described above.
A fourth aspect of the embodiments of the present invention provides a terminal device, including a memory, a processor, and computer readable instructions stored in the memory and executable on the processor, where the processor implements the functions of the data storage apparatus when executing the computer readable instructions.
Compared with the prior art, the embodiment of the invention has the following beneficial effects: in the embodiment of the invention, a source node firstly divides data to be stored into each data fragment, determines the backup number of each data fragment, then determines a level 1 storage node corresponding to an s-th data fragment in a point-to-point network, and sends a data storage request to the level 1 storage node, wherein the data storage request comprises a count value and a storage node set, the count value is equal to the backup number of the s-th data fragment, and the storage node set is empty. After receiving the data storage request, the level 1 storage node first downloads and stores the s-th data fragment from the source node, adds its node identifier to the storage node set, subtracts a count unit from the count value, then determines the level 2 storage node corresponding to the s-th data fragment in the peer-to-peer network, and sends the data storage request to the level 2 storage node, and after receiving the data storage request, the level 2 storage node downloads and stores the s-th data fragment from each target node (i.e., the node corresponding to each node identifier in the storage node set of the data storage request), … …, and repeats the above processes until the count value is 0. By the mode, the storage consumer (namely the source node) only needs to send one copy of data to the point-to-point network, and other nodes in the network carry out data distribution by themselves and gradually increase the concurrency number, so that the effect of rapid distribution is achieved, and the occupation of bandwidth resources is reduced.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.
FIG. 1 is a flowchart illustrating an embodiment of a peer-to-peer network-based data storage method according to an embodiment of the present invention;
FIG. 2 is a schematic flow diagram of determining a level 1 storage node corresponding to an s-th data slice in a peer-to-peer network;
FIG. 3 is a schematic diagram of a hash ring;
FIG. 4 is a diagram illustrating a node in a peer-to-peer network mapped onto a hash ring;
FIG. 5 is a schematic diagram of a lookup of a level 1 storage node on a hash ring;
FIG. 6 is a block diagram of one embodiment of a data storage device in accordance with one embodiment of the present invention;
fig. 7 is a schematic block diagram of a terminal device in an embodiment of the present invention.
Detailed Description
In order to make the objects, features and advantages of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is obvious that the embodiments described below are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, an embodiment of a data storage method based on a peer-to-peer network according to an embodiment of the present invention may include:
step S101, the source node divides the data to be stored into each data fragment and determines the backup number of each data fragment.
The source node is any one node in the point-to-point network. In an actual use process, if a user needs to store data in a terminal device (i.e., the source node) such as a mobile phone, a tablet, a computer, etc. of the user in the peer-to-peer network, the data to be stored may be selected from a specific application installed on the terminal device of the user, and a determination button is clicked to notify the source node of performing a data storage operation.
The source node may perform fragmentation processing on the data to be stored, that is, according to a certain rule, a data set (i.e., the data to be stored) is divided into a plurality of mutually independent and orthogonal data subsets (i.e., data fragments), so as to facilitate distributed storage.
Specifically, the source node may first calculate the number of data fragments according to the following formula:
the data bytes is the number of bytes of the data to be stored, the maxsard is the maximum number of bytes of the data slices, the specific value may be set according to an actual situation, for example, the specific value may be set to 1024, 2048, 4096 or other values, Ceil is an upward rounding function, and SN is the number of the data slices.
Then, dividing the data to be stored into SN data fragments meeting the following conditions:
wherein s is the serial number of each data fragment, s is more than or equal to 1 and less than or equal to SN, ShdbytessThe number of bytes of the s-th data segment, Max is a function of solving the maximum value, Min is a function of solving the minimum value, Thresh is a preset proportional threshold, and the specific value thereof may be set according to the actual situation, for example, it may be set to 0.1, 0.2, 0.3, or other values.
After the division of the data fragments is completed, the source node may allocate a fragment identifier to each data fragment, and the identifier of each data fragment is unique in the whole peer-to-peer network, and the fragment identifiers of any two data fragments are different.
For the data fragments, the greater the number of backups in the peer-to-peer network, the greater the redundancy, and when one or more storage nodes storing the data fragments in the peer-to-peer network fail, the more other storage nodes storing the data fragments still operate normally, and the higher the security of the data. However, if the number of data fragments to be backed up in the peer-to-peer network is too large, a large amount of network storage resources will be occupied, and therefore, a trade-off between the number of data fragments and the number of backup copies needs to be made to select an appropriate number.
In this embodiment, the backup number of the data segments in the peer-to-peer network may be determined according to the rank of the user, the user is divided into a plurality of ranks, such as a common user, a paid user, and a VIP user, each rank has a corresponding backup number, and the higher the rank of the user, the more the backup number of the data segments in the peer-to-peer network.
For example, the number of backups of data fragments of a general user in the peer-to-peer network may be set to 3, the number of backups of data fragments of a paid user in the peer-to-peer network to 5, the number of backups of data fragments of a VIP user in the peer-to-peer network to 10, and so on.
Preferably, in the present embodiment, the backup number of each data slice may be calculated by the following formula:
wherein NodesNum is the total number of nodes in the peer-to-peer network, BaseDNum is the preset reference node number, HsShardNum is the total number of data fragments stored in the peer-to-peer network counted according to a historical record, HsDmgSdNum is the total number of data fragments lost or damaged in the peer-to-peer network counted according to the historical record, c is a serial number arranged from high to low of a user grade, c is more than or equal to 1 and less than or equal to ClassNum, ClassNum is the total number of the user grade, ClassCoef is the total number of the user gradecFor the weighting coefficient of the c-th user class, the higher the class is, the larger the corresponding coefficient is, and the specific coefficient value may be set according to the actual situation, for example, ClassCoef may be set1＝0.001，ClassCoef2＝0.0005，ClassCoef30.0003, … …, and so on, BkpNumcThe backup number of the data fragmentation in the point-to-point network for the c-th user level.
Step S102, the source node determines a level 1 storage node corresponding to the S-th data fragment in the peer-to-peer network, and sends a data storage request to the level 1 storage node.
Wherein s is more than or equal to 1 and less than or equal to SN. Specifically, the source node may determine, in the peer-to-peer network, a level 1 storage node corresponding to the s-th data slice through a process as shown in fig. 2:
and step S1021, calculating the hash value of the S-th data fragment by using a preset first hash function.
In this embodiment, the hash values of the s-th data slice may be respectively calculated according to the following formula:
ShardHashs＝FsHashFunc(ShardIDs)
wherein ShardIDsFor the fragment identification of the s-th data fragment, FsHashFunc is the first hash function, ShardHashsIs the hash value of the s-th data slice.
All hash functions have the following basic characteristics: if two hash values are not identical (according to the same hash function), then the original inputs for the two hash values are also not identical. This property gives the hash function a deterministic result. On the other hand, however, the input and output of a hash function are not unique correspondences, and if two hash values are the same, the two input values are likely to be the same, but may also be different, which is called "hash collision", which is usually two different input values, and the same output value is calculated intentionally. However, for the hash functions commonly used in the prior art, the probability of collision is very low and can be almost ignored, some data are input to calculate the hash value, then the input value is partially changed, and a hash function with strong confusion characteristics can generate a completely different hash value. The first hash function and other hash functions in this embodiment may include, but are not limited to, any one of the hash functions commonly used in the prior art, such as MD4, MD5, SHA1, and so on.
Step S1022, determining a target location point from a location point on the preset hash ring corresponding to the hash value of the S-th data segment.
The hash ring is a circular ring formed by sequentially connecting all function values of the first hash function clockwise or counterclockwise from small to large, that is, each position point on the hash ring corresponds to one function value of the first hash function, and if the value ranges of the first hash function are [0, N ] and are arranged in a clockwise order, the hash ring shown in fig. 3 can be formed.
And step S1023, acquiring each reference position point on the hash ring.
Each reference position point is a position point corresponding to one node in the point-to-point network. The setting process of the reference position point comprises the following steps:
firstly, the identity of each node in the peer-to-peer network is obtained.
The Identity of a certain node may be an International Mobile Equipment Identity (IMEI) of the node, and the IMEI is an electronic serial number composed of 15 digits and corresponding to each terminal device one by one, and the code is unique worldwide. Each terminal device is assigned a globally unique number after assembly, which number is recorded from manufacture to delivery by the manufacturer.
Respectively representing the acquired identity of each node as: NodeID1、NodeID2、…、NodeIDm、…、NodeIDMWherein m is the serial number of each node, m is more than or equal to 1 and less than or equal to NodesNum, NodeIDmIs the identity of the mth node.
And then, respectively carrying out hash operation on the identity of each node in the point-to-point network by using a preset second hash function to obtain the hash value of each node.
Specifically, the hash value of each node may be calculated according to the following formula:
NodeHashm＝SdHashFunc(NodeIDm)
wherein SdhashFunc is the second hash function, and theThe value range of the second hash function is consistent with that of the first hash function, so that the two hash functions can be mapped to the same hash ring, and NodeHashmIs the hash value of the mth node.
And finally, determining a position point corresponding to the hash value of the mth node on the hash ring as a reference position point corresponding to the mth node.
FIG. 4 shows the case where the hash values of 4 nodes are mapped onto the hash ring, NodeHash1、NodeHash2、NodeHash3、NodeHash4The corresponding position points are all the reference position points.
Step S1024, clockwise or anticlockwise searching the reference position point from the target position point, and determining the node corresponding to the searched first reference position point as the level 1 storage node corresponding to the S-th data fragment.
Taking FIG. 5 as an example, the hash value ShardHash sliced from the 1 st data1The corresponding target position point starts to search clockwise, and the first reference position point is NodeHash2And if the corresponding position point is located, determining the 2 nd node in the point-to-point network as the 1 st level storage node corresponding to the 1 st data fragment.
By the method, the following storage correspondence can be determined:
if the counterclockwise search method is adopted, the specific process is similar to the counterclockwise search process, and the above process may be referred to specifically, and will not be described herein again.
After determining the level 1 storage node, the source node may send a data storage request to the level 1 storage node. The data storage request comprises a count value, a storage node set and an s-th data fragment identifier, wherein in an initial state, the count value is equal to the backup number of the s-th data fragment, and the storage node set is empty.
Step S103, after receiving the data storage request, the level 1 storage node downloads and stores the S-th data fragment from the source node, adds the node identifier of the level 1 storage node to the storage node set, and subtracts a count unit from the count value.
Recording the storage node set as a NodeSet, recording the node identifier of the 1 st-level storage node as a NodeID (1), and recording the Count value as a Count, then:
NodeSet＝NodeSet∪NodeID(1)；
Count＝Count－1。
and step S104, the nth-level storage node determines the (n +1) th-level storage node corresponding to the (S) th data fragment in the point-to-point network, and sends a data storage request to the (n +1) th-level storage node.
n is a positive integer, and in an initial state, n is set to 1.
The process of determining the nth +1 th level storage node corresponding to the s-th data slice by the nth level storage node is similar to the process shown in fig. 2, which may be referred to the foregoing specifically, and is not described herein again. However, it should be noted that in this process, the reference position points corresponding to the nodes in the storage node set need to be removed from the hash ring, otherwise, the finally determined n + 1-th storage node is still a node in the storage node set.
Step S105, after receiving the data storage request, the n +1 th-level storage node downloads and stores the S-th data fragment from each target node, adds the node identifier of the n +1 th-level storage node to the storage node set, and subtracts a count unit from the count value.
And the target node is a node corresponding to each node identifier in the storage node set of the data storage request. It should be noted that the n +1 th-level storage node downloads the s-th data fragment from each target node simultaneously in a concurrent downloading manner, so as to accelerate the downloading speed.
After the downloading and storing are completed, the node identifier of the n +1 th-level storage node is marked as NodeID (n +1), and then:
NodeSet＝NodeSet∪NodeID(n+1)；
Count＝Count－1。
and step S106, judging whether the count value is 0.
If the count value is not 0, it indicates that the number of data segments backed up is not completed, and then step S107 is continuously executed, and if the count value is 0, it indicates that the number of data segments backed up is completed, and then step S108 is executed.
Step S107, increasing n by one counting unit.
Namely, executing: n is n +1, and then returns to execute step S104 and subsequent steps until the count value is 0.
And S108, ending the data storage process.
In summary, in the embodiment of the present invention, a source node first divides data to be stored into data fragments, determines the backup number of each data fragment, then determines a level 1 storage node corresponding to an s-th data fragment in the peer-to-peer network, and sends a data storage request to the level 1 storage node, where the data storage request includes a count value and a storage node set, the count value is equal to the backup number of the s-th data fragment, and the storage node set is empty. After receiving the data storage request, the level 1 storage node first downloads and stores the s-th data fragment from the source node, adds its node identifier to the storage node set, subtracts a count unit from the count value, then determines the level 2 storage node corresponding to the s-th data fragment in the peer-to-peer network, and sends the data storage request to the level 2 storage node, and after receiving the data storage request, the level 2 storage node downloads and stores the s-th data fragment from each target node (i.e., the node corresponding to each node identifier in the storage node set of the data storage request), … …, and repeats the above processes until the count value is 0. By the mode, the storage consumer (namely the source node) only needs to send one copy of data to the point-to-point network, and other nodes in the network carry out data distribution by themselves and gradually increase the concurrency number, so that the effect of rapid distribution is achieved, and the occupation of bandwidth resources is reduced.
It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.
Fig. 6 shows a block diagram of an embodiment of a data storage device according to the present invention, which corresponds to the data storage method based on the peer-to-peer network described in the foregoing embodiment.
In this embodiment, a data storage apparatus may include:
the data fragment dividing module 601 is configured to divide data to be stored into data fragments;
a backup number determining module 602, configured to determine a backup number of each data fragment;
a first determining module 603, configured to determine a level 1 storage node corresponding to an s-th data fragment in the peer-to-peer network, and send a data storage request to the level 1 storage node, where the data storage request includes a count value and a storage node set, the count value is equal to the backup number of the s-th data fragment, the storage node set is empty, s is greater than or equal to 1 and less than or equal to SN, and SN is the number of data fragments;
a first processing module 604, configured to download and store an s-th data fragment from the source node after receiving the data storage request, add a local node identifier to the storage node set, and subtract a count unit from the count value;
a second determining module 605, configured to determine a next-stage storage node corresponding to the s-th data segment in the peer-to-peer network, and send a data storage request to the next-stage storage node;
a second processing module 606, configured to download and store the s-th data fragment from each target node after receiving the data storage request, add a local node identifier to the storage node set, and subtract a count unit from the count value, where the target node is a node corresponding to each node identifier in the storage node set of the data storage request.
Further, the backup number determining module is specifically configured to calculate the backup number of each data fragment according to the following formula:
wherein NodesNum is the total number of nodes in the peer-to-peer network, BaseDNum is the preset reference node number, HsShardNum is the total number of data fragments stored in the peer-to-peer network counted according to a historical record, HsDmgSdNum is the total number of data fragments lost or damaged in the peer-to-peer network counted according to the historical record, c is a serial number arranged from high to low of a user grade, c is more than or equal to 1 and less than or equal to ClassNum, ClassNum is the total number of the user grade, ClassCoef is the total number of the user gradecIs the weight coefficient of the c-th user level, BkpNumcThe backup number of the data fragmentation in the point-to-point network for the c-th user level.
Further, the first determining module may include:
the fragment hash value calculation unit is used for calculating the hash value of the s-th data fragment by using a preset first hash function;
the target position point determining unit is used for determining a target position point from a position point corresponding to the hash value of the s-th data fragment on a preset hash ring, wherein the hash ring is a circular ring formed by sequentially connecting all function values of the first hash function from small to large in a clockwise or anticlockwise mode, namely each position point on the hash ring corresponds to one function value of the first hash function;
a reference position point obtaining unit, configured to obtain each reference position point on the hash ring, where each reference position point is a position point corresponding to a node in the peer-to-peer network;
and the reference position point searching unit is used for searching the reference position point clockwise or anticlockwise from the target position point and determining the node corresponding to the searched first reference position point as the level 1 storage node corresponding to the s-th data fragment.
Further, the first determining module may further include:
an identity obtaining unit, configured to obtain an identity of each node in the peer-to-peer network;
the hash operation unit is used for performing hash operation on the identity of each node in the point-to-point network by using a preset second hash function to obtain a hash value of each node, and the value range of the second hash function is consistent with the value range of the first hash function;
and the reference position point determining unit is used for determining a position point corresponding to the hash value of the mth node on the hash ring as a reference position point corresponding to the mth node, wherein m is more than or equal to 1 and is less than or equal to NodesNum.
Further, the data slice dividing module may include:
a slice number calculation unit, configured to calculate the number of data slices according to the following formula:
the data bytes is the number of bytes of the data to be stored, MaxShard is the maximum number of bytes of the data fragments, and Ceil is an upward rounding function;
the data fragment dividing unit is used for dividing the data to be stored into SN data fragments meeting the following conditions:
among them, ShdBytessThe number of bytes of the s-th data fragment is shown, Max is a maximum value solving function, Min is a minimum value solving function, and Thresh is a preset proportion threshold.
It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described apparatuses, modules and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.
Fig. 7 shows a schematic block diagram of a terminal device according to an embodiment of the present invention, and for convenience of description, only the relevant parts related to the embodiment of the present invention are shown.
In this embodiment, the terminal device 7 may be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The terminal device 7 may include: a processor 70, a memory 71, and computer readable instructions 72 stored in the memory 71 and executable on the processor 70, such as computer readable instructions to perform the peer-to-peer network based data storage method described above. The processor 70, when executing the computer readable instructions 72, implements the functionality of the various modules/units in the apparatus embodiments described above, such as the functionality of the modules 601-606 shown in fig. 6.
Illustratively, the computer readable instructions 72 may be partitioned into one or more modules/units that are stored in the memory 71 and executed by the processor 70 to implement the present invention. The one or more modules/units may be a series of computer-readable instruction segments capable of performing specific functions, which are used for describing the execution process of the computer-readable instructions 72 in the terminal device 7.
The Processor 70 may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic device, discrete hardware component, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory 71 may be an internal storage unit of the terminal device 7, such as a hard disk or a memory of the terminal device 7. The memory 71 may also be an external storage device of the terminal device 7, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are provided on the terminal device 7. Further, the memory 71 may also include both an internal storage unit and an external storage device of the terminal device 7. The memory 71 is used for storing the computer readable instructions and other instructions and data required by the terminal device 7. The memory 71 may also be used to temporarily store data that has been output or is to be output.
Each functional unit in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of computer readable instructions for enabling a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and the like, which can store computer readable instructions.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A data storage method based on a peer-to-peer network is characterized in that the method is applied to the peer-to-peer network comprising more than two nodes, and the method comprises the following steps:
dividing data to be stored into each data fragment by a source node, and determining the backup number of each data fragment, wherein the source node is any one node in the point-to-point network;
the source node determines a level 1 storage node corresponding to an s-th data fragment in the point-to-point network and sends a data storage request to the level 1 storage node, wherein the data storage request comprises a count value and a storage node set, the count value is equal to the backup number of the s-th data fragment, the storage node set is empty, s is more than or equal to 1 and less than or equal to SN, and SN is the number of the data fragments;
after receiving the data storage request, the 1 st-level storage node downloads and stores the s-th data fragment from the source node, adds the node identification of the 1 st-level storage node into the storage node set, and subtracts a counting unit from the counting value;
the nth-level storage node determines an nth + 1-level storage node corresponding to the s-th data fragment in the point-to-point network and sends a data storage request to the nth + 1-level storage node, wherein n is a positive integer;
after receiving the data storage request, the n +1 th-level storage node simultaneously downloads and stores the s-th data fragment from each target node in a concurrent downloading mode, adds the node identification of the n +1 th-level storage node into the storage node set, and subtracts a counting unit from the counting value until the counting value is 0, wherein the target node is a node corresponding to each node identification in the storage node set of the data storage request.
2. The data storage method of claim 1, wherein the determining the number of backups for each data slice comprises:
calculating the backup number of each data fragment according to the following formula:
wherein NodesNum is the total number of nodes in the peer-to-peer network, BaseDNum is the preset reference node number, HsShardNum is the total number of data fragments stored in the peer-to-peer network counted according to a historical record, HsDmgSdNum is the total number of data fragments lost or damaged in the peer-to-peer network counted according to the historical record, c is a serial number arranged from high to low of a user grade, c is more than or equal to 1 and less than or equal to ClassNum, ClassNum is the total number of the user grade, ClassCoef is the total number of the user gradecIs the weight coefficient of the c-th user level, BkpNumcThe backup number of the data fragmentation in the point-to-point network for the c-th user level.
3. The data storage method of claim 1, wherein determining the level 1 storage node corresponding to the s-th data slice in the peer-to-peer network comprises:
calculating the hash value of the s-th data fragment by using a preset first hash function;
determining a target position point of a position point corresponding to the hash value of the s-th data fragment on a preset hash ring, wherein the hash ring is a circular ring formed by sequentially connecting all function values of the first hash function clockwise or anticlockwise from small to large, namely each position point on the hash ring corresponds to one function value of the first hash function;
acquiring each reference position point on the hash ring, wherein each reference position point is a position point corresponding to one node in the point-to-point network;
and clockwise or anticlockwise searching the reference position point from the target position point, and determining the node corresponding to the searched first reference position point as the 1 st-level storage node corresponding to the s-th data fragment.
4. The data storage method according to claim 3, wherein the setting process of the reference position point includes:
acquiring the identity of each node in the point-to-point network;
performing hash operation on the identity of each node in the point-to-point network by using a preset second hash function to obtain a hash value of each node, wherein the value range of the second hash function is consistent with the value range of the first hash function;
and determining a position point corresponding to the hash value of the mth node on the hash ring as a reference position point corresponding to the mth node, wherein m is more than or equal to 1 and less than or equal to NodesNum.
5. The data storage method according to any one of claims 1 to 4, wherein the dividing the data to be stored into the respective data slices comprises:
the number of data slices is calculated according to the following formula:
the data bytes is the number of bytes of the data to be stored, MaxShard is the maximum number of bytes of the data fragments, and Ceil is an upward rounding function;
dividing the data to be stored into SN data fragments meeting the following conditions:
among them, ShdBytessThe number of bytes of the s-th data fragment is shown, Max is a maximum value solving function, Min is a minimum value solving function, and Thresh is a preset proportion threshold.
6. A data storage device, comprising:
the data fragment dividing module is used for dividing the data to be stored into various data fragments;
the backup number determining module is used for determining the backup number of each data fragment;
the first determining module is used for determining a level 1 storage node corresponding to an s-th data fragment in a peer-to-peer network and sending a data storage request to the level 1 storage node, wherein the data storage request comprises a count value and a storage node set, the count value is equal to the backup number of the s-th data fragment, the storage node set is empty, s is more than or equal to 1 and less than or equal to SN, and SN is the number of the data fragments;
the first processing module is used for downloading and storing the s-th data fragment from the source node after receiving the data storage request, adding a local node identifier into the storage node set, and subtracting a counting unit from the counting value;
the second determining module is used for determining a next-level storage node corresponding to the s-th data fragment in the point-to-point network and sending a data storage request to the next-level storage node;
and the second processing module is used for simultaneously downloading and storing the s-th data fragment from each target node by adopting a concurrent downloading mode after receiving the data storage request, adding a local node identifier into the storage node set, and subtracting a counting unit from the counting value, wherein the target node is a node corresponding to each node identifier in the storage node set of the data storage request.
7. The data storage device of claim 6, wherein the first determining module comprises:
the fragment hash value calculation unit is used for calculating the hash value of the s-th data fragment by using a preset first hash function;
the target position point determining unit is used for determining a target position point from a position point corresponding to the hash value of the s-th data fragment on a preset hash ring, wherein the hash ring is a circular ring formed by sequentially connecting all function values of the first hash function from small to large in a clockwise or anticlockwise mode, namely each position point on the hash ring corresponds to one function value of the first hash function;
a reference position point obtaining unit, configured to obtain each reference position point on the hash ring, where each reference position point is a position point corresponding to a node in the peer-to-peer network;
and the reference position point searching unit is used for searching the reference position point clockwise or anticlockwise from the target position point and determining the node corresponding to the searched first reference position point as the level 1 storage node corresponding to the s-th data fragment.
8. The data storage device of claim 7, wherein the first determining module further comprises:
an identity obtaining unit, configured to obtain an identity of each node in the peer-to-peer network;
the hash operation unit is used for performing hash operation on the identity of each node in the point-to-point network by using a preset second hash function to obtain a hash value of each node, and the value range of the second hash function is consistent with the value range of the first hash function;
and the reference position point determining unit is used for determining a position point corresponding to the hash value of the mth node on the hash ring as a reference position point corresponding to the mth node, wherein m is more than or equal to 1 and is less than or equal to NodesNum.
9. A computer readable storage medium storing computer readable instructions, which when executed by a processor implement the functionality of a data storage device as claimed in any one of claims 6 to 8.
10. A terminal device comprising a memory, a processor and computer readable instructions stored in the memory and executable on the processor, wherein the processor when executing the computer readable instructions implements the functionality of the data storage apparatus as claimed in any one of claims 6 to 8.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910347818.3A CN110099112B (en) | 2019-04-28 | 2019-04-28 | Data storage method, device, medium and terminal equipment based on point-to-point network |
| PCT/CN2019/103295 WO2020220540A1 (en) | 2019-04-28 | 2019-08-29 | Point-to-point network-based data storage method and apparatus, medium and terminal device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910347818.3A CN110099112B (en) | 2019-04-28 | 2019-04-28 | Data storage method, device, medium and terminal equipment based on point-to-point network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110099112A CN110099112A (en) | 2019-08-06 |
| CN110099112B true CN110099112B (en) | 2022-03-29 |
Family
ID=67446019
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910347818.3A Active CN110099112B (en) | 2019-04-28 | 2019-04-28 | Data storage method, device, medium and terminal equipment based on point-to-point network |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN110099112B (en) |
| WO (1) | WO2020220540A1 (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110099112B (en) * | 2019-04-28 | 2022-03-29 | 平安科技（深圳）有限公司 | Data storage method, device, medium and terminal equipment based on point-to-point network |
| CN112231398A (en) * | 2020-09-25 | 2021-01-15 | 北京金山云网络技术有限公司 | Data storage method, device, equipment and storage medium |
| CN113794558B (en) * | 2021-09-16 | 2024-02-27 | 烽火通信科技股份有限公司 | L-tree calculation method, device and system in XMS algorithm |
| CN116309002B (en) * | 2022-05-19 | 2024-03-01 | 北京百度网讯科技有限公司 | Graph data storage, access and processing methods, training methods, equipment and media |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2202945A1 (en) * | 2008-12-26 | 2010-06-30 | Huawei Technologies Co., Ltd. | Distributed network construction and storage method, apparatus, and system |
| CN107547657A (en) * | 2017-09-27 | 2018-01-05 | 柏科数据技术（深圳）股份有限公司 | A kind of method, apparatus and storage medium numbered based on one point data in cloud storage system |
| CN109388351A (en) * | 2018-12-18 | 2019-02-26 | 平安科技（深圳）有限公司 | A kind of method and relevant apparatus of Distributed Storage |
Family Cites Families (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103136114B (en) * | 2011-11-30 | 2015-11-25 | 华为技术有限公司 | Storage means and memory storage |
| CN103634401B (en) * | 2013-12-03 | 2015-11-25 | 北京京东尚科信息技术有限公司 | A kind of store data trnascription method and terminal installation and server unit |
| CN105511801B (en) * | 2015-11-12 | 2018-11-16 | 长春理工大学 | The method and apparatus of data storage |
| CN107135661A (en) * | 2016-12-26 | 2017-09-05 | 深圳前海达闼云端智能科技有限公司 | Data processing method, device, system and information collecting device |
| CN107404541B (en) * | 2017-09-20 | 2020-08-04 | 苏州浪潮智能科技有限公司 | Method and system for selecting neighbor node in peer-to-peer network transmission |
| CN107678688A (en) * | 2017-09-27 | 2018-02-09 | 柏域信息科技（上海）有限公司 | A kind of method, apparatus and storage medium of the managing redundant copy based in cloud storage system |
| CN108062419B (en) * | 2018-01-06 | 2021-04-20 | 深圳市网心科技有限公司 | File storage method, electronic equipment, system and medium |
| CN108681583A (en) * | 2018-05-11 | 2018-10-19 | 北京奇虎科技有限公司 | Data proof of possession method, apparatus and readable storage medium storing program for executing based on block chain |
| GB2574076B (en) * | 2018-09-21 | 2022-07-13 | Nationwide Building Soc | Distributed data storage |
| CN110099112B (en) * | 2019-04-28 | 2022-03-29 | 平安科技（深圳）有限公司 | Data storage method, device, medium and terminal equipment based on point-to-point network |
-
2019
- 2019-04-28 CN CN201910347818.3A patent/CN110099112B/en active Active
- 2019-08-29 WO PCT/CN2019/103295 patent/WO2020220540A1/en active Application Filing
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2202945A1 (en) * | 2008-12-26 | 2010-06-30 | Huawei Technologies Co., Ltd. | Distributed network construction and storage method, apparatus, and system |
| CN107547657A (en) * | 2017-09-27 | 2018-01-05 | 柏科数据技术（深圳）股份有限公司 | A kind of method, apparatus and storage medium numbered based on one point data in cloud storage system |
| CN109388351A (en) * | 2018-12-18 | 2019-02-26 | 平安科技（深圳）有限公司 | A kind of method and relevant apparatus of Distributed Storage |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110099112A (en) | 2019-08-06 |
| WO2020220540A1 (en) | 2020-11-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110099112B (en) | Data storage method, device, medium and terminal equipment based on point-to-point network | |
| CN108769111B (en) | Server connection method, computer readable storage medium and terminal device | |
| CN110149373B (en) | Data storage method, device, medium and server based on peer-to-peer network | |
| CN109831487B (en) | Fragmented file verification method and terminal equipment | |
| CN107832062B (en) | Program updating method and terminal equipment | |
| US11627207B2 (en) | Systems and methods for data deduplication by generating similarity metrics using sketch computation | |
| CN110659151B (en) | Data verification method and device and storage medium | |
| CN110347651B (en) | Cloud storage-based data synchronization method, device, equipment and storage medium | |
| CN110543448A (en) | data synchronization method, device, equipment and computer readable storage medium | |
| CN109522314B (en) | Block chain-based data archiving method and terminal equipment | |
| US20210158353A1 (en) | Methods, systems, apparatuses, and devices for processing request in consortium blockchain | |
| CN109542462B (en) | System environment deployment method, storage medium and server | |
| CN111988419A (en) | File uploading method, file downloading method, file uploading device, file downloading device, computer equipment and storage medium | |
| CN112069169B (en) | Block data storage method and device, electronic equipment and readable storage medium | |
| CN113282941A (en) | Method and device for acquiring object identification, electronic equipment and storage medium | |
| CN106685429B (en) | Integer compression method and device | |
| GB2572537A (en) | Generating or obtaining an updated neural network | |
| CN113268439A (en) | Memory address searching method and device, electronic equipment and storage medium | |
| EP4078340A1 (en) | Systems and methods for sketch computation | |
| US11995050B2 (en) | Systems and methods for sketch computation | |
| CN113542405A (en) | Block chain-based network communication system, method, device and storage medium | |
| CN112929424A (en) | Gateway load balancing method, device, equipment and storage medium | |
| US20210191640A1 (en) | Systems and methods for data segment processing | |
| CN109165135B (en) | Data management method, computer readable storage medium and terminal equipment | |
| CN112905542A (en) | File processing method, device, server, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |