CN110430282B - Data transmission method and device between nodes of distributed system - Google Patents
Data transmission method and device between nodes of distributed system Download PDFInfo
- Publication number
- CN110430282B CN110430282B CN201910760714.5A CN201910760714A CN110430282B CN 110430282 B CN110430282 B CN 110430282B CN 201910760714 A CN201910760714 A CN 201910760714A CN 110430282 B CN110430282 B CN 110430282B
- Authority
- CN
- China
- Prior art keywords
- data
- nodes
- calling
- target
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 51
- 230000005540 biological transmission Effects 0.000 title claims abstract description 26
- 230000004044 response Effects 0.000 claims abstract description 48
- 238000012545 processing Methods 0.000 claims abstract description 16
- 238000004590 computer program Methods 0.000 claims description 10
- 238000004891 communication Methods 0.000 claims description 8
- 238000012546 transfer Methods 0.000 claims description 6
- 230000006870 function Effects 0.000 description 7
- 230000000977 initiatory effect Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 238000011161 development Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/55—Push-based network services
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a data transmission method and a device among nodes of a distributed system, wherein the data transmission method comprises the following steps: receiving a calling data packet pushed by the Leader node, and storing a calling data abstract; judging whether the calling data packet further comprises calling data, wherein the calling data corresponds to the calling data abstract; if not, sending a first data query request to a plurality of first target Follower nodes according to a first preset rule, wherein the first data query request is used for requiring the plurality of first target Follower nodes to query whether the first target Follower nodes contain the calling data; and processing the first response information returned by the first target Follower node according to a second preset rule. The invention solves the problem that in the prior art, in a block chain application scene, the strength of pushing data by a Leader node is high and the whole distributed system bottleneck is easily formed.
Description
Technical Field
The present invention relates to the field of data communication application technologies, and in particular, to a method and an apparatus for transmitting data between nodes in a distributed system.
Background
With the development and popularization of internet technology, the coverage population of internet services is more and more extensive, and the coverage scenes are more and more. In particular, the development of mobile internet technology has enabled people to enjoy internet services anytime and anywhere, while also continuously generating personal data. With the dramatic increase of data volume, the computing power of a single computer cannot meet the data processing requirements, and the computer begins to develop towards miniaturization and clustering. Computers in the cluster are in time division and cooperation with each other to form a distributed system, and the problem of data consistency in the distributed system is the primary problem to be solved by the distributed technology.
In order to solve the problem of data consistency in the distributed system, namely, each node in the distributed system is required to achieve uniform consensus on data, a series of consensus algorithms are born and developed, and the algorithms have different fault tolerance, different performances and different application scenes. The Paxos algorithm (a consistency algorithm based on message passing) is very pioneering and representative in the consensus algorithm, but is the biggest disadvantage of the Paxos algorithm, which is difficult to understand and implement. Thus, a well-known Raft algorithm (a consensus algorithm) was proposed that, like the Paxos algorithm, could also be formalized to completely solve the consensus problem. The Raft algorithm requires that a Leader node is selected from all nodes participating in consensus, then the Leader node is responsible for pushing data to other nodes, and the nodes receiving the data pushed by the Leader node are collectively called Follower. And if the data of the Follower node is inconsistent with the data pushed by the Leader node, the data pushed by the Leader node is received and approved by the Follower node, so that the consensus is realized. The consensus method is that a Leader node pushes data to all follow nodes, and as the number of nodes participating in consensus increases, the network pressure of the Leader node becomes greater and greater, and becomes a bottleneck of the whole system. When the Raft algorithm is applied to a block chain, data pushed to a Follower node by a Leader node each time is data of a complete block, so that the data pushing force of the Leader node is large in a block chain application scene, and the problem needs to be solved urgently.
Disclosure of Invention
The embodiment of the invention provides a data transmission method and device among nodes of a distributed system, and aims to solve the problem that in the prior art, in a block chain application scene, the strength of a Leader node for pushing data is high and the Leader node is easy to become a bottleneck of the whole distributed system.
In order to solve the above technical problem, a first technical solution adopted in the embodiments of the present invention is as follows:
a data transmission method between nodes of a distributed system, the distributed system comprises a Leader node and a plurality of Follower nodes which are in data communication with each other, and the method comprises the following steps: receiving a calling data packet pushed by the Leader node, and storing a calling data abstract, wherein the calling data packet comprises the calling data abstract; judging whether the calling data packet further comprises calling data, wherein the calling data corresponds to the calling data abstract; if not, sending a first data query request to a plurality of first target Follower nodes according to a first preset rule, wherein the first data query request is used for requiring the plurality of first target Follower nodes to query whether the first target Follower nodes contain the call data or not, and the total number of the plurality of first target Follower nodes is smaller than that of the Follower nodes; processing first response information returned by the first target Follower node according to a second preset rule, wherein the first response information comprises information that the first target Follower node contains the calling data; the Leader node periodically pushes the calling data packet to all the Follower nodes, all the Follower nodes receive the calling data abstract, and only part of the Follower nodes receive the calling data.
Optionally, the processing, according to a second preset rule, first response information returned by the first target Follower node includes: taking one first target Follower node which returns the first response information firstly as a second target Follower node; sending a data calling request to the second target Follower node, wherein the data calling request is used for requiring the second target Follower node to return the calling data; and receiving the calling data returned by the second target Follower node.
Optionally, before sending the data call request to the second target Follower node, the method further includes: judging whether second response information returned by the second target Follower node is received or not, wherein the second response information comprises information that the second target Follower node contains the calling data; and if so, sending a data calling request to the second target Follower node, wherein the data calling request comprises information for requiring the second target Follower node to return the calling data.
Optionally, after receiving the call data returned by the second target Follower node, the method includes: calculating a first hash value corresponding to the received calling data; acquiring a second hash value corresponding to the calling data abstract, and judging whether the first hash value and the second hash value are the same; if yes, the received call data is saved to the local.
Optionally, when the first hash value and the second hash value are not the same, the method further includes: and sending the first data query request to a plurality of first target Follower nodes again according to a first preset rule.
Optionally, when the received call data packet includes call data, the method includes: storing the calling data, and judging whether the first data query request sent by a plurality of third target Follower nodes is received or not, wherein the total number of the third target Follower nodes is smaller than that of the Follower nodes; if so, sending third response information to a plurality of corresponding third target Follower nodes, wherein the third response information comprises information that a sender contains the calling data; judging whether a second data calling request sent by the third target Follower node is received, wherein the second data calling request comprises information for requiring a sender to return the calling data; and if so, sending the calling data to the corresponding third target Follower node.
Optionally, if the number of the Follower nodes is N (N >0), and the number of the Follower nodes of the part that receives the call data is N, N and N satisfy the following relationship: when N is less than or equal to 2, N is N; when 2< N.ltoreq.11, N ═ N-2)/3+ 2; when N >11, N is log2(N-11) + 5.
In order to solve the above technical problem, a second technical solution adopted in the embodiments of the present invention is as follows:
an apparatus for data transmission between nodes of a distributed system, the distributed system including a Leader node and a plurality of Follower nodes in data communication with each other, comprising: the receiving module is used for receiving a calling data packet pushed by the Leader node and storing a calling data abstract, wherein the calling data packet comprises the calling data abstract; the judging module is used for judging whether the calling data packet further comprises calling data, and the calling data corresponds to the calling data abstract; a sending module, configured to send a first data query request to a plurality of first target nodes according to a first preset rule when the call data packet does not include call data, where the first data query request is used to request the plurality of first target nodes to query whether the first target nodes themselves include the call data, where a total number of the plurality of first target nodes is smaller than a total number of the first target nodes; a processing module, configured to process, according to a second preset rule, first response information returned by the first target Follower node, where the first response information includes information that the first target Follower node contains the call data; the Leader node periodically pushes the calling data packet to all the Follower nodes, all the Follower nodes receive the calling data abstract, and only part of the Follower nodes receive the calling data.
In order to solve the above technical problem, a third technical solution adopted in the embodiments of the present invention is as follows:
a computer-readable storage medium, on which a computer program is stored, which, when executed, implements a method of data transmission between distributed system nodes as described above.
In order to solve the above technical problem, a fourth technical solution adopted in the embodiments of the present invention is as follows:
a computer apparatus comprising a processor, a memory and a computer program stored on the memory and executable on the processor, the processor implementing a method of data transfer between nodes of a distributed system as described above when executing the computer program.
The embodiment of the invention has the beneficial effects that: different from the situation in the prior art, the embodiment of the present invention receives the call data packet pushed by the Leader node, stores the call data digest, determines whether the call data packet further includes call data, if not, sends a first data query request to a plurality of first target nodes according to a first preset rule, and finally processes the first response information returned by the first target nodes according to a second preset rule, so as to solve the problem that in the prior art, in a block chain application scenario, the strength of the Leader node for pushing data is large, and the whole distributed system is easily changed into a bottleneck.
Drawings
Fig. 1 is a flowchart illustrating an implementation of a data transmission method between nodes in a distributed system according to a first embodiment of the present invention;
fig. 2 is a partial structural framework diagram of an embodiment of a data transmission apparatus between nodes of a distributed system according to a second embodiment of the present invention;
FIG. 3 is a partial structural framework diagram of an embodiment of a computer-readable storage medium according to a third embodiment of the present invention;
fig. 4 is a partial structural framework diagram of an embodiment of a computer device according to a fourth embodiment of the present invention.
Detailed Description
Example one
Referring to fig. 1, fig. 1 is a flowchart illustrating an implementation of a data transmission method between nodes of a distributed system according to an embodiment of the present invention, and may be obtained by referring to fig. 1, where the distributed system includes a Leader node and multiple nodes that perform data communication with each other, and the method includes:
step S101: and receiving a calling data packet pushed by the Leader node, and storing a calling data abstract, wherein the calling data packet comprises the calling data abstract. In this step, the data transmission method is described with one of the Follower nodes as the executing party of the data transmission method, that is, from the perspective of standing at one of the Follower nodes.
Step S102: and judging whether the calling data packet further comprises calling data or not, wherein the calling data corresponds to the calling data abstract.
Step S103: if the calling data packet does not include calling data, sending a first data query request to a plurality of first target Follower nodes according to a first preset rule, where the first data query request is used to request the plurality of first target Follower nodes to query whether the first target Follower nodes themselves include the calling data, and a total number of the plurality of first target Follower nodes is smaller than a total number of the Follower nodes.
Step S104: and processing first response information returned by the first target Follower node according to a second preset rule, wherein the first response information comprises information that the first target Follower node contains the call data.
The Leader node periodically pushes the calling data packet to all the Follower nodes, all the Follower nodes receive the calling data abstract, and only part of the Follower nodes receive the calling data.
In this embodiment, optionally, processing, according to a second preset rule, first response information returned by the first target Follower node includes:
first, one first target Follower node that returns the first response information first is used as a second target Follower node.
Secondly, sending a data calling request to the second target follow node, where the data calling request is used to request the second target follow node to return the calling data.
And thirdly, receiving the call data returned by the second target Follower node.
In this embodiment, optionally, before sending the data call request to the second target Follower node, the method further includes:
first, whether second response information returned by the second target Follower node is received or not is judged, and the second response information includes information that the second target Follower node includes the call data.
Secondly, if second response information returned by the second target follow node is received, sending a data call request to the second target follow node, where the data call request includes information that requires the second target follow node to return the call data.
In this embodiment, optionally, after receiving the call data returned by the second target Follower node, the method includes:
firstly, calculating a first hash value corresponding to the received call data.
Secondly, acquiring a second hash value corresponding to the calling data abstract, and judging whether the first hash value and the second hash value are the same;
thirdly, if the first hash value is the same as the second hash value, the received call data is saved locally.
In this embodiment, the Hash value is also called Hash Function (or Hash algorithm, also called Hash Function, english: Hash Function) and is a method for creating a small digital "fingerprint" from any kind of data. The hash function compresses a message or data into a digest so that the amount of data becomes small, fixing the format of the data. This function mixes the data in a hash, recreating a fingerprint called a hash value (hash sums, or hashes). The hash value is typically represented by a short string of random letters and numbers. Good hash functions rarely have hash collisions in the input domain. In hash tables and data processing, data is distinguished without suppressing conflicts, making database records more difficult to find.
In this embodiment, optionally, when the first hash value and the second hash value are not the same, the method further includes:
and sending the first data query request to a plurality of first target Follower nodes again according to a first preset rule.
In this embodiment, optionally, when the received call data packet includes call data, the method includes:
first, the call data is saved, and whether the first data query request sent by a plurality of third target nodes is received or not is judged, wherein the total number of the plurality of third target nodes is smaller than the total number of the nodes.
Secondly, if the first data query requests sent by a plurality of third target nodes are received, sending third response information to the corresponding plurality of third target nodes, where the third response information includes information that a sender includes the call data.
And thirdly, judging whether a second data call request sent by the third target follow node is received, wherein the second data call request comprises information for requiring the sender to return the call data.
Fourthly, if a second data call request sent by the third target Follower node is received, sending the call data to the corresponding third target Follower node.
In this embodiment, optionally, if the number of the Follower nodes is N (N >0), and the number of the Follower nodes of the part that receives the call data is N, N and N satisfy the following relationship: when N is less than or equal to 2, N is N; when 2< N.ltoreq.11, N ═ N-2)/3+ 2; when N >11, N is log2(N-11) + 5.
The following is a detailed description of the entire process from the Raft algorithm to the inventive scheme:
in the native Raft algorithm, there are only three RPCs (remote procedure calls) for communication between the participating nodes:
requestnote (request vote): when the Leader node is not determined in the system, the participating nodes initiate the call to all participating nodes except the participating nodes to request to vote for the participating nodes. If one of the participating nodes gets more than half the total number of votes it will become a Leader node in the system. If no participating node gets more than half the total number of participating nodes, the next round of election is resumed. Whether the participating node votes to another participating node or not is restricted by certain conditions, and the system is ensured to have at most one Leader node at any time and the data state of the Leader node is not older than that of any other participating node.
Appendix entries: after the Leader node in the system is determined, the Leader node starts to periodically initiate the call to all the participating nodes except the Leader node, and pushes data to all the participating nodes. The Leader node maintains the data synchronization state of each Follower node, and sends appropriate incremental data to each Follower node. And each Follower node checks whether own data conflicts with the received incremental data according to a determined rule, and if so, the conflicting data is rolled back, and then the received data is adopted.
Installsnaphot (install snapshot): the value of a certain item of data in the system may be updated repeatedly as the system runs, and after the latest value of the item is saved, all the historical values of the item can be deleted, so as to save the storage space. The set of current values for all items of data in the system, i.e., the current snapshot of the system. All participating nodes in the system can independently take a snapshot according to certain needs and rules, and delete all state data before the snapshot. If some Follower nodes in the system have data states far behind those of the Leader nodes for some reasons, the Leader nodes will initiate the call, and directly push the latest snapshot to the Follower nodes, and ignore the history data in the period. Due to the characteristics of blockchains, the snapshot function is not implemented in blockchain applications.
From the above description, it can be seen that the Leader node pushes data by initiating an appendix calls to all the Follower nodes, and as the total number of participating nodes increases, the amount of data pushed by the Leader node increases linearly. Therefore, in order to solve the problem that the data volume pushed by the Leader node is increased linearly, P2P expansion is carried out on the Raft algorithm, the P2P expansion needs to modify the processing logic of a calling party and a called party called by the appendix entries, and in addition, two RPCs need to be added. The specific scheme is as follows:
firstly, the modification of the application entries calling processing logic is to add a 'data summary' field in the application entries calling data packet, wherein the field stores the hash value of the data to be pushed in the calling, and the field is a necessary field. The 'data' field in the original request is changed from optional to optional. When initiating the application entries calling, the Leader node calculates the value of the data summary field according to the data field, for the data field, a certain number of application entries calling data packets are attached with the data field, and the rest of application entries calling data packets leave the data field empty. The number of the application entries call packets with the "data" field is determined by a certain rule. Assuming that the number of Follower nodes is N (N >0) and the number of AppendEntries call packets with "data" field is N, then: if N is less than or equal to 2, then N is N; if 2< N.ltoreq.11, N ═ N-2)/3+ 2; if N >11, then N is log2(N-11) + 5.
When receiving the AppendEntries call data packet, the Follower node stores a data summary field therein, if the request is attached with the data field, the pushed data is stored according to the original logic, otherwise, the data is acquired in a P2P (point-to-point) mode.
Secondly, adding query entries (query entry) calls, namely the query entries calls are initiated by the Follower node and used for querying other participating nodes whether the participating nodes have missing data. The following steps are that a Follower node randomly initiates a request to a certain number of other participating nodes, a request receiving node inquires whether the request receiving node has data required by the request, and then response is carried out according to the inquiry result. After the request initiating node receives the response, if the response indicates that the request initiating node has corresponding missing data, the initiating node requests the response node to transmit the corresponding data. The relationship between the number of query entries calling data packets initiated at one time and the total number of participating nodes can be continued by using the above-described relationship between the number of requests with "data" fields appended to the application entries calling data packets and the total number of participating nodes.
Thirdly, a transfer entries call is newly added, namely the transfer entries call is initiated by the folower node and is used for requesting missing data of the transfer entries call to other participating nodes. And after the Follower node receives the effective response of the query entries call, initiating a transfer entries call data packet to the response node. And the TransferEntries calling the data packet receiving node attaches the corresponding data to the TransferEntries response and returns the data. And calculating the hash value of the data attached in the response by the TransferEntries response receiving node, comparing the hash value with a corresponding data summary field stored locally, if the hash value is consistent with the data summary field, storing the data according to the original logic, and if the hash value is inconsistent with the data summary field, re-initiating QueryEntries call, and continuously trying to acquire the data in a P2P mode.
The embodiment of the invention receives the calling data packet pushed by the Leader node, stores the abstract of the calling data, judges whether the calling data packet also comprises the calling data, if not, sends a first data query request to a plurality of first target Follower nodes according to a first preset rule, and finally processes the first response information returned by the first target Follower nodes according to a second preset rule, thereby solving the problem that in the prior art, under the scene of block chain application, the strength of pushing data by the Leader node is larger, and the whole distributed system is easy to become a bottleneck.
On the premise of ensuring the correctness and completeness of the Raft algorithm, the invention performs P2P (point-to-point) transmission on the consensus data among the Follower nodes, thereby greatly reducing the network pressure of the Leader node, avoiding the Leader node from becoming the bottleneck of the whole system, and performing independent scheduling during transmission can not influence the overall performance of the system because of too frequent scheduling. In the application and extension of the Raft algorithm, a Leader node pushes a data summary to a Follower node, and data itself is transmitted in a P2P manner among all participating nodes.
The invention carries out P2P expansion on the Raft algorithm, and leads the nodes participating in consensus to adopt the mode of P2P to transmit data on the premise of ensuring the correctness and completeness of the Raft algorithm, thereby achieving the following effects:
1. the overall data synchronization and consensus efficiency of the distributed system is improved.
2. The network pressure of the Leader node in the distributed system is greatly reduced.
3. The performance bottleneck that the Leader node becomes a distributed system is avoided.
4. On the premise of ensuring the performance, the distributed system supports more participating nodes.
Example two
Referring to fig. 2, fig. 2 is a partial structural framework diagram of an embodiment of an apparatus for data transmission between nodes of a distributed system according to an embodiment of the present invention, and as can be obtained by referring to fig. 2, an apparatus 100 for data transmission between nodes of a distributed system includes a Leader node and a plurality of nodes in data communication with each other, and includes:
a receiving module 110, configured to receive a call data packet pushed by the Leader node, and store a call data digest, where the call data packet includes the call data digest.
A determining module 120, configured to determine whether the call data packet further includes call data, where the call data corresponds to the call data digest.
A sending module 130, configured to send a first data query request to a plurality of first target nodes according to a first preset rule when the invoking data packet does not include invoking data, where the first data query request is used to request the plurality of first target nodes to query whether the first target nodes themselves include the invoking data, and a total number of the plurality of first target nodes is smaller than a total number of the first target nodes.
And the processing module 140 is configured to process, according to a second preset rule, first response information returned by the first target Follower node, where the first response information includes information that the first target Follower node contains the call data.
The Leader node periodically pushes the calling data packet to all the Follower nodes, all the Follower nodes receive the calling data abstract, and only part of the Follower nodes receive the calling data.
The embodiment of the invention receives the calling data packet pushed by the Leader node, stores the abstract of the calling data, judges whether the calling data packet also comprises the calling data, if not, sends a first data query request to a plurality of first target Follower nodes according to a first preset rule, and finally processes the first response information returned by the first target Follower nodes according to a second preset rule, thereby solving the problem that in the prior art, under the scene of block chain application, the strength of pushing data by the Leader node is larger, and the whole distributed system is easy to become a bottleneck.
EXAMPLE III
Referring to fig. 3, a computer-readable storage medium 10 according to an embodiment of the present invention can be seen, where the computer-readable storage medium 10 includes: ROM/RAM, magnetic disks, optical disks, etc., on which a computer program 11 is stored, which computer program 11, when executed, implements a method of data transmission between distributed system nodes as described in an embodiment one. Since the data transmission method between the nodes of the distributed system has been described in detail in the first embodiment, the description is not repeated here.
According to the data transmission method between the nodes of the distributed system, which is realized by the embodiment of the invention, by receiving the calling data packet pushed by the Leader node, storing the abstract of the calling data, judging whether the calling data packet also comprises the calling data, if not, sending a first data query request to a plurality of first target Follower nodes according to a first preset rule, and finally, processing the first response information returned by the first target Follower nodes according to a second preset rule, the problem that in the prior art, under the scene of block chain application, the strength of the Leader node for pushing the data is large, and the Leader node is easy to become the bottleneck of the whole distributed system is solved.
Example four
Referring to fig. 4, a computer device 20 according to an embodiment of the present invention includes a processor 21, a memory 22, and a computer program 221 stored in the memory 22 and running on the processor 21, where the processor 21 executes the computer program 221 to implement the data transmission method between nodes of a distributed system according to an embodiment. Since the data transmission method between the nodes of the distributed system has been described in detail in the first embodiment, the description is not repeated here.
According to the data transmission method between the nodes of the distributed system, which is realized by the embodiment of the invention, by receiving the calling data packet pushed by the Leader node, storing the abstract of the calling data, judging whether the calling data packet also comprises the calling data, if not, sending a first data query request to a plurality of first target Follower nodes according to a first preset rule, and finally, processing the first response information returned by the first target Follower nodes according to a second preset rule, the problem that in the prior art, under the scene of block chain application, the strength of the Leader node for pushing the data is large, and the Leader node is easy to become the bottleneck of the whole distributed system is solved.
The above description is only an embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes performed by the present specification and drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (9)
1. A data transmission method between nodes of a distributed system, wherein the distributed system comprises a Leader node and a plurality of Follower nodes which are in data communication with each other, and the method is characterized by comprising the following steps:
receiving a calling data packet pushed by the Leader node, and storing a calling data abstract, wherein the calling data packet comprises the calling data abstract;
judging whether the calling data packet further comprises calling data, wherein the calling data corresponds to the calling data abstract;
if not, sending a first data query request to a plurality of first target Follower nodes according to a first preset rule, wherein the first data query request is used for requiring the plurality of first target Follower nodes to query whether the first target Follower nodes contain the call data or not, and the total number of the plurality of first target Follower nodes is smaller than that of the Follower nodes;
processing first response information returned by the first target Follower node according to a second preset rule, wherein the first response information comprises information that the first target Follower node contains the calling data;
after the first target Follerwer node returns a response, if the response has corresponding missing data, the first target Follower node is requested to transmit the corresponding data;
the Leader node periodically pushes the calling data packet to all the Follower nodes, all the Follower nodes receive the calling data abstract, and only part of the Follower nodes receive the calling data.
2. The method according to claim 1, wherein the processing, according to a second preset rule, the first response information returned by the first target Follower node includes:
taking one first target Follower node which returns the first response information firstly as a second target Follower node;
sending a data calling request to the second target Follower node, wherein the data calling request is used for requiring the second target Follower node to return the calling data;
and receiving the calling data returned by the second target Follower node.
3. The method according to claim 2, wherein after receiving the call data returned by the second target Follower node, the method includes:
calculating a first hash value corresponding to the received calling data;
acquiring a second hash value corresponding to the calling data abstract, and judging whether the first hash value and the second hash value are the same;
if yes, the received call data is saved to the local.
4. The method of claim 3, wherein when the first hash value and the second hash value are not the same, further comprising:
and sending the first data query request to a plurality of first target Follower nodes again according to a first preset rule.
5. The method of claim 1, wherein when the received call packet includes call data, the method further comprises:
storing the calling data, and judging whether the first data query request sent by a plurality of third target Follower nodes is received or not, wherein the total number of the third target Follower nodes is smaller than that of the Follower nodes;
if so, sending third response information to a plurality of corresponding third target Follower nodes, wherein the third response information comprises information that a sender contains the calling data;
judging whether a second data calling request sent by the third target Follower node is received, wherein the second data calling request comprises information for requiring a sender to return the calling data;
and if so, sending the calling data to the corresponding third target Follower node.
6. The method according to claim 1, wherein the number of the nodes of the fowler is N (N >0), and the number of the nodes of the fowler receiving the call data is N, then N and N satisfy the following relationship:
when N is less than or equal to 2, N is N;
when 2< N.ltoreq.11, N ═ N-2)/3+ 2;
when N >11, N is log2(N-11) + 5.
7. A data transmission apparatus between nodes of a distributed system, the distributed system including a Leader node and a plurality of Follower nodes that perform data communication with each other, the apparatus comprising:
the receiving module is used for receiving a calling data packet pushed by the Leader node and storing a calling data abstract, wherein the calling data packet comprises the calling data abstract;
the judging module is used for judging whether the calling data packet further comprises calling data, and the calling data corresponds to the calling data abstract;
a sending module, configured to send a first data query request to a plurality of first target nodes according to a first preset rule when the call data packet does not include call data, where the first data query request is used to request the plurality of first target nodes to query whether the first target nodes themselves include the call data, where a total number of the plurality of first target nodes is smaller than a total number of the first target nodes;
a processing module, configured to process, according to a second preset rule, first response information returned by the first target Follower node, where the first response information includes information that the first target Follower node contains the call data;
after the first target Follerwer node returns a response, if the response has corresponding missing data, the first target Follower node is requested to transmit the corresponding data;
the Leader node periodically pushes the calling data packet to all the Follower nodes, all the Follower nodes receive the calling data abstract, and only part of the Follower nodes receive the calling data.
8. A computer-readable storage medium, having stored thereon a computer program which, when executed, implements a method of data transfer between distributed system nodes according to any of claims 1 to 6.
9. A computer device comprising a processor, a memory, and a computer program stored on the memory and executable on the processor, the processor implementing a method of data transmission between distributed system nodes according to any one of claims 1 to 6 when executing the computer program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910760714.5A CN110430282B (en) | 2019-08-16 | 2019-08-16 | Data transmission method and device between nodes of distributed system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910760714.5A CN110430282B (en) | 2019-08-16 | 2019-08-16 | Data transmission method and device between nodes of distributed system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110430282A CN110430282A (en) | 2019-11-08 |
| CN110430282B true CN110430282B (en) | 2022-03-01 |
Family
ID=68415143
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910760714.5A Active CN110430282B (en) | 2019-08-16 | 2019-08-16 | Data transmission method and device between nodes of distributed system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110430282B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111030930B (en) * | 2019-12-02 | 2022-02-01 | 北京众享比特科技有限公司 | Decentralized network data fragment transmission method, device, equipment and medium |
| CN111324305B (en) * | 2020-02-16 | 2021-02-02 | 西安奥卡云数据科技有限公司 | Data writing/reading method in distributed storage system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109067849A (en) * | 2018-07-11 | 2018-12-21 | 杨国超 | Method of data synchronization based on block |
| CN109327548A (en) * | 2018-11-27 | 2019-02-12 | 北京瑞卓喜投科技发展有限公司 | A kind of block chain update method and block chain more new system |
| CN109379397A (en) * | 2018-08-31 | 2019-02-22 | 阿里巴巴集团控股有限公司 | Transaction common recognition processing method and processing device, electronic equipment based on block chain |
| CN109522362A (en) * | 2018-10-17 | 2019-03-26 | 北京瑞卓喜投科技发展有限公司 | Incomplete markets synchronous method, system and equipment based on block chain data |
| WO2019081917A1 (en) * | 2017-10-24 | 2019-05-02 | Copa Fin Limited | Data storage and verification |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10977260B2 (en) * | 2016-09-26 | 2021-04-13 | Splunk Inc. | Task distribution in an execution node of a distributed execution environment |
| US10795884B2 (en) * | 2016-09-26 | 2020-10-06 | Splunk Inc. | Dynamic resource allocation for common storage query |
-
2019
- 2019-08-16 CN CN201910760714.5A patent/CN110430282B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019081917A1 (en) * | 2017-10-24 | 2019-05-02 | Copa Fin Limited | Data storage and verification |
| CN109067849A (en) * | 2018-07-11 | 2018-12-21 | 杨国超 | Method of data synchronization based on block |
| CN109379397A (en) * | 2018-08-31 | 2019-02-22 | 阿里巴巴集团控股有限公司 | Transaction common recognition processing method and processing device, electronic equipment based on block chain |
| CN109522362A (en) * | 2018-10-17 | 2019-03-26 | 北京瑞卓喜投科技发展有限公司 | Incomplete markets synchronous method, system and equipment based on block chain data |
| CN109327548A (en) * | 2018-11-27 | 2019-02-12 | 北京瑞卓喜投科技发展有限公司 | A kind of block chain update method and block chain more new system |
Non-Patent Citations (2)
| Title |
|---|
| Improving Raft When There Are Failures;Christian Fluri;《2018 Eighth Latin-America Symposium on Dependable Computing》;20190321;全文 * |
| 改进的Raft一致性算法及其研究;陈陆;《江苏科技大学学报（自然科学版）》;20180831;第32卷(第4期);559-563 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110430282A (en) | 2019-11-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7293046B2 (en) | Accounting for update notifications in synchronizing data that may be represented by different data structures | |
| CN110119292B (en) | System operation parameter query method, matching method, device and node equipment | |
| CN110430282B (en) | Data transmission method and device between nodes of distributed system | |
| CN103780679A (en) | Long time delay remote invocation method based on HTTP protocol | |
| CN112069169B (en) | Block data storage method and device, electronic equipment and readable storage medium | |
| CN110099106B (en) | Method and terminal for data synchronization between nodes in block chain | |
| CN113127564B (en) | Parameter synchronization method and device | |
| CN112650812A (en) | Data fragment storage method and device, computer equipment and storage medium | |
| CN111124717A (en) | Message delivery method, system and computer storage medium | |
| WO2019201111A1 (en) | Information processing method, apparatus and device, and computer-readable storage medium | |
| CN113934933B (en) | Message forwarding method, device and storage medium with low delay | |
| CN116320011A (en) | Request processing method and device | |
| CN115811515A (en) | File access method and system based on fragments under edge cloud environment | |
| CN115510161A (en) | Data synchronization method, device, equipment and storage medium | |
| CN116095081A (en) | Event processing method and device based on block chain system, equipment and medium | |
| CN113973135A (en) | Data caching processing method and device, caching grid platform and storage medium | |
| CN109254853B (en) | Data sharing method, data sharing system and computer readable storage medium | |
| CN113535415B (en) | Resource acquisition method and terminal | |
| CN113067848B (en) | Call record synchronization method and system and electronic equipment | |
| CN114915652B (en) | Message pushing method and device, electronic equipment and storage medium | |
| CN114567644B (en) | Data synchronization method and device and electronic equipment | |
| CN115022110B (en) | Message distribution method, readable medium and electronic device | |
| Kanemitsu et al. | Lookup parameter optimization for Kademlia DHT alternative in IPFS | |
| CN112445864B (en) | Organization personnel information synchronization method and device, electronic equipment and storage medium | |
| CN111711639B (en) | Terminal, data transmission method, system, and computer-readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: Data transmission methods and devices between nodes in distributed systems Granted publication date: 20220301 Pledgee: China Minsheng Banking Corp Shenzhen branch Pledgor: WEWAY (SHENZHEN) NETWORK TECHNOLOGY CO.,LTD. Registration number: Y2024980020627 |