CN113420323B - Data sharing method and terminal equipment - Google Patents
Data sharing method and terminal equipment Download PDFInfo
- Publication number
- CN113420323B CN113420323B CN202110626699.2A CN202110626699A CN113420323B CN 113420323 B CN113420323 B CN 113420323B CN 202110626699 A CN202110626699 A CN 202110626699A CN 113420323 B CN113420323 B CN 113420323B
- Authority
- CN
- China
- Prior art keywords
- node
- data
- providing
- nodes
- request
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 67
- 230000008569 process Effects 0.000 claims abstract description 29
- 230000003993 interaction Effects 0.000 claims abstract description 24
- 238000011156 evaluation Methods 0.000 claims description 25
- 238000004590 computer program Methods 0.000 claims description 17
- 238000004891 communication Methods 0.000 claims description 8
- 238000012549 training Methods 0.000 claims description 8
- 230000004044 response Effects 0.000 claims description 6
- 230000007704 transition Effects 0.000 claims description 2
- 238000012545 processing Methods 0.000 description 16
- 238000003860 storage Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 230000010076 replication Effects 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000007423 decrease Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000004931 aggregating effect Effects 0.000 description 1
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/64—Protecting data integrity, e.g. using checksums, certificates or signatures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
- G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A30/00—Adapting or protecting infrastructure or their operation
- Y02A30/60—Planning or developing urban green infrastructure
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Computer Security & Cryptography (AREA)
- Bioethics (AREA)
- Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Computer Hardware Design (AREA)
- General Health & Medical Sciences (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Computing Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Information Transfer Between Computers (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention is suitable for the technical field of data interaction, and discloses a data sharing method and terminal equipment, wherein the method comprises the following steps: receiving a sharing request of a data request node, wherein the sharing request comprises model information to be requested; inquiring whether an available model related to the model information to be requested exists in the block chain network; if the available model exists in the block chain network, returning the available model to the data request node; and if no available model exists in the block chain network, selecting a target data providing node cooperating with the data request node, starting federal learning, returning a global model obtained after the federal learning is finished to the data request node, and recording the target data providing node and model parameters in the federal learning process in the block chain network. According to the invention, by combining the block chain and the federal learning, privacy protection and safety credibility in the data sharing process can be realized, data privacy can be completely ensured, and no data leakage risk exists.
Description
Technical Field
The invention belongs to the technical field of data interaction, and particularly relates to a data sharing method and terminal equipment.
Background
The smart city is the best means for realizing accurate and efficient urban management and management service, but with the construction of the smart city, the urban Internet of things perception system relates to massive concurrent events and multi-element heterogeneous Internet of things data, the problems of data availability, sharability, manageability, credibility and the like are increasingly prominent, and in order to realize value-added service of the Internet of things data for urban development, the data sharing and fusion are core problems which need to be solved urgently.
At present, data sharing is generally realized through a block chain technology, but the method cannot completely ensure data privacy, and the risk of data leakage still exists.
Disclosure of Invention
In view of this, embodiments of the present invention provide a data sharing method and a terminal device, so as to solve the problem that in the prior art, data privacy cannot be completely guaranteed and a data leakage risk still exists.
A first aspect of an embodiment of the present invention provides a data sharing method, including:
receiving a sharing request of a data request node, wherein the sharing request comprises model information to be requested;
inquiring whether an available model related to the model information to be requested exists in the block chain network;
if the available model exists in the block chain network, returning the available model to the data request node;
and if no available model exists in the block chain network, selecting a target data providing node cooperating with the data request node, starting federal learning, returning a global model obtained after the federal learning is finished to the data request node, and recording the target data providing node and model parameters in the federal learning process in the block chain network.
A second aspect of embodiments of the present invention provides a terminal device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the steps of the data sharing method according to the first aspect are implemented.
Compared with the prior art, the embodiment of the invention has the following beneficial effects: the embodiment of the invention receives a sharing request of a data request node, wherein the sharing request comprises model information to be requested; inquiring whether an available model related to the model information to be requested exists in the block chain network; if the available model exists in the block chain network, returning the available model to the data request node; and if no available model exists in the block chain network, selecting a target data providing node cooperating with the data request node, starting federal learning, returning a global model obtained after the federal learning is finished to the data request node, and recording the target data providing node and model parameters in the federal learning process in the block chain network. According to the embodiment of the invention, by combining the block chain and the federal learning, privacy protection and safety credibility in the data sharing process can be realized, data privacy can be completely ensured, and the risk of data leakage does not exist.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.
Fig. 1 is a schematic flow chart illustrating an implementation of a data sharing method according to an embodiment of the present invention;
FIG. 2 is a schematic block diagram of a data sharing apparatus according to an embodiment of the present invention;
fig. 3 is a schematic block diagram of a terminal device according to an embodiment of the present invention.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.
In order to explain the technical means of the present invention, the following description will be given by way of specific examples.
Fig. 1 is a schematic implementation flow diagram of a data sharing method according to an embodiment of the present invention, and for convenience of description, only a part related to the embodiment of the present invention is shown. As shown in fig. 1, the method may include the steps of:
s101: and receiving a sharing request of the data request node, wherein the sharing request comprises the model information to be requested.
The embodiment of the invention provides a three-layer architecture, which comprises a local equipment layer, a block chain network layer and a cloud platform layer.
1) Local device layer: the intelligent city intelligent monitoring system is deployed at various sensing devices in all corners of a city and is responsible for acquiring and collecting internet of things data in the intelligent city. In the federal learning process, the local device trains the local model using its own data.
2) Block chain network layer: the block chain stores data identification and hash, manages data records on the chain, and stores and manages the learning process of the whole life cycle in the federal learning process.
3) Cloud platform layer: and processing a data sharing request of the data request node, and aggregating all local models into a global model in the federal learning process.
The data sharing method provided by the embodiment of the invention can be applied to a cloud platform layer. That is, the cloud platform receives a sharing request of the data requesting node. The data requesting node may be a node in the local device layer. The sharing request may include information about the global model required by the data requesting node, i.e., the model information to be requested.
S102: querying whether an available model associated with the model information to be requested exists in the blockchain network.
Because the federal learning process of the full life cycle is stored and managed in the blockchain network, the cloud platform can inquire whether a global model corresponding to the model information to be requested exists in the blockchain network, namely the available model.
S103: and if the available model exists in the blockchain network, returning the available model to the data request node.
And if the available model exists in the block chain network, directly returning the available model to the data request node, and ending the sharing request.
S104: and if no available model exists in the block chain network, selecting a target data providing node cooperating with the data request node, starting federal learning, returning a global model obtained after the federal learning is finished to the data request node, and recording the target data providing node and model parameters in the federal learning process in the block chain network.
If the available model does not exist in the block chain network, starting a new federal learning task, firstly selecting a target data providing node cooperating with the data request node through a node selection algorithm, and recording the target data providing node in the block chain network; then starting federal learning, and recording all model parameters in the learning process in the block chain network; and finally, returning the global model obtained after the federal learning is finished to the data request node, and finishing the sharing request.
The target data providing node is a node of the local device layer, and the number of the target data providing nodes may be one or multiple.
As can be seen from the above description, in the embodiments of the present invention, a sharing request of a data request node is received, where the sharing request includes model information to be requested; inquiring whether an available model related to the model information to be requested exists in the block chain network; if the available model exists in the block chain network, returning the available model to the data request node; and if no available model exists in the block chain network, selecting a target data providing node cooperating with the data request node, starting federal learning, returning a global model obtained after the federal learning is finished to the data request node, and recording the target data providing node and model parameters in the federal learning process in the block chain network. According to the embodiment of the invention, by combining the block chain and the federal learning, privacy protection and safety credibility in the data sharing process can be realized, data privacy can be completely ensured, and the risk of data leakage does not exist.
In an embodiment of the present invention, the selecting a target data providing node cooperating with the data requesting node includes:
determining a comprehensive trust value of each data providing node by the data request node;
and selecting the data providing node with the comprehensive trust value not less than the preset trust threshold value as a target data providing node.
Optionally, if the number of the data providing nodes of which the comprehensive trust value is not less than the preset trust threshold is greater than the required number, the data providing nodes of the previous required number are selected as the target data providing nodes according to the sequence from large to small of the comprehensive trust value.
In one embodiment of the present invention, determining a comprehensive trust value of a data requesting node for each data providing node comprises:
according toDetermining a comprehensive trust value CT (A, B) of a data requesting node A to a data providing node B;
wherein DT (A, B) provides a direct trust value of node B for the data request node A to the data;n is the number of direct interactions between the data requesting node A and the data providing node B, eBiWhen the ith direct interaction between the data request node A and the data providing node B is carried out, the data request node A provides the evaluation value, t, of the node B to the datanowIs the current time, tiProviding the completion time of the ith direct interaction of the node B for the data request node A and the data, wherein a is a preset constant and 0<a<1;
RT (B) provides the data request node A with the recommended trust value of the node B for the data; n (a, C) ═ SN (a, C) -FN (a, C), in (B) is the set of all nodes that interact directly with the data providing node B, R (a, C) is the trust value of DT (C, B) for the data requesting node a, N (a, C) is the number of valid interactions of the data requesting node a with node C, and N (a) is the number of valid interactions in which the data requesting node a participates; SN (A, C) is the number of times that the data request node A and the node C cooperate to participate in data sharing successfully, FN (A, C) is the number of times that the data request node A and the node C cooperate to participate in data sharing fail;
n (A, B) provides the effective interaction times of the node B for the data request node A and the data; hAIs a preset transaction number threshold.
The above-mentioned node C is a node except the data requesting node a among all nodes directly interacting with the data providing node B.
In the federal learning process, it cannot be guaranteed that all nodes provide good service and reliable resources. In order to ensure the data sharing quality, the embodiment of the invention provides a node selection algorithm based on the credit value, and the data request node selects the data providing node with better performance and more stable service to cooperate according to the credit value.
Based on the social network, the trust relationship between the node A and the node B is mainly established on the basis of direct interaction between the node A and the node B, and the evaluation of the node B by other nodes which have direct interaction with the node B is referred.
When the node A and the node B do not interact effectively, the trust of the node A to the node B only depends on the recommended trust; when the effective interaction times between the node A and the node B exceed a preset transaction time threshold value H set by the node AAWhen the node A is in use, the node A can make judgment according to the direct trust value of the node A to the node B; trust of node A to node B when the number of valid interactions between node A and node B is in the middle rangeDirect trust and recommendation trust need to be combined.
In an embodiment of the present invention, after the federal learning is finished, the data sharing method further includes:
and determining the evaluation value of the data request node to the target data providing node, and recording the evaluation value of the data request node to the target data providing node in the block chain network.
In one embodiment of the present invention, determining an evaluation value of a data requesting node to a target data providing node includes:
Wherein m is the number of the target data providing nodes, and j is more than or equal to 1 and less than or equal to m;providing a node N for target datajThe quote value of (a);for sorting in descending order, the target data provides node NjThe rank of the quote value of (a) in the quote values of all the target data providing nodes;providing a node N for target datajA performance evaluation value of (2);for sorting in descending order, the target data provides node NjThe rank of the performance evaluation value of (b) in the performance evaluation values of all target data providing nodes;
size(Dj) Providing a node N for target datajData set D ofjIs the number of iterations of federated learning,providing a node N for target datajThe model quality at the kth iteration; qkModel quality for the global model at the kth iteration, TkFor the training duration of the global model at the kth iteration,providing a node N for target datajTraining duration at kth iteration; alpha, beta and gamma are all weight coefficients, 0<α<1,0<β<1,0<γ<1, and α + β + γ ═ 1.
The set of target data providing nodes selected by the node selection algorithm is N ═ N1,…Nj,…,NmD ═ D, a set of data sets of the respective target data providing nodes is set as D ═ D1,…Dj,…,DmThe quotation of each target data providing node is formed into a set
The federate learning occurs in one iteration, and the target data provides a node NjThe set of model masses at each iteration isTarget data providing node NjThe training duration at each iteration is comprised of a set ofThe set of model qualities of the global model at each iteration is { Q ═ Q1,..,Qk,…,QlAnd the set of training time lengths of the global model in each iteration is T ═ T { (T)1,..,Tk,…,Tl}。
After the federal learning is finished, performance evaluation needs to be performed on each target data providing node. In addition, whether the quotation of the node is matched with the service provided by the node is judged, and finally, the target data providing node N of the data requesting node is obtainedjThe evaluation value of (1).
The embodiment of the invention selects the federal learning node based on the credit value, and evaluates the performance of the node in the learning process so as to realize higher-quality federal learning.
It should be noted that the nodes are all nodes in federal learning, and the nodes mentioned in the subsequent optimized Raft algorithm are Raft nodes, that is, nodes in the optimized Raft algorithm.
In an embodiment of the present invention, the data sharing method further includes:
in the block chain network, an optimized Raft algorithm is adopted to keep consensus.
In one embodiment of the present invention, the above-mentioned method for maintaining consensus by using optimized Raft algorithm includes:
when the follower node cannot receive the heartbeat information of the leader node, the follower node sends a communication request to the non-leader node;
if the follower node receives the response message of the non-leader node, the follower node is converted into a candidate node and initiates a master selection request;
and if the follower node does not receive the response message of the non-leader node, stopping selecting the owner.
In the Raft algorithm, nodes are called by Remote Procedure Calls (RPCs). Communication is performed. When network partitioning occurs, isolated nodes continuously increase currentTerm (latest number) and initiate a main selection process because the isolated nodes cannot receive heartbeat messages of the leader; when the network is recovered, the currentTerm of the node is larger than that of other nodes, so that the original leader becomes a follower, the normal work of the original leader is interrupted, and the stability of the algorithm is seriously influenced.
Therefore, the embodiment of the invention is designed to add a PreVote link, and before the follower which does not receive heartbeat information initiates selection of the master role, the follower firstly sends a PreVote RPC request to other nodes to judge whether the communication with other nodes can be carried out. If the election can be started, the election request is initiated by the candidate, if the election can not be started, the election is terminated, and the network is waited to recover, so that the election is optimized.
In one embodiment of the present invention, the above-mentioned method for maintaining consensus by using optimized Raft algorithm includes:
when the follower node receives the RPC sent by the leader node, judging whether a log index value in the RPC is matched with a log index value of the follower node;
and if the log index value in the RPC is not matched with the log index value of the leader node, sending the log index value of the leader node to the leader node, so that the leader node resends the RPC according to the log index value sent by the follower node.
In the Raft algorithm, after the follower receives the leader's AppendEntries RPC, if the nextIndex in the RPC is checked to be not matched with itself, the request is rejected. When the leader receives the decline reply, it gradually decreases the nextIndex of the follower and sends AppendEntries RPC again until it finds the nextIndex that is consistent with the follower's log. This approach takes a significant amount of time to communicate, reducing the efficiency of the Raft algorithm. Wherein, the AppendEntries RPC is used for log replication or for heartbeat.
Therefore, the embodiment of the invention optimizes the operation, when the nextIndex is not matched, the follower directly sends the index value consistent with the final log to the leader, and the leader resends the Appendentries RPC according to the received nextIndex, so that the log replication optimization is realized, and the communication times when the nextIndex is not matched are effectively reduced. The nextIndex is a log index value and a log number which should be sent to the follower node next time by the leader node.
In one embodiment of the invention, consensus is maintained using an optimized Raft algorithm, comprising:
the candidate node votes for the candidate node and sends a voting message to other nodes, wherein the voting message comprises a client signature of the last agreed log item in the log list of the candidate node;
after receiving the voting message, the other nodes verify the authenticity of the signature of the client, if the signature of the client is real, the other nodes send the message signature to the candidate node, and if the signature of the client is not real, the other nodes refuse voting;
after receiving the message signatures of other nodes beyond 2/3, the candidate node generates a complete signature and sends a message containing the complete signature to the other nodes;
after receiving the message containing the complete signature, other nodes verify the validity of the complete signature, if the complete signature is legal, positive feedback information is sent to the candidate node, and if the complete signature is illegal, negative feedback information is sent to the candidate node;
when the candidate node receives more than 2/3 positive feedback information sent by other nodes, the candidate node is changed to the leader node.
In an embodiment of the present invention, the message signature is (R)h,sh). Wherein s ish=rh+xhαh*e,e=Hash(R,X,entry),X=∑αhXh,R=∑Rh,αh=Hash(l,Xh),l=Hash(X1,X2,…XH),Xh=xh*G,Rh=rh*G,RhIs the public key of the random number of the node h, rhIs a random number, x, of node hhIs the private key of node h, XhIs the public key of the node h, G is the origin of the elliptic curve, X is the shared public key, R is the shared public key of the random number, l is the first hash value, alphahIs the hash value of node h, e is the second hash value, shAnd the entry is the message corresponding to the message signature for the calculated signature of the node h.
The complete signature is(R,s),s=∑sh。
The verifying the validity of the complete signature includes: and verifying the validity of the complete signature by judging whether the equation sG is satisfied or not, wherein if the equation is satisfied, the complete signature is legal, otherwise, the complete signature is illegal.
The Raft algorithm is a non-Byzantine algorithm, is optimized based on a Musig aggregation signature scheme, and simultaneously introduces a digital signature technology to enable a message sent to a leader by a client to contain an instruction to be executed and a corresponding digital signature, so that the Byzantine fault tolerance in the processes of leader election and log replication is realized, and the safety of the algorithm can be improved.
Each Raft node has a private key xiPublic key XiA random number riAnd a random number public key RiAnd G is the origin of the elliptic curve. All Raft nodes publish public key XiAnd a random number public key RiAnd calculates shared public keys X and R.
The signature of the follower on the message represents the vote for the candidate, which can be selected as the leader when the candidate receives a vote exceeding 2/3. The method specifically comprises the following steps:
in the first step, the candidate node votes for itself and sends RequestVoteRPC to other nodes, and the message contains the client signature of the last agreed log item in the candidate log list.
Secondly, after other nodes receive the message, the authenticity of the signature of the client is verified, and if the authenticity is true, the message is signed (R)i,si) And sending the information to the candidate node, and otherwise rejecting the voting.
And thirdly, the candidate node acquires the complete signature(s) after collecting the partial signatures of other nodes exceeding 2/3, and sends the message attached with the complete signature to other nodes.
And fourthly, after receiving the message, other nodes verify the validity of the complete signature by judging whether the equation sG-R + Xe is true or not, if so, returning a positive feedback, and if not, returning a negative feedback.
And fifthly, the last candidate node becomes a leader after collecting positive feedback of other nodes exceeding 2/3, and the whole election process is completed.
Aiming at the problem that the leader tampers the instruction, the follower can verify the authenticity of the instruction through a client public key, and if the instruction is found to be tampered by the leader, the follower refuses to add the tampered log item and converts the tampered log item into a candidate to start a new round of leader election. Aiming at the problem of tampering the instruction by the follower, the mutual recognition of the rest non-Byzantine followers and the leader is not influenced, and in the later election process, the node cannot become the leader because the node does not possess all the logs with the mutual recognition.
In the Byzantine fault-tolerant log replication process, the signature of the follower on the message represents that the log in the AppendEntries RPC is successfully replicated, the leader receives the partial signature of the follower exceeding 2/3 and then calculates the complete signature, and the follower exceeding 2/3 verifies that the complete signature is correct, and represents that the instruction achieves consensus.
The embodiment of the invention provides an optimized Raft algorithm. The method starts from three aspects of selecting a master, copying the log and safety respectively, and improves the performance of the algorithm.
The above description shows that the embodiment of the invention provides a smart city Internet of things data lightweight credible sharing mechanism, and introduces a block chain and federal learning to realize credible data sharing and privacy protection; a federated learning node selection algorithm and a node evaluation algorithm are designed, and a basis is provided for higher-quality data sharing; and finally, the consensus efficiency of the Raft algorithm is optimized, and the requirements of the smart city on time delay and safety are met.
It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.
Corresponding to the data sharing method, an embodiment of the present invention further provides a data sharing apparatus, which has the same beneficial effects as the data sharing method. Fig. 2 is a schematic block diagram of a data sharing apparatus according to an embodiment of the present invention, and for convenience of description, only the portions related to the embodiment of the present invention are shown.
In the embodiment of the present invention, the data sharing apparatus 30 may include a receiving module 301, a querying module 302, a first processing module 303, and a second processing module 304.
The receiving module 301 is configured to receive a sharing request of a data request node, where the sharing request includes model information to be requested;
a query module 302, configured to query whether an available model associated with the model information to be requested exists in the blockchain network;
the first processing module 303 is configured to, if an available model exists in the blockchain network, return the available model to the data request node;
and the second processing module 304 is configured to, if no available model exists in the blockchain network, select a target data providing node cooperating with the data request node, start federal learning, return a global model obtained after the federal learning to the data request node, and record the target data providing node and model parameters in the federal learning process in the blockchain network.
Optionally, the second processing module 304 may be further configured to:
determining a comprehensive trust value of each data providing node by the data request node;
and selecting the data providing node with the comprehensive trust value not less than the preset trust threshold value as a target data providing node.
Optionally, the second processing module 304 may be further configured to:
according toDetermining a comprehensive trust value CT (A, B) of a data requesting node A to a data providing node B;
wherein DT (A, B) provides a direct trust value of node B for the data request node A to the data;n is the direct interaction time of the data requesting node A and the data providing node BNumber eBiWhen the ith direct interaction between the data request node A and the data providing node B is carried out, the data request node A provides the evaluation value, t, of the node B to the datanowIs the current time, tiProviding the completion time of the ith direct interaction of the node B for the data request node A and the data, wherein a is a preset constant and 0<a<1;
RT (A, B) provides a recommended trust value of the node B for the data request node A to the data; n (a, C) ═ SN (a, C) -FN (a, C), in (B) is the set of all nodes that directly interact with the data providing node B, R (a, C) is the trust value of DT (C, B) for the data requesting node a, N (a, C) is the number of valid interactions of the data requesting node a with node C, and N (a) is the number of valid interactions in which the data requesting node a participates; SN (A, C) is the number of times that the data request node A and the node C cooperate to participate in data sharing successfully, FN (A, C) is the number of times that the data request node A and the node C cooperate to participate in data sharing fail;
n (A, B) provides effective interaction times of the node B for the data request node A and the data; hAIs a preset transaction number threshold.
Optionally, the second processing module 304 may be further configured to:
and determining the evaluation value of the data request node to the target data providing node, and recording the evaluation value of the data request node to the target data providing node in the block chain network.
Optionally, the second processing module 304 may further be configured to:
according toDetermining a data requesting node to a target data providing node NjEvaluation value of (2)
Wherein m is the number of the target data providing nodes, and j is more than or equal to 1 and less than or equal to m;providing a node N for target datajThe quote value of (a);for sorting in descending order, the target data provides node NjThe rank of the quote value of (a) in the quote values of all the target data providing nodes;providing a node N for target datajA performance evaluation value of (2);for sorting in descending order, the target data provides node NjThe rank of the performance evaluation value of (2) among the performance evaluation values of all the target data providing nodes;
size(DD) Providing a node N for target datajData set D ofjL is the number of iterations of federal learning,providing a node N for target datajThe model quality at the kth iteration; qkModel quality for the global model at the kth iteration, TkFor the training duration of the global model at the kth iteration,providing a node N for target datajTraining duration at kth iteration; alpha, beta and gamma are all weight coefficients, 0<α<1,0<β<1,0<γ<1, and α + β + γ ═ 1.
Optionally, the data sharing apparatus may further include: and a consensus module.
And the consensus module is used for keeping consensus by adopting an optimized Raft algorithm in the block chain network.
Optionally, the consensus module may be further configured to:
when the follower node cannot receive the heartbeat information of the leader node, the follower node sends a communication request to the non-leader node;
if the follower node receives the response message of the non-leader node, the follower node is converted into a candidate node and initiates a master selection request;
and if the follower node does not receive the response message of the non-leader node, stopping selecting the owner.
Optionally, the consensus module may be further configured to:
when the follower node receives the RPC sent by the leader node, judging whether a log index value in the RPC is matched with a log index value of the follower node;
and if the log index value in the RPC is not matched with the log index value of the leader node, sending the log index value of the leader node to the leader node so that the leader node resends the RPC according to the log index value sent by the follower node.
Optionally, the consensus module may be further configured to:
the candidate node votes for the candidate node and sends a voting message to other nodes, wherein the voting message comprises a client signature of the last agreed log item in the log list of the candidate node;
after receiving the voting message, the other nodes verify the authenticity of the signature of the client, if the signature of the client is real, the other nodes send the message signature to the candidate node, and if the signature of the client is not real, the other nodes refuse voting;
after receiving the message signatures of other nodes beyond 2/3, the candidate node generates a complete signature and sends a message containing the complete signature to the other nodes;
after receiving the message containing the complete signature, other nodes verify the validity of the complete signature, if the complete signature is legal, positive feedback information is sent to the candidate node, and if the complete signature is illegal, negative feedback information is sent to the candidate node; wherein,
when the candidate node receives more than 2/3 positive feedback information sent by other nodes, the candidate node transitions to the leader node.
It will be apparent to those skilled in the art that, for convenience and simplicity of description, the foregoing functional units and modules are merely illustrated in terms of division, and in practical applications, the foregoing functional allocation may be performed by different functional units and modules as needed, that is, the internal structure of the data sharing apparatus is divided into different functional units or modules to perform all or part of the above described functions. Each functional unit and module in the embodiments may be integrated in one processing unit, or each unit may exist alone physically, or two or more units are integrated in one unit, and the integrated unit may be implemented in a form of hardware, or in a form of software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working processes of the units and modules in the above-mentioned apparatus may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
Fig. 3 is a schematic block diagram of a terminal device according to an embodiment of the present invention. As shown in fig. 3, the terminal device 40 of this embodiment includes: one or more processors 401, a memory 402, and a computer program 403 stored in the memory 402 and executable on the processors 401. The processor 401 implements the steps in the above-mentioned embodiments of the data sharing method, such as the steps S101 to S104 shown in fig. 1, when executing the computer program 403. Alternatively, the processor 401, when executing the computer program 403, implements the functions of the modules/units in the data sharing apparatus embodiment, for example, the functions of the modules 301 to 304 shown in fig. 2.
Illustratively, the computer program 403 may be partitioned into one or more modules/units that are stored in the memory 402 and executed by the processor 401 to accomplish the present application. The one or more modules/units may be a series of computer program instruction segments capable of performing specific functions, which are used for describing the execution process of the computer program 403 in the terminal device 40. For example, the computer program 403 may be divided into a receiving module, a querying module, a first processing module and a second processing module, and each module has the following specific functions:
the receiving module is used for receiving a sharing request of a data request node, wherein the sharing request comprises model information to be requested;
the query module is used for querying whether an available model associated with the model information to be requested exists in the block chain network;
the first processing module is used for returning the available model to the data request node if the available model exists in the block chain network;
and the second processing module is used for selecting a target data providing node cooperating with the data request node if no available model exists in the block chain network, starting federal learning, returning a global model obtained after the federal learning is finished to the data request node, and recording the target data providing node and model parameters in the federal learning process in the block chain network.
Other modules or units can refer to the description of the embodiment shown in fig. 2, and are not described again here.
The terminal device 40 may be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The terminal device 40 includes, but is not limited to, a processor 401 and a memory 402. Those skilled in the art will appreciate that fig. 3 is only one example of a terminal device 40, and does not constitute a limitation to the terminal device 40, and may include more or less components than those shown, or combine some components, or different components, for example, the terminal device 40 may further include an input device, an output device, a network access device, a bus, etc.
The Processor 401 may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic device, discrete hardware component, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The storage 402 may be an internal storage unit of the terminal device 40, such as a hard disk or a memory of the terminal device 40. The memory 402 may also be an external storage device of the terminal device 40, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are provided on the terminal device 40. Further, the memory 402 may also include both an internal storage unit of the terminal device 40 and an external storage device. The memory 402 is used for storing the computer program 403 and other programs and data required by the terminal device 40. The memory 402 may also be used to temporarily store data that has been output or is to be output.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the embodiments provided in the present application, it should be understood that the disclosed data sharing apparatus and method may be implemented in other ways. For example, the above-described data sharing apparatus embodiments are merely illustrative, and for example, the division of the modules or units is only one logical division, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated modules/units, if implemented in the form of software functional units and sold or used as separate products, may be stored in a computer readable storage medium. Based on such understanding, all or part of the flow in the method of the embodiments described above can be realized by a computer program, which can be stored in a computer-readable storage medium and can realize the steps of the embodiments of the methods described above when the computer program is executed by a processor. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U.S. disk, removable hard disk, magnetic diskette, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signal, telecommunications signal, and software distribution medium, etc. It should be noted that the computer readable medium may contain other components which may be suitably increased or decreased as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer readable media which may not include electrical carrier signals and telecommunications signals in accordance with legislation and patent practice.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present application and are intended to be included within the scope of the present application.
Claims (8)
1. A method for sharing data, comprising:
receiving a sharing request of a data request node, wherein the sharing request comprises model information to be requested;
inquiring whether an available model related to the model information to be requested exists in a block chain network;
if the available model exists in the block chain network, returning the available model to the data request node;
if the available model does not exist in the block chain network, selecting a target data providing node cooperating with the data request node, starting federal learning, returning a global model obtained after the federal learning is finished to the data request node, and recording the target data providing node and model parameters in the federal learning process in the block chain network;
the selecting of the target data providing node cooperating with the data requesting node includes:
determining a comprehensive trust value of the data request node on each data providing node;
selecting a data providing node with a comprehensive trust value not less than a preset trust threshold value as a target data providing node;
the determining the comprehensive trust value of the data request node for each data providing node comprises:
according toDetermining a comprehensive trust value CT (A, B) of a data requesting node A to a data providing node B;
wherein DT (A, B) provides the direct trust value of node B for the data request node A to the data;n is the number of direct interactions between the data requesting node A and the data providing node B, eBiWhen the ith direct interaction between the data request node A and the data providing node B is carried out, the data request node A provides the evaluation value, t, of the node B to the datanowIs the current time, tiProviding the completion time of the ith direct interaction of the node B for the data request node A and the data, wherein a is a preset constant and is more than 0 and less than 1;
RT (A, B) provides a recommended trust value of the node B for the data request node A to the data; n (a, C) ═ SN (a, C) -FN (a, C), in (B) is the set of all nodes that directly interact with the data providing node B, R (a, C) is the trust value of DT (C, B) for the data requesting node a, N (a, C) is the number of valid interactions of the data requesting node a with node C, and N (a) is the number of valid interactions in which the data requesting node a participates; SN (A, C) isThe number of times that the data request node A and the node C cooperate to participate in data sharing is successful, and FN (A, C) is the number of times that the data request node A and the node C cooperate to participate in data sharing is failed;
n (A, B) provides the effective interaction times of the node B for the data request node A and the data; hAAnd the preset transaction time threshold value is obtained.
2. The data sharing method according to claim 1, wherein after the federal learning is finished, the data sharing method further comprises:
and determining the evaluation value of the data request node to the target data providing node, and recording the evaluation value of the data request node to the target data providing node in the block chain network.
3. The data sharing method according to claim 2, wherein the determining the evaluation value of the data requesting node to the target data providing node includes:
Wherein m is the number of the target data providing nodes, and j is more than or equal to 1 and less than or equal to m;providing a node N for target datajThe quote value of (a);for sorting in descending order, the target data provides node NjThe rank of the quote value of (a) among the quote values of all the targeted data providing nodes;providing a node N for target datajPerformance evaluation value of (2);for sorting in descending order, the target data provides node NjThe rank of the performance evaluation value of (2) among the performance evaluation values of all the target data providing nodes;
size(Dj) Providing a node N for target datajData set D ofjL is the number of iterations of federal learning,providing a node N for target datajThe model quality at the kth iteration; qkModel quality for the global model at the kth iteration, TkFor the training duration of the global model at the kth iteration,providing a node N for target datajTraining duration at kth iteration; alpha, beta and gamma are weight coefficients, alpha is more than 0 and less than 1, beta is more than 0 and less than 1, gamma is more than 0 and less than 1, and alpha + beta + gamma is equal to 1.
4. A data sharing method according to any one of claims 1 to 3, further comprising:
in the block chain network, an optimized Raft algorithm is adopted to keep consensus.
5. The data sharing method of claim 4, wherein the employing the optimized Raft algorithm to maintain consensus comprises:
when the follower node cannot receive the heartbeat information of the leader node, the follower node sends a communication request to a non-leader node;
if the follower node receives the response message of the non-leader node, the follower node is converted into a candidate node and initiates a master selection request;
and if the follower node does not receive the response message of the non-leader node, stopping selecting the owner.
6. The data sharing method of claim 4, wherein the employing the optimized Raft algorithm to maintain consensus comprises:
when the follower node receives the RPC sent by the leader node, judging whether a log index value in the RPC is matched with a log index value of the follower node;
and if the log index value in the RPC is not matched with the log index value of the leader node, sending the log index value of the leader node to the leader node so that the leader node resends the RPC according to the log index value sent by the follower node.
7. The data sharing method of claim 4, wherein the employing the optimized Raft algorithm to maintain consensus comprises:
the candidate node votes for the candidate node and sends a voting message to other nodes, wherein the voting message comprises a client signature of the last agreed log item in the log list of the candidate node;
after receiving the voting message, the other nodes verify the authenticity of the client signature, if the client signature is true, the other nodes send the message signature to the candidate node, and if the client signature is not true, the other nodes refuse voting;
after receiving message signatures of other nodes exceeding 2/3, the candidate node generates a complete signature and sends messages containing the complete signature to other nodes;
after receiving the message containing the complete signature, other nodes verify the validity of the complete signature, if the complete signature is legal, positive feedback information is sent to the candidate node, and if the complete signature is illegal, negative feedback information is sent to the candidate node; wherein,
when the candidate node receives more than 2/3 positive feedback information sent by other nodes, the candidate node transitions to a leader node.
8. A terminal device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the steps of the data sharing method according to any one of claims 1 to 7 when executing the computer program.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110626699.2A CN113420323B (en) | 2021-06-04 | 2021-06-04 | Data sharing method and terminal equipment |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110626699.2A CN113420323B (en) | 2021-06-04 | 2021-06-04 | Data sharing method and terminal equipment |
Publications (2)
Publication Number | Publication Date |
---|---|
CN113420323A CN113420323A (en) | 2021-09-21 |
CN113420323B true CN113420323B (en) | 2022-06-03 |
Family
ID=77713917
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202110626699.2A Active CN113420323B (en) | 2021-06-04 | 2021-06-04 | Data sharing method and terminal equipment |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN113420323B (en) |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN116701549A (en) * | 2023-06-21 | 2023-09-05 | 黑龙江禹桥科技开发有限公司 | Big data multi-scale fusion supervision system and method based on blockchain |
CN116546095B (en) * | 2023-07-04 | 2023-09-05 | 深圳市威宇智通科技有限公司 | Resource integration data trusted sharing system for different intelligent communities |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110177356A (en) * | 2019-05-31 | 2019-08-27 | 长安大学 | A kind of vehicle-mounted name data network retransmission method based on trust model |
CN111901309A (en) * | 2020-07-03 | 2020-11-06 | 北京邮电大学 | Data security sharing method, system and device |
CN111930698A (en) * | 2020-07-01 | 2020-11-13 | 南京晓庄学院 | Data security sharing method based on Hash diagram and federal learning |
CN111931242A (en) * | 2020-09-30 | 2020-11-13 | 国网浙江省电力有限公司电力科学研究院 | Data sharing method, computer equipment applying same and readable storage medium |
CN112395640A (en) * | 2020-11-16 | 2021-02-23 | 国网河北省电力有限公司信息通信分公司 | Industry Internet of things data lightweight credible sharing technology based on block chain |
Family Cites Families (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110233905B (en) * | 2017-04-20 | 2020-12-25 | 腾讯科技(深圳)有限公司 | Node device operation method, node device, and storage medium |
-
2021
- 2021-06-04 CN CN202110626699.2A patent/CN113420323B/en active Active
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110177356A (en) * | 2019-05-31 | 2019-08-27 | 长安大学 | A kind of vehicle-mounted name data network retransmission method based on trust model |
CN111930698A (en) * | 2020-07-01 | 2020-11-13 | 南京晓庄学院 | Data security sharing method based on Hash diagram and federal learning |
CN111901309A (en) * | 2020-07-03 | 2020-11-06 | 北京邮电大学 | Data security sharing method, system and device |
CN111931242A (en) * | 2020-09-30 | 2020-11-13 | 国网浙江省电力有限公司电力科学研究院 | Data sharing method, computer equipment applying same and readable storage medium |
CN112395640A (en) * | 2020-11-16 | 2021-02-23 | 国网河北省电力有限公司信息通信分公司 | Industry Internet of things data lightweight credible sharing technology based on block chain |
Non-Patent Citations (1)
Title |
---|
蒋小杰等.一种基于信誉的移动自组网区分服务激励机制.《电子与信息学报》.2012,第34卷(第07期),1699-1700. * |
Also Published As
Publication number | Publication date |
---|---|
CN113420323A (en) | 2021-09-21 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10678598B2 (en) | Enforcing compute equity models in distributed blockchain | |
US11057225B2 (en) | Enforcing compute equity models in distributed blockchain | |
CN112395640B (en) | Industry internet of things data light-weight credible sharing technology based on block chain | |
CN112765677B (en) | Federal learning method, device and system based on blockchain | |
EP4318362A1 (en) | Blockchain-based data processing method, apparatus and device, and storage medium | |
CN110163755B (en) | Block chain-based data compression and query method and device and electronic equipment | |
CN113420323B (en) | Data sharing method and terminal equipment | |
CN112632013A (en) | Data security credible sharing method and device based on federal learning | |
CN111382456A (en) | Proposal message processing method, device, equipment and storage medium | |
WO2019024780A1 (en) | Light-weight processing method for blockchain, and blockchain node and storage medium | |
CN111988203A (en) | Node election method, device and storage medium | |
WO2020228531A1 (en) | Consortium blockchain governance method and apparatus, computer device and storage medium | |
CN112597240B (en) | Federal learning data processing method and system based on alliance chain | |
WO2022058183A1 (en) | Integrating device identity into a permissioning framework of a blockchain | |
CN114493865A (en) | Decentralized internet cooperative system and data interaction method based on block chain | |
Liu et al. | Ltsm: Lightweight and trusted sharing mechanism of iot data in smart city | |
CN112837157A (en) | Method, device and system for registering and executing timing intelligent contract in block chain | |
CN112416881A (en) | Intelligent terminal storage sharing method, device, medium and equipment based on block chain | |
CN113221163B (en) | Model training method and system | |
CN110264222A (en) | Responsible investigation method, apparatus and terminal device based on data acquisition | |
CN112307331A (en) | Block chain-based college graduate intelligent recruitment information pushing method and system and terminal equipment | |
CN111861481A (en) | Block chain account checking method and system | |
CN113052329A (en) | Method and device for jointly updating service model | |
CN114039733A (en) | Certificate storage service transfer method, device and equipment for alliance chain | |
Bansal et al. | A fast, secure and distributed consensus mechanism for energy trading among vehicles using hashgraph |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |