CN111950739A - Data processing method, device, equipment and medium based on block chain - Google Patents
Data processing method, device, equipment and medium based on block chain Download PDFInfo
- Publication number
- CN111950739A CN111950739A CN202010822839.9A CN202010822839A CN111950739A CN 111950739 A CN111950739 A CN 111950739A CN 202010822839 A CN202010822839 A CN 202010822839A CN 111950739 A CN111950739 A CN 111950739A
- Authority
- CN
- China
- Prior art keywords
- participant
- federal
- parameter information
- determining
- data processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/30—Authentication, i.e. establishing the identity or authorisation of security principals
- G06F21/44—Program or device authentication
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/602—Providing cryptographic facilities or services
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- Evolutionary Computation (AREA)
- Computing Systems (AREA)
- Medical Informatics (AREA)
- Mathematical Physics (AREA)
- Data Mining & Analysis (AREA)
- Health & Medical Sciences (AREA)
- Bioethics (AREA)
- General Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Artificial Intelligence (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a data processing method, a device, equipment and a medium based on a block chain, wherein the method comprises the following steps: sending the target model data to be processed to a block chain block for each second participant to respectively perform preset federal modeling with the first participant based on the target model data to be processed; receiving all federal parameter information which is encrypted and sent to a block chain block by all second participants in the preset federal modeling process; and determining the participation legality of the corresponding second party based on the federal parameter information, and determining the contribution degree of each legal second party. The method and the device solve the technical problem that in the prior art, the federal model is difficult to safely and quickly construct in the prior art.
Description
Technical Field
The present application relates to the field of artificial intelligence (artificial intelligence) technology for financial technology (Fintech), and in particular, to a data processing method, apparatus, device, and medium based on a block chain.
Background
With the continuous development of financial technologies, especially internet technology and finance, more and more technologies (such as distributed, Blockchain, artificial intelligence and the like) are applied to the financial field, but the financial industry also puts higher requirements on the technologies, for example, the financial industry also has higher requirements on data processing based on the Blockchain.
The federal learning can help a plurality of data sources to jointly train a federal model under the condition of protecting user privacy, however, in the prior art, when the federal model is built, all participants directly exchange data, so that potential safety hazards exist during data exchange, and when the federal model is built, only the model effect is concerned, and the contribution degrees of all the participants are defaulted to be the same, so that the participants with high-quality data are difficult to attract to join, and the high-quality federal model is difficult to be built quickly, namely, the technical problem that the high-quality federal model is difficult to be built quickly and safely exists in the prior art.
Disclosure of Invention
The application mainly aims to provide a data processing method, a data processing device, data processing equipment and a data processing medium based on a block chain, and aims to solve the technical problem that a federal model is difficult to safely and quickly construct in the prior art.
In order to achieve the above object, the present application provides a data processing method based on a block chain, which is applied to a first participant, where the first participant and each second participant perform federated communication connection through the block chain, and the data processing method based on the block chain includes:
sending the target model data to be processed to a block chain block for each second participant to respectively perform preset federal modeling with the first participant based on the target model data to be processed;
receiving all federal parameter information which is encrypted and sent to a block chain block by all second participants in the preset federal modeling process;
and determining the participation legality of the corresponding second party based on the federal parameter information, and determining the contribution degree of each legal second party.
Optionally, the step of determining the participation validity of the corresponding second participant based on each piece of federal parameter information includes:
determining the associated participants respectively associated with the second participants;
receiving the associated participants based on the block link, and verifying the federal parameter information of the corresponding second participants to obtain a verification result;
the method comprises the steps that a correlation participant federation obtains a correlation model corresponding to federation parameter information of a second participant and corresponding local federation parameter information, and the correlation model is verified based on a first preset verification data set of the correlation participant to obtain a verification result;
and determining the participation legality of the corresponding second party based on the verification result.
Optionally, the step of determining the contribution degrees of the legitimate second parties includes:
determining each federal model correspondingly determined by the first participant based on each federal parameter information;
obtaining a second preset validation dataset for the first party;
predicting the second preset verification data set based on each federal model to obtain each prediction result;
and correspondingly determining the contribution degree of each second party based on each prediction result.
Optionally, each second party encrypts and sends each federal parameter information in the preset federal modeling process;
before the step of determining the participation legality of the corresponding second party based on each piece of federal parameter information and determining the contribution degree of each legal second party, the method includes:
determining whether encrypted federal parameter information of all second parties is received;
and if the encrypted federal parameter information of all the second participants is received, cooperatively triggering and decrypting all the federal parameter information.
Optionally, the step of determining the contribution degrees of the legitimate second parties includes:
acquiring sending time for sending each federal parameter information to a block chain block by each legal second participant in the preset federal modeling process;
and determining the contribution degree of each legal second participant based on the sending time.
Optionally, after the step of determining the participation validity of the corresponding second party based on the information of each federal parameter and determining the contribution degree of each legal second party, the method includes:
acquiring preset modeling optimization reward data;
and distributing the reward amount in the modeling optimization reward data to each participant based on the contribution degree.
Optionally, the data processing method based on the blockchain includes:
and saving each federal parameter information reported by each second party based on the block chain block record, and saving the contribution degree of each second party based on the block chain block record.
The present application further provides a data processing apparatus based on a block chain, which is applied to a first participant, where the first participant and each second participant perform federated communication connection via the block chain, and the data processing apparatus based on the block chain includes:
the modeling module is used for sending the target model data to be processed to the block chain block so that each second participant can respectively perform preset federal modeling with the first participant based on the target model data to be processed;
the receiving module is used for receiving each federal parameter information which is encrypted and sent to the block chain block by each second participant in the preset federal modeling process;
and the first determining module is used for determining the participation legality of the corresponding second party based on the federal parameter information and determining the contribution degree of each legal second party.
Optionally, the first determining module includes:
the first determining unit is used for determining the associated participants respectively associated with the second participants;
the first receiving unit is used for receiving the associated participants based on the block link and verifying the federal parameter information of the corresponding second participant to obtain a verification result;
the method comprises the steps that a correlation participant federation obtains a correlation model corresponding to federation parameter information of a second participant and corresponding local federation parameter information, and the correlation model is verified based on a first preset verification data set of the correlation participant to obtain a verification result;
and the second determining unit is used for determining the participation legality of the corresponding second party based on the verification result.
Optionally, the first determining module further includes:
the third determining unit is used for determining each federal model correspondingly determined by the first participant based on each federal parameter information;
a first obtaining unit, configured to obtain a second preset verification data set of the first party;
the prediction unit is used for predicting the second preset verification data set based on each federal model to obtain each prediction result;
and a fourth determining unit, configured to correspondingly determine the contribution degree of each second participant based on each prediction result.
Optionally, each second party encrypts and sends each federal parameter information in the preset federal modeling process;
the block chain-based data processing apparatus further includes:
the second determining module is used for determining whether encrypted federal parameter information of all the second participants is received;
and the decryption module is used for cooperatively triggering decryption of all the federal parameter information if the encrypted federal parameter information of all the second participants is received.
Optionally, the first determining module further includes:
the second acquisition unit is used for acquiring each legal second participant and sending each federal parameter information to each sending time in the block chain block in the preset federal modeling process;
and a fifth determining unit, configured to determine, based on the length of the sending time, the contribution degree of each legitimate second party.
Optionally, the data processing apparatus based on a blockchain further includes:
the acquisition module is used for acquiring preset modeling optimization reward data;
and the distribution module is used for distributing the reward amount in the modeling optimization reward data to each participant based on the contribution degree.
Optionally, the data processing apparatus based on a blockchain further includes:
and the storage module is used for storing the federal parameter information reported by each second participant based on the block chain block record and storing the contribution degree of each second participant based on the block chain block record.
The present application further provides a data processing device based on a blockchain, where the data processing device based on the blockchain is an entity device, and the data processing device based on the blockchain includes: a memory, a processor and a program of the blockchain based data processing method stored on the memory and executable on the processor, which program, when executed by the processor, may implement the steps of the blockchain based data processing method as described above.
The present application also provides a medium having a program stored thereon for implementing the above-mentioned blockchain-based data processing method, where the program implements the steps of the above-mentioned blockchain-based data processing method when executed by a processor.
The application provides a data processing method, a device, equipment and a medium based on a block chain, wherein when a federal model is constructed in the prior art, all participants directly exchange data and only pay attention to the effect of the model, so that a high-quality federal model is difficult to construct quickly and safely; receiving all federal parameter information which is encrypted and sent to a block chain block by all second participants in the preset federal modeling process; and determining the participation legality of the corresponding second party based on the federal parameter information, and determining the contribution degree of each legal second party. In the application, each participant carries out federal modeling based on block chain encryption instead of directly exchanging data by each participant, so that the safety of data exchange is improved, and further, in the application, based on each federal parameter information, the participation legality of the corresponding second participant is determined in a block chain mode, and the contribution degree of each legal second participant is determined, namely, each second participant is audited through the block chain in the application to clarify the legality and the contribution degree of each participant, but the contribution degree of each participant is not defaulted to be the same, so that the participants with high-quality data are attracted to join, and the participation of the participants with high-quality data is attracted, so that the speed of constructing a high-quality federal model can be accelerated, and the high-quality federal model can be quickly and safely constructed.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application.
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, and it is obvious for those skilled in the art to obtain other drawings without inventive exercise.
Fig. 1 is a schematic flowchart of a first embodiment of a data processing method based on a block chain according to the present application;
fig. 2 is a detailed flowchart of the step of determining the participation validity of the corresponding second participant based on each federal parameter information in the first embodiment of the data processing method based on the blockchain according to the present application;
fig. 3 is a schematic device structure diagram of a hardware operating environment according to an embodiment of the present application.
The objectives, features, and advantages of the present application will be further described with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
In a first embodiment of the data processing method based on the blockchain, referring to fig. 1, the method is applied to a first participant, where the first participant and each second participant perform federated communication connection through the blockchain, and the data processing method based on the blockchain includes:
step S10, sending the target model data to be processed to a block chain block, so that each second participant can respectively perform preset federal modeling with the first participant based on the target model data to be processed;
step S20, receiving each federal parameter information sent to a block chain block by each second participant in an encrypted manner in the preset federal modeling process;
and step S30, determining the participation legality of the corresponding second party based on the information of each federal parameter, and determining the contribution degree of each legal second party.
The method comprises the following specific steps:
step S10, sending the target model data to be processed to a block chain block, so that each second participant can respectively perform preset federal modeling with the first participant based on the target model data to be processed;
in this embodiment, the data processing method based on the blockchain is applied to a first participant, the first participant and each second participant perform federated communication connection through the blockchain, and the first participant and the second participants together form a data processing system based on the blockchain, and the data processing system based on the blockchain belongs to a data processing device based on the blockchain.
It should be noted that the first participant may be a coordinator or any common participant, and particularly, in this embodiment, the first participant may create a first block of a block chain based on the first participant, that is, the first participant (coordinator) initiates a first federal learning task and creates a first block chain block, in the first block chain block, the first participant writes block chain contents such as target to-be-processed model data (including an initial model, initial model parameters, and a performance detection protocol), and each second participant acquires block chain contents of the first block chain block based on the block chain (by means of downloading or the like), and then each second participant performs two-to-two federal communications with the first participant based on the downloaded block chain contents of the first block chain block (different federals may be at the same time or at different times), and after each time with the second participant, the first participant can calculate the lifting degree of the corresponding federate post model, that is, for each second participant, after the content of the blockchain of the first blockchain block is obtained, iteratively train the initial model based on local data, and send corresponding second model parameters (or models) to the first participant after first preset iteration times or after the initial model converges, the first participant fuses with the first model parameters obtained by self-training based on the second model parameters (that is, during the training of the participants, the first participant synchronously performs model training based on self-data), so as to obtain first fusion parameters, and determine the lifting degree of the model corresponding to each second participant at this time, so as to obtain the model contribution degree of each second participant.
It should be noted that the first participant may also be any ordinary participant who obtains a write right (with the largest model contribution degree) in the last round of competition, that is, in this embodiment, the first participant may be changed, the first participant issues target to-be-processed model data to the block chain block for other second participants to obtain, after the second participant obtains the target to-be-processed model data, each second participant performs pairwise federation with the first participant based on the downloaded target to-be-processed model data, after each time of federation with the first participant, the first participant calculates a promotion degree of a model after the federation of the second participant at this time, specifically, for the second participant, iteratively trains a first target to-be-processed model in the target to-be-processed model data based on local data, and a fourth participant corresponding to the second iteration after the second iteration is performed for a preset number of times or after the first target to-be-processed model is converged The model parameters are sent to the first participant, the first participant performs fusion with third model parameters obtained by self training based on the fourth model parameters (namely, when each second participant trains, the first participant performs model training synchronously based on self data), second fusion parameters are obtained, and the model promotion degree corresponding to the second participant after the federation at this time is determined.
Step S20, receiving each federal parameter information sent to a block chain block by each second participant in an encrypted manner in the preset federal modeling process;
receiving each piece of federal parameter information which is sent to a block chain block by each second participant in an encrypted manner in the preset federal modeling process, wherein each piece of federal parameter information comprises information such as a model, a model parameter and a gradient of the model parameter, namely, in the embodiment, each second participant iteratively trains an initial model based on local data in the preset federal modeling process, and sends corresponding second model parameters (or model parameter gradients) to the first participant in an encrypted manner after corresponding iteration times or convergence of the initial model.
Specifically, for example, in this embodiment, each participant may be a shopping website, on which data of the user on the item, such as rating data, click data, purchase data, and the like, can be obtained, taking model training as Matrix Factorization (MF for short), the MF decomposes the rating Matrix of the item of each participant user into a product of two sub-matrices (model training is to confirm the way of decomposition, i.e. to confirm U and V, so that the loss function or optimization equation is minimum), where the Matrix X is e.g. Rn×mThe behavior data of the user on the item, n is the number of the user, m is the number of the item, and X (i, j) epsilon R represents the behavior of the user i on the item j, such as scoring data. U is formed by Rn×kRepresenting a user interest matrix, V ∈ Rk×mRepresenting an article clustering matrix, wherein k is less than n, m, and for a participant, the following optimization equation needs to be solved to obtain U and V;
wherein | X-UV | Y phosphor2Is the sum of the squares of the errors between X and UV;
representing the sum of squares of the U and V elements as a regularization term; λ is a hyper-parameter of the regularization term, and it should be noted that U and V may be initialized first, and after initialization, the above optimization equation may be solved by a gradient descent method to obtain U and V that ultimately meet requirements (i.e., each participant obtains federal parameter information based on local data).
In the scenario of joint modeling of N participants, then
If all the participants have the same user, i.e. the number n of users of the behavior matrix is the same, all the behavior matrix X is decomposediAnd sharing a global user interest matrix to construct a joint recommendation system, in this embodiment, the following joint optimization equation needs to be solved to obtain U and V.
In order to solve the joint optimization equation to obtain U and V, and to protect user privacy and data security, an update gradient Δ U (i.e., federal parameter information) of the user interest feature matrix of each second party is required, and the update gradient is encrypted and sent to the first party based on the blockchain.
And step S30, determining the participation legality of the corresponding second party based on the information of each federal parameter, and determining the contribution degree of each legal second party.
Determining participation legitimacy of the corresponding second party based on the federal parameter information, specifically, determining participation legitimacy of the corresponding second party by using a performance detection protocol acquired by the second party, where it should be noted that, in this embodiment, determining participation legitimacy of the corresponding second party based on the performance detection protocol may be: after determining that the model parameters corresponding to the second participant to be subjected to the performance detection are obtained, whether the performance is improved or not is determined, if the performance is improved, the corresponding second participant is a participant participating in a legal state, otherwise, if the performance is reduced, the corresponding second participant is a participant participating in an illegal state, specifically, for example, federate learning based on a block chain is a model for training and identifying cats, the federate parameter information may be parameter information having a cat determination characteristic such as cat ears or parameter information having a cat determination characteristic such as cat hair color, a certain second participant sends the a federate parameter information to block chain blocks for obtaining by other second participants and the first participant, specifically, other second participants perform local training adjustment based on the a federate parameter information, for example, the a federate parameter information is data of cat ears in an interest matrix, and correspondingly selecting data of all cat ears from the local data by each other second party, constructing adjustment data, and performing performance test based on the adjustment data, wherein if the performance of each other second party is improved, the corresponding second party has participation legality, otherwise, if the performance is reduced, the corresponding second party does not have participation legality.
In this embodiment, the participation validity of the corresponding second party may also be determined in other manners, and specifically, the step of determining the participation validity of the corresponding second party based on each piece of federal parameter information includes:
step S31, determining the associated participants respectively associated with each second participant;
in this embodiment, the first participant determines the associated participants respectively associated with the second participants, for example, the participants associated with the participant a, that is, the participants the same as the user, are determined to be the participant b and the participant c, respectively, wherein the associated participants may also be determined in other manners, which is not limited herein.
Step S32, receiving the related participants based on the block link, and verifying the federal parameter information of the corresponding second participants to obtain a verification result;
the method comprises the steps that a correlation participant federation obtains a correlation model corresponding to federation parameter information of a second participant and corresponding local federation parameter information, and the correlation model is verified based on a first preset verification data set of the correlation participant to obtain a verification result;
and receiving the associated participants based on the block chain, and verifying the federal parameter information of the corresponding second participant to obtain a verification result, specifically, after the associated participants verify a certain participant, placing the verification result in the block chain block for the first participant to obtain.
The method comprises the steps that a correlation participant federal corresponds to federal parameter information of a second participant and local federal parameter information to obtain a correlation model, namely, aggregate federal parameter information (including verification gradient information) is obtained through the federal parameter information of the second participant and the local federal parameter information, a target model to be processed is adjusted based on the aggregate federal parameter information to obtain the correlation model, the correlation model is verified based on a first preset verification data set of the correlation participant to obtain a verification result, and the verification result comprises a forward feedback result (accuracy improvement and the like) and a reverse feedback result.
And step S33, determining the participation validity of the corresponding second party based on the verification result.
And determining the participation legality of the corresponding second party based on the verification result, specifically, if the verification result is a forward feedback result, determining that the corresponding second party has the participation legality, and if the verification result is a reverse feedback result, determining that the corresponding second party does not have the participation legality.
Wherein the step of determining the contribution of each legitimate second party comprises:
step S34, determining each federal model correspondingly determined by the first participant based on each federal parameter information;
step S35, obtaining a second preset verification data set of the first party;
step S36, predicting the second preset verification data set based on the federal models respectively to obtain prediction results;
and step S37, determining the contribution degree of each second participant according to each prediction result.
In this embodiment, a manner of determining the contribution of the second party is provided, in which first, each federal model that is determined by the first party correspondingly based on federal parameters (which may include gradients and the like) of each second party is determined, a second preset verification data set of the first party is obtained, the second preset verification data set is input into each federal model, the second preset verification data set is predicted based on each federal model, each prediction result is obtained, and the contribution of each second party is determined according to the level of the accuracy of each prediction result and the incidence relation between the accuracy of each prediction result and the contribution.
In this embodiment, it should be noted that it is determined that each federal model may be simultaneous or at different times, and specifically, for example, after receiving a federal parameter corresponding to a second party, a first party may perform two federals to obtain a federal model, or after receiving a federal parameter corresponding to the second party, the first party may perform two federals at the same time to obtain a federal model, and further obtain a second preset verification data set of the first party; predicting the second preset verification data set based on each federal model to obtain each prediction result; and correspondingly determining the contribution degree of each second party based on each prediction result.
Obtaining a second preset validation dataset for the first party; predicting the second preset verification data set based on each federal model to obtain each prediction result; and correspondingly determining the contribution degree of each second party based on each prediction result. In this embodiment, after determining the contribution degrees of the participants, the contribution degrees may also be recorded through a blockchain.
After the step of determining the participation legality of the corresponding second party based on each piece of federal parameter information and determining the contribution degree of each legal second party, the method includes:
step S40, acquiring preset modeling optimization reward data;
and step S50, distributing the reward amount in the modeling optimization reward data to each participant based on the contribution degree.
In this embodiment, preset modeling optimization reward data is further acquired, specifically, a preset modeling optimization reward amount is acquired, and the reward amount in the modeling optimization reward data is distributed to each participant based on the size of the contribution degree, wherein the reward amount in the modeling optimization reward data is distributed to each participant according to a preset incidence relation between the size of the contribution degree and the reward amount, wherein the larger the contribution degree is, the more the reward amount is divided, and therefore, the participant with high quality is attracted to the federal modeling.
It should be noted that, in this embodiment, after the contribution degree is obtained, the participants in the top rank may be broadcast through the blockchain, so as to promote the participants with the high contribution degree, so as to attract the participants with high-quality data to participate in the federation, and improve the rate of federated modeling.
It should be noted that, in this embodiment, the contribution degree of each legitimate second party may be determined after one iteration, or the contribution degree of each legitimate second party may be determined after multiple iterations, which is not specifically limited herein.
Compared with the prior art that when a federal model is built, all participants directly exchange data and only pay attention to the effect of the model, so that a high-quality federal model is difficult to build quickly and safely, the method, the device, the equipment and the medium for processing the data based on the block chain are provided, and in the method, target model data to be processed are sent to a block chain block so that all second participants can respectively perform preset federal modeling with a first participant based on the target model data to be processed; receiving all federal parameter information which is encrypted and sent to a block chain block by all second participants in the preset federal modeling process; and determining the participation legality of the corresponding second party based on the federal parameter information, and determining the contribution degree of each legal second party. In the present application, instead of each participant directly exchanging data, each participant performs federal modeling based on blockchain encryption, thereby improving the security of data exchange and, further, in the embodiment, based on each federal parameter information, the participation legality of the corresponding second participant is determined in a block chain mode, the contribution degree of each legal second participant is determined, namely, the auditing of each second participant is carried out through the block chain in the application, so as to clarify the legality and contribution degree of each participant, instead of defaulting that the contribution degrees of all the participants are the same, thereby being convenient for attracting the participants with high-quality data to join, since the participation of the participants with high-quality data is attracted, the speed of constructing the high-quality federal model can be increased, and the high-quality federal model can be quickly and safely constructed.
Based on the first embodiment, in another embodiment of the data processing method based on the block chain, each second party encrypts and sends each federal parameter information in the preset federal modeling process;
before the step of determining the participation legality of the corresponding second party based on each piece of federal parameter information and determining the contribution degree of each legal second party, the method includes:
step A1, determining whether encrypted federal parameter information of all second participants is received;
it should be noted that different second participants have different rates of iterating the model based on the target to-be-processed model data, and therefore, different time of sending the federal parameter information to the first participant based on the blockchain is also different.
And step A2, if the encrypted federal parameter information of all the second participants is received, cooperatively triggering and decrypting all the federal parameter information.
In this embodiment, if the encrypted federal parameter information of all the second participants is received, the decryption of each federal parameter information is cooperatively triggered, and if the encrypted federal parameter information of all the second participants is not received, the decryption of each federal parameter information cannot be performed. That is, in the present embodiment, decryption needs to be performed through cooperation of multiple parties, so as to ensure security.
The step of determining the contribution of each legitimate second party comprises:
step B1, acquiring each legal second participant sending each federal parameter information to each sending time in the block chain block in the preset federal modeling process;
in this embodiment, each legal second participant sends each federal parameter information to each sending time in the block chain block in the preset federal modeling process, specifically, if the modeling capability or the calculation capability of a certain second participant is strong, the federal parameter information is obtained first, then each federal parameter information is sent to the block chain block first, if the modeling capability or the calculation capability of another second participant is strong, then the federal parameter information is obtained, then each federal parameter information is sent to the block chain block, in this embodiment, each legal second participant is recorded based on the block chain and sends each federal parameter information to each sending time in the block chain block in the preset federal modeling process, and the first participant can apply for the block chain block to obtain each legal second participant in the preset federal modeling process, and sending each federal parameter information to each sending time in the block chain block.
And step B2, determining the contribution degree of each legal second party based on the sending time.
Determining the contribution degree of each legitimate second participant based on the length of the sending time, specifically, determining the contribution degree of each legitimate second participant based on a preset association relationship between the length of the sending time and the contribution degree, for example, if the sending time is 10S, the contribution degree is 10%, and if the sending time is 5S, the contribution degree is 20%, in this embodiment, ranking each second participant based on the size of the contribution degree.
In this embodiment, each legal second participant is taken to send each federal parameter information to each sending time in the block chain block in the preset federal modeling process; and accurately determining the contribution degree of each legal second party based on the length of the sending time.
In another embodiment of the present application, a method for processing data based on a block chain is provided,
the data processing method based on the block chain comprises the following steps:
and step C1, saving the federal parameter information reported by each second party based on the block chain block record, and saving the contribution degree of each second party based on the block chain block record.
In this embodiment, each piece of federal parameter information reported by each second participant is recorded based on the block chain block, and a timestamp, i.e., a reporting time, when each participant reports is recorded, and in addition, the block chain block also records a gradient value, a verification gradient, a state of a model, and the like, reported by each participant, and records a contribution degree of each second participant.
In this embodiment, the federate parameter information reported by each second party based on the block chain block record is saved, and the contribution degree of each second party based on the block chain block record is saved, so that subsequent query is facilitated.
Referring to fig. 3, fig. 3 is a schematic device structure diagram of a hardware operating environment according to an embodiment of the present application.
As shown in fig. 3, the block chain-based data processing apparatus may include: a processor 1001, such as a CPU, a memory 1005, and a communication bus 1002. The communication bus 1002 is used for realizing connection communication between the processor 1001 and the memory 1005. The memory 1005 may be a high-speed RAM memory or a non-volatile memory (e.g., a magnetic disk memory). The memory 1005 may alternatively be a memory device separate from the processor 1001 described above.
Optionally, the data processing device based on the blockchain may further include a rectangular user interface, a network interface, a camera, an RF (Radio Frequency) circuit, a sensor, an audio circuit, a WiFi module, and the like. The rectangular user interface may comprise a Display screen (Display), an input sub-module such as a Keyboard (Keyboard), and the optional rectangular user interface may also comprise a standard wired interface, a wireless interface. The network interface may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface).
Those skilled in the art will appreciate that the blockchain based data processing apparatus architecture shown in fig. 3 does not constitute a limitation of blockchain based data processing apparatuses and may include more or fewer components than those shown, or some components in combination, or a different arrangement of components.
As shown in fig. 3, a memory 1005, which is a kind of computer medium, may include therein an operating system, a network communication module, and a block chain-based data processing program. The operating system is a program that manages and controls hardware and software resources of the blockchain-based data processing apparatus, and supports the execution of the blockchain-based data processing program as well as other software and/or programs. The network communication module is used to enable communication between components within the memory 1005, as well as with other hardware and software in the blockchain based data processing system.
In the data processing apparatus based on the blockchain shown in fig. 3, the processor 1001 is configured to execute a data processing program based on the blockchain stored in the memory 1005, and implement the steps of the data processing method based on the blockchain according to any one of the above.
The specific implementation of the data processing device based on the block chain in the present application is basically the same as that of each embodiment of the data processing method based on the block chain, and is not described herein again.
The present application further provides a data processing apparatus based on a block chain, which is applied to a first participant, where the first participant and each second participant perform federated communication connection via the block chain, and the data processing apparatus based on the block chain includes:
the modeling module is used for sending the target model data to be processed to the block chain block so that each second participant can respectively perform preset federal modeling with the first participant based on the target model data to be processed;
the receiving module is used for receiving each federal parameter information which is encrypted and sent to the block chain block by each second participant in the preset federal modeling process;
and the first determining module is used for determining the participation legality of the corresponding second party based on the federal parameter information and determining the contribution degree of each legal second party.
Optionally, the first determining module includes:
the first determining unit is used for determining the associated participants respectively associated with the second participants;
the first receiving unit is used for receiving the associated participants based on the block link and verifying the federal parameter information of the corresponding second participant to obtain a verification result;
the method comprises the steps that a correlation participant federation obtains a correlation model corresponding to federation parameter information of a second participant and corresponding local federation parameter information, and the correlation model is verified based on a first preset verification data set of the correlation participant to obtain a verification result;
and the second determining unit is used for determining the participation legality of the corresponding second party based on the verification result.
Optionally, the first determining module further includes:
the third determining unit is used for determining each federal model correspondingly determined by the first participant based on each federal parameter information;
a first obtaining unit, configured to obtain a second preset verification data set of the first party;
the prediction unit is used for predicting the second preset verification data set based on each federal model to obtain each prediction result;
and a fourth determining unit, configured to correspondingly determine the contribution degree of each second participant based on each prediction result.
Optionally, each second party encrypts and sends each federal parameter information in the preset federal modeling process;
the block chain-based data processing apparatus further includes:
the second determining module is used for determining whether encrypted federal parameter information of all the second participants is received;
and the decryption module is used for cooperatively triggering decryption of all the federal parameter information if the encrypted federal parameter information of all the second participants is received.
Optionally, the first determining module further includes:
the second acquisition unit is used for acquiring each legal second participant and sending each federal parameter information to each sending time in the block chain block in the preset federal modeling process;
and a fifth determining unit, configured to determine, based on the length of the sending time, the contribution degree of each legitimate second party.
Optionally, the data processing apparatus based on a blockchain further includes:
the acquisition module is used for acquiring preset modeling optimization reward data;
and the distribution module is used for distributing the reward amount in the modeling optimization reward data to each participant based on the contribution degree.
Optionally, the data processing apparatus based on a blockchain further includes:
and the storage module is used for storing the federal parameter information reported by each second participant based on the block chain block record and storing the contribution degree of each second participant based on the block chain block record.
The specific implementation of the data processing apparatus based on the blockchain in the present application is substantially the same as that of each embodiment of the data processing method based on the blockchain, and is not described herein again.
The present invention provides a medium, and the medium stores one or more programs, which can be further executed by one or more processors for implementing the steps of any one of the above block chain based data processing methods.
The specific implementation of the medium of the present application is substantially the same as that of each embodiment of the above data processing method based on a block chain, and is not described herein again.
The above description is only a preferred embodiment of the present application, and not intended to limit the scope of the present application, and all modifications of equivalent structures and equivalent processes, which are made by the contents of the specification and the drawings, or which are directly or indirectly applied to other related technical fields, are included in the scope of the present application.
Claims (10)
1. The block chain-based data processing method is applied to a first participant, the first participant and each second participant are in federated communication connection through a block chain, and the block chain-based data processing method comprises the following steps:
sending the target model data to be processed to a block chain block for each second participant to respectively perform preset federal modeling with the first participant based on the target model data to be processed;
receiving all federal parameter information which is encrypted and sent to a block chain block by all second participants in the preset federal modeling process;
and determining the participation legality of the corresponding second party based on the federal parameter information, and determining the contribution degree of each legal second party.
2. The blockchain-based data processing method according to claim 1, wherein the step of determining the participation legitimacy of the corresponding second participant based on the respective federal parameter information includes:
determining the associated participants respectively associated with the second participants;
receiving the associated participants based on the block link, and verifying the federal parameter information of the corresponding second participants to obtain a verification result;
the method comprises the steps that a correlation participant federation obtains a correlation model corresponding to federation parameter information of a second participant and corresponding local federation parameter information, and the correlation model is verified based on a first preset verification data set of the correlation participant to obtain a verification result;
and determining the participation legality of the corresponding second party based on the verification result.
3. The blockchain-based data processing method according to claim 1, wherein the step of determining the contribution of each of the legitimate second parties comprises:
determining each federal model correspondingly determined by the first participant based on each federal parameter information;
obtaining a second preset validation dataset for the first party;
predicting the second preset verification data set based on each federal model to obtain each prediction result;
and correspondingly determining the contribution degree of each second party based on each prediction result.
4. The blockchain-based data processing method according to claim 1, wherein each second party encrypts and sends each federal parameter information in the preset federal modeling process;
before the step of determining the participation legality of the corresponding second party based on each piece of federal parameter information and determining the contribution degree of each legal second party, the method includes:
determining whether encrypted federal parameter information of all second parties is received;
and if the encrypted federal parameter information of all the second participants is received, cooperatively triggering and decrypting all the federal parameter information.
5. The blockchain-based data processing method according to claim 4, wherein the step of determining the contribution of the legitimate second parties comprises:
acquiring sending time for sending each federal parameter information to a block chain block by each legal second participant in the preset federal modeling process;
and determining the contribution degree of each legal second participant based on the sending time.
6. The blockchain-based data processing method according to claim 1, wherein after the step of determining the participation legitimacy of the corresponding second participant based on the respective federal parameter information and determining the contribution of the respective legitimate second participant, the method comprises:
acquiring preset modeling optimization reward data;
and distributing the reward amount in the modeling optimization reward data to each participant based on the contribution degree.
7. The blockchain-based data processing method according to any one of claims 1 to 6, wherein the blockchain-based data processing method includes:
and saving each federal parameter information reported by each second party based on the block chain block record, and saving the contribution degree of each second party based on the block chain block record.
8. A data processing device based on a block chain is applied to a first participant, the first participant and each second participant are in federated communication connection through the block chain, and the data processing device based on the block chain comprises:
the modeling module is used for sending the target model data to be processed to the block chain block so that each second participant can respectively perform preset federal modeling with the first participant based on the target model data to be processed;
the receiving module is used for receiving each federal parameter information which is encrypted and sent to the block chain block by each second participant in the preset federal modeling process;
and the first determining module is used for determining the participation legality of the corresponding second party based on the federal parameter information and determining the contribution degree of each legal second party.
9. A blockchain-based data processing apparatus, characterized in that the blockchain-based data processing apparatus comprises: memory, processor and program stored on the memory for implementing the blockchain based data processing method, other participants or blockchain
The memory is used for storing a program for realizing the data processing method based on the block chain;
the processor is configured to execute a program implementing the blockchain based data processing method to implement the steps of the blockchain based data processing method according to any one of claims 1 to 7.
10. A medium having stored thereon a program for implementing a blockchain-based data processing method, the program being executable by a processor to implement the steps of the blockchain-based data processing method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010822839.9A CN111950739A (en) | 2020-08-13 | 2020-08-13 | Data processing method, device, equipment and medium based on block chain |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010822839.9A CN111950739A (en) | 2020-08-13 | 2020-08-13 | Data processing method, device, equipment and medium based on block chain |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111950739A true CN111950739A (en) | 2020-11-17 |
Family
ID=73343516
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010822839.9A Pending CN111950739A (en) | 2020-08-13 | 2020-08-13 | Data processing method, device, equipment and medium based on block chain |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111950739A (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112434818A (en) * | 2020-11-19 | 2021-03-02 | 脸萌有限公司 | Model construction method, device, medium and electronic equipment |
| CN112700012A (en) * | 2020-12-30 | 2021-04-23 | 深圳前海微众银行股份有限公司 | Federal feature selection method, device, equipment and storage medium |
| CN112784994A (en) * | 2020-12-31 | 2021-05-11 | 浙江大学 | Block chain-based federated learning data participant contribution value calculation and excitation method |
| CN112801307A (en) * | 2021-04-13 | 2021-05-14 | 深圳索信达数据技术有限公司 | Block chain-based federal learning method and device and computer equipment |
| CN112949868A (en) * | 2021-01-29 | 2021-06-11 | 北京邮电大学 | Asynchronous federal learning method and device based on block chain and electronic equipment |
| CN112949865A (en) * | 2021-03-18 | 2021-06-11 | 之江实验室 | Sigma protocol-based federal learning contribution degree evaluation method |
| CN113051606A (en) * | 2021-03-11 | 2021-06-29 | 佳讯飞鸿（北京）智能科技研究院有限公司 | Block chain mutual communication method of intelligent agent |
| CN113111124A (en) * | 2021-03-24 | 2021-07-13 | 广州大学 | Block chain-based federal learning data auditing system and method |
| CN113298404A (en) * | 2021-06-03 | 2021-08-24 | 光大科技有限公司 | Method and device for determining workload of federal learning participator |
| CN113627086A (en) * | 2021-08-21 | 2021-11-09 | 深圳前海微众银行股份有限公司 | Method, apparatus, medium, and program product for optimizing horizontal federated learning modeling |
-
2020
- 2020-08-13 CN CN202010822839.9A patent/CN111950739A/en active Pending
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112434818A (en) * | 2020-11-19 | 2021-03-02 | 脸萌有限公司 | Model construction method, device, medium and electronic equipment |
| CN112434818B (en) * | 2020-11-19 | 2023-09-26 | 脸萌有限公司 | Model construction method, device, medium and electronic equipment |
| CN112700012A (en) * | 2020-12-30 | 2021-04-23 | 深圳前海微众银行股份有限公司 | Federal feature selection method, device, equipment and storage medium |
| CN112784994A (en) * | 2020-12-31 | 2021-05-11 | 浙江大学 | Block chain-based federated learning data participant contribution value calculation and excitation method |
| CN112784994B (en) * | 2020-12-31 | 2023-03-03 | 浙江大学 | Block chain-based federated learning data participant contribution value calculation and excitation method |
| CN112949868A (en) * | 2021-01-29 | 2021-06-11 | 北京邮电大学 | Asynchronous federal learning method and device based on block chain and electronic equipment |
| CN113051606A (en) * | 2021-03-11 | 2021-06-29 | 佳讯飞鸿（北京）智能科技研究院有限公司 | Block chain mutual communication method of intelligent agent |
| CN112949865B (en) * | 2021-03-18 | 2022-10-28 | 之江实验室 | Joint learning contribution degree evaluation method based on SIGMA protocol |
| CN112949865A (en) * | 2021-03-18 | 2021-06-11 | 之江实验室 | Sigma protocol-based federal learning contribution degree evaluation method |
| CN113111124A (en) * | 2021-03-24 | 2021-07-13 | 广州大学 | Block chain-based federal learning data auditing system and method |
| CN113111124B (en) * | 2021-03-24 | 2021-11-26 | 广州大学 | Block chain-based federal learning data auditing system and method |
| CN112801307B (en) * | 2021-04-13 | 2021-07-06 | 深圳索信达数据技术有限公司 | Block chain-based federal learning method and device and computer equipment |
| CN112801307A (en) * | 2021-04-13 | 2021-05-14 | 深圳索信达数据技术有限公司 | Block chain-based federal learning method and device and computer equipment |
| CN113298404A (en) * | 2021-06-03 | 2021-08-24 | 光大科技有限公司 | Method and device for determining workload of federal learning participator |
| CN113627086A (en) * | 2021-08-21 | 2021-11-09 | 深圳前海微众银行股份有限公司 | Method, apparatus, medium, and program product for optimizing horizontal federated learning modeling |
| CN113627086B (en) * | 2021-08-21 | 2024-03-26 | 深圳前海微众银行股份有限公司 | Method, device, medium and program product for optimizing horizontal federal learning modeling |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111950739A (en) | Data processing method, device, equipment and medium based on block chain | |
| CN111598254B (en) | Federal learning modeling method, device and readable storage medium | |
| CN113159327B (en) | Model training method and device based on federal learning system and electronic equipment | |
| CN110851869B (en) | Sensitive information processing method, device and readable storage medium | |
| US20190050854A1 (en) | Blockchain-based digital data exchange | |
| WO2022016964A1 (en) | Vertical federated modeling optimization method and device, and readable storage medium | |
| CN111340247A (en) | Longitudinal federated learning system optimization method, device and readable storage medium | |
| CN111324812B (en) | Federal recommendation method, device, equipment and medium based on transfer learning | |
| CN112926073A (en) | Federal learning modeling optimization method, apparatus, medium, and computer program product | |
| CN111339412A (en) | Longitudinal federal recommendation recall method, device, equipment and readable storage medium | |
| CN111553744A (en) | Federal product recommendation method, device, equipment and computer storage medium | |
| CN114785556B (en) | Encryption communication method, device, computer equipment and storage medium | |
| CN111860865A (en) | Model construction and analysis method, device, electronic equipment and medium | |
| CN113065857A (en) | Data processing method, device, equipment, medium and product in payment process | |
| CN114691167A (en) | Method and device for updating machine learning model | |
| CN112989186A (en) | Information recommendation model training method and device, electronic equipment and storage medium | |
| JP7422892B2 (en) | Processing machine learning modeling data to improve classification accuracy | |
| CN113792892A (en) | Federal learning modeling optimization method, apparatus, readable storage medium, and program product | |
| CN112949866A (en) | Poisson regression model training method and device, electronic equipment and storage medium | |
| CN114168295A (en) | Hybrid architecture system and task scheduling method based on historical task effect | |
| CN111652382B (en) | Data processing method, device and equipment based on block chain and storage medium | |
| CN107005576A (en) | The bridge joint matching identification symbol for link identifiers is generated from server log | |
| CN112308238A (en) | Analytical model training method and device, electronic equipment and storage medium | |
| CN111368314A (en) | Modeling and predicting method, device, equipment and storage medium based on cross features | |
| CN114677138A (en) | Data processing method, data processing equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |