WO2023069050A1

WO2023069050A1 - Blockchain and distributed ledger technology-based model and data set management method

Info

Publication number: WO2023069050A1
Application number: PCT/TR2022/051096
Authority: WO
Inventors: Emre SAFAK; Mesut GOZUTOK; Engin FIRAT
Original assignee: Havelsan Hava Elektronik San. Ve Tic. A.S.
Priority date: 2021-10-22
Filing date: 2022-10-05
Publication date: 2023-04-27

Abstract

The invention relates to a method that allows the management of training data sets and models for machine learning applications using blockchain and ensures the security and reliability of the data. The method developed by the invention allows the training data sets and models to be versioned using the blockchain and the validity of a training data set and model used by a developer can be checked. The use of the invention in transfer learning applications is also described.

Description

BLOCKCHAIN AND DISTRIBUTED LEDGER TECHNOLOGY-BASED MODEL AND DATA SET MANAGEMENT METHOD

Technical Field

The invention relates to a method that allows the management of training data sets and models for machine learning applications using blockchain and ensures the security and reliability of the data.

Prior Art

In the present state of the art, the training data sets used in machine learning applications and the models obtained as a result of the training carried out with these sets are stored and used uncontrolled. Failure to check the validity of training data sets and models may cause unexpected problems in the trained systems, as well as making the systems vulnerable to attack with the interventions to be made to the sets and models. This poses a risk in terms of transfer learning applications, where models are reused for different purposes. Therefore, there is a need to develop solutions to ensure the security and reliability of training data sets and models by checking their validity.

In the publication titled “Leveraging blockchain to make machine learning models more accessible” (Justin D. Harris, 2019) (https://www.microsoft.com/en- us/research/blog/leveraging-blockchain-to-make-machine-leaming-models-more-accessible/), it is proposed to create a distributed and collaborative structure for the development of models used in artificial intelligence applications by using blockchain applications.

In the publication titled “Blockchain for Al: Review and open research challenges” (Salah, K., Rehman, M. H.U., Nizamuddin, N., & Al-Fuqaha, A. (2019). Blockchain for Al: Review and open research challenges. IEEE Access, 7, 10127-10149.), blockchain applications for use in the field of artificial intelligence have been compiled. It has been stated that machine learning algorithms work better when data is taken from reliable sources and the effects of blockchain applications on data security have been mentioned. It has been mentioned that the blockchain allows for unchangeable and safe storage of model versions. The use of proof of stake algorithms was also mentioned to create a consensus. Document No. CN110991622A discloses a method based on a blockchain network that enables sharing and updating of a model associated with machine learning. The use of proof of stake was also mentioned to create a consensus.

In document CN109361740 A, a method for forming blocks to be added to the end of the chain is disclosed in a blockchain application. It has been stated that a delegated proof of stake application based on factors including economic situation and hardware competence, etc. has been utilized to create a consensus. It is also mentioned to create a time limit for the creation and validation of the block.

In the publication titled “2-hop blockchain: Combining proof-of-work and proof-of-stake securely” (Duong, T., Fan, L., Katz, J., Thai, P., & Zhou, H. S. (2020, September). 2-hop blockchain: Combining proof-of-work and proof-of-stake securely. In European Symposium on Research in Computer Security (pp. 697-712). Springer, Cham.), a structure that combines the features of proof of work and proof of stake for the creation of a consensus based on both transaction power and stake is explained. The problem solution to be realized concerning the proof of work was used as an indicator of transaction power. Proof of stake corresponds to the user’s assets associated with the blockchain in question. The document also shows the combination of different consensus methods.

Objects and Brief Description of the Invention

The object of the present invention is to develop a method that ensures the security and reliability of training data sets and models for machine learning applications, in particular a method that allows the control of the validity of training data sets and models with blockchain applications.

With the invention, a method for managing training data sets and models using blockchain was developed. With the method, after the approval of a data set, a hash is generated and information and hash related to the data set are stored on the distributed ledger network. The approved data set is also stored on a server. Following the approval of a model developed using a data set that is approved and can be validated using hash information, a hash related to this is generated and information about the model and the relevant hash is also stored on the distributed ledger. The approved model is also stored on a server. A proof of stake consensus algorithm based on transaction power is used to determine and approve the validators. This consensus algorithm is based on both transaction power and the number of historical validations.

To determine the validator to perform the validation transaction, the validators are compared according to the transaction power presented to the network and the number of completed validation transactions. The identified validator is redeveloping the model with the training data set and training information. Then, a hash is calculated from the model and model-related performance metrics, and a record of the model is created on the distributed ledger by validating the model by comparing the hash.

The invention ensures that data sets and models can be used safely in transfer learning applications and model versions can be stored reliably. By checking the validity of the training data sets and models, it can be ensured that the model and training data sets that are not approved are not used. The history of training data sets and models can be monitored and it can be seen who provides the development and approvals.

Detailed Description of the Invention

The distributed ledger technology-based model and data set management method developed to achieve the objects of the present invention are shown in the accompanying figures.

Figure 1 This is a flowchart illustrating an exemplary embodiment of the invention for use on a private network.

Figure 2 This is a flowchart illustrating an exemplary embodiment of the invention for use on a public network.

The method of the invention essentially comprises the following steps: developing a model using a specific training data set, running the developed model with a test data set, obtaining model and performance metrics for the relevant test data set, generating a developer hash from model and performance metrics,

- transmitting the developer hash to a validator, - training of the relevant model by the validator using the same training data set and then reobtaining the model and performance metrics after running with the same test data set, generating a validator hash from the model and performance metrics obtained by the validator, storing the validator hash on the distributed ledger network if the validator hash and developer hash are the same.

The invention can be implemented with different distributed ledger data structures and techniques such as Directed Acyclic Directed Acyclic Graph (Dag), Tempo, and Holochain, especially the blockchain where each record is created to keep the hash information of the previous record. Public or private distributed ledger networks can be used. The invention is exemplified using blockchain below.

The training data sets are also validated in the preferred embodiment of the invention. For this, a training data set hash is generated from a training data set and a related unique identifier and this training data set hash is stored on the distributed ledger network. Before storing the hash of a training data set, the source from which the training data set is obtained, the attribute information of the training data set, the type of data, the number of data, the date of data acquisition, explanations, digital signature, and other information can be added and submitted to the approval of the validators.

The validators consider whether it was signed when approving a training data set. For the training data set approved by the majority of validators, a record is created on the distributed ledger network containing the relevant training data set hash, the source from which it was obtained, the attribute information, the type of data, the number of data, the date of data acquisition, other information, explanations, and approval information. The training data set recorded in the distributed ledger network is stored in a data server. In case the training data set is updated, the previous data information is reported as the source information from which it was obtained, and the blockchain that allows the versioning of the training data set is created by adding the new training data set hash generated as a result of the update. Model development is carried out using specific training data sets stored on the data server. The validity of the training data set used can be checked thanks to the relevant blockchain.

The model consists of the weights calculated as a result of the training and the algorithm used. For the approval of the developed model, the test source code, test data set, model metrics, that is, the training data set hash of the training data set used, training source code, hyperparameters (training step [epoch], learning speed, model optimizer, etc.), environment requirements (programming language, libraries, etc.), training date and performance metrics (accuracy, precision, loss, etc.) are transmitted to the distributed ledger network. By checking whether the training data set used is approved or not, if it is determined that it is approved, the developer hash is calculated using model metrics and performance metrics.

The generated developer hash is broadcasted to the network by a consensus algorithm. The model can be approved by comparing the validator hash obtained with the model and performance metrics obtained by the validators using the same training data set and test data set with the developer and the developer hash. Preferably, the development (training) and testing of the model are carried out by a validator, and the resulting model and performance metrics are broadcasted on the network by this validator. Other validators use these metrics to calculate the validator hash and compare it with the developer hash.

A record is created of the training data set hash of the training data set used, training source code, test source code, test data set hash, hyperparameters, media requirements, training date, performance metrics, developer/validator hash, and approval information on the distributed ledger network by the validator who trained the model for the approved model and obtained the metrics. In the distributed ledger network, the recorded model is stored in the model server and the test data set is stored in the data set server.

In an embodiment of the invention for use in private distributed ledger networks, the validation transaction is performed with a proof of stake algorithm based on transaction power. The validation transaction is performed by a validator, while the control of the transaction is performed by more than one secondary validator other than the validator. The selection of the validator is carried out according to the transaction power reported by the candidate devices and the number of past validation transactions completed by the devices, preferably according equally to transaction power and the number of past validation transactions. For candidate devices that have not completed any validation transaction, the one with the least number of unfinished transactions is selected. It is also considered which of the central processor (CPU) or graphic processor (GPU) will be used according to the type of validation to be performed during the determination of the transaction power. In this embodiment, the stake of the validators is the system source allocated to the validation transaction.

In the selection of the validator, the candidate devices are selected from the nodes that do not perform an active validation transaction. If the validator is disconnected or the validation transaction cannot be performed within a maximum validation period determined according to the transaction power, a new validator is determined.

The proof of stake algorithm based on transaction power also allows multiple validation transactions to be performed simultaneously by selecting more than one validator.

In an embodiment of the invention for use in public distributed ledger networks, the developer is recorded before the validation request. For this, the developer receives the timestamp when the training is started, the developer ID number, and request ticket information obtained from a randomly generated number from a server. After the development of the model, a validation data set is created by the developer using the training data set hash information, model and training source codes, performance metrics, and other information and request ticket information and transmitted to the network as a validation request. The validation fee is deposited into the system by the developer. The validator is selected as the highest stake depositor from a pool of validator candidates depositing an amount corresponding to a minimum stake or a stake above it into the system, and a smart contract is initiated. The validator re-generates hash information related to model and performance metrics by performing model training and testing again. If the hash information obtained by the validator is the same as the hash information obtained by the developer, the validation information is transmitted to the distributed chain network and a validation data set is recorded to the request server, the validation fee is transferred to the validator’s account and the smart contract is terminated. If the hash information obtained by the validator and the hash information obtained by the developer are different, the transaction is canceled and the validation fee is transferred to the account of the validator. Cryptocurrency, digital currency, and similar payment instruments compatible with smart contracts can be used to transfer the fee. If the validation transaction cannot be completed within a maximum validation period determined in the form of certain times of the development period, a new validator is determined by transferring the minimum stake of the stake deposited by the validator to the developer’s account.

Training data set hashes, developer hashes and validator hashes can be created with various algorithms including MD5, SHA-1, SHA-2, SHA-3, RIPEMD-160, Whirlpool, BLAKE2, BLAKE3.

The invention provides an advantage, especially in transfer learning applications where a model can be reused for different purposes. The main problem in transfer learning is to determine whether the model to be used is original or not. The method developed by the invention enables the use of original models for transfer learning, as well as the display of the entire development history of the models. The developer can access the approved original model that s/he wants to use through the model server. In addition, if the models recorded in the model server are redeveloped for transfer learning, less computing power will be required to validate the new model, as the updated model has already been validated. During the storage of the new model, the developer/validator hash of the previous model is also submitted and the blockchain that allows the model to be versioned is also created. If a model is redeveloped from the scope of transfer learning applications, validation and recording can be performed by using the recorded model developer/validator hash, training data set hash, training source code, test source code, test data set, hyperparameters, environment requirements, training date, and performance metrics. Before validation, it is checked that the training data set and the model related to the model developer/validator hash and the training data set hash are approved. The validation of the redeveloped model can also be carried out as described above for the original model using the consensus algorithm.

Claims

1. A method that enables the management of training data sets and models for machine learning applications, characterized in that; it comprises the following steps: developing a model using a specific training data set,

- running the developed model with a test data set, obtaining model and performance metrics for the relevant test data set,

- generating a developer hash from model and performance metrics,

- transmitting the developer hash to a validator,

- training of the relevant model by the validator using the same training data set and then reobtaining the model and performance metrics after running with the same test data set,

- generating a validator hash from the model and performance metrics obtained by the validator, storing the validator hash on the distributed ledger network if the validator hash and developer hash are the same.

2. A method according to claim 1, characterized in that; it comprises the step of generating a training data set hash from a training data set and a related unique identifier for the validation of training data sets and storing this training data set hash on a distributed ledger network.

3. A method according to claim 1, characterized in that; the selection of the validator is carried out according to the transaction power reported by the candidate devices and the number of past validation transactions completed by the devices.

4. A method according to claim 3, characterized in that; the validator hash and the developer hash are compared by more than one secondary validator.

5. A method according to claim 1, characterized in that; the validator is selected as the highest stake depositor from a pool of validator candidates depositing an amount corresponding to a minimum stake or a stake above it into the system.

8

6. A method according to claim 3 or 5, characterized in that; multiple validators are selected for simultaneous execution of multiple validation transactions.

9