CN116663049A

CN116663049A - Medical image segmentation cooperation method based on blockchain network

Info

Publication number: CN116663049A
Application number: CN202310542294.XA
Authority: CN
Inventors: 苏红; 罗阳; 吴锡
Original assignee: Chengdu University of Information Technology
Current assignee: Chengdu University of Information Technology
Priority date: 2023-05-15
Filing date: 2023-05-15
Publication date: 2023-08-29

Abstract

The invention relates to a medical image segmentation cooperation method based on a blockchain network, which comprises the steps that firstly, each medical institution carries out local deep learning training modeling on respective medical data, a local weight file is trained and generated, the local weight file is submitted to the blockchain network as a transaction after being signed by the medical institution to which the local weight file belongs, nodes with downloading authority in the blockchain network download the local weight file until all medical institutions generate the respective local weight file, the local weight file is aggregated to generate global weight after the blockchain network receives the local weight data of a preset quantity, and the global weight with the best effect is saved as a final image segmentation neural network parameter after limited iteration circulation. The frame of the block chain plus deep learning is a decentralization framework, the decentralization mechanism ensures the honest behavior of the participants to a great extent, the medical data of each node cannot be accessed by other devices, and the data privacy is better protected.

Description

Medical image segmentation cooperation method based on blockchain network

Technical Field

The invention relates to the crossing field of blockchain and medical image processing, in particular to a medical image segmentation cooperation method based on a blockchain network.

Background

Deep learning has developed rapidly in recent years and is the main research direction in the field of artificial intelligence [1]. Many ideas of deep learning directions are realized with greatly improved computing power. It is known that to train a highly accurate and robust deep learning model, the core factor is the need for a large amount of high quality training data. As insufficient data can lead to under-fitting of the results of model training. Many methods have been proposed to address the problem of insufficient deep learning input samples. Such as data enhancement, also known as data amplification, over-sampling and under-sampling, artificial data synthesis, etc. This approach does have a significant effect on improving the robustness, generalization of the model, allowing limited data to yield more data value. But the raw data plays a decisive role. How to obtain data of all parties has been a challenge for deep learning applications.

In addition, privacy concerns are also an important factor in hampering data collection. Some data cannot be safely shared due to some specificity, and can be used for further research, such as medical data. The medical institutions cannot communicate with each other due to the reasons of patient privacy, interests of related parties and the like, and a large-size data island is formed. It is also known that in order to train out models at the same level as medical professionals, to achieve the clinically required accuracy, it is necessary to provide AI algorithms with a large number of cases that can adequately represent the clinical environment to participate in the training. For example, a physician of ordinary skill may take tens or even decades to grow into an expert, and the case of a manager may be hundreds of thousands of cases, and the AI algorithm may need to learn about the same scale of cases to achieve the same accuracy. Unfortunately, the largest database currently open is a major segment from meeting the requirements of artificial intelligence training. Even medical calves may see hundreds of cases for life if rare diseases are considered more troublesome. Plus privacy reasons or reasons for local government, local legal regulations. Many medical data is wasted and left idle. Many artificial intelligence enterprises or artificial intelligence teams in hospitals at present can only utilize the very limited data volume at hand to perform other researches such as deep learning or machine learning. This creates a significant bottleneck in AI algorithm learning using medical data. But these precious medical data, if integrated in some way, are expected to break this bottleneck. The existing phenomenon is that most medical institutions are not willing to share data mastered by themselves, and the locally held data quantity of all parties is insufficient to support modeling training of a deep learning model, however, the medical institutions have modeling requirements in practice. For the problem that scattered data are difficult to concentrate, the problem that the model is updated locally by android collection terminal users is solved by the Google in 2016. In 2016, google proposed to solve the problem of Android end user collection, and update the model locally. For reference, these efforts attempt to learn from local data using a central server while protecting privacy. But these designs are central.

Based on the above mentioned background, this patent proposes a framework of "blockchain+deep learning". This is a decentered architecture. The realization of the method can help each medical institution to jointly train a deep learning model under the condition that the data of the medical institution is not exposed, the decentralised framework ensures that each party is not administrated by a third party central institution in the whole training process, each party has equal rights, the whole data exchange process is transparent, and the decentralised mechanism ensures the honest behavior of the parties to a great extent. The mode of realizing multiparty cooperation based on the block chain platform provides a thought for solving the problem that multi-center data is difficult to perform joint modeling in a centralized way. Since data is always kept by the device generating the data and cannot be accessed by other devices, the data privacy is protected to a certain extent. Through experimental verification, the block chain and deep learning provided by the patent can effectively solve the dilemma of data island and data privacy protection.

Distributed machine learning mainly involves how to distribute training tasks, allocate computing resources, coordinate the various functional modules to achieve a balance of training speed and accuracy. Data parallelization and model parallelization are important sub-domains in the distributed learning domain. In the distributed data parallel mode, the data does not need to be collected to a central server, and each node calculates a part of data and then performs summarization and aggregation, so that the storage and calculation capacity of a single machine can be effectively solved. Based on the rise of the current pb-level social media big data, students propose some strategies for solving the scalability challenges of distributed machine learning, and design and develop a high-performance software framework. Model parallelization is more complex than data parallelization. The model is distributed on a plurality of computers, and the global deep learning model is subjected to modularized training to achieve a better learning effect. In addition, a method for updating the synchronous model is proposed by a learner, which allows a processor to access a shared memory and only needs to update a small part of variables during deep learning training. Later researchers proposed distributed machine learning in heterogeneous environments, which can operate dynamically and efficiently in heterogeneous clusters, and is suitable for a data parallel training mechanism. This eliminates the need for uniform batch processing of the data.

Federal learning is mainly classified into longitudinal federal learning and transverse federal learning according to the degree of overlap of samples. This patent is primarily concerned with lateral federal learning. The design of the federal learning system proposed by the scholars is a central communication system architecture. Meaning that the central server needs to collect gradient or model information from other participants, the decentralised federal learning framework is continued to be proposed. George's academy of sub-management and the United states NEC laboratories have proposed a federal learning environment based on blockchains by which global models are stored and shared and aggregated tasks are accomplished by intelligent contracts. Researchers at university of Zhongshan and university of hong Kong's university have proposed that the distributed study of Bayesian resilience in the distributed computing environment of block chain slicing 5G is put aside, have mainly solved the model parameter in the distributed study process to update in the problem such as Bayesian attack in the gradient polymerization process.

The prior art scheme has the defects that:

1. in the distributed machine learning strategy, the device has no control right, and most of data distributed on the nodes are required to be independently and uniformly distributed. These requirements are difficult to meet in practical applications.

2. The federal learning local device has control right, the data does not need to be independently and uniformly distributed, the node load is usually unbalanced, but many systems are centralized. Furthermore, in many papers, the manner in which the global deep learning model is trained is not discussed in detail when joint deep learning modeling is performed using an off-center architecture.

Disclosure of Invention

Aiming at the defects of the prior art, the invention provides a medical image segmentation cooperation method based on a blockchain network, which comprises the following steps: firstly, each medical institution carries out local collaborative segmentation network training modeling on respective medical data, local weight files are generated through training, the local weight files are submitted to a blockchain network as a transaction after the medical institutions to which the local weight files belong sign, nodes with downloading rights in the blockchain network download the local weight files until all medical institutions generate the local weight files, the local weight files are aggregated to generate global weights after the blockchain network receives the local weight data with preset quantity, and the global weights with the best preservation effect are taken as final collaborative segmentation network model parameters after limited iterative loops are carried out, and the method specifically comprises the following steps:

step 1: the block chain network is constructed as a platform for information sharing of each medical institution, and is used for supporting each node to cooperate to complete the whole process of building the global deep learning model, and mainly comprises four parts: data uploading, consensus agreement achievement, data storage and global model generation;

step 2: nodes entering the blockchain network must be authorized and are managed by a member service system MSP in Hyperledger Fabric, wherein the member service system is a user certificate and private key management system, and is not in a traditional user name and password format, and any transaction in the member service system needs to verify an account number by adopting two training modes: the step 3 is executed by adopting a sequential training mode or a parallel training mode, and the step 4 is executed by adopting a parallel training mode;

step 3: each medical institution participating in cooperation adopts a sequential training mode to train a global model together, verifies whether the global model obtained when the model converges is better than a local model, and the training mode mainly comprises two parts of local weight uploading and weight data downloading:

step 31: local weight uploading, namely a first medical institution firstly trains a deep learning model by using local medical data of the first medical institution, when the deep learning model is converged, the first local weight is stored, then the encrypted signature part is uploaded to a blockchain network, each node starts to verify the transaction, if the verification is passed, the transaction is legal, consensus is achieved, the transaction is acknowledged, and then the transaction is written into the blockchain network;

step 32: the weight data is downloaded, the second medical institution downloads the first local weight uploaded by the first medical institution to the local from the blockchain network, the first local weight is loaded into a deep learning model as a pre-training weight when the second medical institution performs deep learning training, the local medical data of the second medical institution is utilized for training, when the deep learning model begins to converge, the second local weight is stored, the second local weight is uploaded to the blockchain platform after the second local weight is encrypted and signed, and after the node is verified, the second local weight is written into the blockchain network;

step 33: all other nodes execute the operation of the step 32 in sequence until all nodes participate in training, model training is finished, a deep learning model generated by the last node is a global model, and model parameters of the last node are global parameters;

step 4: the parallel training mode is that each medical institution participating in the joint training deep learning model adopts a parallel training mode to train a global model simultaneously, nodes in the blockchain network use own local data to train until the convergence and stabilization of the global model are finished, in the parallel training mode, each round of training needs all nodes in the blockchain network to participate, in the process of iteratively generating the global model, each node uses private data to train the model continuously simultaneously, and the training process specifically comprises the following steps:

step 41: initializing parameters, initializing initial parameters by each medical institution by using seed values, initializing sample numbers, performing model training by each node in the first training round by adopting the initial parameters, and obtaining current global weight through a block chain network;

step 42: performing cyclic training, wherein each node takes the current global weight as a model parameter, performs model training in parallel by using private medical data, namely, training is performed simultaneously without sequence, and each node uploads a local weight encrypted signature file after training to a blockchain network;

step 43: aggregating local weights, wherein the block chain network aggregates the local weight data of each node to generate a current new global weight;

step 44: and downloading the currently updated global weight in the blockchain network by each node, training again by taking the global weight as a model parameter, repeating the steps 42 to 44 until the cycle times are reached, completing model training, and outputting the trained model as an optimal model by the blockchain network.

Step 5: after the optimal segmentation model is obtained, each node downloads the global weight and then carries out local test, and the effect of the optimal segmentation model is checked.

The invention has the beneficial effects that:

1. the invention provides a collaborative deep learning modeling method based on a blockchain network, which adopts a Hyperleger fabric alliance blockchain member service system, which is a user certificate and private key management system, wherein any transaction in the member service system needs to verify an account number, and in the training process, local training parameters are transmitted through asymmetric encryption to replace the original remote source data transmission mode so as to protect privacy.

2. The method provided by the invention uses the local data to train each, and each node has equal rights, so that each node is not limited by a third party authority verification end, and the problem of untrustiness and single point of failure caused by a centralized system is avoided to a certain extent.

3. The data is always stored by the equipment for generating the data and cannot be accessed by other equipment, so that the data privacy is protected to a certain extent.

Drawings

FIG. 1 is a system framework diagram of a collaborative training network in accordance with the present invention.

Detailed Description

The objects, technical solutions and advantages of the present invention will become more apparent by the following detailed description of the present invention with reference to the accompanying drawings. It should be understood that the description is only illustrative and is not intended to limit the scope of the invention. In addition, in the following description, descriptions of well-known structures and techniques are omitted so as not to unnecessarily obscure the present invention.

The following detailed description refers to the accompanying drawings.

The node of the present invention refers to an independent medical facility.

Medical data of the present invention includes, but is not limited to, CT images of the eyes, nasopharynx, brain.

The invention provides a framework of 'blockchain + deep learning', belonging to a decentralised framework. The method can help each medical institution to jointly train a deep learning model under the condition that own data are not exposed, the decentralised framework ensures that each party is not administrated by a third party central institution in the whole training process, each party has equal rights, the whole data exchange process is transparent, and the decentralised mechanism ensures the honest behavior of the parties to a great extent. The mode of realizing multiparty cooperation based on the block chain platform provides a thought for solving the problem that multi-center data is difficult to perform joint modeling in a centralized way. Since data is always kept by the device generating the data and cannot be accessed by other devices, the data privacy is protected to a certain extent. Experiments prove that the block chain and deep learning provided by the invention can effectively solve the dilemma of data island and data privacy protection.

The blockchain has the characteristics of decentralization, tamper resistance, distrustation and collective maintenance, and can be well used as a data platform for decentralization safety interaction. Therefore, the invention builds a blockchain network to ensure the information safety of data interaction of each medical organization during the joint training of the deep learning model, protect local data and data privacy thereof, and ensure effective deep learning on the premise of legal compliance. Firstly, the invention firstly describes how to adopt a blockchain as a bottom layer platform for deep learning to support each organization for information exchange.

Each medical institution joining the blockchain network shares its own data information during the training process through the blockchain platform. Firstly, each medical institution carries out local deep learning training modeling on medical data held by the medical institutions, a result generated by training is stored in a file form, each institution only uses the file generated by the local data training, which is called local weight, the local weight file is submitted to a blockchain network as a transaction after being signed by the medical institution, and the nodes with the downloading authority in the blockchain network can be downloaded. Other medical institutions as nodes in the blockchain repeat the above operation, firstly, model training is carried out by utilizing own local data, the local weight result obtained by training is uploaded into the blockchain network after being signed, and when the blockchain network receives a certain amount of local weight data, the local weight data is aggregated to generate global weight. This global weight, generated after a limited number of iteration cycles, is the final best parameter. The method provided by the invention does not expose local data in the training process.

The whole system architecture diagram of the collaborative training network proposed by the present invention is shown in fig. 1, and four medical institutions are taken as an example. The proposed collaborative training network comprises a blockchain network and various medical structures, and specifically comprises:

firstly, each medical institution carries out local deep learning training modeling on respective medical data, local weight files are generated through training, the local weight files are submitted to a blockchain network as a transaction after the medical institutions to which the local weight files belong sign, nodes with downloading authority in the blockchain network, namely other medical institutions, download the local weight files until all the medical institutions generate the respective local weight files, when the blockchain network receives a preset number of local weight data, global weight is generated through aggregation, and after limited iterative loops are carried out, the global weight with the best preservation effect is taken as a final image segmentation neural network parameter, and the method specifically comprises the following steps:

step 1: the block chain network is constructed as a platform for information sharing of each medical institution, and is used for supporting each node to cooperate to complete the whole process of building the global deep learning model, and mainly comprises four parts: data uploading, consensus agreement achievement, data storage and global model generation. And adopting a block chain network as a platform for information sharing of all parties. The blockchain network can support each node to cooperatively complete the overall modeling process of the global deep learning model.

The specific operation of the four parts is as follows:

after the nodes participating in training are trained by using own local data, uploading the local weight file generated by training to a block chain network;

the consensus protocol is achieved by a deterministic consensus algorithm, by a Kafka cluster.

The data storage is data uplink after consensus is reached for each node. Once the data is booted, it cannot be altered.

The generation of the global model means that after all nodes are trained by using the local data, the block chains are aggregated to generate a global weight through a consensus algorithm after all the nodes upload the local weight. All nodes in the same channel can download the global weight and then carry out local test to check the effect of the model.

Step 2: nodes entering the blockchain network must be authorized and managed by a member service system MSP (Membership Service Provider) in Hyperledger Fabric, which is a user certificate and private key management system, rather than a traditional username and password format, in which any transaction requires a verification account, in two training modes: and (3) executing the step (3) by adopting a sequential training mode or a parallel training mode, and executing the step (4) by adopting a parallel training mode.

The alliance chain is a block chain mode which is most applied under the current supervision system of China, and the invention adopts a Hyperleger fabric2.0 framework to build the alliance block chain. The shared account book mechanism of the alliance chain can greatly reduce the account checking cost in the scene, improve the data acquisition efficiency, increase the fault tolerance capability, and is very suitable for the requirement that all parties currently need a trusted platform to cooperatively model on the premise of mutual distrust. The POW consensus protocol consumes a lot of communication and computing resources. While Fabric relies on deterministic consensus algorithms, ledgers do not diverge as in other distributed or public chains. The invention adopts a core consensus algorithm in Fabric, which is realized through a Kafka cluster. Compared to other consensus algorithms. The Kafka consensus algorithm is more efficient, energy-saving and environment-friendly. It also provides a fault tolerant mechanism to facilitate stable operation of the system.

The blockchain storage refers to data uplink after each node achieves consensus. Once the data is booted, it cannot be altered. This is also an advantage of blockchain. Not only can tamper be prevented, but also the data on the chain can be utilized for tracing.

The process of verifying account transactions in step 2 is as follows:

step 21: the client of each node first calls the certificate service CA (Certificate Authority) through the SDK, and registers to acquire a certificate.

Step 22: the node initiates a transaction proposal to the endorsement node through the software development kit SDK (Software Development Kit), wherein the transaction proposal comprises contract identification, contract method and parameter information and node signature required by the transaction call.

Step 23: the endorsement node verifies the signature after receiving the transaction proposal, confirms whether the submitter has the execution authority, simultaneously simulates and executes the intelligent contract according to the endorsement policy, and feeds back the result and the respective certificate signature to the client node.

Step 24: after receiving the information returned by the endorsement node, the client node judges whether the proposal results are consistent or not, whether the endorsement strategy is executed according to the regulations or not, and if not, the processing is terminated; otherwise, the data is packaged and signed and then is transmitted to the ordering service node as a transaction.

Step 25: the ordering service node performs consensus ordering on the transactions, performs transaction packaging according to a block generation strategy, generates blocks and sends the blocks to the transaction writing node.

Step 26: after receiving the block, the transaction writing node verifies the transaction in the block, and adds the verification to the local block chain network.

Step 3: each medical institution participating in cooperation trains a global model together in a sequential training mode, verifies whether the global model obtained when the model converges is better than a local model, verifies by comparing the precision of a deep learning model, and if the precision is higher, the model is better, and the training mode mainly comprises two parts of local weight uploading and medical data downloading:

step 31: local weight is uploaded, a first medical institution firstly trains a deep learning model by using local medical data of the first medical institution, when the deep learning model is converged, the first local weight is stored, then the encrypted signature piece is uploaded to a blockchain network, each node starts to verify the transaction, if the verification is passed, the transaction is legal, consensus is achieved, the transaction is acknowledged, and then the transaction is written into the blockchain network.

Step 32: and downloading the weight, namely downloading the first local weight uploaded by the first medical institution to the local from the blockchain network by the second medical institution, loading the first local weight into a deep learning model as a pre-training weight when the second medical institution performs deep learning training, training by using local medical data of the second medical institution, storing the second local weight when the deep learning model starts converging, uploading the second local weight to a blockchain platform after the second local weight encrypts and signs, and writing the second local weight into the blockchain network after the nodes are verified.

Step 33: all other nodes execute the operation of step 32 in sequence until all nodes participate in training, model training is finished, the deep learning model generated by the last node is a global model, and the model parameters of the last node are global parameters.

The sequential server-client algorithm describes the process of receiving the deep learning global model by the blockchain network by way of sequential training for co-training of nodes, specifically downloading local weights for each node, verifying transactions, and storing weights. Each node downloads local weights from the blockchain network, and then continues the training process using the local data as follows:

server-side algorithm:

1. the server function receives a checkpoint, which is a transaction containing local training weights, from the client along with the data file.

2. Check if the checkpoint is true to verify the checkpoint.

3. If the checkpoint is valid, the checkpoint is saved and a "receive success" message is returned.

4. If the checkpoint is invalid, it is discarded and an "invalid proposal" message is returned.

Client algorithm:

1. the client function is executed by the client.

2. The proposal is loaded from the server along with the public key and the next set of parameters is initialized.

3. The specified number of cycles is cycled.

In each cycle, the client updates the model parameters according to the learning rate and gradient. If the loss converges, the client returns the updated parameters to the server.

step 41: initializing parameters, initializing the initial parameters by each medical institution by using seed values, wherein the seed values are random numbers selected according to historical experience, initializing sample numbers, performing model training by each node in the first training by adopting the initial parameters, and obtaining current global weights through a block chain network.

Step 42: and (3) performing cyclic training, namely performing model training in parallel by using the private medical data by taking the current global weight as a model parameter, namely performing training simultaneously without sequence, and uploading the trained local weight encrypted signature file to a blockchain network by each node.

Step 43: and aggregating the local weights, wherein the block chain network aggregates the local weight data of each node to generate a current new global weight.

The parallel server-client algorithm represents a training strategy in which each node adopts parallel training. In the first round of communication, each node performs local deep learning training by randomly initializing parameters. In the subsequent communication, each node takes the current global weight as a pre-training parameter to load. The blockchain network is responsible for aggregating local weights to generate a current new global weight.

Parallel server-side algorithm:

1. the server initializes model parameters using a given seed value, initializing the number of samples.

2. For each cycle, the server traverses each client in parallel.

3. The server sends the current global weight to each client and receives the updated local weight.

4. And then, the server uses the FedProx algorithm of the federal pocket for the updated local parameter weight to obtain the updated global weight.

Parallel client algorithm:

1. each client updates the weights locally from the first arrival to the set number of cycles.

2. In each cycle, the client updates the local weights based on the specified batch size, learning rate, and gradient.

3. The updated local weights are then returned to the server.

It should be noted that the above-described embodiments are exemplary, and that a person skilled in the art, in light of the present disclosure, may devise various solutions that fall within the scope of the present disclosure and fall within the scope of the present disclosure. It should be understood by those skilled in the art that the present description and drawings are illustrative and not limiting to the claims. The scope of the invention is defined by the claims and their equivalents.

Claims

1. The medical image segmentation cooperation method based on the blockchain network is characterized in that firstly, each medical institution carries out local cooperation segmentation network training modeling on respective medical data, local weight files are trained and generated, the local weight files are submitted to the blockchain network as a transaction after being signed by the medical institution to which the local weight files belong, nodes with downloading authority in the blockchain network, namely other medical institutions, download the local weight files until all the medical institutions generate the respective local weight files, the local weight files are aggregated to generate global weights after the blockchain network receives the local weight data of a preset quantity, and the global weights with the best preservation effect are taken as final cooperation segmentation network model parameters after limited iteration loops are carried out, and the method specifically comprises the following steps:

step 2: nodes entering the blockchain network must be authorized and are managed by a member service system MSP in Hyperledger Fabric, wherein the member service system is a user certificate and private key management system, and any transaction in the member service system needs to verify an account number by training modes including: the step 3 is executed by adopting a sequential training mode or a parallel training mode, and the step 4 is executed by adopting a parallel training mode;