CN117095227A - Convolutional neural network training method based on non-intersection differential privacy federal learning - Google Patents

Convolutional neural network training method based on non-intersection differential privacy federal learning Download PDF

Info

Publication number
CN117095227A
CN117095227A CN202311116879.1A CN202311116879A CN117095227A CN 117095227 A CN117095227 A CN 117095227A CN 202311116879 A CN202311116879 A CN 202311116879A CN 117095227 A CN117095227 A CN 117095227A
Authority
CN
China
Prior art keywords
client
model
value
local
neural network
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202311116879.1A
Other languages
Chinese (zh)
Inventor
王子龙
柴政
陈谦
王鸿波
闫浩楠
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Xidian University
Original Assignee
Xidian University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Xidian University filed Critical Xidian University
Priority to CN202311116879.1A priority Critical patent/CN117095227A/en
Publication of CN117095227A publication Critical patent/CN117095227A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • General Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Software Systems (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

The invention discloses a convolutional neural network training method based on non-intersection differential privacy federal learning, which mainly solves the problems of low privacy protection capability and low model classification precision of a convolutional neural network model trained in the prior art. The implementation scheme is as follows: constructing a federal learning system; the client initializes the federal learning training data set; the client initializes a convolutional neural network model; the client initializes a local convolutional neural network model and a local privacy budget value; the client performs iterative training on the local convolutional neural network model; the client calculates non-intersection elements of the trained local model; the client adds noise to the quantized local model according to the non-intersection element result; and the central server carries out iterative training on the convolutional neural network model through the local convolutional neural network model after noise is added in an aggregation mode, and a trained convolutional neural network model is obtained. The convolutional neural network trained by the method has strong privacy protection capability and high classification precision, and can be used for image classification.

Description

Convolutional neural network training method based on non-intersection differential privacy federal learning
Technical Field
The invention belongs to the technical field of image processing, and particularly relates to a training method of a convolutional neural network, which can be used for classifying images.
Background
Image classification is a common machine learning task, which accurately identifies the target class of an image according to the characteristics in the existence of the image, and in machine learning, a convolutional neural network is often adopted to solve the image classification task, so that a training model of the image classification is required to be trained, and a large amount of training data is required. However, these training data often exist in multiple data sources and contain sensitive information about individuals, and with the arousal of people's privacy awareness, personal privacy requirements are becoming more stringent.
To address privacy concerns, mcMahan et al in 2017 proposed the concept of federal learning to compromise efficiency and safety. Federal learning is now becoming a popular distributed machine learning paradigm, with shared model updates, e.g., gradients, model parameters, to reduce communication overhead and enhance privacy protection of communication processes. Researchers have found, however, that the analysis of gradients, model parameters, such as model inversion attacks, membership inference attacks, often violate user privacy. The differential privacy technology is deployed in federal learning to meet the privacy requirement, but the method reduces the prediction classification effect or the system efficiency of the model. Therefore, in federal learning based on differential privacy, how to improve the prediction classification effect of the model and reduce the risk of privacy leakage of the model is a problem to be solved in order to realize an efficient and safe federal learning system.
Patent application publication No. CN113762525a discloses a "federal learning model training method with differential privacy protection", which comprises the following implementation steps: (1) Establishing connection between a model and a terminal, selecting a learning model, and establishing connection between the learning model and a terminal server; (2) Distinguishing and distributing the models, distinguishing sample level and user level of the models, and selecting the distinguished models to participate in the federal learning of the round by the terminal server according to the standard; (3) Carrying out local training, loading a current global model by each device, and training by using own training data to obtain a new local model; (4) Establishing privacy protection, and encrypting and protecting information by utilizing differential privacy protection in different modes according to the distinction of model types; (5) Updating the global model, uploading an updated value of the model to a server by each device, collecting the updated value by the server, and then, aggregating to obtain an aggregation result, thereby updating the global model; (6) Monitoring and feeding back, namely monitoring the updated value of the data, judging whether a specific sample is leaked from a trained model or a user participates in training, and feeding back the obtained result to a terminal server; (7) And the data backup record is used for carrying out backup archiving on the updated global model data, so that the damage to the data acquisition during data transmission is avoided. According to the method, the disturbance degree of the federal learning model is controlled through the privacy budget value, and equal privacy budget values are distributed for all model weight parameters of the federal learning model, so that the disturbance degree of all weight parameters is the same, and differences of privacy protection requirements of shared weight parameters and non-shared weight parameters of clients are not considered, so that the disturbance degree of the shared weight parameters with low privacy protection requirements is overlarge, the disturbance degree of the non-shared weight parameters with high privacy protection requirements is too small, the prediction effect and the privacy protection capability of the model are reduced, and the classification precision of images is affected.
Disclosure of Invention
Aiming at the defects of the prior art, the invention provides a convolutional neural network training method based on non-intersection differential privacy federal learning, so as to reduce the influence of different privacy protection requirements on mismatching of disturbance degrees of two models, namely a shared weight parameter and a non-shared weight parameter of a client, improve the prediction effect and privacy protection capability of a network model, and further improve the classification precision of images.
The technical idea for achieving the purpose of the invention is that the common intersection parameters and the non-common intersection parameters of all local model weight parameters are obtained by each client in each iteration process, and then different privacy budgeting values are distributed for different weight parameters according to different privacy protection requirements, namely, the common intersection parameters are subjected to smaller disturbance degree, the non-common intersection parameters are subjected to larger disturbance degree, and further the prediction effect and the privacy protection capability of the network model are improved.
According to the above thought, the implementation steps of the invention are as follows:
(1) The initialization includes a central server, a private set intersection server and N clients C= { C 1 ,c 2 ,…,c n ,…,c N Federal learning system in which c n Representing the nth client, n= {1,2, …, N }, N being greater than or equal to 2;
(2) Each client c n Initializing a local training data set containing M images of L target categories, and labeling targets of each image, wherein L is more than or equal to 2, and M is more than or equal to 100;
(3) Each client c n Acquiring convolutional neural network model X comprising I model layers from a central server n The X is n The weight parameter of (2) is omega n ，X n The weight parameter of the ith model layer of (2) is omega n,i ；
(4) Each client initiates a local convolutional neural network modelSet->Privacy budget and weight parameters of (2) are +.>And-> The weight parameter of the ith model layer of (2) is +.>
(5) Client-to-local convolutional neural network modelIterative training is carried out:
each client c n Randomly selecting B training data from the local training data set with a place back as a local convolutional neural network modelTo obtain B predictive labels +.>The label is used for adopting a random gradient descent method to carry out weight parameter +.>Performing iterative update to obtain a weight parameter of +.>Post-training local model of->
(6) Client computing post-training local modelIs a non-intersecting element of (a):
(6a) Each client c n Weight parameters for each layer of updated local modelPerforming quantization calculation, and adding weight parameter +.>The decimal part of (2) is precisely of fixed length, and the quantized weight parameter is obtained>
(6b) Each client c n Calculating the model weight parameters after each layer of quantizationHash value +.>And upload to the private collection intersection server, the private collection intersection server according to each client c n Uploaded hash value +.>Calculating to obtain non-intersection element results of each layer of weight parameters of the local model after client quantification>Summarizing the data and then issuing the summarized data to each client;
(7) The client adds noise to the quantized local model according to the non-intersection element result:
(7a) Each client c n Based on the aggregated non-intersection element results and the local modelPrivacy budget value +.>Calculating a noise value comprising two different differential privacy guarantees +.>
(7b) Each client c n Will noise valueWeight parameter added to quantized local model +.>The weight parameter after disturbance is obtained is +.>Local model->Local model after disturbance +.>Uploading to a central server;
(8) The central server trains the convolutional neural network model:
(8a) The central server initializes the aggregation round to t=0, and sets the maximum aggregation round to be T more than or equal to 100;
(8b) Randomly select V t Personal client c v Uploaded post-disturbance local modelWeight parameter->Polymerizing to obtain the current polymerization result omega t And determines whether t=t holds:
if yes, obtaining the weight parameter omega T Is a post-training convolutional neural network model X T ；
Otherwise, let t=t+1, send the aggregate result to each client c n Returning to the step (3).
Compared with the prior art, the invention has the following advantages:
firstly, the client distributes different disturbance noise for different weight parameters of the local model according to different privacy protection requirements of the weight parameters shared by the local convolutional neural network model and the non-shared weight parameters, so that the disturbance degree of the weight parameters of the local model can be accurately controlled;
secondly, the client calculates the Gaussian noise value meeting the differential privacy guarantee for the non-intersection element of the local model weight parameter, so that the defects of too small disturbance noise, too low disturbance process of the local model and low privacy protection capability caused by too large privacy budget in the prior art can be avoided, and the privacy protection capability of the local model is effectively improved;
third, the client calculates the noise value which does not meet the differential privacy guarantee for the intersection element of the local model weight parameter, so that the defects of overlarge disturbance noise, overlarge disturbance degree of the local model and low classification precision of the convolutional neural network model caused by overlarge privacy budget value in the prior art can be avoided, and the prediction classification precision of the convolutional neural network model is effectively improved.
Drawings
FIG. 1 is a flow chart of an implementation of the present invention.
FIG. 2 is a non-intersection element sub-flowchart of the client computing local model weight parameters in the present invention;
FIG. 3 is a sub-flowchart of the noise adding process performed by the client according to the non-intersection element result in the present invention;
FIG. 4 is a graph of results of simulation training of MNIST data sets using the present invention and prior art methods;
FIG. 5 is a graph comparing the effects of performing privacy attacks using convolutional neural network models trained using the present invention and prior art methods.
Detailed Description
Embodiments and effects of the present invention are described in further detail below with reference to the accompanying drawings.
Referring to fig. 1, the implementation steps of this example include the following:
step 1, constructing a federal learning system.
Setting a central server, a private set intersection server and N clients C= { C 1 ,c 2 ,…,c n ,…,c N Each client c n Respectively establishing transmission channels with a central server and a private set intersection server to form a federal learning system, wherein N is more than or equal to 2, c n Represents the nth client, n=100 in this example.
Step 2, the client initializes the federal learning training data set.
Each client c n Initializing a local training data set containing M images of L target categories, and labeling targets of each image, wherein L is more than or equal to 2, and M is more than or equal to 100;
the data set used in this example is an MNIST handwriting digital image data set, where the number of target classes corresponding to the data set l=10, and is a number of 0-9, respectively, and includes 60000 images, each of which has a pixel size of 28×28, and each of the target classes has 6000 images, and in this example, the local training data set of each client is set to include m=600 images.
And step 3, initializing a convolutional neural network model by the client.
Each client c n Acquiring a convolutional neural network model X of I model layers consisting of K sequentially laminated composite layers and F fully connected layers from a central server n Each composite layer comprises a convolution layer, a ReLU layer, a Dropout layer and a pooling layer which are sequentially stacked;
initializing X n The weight parameter of (2) is omega n ，X n The weight parameter of the ith model layer of (2) is omega n,i ；
In this example, the number of composite layers of the convolutional neural network is k=2, the number of fully connected layers is f=2, and i=k+f=4.
And step 4, initializing a local convolutional neural network model and a local privacy budget value by the client.
Each client initiates a local convolutional neural network model
Setting upPrivacy budget value of +.>Initialization->The weight parameter of (2) is->
Is provided withThe weight parameter of the ith model layer of (2) is +.>
In this example set up
Step 5, client-side pair local convolutional neural network modelAnd performing iterative training.
5.1 Each client c) n Randomly selecting B training data from the local training data set with a place back as a local convolutional neural network modelTo obtain B predictive labels +.>In this example, b=32;
5.2 Each client c) n For the B obtained predictive labelsCalculating each predictive tag +.>Corresponding to the real label->Loss value of +.>
5.3 Each client c) n Using loss valuesFor local convolutional neural network model->Weight parameter->Deviation is calculated to obtain the weight parameter +.>Updating to obtain the weight parameter of the current update +.>
Where η > 0 represents a learning rate, in this example, η=0.01;
5.4 Each client c) n Repeating the steps 5.1) to 5.3) for tau times to finally obtain the weight parameter asPost-training local model of->τ+.10, in this example τ=10.
Step 6, the client calculates the trained local modelIs a non-intersecting element of (a).
Referring to fig. 2, the implementation steps of this step include the following:
6.1 Each client c) n By weighting parameters for each layer of the updated local modelCalculate->And judge->The interval range of the value is taken to calculate a quantization threshold iota:
wherein,is the weight parameter of each layer of the local model after updating,/>An mth weight parameter representing an ith model layer of the local model; s is the weight parameter->The fixed length value of the fractional part is set by the client, M is the number of quantized model i-th layer weight parameters, in this example, s=5;
6.2 Each client c) n Weighting parameters of each layer of the updated local model by utilizing quantized threshold iotaCalculating random valued function->Output result probability value +.>Which represents a random valued functionOutputting the probability of iota/s as a result;
6.3 Each client c) n According to the probability value and the weight parameterCalculating a random value functionOutput result of (2):
when (when)When (I)>
When (when)When in use, let->
6.4 Each client c) n By means ofThe weight parameter for each layer of the updated local model +.>Performing quantization calculation, namely, weighting parameter +.>The decimal part of (2) is precisely the fixed length s, and the quantized weight parameter +.>
Wherein II 2 Is L 2 A norm operator, sgn (·) is a sign function;
6.5 Each client c) n Negotiating and selecting any one hash function H:
H:{0,1} * →{0,1} κ
wherein, the step of calculating the hash function is that a group of binary strings with any length are output as a group of values with the length of kappa; * the length of the hash function input value is an arbitrary value; kappa is the length of the hash function output value, kappa > 0,
in this example, each client c n Selecting the SHA-1 function as a hash function H, wherein the length k=160 of the hash function output value;
6.6 Each client c) n Each weight parameter of the quantized modelConverting into binary format, inputting hash function H, and calculating model weight parameter +_after quantization of each layer>Is a hash value of (2):
6.7 Each client c) n Model weight parameters after each layer is quantizedHash value +.>Uploading to a private set intersection server, wherein the private set intersection server judges the parity of the number N of the clients and then calculates the number N of the clients according to each client c n Uploaded hash value +.>Calculating to obtain the non-intersection element result of each layer of weight parameters of the local model after client quantification
If N is even, executing the step 6.8);
if N is odd, executing the step 6.9);
6.8 The private-set intersection server calculates the exclusive or value of all hash values as:
wherein the exclusive OR valuePosition m of (2) corresponds to the weight parameter +.>The other positions are corresponding to the quantized weight parameters in the local model as non-intersection elements;
6.9 Random selection of a client c by the private-set intersection server k Uploaded hash valueAnd calculates the exclusive or value +.1 of the remaining N-1 hash values>
6.10 Private set intersection server to determine exclusive or valueWhether or not 0:
if it isDoes not change->Is a value of (2);
if it isThe private collection intersection server calculates that position m corresponds to client c k Hash value +.>With any client c j Hash value +.>Exclusive OR value +.>
6.11 Private collection intersection server judgment client c) k Hash value of (a)With any client c j Hash value +.>Exclusive OR value +.>Whether or not 0:
if it isWill->The value of (2) is modified to 1;
if it isDoes not change->The value of (i.e.)>
6.12 The private set intersection server will modify the exclusive or valuePosition m of (2) corresponds to the weight parameter +.>As intersection elements, taking weight parameters in the local model after the corresponding quantization of the rest positions as non-intersection elements;
6.13 The private collection intersection server sums the obtained intersection elements and non-intersection elements and then sends the elements to each client c n ；
N=100 in this example, each client c n Step 6.1) to step 6.8) are performed, and then step 6.13) is performed, so as to obtain the non-intersection element result of the quantized local model weight parameters.
And 7, adding noise to the quantized local model according to the non-intersection element result by the client.
Referring to fig. 3, the implementation steps of this step include the following:
7.1 Each client c) n Utilizing the aggregated non-intersection element results and local modelPrivacy budget value +.>Computing a Gaussian noise scale value satisfying a differential privacy guarantee for non-intersection elements of the quantized local model>
Wherein,representing sensitivity, delta representing probability that the quantized local model does not satisfy the differential privacy technique after adding noise; in this example, <' > a->
7.2 Each client c) n Using gaussian noise scale valuesCalculating noise values satisfying differential privacy guarantees +.>
Wherein s represents a random number,μ represents the mathematical expectation of gaussian noise, μ=0 in this example;
7.3 Each client c) n Calculating noise values for intersection elements that do not meet differential privacy guaranteesAnd will->And->Combining corresponding positions of intersection elements and non-intersection elements to obtain noise value +.>
In the step, the larger the noise value added for the non-intersection elements of the quantized local model is, the higher the disturbance degree of the non-intersection elements is, the lower the probability of leakage of the local training data is, and the higher the privacy protection capability is; the smaller the noise value added to the intersection element of the local model after quantization, the lower the disturbance degree to the intersection element, the smaller the influence on the model prediction classification effect, and the higher the prediction classification precision of the model;
7.4 Each client c) n Will noise valueWeight parameter added to quantized local model +.>The weight parameter after disturbance is obtained is +.>Local model->Local model after disturbance +.>Uploading to a central server.
And 8, training the convolutional neural network model by the central server.
8.1 The central server initializes the aggregation round to t=0, sets the maximum aggregation round to T not less than 100, and sets the maximum aggregation round t=100 in this example;
8.2 Random selection of V by central server t Personal client c v Uploaded post-disturbance local modelWeight parameter->Polymerizing to obtain the currentPolymerization result omega t ：
Wherein omega t For the polymerization result, V was set in this example t ＝10；
8.3 The central server determines whether t=t holds:
if yes, obtaining the weight parameter omega T Is a post-training convolutional neural network model X T ；
Otherwise, let t=t+1, the polymerization result ω t To each client c n And (3) returning to the step (3).
The above steps are numbered for clarity in describing the implementation of the present invention, and the sequence numbers are not limited.
The effect of the invention can be further illustrated by the following simulation experiment:
1. simulation conditions
Simulation experiments were performed using Tensorflow and Keras libraries in a Windows 10 system, intel (R) Core (TM) i5-10400 CPU@2.90GHz, NVIDIA GeForce RTX 3070, 16GB RAM, 4T memory computing environment.
2. Emulation content
Simulation 1, the convolutional neural network model is simulated and trained by using MNIST data sets respectively by using the method for training the federal learning model, and the classification precision of the convolutional neural network model in the respective training process is compared with the association classification precision of the convolutional neural network model training without noise, and the result is shown in figure 4.
As can be seen from fig. 4, the convolutional neural network model without noise has the highest association classification accuracy, but the model has no privacy protection capability, and can be used as a reference standard for evaluating the classification effect of the model, namely, the closer to an ideal result, the better the model prediction classification effect; the classification precision of the convolutional neural network model obtained by training the method is higher than that of the convolutional neural network model obtained by training the prior method, and is closer to an ideal result. The method can effectively improve the model prediction classification precision of the convolutional neural network.
And 2, respectively executing privacy attack on the convolutional neural network model trained by the MNIST data set by using the method and the existing federal learning model training method to obtain respective attack results, as shown in figure 5. Wherein,
FIG. 5 (a) is a sample sub-graph of original training data of a client, which can be used as a reference standard for evaluating the privacy protection capability of a model, namely, the worse the model privacy protection capability is the closer the attack result is to the sample sub-graph of original training data;
FIG. 5 (b) is a graph of the results of a convolutional neural network model trained with prior art methods to perform a privacy attack;
fig. 5 (c) is a graph of the results of a privacy attack performed using the convolutional neural network model trained with the present invention.
As can be seen from fig. 5, the convolutional neural network model trained by the existing method performs a privacy attack, and the attack result is very close to the original training data sample, because the disturbance degree of the existing method to the local model is too small, and the privacy protection capability of the local model is insufficient; the convolutional neural network model trained by the invention executes privacy attack, and the attack result is quite obvious from the original training data sample, so that an attacker cannot obtain the original training data sample by executing the privacy attack, because the disturbance degree of the invention to the local model is larger, and the privacy protection capability of the local model is higher. The method and the device indicate that the privacy protection capability of the local model can be effectively improved.
The above description is only one specific example of the invention and does not constitute any limitation of the invention, and it will be apparent to those skilled in the art that various modifications and changes in form and details may be made without departing from the principles, construction of the invention, but these modifications and changes based on the idea of the invention are still within the scope of the claims of the invention.

Claims (8)

1. A convolutional neural network training method based on non-intersection differential privacy federal learning is characterized by comprising the following steps:
(1) The initialization includes a central server, a private set intersection server and N clients C= { C 1 ,c 2 ,…,c n ,,c N Federal learning system in which c n Representing the nth client, n= {1,2, …, N }, N being greater than or equal to 2;
(2) Each client c n Initializing a local training data set containing M images of L target categories, and labeling targets of each image, wherein L is more than or equal to 2, and M is more than or equal to 100;
(3) Each client c n Acquiring convolutional neural network model X comprising I model layers from a central server n The X is n The weight parameter of (2) is omega n ，X n The weight parameter of the ith model layer of (2) is omega n,i ；
(4) Each client initiates a local convolutional neural network modelSet->Privacy budget and weight parameters of (2) are +.>And->The weight parameter of the ith model layer of (2) is +.>
(5) Client-to-local convolutional neural network modelIterative training is carried out:
each client c n From local trainingRandomly selecting B training data with place back in the training data set as a local convolutional neural network modelTo obtain B predictive labels +.>The label is used for adopting a random gradient descent method to carry out weight parameter +.>Performing iterative update to obtain a weight parameter of +.>Post-training local model of->
(6) Client computing post-training local modelIs a non-intersecting element of (a):
(6a) Each client c n Weight parameters for each layer of updated local modelPerforming quantization calculation, and adding weight parameter +.>The decimal part of (2) is precisely of fixed length, and the quantized weight parameter is obtained>
(6b) Each client c n Calculating the model weight parameters after each layer of quantizationHash value +.>And upload to the private collection intersection server, the private collection intersection server according to each client c n Uploaded hash value +.>Calculating to obtain non-intersection element results of each layer of weight parameters of the local model after client quantification>Summarizing the data and then issuing the summarized data to each client;
(7) The client adds noise to the quantized local model according to the non-intersection element result:
(7a) Each client c n Based on the aggregated non-intersection element results and the local modelPrivacy budget value +.>Calculating a noise value comprising two different differential privacy guarantees +.>
(7b) Each client c n Will noise valueWeight parameter added to quantized local model +.>The weight parameter after disturbance is obtained is +.>Local model->Local model after disturbance +.>Uploading to a central server;
(8) The central server trains the convolutional neural network model:
(8a) The central server initializes the aggregation round to t=0, and sets the maximum aggregation round to be T more than or equal to 100;
(8b) Randomly select V t Personal client c v Uploaded post-disturbance local modelWeight parameter->Polymerizing to obtain the current polymerization result omega t And determines whether t=t holds:
if yes, obtaining the weight parameter omega T Is a post-training convolutional neural network model X T ；
Otherwise, let t=t+1, send the aggregate result to each client c n Returning to the step (3).
2. The method according to claim 1, characterized in that: the I model layers in the step (3) include K composite layers and F full-connection layers which are sequentially stacked, i.e., i=k+f; each composite layer comprises a convolution layer, a ReLU layer, a Dropout layer and a pooling layer which are sequentially stacked, wherein K is more than or equal to 2,F and more than or equal to 2.
3. The method according to claim 1, characterized in that: using tags in step (5)The random gradient descent method is adopted to carry out weight parameter +.>The implementation steps of the iterative updating method comprise the following steps:
(5a) Each client c n For the B obtained predictive labelsCalculating each predictive tag +.>Corresponding to the real label->Loss value of +.>
(5b) Using loss valuesFor local convolutional neural network model->Weight parameter->To the partial derivative of the weight parameterUpdating to obtain the weight parameter of the current update +.>
Wherein η > 0 represents a learning rate;
(5c) Repeating (5 a) and (5 b) for tau times to obtain the weight parameter asPost-training local model of->
4. The method according to claim 1, characterized in that: obtaining the quantized weight parameters in the step (6 a)The expression is as follows:
wherein,is the weight parameter of each layer of the local model after updating,/>An mth weight parameter representing an ith model layer of the local model;
‖·‖ 2 is L 2 The norm operator, sgn (·) is the sign function, whenWhen (I)>The definition is as follows:
s is a weight parameterThe fixed length value of the fractional part, iota is an integer, 0.ltoreq.iota < s,when->When in use, let->
5. The method according to claim 1, characterized in that: each client c in step (6 b) n Calculating the model weight parameters after each layer of quantizationHash value +.>The implementation steps comprise the following steps:
(6b1) Each client c n Negotiating and selecting any one hash function H:
H:{0,1} * →{0,1} κ
wherein, the calculation process of the hash function is that a group of binary strings with arbitrary length are output as a group of binary strings with length of kappaA value; * the length of the hash function input value is an arbitrary value; kappa is the length of the hash function output value, kappa > 0;
(6b2) Model weight parameters after quantizationEach weight parameter of (a)>Converting into binary format, inputting into hash function H, calculating to obtain quantized model weight parameter ++>Is a hash value of (2):m is the number of quantized model i-th layer weight parameters.
6. The method according to claim 1, characterized in that: the private-set intersection server in step (6 b) is based on each client c n Uploaded hash valueCalculating to obtain non-intersection element results of each layer of weight parameters of the local model after client quantification>The implementation steps comprise the following steps:
(6b3) Judging parity of the number N of the clients:
if N is even, then executing (6 b 4);
if N is odd, executing (6 b 5);
(6b4) The private-set intersection server calculates the exclusive or value of all hash values as:
wherein the exclusive OR valuePosition m of (2) corresponds to the weight parameter +.>The other positions are corresponding to the quantized weight parameters in the local model as non-intersection elements;
(6b5) The private collection intersection server randomly selects one client c k Uploaded hash valueAnd calculates the exclusive or value +.1 of the remaining N-1 hash values>
(6b6) Determining exclusive or valueWhether or not 0:
if it isDoes not change->Is a value of (2);
if it isThe private collection intersection server calculates that position m corresponds to client c k Hash value +.>With any client c j Hash value +.>Exclusive OR value +.>
(6b7) Judging client c k Hash value of (a)With any client c j Hash value +.>Exclusive or value of (2)Whether or not 0:
if it isWill->The value of (2) is modified to 1;
if it isDoes not change->The value of (i.e.)>
(6b8) Will modify the exclusive OR valuePosition m of (2) corresponds to the weight parameter +.>And taking the weight parameters in the local model after the corresponding quantization of the rest positions as non-intersection elements.
7. The method according to claim 1, characterized in that: each client c in step (7 a) n Computing noise values comprising two different differential privacy guaranteesThe implementation steps comprise the following steps:
(7a1) Each client c n Utilizing privacy budget valuesComputing a Gaussian noise scale value satisfying a differential privacy guarantee for non-intersection elements of the quantized local model>
Wherein,representing sensitivity, delta representing probability that the quantized local model does not satisfy the differential privacy technique after adding noise;
(7a2) Using gaussian noise scale valuesCalculating noise values satisfying differential privacy guarantees +.>
Wherein s represents a random number,μ represents the mathematical expectation of gaussian noise;
(7a3) Calculating noise values for intersection elements that do not meet differential privacy guaranteesAnd will->And->Combining corresponding positions of intersection elements and non-intersection elements to obtain noise value +.>
8. The method according to claim 1, characterized in that: randomly selecting V in step (8 b) t Personal client c v Uploaded post-disturbance local modelWeight parameter->Polymerization was carried out as follows:
wherein omega t As a result after polymerization, 1.ltoreq.V t ≤N。
CN202311116879.1A 2023-08-31 2023-08-31 Convolutional neural network training method based on non-intersection differential privacy federal learning Pending CN117095227A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311116879.1A CN117095227A (en) 2023-08-31 2023-08-31 Convolutional neural network training method based on non-intersection differential privacy federal learning

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311116879.1A CN117095227A (en) 2023-08-31 2023-08-31 Convolutional neural network training method based on non-intersection differential privacy federal learning

Publications (1)

Publication Number Publication Date
CN117095227A true CN117095227A (en) 2023-11-21

Family

ID=88782659

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311116879.1A Pending CN117095227A (en) 2023-08-31 2023-08-31 Convolutional neural network training method based on non-intersection differential privacy federal learning

Country Status (1)

Country Link
CN (1) CN117095227A (en)

Similar Documents

Publication Publication Date Title
CN113408743B (en) Method and device for generating federal model, electronic equipment and storage medium
US11295208B2 (en) Robust gradient weight compression schemes for deep learning applications
CN113191484A (en) Federal learning client intelligent selection method and system based on deep reinforcement learning
CN113609398B (en) Social recommendation method based on heterogeneous graph neural network
CN111695719A (en) User value prediction method and system
CN114912705A (en) Optimization method for heterogeneous model fusion in federated learning
CN113011587B (en) Privacy protection model training method and system
CN113850272A (en) Local differential privacy-based federal learning image classification method
CN111127435B (en) No-reference image quality evaluation method based on double-current convolution neural network
CN114332578A (en) Image anomaly detection model training method, image anomaly detection method and device
CN107545301B (en) Page display method and device
US20230049817A1 (en) Performance-adaptive sampling strategy towards fast and accurate graph neural networks
CN110889759A (en) Credit data determination method, device and storage medium
CN110264270A (en) A kind of behavior prediction method, apparatus, equipment and storage medium
CN117574429A (en) Federal deep learning method for privacy enhancement in edge computing network
CN110610140B (en) Training method, device and equipment of face recognition model and readable storage medium
Gurung et al. Decentralized quantum federated learning for metaverse: Analysis, design and implementation
CN111079930A (en) Method and device for determining quality parameters of data set and electronic equipment
CN117095227A (en) Convolutional neural network training method based on non-intersection differential privacy federal learning
Dash DECPNN: A hybrid stock predictor model using Differential Evolution and Chebyshev Polynomial neural network
CN115577797A (en) Local noise perception-based federated learning optimization method and system
CN115358418A (en) Federal learning classification model training method based on model disturbance
CN115688569A (en) Gain adjustment method, gain adjustment device, computer equipment and storage medium
CN115660147A (en) Information propagation prediction method and system based on influence modeling between propagation paths and in propagation paths
WO2022235599A1 (en) Generation and implementation of dedicated feature-based techniques to optimize inference performance in neural networks

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination