CN111242196B - Differential privacy protection method for interpretable deep learning - Google Patents

Differential privacy protection method for interpretable deep learning Download PDF

Info

Publication number
CN111242196B
CN111242196B CN202010011049.2A CN202010011049A CN111242196B CN 111242196 B CN111242196 B CN 111242196B CN 202010011049 A CN202010011049 A CN 202010011049A CN 111242196 B CN111242196 B CN 111242196B
Authority
CN
China
Prior art keywords
deep learning
learning model
data set
layer
interpretable
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202010011049.2A
Other languages
Chinese (zh)
Other versions
CN111242196A (en
Inventor
王金艳
李德
胡宇航
李先贤
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guangxi Normal University
Original Assignee
Guangxi Normal University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guangxi Normal University filed Critical Guangxi Normal University
Priority to CN202010011049.2A priority Critical patent/CN111242196B/en
Publication of CN111242196A publication Critical patent/CN111242196A/en
Application granted granted Critical
Publication of CN111242196B publication Critical patent/CN111242196B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/23—Clustering techniques
    • G06F18/232—Non-hierarchical techniques
    • G06F18/2321—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions
    • G06F18/23213—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions with fixed number of clusters, e.g. K-means clustering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60—Protecting data
    • G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
    • G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
    • G06F21/6245—Protecting personal data, e.g. for financial or medical purposes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Software Systems (AREA)
  • Evolutionary Computation (AREA)
  • Computational Linguistics (AREA)
  • Computing Systems (AREA)
  • Molecular Biology (AREA)
  • Mathematical Physics (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Bioethics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Biology (AREA)
  • Probability & Statistics with Applications (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Computer Hardware Design (AREA)
  • Computer Security & Cryptography (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a differential privacy protection method for interpretable deep learning, which ensures the safety of the input end and the output end of a model by adding differential privacy protection in a first convolution layer of an FF-CNN and adding differential privacy in a loss function of an output layer of the model, thereby protecting the personal privacy of a model data provider, and enhancing data by a mixup interpolation method after clustering data characteristics obtained in a second sampling layer by using a k-means + + algorithm, thereby improving the robustness of the whole model. The invention is based on the privacy protection strategy of the interpretable deep learning model, so that a model user can obtain an interpretable result by using the deep learning model and does not reveal personal privacy information.

Description

Differential privacy protection method for interpretable deep learning
Technical Field
The invention relates to the technical field of deep learning and privacy protection, in particular to a differential privacy protection method for interpretable deep learning.
Background
In recent years, deep learning is a new field of machine learning research. It has the ability, just like the human brain, to learn and process complex data and attempt to solve responsible tasks. Due to this capability, it is used in various fields such as text extraction, voice recognition, image classification and recognition, and the like. The Convolutional Neural Network (CNN) is a representative network structure for deep learning, is widely applied to various image recognition and semantic segmentation scenes, can be divided into two parts of feature extraction and feature recognition from the macroscopic concept, performs effective feature extraction on an original image through convolution (convolution) operation and sampling (firing) operation, and then performs accurate recognition on the effective features extracted through transformation of a full connection layer (full connection). However, the training of the convolutional neural network requires a large amount of data, the data contains personal sensitive information of the user, and if the convolutional neural network model is directly released without protection, the privacy of the data provider is leaked to a certain extent, and the benefit of the data provider is also damaged, so that the result of providing the data is not desired, and the training of the model is affected.
For a deep learning model, which is a black box, a data owner cannot see what action or reason the model is based on to obtain such a judgment result, so that for a decision made by the deep neural network model, a user often holds a questionable attitude. Even though the prediction accuracy of the model has reached a high level. Therefore, the interpretable deep neural network has a great promotion effect on the development of deep learning, but the interpretable deep neural network still has the problem of privacy disclosure, and a certain privacy protection means needs to be adopted to ensure the privacy of a data provider.
At present, in the aspect of machine learning privacy protection, differential privacy has become one of the most potential privacy protection technologies. The differential privacy is to ensure that the difference of one record between adjacent data sets has almost no influence on the output of the two data sets, the protection mode is realized by adding a proper amount of noise into the return value of the query function, and an attacker cannot steal the original data of the model through the designed attack model. However, the existing convolutional neural network models satisfying the differential privacy are all of common convolutional neural network structures, almost no privacy protection means is provided for the interpretable convolutional neural network models, the interpretable convolutional neural network is more dependent on parameters of the models, and thus the privacy of data providers is more easily revealed for attack means of white boxes or black boxes. The privacy protection of the interpretable convolutional neural network is more meaningful and challenging, and is mainly reflected in the following aspects:
(1) the combination of interpretability and privacy can not only provide the data provider with the explanation of the decision result of the model, but also provide the privacy of the provider on the premise of not losing the interpretability, and provide the data provider with sufficient guarantee on the practicability and the safety.
(2) On the premise of ensuring that the precision of the convolution neural network model is not reduced and the definition of interpretability is not weakened, how to accurately add noise and reduce the influence of the noise on the model is very critical.
Disclosure of Invention
The invention provides a differential privacy protection method for interpretable deep learning, aiming at the problem of privacy disclosure generated in the training and reasoning processes of an interpretable convolutional neural network model.
In order to solve the problems, the invention is realized by the following technical scheme:
the differential privacy protection method for the interpretable deep learning comprises the following steps:
step 1, initializing an interpretable deep learning model, wherein the interpretable deep learning model is based on an interpretable convolutional neural network of forward propagation and sequentially comprises an input layer, a first convolutional layer, a first sampling layer, a second convolutional layer, a second sampling layer, a first fully-connected layer, a second fully-connected layer and an output layer;
step 2, normalizing the given data set to be used as a training data set of the interpretable deep learning model;
step 3, training the first convolution layer of the interpretable deep learning model obtained in the step 1 by using the training data set in the step 2 so as to update the interpretable deep learning model; namely:
step 3.1, carrying out principal component analysis on the training data set to obtain the characteristic value and the characteristic vector of each principal component;
3.2, sorting the characteristic values of the main components in a descending order, and selecting the characteristic vectors corresponding to the main components with the characteristic values arranged at the top 6 bits;
step 3.3, based on the allocated first privacy budget ε1Carrying out Laplacian noise on the eigenvector selected in the step 3.2, namely the initial eigenvector, so as to obtain eigenvectors meeting the difference privacy;
step 3.4, performing truncation operation on the feature vector meeting the difference privacy obtained in the step 3.3 to obtain a final feature vector, and using the final feature vector as 6 convolution kernels of a first convolution layer of the interpretable deep learning model to update the interpretable deep learning model;
step 4, inputting the training data set obtained in the step 2 from the input of the first convolution layer of the interpretable deep learning model obtained in the step 3, and outputting the output of the second sampling layer to obtain a first characteristic data set;
step 5, firstly, clustering sample data in the first characteristic data set obtained in the step 4 by adopting a k-means + + algorithm to obtain a first characteristic data set with a label; then, enhancing the first characteristic data set with the label by using a Mixup algorithm to obtain a first enhanced data set;
step 6, performing least square regression calculation on the first enhanced data set, and using the calculated mapping coefficient vector as a connection parameter between a second sampling layer and a first full connection layer of the interpretable deep learning model to update the interpretable deep learning model;
step 7, inputting the first enhanced data set obtained in the step 5 from the input of the first full-connection layer of the interpretable deep learning model obtained in the step 6, and outputting the output of the first full-connection layer to obtain a second characteristic data set;
step 8, firstly, clustering sample data in the second characteristic data set obtained in the step 7 by adopting a k-means + + algorithm to obtain a second characteristic data set with labels; then, enhancing the second characteristic data set with the label by using a Mixup algorithm to obtain a second enhanced data set;
step 9, performing least square regression calculation on the second enhanced data set, and using the calculated mapping coefficient vector as a connection parameter of a first full connection layer and a second full connection layer of the interpretable deep learning model to update the interpretable deep learning model;
step 10, based on the allocated second privacy budget ε2Performing laplacian denoising on the coefficient of the expansion of the square error loss function of the output layer of the interpretable deep learning model obtained in the step 9 to obtain a denoised square error loss function, and performing denoising on the denoised square error loss functionThe function is used as a square error loss function of an output layer of the interpretable deep learning model to update the interpretable deep learning model;
step 11, inputting the second enhanced data set obtained in the step 8 from the input of the second fully-connected layer of the interpretable deep learning model obtained in the step 10, and outputting the output of the second fully-connected layer to obtain a third feature data set;
step 12, inputting the third feature data set obtained in the step 12 into the noisy square error loss function obtained in the step 10, and obtaining connection parameters of the second full connection layer and the output layer by minimizing the noisy square error loss function so as to update the interpretable deep learning model;
step 13, taking the current interpretable deep learning model as a final interpretable deep learning model;
and 14, inputting the data to be protected into the final interpretable deep learning model obtained in the step 13, wherein the output of the final interpretable deep learning model is the data after privacy protection.
In step 3.3 above, the privacy budget ε of the jth initial feature vectorjComprises the following steps:
Figure GDA0003605513610000031
wherein epsilon1For a given first privacy budget, λjJ is the eigenvalue corresponding to the jth initial eigenvector, j is 1, 2.
In step 3.3, when laplacian denoising is performed, the global sensitivity Δ f of the jth initial eigenvector isjComprises the following steps:
Figure GDA0003605513610000032
wherein the content of the first and second substances,
Figure GDA0003605513610000033
is the jth initialThe maximum value of the values in all elements of the feature vector,
Figure GDA0003605513610000034
is the numerical minimum in all elements of the jth initial feature vector, | | · | | computationally1Is L-1 norm, j ═ 1, 2.
In the step 3.4, the process of performing truncation operation on the feature vector meeting the difference privacy is as follows: traversing each element in the jth feature vector satisfying the differential privacy: if the element is larger than the maximum value of the numerical values in all the elements of the jth initial feature vector
Figure GDA0003605513610000035
Then let the element value be
Figure GDA0003605513610000036
If the element is smaller than the minimum value of the numerical values in all the elements of the jth initial feature vector
Figure GDA0003605513610000037
Then let the element value be
Figure GDA0003605513610000038
Otherwise, the element value is kept unchanged. Wherein j is 1, 2.
The differential privacy protection method for interpretable deep learning according to claim 1, wherein in step 10, when laplacian denoising is performed, the global sensitivity Δ f of the coefficient of the expansion of the square error loss function is:
Δf＝||λmax-λmin||1
wherein λ ismaxMaximum value of coefficient, lambda, representing expansion of squared error loss functionminRepresents the minimum value of the coefficients of the squared error loss function expansion, | · | | non-calculation1Representing the L-1 norm.
Compared with the prior art, the invention has the following characteristics:
1. for the convolution layer of the first layer of the model, the invention adopts a dynamic noise adding mode for the weight value of the convolution kernel, and adds less Laplace noise for the invention with large weight value of the correlation of the output of the model, or vice versa, thereby ensuring that an attacker cannot reversely push out an original data set by destroying the convolution layer of the first layer, and also being capable of accurately adding noise and improving the usability of subsequent training data.
2. And clustering the data sampled for the second time by using a k-means + + algorithm, and then enhancing the data by using a mixup interpolation method, thereby improving the robustness of the model.
3. In an output layer of the model, approximate Taylor expansion is carried out on an output loss function of the model, corresponding Laplacian noise is added into a coefficient in the expansion, the safety of the model at an output end is guaranteed, and an attacker is prevented from successfully attacking the model through member reasoning attack.
Drawings
FIG. 1 is a diagram of an interpretable convolutional neural network.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail below with reference to specific examples.
A differential privacy protection method for interpretable deep learning comprises the following specific steps:
the training process of the interpretable deep learning model comprises the following steps:
step 1, initializing an interpretable deep learning model, wherein the interpretable deep learning model is based on a forward propagation interpretable Convolutional Neural network FF-CNN (Interpretetable Convolutional Neural Networks Via fed forward design), and sequentially comprises an input layer, a first Convolutional layer, a first sampling layer, a second Convolutional layer, a second sampling layer, a first full-connection layer, a second full-connection layer and an output layer.
And 2, normalizing the given data set to be used as a training data set of the interpretable deep learning model.
In the invention, a given data set is a Minst & ciafr-10 data set, and the pixel values of all images in the data set are reduced to be in the range of 0-1 by adopting a normalization operation of image/255.0, so that a training data set required by model construction is obtained.
And 3, training the first convolution layer of the interpretable deep learning model obtained in the step 1 by using the training data set in the step 2 so as to update the interpretable deep learning model. Namely:
calculating the training data in the step 1 by utilizing Principal Component Analysis (PCA) to construct a model with 6 convolution kernels WCONV-1Size of [5 x 5]Using differential privacy technique on convolution kernel WCONV-1And adding noise, and constructing a convolution layer added with Laplace noise to replace the first convolution layer of the FF-CNN model.
Step 3.1, carrying out principal component analysis on the training data set to obtain the characteristic value lambda of each principal componentiAnd a feature vector ui。
Step 3.2, characteristic value lambda of each principal componentiSorting in descending order and selecting the eigenvalue lambdaiFeature vector u corresponding to principal component arranged at the first 6 bitsi. The 6 selected eigenvectors have the same size as the 6 convolution kernels of the first convolution layer of the model, and are all [5 x 5]]。
Step 3.3, based on the allocated first privacy budget ε1And (3) carrying out Laplacian noise on the eigenvector selected in the step (3.2), namely the initial eigenvector, so as to obtain the eigenvector meeting the difference privacy.
The feature vector (i.e., initial convolution kernel) W selected for the 6 firstCONV-1Allocating different privacy budgets epsilonjCalculating the global sensitivity Δ f of the 6 selected feature vectorsjThen based on the global sensitivity of the privacy budget, noise is added to the 6 selected feature vectors to obtain feature vectors (namely noise-added convolution kernels) W 'meeting the differential privacy'CONV-1。
In the present invention, 6 selected eigenvectors are assigned a privacy budget εjMay be the same, but in order to be able to improve the anonymity, in the present invention the privacy assigned to the 6 selected feature vectors isBudget εjAre not equal. A larger privacy budget epsilon (less noise added) is allocated to eigenvectors with large eigenvalues, and vice versa.
Privacy budget ε of jth initial feature vectorjComprises the following steps:
Figure GDA0003605513610000051
wherein epsilon1For a given first privacy budget, λjJ is the eigenvalue corresponding to the jth initial eigenvector, j is 1, 2.
Global sensitivity Δ f of jth initial feature vectorjComprises the following steps:
Figure GDA0003605513610000052
wherein the content of the first and second substances,
Figure GDA0003605513610000053
is the maximum value of the values in all elements of the jth initial feature vector,
Figure GDA0003605513610000054
is the numerical minimum value in all elements of the jth initial feature vector, | · | | calcualting1Is L-1 norm, j ═ 1, 2.
Different Privacy budgets are allocated to 6 feature vectors using Differential Privacy (Differential Privacy) protection techniques, and unequal noise addition, i.e. adding of noise, is performed
Figure GDA0003605513610000055
Wherein, WCONV-1Is an uncoded eigenvector, i.e., initial eigenvector (i.e., initial convolution kernel), W'CONV-1The feature vectors after being subjected to noise addition are the feature vectors meeting the difference privacy (namely the noise addition convolution kernels).
And 3.4, performing truncation operation on the feature vector meeting the difference privacy obtained in the step 3.3 to obtain a final feature vector, and using the final feature vector as 6 convolution kernels of the first convolution layer of the interpretable deep learning model to update the interpretable deep learning model.
6 noisy feature vectors (namely noisy convolution kernels) W 'are subjected to truncation'CONV-1And adjusting the elements of (2), and taking the 6 feature vectors obtained after adjustment as the final 6 convolution kernels of the first convolution layer of the model. The truncation operation process of the feature vectors meeting the differential privacy is as follows: traversing each element w in the jth feature vector satisfying differential privacypq：
If the element wpqNumerical maximum in all elements greater than jth initial feature vector
Figure GDA0003605513610000061
Namely, it is
Figure GDA0003605513610000062
Then order
Figure GDA0003605513610000063
I.e. the element value is
Figure GDA0003605513610000064
If the element wpqNumerical minimum of all elements smaller than jth initial feature vector
Figure GDA0003605513610000065
Namely, it is
Figure GDA0003605513610000066
Then order
Figure GDA0003605513610000067
I.e. the element value is
Figure GDA0003605513610000068
If the elementwpqMinimum value of numerical values among all elements of jth initial feature vector
Figure GDA0003605513610000069
And the maximum value of the values in all the elements of the jth initial feature vector
Figure GDA00036055136100000610
In between, i.e
Figure GDA00036055136100000611
Then wpq＝wpqI.e. the element w is maintainedpqThe value is unchanged.
And 4, inputting the training data set obtained in the step 2 from the input of the first convolution layer of the interpretable deep learning model obtained in the step 3, and outputting the output of the second sampling layer to obtain a first characteristic data set.
Step 4.1, inputting the data of the step 1 into the first one of FF-CNN model with 6 convolution kernels W'CONV-1Size of [5 x 5]The convolution layer performs convolution operation to extract convolution characteristics of data, and the size of the output convolution characteristics [28 x 6]]。
And 4.2, continuously inputting the convolution characteristics obtained in the step 4.1 into a first sampling layer (posing) of the FF-CNN model to perform dimensionality reduction and characteristic extraction operations, wherein the size of the output characteristic is [14 x 6 ].
And 4.3, inputting the features obtained in the step 4.2 into a second convolution layer with 16 convolution kernels with the size of [5 x 6] and a second sampling layer of the FF-CNN for feature extraction, and outputting the features with the size of [5 x 15 ].
And 5, firstly, clustering sample data in the first characteristic data set obtained in the step 4 by adopting a k-means + + algorithm to obtain a first characteristic data set with a label. And then, enhancing the first characteristic data set with the label by using a Mixup algorithm to obtain a first enhanced data set.
And (4) performing enhancement operation on the feature data extracted in the step (4), providing more data samples for the subsequent classification decision process of the full-connection layer, and improving the robustness of model training.
And 5.1, clustering the characteristic data x extracted in the step 4 by using a k-means + + (enhanced randomness k-means clustering) algorithm, and taking the center of the cluster of the characteristic data x as a label y of the characteristic data x, thereby obtaining the labeled characteristic data. Wherein the value of the label y is 0-9.
And 5.2, performing data enhancement operation on the characteristic data with the label processed in the step 5.1 by utilizing a Mixup (data enhancement interpolation) algorithm to expand and obtain more characteristic data with the label, thereby obtaining an enhanced data set.
The method for generating the feature data by the enhanced operation is as follows:
xn＝λxi+(1-λ)xj
the method for generating the label by the enhancement operation comprises the following steps:
yn＝λyi+(1-λ)yj
wherein λ is an enhancement factor, ranging between (0, 1); (x)i,yi)、(xj,yj) For the data obtained in step 4.1, (x)n,yn) Is newly generated data. x is a radical of a fluorine atomi、xjAnd xnRespectively representing the characteristic data itself, yi、yjAnd ynRespectively representing characteristic data xi、xjAnd xnThe label of (1).
Step 6, carrying out least square regression calculation on the first enhanced data set, and taking the calculated mapping coefficient vector as a connection parameter W of a second sampling layer and a first full-connection layer of the interpretable deep learning modelfull-1To update the interpretable deep learning model.
Calculating the data obtained in the step 5 by utilizing a least square regression method (LSR) to obtain a connection parameter W of the second sampling layer and the first full connection layer of the FF-CNN modelfull-1。
The least squares regression method is:
Figure GDA0003605513610000071
(n is the dimension of the output layer)
Wherein the content of the first and second substances,
Figure GDA0003605513610000072
in order to be a vector of the feature data,
Figure GDA0003605513610000073
is a label vector for the feature data,
Figure GDA0003605513610000074
is a mapping coefficient vector, i.e. a connection parameter.
Solving the model weight (parameter) W between the second sampling layer and the first full-connection layer by establishing 120 linear equation setsfull-1。
And 7, inputting the first enhanced data set obtained in the step 5 from the input of the first full connection layer of the interpretable deep learning model obtained in the step 6, and outputting the output of the first full connection layer to obtain a second characteristic data set.
And 8, firstly clustering the sample data in the second characteristic data set obtained in the step 7 by adopting a k-means + + algorithm to obtain a second characteristic data set with labels. And then, enhancing the second characteristic data set with the label by using a Mixup algorithm to obtain a second enhanced data set.
Step 9, performing least square regression calculation on the second enhanced data set, and taking the calculated mapping coefficient vector as a connection parameter W of the first full connection layer and the second full connection layer of the interpretable deep learning modelfull-2To update the interpretable deep learning model.
The least square regression method comprises the following steps:
Figure GDA0003605513610000075
(n is the dimension of the output layer)
Wherein the content of the first and second substances,
Figure GDA0003605513610000081
in order to be a vector of the feature data,
Figure GDA0003605513610000082
is a label vector for the feature data,
Figure GDA0003605513610000083
is a mapping coefficient vector, i.e. a connection parameter.
Solving the model weight (parameter) W between the first fully-connected layer and the second fully-connected layer by establishing 84 linear equation setsfull-2。
Step 10, based on the allocated second privacy budget ε2And (4) performing Laplace denoising on the coefficient of the expansion of the square error loss function of the output layer of the interpretable deep learning model obtained in the step (9) to obtain a denoised square error loss function, and taking the denoised square error loss function as the square error loss function of the output layer of the interpretable deep learning model to update the interpretable deep learning model.
Approximately expanding a square error loss function (loss function) of a model output layer, and adding a privacy budget to a coefficient of the approximate expansion to form epsilon2(ε2＝ε-ε1) The model output end is protected from privacy. Wherein epsilon2＝ε-ε1And epsilon is the given total privacy budget.
The loss function of the output layer adopts a square error function, and aims to measure the difference between the real label and the calculated prediction label, namely to measure the evaluation index of the model training. The squared error loss function of the output layer and its approximation spread as follows:
f(X,Wfull-3)＝(y-XWfull-3)2＝yTy-2XyWfull-3+XTXWfull-3
wherein X represents the characteristic data input to the output layer, Wfull-3And representing a connection parameter between the second full connection layer and the output layer, y representing a real label corresponding to the characteristic data X, and f (X, w) representing a loss function of the output layer.
Global sensitivity Δ f:
Δf＝||λmax-λmin||1＝2(d2+2d+1)
where d represents the dimension of the data features of the output layer, λmaxMaximum of coefficient, lambda, representing expansion of loss functionminRepresents the minimum value of the coefficients of the expansion of the loss function, | · | | non-woven phosphor1Representing the L-1 norm.
Coefficient lambda in the expansion for the squared error loss function0＝1，λ1＝-2Xy，λ2＝XTX, adding laplacian noise, i.e.:
Figure GDA0003605513610000084
wherein λ represents the coefficient of expansion without noise,
Figure GDA0003605513610000085
representing the noisy expansion coefficients.
And 11, inputting the second enhanced data set obtained in the step 8 from the input of the second fully-connected layer of the interpretable deep learning model obtained in the step 10, and outputting the output of the second fully-connected layer to obtain a third feature data set.
Step 12, inputting the third feature data set obtained in step 12 into the squared error loss function after noise addition obtained in step 10, and obtaining the connection parameter W of the second full connection layer and the output layer by minimizing the squared error loss function after noise additionfull-3To update the interpretable deep learning model.
Obtaining parameter W of last full connection layer of FF-CNN model by minimizing loss functionfull-3Namely:
Figure GDA0003605513610000091
wherein X represents the characteristic data input to the output layer, Wfull-3Indicating a second full connectionThe connection parameter between the layer and the output layer, y represents the real label corresponding to the characteristic data X,
Figure GDA0003605513610000092
w when representing the minimization of the f (-) functionfull-3The value of (a).
And step 13, taking the current interpretable deep learning model as a final interpretable deep learning model. As shown in fig. 1.
And (II) carrying out a differential privacy protection process by using the trained interpretable deep learning model:
and 14, inputting the data to be protected into the final interpretable deep learning model obtained in the step 13, wherein the output of the final interpretable deep learning model is the data after privacy protection.
The invention provides a differential privacy protection method based on an FF-CNN (interpretable deep learning model), which ensures the safety of the input end and the output end of the model by adding differential privacy protection in a first layer convolution layer of the FF-CNN and adding differential privacy in a loss function of an output layer of the model, thereby protecting the personal privacy of a model data provider, and performing data enhancement by a mixup interpolation method after clustering by using a k-means + + algorithm on data characteristics obtained in a second sampling layer, thereby improving the robustness of the whole model. The invention is based on the privacy protection strategy of the interpretable deep learning model, so that a model user can obtain an interpretable result by using the deep learning model and does not reveal personal privacy information.
It should be noted that, although the above-mentioned embodiments of the present invention are illustrative, the present invention is not limited thereto, and thus the present invention is not limited to the above-mentioned embodiments. Other embodiments, which can be made by those skilled in the art in light of the teachings of the present invention, are considered to be within the scope of the present invention without departing from its principles.

Claims (5)

1. The differential privacy protection method for the interpretable deep learning is characterized by comprising the following steps of:
step 1, initializing an interpretable deep learning model, wherein the interpretable deep learning model is based on an interpretable convolutional neural network of forward propagation and sequentially comprises an input layer, a first convolutional layer, a first sampling layer, a second convolutional layer, a second sampling layer, a first fully-connected layer, a second fully-connected layer and an output layer;
step 2, normalizing the given data set to be used as a training data set of the interpretable deep learning model;
step 3, training the first convolution layer of the interpretable deep learning model obtained in the step 1 by using the training data set in the step 2 so as to update the interpretable deep learning model; namely:
step 3.1, carrying out principal component analysis on the training data set to obtain the characteristic value and the characteristic vector of each principal component;
3.2, sorting the eigenvalues of the principal components in a descending order, and selecting the eigenvectors corresponding to the principal components with the eigenvalues ranked at the top 6 bits;
step 3.3, based on the allocated first privacy budget ε1Carrying out Laplacian noise on the eigenvector selected in the step 3.2, namely the initial eigenvector, so as to obtain eigenvectors meeting the difference privacy;
step 3.4, performing truncation operation on the feature vector meeting the difference privacy obtained in the step 3.3 to obtain a final feature vector, and using the final feature vector as 6 convolution kernels of a first convolution layer of the interpretable deep learning model to update the interpretable deep learning model;
step 4, inputting the training data set obtained in the step 2 from the input of the first convolution layer of the interpretable deep learning model obtained in the step 3, and outputting the output of the second sampling layer to obtain a first characteristic data set;
step 5, firstly, clustering sample data in the first characteristic data set obtained in the step 4 by adopting a k-means + + algorithm to obtain a first characteristic data set with a label; then, enhancing the first characteristic data set with the label by using a Mixup algorithm to obtain a first enhanced data set;
step 6, performing least square regression calculation on the first enhanced data set, and using the calculated mapping coefficient vector as a connection parameter between a second sampling layer and a first full connection layer of the interpretable deep learning model to update the interpretable deep learning model;
step 7, inputting the first enhanced data set obtained in the step 5 from the input of the first full-connection layer of the interpretable deep learning model obtained in the step 6, and outputting the output of the first full-connection layer to obtain a second characteristic data set;
step 8, firstly, clustering sample data in the second characteristic data set obtained in the step 7 by adopting a k-means + + algorithm to obtain a second characteristic data set with labels; then, enhancing the second characteristic data set with the label by using a Mixup algorithm to obtain a second enhanced data set;
step 9, performing least square regression calculation on the second enhanced data set, and using the calculated mapping coefficient vector as a connection parameter of a first full connection layer and a second full connection layer of the interpretable deep learning model to update the interpretable deep learning model;
step 10, based on the allocated second privacy budget ε2Performing laplacian denoising on the coefficient of the expansion of the square error loss function of the output layer of the interpretable deep learning model obtained in the step 9 to obtain a denoised square error loss function, and using the denoised square error loss function as the square error loss function of the output layer of the interpretable deep learning model to update the interpretable deep learning model;
step 11, inputting the second enhanced data set obtained in the step 8 from the input of the second fully-connected layer of the interpretable deep learning model obtained in the step 10, and outputting the output of the second fully-connected layer to obtain a third feature data set;
step 12, inputting the third feature data set obtained in the step 12 into the noisy square error loss function obtained in the step 10, and obtaining connection parameters of the second full connection layer and the output layer by minimizing the noisy square error loss function so as to update the interpretable deep learning model;
step 13, taking the current interpretable deep learning model as a final interpretable deep learning model;
and 14, inputting the data to be protected into the final interpretable deep learning model obtained in the step 13, wherein the output of the final interpretable deep learning model is the data after privacy protection.
2. The differential privacy preserving method of interpretable deep learning of claim 1, wherein in step 3.3, the privacy budget ε of jth initial feature vectorjComprises the following steps:
Figure FDA0003605513600000021
wherein epsilon1For a given first privacy budget, λjJ is the eigenvalue corresponding to the jth initial eigenvector, j is 1, 2.
3. The differential privacy protection method for interpretable deep learning of claim 1, wherein in step 3.3, the global sensitivity Δ f of the jth initial eigenvector is determined during laplacian denoisingjComprises the following steps:
Figure FDA0003605513600000022
wherein the content of the first and second substances,
Figure FDA0003605513600000023
is the maximum value of the values in all elements of the jth initial feature vector,
Figure FDA0003605513600000024
is the numerical minimum in all elements of the jth initial feature vector, | | · | | computationally1Is L-1 norm, j ═ 1, 2.
4. The differential privacy protection method for interpretable deep learning according to claim 1, wherein in step 3.4, the truncation operation on the feature vector satisfying the differential privacy is performed as follows:
traversing each element in the jth feature vector satisfying the differential privacy:
if the element is larger than the maximum value of the numerical values in all the elements of the jth initial feature vector
Figure FDA0003605513600000025
Then let the element value be
Figure FDA0003605513600000026
If the element is smaller than the minimum value of the numerical values in all the elements of the jth initial feature vector
Figure FDA0003605513600000027
Then let the element value be
Figure FDA0003605513600000028
Otherwise, keeping the element value unchanged;
wherein j is 1, 2.
5. The differential privacy protection method for interpretable deep learning according to claim 1, wherein in step 10, when laplacian denoising is performed, the global sensitivity Δ f of the coefficient of the expansion of the square error loss function is:
Δf＝||λmax-λmin｜｜1
wherein λ ismaxMaximum value of coefficient, lambda, representing expansion of squared error loss functionminMinimum value of coefficient representing square error loss function expansion equation, | · | | calving1Representing the L-1 norm.
CN202010011049.2A 2020-01-06 2020-01-06 Differential privacy protection method for interpretable deep learning Active CN111242196B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010011049.2A CN111242196B (en) 2020-01-06 2020-01-06 Differential privacy protection method for interpretable deep learning

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010011049.2A CN111242196B (en) 2020-01-06 2020-01-06 Differential privacy protection method for interpretable deep learning

Publications (2)

Publication Number Publication Date
CN111242196A CN111242196A (en) 2020-06-05
CN111242196B true CN111242196B (en) 2022-06-21

Family

ID=70864848

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010011049.2A Active CN111242196B (en) 2020-01-06 2020-01-06 Differential privacy protection method for interpretable deep learning

Country Status (1)

Country Link
CN (1) CN111242196B (en)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112232349B (en) * 2020-09-23 2023-11-03 成都佳华物链云科技有限公司 Model training method, image segmentation method and device
CN112487482B (en) * 2020-12-11 2022-04-08 广西师范大学 Deep learning differential privacy protection method of self-adaptive cutting threshold
CN112765662B (en) * 2021-01-22 2022-06-03 电子科技大学 Method for supporting privacy protection of training integrator under deep learning
CN113378859B (en) * 2021-06-29 2022-07-15 中国科学技术大学 Image privacy detection method with interpretability
CN114118407B (en) * 2021-10-29 2023-10-24 华北电力大学 Differential privacy availability measurement method for deep learning

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106557812A (en) * 2016-11-21 2017-04-05 北京大学 The compression of depth convolutional neural networks and speeding scheme based on dct transform
CN107766740A (en) * 2017-10-20 2018-03-06 辽宁工业大学 A kind of data publication method based on difference secret protection under Spark frameworks
CN108427891A (en) * 2018-03-12 2018-08-21 南京理工大学 Neighborhood based on difference secret protection recommends method
CN109102157A (en) * 2018-07-11 2018-12-28 交通银行股份有限公司 A kind of bank's work order worksheet processing method and system based on deep learning
WO2019122854A1 (en) * 2017-12-18 2019-06-27 Privitar Limited Data product release method or system
CN110334757A (en) * 2019-06-27 2019-10-15 南京邮电大学 Secret protection clustering method and computer storage medium towards big data analysis

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10817774B2 (en) * 2016-12-30 2020-10-27 Facebook, Inc. Systems and methods for providing content

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106557812A (en) * 2016-11-21 2017-04-05 北京大学 The compression of depth convolutional neural networks and speeding scheme based on dct transform
CN107766740A (en) * 2017-10-20 2018-03-06 辽宁工业大学 A kind of data publication method based on difference secret protection under Spark frameworks
WO2019122854A1 (en) * 2017-12-18 2019-06-27 Privitar Limited Data product release method or system
CN111971675A (en) * 2017-12-18 2020-11-20 普威达有限公司 Data product publishing method or system
CN108427891A (en) * 2018-03-12 2018-08-21 南京理工大学 Neighborhood based on difference secret protection recommends method
CN109102157A (en) * 2018-07-11 2018-12-28 交通银行股份有限公司 A kind of bank's work order worksheet processing method and system based on deep learning
CN110334757A (en) * 2019-06-27 2019-10-15 南京邮电大学 Secret protection clustering method and computer storage medium towards big data analysis

Also Published As

Publication number Publication date
CN111242196A (en) 2020-06-05

Similar Documents

Publication Publication Date Title
CN111242196B (en) Differential privacy protection method for interpretable deep learning
CN111462126B (en) Semantic image segmentation method and system based on edge enhancement
Zhao et al. Multisensor image fusion and enhancement in spectral total variation domain
Montazer et al. An improved radial basis function neural network for object image retrieval
Cui et al. Superpixel-based extended random walker for hyperspectral image classification
US20050105795A1 (en) Classification in likelihood spaces
Sun et al. Extracting nonlinear features for multispectral images by FCMC and KPCA
CN112884856A (en) Text image generation method for generating confrontation network based on spectrum normalization hierarchical level
CN111160407A (en) Deep learning target detection method and system
Wang et al. An active contour model based on local pre-piecewise fitting bias corrections for fast and accurate segmentation
CN111325275B (en) Robust image classification method and device based on low-rank two-dimensional local identification map embedding
Wang et al. A double dictionary-based nonlinear representation model for hyperspectral subpixel target detection
Poon et al. Improved methods on PCA based human face recognition for distorted images
CN111401440B (en) Target classification recognition method and device, computer equipment and storage medium
Wang et al. Dynamic super-pixel normalization for robust hyperspectral image classification
CN103310210B (en) Character recognition device, recognition dictionary generate device and method for normalizing
Hundley et al. Estimation of topological dimension
Kumar et al. Pixel-based skin color classifier: A review
Li et al. Shadow determination and compensation for face recognition
Kar et al. RBECA: A regularized Bi-partitioned entropy component analysis for human face recognition
Wijaya et al. Pornographic image rejection using eigenporn of simplified LDA of skin ROIs images
Yin A comparative study on the method of extracting edge and contour information of multifunctional digital ship image
Lu et al. Frontal view synthesis based on a novel GAN with global and local discriminators
Rao et al. Texture classification based on statistical Properties of local units
Razzaq et al. Human Face Recognition Based on Local Ternary Pattern and Singular Value Decomposition

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant