CN112766422B - Privacy protection method based on lightweight face recognition model - Google Patents

Abstract

In order to solve the problem that data are easily leaked in a training phase of a face recognition technology in practical application, the invention provides a privacy protection method based on a lightweight face recognition model. Secondly, to solve the problem that the network model parameters are easily attacked, the method designs use a voting mechanism and noise to increase the probability to protect the network parameters. Finally, the invention innovatively provides a 'teacher-student' fusion architecture based on a face recognition network to isolate sensitive and non-sensitive data. The method can accelerate the network operation speed, reduce the training parameter quantity and ensure the safety of the training stage and the reasoning stage under the condition of ensuring that the recognition rate is not reduced, and the method is fully proved to have practical value on the reference indexes of speed and accuracy.

Description

A Privacy Preservation Method Based on Lightweight Face Recognition Model

technical field

The invention relates to a privacy protection method based on a lightweight face recognition model, and belongs to the technical fields of image processing and privacy security protection.

technical background

The rapid development of science has brought about earth-shaking changes in today's society. Smartphones, cloud processors and other products can meet the material and spiritual needs of the society relying on the analysis and processing of a large amount of data. However, product companies that use such data have not properly protected and regulated personal user information. As for this type of data protection issues, what needs to be solved urgently is the part related to artificial intelligence, deep learning and other technologies. Privacy protection under deep learning usually involves making changes to the model itself or the training mechanism, including the design of the network structure and the adjustment of the training method.

In the initial stage of privacy protection technology, relatively direct protection methods are mainly used, such as encryption, masking, etc., that is, to protect sensitive information by blocking it or to prevent third-party access by encryption and decryption, such as RSA encryption technology. length of the public key to encrypt to protect the data. With the development of technology, there is an increasing demand for a large amount of image or video data. Using simple encryption and decryption methods to protect data is more complex and difficult to implement in deep learning algorithms. As a result, many new algorithms have been developed one after another, and differential privacy is one of them. The earliest proposal of this technology is to protect the differential attack. The so-called differential attack is an attack method in which the attacker uses a limited number of queries to obtain the undisclosed part of the data. Differential privacy is to use a specific algorithm in the data set to increase the probability. Sexual purpose to protect the training data. Later, with the improvement of the algorithm, it was also applied to many other privacy protection issues.

In the existing visual privacy protection technology, the common task is to protect and detect human faces, because in special samples such as visual data, human faces have the characteristics of uniquely identifying individuals, and the application range of face recognition is the widest. widely. Face recognition under deep learning is the focus of practical applications. The main application scenarios are the monitoring and security of home scenes or the protection of visual data under intelligent application systems, because the data involves the privacy of everyone and every family. In addition, major companies pay more attention to the development of smart home and other products. The wide application of smart home needs to be based on data analysis. Therefore, the research on face recognition algorithms in this type of scene has a very large potential. practical application value.

Contents of the invention

The traditional face recognition algorithm network has the disadvantages of a large number of parameters, not easy to be transplanted and widely used, and there is a risk of data leakage when using private data sets for training. Aiming at this problem, the present invention proposes a privacy protection method based on face recognition network. The network model designed by this method has the advantages of less parameters, easy convergence, and fast operation speed, which improves the degree of network transplantation and is convenient for large-scale , and the training mechanism using differential privacy effectively protects the training data.

The technical scheme adopted in the present invention is:

A privacy protection method based on a lightweight face recognition model, which uses a partial linear mapping method to reduce computational complexity, and uses a multi-layer cascaded lightweight network to extract features from a face dataset; add " Teacher-student" fusion architecture, train multiple teacher models and add Laplacian noise, introduce a voting mechanism to obtain the labels marked to the student model, so as to increase the probability of data; the newly obtained marked non-private data set Put in the student model for training, and finally use the obtained student model as the final network model. The specific steps include:

Step (1): Prepare the sensitive data set LFW for training the network, then divide the training set pictures and the verification set pictures into n parts, and regard n-1 parts of the data as sensitive data, and the other part as non-sensitive data;

Step (2): Generate n-1 improved lightweight networks, put n-1 training sets in sensitive data into each network for training, and the lightweight network is divided into four sub-modules during training, Feature extraction and fusion are carried out separately. Each sub-module divides the original features into two parts, one part uses convolution operation to generate key feature maps, and the other part uses linear transformation to perform simple mapping operations on the generated feature maps to generate other Auxiliary feature map, and then fuse the two parts to get the final feature; get the output through the entire network to calculate the loss function and perform reverse derivation, and finally get n-1 teacher models that have been trained;

Step (3): Use the divided n-1 verification sets for verification, and observe whether the accuracy rate on the verification set of the teacher model meets the recognition requirements;

Step (4): Prepare the remaining non-sensitive data as the input of all n-1 teacher models, put each sample into the teacher model in turn to get the prediction result, find the sample category with the largest number of votes through the softmax function and Add Laplace noise, and use the highest probability as the data label of each sample;

Step (5): Prepare the obtained non-sensitive data and labels as a new training data set, and then regenerate the lightweight network in step (2), retrain the non-sensitive data as the training set of the network, extract features and Fusion, finally get the student model;

Step (6): Join the face detection model, obtain the face coordinates, and then put the student model into practical use.

In particular, the proportion of the convolution operation part and the linear transformation part of the lightweight network in step (2) is as follows:

Among them, m is the number of channels input by the lightweight network, s is the folding ratio, and n is the number of output channels. The calculation complexity of the model is controlled by selecting the ratio of the linear combination through the folding ratio.

In particular, in step (4), the voting formula used is as follows:

n _j (x)=|{i:i∈[n],f _i (x)=j}|

In the formula, n _j represents the probability of being classified into class j. The meaning of the complete formula is the category of the sample with the most votes from the teacher model, and then adding Laplacian noise,

The final complete voting mechanism is as follows:

代表加入参数为γ的拉普拉斯噪声后被划分到j类的概率，而完整公式含义为加入拉普拉斯噪声后概率最大时j所对应的类别。in,

Represents the probability of being classified into class j after adding Laplacian noise with a parameter of γ, and the meaning of the complete formula is the category corresponding to j when the probability is the largest after adding Laplacian noise.

It can be seen from the above scheme that this method is quite different from other privacy protection technologies in terms of ideas, and has the following advantages. First of all, in the n-1 teacher model, the data selection should use mutually exclusive data, and use mutually exclusive data to protect the model. Although the utilization rate of the data is reduced, the less used data will effectively protect and reduce the exposure. risk. Second, the voting mark prediction using the teacher model actually belongs to the semi-supervised training model, and the realization of this type of model protection is not in the training stage of the teacher model, but in the fusion stage of forming a non-sensitive data set, which uses the voting The method effectively isolates the real data and the attacker, making the two relatively independent and unable to conduct privacy attacks.

To sum up, this method effectively solves the problem of complex algorithm calculation and difficult protection of training data, increases the inference speed of the network, reduces the number of parameters of the model, and ensures the security of the model.

Description of drawings

Fig. 1 is method flowchart of the present invention;

Fig. 2 is a lightweight network structure diagram of the present invention;

Fig. 3 is a diagram of the training mechanism of the present invention.

Detailed ways

The invention proposes a privacy protection method based on a lightweight face recognition model. On the basis of the traditional face recognition model, a lightweight network module is added to replace part of the traditional convolutional network operations in the form of linear mapping. While reducing the computational complexity, it ensures the fusion of features and reduces the network complexity. The number of parameters increases the running speed of the inference stage, making the model more practical. Secondly, the "teacher-student" fusion architecture is applied in the algorithm of the face recognition model, using specific Laplacian noise and voting mechanism to generate labels for non-sensitive data. Finally, the generated new data set is used as the training set of the student model to achieve the purpose of cutting off the connection between the test set and the training set. The flowchart of the method of the present invention has been clearly shown in Fig. 1, and the specific implementation steps are as follows:

(1) Divide the sensitive data set into n parts;

(2) Generating n neural networks as teacher models, each teacher model uses a lightweight network for convolution and linear transformation, and extracts deep-level features without increasing the complexity of the algorithm. The specific process is as follows:

As shown in Figure 2, for each lightweight network module, the convolution operation is first used to extract features from the original number of input channels and convert them into features of m channels, and then the features of m channels The features are linearly transformed with low algorithm complexity to generate m(s-1) auxiliary feature maps, and then the features of the two parts are superimposed to obtain n feature results, which is ms.

It can be seen from Figure 2 that the traditional feature extraction structure is a common convolution operation, which has high computational complexity, and the generated feature maps have high redundancy and many repetitions. Through the lightweight network module, the traditional convolutional network operation first extracts the more critical main features, and then performs linear transformation on the main features. Such a replacement will reduce the complexity of the algorithm. Then use the lightweight network constructed by this transformation to build a network model to realize the final model structure. The model is divided into four parts. Each part uses a different number of lightweight networks to extract features, and then performs operations such as pooling and activation functions to further achieve the purpose of reducing redundancy and fitting data distribution.

(3) Use arcface as a common loss function in the stage of outputting features. This function is a commonly used loss function for face recognition and is improved from the softmax function. The specific expression of the softmax function is as follows:

表示，其次分别对权重和输入进行L2正则化处理并乘以缩放系数s得到具体的公式如下所示：Firstly, the inner product in the formula is converted from the modulo expression

Indicates that, secondly, L2 regularization is performed on the weight and input respectively and multiplied by the scaling factor s to obtain the specific formula as follows:

因此，可以由内积表达式得到||W₁||||x||cosθ₁＞||W₂||||x||cosθ₂。所以为了进一步约束函数，引入了常数m作为严格约束得到||W₁||||x||cos(θ₁+m)＞||W₂||||x||cos(θ₂+m)。最后，通过函数的变换可以得到arcface的表达式如下所示：For general binary classification, it is usually desirable

Therefore, ||W ₁ ||||x||cosθ ₁ >||W ₂ ||||x||cosθ ₂ can be obtained from the inner product expression. So in order to further constrain the function, a constant m is introduced as a strict constraint to get ||W ₁ ||||x||cos(θ ₁ +m)＞||W ₂ ||||x||cos(θ ₂ +m ). Finally, through the transformation of the function, the expression of arcface can be obtained as follows:

(4) Through the output result plus the loss function for reverse transmission, multiple iterations to train m networks, and generate the final teacher model;

(5) Use a differential privacy-based algorithm to train the network model, label the samples of the non-sensitive data set, and use the teacher model to predict the voting mark. In fact, it belongs to the semi-supervised training model. This type of model protection The implementation is not in the training stage of the teacher model, but in the fusion stage of forming a non-sensitive data set. In this stage, the voting method is used to effectively isolate the real data and the attacker, making the two relatively independent and unable to conduct privacy attacks. The specific voting method is as follows:

n _j (x)=|{i:i∈[n],f _i (x)=j}|

In the formula, n _j (x) represents the probability that the input is classified into the j-th class, n represents the total teacher models, and f _i (x) is the prediction made by the i-th teacher model. Then add Laplace noise.

Therefore, the final complete voting mechanism obtained by adding noise is as follows:

The meaning of the formula of f(x) is the category of the label with the most votes, which increases interference by adding Laplacian noise or Gaussian noise to improve security.

(6) Use the obtained complete non-sensitive data set for training, and obtain the final student model through multiple iterations;

(7) When put into the actual family scene application, whether it is a student model or a teacher model, it is a face that has been detected or cut, and when the network actually applies the student model, because it is a complete natural image, it needs to be processed. Face Detection. Accurate face coordinates can only be recognized through detection. In the face detection part, the design of this method is divided into four stages, which are the image size adjustment stage, the face detection candidate frame generation stage, the candidate frame filter selection stage, and the final boundary determination stage.

First, in the image size adjustment stage, the network reduces and crops an image into different detection reference frames by setting the minimum face size, the minimum detection size, and the magnification factor, which is convenient for different face sizes. Do reasonable testing.

Second, put the selected images of different sizes into the detection frame generation network. After several layers of convolution operations to extract features, the face judgment branch, facial key point location branch and bounding box regression branch are generated. The data output by the network is used as the initial selection of the face detection frame.

Then, the output result of the detection frame generation network is further passed through a complex network, and the candidate frames are carefully screened in the network. After filtering out the poorly effective prediction frames, boundary regression and removal are performed on the remaining prediction candidate frames. Re-operation, the candidate box obtained by this input is better and can retain more features.

Finally, the network structure is determined by bounding boxes. The network structure is more complex, and its purpose is to enable more supervision to perform better regression on facial feature points. On the basis of retaining more features, information such as the coordinates of the detection frame and facial key points can be determined.

In this way, accurate face detection coordinates are obtained by performing operations such as boundary regression and feature point positioning separately through three network structures with different functions.

(8) After the coordinates are obtained through the detection of the face, the lightweight network of the student model is used to perform face recognition, which can ensure the security of data privacy without changing the accuracy rate.

The training data set used in this application is CASIA-WebFace, which has 10,575 subjects and 494,414 pictures in total, and is currently the largest face recognition data set in the industry. In the test set LFW, the data provider has given samples and labels of matching face pairs, so the accuracy can be tested directly. In the experimental training, the training data set is divided into n=4, 6, 8, 10... data of multiple models. One of the data is selected as non-sensitive data, that is, as a test set. The label of this type of data is obtained through the voting mechanism in the teacher model, so as to learn as a student model. Also in the test set of LFW, the data is also divided correspondingly. In the selection of the privacy noise parameter ε, the parameters used in the method include 0.6, 2.04, 5.04, 8.03, etc. The privacy cost represents the impact of adding noise on the original data training, so the larger the choice of ε, the better the protection of the data, but also the worse the availability of the data. In the selection of the parameter δ, 10 ^-5 is used. Table 1 compares face recognition model algorithms from two aspects of parameter quantity and speed. In this application, the parameter quantity and speed are greatly reduced when the recognition rate hardly changes. Table 2 analyzes and measures the recognition rate from the perspective of the number of different teacher models and the size of noise parameters. It is found that when the amount of data reaches a certain requirement, as the number of teacher models increases, the influence of noise gradually decreases.

Table 1

Table 2

Claims (3)

1. A privacy protection method based on a lightweight face recognition model is characterized in that a partial linear mapping method is used for reducing the operation complexity, and a multilayer cascade lightweight network is used for extracting the features of a face data set; adding a 'teacher-student' fusion framework, training a plurality of teacher models, adding Laplace noise, and introducing a voting mechanism to obtain labels labeled to the student models so as to increase the probability of data; putting a newly obtained labeled non-private data set into a student model for training, and finally using the obtained student model as a final network model, wherein the method specifically comprises the following steps:

step (1): preparing a sensitive data set LFW for training a network, dividing a training set picture and a verification set picture into n parts, and regarding n-1 parts of data as sensitive data and the other part as non-sensitive data;

step (2): generating n-1 improved lightweight networks, respectively putting training sets in n-1 parts of sensitive data into each network for training, dividing the lightweight network into four sub-modules during training, respectively extracting and fusing features, dividing the original features into two parts by each sub-module, generating a key feature map by using convolution operation on one part, performing simple mapping operation on the generated feature map by using linear transformation on the other part to generate other auxiliary feature maps, and fusing the two parts to obtain final features; obtaining an output calculation loss function through the whole network, and performing reverse derivation to finally obtain n-1 trained teacher models;

and (3): verifying by using n-1 verification sets which are divided, and observing whether the accuracy rate on the teacher model verification set meets the identification requirement;

and (4): preparing the rest non-sensitive data as input of all the n-1 teacher models, sequentially putting each sample into the teacher models to obtain a prediction result, finding the sample class with the maximum voting number through a softmax function, adding Laplace noise, and taking the sample with the maximum probability as a data label of each sample;

and (5): preparing the obtained non-sensitive data and the label as a new training data set, regenerating the lightweight network in the step (2), retraining the non-sensitive data as the training set of the network, extracting the characteristics and fusing to finally obtain a student model;

and (6): and adding a face detection model, acquiring face coordinates, and then carrying out face recognition on the student model.

2. The privacy protection method based on the lightweight face recognition model according to claim 1, characterized in that: the proportion of the convolution operation part and the linear transformation part of the lightweight network in the step (2) is as follows:

wherein m is the number of channels input by the lightweight network, s is the folding ratio, n is the number of output channels, and the operation complexity of the model is controlled by selecting the proportion of the linear combination according to the folding ratio.

3. The privacy protection method based on the lightweight face recognition model as claimed in claim 1, characterized in that: in step (4), the voting formula used is as follows:

n _j (x)＝|{i:i∈[n],f _i (x)＝j}|

in the formula, n _j Representing the probability of being divided into j classes, meaning the class of the sample for which the teacher model votes the most, followed by the addition of laplacian noise,

the final complete voting mechanism is as follows:

wherein,

representing the probability of being divided into j classes after adding laplacian noise with the parameter gamma, and the meaning of the complete formula is the class corresponding to j when the probability is maximum after adding laplacian noise.

Images