CN112115963A - A method for generating unbiased deep learning models based on transfer learning - Google Patents

Abstract

The invention discloses a method for generating an unbiased deep learning model based on migration learning, comprising the following steps: (1) constructing an original data set with task labels and prejudice labels of sample images; (2) using the original data set to The biased deep learning model is trained; (3) the adversarial attack network is constructed and trained, and the trained adversarial network is used to attack the original data set, the unbiased data set; (4) the unbiased data set is used to train and the biased deep learning model The initial unbiased deep learning model with the same structure; (5) prepare the third feature extractor, based on the transfer learning strategy, the parameters of the third feature extractor, the third feature extractor determined by the parameters and the trained initial unbiased deep learning model The included second classifier forms an unbiased deep learning model to ensure the fairness of the deep learning model when making automatic decisions based on the input image, so as to improve the accuracy of image recognition.

Description

A method for generating unbiased deep learning models based on transfer learning

technical field

The invention belongs to the field of deep learning, and in particular relates to a method for generating an unbiased deep learning model based on migration learning.

Background technique

With its powerful ability to learn the inherent laws and highly abstract features of sample data sets, deep learning helps people make decisions automatically and solve many complex pattern recognition problems, so it is used in medical diagnosis, speech recognition, image recognition, natural Language comprehension, advertising, credit, employment, education, and criminal justice, and to great effect. With the continuous exploration and innovation of researchers, the performance of deep learning has been continuously improved, and its applications have become more and more widely, which has had a profound impact on people's daily life.

Although deep learning can help people get more accurate predictions, the latest research shows that deep learning models are also biased in automatic decision-making. This bias may be manifested in: predicting that black defendants are far more likely to reoffend than Whites are far more accurate than blacks in predicting the gender of a person in a photo, and males are far more likely than females to search for software engineers. In some important occasions, such as enterprises, if the deep learning model is used to make decisions, this bias may make the enterprise in a high-risk business environment. If the enterprise abandons the deep learning model, it may lose in business competition. The advantage is eliminated, because the automatic decision support of deep learning is the trend of the times. It can be seen that the biases of deep learning models will have many negative impacts on society, and these biases have penetrated into various fields, so it is particularly important to study the measurement of deep learning algorithms and their fairness.

The main reasons for the bias of deep learning models are that the sample data set itself is biased, the deep learning model will amplify this bias, and the evaluation of the deep learning model is biased. Therefore, the current work of researchers to eliminate bias in deep learning models mainly includes preprocessing of sample data sets to eliminate bias, small-scale modification of deep learning model parameters to eliminate bias in the model, and deep learning models. fairness assessment. However, in the existing methods for eliminating the bias of deep learning models, only one factor of the model's bias is often considered. For example, preprocessing the sample data set directly to eliminate bias. The problem with this approach is that the trained model has not learned the biased data set, so some biased data may be recognized when identifying the original biased data. or irrelevant features are sensitive, and since deep learning models are not considered to amplify the impact of this bias, the trained model is only able to remove part of the bias.

In view of the above-mentioned biases in deep learning models and the limitations of current methods for eliminating biases, it is extremely important in theory and practice to study a method to generate unbiased deep learning models based on transfer learning, and to generate unbiased deep learning models to help people make automatic decisions. significance.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a method for generating an unbiased deep learning model based on transfer learning. Through knowledge transfer, the deep learning model can automatically filter biased features when learning sample data, so as to ensure the fairness of the deep learning model when it automatically makes decisions based on the input image, so as to improve the accuracy of image recognition.

In order to realize the above-mentioned purpose of the invention, the present invention provides the following technical solutions:

A method for generating unbiased deep learning models based on transfer learning, including the following steps:

(1) Obtain a sample image, and mark the task label and bias label of the sample image to construct the original data set;

(2) Use the image data and task labels in the original data set to train the biased deep learning model composed of the first feature extractor and the first classifier, and obtain a trained biased deep learning model;

(3) Construct and train an adversarial attack network, use the trained adversarial network to attack the original data set, and obtain an unbiased data set corresponding to the original data set, so that the biased labels in the unbiased data set are unpredictable;

(4) Use the unbiased dataset to train the initial unbiased deep learning model with the same structure as the biased deep learning model;

(5) Prepare a third feature extractor, and use the feature distribution extracted from the original sample image by the third feature extractor and the features extracted from the unbiased image corresponding to the original sample image by the second feature extractor using the unbiased deep learning model A loss function is constructed by distribution, and the parameters of the third feature extractor are optimized by the loss function. The third feature extractor determined by the parameters and the second classifier included in the trained initial unbiased deep learning model form an unbiased deep learning model.

Preferably, the constructing and training the adversarial attack network comprises:

Construct the adversarial attack network, including the convolution layer and the fully connected layer, the activation function adopts the ReLU function, the input of the adversarial attack network is the feature distribution extracted by the first feature extractor trained on the original sample image, and the output is the logits layer. The softmax function obtains the predicted probability distribution of the bias label of the original sample image;

Construct the loss function Loss_NAdv of the adversarial attack network. The loss function is designed to make the adversarial attack network predict the probability distribution of bias labels according to the feature distribution corresponding to the original sample image. The calculation formula is:

Among them, _zi is the output of the first feature extractor of the pre-trained biased deep learning model for the original sample image _{xi; B i is} the true bias label of the original sample image _xi ; nadv( ) represents the value of the adversarial attack network Output; L( ) represents the cross-entropy function, i is the index of the original sample image, and N is the total number of original sample images;

The adversarial attack network is trained with the loss function Loss_NAdv to optimize the model parameters of the adversarial attack network.

Preferably, the trained adversarial network is used to attack the original data set, and the unbiased data set corresponding to the original data set is obtained including:

(a) Design disturbance variable r;

(b) The disturbance variable r is added to the original sample image _xi to obtain the disturbance sample image, and the first feature extractor of the trained biased deep learning model is used to extract the disturbance feature distribution of the disturbance sample image;

(c) Use the trained adversarial attack network to calculate the disturbance feature distribution to obtain the predicted probability distribution, and calculate the loss Loss_Adv according to the predicted probability distribution. When the number of iterations does not reach the maximum number of iterations, update the disturbance variable r according to the loss Loss_Adv, jump Execute step (b) until the maximum number of iterations is reached, and output the perturbed sample image obtained by using the latest perturbation variable r as an unbiased image to form an unbiased data set;

The formula for calculating the loss Loss_Adv is:

Loss_Adv=-αLoss_NAdv+Loss_Y

Among them, α is a hyperparameter with a value range of 0 to 1, Loss_Y is the loss function value of the task label except the bias label, and the calculation formula is:

where c ₁ ( ) represents the predicted output of the first classifier of the biased deep learning model, and _yi represents the task label of the original sample image _xi .

Among them, the loss function Loss_tl for optimizing the parameters of the third feature extractor is:

Loss_tl=∑L(h,h')

Among them, h represents the feature distribution of the original sample image _xi output by the third feature extractor, h' represents the feature distribution obtained by the unbiased image corresponding to the original sample image _xi extracted by the second feature extractor of the unbiased deep learning model .

Among them, after the sample image is obtained, the sample image is rotated, flipped, color enhanced, added with Gaussian noise, and randomly scaled to expand the sample image. The bias labels include ethnic labels, regional labels, and gender labels.

Preferably, the first feature extractor and the second feature extractor use the ResNet-50 model;

The first classifier and the second classifier of the initial unbiased deep learning model employ a network consisting of fully connected layers.

The first classifier and the second classifier of the initial unbiased deep learning model use a network consisting of 4 fully connected layers;

The adversarial attack network adopts a network composed of 3 convolutional layers and 4 fully connected layers, and the activation function adopts the ReLU function.

Preferably, the training parameters of the adversarial attack network, the initial unbiased deep learning model and the third feature extractor are set as follows: the batch size is set to 32, the maximum number of training iterations is set to 60, the optimizer is Adam, and the learning rate is set to 0.001, The exponential decay rates for the first and second estimates were set to 0.9 and 0.999, respectively.

Compared with the prior art, the beneficial effects of the present invention at least include:

The method for generating an unbiased deep learning model based on migration learning provided by the embodiment of the present invention enables the deep learning model to obtain the ability to automatically filter biased features of sample data based on the strategy of migration learning, ensures the fairness of model decision-making, and improves the performance of image recognition. accuracy.

Description of drawings

In order to illustrate the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative efforts.

1 is a schematic flowchart of a method for generating an unbiased deep learning model based on transfer learning according to an embodiment of the present invention;

2 is a system framework diagram of an unbiased deep learning model based on transfer learning provided by an embodiment of the present invention;

Fig. 3 is a construction flow chart of generating an unbiased data set provided by an embodiment of the present invention;

FIG. 4 is a training flowchart of an unbiased deep learning model based on transfer learning provided by an embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, and do not limit the protection scope of the present invention.

In order to solve the problem of inaccurate image recognition due to the bias problem of deep learning models. This embodiment provides a method for generating an unbiased deep learning model based on migration learning. As shown in FIG. 1 , the method for generating an unbiased deep learning model based on migration learning includes the following steps:

(1) Definition of deep learning model bias.

In the present invention, the erroneous decision-making caused by the deep learning model relying on false relevant features during automatic decision-making is defined as the decision bias of the deep learning model. Among them, false related features are bias features, which may be race, region, gender, etc. For example, when the bias feature is gender, when using deep learning models to screen software engineers' resumes, the elimination rate of women is often higher. This results in discrimination against women in the profession and unfair model decision-making. Therefore, the present invention removes the bias of the model by filtering irrelevant features for classification.

(2) Data set preparation and preprocessing.

This embodiment selects an image dataset with multi-label classification, such as the COCO dataset, and uses one of the biased labels B as a biased feature, such as a gender feature. Select one or more of the other tags as task tags. The task tags can be occupation tags, etc., and preprocess the data set. In order to improve the recognition accuracy of the deep learning model, data augmentation operations can be added to expand the data set, including rotation. , flip, color enhancement, add Gaussian noise, random scaling. The images after the data enhancement operation are composed of the original data set, and the original data set is divided into a training set and a test set according to the ratio of 7:3, which are used for the transfer learning and testing of the unbiased model respectively.

(3) Build and train a biased deep learning model.

In this embodiment, the constructed biased deep learning model includes a first feature extractor and a first classifier, wherein the first feature extractor adopts the ResNet-50 model structure, and the first classifier adopts four fully connected layers. network of. Use the training set of the original data set to train the biased deep learning model, and use the test set to test and optimize the biased deep learning model, so that the biased deep learning model can reach the preset recognition accuracy. Since the original dataset used for training contains biased features, the trained model is called a biased deep learning model.

(4) Build and train the adversarial attack network.

This adversarial attack network aims to predict the biased label B through the output features of the first feature extractor of a biased deep learning model, and is mainly used to generate an unbiased dataset. The specific process is as follows:

(4.1) Construct the structure of the adversarial attack network. The adversarial attack network adopts a network composed of 3 convolutional layers and 4 fully connected layers, and the activation function adopts the ReLU function. The input of the adversarial attack network is the output z of the first feature extractor of the biased deep learning model on the original dataset in step (3), where z represents the feature distribution of the original dataset. The output of the adversarial attack network is the logits layer, and the predicted probability distribution of the bias labels of the original data set is obtained through the softmax function.

(4.2) Design the loss function of the adversarial attack network. The loss function is designed to make the adversarial attack network predict the probability distribution of bias labels according to the feature distribution corresponding to the original sample image. The calculation formula is as follows:

Among them, _zi is the output of the original sample image _xi through the first feature extractor of the biased deep learning model in step (3); B _i is the true bias label of the original sample image _xi ; nadv( ) represents the adversarial attack The output of the network; L( ) represents the cross-entropy function.

(4.3) Train the adversarial attack network, set the batch size to 32, the maximum number of training iterations to 60, the optimizer to use Adam, the learning rate to 0.001, and the exponential decay rates of the first and second estimates to be 0.9, respectively and 0.999. The adversarial attack network is trained using the training set of the original data set and its bias label B, and the adversarial attack network is tested and optimized with the test set, so that the adversarial attack network can reach the preset recognition accuracy.

(5) Generate an unbiased dataset, and use the adversarial attack network to attack the original dataset. The attack is to add disturbance to the original dataset to make the biased label B of the generated unbiased dataset unpredictable, but other labels are not affected. . The original dataset corresponds to the unbiased dataset one-to-one.

Design the loss function against the attack, and the calculation formula is as follows:

Loss_Adv=-αLoss_NAdv+Loss_Y (2)

Among them, α is a hyperparameter, Loss_Y is the loss function value of other classification labels except the bias label B, and its calculation formula is as follows:

Among them, c ₁ (·) represents the output of the first classifier of the biased deep learning model in step (3); _yi represents the true label of the original sample image _xi .

The disturbance variable r is designed, where r is a matrix of weight×hight×3, where weight and hight are the width and height of the sample data image respectively, 3 represents the RGB three channels of the sample data image, and the disturbance variable r is initialized to a zero matrix. The disturbance variable r is optimized by the Adam optimizer, and the optimizer parameters are the same as in step (4.3).

The maximum number of iterations for a single sample is set to 1000, as shown in Figure 3. The specific process of generating unbiased data for each original dataset sample is as follows:

(5.1) Input the original sample image _xi and go to step (5.2);

(5.2) The original sample image _xi and the disturbance variable r are superimposed to obtain the disturbance sample image _xi ', and then go to step (5.3);

(5.3) Input the perturbed sample image _xi ' into the first feature extractor of the biased deep learning model in step (3) and output _zi , and go to step (5.4);

(5.4) Input _zi into the adversarial attack network of step (4), and output nadv( _zi ), go to step (5.5);

(5.5) Calculate the loss function value according to formulas (1) to (3), and go to step (5.6);

(5.6) Judging whether the maximum number of iterations is reached, if so, output the disturbed sample image _xi ' as an unbiased image to form an unbiased data set, and end the iteration, if not, go to step (5.7);

(5.7) Use Adam to update the disturbance variable r according to the loss function value, and go to step (5.2).

(6) Build and train an initial unbiased deep learning model.

The initial unbiased deep learning model structure is the same as the biased deep learning model in step (3). That is, the initial unbiased deep learning model includes a second feature extractor with the same structure as the first feature extractor, and also includes a second classifier with the same result as the first classifier.

Set the batch size to 32, the maximum number of training iterations to 60, the optimizer to use Adam, the learning rate to 0.001, and the exponential decay rates for the first and second estimates to be 0.9 and 0.999, respectively. Use the unbiased data set generated in step (5) to train the initial unbiased deep learning model, and use the test set to test and optimize the model, so that the initial unbiased deep learning model reaches a preset recognition accuracy. Let the output of the second feature extractor of the initial unbiased deep learning model be h', and let the second classifier of the initial unbiased deep learning model be c ₂ (·).

(7) Design an unbiased deep learning model training framework based on transfer learning.

As shown in Figure 2, the concrete process of step (7) is:

(7.1) To design the structure of an unbiased deep learning model based on transfer learning, a third feature extractor can be used, whose input is the original sample image _xi , and the output is an intermediate feature h; the third classifier is trained by the initial unbiased deep learning model good second classifier c ₂ ( );

(7.2) Design the loss function of the unbiased deep learning model based on transfer learning. The loss function is designed to make the unbiased deep learning model learn the knowledge of the initial unbiased deep learning model, so that the unbiased deep learning model can face the original data set. It can automatically filter the biased features when the loss function is calculated as follows:

Loss_tl=∑L(h,h') (4)

where h represents the output of the third feature extractor of the unbiased deep learning model, and h' represents the output of the second feature extractor of the initial unbiased deep learning model.

(7.3) Design the initialized training parameters, set the batch size to 32, the maximum number of training iterations to 60, the optimizer to use Adam, the learning rate to 0.001, and the exponential decay rates of the first and second estimates to be set as 0.9 and 0.999.

(7.4) The training process of designing an unbiased deep learning model, combined with Figure 4, the specific process is as follows:

(7.4.1) Input the original sample image _xi into the third feature extractor and output h, go to step (7.4.2);

(7.4.2) Input the unbiased image _xi ' corresponding to the original sample image _xi into the second feature extractor of the initial unbiased deep learning model and output h', go to step (7.4.3);

(7.4.3) Calculate the loss function value according to formula (4), go to step (7.4.4);

(7.4.4) Update the parameters of the unbiased deep learning model according to the loss function value, go to step (7.4.5);

(7.4.5) Determine whether the maximum number of iterations is reached, if so, save the model and end the training; if not, go to step (7.4.1).

(8) Train and test an unbiased deep learning model.

According to the training process of step (7.4), use the original data set and the unbiased data set to train the third feature extractor of the unbiased deep learning model. After the training, the third feature extractor and the initial unbiased deep learning of step (6) The second classifier trained by the model is connected as an unbiased deep learning model, that is, an unbiased deep learning model generated based on transfer learning in the present invention. The test set of the original data set is used to test the prejudice degree λ of the unbiased deep learning model generated based on transfer learning. The smaller the prejudice degree λ value of the model, the fairer the model is in decision-making. The calculation formula is as follows:

Among them, n represents the total number of test set sample data in the original data set; nadv(h _i ) represents the output of the feature extractor of the unbiased model and the adversarial attack network of the sample data _xi through the transfer learning; the function l[ ] represents the indicator function, when The value in parentheses is 1 when the formula is established, and 0 otherwise.

The method for generating an unbiased deep learning model based on transfer learning provided above proposes a new unbiased model training framework, which generates an unbiased deep learning model through the knowledge transfer of the unbiased model and the adversarial attack network. The model training method simultaneously solves the problem of bias in training data and model structure parameters, and further ensures the fairness of model decision-making. The proposed unbiased model for transfer learning can choose a simple structure, which greatly reduces the computational time of deep learning models in practical applications. The proposed strategy based on transfer learning can enable the deep learning model to automatically filter biased features while ensuring the accuracy of the original target classification task, ensure the fairness of the deep learning model in decision-making, and provide guidance for research on eliminating the bias of deep learning models. .

The above-mentioned specific embodiments describe in detail the technical solutions and beneficial effects of the present invention. It should be understood that the above-mentioned embodiments are only the most preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, additions and equivalent substitutions made within the scope shall be included within the protection scope of the present invention.

Claims (8)

1. a method for generating an unbiased deep learning model based on migration learning, is characterized in that, comprises the following steps: (1) Obtain a sample image, and mark the task label and bias label of the sample image to construct the original data set; (2) Use the image data and task labels in the original data set to train the biased deep learning model composed of the first feature extractor and the first classifier, and obtain a trained biased deep learning model; (3) Construct and train an adversarial attack network, use the trained adversarial network to attack the original data set, and obtain an unbiased data set corresponding to the original data set, so that the biased labels in the unbiased data set are unpredictable; (4) Use the unbiased dataset to train the initial unbiased deep learning model with the same structure as the biased deep learning model; (5) Prepare a third feature extractor, and use the feature distribution extracted from the original sample image by the third feature extractor and the features extracted from the unbiased image corresponding to the original sample image by the second feature extractor using the unbiased deep learning model A loss function is constructed by distribution, and the parameters of the third feature extractor are optimized by the loss function. The third feature extractor determined by the parameters and the second classifier included in the trained initial unbiased deep learning model form an unbiased deep learning model. 2. The method for generating an unbiased deep learning model based on transfer learning as claimed in claim 1, wherein the constructing and training the adversarial attack network comprises: Construct an adversarial attack network, including a convolutional layer and a fully connected layer, the activation function adopts the ReLU function, the input of the adversarial attack network is the feature distribution extracted by the first feature extractor trained on the original sample image, and the output is the logits layer. The softmax function obtains the predicted probability distribution of the bias label of the original sample image; Construct the loss function Loss_NAdv of the adversarial attack network. The loss function is designed to make the adversarial attack network predict the probability distribution of bias labels according to the feature distribution corresponding to the original sample image. The calculation formula is:

Among them, _zi is the output of the first feature extractor of the pre-trained biased deep learning model for the original sample image _{xi; B i is} the true bias label of the original sample image _xi ; nadv( ) represents the value of the adversarial attack network Output; L( ) represents the cross-entropy function, i is the index of the original sample image, and N is the total number of original sample images; The adversarial attack network is trained with the loss function Loss_NAdv to optimize the model parameters of the adversarial attack network. 3. The method for generating an unbiased deep learning model based on migration learning as claimed in claim 2, wherein the trained adversarial network is used to attack the original data set, and the unbiased data set corresponding to the original data set includes: : (a) Design disturbance variable r; (b) The disturbance variable r is added to the original sample image _xi to obtain the disturbance sample image, and the first feature extractor of the trained biased deep learning model is used to extract the disturbance feature distribution of the disturbance sample image; (c) Use the trained adversarial attack network to calculate the disturbance feature distribution to obtain the predicted probability distribution, and calculate the loss Loss_Adv according to the predicted probability distribution. When the number of iterations does not reach the maximum number of iterations, update the disturbance variable r according to the loss Loss_Adv, jump Execute step (b) until the maximum number of iterations is reached, and output the perturbed sample image obtained by using the latest perturbation variable r as an unbiased image to form an unbiased data set; The formula for calculating the loss Loss_Adv is: Loss_Adv=-αLoss_NAdv+Loss_Y Among them, α is a hyperparameter with a value range of 0 to 1, Loss_Y is the loss function value of the task label except the bias label, and the calculation formula is:

where c ₁ ( ) represents the predicted output of the first classifier of the biased deep learning model, and _yi represents the task label of the original sample image _xi . 4. The method for generating an unbiased deep learning model based on migration learning as claimed in claim 1, wherein the loss function Loss_t1 of optimizing the third feature extractor parameter is: Loss_tl=∑L(h,h') Among them, h represents the feature distribution of the original sample image _xi output by the third feature extractor, h' represents the feature distribution obtained by the unbiased image corresponding to the original sample image _xi extracted by the second feature extractor of the unbiased deep learning model . 5. The method for generating an unbiased deep learning model based on migration learning as claimed in claim 1, wherein after acquiring the sample image, the sample image is rotated, flipped, color enhanced, added Gaussian noise, and randomly scaled to The sample images are augmented, and bias labels include ethnic labels, geographical labels, and gender labels. 6. The method for generating an unbiased deep learning model based on migration learning as claimed in claim 1, wherein the first feature extractor and the second feature extractor use the ResNet-50 model; The first classifier and the second classifier of the initial unbiased deep learning model employ a network consisting of fully connected layers. 7. The method for generating an unbiased deep learning model based on transfer learning according to claim 1, wherein the first classifier and the second classifier of the initial unbiased deep learning model are composed of four fully connected layers. formed network; The adversarial attack network adopts a network composed of 3 convolutional layers and 4 fully connected layers, and the activation function adopts the ReLU function. 8. The method for generating an unbiased deep learning model based on migration learning as claimed in claim 1, wherein the training parameters of the adversarial attack network, the initial unbiased deep learning model and the third feature extractor are set as: Batch size is set as follows. is 32, the maximum number of training iterations is set to 60, the optimizer adopts Adam, the learning rate is set to 0.001, and the exponential decay rates of the first and second estimates are set to 0.9 and 0.999, respectively.

Images