CN116386111A - Face recognition-oriented patch attack countermeasure method - Google Patents

Abstract

The invention relates to a face recognition-oriented patch attack countermeasure method, and belongs to the field of deep learning. Compared with the common counterpatch method, the countersample attack method based on the countermask patch provided by the invention provides the mask shape more suitable for the face recognition model and provides the hidden generation method more suitable for the application scene, so that the attack robustness of the method is improved, and the possibility of successful attack in the real world is increased. The main innovation of the invention is to design a disturbance generation method aiming at a face recognition system, and the related loss function enhances the concealment of patches by aiming at the dependence of the face recognition network on embedded vectors and by the links of design style loss, so that the possibility of attacking a real system is greatly improved, and the consideration of concealment factors is increased on the premise of considering the success rate of the method. In real life, for example, an attacker can impersonate other identities to perform face verification, steal account information of other people after passing, and the like. The attack method provides a hidden related evaluation standard for defense.

Description

Face recognition-oriented patch attack countermeasure method

Technical Field

The invention belongs to the field of deep learning, and relates to a face recognition-oriented patch attack resisting method.

Background

The method for resisting the sample attack refers to adding disturbance which is not easy to be perceived by naked eyes on the local or the global of the original input image so as to intervene in the judgment of the input by the deep learning network such as classification or regression. The method is generally characterized in that extremely small noise exists in the picture, the noise is specially formulated by utilizing the structural characteristics of complex iteration of the network, and the picture is input by iteration by utilizing the characteristics of gradient, output vector and the like, so that the network misjudgment is induced. The existence of the attack against the sample brings great potential threat to most identity information verification systems, automatic driving systems and target detection systems in the market, and researches and puts forward a corresponding attack method to bring great value to successful defense and strengthening of networks in practical application.

The current method of countering the sample can be divided into a local attack method and a global attack method, wherein the global attack method is to cover the whole input image with a layer of generated countering noise, thereby changing the original input image into the countering sample. This approach is of little interest in real life because it is difficult for an attacker to capture and modify the relevant pictures before entering them into the system. The local disturbance attack method is easier to migrate to the physical world, and the local disturbance attack method falsifies the input image in a mode of hiding a tiny countermeasure patch in a local area of the input image, so that the network is induced to make error judgment. The perturbation in the form of such patches is more conducive to migration into the real world, thereby posing a significant threat to real world applications. Because the methods for resisting the samples do not need to operate a network and a training set, and only tamper is carried out on the tested samples, the difficulty in implementation is low, and the robustness of the system can be improved by more effectively adding corresponding defenses for the application by researching the attack methods.

The existing attack method based on patches is generated by carrying out disturbance on the network gradient, and because the deep learning network carries out continuous iteration and convolution to infer the final result, the output result of each layer is calculated as the input of the next layer, and the judgment of each layer on the correct category of the network can be effectively misled by designing a loss function in the opposite direction of the original correct category gradient, so that the final judgment errors are accumulated. Most of the existing methods utilize the idea to design the loss function of the iterative patch, neglecting the consideration of generating the concealment of the patch, and the concealment is one of important standards for the consideration of the effect of the patch.

Disclosure of Invention

In view of the above, the present invention aims to provide a method for countering patch attack for face recognition.

In order to achieve the above purpose, the present invention provides the following technical solutions:

a face recognition-oriented patch attack countermeasure method comprises the following steps:

s1: the random patch is selected for pretreatment, the shape of the outlet mask is cut and spliced with the input original-like image, and the image of the simulated mask on the face is used as initial input;

s2: generating a picture and an original image, delivering the picture and the original image to a network for feature extraction, respectively obtaining mapping vector outputs of the two pictures, and recording feature values;

s3: carrying out loss function calculation by using an output vector of the contrast patch synthesized picture and an output vector of a real class, updating patch pixels through a gradient direction, and generating a brand new contrast patch;

s4: and (3) iterating, repeating the first step operation until the iteration number threshold is set, and outputting a final countermeasure patch.

Optionally, in the step S4, the iterative method for performing iteration is to continuously repeat the training for m times for each picture x, calculate the gradient of the last step of multiplexing when disturbing r, and divide the whole epoch by m in order to ensure the speed and enhance the concealment; wherein the update formula of r is:

r _t+1 ＝r _t +∈·sign(g)

wherein sign (g) represents the gradient direction, and E is a preset super parameter to control the gradient direction to be the gradient direction; wherein x is input, and y is a label for the loss function

The gradient write of (2) is:

the specific iterative step algorithm is as follows:

for the original input sample x and the corresponding label y, selecting a book patch p ₀ Training is carried out; in the N/m iterative processes, updating the patch in each round; in each round, firstly, covering the picture x by patches to obtain a brand new face picture so as to simulate wearing a mask; the second step is to pass through the loss function L (x ₀ Y, θ) calculating a loss value; third step, calculating gradient corresponding to the loss function

And fourthly, updating the model parameter theta by a random gradient descent method, wherein the updating formula is as follows: />

Updating the gradient of the input x by the updated theta to obtain the gradient direction of the updating countermeasure patch +.>

Finally, the unit value of the gradient direction is accumulated on the original patch, namely p ₀ ←p ₀ +∈·sign(g _adv ) The method comprises the steps of carrying out a first treatment on the surface of the Repeating the fourth step m times until gradient descent is obtained;

the total loss function is:

wherein P is _adv Representing the generated challenge patch, P ₀ Represents an initial patch that is not iterated, where random noise is selected, x represents the input face image, x _t Representing a target object;

the similar loss function is written for a directional attack as:

l _tsim (P _adv ，x，x _t )＝cos(em(T(P _adv ，x))，em(x _t ))

where em (.) represents a mapping vector (embedded) that inputs a picture to the face recognition network output, T (.) represents a transformation operation that passes the patch onto the face, cos (.) is the cosine distance between the vectors; for a non-directional attack, the form of the similarity loss function is:

l _utsim (P _adv ，x，x _t )＝-cos(em(T(P _adv ，x)，em(x))

inducing network misjudgment by minimizing the cosine distance between the generated mapping vector and the target vector; the style loss is expressed as the difference in the gram matrix between the generated patch and the original patch:

wherein G (.) is a gram matrix used to represent the structure of a particular graph and the matrix used to calculate the correlation coefficient, describing the relationships between nodes in the graph and the structure of one intranet layer; using a gram matrix to maintain the stylistic nature of the patch from the original; the original gram matrix requires the bottom to the top layers of the iterative network, taking the input of the top layer into account with the loss each time.

The invention has the beneficial effects that: compared with the common counterpatch method, the countersample attack method based on the countermask patch provided by the invention provides the mask shape more suitable for the face recognition model and provides the hidden generation method more suitable for the application scene, so that the attack robustness of the method is improved, and the possibility of successful attack in the real world is increased. The main innovation of the invention is to design a disturbance generating method aiming at a face recognition system, and the related loss function enhances the concealment of the patch by aiming at the dependence of the face recognition network on the embedded vector and by the link of design style loss, thereby greatly improving the possibility of attacking the real system. In real life, for example, an attacker can impersonate other identities to perform face verification, steal account information of other people after passing, and the like. The attack method provides a hidden related evaluation standard for defense.

Compared with a countermeasure sample of a general attack classification type, the method can better utilize the information of the face recognition network to generate a more targeted method patch, and for attack data with larger inter-class distance and smaller intra-class distance, such as a face recognition algorithm, the method is more suitable for a loss function of the face recognition model to better finish the attack in reality. The novel breakthrough in the fields of countermeasure defense, countermeasure training and the like can be brought by better attack effects.

Compared with the classified attack patch of the related face method, the newly added hidden method of the method can better patch, thereby being capable of effectively aiming at the face recognition system in the real world. The target characteristic information can be more gently hidden into the surrounding pixel information through the assistance of style loss and smoothing loss, so that the effect is more natural and hidden.

Additional advantages, objects, and features of the invention will be set forth in part in the description which follows and in part will become apparent to those having ordinary skill in the art upon examination of the following or may be learned from practice of the invention. The objects and other advantages of the invention may be realized and obtained by means of the instrumentalities and combinations particularly pointed out in the specification.

Drawings

For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in the following preferred detail with reference to the accompanying drawings, in which:

FIG. 1 is a flow chart of an overall method of single-step iterative challenge training with increased disturbance diversity;

FIG. 2 is a non-directional attack-versus-mask similarity experiment;

FIG. 3 is a graph of similarity experiment of orientation attack versus mask;

fig. 4 is a comparative test of migration between models.

Detailed Description

Other advantages and effects of the present invention will become apparent to those skilled in the art from the following disclosure, which describes the embodiments of the present invention with reference to specific examples. The invention may be practiced or carried out in other embodiments that depart from the specific details, and the details of the present description may be modified or varied from the spirit and scope of the present invention. It should be noted that the illustrations provided in the following embodiments merely illustrate the basic idea of the present invention by way of illustration, and the following embodiments and features in the embodiments may be combined with each other without conflict.

Wherein the drawings are for illustrative purposes only and are shown in schematic, non-physical, and not intended to limit the invention; for the purpose of better illustrating embodiments of the invention, certain elements of the drawings may be omitted, enlarged or reduced and do not represent the size of the actual product; it will be appreciated by those skilled in the art that certain well-known structures in the drawings and descriptions thereof may be omitted.

The same or similar reference numbers in the drawings of embodiments of the invention correspond to the same or similar components; in the description of the present invention, it should be understood that, if there are terms such as "upper", "lower", "left", "right", "front", "rear", etc., that indicate an azimuth or a positional relationship based on the azimuth or the positional relationship shown in the drawings, it is only for convenience of describing the present invention and simplifying the description, but not for indicating or suggesting that the referred device or element must have a specific azimuth, be constructed and operated in a specific azimuth, so that the terms describing the positional relationship in the drawings are merely for exemplary illustration and should not be construed as limiting the present invention, and that the specific meaning of the above terms may be understood by those of ordinary skill in the art according to the specific circumstances.

A method of countering sample attacks based on a countermask patch, comprising the steps of:

step S1: the random patches are selected for pretreatment, the shapes of the cut outlet covers are spliced with the input original-like images, and the pictures of the simulated masks on the faces are used as initial input.

Step S2: and (5) generating a picture and an original image, delivering the picture and the original image to a network for feature extraction, respectively obtaining mapping vector outputs of the two pictures, and recording feature values.

Step S3: and carrying out loss function calculation on the output vector of the contrast patch synthesized picture and the output vector of the real class, updating patch pixels through the gradient direction, and generating a brand new contrast patch.

Step S4: repeating the first step to reach about the threshold value of the set iteration times, and outputting the final countermeasure patches.

Further, the iterative method is to continuously repeat the training for m times for each picture x, multiplex the gradient of the previous step when calculating the disturbance r, and divide the whole epoch by m in order to ensure the speed and enhance the concealment. Wherein the update formula of r is:

r _t+1 ＝r _t +∈·sign(g)，

wherein sign (g) represents the gradient direction, and E is a preset super parameter to control the gradient direction to be the gradient direction. Wherein x is input, and y is a label for the loss function

Can be written as a gradient of

The iterative method can effectively prevent the patch from excessively meeting the gradient direction, so that the image is too similar to the target feature, and the concealment of the attack method is reduced.

The specific iterative step algorithm is as follows:

the total loss function can be written as

Wherein P is _adv Representing the generated challenge patch, P ₀ Represents an initial patch that is not iterated, where random noise is selected, x represents the input face image, x _t Representing the target object.

The similar loss function can be written for directional attacks

l _utsim (P _adv ，x，x _t )＝cos(em(T(P _adv ，x)，em(x _t ))

Where em (.) represents the mapping vector (embedded) that inputs the picture to the face recognition network output, T (.) represents the transformation operation that passes the patch onto the face, and cos (.) is the cosine distance between the vectors. For a non-directional attack, the form of the similarity loss function is:

l _utsim (P _adv ，x，x _t )＝-cos(em(T(P _adv ，x))，em(x))

network misjudgment is induced by minimizing the cosine distance between the generated mapping vector and the target vector. The style loss may be expressed as the difference in the gram matrix between the generated patch and the original patch:

where G (..) is a Gram matrix, a matrix used to represent a particular graph structure and calculate correlation coefficients, which can describe the relationships between nodes in the graph, used to describe the structure of an intranet layer. The gram matrix is used here to keep the patch from the original style, thereby enhancing the concealment of the method. The original gram matrix needs to iterate the bottom layer to the high layer of the network, and as the method needs to repeatedly input the input of the highest layer to the network for updating the patch, the input of the highest layer is only needed to be brought into loss to participate in iteration at each time.

Example 1

Fig. 1 is a flowchart showing the whole method of challenge sample attack based on the challenge mask patch according to the present invention. To verify the effectiveness of the method we designed experiments for the relevant concealment and similarity tests.

According to the requirements of related experiments, the invention provides related evaluation indexes in total:

cosine similarity: and measuring a measurement index which is very important in face recognition, and verifying the identity by measuring cosine pinch between feature vectors in two face images. In the experiment, facial features of a face are extracted, and the cosine distance is calculated to measure the reliability of generating an countermeasure sample. This evaluation method was employed in both physical and digital world experiments herein to verify the aggressiveness of the mask.

Attack success rate: an index for measuring whether the whole method is successful or not is that the recognition rate cannot be guaranteed in a physical world experiment, and a plurality of ineffective screenshots exist in one frame of video, so that batch experiments are required to be carried out on a plurality of groups of pictures, and whether the whole system can be attacked successfully or not is measured by counting the proportion of the number of pictures with the similarity passing a specific threshold value to the total number of the counted batch pictures. For a directional attack, the expression of the success rate can be written as:

wherein I represents the image of the test specimen,

representing an attack target image.

Peak Signal-to-Noise Ratio (Peak Signal-to-Noise Ratio): since the digital world method aims at guaranteeing the effectiveness of patch attack on the premise of improving the concealment of the sample, the measurement of the concealment is still an important index. The clean image X and the incidental patch noise picture Y are compared herein by peak signal to noise ratio. First, the mean square error of two pictures input as mxn is defined as:

the expression giving the peak signal-to-noise ratio on the basis is:

wherein the method comprises the steps of

Is the maximum pixel value possible in the image, here represented by 8-bit binary per pixel, with a maximum pixel value of 255. For color images, the final signal-to-noise ratio is obtained by averaging the calculated PSNR values of the RGB three channels.

Method concealment experiment

For the hidden experiments, we select ResNet-18 backbone network to be carried out in a pre-training model by adopting an MS-Celeb-1M face data set, the parameter setting in the whole training process is batch size (batch size) 64, the learning rate is initialized to 0.01, and the iteration times and the epoch round number are set to 100. All experiments herein were run on NVIDIA Tesla V100 server, the main deep learning framework was Pytorch 1.7, and the image processing framework was OpenCV 4.5.2. After iteration, an anti-mask is generated and subjected to a concealment test, three experiments are respectively carried out on three different random number seeds, the average value of the three experimental results is taken as a final result, and the data index results are shown in Table 1.

As can be seen from table 1, the patch can still maintain a certain masking effect after a sufficient number of iterations. For the countersample method of patches, the patch needs to try to make the characteristic information of the misleading network contained in the patch easy to expose, the test of the experiment is to compare the characteristic information with the random disturbance of the initial speech as a reference, and the iteration generating characteristic is hidden by the hidden loss of the method. In fig. 3, the first column is the face image of the mask wearing the anti-patch mask, the second column is the face image of the mask wearing the random disturbance initial patch, and the third column is the original image in the database. In table 1, the first column shows the PSRN score of the original image after wearing the antigen mask, and the second column shows the PSRN score of the original image after wearing the random mask, and the signal to noise ratio score of the blue mask is about 20, so as to simulate the general difference of human eyes to observe whether the face is worn on the mask.

Method similarity experiments

Through the above experimental analysis, a relatively successful method of counterpatch is presented herein in terms of concealment. On the premise that the sample has stronger concealment, similarity score comparison is carried out, experiments are carried out on the first thousand classes of CASIA-WebFace, and cosine similarity comparison is carried out on the original sample, the random disturbance sample and the mask contrast patch sample method and the true class, so that the results shown in the figures 2 and 3 are obtained.

The abscissa in fig. 2 and 3 represents the mask type, and No. 0 mask, no. 1 countermask, no. 2 random disturbance mask, and No. 3 white mask are sequentially arranged from left to right. The ordinate represents the cosine similarity score obtained by testing the sample in the network. Wherein the green dot is an outlier representing the number of times the sample distribution deviates from the majority. The experimental result shows that the anti-mask and the random disturbance mask have better attack effects, and the directional attack method has better effect than the random disturbance. Because in the non-directional attack, the types of the attack targets are complex and various, random noise hits part of the features more easily to cause interference, but in the directional attack, the target domain required by the experiment is limited and the success rate is low. The non-directional attack method can reduce the accuracy of the face recognition model to be within the interval of 0.2 and 0.4, and compared with the similarity above 0.5, the similarity of the face recognition model is obviously reduced in the attack effect; non-directional attack attacks can induce the original erroneous [0.2,0.3] similarity value to above 0.5.

The migration test experiments are performed under three models of ArcFace, cosFace and MegFace in all backbone networks, and the backbone networks generated by the mask are based on ResNet50, and the experimental results under different face recognition models of different backbone networks are shown in fig. 4, wherein x-coordinate represents the type of attack patch mask, the original picture of the attack patch mask, the random disturbance mask, the blue mask and the mask-free mask are sequentially from left to right, y-coordinate represents the type of network model, and ArcFace-ResNet18, cosFace-ResNet18, arcFace-ResNet34, cosFace-ResNet34, arcFace-ResNet50, cosFace-ResNet50, arcFace-ResNet101, megface-ResNet101 and z-coordinate represent similarity scores. From experimental results, it can be seen that the challenge patches trained on the ResNet50 based backbone network can exhibit effective attack capability between models. In particular, it can be found that there is a slightly better attack effect when the depth of the ResNet network model is deeper, i.e., there is a similarity score for each mask type that decreases with increasing depth. This is also common in attack methods, where network iterations through deeper layers can lead to errors that are obtained each time deepening as the number of layers is updated. Particularly, when the backbone network of ResNet50 is adopted, the attack belongs to a white box attack method, the performance of the attack effect is more prominent, when the method attacks other class models of other backbone networks, the attack method belongs to a black box method, and good scores are also provided for the attack result. Because the attack loss function of the method is based on the cosine distance between network output vectors, the method is mostly a common processing mode of a face recognition model, so that certain mobility exists between the methods.

TABLE 1 hidden PSNR index score

The effectiveness of the countermeasure training method provided by the invention is verified by the experiment and the related result analysis.

Finally, it is noted that the above embodiments are only for illustrating the technical solution of the present invention and not for limiting the same, and although the present invention has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications and equivalents may be made thereto without departing from the spirit and scope of the present invention, which is intended to be covered by the claims of the present invention.

Claims (2)

1. A face recognition-oriented patch attack countermeasure method is characterized in that: the method comprises the following steps:

s1: the random patch is selected for pretreatment, the shape of the outlet mask is cut and spliced with the input original-like image, and the image of the simulated mask on the face is used as initial input;

s2: generating a picture and an original image, delivering the picture and the original image to a network for feature extraction, respectively obtaining mapping vector outputs of the two pictures, and recording feature values;

s3: carrying out loss function calculation by using an output vector of the contrast patch synthesized picture and an output vector of a real class, updating patch pixels through a gradient direction, and generating a brand new contrast patch;

s4: and (3) iterating, repeating the first step operation until the iteration number threshold is set, and outputting a final countermeasure patch.

2. The face recognition-oriented patch attack countermeasure method of claim 1, wherein: in the step S4, the iterative method is that m times of training are continuously repeated for each picture x, the gradient of the last step is multiplexed when the disturbance r is calculated, and in order to ensure the speed and enhance the concealment, the integral epoch is divided by m; wherein the update formula of r is:

r _t+1 ＝r _t +∈·sign(g)

wherein sign (g) represents the gradient direction, and E is a preset super parameter to control the gradient direction to be the gradient direction; wherein x is input, and y is a label for the loss function

The gradient write of (2) is:

the specific iterative step algorithm is as follows:

for the original input sample x and the corresponding label y, selecting a book patch p ₀ Training is carried out; in the N/m iterative processes, updating the patch in each round; in each round, firstly, covering the picture x by patches to obtain a brand new face picture so as to simulate wearing a mask; the second step is to pass through the loss function L (x ₀ Y, θ) calculating a loss value; third step, calculating the corresponding loss functionGradient of (2)

And fourthly, updating the model parameter theta by a random gradient descent method, wherein the updating formula is as follows:

updating the gradient of the input x by the updated theta to obtain the gradient direction of the updating countermeasure patch +.>

Finally, the unit value of the gradient direction is accumulated on the original patch, namely p ₀ ←p ₀ +∈·sign(g _adv ) The method comprises the steps of carrying out a first treatment on the surface of the Repeating the fourth step m times until gradient descent is obtained;

the total loss function is:

wherein P is _adv Representing the generated challenge patch, P ₀ Represents an initial patch that is not iterated, where random noise is selected, x represents the input face image, x _t Representing a target object;

the similar loss function is written for a directional attack as:

l _tsim (P _adv ，x，x _t )＝cos(em(T(P _adv ，x))，em(x _t ))

where em (.) represents a mapping vector (embedded) that inputs a picture to the face recognition network output, T (.) represents a transformation operation that passes the patch onto the face, cos (.) is the cosine distance between the vectors; for a non-directional attack, the form of the similarity loss function is:

l _utsim (P _adv ，x，x _t )＝-cos(em(T(P _adv ，x))，em(x))

inducing network misjudgment by minimizing the cosine distance between the generated mapping vector and the target vector; the style loss is expressed as the difference in the gram matrix between the generated patch and the original patch:

wherein G (.) is a gram matrix used to represent the structure of a particular graph and the matrix used to calculate the correlation coefficient, describing the relationships between nodes in the graph and the structure of one intranet layer; using a gram matrix to maintain the stylistic nature of the patch from the original; the original gram matrix requires the bottom to the top layers of the iterative network, taking the input of the top layer into account with the loss each time.

Images