CN116386111A - Face recognition-oriented patch attack countermeasure method - Google Patents

Abstract

The invention relates to a face recognition-oriented patch attack countermeasure method, and belongs to the field of deep learning. Compared with the common counterpatch method, the countersample attack method based on the countermask patch provided by the invention provides the mask shape more suitable for the face recognition model and provides the hidden generation method more suitable for the application scene, so that the attack robustness of the method is improved, and the possibility of successful attack in the real world is increased. The main innovation of the invention is to design a disturbance generation method aiming at a face recognition system, and the related loss function enhances the concealment of patches by aiming at the dependence of the face recognition network on embedded vectors and by the links of design style loss, so that the possibility of attacking a real system is greatly improved, and the consideration of concealment factors is increased on the premise of considering the success rate of the method. In real life, for example, an attacker can impersonate other identities to perform face verification, steal account information of other people after passing, and the like. The attack method provides a hidden related evaluation standard for defense.

Description

A Method of Adversarial Patch Attack for Face Recognition

technical field

The invention belongs to the field of deep learning and relates to a face recognition-oriented anti-patch attack method.

Background technique

The adversarial example attack method refers to adding perturbations imperceptible to the naked eye to the original input image locally or globally to interfere with the input judgment of deep learning networks such as classification or regression. It usually exists in the picture through very small noise, and these noises are specially formulated by using the structural characteristics of the complex iteration of the network, and iterates the input picture by using the gradient, output vector and other characteristics to induce network misjudgment. The existence of adversarial sample attacks has brought great potential threats to the vast majority of identity information verification systems, automatic driving systems, and target detection systems on the market. Researching and proposing corresponding attack methods is crucial to successful defense and strengthening of network belts in practical applications. Comes great value.

At present, the methods of adversarial examples can be divided into local attack methods and global attack methods. The global attack method is to cover the input image with a layer of generated adversarial noise, thereby turning the original input image into an adversarial example. This approach is less relevant in real life as it is difficult for an attacker to capture and modify the relevant images before they are fed into the system. The local perturbation attack method is easier to migrate to the physical world. It tampers with the input image by hiding a small adversarial patch in a local area of the input image, thereby inducing the network to make wrong judgments. Perturbations in the form of such patches are more conducive to migration to the real world, which poses a great threat to real-world applications. Since these adversarial sample methods do not need to operate the network and training sets, but only tamper with the test samples, the difficulty of implementation is relatively low. Only by studying these attack methods can it be more effective to add corresponding defenses to the application and improve the robustness of the system. .

Existing patch-based attack methods are generated by perturbation on the network gradient. Since the deep learning network infers the final result through continuous iteration and convolution, the output of each layer will be used as the input of the next layer. Therefore, by designing the loss function in the opposite direction of the gradient of the original correct category, it can effectively mislead the judgment of each layer of the network on the correct category, so that the cumulative final judgment is wrong. Most of the existing methods use this idea to design the loss function of the iterative patch and ignore the consideration of the concealment of the generated patch. The concealment is also one of the important criteria for considering the effect of a patch.

Contents of the invention

In view of this, the object of the present invention is to provide a face recognition-oriented anti-patch attack method.

To achieve the above object, the present invention provides the following technical solutions:

A face recognition-oriented anti-patch attack method, the method comprises the following steps:

S1: Select a random patch for preprocessing, cut out the shape of the mask and stitch it with the original input image, and simulate the image of the mask on the face as the initial input;

S2: The generated picture and the original image are handed over to the network for feature extraction, and the mapping vector outputs of the two pictures are respectively obtained, and the feature values are recorded;

S3: Bring the output vector of the anti-patch synthetic image and the output vector of the real class into the loss function calculation, update the patch pixels through the gradient direction and generate a new anti-patch;

S4: Iterate, repeat the first step to reach the set iteration threshold, and output the final confrontation patch.

Optionally, in S4, the iterative method for iteration is to repeat the training m times for each picture x continuously, and reuse the gradient of the previous step when calculating the disturbance r. In order to ensure the speed and enhance the concealment, the overall epoch will be divided by m; where the update formula of r is:

r _t+1 ＝r _t +∈·sign(g)

的梯度写为：Where sign(g) represents the gradient direction, ∈ is the preset hyperparameter to control the gradient direction to size; where the input x, label y is for the loss function

The gradient of is written as:

The specific iterative step algorithm is as follows:

第四步通过随机梯度下降方法更新模型参数θ，更新公式为：/>

通过更新后的θ对输入x更新梯度，从而得到更新对抗补丁的梯度方向/>

最终在原始补丁上累加上梯度方向单位值，即p₀←p₀+∈·sign(g_adv)；重复第四步m次直到得到梯度下降结束；For the original input sample x and the corresponding label y, select the book patch p ₀ for training; in the N/m iteration process, the patch is updated in each round; in each round, the first step is to image x Cover the patch to get a brand new face picture, and then simulate wearing a mask; the second step calculates the loss value through the loss function L(x ₀ , y, θ); the third step calculates the gradient corresponding to the loss function

The fourth step is to update the model parameter θ through the stochastic gradient descent method, and the update formula is: />

Update the gradient of the input x through the updated θ, so as to obtain the gradient direction of the updated anti-patch />

Finally, the gradient direction unit value is accumulated on the original patch, that is, p ₀ ←p ₀ +∈·sign(g _adv ); repeat the fourth step m times until the end of the gradient descent;

The overall loss function is:

Where P _adv represents the generated confrontation patch, P ₀ represents the initial patch that has not been iterated, here we choose random noise, x represents the input face image, and x _t represents the target object;

The similarity loss function is written as:

l _tsim (P _adv , x, x _t ) = cos(em(T(P _adv , x)), em(x _t ))

Among them, em(...) represents the mapping vector (Embedding) that inputs the image to the output of the face recognition network, T(...) represents the transformation operation that will transfer the patch to the face, and cos(...) is cosine distance between vectors; for non-directed attacks, the similarity loss function is of the form:

l _utsim (P _adv , x, x _t )=-cos(em(T(P _adv , x), em(x))

The network misjudgment is induced by minimizing the cosine distance between the generated mapping vector and the target vector; the style loss is expressed as the difference of the Graham matrix between the generated patch and the original patch:

Among them, G(...) is a Graham matrix, which is used to represent a specific graph structure and a matrix for calculating correlation coefficients, describing the relationship between nodes in the graph and the structure of an internal network layer; the Graham matrix is used to maintain The patch comes from the original style; the original Gram matrix needs to iterate from the bottom layer to the top layer of the network, and each time the input of the highest layer is brought into the loss to participate in the iteration together.

The beneficial effect of the present invention is that: compared with the general adversarial patch method, the adversarial sample attack method based on the adversarial mask patch proposed by the present invention proposes a mask shape that is more suitable for the face recognition model and proposes a concealment generation that is more suitable for the application scene method, which improves the attack robustness of such methods and increases the possibility of successful attacks in the real world. The main innovation of the present invention is to design a method for generating disturbances for the face recognition system. The correlation loss function enhances the concealment of the patch by aiming at the dependence of the face recognition network on the embedding vector and through the link of design style loss, thereby greatly improving the attack Possibility of real systems. In real life, for example, attackers can pretend to be other identities for face verification, and then steal other people's account information after passing. This attack method proposes a stealth-related evaluation criterion for the defense.

Compared with adversarial samples of general attack classification types, this method can better use the information of the face recognition network to generate more targeted method patches. For the face recognition algorithm, the inter-class distance is large but the intra-class distance is small For the attack data, the loss function that is more suitable for the face recognition model can better complete the attack in reality. With better attack effects, it can bring new breakthroughs in areas such as confrontation defense and confrontation training.

Compared with the classification attack patch of the related face method, the new concealment method of this method can be better patched, so that it can more effectively target the face recognition system in the real world. With the assistance of style loss and smoothing loss, the target feature information can be softly hidden into the surrounding pixel information, so that the effect is more natural and concealed.

Other advantages, objects and features of the present invention will be set forth in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the investigation and research below, or can be obtained from Taught in the practice of the present invention. The objects and other advantages of the invention may be realized and attained by the following specification.

Description of drawings

In order to make the purpose of the present invention, technical solutions and advantages clearer, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

Figure 1 is the overall flow chart of the single-step iterative confrontation training method based on the enhancement of disturbance diversity;

Figure 2 is a non-directional attack against mask similarity experiment;

Figure 3 shows the similarity experiment of directed attacks against masks;

Figure 4 is a comparison test of the mobility between the models.

Detailed ways

Embodiments of the present invention are described below through specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific implementation modes, and various modifications or changes can be made to the details in this specification based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the diagrams provided in the following embodiments are only schematically illustrating the basic concept of the present invention, and the following embodiments and the features in the embodiments can be combined with each other in the case of no conflict.

Wherein, the accompanying drawings are for illustrative purposes only, and represent only schematic diagrams, rather than physical drawings, and should not be construed as limiting the present invention; in order to better illustrate the embodiments of the present invention, some parts of the accompanying drawings may be omitted, Enlargement or reduction does not represent the size of the actual product; for those skilled in the art, it is understandable that certain known structures and their descriptions in the drawings may be omitted.

In the drawings of the embodiments of the present invention, the same or similar symbols correspond to the same or similar components; , "front", "rear" and other indicated orientations or positional relationships are based on the orientations or positional relationships shown in the drawings, which are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the referred devices or elements must It has a specific orientation, is constructed and operated in a specific orientation, so the terms describing the positional relationship in the drawings are for illustrative purposes only, and should not be construed as limiting the present invention. For those of ordinary skill in the art, the understanding of the specific meaning of the above terms.

An adversarial sample attack method based on an anti-mask patch, characterized in that it comprises the following steps:

Step S1: Select a random patch for preprocessing, cut out the shape of the mask and stitch it with the original input image, and use the image of the simulated mask on the face as the initial input.

Step S2: The generated picture and the original image are sent to the network for feature extraction, and the mapping vector outputs of the two pictures are respectively obtained, and the feature values are recorded.

Step S3: Bring the output vector of the anti-patch synthetic image and the output vector of the real class into the loss function calculation, update the patch pixels through the gradient direction and generate a new anti-patch.

Step S4: Repeat the first step to reach the set iteration threshold, and output the final confrontation patch.

Further, the iterative method is to repeat the training m times continuously for each image x, and reuse the gradient of the previous step when calculating the disturbance r. In order to ensure the speed and enhance the concealment, the overall epoch will be divided by m. The update formula of r is:

r _{t + 1} = r _t + ∈ · sign(g),

的梯度可以写为Where sign(g) represents the gradient direction, and ∈ is a preset hyperparameter to control the gradient direction to size. where the input x, the label y for the loss function

The gradient of can be written as

This iterative method can effectively prevent the patch from over-satisfying the gradient direction, which will cause the image to be too similar to the target feature, thereby reducing the concealment of the attack method.

The specific iterative step algorithm is as follows:

The overall loss function can be written as

Where P _adv represents the generated adversarial patch, P ₀ represents the initial patch without iteration, where random noise is selected, x represents the input face image, and x _t represents the target object.

The similarity loss function can be written as

l _utsim (P _adv , x, x _t ) = cos(em(T(P _adv , x), em(x _t ))

Among them, em(...) represents the mapping vector (Embedding) that inputs the image to the output of the face recognition network, T(...) represents the transformation operation that will transfer the patch to the face, and cos(...) is Cosine distance between vectors. For non-directed attacks, the form of the similarity loss function is:

l _utsim (P _adv , x, x _t )=-cos(em(T(P _adv , x)), em(x))

The network misjudgment is induced by minimizing the cosine distance between the generated mapping vector and the target vector. The style loss can be expressed as the difference of the Graham matrix between the generated patch and the original patch:

Among them, G(...) is the Gram matrix (Gram matrix), which is a matrix used to represent a specific graph structure and calculate the correlation coefficient. It can describe the relationship between nodes in the graph and is used to describe An intranet layer structure. Here the Gram matrix is used to keep the patch from the original style, thus enhancing the concealment of the method. The original Gram matrix needs to iterate from the bottom layer to the top layer of the network, and since this method itself needs to be repeatedly input to the network for patch update, it only needs to bring the input of the highest layer into the loss and participate in the iteration together.

Embodiment one

As shown in FIG. 1 , it is an overall flow chart of the method of adversarial sample attack based on the anti-mask patch of the present invention. In order to verify the effectiveness of the method, we designed experiments related to concealment and similarity testing.

According to the needs of relevant experiments, the present invention proposes relevant evaluation indexes in total:

Cosine similarity: It is a very important metric in face recognition, and the identity is verified by measuring the cosine clamp between the feature vectors in two face images. In the experiment in this paper, the facial features of the face are extracted, and the cosine distance is calculated to measure the reliability of generating adversarial samples. This evaluation method is adopted in both the physical world and digital world experiments in this paper to verify the attack capability of masks.

Attack success rate: an index to measure the success of the overall method. In the physical world experiment, the recognition rate cannot be guaranteed, and there are many invalid screenshots in a frame of video, so it is necessary to conduct batch experiments on multiple groups of pictures , measure whether the overall system can be successfully attacked by counting the ratio of the number of pictures whose similarity passes a specific threshold to the total number of pictures in the statistical batch. For targeted attacks, the expression for the success rate can be written as:

代表攻击目标图像。where I represents the test sample image,

Represents an attack target image.

Peak Signal-to-Noise Ratio (Peak Signal-to-Noise Ratio): Since the digital world method is committed to ensuring the effectiveness of patch attacks on the premise of improving the concealment of samples, measuring concealment is still a very important indicator. In this paper, we compare the clean image X and the image Y with anti-patch noise by peak signal-to-noise ratio. First, define the mean square error of two input mxn images as:

The expression for the peak signal-to-noise ratio given on the basis is:

是图像中可能的最大像素值，这里按照每个像素都由8位的二进制表示，最大像素取值为255。对于彩色图像，需要计算RGB三个通道分别的PSNR值之后取平均值得出最终的信噪比。in

Is the maximum possible pixel value in the image, where each pixel is represented by 8-bit binary, and the maximum pixel value is 255. For color images, it is necessary to calculate the PSNR values of the three channels of RGB and then take the average value to obtain the final signal-to-noise ratio.

method concealment experiment

For the concealment experiment, we choose the pre-training model of the ResNet-18 backbone network to use the MS-Celeb-1M face dataset. The parameter settings during the entire training process are respectively batch size (batch size) 64, and the learning rate is initialized as 0.01, the number of iterations and the number of epoch rounds are set to 100. All experiments in this paper are run on NVIDIA Tesla V100 server, the main deep learning framework is Pytorch 1.7, and the image processing framework is OpenCV 4.5.2. After iterative generation of confrontation masks for concealment testing, we conducted three experiments on three different random number seeds, and took the average of the three experimental results as the final result. The data index results are shown in Table 1.

It can be seen from Table 1 that the patch can still maintain a certain concealment effect after a sufficient number of iterations. For the patch-type adversarial sample method, it is necessary to try to make the patch contain characteristic information that misleads the network so as to be easily exposed. The test in this experiment is based on the random disturbance of the initial words for comparison, and the hidden loss of this method is used to hide the characteristics of the iterative generation. . The first column in Figure 3 is the face image wearing the anti-patch mask, the second column is the face image wearing the initial random perturbation patch, and the third column is the original image in the database. In Table 1, the first column is the PSRN score of the original image after wearing the confrontation mask, and the second column is the PSRN score of the original image and the random mask. The SNR score of wearing a blue mask is about 20, to simulate the human eye Observe the general difference between face pairs and whether they wear masks or not.

method similarity experiment

Through the above experimental analysis, this paper proposes a relatively successful anti-patch method in terms of concealment. On the premise that the sample has strong concealment, compare the similarity score, conduct experiments on the first thousand classes of CASIA-WebFace, analyze the original sample, random perturbation sample and anti-mask patch sample method and compare the cosine similarity with the real class , get the results shown in Figure 2 and Figure 3.

The abscissas in Figure 2 and Figure 3 represent the mask types, from left to right there are No. 0 no mask, No. 1 confrontation mask, No. 2 random perturbation mask, and No. 3 white mask. The vertical axis represents the cosine similarity score obtained by testing the samples in the network. Among them, the green points are outliers, which represent the number of sample result values that deviate from most sample distributions. From the experimental results, it can be seen that the anti-mask and random perturbation mask have a better attack effect, and the effect of the directional attack method is better than that of random perturbation. Because in non-directional attacks, the types of attack targets are more complex and diverse, random noise is more likely to hit some features and cause interference, but in directional attacks, the target domain required by the experiment is limited and the success rate is low. The non-directional attack method in this paper can reduce the accuracy of the face recognition model to the interval [0.2, 0.4], and the attack effect is significantly reduced compared with the original similarity above 0.5; the non-directional attack can reduce the original False [0.2, 0.3] similarity values induced above 0.5.

Migration test experiments were carried out under three models of ArcFace, CosFace and MegFace under all backbone networks. The backbone network generated by masks is based on ResNet50. The experimental results under different face recognition models of different backbone networks are shown in Figure 4, where x The coordinates represent the type of attack patch mask, from left to right are confrontation mask, random perturbation mask, blue mask and the original picture without mask, the y coordinate represents the network model category, from bottom to top are ArcFace-ResNet18, CosFace-ResNet18, ArcFace-ResNet34, CosFace-ResNet34, ArcFace-ResNet50, CosFace-ResNet50, ArcFace-ResNet101, CosFace-ResNet101, MegFace-ResNet101, the z coordinate represents the similarity score. From the experimental results, it can be seen that the adversarial patches trained on the ResNet50-based backbone network can show effective attack capabilities between models. In particular, it can be found that when the depth of the ResNet network model is deeper, there is a slightly better attack effect, that is, for each mask type, there is a similarity score that decreases as the depth increases. This situation is also common in attack methods. After deeper network iterations, the errors obtained each time will deepen as the number of layers is updated. In particular, when the backbone network of ResNet50 is used, the attack belongs to the white-box attack method, and the performance of the attack effect is more prominent. When the method attacks other types of models of other backbone networks, the attack method belongs to the black-box method. For the attack results Also has a good score. Since the attack loss function of this method is based on the cosine distance between network output vectors, this method is mostly a common processing method of face recognition models, so there is a certain degree of migration between methods.

Table 1 Covert PSNR index score

The above experiments and analysis of related results verify the effectiveness of the adversarial training method proposed in the present invention.

Finally, it is noted that the above embodiments are only used to illustrate the technical solutions of the present invention without limitation. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be carried out Modifications or equivalent replacements, without departing from the spirit and scope of the technical solution, should be included in the scope of the claims of the present invention.

Claims (2)

1. A face recognition-oriented anti-patch attack method, characterized in that: the method may further comprise the steps: S1: Select a random patch for preprocessing, cut out the shape of the mask and stitch it with the original input image, and simulate the image of the mask on the face as the initial input; S2: The generated picture and the original image are handed over to the network for feature extraction, and the mapping vector outputs of the two pictures are respectively obtained, and the feature values are recorded; S3: Bring the output vector of the anti-patch synthetic image and the output vector of the real class into the loss function calculation, update the patch pixels through the gradient direction and generate a new anti-patch; S4: Iterate, repeat the first step to reach the set iteration threshold, and output the final confrontation patch. 2. a kind of face recognition-oriented anti-patch attack method according to claim 1, is characterized in that: in described S4, the iterative method that carries out iteration is to repeat m times training continuously to each picture x, calculates disturbance r When reusing the gradient of the previous step, in order to ensure the speed and enhance the concealment, the overall epoch will be divided by m; where the update formula of r is: r _t+1 ＝r _t +∈·sign(g) Where sign(g) represents the gradient direction, ∈ is the preset hyperparameter to control the gradient direction to size; where the input x, label y is for the loss function

The gradient of is written as:

The specific iterative step algorithm is as follows: For the original input sample x and the corresponding label y, select the book patch p ₀ for training; in the N/m iteration process, the patch is updated in each round; in each round, the first step is to image x Cover the patch to get a brand new face picture, and then simulate wearing a mask; the second step calculates the loss value through the loss function L(x ₀ , y, θ); the third step calculates the gradient corresponding to the loss function

The fourth step is to update the model parameter θ through the stochastic gradient descent method, and the update formula is:

Update the gradient of the input x through the updated θ, so as to obtain the gradient direction of the updated anti-patch />

Finally, the gradient direction unit value is accumulated on the original patch, that is, p ₀ ←p ₀ +∈·sign(g _adv ); repeat the fourth step m times until the end of the gradient descent; The overall loss function is:

Where P _adv represents the generated confrontation patch, P ₀ represents the initial patch that has not been iterated, here we choose random noise, x represents the input face image, and x _t represents the target object; The similarity loss function is written as: l _tsim (P _adv , x, x _t ) = cos(em(T(P _adv , x)), em(x _t )) Among them, em(...) represents the mapping vector (Embedding) that inputs the image to the output of the face recognition network, T(...) represents the transformation operation that will transfer the patch to the face, and cos(...) is cosine distance between vectors; for non-directed attacks, the similarity loss function is of the form: l _utsim (P _adv , x, x _t )=-cos(em(T(P _adv , x)), em(x)) The network misjudgment is induced by minimizing the cosine distance between the generated mapping vector and the target vector; the style loss is expressed as the difference of the Graham matrix between the generated patch and the original patch:

Among them, G(...) is a Graham matrix, which is used to represent a specific graph structure and a matrix for calculating correlation coefficients, describing the relationship between nodes in the graph and the structure of an internal network layer; the Graham matrix is used to maintain The patch comes from the original style; the original Gram matrix needs to iterate from the bottom layer to the top layer of the network, and each time the input of the highest layer is brought into the loss to participate in the iteration together.

Images