CN112329894A - Countermeasure sample generation method and device based on proxy model and computing equipment - Google Patents
Countermeasure sample generation method and device based on proxy model and computing equipment Download PDFInfo
- Publication number
- CN112329894A CN112329894A CN202110000722.7A CN202110000722A CN112329894A CN 112329894 A CN112329894 A CN 112329894A CN 202110000722 A CN202110000722 A CN 202110000722A CN 112329894 A CN112329894 A CN 112329894A
- Authority
- CN
- China
- Prior art keywords
- sample
- proxy model
- convolution kernel
- iteration
- convolution
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
- G06N20/20—Ensemble learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Artificial Intelligence (AREA)
- Software Systems (AREA)
- Evolutionary Computation (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Biophysics (AREA)
- Molecular Biology (AREA)
- General Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Biomedical Technology (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Medical Informatics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention provides a countermeasure sample generation method, a countermeasure sample generation device and computing equipment based on a proxy model. The method comprises the following steps: providing an agent model, an original sample and an iteration-based confrontation sample generation algorithm and iterating to generate a confrontation sample until a preset termination condition is reached; in each iteration round: acquiring the weight and gradient of each parameter of each convolution kernel of the agent model when the confrontation sample is generated in the previous iteration; calculating the importance score of each convolution kernel according to the weight and the gradient of each parameter of each convolution kernel; cutting off partial convolution kernels of the proxy model according to a preset rule and importance scores of all convolution kernels of the proxy model; updating the proxy model according to each convolution kernel reserved after cutting off part of the convolution kernels; and taking the confrontation sample generated when the preset termination condition is reached as a final confrontation sample. The method has stronger migration performance and higher success rate of black box attack.
Description
Technical Field
The embodiment of the invention relates to the technical field of neural networks, in particular to a countermeasure sample generation method, a countermeasure sample generation device and computing equipment based on a proxy model.
Background
This section is intended to provide a background or context to the embodiments of the invention that are recited in the claims. The description herein is not admitted to be prior art by inclusion in this section.
Image recognition, an important task in computer vision, has also been greatly developed due to the drive of deep neural networks. And the image recognition system has a plurality of applications in the real scenes of finance/payment, public transportation, criminal recognition and the like. Although image recognition systems have been used with great success and practice, they have not fully ensured adequate security.
In recent years, deep learning has achieved breakthrough in the fields of images, speech, natural language, and the like. However, for some deep neural network models which can achieve high accurate recognition rate, the deep neural network models are easy to attack in the fighting environment. In the countermeasure environment, the deep neural network model is input with some countermeasure samples (e.g., pictures or voice information) based on normal sample malicious constructions, and under the attack of the countermeasure samples, the deep neural network model can make wrong predictions. Therefore, the attack on the deep neural network by adopting the countermeasure sample can detect the potential vulnerability of the deep neural network model, and then the vulnerability can be used for evaluating and improving the safety of the deep neural network model.
Disclosure of Invention
In this context, embodiments of the present invention are intended to provide a method, apparatus, medium, and computing device for generating confrontational samples based on a proxy model.
In a first aspect of embodiments of the present invention, there is provided a countermeasure sample generation method based on a proxy model, including:
providing a proxy model, original samples and an iteration-based countermeasure sample generation algorithm;
iteratively generating a countermeasure sample based on the agent model, the original sample and a countermeasure sample generation algorithm until a preset termination condition is reached;
in each iteration round:
acquiring the weight and gradient of each parameter of each convolution kernel of the agent model when the confrontation sample is generated in the previous iteration;
calculating an importance score of each convolution kernel according to the weight and the gradient of each parameter of each convolution kernel, wherein the importance score is used for representing the influence degree of the countermeasure sample generated by the corresponding convolution kernel of the agent model in the corresponding round;
cutting off partial convolution kernels of the proxy model according to a preset rule and importance scores of all convolution kernels of the proxy model;
updating the proxy model according to each convolution kernel reserved after cutting off part of the convolution kernels;
and taking the confrontation sample generated when the preset termination condition is reached as a final confrontation sample.
In an embodiment of the present invention, the importance score of a convolution kernel is an absolute value of a product of a gradient vector of the convolution kernel after being inverted and a weight.
In an embodiment of this embodiment, the preset rule includes pruning a convolution kernel with an importance score lower than a preset threshold.
In an embodiment of this embodiment, the preset rule includes performing convolution kernel pruning at a preset pruning rate.
In an embodiment of the present invention, pruning a part of convolution kernels of the proxy model according to a preset rule and an importance score of each convolution kernel of the proxy model includes:
sorting the convolution kernels in order from high to low based on the importance scores of the convolution kernels;
and pruning the sequenced partial convolution kernels according to the pruning rate.
In one example of the present embodiment, the convolution kernels are pruned by setting the value of each parameter in one of the convolution kernels to 0.
In one embodiment of this embodiment, the iterative-based confrontation sample generation algorithm comprises a momentum iterative confrontation sample generation algorithm;
in the first placeiIn round iterations, confrontation samples are generated in the following manner
：
Wherein the content of the first and second substances,
is shown asiThe momentum gained by the update of the wheel,
the velocity of the momentum decay is represented as,
loss result calculated by loss function representing proxy model
Challenge samples generated with respect to round i-1
The gradient of (a) of (b) is,
the norm of L1 is shown,yto represent
Corresponding original sampleXIn the category of (a) to (b),
representing antagonistic samples in a to-be-modified
Projection to distance original sampleXNot exceeding ϵ, where α is a hyperparameter.
In one example of this embodiment, in the second embodimentiIn the iteration round, the confrontation sample generated in the previous round by a certain convolution check is calculated in the following way
Importance score of (a):
where d denotes a differential operation,wrepresents the weight of the convolution kernel and T represents the vector transpose.
In a second aspect of the embodiments of the present invention, there is provided a countermeasure sample generation apparatus based on a proxy model, including:
a preprocessing module configured to provide a proxy model, raw samples, and an iteration-based confrontation sample generation algorithm;
the iteration module is configured to iteratively generate a confrontation sample based on the agent model, the original sample and the confrontation sample generation algorithm until a preset termination condition is reached;
the obtaining unit is configured to obtain the weight and gradient of each parameter of each convolution kernel of the proxy model when the countermeasure sample is generated in the previous iteration in each iteration round;
a calculating unit configured to calculate, in each iteration turn, an importance score of each convolution kernel according to the weight and the gradient of each parameter of each convolution kernel, the importance score being used to represent the degree of influence of the countermeasure samples generated by the corresponding convolution kernel of the proxy model in the corresponding turn;
a pruning unit configured to prune, in each iteration round, a part of convolution kernels of the proxy model according to a preset rule and importance scores of the respective convolution kernels of the proxy model;
an updating unit configured to update the proxy model according to each convolution kernel retained after a part of the convolution kernels is cut out in each iteration round;
and the determining unit is configured to take the confrontation sample generated when the preset termination condition is reached as a final confrontation sample in each iteration turn.
In a third aspect of embodiments of the present invention, a storage medium is provided, which stores a computer program that, when executed by a processor, may implement the countermeasure sample generation method based on a proxy model.
In a fourth aspect of embodiments of the present invention, there is provided a computing device comprising: a processor; a memory for storing the processor-executable instructions; the processor is used for executing the countermeasure sample generation method based on the agent model.
According to the method and the device for generating the confrontation sample based on the proxy model, the original sample and an iteration-based confrontation sample generation algorithm are provided; iteratively generating a countermeasure sample based on the agent model, the original sample and a countermeasure sample generation algorithm until a preset termination condition is reached; in each iteration round: acquiring the weight and gradient of each parameter of each convolution kernel of the agent model when the confrontation sample is generated in the previous iteration; calculating an importance score of each parameter according to the weight and the gradient of each parameter of each convolution kernel, wherein the importance score is used for representing the influence degree of the corresponding parameter of the proxy model in the corresponding round on the generated countermeasure sample; cutting off partial convolution kernels of the proxy model according to a preset rule and importance scores of all convolution kernels of the proxy model; updating the proxy model according to each cut convolution kernel; and taking the confrontation sample generated when the preset termination condition is reached as a final confrontation sample.
The countermeasure sample generated according to the technical scheme of the application has stronger migration performance and higher success rate of black box attack compared with the countermeasure sample generated based on primary physiological model iteration.
Compared with the prior art, the invention mainly has the following beneficial effects:
the invention belongs to a black box confrontation sample generation method based on migration, so that a victim model does not need to be acquired and a large amount of access to the victim model is not needed.
Secondly, the invention can be combined with any iterative-based attack method (such as a TIM attack method) to improve the migration performance.
Thirdly, the countermeasure sample finally generated by the method is based on the proxy model after the convolution kernels are cut off for multiple times, and the convolution kernels reserved in the proxy model are more important to the image recognition result compared with the cut-off convolution kernels, namely more attention is paid to the important characteristics of the image, so that the countermeasure sample generated by the method is beneficial to the abnormal detection or interpretability of the image.
Drawings
The above and other objects, features and advantages of exemplary embodiments of the present invention will become readily apparent from the following detailed description read in conjunction with the accompanying drawings. Several embodiments of the invention are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which:
FIG. 1 is a flow diagram schematically illustrating a countermeasure sample generation method based on a proxy model according to an embodiment of the invention;
FIG. 2 schematically illustrates a flow diagram for iteratively generating a challenge sample of the embodiment shown in FIG. 1;
FIG. 3 is a block diagram of a countermeasure sample generation apparatus based on a proxy model according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention;
fig. 5 is an illustration of a computing device provided by an embodiment of the invention.
In the drawings, the same or corresponding reference numerals indicate the same or corresponding parts.
Detailed Description
The principles and spirit of the present invention will be described with reference to a number of exemplary embodiments. It is understood that these embodiments are given solely for the purpose of enabling those skilled in the art to better understand and to practice the invention, and are not intended to limit the scope of the invention in any way. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
As will be appreciated by one skilled in the art, embodiments of the present invention may be embodied as a system, apparatus, device, method, or computer program product. Accordingly, the present disclosure may be embodied in the form of: entirely hardware, entirely software (including firmware, resident software, micro-code, etc.), or a combination of hardware and software.
According to the embodiment of the invention, a countermeasure sample generation method, a countermeasure sample generation device, a countermeasure sample generation medium and computing equipment based on a proxy model are provided.
Moreover, any number of elements in the drawings are by way of example and not by way of limitation, and any nomenclature is used solely for differentiation and not by way of limitation.
The principles and spirit of the present invention are explained in detail below with reference to several representative embodiments of the invention.
Summary of The Invention
The inventor finds that in the method of attacking the neural network discrimination/classification/prediction model, different countersample generation methods are mainly different in an optimization algorithm and a disturbance form. For common white-box-based attack methods, score-based black-box attack methods, decision-based black-box attack methods and migration-based black-box attack methods, the perturbation form is usually that a noise is linearly superimposed on a normal sample. They differ only in the use of different optimization procedures. Each of these methods has some disadvantages, such as:
first, a white-box-based attack method needs to acquire a victim model, and in reality, the model is often not easy to acquire.
Secondly, the black box attack method based on the scores and the black box attack method based on the decision usually need to visit the victim model for many times, and the attack efficiency is reduced.
And thirdly, in the black box attack method based on migration, the similarity between the black box model and the damaged model has a large influence on the success rate of attack.
The discrimination/classification/prediction model constructed based on the neural network is often based on similar feature extraction and feature processing modes, for example, the features of a sample are extracted, and then corresponding discrimination/classification/prediction operation is performed on the extracted features.
However, in the image discrimination/classification/prediction model constructed based on the neural network, a large number of convolutional neural networks with different convolution kernels often exist, and the final output results of the convolutional kernel models play different influence roles, so that the inventor conceives a countermeasure sample generation method adopting a network pruning technology, and when a countermeasure sample is generated in each iteration step, an unimportant convolution kernel of the agent model relative to the original sample is searched and pruned, thereby obtaining an agent model which focuses more on the important characteristics of the sample. The countermeasure sample generated on the proxy model has stronger migration performance and higher success rate of black box attack compared with the original model.
Exemplary method
A countermeasure sample generation method based on a proxy model according to an exemplary embodiment of the present invention is described below with reference to fig. 1 and 2. It should be noted that the above application scenarios are merely illustrated for the convenience of understanding the spirit and principles of the present invention, and the embodiments of the present invention are not limited in this respect. Rather, embodiments of the present invention may be applied to any scenario where applicable.
The embodiment of the invention provides a countermeasure sample generation method based on a proxy model, which comprises the following steps:
step S110, providing a proxy model, an original sample and a confrontation sample generation algorithm based on iteration;
step S120, iteratively generating a confrontation sample based on the agent model, the original sample and the confrontation sample generation algorithm until a preset termination condition is reached;
in each iteration round:
step S121, acquiring the weight and gradient of each parameter of each convolution kernel of the agent model when the countermeasure sample is generated in the previous iteration;
step S122, calculating an importance score of each convolution kernel according to the weight and the gradient of each parameter of each convolution kernel, wherein the importance score is used for expressing the influence degree of the corresponding convolution kernel of the proxy model in the corresponding round on the generated countermeasure sample;
step S123, cutting off partial convolution kernels of the proxy model according to a preset rule and importance scores of all convolution kernels of the proxy model;
step S124, updating the proxy model according to each convolution kernel reserved after cutting off part of the convolution kernels;
in step S125, the confrontation sample generated when the preset termination condition is reached is taken as the final confrontation sample.
It can be understood that the countermeasure sample generation method of the present invention generates the countermeasure sample based on a proxy model iteration, and the unimportant convolution kernel of the proxy model with respect to the original sample is found and cut off when generating the countermeasure sample in each iteration, so as to obtain a proxy model with more attention to the important features of the sample. The countermeasure samples generated on the proxy model after the unimportant convolution kernels are cut off have stronger migration performance and higher success rate of black box attack compared with the countermeasure samples generated on the primary physical model.
The method is suitable for generation of countermeasure samples of various types of data, the original samples include but are not limited to picture samples, voice samples and text samples, and the corresponding proxy models are image processing models, voice processing models and text processing models which are constructed on the basis of a neural network and correspond to the original samples.
The following describes how to perform the generation of photo-like confrontation samples based on the proxy model with reference to the accompanying drawings:
firstly, executing step S110, providing a proxy model, an original sample and an iteration-based confrontation sample generation algorithm;
in this embodiment, an original sample and a proxy model are first provided, where the original sample may be one or more image samples or image sample sets prepared in advance, and when the original sample needs to be obtained from an image sample set, the original sample may be obtained by a random sampling manner or other preset rules, which is not limited in this embodiment, the proxy model is a white-box neural network model that can be distinguished, classified, and predicted based on the original sample, that is, we can obtain details of a neural network structure and parameters of the proxy model, and the proxy model has the same or similar purpose as a model intended to be attacked (which may also be referred to as a victim model), for example, the victim model is used for image classification, and the proxy model is also preferably used for image classification, and in this embodiment, an attack image classification model is taken as an example for explanation, specifically, any one of VGG16, VGG19, inclusion v3, Xception, MobileNet, AlexNet, LeNet, ZF _ Net, ResNet18, ResNet34, ResNet50, ResNet _101, and ResNet _152 may be used.
The iteration-based countermeasure sample generation algorithm refers to a Method for performing Iterative optimization according to an output result of a countermeasure sample generated in the previous round after being input into an agent model so as to gradually improve the success rate of subsequently generated countermeasure sample attacks, and when a countermeasure sample is constructed based on an image sample and an image classification model, the iteration-based countermeasure sample generation algorithm may be a Basic Iterative Method: a confrontation sample generation method based on gradient descent; or a Momentum Iterative Fast Gradient Signal Method, an optimization-based countermeasure sample generation Method using Momentum; or in other ways, this embodiment is not limited to this.
In this embodiment, how to generate a picture-class countermeasure sample is described as an example, but this does not mean that the present invention can be applied only to the generation of the picture-class countermeasure sample, and when other classes of countermeasure samples such as voice or text need to be performed, corresponding data processing models and iterative attack modes can be correspondingly adopted according to each step of the present invention.
Next, executing step S120, iteratively generating a countermeasure sample based on the agent model, the original sample and a countermeasure sample generation algorithm until a preset termination condition is reached; specifically, in each iteration round:
step S121, acquiring the weight and gradient of each parameter of each convolution kernel of the agent model when the countermeasure sample is generated in the previous iteration;
in a neural network, it is often seen that the expression for a certain neuron to process input data x is z =wx + b, whereinwIs weight, b is bias term; the weights of the convolution kernels obtained in this step are the weights of the corresponding neuronsw。
Accordingly, the gradient acquisition or calculation of each convolution kernel is well known to those skilled in the art and will not be illustrated in detail in this embodiment.
Step S122, calculating an importance score of each convolution kernel according to the weight and the gradient of each parameter of each convolution kernel, wherein the importance score is used for expressing the influence degree of the corresponding convolution kernel of the proxy model in the corresponding round on the generated countermeasure sample;
in this embodiment, for the obtained weight and gradient of each convolution kernel, any reasonable evaluation index may be used to measure the importance of the convolution kernel, for example, a scale scaling factor of a batch normalization layer is used to evaluate the importance of the convolution kernel, or a greedy strategy is used to find unimportant convolution kernels, or parameters are usedlThe 1 norm serves as the importance score of the convolution kernel.
Considering that each convolution kernel includes a plurality of parameters, the weight of the convolution kernel is a vector, and the gradient of the convolution kernel is also a vector. Therefore, in an embodiment of the present embodiment, the importance score of a certain convolution kernel is an absolute value of a product of the transformed gradient vector of the convolution kernel and the weight, and the vector transformation specifically is to calculate a first-order term of taylor expansion of the loss function to approximate a first-order variation of the parameter w of the loss function.
It should be noted that the weight or gradient of a certain parameter itself may be positive or negative. If the importance score of a convolution kernel is calculated using the weight or gradient of a certain parameter itself directly as a basis, it may be possible to simply assume that a positive weight or gradient is more important than a negative one. In practice, a very negative weight or gradient means that the parameter can have a very large side effect, and its influence on the model prediction result is comparable to a very large positive weight or gradient. The present application measures the importance of the model convolution kernel by absolute value, i.e., the magnitude of its value (and neglecting sign).
Next, step S123 is executed to prune a part of the convolution kernels of the proxy model according to a preset rule and the importance scores of the convolution kernels of the proxy model;
in an embodiment of this embodiment, the preset rule includes pruning a convolution kernel with an importance score lower than a preset threshold.
In another embodiment of this embodiment, the pre-set rule includes performing convolution kernel pruning at a pre-set pruning rate, and in this embodiment, pruning a part of convolution kernels of the proxy model according to the pre-set rule and an importance score of each convolution kernel of the proxy model includes:
sorting the convolution kernels in order from high to low based on the importance scores of the convolution kernels;
and pruning the sequenced partial convolution kernels according to the pruning rate.
For example, there are N convolution kernels in the current proxy model: a1 and a2 · an are respectively and correspondingly calculated to obtain importance scores s1 and s2 · sn, then the convolution kernels are sorted from high to low based on the importance scores of the convolution kernels, and the last 20% of the convolution kernels are cut off according to a pruning rate (namely, the proportion of the convolution kernels needing to be cut off, for example, 20%), specifically, the last 20% of the convolution kernels can be cut off by setting the value of each parameter in a certain convolution kernel to be 0.
Step S124 is executed next, and the proxy model is updated according to each convolution kernel reserved after the partial convolution kernels are cut out;
in step S125, the confrontation sample generated when the preset termination condition is reached is taken as the final confrontation sample.
In this embodiment, the preset termination condition may be that iteration reaches a preset number of times, or that a cut-off of a convolution kernel of the proxy model exceeds a certain proportion, or that an attack success rate of an anti-sample generated by iteration reaches a preset threshold, which is not limited in this embodiment.
In the following, a detailed description is given to a scheme of an embodiment of the present application, where an MI-FGSM attack algorithm is used as the iterative-based countermeasure sample generation algorithm, three networks, namely AlexNet, DenseNet161, and ResNet18, are used as the proxy model and the migration attack model, respectively, an original sample includes 1000 ImageNet pictures, and a pruning rate r is 20%.
First, a victim image is selectedXAnd a proxy modelM. The victim imageXThe image is classified intoy。
Then, a hyper-parameter is chosen, such as a predetermined stop condition: number of iterations of attackNVelocity of momentum decayμAttack amplitude per stepαTotal size of disturbance allowedϵAnd pruning rate r.
Next, the countermeasure sample is initialized
Momentum
0 is and
the full zero tensor of the same dimension.
In the first placeiIn round iterations, confrontation samples are generated in the following manner
：
Wherein the content of the first and second substances,
is shown asiWheel holderThe newly obtained momentum is the momentum of the magnetic field,
the velocity of the momentum decay is represented as,
loss result calculated by loss function representing proxy model
Challenge samples generated with respect to round i-1
The gradient of (a) of (b) is,
the norm of L1 is shown,yto represent
Corresponding original sampleXIn the category of (a) to (b),
representing antagonistic samples in a to-be-modified
Projection to distance original sampleXNot exceeding ϵ, where α is a hyperparameter.
Loss result obtained by calculating loss function of proxy model
About confrontational sample
Gradient of (2)
Cross entropy loss result of neural networkXOf the gradient of (c).
Then, byCalculating the confrontation sample generated in the last round by a certain convolution check
The absolute value of the product of the convolution kernel self weight and the self gradient:
where d denotes a differential operation,wrepresents the weight of the convolution kernel and T represents the vector transpose.
When pruning of unimportant convolution kernels is performed, proxy models are processedMThe individual convolution kernels are sorted by importance scores. Setting n convolution kernels in the current layer, setting each parameter in the nr convolution kernels with the lowest importance scores as 0, and obtaining a temporary modelM i And back.
In the iteration of this round (the first one)iRound robin iteration) based on the temporary model
Generating challenge samples
。
While iterating N rounds, i.e.i=NWhen in use, will
As a final output challenge sample.
Exemplary devices
Having described the method of the exemplary embodiment of the present invention, next, a countermeasure sample generation apparatus based on a proxy model of the exemplary embodiment of the present invention will be described with reference to fig. 3, the apparatus including:
a preprocessing module 310 configured to provide a proxy model, raw samples, and an iteration-based confrontation sample generation algorithm;
an iteration module 320 configured to iteratively generate a countermeasure sample based on the agent model, the original sample, and a countermeasure sample generation algorithm until a preset termination condition is reached;
an obtaining unit 321 configured to obtain, in each iteration round, a weight and a gradient of each parameter of each convolution kernel of the proxy model when the countermeasure sample is generated in the previous iteration round;
a calculating unit 322 configured to calculate, in each iteration round, an importance score of each convolution kernel according to the weight and the gradient of each parameter of each convolution kernel, the importance score being used to represent the degree of influence of the countermeasure sample generated by the corresponding convolution kernel of the proxy model in the corresponding round;
a pruning unit 323 configured to prune, in each iteration round, a part of convolution kernels of the proxy model according to a preset rule and an importance score of each convolution kernel of the proxy model;
an updating unit 324 configured to update the proxy model according to each convolution kernel remaining after cutting off a part of the convolution kernels in each iteration round;
the determining unit 325 is configured to take the confrontation sample generated when the preset termination condition is reached as a final confrontation sample in each iteration turn.
In an embodiment of this embodiment, the preset rule includes pruning a convolution kernel with an importance score lower than a preset threshold.
In an embodiment of this embodiment, the preset rule includes performing convolution kernel pruning at a preset pruning rate.
In an embodiment of the present embodiment, the pruning unit 323 includes:
a sorting subunit configured to sort the respective convolution kernels in order from high to low based on the importance scores of the respective convolution kernels;
and the pruning subunit is configured to prune the sorted partial convolution kernels according to the pruning rate.
In an example of this embodiment, the pruning unit 323 is further configured to prune the convolution kernels by setting the values of the respective parameters within a certain convolution kernel to 0.
In one embodiment of this embodiment, the iterative-based confrontation sample generation algorithm comprises a momentum iterative confrontation sample generation algorithm;
in the first placeiIn round iterations, confrontation samples are generated in the following manner
：
Wherein the content of the first and second substances,
is shown asiThe momentum gained by the update of the wheel,
the velocity of the momentum decay is represented as,
loss result calculated by loss function representing proxy model
Challenge samples generated with respect to round i-1
The gradient of (a) of (b) is,
the norm of L1 is shown,yto represent
Corresponding original sampleXIn the category of (a) to (b),
representing antagonistic samples in a to-be-modified
Projection to distance original sampleXNot exceeding ϵ, where α is a hyperparameter.
In one example of this embodiment, in the second embodimentiIn the iteration round, the confrontation sample generated in the previous round by a certain convolution check is calculated in the following way
Importance score of (a):
where d denotes a differential operation,wrepresents the weight of the convolution kernel and T represents the vector transpose.
Exemplary Medium
Having described the method and apparatus of the exemplary embodiments of this invention, a computer-readable storage medium of the exemplary embodiments of this invention is described with reference to fig. 4, which refers to fig. 4, which illustrates an optical disc 40 having a computer program (i.e., a program product) stored thereon, which when executed by a processor, performs the steps described in the method embodiments, e.g., providing a proxy model, original samples, and an iteration-based challenge sample generation algorithm; iteratively generating a countermeasure sample based on the agent model, the original sample and a countermeasure sample generation algorithm until a preset termination condition is reached; in each iteration round: acquiring the weight and gradient of each parameter of each convolution kernel of the agent model when the confrontation sample is generated in the previous iteration; calculating an importance score of each convolution kernel according to the weight and the gradient of each parameter of each convolution kernel, wherein the importance score is used for representing the influence degree of the countermeasure sample generated by the corresponding convolution kernel of the agent model in the corresponding round; cutting off partial convolution kernels of the proxy model according to a preset rule and importance scores of all convolution kernels of the proxy model; updating the proxy model according to each convolution kernel reserved after cutting off part of the convolution kernels; taking the confrontation sample generated when the preset termination condition is reached as a final confrontation sample; the specific implementation of each step is not repeated here.
It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory, or other optical and magnetic storage media, which are not described in detail herein.
Exemplary computing device
Having described the methods, apparatus and media of exemplary embodiments of the present invention, a computing device for countermeasure sample generation based on a proxy model of an exemplary embodiment of the present invention is next described with reference to FIG. 5.
FIG. 5 illustrates a block diagram of an exemplary computing device 50 suitable for use in implementing embodiments of the present invention, the computing device 50 may be a computer system or server. The computing device 50 shown in FIG. 5 is only one example and should not be taken to limit the scope of use and functionality of embodiments of the present invention.
As shown in fig. 5, components of computing device 50 may include, but are not limited to: one or more processors or processing units 501, a system memory 502, and a bus 503 that couples the various system components (including the system memory 502 and the processing unit 501).
The system memory 502 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM) 5021 and/or cache memory 5022. Computing device 50 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, the ROM5023 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 5, which is commonly referred to as a "hard drive"). Although not shown in FIG. 5, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 503 by one or more data media interfaces. At least one program product may be included in system memory 502 having a set (e.g., at least one) of program modules configured to carry out the functions of embodiments of the invention.
A program/utility 5025 having a set (at least one) of program modules 5024 may be stored in, for example, system memory 502, and such program modules 5024 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment. The program modules 5024 generally perform the functions and/or methodologies of the described embodiments of the invention.
The processing unit 501 executes various functional applications and data processing by executing programs stored in the system memory 502, for example, providing a proxy model, original samples, and an iteration-based countermeasure sample generation algorithm; iteratively generating a countermeasure sample based on the agent model, the original sample and a countermeasure sample generation algorithm until a preset termination condition is reached; in each iteration round: acquiring the weight and gradient of each parameter of each convolution kernel of the agent model when the confrontation sample is generated in the previous iteration; calculating an importance score of each convolution kernel according to the weight and the gradient of each parameter of each convolution kernel, wherein the importance score is used for representing the influence degree of the countermeasure sample generated by the corresponding convolution kernel of the agent model in the corresponding round; cutting off partial convolution kernels of the proxy model according to a preset rule and importance scores of all convolution kernels of the proxy model; updating the proxy model according to each convolution kernel reserved after cutting off part of the convolution kernels; and taking the confrontation sample generated when the preset termination condition is reached as a final confrontation sample. The specific implementation of each step is not repeated here. It should be noted that although several units/modules or sub-units/sub-modules of the countermeasure sample generation apparatus based on the proxy model are mentioned in the above detailed description, such division is merely exemplary and not mandatory. Indeed, the features and functionality of two or more of the units/modules described above may be embodied in one unit/module according to embodiments of the invention. Conversely, the features and functions of one unit/module described above may be further divided into embodiments by a plurality of units/modules.
In the description of the present invention, it should be noted that the terms "first", "second", and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the embodiments provided in the present invention, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one logical division, and there may be other divisions when actually implemented, and for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or units through some communication interfaces, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
Finally, it should be noted that: the above-mentioned embodiments are only specific embodiments of the present invention, which are used for illustrating the technical solutions of the present invention and not for limiting the same, and the protection scope of the present invention is not limited thereto, although the present invention is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art can modify or easily conceive the technical solutions described in the foregoing embodiments or equivalent substitutes for some technical features within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the embodiments of the present invention, and they should be construed as being included therein. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Moreover, while the operations of the method of the invention are depicted in the drawings in a particular order, this does not require or imply that the operations must be performed in this particular order, or that all of the illustrated operations must be performed, to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step execution, and/or one step broken down into multiple step executions.
Claims (10)
1. A countermeasure sample generation method based on a proxy model comprises the following steps:
providing a proxy model, original samples and an iteration-based countermeasure sample generation algorithm;
iteratively generating a countermeasure sample based on the agent model, the original sample and a countermeasure sample generation algorithm until a preset termination condition is reached;
in each iteration round:
acquiring the weight and gradient of each parameter of each convolution kernel of the agent model when the confrontation sample is generated in the previous iteration;
calculating an importance score of each convolution kernel according to the weight and the gradient of each parameter of each convolution kernel, wherein the importance score is used for representing the influence degree of the countermeasure sample generated by the corresponding convolution kernel of the agent model in the corresponding round;
cutting off partial convolution kernels of the proxy model according to a preset rule and importance scores of all convolution kernels of the proxy model;
updating the proxy model according to each convolution kernel reserved after cutting off part of the convolution kernels;
and taking the confrontation sample generated when the preset termination condition is reached as a final confrontation sample.
2. The method of generating resist samples based on a surrogate model as claimed in claim 1, wherein the importance score of a certain convolution kernel is the absolute value of the product of the shifted gradient vector of the convolution kernel and the weight.
3. The method of generating antagonistic samples based on the surrogate model of claim 1, wherein the preset rule comprises pruning convolution kernels with importance scores below a preset threshold.
4. The method of generating a confrontation sample based on a proxy model of claim 1, wherein the predetermined rule includes convolution kernel pruning at a predetermined pruning rate.
5. The method for generating countermeasure samples based on the proxy model according to claim 4, wherein pruning partial convolution kernels of the proxy model according to a preset rule and an importance score of each convolution kernel of the proxy model comprises:
sorting the convolution kernels in order from high to low based on the importance scores of the convolution kernels;
and pruning the sequenced partial convolution kernels according to the pruning rate.
6. The method of generating a confrontation sample based on a proxy model according to any of claims 1 to 5, wherein the convolution kernel is pruned by setting the value of each parameter within a certain convolution kernel to 0.
7. The countermeasure sample generation method based on the proxy model of claim 6, wherein the iterative-based countermeasure sample generation algorithm comprises a momentum iterative countermeasure sample generation algorithm;
in the first placeiIn round iterations, confrontation samples are generated in the following manner
：
Wherein the content of the first and second substances,
is shown asiThe momentum gained by the update of the wheel,
the velocity of the momentum decay is represented as,
loss result calculated by loss function representing proxy model
Challenge samples generated with respect to round i-1
The gradient of (a) of (b) is,
the norm of L1 is shown,yto represent
Corresponding original sampleXIn the category of (a) to (b),
representing antagonistic samples in a to-be-modified
Projection to distance original sampleXNot exceeding ϵ, where α is a hyperparameter.
8. The countermeasure sample generation method based on the proxy model of claim 7, wherein at the second placeiIn the iteration round, the confrontation sample generated in the previous round by a certain convolution check is calculated in the following way
Importance score of (a):
where d denotes a differential operation,wrepresents the weight of the convolution kernel and T represents the vector transpose.
9. A countermeasure sample generation apparatus based on a proxy model, comprising:
a preprocessing module configured to provide a proxy model, raw samples, and an iteration-based confrontation sample generation algorithm;
the iteration module is configured to iteratively generate a confrontation sample based on the agent model, the original sample and the confrontation sample generation algorithm until a preset termination condition is reached; the method comprises the following steps:
the obtaining unit is configured to obtain the weight and gradient of each parameter of each convolution kernel of the proxy model when the countermeasure sample is generated in the previous iteration in each iteration round;
a calculating unit configured to calculate, in each iteration turn, an importance score of each convolution kernel according to the weight and the gradient of each parameter of each convolution kernel, the importance score being used to represent the degree of influence of the countermeasure samples generated by the corresponding convolution kernel of the proxy model in the corresponding turn;
a pruning unit configured to prune, in each iteration round, a part of convolution kernels of the proxy model according to a preset rule and importance scores of the respective convolution kernels of the proxy model;
an updating unit configured to update the proxy model according to each convolution kernel retained after a part of the convolution kernels is cut out in each iteration round;
and the determining unit is configured to take the confrontation sample generated when the preset termination condition is reached as a final confrontation sample in each iteration turn.
10. A computing device, the computing device comprising: a processor; a memory for storing the processor-executable instructions; the processor configured to perform the method of any of the preceding claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110000722.7A CN112329894A (en) | 2021-01-04 | 2021-01-04 | Countermeasure sample generation method and device based on proxy model and computing equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110000722.7A CN112329894A (en) | 2021-01-04 | 2021-01-04 | Countermeasure sample generation method and device based on proxy model and computing equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112329894A true CN112329894A (en) | 2021-02-05 |
Family
ID=74302503
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110000722.7A Withdrawn CN112329894A (en) | 2021-01-04 | 2021-01-04 | Countermeasure sample generation method and device based on proxy model and computing equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112329894A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109523017A (en) * | 2018-11-27 | 2019-03-26 | 广州市百果园信息技术有限公司 | Compression method, device, equipment and the storage medium of deep neural network |
| US20190122113A1 (en) * | 2017-10-19 | 2019-04-25 | International Business Machines Corporation | Pruning Redundant Neurons and Kernels of Deep Convolutional Neural Networks |
| CN110245598A (en) * | 2019-06-06 | 2019-09-17 | 北京瑞莱智慧科技有限公司 | It fights sample generating method, device, medium and calculates equipment |
| CN110942143A (en) * | 2019-12-04 | 2020-03-31 | 卓迎 | Toy detection acceleration method and device based on convolutional neural network |
| CN111429415A (en) * | 2020-03-18 | 2020-07-17 | 东华大学 | Efficient model construction method for product surface defects based on network collaborative pruning |
| CN111914928A (en) * | 2020-07-30 | 2020-11-10 | 南京大学 | Method for defending confrontation sample for image classifier |
-
2021
- 2021-01-04 CN CN202110000722.7A patent/CN112329894A/en not_active Withdrawn
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190122113A1 (en) * | 2017-10-19 | 2019-04-25 | International Business Machines Corporation | Pruning Redundant Neurons and Kernels of Deep Convolutional Neural Networks |
| CN109523017A (en) * | 2018-11-27 | 2019-03-26 | 广州市百果园信息技术有限公司 | Compression method, device, equipment and the storage medium of deep neural network |
| CN110245598A (en) * | 2019-06-06 | 2019-09-17 | 北京瑞莱智慧科技有限公司 | It fights sample generating method, device, medium and calculates equipment |
| CN110942143A (en) * | 2019-12-04 | 2020-03-31 | 卓迎 | Toy detection acceleration method and device based on convolutional neural network |
| CN111429415A (en) * | 2020-03-18 | 2020-07-17 | 东华大学 | Efficient model construction method for product surface defects based on network collaborative pruning |
| CN111914928A (en) * | 2020-07-30 | 2020-11-10 | 南京大学 | Method for defending confrontation sample for image classifier |
Non-Patent Citations (1)
| Title |
|---|
| YIREN ZHAO等: "TO COMPRESS OR NOT TO COMPRESS:UNDERSTANDING THE INTERACTIONS BETWEEN ADVERSARIAL ATTACKS AND NEURAL NETWORK COMPRESSION", 《ARXIV》 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112329930B (en) | Countermeasure sample generation method and device based on proxy model | |
| Fang et al. | Phishing email detection using improved RCNN model with multilevel vectors and attention mechanism | |
| Chawla et al. | Host based intrusion detection system with combined CNN/RNN model | |
| CN108737406B (en) | Method and system for detecting abnormal flow data | |
| CN109302410B (en) | Method and system for detecting abnormal behavior of internal user and computer storage medium | |
| CN111027069B (en) | Malicious software family detection method, storage medium and computing device | |
| AU2020202542A1 (en) | Transforming attributes for training automated modeling systems | |
| CN109902018B (en) | Method for acquiring test case of intelligent driving system | |
| US20200387783A1 (en) | Fast Nearest Neighbor Search for Output Generation of Convolutional Neural Networks | |
| Chen et al. | Learning spatial-temporal varying graphs with applications to climate data analysis | |
| Arafiyah et al. | Classification of Dengue Haemorrhagic Fever (DHF) using SVM, naive bayes and random forest | |
| CN113919497A (en) | Attack and defense method based on feature manipulation for continuous learning ability system | |
| Shoohi et al. | DCGAN for Handling Imbalanced Malaria Dataset based on Over-Sampling Technique and using CNN. | |
| Prima et al. | Using transfer learning for malware classification | |
| CN115051864A (en) | PCA-MF-WNN-based network security situation element extraction method and system | |
| CN112329929B (en) | Countermeasure sample generation method and device based on proxy model | |
| Ravi et al. | Hybrid classification and regression models via particle swarm optimization auto associative neural network based nonlinear PCA | |
| CN114511756A (en) | Attack method and device based on genetic algorithm and computer program product | |
| Finn et al. | Deep learning subgrid-scale parametrisations for short-term forecasting of sea-ice dynamics with a Maxwell elasto-brittle rheology | |
| CN117134958A (en) | Information processing method and system for network technology service | |
| CN111062442B (en) | Method and device for explaining service processing result of service processing model | |
| CN112329931B (en) | Countermeasure sample generation method and device based on proxy model | |
| CN109389400A (en) | The system and method for potential danger equipment are identified during user interacts with bank service | |
| Gogineni et al. | Eye disease detection using yolo and ensembled googlenet | |
| CN112329894A (en) | Countermeasure sample generation method and device based on proxy model and computing equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20210205 |
|
| WW01 | Invention patent application withdrawn after publication |