CN111858343A

CN111858343A - Countermeasure sample generation method based on attack capability

Info

Publication number: CN111858343A
Application number: CN202010712768.7A
Authority: CN
Inventors: 房春荣; 李珍鸿; 顾逸飞; 刘佳玮
Original assignee: Shenzhen Muzhi Technology Co ltd
Current assignee: Shenzhen Muzhi Technology Co ltd
Priority date: 2020-07-23
Filing date: 2020-07-23
Publication date: 2020-10-30

Abstract

A confrontation sample generation method based on attack capability comprises a data set and neural network model processing module, a confrontation sample generation module and a distance evaluation module. The dataset and neural network model processing module converts the dataset into numpy data of python for numerical computation in the project. The challenge sample generation module randomly screens out a part of data, generates challenge samples by using different challenge sample generation methods, and evaluates the average time of generating a single challenge sample by each generation method. In the distance evaluation module, we analyze the number of misclassifications and the highest probability of misclassification from two dimensions according to the distribution of output information. Secondly, if the original data has no label, the correct label of the original data cannot be known, and the validity of the original data can be calculated only according to the output classification quantity and the classification probability distribution.

Description

Countermeasure sample generation method based on attack capability

Technical Field

The invention belongs to the field of software testing, and particularly relates to generation of a countermeasure sample, and evaluation of quality of the generated countermeasure sample is carried out based on attack capability of the generated countermeasure sample on a neural network.

Background

In recent years, deep learning and deep neural networks are rapidly developed, and deep learning models have been related to aspects of social life. Therefore, the testing technology for guaranteeing the quality of the deep learning model is also in an increasingly important position. The traditional software testing technology cannot meet the characteristics that a deep learning model is complex and various and is difficult to understand. Therefore, new testing techniques for deep learning models become an important hotspot problem in the current academic and industrial fields. Proved by facts, in some fields, the confrontation samples can effectively find out the bug of the deep learning model, and the quality of the deep learning model is improved. Countermeasures are the deliberate addition of some imperceptible subtle perturbations to the input samples, causing the model to give an erroneous output with high confidence. The confrontation sample generation only needs to calculate the original data, the effective generation property is consistent with the original property, but the error data of the neural network is caused, and a large amount of labor cost is saved.

In the deep learning model test, the countermeasure sample generation test has become the mainstream method because of fragmentation of deep learning data, inexplicability of the deep learning model, and diversity of test scenarios. However, openness against sample generation also tends to result in poor quality results. Quality control is a significant challenge for challenge sample generation, especially for multiple iterations of a challenge sample generation method, such as up. The quality control strategy in the generation of the countersample is mainly divided into the countersample attack capability, the countersample disturbance magnitude and the like. For most data (even common application users), it is easy to get countersamples in iterative calculations. However, the unverified challenge sample may not be able to make the neural network, resulting in a low quality challenge sample, and cannot be used to repair the neural network. These problems cause great inconvenience and burden to the defect location and repair work of the deep learning model. At present, there is no effective means for quality management of the challenge sample, and therefore, it is desirable to research an auxiliary means to help evaluate the property of the challenge sample, so as to achieve the purpose of quality management of the challenge sample.

Some preliminary work has been done by some researchers on the challenge samples, but the existing work is limited to the generation of the challenge samples and supports poor quality assessment of the challenge samples.

Based on the work, the invention deeply mines the output information of the confrontation sample of the deep learning model in the neural network. This information is simple and understandable for the person. However, further technical processing work is required for the machine to automatically recognize this information and to generalize it into a failed countermeasure sample. Therefore, based on the existing research results, the invention summarizes and modifies the corresponding challenge sample generation technology, adds a new technical method, applies the new technical method to the generation and evaluation of the challenge sample in the deep learning model, enhances the quality screening of the challenge sample, and automatically generates the challenge sample.

Disclosure of Invention

The invention aims to solve the problems that: the quality of the confrontation sample generated by the confrontation sample generation method in the deep learning model test is underground, and the effective defect repair effect is difficult to form. The invention can evaluate the attack capability of the invention on the neural network, automatically generate high-quality countersample and solve the problem of low quality of the countersample.

The technical scheme of the invention is as follows: a method for generating a countermeasure sample based on attack capability assessment is characterized in that a reliable countermeasure sample can be generated according to deep learning raw data. The generation method comprises the following three modules/steps:

1) the data set and neural network model processing module: firstly, acquiring a data set and a neural network model file input by a user, acquiring structural information, neuron quantity, weight of each neuron and an activation function of the neural network model, restoring the neural network model, and locally generating the neural network model of the user. And decoding the input data set, and converting the data set into numpy data of python for numerical calculation in the project.

2) And a confrontation sample generation module. The system firstly randomly screens a part of data aiming at the data set, generates the confrontation samples by using different confrontation sample generation methods, and evaluates the average time of generating a single confrontation sample by each generation method. Based on the average time of generation, the challenge sample generation method that takes time to exceed the threshold is eliminated, because this method may be too costly for this data set, which may affect the cost of the overall process of generating the challenge samples later. For each data in the data set, a screened countermeasure sample generation method is adopted to generate a countermeasure sample, and effective countermeasure disturbance can be performed on each iteration by calculating the gradient of each iteration. After the end of the iteration cycle, each confrontation sample generation method generates one confrontation sample by default, and of course, the user can also set the number of generated confrontation samples. There are several different ways of generating challenge samples for each datum.

3) A distance evaluation module: firstly, the generated confrontation sample is input into a neural network, and the output information of the confrontation sample is analyzed. The processing of the output information is different according to whether the original data has a label, i.e. if the original data has a label, the property label of the confrontation sample should be consistent with the original data, so according to the distribution of the output information, from two dimensions: the number of misclassifications, the highest probability of misclassification. Secondly, if the original data has no label, the correct label of the original data cannot be known, and the validity of the original data can be calculated only according to the output classification quantity and the classification probability distribution.

The invention is characterized in that:

1. the use of time-based evaluation challenge sample generation techniques was first proposed in the field of challenge sample generation.

2. The attack capability assessment is applied for the first time to the generation of the challenge sample.

Drawings

Fig. 1 is a general flow chart of the implementation of the present invention.

Figure 2 is a flow chart of key step 1.

Figure 3 is a flow chart of key step 2.

Figure 4 is a flow chart of key step 3.

Detailed Description

The embodiments of the present invention are described below with reference to specific examples, and other advantages and effects of the present invention will be readily apparent to those skilled in the art from the disclosure of the present specification.

The method is used for automatically generating the countermeasure sample based on distance evaluation, mainly adopts a countermeasure sample generation technology, and relates to specific key technologies including a deep Convolutional Neural Network (CNN), a gradient descent technology and an attack capability evaluation technology.

1. Feature extraction

In the invention, a convolutional neural network method is adopted to extract the characteristics of a data file uploaded by a user, and a picture is converted into a characteristic vector. Convolutional neural networks are a class of feed-forward neural networks that contain convolutional computations and have a deep structure. The convolutional neural network has the characteristic learning ability and can carry out translation invariant classification on input information according to the hierarchical structure of the convolutional neural network. The convolutional neural network is constructed by imitating a visual perception mechanism of a living being, and can perform supervised learning and unsupervised learning, and the convolutional neural network can perform lattice characterization with smaller calculation amount due to parameter sharing of convolution kernels in hidden layers and sparsity of interlayer connection.

2. Deep neural network reduction

In the invention, the network to which the method is directed is a deep neural network which mainly comprises convolution, pooling and activation functions, and the deep neural network has inexplicability, so that the sufficiency of an original data set is difficult to measure from the point of white box testing. However, the neuron is almost a general feature representing the deep neural network, the structure and weight of the neuron represent the cognitive behavior of the deep neural network, and the output of the neuron finally determines the output of the neural network. Therefore, when the deep neural network is introduced, the neuron is inserted, and the output change of the neuron after data input is calculated, so that the gradient direction after data disturbance is calculated.

3. Countermeasure sample method time assessment

In the invention, the original data is calculated by a confrontation sample generation method, and the confrontation sample is generated by iterative disturbance. The time cost is an important cost for the generation of the confrontation, so the time length required for different confrontation samples is measured, thereby excluding the method that the time cost cannot bear. In the invention, a method of sampling a data set is adopted to replace the whole data set, and the average generation time of different countermeasure sample generation methods for the sampled data is calculated, so that the approximate time cost of the method for the data set is obtained.

4. Antagonistic sample generation

In the invention, gradient calculation is adopted to perturb the image feature vector generated by the convolutional neural network into a countermeasure sample. In the machine learning algorithm, when the loss function is minimized, iterative solution can be performed step by step through a gradient method, and the minimized loss function and the model parameter value are obtained. When the countermeasure sample is generated, by applying perturbation to the data in the direction along the gradient, the perturbation can be made most effective, and the data making the neural network erroneous can be found efficiently.

5. Attack capability assessment

In the invention, the output distribution of the confrontation sample relative to the neural network is calculated, and the confrontation sample generated by disturbance is input into the original neural network to obtain the output probability distribution condition. The probability distribution refers to the probability of various classifications of the output of the sample by the neural network, and the attack capability of the countermeasure sample can be effectively calculated through the probability distribution, so that the countermeasure sample is evaluated

In this example, the deep neural network computation of the data feature extraction part converts the input picture into a data vector. In the neural network model reduction part, a publicly available neural network model reduction method is adopted to generate a neural network for calculating the generation of the confrontation sample. The confrontation sample list used in the confrontation sample filtering is generated by automatically perturbing from our data set, and there are five generation methods, each of which generates several confrontation samples. And in the final resisting sample evaluation part, an attack ability evaluation method is adopted, the output distribution of the generated resisting samples input into the neural network is effectively calculated, and the resisting samples with poor attack ability are eliminated.

Claims

1. A counterattack sample generation method based on attack ability is characterized in that a data set and a neural network model processing module complete the conversion of the data set; configuring the correlation attributes generated by the samples in the confrontation sample generation module; and evaluating the generated result according to the distribution of the output information.

2. The method of transforming a data set with a neural network model processing module as claimed in claim 1, wherein: firstly, acquiring a data set and a neural network model file input by a user, acquiring structural information, neuron quantity, weight of each neuron and an activation function of the neural network model, restoring the neural network model, and locally generating the neural network model of the user. And decoding the input data set, and converting the data set into numpy data of python for numerical calculation in the project.

3. The correlation property for configuring sample generation in an antagonistic sample generation module as described in claim 1, wherein: the system firstly randomly screens a part of data aiming at the data set, generates the confrontation samples by using different confrontation sample generation methods, and evaluates the average time of generating a single confrontation sample by each generation method. Based on the average time of generation, the challenge sample generation method that takes time to exceed the threshold is eliminated, because this method may be too costly for this data set, which may affect the cost of the overall process of generating the challenge samples later. For each data in the data set, a screened countermeasure sample generation method is adopted to generate a countermeasure sample, and effective countermeasure disturbance can be performed on each iteration by calculating the gradient of each iteration.

4. The evaluation of the generated result based on the distribution of the output information as described in claim 1, wherein: firstly, the generated confrontation sample is input into a neural network, and the output information of the confrontation sample is analyzed. The processing of the output information is different according to whether the original data has a label or not, and if the original data has a label, the property label of the confrontation sample is consistent with the original data, so that the number of error classifications and the highest probability of error classification are analyzed from two dimensions according to the distribution of the output information. Secondly, if the original data has no label, the correct label of the original data cannot be known, and the validity of the original data can be calculated only according to the output classification quantity and the classification probability distribution.