CN114510715B - Method and device for testing functional safety of model, storage medium and equipment - Google Patents
Method and device for testing functional safety of model, storage medium and equipment Download PDFInfo
- Publication number
- CN114510715B CN114510715B CN202210042146.7A CN202210042146A CN114510715B CN 114510715 B CN114510715 B CN 114510715B CN 202210042146 A CN202210042146 A CN 202210042146A CN 114510715 B CN114510715 B CN 114510715B
- Authority
- CN
- China
- Prior art keywords
- model
- test
- original
- data set
- value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/50—Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
- G06F21/55—Detecting local intrusion or implementing counter-measures
- G06F21/56—Computer malware detection or handling, e.g. anti-virus arrangements
- G06F21/566—Dynamic detection, i.e. detection performed at run-time, e.g. emulation, suspicious activities
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/50—Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
- G06F21/55—Detecting local intrusion or implementing counter-measures
- G06F21/56—Computer malware detection or handling, e.g. anti-virus arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Abstract
The invention discloses a method, a device, a storage medium and equipment for testing functional safety of a model, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring an original model and an original data set; generating a test data set based on the original data set, the test data set including a perturbation sample; carrying out deep learning model test on the original model by adopting a test data set to obtain an original test value; performing target function safety protection on the original model to obtain a protection model; carrying out deep learning model test on the protection model by adopting a test data set to obtain a protection test value; and determining a functional safety test result of the original model based on the original test value and the protection test value. The invention realizes the data processing-oriented functional safety test of the deep learning model.
Description
Technical Field
The invention relates to the technical field of artificial intelligence, in particular to a method, a device, a storage medium and equipment for testing functional safety of a model.
Background
Although artificial intelligence models have been used in many industries, security challenges for artificial intelligence models have been presented.
To eliminate safety challenges for artificial intelligence models as much as possible, functional safety testing of artificial intelligence models has become very important. After the artificial intelligence model finishes training, the functional safety test can be carried out on the artificial intelligence model firstly, and if the artificial intelligence model passes the functional safety test, the artificial intelligence model can be put into use. The functional security test in the related art is mainly a security test oriented to the traditional software, namely, the functional security test focuses on testing software bugs, defects in model design and the like.
However, data processing also plays a crucial role in the security of the artificial intelligence model, and a data processing-oriented functional security test for the artificial intelligence model is lacking in the related art.
Disclosure of Invention
The invention provides a method, a device, a storage medium and equipment for testing functional safety of a model, which realize data processing-oriented functional safety testing of an artificial intelligent model.
The technical scheme of the invention comprises the following steps:
a method of functional security testing of a model, the method comprising:
acquiring an original model and an original data set;
generating a test dataset based on the raw dataset, the test dataset comprising a perturbation sample s;
performing deep learning model test on the original model by adopting the test data set to obtain an original test value of a model test index;
performing target function security protection on the original model to obtain a protection model, wherein the target function security protection is used for protecting the disturbance sample s from attacking the original model;
performing deep learning model test on the protection model by adopting the test data set to obtain a protection test value of the model test index;
and determining a functional safety test result of the original model based on the original test value and the protection test value.
Optionally, the performing target function security protection on the original model to obtain a protection model includes:
adding a detector before the original model to obtain the protection model, wherein the target function safety protection comprises the step of eliminating the disturbance sample s in the test data set through the detector;
and/or the presence of a gas in the gas,
adding a reconstructor before the original model to obtain the protection model, wherein the target function safety protection comprises reconstructing a sample in the test data set through the reconstructor.
Optionally, the detector and/or the reconstructor are implemented as an auto-encoder, and a training process of the auto-encoder is as follows:
constructing a training loss value;
training an auto-encoder using the raw data set
And adjusting the auto-encoder based on the training loss value
Parameter θ of * To obtain an automatic encoder
t is the training round;
at the automatic encoder
Is less than a loss threshold, the auto-encoder is configured to perform the auto-encoder
As an automatic encoder for completing the training;
at the automatic encoder
Let t = t +1, in case the training loss value of (a) is greater than a loss threshold, from training an automatic encoder with the original data set
The steps of (1) are started again.
Optionally, the determining a functional safety test result of the original model based on the original test value and the protection test value includes:
calculating a difference between the original test value and the protection test value;
determining that the functional safety test result is that the functional safety of the original model is low under the condition that the difference is larger than a threshold value T;
and under the condition that the difference is smaller than a threshold value T, determining that the function safety test result is the function safety of the original model.
Optionally, the generating a test data set based on the raw data set includes:
performing sample change processing on samples in the original data set to obtain at least one change sample, wherein the sample change processing comprises at least one of the following steps: data change, random noise addition, and countermeasure generation;
screening the perturbation sample s from the at least one variation sample;
and constructing the test data set based on the perturbation sample s.
Optionally, the screening the disturbance sample s from the at least one change sample comprises:
for each of the at least one variation sample, calculating a difference value between the variation sample and a corresponding sample in the original data set, the difference value including any one of: the drop value of the accuracy, the Euclidean distance and the absolute value loss;
and screening the change sample as the disturbance sample s under the condition that the difference value meets a difference condition.
Optionally, after obtaining the original model and the original data set, the method further includes:
carrying out deep learning model test on the original model by adopting the original data set to obtain an evaluation test value of the model test index;
wherein the test data set is generated based on the original data set if the evaluation test value is greater than a preset test value.
A functional safety testing device of a model, comprising:
the acquisition module is used for acquiring an original model and an original data set;
a generating module for generating a test data set based on the original data set, the test data set comprising a perturbation sample s;
the first testing module is used for carrying out deep learning model testing on the original model by adopting the testing data set to obtain an original testing value of a model testing index;
the safety protection module is used for carrying out target function safety protection on the original model to obtain a protection model, and the target function safety protection is used for protecting the attack of the disturbance sample s on the original model;
the second testing module is used for carrying out deep learning model testing on the protection model by adopting the testing data set to obtain a protection testing value of the model testing index;
and the calculation module is used for determining a functional safety test result of the original model based on the original test value and the protection test value.
A storage medium having stored therein a computer program, wherein the computer program is arranged to execute the functional safety testing method of the model described above when running.
A computer device comprising a memory having a computer program stored therein and a processor arranged to run the computer program to perform the functional safety testing method of the model described above.
A computer program product for causing a computer device to perform a method for functional security testing of the above model when the computer program product is run on the computer device.
Compared with the related art, the invention has at least the following beneficial effects:
according to the method, the original model and the original model configured with the target function safety protection are subjected to deep learning model test respectively by adopting the test data set comprising the disturbance sample so as to obtain the original test value and the protection test value of the model test index, and finally, the function safety of the original model is determined based on the original test value and the protection test value, so that the data processing-oriented function safety test of the deep learning model is realized.
In addition, the invention provides a plurality of target function safety protection modes, such as adding a detector and/or a reconstructor before the original model, and can flexibly and effectively protect the attack of the disturbance sample on the original model.
In addition, the test data set used by the method is generated based on the original data set, the disturbance sample is generated by performing data change, random noise addition, countermeasure generation and the like on the sample in the original data set, and the test data set is constructed based on the disturbance sample.
In addition, before the original model is subjected to the functional safety test, the original data set is adopted to carry out the deep learning model test on the original model, and the subsequent functional safety test is continuously executed under the condition that the original model passes the evaluation test, so that the inaccuracy of the functional safety test result caused by the factors of insufficient training of the original model, poor quality of the original data set and the like can be avoided, the accuracy of the functional safety test result is further enhanced, and the efficiency of the functional safety test is improved.
Drawings
Fig. 1 is a flowchart of a functional safety testing method for a model according to an embodiment of the present invention.
FIG. 2 is a schematic diagram of a protection model according to an embodiment of the present invention.
FIG. 3 is a schematic diagram of a protection model according to another embodiment of the present invention.
FIG. 4 is a schematic diagram of a protection model according to yet another embodiment of the present invention.
FIG. 5 is a schematic diagram of a protection model according to another embodiment of the present invention.
FIG. 6 is a block diagram of a functional safety test device for a model according to an embodiment of the present invention.
Detailed Description
In the following, technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the embodiments of the present invention, and it is obvious that the described embodiments are only specific embodiments of the present invention, and not all embodiments. All other embodiments, which can be obtained by a person skilled in the art without any inventive step based on the embodiments of the present invention, are within the scope of the present invention.
The functional safety test method of the present invention, as shown in fig. 1, includes the following steps (110 to 160).
Step 110: an original model and an original data set are obtained.
The computer device first obtains the original model and the original data set X = { X } required by the functional security test task i I =1, 2.., N }, where x i Is the sample in the original data set (or called original sample), N is the number of original samples, and N is a positive integer. The original model in this embodiment refers to a deep learning model.
For the acquisition of the original model and the original data set, the acquisition from local upload of a user or the selection from the existing model and data set is supported.
In order to ensure the accuracy of the functional safety test and improve the efficiency of the functional safety test, in an example, after the step 110, the method further includes: performing deep learning model test on the original model by adopting an original data set to obtain an evaluation test value of a model test index; wherein in case the evaluation test value is larger than the preset test value, it is executed starting from step 120 described below.
The model test index is an index reflecting the prediction effect of the model, and may be different for different types of models. Illustratively, for models used for image classification tasks, model test metrics include accuracy, precision, and the like; aiming at a model for a target detection task, model test indexes comprise mAP (mean Average Precision), recall rate and the like; for the model for the semantic segmentation task, the model test indexes include accuracy, pixel accuracy, ioU (Intersection over Union), and the like.
After obtaining the raw model and the raw data set, the computer device employs the raw data setAnd carrying out deep learning test on the original model to obtain an evaluation test value of the model test index. Illustratively, the computer device inputs samples in the raw data set (i.e., raw samples) into the raw model, resulting in a predicted value y 'of the raw samples' 0 And then based on predicted values y 'of the respective original samples' 0 And its label value y 0 And calculating an evaluation test value. Then, the computer device compares the evaluation test value with the preset test value, if the evaluation test value is greater than the preset test value, or the evaluation test value is equal to the preset test value, it indicates that the original model training effect is better, the quality of the original data set is higher, and step 120 can be continuously executed; if the evaluation test value is smaller than the preset test value, the original model is poor in training effect and needs to be retrained, or the original data set is poor in quality and needs to be replaced or optimized.
Step 120: a test data set is generated based on the raw data set, the test data set including a perturbation sample s.
The test data set used by the function safety test task can be further generated based on the original data set, the test data set comprises a disturbance sample s, and the disturbance sample s can attack the original model, for example, the prediction accuracy of the original model is reduced, and even the system using the original model may have operation errors. The embodiment does not limit the manner in which the computer device generates the test data set, and optionally, the computer device may perform one or more manners of screening, sample change processing, combination and splicing, and the like on the original sample to construct the test data set.
In one example, the above-mentioned step 120 includes several sub-steps (122 to 126) as follows.
Step 122: and carrying out sample change processing on samples (namely original samples) in the original data set to obtain at least one change sample.
The sample change processing refers to performing change processing on one or more items of the structure, format, and structure of the original sample, and optionally, the sample change processing includes at least one of the following items: data change, random noise addition, challenge generation. Exemplary algorithms used by the sample change process include, but are not limited to: FGSM (Fast Gradient Sign Method), depfool (an Attack Method based on hyperplane classification), JSMA (Jacobian-based collaborative Map Attack), C & W (Carlini and ware Attacks), BIM (Basic Iterative Method), elastic net (elastic network), MI-FGSM (motion objective Fast Gradient Sign Method, FGSM with addition of a Momentum term), PGD (Project Gradient decision), and the like.
Step 124: and screening the disturbance sample s from the at least one change sample.
In order to obtain a higher-quality disturbance sample s and implement a sufficient functional safety test on the original model, in this embodiment, after the change sample is obtained through sample change processing, the change sample is further screened to obtain the disturbance sample s. Optionally, screening the perturbed sample s from the at least one variation sample comprises: calculating a difference value between each of the at least one variation sample and a corresponding sample in the original data set; and screening the variation sample as a disturbance sample s under the condition that the difference value meets the difference condition.
The variation sample is obtained based on the sample (original sample) processing in the original data set, and the perturbation sample s can be screened by the difference value between the variation sample and the original sample. Optionally, the difference value includes any one of: the accuracy drop, the Euclidean loss, the absolute loss, etc. Based on different types of difference values, different difference conditions are set. For example, a difference condition of a drop value threshold value may be set based on the drop value of the accuracy, and when the accuracy of the change sample is compared with the accuracy of the original sample, if the drop value of the accuracy is greater than or equal to the drop value threshold value, the change sample is screened as a disturbance sample s; and setting a difference condition of a loss value threshold value based on the Euclidean loss, and when the Euclidean loss between the change sample and the original sample is calculated, if the Euclidean loss is less than the loss value threshold value, the similarity between the change sample and the original sample is higher, and screening the change sample into a disturbance sample s.
Step 126: and constructing a test data set based on the disturbance sample s.
The perturbed sample s with higher quality can be obtained through the above steps 122 and 124, and then the test data set is constructed based on the perturbed sample s, so that the test data set includes the perturbed sample s. Optionally, in step 126, the computer device may construct a test data set based only on the perturbation samples s, that is, only the perturbation samples s are included in the test data set; the test data set may also be constructed based on the perturbed sample s and the original sample together, that is, the test data set includes not only the perturbed sample s but also some or all of the samples in the original data set.
Step 130: and performing deep learning model test on the original model by adopting a test data set to obtain an original test value of the model test index.
When the original model is subjected to the functional safety test, the deep learning model test needs to be performed on the original model by adopting the constructed test data set, and the original test value of the model test index is obtained. For an introduction description of the deep learning model test and the model test index, please refer to the above embodiments, which are not repeated herein.
Step 140: and performing target function safety protection on the original model to obtain a protection model.
In this embodiment, when performing the functional safety test on the original model, in addition to acquiring the original test value, the original model also needs to be subjected to the target functional safety protection to obtain the protection model, and the protection test value is obtained based on the protection model. And the target function safety protection is used for protecting the attack of the disturbance sample s on the original model.
The embodiment does not limit the specific manner of the target function security protection, and in an example, the step 140 includes: adding a detector before the original model to obtain a protection model, so that the target function safety protection comprises the step of removing disturbance samples s in a test data set through the detector; and/or adding a reconstructor before the original model to obtain the protection model, so that the target function safety protection comprises reconstructing a sample in the test data set through the reconstructor. Based on this, the target function security protection in this embodiment includes at least the following four implementation manners.
(1) And adding a detector before the original model to obtain a protection model.
In this implementation, the target function safety protection includes culling, by the detector, the perturbed samples s in the test dataset. That is to say, the detector identifies the disturbance sample s, the identified disturbance sample s is removed from the test data set, and then the residual samples in the test data set are used for carrying out deep learning model test on the original model to obtain a protection test value.
Illustratively, FIG. 2 is a schematic diagram of a guard model with a detector added before the original model. For any given test sample z i Using detector AE (θ) * ) Calculation output AE (θ) * ,z i ) And calculates the output AE (theta) * ,z i ) And test specimen z i The calculation formula is | z i -AE(θ * ,z i )‖ 2 (ii) a Wherein, theta * Are detector parameters. If the reconstruction loss value is larger than the set threshold value, the test sample z i Is a disturbance sample, and a test sample z is rejected i (ii) a If the reconstruction loss value is less than or equal to the set threshold value, testing the sample z i Instead of perturbing the sample, a test sample z may be used i Deep learning model testing is performed on the original model.
(2) And adding a reconstructor before the original model to obtain the protection model.
In this implementation, the target functional safety protection includes reconstructing a sample in the test data set by a reconstructor. That is, the sample in the test data set is reconstructed by the reconstructor, and then, the original model is subjected to the deep learning model test by using the reconstructed sample, so as to obtain the protection test value. The non-disturbance samples in the test data set can not generate obvious difference through reconstruction, and the disturbance samples can be closer to the corresponding samples in the original data set through reconstruction, so that the attack of the disturbance samples to the original model can be effectively reduced.
Illustratively, FIG. 3 is a schematic diagram of another protection model, with a reconstructor added before the original model. For any given test sample z i Testing the sample z by a reconstructor i Reconstructing to obtain a reconstructed sample z' i . Afterwards, reconstructed sample z 'is taken' i Deep learning model testing is performed on the original model.
(3) And adding a detector and a reconstructor before the original model to obtain a protection model.
In this implementation, the target function security protection includes rejecting a disturbance sample s in the test data set by the detector, and then reconstructing the remaining samples in the test data set by the reconstructor. Namely, the detector identifies the disturbance sample s, the identified disturbance sample s is removed from the test data set, then the reconstructor reconstructs the rest samples in the test data set, and the reconstructed samples are used for carrying out deep learning model test on the original model to obtain a protection test value.
Illustratively, FIG. 4 is a schematic diagram of yet another guard model with a reconstructor added before the original model and a detector added before the reconstructor. If the test sample z is determined by the detector i If the sample is a disturbance sample, rejecting the test sample z i (ii) a If the test sample z is determined by the detector i If not, the test sample z is reconstructed by the reconstructor i Reconstructing to obtain a reconstructed sample z' i After that, reconstructed sample z 'is taken' i Deep learning model testing is performed on the original model.
(4) And adding a reconstructor and a detector before the original model to obtain a protection model.
In this implementation, the target function safeguard includes reconstructing samples in the test data set by a reconstructor and then rejecting perturbed samples in the test data set by a detector. That is to say, a sample in the test data set is reconstructed by the reconstructor, then, a disturbance sample in the reconstructed sample is identified by the detector, the identified disturbance sample is removed, and then, the original model is subjected to deep learning model test by using the residual sample, so as to obtain a protection test value.
It should be noted that, in this implementation, the disturbed samples identified by the detector are not necessarily the disturbed samples s initially included in the test data set, and it is possible that the disturbed samples s are converted into non-disturbed samples through reconstruction by the reconstructor, and the non-disturbed samples initially included in the test data set are converted into disturbed samples through reconstruction by the reconstructor.
Illustratively, FIG. 5 is a schematic diagram of yet another guard model with a detector added before the original model and a reconstructor added before the detector. Firstly, the test sample z is processed by a reconstructor i Reconstructing to obtain a reconstructed sample z' i If reconstructed sample z 'is determined by the detector' i If the sample is a disturbance sample, rejecting a reconstruction sample z' i (ii) a If the reconstructed sample z 'is determined by the detector' i If the sample is not a disturbance sample, adopting a reconstructed sample z' i Deep learning model testing is performed on the original model.
In the above embodiments, the detector and/or reconstructor may be implemented as an auto-encoder, and the training process of the auto-encoder may comprise several steps (1401 to 1404) as follows.
Step 1401: and constructing a training loss value.
In this embodiment, the construction method of the training loss value is not limited, and optionally, the training loss value may be constructed by adopting a sample reconstruction loss, such as an euclidean loss, an absolute value loss, and the like; alternatively, the training loss value may be constructed by using similarity between the sample prediction value and the sample label value, such as KL (Kullback-Leibler divergence) divergence, JS (Jensen-Shannon divergence), cosine similarity, and the like.
Illustratively, the training Loss value includes a sample reconstruction Loss, which may be implemented as a euclidean Loss (L2 Loss) or an absolute value Loss (L1 Loss). Training the automatic encoder through sample reconstruction loss can enable the input original sample x i Is reconstructed with higher quality, this reconstruction step also actually involves modeling the distribution of the test data set that differs significantly from this distributionThe sample reconstruction loss corresponding to the disturbance sample s is also certain to be large, so that whether the current input sample is the disturbance sample can be judged by setting a threshold value according to the sample reconstruction loss. Assuming that the auto-encoder is fitted using neural network AE (θ) to approximate an identity map, the sample reconstruction loss L (θ) is as follows:
illustratively, the training loss value includes the JS divergence between the sample prediction value and the sample label value. Suppose P 1 Is the sample tag value, P 2 Is the sample prediction value (prediction result of data reconstructed by the auto-encoder), then P 1 And P 2 The JS divergence between (1) is as follows:
wherein, if P 1 And P 2 For an n-dimensional discrete probability distribution, then:
therefore, the JS divergence solves the problem that the KL divergence is asymmetric in a symmetrical mode, and distance calculation can be better carried out on two probability distributions.
Step 1402: automatic encoder for training by adopting original data set
And adjusting the automatic encoder based on the training loss value
Parameter θ of * To obtain an automatic encoder
t is the training round.
The automatic encoder performs a minimization of sample reconstruction loss by adjusting parameters
The value of (b) enables the sample reconstruction loss to be converged, and the output AE (theta, x) of the automatic encoder is realized i ) Approach to the original sample x i . Wherein the content of the first and second substances,
and obtaining the optimal parameters of the automatic encoder for the training round t-1.
In step 1402, a comparison auto-encoder is also needed
A converged training loss value and a loss threshold value, and in case the training loss value is less than the loss threshold value, the following step 1403 is executed; in the event that the training loss value is greater than the loss threshold, step 1404, described below, is performed. It should be understood that for the case where the training loss value is equal to the loss threshold, the computer device may be set to perform either step 1403 or 1404.
Step 1403: in an automatic encoder
Is less than the loss threshold, the automatic encoder is started
As an automatic encoder for completing the training.
Step 1404: in an automatic encoder
Let t = t +1 to train the automatic encoder using the original data set when the training loss value of (2) is greater than the loss threshold value
The steps of (1) are performed again. If the sample reconstruction loss is less thanIf the threshold is lost, it is considered that the input change is too large due to the reconstruction performed by the automatic encoder, and there is a certain influence on the subsequent detection and/or prediction, let t = t +1, return to step 1402, and retrain the automatic encoder.
Step 150: and carrying out deep learning model test on the protection model by adopting the test data set to obtain a protection test value of the model test index.
According to the embodiment, the protection model is a structure of adding the target function safety protection before the original model, and the deep learning model test is performed on the protection by adopting the test data set, namely, the test data set is firstly input into the structure of the target function safety protection, and then the deep learning model test is performed on the original model by adopting the sample output by the structure, so that the protection test value of the model test index is obtained. For a specific obtaining process of the protection test value, an introduction description of the model test index, and the like, please refer to the above embodiments, which are not described herein again.
Illustratively, the guard model includes a detector and a raw model, assuming a sample z is included in the test dataset 1 、z 2 、z 3 、z 4 The label values of the four samples are all y 0 Then the sample z is first detected by the detector 1 、z 2 、z 3 、z 4 Whether the sample is a disturbance sample; suppose that the detection result represents a sample z 1 、z 2 、z 3 、z 4 If none is the disturbance sample, then the sample z 1 、z 2 、z 3 、z 4 Inputting an original model, and obtaining predicted values of the four samples to be y' 1 、y′ 2 、y′ 3 、y′ 4 (ii) a Finally, based on the tag value y 0 And predicted value y' 1 、y′ 2 、y′ 3 、y′ 4 The protection test value of the model test index can be calculated.
Illustratively, the protection model includes a reconstructor and an original model, assuming that a sample z is included in the test dataset 1 、z 2 、z 3 、z 4 The label values of the four samples are all y 0 Then the sample z is first reconstructed by the reconstructor 1 、z 2 、z 3 、z 4 Obtaining a reconstructed sample z' 1 、z′ 2 、z′ 3 、z′ 4 (ii) a Thereafter, sample z 'will be reconstructed' 1 、z′ 2 、z′ 3 、z′ 4 Inputting an original model, and obtaining predicted values of the four samples to be y' 1 、y′ 2 、y′ 3 、y′ 4 (ii) a Finally, based on the tag value y 0 And predicted value y' 1 、y′ 2 、y′ 3 、y′ 4 The protection test value of the model test index can be calculated.
Step 160: and determining a functional safety test result of the original model based on the original test value and the protection test value.
And calculating based on the original test value and the protection test value to determine the functional safety test result of the original model. The embodiment does not limit the specific calculation manner, and optionally, one or more of ratio calculation, subtraction calculation, and comparison calculation may be performed on the original test value and the protection test value. For example, the original test value and the protection test value are respectively compared with a set threshold, and if the original test value is smaller than the set threshold and the protection test value is greater than the set threshold, it is determined that the functional safety test result of the original model is that the functional safety of the original model is low.
In one example, the step 160 includes: calculating a difference between the original test value and the protection test value; determining that the functional safety of the original model is low in the functional safety test result under the condition that the difference is larger than the threshold T; and under the condition that the difference is smaller than the threshold value T, determining that the function safety test result is the function safety of the original model.
The protection test value is obtained based on the condition that the original model is configured with target function safety protection, so that the protection test value of the model test index is theoretically superior to the original test value. Based on this, the difference between the original test value and the guard test value needs to be calculated first. In an example, if the guard test values are obtained based on a case where the detector is added before the original model, based on a case where the reconstructor is added before the original model, based on a case where the detector and the reconstructor are added before the original model, and based on a case where the reconstructor and the detector are added before the original model, respectively, the guard test value used in step 160 may be a maximum value of the four guard test values, an average value of the four guard test values, or a weighted average value of the four guard test values, which is not limited in this embodiment.
Comparing the difference between the original test value and the protection test value with a preset threshold value T: under the condition that the difference value is larger than the threshold value T, the original model has a larger improvement space, the functional safety test result is that the functional safety of the original model is low, and the original model does not pass the functional safety test; and under the condition that the difference is smaller than the threshold value T, the improvement space of the original model is smaller, the functional safety test result is that the functional safety of the original model is high, and the original model passes the functional safety test.
In this embodiment, after the functional safety test result of the original model is determined, the functional safety test result may be further stored for subsequent query and comparison. Optionally, the saved data includes, but is not limited to: basic information of the functional safety test task, an original test value of the model test index, a protection test value of the model test index, a difference value between the original test value and the protection test value, a chart display of the test values, a functional safety test result, a prediction result and the like. The basic information of the functional safety test task comprises a test task name, creation time, an original model, an original data set, test task description, threshold setting, target functional safety protection configuration and the like; the prediction result refers to the prediction values of different samples under the original model and the protection model respectively.
In summary, the invention adopts the test data set including the disturbance sample to perform the deep learning model test on the original model and the original model configured with the target function safety protection respectively so as to obtain the original test value and the protection test value of the model test index, and finally determines the function safety of the original model based on the original test value and the protection test value, thereby realizing the data processing-oriented function safety test of the deep learning model.
In addition, the invention provides a plurality of target function safety protection modes, such as adding a detector and/or a reconstructor before the original model, and can flexibly and effectively protect the attack of the disturbance sample on the original model.
In addition, the test data set used by the method is generated based on the original data set, the disturbance sample is generated by performing data change, random noise addition, countermeasure generation and the like on the sample in the original data set, and the test data set is constructed based on the disturbance sample.
In addition, before the original model is subjected to the functional safety test, the original data set is adopted to carry out the deep learning model test on the original model, and the subsequent functional safety test is continuously executed under the condition that the original model passes the evaluation test, so that the inaccuracy of the functional safety test result caused by the factors of insufficient training of the original model, poor quality of the original data set and the like can be avoided, the accuracy of the functional safety test result is further enhanced, and the efficiency of the functional safety test is improved.
Referring to fig. 6, a block diagram of a functional safety testing device of a model according to an embodiment of the present invention is shown. The device can be computer equipment and can also be arranged in the computer equipment. As shown in fig. 6, the apparatus 600 includes: the system comprises an acquisition module 610, a generation module 620, a first testing module 630, a security protection module 640, a second testing module 650 and a calculation module 660.
An obtaining module 610 is configured to obtain an original model and an original data set.
A generating module 620, configured to generate a test data set based on the original data set, where the test data set includes a perturbation sample s.
The first testing module 630 is configured to perform a deep learning model test on the original model by using the test data set, and obtain an original test value of a model test index.
And the safety protection module 640 is configured to perform target function safety protection on the original model to obtain a protection model, where the target function safety protection is used to protect the disturbance sample s from attacking the original model.
And the second testing module 650 is configured to perform deep learning model testing on the protection model by using the test data set, and obtain a protection test value of the model test index.
A calculating module 660, configured to determine a functional safety test result of the original model based on the original test value and the protection test value.
Optionally, the apparatus 600 further comprises a third testing module, configured to: carrying out deep learning model test on the original model by adopting the original data set to obtain an evaluation test value of the model test index; wherein the test data set is generated based on the original data set if the evaluation test value is greater than a preset test value.
Optionally, the generating module 620 includes: a changing unit, configured to perform sample change processing on samples in the original data set to obtain at least one change sample, where the sample change processing includes at least one of: data change, random noise addition, and countermeasure generation; a screening unit for screening the perturbation sample s from the at least one variation sample; a construction unit for constructing the test data set based on the perturbation samples s.
Optionally, the screening unit is configured to: for each of the at least one variation sample, calculating a difference value between the variation sample and a corresponding sample in the original data set, the difference value including any one of: the accuracy rate drop, euclidean distance, and absolute value loss; and screening the change sample as the disturbance sample s under the condition that the difference value meets a difference condition.
Optionally, the safety protection module 640 is configured to: adding a detector in front of the original model to obtain the protection model, wherein the target function safety protection comprises eliminating the disturbance sample s in the test data set through the detector; and/or adding a reconstructor before the original model to obtain the protection model, wherein the target function safety protection comprises reconstructing a sample in the test data set through the reconstructor.
Optionally, the detector and/or the reconstructor are implemented as an auto-encoder, the apparatus further comprising a training module for: constructing a training loss value; training an autoencoder using the raw data set
And adjusting the auto-encoder based on the training loss value
Parameter theta of * To obtain an automatic encoder
t is the training round; at the automatic encoder
Is less than a loss threshold, the auto-encoder is configured to perform the method of (a) when the training loss value of (b) is less than a loss threshold
As an automatic encoder for completing the training; at the automatic encoder
Let t = t +1, in case the training loss value of (a) is greater than a loss threshold, from training an automatic encoder with the original data set
The steps of (1) are performed again.
Optionally, the calculating module 660 is configured to: calculating a difference between the original test value and the protection test value; determining that the functional safety test result is that the functional safety of the original model is low under the condition that the difference is larger than a threshold value T; and determining that the functional safety test result is the functional safety of the original model under the condition that the difference is smaller than a threshold value T.
Optionally, the apparatus 600 further includes a task management module, configured to view an existing task, rerun an existing task, delete a task, and the like.
Optionally, the apparatus 600 further includes a task configuration module, configured to create a new functional security testing task.
Optionally, the apparatus 600 further includes an operation management module, configured to display basic task information and a task operation log, terminate an operating task according to a test requirement, and display a task progress in real time in task management during task execution, where the content of the task progress includes, but is not limited to, a predicted value of a sample, a test index of a model, a model test progress, and the like.
Optionally, the apparatus 600 further includes a result output module, configured to output a functional safety test report, where the functional safety test report includes basic information of a functional safety test task, an original test value of a model test index, a protection test value of the model test index, a difference between the original test value and the protection test value, a chart display of the test value, a functional safety test result, a prediction result, and the like.
Optionally, the apparatus 600 further comprises a database for storing the created tasks, the existing models and data sets, the reconstruction and detection methods, the operation results, and the like.
For the explanation of the specific execution process, beneficial effects, etc. of the device module, please refer to the description of the above method embodiment, which is not described herein again.
In an exemplary embodiment, a computer device is further provided, which comprises a memory and a processor, wherein the memory stores a computer program, and the computer program is loaded and executed by the processor to implement the functional safety testing method of the model.
In an exemplary embodiment, a computer-readable storage medium is also provided, on which a computer program is stored which, when being executed by a processor, carries out the functional safety testing method of the model as described above.
In an exemplary embodiment, a computer program product is also provided, which, when run on a computer device, causes the computer device to perform the functional safety testing method of the model as described above.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (6)
1. A method for functional security testing of a model, the method comprising:
acquiring an original model and an original data set;
performing deep learning model test on the original model by adopting the original data set to obtain an evaluation test value of a model test index; if the original model is used for an image classification task, the model test indexes comprise accuracy and precision; if the original model is used for a target detection task, the model test indexes comprise an average precision mean value mAP and a recall rate; if the original model is used for a semantic segmentation task, the model test indexes comprise accuracy, pixel accuracy and an intersection ratio IoU;
in the case where the evaluation test value is less than a preset test value, performing any one of: retraining the original model, replacing the original data set and optimizing the original data set;
generating a test data set based on the original data set under the condition that the evaluation test value is greater than a preset test value, wherein the test data set comprises a disturbance sample s and part or all of samples in the original data set; wherein the generating a test data set based on the raw data set comprises: performing sample change processing on samples in the original data set to obtain at least one change sample, wherein the sample change processing comprises at least one of the following steps: data change, random noise addition, and countermeasure generation; for each of the at least one variation sample, calculating a difference value between the variation sample and a corresponding sample in the original data set, the difference value including any one of: the accuracy rate drop, euclidean distance, and absolute value loss; screening the change sample as the disturbance sample s under the condition that the difference value meets a difference condition; constructing the test data set based on the perturbation samples s and the samples in the original data set;
performing deep learning model test on the original model by adopting the test data set to obtain an original test value of the model test index;
performing target function security protection on the original model to obtain a protection model, wherein the target function security protection is used for protecting the disturbance sample s from attacking the original model;
carrying out deep learning model test on the protection model by adopting the test data set to obtain a protection test value of the model test index;
determining a functional safety test result of the original model based on the original test value and the protection test value; wherein the determining a functional safety test result of the original model based on the original test value and the guard test value comprises: calculating a difference between the original test value and the protection test value; determining that the functional safety test result is that the functional safety of the original model is low under the condition that the difference value is larger than a threshold value T; and under the condition that the difference is smaller than a threshold value T, determining that the function safety test result is the function safety of the original model.
2. The method of claim 1, wherein the performing target function security protection on the original model to obtain a protection model comprises:
adding a detector in front of the original model to obtain the protection model, wherein the target function safety protection comprises eliminating the disturbance sample s in the test data set through the detector;
and/or the presence of a gas in the atmosphere,
adding a reconstructor before the original model to obtain the protection model, wherein the target function safety protection comprises reconstructing a sample in the test data set through the reconstructor.
3. The method of claim 2, wherein the detector and/or the reconstructor are implemented as an auto-encoder, the auto-encoder training process is as follows:
constructing a training loss value;
training an autoencoder using the raw data set
And adjusting the auto-encoder based on the training loss value
Parameter θ of * Obtaining an automatic encoder
t is the training round;
at the automatic encoder
Is less than a loss threshold, the auto-encoder is configured to perform the auto-encoder
As an automatic encoder for completing the training;
at the automatic encoder
Let t =under the condition that the training loss value is greater than the loss threshold valuet +1, from training an automatic encoder using said raw data set
The steps of (1) are performed again.
4. A functional safety testing device for a model, comprising:
the acquisition module is used for acquiring an original model and an original data set;
the third testing module is used for carrying out deep learning model testing on the original model by adopting the original data set to obtain an evaluation testing value of a model testing index; if the original model is used for an image classification task, the model test indexes comprise accuracy and precision; if the original model is used for a target detection task, the model test indexes comprise an average precision mean value mAP and a recall rate; if the original model is used for a semantic segmentation task, the model test indexes comprise accuracy, pixel accuracy and an intersection ratio IoU; wherein, in the case that the evaluation test value is less than a preset test value, any one of the following is performed: retraining the original model, replacing the original data set and optimizing the original data set;
the generating module is used for generating a test data set based on the original data set under the condition that the evaluation test value is larger than a preset test value, wherein the test data set comprises a disturbance sample s and part or all of samples in the original data set; wherein the generating a test data set based on the raw data set comprises: performing sample change processing on samples in the original data set to obtain at least one change sample, wherein the sample change processing comprises at least one of the following steps: data change, random noise addition, and countermeasure generation; for each of the at least one variation sample, calculating a difference value between the variation sample and a corresponding sample in the original data set, the difference value including any one of: the drop value of the accuracy, the Euclidean distance and the absolute value loss; screening the change sample as the disturbance sample s under the condition that the difference value meets a difference condition; constructing the test data set based on the perturbed samples s and the samples in the original data set;
the first testing module is used for carrying out deep learning model testing on the original model by adopting the testing data set to obtain an original testing value of the model testing index;
the safety protection module is used for carrying out target function safety protection on the original model to obtain a protection model, and the target function safety protection is used for protecting the attack of the disturbance sample s on the original model;
the second testing module is used for carrying out deep learning model testing on the protection model by adopting the testing data set to obtain a protection testing value of the model testing index;
the calculation module is used for determining a functional safety test result of the original model based on the original test value and the protection test value; wherein the determining a functional safety test result of the original model based on the original test value and the guard test value comprises: calculating a difference between the original test value and the protection test value; determining that the functional safety test result is that the functional safety of the original model is low under the condition that the difference value is larger than a threshold value T; and determining that the functional safety test result is the functional safety of the original model under the condition that the difference is smaller than a threshold value T.
5. A storage medium having a computer program stored thereon, wherein the computer program is arranged to perform the method of any of claims 1-3 when executed.
6. A computer device comprising a memory having a computer program stored therein and a processor arranged to run the computer program to perform the method according to any of claims 1-3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210042146.7A CN114510715B (en) | 2022-01-14 | 2022-01-14 | Method and device for testing functional safety of model, storage medium and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210042146.7A CN114510715B (en) | 2022-01-14 | 2022-01-14 | Method and device for testing functional safety of model, storage medium and equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114510715A CN114510715A (en) | 2022-05-17 |
| CN114510715B true CN114510715B (en) | 2022-10-14 |
Family
ID=81549606
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210042146.7A Active CN114510715B (en) | 2022-01-14 | 2022-01-14 | Method and device for testing functional safety of model, storage medium and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114510715B (en) |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TW202139075A (en) * | 2020-03-31 | 2021-10-16 | 群光電子股份有限公司 | Deep learning model training system, deep learning model training method, and non-transitory computer readable storage medium |
Family Cites Families (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110826059B (en) * | 2019-09-19 | 2021-10-15 | 浙江工业大学 | Method and device for defending black box attack facing malicious software image format detection model |
| CN110851835A (en) * | 2019-09-23 | 2020-02-28 | 平安科技（深圳）有限公司 | Image model detection method and device, electronic equipment and storage medium |
| CN111600835B (en) * | 2020-03-18 | 2022-06-24 | 宁波送变电建设有限公司永耀科技分公司 | Detection and defense method based on FGSM (FGSM) counterattack algorithm |
| US11494496B2 (en) * | 2020-03-30 | 2022-11-08 | International Business Machines Corporation | Measuring overfitting of machine learning computer model and susceptibility to security threats |
| CN111598210B (en) * | 2020-04-30 | 2023-06-02 | 浙江工业大学 | Anti-attack defense method for anti-attack based on artificial immune algorithm |
| EP3910479A1 (en) * | 2020-05-15 | 2021-11-17 | Deutsche Telekom AG | A method and a system for testing machine learning and deep learning models for robustness, and durability against adversarial bias and privacy attacks |
| CN112215298A (en) * | 2020-10-21 | 2021-01-12 | 平安国际智慧城市科技股份有限公司 | Model training method, device, equipment and readable storage medium |
| CN113343247A (en) * | 2021-06-17 | 2021-09-03 | 公安部第三研究所 | Biological characteristic identification counterattack sample attack safety evaluation method, system, device, processor and computer readable storage medium thereof |
| CN113822328B (en) * | 2021-08-05 | 2022-09-16 | 厦门市美亚柏科信息股份有限公司 | Image classification method for defending against sample attack, terminal device and storage medium |
| CN113723564A (en) * | 2021-09-14 | 2021-11-30 | 燕山大学 | Method and system for training defense model of confrontation sample and application of method and system |
| CN113902959A (en) * | 2021-10-28 | 2022-01-07 | 中国工商银行股份有限公司 | Image recognition method and device, computer equipment and storage medium |
-
2022
- 2022-01-14 CN CN202210042146.7A patent/CN114510715B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TW202139075A (en) * | 2020-03-31 | 2021-10-16 | 群光電子股份有限公司 | Deep learning model training system, deep learning model training method, and non-transitory computer readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114510715A (en) | 2022-05-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Zhong et al. | A multi-level deep learning system for malware detection | |
| CN109189767B (en) | Data processing method and device, electronic equipment and storage medium | |
| CN110704840A (en) | Convolutional neural network CNN-based malicious software detection method | |
| CN111783442A (en) | Intrusion detection method, device, server and storage medium | |
| JP2015087903A (en) | Apparatus and method for information processing | |
| Mai et al. | Metamixup: Learning adaptive interpolation policy of mixup with metalearning | |
| CN113204745B (en) | Deep learning back door defense method based on model pruning and reverse engineering | |
| CN114692156B (en) | Memory segment malicious code intrusion detection method, system, storage medium and equipment | |
| US11954202B2 (en) | Deep learning based detection of malicious shell scripts | |
| CN109656818B (en) | Fault prediction method for software intensive system | |
| WO2023030322A1 (en) | Methods, systems, and media for robust classification using active learning and domain knowledge | |
| CN110991246A (en) | Video detection method and system | |
| CN117115581A (en) | Intelligent misoperation early warning method and system based on multi-mode deep learning | |
| CN117557872B (en) | Unsupervised anomaly detection method and device for optimizing storage mode | |
| CN114037001A (en) | Mechanical pump small sample fault diagnosis method based on WGAN-GP-C and metric learning | |
| JP2008009548A (en) | Model preparation device and discrimination device | |
| CN116051924B (en) | Divide-and-conquer defense method for image countermeasure sample | |
| CN114510715B (en) | Method and device for testing functional safety of model, storage medium and equipment | |
| Nakhwan et al. | Comparison Analysis of Data Augmentation using Bootstrap, GANs and Autoencoder | |
| CN115080745A (en) | Multi-scene text classification method, device, equipment and medium based on artificial intelligence | |
| CN111581640A (en) | Malicious software detection method, device and equipment and storage medium | |
| US11727109B2 (en) | Identifying adversarial attacks with advanced subset scanning | |
| CN114463574A (en) | Scene classification method and device for remote sensing image | |
| Valle et al. | Assessing the reliability of visual explanations of deep models with adversarial perturbations | |
| Liu et al. | A selective quantization approach for optimizing quantized inference engine |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |