CN115658515A - Deep learning metamorphic test case sequencing method and computer readable medium - Google Patents
Deep learning metamorphic test case sequencing method and computer readable medium Download PDFInfo
- Publication number
- CN115658515A CN115658515A CN202211344266.9A CN202211344266A CN115658515A CN 115658515 A CN115658515 A CN 115658515A CN 202211344266 A CN202211344266 A CN 202211344266A CN 115658515 A CN115658515 A CN 115658515A
- Authority
- CN
- China
- Prior art keywords
- test case
- deep learning
- metamorphic
- software
- tested
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
The invention discloses a deep learning metamorphic test case sequencing method and a computer readable medium. The method of the invention firstly divides the measured deep learning model into two parts. And executing all metamorphic test case pairs in the first part of the tested deep learning model to obtain output, converting neuron output obtained by each metamorphic test case pair into distribution output, and calculating the distribution difference of the two to obtain the diversity of each metamorphic test case pair. And sequencing according to the diversity of the metamorphic test case pairs to obtain the execution sequence of the whole metamorphic test case pair set. And then executing the metamorphic test case pair on the second part of the tested deep learning model according to the obtained execution sequence to obtain the output of the source test case and the derived test case, so that the metamorphic test case pair which can trigger the error behavior of the tested model more easily can be preferentially executed. The invention improves the efficiency of testing the deep learning software by the tester.
Description
Technical Field
The invention relates to the technical field of computer software testing, in particular to a deep learning metamorphic test case sequencing method and a computer readable medium.
Background
Deep learning software is now widely used in people's daily lives and undertakes many important and complex tasks. However, deep learning software still has some hidden and fatal errors, which can even cause serious accidents. For example, in an automatic driving system, some traffic accidents occur because it erroneously recognizes an obstacle. Therefore, it is very important and necessary to adequately test a deep learning system and reveal the underlying errors.
At present, metamorphic testing is one of the most popular and effective testing techniques among testing techniques for deep learning software. Because the metamorphic test can effectively relieve the Oracle problem, a large amount of test data which are not marked manually can be widely applied to test deep learning software, so that the test data can be tested fully.
However, currently, the metamorphic test technology on deep learning software is often more concerned with exploring new application fields and creating novel metamorphic relations, but neglects the importance of optimizing metamorphic test case sets. The metamorphic test case set is optimized, so that the test efficiency can be improved, software defects possibly causing serious errors can be discovered in advance, and serious accidents can be avoided.
Disclosure of Invention
In order to solve the technical problem, the invention provides a deep learning metamorphic test case sequencing method and a computer readable medium.
The technical scheme of the method is a deep learning metamorphic test case sequencing method, which specifically comprises the following steps:
step 1: introducing a to-be-tested software deep learning classification model, selecting an intermediate neuron layer of the to-be-tested software deep learning classification model as a breakpoint layer, constructing a first part of the to-be-tested software deep learning classification model according to the input layer to the breakpoint layer of the to-be-tested software deep learning classification model, and constructing a second part of the to-be-tested software deep learning classification model by combining the breakpoint layer of the to-be-tested software deep learning classification model and the output layer;
step 2: introducing a plurality of original test cases, deriving the original test cases according to the metamorphic relation to obtain a plurality of derived test cases, and forming a plurality of metamorphic test case pairs;
and 3, step 3: inputting a plurality of original test cases into a first part of a deep learning classification model of the software to be tested to calculate to obtain an output result of each original test case, inputting a plurality of derivative test cases into the first part of the deep learning classification model of the software to be tested to calculate to obtain an output result of each derivative test case, and further calculating to obtain a diversity value of each original test case and the corresponding derivative test case by a method for calculating neuron output distribution differences;
and 4, step 4: sequencing the diversity values of a plurality of metamorphic test case pairs consisting of a plurality of original test cases and a plurality of corresponding derivative test cases from large to small to obtain the execution sequence of the metamorphic test case pairs;
and 5: and executing original test cases and derivative test cases of the plurality of metamorphic test case pairs in the second part of the deep learning classification model of the software to be tested according to the execution sequence of the metamorphic test case pairs, so that the metamorphic test case pairs which can more easily cause the error behavior of the deep learning classification model of the software to be tested can be preferentially executed.
Preferably, the first part of the deep learning classification model of the software to be tested in the step 1 is used for inputting a source test case and a derived test case of the metamorphic test case pair, outputting neuron outputs of the source test case and the derived test case of the metamorphic test case pair at a breakpoint layer, and further calculating a diversity value and inputting the second part of the deep learning classification model of the software to be tested;
the second part of the software classification deep learning model to be tested is used for inputting neuron outputs of source test cases and derived test cases of metamorphic test case pairs at a breakpoint layer, outputting classification results of the source test cases and the derived test cases of the metamorphic test case pairs, and further judging whether metamorphic relations are violated and error behaviors occur;
preferably, in step 2, the plurality of original test cases are subjected to derivation processing sequentially through a metamorphic relation to obtain a plurality of derived test cases and form a plurality of metamorphic test case pairs, which specifically includes the following steps:
step 2.1: and according to the rule defined by the metamorphic relation, carrying out derivative processing on the plurality of original test cases to obtain a plurality of corresponding derivative test cases.
Step 2.2: obtaining a plurality of corresponding metamorphic test case pairs according to the original test cases and the derived test cases;
preferably, in step 3, the diversity value of each original test case and the corresponding derived test case is obtained by calculating through a method for calculating the difference of the output distribution of the neurons, and the specific process is as follows:
obtaining the normalized output result of each original test case by the normalization method of the output result of each original test case;
obtaining the normalized output result of each derived test case by the normalization method of the output result of each derived test case;
calculating to obtain a normalized output result of each original test case and a normalized output result of each derived test case by a normalization method, and calculating a distribution difference between the normalized output result and the normalized output result of each derived test case to obtain a diversity value of each metamorphic test case pair formed by each original test case and the corresponding derived test case;
preferably, the calculating of the distribution difference in step 3 is as follows:
Diversity(t s ，t f )＝Discrepancy(O′(D 1 ，t s ，L k )，O′(D 1 ，t f ，L k ))
k∈[1，N]，s∈[1，K]，f∈[1，K]
wherein, the relationship (x) represents a calculation distribution difference algorithm model, the relationship (O) represents a normalization calculation model, and the relationship (t) represents the relationship (f) s ，t f ) Representing the s-th original test case and the f-th derived testValue of diversity, t, between test cases s Represents the s-th original test case, t f Represents the f-th derived test case, O' (D) 1 ，t s ，L k ) The normalized output result, O' (D), obtained by executing the s-th original test case on the k-th middle neuron layer of the part 1 of the software classification deep learning model to be tested is shown 1 ，t f ，L k ) The normalized output result obtained by executing the f derived test case on the k middle neuron layer of the part 1 of the software classification deep learning model to be tested is shown,
D 1 the method comprises the steps of representing a part 1 of a software classification deep learning model to be tested, representing the number of middle neuron layers of the software classification deep learning model to be tested, representing the number of original test cases by K, and representing the K-th middle neuron layer as a breakpoint layer.
Preferably, the error behavior of the deep learning classification model of the software to be tested in step 5 is as follows:
and if the metamorphic test case pair has the condition that the output result of the original test case and the output result of the corresponding derivative test case violate the output relationship specified by the metamorphic relationship, judging that the deep learning classification model of the test software has wrong behaviors.
The invention further provides a computer readable medium, which stores a computer program executed by an electronic device, and when the computer program runs on the electronic device, the electronic device executes the steps of the deep learning transmutation test case sequencing method.
The technical scheme provided by the invention has the beneficial effects that:
the invention ensures that the metamorphic test case pair with higher error sensitivity can be preferentially executed, improves the speed of finding the error of the deep learning software by a tester, improves the efficiency of performing metamorphic test on the deep learning software by the tester, and further relieves the problem of huge expense for testing on the deep learning software.
The invention can accurately measure the diversity of the metamorphic test case pairs, so that the metamorphic test case pairs which are easy to expose errors can be preferentially executed, thereby greatly improving the speed of finding errors in the process of testing deep learning software and greatly improving the testing efficiency.
The method can enable metamorphic test case pairs with higher error sensitivity to be preferentially executed, so that the speed of finding the deep learning software errors by testers is increased.
Drawings
FIG. 1: a method flow diagram of an embodiment of the invention;
FIG. 2: an illustration of the embodiment of the invention oriented to the VGG16 model;
FIG. 3: an overall framework diagram of an embodiment of the invention;
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In specific implementation, a person skilled in the art can implement the automatic operation process by using a computer software technology, and a system device for implementing the method, such as a computer-readable storage medium storing a corresponding computer program according to the technical solution of the present invention and a computer device including a corresponding computer program for operating the computer program, should also be within the scope of the present invention.
The invention aims to provide a deep learning metamorphic test case sequencing method based on neuron output distribution difference aiming at the problem of low efficiency in the current deep learning software test.
In order to achieve the technical effects, the main concept of the invention is as follows:
the method comprises the steps of firstly selecting a neuron layer of a measured deep learning model as a breakpoint, and dividing the measured deep learning model into two parts according to the breakpoint. And then executing the first part of the tested deep learning model to obtain the output of the source test case and the derivative test case of each metamorphic test case on the breakpoint neuron layer. And converting neuron outputs obtained by the source test case and the derived test case in each metamorphic test case pair into distribution outputs through normalization respectively, and obtaining the diversity of each metamorphic test case pair through calculating the distribution difference of the source test case and the derived test case. And obtaining the execution sequence of the set of the whole metamorphic test case pair according to the diversity size of the metamorphic test case pair. And then sequentially executing the metamorphic test case pairs on the second part of the tested deep learning model according to the obtained execution sequence to obtain the outputs of the source test case and the derivative test case, and checking whether the metamorphic test case pairs violate the corresponding metamorphic relation or not according to the obtained outputs.
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The technical scheme of the method of the embodiment of the invention is introduced in combination with fig. 1 to 3, and is a deep learning metamorphic test case sequencing method, which comprises the following specific steps:
step 1: introducing a to-be-tested software deep learning classification model, selecting an intermediate neuron layer of the to-be-tested software deep learning classification model as a breakpoint layer, constructing a first part of the to-be-tested software deep learning classification model according to the input layer to the breakpoint layer of the to-be-tested software deep learning classification model, and constructing a second part of the to-be-tested software deep learning classification model by combining the breakpoint layer of the to-be-tested software deep learning classification model and the output layer;
the deep learning classification model of the software to be tested is a deep learning classification model, here, a VGG16 model is taken as an example, and a specific model structure thereof can be shown in fig. 2. The model structure of the deep learning classification model of the software to be tested can be expressed as follows:
D＝<L 1 ，L 2 ，...，L i ，...，L N >
d represents a deep learning classification model of the software to be tested, L represents a neuron layer of the deep learning classification model of the software to be tested, and L i And the number of the neuron layers of the i-th layer of the deep learning classification model of the software to be tested is represented by N.
Step 1, a first part of the deep learning classification model of the software to be tested is used for inputting a source test case and a derived test case of a metamorphic test case pair, outputting neuron outputs of the source test case and the derived test case of the metamorphic test case pair at a breakpoint layer, and further calculating a diversity value and inputting a second part of the deep learning classification model of the software to be tested;
the first part of the deep learning classification model of the software to be tested can be expressed as:
D 1 ＝{L 1 ，L 2 ，...，L k }
wherein D 1 Representing a first part, L, of a deep-learning classification model of the software under test k And representing the k-th layer neuron layer, namely the breakpoint layer, of the selected software deep learning classification model to be tested.
Wherein the breakpoint layer L k Can be expressed as:
L k ＝<n k1 ，n k2 ，...，n ki ，...，n kM >
wherein n is ki K-layer neuron layer L representing deep learning classification model of software to be tested k The ith neuron of (1), and the breakpoint layer L k There are a total of M neurons in the population, i ∈ [1, M ]]；
The second part of the software classification deep learning model to be tested is used for inputting neuron outputs of source test cases and derived test cases of metamorphic test case pairs at a breakpoint layer, outputting classification results of the source test cases and the derived test cases of the metamorphic test case pairs, and further judging whether metamorphic relations are violated and error behaviors occur;
the second part of the deep learning classification model of the software to be tested can be expressed as:
D 2 ＝{L k+1 ，L k+2 ，…，L N }
wherein D is 2 Representing a second part, L, of the deep-learning classification model of the software under test k+1 Representing the k +1 layer neuron layer, L, of the deep learning classification model of the software to be tested k+2 Representing the selected k +2 layer neuron layer, L, of the deep learning classification model of the software to be tested N The second one representing the selected deep learning classification model of the software to be tested
N neuronal layers;
and 2, step: introducing a plurality of original test cases, sequentially carrying out derivative processing on the original test cases according to the metamorphic relation to obtain a plurality of derivative test cases, and forming a plurality of metamorphic test case pairs;
2, sequentially carrying out derivation treatment on the plurality of original test cases through a metamorphic relation to obtain a plurality of derived test cases and form a plurality of metamorphic test case pairs, wherein the specific process is as follows:
step 2.1: and according to a rule defined by the metamorphic relation, sequentially carrying out derivative processing on the plurality of original test cases to obtain a plurality of corresponding derivative test cases.
Step 2.2: obtaining a plurality of corresponding metamorphic test case pairs according to the original test cases and the derived test cases;
illustratively, in embodiments of the present invention, the transmutation test is used to verify expected relationships between inputs and outputs over multiple executions of a program, and these expected relationships may be represented as transmutation relationships. If the output result of multiple executions violates a specific metamorphic relation, a certain problem exists in the tested program. For example, a deep learning classification model is taken as an example.
For a picture x which is not labeled manually, manually checking whether the image classification procedure is successfully classified requires a lot of manpower and is prone to errors. If metamorphic testing is utilized, a simple metamorphic relation is designed, a large number of test cases are automatically generated, and whether the tested model has wrong behaviors or not can be checked by comparing the output results of the original test cases and the derived test cases. Assuming that the metamorphic relation is a flip metamorphic relation, an image x is used as an original test case x, and is flipped to obtain a derived test case x ', and a metamorphic test case pair is formed by the original test case x and the derived test case x'. And the classification results of the deep learning image classification program on the original test case x and the derived test case x 'should be kept the same, and because the turning picture does not change the subject object of the picture, the classification results of the deep learning classification model on the original test case x and the derived test case x' should be kept the same. If the difference is not the same, the metamorphic test case pair is composed of x and x', and the corresponding metamorphic relation is violated, which indicates that the deep learning classification model has certain error behaviors.
And step 3: sequentially inputting a plurality of original test cases to a first part of a deep learning classification model of the software to be tested to obtain an output result of each original test case, sequentially inputting a plurality of derivative test cases to the first part of the deep learning classification model of the software to be tested to obtain an output result of each derivative test case, and further calculating by a method for calculating output distribution differences of neurons to obtain diversity values of each original test case and the corresponding derivative test case;
the output result of the original test case can be expressed as:
O(D 1 ，t s ，L k )＝<x ks1 ，x ks2 ，...，x ksi ，...，x ksM >
wherein, O (D) 1 ，t s ，L k ) Representing the original test case t s Is under testFirst part D of deep learning classification model 1 Breakpoint layer L of k The resulting neuron output sequence, x, is executed ksi Representing the original test case t s Learning the first part D of the classification model at the depth to be measured 1 Breakpoint layer L of k The ith neuron above executes the resulting output, and there are a total of M neurons on the breakpoint layer, i ∈ [1, M ∈ []；
Wherein the output result of the derived test case can be expressed as:
O(D 1 ，t f ，L k )＝<x k_f_1 ，x k_f_2 ，...，x k_f_i ，...，x k_f_M >
wherein, O (D) 1 ，t f ，L k ) Representing derived test case t f Learning the first part D of the classification model at the depth to be measured 1 Breakpoint layer L of k Performing the resulting neuron output sequence; x is a radical of a fluorine atom k_f_i Representing derived test case t f Learning the first part D of the classification model at the depth to be measured 1 Breakpoint layer L of k The ith neuron above executes the resulting output, and there are a total of M neurons on the breakpoint layer, i ∈ [1, M ∈ []；
Specifically, step 3 calculates diversity of metamorphic test case pairs based on the neuron outputs at the breakpoint obtained by executing the first part of the tested deep learning model.
Exemplarily, in the embodiment of the present invention, the deep learning classification model VGG16 is taken as a specific example in the present invention. The model structure is shown in fig. 2. The VGG16 model is composed of 23 neuron layers, including an Input layer Input _1, a plurality of hidden layers, and an output layer Predictions. Wherein the input layer receives picture data having dimensions 224 x 3. And after the input layer receives the data, a final result is obtained through layer neuron calculation of the hidden layer. The hidden layer mainly comprises a convolution layer, a ReLU layer, a pooling layer and a full-connection layer. In practical applications, the convolutional layer and the ReLU layer are often laminated together to form the convolutional layer. For example, in the Block1_ Conv1 layer in fig. 2, not only the convolution operation is required for calculation, but also the operation is required to be calculated by an activation function and then input to the neuron of the next layer for further calculation.
As shown in fig. 2, assume Block3_ Conv1 is selected as the breakpoint layer. Preprocessing a source test case as follows: and inputting the data structure in the form of a 224 × 224 × 3 three-dimensional matrix into an Input _1 layer of the VGG16 model, and performing a series of convolution operations, activation function calculation and pooling operations to obtain neuron output on a Block3_ Cony1 layer.
And 3, calculating to obtain a diversity value of each original test case and the corresponding derivative test case by the method for calculating the output distribution difference of the neurons, wherein the specific process is as follows:
firstly, obtaining the normalized output result O' (D) of each original test case by the output result of each original test case through a normalization method 1 ，t s ，L k ) It can be represented as:
O′(D 1 ，t s ，L k )-<x′ k_s_1 ，x’ k_s_2 ，…，x′ k_s_i ，…，x’ k_s_M >
wherein, O' (D) 1 ，t s ，L k ) Representing the original test case t s Learning the first part D of the classification model at the depth to be measured 1 Breakpoint layer L of k The neuron element obtained by the execution outputs a normalized distribution output result, x' k_s_i Representing the original test case t s Learning the first part D of the classification model at the depth to be measured 1 Breakpoint layer L of k The ith neuron on the table executes the obtained output normalized distribution output result, and the breakpoint layer has M neurons in total, i belongs to [1, M ]]；
Then obtaining the normalized output result O' (D) of each derived test case by normalizing the output result of each derived test case 1 ，t f ，L k ) It can be expressed as:
O′(D 1 ，t f ，L k )＝<x′ k_f_1 ，x′ k_f_2 ，...，x’ k_f_i ，...，x’ k_f_M >
wherein O' (D) 1 ，t f ，L k ) Representing derived test case t f Learning the first part D of the classification model in depth to be measured 1 Breakpoint layer L of k The neuron element obtained by the execution outputs a normalized distribution output result, x' k_f_i Representing derived test cases t f Learning the first part D of the classification model at the depth to be measured 1 Breakpoint layer L of k The ith neuron on the layer is executed to obtain the output normalized distribution output result, and the breakpoint layer has M neurons in total, i belongs to [1, M ]]；
And calculating to obtain the normalized output result of each original test case and the normalized output result of each derivative test case by a normalization method, and calculating the distribution difference between the normalized output result of each original test case and the normalized output result of each derivative test case to obtain the diversity value of each metamorphic test case pair formed by each original test case and the corresponding derivative test case.
Wherein the value of diversity is expressed as:
Diversity(t s ，t f )＝Discrepancy(O′(D 1 ，t s ，L k )，O′(D 1 ，t f ，L k ))
the relationship represents a specific distribution difference calculation mode, such as a KL divergence, a JS divergence, a Hellinger distance or a Wasserstein distance. Diversity (t) s ，t f ) Representing the original test case t s And its corresponding derived test case t f And (3) the diversity value of the formed metamorphic test case pairs. Wherein O' (D) 1 ，t s ，L k ) Representing the original test case t s Learning the first part D of the classification model at the depth to be measured 1 Breakpoint layer L of k The obtained neurons output the normalized distribution output result. Wherein O' (D) 1 ，t f ，L k ) Representing derived test cases t f Learning the first part D of the classification model in depth to be measured 1 Breakpoint layer L of k And outputting the normalized distribution output result by the obtained neurons.
Illustratively, the process flow is set forth below using a specific example:
hypothesis source test case t s Learning the first part D of the classification model at the depth to be measured 1 Breakpoint neuron layer L of k Neuron output of (D) 1 ，t s ，L k ) Is (1, 2,3, 4), then its original test case t s Learning the first part D of the classification model at the depth to be measured 1 Breakpoint layer L of k The obtained neuron outputs a normalized distribution output result O' (D) 1 ，t s ，L k ) = (0.1, 0.2,0.3, 0.4). Assume the source test case t s Deriving test case t obtained through deriving treatment f Learning the first part D of the classification model at the depth to be measured 1 Breakpoint neuron layer L of k Neuron output of (D) 1 ，t f ，L k ) Is (2, 4, 10), then it is derived test case t f Learning the first part D of the classification model at the depth to be measured 1 Breakpoint layer L of k The obtained neuron outputs a normalized distribution output result O' (D) 1 ，t f ，L k ) = (0.1, 0.2, 0.5). Assuming that KL divergence is chosen as the measure here, the computation of the diversity of this pair of transmutation test cases is as follows:
wherein the Diversity is KL (t s ，t f ) Representing the original test case t s And its corresponding derived test case t f The formed disintegration test case is applied to a diversity value, x 'calculated by using a KL divergence distribution measurement mode' k_f_i Representing derived test case t f Learning the first part D of the classification model in depth to be measured 1 Breakpoint layer L of k Output normalized distribution output result, x 'obtained by the i-th neuron on' k_s_i Representing the original test case t s At the depth to be measuredLearning the first part D of the classification model 1 Breakpoint layer L of k The ith neuron executes the obtained distribution output result after output normalization;
and 4, step 4: sequencing the diversity values of a plurality of metamorphic test case pairs consisting of a plurality of original test cases and a plurality of corresponding derivative test cases from large to small to obtain the execution sequence of the metamorphic test case pairs;
illustratively, the process flow is set forth below using a specific example:
assume that there are five transmutation test case pairs in the SET of current transmutation test case pairs MP _ SET:
MP_SET＝(MP 1 ，MP 2 ，MP3，MP 4 ，MP 5 )
where MP _ SET represents a collection of metamorphic test case pairs.
Wherein, MP 1 ，MP 2 ，MP 3 ，MP 4 ，MP 5 Respectively representing a first metamorphic test case pair, a second metamorphic test case pair, a third metamorphic test case pair, a fourth metamorphic test case pair and a fifth metamorphic test case pair.
According to the steps 1-3, the diversity of each metamorphic test case pair (MP) can be obtained as follows:
the MP _ SET is ordered according to the size of the diversity from large to small, and the following execution order can be obtained:
wherein the execution order indicates, MP 2 To be executed first, MP 3 Will be executed by the second, MP 4 Will be executed by the third, MP 1 Will be executed by the fourth, MP 5 Will be executed last.
And 5: according to the execution sequence of the metamorphic test case pairs, original test cases and derivative test cases of the plurality of metamorphic test case pairs are executed on the second part of the deep learning classification model of the software to be tested, so that the metamorphic test case pairs which can easily cause the error behavior of the deep learning classification model of the software to be tested can be preferentially executed, the speed of finding the error behavior of the deep learning classification model of the software to be tested is increased, and the test efficiency is improved.
According to steps 1-4, the execution sequence of MP _ SET can be obtained as follows:
wherein MP _ SET _ PRIORITY represents the execution order of the pair of transmutation test cases.
The metamorphic test case pairs are executed according to the execution sequence MP _ SET _ PRIORITY, so that the metamorphic test case pairs which can more easily cause the error behavior of the deep learning classification model of the software to be tested can be preferentially executed, the speed of finding the error behavior of the deep learning classification model of the software to be tested is increased, and the test efficiency is improved.
And 5, the error behaviors of the deep learning classification model of the software to be tested are as follows:
if the metamorphic test case pair has the condition that the output result of the original test case and the output result of the corresponding derivative test case violate the output relation specified by the metamorphic relation, judging that the deep learning classification model of the test software has wrong behaviors;
illustratively, the process flow is set forth below using a specific example:
according to the execution sequence MP _ SET _ PRIORITY, MP of metamorphic test case pairs 2 The test case is preferentially executed, and the prediction result of the original test case on the VGG16 model is assumed as follows:
wherein, source _ Predictions represents the prediction result of the original test case of the metamorphic test case pair on the VGG16 model. Wherein CAT represents a feline, DOG represents a canine, and BIRD represents an avian.
Wherein the predictive value for CAT is 0.8, which is greater than other predictive classifications for DOG and BIRD. So the metamorphic test case pair MP 2 The predicted classification result of the original test case of (2) was CAT.
The prediction result of the derived test case on the VGG16 model is assumed as follows:
wherein Folow _ Predictions represents a prediction result of a derived test case of the transmutation test case pair on the VGG16 model.
Wherein the predicted value of DOG animals is 0.8, which is greater than other prediction categories CAT animals and BIRD animals BIRD. So the metamorphic test case pair MP 2 The predicted classification result of the derived test case of (a) is DOG animals.
MP pair according to the metamorphic test case 2 The output results of the original test case and the derived test case can judge the metamorphic test case to MP 2 And violating the corresponding metamorphic relation, and revealing that the tested deep learning classification model has unexpected and unexpected wrong behaviors.
In order to more clearly illustrate the practical process of the present invention, the following is described in detail by way of example:
FIG. 3 illustrates a process for deep learning transmutation test case ordering based on neuron output distribution variability according to the present invention. Firstly, a tester forms a metamorphic test case pair set through a source test case and a derivative test case generated through a predefined metamorphic relation. And then selecting a certain neuron layer of the deep learning model to be detected as a breakpoint to divide the neuron layer into two parts. Firstly, the output distribution of the source test case and the derived test case on the breakpoint neuron layer is obtained through the execution of the metamorphic test case pair on the first part of the deep learning model to be tested, and the diversity of the metamorphic test case pair is further obtained through measuring the distribution difference of the source test case and the derived test case. And secondly, obtaining an execution sequence of the metamorphic test case pair set according to the diversity size and the greedy algorithm of the metamorphic test case pairs, and executing the execution sequence on a second part of the deep learning model to be tested in sequence. Therefore, the metamorphic test case pairs which are more sensitive to errors are preferentially executed, and the speed of finding the errors in the deep learning test is improved.
Embodiments of the present invention also provide a computer-readable medium.
The computer readable medium is a server workstation;
the server workstation stores a computer program executed by the electronic device, and when the computer program runs on the electronic device, the electronic device executes the steps of the deep learning metamorphic test case sequencing method in the embodiment of the invention.
It should be understood that parts of the specification not set forth in detail are well within the prior art.
It should be understood that the above description of the preferred embodiments is given for clarity and not for any purpose of limitation, and that various changes, substitutions and alterations can be made herein without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (9)
1. A deep learning metamorphic test case sequencing method is characterized by comprising the following steps: the method comprises the following steps:
step 1: introducing a to-be-tested software deep learning classification model, selecting an intermediate neuron layer of the to-be-tested software deep learning classification model as a breakpoint layer, constructing a first part of the to-be-tested software deep learning classification model and constructing a second part of the to-be-tested software deep learning classification model;
step 2: introducing a plurality of original test cases, deriving the original test cases according to the metamorphic relation to obtain a plurality of derived test cases, and forming a plurality of metamorphic test case pairs;
and step 3: inputting a plurality of original test cases into a first part of a deep learning classification model of the software to be tested to calculate to obtain an output result of each original test case, inputting a plurality of derivative test cases into the first part of the deep learning classification model of the software to be tested to calculate to obtain an output result of each derivative test case, and further calculating to obtain a diversity value of each original test case and the corresponding derivative test case by a method for calculating neuron output distribution differences;
and 4, step 4: sequencing the diversity values of a plurality of metamorphic test case pairs consisting of a plurality of original test cases and a plurality of corresponding derivative test cases from large to small to obtain the execution sequence of the metamorphic test case pairs;
and 5: and executing original test cases and derivative test cases of the plurality of metamorphic test case pairs on the second part of the deep learning classification model of the software to be tested according to the execution sequence of the metamorphic test case pairs, so that metamorphic test case pairs which can more easily cause the error behaviors of the deep learning classification model of the software to be tested can be preferentially executed.
2. The deep learning transmutation test case sequencing method of claim 1, wherein:
step 1, the first part for constructing the classification deep learning model of the software to be tested is as follows:
constructing a first part of a to-be-tested software classification deep learning model according to an input layer to a breakpoint layer of the to-be-tested software deep learning classification model;
the second part for constructing the classification deep learning model of the software to be tested in the step 1 is as follows:
and constructing a second part of the deep learning classification model of the software to be tested from the next layer to the output layer by combining the breakpoint layer of the deep learning classification model of the software to be tested.
3. The deep learning transmutation test case sequencing method of claim 2, wherein:
the first part of the deep learning classification model of the software to be tested is used for inputting a source test case and a derivative test case of the metamorphic test case pair, outputting neuron outputs of the source test case and the derivative test case of the metamorphic test case pair at a breakpoint layer, and further calculating a diversity value and inputting a second part of the deep learning classification model of the software to be tested.
4. The deep learning transmutation test case sequencing method of claim 2, characterized in that:
the second part of the software classification deep learning model to be tested in the step 1 is used for inputting the neuron output of the source test case and the derived test case of the metamorphic test case pair at the breakpoint layer, and outputting the classification results of the source test case and the derived test case of the metamorphic test case pair, and is used for further judging whether the metamorphic relation is violated and an error behavior occurs.
5. The deep learning transmutation test case sequencing method of claim 1, characterized in that:
step 2, sequentially carrying out derivation treatment on the plurality of original test cases through metamorphic relations to obtain a plurality of derivative test cases and form a plurality of metamorphic test case pairs, wherein the specific process is as follows:
step 2.1: according to a rule defined by the metamorphic relation, carrying out derivative processing on a plurality of original test cases to obtain a plurality of corresponding derivative test cases;
step 2.2: and obtaining a plurality of corresponding metamorphic test case pairs according to the plurality of original test cases and the derived test cases.
6. The deep learning transmutation test case sequencing method of claim 1, wherein:
and 3, calculating to obtain a diversity value of each original test case and the corresponding derivative test case by the method for calculating the output distribution difference of the neurons, wherein the specific process is as follows:
obtaining the normalized output result of each original test case by the normalization method;
obtaining the normalized output result of each derived test case by the normalization method of the output result of each derived test case;
and calculating to obtain the normalized output result of each original test case and the normalized output result of each derived test case by a normalization method, and calculating the distribution difference between the normalized output result of each original test case and the normalized output result of each derived test case to obtain the diversity value of each metamorphic test case pair formed by each original test case and the corresponding derived test case.
7. The deep learning transmutation test case sequencing method of claim 6, wherein:
the distribution difference between the two is calculated as follows:
Diversity(t s ，t f )＝Discrepancy(O′(D 1 ，t s ，L k )，O′(D 1 ，t f ，L k ))
k∈[1，N]，s∈[1，K]，f∈[1，K]
wherein, the Discrepancy (x) represents a calculation distribution difference algorithm model, the O (x) represents a normalization calculation model, and the Diversity (t) represents s ，t f ) Representing a diversity value, t, between the s-th original test case and the f-th derived test case s Represents the s-th original test case, t f Represents the f-th derived test case, O' (D) 1 ，t s ，L k ) The normalized output result, O' (D), obtained by executing the s-th original test case on the k-th middle neuron layer of the part 1 of the software classification deep learning model to be tested is shown 1 ，t f ，L k ) The normalized output result obtained by executing the f derived test case on the k middle neuron layer of the part 1 of the software classification deep learning model to be tested is shown,
D 1 part 1 of the classification deep learning model of the software to be tested is shown, and N shows the software to be testedThe number of the intermediate neuron layers of the similar deep learning model is K, the K represents the number of original test cases, and the K-th intermediate neuron layer is a breakpoint layer.
8. The deep learning transmutation test case sequencing method of claim 1, characterized in that:
and 5, the error behaviors of the deep learning classification model of the software to be tested are as follows:
and if the metamorphic test case pair has the condition that the output result of the original test case and the output result of the corresponding derivative test case violate the output relation specified by the metamorphic relation, judging that the deep learning classification model of the test software has wrong behaviors.
9. A computer-readable medium, storing a computer program for execution by an electronic device, which, when run on the electronic device, causes the electronic device to perform the steps of the method of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211344266.9A CN115658515A (en) | 2022-10-31 | 2022-10-31 | Deep learning metamorphic test case sequencing method and computer readable medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211344266.9A CN115658515A (en) | 2022-10-31 | 2022-10-31 | Deep learning metamorphic test case sequencing method and computer readable medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115658515A true CN115658515A (en) | 2023-01-31 |
Family
ID=84993926
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211344266.9A Pending CN115658515A (en) | 2022-10-31 | 2022-10-31 | Deep learning metamorphic test case sequencing method and computer readable medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115658515A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117056203A (en) * | 2023-07-11 | 2023-11-14 | 南华大学 | Numerical expression type metamorphic relation selection method based on complexity |
-
2022
- 2022-10-31 CN CN202211344266.9A patent/CN115658515A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117056203A (en) * | 2023-07-11 | 2023-11-14 | 南华大学 | Numerical expression type metamorphic relation selection method based on complexity |
| CN117056203B (en) * | 2023-07-11 | 2024-04-09 | 南华大学 | Numerical expression type metamorphic relation selection method based on complexity |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107633265B (en) | Data processing method and device for optimizing credit evaluation model | |
| Gong et al. | Evolutionary generation of test data for many paths coverage based on grouping | |
| EP4075281A1 (en) | Ann-based program test method and test system, and application | |
| Uchiyama et al. | Design pattern detection using software metrics and machine learning | |
| CN113761444A (en) | Code scoring-based course recommendation method, course recommendation device and terminal equipment | |
| CN112419268A (en) | Method, device, equipment and medium for detecting image defects of power transmission line | |
| CN115658515A (en) | Deep learning metamorphic test case sequencing method and computer readable medium | |
| CN111581086B (en) | Hybrid software error positioning method and system based on RankNet | |
| CN114036531A (en) | Multi-scale code measurement-based software security vulnerability detection method | |
| WO2023115856A1 (en) | Task exception alert method and apparatus | |
| WO2019019429A1 (en) | Anomaly detection method, device and apparatus for virtual machine, and storage medium | |
| Asmono et al. | Absolute correlation weighted naïve bayes for software defect prediction | |
| CN112131587A (en) | Intelligent contract pseudo-random number security inspection method, system, medium and device | |
| CN116955059A (en) | Root cause positioning method, root cause positioning device, computing equipment and computer storage medium | |
| CN116166967A (en) | Data processing method, equipment and storage medium based on meta learning and residual error network | |
| CN112667569A (en) | Feature method, system, computer device and computer-readable storage medium | |
| Fu | Verifying probabilistic systems: new algorithms and complexity results | |
| CN110851344A (en) | Big data testing method and device based on computational formula complexity and electronic equipment | |
| CN113434408B (en) | Unit test case sequencing method based on test prediction | |
| Chandrasekaran | Testing Artificial Intelligence-Based Software Systems | |
| CN117035563B (en) | Product quality safety risk monitoring method, device, monitoring system and medium | |
| CN116362628B (en) | Method and device for analyzing and evaluating running state of power system | |
| CN117522174A (en) | Territorial space planning space data mutation checking method, application system and cloud platform | |
| US10839128B2 (en) | Error detection technique based on identifying data trend issues | |
| CN112883988B (en) | Training and feature extraction method of feature extraction network based on multiple data sets |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |