CN112035338B - Coverage rate calculation method of stateful deep neural network - Google Patents
Coverage rate calculation method of stateful deep neural network Download PDFInfo
- Publication number
- CN112035338B CN112035338B CN202010662117.1A CN202010662117A CN112035338B CN 112035338 B CN112035338 B CN 112035338B CN 202010662117 A CN202010662117 A CN 202010662117A CN 112035338 B CN112035338 B CN 112035338B
- Authority
- CN
- China
- Prior art keywords
- time sequence
- state
- list
- training
- calculating
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3676—Test management for coverage analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Artificial Intelligence (AREA)
- Biophysics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Tests Of Electronic Circuits (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a method for calculating coverage rate of a stateful deep neural network, which aims at the stateful deep neural network, such as RNN, and realizes the coverage rate of the stateful deep neural network by six steps of extraction, dimension reduction plane establishment, state base number confirmation, state conversion confirmation, state coverage rate calculation and conversion coverage rate calculation. The method can solve some defects of neuron coverage, can better adapt to the state deep neural network by incorporating the time sequence relation among samples into the coverage criterion, can analyze according to the state and time sequence characteristics of the state deep neural network, and realizes the state level coverage rate calculation and the conversion level coverage rate calculation of the state deep neural network, thereby having certain understanding on the accuracy and the robustness of the current model.
Description
Technical Field
The invention relates to coverage rate calculation of a deep neural network model, in particular to a coverage rate calculation method based on a stateful deep neural network, and belongs to the field of artificial intelligence tests.
Background
Code coverage is one of the indexes for measuring the test performance of the software system, and generally, the higher the code coverage, the better the software performance. However, it is not feasible for deep learning systems to use only traditional software test metrics for scaling. Because the decision logic of the deep learning model is not written manually, but rather is learned automatically from the training data. In the research, it was found that code coverage reached 100% when deep learning tests were performed with a randomly selected test input. It is clear that in this case the value of the measure of conventional coverage has lost its meaning. Therefore, it is necessary to adopt new coverage criteria to effectively measure the test performance of the deep learning system.
At present, the coverage rate of a deep neural network is studied, and the coverage is a coverage criterion which is proposed from the perspective of a neuron. In the earliest proposed neuron coverage, a neuron was considered to be an activated neuron if its output value was greater than a given threshold. And observing the coverage rate of the model by observing the number ratio of the activated neurons. However, such coverage rate is only applicable to general deep neural networks such as CNN, DNN, etc., and since such neural networks only analyze the content of training sample data itself during training and do not analyze the timing relationship between training samples, the research on the coverage rate of such networks mainly covers the data content of the samples itself, but does not cover the timing relationship between the samples. The inputs and outputs of such networks have no timing related information. However, for a state-based deep neural network such as RNN, since timing information is captured by memory of loop and internal state, the timing relationship between training samples is considered in the model training period, and the influence of the input timing of the model and the timing relationship between samples on the quality of the model is large. If the coverage test is performed on the stateful deep neural network by using the neuron coverage, only the output condition of each neuron of the current network can be obtained, and the time-series relationship cannot be well covered, so that the current coverage criterion is not suitable for the stateful neural network.
The method is different from the neuron coverage rate of the existing deep neural network at present, and starts from the characteristic of the stateful deep neural network, and considers the distribution condition of the training samples in the data space and the time sequence relation among the training samples. The data space distribution and time sequence conversion relation between the training sample and the test sample can be observed through the state level coverage rate, the weighted state level coverage rate, the conversion level coverage rate and the weighted conversion level coverage rate in the invention.
Disclosure of Invention
In view of the time sequence characteristic of the stateful deep neural network, the traditional software test indexes and the common deep neural network coverage criterion cannot be suitable for the stateful deep neural network. The coverage rate test method is based on the characteristics of the stateful deep neural network, focuses on the characteristic of the time sequence of the stateful deep neural network, and realizes the coverage rate test of the stateful deep neural network by researching the time sequence relation of the training sample and the test sample and through six steps of extraction, dimension reduction, scanning, state base number confirmation, state conversion confirmation and coverage rate calculation.
In order to realize the purpose of the invention, the following technical scheme is adopted:
the invention discloses a deep neural network coverage rate calculation method based on a stateful state, which comprises the following steps:
step 1: deploying an RNN training model and a static data set which are trained off line, confirming the number m x m of states to be determined and test data, loading the model, inputting the training data and the test data, and collecting the training time sequence state and the test time sequence state.
Step 2: and reducing the dimensions of the collected training time sequence state and the collected testing time sequence state, extracting a critical point after dimension reduction, establishing a plane P by using the critical point, and dividing the plane P into a plurality of states.
And step 3: and classifying and confirming the training time sequence state and the testing time sequence state after dimension reduction, and determining the relative positions of the training time sequence state and the testing time sequence state in a plane.
And 4, step 4: and according to the input time sequence, performing time sequence connection on the results after classification confirmation according to a linked list to obtain the conversion relation between the time sequence states of the training time sequence state and the test time sequence state.
And 5: and (4) comparing the classification results of the training time sequence state and the testing training time sequence state according to the classification result of the step (4) to obtain a state level coverage BSCov and a weighted state level coverage WSCov of the deep neural network model.
Step 6: and (5) comparing the time sequence conversion relation of the classification results of the training test time sequence state and the test training time sequence state according to the time sequence connection in the step (5) to obtain the conversion level coverage BTCov and the weighted conversion level coverage WTCov of the deep neural network.
The static data set in step 1 includes an original training set and a test set of the stateful deep neural network to be tested, and the training time sequence state and the test time sequence state in step 1 are obtained by loading the static data set.
After the test time sequence state and the training time sequence state are obtained, a data basis is provided for subsequent dimension reduction, and a critical point support is provided for establishing a classification plane.
Establishing a dimensionality reduction plane: and performing PCA processing on the extracted data, and mapping the data onto a two-dimensional plane. Scanning the data after dimensionality reduction, searching critical values of the data in the x and y directions, establishing a plane P by taking the critical values as basic points, and dividing the plane into a plurality of states. The step 2 is further as follows:
step 21: and performing data conversion processing on the data types of the training time sequence state and the testing time sequence state to enable the data types to meet the requirement of dimension reduction processing.
Step 22: and performing dimensionality reduction on the processed data, and reducing the processed data to 2 dimensions by using a PCA (principal component analysis) method, so that the original high-dimensional data is mapped onto a 2-dimensional plane.
Step 23: and scanning the time sequence state after dimension reduction, and finding the maximum value MAX and the minimum value MIN in the x and y directions.
Step 24: and (MAX _ X,0) (MIN _ X,0) (0, MAX _ Y) (0, MIN _ Y) is used for establishing a plane P, m equal division is carried out on the X direction of the plane P, m equal division is carried out on the Y direction of the plane P, and the whole plane is divided into m states.
Status cardinality validation: and traversing the data after dimensionality reduction, confirming the specific position of the data in the P, namely which state belongs to the P, and finding out the cardinal number of the corresponding state. The step 3 is further as follows:
step 31: and traversing the training time sequence state and the testing time sequence state after dimension reduction, and comparing the state in which the value of the training time sequence state and the testing time sequence state is positioned in the plane P.
Step 32: a list TrainList is created for the training sequence states, each element of the list containing two attributes list1, list2, the first attribute list1 indicating the training sequence state and the second attribute list2 indicating which state in the plane P the training sequence state lies in, i.e. the corresponding index x, y.
Step 33: a list TestList is built for the test timing states, each element of the list containing two attributes list1, list2, the first attribute list1 indicating the test timing state and the second attribute list2 indicating which state in P the test timing state is located in, i.e. the corresponding index x, y.
And (3) confirming state transition: performing time sequence connection on the confirmed time sequence states according to a loading sequence, wherein the step 4 is further as follows:
step 41: and 2, obtaining the state of each time sequence state in the plane, and respectively establishing a time sequence linked list TrainLinkedList and a test LinkedList for the training time sequence state and the test time sequence state. According to the input time sequence, the time sequence relation between the states after the classification confirmation can be obtained.
Step 42: and connecting the training time sequence states through a chain table TrainLinkedList according to the input time sequence, wherein the connected chain table TrainLinkedList represents the time sequence conversion relation among the training time sequence states.
Step 43: connecting the test time sequence states through a linked list TestLinkedList according to the input time sequence, wherein the linked list TestLinkedList represents the time sequence conversion relation between the training time sequence states
State level coverage calculation: obtaining a dimension reduction classification result of the training data and the test data, and comparing a training state base number and a test state base number of the classification result to obtain a state level coverage rate, wherein the step 4 further comprises the following steps:
step 51: for the training time sequence state, a list TrainState is defined, the list TrainList is traversed, the TrainState records the type of the attribute list2 in each element in the list TrainList, the length of the TrainState is calculated, and the number of states covered by the training time sequence state in the plane P can be calculated and is recorded as TrainStaenumber.
Step 52: for testing the time sequence state, traversing the list TestList, and calculating the intersection of the second element list2 and TrainState in the list TestList, i.e. the same number of TrainAndStastState Numbers as the second element list2 and TrainState in the list TestList.
Step 53: and calculating the ratio of TrainAndTestStatemumberber to TestStatemumber to obtain the state level coverage BSCov.
Step 54: calculating the number TrainNumber and TestNumber of input training time sequence state and test time sequence state, calculating the ratio TrainWeight of each state in TrainState to TrainNumber, and calculating the ratio TrainAndStastStatmumber to TrainNumber TrainAndStatmWeight.
Step 55: and calculating the ratio of each state weight sum in TrainAndTestweight to each state weight sum in Testweight to obtain the weighted state level coverage WSCov.
Conversion level coverage calculation: after obtaining the time sequence connection result, analyzing the result, comparing the training time sequence conversion with the testing time sequence conversion, and obtaining the conversion level coverage rate, wherein the step 6 further comprises the following steps:
step 61: for the training time sequence state, the time sequence linked list TrainLinkedList obtained in the step 3 is utilized, for every two adjacent elements in the TrainLinkedList, a directed edge can be obtained, a list TrainChange is created, the elements in the list represent the directed edges obtained by the two adjacent elements in the TrainLinkedList, and the TrainChange number can be obtained by calculating the length of the TrainChange.
Step 62: for testing the time sequence state, a time sequence linked list TestLinkedList obtained in the step 3 is utilized, for each two adjacent elements in the TestLinkedList, a directed edge can be obtained, a list TestChange is created, the elements in the list represent the directed edges obtained by the two adjacent elements in the TestLinkedList, and the TestAndCrainChange number which is the same as the TrainChange in the TestChange is calculated.
And step 63: and calculating the ratio of the TestAndTracChangeNumber to the TrainChangeNumber to obtain the conversion level coverage BTCov of the conversion level coverage.
Step 64: for TrainChange obtained in step 51, a definition list TrainOut is defined, which indicates the number of starting vertices from a vertex in TrainChange, the first element in TrainOut indicates the starting vertex, and the second element indicates the number of edges from the vertex.
Step 65: for TestChange obtained in step 52, a list TestOut is defined, which indicates the number of starting vertices from a vertex in TestChange, the first element in TestOut indicates the starting vertex, and the second element indicates the number of edges from the starting vertex.
And step 66: and calculating the number of each type edge in the TrainChange, and calculating the ratio of each type edge to the number of top outgoing edges TrainOut of each directed edge to obtain TrainOutWeigh which represents the corresponding weight of each edge.
Step 67: and calculating the number of the same edges in the TestChange and the TrainChange, and calculating the ratio of the same edges to the number of the top outgoing edges TestOut of each same edge to obtain TestAndTracOutWeigh which represents the corresponding weight of the same edges.
Step 68: and calculating the ratio of each weight sum in the TestAndTrainOutWeigh to each weight sum in the TrainOutWeigh to obtain the weighted conversion level coverage WTCov.
The coverage rate calculation method based on the stateful deep neural network comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, wherein when the computer program is loaded to the processor, the coverage rate calculation method based on the stateful deep neural network is realized.
The invention principle is as follows: the main principle of the invention is to calculate the coverage rate of the state level and the conversion level of the model by observing the distribution condition of the test sample data space in the training sample data space, observing the fitting performance of the time sequence relationship of the test sample and the time sequence relationship of the training sample and taking the distribution condition of the training sample in the data space and the time sequence relationship between the training samples as the reference. Compared with neuron coverage, the method brings the time sequence of the stateful deep neural network into coverage consideration, and solves the problem that the stateful deep neural network cannot obtain better feedback about the time sequence from the neuron coverage. By the method, the stateful deep neural network can be tested, and the accuracy and the robustness of the model can be specifically known.
The invention has the beneficial effects that:
the invention carries out dimensionality reduction processing on training data and test data, establishes a critical point plane for classification and time sequence analysis, carries out state analysis on the training data and the test data through a classification result, carries out conversion analysis on the training data and the test data through a time sequence result, and can preliminarily measure the training effect of the model through the state analysis and the conversion analysis.
The method can solve some defects of neuron coverage, can better adapt to the state deep neural network by incorporating the time sequence relation among samples into the coverage criterion, can analyze according to the state and time sequence characteristics of the state deep neural network, and realizes the state level coverage rate calculation and the conversion level coverage rate calculation of the state deep neural network, thereby having certain understanding on the accuracy and the robustness of the current model.
Drawings
FIG. 1 is an overall process diagram of an embodiment of the present invention.
FIG. 2 is a flowchart of a method according to an embodiment of the present invention.
Fig. 3 is a plane established based on four critical points.
Fig. 4 is a table listing training timing states.
FIG. 5 is a table listing test timing states.
FIG. 6 is a linked list obtained by performing timing concatenation on training timing states.
FIG. 7 is a linked list obtained by performing timing connection on test timing states.
Detailed Description
The present invention is further illustrated by the following examples, which are intended to be purely exemplary and are not intended to limit the scope of the invention, as various equivalent modifications of the invention will occur to those skilled in the art upon reading the present disclosure and fall within the scope of the appended claims.
As shown in fig. 2, the embodiment of the present invention takes an example of a specific LSTM for use, and in a python environment, describes detailed steps of a coverage calculation method based on a stateful deep neural network disclosed in the embodiment of the present invention, specifically as follows:
step 1: and deploying an offline trained RNN training model and a static data set, and confirming the state number m to be determined and the test data. And loading the model, inputting training data and test data, and collecting a training time sequence state and a test time sequence state.
Step 11: establishing an LSTM model, setting corresponding parameters such as return _ sequence ═ True to obtain hiddenstate, and storing a training data set and a testing data set.
Step 12: the model is loaded by a load () function, and the training and test timing states of the training and test data sets are obtained by a predict () function.
Step 2: and reducing the dimensions of the collected training time sequence state and the collected testing time sequence state, extracting a critical point after dimension reduction, establishing a plane P by using the critical point, and dividing the plane P into a plurality of states.
Step 21: reducing the dimension of the data by a PCA method, and setting the dimension parameter to be 2
Step 22: and carrying out dimensionality reduction on the training time sequence state and the testing time sequence state to obtain dimensionality-reduced data. The following were used:
training data:
[[0.53891901 -0.08390925][-0.27906484 -0.08726882][0.07459255 0.19205766][-0.08626248 0.01626995][0.02121108 -0.00710193][-0.26939532 -0.03004761]]
test data:
[[0.146696 0.05898452][0.25280694 -0.03033121][-0.19534093 0.01721164][-0.21780353 -0.05641372][-0.22871543 0.02646384][0.24235696 -0.01591508]]
step 23: traversing the dimension reduction data to find the maximum value and the minimum value in the x and y directions
MAX_X:0.5389190128669967
MIN_X:-0.27906484365571066
MAX_Y:0.19205766331754384
MIN_Y:-0.08726882401210687
Step 24: based on the four critical points, a plane P is created, which is divided into m states, assuming that m is 6. As shown in fig. 3.
And step 3: and classifying and confirming the training time sequence state and the testing time sequence state after dimension reduction, and determining the relative positions of the training time sequence state and the testing time sequence state in a plane.
Step 31: traversing the data after dimensionality reduction, and confirming the specific position of the data in P, namely which state in P belongs to
Step 32: a list of training timing states can be obtained as in fig. 4.
TrainList＝[[t0,(2,2)],[t1,(3,1)],[t2,(0,1)],[t3,(0,0)],[t4,(0,2)],[t5,(3,1)]]
Step 33: a list of test timing states can be obtained as in fig. 5.
TestList＝[[t0,(5,0)],[t1,(0,0)],[t2,(2,5)],[t3,(1,1)],[t4,(1,1)],[t5,(0,1)]]
And 4, step 4: and according to the input time sequence, performing time sequence connection on the results after classification confirmation according to a linked list to obtain the conversion relation between the time sequence states of the training time sequence state and the test time sequence state.
Step 41: and connecting the sorted time sequence states according to the sequence of the data set entering the model.
Step 42: the training sequence states are connected in sequence to obtain a linked list, as shown in fig. 6.
TrainLinkedList＝[[2,2],[3,1],[0,1],[0,0],[0,2],[3,1]]
Step 43: the test timing states are connected in time sequence to obtain a linked list, as shown in fig. 7.
TestLinkedList＝[[5,0],[0,0],[2,5],[1,1],[1,1],[0,1]]
And 5: and (4) comparing the classification results of the training time sequence state and the testing training time sequence state according to the classification result of the step (4) to obtain a state level coverage BSCov and a weighted state level coverage WSCov of the deep neural network model.
Step 51: the TrainList is traversed to obtain
TrainState [ [2,2], [3,1], [0,1], [0,0], [0,2] ], and the number of states covered by the training sequence state in the plane P is calculated to be 5.
Step 52: traversing the list TestList to obtain
TestState [ [5,0], [0,0], [2,5], [1,1], [0,1] ], and the number of states covered by the test time series state in the plane P is calculated to be 5. Observing the TrainState and the TestState, the intersection of TrainState and TestState is [ [0,0], [0,1] ], so TrainAndStatemumber [ [ 2]
Step 53: available BSCov ═ trainandstatembuber/trainarstate ═ 0.4
Step 54: the total number of training time series states inputted, train number 6, the total number of test time series states inputted, test number 6, the ratio of each state in train state to train number, train weight [1/6,1/3,1/6,1/6,1/6], the ratio of train andteststatenumber to train number, train andtestweight [1/6,1/6]
Step 55: and calculating the ratio of the sum of each state weight in TrainAndTestweight to the sum of each state weight in Testweight to obtain the weighted state level coverage WSCov of 1/3.
Step 6: and (5) comparing the time sequence conversion relation of the classification results of the training test time sequence state and the test training time sequence state according to the time sequence connection in the step (5) to obtain the conversion level coverage BTCov and the weighted conversion level coverage WTCov of the deep neural network.
Step 61: for the training time sequence state, a training time sequence linked list TrainLinkedList can obtain a directed edge for every two adjacent elements in the TrainLinkedList, and a list TrainChange is created to store the directed edges, so that the directed edges can be obtained
TrainChange＝[[(2,2),(3,1)],[(3,1),(0,1)],[(0,1),(0,0)],[(0,0),(0,2)],[(0,2),(3,1)]]
The length of the list is calculated, and the number of transitions trainchangenumberof the training sequence state is 5.
Step 62: for the test time sequence state, a test time sequence linked list TestLinkedList can obtain a directed edge for every two adjacent elements in the TestLinkedList, and a list TestChange is created to store the directed edges, so that the directed edges can be obtained
TestChange [ [ (5,0), (0,0) ], [ (0,0), (2,5) ], [ (2,5), (1,1) ], [ (1,1), (1,1) ], [ (1,1), (0,1) ] ], the same number of edges TestAndTrainChangeNumber ═ 0 in TestChange and TrainChange is calculated.
And step 63: the ratio of testanddatachangenumber to TrainChangeNumber is calculated to obtain its conversion level coverage BTCov equal to 0.
Step 64: for the TrainChange, the number of starting vertices at a vertex in the calculation is "TrainOut [ [ (2,2),1], [ (3,1),1], [ (0,1),1], [ (0,0),1], [ (0,2),1] ]
Step 65: for TestChange, the number of starting vertices is determined from a vertex, TestOut [ [ (5,0),1], [ (0,0),1], [ (2,5),1], [ (1,1),2] ]
And step 66: calculating the number of each type edge in TrainChange, and calculating the ratio of each type edge to the outgoing edge number TrainOut to obtain the value
TrainOutWeight＝[[[(2,2),(3,1)],1],[[(3,1),(0,1)],1],[[(0,1),(0,0)],1],[[(0,0),(0,2)],1],[[(0,2),(3,1)],1]]
And step 66: calculating the number of each type edge in the TestChange, and calculating the ratio of the number of the type edge to the number of outgoing edges TestOut for each directed edge to obtain
TestOutWeight＝[[[(5,0),(0,0)],1],[[(0,0),(2,5)],1],[[(2,5),(1,1)],1],[[(1,1),(1,1)],0.5],[[(1,1),(0,1)],0.5]]
Step 67: and calculating the number of the same edges in the TestChange and the TrainChange, and calculating the ratio of the same edges to the number of the top outgoing edges TestOut for each same edge to obtain the TestAndTrainOutWeigh as 0.
Step 68: and calculating the ratio of each weight sum in the TestAndTrainOutWeigh to each weight sum in the TrainOutWeigh to obtain the weighted conversion level coverage WTCov which is 0.
The state level coverage BSCov of the model is 0.4, the weighted state transition coverage WSCov is 1/3, the transition level coverage BTCov is 0, and the weighted state level coverage is 0. The coverage rate of the model is found to be low, which indicates that the model is low in correctness and robustness, and the model needs to be further trained. The method is used as a measuring criterion and is mainly used for measuring the quality of the model, and the improvement of the quality of the model has little relation with the method.
Based on the same inventive concept, the coverage calculation device based on the stateful deep neural network disclosed by the embodiment of the invention comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, and when the computer program is loaded to the processor, the coverage calculation method based on the stateful deep neural network is realized.
Claims (4)
1. A method for calculating coverage rate of a stateful deep neural network is characterized by comprising the following steps:
step 1: deploying an RNN training model and a static data set which are trained off line, confirming the number m x m of states to be determined and test data, loading the model, inputting the training data and the test data, and collecting a training time sequence state and a test time sequence state;
step 2: reducing the dimensions of the collected training time sequence state and the collected testing time sequence state, extracting a critical point after dimension reduction, establishing a plane P by using the critical point, and dividing the plane P into a plurality of states;
and step 3: carrying out classification confirmation on the training time sequence state and the testing time sequence state after dimension reduction, and determining the relative positions of the training time sequence state and the testing time sequence state in a plane;
and 4, step 4: according to the input time sequence, performing time sequence connection on the results after classification confirmation according to a linked list to obtain the conversion relation between the time sequence states of the training time sequence state and the test time sequence state, specifically:
step 41, respectively establishing a train timing sequence state and a test timing sequence state as a timing sequence linked list TrainLinkedList and a test LinkedList, and obtaining the timing sequence relation among the states after classification confirmation according to the input timing sequence;
step 42: connecting training time sequence states through a chain table TrainLinkedList according to the input time sequence, wherein the connected chain table TrainLinkedList represents the time sequence conversion relation among the training time sequence states;
step 43: connecting the test time sequence states through a linked list TestLinkedList according to the input time sequence, wherein the linked list TestLinkedList represents the time sequence conversion relation among the training time sequence states;
and 5: according to the classification result of the step 4, comparing the classification results of the training time sequence state and the testing training time sequence state to obtain a state level coverage BSCov and a weighted state level coverage WSCov of the deep neural network model;
step 6: according to the time sequence connection in the step 5, comparing the time sequence conversion relation of the classification results of the training test time sequence state and the test training time sequence state to obtain the conversion level coverage BTCov and the weighted conversion level coverage WTCov of the deep neural network, which specifically comprises the following steps:
step 61: for the training time sequence state, a time sequence linked list TrainLinkedList obtained in the step 3 is utilized, for every two adjacent elements in the TrainLinkedList, a directed edge can be obtained, a list TrainChange is created, the elements in the list represent the directed edges obtained by the two adjacent elements in the TrainLinkedList, and the length of the TrainChange is calculated, so that the number TrainChange number of the conversion times of the training time sequence data can be obtained;
step 62: for testing the time sequence state, utilizing the time sequence linked list TestLinkedList obtained in the step 3, obtaining a directed edge for every two adjacent elements in the TestLinkedList, creating a list TestChange, wherein the elements in the list represent the directed edges obtained by the two adjacent elements in the TestLinkedList, and calculating the TestAndCrainChange number which is the same as the TrainChange in the TestChange;
and step 63: calculating the ratio of the TestAndTracChangeNumber to the TrainChangeNumber to obtain the conversion level coverage BTCov of the conversion level coverage BTCov;
step 64: for the TrainChange obtained in step 51, a definition list TrainOut is defined, which indicates the number of the TrainChange with a certain vertex as a starting vertex, the first element in the TrainOut indicates the starting vertex, and the second element indicates the number of edges with the vertex as the starting vertex;
step 65: for TestChange obtained in step 52, defining a list TestOut, which indicates the number of the TestChange using a certain vertex as a starting vertex, wherein the first element in the TestOut indicates the starting vertex, and the second element indicates the number of edges using the vertex as the starting vertex;
and step 66: calculating the number of each type edge in the TrainChange, and calculating the ratio of each type edge to the number of top outgoing edges TrainOut of each directed edge to obtain TrainOutWeigh which represents the corresponding weight of each edge;
step 67: calculating the number of the same edges in the TestChange and the TrainChange, and calculating the ratio of the same edges to the number of the top outgoing edges TestOut for each same edge to obtain TestAndTracOutWeigh which represents the corresponding weight of the same edge;
step 68: and calculating the ratio of each weight sum in the TestAndTrainOutWeigh to each weight sum in the TrainOutWeigh to obtain the weighted conversion level coverage WTCov.
2. The method according to claim 1, wherein the step 2 specifically comprises the following steps:
step 21: performing data conversion processing on the data types of the training time sequence state and the testing time sequence state to enable the data types to meet the requirement of dimension reduction processing;
step 22: performing dimensionality reduction on the processed data, and reducing the processed data to 2 dimensions by using a PCA (principal component analysis) method, so that the original high-dimensional data is mapped onto a 2-dimensional plane;
step 23: scanning the time sequence state after dimensionality reduction, and finding out the maximum value MAX and the minimum value MIN in the x and y directions;
step 24: and (MAX _ X,0) (MIN _ X,0) (0, MAX _ Y) (0, MIN _ Y) is used for establishing a plane P, m equal division is carried out on the X direction of the plane P, m equal division is carried out on the Y direction of the plane P, and the whole plane is divided into m states.
3. The method according to claim 1, wherein the step 3 specifically comprises the following steps:
step 31: traversing the training time sequence state and the testing time sequence state after dimension reduction, and comparing the state in which the value of the training time sequence state and the testing time sequence state is positioned in the plane P;
step 32: establishing a list TrainList for the training time sequence state, wherein each element in the list comprises two attributes list1 and list2, the first attribute list1 represents the training time sequence state, and the second attribute list2 represents which state of the training time sequence state is located in the plane P, namely the corresponding subscript x, y;
step 33: a list TestList is built for the test timing states, each element of the list containing two attributes list1, list2, the first attribute list1 indicating the test timing state and the second attribute list2 indicating which state in P the test timing state is located in, i.e. the corresponding index x, y.
4. The method according to claim 1, wherein the step 5 comprises the following steps:
step 51: for the training time sequence state, defining a list TrainState, traversing the list TrainList, recording the type of the attribute list2 in each element in the list TrainList by the TrainState, and calculating the length of the TrainState, namely calculating the number of states covered by the training time sequence state in a plane P and recording the number as TrainStaenumber;
step 52: for the test time sequence state, traversing the list TestList, and calculating the intersection of the second element list2 and TrainState in the list TestList, namely the same number of TrainAndStatEnumber of the second element list2 and TrainState in the list TestList;
step 53: calculating the ratio of TrainAndTestStatemumber to TestStatEnumber to obtain state level coverage BSCov;
step 54: calculating the number TrainNumber and TestNumber of input training time sequence states and test time sequence states, calculating the ratio TrainWeight of each state in the TrainState to the TrainNumber, and calculating the ratio TrainAndStastTimeber to the TrainNumber;
step 55: and calculating the ratio of each state weight sum in TrainAndTestweight to each state weight sum in Testweight to obtain the weighted state level coverage WSCov.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010662117.1A CN112035338B (en) | 2020-07-10 | 2020-07-10 | Coverage rate calculation method of stateful deep neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010662117.1A CN112035338B (en) | 2020-07-10 | 2020-07-10 | Coverage rate calculation method of stateful deep neural network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112035338A CN112035338A (en) | 2020-12-04 |
| CN112035338B true CN112035338B (en) | 2022-01-28 |
Family
ID=73579030
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010662117.1A Active CN112035338B (en) | 2020-07-10 | 2020-07-10 | Coverage rate calculation method of stateful deep neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112035338B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102073586A (en) * | 2010-12-23 | 2011-05-25 | 北京航空航天大学 | Gray generalized regression neural network-based small sample software reliability prediction method |
| CN110135558A (en) * | 2019-04-22 | 2019-08-16 | 南京邮电大学 | Deep neural network testing adequacy method based on variable intensity combined test |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110110854B (en) * | 2019-04-01 | 2022-04-22 | 南京邮电大学 | Method for testing sufficiency of deep neural network based on edge state |
| CN110390347B (en) * | 2019-06-10 | 2021-09-07 | 河海大学 | Condition-guided countermeasure generation test method and system for deep neural network |
| GB2586868A (en) * | 2019-09-06 | 2021-03-10 | Nec Corp | Coverage and capacity optimisation using deep reinforcement learning |
-
2020
- 2020-07-10 CN CN202010662117.1A patent/CN112035338B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102073586A (en) * | 2010-12-23 | 2011-05-25 | 北京航空航天大学 | Gray generalized regression neural network-based small sample software reliability prediction method |
| CN110135558A (en) * | 2019-04-22 | 2019-08-16 | 南京邮电大学 | Deep neural network testing adequacy method based on variable intensity combined test |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112035338A (en) | 2020-12-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109614981B (en) | Power system intelligent fault detection method and system based on spearman level-dependent convolutional neural network | |
| CN109934293B (en) | Image recognition method, device, medium and confusion perception convolutional neural network | |
| CN107766929B (en) | Model analysis method and device | |
| CN111796957B (en) | Transaction abnormal root cause analysis method and system based on application log | |
| CN109325020A (en) | Small sample application method, device, computer equipment and storage medium | |
| CN116132104A (en) | Intrusion detection method, system, equipment and medium based on improved CNN-LSTM | |
| CN108681505B (en) | Test case ordering method and device based on decision tree | |
| CN114548586A (en) | Short-term power load prediction method and system based on hybrid model | |
| CN115358481A (en) | Early warning and identification method, system and device for enterprise ex-situ migration | |
| CN116823793A (en) | Device defect detection method, device, electronic device and readable storage medium | |
| CN114882497A (en) | Method for realizing fruit classification and identification based on deep learning algorithm | |
| CN112580616B (en) | Crowd quantity determination method, device, equipment and storage medium | |
| US20210319269A1 (en) | Apparatus for determining a classifier for identifying objects in an image, an apparatus for identifying objects in an image and corresponding methods | |
| CN112035338B (en) | Coverage rate calculation method of stateful deep neural network | |
| KR20220160974A (en) | Method and system for determining abnormalities in air quality data using the ensemble structure of supervised and unsupervised learning models | |
| CN116503026B (en) | Operation and maintenance risk assessment method, system and storage medium for science and technology items | |
| CN116304721A (en) | Data standard making method and system for big data management based on data category | |
| CN116662866A (en) | End-to-end incomplete time sequence classification method based on data interpolation and characterization learning | |
| CN115756919A (en) | Root cause positioning method and system for multidimensional data | |
| CN113240213B (en) | Method, device and equipment for selecting people based on neural network and tree model | |
| CN114186644A (en) | Defect report severity prediction method based on optimized random forest | |
| Kvinge et al. | Sheaves as a framework for understanding and interpreting model fit | |
| Premalatha et al. | Software Fault Prediction and Classification using Cost based Random Forest in Spiral Life Cycle Model. | |
| Borkar et al. | Comparative study of supervised learning algorithms for fake news classification | |
| CN111382191A (en) | Machine learning identification method based on deep learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |