CN111858341A

CN111858341A - Test data measurement method based on neuron coverage

Info

Publication number: CN111858341A
Application number: CN202010712762.XA
Authority: CN
Inventors: 房春荣; 张子杰; 高新宇; 尹伊宁
Original assignee: Shenzhen Muzhi Technology Co ltd
Current assignee: Shenzhen Muzhi Technology Co ltd
Priority date: 2020-07-23
Filing date: 2020-07-23
Publication date: 2020-10-30

Abstract

A test data measurement method based on neuron coverage is characterized in that a large amount of data which are not artificially labeled are subjected to priority ranking according to certain measurement, so that the cost of artificial data labeling is reduced. The invention comprises three components: the device comprises a model data module, an algorithm measurement module and a result display module. The priority ranking of a large amount of test data is implemented through a test data measurement algorithm, and a test priority technology is mainly adopted. The invention can effectively improve the reasoning performance of the model and greatly reduce the cost of manual data marking.

Description

Test data measurement method based on neuron coverage

Technical Field

The invention belongs to the field of software testing, and particularly relates to test data measurement and test priority sequencing. A large amount of data which are not manually marked can be prioritized according to a certain measure, so that the cost of manual data marking is reduced.

Background

Deep Neural Networks (DNNs) have been widely deployed in numerous software systems. DNN has shown surprising effects in certain specific areas, such as face recognition, data prediction, etc. At the same time, DNN may exhibit erroneous behavior under certain inputs, which may lead to significant accidents and even loss of life under certain safety-critical situations. Therefore, how to guarantee the reliability of DNN becomes a critical issue. In consideration of the limitation of formal verification and low engineering efficiency, the current method for guaranteeing the reliability of the software system is to perform large-scale software test. For a neural network software system, the acquisition of the data set tags is generally more difficult than the acquisition of the data set itself, which requires a lot of manpower for the tagging work. This not only increases the development cost of the entire project, but also slows down the development efficiency of the project.

However, generating the correct test label for a given input is generally not available in automated testing. To obtain test label information, the DNN system-based test task typically requires a significant amount of manpower to label the test data, which greatly slows the progress of software quality assurance. At present, a widely popular solution is to filter a data set with a large total amount, select a test case from the data set, where the test case is effective in improving the system performance, and then manually label the test case, so that the cost of data labeling can be reduced to the greatest extent. The idea is based on the fact that DNN-based intelligent software systems are often already capable of performing expected behavior on most input data, and therefore do not necessarily label the entire test set data. The key problem with this approach is how to measure the validity of the test case. Considering that the DNN software system is different from the traditional software system, and it does not have a clear business logic, the methods based on branch coverage, data flow, etc. in the traditional software testing field are not suitable for the intelligent software system. The test measurement platform starts from the characteristic that an intelligent software system is driven by data, combines measurement indexes such as neuron coverage and neuron activation diversity, and can perform priority sequencing on multi-type data according to a certain measurement standard. Data annotation personnel only need to manually label the previous part of the sorted test set, and the reliability of the software system can be effectively improved due to the part of test data. The test measurement platform can effectively reduce a large amount of unnecessary manual labeling work, and meanwhile, the reliability quality guarantee work of the intelligent software system cannot be negatively affected.

Disclosure of Invention

The invention aims to solve the problems that: to ensure that the deep neural network system can correctly process in a service scenario, a large amount of test data needs to be labeled manually, which results in extremely high project development cost and slows down the development efficiency of software projects.

The technical scheme of the invention is as follows: a test data priority ranking algorithm based on neuron measurement is characterized in that a large amount of input test data is ranked according to coverage rate of neurons, test cases are ranked in front of the test cases when the coverage rate is higher, and then the previous test data is marked manually. The method comprises the following three modules:

1) a model data module: the model file system is used for receiving a data set and a model file submitted by a user, and the model can be a trained model file or a definition file of the model. The main complexity of this module is in supporting different deep learning frameworks. In order to reduce the complexity of engineering, a user only needs to describe the model structure by using a predefined description language, and does not need to write codes by using a specific deep learning framework. The code translation module can automatically convert the description language into the implementation code of the specified frame, so that one-time writing can be realized, and the code translation module can run on any deep learning frame.

2) An algorithm measurement module: the module integrates a key test data measurement algorithm, and the number of neurons covered by the test data can be obtained by calculating the neuron activation coverage rate of the neural network hidden layer. By performing such calculation on the entire data set and finally sorting the data set according to the standard, valuable test data for the network model can be obtained.

3) And a result display module: the module is mainly responsible for front-end interaction components, for example, displaying the final sorting result and the CAM and CTM indexes of the part of test cases, so that a user can better understand the sorting process of the test priority.

The invention is characterized in that:

1. the idea of testing priority is introduced into neural network testing for the first time.

2. The model and a specific deep learning framework are decoupled for the first time, and the model can be designed once and can be operated on each platform.

3. An interactive real-time response front end is provided, and results after test sequencing are better visualized.

Drawings

Fig. 1 is a general flow chart of the implementation of the present invention.

FIG. 2 is a diagram of a pretreatment process.

Fig. 3 is an algorithmic metrology process.

Detailed Description

The priority ranking of a large amount of test data is implemented through a test data measurement algorithm, a test priority technology is mainly adopted, and specific key technologies comprise a deep Convolutional Neural Network (CNN), neuron coverage, information entropy, an average error detection rate (APFD) and the like.

1. Model-defined translation

In the invention, a unified intermediate language is defined to complete the construction work of the user model logic, specifically, a user can define a model structure in an XML mode, and a code translator can automatically convert an XML model definition file into a Python code under a corresponding deep learning framework. Thereafter, each new framework is adapted by writing the corresponding transcoding logic.

2. Measurement of test data

In the invention, an activation threshold value is set, then the output value of each neuron in a given test sample is recorded, whether the neuron is activated or not is judged according to the threshold value, and the number of the neurons activated by the test case is calculated at the same time, so that the coverage rate is calculated.

3. Measurement result display

In the present invention, echart and other ways are adopted to show the measured results, including the actual CAM and CTM values and APFD, and the test data of Top10 is shown in a visual way, so as to facilitate the user to have a better understanding of the characteristics of the test data.

In this example, the measurement algorithm uses the CIFAR-10 dataset on the task of image classification, which can help software developers reduce the cost of manual data labeling by about 50%, and meanwhile, by retraining the sorted data, the performance of the model can be improved by about 5%. Therefore, the method can effectively improve the reasoning performance of the model and greatly reduce the cost of manual data annotation.

Claims

1. A test data measurement method based on neuron coverage is characterized in that a large amount of input test data is sequenced according to coverage rate of neurons, test cases are arranged in front of the test cases when the coverage rate is higher, and then the previous part of the test data is marked manually.

2. The model data module of claim 1, wherein the user only needs to describe the model structure in a predefined description language, without using a specific deep learning framework for code writing; the code translation module can automatically convert the description language into the implementation code of the specified frame, so that one-time writing can be realized, and the code translation module can run on any deep learning frame.

3. The algorithmic measurement module of claim 1, wherein the neuron activation coverage of hidden layers of a neural network is calculated by integrating a critical test data measurement algorithm to determine how many neurons the test data can cover; the calculation is carried out on the whole data set, and finally the data set is sequenced according to the standard, so that valuable test data for the network model can be obtained.

4. The result presentation module of claim 1, wherein the front-end interaction component is designed to present the final sorting result and the CAM and CTM indicators of the part of the test case, for example, so that the user can better understand the sorting process of the test priority.