CN111858341A - Test data measurement method based on neuron coverage - Google Patents
Test data measurement method based on neuron coverage Download PDFInfo
- Publication number
- CN111858341A CN111858341A CN202010712762.XA CN202010712762A CN111858341A CN 111858341 A CN111858341 A CN 111858341A CN 202010712762 A CN202010712762 A CN 202010712762A CN 111858341 A CN111858341 A CN 111858341A
- Authority
- CN
- China
- Prior art keywords
- test
- data
- test data
- module
- measurement
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3688—Test management for test execution, e.g. scheduling of test suites
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Debugging And Monitoring (AREA)
Abstract
A test data measurement method based on neuron coverage is characterized in that a large amount of data which are not artificially labeled are subjected to priority ranking according to certain measurement, so that the cost of artificial data labeling is reduced. The invention comprises three components: the device comprises a model data module, an algorithm measurement module and a result display module. The priority ranking of a large amount of test data is implemented through a test data measurement algorithm, and a test priority technology is mainly adopted. The invention can effectively improve the reasoning performance of the model and greatly reduce the cost of manual data marking.
Description
Technical Field
The invention belongs to the field of software testing, and particularly relates to test data measurement and test priority sequencing. A large amount of data which are not manually marked can be prioritized according to a certain measure, so that the cost of manual data marking is reduced.
Background
Deep Neural Networks (DNNs) have been widely deployed in numerous software systems. DNN has shown surprising effects in certain specific areas, such as face recognition, data prediction, etc. At the same time, DNN may exhibit erroneous behavior under certain inputs, which may lead to significant accidents and even loss of life under certain safety-critical situations. Therefore, how to guarantee the reliability of DNN becomes a critical issue. In consideration of the limitation of formal verification and low engineering efficiency, the current method for guaranteeing the reliability of the software system is to perform large-scale software test. For a neural network software system, the acquisition of the data set tags is generally more difficult than the acquisition of the data set itself, which requires a lot of manpower for the tagging work. This not only increases the development cost of the entire project, but also slows down the development efficiency of the project.
However, generating the correct test label for a given input is generally not available in automated testing. To obtain test label information, the DNN system-based test task typically requires a significant amount of manpower to label the test data, which greatly slows the progress of software quality assurance. At present, a widely popular solution is to filter a data set with a large total amount, select a test case from the data set, where the test case is effective in improving the system performance, and then manually label the test case, so that the cost of data labeling can be reduced to the greatest extent. The idea is based on the fact that DNN-based intelligent software systems are often already capable of performing expected behavior on most input data, and therefore do not necessarily label the entire test set data. The key problem with this approach is how to measure the validity of the test case. Considering that the DNN software system is different from the traditional software system, and it does not have a clear business logic, the methods based on branch coverage, data flow, etc. in the traditional software testing field are not suitable for the intelligent software system. The test measurement platform starts from the characteristic that an intelligent software system is driven by data, combines measurement indexes such as neuron coverage and neuron activation diversity, and can perform priority sequencing on multi-type data according to a certain measurement standard. Data annotation personnel only need to manually label the previous part of the sorted test set, and the reliability of the software system can be effectively improved due to the part of test data. The test measurement platform can effectively reduce a large amount of unnecessary manual labeling work, and meanwhile, the reliability quality guarantee work of the intelligent software system cannot be negatively affected.
Disclosure of Invention
The invention aims to solve the problems that: to ensure that the deep neural network system can correctly process in a service scenario, a large amount of test data needs to be labeled manually, which results in extremely high project development cost and slows down the development efficiency of software projects.
The technical scheme of the invention is as follows: a test data priority ranking algorithm based on neuron measurement is characterized in that a large amount of input test data is ranked according to coverage rate of neurons, test cases are ranked in front of the test cases when the coverage rate is higher, and then the previous test data is marked manually. The method comprises the following three modules:
1) a model data module: the model file system is used for receiving a data set and a model file submitted by a user, and the model can be a trained model file or a definition file of the model. The main complexity of this module is in supporting different deep learning frameworks. In order to reduce the complexity of engineering, a user only needs to describe the model structure by using a predefined description language, and does not need to write codes by using a specific deep learning framework. The code translation module can automatically convert the description language into the implementation code of the specified frame, so that one-time writing can be realized, and the code translation module can run on any deep learning frame.
2) An algorithm measurement module: the module integrates a key test data measurement algorithm, and the number of neurons covered by the test data can be obtained by calculating the neuron activation coverage rate of the neural network hidden layer. By performing such calculation on the entire data set and finally sorting the data set according to the standard, valuable test data for the network model can be obtained.
3) And a result display module: the module is mainly responsible for front-end interaction components, for example, displaying the final sorting result and the CAM and CTM indexes of the part of test cases, so that a user can better understand the sorting process of the test priority.
The invention is characterized in that:
1. the idea of testing priority is introduced into neural network testing for the first time.
2. The model and a specific deep learning framework are decoupled for the first time, and the model can be designed once and can be operated on each platform.
3. An interactive real-time response front end is provided, and results after test sequencing are better visualized.
Drawings
Fig. 1 is a general flow chart of the implementation of the present invention.
FIG. 2 is a diagram of a pretreatment process.
Fig. 3 is an algorithmic metrology process.
Detailed Description
The priority ranking of a large amount of test data is implemented through a test data measurement algorithm, a test priority technology is mainly adopted, and specific key technologies comprise a deep Convolutional Neural Network (CNN), neuron coverage, information entropy, an average error detection rate (APFD) and the like.
1. Model-defined translation
In the invention, a unified intermediate language is defined to complete the construction work of the user model logic, specifically, a user can define a model structure in an XML mode, and a code translator can automatically convert an XML model definition file into a Python code under a corresponding deep learning framework. Thereafter, each new framework is adapted by writing the corresponding transcoding logic.
2. Measurement of test data
In the invention, an activation threshold value is set, then the output value of each neuron in a given test sample is recorded, whether the neuron is activated or not is judged according to the threshold value, and the number of the neurons activated by the test case is calculated at the same time, so that the coverage rate is calculated.
3. Measurement result display
In the present invention, echart and other ways are adopted to show the measured results, including the actual CAM and CTM values and APFD, and the test data of Top10 is shown in a visual way, so as to facilitate the user to have a better understanding of the characteristics of the test data.
In this example, the measurement algorithm uses the CIFAR-10 dataset on the task of image classification, which can help software developers reduce the cost of manual data labeling by about 50%, and meanwhile, by retraining the sorted data, the performance of the model can be improved by about 5%. Therefore, the method can effectively improve the reasoning performance of the model and greatly reduce the cost of manual data annotation.
Claims (4)
1. A test data measurement method based on neuron coverage is characterized in that a large amount of input test data is sequenced according to coverage rate of neurons, test cases are arranged in front of the test cases when the coverage rate is higher, and then the previous part of the test data is marked manually.
2. The model data module of claim 1, wherein the user only needs to describe the model structure in a predefined description language, without using a specific deep learning framework for code writing; the code translation module can automatically convert the description language into the implementation code of the specified frame, so that one-time writing can be realized, and the code translation module can run on any deep learning frame.
3. The algorithmic measurement module of claim 1, wherein the neuron activation coverage of hidden layers of a neural network is calculated by integrating a critical test data measurement algorithm to determine how many neurons the test data can cover; the calculation is carried out on the whole data set, and finally the data set is sequenced according to the standard, so that valuable test data for the network model can be obtained.
4. The result presentation module of claim 1, wherein the front-end interaction component is designed to present the final sorting result and the CAM and CTM indicators of the part of the test case, for example, so that the user can better understand the sorting process of the test priority.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202010712762.XA CN111858341A (en) | 2020-07-23 | 2020-07-23 | Test data measurement method based on neuron coverage |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202010712762.XA CN111858341A (en) | 2020-07-23 | 2020-07-23 | Test data measurement method based on neuron coverage |
Publications (1)
Publication Number | Publication Date |
---|---|
CN111858341A true CN111858341A (en) | 2020-10-30 |
Family
ID=72949274
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202010712762.XA Pending CN111858341A (en) | 2020-07-23 | 2020-07-23 | Test data measurement method based on neuron coverage |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN111858341A (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113255810A (en) * | 2021-06-04 | 2021-08-13 | 杭州新州网络科技有限公司 | Network model testing method based on key decision logic design test coverage rate |
Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN105446885A (en) * | 2015-12-28 | 2016-03-30 | 西南大学 | Regression testing case priority ranking technology based on needs |
CN109710514A (en) * | 2018-12-10 | 2019-05-03 | 江苏大学 | The solution and system of tie-breaking in priorities of test cases sequence |
CN110134588A (en) * | 2019-04-16 | 2019-08-16 | 江苏大学 | A kind of priorities of test cases sort method and test macro based on code and combined covering |
CN110135558A (en) * | 2019-04-22 | 2019-08-16 | 南京邮电大学 | Deep neural network testing adequacy method based on variable intensity combined test |
CN110610193A (en) * | 2019-08-12 | 2019-12-24 | 大箴(杭州)科技有限公司 | Method and device for processing labeled data |
CN111061626A (en) * | 2019-11-18 | 2020-04-24 | 北京工业大学 | Test case priority ranking method based on neuron activation frequency analysis |
CN111191732A (en) * | 2020-01-03 | 2020-05-22 | 天津大学 | Target detection method based on full-automatic learning |
CN111382055A (en) * | 2018-12-29 | 2020-07-07 | 贝壳技术有限公司 | Automatic unit testing method and device based on unified description language |
-
2020
- 2020-07-23 CN CN202010712762.XA patent/CN111858341A/en active Pending
Patent Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN105446885A (en) * | 2015-12-28 | 2016-03-30 | 西南大学 | Regression testing case priority ranking technology based on needs |
CN109710514A (en) * | 2018-12-10 | 2019-05-03 | 江苏大学 | The solution and system of tie-breaking in priorities of test cases sequence |
CN111382055A (en) * | 2018-12-29 | 2020-07-07 | 贝壳技术有限公司 | Automatic unit testing method and device based on unified description language |
CN110134588A (en) * | 2019-04-16 | 2019-08-16 | 江苏大学 | A kind of priorities of test cases sort method and test macro based on code and combined covering |
CN110135558A (en) * | 2019-04-22 | 2019-08-16 | 南京邮电大学 | Deep neural network testing adequacy method based on variable intensity combined test |
CN110610193A (en) * | 2019-08-12 | 2019-12-24 | 大箴(杭州)科技有限公司 | Method and device for processing labeled data |
CN111061626A (en) * | 2019-11-18 | 2020-04-24 | 北京工业大学 | Test case priority ranking method based on neuron activation frequency analysis |
CN111191732A (en) * | 2020-01-03 | 2020-05-22 | 天津大学 | Target detection method based on full-automatic learning |
Non-Patent Citations (2)
Title |
---|
FENGYANG等: "DeepGini: Prioritizing Massive Tests to Enhance the Robustness of Deep Neural Networks", 《ARXIV:HTTPS://ARXIV.ORG/ABS/1903.00661》 * |
刘二火: "DeepGini:优先进行大规模测试以增强深度神经网络的鲁棒性", 《CSDN:HTTPS://BLOG.CSDN.NET/WEIXIN_43482279/ARTICLE/DETAILS/107026244》 * |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113255810A (en) * | 2021-06-04 | 2021-08-13 | 杭州新州网络科技有限公司 | Network model testing method based on key decision logic design test coverage rate |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11790256B2 (en) | Analyzing test result failures using artificial intelligence models | |
CN113884961B (en) | SOC calibration method, modeling device, computer equipment and medium | |
CN111598860A (en) | Lithium battery defect detection method based on yolov3 network embedded in self-attention door module | |
CN109087667B (en) | Voice fluency recognition method and device, computer equipment and readable storage medium | |
CN114037653B (en) | Industrial machine vision defect detection method and system based on two-stage knowledge distillation | |
CN111427775A (en) | Method level defect positioning method based on Bert model | |
CN112419268A (en) | Method, device, equipment and medium for detecting image defects of power transmission line | |
CN112035345A (en) | Mixed depth defect prediction method based on code segment analysis | |
CN115456107A (en) | Time series abnormity detection system and method | |
CN113870846B (en) | Speech recognition method, device and storage medium based on artificial intelligence | |
CN116361147A (en) | Method for positioning root cause of test case, device, equipment, medium and product thereof | |
CN111858341A (en) | Test data measurement method based on neuron coverage | |
CN117217163B (en) | Script-based SOC chip testing method | |
CN117009223A (en) | Software testing method, system, storage medium and terminal based on abstract grammar | |
CN116910657A (en) | Fault diagnosis method and equipment based on unsupervised learning | |
CN111858348A (en) | Test data measurement method based on neural network output vector | |
CN114708470A (en) | Illegal picture identification method, medium and computing device | |
CN116883709A (en) | Carbonate fracture-cavity identification method and system based on channel attention mechanism | |
CN113360649A (en) | Flow error control method and system based on natural language processing in RPA system | |
Choi et al. | Just-in-Time Defect Prediction for Self-driving Software via a Deep Learning Model | |
CN113139187B (en) | Method and device for generating and detecting pre-training language model | |
CN112631930B (en) | Dynamic system testing method and related device | |
CN116071768A (en) | Table identification method, apparatus, electronic device and storage medium | |
CN117251376B (en) | Software defect prediction method and system | |
US20230222360A1 (en) | Context similarity detector for artificial intelligence |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
WD01 | Invention patent application deemed withdrawn after publication | ||
WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20201030 |