CN112527676A - Model automation test method, device and storage medium - Google Patents
Model automation test method, device and storage medium Download PDFInfo
- Publication number
- CN112527676A CN112527676A CN202011535850.3A CN202011535850A CN112527676A CN 112527676 A CN112527676 A CN 112527676A CN 202011535850 A CN202011535850 A CN 202011535850A CN 112527676 A CN112527676 A CN 112527676A
- Authority
- CN
- China
- Prior art keywords
- model
- test
- tested
- code
- testing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000010998 test method Methods 0.000 title claims description 6
- 238000012360 testing method Methods 0.000 claims abstract description 255
- 238000000034 method Methods 0.000 claims description 24
- 238000011156 evaluation Methods 0.000 claims description 16
- 238000004458 analytical method Methods 0.000 claims description 13
- 238000013215 result calculation Methods 0.000 claims description 11
- 238000002372 labelling Methods 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 4
- 239000000126 substance Substances 0.000 claims 1
- 238000004422 calculation algorithm Methods 0.000 description 11
- 238000013135 deep learning Methods 0.000 description 9
- 238000005516 engineering process Methods 0.000 description 8
- 238000004364 calculation method Methods 0.000 description 7
- 230000008569 process Effects 0.000 description 6
- 238000013136 deep learning model Methods 0.000 description 4
- 238000001514 detection method Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000012827 research and development Methods 0.000 description 3
- 238000012916 structural analysis Methods 0.000 description 3
- 238000013473 artificial intelligence Methods 0.000 description 2
- 230000006399 behavior Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- HPTJABJPZMULFH-UHFFFAOYSA-N 12-[(Cyclohexylcarbamoyl)amino]dodecanoic acid Chemical compound OC(=O)CCCCCCCCCCCNC(=O)NC1CCCCC1 HPTJABJPZMULFH-UHFFFAOYSA-N 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 230000001934 delay Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000011056 performance test Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3684—Test management for test design, e.g. generating new test cases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3688—Test management for test execution, e.g. scheduling of test suites
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the invention relates to the technical field of model automatic testing, and discloses a model automatic testing method, which comprises the following steps: obtaining a model to be tested; configuring a test environment for the model to be tested; generating a model test code according to the model to be tested and the test environment; and executing the model test code to test the model to be tested according to a pre-stored test data set. The automatic model testing method can automatically match the required testing environment, realizes the automatic testing of the model to be tested, and has simple flow and high efficiency.
Description
Technical Field
The embodiment of the invention relates to the technical field of model automatic testing, in particular to a method and a device for model automatic testing and a storage medium.
Background
Since the development of computer technology, people always hope that machines can be intelligent like people, help people to solve problems, reduce the labor of people, and create artificial intelligence. The artificial intelligence adopts the related technology in the fields of computer vision and the like, so that the machine becomes more and more intelligent. At present, deep learning is the most advanced technology in the field of computers, and the development of deep learning technology in the technical field is hot.
In the research and development of the deep learning technology, the result is a model completed by deep learning training, and the performance of the model is good and bad, so that the final use effect is influenced. In the research and development process of the deep learning technology, the model is trained in various ways, and the quality of the model is evaluated; and continuously optimizing the model, and evaluating the quality of the model, so as to repeat and continuously iterate. Different technicians obtain the deep learning model through different technologies, and need to evaluate the relative quality of the deep learning model. Therefore, the evaluation of the model is extremely important and indispensable in the development of deep learning technology.
However, the inventor finds that the evaluation of the performance of the current model is mostly realized manually by technicians, and is not efficient.
Disclosure of Invention
The embodiment of the invention aims to provide an automatic model testing method which can automatically match a required testing environment, realizes automatic testing of a model to be tested, and has the advantages of simple flow and high efficiency.
In order to solve the above technical problem, an embodiment of the present invention provides an automated model testing method, including: obtaining a model to be tested; configuring a test environment for the model to be tested; generating a model test code according to the model to be tested and the test environment; and executing the model test code to test the model to be tested according to a pre-stored test data set.
Additionally, the test environment includes: a test platform and a model operation frame; the generating of the model test code according to the model to be tested and the test environment includes: analyzing the model to be tested according to the model operation framework to obtain model parameters and keywords of the model to be tested; determining the code type of the model test code according to the test platform; and generating a model test code according to the model parameters, the keywords and the code category.
Additionally, the model test code includes: a data set analysis code, a model operation code and a result calculation code; generating a model test code according to the model parameters, the keywords and the code category comprises the following steps: generating the data set analysis code according to the model type of the model to be tested and the code type; generating a model operation code according to the model type of the model to be tested, the model parameter, the keyword and the code type; and generating the result calculation code according to the evaluation index of the model to be tested and the code type.
In addition, the executing the model test code to test the model to be tested according to a pre-stored test data set and test the model to be tested according to a test result includes: executing the data set analysis code to call a data set of a format required by the model to be tested from the pre-stored test data set, and reading data in the data set and the labeling information of the data according to the format; executing the model operation code to input the data and the labeling information of the data into the model to be tested for testing so as to obtain the test result; and executing the result calculation code to calculate the evaluation index of the model to be tested according to the test result.
In addition, the executing the model running code inputs the data and the labeling information of the data into the model to be tested for testing, and the testing comprises the following steps: executing the model operation code to convert the data into a format required by the model to be tested; and inputting the data in the format required by the model to be tested and the labeling information of the data into the model to be tested for testing.
In addition, the testing the model to be tested according to the testing result, and then, the method further comprises the following steps: generating a test packet report, the test report at least comprising: the model to be tested, the test data set and the evaluation index.
Additionally, the test environment includes: a test platform; before configuring a test environment for the model to be tested, the method further includes: presetting a plurality of test platforms; the configuring of the test environment for the model to be tested comprises: and configuring different test platforms for the same model to be tested.
Additionally, the pre-stored test data set includes: a public data set and a private data set stored in a database.
The embodiment of the invention also provides a model automatic testing device, which comprises: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the above-described model automated testing method.
The embodiment of the invention also provides a computer readable storage medium, which stores a computer program, and the computer program realizes the automatic testing method of the model when being executed by a processor.
Compared with the prior art, the embodiment of the invention provides an automatic model testing method, which is used for obtaining a model to be tested, configuring a testing environment for the model to be tested and automatically configuring a required testing environment for the model to be tested, thereby avoiding the problem that the testing cannot be carried out due to the mismatching of the testing environment and the model to be tested; after the required test environment is automatically configured for the model to be tested, the model test code is automatically generated according to the model to be tested and the test environment, and the model test code is executed to test the model to be tested according to the pre-stored test data set.
Drawings
One or more embodiments are illustrated by way of example in the accompanying drawings, which correspond to the figures in which like reference numerals refer to similar elements and which are not to scale unless otherwise specified.
FIG. 1 is a schematic flow chart of a model automated testing method according to a first embodiment of the present invention;
FIG. 2 is a schematic flow chart of a model automated testing method according to a second embodiment of the present invention;
fig. 3 is a schematic structural diagram of a model automated testing apparatus according to a third embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, it will be appreciated by those of ordinary skill in the art that numerous technical details are set forth in order to provide a better understanding of the present application in various embodiments of the present invention. However, the technical solution claimed in the present application can be implemented without these technical details and various changes and modifications based on the following embodiments.
The current model performance test method comprises the following steps: manually downloading an online public data set, and arranging the online public data set into a test data set; then, a test environment, a mounting frame and various required third party libraries are manually built; finally, test codes are compiled, reports are output, and the test reports of the multiple models are compared to obtain the conclusion of the performance of the models.
This evaluation method has many problems:
(1) the arrangement of the test data sets is time-consuming, and large-space storage is needed after the data sets are excessive, so that data management is inconvenient.
(2) The environment is built, a third party library required by compiling a plurality of test codes is required according to the configuration of a test machine, various famous errors can be encountered in the process, time and labor are wasted, and the repeated work is basically realized.
(3) Different test codes need to be written according to respective models for comparison with other models, and the test codes are time-consuming. The method is a repetitive work which wastes time and labor, and greatly delays the research and development efficiency.
In view of the above, a first embodiment of the present invention relates to an automated model testing method, which is characterized in that a model to be tested is obtained, a testing environment is configured for the model to be tested, a testing environment required for the model to be tested is automatically configured for the model to be tested, and the problem that the testing cannot be performed due to the mismatching between the testing environment and the model to be tested is avoided; after the required test environment is automatically configured for the model to be tested, the model test code is automatically generated according to the model to be tested and the test environment, and the model test code is executed to test the model to be tested according to the pre-stored test data set.
The following describes implementation details of the model automation testing method according to the present embodiment in detail, and the following is only provided for easy understanding and is not necessary for implementing the present embodiment.
The flow diagram of the model automation test method in this embodiment is shown in fig. 1:
step 101: and obtaining a model to be tested.
Specifically, a deep learning model file uploaded by a user is obtained as a model to be tested, and the obtaining mode may be obtaining a local model file of the user, obtaining a model file on mobile equipment of the user, and obtaining a model file at the cloud of the user.
Step 102: and configuring a test environment for the model to be tested.
Specifically, the test environment in this embodiment includes: a test platform and a model operation framework.
Because different models to be tested need different test environments, a variety of test platforms are preset in this embodiment, including common platforms such as a CPU, a GPU, an embedded type, a Mac multiple access computer, and an android. After the model to be tested is obtained, a proper test platform can be directly selected for a user from a plurality of preset test platforms, and the operation is simple. The model operation of the CPU test platform is realized by using a CPU; the GPU test platform is used for realizing model operation by using a GPU; the model operation of the Mac test platform is realized by using Mac equipment; and the Android test platform is implemented by using Android equipment for model operation. In practical use, other hardware platform environments may also be set, which is not illustrated in this embodiment.
In this embodiment, different hardware configurations of the test platform may be implemented in the form of hardware devices, and different hardware resources may also be scheduled by the unified platform, for example: CPU resources, GPU resources, or Android system resources on the hardware may be scheduled by a fixed server.
Because the models and formats of different depth learning frames are different, if the frame selection is not matched with the model, the analysis will fail, and the test cannot be completed, so that various algorithm frames including common depth learning algorithm frames such as TensorFlow, Caffe, Mxnet and the like can be preset in the embodiment.
In this embodiment, because the type of the model to be tested that can be tested by a testing platform is determined, the deep learning framework of all models to be tested that can be tested by the testing platform can be directly deployed on the testing platform in advance. After the model to be tested is positioned on the test platform, a user can directly select an applicable algorithm frame for the model to be tested from a plurality of preset algorithm frames, and the operation is simple.
Step 103: and generating a model test code according to the model to be tested and the test environment.
Specifically, the model test code needs to match the previously selected test platform, and the test code is different from platform to platform. For example: the CPU test platform only uses CPU resources to execute codes and only needs to compile normal codes; the GPU test platform needs to call a GPU computing unit, so that codes containing GPU programming such as CUDA are generated; the Android platform needs to generate codes which can run on the Android. Specifically, the code may be implemented in a variety of languages such as C/C + +, python, Java, and so on.
In some embodiments, the test environment includes: a test platform and a model operation frame; generating a model test code according to a model to be tested and a test environment, comprising: analyzing the model to be tested according to the model operation framework to obtain model parameters and keywords of the model to be tested; determining the code type of the model test code according to the test platform; and generating a model test code according to the model parameters, the keywords and the code category.
Specifically, before generating the model test code, the structural analysis of the model to be tested is required, so as to ensure that the generated model test code can be adapted to the model. Therefore, the embodiment provides an implementation manner of how to perform structural analysis on the model to be tested.
The deep learning model is substantially composed of multiple layers of complex neural networks, and generally includes two parts, one part is keywords for representing the network structure, and the other part is parameters for representing the network, i.e. parameters are numerical values, such as: a 3x3 convolutional layer network consists of keys representing convolutional layers and 9 numeric parameters. Therefore, the structural analysis of the model to be tested is as follows: and analyzing the model parameters according to the selected model operation framework and acquiring the keywords of the model to be tested. When analyzing the model parameters according to the model operation framework, the parameters of the model to be tested need to be read one by one according to the network structure under the model operation framework, and one model to be tested may contain tens of thousands, hundreds of thousands, millions, or even tens of millions of parameters.
For example: for the Caffe framework, the model has two files, one is a prototxt file, and the network structure of the model is stored in the prototxt file; the other is that the capacity model file and the storage parameter are all numerical values, such as 1.202, and the parameters are converted by binary.
An example of a network where the prototxt file resides is as follows:
wherein layer indicates that it is a complete network layer, named conv1, and type is Convolition, i.e. Convolution operation. Several important keywords were resolved as follows: num _ output: the number of convolution kernels; kernel _ size: the size of the convolution kernel, if the kernel _ size is different in length and width, the kernel _ h and the kernel _ w need to be set respectively; stride: the default of the step size of the convolution kernel is 1, and stride _ h and stride _ w can also be used for setting; weight _ filter: the weight is initialized, defaults to "constant", the values are all 0, and can be set to "gaussian"; bias _ filter: the offset term is initialized, typically to "constant", and all values are 0. Through the analysis, the number of the Caffe network layer convolution kernel parameters can be known, and the corresponding parameters can be read from the corresponding positions of the ca ffeemodel.
In some embodiments, the model test code includes: a data set analysis code, a model operation code and a result calculation code; generating a model test code according to the model parameters, the keywords and the code types, wherein the method comprises the following steps: generating a data set analysis code according to the model type and the code type of the model to be tested; generating a model operation code according to the model type, the model parameter, the keyword and the code type of the model to be tested; and generating a result calculation code according to the evaluation index and the code type of the model to be tested.
Specifically, since the format and labeling manner of each data set are different, for example: the face data set CelebA data is presented in a picture format, the label information is stored in a corresponding text file, and the picture and the text file (including the label information) are required to be read respectively to be used as test input information. The human behavior data set HMDB is presented in a video form, names of folders, namely label information of data, are stored, when the video data set is used, video streams need to be analyzed, and names (label information) of the folders are read and are jointly used as test input information. Therefore, in the embodiment, a dataset analysis code is generated according to the model type and the code type of the model to be tested, and the dataset analysis code is used for analyzing the format of the test dataset and reading the test data according to the format.
The annotation refers to extracting information that we want from an image, and storing the extracted information in a corresponding annotation file, and the annotation file is opened to obtain the information in the image, such as: in the face detection algorithm, whether a face exists in an image or not and where the face is located are detected. Assuming that the picture name is 1.jpg and the markup file is named as 1.txt, the format of the markup information stored in the markup file can be set as follows: num, x1, y1, w1, h1... xi, yi, wi, hi. The number of the faces in the picture is represented by num, xi represents the horizontal coordinate of the upper left corner of the ith face frame in the picture, yi represents the vertical coordinate of the upper left corner of the ith face frame in the picture, wi represents the width of the ith face frame, and hi represents the height of the ith face frame. Assuming that there are two faces, the annotation information content can be as follows: 2,15, 18, 100, 120, 205, 208, 200, 150.
After the test data is read, the test data needs to be called and input into the model to be tested to run, so that the model running code is generated according to the model type, the model parameter, the keyword and the code type of the model to be tested in this embodiment.
In some embodiments, in order to match the input format of the model to be tested, the executing the model running code in this embodiment inputs the data and the label information of the data into the model to be tested for testing, including: executing the model operation code to convert the data into a format required by the model to be tested; and inputting the data in the format required by the model to be tested and the labeling information of the data into the model to be tested for testing. For example: assuming that the input of the model to be tested is 112 × 112 images, the images of the test data set are adjusted to 112 × 112 images as input data to perform the calculation.
After the test data is input into the model to be tested for calculation, the calculation result needs to be evaluated, so the result calculation code is generated according to the evaluation index and the code type of the model to be tested in this embodiment.
Step 104: the model test code is executed to test the model to be tested according to a pre-stored test data set.
Specifically, after inputting a model to be tested, selecting a test platform and a model operation framework, an inventor can execute a model test code to test the model to be tested according to a pre-stored test data set, so that the model to be tested can be tested only by executing the input operation of the model to be tested, the manual operation process is simple, and the test efficiency is high.
Wherein the pre-stored test data set comprises: a public data set and a private data set stored in a database. Public data sets such as: for the Face recognition algorithm, common data sets such as VGG Face and CelebA are included; for the behavior recognition algorithm, common data sets such as UCF101 and HMDB are included; for the target detection algorithm, common data sets such as PASCAL VOC, MSCOCO and the like exist. And meanwhile, providing corresponding test data sets of other deep learning algorithms, such as Kaggle-DSB2018, FDDB and the like. The private data set includes: face recognition data sets, target detection data sets, object classification data sets, and the like. In the embodiment, various selectable data sets are provided for the model to be tested, and the various test data can know the performance of the model to be tested in different scenes, so that the evaluation of the quality of the model to be tested is facilitated more comprehensively.
In some embodiments, executing the model test code to test the model to be tested according to the pre-stored test data set, and testing the model to be tested according to the test result includes: executing a data set analysis code to call a data set of a format required by a model to be tested from a pre-stored test data set, and reading data in the data set and label information of the data according to the format; executing the model operation code to input the data and the labeling information of the data into a model to be tested for testing so as to obtain a test result; and the execution result calculation code calculates the evaluation index of the model to be tested according to the test result.
Specifically, after the test result is obtained, the execution result calculation code calculates the evaluation index of the model to be tested according to the test result. Among the commonly used evaluation indices are: accuracy, precision, recall, error, Average Precision (AP), mean average precision (mapp). Where accuracy and recall are the two highest indicators used, these two are explained here: in a dataset test, four test results will be generated: the 1 st: TP is originally a positive sample and is detected as a positive sample; the 2 nd: TN is originally a negative sample and is detected as the negative sample; and (3) type: FP, which is originally a negative sample, is detected as a positive sample; and 4, the method comprises the following steps: FN, originally positive samples, detect negative samples.
The calculation formula of the accuracy rate is shown in the following formula (1):
the calculation formula of the recall ratio is shown in the following formula (2):
according to the above calculation formula, corresponding calculation codes can be written.
In some embodiments, the testing of the model to be tested is performed according to the test result, and then, the method further includes: generating a test packet report, wherein the test report at least comprises: a model to be tested, a test data set and an evaluation index. Specifically, a test report is generated, including: the electronic version test report can be generated and sent to the place designated by the user through the network, and can also be saved as a local file. For example, the test report may be sent to a user terminal, and the user terminal may view the test report on a display screen, so that the user may find the test result in a glaring manner.
The present embodiment has the following advantages:
(1) the implementation mode provides various platform environments for deep learning to use, and the platform environments are selected by a user to use, so that the problem that the model cannot be tested due to the problem of the platform environment is solved. The user can select a platform environment suitable for the model of the user, and according to the platform environment selected by the user, the embodiment can select equipment of a corresponding platform to be tested.
(2) The embodiment provides various data sets for the user to test and use, so that the time required by the user to upload data is avoided, the process that the user downloads the data from the internet and then cleans and arranges the data is also avoided, and the data collection time is greatly reduced.
(3) The method and the device have the advantages that the efficiency is high, the flow is simple, a user only needs to upload the model to be tested, the testing environment and the testing data set are configured, other operations are not needed, and compared with a related testing method, the method and the device have the advantages that the automation degree is higher, and the efficiency is higher.
Compared with the prior art, the embodiment of the invention provides an automatic model testing method, for a user, the automatic model testing method can be used as a black box, the user only needs to upload a model to be tested and configure a testing environment and a testing data set, and other operations are not needed.
The second embodiment of the invention relates to a model automatic testing method. The second embodiment is substantially the same as the first embodiment, except that different test platforms are configured for the same model to be tested, so that the influence of the different test platforms on the performance of the model to be tested can be known, and the performance of the model to be tested can be judged more comprehensively.
A flow diagram of the model automation test method in this embodiment is shown in fig. 2, and specifically includes:
step 201: and obtaining a model to be tested.
Step 201 is substantially the same as step 101 in the first embodiment, and is not described again in this embodiment to avoid repetition.
Step 202: a plurality of test platforms are preset.
Step 203: and configuring different test platforms for the same model to be tested.
Specifically, in the present embodiment, multiple test platforms are preset in the steps 202 and 203, including common platforms such as a CPU, a GPU, an embedded type, a Mac multiple access computer, and an android. After the model to be tested is obtained, a proper test platform can be directly selected for a user from a plurality of preset test platforms, and the operation is simple. The model operation of the CPU test platform is realized by using a CPU; the GPU test platform is used for realizing model operation by using a GPU; the model operation of the Mac test platform is realized by using Mac equipment; and the Android test platform is implemented by using Android equipment for model operation. In practical use, other hardware platform environments may also be set, which is not illustrated in this embodiment.
In this embodiment, different hardware configurations of the test platform may be implemented in the form of hardware devices, and different hardware resources may also be scheduled by the unified platform, for example: CPU resources, GPU resources, or Android system resources on the hardware may be scheduled by a fixed server.
In the embodiment, different test platforms are configured for the same model to be tested, so that the influence of the different test platforms on the performance of the model to be tested can be known, and the performance quality of the model to be tested can be judged more comprehensively. In this embodiment, except that different test platforms are configured for the model to be tested, other implementation manners are the same as those of the first embodiment, and the implementation details of the first embodiment may be directly applied to this embodiment, and are not repeated in this embodiment to avoid repetition.
Step 204: and generating a model test code according to the model to be tested and the test environment.
Step 205: the model test code is executed to test the model to be tested according to a pre-stored test data set.
Step 204 and step 205 are substantially the same as step 103 and step 104 in the first embodiment, and are not described again in this embodiment to avoid repetition.
Compared with the prior art, the embodiment of the invention provides the automatic model testing method, and different testing platforms are configured for the same model to be tested, so that the influence of the different testing platforms on the performance of the model to be tested can be known, and the performance quality of the model to be tested can be judged more comprehensively.
The steps of the above methods are divided for clarity, and the implementation may be combined into one step or split some steps, and the steps are divided into multiple steps, so long as the same logical relationship is included, which are all within the protection scope of the present patent; it is within the scope of the patent to add insignificant modifications to the algorithms or processes or to introduce insignificant design changes to the core design without changing the algorithms or processes.
The third embodiment of the present invention relates to a model automation test apparatus, as shown in fig. 3, including at least one processor 301; and a memory 302 communicatively coupled to the at least one processor 301; the memory 302 stores instructions executable by the at least one processor 301, and the instructions are executed by the at least one processor 301, so that the at least one processor 301 can execute the model automation test method according to the first or second embodiment.
Where the memory 302 and the processor 301 are coupled in a bus, the bus may comprise any number of interconnected buses and bridges, the buses coupling one or more of the various circuits of the processor 301 and the memory 302. The bus may also connect various other circuits such as peripherals, voltage regulators, power management circuits, and the like, which are well known in the art, and therefore, will not be described any further herein. A bus interface provides an interface between the bus and the transceiver. The transceiver may be one element or a plurality of elements, such as a plurality of receivers and transmitters, providing a means for communicating with various other apparatus over a transmission medium. The data processed by the processor 301 is transmitted over a wireless medium through an antenna, which further receives the data and transmits the data to the processor 301.
The processor 301 is responsible for managing the bus and general processing and may also provide various functions including timing, peripheral interfaces, voltage regulation, power management, and other control functions. And memory 302 may be used to store data used by processor 301 in performing operations.
The fourth embodiment of the present invention also provides a computer-readable storage medium storing a computer program which, when executed by a processor, implements the model automation testing method of the first or second embodiment.
That is, as can be understood by those skilled in the art, all or part of the steps in the method for implementing the embodiments described above may be implemented by a program instructing related hardware, where the program is stored in a storage medium and includes several instructions to enable a device (which may be a single chip, a chip, or the like) or a processor (processor) to execute all or part of the steps of the method described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
It will be understood by those of ordinary skill in the art that the foregoing embodiments are specific examples for carrying out the invention, and that various changes in form and details may be made therein without departing from the spirit and scope of the invention in practice.
Claims (10)
1. An automated model testing method, comprising:
obtaining a model to be tested;
configuring a test environment for the model to be tested;
generating a model test code according to the model to be tested and the test environment;
and executing the model test code to test the model to be tested according to a pre-stored test data set.
2. The method for automated testing of models of claim 1, wherein the test environment comprises: a test platform and a model operation frame;
the generating of the model test code according to the model to be tested and the test environment includes:
analyzing the model to be tested according to the model operation framework to obtain model parameters and keywords of the model to be tested;
determining the code type of the model test code according to the test platform;
and generating a model test code according to the model parameters, the keywords and the code category.
3. The model automated testing method of claim 2, wherein the model testing code comprises: a data set analysis code, a model operation code and a result calculation code;
generating a model test code according to the model parameters, the keywords and the code category comprises the following steps:
generating the data set analysis code according to the model type of the model to be tested and the code type;
generating a model operation code according to the model type of the model to be tested, the model parameter, the keyword and the code type;
and generating the result calculation code according to the evaluation index of the model to be tested and the code type.
4. The automated model testing method of claim 3, wherein said executing the model testing code to test the model to be tested according to a pre-stored test data set and to test the model to be tested according to the test result comprises:
executing the data set analysis code to call a data set of a format required by the model to be tested from the pre-stored test data set, and reading data in the data set and the labeling information of the data according to the format;
executing the model operation code to input the data and the labeling information of the data into the model to be tested for testing so as to obtain the test result;
and executing the result calculation code to calculate the evaluation index of the model to be tested according to the test result.
5. The automated model test method of claim 4, wherein the executing the model run code inputs the data and the label information of the data into the model to be tested for testing, comprising:
executing the model operation code to convert the data into a format required by the model to be tested;
and inputting the data in the format required by the model to be tested and the labeling information of the data into the model to be tested for testing.
6. The automated model testing method of claim 4, wherein the testing the model to be tested according to the testing result further comprises:
generating a test packet report, the test report at least comprising: the model to be tested, the test data set and the evaluation index.
7. The method for automated testing of models of claim 1, wherein the test environment comprises: a test platform;
before configuring a test environment for the model to be tested, the method further includes: presetting a plurality of test platforms;
the configuring of the test environment for the model to be tested comprises: and configuring different test platforms for the same model to be tested.
8. The model automated testing method of claim 1, wherein the pre-stored test data set comprises: a public data set and a private data set stored in a database.
9. An automated model testing apparatus, comprising:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of model automated testing of any of claims 1 to 8.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out a method for automated testing of a model according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011535850.3A CN112527676A (en) | 2020-12-23 | 2020-12-23 | Model automation test method, device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011535850.3A CN112527676A (en) | 2020-12-23 | 2020-12-23 | Model automation test method, device and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112527676A true CN112527676A (en) | 2021-03-19 |
Family
ID=74975856
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011535850.3A Pending CN112527676A (en) | 2020-12-23 | 2020-12-23 | Model automation test method, device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112527676A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113836012A (en) * | 2021-09-17 | 2021-12-24 | 上海瑾盛通信科技有限公司 | Algorithm testing method and device, electronic equipment and storage medium |
| CN116578500A (en) * | 2023-07-14 | 2023-08-11 | 安徽华云安科技有限公司 | Method, device and equipment for testing codes based on reinforcement learning |
| CN117407299A (en) * | 2023-10-18 | 2024-01-16 | 北京大学 | Model test method and system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107451064A (en) * | 2017-08-16 | 2017-12-08 | 北京车和家信息技术有限责任公司 | Automatic test approach, device, computer equipment and readable storage medium storing program for executing |
| CN109634843A (en) * | 2018-10-31 | 2019-04-16 | 中国科学院软件研究所 | A kind of distributed automatization method for testing software and platform towards AI chip platform |
| CN111949514A (en) * | 2020-07-13 | 2020-11-17 | 深圳地平线机器人科技有限公司 | Model prediction method, device and storage medium |
| CN112069057A (en) * | 2020-08-07 | 2020-12-11 | 长沙市到家悠享网络科技有限公司 | Code testing method, device, apparatus and storage medium |
-
2020
- 2020-12-23 CN CN202011535850.3A patent/CN112527676A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107451064A (en) * | 2017-08-16 | 2017-12-08 | 北京车和家信息技术有限责任公司 | Automatic test approach, device, computer equipment and readable storage medium storing program for executing |
| CN109634843A (en) * | 2018-10-31 | 2019-04-16 | 中国科学院软件研究所 | A kind of distributed automatization method for testing software and platform towards AI chip platform |
| CN111949514A (en) * | 2020-07-13 | 2020-11-17 | 深圳地平线机器人科技有限公司 | Model prediction method, device and storage medium |
| CN112069057A (en) * | 2020-08-07 | 2020-12-11 | 长沙市到家悠享网络科技有限公司 | Code testing method, device, apparatus and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| JUNYUSEU: "Experiments on CIFAR datasets with PyTorch", pages 1 - 13, Retrieved from the Internet <URL:https://github.com/junyuseu/pytorch-cifar-models/blob/master> * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113836012A (en) * | 2021-09-17 | 2021-12-24 | 上海瑾盛通信科技有限公司 | Algorithm testing method and device, electronic equipment and storage medium |
| CN113836012B (en) * | 2021-09-17 | 2024-05-03 | 上海瑾盛通信科技有限公司 | Algorithm testing method and device, electronic equipment and storage medium |
| CN116578500A (en) * | 2023-07-14 | 2023-08-11 | 安徽华云安科技有限公司 | Method, device and equipment for testing codes based on reinforcement learning |
| CN116578500B (en) * | 2023-07-14 | 2023-09-26 | 安徽华云安科技有限公司 | Method, device and equipment for testing codes based on reinforcement learning |
| CN117407299A (en) * | 2023-10-18 | 2024-01-16 | 北京大学 | Model test method and system |
| CN117407299B (en) * | 2023-10-18 | 2024-05-07 | 北京大学 | Model test method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112527676A (en) | Model automation test method, device and storage medium | |
| US7333962B2 (en) | Techniques to organize test results | |
| CN109062780B (en) | Development method of automatic test case and terminal equipment | |
| CN110221975B (en) | Method and device for creating interface case automation test script | |
| EP3846034B1 (en) | Systems and methods for automated testing using artificial intelligence techniques | |
| CN111900694B (en) | Relay protection equipment information acquisition method and system based on automatic identification | |
| CN111144493A (en) | Method for automatically identifying algorithm index test, storage medium and electronic terminal | |
| CN112148766A (en) | Method and system for sampling data using artificial neural network model | |
| CN113778864A (en) | Test case generation method and device, electronic equipment and storage medium | |
| CN115562656A (en) | Page generation method and device, storage medium and computer equipment | |
| CN110232130B (en) | Metadata management pedigree generation method, apparatus, computer device and storage medium | |
| CN111768247A (en) | Order-placing rate prediction method, device and readable storage medium | |
| CN117435505B (en) | Visual generation method of performance test script | |
| CN113901647A (en) | Part process rule compiling method and device, storage medium and electronic equipment | |
| CN112712181A (en) | Model construction optimization method, device, equipment and readable storage medium | |
| CN117235527A (en) | End-to-end containerized big data model construction method, device, equipment and medium | |
| CN115345600B (en) | RPA flow generation method and device | |
| US11681511B2 (en) | Systems and methods for building and deploying machine learning applications | |
| CN113010687B (en) | Exercise label prediction method and device, storage medium and computer equipment | |
| US20220215034A1 (en) | Electronic apparatus and controlling method thereof | |
| CN115905016A (en) | BIOS Setup search function test method and device, electronic equipment and storage medium | |
| CN115631374A (en) | Control operation method, control detection model training method, device and equipment | |
| CN115576834A (en) | Software test multiplexing method, system, terminal and medium for supporting fault recovery | |
| CN110838338A (en) | System, method, storage medium, and electronic device for creating biological analysis item | |
| CN115292178A (en) | Test data searching method, device, storage medium and terminal |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |