CN117724980A - Method and device for testing software framework performance, electronic equipment and storage medium - Google Patents
Method and device for testing software framework performance, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN117724980A CN117724980A CN202311848072.7A CN202311848072A CN117724980A CN 117724980 A CN117724980 A CN 117724980A CN 202311848072 A CN202311848072 A CN 202311848072A CN 117724980 A CN117724980 A CN 117724980A
- Authority
- CN
- China
- Prior art keywords
- configuration
- inheritance
- operator
- model training
- tested
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 57
- 238000012549 training Methods 0.000 claims abstract description 125
- 238000012360 testing method Methods 0.000 claims abstract description 105
- 238000011056 performance test Methods 0.000 claims abstract description 32
- 230000006870 function Effects 0.000 claims description 68
- 238000012545 processing Methods 0.000 claims description 23
- 238000004590 computer program Methods 0.000 claims description 16
- 238000005259 measurement Methods 0.000 claims description 12
- 238000001514 detection method Methods 0.000 claims description 7
- 238000010998 test method Methods 0.000 claims description 7
- XSQUKJJJFZCRTK-NJFSPNSNSA-N UREA C 14 Chemical compound N[14C](N)=O XSQUKJJJFZCRTK-NJFSPNSNSA-N 0.000 claims description 6
- 238000007781 pre-processing Methods 0.000 claims description 6
- 238000011156 evaluation Methods 0.000 abstract description 18
- 238000011161 development Methods 0.000 abstract description 9
- 230000009286 beneficial effect Effects 0.000 abstract description 3
- 238000013135 deep learning Methods 0.000 description 21
- 230000008569 process Effects 0.000 description 13
- 238000010586 diagram Methods 0.000 description 11
- 238000004891 communication Methods 0.000 description 8
- 238000005457 optimization Methods 0.000 description 6
- 238000013136 deep learning model Methods 0.000 description 5
- 230000006399 behavior Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000013461 design Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000007547 defect Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 241000287530 Psittaciformes Species 0.000 description 1
- BUGBHKTXTAQXES-UHFFFAOYSA-N Selenium Chemical compound [Se] BUGBHKTXTAQXES-UHFFFAOYSA-N 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000004883 computer application Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000012854 evaluation process Methods 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 235000019580 granularity Nutrition 0.000 description 1
- -1 junit Chemical compound 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 229910052711 selenium Inorganic materials 0.000 description 1
- 239000011669 selenium Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the invention provides a method and a device for testing the performance of a software framework, electronic equipment and a storage medium, wherein the method comprises the following steps: acquiring at least two types of model training frames to be tested, and determining operator configuration corresponding to the model training frames to be tested; determining inheritance relations among different operator configurations according to the configuration inheritance of the operator configurations; and generating performance test results corresponding to the at least two types of model training frames to be tested based on a preset test template, the operator configuration and the inheritance relationship. The embodiment of the invention realizes the transverse evaluation of the model training frame, can improve the accuracy of the evaluation of the model training frame, and is beneficial to improving the development efficiency of the frame and optimizing the performance of the frame.
Description
Technical Field
The present invention relates to the field of computer application technologies, and in particular, to a method and apparatus for testing performance of a software framework, an electronic device, and a storage medium.
Background
With the rapid development of deep learning technology, the generation of numerous open-source deep learning training frameworks, such as TensFlow, pyTorch, caffe, is promoted, and more institutions develop dedicated deep learning training frameworks to meet their own specific requirements. However, due to the design principles and the optimization methods of different institutions for deep learning training frames, the training results of the same deep learning model under different frames may be different, and how to evaluate and verify the performance of the different deep learning training frames is a complex and time-consuming task.
At present, the evaluation method of the deep learning framework performance of the scene is mainly evaluated in the mode of the same model and the same data set, but the method cannot provide information of performance difference and cannot judge the source of the difference. The performance of the deep learning framework is evaluated by means of the special tool, and the special tool is only suitable for part of the deep learning frameworks, so that the tool has poor universality, and the evaluation process is mainly limited to the index evaluation of the deep learning frameworks, so that the transverse evaluation of a plurality of deep learning frameworks cannot be performed, and the performance evaluation of the deep learning frameworks in the prior art has defects.
Disclosure of Invention
The invention provides a method, a device, electronic equipment and a storage medium for testing the performance of a software framework, which are used for realizing the transverse evaluation of multiple deep learning frameworks, improving the accuracy of the evaluation of the deep learning frameworks and being beneficial to improving the development and optimization efficiency of the frameworks.
According to an aspect of the present invention, there is provided a method for testing performance of a software framework, wherein the method includes:
acquiring at least two types of model training frames to be tested, and determining operator configuration corresponding to the model training frames to be tested;
Determining inheritance relations among different operator configurations according to the configuration inheritance of the operator configurations;
and generating performance test results corresponding to the at least two types of model training frames to be tested based on a preset test template, the operator configuration and the inheritance relationship.
According to another aspect of the present invention, there is provided a testing apparatus for software framework performance, wherein the apparatus comprises:
the configuration determining module is used for obtaining at least two types of model training frames to be tested and determining operator configuration corresponding to the model training frames to be tested;
the preprocessing module is used for determining inheritance relations among different operator configurations according to the configuration inheritance of the operator configurations;
and the test execution module is used for generating performance test results corresponding to the at least two types of model training frames to be tested based on a preset test template, the operator configuration and the inheritance relationship.
According to another aspect of the present invention, there is provided an electronic apparatus including:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the method of testing the performance of a software framework according to any of the embodiments of the present invention.
According to another aspect of the present invention, there is provided a computer readable storage medium storing computer instructions for causing a processor to execute a method for testing the performance of a software framework according to any of the embodiments of the present invention.
According to the technical scheme, the corresponding operator configuration is determined for the obtained at least two types of model training frames to be tested, the inheritance relation among different operator configurations is determined according to the inheritance of the operator configuration, performance test is carried out on the model training frames to be tested based on the preset test template and the operator configuration to determine performance test results, so that transverse evaluation of different deep learning frames is achieved, accuracy of evaluation of the deep learning frames can be improved, and development and optimization efficiency of the frames are improved.
It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the invention or to delineate the scope of the invention. Other features of the present invention will become apparent from the description that follows.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a method for testing the performance of a software framework according to a first embodiment of the present invention;
FIG. 2 is a flow chart of another method for testing the performance of a software framework according to a second embodiment of the present invention;
FIG. 3 is an overall architecture diagram of a software framework performance test provided in accordance with a third embodiment of the present invention;
FIG. 4 is an exemplary diagram of a configuration inheritance process provided in accordance with third embodiment of the present invention;
FIG. 5 is an exemplary diagram of a parameter processing procedure provided according to a third embodiment of the present invention;
fig. 6 is an exemplary diagram of a code generation process provided according to a third embodiment of the present invention;
FIG. 7 is a diagram illustrating the generation of a result comparison report according to a third embodiment of the present invention;
FIG. 8 is a schematic structural diagram of a testing device for software framework performance according to a fourth embodiment of the present invention;
fig. 9 is a schematic structural diagram of an electronic device implementing a method for testing the performance of a software framework according to an embodiment of the present invention.
Detailed Description
In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and the claims of the present invention and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Example 1
Fig. 1 is a flowchart of a method for testing the performance of a software framework according to an embodiment of the present invention, where the method may be implemented by a device for testing the performance of a software framework, where the device may be implemented in hardware and/or software, and the device may be configured in a server or a server cluster. As shown in fig. 1, the method includes:
Step 110, obtaining at least two types of model training frames to be tested, and determining operator configuration of the model training frames to be tested.
The model training frames to be tested can be software frames for training the deep learning model, different types of model training frames to be tested can be developed by adopting different development languages or by different design principles, and different types of model training frames to be tested can be used for training the deep learning model with the same business requirement. The operator configuration may be configuration information for implementing a computational unit in the deep learning model, the operator configuration may be a function-dependent configuration of the deep learning model trained by the model training framework to be tested, the operator configuration may include, but is not limited to, weight coefficients, computational functions, and the like.
In the embodiment of the invention, different operator configurations can be configured in advance for different model training frames to be tested, and when at least two types of model training frames to be tested are acquired, operator configurations respectively corresponding to the different types of model training frames to be tested can be searched in the operator configurations which are configured in advance, and it can be understood that the operator configurations can comprise parameters corresponding to the model training frames to be tested and behavior configuration parameters.
Step 120, determining the inheritance relationship between different operator configurations according to the configuration inheritance of the operator configurations.
The configuration inheritance may be a parameter preset in the operator configuration and used for representing an inheritance relationship between different operator configurations, the configuration inheritance may be located in a configuration file of the operator configuration, the inheritance relationship may be an inheritance relationship between different operator configurations or between different functions in the operator configuration, for example, when the inheritance relationship includes an operator configuration inherited by the operator configuration a for the operator configuration B, the operator configuration B may use the configuration parameter in the operator configuration a.
In the embodiment of the invention, the configuration information inherited by the corresponding configuration can be extracted from the operator configuration, and the inheritance relationship among different operator configurations can be established according to the configuration information, wherein the inheritance relationship can comprise the inheritance relationship between the operator configuration and the inheritance relationship between functions in the operator configuration. For example, declaration information corresponding to the configuration inheritance may be extracted from the operator configuration, and the operator configuration corresponding to the declaration information may be used as the operator configuration inherited by the current operator configuration.
And 130, generating performance test results corresponding to at least two types of model training frames to be tested based on a preset test template, operator configuration and inheritance relationship.
In the embodiment of the invention, operators corresponding to different model training frames to be tested can be formed through different operator configurations and inheritance relations of the operator configurations, test codes corresponding to different model training frames to be tested can be generated according to a preset test template, the constructed operators can be tested according to the test codes, and the test results are transversely compared according to the different model training frames to be tested, so that performance test results are determined.
According to the embodiment of the invention, the corresponding operator configuration is determined for the obtained at least two types of model training frames to be tested, the inheritance relation between different operator configurations is determined according to the inheritance of the operator configuration, and the performance test is carried out on the model training frames to be tested based on the preset test template and the operator configuration so as to determine the performance test result, so that the transverse evaluation of different deep learning frames is realized, the evaluation accuracy of the deep learning frames can be improved, and the development and optimization efficiency of the frames can be improved.
Example two
Fig. 2 is a flowchart of another method for testing the performance of a software framework according to the second embodiment of the present invention, where the embodiment of the present invention is embodied on the basis of the above embodiment of the present invention, and referring to fig. 2, the method provided by the embodiment of the present invention specifically includes the following steps:
Step 210, receiving a control instruction for triggering testing of at least two types of model training frames to be tested.
The control instruction may be information triggering the to-be-tested model training frame to perform performance evaluation test, and the control instruction may carry indication information of the to-be-tested model training frame, for example, the control instruction may carry identification information of the to-be-tested model training frame.
In the embodiment of the invention, a user can trigger measurement of the model training frame to be tested through the control instruction, and when the server or the server cluster receives the control instruction, the model to be tested which needs to be tested can be determined through the information carried in the control instruction.
And 220, extracting the identification information of the model training frame to be tested corresponding to the control instruction.
The identification information may be information for identifying different types of training frames of the model to be tested, and the identification information may include information such as a frame name, a frame version number, a frame number, and the like.
In the embodiment of the invention, the received control instruction can be analyzed, the identification information corresponding to different training frames to be tested in the control instruction is extracted, it can be understood that the format of the control instruction message is not limited herein, the control instruction can be used for transmitting the identification information of the training frames to be tested, for example, the control instruction can comprise an HTTP message format, a UDP message format and the like.
Step 230, searching an operator configuration file stored in association with the identification information as operator configuration, wherein the operator configuration file at least comprises an operator unified configuration file and an operator customized configuration file.
In the embodiment of the invention, different operator configuration files can be configured for different model training frames to be tested, each operator configuration file can be associated and stored with the corresponding model training frame to be tested, the association mode can include that the operator configuration file and the model training frame to be tested have physical association or logic management, for example, the file name of the operator configuration file can contain label information of the model training frame to be tested, or the operator configuration file is stored in a storage area and the like which are arranged for the model training frame to be tested. When performance evaluation is required to be performed on the model training frames to be tested, a corresponding operator configuration file can be searched according to the extracted identification information, the operator configuration file can be used as operator configuration of the model training frames to be tested corresponding to the identification information, wherein the operator unified configuration file can be operator configuration shared by all different types of model training frames to be tested, the operator customized configuration file can be operator configuration unique to each type of model training frames to be tested, and the configuration repetition amount of the different types of model training frames to be tested can be reduced by dividing the operator configuration file into the operator unified configuration file and the operator customized configuration file.
Step 240, reading the inheritance declaration information of the configuration inheritance in the operator configuration file of the operator configuration.
The inheritance declaration information can be information representing inheritance relations between operator configurations in the operator configuration file, the inheritance declaration information can comprise module level inheritance information and function level inheritance information, the inheritance information of different levels can act on different granularities of the operator configurations, the module level inheritance information can declare inheritance relations between different operator configurations, and the function level inheritance information can declare inheritance relations between different functions in the operator configuration file.
In the embodiment of the invention, the inheritance declaration information can be extracted from the operator configuration file corresponding to the operator configuration, so that the configuration inheritance corresponding to each operator configuration is determined, and it can be understood that the inheritance declaration information can be in various levels, the inheritance declaration information can be read at different file positions of the operator configuration file according to the corresponding levels, for example, the module level inheritance information can be read at the file header of the operator configuration file, and the function level inheritance information can be read at the function header of the operator configuration file.
Step 250, extracting inheritance configuration information in the inheritance declaration information.
The inheritance configuration information can be parameters representing inheritance relations in the inheritance declaration information, the inheritance configuration information can represent operator configuration inherited by the current operator configuration or parent function inherited by the current function, and the like, and the inheritance configuration information can be extracted in a key sub-matching or character recognition mode.
In the embodiment of the invention, the inheritance configuration information can be extracted in the inheritance declaration information, so that the operator configuration inherited by the previous operator configuration or the parent function inherited by the current function can be determined.
And 260, when the inheritance configuration information is module level inheritance information, establishing an inheritance relationship between operator configuration and target operator configuration corresponding to the inheritance configuration information.
Specifically, when the inheritance configuration information is module level inheritance information, for example, the inheritance configuration information is located at the head of the operator configuration file, the target operator configuration inherited by the current operator configuration can be searched according to the operator configuration name or operator configuration identification information in the inheritance configuration information, the inheritance relationship between the current operator configuration and the target operator configuration can be established, the inheritance relationship can comprise a physical relationship or a logical relationship, and after the inheritance relationship is established, the current operator configuration can directly use the configuration information in the target operator configuration.
Step 270, when the inheritance configuration information is the function level inheritance information, establishing an inheritance relationship between the source function and the target function corresponding to the inheritance configuration information in the operator configuration.
In the embodiment of the invention, when the inheritance configuration information is the function level inheritance information, the corresponding source function and target function can be searched in the operator configuration file configured by the current operator according to the parameters in the inheritance configuration information, and an inheritance relationship can be established between the source function and the target function, so that the source function can use the configuration information in the target function.
And 280, constructing operators corresponding to each type of model training frame to be tested according to the operator configuration and the inheritance relationship.
In the embodiment of the invention, the information such as the function, the parameter default value and the like can be extracted according to the operator configuration and the inheritance relation for each type of model training frame to be tested, and the calculation unit can be constructed according to the information such as the function, the parameter default value and the like to serve as an operator of the model training frame to be tested.
And 290, filling the measurement parameters in the operator configuration into preset test templates corresponding to each type of model training frames to be tested respectively as test codes.
In the embodiment of the invention, the corresponding preset test templates can be searched for each type of to-be-tested model training frame, and the measurement parameters in the operator configuration of the type of to-be-tested model training frame can be filled into the preset test templates, so that the preset test templates are used as test codes for testing the type of to-be-tested model training frame.
And 2100, invoking test codes in a preset test tool to train each type of model training framework to be tested so as to determine performance test results.
The preset test tool may be a test tool for running test codes, and may provide corresponding software running environments for different types of training frames of the model to be tested, for example, the preset test tool may include pytest, junit, selenium and the like.
In the embodiment of the invention, the test codes of different types of model training frames to be tested can be respectively called in the preset test tool, so that the test of the model training frames to be tested corresponding to the test codes is realized, the performance test result can be determined through the training result of each type of model training frames to be tested, for example, the training result of each type of model training frames to be tested can be processed as a part of the performance test result, or the training result of each type of model training frames to be tested can be directly used as a part of the performance test result.
According to the embodiment of the invention, the control instruction triggering the to-be-tested model training frame is received, the identification information is extracted, the identification information corresponds to the to-be-tested model training frame, the operator configuration file stored in association with the identification information is searched and used as the operator configuration corresponding to the to-be-tested model training frame, the inheritance declaration information of the operator configuration file is read and used as configuration inheritance, the inheritance configuration information is extracted from the inheritance declaration information, the inheritance relation between the operator configuration and the target operator configuration or the inheritance relation between the source function and the target function is established according to the inheritance configuration information, the operator of each type of to-be-tested model training frame is constructed through the operator configuration and the inheritance relation, the test code is generated according to the test parameters of the operator configuration and the preset test target, and the test code is called in the preset test tool to realize the training corresponding to each type of to-be-tested model training frame, so that the performance test result is generated, the lateral evaluation of different deep learning frames can be realized, the accuracy of the evaluation of the deep learning frame can be improved, and the development and optimization efficiency of the frame can be improved.
Further, on the basis of the above embodiment of the present invention, inheritance declaration information is located in a first line of execution codes in a file of the operator configuration file or a first line of execution codes in a function of the operator configuration file.
In the embodiment of the invention, the inheritance declaration information can be arranged in the first line of execution codes in the file of the operator configuration file or the first line of codes in the function, so that the searching efficiency of the inheritance declaration information is improved, and the inheritance configuration is convenient to manage.
In other inventive embodiments, further comprising:
extracting parameter configuration files of corresponding measurement parameters from operator configuration files corresponding to the operator configuration; and calling the parameter configuration file to perform unified processing on the measurement parameters, wherein the unified processing at least comprises: function parameter detection, key parameter default value configuration and parameter formatting.
In the embodiment of the invention, the operator configuration file can comprise a parameter configuration file for measuring parameters, the parameter configuration file can comprise a rule for processing the measuring parameters, unified processing for the measuring parameters can be realized by calling the parameter configuration file, the unified processing can carry out processing such as function parameter detection, key parameter default value configuration, parameter formatting and the like on the measuring parameters, wherein the function parameter detection can comprise determining whether parameters belonging to the function in the measuring parameters meet the relevant information such as type, numerical value, length and the like; the key parameter default configuration may include setting a default value for a key parameter of the measured parameters; the parameter formatting may include initializing the measurement parameters and uniformly adjusting the data length.
In some embodiments of the present invention, invoking the test code on a preset test tool to train each type of the model training frame to be tested to determine the performance test result includes:
respectively calling the test codes corresponding to each type of the model training frame to be tested in a pytest tool of the preset test tool; collecting running results and executing speeds of different test codes; determining a first comparison result between different running results and a second comparison result between different execution speeds; and generating a result comparison report of the performance test result according to the first comparison result and the second comparison result.
In the embodiment of the invention, the generating process of the performance test result can comprise the steps of calling test codes in a Pytest tool for each type of model training frame to be tested, collecting the running result and the execution speed of the test codes in the test process, comparing the running results of different types of model training frames to be tested, comparing the running results of various types of model training frames to be tested, taking the comparison result as a first comparison result, comparing the execution speed of various types of model training frames to be tested, taking the comparison result of the execution speed as a second comparison result, and generating a result comparison report through the first comparison result and the second comparison result. For example, the execution speeds of the same functions may be selected for comparison, and the model training framework to be tested with the minimum execution speed therein may be populated into a result comparison report of the performance test results.
Example III
Fig. 3 is an overall architecture diagram of a software framework performance test according to a third embodiment of the present invention, and referring to fig. 3, a method for testing software framework performance test according to an embodiment of the present invention may include modules such as operator configuration, configuration inheritance, parameter processing, code generation, code execution, and result comparison report generation.
Wherein operator configuration may be a process of configuring parameters and behavior of an operator. Mainly has two functions: unified configuration and custom configuration.
Unified configuration: refers to placing the configuration of different operators in a unified way in one configuration file, e.g. config_common. The method has the advantages that unified configuration can be conveniently carried out on all operators, and repeated work is reduced. The configuration file of each platform inherits the unified configuration, so that the operator is ensured to have consistent behavior on different platforms.
Custom configuration: the operator and the unified configuration of each platform are different by the pointer, and inheritance and modification can be carried out in the configuration file of each platform. Similar to the principle of class inheritance, the configuration can be modified in a personalized way on the basis of inheritance so as to meet the requirements of different platforms. The custom configuration is extensive, in order to ensure the uniformity of the operator test and the accuracy of the result, the custom configuration of the function name, the parameter name and the parameter default value is mainly completed.
Through unified configuration and custom configuration, operators can be flexibly configured, so that the operators have consistent behaviors on different platforms, and personalized customization can be performed according to specific requirements. The configuration mode can improve development efficiency, reduce repeated labor and facilitate maintenance and operator configuration.
Configuration inheritance mainly occurs between operators because the operator tests many parameters and attributes to be the same, and if each operator configuration is repeatedly filled, the total amount of work is increased, and the configuration file becomes huge and tedious. Configuration inheritance has the main effects of reducing the repeated workload and improving the maintainability of the configuration. Through configuration inheritance, the same configuration information can be managed in a unified way, repeated filling is avoided, and the possibility of errors is reduced. Meanwhile, the configuration inheritance also enables the configuration file to be clearer and easier to understand, and is convenient for subsequent maintenance and modification. The configuration inheritance may be divided into a module level and a function level from the size of the scope, as shown in fig. 4, an inheritance relationship may exist between a model configuration config_pt_module of the pt model training framework and a model configuration config_ms_module of the ms model training framework, a function configuration of a function of the pt model training framework may have an inheritance relationship with a model configuration config_pt_module of the pt model training framework, and a function configuration of a function of the t model training framework may also have an inheritance relationship with a function configuration of a function of the ms model training framework. Configuration inheritance may include module level and function level configuration inheritance.
The module-level configuration inheritance mainly completes inheritance relation among modules, for example:
configuration inheritance of the function level is mainly in a module, the configuration inheritance of the function piece is completed, and a parent object of keyword identification inheritance, such as a base, can be configured. The default priority order of the cases can be searched in the configuration inheritance of the function level, the parent object in the current config inheritance configuration can be searched first, and then the parent object in the '__ Dependencies __' = [ config_pt_module ] list according to the sequence order is searched. In one exemplary embodiment, function-level configuration inheritance may be implemented by:
in the embodiment of the invention, the parameter processing can comprise the processing procedures of function parameter detection, more value addition, parameter formatting and the like.
Function parameter detection: whether the parameters of the check function satisfy the information related to type, value, existence, length and the like.
Increase default value: the key parameter is increased by a default value.
Parameter formatting: and finally, carrying out fine tuning on the initialization, the length and the like of all parameters in the last link of parameter processing.
In order to compare the multi-platform effect, a set of unified keys can be generated for comparison use through parameter processing, so that performance comparison of different types of model training frames is realized.
For example, a parameter configuration file that may be fun_key_hash may be stored in the configuration file for use in performance testing and as a key error-reporting data result. The parameter profile of the fun_key_hash may be composed of function names and key configurations. In some embodiments of the present invention, inheritance relationships may also exist between parameter configuration files, where in the process of configuring an operator, the corresponding inheritance relationship may be configured for the parameter configuration files, and a parent class corresponding to the parameter configuration files may be found through a keyword base through a process similar to inheritance configuration, and then a fun_key_hash in the parent class is referred to. For example:
referring to fig. 5, the fun_key_hash of different types of model training frameworks can be obtained by inheriting relationships between each other, pp: add and ms: add is all from pt: add inherits, then fun_key_hash is derived from pt: add acquisition, guarantee pt: add, pp: add, ms: the fun_key_hash of the add is consistent.
In the embodiment of the invention, the operator test codes specific to each platform are generated by combining templates corresponding to the platforms to which each model training frame belongs according to configuration of code generation. Referring to fig. 6, the templates pt_template.py of the pt model training framework, pp_template.py of the pp model training framework, and ms_template.py of the ms model training framework may generate test codes corresponding to respective platforms by initializing init, setting model, setting method func, and setting tensor, respectively, and may execute the test codes in respective execution environments corresponding to respective platforms, thereby generating test results corresponding to respective platforms.
Specifically, the code executing process may mainly use a pytest tool to run the operator test code of each platform, and generate a standard test result, where the test result may keep information such as a function key, a function name, a running result, running time consumption, and the like of the model training framework as the test result.
In the embodiment of the invention, the test results obtained by the code execution process can be compared to generate a comparison report, and the result comparison report process can comprise the comparison of two parts:
result accuracy contrast: because the results of each platform cannot be directly compared, the execution results of each platform need to be converted into a unified numpy format for unified comparison, so that the results can be directly compared, and operators with difference results can be found.
Perform speed comparison: this part simply gathers the functions of the same keys and then makes a lateral comparison.
Referring to fig. 7, test results of the parrots platform, the pp platform and the ms platform in respective execution environments can be collected, the test results are processed to generate unified data numpy, and the execution speed and the result accuracy of the unified data are respectively compared, so that a result comparison report is generated.
Example IV
Fig. 8 is a schematic structural diagram of a testing device for software framework performance according to a fourth embodiment of the present invention. As shown in fig. 8, the apparatus includes:
the configuration determining module 301 is configured to obtain at least two types of model training frames to be tested, and determine operator configurations corresponding to the model training frames to be tested.
A preprocessing module 302, configured to determine inheritance relationships between different operator configurations according to configuration inheritance of the operator configurations.
The test execution module 303 is configured to generate a performance test result corresponding to the at least two types of model training frames to be tested based on a preset test template, the operator configuration and the inheritance relationship.
According to the embodiment of the invention, the configuration determining module determines the corresponding operator configuration for the obtained at least two types of model training frames to be tested, the preprocessing module determines the inheritance relation among different operator configurations according to the inheritance of the operator configuration, and the test execution module performs performance test on the model training frames to be tested based on the preset test template and the operator configuration to determine the performance test result, so that the transverse evaluation of different deep learning frames is realized, the evaluation accuracy of the deep learning frames is improved, and the development and optimization efficiency of the frames are improved.
Further, on the basis of the above embodiment of the present invention, the configuration determining module 301 includes:
and the test triggering unit is used for receiving control instructions for triggering the test of the at least two types of model training frames to be tested.
And the information extraction unit is used for extracting the identification information of the model training frame to be tested corresponding to the control instruction.
The file searching unit is used for searching an operator configuration file stored in association with the identification information as the operator configuration, wherein the operator configuration file at least comprises an operator unified configuration file and an operator customized configuration file.
Further, on the basis of the above embodiment of the present invention, the preprocessing module 302 includes:
the information extraction unit is used for reading inheritance declaration information inherited by the configuration in the operator configuration file of the operator configuration; and extracting inheritance configuration information in the inheritance declaration information.
And the module relation unit is used for establishing the inheritance relation of the operator configuration and the target operator configuration corresponding to the inheritance configuration information when the inheritance configuration information is module level inheritance information.
And the function relation unit is used for establishing the inheritance relation between the source function and the target function corresponding to the inheritance configuration information in the operator configuration when the inheritance configuration information is the function level inheritance information.
Further, on the basis of the above embodiment of the invention, the preprocessing module 302 relays that the declaration information is located in the first line of execution code in the file of the operator configuration file or the first line of execution code in the function of the operator configuration file.
Further, on the basis of the embodiment of the invention, the method further comprises the following steps: the parameter processing module is used for extracting a parameter configuration file corresponding to the measurement parameter from an operator configuration file corresponding to the operator configuration;
and calling the parameter configuration file to perform unified processing on the measurement parameters, wherein the unified processing at least comprises: function parameter detection, key parameter default value configuration and parameter formatting.
Further, on the basis of the above embodiment of the present invention, the test execution module 303 includes:
and the operator construction unit is used for constructing operators corresponding to the model training frames to be tested according to the operator configuration and the inheritance relationship.
And the code generation unit is used for respectively filling the measurement parameters in the operator configuration into the preset test templates corresponding to each type of the model training frame to be tested as test codes.
And the result determining unit is used for calling the test codes in a preset test tool to train each type of model training framework to be tested so as to determine the performance test result.
Further, on the basis of the above embodiment of the present invention, the result determining unit is specifically configured to:
respectively calling the test codes corresponding to each type of the model training frame to be tested in a pytest tool of the preset test tool; collecting running results and executing speeds of different test codes; determining a first comparison result between different running results and a second comparison result between different execution speeds; and generating a result comparison report of the performance test result according to the first comparison result and the second comparison result.
The device for testing the performance of the software framework provided by the embodiment of the invention can execute the method for testing the performance of the software framework provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.
Example five
Fig. 9 is a schematic structural diagram of an electronic device implementing a method for testing the performance of a software framework according to an embodiment of the present invention. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Electronic equipment may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.
As shown in fig. 9, the electronic device 10 includes at least one processor 11, and a memory, such as a Read Only Memory (ROM) 12, a Random Access Memory (RAM) 13, etc., communicatively connected to the at least one processor 11, in which the memory stores a computer program executable by the at least one processor, and the processor 11 may perform various appropriate actions and processes according to the computer program stored in the Read Only Memory (ROM) 12 or the computer program loaded from the storage unit 18 into the Random Access Memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 may also be stored. The processor 11, the ROM 12 and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to bus 14.
Various components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, etc.; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, an optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.
The processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, digital Signal Processors (DSPs), and any suitable processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as the test method of software framework performance.
In some embodiments, the method of testing the performance of a software framework may be implemented as a computer program tangibly embodied on a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 10 via the ROM 12 and/or the communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the method of testing the performance of a software framework described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the method of testing the performance of the software framework in any other suitable manner (e.g., by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
A computer program for carrying out methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be implemented. The computer program may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which a user can provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the internet.
The computing system may include clients and servers. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical hosts and VPS service are overcome.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps described in the present invention may be performed in parallel, sequentially, or in a different order, so long as the desired results of the technical solution of the present invention are achieved, and the present invention is not limited herein.
The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.
Claims (10)
1. A method for testing the performance of a software framework, the method comprising:
acquiring at least two types of model training frames to be tested, and determining operator configuration corresponding to the model training frames to be tested;
determining inheritance relations among different operator configurations according to the configuration inheritance of the operator configurations;
and generating performance test results corresponding to the at least two types of model training frames to be tested based on a preset test template, the operator configuration and the inheritance relationship.
2. The method of claim 1, wherein the obtaining at least two types of model training frames to be tested, and obtaining an operator configuration corresponding to the model training frames to be tested, comprises:
receiving a control instruction for triggering the test of the at least two types of model training frames to be tested;
extracting identification information of the model training frame to be tested corresponding to the control instruction;
and searching an operator configuration file stored in association with the identification information as the operator configuration, wherein the operator configuration file at least comprises an operator unified configuration file and an operator customized configuration file.
3. The method of claim 1, wherein said determining inheritance relationships between different operator configurations based on configuration inheritance of said operator configurations comprises:
reading inheritance declaration information inherited by the configuration in an operator configuration file of the operator configuration;
extracting inheritance configuration information in the inheritance declaration information;
when the inheritance configuration information is module level inheritance information, establishing an inheritance relation of the operator configuration and target operator configuration corresponding to the inheritance configuration information;
and when the inheritance configuration information is function level inheritance information, establishing an inheritance relation between a source function and a target function corresponding to the inheritance configuration information in the operator configuration.
4. A method according to claim 3, wherein the inheritance declaration information is located in a first line of execution code within a file of the operator profile or in a first line of execution code within a function of the operator profile.
5. The method as recited in claim 1, further comprising:
extracting parameter configuration files of corresponding measurement parameters from operator configuration files corresponding to the operator configuration;
and calling the parameter configuration file to perform unified processing on the measurement parameters, wherein the unified processing at least comprises: function parameter detection, key parameter default value configuration and parameter formatting.
6. The method of claim 1, wherein the generating performance test results corresponding to the at least two types of model training frameworks to be tested based on the preset test templates, the operator configuration, and the inheritance relationship comprises:
constructing operators corresponding to each type of model training frames to be tested according to the operator configuration and the inheritance relationship;
filling measurement parameters in the operator configuration into the preset test templates corresponding to each type of the model training frames to be tested respectively as test codes;
and calling the test codes in a preset test tool to train each type of model training framework to be tested so as to determine the performance test result.
7. The method of claim 6, wherein invoking the test code at a preset test tool trains each type of the model training framework under test to determine the performance test results, comprising:
respectively calling the test codes corresponding to each type of the model training frame to be tested in a pytest tool of the preset test tool;
collecting running results and executing speeds of different test codes;
determining a first comparison result between different running results and a second comparison result between different execution speeds;
and generating a result comparison report of the performance test result according to the first comparison result and the second comparison result.
8. A device for testing the performance of a software framework, the device comprising:
the configuration determining module is used for obtaining at least two types of model training frames to be tested and determining operator configuration corresponding to the model training frames to be tested;
the preprocessing module is used for determining inheritance relations among different operator configurations according to the configuration inheritance of the operator configurations;
and the test execution module is used for generating performance test results corresponding to the at least two types of model training frames to be tested based on a preset test template, the operator configuration and the inheritance relationship.
9. An electronic device, the electronic device comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the method of testing the performance of the software framework of any one of claims 1-7.
10. A computer readable storage medium storing computer instructions for causing a processor to perform the method of testing the performance of a software framework according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311848072.7A CN117724980A (en) | 2023-12-28 | 2023-12-28 | Method and device for testing software framework performance, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311848072.7A CN117724980A (en) | 2023-12-28 | 2023-12-28 | Method and device for testing software framework performance, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117724980A true CN117724980A (en) | 2024-03-19 |
Family
ID=90201559
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311848072.7A Pending CN117724980A (en) | 2023-12-28 | 2023-12-28 | Method and device for testing software framework performance, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117724980A (en) |
-
2023
- 2023-12-28 CN CN202311848072.7A patent/CN117724980A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20230029359A1 (en) | Method and apparatus for testing ai chip computing performance, and non-transitory computer-readable storage medium | |
| CN115576828A (en) | Test case generation method, device, equipment and storage medium | |
| CN113127357A (en) | Unit testing method, device, equipment, storage medium and program product | |
| CN115687406B (en) | Sampling method, device, equipment and storage medium for call chain data | |
| CN115481594B (en) | Scoreboard implementation method, scoreboard, electronic equipment and storage medium | |
| CN116303013A (en) | Source code analysis method, device, electronic equipment and storage medium | |
| CN115659347A (en) | Safety testing method and device, electronic equipment and storage medium | |
| CN115905016A (en) | BIOS Setup search function test method and device, electronic equipment and storage medium | |
| CN114443493A (en) | Test case generation method and device, electronic equipment and storage medium | |
| CN115576831A (en) | Test case recommendation method, device, equipment and storage medium | |
| CN115328736A (en) | Probe deployment method, device, equipment and storage medium | |
| CN114722401A (en) | Equipment safety testing method, device, equipment and storage medium | |
| CN117724980A (en) | Method and device for testing software framework performance, electronic equipment and storage medium | |
| CN117271373B (en) | Automatic construction method and device for test cases, electronic equipment and storage medium | |
| CN115098405B (en) | Software product evaluation method and device, electronic equipment and storage medium | |
| CN115860055B (en) | Performance determination method, performance optimization method, device, electronic equipment and medium | |
| CN116401113B (en) | Environment verification method, device and medium for heterogeneous many-core architecture acceleration card | |
| CN115312114A (en) | Method, apparatus, device and medium for testing RAM | |
| CN117827637A (en) | Automatic test method and device and electronic equipment | |
| CN117493203A (en) | Method, device, equipment and storage medium for testing server software | |
| CN117421452A (en) | Data blood edge testing method and device, electronic equipment, storage medium and product | |
| CN115567624A (en) | Message processing method and device, electronic equipment and medium | |
| CN117472751A (en) | Vehicle system function analysis method, device, equipment and medium | |
| CN116303071A (en) | Interface testing method and device, electronic equipment and storage medium | |
| CN116991737A (en) | Software testing method, system, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |