CN117035065A - Model evaluation method and related device - Google Patents
Model evaluation method and related device Download PDFInfo
- Publication number
- CN117035065A CN117035065A CN202311302188.0A CN202311302188A CN117035065A CN 117035065 A CN117035065 A CN 117035065A CN 202311302188 A CN202311302188 A CN 202311302188A CN 117035065 A CN117035065 A CN 117035065A
- Authority
- CN
- China
- Prior art keywords
- model
- target
- candidate
- models
- platform
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000011156 evaluation Methods 0.000 title claims abstract description 41
- 238000012795 verification Methods 0.000 claims abstract description 105
- 238000012549 training Methods 0.000 claims abstract description 98
- 238000000034 method Methods 0.000 claims abstract description 71
- 238000006243 chemical reaction Methods 0.000 claims abstract description 60
- 230000000694 effects Effects 0.000 claims abstract description 22
- 238000012545 processing Methods 0.000 claims abstract description 12
- 238000012216 screening Methods 0.000 claims abstract description 11
- 238000013507 mapping Methods 0.000 description 7
- 238000013528 artificial neural network Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 6
- 238000003062 neural network model Methods 0.000 description 5
- 238000013479 data entry Methods 0.000 description 4
- 230000001360 synchronised effect Effects 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 240000007651 Rubus glaucus Species 0.000 description 1
- 235000011034 Rubus glaucus Nutrition 0.000 description 1
- 235000009122 Rubus idaeus Nutrition 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000003631 expected effect Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/10—Interfaces, programming languages or software development kits, e.g. for simulating neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/042—Knowledge-based neural networks; Logical representations of neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/04—Inference or reasoning models
Abstract
The application discloses a model evaluation method and a related device. The method for evaluating the model comprises the following steps: obtaining a plurality of candidate training models, wherein the plurality of candidate training models are training results of an initial model in a plurality of different iteration periods; converting each candidate training model into a candidate target model by using model conversion parameters of a target platform to obtain a plurality of candidate target models; and processing the material data by using the candidate target models by the equipment of the target platform to obtain verification results of the candidate target models on the material data, and screening out the optimal candidate target models for being deployed to the target platform based on the verification results. According to the method and the device, the optimal model of the device conforming to the target platform can be selected according to the running condition of the device of the target platform, and the deployment effect of the model on the device of the target platform can be improved.
Description
Technical Field
The application relates to the technical field of deep learning, in particular to a model evaluation method and a related device.
Background
At present, a neural network algorithm is applied to electronic equipment, a trained neural network model is firstly converted into a model of a target electronic equipment platform, and then the model is deployed to the target electronic equipment, so that reasoning work can be performed by using the neural network algorithm. However, the reasoning effect of the converted neural network model often cannot be completely consistent with the reasoning effect of the original neural network model, so that the deployment effect of the model on the target electronic device is difficult to ensure.
Disclosure of Invention
The application provides a model evaluation method and a related device, which can select an optimal model of equipment conforming to a target platform according to the running condition of the equipment of the target platform, and can improve the deployment effect of the model on the equipment of the target platform.
To achieve the above object, the present application provides a method for model evaluation, the method comprising:
obtaining a plurality of candidate training models, wherein the plurality of candidate training models are training results of an initial model in a plurality of different iteration periods;
converting each candidate training model into a candidate target model by using model conversion parameters of a target platform to obtain a plurality of candidate target models;
and processing the material data by using the candidate target models by the equipment of the target platform to obtain verification results of the candidate target models on the material data, and screening out the optimal candidate target models for being deployed to the target platform based on the verification results.
To achieve the above object, the present application provides a method for model evaluation, the method comprising:
the method comprises the steps that equipment of a target platform obtains a plurality of candidate target models, wherein the candidate target models are obtained by converting a plurality of candidate training models by using model conversion parameters of the target platform, and the candidate training models are training results of an initial model in a plurality of different iteration periods;
And processing the material data by utilizing each candidate target model to obtain verification results of a plurality of candidate target models on the material data, and screening out an optimal candidate target model for deployment to the target platform based on the verification results.
To achieve the above object, the present application provides an electronic device including a processor; the processor is configured to execute instructions to implement the steps of the above method.
To achieve the above object, the present application also provides a computer-readable storage medium storing instructions/program data capable of being executed to implement the above method.
The model evaluation method converts a plurality of candidate training models into a plurality of candidate target models by using model conversion parameters of a target platform, wherein the plurality of candidate training models are training results of an initial model in a plurality of different iteration periods; and the equipment of the target platform runs each candidate target model to process the material data, so that verification results of the candidate target models on the material data are obtained, and an optimal candidate target model for being deployed to the target platform is screened out, so that the equipment of the candidate target platform converted by the candidate training models is run through the equipment of the target platform, and the optimal candidate target model for being deployed to the target platform is screened out from the equipment of the candidate target platform based on the running results; a plurality of models generated in different iteration periods during iterative training of the same neural network can select an optimal model of equipment conforming to a target platform according to the running condition of the equipment of the target platform under the condition that the effect quality sequence of the training environment is inconsistent with the effect quality sequence of the several models on the equipment of a certain target platform, and the deployment effect of the models on the equipment of the target platform can be improved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute a limitation on the application. In the drawings:
FIG. 1 is a schematic diagram of a model evaluation system of the present application;
FIG. 2 is a flow chart of an embodiment of a method of model evaluation of the present application;
FIG. 3 is a flow diagram of one embodiment of model distribution in a method of model evaluation of the present application;
FIG. 4 is a schematic flow chart of another embodiment of a method of model evaluation of the present application;
FIG. 5 is a flow chart of one embodiment of model conversion in the method of model evaluation of the present application;
FIG. 6 is a flow chart of one embodiment of executable program compilation in a method of model evaluation according to the present application;
FIG. 7 is a flow chart of one embodiment of verification result checking in the method of model evaluation of the present application;
FIG. 8 is a schematic diagram of an embodiment of an electronic device of the present application;
fig. 9 is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application.
Detailed Description
The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application. In addition, the term "or" as used herein refers to a non-exclusive "or" (i.e., "and/or") unless otherwise indicated (e.g., "or otherwise" or in the alternative "). Also, the various embodiments described herein are not necessarily mutually exclusive, as some embodiments may be combined with one or more other embodiments to form new embodiments.
With the rapid development of artificial intelligence, various platforms are layered endlessly, mainly including IOS, linux, android, windows, raspberry pie and other platforms. In the process from algorithm development to algorithm deployment, different platforms may be required, for example, windows may be used for algorithm development and debugging, and in the deployment stage, due to limitations of hardware and deployment software, the algorithm may need to be converted into an IOS model for adaptation. When the platforms used by the training device and the deployment device (i.e. the device of the target platform) are different, the model conversion is needed, and the source network can be converted into the model of the target platform by using the model conversion parameters of the target platform. And a plurality of models generated in different iteration periods during iterative training of the same neural network are not completely consistent with the effect quality ordering of the models on other platforms in the training environment.
Based on the method, the application provides a model evaluation method, which comprises the steps of converting a plurality of candidate training models into a plurality of candidate target models by using model conversion parameters of a target platform, wherein the plurality of candidate training models are training results of an initial model in a plurality of different iteration periods; and the equipment of the target platform runs each candidate target model to process the material data, so that verification results of a plurality of candidate target models on the material data are obtained, and an optimal candidate target model for deployment to the target platform is screened out, so that equipment of a plurality of candidate target platforms converted from the candidate training models is run through the equipment of the target platform, and the optimal candidate target model for deployment to the target platform is screened out from the equipment of the candidate target platforms based on the running results; a plurality of models generated in different iteration periods during iterative training of the same neural network can select an optimal model of equipment conforming to a target platform according to the running condition of the equipment of the target platform under the condition that the effect quality sequence of the training environment is inconsistent with the effect quality sequence of the several models on the equipment of a certain target platform, and the deployment effect of the models on the equipment of the target platform can be improved.
The present application provides a model evaluation system, as shown in FIG. 1, which may include a model conversion module and a verification task scheduling module.
The model conversion module is used for reading the training model file and the model conversion parameters of the target platform so as to convert the corresponding training model into the model of the target platform based on the model conversion parameters of the target platform.
And the verification task scheduling module is used for enabling the equipment of the target platform to run each candidate target model to process the material data, so as to obtain verification results of the candidate target models on the material data, and screening out the optimal candidate target model for the equipment deployed to the target platform. Further, the verification task scheduling module can search hardware equipment of a corresponding platform according to target platform information corresponding to the model of the target platform, and transmit the model of the target platform to the hardware equipment matched with the platform information; and running each candidate object model on corresponding hardware equipment to process the material data.
In addition, the model evaluation system may further include a model verification software package generation module. The model verification software package generating module processes a model (such as a candidate target model) of a target platform, an inference program basic compiling command and material data to generate a model inference software package so as to send the model inference software package to equipment of the target platform, and the equipment of the target platform runs each candidate target model to process the material data to obtain verification results of a plurality of candidate target models on the material data, so that optimal candidate target models for equipment deployed to the target platform are screened out.
The model deployment system may further include a verification result reconciliation module. The verification result checking module can be used for checking the verification result of the model by taking the label of the material data as a checking standard, and finally can output statistical data for evaluating the effect of the model.
As shown in fig. 2, the model evaluation method according to an embodiment of the present application includes the following steps. It should be noted that the following step numbers are only for simplifying the description, and are not intended to limit the execution order of the steps, and the steps of this embodiment may be arbitrarily replaced without departing from the technical idea of the present application.
S101: a plurality of candidate training models is obtained.
A plurality of candidate training models may be first obtained, where the plurality of candidate training models are training results of the initial model in a plurality of different iteration cycles, so as to subsequently convert the plurality of candidate training models into a plurality of candidate target models, and then determine an optimal candidate target model on the device of the target platform.
Alternatively, the model evaluation method of the present embodiment may be applied to a model training apparatus, so step S101 may include: iterative training is performed on the initial model to obtain a plurality of candidate training models. The total iteration number of model training may be greater than or equal to the number of multiple candidate training models, that is, a model generated by at least part of the iteration cycles of the initial model in all the iteration cycles may be used as the candidate training model.
Alternatively, the model evaluation method of the present embodiment may be applied to a device of the target platform, or a device other than the model training device and the device of the target platform (e.g., a server). The step S101 may be as follows: a plurality of candidate training models generated by the initial model iterative training are obtained from the model training device.
S102: and converting each candidate training model into a candidate target model by using the model conversion parameters of the target platform to obtain a plurality of candidate target models.
Wherein the target platform is different from the platform of the training device, such that a conversion of the model, in particular a conversion of the model under the platform of the training device, to the target platform is required before deploying the training model to the device of the target platform.
The model conversion parameters are related to the platform type, and can also be related to information such as network structures and/or network weights of the training model. Preferably, the plurality of candidate training models generated by the initial model in a plurality of different iteration cycles can adopt the same conversion parameters corresponding to the equipment of the same target platform so as to maintain the consistency of the conversion parameters of the models, thereby better judging the candidate target model optimal for the equipment of the target platform.
In step S102, model conversion parameters of each candidate training model and target platform may be synchronized to the container environment; and running the container to obtain the corresponding candidate training model and model conversion parameters in the container, and generating a candidate target model of the candidate training model running on the equipment of the target platform by using the model conversion parameters.
Further, in step S102, each candidate training model, information of the target platform, and model conversion parameters of the target platform may be synchronized to the container environment; and running the container to obtain the corresponding candidate training model, the information of the target platform and the model conversion parameters in the container, and generating a candidate target model of the candidate training model running on the equipment of the target platform by using a conversion method and the model conversion parameters corresponding to the platform.
To improve model conversion efficiency, conversion operations of multiple candidate training models may be performed in parallel. Specifically, a plurality of candidate training models can be traversed in sequence, and the traversed candidate training models, model conversion parameters and other information are synchronized to a container environment and a container is started during the traversal; traversing the next candidate training model after the container is started until all candidate training models are synchronized into the corresponding containers; after all containers corresponding to all candidate training models are started, all started containers can be operated concurrently, so that multiple candidate training models are converted into multiple candidate target models through all containers in parallel.
If the conversion operation is completed, that is, the container is completed, whether a candidate target model is generated in the container mapping environment can be searched, and if so, the candidate target model can be saved. Alternatively, the candidate object models may be stored inside the model conversion module. In order to facilitate the evaluation and screening of a plurality of candidate target models, the information of the candidate target models and the target platforms and/or the model conversion parameters can be stored in a combined mode so as to send the candidate target models to the corresponding target platform equipment, so that the situation that the evaluation and screening of the candidate target models cannot be performed due to the fact that the candidate target models are sent incorrectly is avoided.
S103: and processing the material data by using each candidate target model by the equipment of the target platform to obtain verification results of the material data by a plurality of candidate target models, so as to screen out the optimal candidate target models for being deployed to the target platform based on the verification results.
After converting the plurality of candidate training models into a plurality of candidate target models, the equipment of the target platform can process the material data by utilizing each candidate target model to obtain verification results of the plurality of candidate target models on the material data so as to screen out an optimal candidate target model for deploying to the equipment of the target platform, so that the optimal model of the equipment conforming to the target platform can be selected through the running condition of the equipment of the target platform, and the deployment effect of the training model on the equipment of the target platform can be improved.
Optionally, under the condition that the model evaluation method is applied to the equipment of the target platform, after a plurality of candidate target models are obtained through conversion, the obtained candidate target models can be directly utilized to process the material data, so that verification results of the candidate target models on the material data can be obtained.
In some embodiments, when the model evaluation method is applied to a device of a target platform or a training device, etc., after the multiple candidate target models are obtained by conversion in step S102, the multiple candidate target models obtained by conversion may be sent to at least one device of the target platform, so that the device of the target platform processes the material data by using each candidate target model to obtain verification results of the material data by the multiple candidate target models, that is, the candidate target models need to be transmitted to an electronic device with a matched platform structure, so that model reasoning is performed on the electronic device by using the candidate target models to obtain verification results of the material data.
Alternatively, the distribution of the plurality of candidate object models converted may be performed using the method shown in fig. 3. Specifically, the equipment with the platform information conforming to and in the idle state can be searched in the electronic equipment cluster; if the qualified equipment is found, at least one candidate target model can be sent to the qualified equipment, so that the qualified equipment can process the material data by utilizing the at least one candidate target model to obtain a verification result of the material data, and the state of the qualified equipment can be set to be a use state; if no eligible devices are found then the search may continue, for example once at intervals, or may continue until an eligible device is found.
Further, if the equipment with the platform information conforming to the idle state is found, the storage path and the task execution output result path of at least one candidate target model can be mapped to the equipment conforming to the condition, so that the at least one candidate target model is distributed to the equipment conforming to the condition through the mapping; then, in the apparatus, a program is started to process the material data using the candidate object model; after the equipment is executed, a verification result of the corresponding candidate target model on the material data can be obtained in a corresponding task execution output result path, and a task output result file is obtained; and then, the verification result of the candidate target model on the material data can be associated and stored with the corresponding candidate target model.
In order to facilitate the inquiry of the execution status of the device, after mapping at least one candidate target model storage path and a task execution output result path to the device meeting the conditions, a piece of task information can be created and added to a task queue, and the task information can comprise the device information meeting the conditions, the candidate target model information, the output result path information and the like; so that whether the task in the task queue is executed or not can be inquired at regular time; if the task is already executed, the task output result file can be queried from the corresponding output result path.
In an implementation manner, after the device of the target platform processes the material data to obtain the verification result of at least one candidate target model on the material data, the device of the target platform may compare the verification result of each candidate target model on each material data with the label of each material data to confirm whether the verification result of each candidate target model on each material data is correct, so that the verification accuracy of each candidate target model on all material data may be determined, and then all candidate target models may be screened based on the verification accuracy of all candidate target models and other data to determine the optimal candidate target model for deploying to the device of the target platform.
In another implementation manner, the verification results of all candidate target models on all material data may be obtained from at least one device of the target platform, and then the device applying the model evaluation method of the embodiment compares the verification results of each candidate target model on each material data with the labels of each material data to confirm whether the verification results of each candidate target model on each material data are correct, so that the verification accuracy of each candidate target model on all material data can be confirmed, and further, the optimal candidate target model for the device deployed to the target platform may be determined by screening all candidate target models based on the verification accuracy and other data of all candidate target models.
In order to facilitate verification of the verification result, after each candidate target model is utilized by the device of the target platform to process the material data, the verification result of the material data can be named by utilizing the identification of the material data, so that the verification result of the material data can be correspondingly matched with the label of the material data subsequently, and the verification accuracy of the verification result is improved. Namely, the result output file contained in the model verification result is in one-to-one correspondence with the material data identifiers, and the material data identifiers are in one-to-one correspondence with the tag data identifiers, so that the identifiers of the verification result can be in one-to-one correspondence with the tag data identifiers.
When the equipment of the target platform has the operation requirement for obtaining the verification results of the material data by a plurality of candidate target models, the equipment of the target platform can name the verification results of the material data by utilizing the identification of the material data and the identification of the candidate target models so as to distinguish the verification results of different candidate target models on the same material data; of course, the device of the target platform may only use the identifier of the material data to name the verification result of the material data, but after the device runs a candidate target model to process the material data to obtain the verification result of the candidate target model on the material data, the verification result of the material data may be directly stored in the verification result packet of the candidate target model, for example, the verification result of the material data may be directly stored in the output result path of the candidate target model, where the verification result packet of each candidate target model may include the verification result of the corresponding candidate target model on each material data, so that when verifying the verification result, verification may be directly performed on all the verification results of each candidate target model based on the verification result packet of each candidate target model.
When verifying the verification result, not only can the verification result of each candidate target model on each material data be confirmed whether to be correct, but also information such as difference data between the verification result of each candidate target model on each material data and the labels of each material data can be confirmed, and then the correctness and/or the difference data of the verification result of all candidate target models on all material data can be utilized to screen and evaluate all candidate target models.
Optionally, a series of indexes such as accuracy and/or recall of each candidate target model may be evaluated by using correctness and/or difference data of verification results of each candidate target model on all material data, and then all candidate target models may be screened based on indexes such as accuracy and/or recall of all candidate target models to determine an optimal candidate target model for equipment deployed to the target platform.
After verification of the verification results, the verification results and/or the verification results thereof (e.g., identification, correctness, and/or difference data of the verification results, etc.) may be saved for subsequent querying.
Under the condition that the optimal candidate target model of the equipment for being deployed to the target platform is determined, the equipment for determining the optimal candidate target model can send the optimal candidate target model to the equipment of the target platform so as to enable the equipment of the target platform to deploy the optimal candidate target model; or under the condition that the optimal candidate target model is received before the equipment of the target platform, the equipment determining the optimal candidate target model directly informs the information of the optimal candidate target model to the equipment of the target platform so as to enable the equipment of the target platform to find the optimal candidate target model and deploy the optimal candidate target model.
In order to facilitate the device of the target platform to process the material data by using each candidate target model, before step S103, an executable program running the candidate target model may be generated, so that the candidate target model and the material data are read by the device of the target platform through the executable program, and a forward reasoning process of the neural network is run, so as to obtain a verification result of the candidate target model on the material data. Because of the difference of the platform structures used by different electronic devices, the corresponding executable programs of the electronic devices need to be generated for completing the process. Wherein the executable program can be compiled and generated, and the executable programs of different platforms can be generated based on basic compiling commands and device platform parameters. Specifically, an executable program basic compiling command (which can be obtained through user input or other modes) and a platform parameter of the device can be obtained first, and the executable program basic compiling command is used for generating the executable program compiling command of the device in combination with the platform parameter of the device; submitting an executable program compiling task of the equipment to a compiling server, wherein the task information carries executable program compiling commands of the equipment so that the compiling server responds to the task request and compiles and generates the executable program of the equipment based on the executable program compiling commands of the equipment; the executable program of the device is obtained from a compiling server.
In order to improve the comparison of verification results of candidate training models of different iteration periods on equipment of the same target platform, the equipment of the target platform can use the same executable program to infer the same set of material data so as to keep consistency of the executable program and the material data, thereby improving the comparison of the verification results of the candidate training models. In this way, the same executable program, the same set of material data and the plurality of candidate target models can be distributed to at least one device of the target platform, so that the device of the target platform runs the same executable program and uses the plurality of candidate target models to infer the same set of material data. In order to facilitate the distribution of the same executable program, the same set of material data and a plurality of candidate object models, the executable program, the material data and each candidate object model can be assembled into a software package to obtain the software package of the plurality of candidate object models; therefore, the software packages of a plurality of candidate target models can be directly distributed, and the data distribution efficiency and accuracy are improved. Further, the distribution of the storage paths of the software packages of the candidate target models can be performed first, and when the device of the target platform starts to execute the corresponding task, the device of the target platform can acquire the software packages of the candidate target models through the storage paths of the software packages of the candidate target models to execute the corresponding task.
Accordingly, the present application may provide a model evaluation method of another embodiment, which may include: the method comprises the steps that equipment of a target platform obtains a plurality of candidate target models, wherein the candidate target models are obtained by converting a plurality of candidate training models by using model conversion parameters of the target platform, and the candidate training models are training results of an initial model in a plurality of different iteration periods; and processing the material data by utilizing each candidate target model to obtain verification results of a plurality of candidate target models on the material data, and screening out an optimal candidate target model for deployment to the target platform based on the verification results.
As shown in FIG. 4, the present application also provides a model evaluation method of another embodiment, which includes the following steps. It should be noted that the following step numbers are only for simplifying the description, and are not intended to limit the execution order of the steps, and the steps of this embodiment may be arbitrarily replaced without departing from the technical idea of the present application.
S201: and converting the training model into a model of each target platform by using the model conversion parameters of each target platform to obtain models of a plurality of target platforms.
When the training model is converted into the equipment model, conversion parameters are needed, and the conversion parameters are related to information such as different electronic equipment platforms. A training model converts one set of conversion parameters to the equipment of one target platform, and converts the plurality of sets of conversion parameters to the equipment of a plurality of target platforms. Thus, in this embodiment, the input of the model conversion module may include: training a model, a plurality of target platforms and conversion parameters of each platform; the output is: a model of a plurality of target platforms; i.e. by this embodiment the training model is converted into a model on a plurality of target platforms. As shown in fig. 5, the training model (i.e., the source network model in fig. 5) can be read and saved to the internal storage path of the model conversion module for later conversion to read the model content; acquiring platform information of equipment of a target platform to which a user needs to deploy and corresponding model conversion parameters of the equipment, wherein the equipment of the target platform can be a plurality of equipment, and establishing a platform-model conversion parameter combination list; traversing a platform-model conversion parameter combination list, using a mirror image starting container containing an environment required by a conversion model, mapping a stored training model to a container environment, storing a file of the currently traversed platform and model conversion parameters, and mapping the file to the container environment; traversing the next platform after the container is started; if all the target platforms are traversed, starting all the containers and enabling all the containers to run concurrently, acquiring training models, platforms and conversion parameters in the containers, and generating equipment models for running the training models on the platforms by using conversion methods and conversion parameters corresponding to the platforms; searching whether a model of the target platform (i.e. the equipment model in fig. 5) is generated in the container mapping environment after the container runs, and if so, storing the model of the target platform into a storage path in the module; and combining and storing information such as a model storage path and file size of the target platform with the platform and model conversion parameters.
S202: and processing the material data by the equipment of each target platform by using the model of each target platform to obtain verification results of the material data by the models of a plurality of target platforms.
In step S202, the device of each target platform may start the executable program of each target platform and process the same set of material data by using the model of each target platform, so as to obtain verification results of the models of the target platforms on the material data; the executable program of each target platform is compiled based on executable program basic compiling commands and parameters of each target platform. Compared with the prior art, the method only carries out model deployment after model conversion aiming at the target electronic equipment; in the embodiment, the training model and the equipment model are respectively deployed on respective corresponding electronic equipment to perform model reasoning; the reasoning effect of the equipment model can be evaluated more clearly and accurately by comparing the verification and check results of the training model; and meanwhile, whether the model deployment process has problems or not is facilitated to be analyzed. On the premise that factors such as an original neural network model (namely a training model), an inference program, verification materials, labels and the like are kept consistent, deducing on equipment of a plurality of platforms to obtain an inference effect of each equipment model; the feasibility and expected effect of deploying the neural network model on a plurality of platforms are convenient to longitudinally compare.
Before step S202, an executable program of each target platform may be generated, and a software package of each target platform may be generated through the model, the executable program and the material data of each target platform, so that the device of each target platform starts the executable program of each target platform to process the same set of material data by using the model of each target platform.
In one embodiment, before step S202, the model verification software package generating module may generate a corresponding executable program according to the target platform, and package the executable program and the device model and the material data read during the operation reasoning into a software package. Inputs to the model verification software package generation module are as follows: a plurality of models (training models and/or models of target platforms), executable program basic compiling commands, material data, and outputs as: a plurality of models validates the software package. Specifically, as shown in fig. 6, an executable program base compiling command input by a user may be read; reading all input models (the model of the target platform comprises equipment model information and a platform generated by a model conversion module, model conversion parameters and the like, and the training model comprises training model information and a platform) to generate a model list; traversing the model list; reading a platform carried by the current model, and generating an executable program compiling command of the platform by combining the executable program basic compiling command with platform parameters; submitting executable program compiling tasks of the current platform to a compiling server, wherein task information carries relevant compiling commands of the current platform; traversing the next model after the task is submitted; tasks in the compiling server may be performed concurrently; the compiling server generates executable programs of the corresponding platforms according to the received platform related compiling commands; each compiling task completes an executable program which can be downloaded to a corresponding platform; and reading a model of the same platform as the executable program, reading material data, combining the model and the executable program into a model verification software package, and storing the model verification software package into a module storage path.
The model verification software package needs to be transmitted to the device with the matched platform structure (i.e. the device corresponding to the target platform), and the executable program in the model verification software package is started on the relevant device, so as to perform model reasoning to execute step S202. And outputting the reasoning result after the executable program is run, and taking the output result as a model verification result corresponding to the model of the target platform. The input of the verification task scheduling module is a model verification software package, and the output is a model verification result. As shown in fig. 3, platform information carried by the model can be obtained from the model verification software package; searching equipment with platform information conforming to the equipment in an idle state in the electronic equipment cluster, and searching after waiting for a certain time if the equipment does not conform to the condition; setting the found target electronic equipment as equipment in a use state, and mapping a model verification software package storage path and a software package execution output result path to equipment of a target platform; in the target electronic device, starting an executable program; creating task information, including target electronic equipment information, model verification software package information and output result path information, and adding the information into a task queue; the task output result file can be obtained from the output result path of the task information after the task in the task queue is finished being executed or not; platform information is obtained from a model verification software package of the task and is bound with a task output result file, and the platform information is stored as a model verification result.
S203: and evaluating the variability of the reasoning effect of the training model on different target platforms based on the verification result.
After the verification results of the target platforms are obtained based on the steps, the reasoning effect of the training model on the target platforms can be estimated based on the verification results.
Wherein the correctness of the verification result of each target platform can be analyzed. Specifically, the tag data corresponding to the material data can be used as a standard value, the verification result is compared with the standard value, indexes such as accuracy, recall rate and the like of the model in the package can be calculated, and then verification results of a plurality of platforms are compared to evaluate the difference of the reasoning effect of the training model on different target platforms. Further, as shown in fig. 7, the model verification result and the tag data may be read, where the model verification result includes a result output file in which an entry corresponds to a material data entry input by the user, and the material data entry corresponds to a tag data entry, so that an inference result entry included in the verification result corresponds to the tag data entry; traversing all items of the verification result, reading the data of the current item from the tag data, and comparing the data with the verification result data of the current item to obtain information such as consistency, difference data and the like of the data and the data; storing the current item information, consistency, difference data and the like as a check result; according to all the above checking results, a series of indexes such as accuracy, recall and the like for model effect evaluation can be calculated.
If the inference effect of the evaluation training model on a target platform is poor, whether the processes of model conversion, executable program compiling, software package assembling and the like of the target platform have problems or not can be traced.
Referring to fig. 8, fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the application. The present electronic device 20 comprises a processor 22, the processor 22 being adapted to execute instructions to implement the method of model evaluation described above. The specific implementation process is described in the above embodiments, and will not be described herein.
The processor 22 may also be referred to as a CPU (Central Processing Unit ). The processor 22 may be an integrated circuit chip having signal processing capabilities. Processor 22 may also be a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components. The general purpose processor may be a microprocessor or the processor 22 may be any conventional processor or the like.
The electronic device 20 may further comprise a memory 21 for storing instructions and data needed for the operation of the processor 22.
The processor 22 is configured to execute instructions to implement the methods provided by any of the embodiments of the methods of model evaluation of the present application and any non-conflicting combinations described above.
Referring to fig. 9, fig. 9 is a schematic structural diagram of a computer readable storage medium according to an embodiment of the application. The computer readable storage medium 30 of an embodiment of the present application stores instruction/program data 31 that, when executed, implements the methods provided by any of the embodiments of the method of model evaluation of the present application, as well as any non-conflicting combinations. Wherein the instructions/program data 31 may be stored in the storage medium 30 as a software product in a form of a program file, so that a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) performs all or part of the steps of the methods according to the embodiments of the present application. And the aforementioned storage medium 30 includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, an optical disk, or other various media capable of storing program codes, or a terminal device such as a computer, a server, a mobile phone, a tablet, or the like.
In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of elements is merely a logical functional division, and there may be additional divisions of actual implementation, e.g., multiple elements or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.
In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises an element.
The foregoing is only the embodiments of the present application, and therefore, the patent scope of the application is not limited thereto, and all equivalent structures or equivalent processes using the descriptions of the present application and the accompanying drawings, or direct or indirect application in other related technical fields, are included in the scope of the application.
Claims (10)
1. A method of model evaluation, the method further comprising:
obtaining a plurality of candidate training models, wherein the plurality of candidate training models are training results of an initial model in a plurality of different iteration periods;
converting each candidate training model into a candidate target model by using model conversion parameters of a target platform to obtain a plurality of candidate target models;
and processing the material data by using the candidate target models by the equipment of the target platform to obtain verification results of the candidate target models on the material data, and screening out the optimal candidate target models for being deployed to the target platform based on the verification results.
2. The model evaluation method according to claim 1, wherein the causing the device of the target platform to process the material data using each of the candidate target models to obtain verification results of the material data by a plurality of the candidate target models includes:
And searching at least one device with platform information conforming to and in an idle state in the device cluster, and sending a plurality of candidate target models to the at least one device so that the at least one device processes the material data by utilizing each candidate target model to obtain verification results of the candidate target models on the material data.
3. The model evaluation method according to claim 1, wherein the causing the device of the target platform to process the material data with each of the candidate target models includes, before:
acquiring an executable program basic compiling command;
determining an executable program for running the candidate target model based on the executable program basic compiling command and the parameters of the target platform;
distributing the executable program and the plurality of candidate object models to at least one device of the object platform, so that the at least one device starts the executable program to process the material data by using each candidate object model.
4. A model evaluation method according to claim 3, wherein said determining the executable program based on the executable program base compilation command and the parameters of the target platform comprises:
Sending a compiling request to a compiling server, wherein the compiling request carries the executable program basic compiling command and the parameters of the target platform, so that the compiling server responds to the compiling request and generates the executable program based on the executable program basic compiling command and the parameters of the target platform;
the executable program is obtained from the compiling server.
5. A model evaluation method according to claim 3, wherein said determining an executable program running said candidate object model, then comprises:
assembling the executable program, the material data and each candidate object model into a software package to obtain a plurality of software packages;
the at least one device for distributing the executable program and the plurality of candidate object models to the object platform comprises: distributing a plurality of said software packages to said at least one device.
6. The model evaluation method according to claim 1, wherein the causing the device of the target platform to process the material data with each of the candidate target models to obtain verification results of the material data by a plurality of candidate target models, to screen out an optimal candidate target model for deployment to the target platform based on the verification results, includes:
Obtaining verification results of a plurality of candidate target models on the material data from equipment of the target platform;
verifying the verification results of a plurality of candidate target models by using the labels of the material data;
and evaluating and screening a plurality of candidate target models based on the verification result.
7. The model evaluation method according to claim 1, characterized in that the method further comprises:
converting the training model into a model of each target platform by using the model conversion parameters of each target platform to obtain a plurality of models of the target platforms;
enabling the equipment of each target platform to start an executable program of each target platform, and processing the same set of material data by using the model of each target platform to obtain verification results of the models of a plurality of target platforms on the material data; the executable program of each target platform is compiled based on executable program basic compiling commands and parameters of each target platform;
and evaluating the variability of the training model in the reasoning effect of different target platforms based on the verification result.
8. A method of model evaluation, the method further comprising:
The method comprises the steps that equipment of a target platform obtains a plurality of candidate target models, wherein the candidate target models are obtained by converting a plurality of candidate training models by using model conversion parameters of the target platform, and the candidate training models are training results of an initial model in a plurality of different iteration periods;
and processing the material data by utilizing each candidate target model to obtain verification results of a plurality of candidate target models on the material data, and screening out an optimal candidate target model for deployment to the target platform based on the verification results.
9. An electronic device, the electronic device comprising a processor; the processor is configured to execute instructions to implement the steps of the method according to any of claims 1-8.
10. A computer readable storage medium having stored thereon instructions/program data, which when executed, implement the steps of the method of any of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311302188.0A CN117035065A (en) | 2023-10-10 | 2023-10-10 | Model evaluation method and related device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311302188.0A CN117035065A (en) | 2023-10-10 | 2023-10-10 | Model evaluation method and related device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117035065A true CN117035065A (en) | 2023-11-10 |
Family
ID=88637657
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311302188.0A Pending CN117035065A (en) | 2023-10-10 | 2023-10-10 | Model evaluation method and related device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117035065A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117455319A (en) * | 2023-12-25 | 2024-01-26 | 凯美瑞德（苏州）信息科技股份有限公司 | Financial index model calculation processing method, electronic equipment and processing medium |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111191789A (en) * | 2020-01-20 | 2020-05-22 | 上海依图网络科技有限公司 | Model training method, system, chip, electronic device and medium |
| CN111861020A (en) * | 2020-07-27 | 2020-10-30 | 深圳壹账通智能科技有限公司 | Model deployment method, device, equipment and storage medium |
| CN111898706A (en) * | 2020-08-24 | 2020-11-06 | 深圳市富之富信息科技有限公司 | Intelligent iterative deployment method and device of model, computer equipment and storage medium |
| CN112507102A (en) * | 2020-12-18 | 2021-03-16 | 北京百度网讯科技有限公司 | Predictive deployment system, method, apparatus and medium based on pre-training paradigm model |
| CN112540767A (en) * | 2020-12-25 | 2021-03-23 | 北京百度网讯科技有限公司 | Program code generation method, program code generation device, electronic device and storage medium |
| CN113988212A (en) * | 2021-11-10 | 2022-01-28 | 深圳思为科技有限公司 | Management method and device of algorithm model |
| CN114546592A (en) * | 2022-02-18 | 2022-05-27 | 北京市商汤科技开发有限公司 | Model deployment evaluation method, server, electronic device and storage medium |
| CN114861836A (en) * | 2022-07-05 | 2022-08-05 | 浙江大华技术股份有限公司 | Model deployment method based on artificial intelligence platform and related equipment |
| CN114913375A (en) * | 2022-05-17 | 2022-08-16 | 成都商汤科技有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN115049057A (en) * | 2022-08-11 | 2022-09-13 | 浙江大华技术股份有限公司 | Model deployment method and device, electronic equipment and storage medium |
| WO2023019397A1 (en) * | 2021-08-16 | 2023-02-23 | Baidu.Com Times Technology (Beijing) Co., Ltd. | Hardware adaptive multi-model scheduling |
| CN115713108A (en) * | 2022-11-08 | 2023-02-24 | 浙江大华技术股份有限公司 | Packing method and device of model, storage medium and electronic device |
| CN115860143A (en) * | 2022-12-28 | 2023-03-28 | 浙江大华技术股份有限公司 | Operator model generation method, device and equipment |
| US20230168880A1 (en) * | 2021-11-29 | 2023-06-01 | Microsoft Technology Licensing, Llc | Dynamic ring structure for deployment policies for improved reliability of cloud service |
-
2023
- 2023-10-10 CN CN202311302188.0A patent/CN117035065A/en active Pending
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111191789A (en) * | 2020-01-20 | 2020-05-22 | 上海依图网络科技有限公司 | Model training method, system, chip, electronic device and medium |
| CN111861020A (en) * | 2020-07-27 | 2020-10-30 | 深圳壹账通智能科技有限公司 | Model deployment method, device, equipment and storage medium |
| CN111898706A (en) * | 2020-08-24 | 2020-11-06 | 深圳市富之富信息科技有限公司 | Intelligent iterative deployment method and device of model, computer equipment and storage medium |
| CN112507102A (en) * | 2020-12-18 | 2021-03-16 | 北京百度网讯科技有限公司 | Predictive deployment system, method, apparatus and medium based on pre-training paradigm model |
| CN112540767A (en) * | 2020-12-25 | 2021-03-23 | 北京百度网讯科技有限公司 | Program code generation method, program code generation device, electronic device and storage medium |
| WO2023019397A1 (en) * | 2021-08-16 | 2023-02-23 | Baidu.Com Times Technology (Beijing) Co., Ltd. | Hardware adaptive multi-model scheduling |
| CN113988212A (en) * | 2021-11-10 | 2022-01-28 | 深圳思为科技有限公司 | Management method and device of algorithm model |
| US20230168880A1 (en) * | 2021-11-29 | 2023-06-01 | Microsoft Technology Licensing, Llc | Dynamic ring structure for deployment policies for improved reliability of cloud service |
| CN114546592A (en) * | 2022-02-18 | 2022-05-27 | 北京市商汤科技开发有限公司 | Model deployment evaluation method, server, electronic device and storage medium |
| CN114913375A (en) * | 2022-05-17 | 2022-08-16 | 成都商汤科技有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN114861836A (en) * | 2022-07-05 | 2022-08-05 | 浙江大华技术股份有限公司 | Model deployment method based on artificial intelligence platform and related equipment |
| CN115049057A (en) * | 2022-08-11 | 2022-09-13 | 浙江大华技术股份有限公司 | Model deployment method and device, electronic equipment and storage medium |
| CN115713108A (en) * | 2022-11-08 | 2023-02-24 | 浙江大华技术股份有限公司 | Packing method and device of model, storage medium and electronic device |
| CN115860143A (en) * | 2022-12-28 | 2023-03-28 | 浙江大华技术股份有限公司 | Operator model generation method, device and equipment |
Non-Patent Citations (3)
| Title |
|---|
| IOANNIS PRAPAS等: "Continuous Training and Deployment of Deep Learning Models", 《SCHWERPUNKTBEITRAG》, vol. 21, pages 203 - 212, XP037624174, DOI: 10.1007/s13222-021-00386-8 * |
| TATSUYA MATSUSHIMA等: "Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization", 《ARXIV》, pages 1 - 21 * |
| 于培华: "面向移动环境的分布式目标检测框架研究与实现", 《中国优秀硕士学位论文全文数据库 信息科技辑》, vol. 2021, no. 4, pages 138 - 748 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117455319A (en) * | 2023-12-25 | 2024-01-26 | 凯美瑞德（苏州）信息科技股份有限公司 | Financial index model calculation processing method, electronic equipment and processing medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108153520B (en) | Software development method and device | |
| CN109857475B (en) | Framework management method and device | |
| Galindo et al. | Testing variability-intensive systems using automated analysis: an application to android | |
| US11030364B2 (en) | Evaluating autonomous vehicle algorithms | |
| CN110276074B (en) | Distributed training method, device, equipment and storage medium for natural language processing | |
| CN117035065A (en) | Model evaluation method and related device | |
| US20230030296A1 (en) | Task processing method based on defect detection, device, apparatus and storage medium | |
| CN109634837A (en) | Automated testing method, device, equipment and storage medium | |
| CN111026634A (en) | Interface automation test system, method, device and storage medium | |
| US20220284286A1 (en) | Method and apparatus for providing recommendations for completion of an engineering project | |
| US11340898B1 (en) | System and method for automating software development life cycle | |
| CN114546592A (en) | Model deployment evaluation method, server, electronic device and storage medium | |
| CN111459621B (en) | Cloud simulation integration and scheduling method and device, computer equipment and storage medium | |
| US11061739B2 (en) | Dynamic infrastructure management and processing | |
| CN114237754A (en) | Data loading method and device, electronic equipment and storage medium | |
| CN114385196A (en) | Software deployment method and device combining RPA and AI | |
| CN112784273A (en) | SQL risk identification method, device and equipment | |
| CN113360300A (en) | Interface calling link generation method, device, equipment and readable storage medium | |
| WO2023231526A1 (en) | Algorithm warehouse management method and system based on digital twin daas platform | |
| CN111291868A (en) | Network model training method, device, equipment and computer readable storage medium | |
| CN115454420A (en) | Artificial intelligence algorithm model deployment system, method, equipment and storage medium | |
| CN114265595A (en) | Cloud native application development and deployment system and method based on intelligent contracts | |
| CN106775629A (en) | Search file generation method and device | |
| CN113806624B (en) | Data processing method and device | |
| CN105320512A (en) | Method and device for distributing trunk permission |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |