CN114676797B - Model precision calculation method and device and computer readable storage medium - Google Patents
Model precision calculation method and device and computer readable storage medium Download PDFInfo
- Publication number
- CN114676797B CN114676797B CN202210589396.2A CN202210589396A CN114676797B CN 114676797 B CN114676797 B CN 114676797B CN 202210589396 A CN202210589396 A CN 202210589396A CN 114676797 B CN114676797 B CN 114676797B
- Authority
- CN
- China
- Prior art keywords
- network model
- precision
- result
- full
- deployment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/217—Validation; Performance evaluation; Active pattern learning techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/10—Interfaces, programming languages or software development kits, e.g. for simulating neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Biophysics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Mathematical Physics (AREA)
- Evolutionary Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Image Analysis (AREA)
Abstract
The application discloses a method and a device for calculating model precision and a computer-readable storage medium. The method comprises the following steps: deploying a full-precision model on designated equipment to obtain a deployment model, wherein the full-precision model comprises a plurality of processing layers; when the deployment model is judged to meet the preset correction condition, acquiring an abnormal sample from a preset sample set; processing the abnormal sample by adopting a full-precision model to obtain a full-precision model result; processing the abnormal sample by adopting a deployment model to obtain a deployment model result; determining the gradient of each processing layer in the full-precision model according to the full-precision model result and the label data of the abnormal sample; and determining the precision value between the processing layer in the full-precision model and the corresponding processing layer in the deployment model based on the gradient, the full-precision model result and the deployment model result. Through the mode, the accuracy of precision comparison can be improved.
Description
Technical Field
The application relates to the technical field of deep learning, in particular to a method and a device for calculating model precision and a computer readable storage medium.
Background
After the trained network model is deployed to a certain device, for some reasons, the model may generate a certain accuracy loss, so that the accuracy of the network model is reduced, and in order to find out which network layers of the network model have problems, the accuracies of the trained network model and the model deployed to the device may be compared, for example: the Euclidean distance-based scheme, the statistical data distribution-based scheme or the cosine similarity-based scheme is adopted, but the effect is poor.
Disclosure of Invention
The application provides a model precision calculation method, a model precision calculation device and a computer readable storage medium, which can improve the precision comparison accuracy.
In order to solve the technical problem, the technical scheme adopted by the application is as follows: a method for calculating model accuracy is provided, which comprises the following steps: deploying a full-precision model on designated equipment to obtain a deployment model, wherein the full-precision model comprises a plurality of processing layers; when the deployment model is judged to meet the preset correction condition, obtaining an abnormal sample from a preset sample set; processing the abnormal sample by using a full-precision model to obtain a full-precision model result; processing the abnormal sample by adopting a deployment model to obtain a deployment model result; determining the gradient of each processing layer in the full-precision model according to the full-precision model result and the label data of the abnormal sample; and determining the precision value between the processing layer in the full-precision model and the corresponding processing layer in the deployment model based on the gradient, the full-precision model result and the deployment model result.
In order to solve the above technical problem, another technical solution adopted by the present application is: there is provided a model accuracy calculation apparatus comprising a memory and a processor connected to each other, wherein the memory is used for storing a computer program, and the computer program is used for implementing the calculation method of model accuracy in the above technical solution when being executed by the processor.
In order to solve the above technical problem, another technical solution adopted by the present application is: the model precision calculating device comprises a deployment module, a processing module, a gradient sampling module and a comparison module, wherein the deployment module is used for deploying a full-precision model on specified equipment to obtain a deployment model, and the full-precision model comprises a plurality of processing layers; the processing module is connected with the deployment module and is used for acquiring an abnormal sample from a preset sample set when the deployment model is judged to meet a preset correction condition; processing the abnormal sample by adopting a full-precision model to obtain a full-precision model result; processing the abnormal sample by adopting a deployment model to obtain a deployment model result; the gradient sampling module is connected with the processing module and is used for determining the gradient of each processing layer in the full-precision model according to the full-precision model result and the label data of the abnormal sample; and the comparison module is connected with the processing module and the gradient sampling module and is used for determining the precision value between the processing layer in the full-precision model and the corresponding processing layer in the deployment model based on the gradient, the full-precision model result and the deployment model result.
In order to solve the above technical problem, another technical solution adopted by the present application is: there is provided a computer-readable storage medium for storing a computer program for implementing the method of calculating model accuracy in the above-described technical solution when the computer program is executed by a processor.
Through the scheme, the beneficial effects of the application are that: deploying a full-precision model to obtain a deployment model; then judging whether the deployment model meets preset correction conditions or not, if so, acquiring abnormal samples from a preset sample set, and respectively inputting a full-precision model and a deployment model to obtain a full-precision model result and a deployment model result; then, calculating the gradient of each processing layer in the full-precision model by using the full-precision model result and the label data of the abnormal sample; calculating the precision value between each processing layer in the full-precision model and the corresponding processing layer in the deployment model by using the gradient of each processing layer in the full-precision model, the full-precision model result and the deployment model result; due to the introduction of the gradient, the calculation accuracy value by utilizing the effective characteristics of the abnormal sample can be ensured, the influence of invalid characteristics is avoided, the accuracy and the reliability of the precision positioning algorithm are improved, the smoothness degree of the precision descending curve is improved, burrs in the precision descending curve are reduced, and effective support is provided for the subsequent precision tuning work based on the precision positioning result.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the description below are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings without creative efforts. Wherein:
FIG. 1 is a schematic illustration of a degradation of accuracy curve provided herein;
FIG. 2 is a schematic flow chart diagram illustrating an embodiment of a method for calculating model accuracy provided herein;
FIG. 3 is a schematic flow chart diagram illustrating a method for calculating model accuracy according to another embodiment of the present disclosure;
FIG. 4 is a schematic diagram of feature sampling in the forward inference process provided herein;
FIG. 5 is a schematic diagram of gradient sampling during reverse transmission based on loss values as provided herein;
FIG. 6 is a schematic diagram of the gradient weighted correction provided by the present application for the same layer feature of different images;
FIG. 7 is a schematic diagram of the gradient weighted correction provided by the present application for correcting different layer features of the same image;
FIG. 8 is a schematic structural diagram of an embodiment of a model accuracy calculation apparatus provided herein;
FIG. 9 is a schematic structural diagram of another embodiment of a model accuracy calculation apparatus provided herein;
FIG. 10 is a schematic diagram of the operation of the modules in the gradient weighting correction process provided herein;
FIG. 11 is a schematic structural diagram of an embodiment of a computer-readable storage medium provided in the present application.
Detailed Description
The present application will be described in further detail with reference to the following drawings and examples. It is to be noted that the following examples are only illustrative of the present application, and do not limit the scope of the present application. Likewise, the following examples are only some examples and not all examples of the present application, and all other examples obtained by a person of ordinary skill in the art without any inventive work are within the scope of the present application.
Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the specification. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.
It should be noted that the terms "first", "second" and "third" in the present application are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implying any number of indicated technical features. Thus, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of the feature. In the description of the present application, "plurality" means at least two, e.g., two, three, etc., unless explicitly specifically limited otherwise. Furthermore, the terms "include" and "have," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements but may alternatively include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
In practical use, for the solution based on the cosine similarity calculation model, a glitch phenomenon occurs in the forward reasoning process after the layer is raised and then lowered or after the layer is lowered and then raised, for example, as shown in fig. 1, a precision descending curve, where the abscissa is the layer number set in the layer connection order and the ordinate is the precision value, and the reason for this phenomenon is: on one hand, one image is not enough to show the overall distribution, and on the other hand, the redundancy degree and the characteristic dimension of data of each layer are different; the phenomenon of burrs is most pronounced at the key functional layers of some feature dimension transitions, for example: a Linear rectification function (relu) layer or a normalized exponential function (softmax) layer, etc. Based on the above, the present application provides a precision loss positioning method based on gradient correction, which introduces the gradient of the full-precision model into a precision comparison framework, and is described in detail below.
Referring to fig. 2, fig. 2 is a schematic flowchart illustrating an embodiment of a method for calculating model accuracy according to the present application, where an execution subject of the method is a model accuracy calculation apparatus, and the method includes:
s21: and deploying the full-precision model on the specified equipment to obtain a deployment model.
The full-precision model comprises a plurality of processing layers, and the structure of the deployment model is the same as that of the full-precision model; preparing a sufficient, necessary and evenly distributed data set according to an application scene design scheme, wherein the data set comprises a plurality of training samples; and carrying out data annotation on each training sample to obtain corresponding label data. Further, the training samples may be preprocessed to increase the number of training samples or to increase the quality of the training samples.
And constructing a deep learning model, and designing a layer connection structure and a layer dimension structure of the deep learning model, namely designing a connection mode of processing layers, a dimension of input data of each processing layer and a dimension of an output result of each processing layer. After the deep learning model is constructed, a loss function can be designed according to the distribution and the labeling form of the data set and the final layer structure of the deep learning model. Then, the deep learning model is trained by using the data set, and a full-precision model is obtained. The full-precision model is then deployed to a specified device, which may be a model precision computation means, such as: a computer or a server.
S22: and when the deployment model is judged to meet the preset correction condition, acquiring an abnormal sample from a preset sample set.
After the full-precision model is deployed to the model precision computing device, a preset sample set can be adopted to test the deployment model so as to judge whether the deployment model meets a preset correction condition, namely, the precision value of the deployment model is detected. Specifically, if the deployment model meets the preset correction condition, it indicates that the precision value of the deployment model is low, and at this time, the full-precision model and the deployment model need to be processed to determine which processing layers in the deployment model have problems.
Further, the sample set includes a plurality of test samples, and when the deployment model satisfies the preset correction condition, a test sample with poor accuracy may be selected from all the test samples as an abnormal sample, that is, the sample set includes an abnormal sample. For example, the precision value is an identification rate, assuming that the deployment model is a face identification model, the test samples are marked as A1-A2, the identification rate of the test sample A1 is smaller than a preset threshold, and the identification rate of the test sample A2 is larger than the preset threshold, the test sample A1 is determined as an abnormal sample, and subsequent processing is performed.
S23: processing the abnormal sample by adopting a full-precision model to obtain a full-precision model result; and processing the abnormal sample by adopting a deployment model to obtain a deployment model result.
After the abnormal sample is obtained, inputting the abnormal sample into a full-precision model so that the full-precision model processes the abnormal sample to obtain a full-precision model result, wherein the full-precision model result comprises a characteristic diagram output by each processing layer in the full-precision model; and inputting the abnormal sample into the deployment model, so that the deployment model processes the abnormal sample to obtain a deployment model result, wherein the deployment model result comprises a feature diagram output by each processing layer in the deployment model.
S24: and determining the gradient of each processing layer in the full-precision model according to the full-precision model result and the label data of the abnormal sample.
Taking a training sample of the full-precision model as an example, because images adopted for training are widely distributed and are huge in quantity, the effect of the network weight of the full-precision model on different images is different from image to image, and the gradient can describe whether each feature tensor in the feature graph corresponding to the processing layer plays an important role in reasoning of a specific image, so that the gradient of each processing layer in the full-precision model can be obtained.
Further, after the full-precision model result is obtained, a feature map output by the last processing layer in the full-precision model result is obtained, the feature map is the final output of the full-precision model, the difference between the feature map and the label data corresponding to the abnormal sample is calculated, a loss value can be obtained, and the loss value is input into the full-precision model to obtain the gradient of each processing layer in the full-precision model.
S25: and determining the precision value between the processing layer in the full-precision model and the corresponding processing layer in the deployment model based on the gradient, the full-precision model result and the deployment model result.
After the gradients of the processing layers in the full-precision model are obtained, the full-precision model result and the deployment model result can be combined to calculate the precision value between the full-precision model and the deployment model, and a specific implementation scheme is introduced as follows:
(1) Determining the weighting coefficient of each processing layer based on the gradient, and respectively correcting the full-precision model result and the deployment model result based on the weighting coefficient; and calculating a precision value based on the corrected full-precision model result and the corrected deployment model result.
Firstly, calculating the weighting coefficient of each processing layer based on the gradient; then, correcting the full-precision model result and the deployment model result by using the weighting coefficient of the processing layer to obtain a corrected full-precision model result and a corrected deployment model result; and calculating the corrected full-precision model result and the corrected deployment model result by adopting a precision comparison function to obtain a precision value.
In a specific embodiment, taking an abnormal sample as an image and an output result of a processing layer as a feature map as an example, considering that the accuracy evaluation loses alignment due to different feature dimensions and different redundancy degrees of different layers or local tensor sets in a full-accuracy model, the embodiment corrects the dimension difference between the feature maps by using gradients, and corrects the accuracy comparison to be performed under a relatively close dimension. Specifically, a formula may be set, and the formula is used to calculate the original dimensionality and the weighting coefficient of the data output by the processing layer to obtain a dimensionality reduction difference value so as to describe the degree of the drawn dimensionality; it is understood that after the weighting method is selected, the degree of dimensional pull-up of different layers of different abnormal samples can be calculated.
After the corrected full-precision model result and the corrected deployment model result are obtained, the corrected full-precision model result and the corrected deployment model result may be processed by using a preset precision comparison function to obtain a comparison result, where the comparison result includes precision values between each processing layer in the full-precision model and a corresponding processing layer in the deployment model. Specifically, the modified full-precision model result includes a modified output result of each processing layer in the full-precision model, and the modified deployment model result includes a modified output result of each processing layer in the deployment model; the precision comparison function may be a function of the calculation precision in the related art, such as: and calculating the characteristic diagram of the ith processing layer in the corrected full-precision model result and the characteristic diagram of the ith processing layer in the corrected deployment model result based on the Euclidean distance scheme, the statistical data distribution scheme or the cosine similarity scheme to obtain the precision value of the ith processing layer, wherein i is more than or equal to 1 and less than or equal to N, and N is the total number of the processing layers in the full-precision model.
(2) Modifying the precision comparison function based on the gradient to obtain a modified precision comparison function; and calculating the full-precision model result and the deployment model result by adopting the modified precision comparison function to obtain a precision value.
In addition to modifying the input data to the precision comparison function, the precision comparison function itself may also be modified, such as: modifying the values of some parameters in the precision comparison function to obtain a modified precision comparison function; then, the full-precision model result and the deployment model result are input into the modified precision comparison function, and a corresponding precision value can be obtained.
In other implementation manners, the solution may also be implemented by cooperation of an Artificial Intelligence (AI) open platform and a designated device, where the AI open platform is a platform commonly used in the related art and is responsible for tasks of model training and calculating precision values, and the specific scheme is as follows:
1) And after the AI open platform obtains a plurality of training samples, training the training samples to obtain a full-precision model.
2) And after receiving a deployment instruction issued by a user, the AI open platform deploys the full-precision model to the specified equipment to obtain a deployment model arranged on the specified equipment.
3) The AI open platform acquires an output result of the deployment model, and detects the output result of the deployment model to judge whether the deployment model meets a preset correction condition; and if the AI open platform judges that the deployment model meets the preset correction condition, sending a control instruction to the specified equipment.
4) And after receiving the control instruction, the designated equipment acquires an abnormal sample from a preset sample set, and processes the abnormal sample by adopting a deployment model to obtain a deployment model result.
5) The AI open platform acquires an abnormal sample from the specified equipment, and a full-precision model is adopted to process the abnormal sample to obtain a full-precision model result; and the AI open platform determines the gradient of each processing layer in the full-precision model according to the full-precision model result and the label data of the abnormal sample.
6) And the AI open platform acquires a deployment model result from the specified equipment, and determines the precision value between the processing layer in the full-precision model and the corresponding processing layer in the deployment model based on the gradient, the full-precision model result and the deployment model result.
And after the AI open platform acquires the precision value between the processing layer in the full-precision model and the corresponding processing layer in the deployment model, performing other service processing by using the acquired precision value. For example, the AI open platform may retrain the full-precision model using the precision value, etc. This is merely an example and is not particularly limited.
The embodiment provides a precision positioning algorithm of a deep learning model based on a gradient correction mechanism, which corrects output results of a full-precision model and a deployment model by adopting the gradient of the full-precision model respectively, so that the effect is better when precision values of all processing layers of the full-precision model and the deployment model are calculated subsequently; the scheme focuses more on the difference of feature distribution under different images and different layers, especially on the part of data distribution which has influence on final errors. For the scheme without adopting gradient correction, when precision comparison is carried out, the effective data distribution and the ineffective data distribution cannot be distinguished, so that the interpretability is reduced; and the effective characteristic distribution in the characteristic diagram of the abnormal sample can be introduced into the precision comparison function by introducing the gradient, so that the influence of an ineffective characteristic tensor is avoided, the effectiveness and the fairness of a precision positioning algorithm are improved, a precision descending curve is smoother, and effective support is provided for subsequent precision tuning work based on a precision positioning result, such as: and code reconfiguration optimization, parameter reconfiguration or model retraining and the like are performed, so that the accuracy of the deployment model is improved.
Referring to fig. 3, fig. 3 is a schematic flowchart illustrating another embodiment of the method for calculating model accuracy provided in the present application, the method including:
s301: and deploying the full-precision model on the specified equipment to obtain a deployment model.
The full-precision model includes a plurality of processing layers, and S301 is the same as S21 in the above embodiment, and is not described herein again.
S302: and acquiring a sample set, inputting the test samples in the sample set into the deployment model, and acquiring a current processing result.
The sample set comprises a plurality of test samples, the current processing result comprises a processing result corresponding to each test sample, and the test samples are input into the deployment model, so that the deployment model processes the test samples to obtain corresponding processing results (recorded as the current processing results).
S303: and calculating the current accuracy based on the label data of the test sample and the current processing result.
The sample set also comprises label data corresponding to the test sample, and whether the processing result of the deployment model on the test sample is the same as or higher in similarity with the label data can be judged by comparing the label data with the corresponding processing result, if so, the processing is considered to be correct, and if not, the processing is considered to be wrong; the current accuracy can be obtained by counting whether the processing results of all the test samples are correct or not. For example, assuming that there are M test samples, if the processing results of N (N < M) test samples are correct, the current accuracy is N/M.
It can be understood that the current accuracy may be a loss value, and the scheme for calculating the current accuracy by using the tag data and the current processing result is the same as the scheme adopted for calculating the loss value in the related art, that is, the tag data and the current processing result are substituted into the loss function to obtain the loss value, and the loss value is taken as the current accuracy. Alternatively, the current accuracy may be an average of at least two loss values, that is, the current accuracy is generated by averaging the loss values corresponding to different abnormal samples.
S304: and judging whether the deployment model meets a preset correction condition or not based on the current accuracy.
Judging whether the current accuracy reaches a preset accuracy threshold value or not; if the current accuracy rate does not reach the preset accuracy threshold, the processing accuracy of the deployment model is not met, the deployment model is determined to meet the preset correction condition, and the deployment model needs to be analyzed to determine which processing layers in the deployment model have problems; if the current accuracy reaches the preset accuracy threshold, the deployment model is indicated to not meet the preset correction condition, at the moment, the processing accuracy of the deployment model is high, and the deployment model does not need to be corrected.
S305: if the deployment model meets the preset correction condition, acquiring an abnormal sample from a preset sample set, and processing the abnormal sample by adopting a full-precision model to obtain a full-precision model result; and processing the abnormal sample by adopting a deployment model to obtain a deployment model result.
If the deployment model is detected to meet the preset correction condition, selecting an abnormal sample from the multiple test samples, and taking the abnormal sample as the input of the precision positioning process, namely inputting the abnormal sample into the full-precision model and the deployment model respectively to generate a full-precision model result and a deployment model result, wherein the full-precision model result comprises the output result of each processing layer in the full-precision model, the output result of the processing layer in the full-precision model is a feature map generated by the processing layer in the full-precision model, the deployment model result comprises the output result of each processing layer in the deployment model, and the output result of the processing layer in the deployment model is a feature map generated by the processing layer in the deployment model.
After the full-precision model result is obtained, performing gradient sampling operation on the abnormal sample, namely obtaining the gradient of each processing layer in the full-precision model; since the determinants of the gradient are the network structure, the weight parameters, the input data and the loss function of the full-precision model, the loss function is the only factor for determining the gradient given the network structure, the weight parameters and the input data, and the optimal selection scheme of the loss function is the loss function used in the training for reusing the full-precision model. Understandably, in the training process, under the condition that the adopted loss function cannot be known, the label data can be used for constructing an imaginary loss function; alternatively, when there is no tag data, the input data may be input to the full-precision model to obtain tag data corresponding to the input data.
S306: sampling an output result of a processing layer in the full-precision model to obtain a first sampling result; and sampling the output result of the processing layer in the deployment model to obtain a second sampling result.
Assuming that the number of processing layers in the full-precision model is N, the principle of feature sampling is shown in fig. 4, each abnormal sample (i.e., abnormal sample 1 to abnormal sample p) is subjected to two forward reasoning processes, one is the reasoning of the full-precision model, and the other is the reasoning of the actual deployment model; the feature sampling module samples each layer of feature map.
Further, in the forward reasoning process of the model, the characteristic sampling module is used for sampling the processing layers 1-N. It will be appreciated that one or more feature sampling modules may be used for the entire sampling process, and that sampling of all processing layers may be done at one time without limitation, depending on the limitations of the device sampling and computational power.
S307: and acquiring an output result of the last processing layer in the full-precision model result, and calculating a loss value between the label data and the output result of the last processing layer to obtain a loss value.
The output result of the last processing layer in the full-precision model result can be recorded as output data, and the output data and the label data of the abnormal sample are input into a loss function, so that a loss value corresponding to the abnormal sample can be obtained.
S308: the loss values are transmitted back to each processing layer in the full-precision model to obtain the corresponding gradient.
The loss values are transmitted back to each processing layer in the full-precision model using methods in the related art to obtain the gradient of each processing layer. For example, the principle of gradient sampling is shown in fig. 5, the gradient acquired by the gradient sampling module is the global gradient of the feature tensor in the full-precision model under each abnormal sample, and each abnormal sample only undergoes one gradient sampling process.
S309: based on the weighting coefficient, correcting the first sampling result to obtain a corrected full-precision model result; and modifying the second sampling result based on the weighting coefficient to obtain a modified deployment model result.
Performing the same weighted correction operation on the results of the two times of characteristic sampling by using the same gradient sampling result; specifically, the modified full-precision model result includes a modified output result for each processing layer in the full-precision model; the revised deployment model result includes a revised output result for the processing layer in the revised deployment model for each processing layer in the deployment model.
In a specific embodiment, the gradient of each processing layer in the full-precision model can be adjusted to obtain a weighting coefficient; multiplying the weighting coefficient of the processing layer by the corresponding first sampling result to obtain a corrected output result; and multiplying the weighting coefficient of the processing layer by the corresponding second sampling result to obtain a corrected output result.
Further, the calculation method of the weighting coefficient is flexible, for example: calculating the absolute value of the gradient of the processing layer, and normalizing the absolute value to obtain a weighting coefficient; alternatively, the even power of the gradient of the processing layer is calculated and normalized to obtain the weighting coefficient, for example, the gradient may be normalized in an energy manner, or the fourth power of the gradient may be calculated and normalized. It will be appreciated that other ways of calculating the weighting coefficients may be used, following the principle: the weighting coefficient of the feature tensor having a larger influence on the loss value is larger so as to reduce the influence of the redundant feature in the precision comparison.
And each abnormal sample is corrected by adopting respective gradient, after gradient weighting correction, the influence of the feature tensor in the invalid processing layer is eliminated, effective features are extracted for comparison, inaccuracy caused by directly using all the features for comparison is avoided, and the accuracy of comparison is favorably improved. In addition, the weighting coefficient can be adjusted, so that an uncorrected result and a corrected result can be given at the same time, the improvement of a correction mode is guided, and an expandable correction comparison scheme is realized.
S310: and calculating the distance between the corrected output result of each processing layer in the full-precision model and the corrected output result of the corresponding processing layer in the deployment model to obtain the precision value corresponding to the processing layer.
And inputting the corrected output result of the ith processing layer in the full-precision model and the corrected output result of the ith processing layer in the deployment model into a precision comparison function to obtain the precision value of the ith processing layer. Specifically, the precision comparison function may be various distance formulas, various statistical distribution formulas, cosine similarity, or the like, and the input data of the precision comparison function is corrected in this embodiment, so that different images are transmitted to feature maps of each layer to be corrected to different degrees, thereby reducing the dimension difference between the feature maps.
Furthermore, the characteristic diagram with large parameter quantity has more redundant characteristics and small gradient, and is corrected more; the feature map with less parameters, less redundant features, large gradient and less correction. Meanwhile, the characteristic difference between different types of processing layers is also corrected, the processing layer with large nonlinear distortion has large gradient correction; and the characteristic parameter correction is small for a processing layer with small nonlinear distortion. For example, the effect of relu is corrected, and when the input of relu is positive, the gradient correction is minimal; when the input of relu is negative, the gradient is 0, and the correction effect is large, which corresponds to the value being deleted in the feature. It is worth to be noted that when the input of relu is a negative number with a large value, the corrected value is equivalent to an abnormal value and is deleted, so that the comparison results before and after the relu operation are consistent, and the results are reliable; however, in the unmodified scheme, the comparison result before and after the relu operation is executed has a large difference, so that a burr phenomenon is generated, and the comparison result is very inaccurate.
In an embodiment, the sampling, correcting and comparing processes may be performed for different images for multiple times, the correction principle of the gradient on the same layer feature of different images is shown in fig. 6, and fig. 6 shows the output features of the layers K (i.e., the kth processing layer) of pictures 1 to N; the principle of correction of the same layer of different images consists in: the gradient distribution difference of different images is large, the weight of the feature tensor with large features and small gradients is reduced, the interference of the invalid feature tensor can be reduced, and the characteristic dimensionality can be favorably drawn.
The label data of different images are different, under the same loss function, the errors are different, so that the overall gradients reversely transmitted to a specific layer are different, the correction principle of the gradients on the characteristics of different layers of the same image is shown in fig. 7, and fig. 7 shows the output characteristics of the image in layers 1 to K; the correction principle of different layer features of the same image is as follows: the characteristic dimensions and the nonlinear distortion degrees of different layers are different, the influence of the nonlinear distortion is accumulated on the gradient which changes layer by layer, the gradient weighting has a correction effect on the nonlinear distortion on one hand, and on the other hand, the characteristic dimensions of different layers can be drawn closer, so that the burr phenomenon which appears in the comparison result of the characteristics of different layers before correction can be gentle through the gradient weighting correction, as shown in fig. 7, the accuracy descending curve is smoother and the burr phenomenon is not obvious compared with that before correction.
Considering that information of different layers in the deep learning model is complex, dimension difference is large, nonlinear distortion exists, and only the feature sampling module is used to obtain feature data of forward reasoning in the deep learning model, which is not enough to obtain a reliable comparison result, the embodiment provides a mechanism of gradient correction, calculates a loss value by using a full-precision model, reversely propagates the loss value to each local feature tensor, and generates a gradient; the precision comparison function is guided based on a gradient correction mode, the effectiveness of accurate numerical values (namely gradients) representing data characteristic distribution is introduced, high-dimensional information is efficiently reduced in dimension, extra judgment and branch logic do not need to be added, and the instantaneity is high; moreover, the introduction of the gradient also corrects local information loss caused by a nonlinear function, and the interpretability of a local precision calculation result is enhanced.
In other embodiments, the deployment model may be updated based on the precision value, and the step of inputting the test samples in the sample set to the deployment model is returned until the deployment model meets the preset correction condition. Specifically, a tuning strategy is implemented according to the precision value, such as: code reconstruction optimization, parameter reconfiguration, model retraining, compression cutting or knowledge distillation and the like; after the tuning strategy is executed, S301 may be executed again.
The scheme provided by the embodiment has strong adaptability, can be applied to correction of a plurality of common schemes, and can adapt to different computing resources and data distribution scenes; compared with a scheme without correction, the scheme not only reduces the dimension characteristic difference, reduces the influence of nonlinear distortion, highlights the effect of effective characteristics and avoids the burr phenomenon, but also is simple to implement, does not change the traditional comparison process, and is expanded on the basis of the traditional comparison process; compared with a scheme based on cosine similarity and various distances, the method and the device have the advantages that the extra information of the gradient is used, and the effective angles and the effective distances of the features are more reliably obtained; compared with a scheme based on statistical distribution, the method can calculate the precision values of all processing layers in the deployment model by adopting a small number of represented abnormal samples, so that the calculation amount is small, and a reliable result can be obtained.
Referring to fig. 8, fig. 8 is a schematic structural diagram of an embodiment of a model accuracy calculation apparatus provided in the present application, the model accuracy calculation apparatus 80 includes a memory 81 and a processor 82 connected to each other, the memory 81 is used for storing a computer program, and when the computer program is executed by the processor 82, the computer program is used for implementing a calculation method of model accuracy in the foregoing embodiment.
Referring to fig. 9, fig. 9 is a schematic structural diagram of another embodiment of the model accuracy calculation apparatus provided in the present application, and the model accuracy calculation apparatus 90 includes a deployment module 91, a processing module 92, a gradient sampling module 93, and a comparison module 94.
The deployment module 91 is configured to deploy the full-precision model on the specified device to obtain a deployment model, where the full-precision model includes multiple processing layers.
The processing module 92 is connected to the deployment module 91, and is configured to obtain an abnormal sample from a preset sample set when it is determined that the deployment model meets a preset correction condition; processing the abnormal sample by using a full-precision model to obtain a full-precision model result; and processing the abnormal sample by adopting a deployment model to obtain a deployment model result.
The gradient sampling module 93 is connected to the processing module 92, and is configured to determine a gradient of each processing layer in the full-precision model according to the full-precision model result and the label data of the abnormal sample.
A comparison module 94 is connected to the processing module 92 and the gradient sampling module 93 and is configured to determine a precision value between a processing layer in the full-precision model and a corresponding processing layer in the deployment model based on the gradient, the full-precision model result, and the deployment model result.
Further, based on the gradient, correcting the full-precision model result and the deployment model result to obtain a corrected full-precision model result and a corrected deployment model result; calculating the corrected full-precision model result and the corrected deployment model result by adopting a precision comparison function to obtain a comparison result, wherein the comparison result comprises precision values between a processing layer in the full-precision model and a corresponding processing layer in the deployment model; or modifying the precision comparison function based on the gradient to obtain a modified precision comparison function; and calculating the full-precision model result and the deployment model result by adopting the modified precision comparison function to obtain a comparison result.
Compared with the scheme in the related technology, the scheme is additionally provided with the gradient sampling module 93, the gradient of the full-precision model is obtained by adopting the gradient sampling module 93, the precision comparison function is corrected by using the gradient in the comparison module 94 or the characteristic diagram of the prepared input precision comparison function is corrected, so that the accurate characteristic information of the corresponding image in the deep learning model can be reserved, and the part of the characteristic which directly contributes to the loss function is effectively extracted.
In an embodiment, the working process of each module is as shown in fig. 10, the model accuracy calculation apparatus 90 further includes a feature sampling module 95, the feature sampling module 95 is connected to the comparison module 94, and the feature sampling module 95 is configured to sample feature maps output by the processing layers in the full-accuracy model and the deployment model; the gradient sampled by the gradient sampling module 93 is used to modify the precision comparison function used in the comparison module 94 or modify the input of the precision comparison function, thereby changing the output of the precision comparison function and obtaining a reliable comparison result.
The embodiment provides a feature comparison scheme based on gradient correction aiming at the problem of model precision positioning, and the feature comparison is guided by the gradient of a full-precision model, so that the comparison result is more reliable, the interpretability is stronger, and the smoothness of the comparison result is higher.
Referring to fig. 11, fig. 11 is a schematic structural diagram of an embodiment of a computer-readable storage medium provided in the present application, where the computer-readable storage medium 110 is used for storing a computer program 111, and the computer program 111 is used for implementing a method for calculating model accuracy in the foregoing embodiment when being executed by a processor.
The computer readable storage medium 110 may be a server, a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and various media capable of storing program codes.
In the several embodiments provided in the present application, it should be understood that the disclosed method and apparatus may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, a division of modules or units is merely a logical division, and an actual implementation may have another division, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one position, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each of the units may exist alone physically, or two or more units may be integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
If the technical scheme of the application relates to personal information, a product applying the technical scheme of the application clearly informs personal information processing rules before processing the personal information, and obtains personal independent consent. If the technical scheme of the application relates to sensitive personal information, a product applying the technical scheme of the application obtains individual consent before processing the sensitive personal information, and simultaneously meets the requirement of 'express consent'. For example, at a personal information collection device such as a camera, a clear and significant identifier is set to inform that the personal information collection range is entered, the personal information is collected, and if the person voluntarily enters the collection range, the person is regarded as agreeing to collect the personal information; or on the device for processing the personal information, under the condition of informing the personal information processing rule by using obvious identification/information, obtaining personal authorization by modes of popping window information or asking a person to upload personal information of the person by himself, and the like; the personal information processing rule may include information such as a personal information processor, a personal information processing purpose, a processing method, and a type of personal information to be processed.
The above embodiments are merely examples, and not intended to limit the scope of the present application, and all modifications, equivalents, and flow charts using the contents of the specification and drawings of the present application, or those directly or indirectly applied to other related arts, are included in the scope of the present application.
Claims (13)
1. A method for calculating the accuracy of a network model, which is executed by an artificial intelligence open platform, and comprises the following steps:
deploying a full-precision network model on a designated device to obtain a deployed network model, wherein the full-precision network model comprises a plurality of processing layers;
when the deployment network model is judged to meet a preset correction condition, sending a control instruction to the designated equipment so that the designated equipment responds to the control instruction, acquiring an abnormal sample from a preset sample set, and processing the abnormal sample by adopting the deployment network model to obtain a deployment network model result;
acquiring the abnormal sample from the specified equipment, and processing the abnormal sample by adopting the full-precision network model to obtain a full-precision network model result; the abnormal sample is a sample with the precision smaller than a preset precision value in the sample set;
determining the gradient of each processing layer in the full-precision network model according to the full-precision network model result and the label data of the abnormal sample;
determining a weighting coefficient of each processing layer based on the gradient, and respectively correcting the full-precision network model result and the deployment network model result based on the weighting coefficient; calculating an accuracy value based on the corrected full-accuracy network model result and the corrected deployment network model result; or
Modifying a precision comparison function based on the gradient; and calculating the full-precision network model result and the deployment network model result by adopting the modified precision comparison function to obtain the precision value.
2. The method according to claim 1, wherein before the step of sending a control instruction to the designated device when the deployed network model is determined to meet a preset correction condition, so that the designated device obtains an abnormal sample from a preset sample set in response to the control instruction, the method comprises:
acquiring the sample set, and inputting the test samples in the sample set into the deployment network model to obtain a current processing result;
calculating the current accuracy rate based on the label data of the test sample and the current processing result;
and judging whether the deployment network model meets the preset correction condition or not based on the current accuracy.
3. The method according to claim 2, wherein the step of determining whether the deployed network model satisfies the preset correction condition based on the current accuracy includes:
judging whether the current accuracy reaches a preset accuracy threshold value or not;
if not, determining that the deployment network model meets the preset correction condition.
4. The method of claim 2, wherein the full-precision network model result comprises an output result of each processing layer in the full-precision network model, and the step of determining the gradient of each processing layer in the full-precision network model according to the full-precision network model result and the label data of the abnormal sample comprises:
obtaining an output result of the last processing layer in the full-precision network model result;
calculating a loss value between the tag data and an output result of the last processing layer to obtain a loss value;
and transmitting the loss value back to each processing layer in the full-precision network model to obtain a corresponding gradient.
5. The method of claim 4, wherein the deployed network model result comprises an output result of each processing layer in the deployed network model; the step of correcting the full-precision network model result and the deployment network model result based on the weighting coefficients respectively comprises:
sampling an output result of a processing layer in the full-precision network model to obtain a first sampling result;
sampling an output result of a processing layer in the deployment network model to obtain a second sampling result;
based on the weighting coefficient, correcting the first sampling result to obtain a corrected full-precision network model result, wherein the corrected full-precision network model result comprises a corrected output result of each processing layer in the full-precision network model;
and modifying the second sampling result based on the weighting coefficient to obtain a modified deployment network model result, wherein the modified deployment network model result comprises a modified output result of each processing layer in the deployment network model.
6. The method according to claim 5, wherein the step of modifying the first sampling result based on the weighting factor to obtain the modified full-precision network model result comprises:
multiplying the weighting coefficient of the processing layer by the corresponding first sampling result to obtain the corrected output result;
the step of modifying the second sampling result based on the weighting coefficient to obtain the modified deployment network model includes:
and multiplying the weighting coefficient of the processing layer by the corresponding second sampling result to obtain the corrected output result.
7. The method of calculating the accuracy of a network model of claim 2, further comprising:
updating the deployment network model based on the precision value, and returning to the step of inputting the test sample in the sample set to the deployment network model until the deployment network model meets the preset correction condition.
8. The method of claim 1, wherein the step of determining the weighting coefficients of the processing layers based on the gradients comprises:
calculating an absolute value of the gradient of the processing layer, and normalizing the absolute value to obtain the weighting coefficient; or alternatively
And calculating the even power of the gradient of the processing layer and normalizing to obtain the weighting coefficient.
9. The method according to claim 1, wherein the step of calculating the full-precision network model result and the deployed network model result by using the modified precision comparison function to obtain the precision value comprises:
and calculating the distance between the corrected output result of each processing layer in the full-precision network model and the corrected output result of the corresponding processing layer in the deployment network model to obtain the precision value corresponding to the processing layer.
10. The method of calculating network model accuracy of claim 1,
the abnormal sample is an image, and the output result of the processing layer is a feature map.
11. A network model accuracy computation apparatus, comprising a memory and a processor connected to each other, wherein the memory is used for storing a computer program, and the computer program is used for implementing the network model accuracy computation method according to any one of claims 1 to 10 when being executed by the processor.
12. A network model accuracy calculation apparatus, comprising:
the deployment module is used for deploying the full-precision network model on the specified equipment to obtain a deployed network model, and the full-precision network model comprises a plurality of processing layers;
the processing module is connected with the deployment module and used for sending a control instruction to the designated equipment when the deployment network model is judged to meet a preset correction condition, so that the designated equipment responds to the control instruction, obtains an abnormal sample from a preset sample set, and processes the abnormal sample by adopting the full-precision network model to obtain a full-precision network model result; acquiring the abnormal sample from the specified equipment, and processing the abnormal sample by adopting the deployment network model to obtain a deployment network model result; the abnormal sample is a sample with the precision smaller than a preset precision value in the sample set;
the gradient sampling module is connected with the processing module and is used for determining the gradient of each processing layer in the full-precision network model according to the full-precision network model result and the label data of the abnormal sample;
the comparison module is connected with the processing module and the gradient sampling module and used for determining the weighting coefficient of each processing layer based on the gradient and respectively correcting the full-precision network model result and the deployment network model result based on the weighting coefficient; calculating an accuracy value based on the corrected full-accuracy network model result and the corrected deployment network model result; or
Modifying a precision comparison function based on the gradient; and calculating the full-precision network model result and the deployment network model result by adopting the modified precision comparison function to obtain the precision value.
13. A computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, is configured to implement the method for calculating the accuracy of a network model according to any one of claims 1 to 10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210589396.2A CN114676797B (en) | 2022-05-27 | 2022-05-27 | Model precision calculation method and device and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210589396.2A CN114676797B (en) | 2022-05-27 | 2022-05-27 | Model precision calculation method and device and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114676797A CN114676797A (en) | 2022-06-28 |
| CN114676797B true CN114676797B (en) | 2022-10-04 |
Family
ID=82080308
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210589396.2A Active CN114676797B (en) | 2022-05-27 | 2022-05-27 | Model precision calculation method and device and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114676797B (en) |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11842280B2 (en) * | 2017-05-05 | 2023-12-12 | Nvidia Corporation | Loss-scaling for deep neural network training with reduced precision |
| KR20210108413A (en) * | 2018-12-18 | 2021-09-02 | 모비디어스 리미티드 | Neural Network Compression |
| CN112749798A (en) * | 2020-07-22 | 2021-05-04 | 腾讯科技（深圳）有限公司 | Compression method of neural network model and related device |
| CN112906502B (en) * | 2021-01-29 | 2023-08-01 | 北京百度网讯科技有限公司 | Training method, device, equipment and storage medium of target detection model |
| CN113076663A (en) * | 2021-05-06 | 2021-07-06 | 华南理工大学 | Dynamic hybrid precision model construction method and system |
| CN113449854A (en) * | 2021-08-30 | 2021-09-28 | 阿里云计算有限公司 | Method and device for quantifying mixing precision of network model and computer storage medium |
| CN113780466B (en) * | 2021-09-27 | 2024-02-02 | 重庆紫光华山智安科技有限公司 | Model iterative optimization method, device, electronic equipment and readable storage medium |
| CN114118384B (en) * | 2021-12-09 | 2024-06-04 | 安谋科技（中国）有限公司 | Quantification method of neural network model, readable medium and electronic device |
-
2022
- 2022-05-27 CN CN202210589396.2A patent/CN114676797B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN114676797A (en) | 2022-06-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP1739580B1 (en) | Categorization including dependencies between different category systems | |
| CN110472675B (en) | Image classification method, image classification device, storage medium and electronic equipment | |
| US20080208836A1 (en) | Regression framework for learning ranking functions using relative preferences | |
| US6931351B2 (en) | Decision making in classification problems | |
| US20210073635A1 (en) | Quantization parameter optimization method and quantization parameter optimization device | |
| CN117131348B (en) | Data quality analysis method and system based on differential convolution characteristics | |
| CN114676797B (en) | Model precision calculation method and device and computer readable storage medium | |
| CN112270334B (en) | Few-sample image classification method and system based on abnormal point exposure | |
| CN116522912B (en) | Training method, device, medium and equipment for package design language model | |
| CN112801231A (en) | Decision model training method and device for business object classification | |
| CN111523604A (en) | User classification method and related device | |
| CN114495114B (en) | Text sequence recognition model calibration method based on CTC decoder | |
| CN116956171A (en) | Classification method, device, equipment and storage medium based on AI model | |
| CN113688875B (en) | Industrial system fault identification method and device | |
| US20240020531A1 (en) | System and Method for Transforming a Trained Artificial Intelligence Model Into a Trustworthy Artificial Intelligence Model | |
| KR20230127509A (en) | Method and apparatus for learning concept based few-shot | |
| CN114492657A (en) | Plant disease classification method and device, electronic equipment and storage medium | |
| CN112069800A (en) | Sentence tense recognition method and device based on dependency syntax and readable storage medium | |
| CN109800923A (en) | distributed wind power generation short-term power combination prediction method | |
| CN117389821A (en) | Log abnormality detection method, device and storage medium | |
| EP4083870A1 (en) | Method and system for classifying data | |
| CN117666981B (en) | TTS voice broadcasting method, device and printer | |
| JP5652250B2 (en) | Image processing program and image processing apparatus | |
| CN116755910B (en) | Host machine high availability prediction method and device based on cold start and electronic equipment | |
| CN117113148B (en) | Risk identification method, device and storage medium based on time sequence diagram neural network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |