CN113222149B - Model training method, device, equipment and storage medium - Google Patents
Model training method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN113222149B CN113222149B CN202110597904.7A CN202110597904A CN113222149B CN 113222149 B CN113222149 B CN 113222149B CN 202110597904 A CN202110597904 A CN 202110597904A CN 113222149 B CN113222149 B CN 113222149B
- Authority
- CN
- China
- Prior art keywords
- model
- data
- training
- test set
- tested
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012549 training Methods 0.000 title claims abstract description 137
- 238000000034 method Methods 0.000 title claims abstract description 63
- 238000003860 storage Methods 0.000 title claims abstract description 14
- 238000012360 testing method Methods 0.000 claims abstract description 137
- 238000011156 evaluation Methods 0.000 claims abstract description 77
- 238000002372 labelling Methods 0.000 claims abstract description 67
- 238000013136 deep learning model Methods 0.000 claims description 18
- 238000001514 detection method Methods 0.000 claims description 16
- 230000008569 process Effects 0.000 claims description 11
- 230000006870 function Effects 0.000 claims description 7
- 230000009466 transformation Effects 0.000 claims description 6
- 238000000844 transformation Methods 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 5
- 230000003902 lesion Effects 0.000 claims description 5
- 230000000694 effects Effects 0.000 abstract description 5
- 238000005457 optimization Methods 0.000 description 10
- 238000004364 calculation method Methods 0.000 description 6
- 238000013135 deep learning Methods 0.000 description 6
- 230000007246 mechanism Effects 0.000 description 6
- 238000004422 calculation algorithm Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 238000010801 machine learning Methods 0.000 description 4
- 238000013473 artificial intelligence Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000009826 distribution Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 239000005337 ground glass Substances 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 238000003058 natural language processing Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 206010056342 Pulmonary mass Diseases 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000007418 data mining Methods 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 201000010099 disease Diseases 0.000 description 1
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 1
- 238000007667 floating Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/04—Inference or reasoning models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Molecular Biology (AREA)
- General Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a model training method, a model training device, model training equipment and a storage medium. The model training method comprises the following steps: model training is carried out based on the training data set, and an intermediate model is obtained; based on a preset data acquisition type, acquiring data from a data set to be marked as data to be tested, and generating a model reasoning result of the data to be tested based on a current intermediate model; respectively carrying out test evaluation on the model reasoning result based on the dynamic test set and the fixed test set; if the evaluation result of the fixed test set does not meet the standard condition, adjusting the preset data acquisition type based on the evaluation result of the dynamic test set, and performing iterative training on the intermediate model based on the data to be tested corresponding to the adjusted preset data acquisition type until the evaluation result of the fixed test set meets the standard condition, so as to obtain the target model. The dynamic model evaluation is carried out under the driving of the small sample labeling data, the model iteration is carried out rapidly, and the effect of the model conforming to the labeling is further produced rapidly.
Description
Technical Field
The embodiment of the invention relates to a deep learning technology, in particular to a model training method, a device, equipment and a storage medium.
Background
Deep learning, which is an important research direction in the field of machine learning, was introduced into machine learning to make it closer to the original goal of artificial intelligence. In the process of deep learning modeling, generally, starting from original data, data analysis is performed by using a given multi-layer algorithm, initial parameters and initial weights, factors generating differences are searched compared with standard results, related parameters and weights are adjusted, a new round of simulation is performed, and finally, the total tolerance between a calculation result and an actual result is minimized. In the practical application of deep learning, if the deep learning needs to be combined into daily business, one basic problem to be solved is the construction of a network model.
Most of the computation of the artificial intelligence technology belongs to supervised learning computation, and deep learning is applied to medical text NLP, data mining, machine learning, image classification, focus segmentation and other technologies in medical application scenes, and in the scenes, how to perform dynamic model evaluation based on as few labeling data as possible and perform model iteration quickly, so that the time for generating a model conforming to labeling is shortened.
The current model evaluation and iteration methods mainly comprise three types, namely 1) a model iteration method based on an attention mechanism or provided with an neglected area, wherein training data is mainly input into a rapid imaging model, feature extraction is carried out on the training data through N multi-granularity attention modules according to multi-scale information of images and the attention mechanism, feature images extracted by the attention modules are fused, and imaging training is carried out according to gradient updating. 2) The model optimization method based on parameter combination is mainly characterized in that data processing is carried out on the data to be processed through an optimized deep learning model with optimized combination parameters, so that the additional calculation cost in the deep learning model is saved, and the reasoning calculation time and response delay are reduced. 3) The model optimization method based on the preset range mainly comprises the steps of calculating a target floating point value of a target network layer by acquiring an initial output value of the target network layer and the maximum output distribution of the target network layer in a target deep learning model, converting the initial output value based on an output scaling factor under the condition that the initial output value exceeds a first preset range to obtain an initial output value, and optimizing the target deep learning model according to the target output value.
However, the existing model iteration method based on the attention mechanism or the set neglect zone mainly adopts the attention mechanism to perform model iteration, and the attention mechanism has some defects, such as using the attention mechanism in natural language processing, the position information cannot be captured, that is, the sequence relation in the sequence cannot be learned. The existing model optimization method based on parameter combination is mainly used for combining convolution corresponding to a model and batch normalization parameters in an optimization combination mode in a deep learning model optimization process, and the method is an optimization mode of a model parameter layer and cannot solve the problem of rapid iteration in a short time based on a small amount of data. The existing model optimization method based on the preset range mainly takes the artificial setting range as the thought of model optimization, and because of the dependence on the range setting, a large number of inter-class and intra-class changes exist in the actual model iteration and optimization, a large amount of marking data are needed, the data cannot be flexibly utilized, and the model evaluation cannot be dynamically achieved.
Disclosure of Invention
The invention provides a model training method, a device, equipment and a storage medium, which are used for realizing dynamic model evaluation under the driving of small sample labeling data, and rapidly carrying out model iteration so as to rapidly generate a model conforming to labeling.
In a first aspect, an embodiment of the present invention provides a model training method, including:
acquiring a training data set, wherein the training data set comprises initial annotation data;
Model training is carried out based on the training data set, and an intermediate model is obtained;
Based on a preset data acquisition type, acquiring data from a data set to be marked as data to be tested, generating a model reasoning result of the data to be tested based on a current intermediate model, and updating a dynamic test set based on the data to be tested, the model reasoning result of the data to be tested and marking information;
respectively carrying out test evaluation on the model reasoning result based on the dynamic test set and the fixed test set;
and if the evaluation result of the fixed test set does not meet the standard condition, adjusting a preset data acquisition type based on the evaluation result of the dynamic test set, and performing iterative training on the intermediate model based on the data to be tested corresponding to the adjusted preset data acquisition type until the evaluation result of the fixed test set meets the standard condition, so as to obtain a target model.
In an optional embodiment of the present invention, if the evaluation result of the fixed test set does not meet the standard reaching condition, the method further includes:
And extracting difficult samples in the dynamic test set and the fixed test set, and determining the difficult samples as to-be-tested data of iterative training, wherein the difficult samples are samples with a difference value between model reasoning results and corresponding labeling information in the dynamic test set and the fixed test set being larger than a preset value.
In an alternative embodiment of the invention, the loss function during training based on the data to be tested comprising the difficult sample is:
Where ti is labeling information of the ith sample xi, pi is probability that the model prediction xi predicts correctly, and r is a super parameter.
In an alternative embodiment of the invention, the performing model training includes:
Determining the data complexity of unlabeled data, and distributing each data to be tested to a labeling object of a corresponding grade based on the data complexity;
And receiving labeling information of the data to be tested fed back by each labeling object.
In an alternative embodiment of the present invention, the determining the data complexity of each data to be tested in the training data set includes:
Inputting the unlabeled data into a deep learning model to obtain a test value, and determining the data complexity for the test value based on the following formula:
di: the complexity of the image, pi, the test value of the deep learning model; m, performing m geometric transformations on unlabeled data; j and k are the sequence numbers of one of the "m geometric transformations", respectively, so that m is the maximum. And y, the category number of the data trained by the model.
In an alternative embodiment of the present invention, after performing test evaluation on the model inference results based on the dynamic test set and the fixed test set, the method further includes:
And storing each intermediate model and the evaluation result thereof in the iteration process into a model library.
In an alternative embodiment of the present invention, the determining the object model includes:
Determining a target model based on the evaluation result of each intermediate model in the model library;
and when the target model does not meet the standard reaching condition of the fixed test set, continuing to execute the iterative training of the target model.
In an alternative embodiment of the present invention, the evaluation result includes accuracy and recall of intermediate models, and the determining the target model based on the evaluation result of each intermediate model includes:
Determining a model selection parameter F1 value based on the accuracy and recall rate of each intermediate model;
and determining the intermediate model with the maximum F1 value as a target model.
In an alternative embodiment of the present invention, when the target model does not meet the standard reaching condition of the fixed test set, the method further includes:
and obtaining a model adjustment operation, and executing the model adjustment operation, wherein the model adjustment operation comprises one or more of model index adjustment, training data adjustment, test data adjustment and training method adjustment.
In a second aspect, an embodiment of the present invention further provides a model training apparatus, where the model training apparatus includes:
The acquisition module is used for acquiring a training data set, wherein the training data set comprises initial annotation data;
The obtaining module is used for carrying out model training based on the training data set to obtain an intermediate model;
The generation module is used for acquiring data from a data set to be marked as data to be tested based on a preset data acquisition type, generating a model reasoning result of the data to be tested based on a current intermediate model, and updating a dynamic test set based on the data to be tested, the model reasoning result of the data to be tested and marking information;
The evaluation module is used for respectively carrying out test evaluation on the model reasoning result based on the dynamic test set and the fixed test set;
And the adjustment module is used for adjusting the preset data acquisition type based on the evaluation result of the dynamic test set if the evaluation result of the fixed test set does not meet the standard condition, and carrying out iterative training on the intermediate model based on the adjusted data to be tested corresponding to the adjusted preset data acquisition type until the evaluation result of the fixed test set meets the standard condition to obtain a target model.
In a third aspect, embodiments of the present invention further provide a computer device, where the computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the model training method according to any of the embodiments of the present invention when the program is executed.
In a fourth aspect, embodiments of the present invention also provide a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a model training method according to any of the embodiments of the present invention.
According to the invention, the model is dynamically evaluated through the dynamic test set, the conventional evaluation is carried out on the model through the fixed test set, and the preset data acquisition type is adjusted based on the evaluation result of the dynamic test set when the evaluation result of the fixed test set does not meet the standard condition, so that unnecessary homologous data labeling can be saved, labeling and training cost is saved, the problem that a model conforming to the labeling cannot be quickly generated under the driving of small sample labeling data is solved, the dynamic model evaluation is carried out under the driving of the small sample labeling data, and the model iteration is quickly carried out, thereby the effect conforming to the labeled model is quickly generated.
Drawings
FIG. 1 is a flowchart of a model training method according to a first embodiment of the present invention;
FIG. 2 is a block flow diagram of a model training apparatus according to a second embodiment of the present invention;
fig. 3 is a schematic structural diagram of a computer device according to a third embodiment of the present invention.
Detailed Description
The invention is described in further detail below with reference to the drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting thereof. It should be further noted that, for convenience of description, only some, but not all of the structures related to the present invention are shown in the drawings.
Example 1
Fig. 1 is a flowchart of a model training method according to a first embodiment of the present invention, where the embodiment is applicable to the case of medical image labeling, and the trained model may be a focus detection model of a medical image, and by labeling a medical image in an electronic medical record, whether the medical image corresponds to a disease of a certain type can be detected. The method can be executed by a model training device and specifically comprises the following steps:
S110, acquiring a training data set, wherein the training data set comprises initial annotation data.
Wherein the training data set refers to the data set that was originally used to train the model. For example, the model is a focus detection model of a medical image, and the training data set is the medical image. According to the defined business scene, data source and labeling rules, initial labeling data of model training are needed to be prepared, when the business scene is focus detection of a hospital, the initial labeling data can be classified historical medical images, namely the focus types corresponding to the medical images in the historical medical images are known, and the corresponding focus types are labeled on the medical images. The initial data size is assessed by combining the service response period and manpower and material resources. When the embodiment is suitable for medical image labeling, the data source may be a medical image in the electronic medical record.
In addition, when the embodiment is applicable to credit scoring, the business scenario may be that the bank evaluates the loan of the applicant, the credit of the applicant is scored through the credit scoring model, and the loan is issued to the applicant when the credit of the applicant is higher than a preset value. The data source is the historical credit information of the applicant, such as the record of overdue credit card, the record of overdue bar, and the overdue number. The initial labeling set is the credit score corresponding to the historical credit condition of the historical applicant.
And S120, performing model training based on the training data set to obtain an intermediate model.
Among them, in artificial intelligence, it is not so easy to output images/voices expected to be output if the data inputted by a large number of users are accurately and easily recognized in the disordered content, so that the algorithm is particularly important, and the algorithm is a model which we say. Training refers to a process of identifying a target with high identification rate and finding out optimal configuration parameters by using big data. The model training is performed based on the training data set, namely the parameter values of the model are continuously adjusted by utilizing the data in the training data set, and finally a group of parameter values with balanced comparison and higher recognition rate of all parties are counted in the result, wherein the group of parameter values are the results obtained after training, namely the intermediate model.
When model training is carried out, a network structure obtained by using technology type selection can be selected from fast-rcnn series, yolo series or a self-constructed structure if the network structure is a detection task; and if the model is other tasks, the model training work can be carried out by selecting the corresponding model structure.
For example, when the business scenario is a lesion detection of a hospital, the intermediate model is a lesion detection model of a medical image. When the business scenario is a loan assessment of the applicant by a bank, the intermediate model is a credit scoring model.
S130, acquiring data from a data set to be marked as data to be tested based on a preset data acquisition type, generating a model reasoning result of the data to be tested based on a current intermediate model, and updating a dynamic test set based on the data to be tested, the model reasoning result of the data to be tested and marking information.
The data are of different types, for example, when the model is a focus detection model of a medical image, the data type is the focus type, for example, a ground glass nodule, a solid nodule and the like in the lung nodule classification, and the ground glass nodule type. Of course, the types of data may be correspondingly different according to the application scenarios, and are only illustrated herein, and specific data type limitations are not involved.
And selecting the data of the corresponding type from the data set to be marked based on the preset data acquisition type. Labeling the data to be tested through the current intermediate model, wherein the model reasoning result is the result obtained by predicting the data to be tested through the current intermediate model. The dynamic test set refers to a marked data set generated after marking the current batch of data to be tested manually, and marking information is the selection marking performed manually.
And S140, respectively carrying out test evaluation on the model reasoning result based on the dynamic test set and the fixed test set.
Wherein the fixed test set does not change following the iteration of the model, and the model index of the fixed test set is a constant determined by the original business. The fixed test set is a labeling value obtained after all data of the data set to be labeled are labeled, and the labeling value obtained after the current batch of data to be tested is labeled by the dynamic test set. The test evaluation means that the model reasoning result is compared with the labeling values in the fixed test set and the dynamic test set to obtain samples which are predicted to be correct and incorrect in the model reasoning result, and then indexes such as precision, recall rate and the like in the fixed test set can be calculated according to the number of samples which are preset to be correct and preset to be incorrect respectively, and the model quality degree of the current intermediate model is obtained according to the specific values of the indexes. The test evaluation of the model reasoning results is carried out through the fixed test set and the dynamic test set respectively, so that double guarantees are achieved, and the model training effect is improved.
And S150, if the evaluation result of the fixed test set does not meet the standard condition, adjusting a preset data acquisition type based on the evaluation result of the dynamic test set, and performing iterative training on the intermediate model based on the data to be tested corresponding to the adjusted preset data acquisition type until the evaluation result of the fixed test set meets the standard condition, so as to obtain a target model.
The evaluation result refers to a difference condition obtained by comparing the data in the fixed test set with the model reasoning result, for example, the accuracy and recall rate of the current intermediate model are calculated according to the data in the fixed test set and the model reasoning result. The standard condition means that the difference condition between the model reasoning result and the fixed test set reaches a preset value, for example, the standard condition means that the accuracy and the recall rate are greater than 95%, and when the model reasoning result of the current intermediate model is compared with the fixed test set, the accuracy and the recall rate are greater than 95%, the evaluation result is proved to meet the standard condition.
And determining the main type of the data selected from the data set to be marked in the next round according to the evaluation result of the dynamic test set, so that unnecessary homologous data marking can be saved, and marking and training costs can be saved. Data homology is a fundamental assumption that traditional machine learning relies on, i.e., training data and test data obey the same distribution.
Optionally, if the evaluation result of the fixed test set does not meet the standard reaching condition, the method further includes: and extracting difficult samples in the dynamic test set and the fixed test set, and determining the difficult samples as to-be-tested data of iterative training, wherein the difficult samples are samples with a difference value between model reasoning results and corresponding labeling information in the dynamic test set and the fixed test set being larger than a preset value.
Specifically, samples having differences between the model inference results and the corresponding labeling information are also classified into difficult samples (HARD SAMPLE) and easy samples (EASY SAMPLE). The difficult sample refers to a sample with a larger difference between the model reasoning result and the corresponding labeling information, and the easy sample refers to a sample with a smaller difference between the model reasoning result and the corresponding labeling information. For example, when the preset value is 10%, the information [1, 0] is marked, and the model reasoning result is [0.3,0.3,0.4], the difference between the preset value and the model reasoning result is more than 10%, the sample is a difficult sample. When [0.98,0.01,0.01] is predicted to be different from the two by less than 10%, the sample is an easy sample.
Optionally, in the training process based on the data to be tested including the difficult sample, the loss function is:
Where ti is labeling information of the ith sample xi, pi is probability that the model prediction xi predicts correctly, and r is a super parameter.
The labeling information refers to a labeling value of the ith sample manually, and the easy sample refers to a sample with a smaller difference between the model reasoning result and the corresponding labeling information, so if pi is larger than a preset value, it is explained that the difference between the model reasoning result and the corresponding labeling information is smaller, namely, if pi is one easy sample (EASY SAMPLE), the picture is smaller, and if r is the next power, the picture is smaller. The Loss function of Loss allows the difficult samples (HARD SAMPLE) to contribute more to Loss, thus allowing better training results for difficult samplings. The optimization can make the difficult sample contribute more to loss, thereby achieving the effect of loss improvement.
Optionally, performing model training further includes: determining the data complexity of unlabeled data in the training data set, distributing the data to be tested to the labeling objects of corresponding grades based on the data complexity, and receiving labeling information of the data to be tested fed back by the labeling objects.
In the initial stage of model training, the deep learning model needs to be manually trained, and data calibration of corresponding tasks is carried out through active learning. The deep learning model refers to a model iterated in the process of generating an intermediate model from the initial annotation data. Unlabeled data herein refers to data that is unlabeled by the deep learning model when the data for the corresponding task is labeled. And distributing the unlabeled data to the labeling object again for labeling, and finding out the data of the deep learning model missing label. With continuous fine tuning, the latest marked data plus the data that was mispredicted by the current model, i.e., the "forgotten" data, in the originally marked data is used. The model forgetting problem which often occurs in continuous fine tuning can be solved, the data utilization rate is improved, the fine tuning model needs less calculation resources, and the convergence speed is faster than that of the head training.
The labeling object refers to electronic equipment corresponding to personnel for manually labeling unlabeled data, for example, labeling personnel are technicians and specialists, the labeling object refers to electronic equipment of different technicians and specialists, the grades can be 1,2,3 and the like, the specialist corresponds to 1 grade, and the technician corresponds to 2 grades. But also can be a master grade, an expert grade and the like. And the unlabeled data with high data complexity is sent to the labeling object with high grade for labeling, so that the quick labeling is facilitated.
Optionally, determining the data complexity of each data to be tested in the training data set includes: inputting the unlabeled data into a deep learning model to obtain a test value, and determining the data complexity for the test value based on the following formula:
di: the complexity of the data, pi, the test value of the deep learning model; m, performing m geometric transformations on unlabeled data; j and k are the serial numbers of one of the m geometric transformations, so that the maximum is m; and y, the number of categories of the data trained by the model, namely how many categories the data are.
In the initial stage of model training, the deep learning model needs to be manually trained, and data calibration of corresponding tasks is carried out through active learning. The deep learning model refers to a model iterated in the process of generating an intermediate model from the initial annotation data. By the formula, the complexity of the data can be conveniently calculated.
Optionally, after performing test evaluation on the model reasoning result based on the dynamic test set and the fixed test set, the method further includes: and storing each intermediate model and the evaluation result thereof in the iteration process into a model library.
The evaluation result refers to a difference condition obtained by comparing the data in the fixed test set with the model reasoning result, for example, the accuracy and recall rate of the current intermediate model are calculated according to the data in the fixed test set and the model reasoning result.
On the basis of the above embodiment, the determining the target model includes: determining a target model based on the evaluation result of each intermediate model in the model library; and when the target model does not meet the standard reaching condition of the fixed test set, continuing to execute the iterative training of the target model.
The target model is an intermediate model with the optimal evaluation result, and when the target model does not meet the standard reaching condition of the fixed test set, the target model cannot be delivered, and training is needed again.
On the basis of the above embodiment, the evaluation result includes accuracy and recall of intermediate models, and the determining the target model based on the evaluation result of each intermediate model includes:
the model selection parameter F1 value is determined based on the accuracy and recall of each intermediate model.
And determining the intermediate model with the maximum F1 value as a target model.
Wherein TP (a): the actual positive class is predicted to be the number of positive classes; FN (b): the actual positive class predicts the number of negative classes; FP (c): the actual negative class is predicted as the number of positive classes; TN (d): the actual negative class is predicted as the number of negative classes; t=true, f=false, indicating whether the prediction is correct; p=positive, n=negative, indicating whether the prediction result is a Positive class or a Negative class.
Recall (Recall), recall, and correct prediction of positive proportion of all actual positive proportions. The calculation formula of the recall rate is as follows: recall = TP/(tp+fn).
Precision (Precision): the predicted positive samples are correct. The formula of the precision is: precision = TP/(tp+fp).
F1 value (H-mean value), in order to be able to evaluate the merits of different algorithms, the concept of F1 value was proposed on the basis of Precision and Recall to evaluate the Precision and Recall as a whole. The F1 value is the arithmetic mean divided by the geometric mean, and the larger the better, bringing the above formulas for Precision and Recall will find that when the F1 value is small, true Positive increases relatively while false decreases relatively, i.e., both Precision and Recall increase relatively, i.e., F1 weights both Precision and Recall. The calculation formula of the F1 value is as follows:
the larger the F1 value is, the better the model is, and the intermediate model with the largest F1 value is determined as the target model, so that the target model can be the optimal model in the model library.
On the basis of the foregoing embodiment, when the target model does not meet the standard reaching condition of the fixed test set, the method further includes:
and obtaining a model adjustment operation, and executing the model adjustment operation, wherein the model adjustment operation comprises one or more of model index adjustment, training data adjustment, test data adjustment and training method adjustment.
When all the data to be tested in the training data set are tested, the model iteration process is finished because no data is tested at the moment, and if the target model still does not meet the standard reaching condition of the fixed test set, the model itself may be problematic, for example, misselection of model indexes, misselection of training data, misselection of testing data, misselection of training methods and the like are described. The model adjustment operation can be manually input, and the model can be adjusted by acquiring the model adjustment operation and executing the model adjustment operation, so that the condition that the training efficiency is low in the wrong model continuously and automatically is prevented from happening. Correspondingly, continuing to perform iterative training on the target model at this time specifically includes continuing to perform iterative training on the target model after performing the model adjustment operation.
According to the technical scheme, the model is dynamically evaluated through the dynamic test set, the model is conventionally evaluated through the fixed test set, and the preset data acquisition type is adjusted based on the evaluation result of the dynamic test set when the evaluation result of the fixed test set does not meet the standard condition, so that unnecessary homologous data labeling can be saved, labeling and training cost can be saved, the problem that a model conforming to the labeling cannot be quickly generated under the driving of small sample labeling data is solved, dynamic model evaluation is performed under the driving of the small sample labeling data, and model iteration is quickly performed, so that the effect conforming to the labeled model is quickly generated.
Example two
Fig. 2 is a flow chart of a model training device provided in the second embodiment of the present invention, and as shown in fig. 2, the model training device in the embodiment of the present invention may specifically include the following modules:
an obtaining module 61 is configured to obtain a training data set, where the training data set includes initial annotation data.
The obtaining module 62 is configured to perform model training based on the training data set to obtain an intermediate model.
The generating module 63 is configured to obtain data from the data set to be marked as data to be tested based on a preset data obtaining type, generate a model reasoning result of the data to be tested based on a current intermediate model, and update a dynamic test set based on the data to be tested, the model reasoning result of the data to be tested, and labeling information.
And the evaluation module 64 is used for respectively carrying out test evaluation on the model reasoning result based on the dynamic test set and the fixed test set.
And the adjusting module 65 is configured to adjust the preset data acquisition type based on the evaluation result of the dynamic test set if the evaluation result of the fixed test set does not meet the standard condition, and perform iterative training on the intermediate model based on the adjusted data to be tested corresponding to the adjusted preset data acquisition type until the evaluation result of the fixed test set meets the standard condition, so as to obtain a target model.
In an optional embodiment of the present invention, if the evaluation result of the fixed test set does not meet the standard reaching condition, the model training apparatus further includes:
The extraction module is used for extracting difficult samples in the dynamic test set and the fixed test set and determining the difficult samples as data to be tested of iterative training, wherein the difficult samples are samples with model reasoning results different from corresponding labeling information in the dynamic test set and the fixed test set.
In an alternative embodiment of the present invention, the model training apparatus further includes:
the determining module is used for determining the data complexity of each data to be tested in the training data set, and distributing each data to be tested to the labeling object of the corresponding grade based on the data complexity.
And the receiving module is used for receiving the labeling information of the data to be tested fed back by each labeling object.
In an alternative embodiment of the present invention, the determining module is further configured to input the unlabeled data into a deep learning model, obtain a test value, and determine a data complexity for the test value based on the following formula:
in an alternative embodiment of the present invention, the model training apparatus further includes:
And the storage module is used for storing each intermediate model and the evaluation result thereof in the iteration process into a model library.
In an alternative embodiment of the present invention, the model training apparatus further includes:
and the model determining module is used for determining a target model based on the evaluation result of each intermediate model in the model library.
And the iteration executing module is used for continuously executing the iteration training of the target model when the target model does not meet the standard reaching condition of the fixed test set.
In an alternative embodiment of the present invention, the evaluation result includes accuracy and recall of an intermediate model, and the model determination module further includes:
and the selection submodule is used for determining a model selection parameter F1 value based on the accuracy and recall rate of each intermediate model.
And the determining submodule is used for determining the intermediate model with the maximum F1 value as a target model.
In an alternative embodiment of the present invention, the model training apparatus further includes:
the model adjustment module is used for acquiring model adjustment operation and executing the model adjustment operation, and the model adjustment operation comprises one or more of model index adjustment, training data adjustment, test data adjustment and training method adjustment.
Example III
Fig. 3 is a schematic structural diagram of a computer device according to a third embodiment of the present invention, as shown in fig. 3,
Comprising a memory 71, a processor 72 and a computer program stored on the memory 71 and executable on the processor 72, said processor 72 implementing the model training method according to any of the embodiments described above when said program is executed.
The computer device comprises a processor 72, a memory 71, input means 73 and output means 74; the number of processors 72 in the computer device may be one or more, one processor 72 being taken as an example in fig. 3; the processor 72, memory 71, input means 73 and output means 74 in the computer device may be connected by a bus or other means, in fig. 3 by way of example.
The memory 71 is used as a computer readable storage medium for storing software programs, computer executable programs, and modules, such as corresponding program instructions/modules (e.g., acquisition module, generation module, evaluation module, and adjustment module in a model training apparatus) of a model training method in an embodiment of the present invention.
The memory 71 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, at least one application program required for functions; the storage data area may store data created according to the use of the terminal, etc. In addition, the memory 71 may include high-speed random access memory 71, and may also include non-volatile memory 71, such as at least one disk memory 71, flash memory device, or other non-volatile solid-state memory 71. In some examples, memory 71 may further include memory 71 located remotely from processor 72, and such remote memory 71 may be connected to the device/terminal/server via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input means 73 may be used to receive input numeric or character information and to generate key signal inputs related to the power user settings and function control of the device.
The output device 74 may include a display device that displays numeric or character information that may be used to receive input, as well as a production and screen.
The processor 72 executes various functional applications of the computer device and data processing, i.e., implements the model training method described above, by running software programs, instructions, and modules stored in the memory 71.
Example IV
A fourth embodiment of the present invention also provides a storage medium containing computer-executable instructions, which when executed by a computer processor, are for performing a model training method, the method comprising:
A training dataset is obtained, wherein the training dataset comprises initial annotation data.
And performing model training based on the training data set to obtain an intermediate model.
Based on a preset data acquisition type, acquiring data from a data set to be marked as data to be tested, generating a model reasoning result of the data to be tested based on a current intermediate model, and updating a dynamic test set based on the data to be tested, the model reasoning result of the data to be tested and marking information.
And respectively carrying out test evaluation on the model reasoning result based on the dynamic test set and the fixed test set.
And if the evaluation result of the fixed test set does not meet the standard condition, adjusting a preset data acquisition type based on the evaluation result of the dynamic test set, and performing iterative training on the intermediate model based on the data to be tested corresponding to the adjusted preset data acquisition type until the evaluation result of the fixed test set meets the standard condition, so as to obtain a target model.
Of course, the storage medium containing the computer executable instructions provided in the embodiments of the present invention is not limited to the method operations described above, and may also perform the related operations in the model training method provided in any embodiment of the present invention.
From the above description of embodiments, it will be clear to a person skilled in the art that the present invention may be implemented by means of software and necessary general purpose hardware, but of course also by means of hardware, although in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a FLASH Memory (FLASH), a hard disk, or an optical disk of a computer, etc., and include several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method according to the embodiments of the present invention.
It should be noted that, in the embodiment of the model training apparatus, each unit and module included are only divided according to the functional logic, but are not limited to the above-mentioned division, so long as the corresponding functions can be implemented; in addition, the specific names of the functional units are also only for distinguishing from each other, and are not used to limit the protection scope of the present invention.
Note that the above is only a preferred embodiment of the present invention and the technical principle applied. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, while the invention has been described in connection with the above embodiments, the invention is not limited to the embodiments, but may be embodied in many other equivalent forms without departing from the spirit or scope of the invention, which is set forth in the following claims.
Claims (12)
1. A method of model training, comprising:
Acquiring a training data set, wherein the training data set comprises initial labeling data, and the initial labeling data comprises historical medical images labeled with focus types;
Model training is carried out based on the training data set, and an intermediate model is obtained;
Based on a preset data acquisition type, acquiring a medical image from a medical image set to be marked as data to be tested, generating a model reasoning result of the data to be tested based on a current intermediate model, and updating a dynamic test set based on the data to be tested, the model reasoning result of the data to be tested and marking information;
respectively carrying out test evaluation on the model reasoning result based on the dynamic test set and the fixed test set;
And if the evaluation result of the fixed test set does not meet the standard condition, adjusting a preset data acquisition type based on the evaluation result of the dynamic test set, and performing iterative training on the intermediate model based on the data to be tested corresponding to the adjusted preset data acquisition type until the evaluation result of the fixed test set meets the standard condition, so as to obtain a focus detection model.
2. The model training method of claim 1, wherein if the evaluation result of the fixed test set does not meet the criteria for compliance, the method further comprises:
And extracting difficult samples in the dynamic test set and the fixed test set, and determining the difficult samples as to-be-tested data of iterative training, wherein the difficult samples are samples with a difference value between model reasoning results and corresponding labeling information in the dynamic test set and the fixed test set being larger than a preset value.
3. Model training method according to claim 2, characterized in that in training based on the data to be tested comprising the difficult sample, the loss function is:
Where ti is labeling information of the ith sample xi, pi is probability that the model prediction xi predicts correctly, and r is a super parameter.
4. The model training method of claim 1, wherein the performing model training comprises:
Determining the data complexity of unlabeled data, and distributing each data to be tested to a labeling object of a corresponding grade based on the data complexity;
And receiving labeling information of the data to be tested fed back by each labeling object.
5. The model training method of claim 4, wherein determining the data complexity of each data to be tested in the training dataset comprises:
Inputting the unlabeled data into a deep learning model to obtain a test value, and determining the data complexity for the test value based on the following formula:
di: the complexity of the image, pi, the test value of the deep learning model; m, performing m geometric transformations on unlabeled data; j and k are the serial numbers of one of the m geometric transformations, so that the maximum is m; and y, the category number of the data trained by the model.
6. The model training method according to claim 1, further comprising, after performing test evaluation on the model reasoning results based on the dynamic test set and the fixed test set, respectively:
And storing each intermediate model and the evaluation result thereof in the iteration process into a model library.
7. The model training method of claim 6, wherein the determining a lesion detection model comprises:
Determining a focus detection model based on the evaluation result of each intermediate model in the model library;
And when the focus detection model does not meet the standard reaching condition of the fixed test set, continuing to execute iterative training of the focus detection model.
8. The model training method of claim 7, wherein the evaluation results include accuracy and recall of intermediate models, and wherein the determining the lesion detection model based on the evaluation results of each intermediate model comprises:
Determining a model selection parameter F1 value based on the accuracy and recall rate of each intermediate model;
and determining the intermediate model with the maximum F1 value as a focus detection model.
9. The model training method of claim 7, wherein when the lesion detection model does not meet the criteria for compliance of the fixed test set, the method further comprises:
and obtaining a model adjustment operation, and executing the model adjustment operation, wherein the model adjustment operation comprises one or more of model index adjustment, training data adjustment, test data adjustment and training method adjustment.
10. A model training device, comprising:
The acquisition module is used for acquiring a training data set, wherein the training data set comprises initial labeling data, and the initial labeling data comprises historical medical images labeled with focus types;
The obtaining module is used for carrying out model training based on the training data set to obtain an intermediate model;
The generation module is used for acquiring medical images from a medical image set to be marked as data to be tested based on a preset data acquisition type, generating a model reasoning result of the data to be tested based on a current intermediate model, and updating a dynamic test set based on the data to be tested, the model reasoning result of the data to be tested and marking information;
The evaluation module is used for respectively carrying out test evaluation on the model reasoning result based on the dynamic test set and the fixed test set;
And the adjusting module is used for adjusting the preset data acquisition type based on the evaluation result of the dynamic test set if the evaluation result of the fixed test set does not meet the standard condition, and carrying out iterative training on the intermediate model based on the data to be tested corresponding to the adjusted preset data acquisition type until the evaluation result of the fixed test set meets the standard condition, so as to obtain the focus detection model.
11. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the model training method according to any of claims 1-9 when executing the program.
12. A computer readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements a model training method according to any of claims 1-9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110597904.7A CN113222149B (en) | 2021-05-31 | 2021-05-31 | Model training method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110597904.7A CN113222149B (en) | 2021-05-31 | 2021-05-31 | Model training method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113222149A CN113222149A (en) | 2021-08-06 |
| CN113222149B true CN113222149B (en) | 2024-04-26 |
Family
ID=77099322
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110597904.7A Active CN113222149B (en) | 2021-05-31 | 2021-05-31 | Model training method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113222149B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113688989B (en) * | 2021-08-31 | 2024-04-19 | 中国平安人寿保险股份有限公司 | Deep learning network acceleration method, device, equipment and storage medium |
| WO2023060954A1 (en) * | 2021-10-14 | 2023-04-20 | 北京百度网讯科技有限公司 | Data processing method and apparatus, data quality inspection method and apparatus, and readable storage medium |
| CN114067109B (en) * | 2022-01-13 | 2022-04-22 | 安徽高哲信息技术有限公司 | Grain detection method, grain detection device and storage medium |
| CN114741269B (en) * | 2022-04-14 | 2022-09-23 | 网思科技股份有限公司 | Method for estimating service performance of reasoning system |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109829375A (en) * | 2018-12-27 | 2019-05-31 | 深圳云天励飞技术有限公司 | A kind of machine learning method, device, equipment and system |
| WO2020119075A1 (en) * | 2018-12-10 | 2020-06-18 | 平安科技（深圳）有限公司 | General text information extraction method and apparatus, computer device and storage medium |
-
2021
- 2021-05-31 CN CN202110597904.7A patent/CN113222149B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020119075A1 (en) * | 2018-12-10 | 2020-06-18 | 平安科技（深圳）有限公司 | General text information extraction method and apparatus, computer device and storage medium |
| CN109829375A (en) * | 2018-12-27 | 2019-05-31 | 深圳云天励飞技术有限公司 | A kind of machine learning method, device, equipment and system |
Non-Patent Citations (1)
| Title |
|---|
| 标准化全连接残差网络空战目标威胁评估;翟翔宇;杨风暴;吉琳娜;李书强;吕红亮;;火力与指挥控制;20200615(06);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113222149A (en) | 2021-08-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113222149B (en) | Model training method, device, equipment and storage medium | |
| CN107633265B (en) | Data processing method and device for optimizing credit evaluation model | |
| CN111369016B (en) | Image recognition-based on-site operation and maintenance method and system | |
| CN109271539B (en) | Image automatic labeling method and device based on deep learning | |
| US11861925B2 (en) | Methods and systems of field detection in a document | |
| KR101953190B1 (en) | A multidimensional recursive learning process and system used to discover complex dyadic or multiple counterparty relationships | |
| JPWO2015146113A1 (en) | Identification dictionary learning system, identification dictionary learning method, and identification dictionary learning program | |
| CN114202243A (en) | Engineering project management risk early warning method and system based on random forest | |
| CN110310012B (en) | Data analysis method, device, equipment and computer readable storage medium | |
| Nababan et al. | Implementation of K-Nearest Neighbors (KNN) algorithm in classification of data water quality | |
| CN116775879A (en) | Fine tuning training method of large language model, contract risk review method and system | |
| CN113192028B (en) | Quality evaluation method and device for face image, electronic equipment and storage medium | |
| CN111914166B (en) | Correction strategy personalized recommendation system applied to community correction personnel | |
| CN113569018A (en) | Question and answer pair mining method and device | |
| Sameki et al. | ICORD: Intelligent Collection of Redundant Data-A Dynamic System for Crowdsourcing Cell Segmentations Accurately and Efficiently. | |
| CN116665272A (en) | Airport scene face recognition fusion decision method and device, electronic equipment and medium | |
| CN111582404B (en) | Content classification method, device and readable storage medium | |
| CN115410250A (en) | Array type human face beauty prediction method, equipment and storage medium | |
| CN114238768A (en) | Information pushing method and device, computer equipment and storage medium | |
| CN114240318A (en) | Target object oriented information processing method and device and computer equipment | |
| CN113987351A (en) | Artificial intelligence based intelligent recommendation method and device, electronic equipment and medium | |
| CN112184417A (en) | Business approval method, device, medium and electronic equipment | |
| CN112861689A (en) | Searching method and device of coordinate recognition model based on NAS technology | |
| CN116701962B (en) | Edge data processing method, device, computing equipment and storage medium | |
| CN117035434B (en) | Suspicious transaction monitoring method and suspicious transaction monitoring device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |