CN117496191B - Data weighted learning method based on model collaboration - Google Patents
Data weighted learning method based on model collaboration Download PDFInfo
- Publication number
- CN117496191B CN117496191B CN202410004710.5A CN202410004710A CN117496191B CN 117496191 B CN117496191 B CN 117496191B CN 202410004710 A CN202410004710 A CN 202410004710A CN 117496191 B CN117496191 B CN 117496191B
- Authority
- CN
- China
- Prior art keywords
- image
- target
- sample
- auxiliary
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 56
- 238000011156 evaluation Methods 0.000 claims abstract description 15
- 238000012549 training Methods 0.000 claims description 34
- 230000011218 segmentation Effects 0.000 claims description 14
- 238000013441 quality evaluation Methods 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 5
- 238000012795 verification Methods 0.000 claims description 3
- 238000012935 Averaging Methods 0.000 claims description 2
- 238000001514 detection method Methods 0.000 abstract description 5
- 238000003709 image segmentation Methods 0.000 abstract description 3
- 238000012216 screening Methods 0.000 abstract description 3
- 238000002372 labelling Methods 0.000 description 4
- 238000013528 artificial neural network Methods 0.000 description 3
- 230000006978 adaptation Effects 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 108010015780 Viral Core Proteins Proteins 0.000 description 1
- 230000004931 aggregating effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 210000002569 neuron Anatomy 0.000 description 1
- 230000008092 positive effect Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/72—Data preparation, e.g. statistical preprocessing of image or video features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/776—Validation; Performance evaluation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/98—Detection or correction of errors, e.g. by rescanning the pattern or by human intervention; Evaluation of the quality of the acquired patterns
- G06V10/993—Evaluation of the quality of the acquired pattern
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/70—Labelling scene content, e.g. deriving syntactic or semantic representations
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Artificial Intelligence (AREA)
- Quality & Reliability (AREA)
- Computational Linguistics (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a data weighted learning method based on model collaboration, and defines a sample re-weighting method for image data. And calculating auxiliary weights and target weights of an input image sample by utilizing a pre-trained auxiliary model set and a target model to be trained and utilizing a prediction result obtained by each auxiliary model set and a real label, and then obtaining final sample weights by fusing a certain proportion. The weighting loss of the samples is calculated using the above strategy, and the target model updates the parameters by minimizing the weighting loss, focusing the target model on learning valuable samples. The invention can greatly reduce the labor cost for screening low-quality data and improve the efficiency. Meanwhile, the invention is suitable for various computer vision tasks, such as image segmentation, image classification, target detection and the like, and can establish a new standard universal frame for a data validity evaluation scheme.
Description
Technical Field
The invention relates to the technical field of image instance segmentation, in particular to a data weighted learning method based on model collaboration.
Background
In recent years, artificial intelligence technology has been rapidly developed, and new sparks are continuously collided with by the combination of computer vision, and penetrate into various aspects of learning, work and life, so that the artificial intelligence technology is closely related to human science and technology development. Image classification, object detection, and image segmentation are three major core problems of computer vision. Image instance segmentation is an important branch of research in the field of image segmentation. The example segmentation task refers to locating the position of a potential target in an image, wherein the position is represented by a detection target frame, and pixel-by-pixel marking is performed in different target areas by using a semantic segmentation mode.
With the proposal of the full convolution network, image instance segmentation algorithm research based on deep learning starts to appear in the field of view of people. The method has the outstanding advantages of automatic and multi-layer feature extraction, training of the network by inputting a large number of image samples with labels or notes into the learning network, adjusting the weight of connection between neurons according to error detection, repeatedly optimizing the network to realize end-to-end classification learning, and predicting the image samples without labels after completion. The application effect of the method in the aspect of image instance segmentation is greatly improved in accuracy compared with the traditional algorithm.
However, deep learning requires a large number of training image samples to be accurately labeled when fitting a model, and such a finely labeled image dataset is mainly completed through manual labeling, so that the cost is high and the efficiency is low, and therefore, the training image dataset is difficult to acquire in a real scene. In the process of collecting and labeling an image data set, the problems of low quality of an image sample and inconsistent labeling quality generally occur, and the problem of inconsistent difficulty of the image sample also exists in model training, so that the difficulty of identifying and processing the low-quality image sample is great.
Learning data-driven curriculum for very deep neural networks on corrupted labls, in ICML, 2018 pre-trains an additional auxiliary network and then uses it to select a clean instance to guide training of the target network. The two networks are trained simultaneously by the Decoupling "white to update" from "how to update". In NeurIPS, 2017, and for each batch of samples, the prediction results are given to the two networks respectively, and when the prediction results are inconsistent, the backward propagation gradient update is carried out, but as the training times increase, the two networks gradually tend to be consistent, and functionally gradually degenerate into a self-training single target model. In order to solve the problems, co-training is Robust training of deep neural networks with extremely noisy labes, in NeurIPS, 2018 trains two deep neural networks at the same time, in each small batch of data, the two networks select label data which is as clean as possible according to the acquired loss result when feeding all data forward, the two networks determine the trained data by selecting a small loss method, and finally the other network performs back propagation on the selected data to update the super parameters of the network weight. "How does disagreement help generalization against label corruption". Then, among these inconsistent data, each network selects its own small-loss data to teach the other network, propagates the small-loss data from the other network, and updates its own parameters.
The method is focused on detecting the low-quality image sample, and re-labeling the image data set after discarding the low-quality image sample, however, the method does not dig the potential value of the low-quality image sample, so that the waste of invalid acquisition is caused.
Disclosure of Invention
The invention aims to: aiming at the problems in the background art, the invention provides a data weighted learning method based on model cooperation, which considers that the existing data effectiveness evaluation is mainly focused on detecting low-quality image samples, and the data is re-marked after being discarded, and the value of the image samples is not dug out, so that a new standard general framework is designed for the data effectiveness evaluation scheme, namely, the quality of the image samples is evaluated, the image sample re-weighting scheme is adopted to enable a target model to tend to learn the image samples with high quality, and the discarding of too many low-quality image samples is avoided, so that the model training is assisted by using the model.
The technical scheme is as follows: in order to achieve the above purpose, the invention adopts the following technical scheme:
a data weighted learning method based on model collaboration comprises the following steps:
step S1, giving a general image datasetTarget image dataset->Verification set->Giving an auxiliary model set consisting of a plurality of auxiliary models differing in both decision boundary and learning ability>Wherein->Indicate->Auxiliary model->For natural numbers greater than 0, give the target model +.>The method comprises the steps of carrying out a first treatment on the surface of the Wherein the set of auxiliary models->In general image dataset->Pre-training is carried out on the device;
step S2, randomly selecting the target image datasetImage sample->Auxiliary model set->Is->The individual auxiliary models are all for image samples +.>Generating a prediction result to obtain a prediction result setWherein->Indicate->The individual auxiliary model is +.>The prediction result generated is then passed through the image samples +.>The set of prediction results generated->And image sample->Corresponding real tag calculate image sample +.>Is->Auxiliary weights are obtained to obtain auxiliary weight set +.>Wherein->Indicate->Auxiliary weights generated by the auxiliary model are then calculated +.>Averaging the auxiliary weights to obtain an image sampleFinal auxiliary weights of (2);
finally, the target image dataset is calculatedFinal auxiliary weight of other image samples in the target image dataset, and according to the size of the final auxiliary weight +.>Ordering all the image samples in the image, if the final auxiliary weight of a certain image sample is smaller than the set threshold +.>Then the image sample is +.>Discarding to obtain cleaned target image dataset +.>；
Step S3, randomly selecting a target image datasetThe image sample is defined as +.>Through the object model->Generating an image sample->Is not yet controlled by the pre-processing of (2)Measurement of->Through image sample->Generated prediction result->Image sample->Corresponding real tag calculate image sample +.>Target weight and sample loss of (2), then using the target model +.>Calculating the target image dataset +.>Target weights and sample losses for all other image samples;
step S4, collecting the target image data setThe final auxiliary weight and the target weight obtained by each image sample in the image are fused to obtain sample weight;
step S5, collecting the target image data setRe-weighting the sample loss obtained for each image sample in the model with the sample weight to obtain a weighted sample loss, and finally updating the target model by using gradient calculation>Is->Is used for training;
step S6, when the target image data setThe training is completed, and when all training rounds are completed, the target model +.>After training, each training round comprises a training process of steps S3-S5;
step S7, verifying the setInput to the trained object model +.>And outputting a result after the image instance to be segmented is segmented.
Preferably, the implementation process of obtaining the final auxiliary weight in step S2 is as follows:
step S2.1, randomly selecting a target image datasetImage sample->Based on image sample->Is->Obtaining an image sample->Predicted target bounding box setsPredicted target mask setAnd predicted target edge setsWherein->Representing image samples +.>First->Target bounding box predicted by the auxiliary model +.>Representing image samples +.>First->Target mask for individual auxiliary model prediction, +.>Representing image samples +.>First->Target edges predicted by the auxiliary models; calculate->、/>Andconfidence score of (2) and calculate +.>、/>And->An average confidence score of the confidence scores of +.>The method comprises the steps of carrying out a first treatment on the surface of the Then calculate the image sample +.>Average confidence scores obtained from other auxiliary models respectively, all of the average confidence scores forming a setThe method comprises the steps of carrying out a first treatment on the surface of the Aggregating average confidence scoresIn combination with the set of evaluation index scores as image sample +.>Is a difficulty score set of (1)The evaluation index score set is expressed asWhereinRepresenting image samples +.>Is>Score of each evaluation index->Representing image samples +.>Is>The individual difficulty scores:
；
；
；
wherein the method comprises the steps ofRepresenting a confidence score; />Representing image samples +.>Is set to be the target bounding box true value of (1),representing image samples +.>Target mask true value,/->Representing image samples +.>Target edge truth values of (2); />For measuring the image sample->First->Degree of overlap between target bounding box and target bounding box truth values predicted by the respective auxiliary model, ++>Is used for measuring the image sample +.>First->An average indicator of the degree of overlap between the target mask and the target mask truth values predicted by the respective auxiliary models,is used for measuring the image sample +.>First->Matching degree between the target edge predicted by the auxiliary model and the target edge true value;
step S2.2, based on the prediction result setAnd image sample->At the coordinates ofCorresponding real label at pixel->Calculate image sample +.>Is a set of tag quality scores for (a)Wherein->Represents the>The mass fraction of each label is as follows:
；
；
wherein the method comprises the steps ofRepresenting image samples +.>Width of->Representing image samples +.>Height of->、/>For parameters->For counting symbols +.>Expressed in coordinates +.>Corresponding real label at pixels of (2)>Confidence of category->For the set threshold value, calculate the image sample +.>True mark at other pixelsConfidence of sign category, statistics of +.>The individual auxiliary models are +.>Confidence in the prediction results at all pixels in (a) is greater than the set threshold +.>The number of pixels of (2) is recorded as +.>；
S2.3, obtaining an image sample based on the UNIQUE without reference quality evaluation indexIs>Wherein->Representing image samples +.>Is>The image quality scores are:
；
step S2.4, sample the imageThe difficulty score set, the label quality score set and the image quality score set are fused to obtain an image sample +.>Auxiliary weight set ∈ ->：；
Wherein the method comprises the steps of，/>，/>Is a superparameter for balancing the difficulty score set +.>Tag quality score set->Image quality score set +.>For->Contribution to->The auxiliary weights are averaged to obtain an image sample +.>Then calculates the final auxiliary weights of the target image dataset +.>Setting a threshold t for final auxiliary weights of other image samples, and if the final auxiliary weight is smaller than the threshold t, indicating that the quality of the image sample corresponding to the final auxiliary weight is low, and taking the image sample from the target image datasetDiscarding to obtain cleaned target image dataset +.>。
Preferably, the implementation process of step S3 is as follows:
step S3.1, calculating target weight: for the target image datasetIs a randomly selected image sample +.>Based on the object model->Predicted outcome of->Obtaining a target model->Confidence score of predicted target bounding box, confidence score of predicted target mask and confidence score of predicted target edge, and calculate average confidence score of predicted target bounding box, confidence score of predicted target mask and confidence score of predicted target edge, combine average confidence score with evaluation index score as image sample>Is then based on the objective model +.>Predicted outcome of->And image sample->Corresponding real label calculation image sample +.>Then obtaining an image sample ++based on the no-reference quality evaluation index UNIQUE>Is to sample the image +.>The difficulty score, the label quality score and the image quality score are fused to obtain an image sample +.>The operation of fusing is as follows: />；
Wherein the method comprises the steps ofFor image sample->Target weight of->For image sample->Difficulty score of->For image sample->Label mass fraction,/>For the image quality score +.>，/>，/>Is a super parameter for balancing the difficulty fraction +.>Tag quality score set->And image quality score->Weight of target->Contribution of (2);
step S3.2, calculating an image sampleSample loss of->：
；
；
；
；
Wherein,representing class loss of target frame, using class cross entropy loss to measure differences between predicted and real target frame classes，/>True value representing target frame class, ++>A probability value representing a predicted target class for the target frame; />Representing regression loss of target frame, using average absolute error loss, measuring difference between predicted target frame and real target frame, +.>True value representing the position coordinates of the target frame, +.>Predicted value representing target frame position coordinates, +.>Representing the number of target boxes +.>Is a parameter; />Representing the segmentation penalty of the target mask, using the cross entropy penalty at pixel level,/for the target mask>True value representing target mask, +_>A predicted value representing a target mask, wherein +.>Representing image samples +.>Width of->Representing image samples +.>Height of->，/>For parameters->The number of categories is indicated and,cis a parameter.
Preferably, the sample weight obtained in S4 is:
；
wherein,for sample weight, ++>For the target weight, ++>For final auxiliary weight, ++>For adjusting target weight->And final auxiliary weight +.>Is a parameter of (a).
Preferably, in the step S5, the target model is updatedIs a weighted sample loss->The method comprises the following steps:
。
the beneficial effects of the invention include the following aspects:
(1) According to the invention, the model is used for evaluating the data validity instead of manual work, and the process of evaluating the data validity is automated, so that the labor cost of data screening can be greatly reduced, and the efficiency is improved. Data in scenes such as crowdsourcing are cleaned, so that higher-quality data can be obtained while the data collection cost is reduced;
(2) The invention calculates the loss weight of each image sample by evaluating the quality of the label sample together with the auxiliary model and the target model, and under the weighted loss, the target model tends to learn the image sample with high quality, thereby improving the performance of the image data set containing a large number of low-quality image samples;
(3) The method is not only suitable for the field of image instance segmentation, but also hopefully achieves positive effects in the visual fields of image semantic segmentation, image classification, target detection and the like.
Drawings
FIG. 1 is a flow chart of a data weighted learning method based on model collaboration provided by the invention;
FIG. 2 is a simplified flow chart of the method for deriving a cleaned dataset based on auxiliary weights;
fig. 3 is a schematic diagram of a data weighting framework based on model collaboration provided by the invention.
Description of the embodiments
The invention will be further described with reference to the accompanying drawings. It will be apparent that the described embodiments are some, but not all, embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The invention provides a data weighted learning method based on model collaboration, the specific principle is shown in figure 1, and the method comprises the following steps:
step S1, giving a general image datasetTarget image dataset->Verification set->Giving an auxiliary model set consisting of a plurality of auxiliary models differing in both decision boundary and learning ability>Wherein->Indicate->Auxiliary model->For natural numbers greater than 0, give the target model +.>The method comprises the steps of carrying out a first treatment on the surface of the Wherein the set of auxiliary models->In general image dataset->Pre-training is carried out on the device;
step S2, randomly selecting the target image datasetImage sample->Auxiliary model set->Is->The individual auxiliary models are all for image samples +.>Generating a prediction result to obtain a prediction result setWherein->Indicate->The individual auxiliary model is +.>The prediction result generated is then passed through the image samples +.>The set of prediction results generated->And image sample->The corresponding real tag calculates the image sample +.>Sets of individual auxiliary weights->Wherein->Indicate->Auxiliary weights generated by the auxiliary model are then applied to this +.>The auxiliary weights are averaged to obtain an image sample +.>Final auxiliary weights of (2);
finally, the target image dataset is calculatedFinal auxiliary weight of other image samples in the target image dataset, and according to the size of the final auxiliary weight +.>Ordering all the image samples in the image, if the final auxiliary weight of a certain image sample is smaller than the set threshold +.>Then the image sample is +.>Discarding to obtain cleaned target image dataset +.>。
Specifically, referring to fig. 2, the method includes the following steps:
step S2.1, randomly selecting a target image datasetImage sample->Based on image sample->Is->Obtaining an image sample->Predicted target bounding box setsPredicted target mask setAnd predicted target edge setsWherein->Representing image samples +.>First->Target bounding box predicted by the auxiliary model +.>Representing image samples +.>First->Target mask for individual auxiliary model prediction, +.>Representing image samples +.>First->Target edges predicted by the auxiliary models; calculate->、/>Andand calculates the average confidence score +.>The method comprises the steps of carrying out a first treatment on the surface of the Then calculate the image sample +.>Average confidence scores obtained from other auxiliary models respectively, all the average confidence scores constituting the set +.>The method comprises the steps of carrying out a first treatment on the surface of the The average confidence score set +.>In combination with the set of evaluation index scores as image sample +.>Is ++A of difficulty score sets>The evaluation index score set is expressed as +.>Wherein->Representing image samples +.>Is>Score of each evaluation index->Representing image samples +.>Is the first of (2)The individual difficulty scores:
；
；
；
wherein the method comprises the steps ofRepresenting a confidence score; />Representing image samples +.>Is set to be the target bounding box true value of (1),representing image samples +.>Target mask true value,/->Representing image samples +.>Target edge truth values of (2); />For measuring the image sample->First->Target bounding box and target boundary for individual auxiliary model predictionDegree of overlap between frame truth values, +.>Is used for measuring the image sample +.>First->An average indicator of the degree of overlap between the target mask and the target mask truth values predicted by the respective auxiliary models,is used for measuring the image sample +.>First->Matching degree between the target edge predicted by the auxiliary model and the target edge true value;
step S2.2, based on the prediction result setAnd image sample->At the coordinates ofCorresponding real label at pixel->Calculate image sample +.>Is a set of tag quality scores for (a)Wherein->Represents the>The mass fraction of each label is as follows:
；
；
wherein the method comprises the steps ofRepresenting image samples +.>Width of->Representing image samples +.>Height of->、/>As a function of the parameters,for counting symbols +.>Expressed in coordinates +.>Corresponding real label at pixels of (2)>Confidence of category->For the set threshold value, calculate +.>Confidence of true tag class at other pixels, statistics of +.>Confidence in the prediction results of the auxiliary models for all pixels in the image sample is greater than a set threshold +.>The number of pixels of (2) is recorded as +.>；
S2.3, obtaining an image sample based on the UNIQUE without reference quality evaluation indexIs>Wherein->Representing image samples +.>Is>The image quality scores are:
；
step S2.4, sample the imageThe difficulty score set, the label quality score set and the image quality score set are fused to obtain an image sample +.>Auxiliary weight set ∈ ->：；
Wherein the method comprises the steps of，/>，/>Is a superparameter for balancing the difficulty score set +.>Tag quality score set->Image quality score set +.>For->Contribution to->The auxiliary weights are averaged to obtain an image sample +.>Then calculates the final auxiliary weights of the target image dataset +.>Final auxiliary weights of other image samples in the image data set, setting a threshold t, and adding the final auxiliary weights to the target image data set>Screening and discarding low-quality image samples, wherein the method comprises the following steps: if the final auxiliary weight is less than the thresholdA value t indicating that the quality of the image sample corresponding to the final auxiliary weight is low, and this image sample is taken from the target image dataset +.>Discarding to obtain cleaned target image dataset +.>。
Step S3, randomly selecting a target image datasetThe image sample is defined as +.>Through the object model->Generating an image sample->Predicted outcome of->Through image sample->Generated prediction result->Image sample->Corresponding real tag calculate image sample +.>Target weight and sample loss of (2), then using the target model +.>Calculating the target image dataset +.>Target weights and sample losses for all other image samples; specifically, please see fig. 3:
step S3.1, calculating target weight: for the target image datasetIs a randomly selected image sample +.>Based on the object model->Predicted outcome of->Obtaining a target model->Confidence scores of the predicted target bounding box, the predicted target mask and the predicted target edge, and calculating an average confidence score of the three confidence scores, combining the average confidence score with the evaluation index score as an image sample>Is then based on the objective model +.>Predicted outcome of->And image sample->Corresponding real label calculation image sample +.>Then obtaining an image sample ++based on the no-reference quality evaluation index UNIQUE>Is to sample the image +.>The difficulty score, the label quality score and the image quality score are fused to obtain an image sample +.>The operation of fusing is as follows: />；
Wherein the method comprises the steps ofFor image sample->Target weight of->For image sample->Difficulty score of->For image sample->Label mass fraction,/>For the image quality score +.>，/>，/>Is a super parameter for balancing the difficulty and the easinessCount->Tag quality score set->And image quality score->Weight of target->Contribution of (2);
step S3.2, calculating an image sampleSample loss of->：
；
；
；
；
Wherein,representing class loss of target frame, using class cross entropy loss to measure difference between class of predicted target frame and class of real target frame, +.>True value representing target frame class, ++>A probability value representing a predicted target class for the target frame; />Representing regression loss of target frame, using average absolute error loss, measuring difference between predicted target frame and real target frame, +.>True value representing the position coordinates of the target frame, +.>Predicted value representing target frame position coordinates, +.>Representing the number of target boxes +.>Is a parameter; />Representing the segmentation penalty of the target mask, using the cross entropy penalty at pixel level,/for the target mask>True value representing target mask, +_>A predicted value representing a target mask, wherein +.>Representing image samples +.>Width of->Representing image samples +.>Height of->，/>For parameters->The number of categories is indicated and,cis a parameter.
Step S4, collecting the target image data setThe final auxiliary weight and the target weight obtained by each image sample are fused to obtain sample weight:
；
wherein,for sample weight, ++>For the target weight, ++>For final auxiliary weight, ++>For adjusting target weights->And final auxiliary weight +.>Is a parameter of (a).
Step S5, object image data setRe-weighting the sample loss obtained by each image sample and the sample weight to obtain weighted sample loss, and finally utilizing gradientIs used for updating the target model>Is->Is to (1) training:
；
s6, when the training set training is completed and all training rounds are completed, the target modelAfter training, each training round comprises a training process of steps S2-S5;
step S7, verifying the setInput to the trained object model +.>And outputting a result after the picture instance to be segmented is segmented.
A data weighted learning method based on model collaboration is implemented in a specific example segmentation task. Two models, mask-RCNN, pointRend, were chosen as the auxiliary model, the yolact model as the target model, and experiments were performed on CoCo datasets, with the results shown in table 1.
Table 1: experimental results of training the yolact example segmentation model under the CoCo dataset:
in table 1, the first row shows the results obtained by training the yolact model directly on the CoCo dataset. The second line shows that re-weighting the final sample weights to the loss of the target model proceeds using only the difficulty score of the image sample as the sample score for the auxiliary model and the target modelAnd updating the row model to obtain a result. The third line shows the results obtained after model updating by re-weighting the final sample weights to the loss of the target model using only the label quality scores of the image samples as the sample scores for the auxiliary model and the target model. The last line shows the difficulty score for the simultaneous use of the image samplesTag mass fraction of image sample->Image quality fraction of image sample->And weighting the final sample weight to the loss of the target model according to a certain proportion as the sample fraction of the auxiliary model and the target model, and carrying out model updating to obtain a result.
By observing the table, the used sample fraction indexes can be found to improve the segmentation performance of the model to a certain extent, and the effectiveness of the method is further verified.
The foregoing is only a preferred embodiment of the invention, it being noted that: it will be apparent to those skilled in the art that various modifications and adaptations can be made without departing from the principles of the present invention, and such modifications and adaptations are intended to be comprehended within the scope of the invention.
Claims (3)
1. The data weighted learning method based on model collaboration is characterized by comprising the following steps of:
step S1, giving a general image datasetTarget image dataset->Verification set->Giving an auxiliary model set consisting of a plurality of auxiliary models differing in both decision boundary and learning ability>WhereinIndicate->Auxiliary model->For natural numbers greater than 0, give the target model +.>The method comprises the steps of carrying out a first treatment on the surface of the Wherein the set of auxiliary models->In general image dataset->Pre-training is carried out on the device;
step S2, randomly selecting the target image datasetImage sample->Auxiliary model set->In (a) and (b)The individual auxiliary models are all for image samples +.>Generating a prediction result to obtain a prediction result setWherein->Indicate->The individual auxiliary model is +.>The prediction result generated is then passed through the image samples +.>The set of prediction results generated->And image sample->Corresponding real tag calculate image sample +.>Is->Auxiliary weights are obtained to obtain auxiliary weight set +.>Wherein->Indicate->Auxiliary weights generated by the auxiliary model are then calculated +.>Averaging the auxiliary weights to obtain an image sampleFinal auxiliary weights of (2);
finally, the target image dataset is calculatedFinal auxiliary weight of other image samples in the target image dataset, and according to the size of the final auxiliary weight +.>Ordering all the image samples in the image, if the final auxiliary weight of a certain image sample is smaller than the set threshold +.>Then the image sample is +.>Discarding to obtain cleaned target image dataset +.>；
Step S3, randomly selecting a target image datasetThe image sample is defined as +.>Through the object model->Generating an image sample->Predicted outcome of->Through image sample->Generated prediction result->Image sample->Corresponding real tag calculate image sample +.>Target weight and sample loss of (2), then using the target model +.>Calculating the target image dataset +.>Target weights and sample losses for all other image samples;
step S4, collecting the target image data setThe final auxiliary weight and the target weight obtained by each image sample in the image are fused to obtain sample weight;
step S5, collecting the target image data setRe-weighting the sample loss obtained for each image sample in the model with the sample weight to obtain a weighted sample loss, and finally updating the target model by using gradient calculation>Is->Is used for training;
step S6, when the target image data setThe training is completed, and when all training rounds are completed, the target model +.>After training, each training round comprises a training process of steps S3-S5;
step S7, verifying the setInput to the trained object model +.>Outputting a result after the image instance to be segmented is segmented;
the implementation process of obtaining the final auxiliary weight in the step S2 is as follows:
step S2.1, randomly selecting a target image datasetImage sample->Based on image sample->Is->Obtaining an image sample->Predicted target bounding box setsPredicted target mask setAnd predicted target edge setsWherein->Representing image samples +.>First->Target bounding box predicted by the auxiliary model +.>Representing image samples +.>First->Target mask for individual auxiliary model prediction, +.>Representing image samples +.>First->Target edges predicted by the auxiliary models; calculate->、/>Andconfidence score of (2) and calculate +.>、/>And->An average confidence score of the confidence scores of +.>The method comprises the steps of carrying out a first treatment on the surface of the Then calculate the image sample +.>Average confidence scores obtained from other auxiliary models respectively, all the average confidence scores constituting the set +.>The method comprises the steps of carrying out a first treatment on the surface of the The average confidence score set +.>In combination with the set of evaluation index scores as image sample +.>Is ++A of difficulty score sets>The evaluation index score set is expressed asWherein->Representing image samples +.>Is>Score of each evaluation index->Representing image samples +.>Is>The individual difficulty scores:
；
；
；
wherein the method comprises the steps ofRepresenting a confidence score; />Representing image samples +.>Target bounding box true value,/->Representing image samples +.>Is true of the target mask of (2)Value of->Representing image samples +.>Target edge truth values of (2);for measuring the image sample->First->Degree of overlap between target bounding box and target bounding box truth values predicted by the respective auxiliary model, ++>Is used for measuring the image sample +.>First->An average indicator of the degree of overlap between the target mask and the target mask truth values predicted by the respective auxiliary models,is used for measuring the image sample +.>First->Matching degree between the target edge predicted by the auxiliary model and the target edge true value;
step S2.2, based on the prediction result setAnd image sample->At the coordinates ofCorresponding real label at pixel->Calculate image sample +.>Is a set of tag quality scores for (a)Wherein->Represents the>The mass fraction of each label is as follows:
；
；
wherein the method comprises the steps ofRepresenting image samples +.>Width of->Representing image samples +.>Height of->、/>For parameters->For counting symbols +.>Expressed in coordinates +.>Corresponding real label at pixels of (2)>Confidence of category->For the set threshold value, calculate the image sample +.>Confidence of true tag class at other pixels, statistics of +.>The individual auxiliary models are +.>Confidence in the prediction results at all pixels in (a) is greater than the set thresholdThe number of pixels of (2) is recorded as +.>；
S2.3, obtaining an image sample based on the UNIQUE without reference quality evaluation indexImage quality score set of (2)Wherein->Representing image samples +.>Is>The image quality scores are:
；
step S2.4, sample the imageThe difficulty score set, the label quality score set and the image quality score set are fused to obtain an image sample +.>Auxiliary weight set ∈ ->：；
Wherein the method comprises the steps of，/>，/>Is a superparameter for balancing the difficulty score set +.>Tag quality score set->Image quality score set +.>For a pair ofContribution to->The auxiliary weights are averaged to obtain an image sample +.>Then calculates the final auxiliary weights of the target image dataset +.>Setting a threshold t for final auxiliary weight of other image samples in the image data set, if the final auxiliary weight is smaller than the threshold t, indicating that the quality of the image sample corresponding to the final auxiliary weight is low, and taking the image sample from the target image data set->Discarding to obtain cleaned target image dataset +.>；
The implementation process of the step S3 is as follows:
step S3.1, calculating target weight: for the targetImage data setRandomly selected one of the image samplesBased on the object model->Predicted outcome of->Obtaining a target model->Confidence score of predicted target bounding box, confidence score of predicted target mask and confidence score of predicted target edge, and calculate average confidence score of predicted target bounding box, confidence score of predicted target mask and confidence score of predicted target edge, combine average confidence score with evaluation index score as image sample>Is then based on the objective model +.>Predicted outcome of->And image sample->Corresponding real label calculation image sample +.>Then obtaining an image sample ++based on the no-reference quality evaluation index UNIQUE>Is to sample the image +.>The difficulty score, the label quality score and the image quality score are fused to obtain an image sample +.>The operation of fusing is as follows: />；
Wherein the method comprises the steps ofFor image sample->Target weight of->For image sample->Difficulty score of->For image sample->Label mass fraction,/>For the image quality score +.>，/>，/>Is a superparameter for balancing difficulty scoresTag quality score set->And image quality score->Weight of target->Contribution of (2);
step S3.2, calculating an image sampleSample loss of->：
；
；
；
；
Wherein,representing class loss of target frame, using class cross entropy loss to measure difference between class of predicted target frame and class of real target frame, +.>True value representing target frame class, ++>A probability value representing a predicted target class for the target frame; />Representing regression loss of target frame, using average absolute error loss, measuring difference between predicted target frame and real target frame, +.>True value representing the position coordinates of the target frame, +.>Predicted value representing target frame position coordinates, +.>Representing the number of target boxes +.>Is a parameter; />Representing the segmentation penalty of the target mask, using the cross entropy penalty at pixel level,/for the target mask>True value representing target mask, +_>Representing predicted values of a target mask, whereRepresenting image samples +.>Width of->Representing image samples +.>Height of->，/>For parameters->The number of categories is indicated and,cis a parameter.
2. The data weighted learning method based on model collaboration according to claim 1, wherein the sample weights obtained in S4 are:
；
wherein,for sample weight, ++>For the target weight, ++>For final auxiliary weight, ++>For adjusting target weight->And final auxiliary weight +.>Is a parameter of (a).
3. The method for model-collaboration-based data weighted learning of claim 2, wherein the step S5 is performed to update the target modelIs a weighted sample loss->The method comprises the following steps:
。
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410004710.5A CN117496191B (en) | 2024-01-03 | 2024-01-03 | Data weighted learning method based on model collaboration |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410004710.5A CN117496191B (en) | 2024-01-03 | 2024-01-03 | Data weighted learning method based on model collaboration |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117496191A CN117496191A (en) | 2024-02-02 |
| CN117496191B true CN117496191B (en) | 2024-03-29 |
Family
ID=89678670
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410004710.5A Active CN117496191B (en) | 2024-01-03 | 2024-01-03 | Data weighted learning method based on model collaboration |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117496191B (en) |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3731144A1 (en) * | 2019-04-25 | 2020-10-28 | Koninklijke Philips N.V. | Deep adversarial artifact removal |
| CN112233095A (en) * | 2020-10-16 | 2021-01-15 | 哈尔滨市科佳通用机电股份有限公司 | Method for detecting multiple fault forms of railway wagon locking plate device |
| WO2022045877A1 (en) * | 2020-08-28 | 2022-03-03 | Mimos Berhad | A system and method for identifying occupancy of parking lots |
| CN114252423A (en) * | 2021-12-24 | 2022-03-29 | 汉姆德(宁波)智能医疗科技有限公司 | Method and device for generating fully sampled image of super-resolution microscope |
| CN115310130A (en) * | 2022-08-15 | 2022-11-08 | 南京航空航天大学 | Multi-site medical data analysis method and system based on federal learning |
| WO2023077821A1 (en) * | 2021-11-07 | 2023-05-11 | 西北工业大学 | Multi-resolution ensemble self-training-based target detection method for small-sample low-quality image |
| WO2023093346A1 (en) * | 2021-11-25 | 2023-06-01 | 支付宝(杭州)信息技术有限公司 | Exogenous feature-based model ownership verification method and apparatus |
| CN116824216A (en) * | 2023-05-22 | 2023-09-29 | 南京信息工程大学 | Passive unsupervised domain adaptive image classification method |
| WO2023185785A1 (en) * | 2022-03-28 | 2023-10-05 | 华为技术有限公司 | Image processing method, model training method, and related apparatuses |
| CN117036897A (en) * | 2023-05-29 | 2023-11-10 | 中北大学 | Method for detecting few sample targets based on Meta RCNN |
| CN117173497A (en) * | 2023-11-02 | 2023-12-05 | 腾讯科技（深圳）有限公司 | Image generation method and device, electronic equipment and storage medium |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20220114399A1 (en) * | 2020-10-08 | 2022-04-14 | Royal Bank Of Canada | System and method for machine learning fairness testing |
| US20230360246A1 (en) * | 2022-05-05 | 2023-11-09 | Elm Company | Method and System of Real-Timely Estimating Dimension of Signboards of Road-side Shops |
-
2024
- 2024-01-03 CN CN202410004710.5A patent/CN117496191B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3731144A1 (en) * | 2019-04-25 | 2020-10-28 | Koninklijke Philips N.V. | Deep adversarial artifact removal |
| WO2022045877A1 (en) * | 2020-08-28 | 2022-03-03 | Mimos Berhad | A system and method for identifying occupancy of parking lots |
| CN112233095A (en) * | 2020-10-16 | 2021-01-15 | 哈尔滨市科佳通用机电股份有限公司 | Method for detecting multiple fault forms of railway wagon locking plate device |
| WO2023077821A1 (en) * | 2021-11-07 | 2023-05-11 | 西北工业大学 | Multi-resolution ensemble self-training-based target detection method for small-sample low-quality image |
| WO2023093346A1 (en) * | 2021-11-25 | 2023-06-01 | 支付宝(杭州)信息技术有限公司 | Exogenous feature-based model ownership verification method and apparatus |
| CN114252423A (en) * | 2021-12-24 | 2022-03-29 | 汉姆德(宁波)智能医疗科技有限公司 | Method and device for generating fully sampled image of super-resolution microscope |
| WO2023185785A1 (en) * | 2022-03-28 | 2023-10-05 | 华为技术有限公司 | Image processing method, model training method, and related apparatuses |
| CN115310130A (en) * | 2022-08-15 | 2022-11-08 | 南京航空航天大学 | Multi-site medical data analysis method and system based on federal learning |
| CN116824216A (en) * | 2023-05-22 | 2023-09-29 | 南京信息工程大学 | Passive unsupervised domain adaptive image classification method |
| CN117036897A (en) * | 2023-05-29 | 2023-11-10 | 中北大学 | Method for detecting few sample targets based on Meta RCNN |
| CN117173497A (en) * | 2023-11-02 | 2023-12-05 | 腾讯科技（深圳）有限公司 | Image generation method and device, electronic equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 一种基于掩膜组合的多类弹载图像目标分割算法;袁汉钦;陈栋;杨传栋;王昱翔;刘桢;;舰船电子工程;20200620(第06期);112-117 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117496191A (en) | 2024-02-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107610087B (en) | Tongue coating automatic segmentation method based on deep learning | |
| CN111462175A (en) | Space-time convolution twin matching network target tracking method, device, medium and equipment | |
| CN110490239B (en) | Training method, quality classification method, device and equipment of image quality control network | |
| Li et al. | Weather GAN: Multi-domain weather translation using generative adversarial networks | |
| CN114529819B (en) | Household garbage image recognition method based on knowledge distillation learning | |
| CN112949408B (en) | Real-time identification method and system for target fish passing through fish channel | |
| CN114998603B (en) | Underwater target detection method based on depth multi-scale feature factor fusion | |
| CN114663426B (en) | Bone age assessment method based on key bone region positioning | |
| CN109543693A (en) | Weak labeling data noise reduction method based on regularization label propagation | |
| CN118379288B (en) | Embryo prokaryotic target counting method based on fuzzy rejection and multi-focus image fusion | |
| CN112883931A (en) | Real-time true and false motion judgment method based on long and short term memory network | |
| CN111126155B (en) | Pedestrian re-identification method for generating countermeasure network based on semantic constraint | |
| CN115019039A (en) | Example segmentation method and system combining self-supervision and global information enhancement | |
| CN114723994A (en) | Hyperspectral image classification method based on dual-classifier confrontation enhancement network | |
| CN117152503A (en) | Remote sensing image cross-domain small sample classification method based on false tag uncertainty perception | |
| CN112991280A (en) | Visual detection method and system and electronic equipment | |
| CN117853486B (en) | Automatic evaluation method for rock mass quality of tunnel working face under condition of data loss | |
| CN116758539A (en) | Embryo image blastomere identification method based on data enhancement | |
| CN116977844A (en) | Lightweight underwater target real-time detection method | |
| CN114494941A (en) | Comparison learning-based weak supervision time sequence action positioning method | |
| CN117496191B (en) | Data weighted learning method based on model collaboration | |
| CN112991281A (en) | Visual detection method, system, electronic device and medium | |
| CN116994320A (en) | Train driver in-transit fatigue driving detection method, system and equipment | |
| CN113792574B (en) | Cross-dataset expression recognition method based on metric learning and teacher student model | |
| Pang et al. | Target tracking based on siamese convolution neural networks |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |