CN116028878A - Diversified query active learning method and device for image classification - Google Patents

Abstract

The invention provides an image classification-oriented diversified query active learning method and device, wherein the method comprises the following steps: generating a pseudo tag with high confidence by using the auxiliary classification network as a main classification network, and removing redundant samples which are similar to the tag data from the non-tag data; in each active learning cycle, the main classification network and the auxiliary classification network exchange respective high-confidence pseudo-mark samples in a cooperative training mode, meanwhile, unlabeled samples predicted by the main classification network and the auxiliary classification network as different labels are added into a candidate pool for active learning, the auxiliary learning and multi-level diversity screening strategy is utilized for carrying out artificial expert marking on more diversified samples, and a small number of labeled samples are utilized for carrying out active learning cycle training on the classification model, so that the training effect close to using a large number of labeled samples can be achieved.

Description

Diversified query active learning method and device for image classification

Technical Field

The invention relates to the technical field of image processing, in particular to an image classification-oriented diversified query active learning method and device.

Background

The depth model learns the characteristics with strong robustness and expressive property from a large number of marked image samples, so that higher classification precision can be obtained on an image classification task. However, the labor and time costs required to annotate a large number of images are very high. In many practical applications, the number of available tagged images is very limited. Therefore, how to improve classification performance in the case of limited labeling data has become a challenge in the field of machine learning. Active learning is a very popular technique at present, which can train a depth model with high generalization capability by iteratively screening sample images and delivering limited annotation data obtained by expert annotation. Sample query strategies are the core of active learning methods. If the query function can select the sample with the largest information amount in each active learning cycle, the generalization capability of the classification model can be obviously improved through a small number of marked samples, so that the manual labeling cost is reduced. The query strategy in active learning is designed based mainly on two criteria, uncertainty and representativeness. Uncertainty criteria mainly indicate whether the predictions of the samples have a high confidence, and many active learning methods are designed based on this criteria, which select the samples with the largest information content and the lowest classification confidence, such as marginal sampling and entropy sampling, in each cycle. While the uncertainty criteria are very popular for use in active learning, it does not effectively improve the performance of classification. For example, if a batch of samples is selected in each cycle, there may be redundancy between samples with similar uncertainties; however, if only the least deterministic sample is selected in each cycle, the number of active learning cycles will increase significantly, resulting in an unacceptable time cost. To more fully explore the structural information in the unlabeled data, a representative method of selecting independent co-distributed samples in an exponential search space is proposed. But the computational cost of selecting a batch of independent co-distributed samples increases significantly over a large data set. Therefore, how to design an active learning strategy which is efficient and can improve the generalization capability of the depth model and reduce the cost of manual labeling is a problem with important practical significance.

Disclosure of Invention

In order to solve at least one technical problem in the prior art, the invention provides a diversified query active learning method and device for image classification.

In a first aspect, the present invention provides an image-classification-oriented diversified query active learning method, including:

step 1, acquiring an unlabeled image set U consisting of a first number of unlabeled images and a labeled image set L consisting of a second number of labeled images, wherein the first number is greater than the second number;

step 2, using the labeled image set L as a training set to respectively classify the main classification model f ₁ And an auxiliary classification model f ₂ Training to obtain f ₁' and f₂ ' and utilize f ₁' and f₂ ' extracting and predicting the characteristics of all the unlabeled images in the unlabeled image set U;

step 3, from f ₁' and f₂ ' predicting all the unlabeled images in the unlabeled image set to obtain the target image of the target image ₁' and f₂ ' image set S consisting of unlabeled images predicted to be different labels and composed of a quilt f ₁' and f₂ ' image set H of unlabeled images predicted to be the same label;

step 4, clustering, sorting and screening the unlabeled images in the image set S by using a multi-level diversity criterion to form an unlabeled sample candidate set to be labeled;

Step 5, labeling the label-free sample candidate set based on labeling operation, and removing the labeled sample from the label-free image set U and adding the labeled sample into the label-free image set L;

step 6, the images in the image set H are respectively processed according to f ₁' and f₂ Sequencing the prediction confidence degrees, and respectively screening out a sample set H with highest confidence degree ₁ and H₂ And respectively giving the samples of the sample with the values of f ₁' and f₂ 'pseudo tag output' to obtain sample set H ₁' and H₂ '；

Step 7Using sample sets H ₂ ' sum tagged image set L vs f ₁ ' training to obtain f ₁ ", use sample set H ₁ ' sum tagged image set L vs f ₂ ' training to obtain f ₂ ”；

Step 8, f ₁” and f₂ "as the main classification model f ₁ And an auxiliary classification model f ₂ Returning to the step 2 until the main classification model f ₁ Stopping iteration when the expected classification performance or the marked sample exceeds the set upper limit, and entering a step 9;

step 9, utilizing the main classification model f ₁ And performing image classification prediction tasks.

In a second aspect, the present invention further provides an image-classification-oriented diversified query active learning apparatus, where the image-classification-oriented diversified query active learning apparatus includes:

the acquisition module is used for acquiring an unlabeled image set U consisting of a first number of unlabeled images and a labeled image set L consisting of a second number of labeled images, wherein the first number is larger than the second number;

A first training module for using the labeled image set L as training set to respectively classify the main classification model f ₁ And an auxiliary classification model f ₂ Training to obtain f ₁' and f₂ ' and utilize f ₁′ and f₂ ' extracting and predicting the characteristics of all the unlabeled images in the unlabeled image set U;

a first building block for building up the first building block from f ₁' and f₂ ' predicting all the unlabeled images in the unlabeled image set to obtain the target image of the target image ₁′ and f₂ ' image set S consisting of unlabeled images predicted to be different labels and composed of a quilt f ₁' and f₂ ' image set H of unlabeled images predicted to be the same label;

the second construction module is used for clustering, sorting and screening the unlabeled images in the image set S by using a multi-level diversity criterion to form an unlabeled sample candidate set to be labeled;

the labeling module is used for labeling the label-free sample candidate set based on labeling operation, removing the labeled sample from the label-free image set U and adding the labeled sample into the label-free image set L;

a third construction module for respectively setting the images in the image set H according to f ₁' and f₂ Sequencing the prediction confidence degrees, and respectively screening out a sample set H with highest confidence degree ₁ and H₂ And respectively giving the samples of the sample with the values of f ₁' and f₂ 'pseudo tag output' to obtain sample set H ₁' and H₂ ′；

A second training module for utilizing the sample set H ₂ ' sum tagged image set L vs f ₁ ' training to obtain f ₁ ", use sample set H ₁ ' sum tagged image set L vs f ₂ ' training to obtain f ₂ ”；

A circulation module for at f ₁” and f₂ "as the main classification model f ₁ And an auxiliary classification model f ₂ Returning to the step executed by the first training module until the main classification model f ₁ Stopping iteration when the expected classification performance or the labeling sample exceeds a set upper limit;

a prediction module for utilizing the main classification model f ₁ And performing image classification prediction tasks.

In the present invention, the following steps are performed:

step 1, acquiring an unlabeled image set U consisting of a first number of unlabeled images and a labeled image set L consisting of a second number of labeled images, wherein the first number is greater than the second number;

step 2, using the labeled image set L as a training set to respectively classify the main classification model f ₁ And an auxiliary classification model f ₂ Training to obtain f ₁' and f₂ ' and utilize f ₁' and f₂ ' extracting and predicting the characteristics of all the unlabeled images in the unlabeled image set U;

step 3, from f ₁' and f₂ ' predicting all the unlabeled images in the unlabeled image set to obtain the target image of the target image ₁' and f₂ ' image set S consisting of unlabeled images predicted to be different labels and composed of a quilt f ₁′ and f₂ ' image set H of unlabeled images predicted to be the same label;

step 4, clustering, sorting and screening the unlabeled images in the image set S by using a multi-level diversity criterion to form an unlabeled sample candidate set to be labeled;

step 5, labeling the label-free sample candidate set based on labeling operation, and removing the labeled sample from the label-free image set U and adding the labeled sample into the label-free image set L;

step 6, the images in the image set H are respectively processed according to f ₁' and f₂ Sequencing the prediction confidence degrees, and respectively screening out a sample set H with highest confidence degree ₁ and H₂ And respectively giving the samples of the sample with the values of f ₁′ and f₂ 'pseudo tag output' to obtain sample set H ₁′ and H₂ ′；

Step 7, utilizing sample set H ₂ ' sum tagged image set L vs f ₁ ' training to obtain f ₁ ", use sample set H ₁ ' sum tagged image set L vs f ₂ ' training to obtain f ₂ ”；

Step 8, f ₁” and f₂ "as the main classification model f ₁ And an auxiliary classification model f ₂ Returning to the step 2 until the main classification model f ₁ Stopping iteration when the expected classification performance or the marked sample exceeds the set upper limit, and entering a step 9;

Step 9, utilizing the main classification model f ₁ And performing image classification prediction tasks.

According to the invention, a large amount of manual labeling cost is reduced, and the training effect close to that of using a large amount of labeled samples can be achieved by using a small amount of labeled samples to perform active learning and circulating training on the classification model.

Drawings

FIG. 1 is a flow chart of an embodiment of an image classification oriented diversified query active learning method of the present invention;

fig. 2 is a schematic diagram of functional modules of an embodiment of the image classification-oriented diversified query active learning device according to the present invention.

The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.

Detailed Description

It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.

In a first aspect, an embodiment of the present invention provides an image classification-oriented active learning method for diversified queries.

In an embodiment, referring to fig. 1, fig. 1 is a flowchart illustrating an embodiment of a method for learning a diversified query for image classification according to the present invention. As shown in fig. 1, the method for actively learning the diversified queries for image classification includes:

Step 1, acquiring an unlabeled image set U consisting of a first number of unlabeled images and a labeled image set L consisting of a second number of labeled images, wherein the first number is greater than the second number;

in this embodiment, the first number is much larger than the second number, i.e. the set of unlabeled images U comprises a large number of unlabeled images and the set of labeled images L comprises a small number of labeled images.

Step 2, using the labeled image set L as a training set to respectively classify the main classification model f ₁ And an auxiliary classification model f ₂ Training to obtain f ₁′ and f₂ ' and utilize f ₁′ and f₂ ' extracting and predicting the characteristics of all the unlabeled images in the unlabeled image set U;

in this embodiment, the labeled image set L is used as a training set, and the main classification model f is respectively identified ₁ And an auxiliary classification model f ₂ Training to obtain f ₁′ and f₂ ′。

Further, in an embodiment, the labeled image set L is used as a training set, and the main classification model f is respectively calculated ₁ And an auxiliary classification model f ₂ Proceeding withTraining to obtain f ₁′ and f₂ The' step comprises:

for the ith labeled image x in the labeled image set L _i Obtaining x _i Are respectively f ₁ and f₂ Classification probability for class j:

wherein ,W₁ and W₂ Respectively f ₁ and f₂ Parameters of (2);

is based on f ₁ The x obtained _i Classification probability belonging to class j +.>

Is based on f ₂ The x obtained _i Classification probability belonging to class j;

by minimizing the primary classification model f ₁ And an auxiliary classification model f ₂ Cross entropy loss on labeled image set L

and />

For the main classification model f ₁ And an auxiliary classification model f ₂ Training to obtain f ₁′ and f₂ ' wherein for tagged images in the tagged image set L, the model f is classified in the main classification ₁ And an auxiliary classification model f ₂ The cross entropy losses above are respectively:

wherein ,

and />

Respectively, training a main classification model f ₁ And an auxiliary classification model f ₂ Is a target function of (2); c is the total number of categories contained in the tagged image set L; l is the number of tagged images contained by the tagged image set L; y is _i Is the ith labeled image x _i Is a real tag of (1); j is x _i Is a predictive tag of (1); />

Is based on the x obtained by the main classification model _i The prediction probabilities belonging to class j,

is obtained based on auxiliary classification model _i Prediction probability belonging to class j; i { y } _i =j } is an indication function if y _i Equal to j, 1, otherwise 0.

After obtaining f ₁′ and f₂ ' after using f ₁′ and f₂ Feature extraction and prediction are performed on all the unlabeled images in the unlabeled image set U. I.e. all the unlabeled images in the unlabeled image set U are sequentially input into f ₁′ and f₂ ' obtain f ₁′ and f₂ ' prediction result for each unlabeled image, and f ₁ ' feature extraction results for each unlabeled image.

Further, in one embodiment, f is utilized ₁′ and f₂ The step of' feature extraction and prediction of all the unlabeled images in the unlabeled image set U comprises:

for the ith unlabeled image z in the unlabeled image set U _i Obtaining z _i Are respectively f ₁′ and f₂ The classification probability of' class j:

wherein ,W₁′ and W₂ ' f is respectively ₁′ and f₂ ' parameters;

is based on f ₁ ' z obtained _i Classification probability belonging to class j +.>

Is based on f ₂ ' z obtained _i Classification probability belonging to class j;

the method comprises the following steps:

obtaining f ₁′ and f₂ ' respectively to z _i Is a predictive pseudo tag of (1)

and />

According to f ₁ The output of the last pooling layer yields the pair z _i Extracted feature orientationAmount of the components.

Step 3, from f ₁′ and f₂ ' predicting all the unlabeled images in the unlabeled image set to obtain the target image of the target image ₁′ and f₂ ' image set S consisting of unlabeled images predicted to be different labels and composed of a quilt f ₁′ and f₂ ' image set H of unlabeled images predicted to be the same label;

step 4, clustering, sorting and screening the unlabeled images in the image set S by using a multi-level diversity criterion to form an unlabeled sample candidate set to be labeled;

In this embodiment, according to the feature vector corresponding to each unlabeled image in the image set S, the image set S is divided into C clusters by using a k-means clustering algorithm, which is expressed as u=u ₁ ∪U ₂ ∪...∪U _C For the above C clusters, according to the sum of f in each cluster ₁ The predicted category label number is further divided into a plurality of sub-clusters, the iterative clustering operation is carried out until only one category label is in each minimum sub-cluster, the clustering operation is stopped, m minimum sub-clusters are obtained and expressed as

And selecting one with highest uncertainty from each minimum sub-cluster according to a BvSB criterion to obtain a query data set F with m samples, wherein the BvSB criterion is expressed as:

wherein, for each smallest sub-cluster,

refers to samples within a cluster being f ₁ ' classification probability of class j; r is (r) _max1 Refer to f ₁ ' class subscript with maximum probability of classifying samples, r _max2 Refer to f ₁ ' category subscript with second greatest probability of classifying samples->

Representing the maximum predictive probability of a sample +.>

And the second highest prediction probability->

The difference between them, p is chosen for each smallest sub-cluster ^diff The smallest sample forms the query dataset F, p ^diff The smallest sample is the sample with the highest uncertainty;

If t samples have the same pseudo tag k in F, then the threshold for selecting different samples with the same pseudo tag k is expressed as:

where Var (·) is the variance operator, β is the hyper-parameter used to adjust the threshold,

represents f ₁ ' Classification probability for sample outputs in F with the same pseudo tag k, +.>

Represents f ₁ ' Classification probability for sample outputs in F with the same pseudo tag k, +.>

Ordering F by uncertainty of the samples and defining as

Wherein the uncertainty of the sample is from->

Decrementing to +.>

From F _u Sequentially selecting samples and adding the samples into a label-free sample candidate set Q, wherein +_>

Is F _u The j sample in (1) is predicted as the k-th class, if t with a pseudo tag k exists in Q _k Sample number, calculate->

And t _k The similarity of the individual samples is:

if t _k 0, then

Added to Q; otherwise, u _k When greater than v _k When in use, will->

Added to Q; repeating the screening process until h samples exist in the Q, wherein h is a hyper-parameter controlling the size of the active learning screening batch, and +.>

Represents f ₁ ' classification probability for sample output with false tag k in Q.

Step 5, labeling the label-free sample candidate set based on labeling operation, and removing the labeled sample from the label-free image set U and adding the labeled sample into the label-free image set L;

In this embodiment, a manual expert specifically marks samples in the unlabeled sample candidate set, and a label can be added to each sample in the unlabeled sample candidate set based on the marking operation of the manual expert. Then, the marked sample is removed from the unlabeled image set U and added to the labeled image set L.

Step 6, the images in the image set H are respectively processed according to f ₁′ and f₂ Sequencing the prediction confidence degrees, and respectively screening out a sample set H with highest confidence degree ₁ and H₂ And respectively giving the samples of the sample with the values of f ₁′ and f₂ 'pseudo tag output' to obtain sample set H ₁′ and H₂ ′；

In this embodiment, the images in the image set H are respectively expressed as f ₁′ and f₂ Sequencing the 'prediction confidence coefficient' to obtain two sequences, and screening a plurality of samples with highest confidence coefficient from the sequence 1 to form a sample set H ₁ The method comprises the steps of carrying out a first treatment on the surface of the Likewise, a plurality of samples with highest confidence are screened from the sequence 2 to form a sample set H ₂ 。

For H ₁ Each sample in (2) is assigned the value of f ₁ 'pseudo tag output' to obtain sample set H ₁ 'A'; similarly, for H ₂ Each sample in (2) is assigned the value of f ₂ 'pseudo tag output' to obtain sample set H ₂ ′。

Further, in an embodiment, the images in the image set H are respectively according to f ₁′ and f₂ Sequencing the prediction confidence degrees, and respectively screening out a sample set H with highest confidence degree ₁ and H₂ The method comprises the following steps:

the images in the image set H are processed according to f ₁ Sequencing the 'prediction confidence degrees', screening out the top l multiplied by alpha images with the highest confidence degrees, and forming a sample set H ₁ ；

The images in the image set H are processed according to f ₂ The 'confidence in prediction' is ranked,screening out the first l multiplied by alpha images with highest confidence coefficient to form a sample set H ₂ ；

wherein ,f₁ The' confidence in prediction for the ith image in image set H is:

f ₂ the' confidence in prediction for the ith image in image set H is:

is f for the ith image in the image set H ₁ ' Classification probability of class j +.>

Is f for the ith image in the image set H ₂ The' classification probability of j classes, i is the number of tagged images contained in the tagged image set L, and alpha is the super parameter.

Step 7, utilizing sample set H ₂ ' sum tagged image set L vs f ₁ ' training to obtain f ₁ "utilize sample set H ₁ ' sum tagged image set L vs f ₂ ' training to obtain f ₂ ″；

In this embodiment, the training process refers to the training process in step 2, and will not be described herein.

Step 8, f ₁” and f₂ "as the main classification model f ₁ And an auxiliary classification model f ₂ Returning to the step 2 until the main classification model f ₁ Stopping iteration when the expected classification performance or the marked sample exceeds the set upper limit, and entering a step 9;

in the present embodiment, f is ₁” and f₂ "as the main classification model f ₁ And an auxiliary classification model f ₂ Returning to the step 2 until the stopping condition is met, stopping iteration, and obtaining f finally ₁ As an image classification model, for performing image classification prediction tasks.

Step 9, utilizing the main classification model f ₁ And performing image classification prediction tasks.

In this example, the final f is used ₁ And performing image classification prediction tasks.

In this embodiment, the artificial expert labeling is performed on the more diversified samples by using the auxiliary learning and multi-level diversity screening strategy, so as to reduce the cost of artificial labeling. Specifically, in the auxiliary learning, the embodiment uses the auxiliary classification network as the main classification network to generate the pseudo tag with high confidence, and removes the redundant sample with similar representation with the tag data from the non-tag data; in each active learning cycle, the primary and secondary classification networks exchange respective highly-trusted pseudo-labeled samples in a co-training manner, while unlabeled samples predicted to be different labels by the primary and secondary classification networks are added to the candidate pool for active learning. Based on the predictions of the primary classification network, the present embodiment employs uncertainty criteria to select the most uncertain samples among them, followed by a multi-level diversity screening strategy to remove redundant samples from the uncertain samples. The multi-level diversity criterion comprises feature level diversity and class level diversity, wherein the feature level diversity can effectively mine the distribution structure of unlabeled data, unlabeled samples are clustered according to pseudo labels of a main classification model until the pseudo labels in each sub-cluster are the same, and the most uncertain sample is selected from each sub-cluster, so that samples with uncertainty and representativeness can be screened out; based on feature level diversity, the embodiment designs class level diversity criteria by considering redundancy inside query samples with the same pseudo tags, and effectively removes redundancy in uncertain samples with the same prediction tags, thereby ensuring that each query sample is beneficial to improving generalization capability of a classification network. Therefore, the multi-level diversity screening criteria adopted in the embodiment can ensure the representativeness and diversity of the selected samples, thereby reducing a large amount of manual labeling cost, and the training effect of using a large amount of labeled samples can be achieved by using a small amount of labeled samples to perform active learning and circulating training on the classification model.

In a second aspect, the embodiment of the invention further provides a diversified query active learning device oriented to image classification.

In an embodiment, referring to fig. 2, fig. 2 is a schematic functional block diagram of an embodiment of the image classification-oriented diversified query active learning device according to the present invention. As shown in fig. 2, the image classification-oriented diversified query active learning apparatus includes:

an acquisition module 10, configured to acquire an unlabeled image set U composed of a first number of unlabeled images and a labeled image set L composed of a second number of labeled images, where the first number is greater than the second number;

a first training module 20 for using the labeled image set L as training set for the main classification model f ₁ And an auxiliary classification model f ₂ Training to obtain f ₁′ and f₂ ' and utilize f ₁′ and f₂ ' extracting and predicting the characteristics of all the unlabeled images in the unlabeled image set U;

a first building block 30 for the first building block defined by f ₁′ and f₂ ' predicting all the unlabeled images in the unlabeled image set to obtain the target image of the target image ₁′ and f₂ ' image set S consisting of unlabeled images predicted to be different labels and composed of a quilt f ₁′ and f₂ ' image set H of unlabeled images predicted to be the same label;

The second construction module 40 is configured to cluster and sort the unlabeled images in the image set S by using a multi-level diversity criterion, so as to form an unlabeled sample candidate set to be labeled;

the labeling module 50 is configured to label the label-free sample candidate set based on a labeling operation, and remove the labeled sample from the label-free image set U and add the labeled sample to the labeled image set L;

third constructionA module 60 for respectively converting the images in the image set H according to f ₁′ and f₂ Sequencing the prediction confidence degrees, and respectively screening out a sample set H with highest confidence degree ₁ and H₂ And respectively giving the samples of the sample with the values of f ₁′ and f₂ 'pseudo tag output' to obtain sample set H ₁′ and H₂ ′；

A second training module 70 for utilizing the sample set H ₂ ' sum tagged image set L vs f ₁ ' training to obtain f ₁ ", use sample set H ₁ ' sum tagged image set L vs f ₂ ' training to obtain f ₂ ”；

A circulation module 80 for at f ₁” and f₂ "as the main classification model f ₁ And an auxiliary classification model f ₂ Returning to the step executed by the first training module until the main classification model f ₁ Stopping iteration when the expected classification performance or the labeling sample exceeds a set upper limit;

A prediction module 90 for utilizing a primary classification model f ₁ And performing image classification prediction tasks.

Further, in an embodiment, the first training module 20 is configured to:

for the ith labeled image x in the labeled image set L _i Obtaining x _i Are respectively f ₁ and f₂ Classification probability for class j:

wherein ,W₁ and W₂ Respectively f ₁ and f₂ Parameters of (2);

is based on f ₁ The x obtained _i Classification probability belonging to class j +.>

Is based on f ₂ The x obtained _i Classification probability belonging to class j;

by minimizing the primary classification model f ₁ And an auxiliary classification model f ₂ Cross entropy loss on labeled image set L

and />

For the main classification model f1 and the auxiliary classification model f ₂ Training to obtain f ₁′ and f₂ ' wherein for tagged images in the tagged image set L, the model f is classified in the main classification ₁ And an auxiliary classification model f ₂ The cross entropy losses above are respectively:

wherein ,

and />

Respectively, training a main classification model f ₁ And an auxiliary classification model f ₂ Is a target function of (2); c is the total number of categories contained in the tagged image set L; l is the number of tagged images contained by the tagged image set L; y is _i Is the ith labeled image x _i Is a real tag of (1); j is x _i Is a predictive tag of (1); />

Is based on the x obtained by the main classification model _i The prediction probabilities belonging to class j,

is obtained based on auxiliary classification model _i Prediction probability belonging to class j; i { y } _i =j } is an indication function if y _i Equal to j, 1, otherwise 0.

Further, in an embodiment, the first training module 20 is configured to:

for the ith unlabeled image z in the unlabeled image set U _i Obtaining z _i Are respectively f ₁′ and f₂ The classification probability of' class j:

wherein ,W₁′ and W₂ ' f is respectively ₁′ and f₂ ' parameters;

is based on f ₁ ' z obtained _i Classification probability belonging to class j +.>

Is based on f ₂ ' z obtained _i Classification probability belonging to class j;

the method comprises the following steps:

obtaining f ₁′ and f₂ ' respectively to z _i Is a predictive pseudo tag of (1)

and />

According to f ₁ The output of the last pooling layer yields the pair z _i The extracted feature vector.

Further, in an embodiment, the second construction module 40 is configured to:

according to the feature vector corresponding to each unlabeled image in the image set S, the image set S is divided into C clusters by using a k-means clustering algorithm, and the C clusters are expressed as U=U ₁ ∪U ₂ ∪...∪U _C For the above C clusters, according to the sum of f in each cluster ₁ The predicted category label number is further divided into a plurality of sub-clusters, the iterative clustering operation is carried out until only one category label is in each minimum sub-cluster, the clustering operation is stopped, m minimum sub-clusters are obtained and expressed as

And selecting one with highest uncertainty from each minimum sub-cluster according to a BvSB criterion to obtain a query data set F with m samples, wherein the BvSB criterion is expressed as:

wherein, for each smallest sub-cluster,

refers to samples within a cluster being f ₁ ' classification probability of class j; r is (r) _max1 Refer to f ₁ ' class subscript with maximum probability of classifying samples, r _max2 Refer to f ₁ ' category subscript with second greatest probability of classifying samples->

Representing the maximum predictive probability of a sample +.>

And the second highest prediction probability->

The difference between them, p is chosen for each smallest sub-cluster ^diff The smallest sample forms the query dataset F, p ^diff The smallest sample is the sample with the highest uncertainty;

if t samples have the same pseudo tag k in F, then the threshold for selecting different samples with the same pseudo tag k is expressed as:

where Var (·) is the variance operator, β is the hyper-parameter used to adjust the threshold,

represents f ₁ ' Classification probability for sample outputs in F with the same pseudo tag k, +.>

Represents f ₁ ' Classification probability for sample outputs in F with the same pseudo tag k, +.>

Ordering F by uncertainty of the samples and defining as

Wherein the uncertainty of the sample is from- >

Decrementing to +.>

From F _u Sequentially selecting samples and adding the samples into a label-free sample candidate set Q, wherein +_>

Is F _u The j sample in (1) is predicted as the k-th class, if t with a pseudo tag k exists in Q _k Sample number, calculate->

And t _k The similarity of the individual samples is:

if t _k 0, then

Added to Q; otherwise, u _k When greater than v _k When in use, will->

Added to Q; repeating the screening process until h samples exist in the Q, wherein h is a hyper-parameter controlling the size of the active learning screening batch, and +.>

Represents f ₁ ' classification probability for sample output with false tag k in Q.

Further, in an embodiment, the third construction module 60 is configured to:

the images in the image set H are processed according to f ₁ Sequencing the 'prediction confidence degrees', screening out the top l multiplied by alpha images with the highest confidence degrees, and forming a sample set H ₁ ；

The images in the image set H are processed according to f ₂ Sequencing the 'prediction confidence degrees', screening out the top l multiplied by alpha images with the highest confidence degrees, and forming a sample set H ₂ ；

wherein ,f₁ The' confidence in prediction for the ith image in image set H is:

f ₂ the' confidence in prediction for the ith image in image set H is:

is f for the ith image in the image set H ₁ ' Classification probability of class j +. >

Is f for the ith image in the image set H ₂ The' classification probability of j classes, i is the number of tagged images contained in the tagged image set L, and alpha is the super parameter.

The function implementation of each module in the image-classification-oriented diversified query active learning device corresponds to each step in the embodiment of the image-classification-oriented diversified query active learning method, and the function and implementation process of the method are not described in detail herein.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.

The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.

From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) as described above, comprising several instructions for causing a terminal device to perform the method according to the embodiments of the present invention.

The foregoing description is only of the preferred embodiments of the present invention, and is not intended to limit the scope of the invention, but rather is intended to cover any equivalents of the structures or equivalent processes disclosed herein or in the alternative, which may be employed directly or indirectly in other related arts.

Claims (10)

1. The image classification-oriented diversified query active learning method is characterized by comprising the following steps of:

Step 1, acquiring an unlabeled image set U consisting of a first number of unlabeled images and a labeled image set L consisting of a second number of labeled images, wherein the first number is greater than the second number;

step 2, using the labeled image set L as a training set to respectively classify the main classification model f ₁ And an auxiliary classification model f ₂ Training to obtain f ₁′ and f₂ ' and utilize f ₁′ and f₂ ' extracting and predicting the characteristics of all the unlabeled images in the unlabeled image set U;

step 3, from f ₁′ and f₂ ' predicting all the unlabeled images in the unlabeled image set to obtain the target image of the target image ₁′ and f₂ ' image set S consisting of unlabeled images predicted to be different labels and composed of a quilt f ₁′ and f₂ ' image set H of unlabeled images predicted to be the same label;

step 4, clustering, sorting and screening the unlabeled images in the image set S by using a multi-level diversity criterion to form an unlabeled sample candidate set to be labeled;

step 5, labeling the label-free sample candidate set based on labeling operation, and removing the labeled sample from the label-free image set U and adding the labeled sample into the label-free image set L;

step 6, the images in the image set H are respectively processed according to f ₁′ and f₂ Sequencing the prediction confidence degrees, and respectively screening out a sample set H with highest confidence degree ₁ and H₂ And respectively giving the samples of the sample with the values of f ₁′ and f₂ 'pseudo tag output' to obtain sample set H ₁′ and H₂ ′；

Step 7, utilizing sample set H ₂ ' sum tagged image set L vs f ₁ ' training to obtain f ₁ ", use sample set H ₁ ' sum tagged image set L vs f ₂ ' training to obtain f ₂ ”；

Step 8, f ₁” and f₂ "as the main classification model f ₁ And an auxiliary classification model f ₂ Returning to the step 2 until the main classification model f ₁ Stopping iteration when the expected classification performance or the marked sample exceeds the set upper limit, and entering a step 9;

step 9, utilizing the main classificationModel f ₁ And performing image classification prediction tasks.

2. The method for actively learning image-classification-oriented diversified queries according to claim 1, wherein the labeled image set L is used as a training set for the main classification model f ₁ And an auxiliary classification model f ₂ Training to obtain f ₁' and f₂ The' step comprises:

for the ith labeled image x in the labeled image set L _i Obtaining x _i Are respectively f ₁ and f₂ Classification probability for class j:

wherein ,

and />

Respectively f ₁ and f₂ Parameters of (2); / >

Is based on f ₁ The x obtained _i Classification probability belonging to class j +.>

Is based on f ₂ The x obtained _i Classification probability belonging to class j;

by minimizing the primary classification model f ₁ And an auxiliary classification model f ₂ Cross entropy loss on labeled image set L

and />

For the main classification model f ₁ And an auxiliary classification model f ₂ Training to obtain f ₁' and f₂ ' wherein for tagged images in the tagged image set L, the model f is classified in the main classification ₁ And an auxiliary classification model f ₂ The cross entropy losses above are respectively:

/>

wherein ,

and />

Respectively, training a main classification model f ₁ And an auxiliary classification model f ₂ Is a target function of (2); c is the total number of categories contained in the tagged image set L; l is the number of tagged images contained by the tagged image set L; y is _i Is the ith labeled image x _i Is a real tag of (1); j is x _i Is a predictive tag of (1); />

Is based on the x obtained by the main classification model _i Prediction probability belonging to class j, +.>

Is obtained based on auxiliary classification model _i Prediction probability belonging to class j;I{y _i =j } is an indication function if y _i Equal to j, 1, otherwise 0.

3. The method for actively learning image-classification-oriented diversified queries of claim 2 utilizing f ₁′ and f₂ The step of' feature extraction and prediction of all the unlabeled images in the unlabeled image set U comprises:

For the ith unlabeled image z in the unlabeled image set U _i Obtaining z _i Are respectively f ₁' and f₂ The classification probability of' class j:

wherein ,

and />

Respectively f ₁′ and f₂ ' parameters; />

Is based on f ₁ ' z obtained _i The probability of classification belonging to class j,

is based on f ₂ ' z obtained _i Classification probability belonging to class j;

the method comprises the following steps:

obtaining f ₁′ and f₂ ' respectively to z _i Is a predictive pseudo tag of (1)

and />

According to f ₁ The output of the last pooling layer yields the pair z _i The extracted feature vector.

4. The method for actively learning image-classification-oriented diversified query of claim 3 wherein said step of clustering and sorting unlabeled images in image set S using a multi-level diversity criterion to form a set of unlabeled sample candidates to be labeled comprises:

according to the feature vector corresponding to each unlabeled image in the image set S, the image set S is divided into C clusters by using a k-means clustering algorithm, and the C clusters are expressed as U=U ₁ ∪U ₂ ∪…∪U _C For the above C clusters, according to the sum of f in each cluster ₁ The predicted category label number is further divided into a plurality of sub-clusters, the iterative clustering operation is carried out until only one category label is in each minimum sub-cluster, the clustering operation is stopped, m minimum sub-clusters are obtained and expressed as

And selecting one with highest uncertainty from each minimum sub-cluster according to a BvSB criterion to obtain a query data set F with m samples, wherein the BvSB criterion is expressed as:

/>

wherein, for each smallest sub-cluster,

refers to samples within a cluster being f ₁ ' classification probability of class j; r is (r) _max1 Refer to f ₁ ' class subscript with maximum probability of classifying samples, r _max2 Refer to f ₁ ' category subscript with second greatest probability of classifying samples->

Representing the maximum predictive probability of a sample +.>

And the second highest prediction probability->

The difference between them, p is chosen for each smallest sub-cluster ^diff The smallest sample forms the query dataset F, p ^diff The smallest sample is the sample with the highest uncertainty;

if t samples have the same pseudo tag k in F, then the threshold for selecting different samples with the same pseudo tag k is expressed as:

where Var (·) is the variance operator, β is the hyper-parameter used to adjust the threshold,

represents f ₁ ' Classification probability for sample outputs in F with the same pseudo tag k, +.>

Represents f ₁ ' Classification probability for sample outputs in F with the same pseudo tag k, +.>

Ordering F by uncertainty of the samples and defining as

Wherein the uncertainty of the sample is from- >

Decrementing to +.>

From F _u Sequentially selecting samples and adding the samples into a label-free sample candidate set Q, wherein +_>

Is F _u The j sample in (1) is predicted as the k-th class, if t with a pseudo tag k exists in Q _k Sample number, calculate->

And t _k The similarity of the individual samples is:

if t _k 0, then

Added to Q; otherwise, u _k When greater than v _k When in use, will->

Added to Q; repeating the screening process until h samples exist in the Q, wherein h is a hyper-parameter controlling the size of the active learning screening batch, and +.>

Represents f ₁ ' classification probability for sample output with false tag k in Q.

5. The method for actively learning image-classification-oriented diversified queries of claim 4 wherein images in said image set H are each identified as f ₁′ and f₂ Sequencing the prediction confidence degrees, and respectively screening out a sample set H with highest confidence degree ₁ and H₂ The method comprises the following steps:

the images in the image set H are processed according to f ₁ Sequencing the 'prediction confidence degrees', screening out the top l multiplied by alpha images with the highest confidence degrees, and forming a sample set H ₁ ；

The images in the image set H are processed according to f ₂ Sequencing the 'prediction confidence degrees', screening out the top l multiplied by alpha images with the highest confidence degrees, and forming a sample set H ₂ ；

wherein ,f₁ The' confidence in prediction for the ith image in image set H is:

f ₂ the' confidence in prediction for the ith image in image set H is:

is f for the ith image in the image set H ₁ ' Classification probability of class j +.>

Is f for the ith image in the image set H ₂ The' classification probability of j classes, i is the number of tagged images contained in the tagged image set L, and alpha is the super parameter.

6. The utility model provides a diversified inquiry initiative learning device towards image classification which characterized in that, diversified inquiry initiative learning device towards image classification includes:

the acquisition module is used for acquiring an unlabeled image set U consisting of a first number of unlabeled images and a labeled image set L consisting of a second number of labeled images, wherein the first number is larger than the second number;

a first training module for using the labeled image set L as training set to respectively classify the main classification model f ₁ And an auxiliary classification model f ₂ Training to obtain f ₁′ and f₂ ' and utilize f ₁′ and f₂ ' extracting and predicting the characteristics of all the unlabeled images in the unlabeled image set U;

a first building block for building up the first building block from f ₁′ and f₂ ' predicting all the unlabeled images in the unlabeled image set to obtain the target image of the target image ₁' and f₂ ' image set S consisting of unlabeled images predicted to be different labels and composed of a quilt f ₁′ and f₂ ' predicted as phaseAn image set H consisting of unlabeled images of the same label;

the second construction module is used for clustering, sorting and screening the unlabeled images in the image set S by using a multi-level diversity criterion to form an unlabeled sample candidate set to be labeled;

the labeling module is used for labeling the label-free sample candidate set based on labeling operation, removing the labeled sample from the label-free image set U and adding the labeled sample into the label-free image set L;

a third construction module for respectively setting the images in the image set H according to f ₁′ and f₂ Sequencing the prediction confidence degrees, and respectively screening out a sample set H with highest confidence degree ₁ and H₂ And respectively giving the samples of the sample with the values of f ₁′ and f₂ 'pseudo tag output' to obtain sample set H ₁′ and H₂ ′；

A second training module for utilizing the sample set H ₂ ' sum tagged image set L vs f ₁ ' training to obtain f ₁ ", use sample set H ₁ ' sum tagged image set L vs f ₂ ' training to obtain f ₂ ”；

A circulation module for at f ₁” and f₂ "as the main classification model f ₁ And an auxiliary classification model f ₂ Returning to the step executed by the first training module until the main classification model f ₁ Stopping iteration when the expected classification performance or the labeling sample exceeds a set upper limit;

a prediction module for utilizing the main classification model f ₁ And performing image classification prediction tasks.

7. The image classification-oriented diversified query active learning apparatus of claim 6 wherein the first training module is configured to:

for the ith labeled image x in the labeled image set L _i Obtaining x _i Are respectively f ₁ and f₂ Classification probability for class j:

wherein ,

and />

Respectively f ₁ and f₂ Parameters of (2); />

Is based on f ₁ The x obtained _i Classification probability belonging to class j +.>

Is based on f ₂ The x obtained _i Classification probability belonging to class j;

by minimizing the primary classification model f ₁ And an auxiliary classification model f ₂ Cross entropy loss on labeled image set L

and />

For the main classification model f ₁ And an auxiliary classification model f ₂ Training to obtain f ₁′ and f₂ ' wherein for tagged images in the tagged image set L, the model f is classified in the main classification ₁ And an auxiliary classification model f ₂ The cross entropy losses above are respectively:

wherein ,

and />

Respectively, training a main classification model f ₁ And an auxiliary classification model f ₂ Is a target function of (2); c is the total number of categories contained in the tagged image set L; l is the number of tagged images contained by the tagged image set L; y is _i Is the ith labeled image x _i Is a real tag of (1); j is x _i Is a predictive tag of (1); />

Is based on the x obtained by the main classification model _i Prediction probability belonging to class j, +.>

Is obtained based on auxiliary classification model _i Prediction probability belonging to class j; i { y } _i =j } is an indication function if y _i Equal to j, 1, otherwise 0.

8. The image classification-oriented diversified query active learning apparatus of claim 7 wherein the first training module is configured to:

for the ith unlabeled image z in the unlabeled image set U _i Obtaining z _i Are respectively f ₁' and f₂ The classification probability of' class j:

wherein ,

and />

Respectively f ₁' and f₂ ' parameters; />

Is based on f ₁ ' z obtained _i The probability of classification belonging to class j,

is based on f ₂ ' z obtained _i Classification probability belonging to class j;

the method comprises the following steps:

obtaining f ₁′ and f₂ ' respectively to z _i Is a predictive pseudo tag of (1)

and />

According to f ₁ Output of the last pooling layerObtain the pair z _i The extracted feature vector.

9. The image-classification-oriented diversified query active learning device of claim 8 wherein the second building module is configured to:

according to the feature vector corresponding to each unlabeled image in the image set S, the image set S is divided into C clusters by using a k-means clustering algorithm, and the C clusters are expressed as U=U ₁ ∪U ₂ ∪…∪U _C For the above C clusters, according to the sum of f in each cluster ₁ The predicted category label number is further divided into a plurality of sub-clusters, the iterative clustering operation is carried out until only one category label is in each minimum sub-cluster, the clustering operation is stopped, m minimum sub-clusters are obtained and expressed as

And selecting one with highest uncertainty from each minimum sub-cluster according to a BvSB criterion to obtain a query data set F with m samples, wherein the BvSB criterion is expressed as: />

Wherein, for each smallest sub-cluster,

refers to samples within a cluster being f ₁ ' classification probability of class j; r is (r) _max1 Refer to f ₁ ' Classification of samplesCategory subscript with highest probability, r _max2 Refer to f ₁ ' category subscript with second greatest probability of classifying samples->

Representing the maximum predictive probability of a sample +.>

And the second highest prediction probability->

The difference between them, p is chosen for each smallest sub-cluster ^diff The smallest sample forms the query dataset F, p ^diff The smallest sample is the sample with the highest uncertainty;

if t samples have the same pseudo tag k in F, then the threshold for selecting different samples with the same pseudo tag k is expressed as:

where Var (·) is the variance operator, β is the hyper-parameter used to adjust the threshold,

Represents f ₁ ' Classification probability for sample outputs in F with the same pseudo tag k, +.>

Represents f ₁ ' Classification probability for sample outputs in F with the same pseudo tag k, +.>

Ordering F by uncertainty of the samples and defining as

Wherein the uncertainty of the sample is from->

Decrementing to +.>

From F _u Sequentially selecting samples and adding the samples into a label-free sample candidate set Q, wherein +_>

Is F _u The j sample in (1) is predicted as the k-th class, if t with a pseudo tag k exists in Q _k Sample number, calculate->

And t _k The similarity of the individual samples is:

if t _k 0, then

Added to Q; otherwise, u _k When greater than v _k When in use, will->

Added to Q; repeating the screening process until h samples exist in the Q, wherein h is a hyper-parameter controlling the size of the active learning screening batch, and +.>

Represents f ₁ ' classification probability for sample output with false tag k in Q.

10. The image classification-oriented diversified query active learning device of claim 9 wherein the third building module is configured to:

the images in the image set H are processed according to f ₁ Sequencing the 'prediction confidence degrees', screening out the top l multiplied by alpha images with the highest confidence degrees, and forming a sample set H ₁ ；

The images in the image set H are processed according to f ₂ Sequencing the 'prediction confidence degrees', screening out the top l multiplied by alpha images with the highest confidence degrees, and forming a sample set H ₂ ；

wherein ,f₁ The' confidence in prediction for the ith image in image set H is:

f ₂ the' confidence in prediction for the ith image in image set H is:

is f for the ith image in the image set H ₁ ' Classification probability of class j +.>

Is f for the ith image in the image set H ₂ The' classification probability of j classes, i is the number of tagged images contained in the tagged image set L, and alpha is the super parameter. />

Images