CN113537389B - Robust image classification method and device based on model embedding - Google Patents
Robust image classification method and device based on model embedding Download PDFInfo
- Publication number
- CN113537389B CN113537389B CN202110898433.3A CN202110898433A CN113537389B CN 113537389 B CN113537389 B CN 113537389B CN 202110898433 A CN202110898433 A CN 202110898433A CN 113537389 B CN113537389 B CN 113537389B
- Authority
- CN
- China
- Prior art keywords
- model
- training image
- image sample
- image
- label
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 88
- 238000012549 training Methods 0.000 claims abstract description 194
- 238000003062 neural network model Methods 0.000 claims abstract description 89
- 238000012216 screening Methods 0.000 claims abstract description 39
- 238000007865 diluting Methods 0.000 claims abstract description 16
- 238000010790 dilution Methods 0.000 claims description 32
- 239000012895 dilution Substances 0.000 claims description 32
- 239000011159 matrix material Substances 0.000 claims description 11
- 238000004590 computer program Methods 0.000 claims description 8
- 230000006870 function Effects 0.000 claims description 8
- 238000012545 processing Methods 0.000 claims description 7
- 238000002372 labelling Methods 0.000 abstract description 11
- 239000000463 material Substances 0.000 abstract description 6
- 230000008569 process Effects 0.000 description 16
- 239000000243 solution Substances 0.000 description 11
- 238000010586 diagram Methods 0.000 description 9
- 238000013145 classification model Methods 0.000 description 8
- 238000013528 artificial neural network Methods 0.000 description 6
- 238000012360 testing method Methods 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 238000007635 classification algorithm Methods 0.000 description 4
- 238000003113 dilution method Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 238000013135 deep learning Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 238000002474 experimental method Methods 0.000 description 3
- 230000003446 memory effect Effects 0.000 description 3
- 230000007774 longterm Effects 0.000 description 2
- 210000002569 neuron Anatomy 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 241000282994 Cervidae Species 0.000 description 1
- 241000283073 Equus caballus Species 0.000 description 1
- 241000282326 Felis catus Species 0.000 description 1
- 241000009328 Perro Species 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000002059 diagnostic imaging Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000013100 final test Methods 0.000 description 1
- 239000003550 marker Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Molecular Biology (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Editing Of Facsimile Originals (AREA)
- Image Processing (AREA)
Abstract
The disclosure provides a robust image classification method and device based on model embedding, wherein the method comprises the following steps: inputting the image to be classified into an image classification depth neural network model, and outputting an image class identification result corresponding to the image to be classified; the image classification deep neural network model is that training image samples are input into the deep neural network model to obtain embedded features of the training image samples in the last hidden layer of the neural network model; diluting the noise label of the training image sample according to the embedded characteristics; sample screening is carried out according to the diluted label, and a screened image sample is obtained; and updating model parameters of the deep neural network model according to the screened image samples to obtain the image classification deep neural network model. According to the technical scheme, the robust image classification performance can be guaranteed while the labeling cost is reduced, so that the manpower and material cost of image labeling is saved.
Description
Technical Field
The disclosure relates to the field of computer technology, and in particular relates to a robust image classification method and device based on model embedding.
Background
The image classification problem based on deep learning makes great progress in the full-supervision scene, but the performance of the classification algorithm in the full-supervision scene is often severely dependent on the accuracy of image annotation. In practical application with larger scale, the acquired training data set is often noisy due to professional knowledge limitation of a marker, working fatigue and other artificial factors, namely, a part of pictures are inaccurately marked. How to design a robust classification model to resist negative effects caused by tag noise is a key to expanding the application scene of an image classification model. There are also some technical solutions currently trying to design a robust image classification model. In the prior art, the model prediction is utilized to try to correct the wrong image annotation, and then the corrected image is utilized to train the later model; also, there are technical solutions that use model prediction to screen training samples, and the screened clean image (accurately labeled image) will be further used to train a later model.
However, the above-described prior art schemes have a serious technical disadvantage in that early model prediction, while effective in helping to reduce the effect of labeling noise, still has a portion of intractable noise present, while the "early" phase of early model training is not controllable. More specifically, the prediction results of the early model are not a percentage accurate, and secondary training with the prediction results can produce accumulated errors, resulting in an iterative classification model that is worse and worse. Compared with model prediction, the model embedding-based method provided by the scheme is more robust, and can effectively avoid the problem of accumulated errors.
It is well known that DNN requires a long training process to get a reliable prediction, which necessarily conflicts with "early" in memory effects. Although DNN has the opportunity to learn a clean image during early training, it is not enough to learn a reliable clean classification model, and the noise image participates in training, resulting in the model beginning to fit the noise. As a consequence, the classification performance of early models is improved and less than optimal. The number of discarded noisy images is continuously reduced by using a moving average method to achieve the goal of taking a compromise between the "long term" required for DNN training and the "early" required for memory effect. However, this compromise requires manual adjustment and the adjustment process is laborious.
Disclosure of Invention
The invention provides a robust image classification method and device based on model embedding, which are used for solving the defects that manual adjustment is needed and the adjustment process is labor-consuming in the prior art, realizing the reduction of the labeling cost and guaranteeing the robust image classification performance at the same time so as to save the manpower and material cost of image labeling.
In a first aspect, the present disclosure provides a robust image classification method based on model embedding, comprising:
acquiring an image to be classified;
inputting the image to be classified into an image classification depth neural network model, and outputting an image class identification result corresponding to the image to be classified;
the image classification depth neural network model is obtained through training by the following method:
inputting a training image sample into a deep neural network model, and acquiring embedded features of the training image sample in a last hidden layer of the neural network model;
diluting the noise label of the training image sample according to the embedded characteristics to obtain a diluted label;
sample screening is carried out according to the diluted label, and a screened image sample is obtained;
and inputting the screened image sample into the deep neural network model, and updating model parameters of the deep neural network model to obtain the image classification deep neural network model.
According to the robust image classification method based on model embedding, the noise label of the training image sample is diluted according to the embedding characteristics, and the robust image classification method specifically comprises the following steps:
step one: obtaining K neighbors of each training image sample;
transmitting the label information of the K neighbor to the corresponding training image sample to realize dilution of the noise label of the training image sample;
step two: repeating the first step until convergence occurs, and determining the final dilution result of the noise label of the training image sample.
According to the robust image classification method based on model embedding provided by the disclosure, the image sample screening is performed according to the diluted label, and the robust image classification method specifically comprises the following steps:
determining a noise label of the training image sample and a dilution label corresponding to the training image sample;
inputting the noise label and the dilution label into a first model, and determining the weight of the image sample;
and screening the image samples according to the weights.
According to the robust image classification method based on model embedding, model parameters of the deep neural network model are updated according to a second model, wherein the second model is as follows:
wherein B is a training image sample after screening,for a gradient solver with respect to w, w is a parameter of the deep neural network model, e i Is the weight of the ith training image sample, < +.>Representing noise labels corresponding to training image samples, f (x i ) Is the real label of the training image sample, +.>Is a loss function.
According to the robust image classification method based on model embedding, the method for acquiring the K neighbor of each training image sample is as follows:
determining Euclidean space distance values between each training image sample and other training image samples in the training image sample set;
sorting the Euclidean space distance values from small to large;
and screening out the first K Euclidean space distance values, wherein the training image samples corresponding to the first K Euclidean space distance values are K neighbors of each training image sample.
According to the robust image classification method based on model embedding, the first model is as follows:
wherein e i Is the weight of the ith training image sample,noise label corresponding to the training image sample +.>Is an n x C matrix representing the dilution signature, wherein C is the number of categories, +.>The representation means make->And j takes the value of the maximum value.
In a second aspect, the present disclosure provides a robust image classification apparatus based on model embedding, including:
the first processing module is used for acquiring images to be classified;
the second processing module is used for inputting the image to be classified into an image classification depth neural network model and outputting an image class identification result corresponding to the image to be classified;
the image classification depth neural network model is obtained through training by the following method:
inputting a training image sample into a deep neural network model, and acquiring embedded features of the training image sample in a last hidden layer of the neural network model;
diluting the noise label of the training image sample according to the embedded characteristics to obtain a diluted label;
sample screening is carried out according to the diluted label, and a screened image sample is obtained;
and inputting the screened image sample into the deep neural network model, and updating model parameters of the deep neural network model to obtain the image classification deep neural network model.
According to the robust image classification device based on model embedding, the noise label of the training image sample is diluted according to the embedding characteristics, and the robust image classification device specifically comprises:
step one: obtaining K neighbors of each training image sample;
transmitting the label information of the K neighbor to the corresponding training image sample to realize dilution of the noise label of the training image sample;
step two: repeating the first step until convergence occurs, and determining the final dilution result of the noise label of the training image sample.
According to the robust image classification device based on model embedding provided by the disclosure, the image sample screening is performed according to the diluted label, which specifically comprises:
determining a noise label of the training image sample and a dilution label corresponding to the training image sample;
inputting the noise label and the dilution label into a first model, and determining the weight of the image sample;
and screening the image samples according to the weights.
According to the robust image classification device based on model embedding, model parameters of the deep neural network model are updated according to a second model, wherein the second model is as follows:
wherein B is a training image sample after screening,to solve for the operator for gradients with respect to w, which is deepParameters of the neural network model, e i Is the weight of the ith training image sample, < +.>Representing noise labels corresponding to training image samples, f (x i ) Is the real label of the training image sample, +.>Is a loss function.
According to the robust image classification device based on model embedding, the method for acquiring the K neighbor of each training image sample comprises the following steps:
determining Euclidean space distance values between each training image sample and other training image samples in the training image sample set;
sorting the Euclidean space distance values from small to large;
and screening out the first K Euclidean space distance values, wherein the training image samples corresponding to the first K Euclidean space distance values are K neighbors of each training image sample.
According to the robust image classification device based on model embedding, the first model is:
wherein e i Is the weight of the ith training image sample,noise label corresponding to the training image sample +.>Is an n x C matrix representing the dilution signature, wherein C is the number of categories, +.>The representation means make->And j takes the value of the maximum value.
In a third aspect, the present disclosure also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the model-embedding-based robust image classification method as described in any one of the above when the program is executed.
In a fourth aspect, the present disclosure also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the model-embedding based robust image classification method as described in any of the above.
According to the robust image classification method and device based on model embedding, training image samples are input into a deep neural network model, and embedded features of the training image samples in the last hidden layer of the neural network model are obtained; and diluting the noise label of the training image sample according to the embedded characteristic, screening the sample image based on the diluted label, and updating parameters of the model by using the screened image, so that the technical scheme of the disclosure can ensure the robust image classification performance while reducing the labeling cost, thereby saving the manpower and material cost of image labeling.
Drawings
In order to more clearly illustrate the present disclosure or the prior art solutions, a brief description will be given below of the drawings that are needed in the embodiments or prior art descriptions, it being apparent that the drawings in the following description are some embodiments of the present disclosure and that other drawings may be obtained from these drawings without inventive effort to a person of ordinary skill in the art.
FIG. 1 is a flow diagram of a robust image classification method based on model embedding provided by the present disclosure;
FIG. 2 is a flow diagram of a robust image classification algorithm based on model embedding provided by the present disclosure;
FIG. 3 is a schematic diagram of a find sample k neighbor process provided by the present disclosure;
fig. 4 is a schematic diagram of a label dilution process based on k-nearest neighbor provided by the present disclosure;
FIG. 5 is a schematic diagram showing a comparison of the performance of the model predictive method and the model embedding method provided by the present disclosure;
FIG. 6 is a schematic structural diagram of a robust image classification device based on model embedding provided by the present disclosure;
fig. 7 is a schematic structural diagram of an electronic device provided by the present disclosure.
Detailed Description
For the purposes of making the objects, technical solutions and advantages of the embodiments of the present disclosure more apparent, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure, and it is apparent that the described embodiments are some embodiments, but not all embodiments of the present disclosure. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the disclosed embodiments, are within the scope of the disclosed embodiments.
The following describes a robust image classification method based on model embedding according to an embodiment of the present disclosure with reference to fig. 1 to fig. 2, including:
step 100: acquiring an image to be classified;
specifically, the image classification problem is used as a bottom image processing problem and has important application in the fields of medical imaging, satellite image remote sensing, internet crowdsourcing and the like. In recent years, the problem of image classification by deep learning has achieved a good effect. Therefore, when the deep neural network model is utilized for image classification, an image to be classified is acquired first.
Step 200: inputting the image to be classified into an image classification depth neural network model, and outputting an image class identification result corresponding to the image to be classified;
the image classification depth neural network model is obtained through training by the following method:
inputting a training image sample into a deep neural network model, and acquiring embedded features of the training image sample in a last hidden layer of the neural network model;
diluting the noise label of the training image sample according to the embedded characteristics to obtain a diluted label;
sample screening is carried out according to the diluted label, and a screened image sample is obtained;
and inputting the screened image sample into the deep neural network model, and updating model parameters of the deep neural network model to obtain the image classification deep neural network model.
In particular, a deep neural network (Deep Neural Networks, DNN) can be understood as a neural network with many hidden layers, also called deep feed forward network (DFN), and Multi-Layer perceptron (MLP) is divided by the positions of the different layers, and the neural network layers inside the DNN can be divided into: input layer, hidden layer and output layer, the first layer is input layer, the last layer is output layer, and the middle layer number is hidden layer. The layers are fully connected, i.e., any neuron in the i layer must be connected to any neuron in the i+1 layer.
Before the deep neural network is used for classifying images, the deep neural network model needs to be trained, and in the existing training method, DNN has the opportunity to learn a clean image in the early training process, but a reliable clean classification model cannot be learned, the noise image participates in training, and the model starts to fit noise. As a consequence, the classification performance of early models is improved and less than optimal. Or the number of discarded noisy images is continuously reduced by using a moving average method to achieve the goal of taking a compromise between the "long term" required for DNN training and the "early" required for memory effect. However, this compromise requires manual adjustment and the adjustment process is laborious.
In an actual application scene, considering the cost of manpower and material resources or subjectivity of a classification task, the actually available training data is usually influenced by external noise, so that the performance of the traditional deep learning algorithm is limited. Therefore, how to design a robust image classification algorithm based on noisy training images is of great importance in academic research and industrial applications. The robustness of the model embedded features adopted by the scheme has longer period in the training process, and the problem of manually adjusting parameters is avoided.
In the early training process of the deep neural network, the model embedding has stronger robustness than model prediction, a label noise reduction scheme based on the model embedding is designed based on the model embedding, and a clean image screening scheme is further designed to carry out DNN model training.
Fig. 2 shows a basic flow chart of a robust image classification algorithm based on model embedding proposed by the present patent. Specifically, training image samples (input images) are input to DNN, the embedded features (model embedding) of the last hidden layer of the training model are taken for label noise dilution (label edition), the diluted labels are further used for image sample screening (image selection), the clean samples (selected images) obtained by screening are fed back to the DNN training process, and finally a robust classification model is output. Classifying the images based on the obtained image classification model.
According to the robust image classification method based on model embedding, training image samples are input into a deep neural network model, and embedded features of the training image samples in the last hidden layer of the neural network model are obtained; and diluting the noise label of the training image sample according to the embedded characteristic, screening the sample image based on the diluted label, and updating parameters of the model by using the screened image, so that the technical scheme of the disclosure can ensure the robust image classification performance while reducing the labeling cost, thereby saving the manpower and material cost of image labeling.
According to the robust image classification method based on model embedding provided by the embodiment of the disclosure, the noise label of the training image sample is diluted according to the embedding feature, which specifically includes:
step one: acquiring K neighbors of each training image sample;
transmitting the label information of the K neighbor to the corresponding training image sample to dilute the noise label of the training image sample;
step two: repeating the first step until convergence occurs, and determining a final dilution result of the noise label of the training image sample.
Specifically, during the label dilution process, the K-nearest neighbor (K nearest neighbors, KNN) of the training image sample is used to further extract the robust information provided by the model embedding. Fig. 3 shows a schematic diagram of the process of finding K neighbors of an image sample, i.e. finding K image samples that are closest to each training image sample in the european space (where +.and x respectively indicate that the image sample is correctly/incorrectly labeled). Fig. 4 illustrates a noise tag dilution process based on K-nearest neighbor. In each round of dilution process, firstly searching K neighbor of each training image sample, then transmitting label information of the K neighbor of each training image sample to the image sample and updating labels of the image sample; the process iterates through the T rounds until convergence.
Given input training data(x represents a training image, +.>Representing the noise label to which the image corresponds, n representing the total number of training image samples), training DNNf (x) with this data, desirably given arbitrary test data x t ，f(x t ) The true label y of the label can be accurately predicted t . Let w denote the updated parameters of f (x), phi (x) denote the embedded features of the last hidden layer of f (x), NN k (x i The method comprises the steps of carrying out a first treatment on the surface of the Phi) represents the image sample x i K-nearest neighbor on feature phi (x).
During each round of label dilution, for each image sample x i First, it is embedded byIn-feature phi (x) i ) Obtaining its K neighbor NN k (x i The method comprises the steps of carrying out a first treatment on the surface of the Phi) and then propagates its k-nearest neighbor tag information to image sample x i To obtain the diluted label z i 。
Formally expressing the t-th round iteration process as:
wherein W represents a similarity matrix constructed based on phi (x), and the first term on the right of formula (1) represents sample x i The second right term represents the label information of the k-nearest neighbor of the t-1 th iteration sample x i And alpha is the weight parameter of the two items. The above written matrix form is:
Z (t) ＝(1-α)·WZ (t-1) +α·Z (t-1) (2)
with noise tagsOne-hot representation (one-hot representation) of the initialization dilution label Z (0) Then, Z is updated by equation (2) until convergence.
Next, the process of generating the similarity matrix W is described in detail. Given training sampleIs characterized byFirstly, constructing an n multiplied by n sparse matrix A, wherein the element A is the same as the element A ij The method comprises the following steps:
where γ is a super parameter, typically set to 4; finally, a similarity matrix W=D is constructed -1/2 W′D -1/2 WhereinD＝diag(W′1)。
According to the robust image classification method based on model embedding provided by the embodiment of the disclosure, the image sample screening is performed according to the diluted label, which specifically includes:
determining a noise label of the training image sample and a dilution label corresponding to the training image sample;
inputting the noise label and the dilution label into a first model, and determining the weight of the image sample;
and screening the image samples according to the weights.
Specifically, given a training sampleAnd its dilution label->Obtaining a training weight e of each sample through a formula (4), and further updating DNN parameters w through a formula (5), wherein I (a) is an indication function, and I (a) =1 when a is true, otherwise, I (a) =0; />The operator is solved for the gradient with respect to w.
Wherein B is a training image sample after screening,for a gradient solver with respect to w, w is a parameter of the deep neural network model, e i Is the ith trainingWeights of image samples ∈ ->Representing noise labels corresponding to training image samples, f (x i ) Is the real label of the training image sample, +.>Is a loss function.
In the formula (4)Indicating->The value of j at maximum, wherein +.>Is a matrix of n x C, C being the number of classifications. Each row (i) represents a set of one hot labels; in one hot label, each element is either 0 or 1, and there is one and only one 1. Here->I.e. the position of this 1 in group i one hot label.
Wherein I (a) is an indication function, I (a) =1 when a is true, otherwise I (a) =0; is shown inWhen it is established, the weight value corresponding to the sample is 1, i.e. the image sample is screened, whileWhen not established, the weight of the sample is 0, i.e. the image sample is not screened.
Further, parameters of the deep neural network model are updated according to the filtered image samples in the mode of the formula (5).
According to the robust image classification method based on model embedding provided by the embodiment of the disclosure, the method for acquiring the K neighbor of each training image sample is as follows:
determining Euclidean space distance values between each training image sample and other training image samples in the training image sample set;
sorting the Euclidean space distance values from small to large;
and screening out the first K Euclidean space distance values, wherein the training image samples corresponding to the first K Euclidean space distance values are K neighbors of each training image sample.
In experiments, the validity of the method proposed by this patent was verified using CIFAR-10 and CIFAR-100 datasets. Specifically, CIFAR-10 and CIFAR-100 each contain 50000 training images and 10000 test images, and the number of image categories is 10 and 100, respectively. Note that the data set was initially a clean data set, which was converted to a noisy data set by manual addition of tag noise in this experiment using the following conversion scheme: symmetrical flip and asymmetrical flip. The symmetrical inversion is also called random inversion, and the images of each category are turned into all other labels with a certain probability rho; the asymmetric inversion is also called pairing inversion, and the images of each category are inverted into a semantic information similar label (such as cat, dog, deer and horse) according to a certain probability rho, and the inversion mode can be applied to fine-grained image classification scenes in practical application. The probability ρ is also called the noise ratio, i.e. the proportion of the noisy image. The noise rates considered for asymmetric inversion are: 40%, 50% and 60%; the noise rates considered for symmetric flipping are: 20%, 30% and 40%. The experimental results are shown in tables 1 and 2 below:
table 1: CIFAR-10 dataset experimental results
Table 2: CIFAR-100 dataset experimental results
Tables 1 and 2 show the experimental results (test accuracy and standard deviation) of the present scheme (LEND) and the existing schemes (CE, co-teaching, joCor and GCE) on CIFAR-10 and CIFAR-100 data sets, respectively. The Best line represents the optimal test result in the training process, the Last line represents the final test result in the training process, and the optimal test result of each group of experiments is marked in bold.
In addition, in order to verify that the model embedding method adopted by the scheme is superior to the model prediction method, classification performance curves of the model embedding method and the model prediction method in the training process are further shown, as shown in fig. 5, wherein a dotted line represents the precision of the model embedding method, and a solid line represents the precision of the model prediction method.
The test results show that the robust picture classification method based on model embedding provided by the patent can obtain better results than the prior scheme under various label noise scenes and various noise rates. Meanwhile, the model embedding adopted by the scheme is verified to have stronger robustness than model prediction. Therefore, the technical scheme can be applied to noisy image classification task types, and more accurate classification effect can be ensured.
In a second aspect, in connection with fig. 6, an embodiment of the present disclosure provides a robust image classification apparatus based on model embedding, where the robust image classification apparatus includes:
a first processing module 61, configured to acquire an image to be classified;
the second processing module 62 is configured to input the image to be classified into an image classification deep neural network model, and output an image classification recognition result corresponding to the image to be classified;
the image classification depth neural network model is obtained through training by the following method:
inputting a training image sample into a deep neural network model, and acquiring embedded features of the training image sample in a last hidden layer of the neural network model;
diluting the noise label of the training image sample according to the embedded characteristics to obtain a diluted label;
sample screening is carried out according to the diluted label, and a screened image sample is obtained;
and inputting the screened image sample into the deep neural network model, and updating model parameters of the deep neural network model to obtain the image classification deep neural network model.
Since the apparatus provided by the embodiment of the present invention may be used to perform the method described in the above embodiment, its working principle and beneficial effects are similar, so that details will not be described herein, and reference will be made to the description of the above embodiment.
According to the robust image classification device based on model embedding, training image samples are input into a deep neural network model, and embedding characteristics of the training image samples in the last hidden layer of the neural network model are obtained; and diluting the noise label of the training image sample according to the embedded characteristic, screening the sample image based on the diluted label, and updating parameters of the model by using the screened image, so that the technical scheme of the disclosure can ensure the robust image classification performance while reducing the labeling cost, thereby saving the manpower and material cost of image labeling.
According to an embodiment of the present disclosure, a robust image classification device based on model embedding is provided, where the diluting, according to the embedding feature, a noise label of the training image sample specifically includes:
step one: obtaining K neighbors of each training image sample;
transmitting the label information of the K neighbor to the corresponding training image sample to realize dilution of the noise label of the training image sample;
step two: repeating the first step until convergence occurs, and determining the final dilution result of the noise label of the training image sample.
According to the robust image classification device based on model embedding provided in the embodiment of the present disclosure, the filtering of the image sample according to the diluted label specifically includes:
determining a noise label of the training image sample and a dilution label corresponding to the training image sample;
inputting the noise label and the dilution label into a first model, and determining the weight of the image sample;
and screening the image samples according to the weights.
According to the robust image classification device based on model embedding, model parameters of the deep neural network model are updated according to a second model, wherein the second model is as follows:
wherein B is a training image sample after screening,for a gradient solver with respect to w, w is a parameter of the deep neural network model, e i Is the weight of the ith training image sample, < +.>Representing noise labels corresponding to training image samples, f (x i ) Is the real label of the training image sample, +.>Is a loss function.
According to the robust image classification device based on model embedding provided by the embodiment of the disclosure, the method for acquiring the K neighbor of each training image sample is as follows:
determining Euclidean space distance values between each training image sample and other training image samples in the training image sample set;
sorting the Euclidean space distance values from small to large;
and screening out the first K Euclidean space distance values, wherein the training image samples corresponding to the first K Euclidean space distance values are K neighbors of each training image sample.
According to the robust image classification device based on model embedding, the first model is:
wherein e i Is the weight of the ith training image sample,noise label corresponding to the training image sample +.>Is an n x C matrix representing the dilution signature, wherein C is the number of categories, +.>The representation means make->And j takes the value of the maximum value.
Fig. 7 illustrates a physical schematic diagram of an electronic device, as shown in fig. 7, which may include: processor 710, communication interface (Communications Interface) 720, memory 730, and communication bus 740, wherein processor 710, communication interface 720, memory 730 communicate with each other via communication bus 740. Processor 710 may invoke logic instructions in memory 730 to perform a robust image classification method based on model embedding, the method comprising: acquiring an image to be classified; inputting the image to be classified into an image classification depth neural network model, and outputting an image class identification result corresponding to the image to be classified; the image classification depth neural network model is obtained through training by the following method: inputting a training image sample into a deep neural network model, and acquiring embedded features of the training image sample in a last hidden layer of the neural network model; diluting the noise label of the training image sample according to the embedded characteristics to obtain a diluted label; sample screening is carried out according to the diluted label, and a screened image sample is obtained; and inputting the screened image sample into the deep neural network model, and updating model parameters of the deep neural network model to obtain the image classification deep neural network model.
Further, the logic instructions in the memory 730 described above may be implemented in the form of software functional units and may be stored in a computer readable storage medium when sold or used as a stand alone product. Based on such understanding, the technical solutions of the embodiments of the present disclosure may be essentially or, what contributes to the prior art, or part of the technical solutions, may be embodied in the form of a software product stored in a storage medium, including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the embodiments of the present disclosure. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
In another aspect, the present disclosure also provides a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, are capable of performing the robust image classification method based on model embedding provided by the above methods, the method comprising: acquiring an image to be classified; inputting the image to be classified into an image classification depth neural network model, and outputting an image class identification result corresponding to the image to be classified; the image classification depth neural network model is obtained through training by the following method: inputting a training image sample into a deep neural network model, and acquiring embedded features of the training image sample in a last hidden layer of the neural network model; diluting the noise label of the training image sample according to the embedded characteristics to obtain a diluted label; sample screening is carried out according to the diluted label, and a screened image sample is obtained; and inputting the screened image sample into the deep neural network model, and updating model parameters of the deep neural network model to obtain the image classification deep neural network model.
In yet another aspect, the present disclosure also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, is implemented to perform the above provided method of model-based embedded robust image classification, the method comprising: acquiring an image to be classified; inputting the image to be classified into an image classification depth neural network model, and outputting an image class identification result corresponding to the image to be classified; the image classification depth neural network model is obtained through training by the following method: inputting a training image sample into a deep neural network model, and acquiring embedded features of the training image sample in a last hidden layer of the neural network model; diluting the noise label of the training image sample according to the embedded characteristics to obtain a diluted label; sample screening is carried out according to the diluted label, and a screened image sample is obtained; and inputting the screened image sample into the deep neural network model, and updating model parameters of the deep neural network model to obtain the image classification deep neural network model.
The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective embodiments or some parts of the embodiments.
Finally, it should be noted that: the above embodiments are merely for illustrating the technical solution of the present disclosure, and are not limiting thereof; although the present disclosure has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present disclosure.
Claims (12)
1. A robust image classification method based on model embedding, comprising:
acquiring an image to be classified;
inputting the image to be classified into an image classification depth neural network model, and outputting an image class identification result corresponding to the image to be classified;
the image classification depth neural network model is obtained through training by the following method:
inputting a training image sample into a deep neural network model, and acquiring embedded features of the training image sample in a last hidden layer of the neural network model;
diluting the noise label of the training image sample according to the embedded characteristics to obtain a diluted label;
sample screening is carried out according to the diluted label, and a screened image sample is obtained;
inputting the screened image sample into the deep neural network model, and updating model parameters of the deep neural network model to obtain the image classification deep neural network model;
the diluting the noise label of the training image sample according to the embedded feature specifically includes:
step one: obtaining K neighbors of each training image sample;
transmitting the label information of the K neighbor to the corresponding training image sample to realize dilution of the noise label of the training image sample;
step two: repeating the first step until convergence occurs, and determining the final dilution result of the noise label of the training image sample.
2. The robust image classification method based on model embedding of claim 1, wherein the performing image sample screening according to the diluted label specifically comprises:
determining a noise label of the training image sample and a dilution label corresponding to the training image sample;
inputting the noise label and the dilution label into a first model, and determining the weight of the image sample;
and screening the image samples according to the weights.
3. The robust image classification method based on model embedding of claim 1, wherein model parameters of the deep neural network model are updated according to a second model, wherein the second model is:
；
wherein B is a training image sample after screening,for->Gradient solver of>Is a parameter of the deep neural network model, +.>Is the firstiWeights of individual training image samples, +.>Noise label corresponding to the training image sample +.>Is the real label of the training image sample, +.>Is a loss function.
4. The robust image classification method based on model embedding of claim 1, wherein the method of obtaining K-nearest neighbor of each training image sample is:
determining Euclidean space distance values between each training image sample and other training image samples in the training image sample set;
sorting the Euclidean space distance values from small to large;
and screening out the first K Euclidean space distance values, wherein the training image samples corresponding to the first K Euclidean space distance values are K neighbors of each training image sample.
5. The robust image classification method based on model embedding of claim 2, wherein the first model is:
；
wherein,is the firstiWeights of individual training image samples, +.>Noise label corresponding to the training image sample +.>Is an n x C matrix representing the dilution signature, wherein C is the number of categories, +.>Indicating->And j takes the value of the maximum value.
6. A robust image classification apparatus based on model embedding, comprising:
the first processing module is used for acquiring images to be classified;
the second processing module is used for inputting the image to be classified into an image classification depth neural network model and outputting an image class identification result corresponding to the image to be classified;
the image classification depth neural network model is obtained through training by the following method:
inputting a training image sample into a deep neural network model, and acquiring embedded features of the training image sample in a last hidden layer of the neural network model;
diluting the noise label of the training image sample according to the embedded characteristics to obtain a diluted label;
sample screening is carried out according to the diluted label, and a screened image sample is obtained;
inputting the screened image sample into the deep neural network model, and updating model parameters of the deep neural network model to obtain the image classification deep neural network model;
the diluting the noise label of the training image sample according to the embedded feature specifically includes:
step one: obtaining K neighbors of each training image sample;
transmitting the label information of the K neighbor to the corresponding training image sample to realize dilution of the noise label of the training image sample;
step two: repeating the first step until convergence occurs, and determining the final dilution result of the noise label of the training image sample.
7. The robust image classification apparatus based on model embedding of claim 6, wherein the image sample screening according to the diluted label specifically comprises:
determining a noise label of the training image sample and a dilution label corresponding to the training image sample;
inputting the noise label and the dilution label into a first model, and determining the weight of the image sample;
and screening the image samples according to the weights.
8. The model-embedding-based robust image classification apparatus of claim 6, wherein model parameters of the deep neural network model are updated according to a second model, wherein the second model is:
；
wherein B is a training image sample after screening,for->Gradient solver of>Is a parameter of the deep neural network model, +.>Is the firstiWeights of individual training image samples, +.>Noise label corresponding to the training image sample +.>Is the real label of the training image sample, +.>Is a loss function.
9. The robust image classification apparatus based on model embedding of claim 6, wherein the method of obtaining K-nearest neighbor of each training image sample is:
determining Euclidean space distance values between each training image sample and other training image samples in the training image sample set;
sorting the Euclidean space distance values from small to large;
and screening out the first K Euclidean space distance values, wherein the training image samples corresponding to the first K Euclidean space distance values are K neighbors of each training image sample.
10. The model-embedding-based robust image classification apparatus of claim 7, wherein the first model is:
；
wherein,is the firstiWeights of individual training image samples, +.>Noise label corresponding to the training image sample +.>Is an n x C matrix representing the dilution signature, wherein C is the number of categories, +.>Indicating->And j takes the value of the maximum value.
11. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the model-embedding based robust image classification method according to any of claims 1 to 5 when the program is executed by the processor.
12. A non-transitory computer readable storage medium having stored thereon a computer program, which when executed by a processor, implements the steps of the model embedding based robust image classification method according to any of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110898433.3A CN113537389B (en) | 2021-08-05 | 2021-08-05 | Robust image classification method and device based on model embedding |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110898433.3A CN113537389B (en) | 2021-08-05 | 2021-08-05 | Robust image classification method and device based on model embedding |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113537389A CN113537389A (en) | 2021-10-22 |
| CN113537389B true CN113537389B (en) | 2023-11-07 |
Family
ID=78122088
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110898433.3A Active CN113537389B (en) | 2021-08-05 | 2021-08-05 | Robust image classification method and device based on model embedding |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113537389B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115618935B (en) * | 2022-12-21 | 2023-05-05 | 北京航空航天大学 | Robustness loss function searching method and system for classification task tag noise |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105335756A (en) * | 2015-10-30 | 2016-02-17 | 苏州大学 | Robust learning model and image classification system |
| CN108171261A (en) * | 2017-12-21 | 2018-06-15 | 苏州大学 | Adaptive semi-supervision image classification method, device, equipment and the medium of robust |
| CN108399421A (en) * | 2018-01-31 | 2018-08-14 | 南京邮电大学 | A kind of zero sample classification method of depth of word-based insertion |
| CN111488904A (en) * | 2020-03-03 | 2020-08-04 | 清华大学 | Image classification method and system based on confrontation distribution training |
| CN112766386A (en) * | 2021-01-25 | 2021-05-07 | 大连理工大学 | Generalized zero sample learning method based on multi-input multi-output fusion network |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111507362B (en) * | 2019-01-30 | 2023-12-19 | 中强光电股份有限公司 | Neural network training method, neural network-based classification method and device |
-
2021
- 2021-08-05 CN CN202110898433.3A patent/CN113537389B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105335756A (en) * | 2015-10-30 | 2016-02-17 | 苏州大学 | Robust learning model and image classification system |
| CN108171261A (en) * | 2017-12-21 | 2018-06-15 | 苏州大学 | Adaptive semi-supervision image classification method, device, equipment and the medium of robust |
| CN108399421A (en) * | 2018-01-31 | 2018-08-14 | 南京邮电大学 | A kind of zero sample classification method of depth of word-based insertion |
| CN111488904A (en) * | 2020-03-03 | 2020-08-04 | 清华大学 | Image classification method and system based on confrontation distribution training |
| CN112766386A (en) * | 2021-01-25 | 2021-05-07 | 大连理工大学 | Generalized zero sample learning method based on multi-input multi-output fusion network |
Non-Patent Citations (1)
| Title |
|---|
| 基于卷积神经网络的多标签图像自动标注;黎健成;袁春;宋友;;计算机科学(07);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113537389A (en) | 2021-10-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Iscen et al. | Label propagation for deep semi-supervised learning | |
| US11935243B2 (en) | Generative adversarial networks for image segmentation | |
| Goodfellow et al. | Generative adversarial nets | |
| US8325999B2 (en) | Assisted face recognition tagging | |
| WO2019100723A1 (en) | Method and device for training multi-label classification model | |
| CN109558942B (en) | Neural network migration method based on shallow learning | |
| CN110276406B (en) | Expression classification method, apparatus, computer device and storage medium | |
| CN114492574A (en) | Pseudo label loss unsupervised countermeasure domain adaptive picture classification method based on Gaussian uniform mixing model | |
| Ghadiyaram et al. | Blind image quality assessment on real distorted images using deep belief nets | |
| CN107945210B (en) | Target tracking method based on deep learning and environment self-adaption | |
| US11816882B2 (en) | Image recognition learning device, image recognition device, method and program | |
| CN112862093A (en) | Graph neural network training method and device | |
| CN113673482B (en) | Cell antinuclear antibody fluorescence recognition method and system based on dynamic label distribution | |
| Nguyen et al. | Robust learning under label noise with iterative noise-filtering | |
| CN113537389B (en) | Robust image classification method and device based on model embedding | |
| CN115731445A (en) | Learning method, information processing apparatus, and recording medium having learning program recorded thereon | |
| JP2020155101A (en) | Information processing device and information processing method | |
| JP2021051589A5 (en) | ||
| Bianchi et al. | Improving image classification robustness through selective cnn-filters fine-tuning | |
| CN116883751A (en) | Non-supervision field self-adaptive image recognition method based on prototype network contrast learning | |
| CN110717402B (en) | Pedestrian re-identification method based on hierarchical optimization metric learning | |
| CN116681128A (en) | Neural network model training method and device with noisy multi-label data | |
| US11328179B2 (en) | Information processing apparatus and information processing method | |
| CN114936890A (en) | Counter-fact fairness recommendation method based on inverse tendency weighting method | |
| CN115481215A (en) | Partner prediction method and prediction system based on temporal partner knowledge graph |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |