CN113569960A - Small sample image classification method and system based on domain adaptation - Google Patents
Small sample image classification method and system based on domain adaptation Download PDFInfo
- Publication number
- CN113569960A CN113569960A CN202110866395.3A CN202110866395A CN113569960A CN 113569960 A CN113569960 A CN 113569960A CN 202110866395 A CN202110866395 A CN 202110866395A CN 113569960 A CN113569960 A CN 113569960A
- Authority
- CN
- China
- Prior art keywords
- domain
- feature
- image
- small sample
- representing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 58
- 230000006978 adaptation Effects 0.000 title claims abstract description 22
- 238000012549 training Methods 0.000 claims abstract description 59
- 238000000605 extraction Methods 0.000 claims abstract description 57
- 238000012360 testing method Methods 0.000 claims abstract description 40
- 238000010801 machine learning Methods 0.000 claims abstract description 33
- 239000011159 matrix material Substances 0.000 claims description 32
- 230000006870 function Effects 0.000 claims description 23
- 230000008569 process Effects 0.000 claims description 11
- 230000004913 activation Effects 0.000 claims description 9
- 238000010586 diagram Methods 0.000 claims description 7
- 239000013598 vector Substances 0.000 claims description 5
- 230000001131 transforming effect Effects 0.000 claims 1
- 238000011160 research Methods 0.000 abstract description 6
- 238000013528 artificial neural network Methods 0.000 description 7
- 238000012986 modification Methods 0.000 description 5
- 230000004048 modification Effects 0.000 description 5
- 238000003062 neural network model Methods 0.000 description 4
- 238000012545 processing Methods 0.000 description 3
- 238000011426 transformation method Methods 0.000 description 3
- 238000005457 optimization Methods 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000011478 gradient descent method Methods 0.000 description 1
- 210000005036 nerve Anatomy 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/211—Selection of the most significant subset of features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Computational Linguistics (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to the field of image classification, and discloses a small sample image classification method and system based on domain adaptation, which comprises the steps of constructing a feature extractor and loading a pre-training parameter initialization model; obtaining a plurality of feature extractors by using a domain feature extraction module; obtaining a final feature extractor suitable for a target data domain by using a weight training module; the final performance of the method is obtained by using the test module, and the problem that the existing small sample machine learning method cannot really solve the small sample problem in the field of computer vision is solved. According to the invention, the small sample machine learning method does not need a strict meta-training set, and the source domain data and the target domain data can be dissimilar, so that the small sample machine learning method can be better applied to actual scenes, and has the advantage of expanding the application range of the current research.
Description
Technical Field
The invention relates to the field of image classification, in particular to a small sample image classification method and system based on domain adaptation.
Background
In the field of computer vision, training of a currently mainstream neural network model often requires a huge number of sample image data sets, and because machine learning requires a large amount of training data to fit data distribution of a target task, insufficient sample data amount can significantly affect the performance of the machine learning model. An image dataset typically includes tens of thousands of image samples, with each image category including hundreds of samples. In an actual application scenario, such sufficient sample data cannot be obtained generally, so that the problem of small sample image classification gradually becomes a research hotspot, and in order to solve the problem, a small sample machine learning method is provided.
The small sample machine learning method mainly aims to solve the problem that the number of sample data for model training is extremely insufficient (each class only comprises 1-5 samples), under the scene, the performance of a general neural network model and a machine learning method is extremely poor, and the small sample machine learning method can achieve better performance under the extreme scene by using a special neural network model and a training thought, so that the field gradually becomes the main direction of current research. Small sample machine learning methods typically artificially divide the available data sets into meta-training data sets and meta-testing data sets, and divide the training process into meta-training and meta-testing phases. In the meta-training phase, a sufficiently large annotated meta-training data set is used to train the method model; in the meta-test phase, a meta-test dataset containing a different class than the meta-training dataset is used to evaluate the ability of the method model to learn and classify these new classes. In order to simulate the actual application scene, 1-5 images of each category in the element test set are taken out by the small sample machine learning method to form a support set for method model learning, and the rest images form a query set to test the classification performance of the method model. At present, research aiming at a small sample machine learning method is mainly carried out from two directions of model optimization and meta-learning, wherein the model optimization direction is optimized aiming at a neural network model used by the small sample machine learning method, so that the method can adapt to the scene of a small sample, for example, graph convolution nerve is introduced and a confrontation network is generated to be used as a backbone network to achieve a better classification effect; the meta-learning direction is to introduce the thought of meta-learning into the learning method of the machine of the small sample, train a migratable meta-parameter (such as gradient, initial parameter of the model) in the meta-training stage, and then use these parameters to make the method model obtain better classification performance in the meta-testing stage.
When the small sample machine learning method is used, because the small sample machine learning method basically comprises a meta-training process, a method model needs to be trained on a meta-training set which has enough image sample data quantity and category number and is highly similar to a meta-testing data set, so strict requirements cause that the current small sample machine learning method usually only performs experiments on a plurality of image data sets specially used for classifying small sample images, and the data which can be obtained in an actual scene usually only has a plurality of categories and only a plurality of sample images of each category, the meta-training set and the meta-testing set cannot be divided, and a public data set similar to current task data cannot be easily obtained, so that the research on the small sample machine learning method is basically in a theoretical research stage and cannot be practically applied, namely the current small sample machine learning method cannot really solve the small sample problem in the field of computer vision, therefore, it is desired to improve the method.
Disclosure of Invention
Aiming at the defects in the prior art, the invention aims to provide a domain-adaptation-based small sample image classification method and system, wherein a domain adaptation method is creatively used in a small sample machine learning method, so that the small sample machine learning method does not need to require a strict meta-training set, but a plurality of feature extractors are trained on a source domain, and then a small sample machine learning method is used on a target domain support set to train and obtain the optimal weight for combining the feature extractors, so that a final feature extractor most suitable for a target domain is obtained, and the optimal performance is finally obtained by testing on a target domain query set.
In order to achieve the purpose, the technical scheme of the invention is as follows:
a small sample image classification method based on domain adaptation comprises the following steps:
constructing a feature extractor, and loading a pre-training parameter initialization model;
obtaining a plurality of feature extractors by using a domain feature extraction module;
obtaining a final feature extractor suitable for a target data domain by using a weight training module;
the test module is used to obtain the final performance of the process.
By adopting the technical scheme, the domain-adaptation-based small sample image classification method creatively uses the domain adaptation method in the small sample machine learning method by means of the three modules of the domain feature extraction module, the weight training module and the prediction module, specifically, after the domain feature extraction module is used for obtaining a plurality of feature extractors which are fully trained on a source domain and are inserted with the domain feature extraction module, the optimal weights of the feature extractors are trained and combined on a support set of a target domain by the weight training module, so that a final feature extractor which is most suitable for the target domain is obtained, and a final small sample image classification result is obtained by the prediction module to measure the performance of the domain-adaptation-based small sample image classification method.
The invention is further configured to: the domain feature extraction module transforms the feature map X output by the feature extractor to extract domain-specific global features of the input image using the following formula:
X1＝γ⊙X+β
wherein, X is a matrix with HW multiplied by C as a dimension, wherein H, W and C respectively represent the height, width and channel number of the matrix, and the domain feature extraction module takes X as input and processes the X; x1Is the transformed matrix; gamma and beta are trainable parameter matrixes, and the dimensionality of the matrix is the same as that of the X; an element indicates a product.
By adopting the technical scheme, the domain feature extraction module can transform the feature map X output by the feature extractor so as to better extract the domain-specific global features of the input image.
The invention is further configured to: the domain feature extraction module also performs local attention operation on the feature map X output by the feature extractor to extract domain-specific local features of the input image using the following formula:
X2＝PWConv2(ReLU(PWConv1(X)))
wherein, X2The matrix after the local attention operation is carried out; PWConv1Representing the first point convolution layer, PWConv2Representing a second point convolution layer; the ReLU is an activation function, when an input value m is less than or equal to 0, the ReLU (m) is 0, and when m is more than 0, the ReLU (m) is m; PWConv1(X) represents the result of the point convolution layer operation performed on the input feature map X by point convolution.
By adopting the technical scheme, the domain feature extraction module further performs local attention operation on the feature map X output by the feature extractor by using the following formula so as to better extract the domain-specific local features of the input image.
The invention is further configured to: the final output of the domain feature extraction module is expressed by the following formula:
wherein,
is the final output of the domain feature extraction module.
By adopting the technical scheme, X is obtained through local attention operation processing2The feature map has the same dimension as the input feature map X, fine details in the bottom-layer features can be reserved and highlighted, and domain-specific local features can be extracted, saved and migrated better. Because the dimension of the input feature diagram X in the transformation operation and the local attention operation is not changed, the two parts of output can be directly added, namely the domain-specific local feature and the domain-specific global feature of the input image are simultaneously extracted, the capability of the feature extractor for extracting the domain-specific feature is greatly improved, so that more feature information is migrated from the source domain, and the domain-specific local feature is a small sample local machine learning party in the target domainThe method provides help, so that the final small sample image classification performance is improved.
The invention is further configured to: the weight training module obtains a weight λ using the following formula:
wherein λ is a weight for combining the plurality of feature extractors, λ is a vector having one dimension of 1 × N; n is the number of feature extractors containing domain-specific features (including domain-specific global features and domain-specific local features); s is a support set in the target domain, and the image label pair in the support set s is (x)i，yi) Wherein x isiRepresenting the ith image sample, yiIndicates its corresponding category label; j represents the image class in the support set s, and N is the totalsClass nsAn image sample, sjA set of image sample indices representing image class labels equal to j, p being a prototype, f (-) representing a feature extractor, d (-) representing a distance function for estimating the similarity between different parameters; l (λ) represents the loss function on the support set with respect to λ.
By adopting the technical scheme, the weight training module trains the weights of the plurality of feature extractors obtained by the combined domain feature extraction module by using the target domain support set data, and the final feature extractor most suitable for the target domain is obtained by combining the optimal weight obtained by training.
The invention is further configured to: the test module obtains a predicted result of the test sample using the following formula:
wherein x iskRepresents the k-th image sample, yk represents its pairThe corresponding category label is used to identify the category,
representing the prediction result for the test sample, j represents the image class in the support set s, and N is totalsClass nsAn image sample, sjA set of image sample indices representing image class labels equal to j, p being a prototype, f (-) representing a feature extractor, d (-) representing a distance function for estimating the similarity between different parameters.
By adopting the technical scheme, the test module can predict the class label of the target domain query set sample data so as to test the final performance of the method.
The invention is further configured to: the method also comprises the following steps after the characteristic extractor is constructed and the pre-training parameter initialization model is loaded:
the domain feature extraction module trains a plurality of feature extractors on a source domain;
the weight training module trains all the feature extractors on a target domain support set by using a small sample machine learning method to obtain weights for combining all the feature extractors, and combines all the feature extractors according to the weights to obtain a final feature extractor suitable for a target domain;
the test module tests final image classification performance on the target domain query set using the final feature extractor.
By adopting the technical scheme, the plurality of feature extractors are trained on the source domain, the model parameters of the feature extractors are stored and carry the domain specific features of the domain, the optimal weight for combining the plurality of feature extractors is obtained by training the support set data of the target domain in combination with a small sample machine learning method, all the feature extractors are combined according to the optimal weight to obtain the final feature extractor capable of obtaining the optimal performance on the target domain, and finally, the final feature extractor can be used for testing the final image classification performance on the target domain query set.
The invention is further configured to: each data field in the source domain trains a feature extractor.
By adopting the technical scheme, the model parameter of one feature extractor trained on each data domain is stored and carries the domain specific features of the domain, so that all the feature extractors can extract the domain specific features of all the data domains in the source domain, the utilization rate of the source domain is improved, and the final image classification performance is also improved.
The invention is further configured to: wherein the model structures of the plurality of feature extractors trained on the source domain are the same.
By adopting the technical scheme, the model structures of the plurality of feature extractors are the same, so that the subsequent combination is facilitated, and the operation difficulty of the combined feature extractor is reduced.
The invention also provides a small sample image classification system based on domain adaptation, which comprises a domain feature extraction module, a weight training module and a test module, wherein the domain feature extraction module transforms the feature map X of the input image by using the following formula to extract the domain-specific global features of the input image:
X1＝γ⊙X+β
wherein, X is a matrix with HW multiplied by C as a dimension, wherein H, W and C respectively represent the height, width and channel number of the matrix, and the domain feature extraction module takes X as input and processes the X; x1Is the transformed matrix; gamma and beta are trainable parameter matrixes, and the dimensionality of the matrix is the same as that of the X; an element indicates a product;
the domain feature extraction module further performs a local attention operation on the feature map X of the input image to extract domain-specific local features of the input image using the following formula:
X2＝PWConv2(ReLU(PWConv1(X)))
wherein, X2The matrix after the local attention operation is carried out; PWConv1Representing the first point convolution layer, PWConv2Representing a second point convolution layer; the ReLU is an activation function, when an input value m is less than or equal to 0, the ReLU (m) is 0, and when m is more than 0, the ReLU (m) is m; PWConv1(X) representing the result of point convolution operation of the input characteristic diagram X after point convolution;
the final output of the domain feature extraction module is expressed by the following formula:
wherein,
is the final output of the domain feature extraction module.
By adopting the technical scheme, the domain feature extraction module can be conveniently embedded into an original neural network, and the domain specific local features and the domain specific global features of the input image can be simultaneously extracted by combining the local attention and the feature map transformation method, so that the capability of the feature extractor for extracting the domain specific features is greatly improved, more feature information is migrated from a source domain, and the final classification performance of the small sample image is finally improved.
In summary, the invention has the following beneficial effects:
(1) the domain-adaptation-based small sample image classification method is characterized in that a domain adaptation method is creatively used in a small sample machine learning method by virtue of a domain feature extraction module, a weight training module and a prediction module, after the domain feature extraction module is used for obtaining a plurality of feature extractors, the weight training module obtains the optimal weight for combining the feature extractors and obtains a final feature extractor which is most suitable for a target domain, and the prediction module obtains a final small sample image classification result to measure the performance of the domain-adaptation-based small sample image classification method;
(2) training a plurality of feature extractors on a source domain, storing model parameters of the feature extractors and carrying domain specific features of the domain, training in a support set data of a target domain by combining a small sample local machine learning method to obtain optimal weights for combining the plurality of feature extractors, combining all the feature extractors according to the optimal weights to obtain a final feature extractor capable of obtaining optimal performance on the target domain, and finally testing the final image classification performance on a target domain query set by using the final feature extractor;
(3) the domain feature extraction module can be conveniently embedded into an original neural network, local attention and a feature map transformation method are combined, domain-specific local features and domain-specific global features of an input image can be simultaneously extracted, the capability of the feature extractor for extracting the domain-specific global features is greatly improved, therefore, more feature information is migrated from a source domain, and finally the classification performance of a final small sample image is improved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments described in the present invention, and those skilled in the art can also obtain other drawings according to the drawings.
FIG. 1 is a flowchart illustrating a small sample image classification method based on domain adaptation according to an embodiment;
FIG. 2 is a diagram of a key model architecture of the present invention;
FIG. 3 is a block diagram of a neural network building block inserted into a domain feature extraction module;
fig. 4 is a flowchart illustrating a small sample image classification method based on domain adaptation according to a second embodiment.
Detailed Description
The technical solutions in the embodiments of the present invention are clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example one
As shown in fig. 1, a method for classifying small sample images based on domain adaptation includes:
s101: constructing a feature extractor, and loading a pre-training parameter initialization model;
as shown in FIG. 2, the key model structure of the present invention, in this embodiment, is describedThe Resnet-18 convolutional neural network (i.e. Resnet model with 18 layers of neural network layers) is used as a backbone network, and the neural network is composed of 4 basic blocks and a classification layer. The basic block structure is shown in fig. 3, where a represents two domain feature extraction modules inserted on each basic block in the present invention, 3 × 3 represents a convolution layer with a convolution kernel of 3 × 3, BN represents a batch normalization layer, ReLU represents an activation function,
it is indicated that the multiplication of the elements,
representing the addition of elements.
Setting the source domain as R ═ R1，r2，…，rN}，riThe number of the image data fields is N;
let the target domain be T ═ s ≦ q, where s represents the support set for small sample machine learning method training, q represents the query set for performance testing, and
in the present embodiment, training is performed using, as source domains, a CUB dataset (i.e., an image dataset containing 200 different birds), an Aircraft dataset (i.e., an image dataset containing 100 airplane classes), and a VGG-Flower dataset (i.e., an image dataset containing 102 different Flower classes), which are each set as r1，r2，r3And randomly taking 5 images from each type of a CIFAR-10 data set (namely an image data set containing 10 types of common articles) to form a support set s, and forming a query set q by using the residual image samples as a target domain to train and test the small sample machine learning method. The classification network consists of a pooling layer, a linear classification layer and a softmax (normalized exponential function) layer and is used for outputting a final prediction result.
A neural network serving as a feature extractor is constructed according to a key model structure in the attached figure 2, a pre-training parameter initialization model is loaded, gamma parameters are initialized to be a full 1 matrix, beta parameters are initialized to be a full 0 matrix, and a random parameter is used for initializing a classification network.
S102: obtaining a plurality of feature extractors by using a domain feature extraction module;
the domain feature extraction module transforms the feature map X output by the feature extractor to better extract domain-specific global features of the input image using the following formula:
X1＝γ⊙X+β
wherein, X is a matrix with HW multiplied by C as a dimension, wherein H, W and C respectively represent the height, width and channel number of the matrix, and the domain feature extraction module takes X as input and processes the X; x1Is the transformed matrix; gamma and beta are trainable parameter matrixes, and the dimensionality of the matrix is the same as that of the X; an element indicates a product.
Meanwhile, in order to better extract local characteristics specific to a domain, the invention also introduces local attention operation, which is composed of two point-wise convolution (point-wise convolution) layers and a ReLU activation function. The point convolution is a special convolution operation, which can realize the fusion of cross-channel information, increase the nonlinearity of the network, and realize the dimension increasing and dimension reducing of the number of channels, and is realized as a point convolution layer in the embodiment. ReLU is an activation function, and when an input value m is less than or equal to 0, ReLU (m) is 0, and when m is greater than 0, ReLU (m) is m. Therefore, the domain feature extraction module performs a local attention operation on the feature map X output by the feature extractor to better extract domain-specific local features of the input image using the following formula:
X2＝PWConv2(ReLU(PWConv1(X)))
wherein, X2The matrix after the local attention operation is carried out; PWConv1Representing the first point convolution layer, PWConv2Representing a second point convolution layer; the ReLU is an activation function, when an input value m is less than or equal to 0, the ReLU (m) is 0, and when m is more than 0, the ReLU (m) is m; PWConv1(X) represents the result of the point convolution layer operation performed on the input feature map X by point convolution.
Finally, the output of the domain feature extraction module is expressed by the following formula:
wherein,
is the final output of the domain feature extraction module. X obtained through local attention operation processing2The feature map has the same dimension as the input feature map X, fine details in the bottom-layer features can be reserved and highlighted, and domain-specific local features can be extracted, saved and migrated better. Because the dimension of the input feature diagram X is not changed in the transformation operation and the local attention operation, the two parts of output can be directly added, namely the domain-specific local feature and the domain-specific global feature of the input image are simultaneously extracted, so that the capability of the feature extractor for extracting the domain-specific feature is greatly improved, more feature information is migrated from the source domain, help is provided for the learning method of the machine of the small sample in the target domain, and the classification performance of the final small sample image is improved.
Respectively in the source domain data field r by using the above-mentioned feature extractor1，r2，r3The model parameters obtained by the training are respectively stored and the feature extractors obtained by the training are respectively set as f1(·)，f2(·)，f3(. The) as an output, so that it is sufficient to extract domain-specific feature information of each domain, and these feature information are stored in the model parameters.
S103: obtaining a final feature extractor suitable for a target data domain by using a weight training module;
the weight training module obtains a weight λ using the following equation:
wherein λ is the weight of combining the plurality of feature extractors, λ is one dimensionA vector of degree 1 × N; n is the number of feature extractors containing domain-specific features (including domain-specific global features and domain-specific local features); s is a support set in the target domain, and the image label pair in the support set s is (x)i，yi) Wherein x isiRepresenting the ith image sample, yiIndicates its corresponding category label; j represents the image class in the support set s, and N is the totalsClass nsAn image sample, sjRepresenting an image sample subscript set with an image class label equal to j, p being a prototype, f (-) representing a feature extractor, d (-) representing a distance function for estimating similarity between different parameters, wherein the practical application can be selected from various options, such as Euclidean distance, Manhattan distance and the like; l (λ) represents the loss function on the support set with respect to λ.
By using the loss function shown in the formula, the optimal combination weight lambda can be obtained by training on the support set by using methods such as gradient descent and the like, and then the final feature extractor f most suitable for the target domain is obtainedλ(·)。
Specifically, in the step of using the weight training module to obtain the final feature extractor suitable for the target data domain, the method further comprises the following substeps:
s1031: output feature extractor f accepting multi-domain feature extraction module1(·)，f2(·)，f3(. The) as input, the combination weight parameters λ of these feature extractors are initialized to all 1 vectors, and the initial f is obtained from the following equationλ(·)：
Where x denotes the target field image, fλ(x) Representing the features extracted by the feature extractor for x, the other settings in the formula are the same as the formula in step S103.
S1032: using fλ(. The) processing the image sample on the support set s to obtain the prototype of each type of the initial support set by the following formula
The setting in the formula is the same as the formula in step S103.
S1033: traversing all image samples on the support set s by using a loss function shown by the following formula to obtain the training loss of the current lambda, and training the lambda by using a random gradient descent method:
wherein λ is a weight for combining the plurality of feature extractors, λ is a vector having one dimension of 1 × N; n is the number of feature extractors containing domain-specific features (including domain-specific global features and domain-specific local features); s is a support set in the target domain, and the image label pair in the support set s is (x)i，yi) Wherein x isiRepresenting the ith image sample, yiIndicates its corresponding category label; j represents the image class in the support set s, and N is the totalsClass nsAn image sample, sjRepresenting an image sample subscript set with an image class label equal to j, p being a prototype, f (-) representing a feature extractor, d (-) representing a distance function for estimating similarity between different parameters, wherein the practical application can be selected from various options, such as Euclidean distance, Manhattan distance and the like; l (λ) represents the loss function on the support set with respect to λ.
S1034: repeating the step S1033 until the training loss of the lambda is not reduced any more, and finally obtaining the optimal weight parameter lambda for combining a plurality of feature extractors, namely obtaining the final feature extractor f most suitable for the target data fieldλ(·)。
S104: the test module is used to obtain the final performance of the process.
The test module obtains a predicted result of the test sample using the following formula:
wherein x iskRepresenting the kth image sample, yk its corresponding class label,
representing the prediction result for the test sample, j represents the image class in the support set s, and N is totalsClass nsAn image sample, sjThe image sample index set which represents the image class label is equal to j, p is a prototype, f (-) represents a feature extractor, d (-) represents a distance function and is used for estimating the similarity between different parameters, and various choices such as Euclidean distance, Manhattan distance and the like can be provided in practical application.
Final feature extractor f for obtaining the most suitable target data field by weight training moduleλAfter (-) the query set q is traversed using the test module to get the final performance of the classification method.
The domain-adaptation-based small sample image classification method in the embodiment creatively uses a domain adaptation method in the small sample machine learning method by means of a domain feature extraction module, a weight training module and a prediction module, and specifically, after a plurality of feature extractors which are fully trained on a source domain and inserted with the domain feature extraction module are obtained by using the domain feature extraction module, the optimal weights of the feature extractors are trained and combined on a support set of a target domain by the weight training module, so that a final feature extractor which is most suitable for the target domain is obtained, and a final small sample image classification result is obtained by the prediction module to measure the performance of the domain-adaptation-based small sample image classification method.
The embodiment also provides a small sample image classification system based on domain adaptation, which comprises a domain feature extraction module, a weight training module and a test module, wherein the domain feature extraction module transforms a feature map X of an input image by using the following formula to extract domain-specific global features of the input image:
X1＝γ⊙X+β
wherein, X is a matrix with HW multiplied by C as a dimension, wherein H, W and C respectively represent the height, width and channel number of the matrix, and the domain feature extraction module takes X as input and processes the X; x1Is the transformed matrix; gamma and beta are trainable parameter matrixes, and the dimensionality of the matrix is the same as that of the X; an element indicates a product;
the domain feature extraction module further performs a local attention operation on the feature map X of the input image to extract domain-specific local features of the input image using the following formula:
X2＝PWConv2(ReLU(PWConv1(X)))
wherein, X2The matrix after the local attention operation is carried out; PWConv1Representing the first point convolution layer, PWConv2Representing a second point convolution layer; the ReLU is an activation function, when an input value m is less than or equal to 0, the ReLU (m) is 0, and when m is more than 0, the ReLU (m) is m; PWConv1(X) representing the result of point convolution operation of the input characteristic diagram X after point convolution;
the final output of the domain feature extraction module is expressed by the following formula:
wherein,
is the final output of the domain feature extraction module.
The domain feature extraction module can be conveniently embedded into an original neural network, and can simultaneously extract domain-specific local features and domain-specific global features of an input image by combining a local attention and a feature map transformation method, so that the capability of the feature extractor for extracting the domain-specific global features is greatly improved, more feature information is migrated from a source domain, and the final classification performance of the small sample image is finally improved.
Example two
As shown in fig. 4, for the domain-adaptive small sample image classification method disclosed in the present invention, different from the first embodiment, the method further includes the following steps after the feature extractor is constructed and the pre-training parameter initialization model is loaded:
s201: the domain feature extraction module trains a plurality of feature extractors on a source domain;
s202: the weight training module trains all the feature extractors on a target domain support set by using a small sample machine learning method to obtain weights for combining all the feature extractors, and combines all the feature extractors according to the weights to obtain a final feature extractor suitable for a target domain;
s203: the test module tests final image classification performance on the target domain query set using the final feature extractor.
In step S201, each data field in the source domain trains a feature extractor.
The model parameters of one feature extractor trained on each data field are stored and carry the field-specific features of the field, so that all the feature extractors can extract the field-specific features of all the data fields in the source field, the utilization rate of the source field is improved, and the final image classification performance is also improved.
Furthermore, the model structures of the plurality of feature extractors trained on the source domain are the same, so that the combination of the plurality of feature extractors is convenient, and the operation difficulty of the combined feature extractor is reduced.
In the embodiment, a plurality of feature extractors are trained on a source domain, model parameters of the feature extractors are stored and carry domain specific features of the domain, optimal weights for combining the plurality of feature extractors are obtained by training support set data of a target domain in combination with a small sample local machine learning method, all the feature extractors are combined according to the optimal weights to obtain a final feature extractor capable of obtaining optimal performance on the target domain, and finally, the final feature extractor can be used for testing the final image classification performance on a target domain query set.
While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all such alterations and modifications as fall within the scope of the invention. It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.
Claims (10)
1. A small sample image classification method based on domain adaptation is characterized by comprising the following steps:
constructing a feature extractor, and loading a pre-training parameter initialization model;
obtaining a plurality of feature extractors by using a domain feature extraction module;
obtaining a final feature extractor suitable for a target data domain by using a weight training module;
the test module is used to obtain the final performance of the process.
2. The method for classifying small sample images based on domain adaptation as claimed in claim 1, wherein the domain feature extraction module transforms the feature map X output by the feature extractor to extract domain-specific global features of the input image using the following formula:
X1＝γ⊙X+β
wherein, X is a matrix with HW multiplied by C as a dimension, wherein H, W and C respectively represent the height, width and channel number of the matrix, and the domain feature extraction module takes X as input and processes the X; x1Is the transformed matrix; gamma and beta are trainable parameter matrixes, and the dimensionality of the matrix is the same as that of the X; an element indicates a product.
3. The method for classifying small sample images based on domain adaptation as claimed in claim 2, wherein the domain feature extraction module further performs local attention operation on the feature map X output by the feature extractor to extract domain-specific local features of the input image by using the following formula:
X2＝PWConv2(ReLU(PWConv1(X)))
wherein, X2The matrix after the local attention operation is carried out; PWConv1Representing the first point convolution layer, PWConv2Representing a second point convolution layer; the ReLU is an activation function, when an input value m is less than or equal to 0, the ReLU (m) is 0, and when m is more than 0, the ReLU (m) is m; PWConv1(X) represents the result of the point convolution layer operation performed on the input feature map X by point convolution.
4. The method for classifying small sample images based on domain adaptation according to claim 3, wherein the final output of the domain feature extraction module is expressed by the following formula:
wherein,
is the final output of the domain feature extraction module.
5. The method for classifying small sample images based on domain adaptation according to claim 1, wherein the weight training module obtains the weight λ by using the following formula:
wherein λ is a groupCombining the weights of the plurality of feature extractors, wherein lambda is a vector with the dimension of 1 multiplied by N; n is the number of feature extractors containing domain-specific features; s is a support set in the target domain, and the image label pair in the support set s is (x)i，yi) Wherein x isiRepresenting the ith image sample, yiIndicates its corresponding category label; j represents the image class in the support set s, and N is the totalsClass nsAn image sample, sjA set of image sample indices representing image class labels equal to j, p being a prototype, f (-) representing a feature extractor, d (-) representing a distance function used to estimate the similarity between different parameters; l (λ) represents the loss function on the support set with respect to λ.
6. The method for classifying small sample images based on domain adaptation according to claim 1, wherein the testing module obtains the predicted result of the testing sample by using the following formula:
wherein x iskRepresenting the kth image sample, yk its corresponding class label,
representing the prediction result for the test sample, j represents the image class in the support set s, and N is totalsClass nsAn image sample, sjA set of image sample indices representing image class labels equal to j, p being a prototype, f (-) representing a feature extractor, d (-) representing a distance function used to estimate the similarity between different parameters.
7. The method for classifying small sample images based on domain adaptation as claimed in claim 1, further comprising the following steps after constructing the feature extractor and loading the pre-training parameter initialization model:
the domain feature extraction module trains a plurality of feature extractors on a source domain;
the weight training module trains all the feature extractors on a target domain support set by using a small sample machine learning method to obtain weights for combining all the feature extractors, and combines all the feature extractors according to the weights to obtain a final feature extractor suitable for a target domain;
the test module tests final image classification performance on the target domain query set using the final feature extractor.
8. The method of claim 6, wherein each data field in the source domain trains a feature extractor.
9. The method of claim 6, wherein the models of the plurality of feature extractors trained on the source domain are identical.
10. The small sample image classification system based on the domain adaptation is characterized by comprising a domain feature extraction module, a weight training module and a testing module, wherein the domain feature extraction module is used for transforming a feature map X of an input image to extract domain-specific global features of the input image by using the following formula:
X1＝γ⊙X+β
wherein, X is a matrix with HW multiplied by C as a dimension, wherein H, W and C respectively represent the height, width and channel number of the matrix, and the domain feature extraction module takes X as input and processes the X; x1Is the transformed matrix; gamma and beta are trainable parameter matrixes, and the dimensionality of the matrix is the same as that of the X; an element indicates a product;
the domain feature extraction module further performs a local attention operation on the feature map X of the input image to extract domain-specific local features of the input image using the following formula:
X2＝PWConv2(ReLU(PWConv1(X)))
wherein, X2The matrix after the local attention operation is carried out; PWConv1Representing the first point convolution layer, PWConv2Representing a second point convolution layer; the ReLU is an activation function, when an input value m is less than or equal to 0, the ReLU (m) is 0, and when m is more than 0, the ReLU (m) is m; PWConv1(X) representing the result of point convolution operation of the input characteristic diagram X after point convolution;
the final output of the domain feature extraction module is expressed by the following formula:
wherein,
is the final output of the domain feature extraction module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110866395.3A CN113569960B (en) | 2021-07-29 | 2021-07-29 | Small sample image classification method and system based on domain adaptation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110866395.3A CN113569960B (en) | 2021-07-29 | 2021-07-29 | Small sample image classification method and system based on domain adaptation |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113569960A true CN113569960A (en) | 2021-10-29 |
| CN113569960B CN113569960B (en) | 2023-12-26 |
Family
ID=78169151
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110866395.3A Active CN113569960B (en) | 2021-07-29 | 2021-07-29 | Small sample image classification method and system based on domain adaptation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113569960B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117150282A (en) * | 2023-09-16 | 2023-12-01 | 石家庄正和网络有限公司 | Secondhand equipment recycling evaluation method and system based on prediction model |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190012526A1 (en) * | 2017-07-04 | 2019-01-10 | Microsoft Technology Licensing, Llc | Image recognition with promotion of underrepresented classes |
| CN109961089A (en) * | 2019-02-26 | 2019-07-02 | 中山大学 | Small sample and zero sample image classification method based on metric learning and meta learning |
| CN111738301A (en) * | 2020-05-28 | 2020-10-02 | 华南理工大学 | Long-tail distribution image data identification method based on two-channel learning |
| US20210003700A1 (en) * | 2019-07-02 | 2021-01-07 | Wuyi University | Method and apparatus for enhancing semantic features of sar image oriented small set of samples |
| CN112419321A (en) * | 2021-01-25 | 2021-02-26 | 长沙理工大学 | X-ray image identification method and device, computer equipment and storage medium |
| CN112668494A (en) * | 2020-12-31 | 2021-04-16 | 西安电子科技大学 | Small sample change detection method based on multi-scale feature extraction |
| CN112784764A (en) * | 2021-01-27 | 2021-05-11 | 南京邮电大学 | Expression recognition method and system based on local and global attention mechanism |
| CN112784879A (en) * | 2020-12-31 | 2021-05-11 | 前线智能科技(南京)有限公司 | Medical image segmentation or classification method based on small sample domain self-adaption |
| CN112801146A (en) * | 2021-01-13 | 2021-05-14 | 华中科技大学 | Target detection method and system |
| CN112990097A (en) * | 2021-04-13 | 2021-06-18 | 电子科技大学 | Face expression recognition method based on countermeasure elimination |
-
2021
- 2021-07-29 CN CN202110866395.3A patent/CN113569960B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190012526A1 (en) * | 2017-07-04 | 2019-01-10 | Microsoft Technology Licensing, Llc | Image recognition with promotion of underrepresented classes |
| CN109961089A (en) * | 2019-02-26 | 2019-07-02 | 中山大学 | Small sample and zero sample image classification method based on metric learning and meta learning |
| US20210003700A1 (en) * | 2019-07-02 | 2021-01-07 | Wuyi University | Method and apparatus for enhancing semantic features of sar image oriented small set of samples |
| CN111738301A (en) * | 2020-05-28 | 2020-10-02 | 华南理工大学 | Long-tail distribution image data identification method based on two-channel learning |
| CN112668494A (en) * | 2020-12-31 | 2021-04-16 | 西安电子科技大学 | Small sample change detection method based on multi-scale feature extraction |
| CN112784879A (en) * | 2020-12-31 | 2021-05-11 | 前线智能科技(南京)有限公司 | Medical image segmentation or classification method based on small sample domain self-adaption |
| CN112801146A (en) * | 2021-01-13 | 2021-05-14 | 华中科技大学 | Target detection method and system |
| CN112419321A (en) * | 2021-01-25 | 2021-02-26 | 长沙理工大学 | X-ray image identification method and device, computer equipment and storage medium |
| CN112784764A (en) * | 2021-01-27 | 2021-05-11 | 南京邮电大学 | Expression recognition method and system based on local and global attention mechanism |
| CN112990097A (en) * | 2021-04-13 | 2021-06-18 | 电子科技大学 | Face expression recognition method based on countermeasure elimination |
Non-Patent Citations (2)
| Title |
|---|
| 李新叶;龙慎鹏;朱婧;: "基于深度神经网络的少样本学习综述", 计算机应用研究, no. 08 * |
| 段萌;王功鹏;牛常勇;: "基于卷积神经网络的小样本图像识别方法", 计算机工程与设计, no. 01 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117150282A (en) * | 2023-09-16 | 2023-12-01 | 石家庄正和网络有限公司 | Secondhand equipment recycling evaluation method and system based on prediction model |
| CN117150282B (en) * | 2023-09-16 | 2024-01-30 | 石家庄正和网络有限公司 | Secondhand equipment recycling evaluation method and system based on prediction model |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113569960B (en) | 2023-12-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111291183B (en) | Method and device for carrying out classification prediction by using text classification model | |
| CN110163234B (en) | Model training method and device and storage medium | |
| CN107358293B (en) | Neural network training method and device | |
| WO2019228122A1 (en) | Training method for model, storage medium and computer device | |
| CN108960330B (en) | Remote sensing image semantic generation method based on fast regional convolutional neural network | |
| CN110503192A (en) | The effective neural framework of resource | |
| CN108804641A (en) | A kind of computational methods of text similarity, device, equipment and storage medium | |
| CN109508717A (en) | A kind of licence plate recognition method, identification device, identification equipment and readable storage medium storing program for executing | |
| CN113065013B (en) | Image annotation model training and image annotation method, system, equipment and medium | |
| CN112231516B (en) | Training method of video abstract generation model, video abstract generation method and device | |
| CN111709493B (en) | Object classification method, training device, object classification equipment and storage medium | |
| CN111723914A (en) | Neural network architecture searching method based on convolution kernel prediction | |
| CN116580257A (en) | Feature fusion model training and sample retrieval method and device and computer equipment | |
| CN113128478B (en) | Model training method, pedestrian analysis method, device, equipment and storage medium | |
| CN114565807A (en) | Method and device for training target image retrieval model | |
| Chen et al. | SA-SinGAN: self-attention for single-image generation adversarial networks | |
| CN112989843B (en) | Intention recognition method, device, computing equipment and storage medium | |
| CN113569960B (en) | Small sample image classification method and system based on domain adaptation | |
| CN114155388A (en) | Image recognition method and device, computer equipment and storage medium | |
| CN114003706A (en) | Keyword combination generation model training method and device | |
| Wang et al. | Towards efficient convolutional neural networks through low-error filter saliency estimation | |
| CN112733724A (en) | Relativity relationship verification method and device based on discrimination sample meta-digger | |
| CN116957006A (en) | Training method, device, equipment, medium and program product of prediction model | |
| CN110826726B (en) | Target processing method, target processing device, target processing apparatus, and medium | |
| Mao et al. | Differentiable light-weight architecture search |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |