CN112132004A - A fine-grained image recognition method based on multi-view feature fusion - Google Patents

Abstract

A fine-grained image recognition method based on multi-view feature fusion, relates to the technical field of image processing, and solves the problem that existing fine-grained image recognition methods ignore the detailed information of images and have poor adaptability to visual differences between images, and the introduced The loss function is complex, which increases the number of parameters of the model, including the bilinear feature extraction step, the suppression branch learning step, the similar comparison module learning step, the center loss calculation step and the model optimization loss function calculation step; the invention introduces the suppression branch, By suppressing the most salient regions in the image, the network is forced to look for subtle discriminative features between confusing classes. A similar comparison learning module is introduced to fuse the feature vectors of similar samples to increase the interactive information of different images under the same category. A center loss function is also introduced to minimize the distance between features and corresponding class centers, making the learned features more discriminative. Improves the accuracy of fine-grained image recognition.

Description

A fine-grained image recognition method based on multi-view feature fusion

technical field

The invention relates to the technical field of image processing, in particular to a fine-grained image recognition method based on multi-view feature fusion.

Background technique

Fine-grained image classification is based on distinguishing basic categories, and then performs more fine-grained sub-categories, such as birds or dogs. Therefore, this problem needs to capture subtle inter-class differences and fully exploit the discriminative features of images.

Fine-grained objects exist widely in real life, and the corresponding fine-grained image recognition is an important research topic in computer vision recognition. The current fine-grained image recognition mainly has the following three challenges: (1) Due to the different poses, backgrounds and shooting angles, the appearance of the same category may appear to be quite different. (2) Since different categories belong to the same parent category, the differences between them only exist in some subtle areas, such as the beak and tail of birds. (3) The collection and labeling of fine-grained images is time-consuming and labor-intensive. As shown in Figure 1.

The existing methods mainly achieve the purpose of recognition through the following three aspects: (1) Fine-grained image recognition based on localization-classification network. (2) Directly learn more discriminative representations by developing powerful deep models for fine-grained recognition. (3) Combine the global features and local features of images to achieve fine-grained classification of images.

Prior art 1, bilinear pooling fine-grained image classification (Bilinear pooling), extracts features through pre-trained twin convolutional neural networks (convolutional neural networks), and performs bilinear pooling at each channel level of the features, A high-order representation of the feature is obtained, thereby enhancing the discriminative ability of the feature. This method benefits from a new pooling method, which improves the accuracy of fine-grained image recognition.

This method proposes a new bilinear pooling method, but there is no effective design for fine-grained image recognition in terms of the relationship between fine-grained image categories, the amount of model parameters, and the number of detail regions. That is, it does not take into account the influence of factors such as fine-grained images containing a variety of detailed information, small differences between classes, and large differences within classes.

Prior art 2, Multi-Attention Multi-Class Constraint, this method extracts multiple attention regions of the input image through a one-squeeze multi-excitation module, Then metric learning is introduced, triplet loss and softmax loss are used to train the network, the same attention of the same features is drawn closer, and the features of different attention or different classes are pushed apart, thereby strengthening the relationship between components and realizing Improved accuracy of fine-grained image recognition.

This method mainly utilizes metric learning to improve the sample distribution in the feature space, so it is less adaptable for mining visual differences between a pair of images. Moreover, the introduced loss function is relatively complex, and a large number of sample pairs need to be constructed, which greatly increases the number of parameters of the model.

SUMMARY OF THE INVENTION

In order to solve the problems that the existing fine-grained image recognition method ignores the detailed information of the image and the visual difference between the images has poor adaptability, the introduced loss function is complex, and the parameter quantity of the model is increased, the present invention provides a method based on A fine-grained image recognition method based on multi-view feature fusion.

A fine-grained image recognition method based on multi-view feature fusion, the method is realized by the following steps:

Step 1. Bilinear feature extraction;

The original image is input into the bilinear feature extraction network, and the feature maps output by different convolutional layers are fused to obtain bilinear feature vectors; the feature extraction network adopts the network structure pre-trained in the dataset ImageNet;

Step 2: Suppress branch learning, the specific process is:

Step 21: Generate an attention map according to the size and threshold of the feature maps output by different convolutional layers of the feature extraction network described in step 1;

Step 22, generating a suppression mask according to the attention map described in step 21, and overlaying the suppression mask on the original image to generate a suppressed image in which the local area is masked;

Step 2 and 3: Use step 1 to perform bilinear feature extraction on the suppressed image described in step 2, obtain bilinear feature vector, input the bilinear feature vector to the fully connected layer, and obtain the predicted class probability value, And calculate the multi-class cross entropy for the predicted class probability value;

Step 3. Similar comparison module learning;

Step 31: Randomly select other N images in the same category as the original image as positive sample images;

Step 32: The target image and the positive sample image described in step 31 are sent to the feature extraction network described in step 1 for bilinear feature vector fusion, and the fusion feature is obtained. Feature vector;

Step 33: Average the bilinear eigenvectors of multiple images of the same category obtained in step 32 to obtain a fused feature vector, and input the fused feature vector into the fully connected layer to obtain the predicted probability. Calculate the multi-class cross entropy of the predicted probability of the same category;

Step 4: Calculate the central loss function _LC ;

Let v _i be the bilinear feature of the ith sample, c _i be the average feature of all samples in the corresponding category of sample i, that is, the class center, and N be the number of samples in the current batch, then the formula of the center loss function L _C is as follows:

Step 5. Model optimization loss function calculation;

The cross-entropy loss function of the bilinear feature vector of the original image, the cross-entropy loss function of the bilinear feature vector of the suppressed image, the cross-entropy loss function of the fusion feature and the central loss function are weighted and summed to obtain the loss function of the model optimization. .

Beneficial effects of the present invention: The present invention comprehensively considers factors such as large intra-class differences, small inter-class differences, and large background noise in fine-grained images, and introduces a suppression branch, which suppresses the most significant area in the image, thereby forcing the network to find easy-to-use Confuse subtle discriminative features between classes. A class comparison learning module is also introduced to fuse the feature vectors of samples of the same class, thereby increasing the interactive information of different images under the same class. At the same time, a center loss function is introduced to minimize the distance between the feature and the corresponding class center, making the learned features more discriminative.

Based on the above points, the present invention comprehensively utilizes global features and local features in the judgment process, and achieves obvious performance improvement in multiple fine-grained image classification tasks, which is more robust and easy to practice compared with existing methods. deploy. Improves the accuracy of fine-grained image recognition.

Description of drawings

Fig. 1 is a schematic diagram of four existing groups of fine-grained images in Fig. 1a, Fig. 1b, Fig. 1c and Fig. 1d;

2 is a schematic diagram of bilinear feature extraction in a fine-grained image recognition method based on multi-view feature fusion according to the present invention;

FIG. 3 is a schematic diagram of similar comparison learning in a fine-grained image recognition method based on multi-view feature fusion according to the present invention;

4 is a schematic diagram of model optimization loss function calculation in a fine-grained image recognition method based on multi-view feature fusion according to the present invention;

FIG. 5 is a feature visualization effect diagram obtained by a fine-grained image recognition method based on multi-view feature fusion according to the present invention.

Detailed ways

The present embodiment, a fine-grained image recognition method based on multi-view feature fusion, is described with reference to FIGS. 2 to 5 , and the method is implemented by the following steps:

Step 1. Bilinear feature extraction step: Using the pre-trained ResNet-50 network structure on ImageNet, input the original image with a fixed size, and fuse the feature maps output by different convolutional layers to obtain bilinear feature vectors.

其中D1、D2表示两种特征的通道数，H和W分别表示特征图的高度和宽度。为了解决融合后特征维数过高的问题，同时保证生成的特征向量中所包含的特征信息足够多，只随机抽取F₂中n个通道的特征图与F₁进行融合。记特征图F₁与F₂中各个位置沿通道的特征向量为

两个特征向量相乘后可得到双线性矩阵

将特征图中各个位置对应的双线性矩阵进行相加，并将矩阵展为一个向量，即为双线性向量

其中D＝D1×D2。该双线性向量提供了比线性模型更强的特征表示。Combined with Figure 2, in the feature extraction step, the network pre-trained in the dataset ImageNet is used as the basic network for feature extraction. Commonly used image classification networks such as VGGNet, GoogLeNet, ResNet, etc., can be fine-tuned (fine-tune). Models are adapted to specific tasks. Specifically, the original image is input to the feature extraction network, and the feature maps output by the last two convolutional layers are obtained, which are recorded as

Among them, D1 and D2 represent the number of channels of the two features, and H and W represent the height and width of the feature map, respectively. In order to solve the problem that the feature dimension is too high after fusion, and at the same time ensure that the generated feature vector contains enough feature information, only the feature maps of _n channels in _F2 are randomly extracted and fused with F1. The feature vectors of each position along the channel in the feature maps F ₁ and F ₂ are written as

The bilinear matrix can be obtained by multiplying the two eigenvectors

Add the bilinear matrices corresponding to each position in the feature map, and expand the matrix into a vector, which is a bilinear vector

where D=D1×D2. This bilinear vector provides a stronger feature representation than the linear model.

Step 2. Suppression branch learning steps:

A. Attention map generation step: generate an attention map according to the size and threshold of the feature map.

B. Image suppression step: generate a suppression mask according to the attention map, and overlay it on the original image to generate a suppressed image with masked local areas.

C. Multi-classification cross-entropy calculation step: The suppression image obtains a bilinear feature vector through step 1, and then inputs it to the fully connected layer to obtain the predicted probability value, and calculates the multi-classification cross-entropy for the obtained class prediction value.

In the suppression branch learning step, the following three aspects are included:

的各个通道求平均值p_d，按该平均值进行排序，选择top-5的值计算熵：Step A: The feature map output by the convolutional layer in the feature extraction network

Calculate the average value of each channel p _d , sort by this average value, and select the top-5 value to calculate the entropy:

Construct the attention map A by comparing the magnitude of entropy and threshold δ:

Step B enlarges the attention map to the original image size, calculates its average value m, uses m*θ as the threshold, sets the elements in the attention map larger than the threshold value to 0, and sets other elements to 1, thereby obtaining the suppression mask M:

Calculate the average value m of the attention map, and set a threshold θ ranging from 0 to 1,

Step C covers the suppression mask to the original image, so as to obtain the suppressed image with the masked local area:

Is (x,y)=I( _x ,y)*M(x,y)

In the formula, I(x,y) is the value of the (x,y) position in I in the original image.

Distraction by suppressing the most salient regions of the image forces the neural network to learn discriminative information from other regions. It can also reduce the network's dependence on training samples, prevent overfitting, and further improve the robustness of the model.

Step 3. Learning steps for similar comparison modules:

A. Image sampling step: randomly select other N images in the same category as positive samples.

B. Feature fusion step: fuse the target image and the randomly sampled positive sample image through the bilinear feature vector obtained in step 1, and the obtained fusion feature integrates the feature information of multiple images under the same category.

C. Calculation of fusion feature loss function: The fused feature vector is directly input into the fully connected layer to obtain the prediction probability, and the multi-class cross entropy is calculated for the obtained category prediction value.

Referring to Figure 3, step A randomly selects N images belonging to the same category as the input image, and sends them to the bilinear feature extraction network in step 1.

Step B averages the bilinear feature vectors of multiple images of the same type output in step A to obtain a fused feature vector:

In the formula, j is the position of the feature vector, V(j) is the value of the feature vector at the jth position, T is the number of positive samples selected; V _r (j) is the rth positive sample at the jth position. the value of;

Step 4: Calculate the center loss;

A. Class center generation step: During the training process, the feature vector of each class center learned by the network is continuously updated.

B. Center loss calculation step: The distance between the bilinear feature vector obtained from each input image and the class center vector is used as the center loss, which is continuously optimized during the training process.

In this embodiment, a feature vector is calculated for each category as the class center of the corresponding category, and this feature vector will be continuously updated as the training progresses. By penalizing the offset of the bilinear feature vector of each sample and the center of the corresponding class of samples, the samples of the same class are aggregated together as much as possible, eliminating the complex sample pair construction process. Let vi be the bilinear feature of the _ith sample, ci be the average feature of all samples of the corresponding category of sample _i , namely the class center, N is the number of samples in the current batch, the formula is as follows:

Step 5. Model optimization loss function calculation steps:

The weighted summation of the cross-entropy loss function of the bilinear feature of the original image, the cross-entropy loss function of the suppressed bilinear feature of the image, the cross-entropy loss function of the fusion feature, and the central loss function can obtain the optimized loss function of the model .

Referring to Figure 4, the cross-entropy loss of the bilinear feature vector of the original image is L _CE1 , the cross-entropy loss of the bi-linear feature vector of the suppressed image is L _CE2 , the cross-entropy loss of the fusion feature is L _CE3 , and the center loss is L _C , the weighted summation of these loss functions, the loss function L optimized by the model can be obtained:

where λ is the weight of the central loss function.

This embodiment is described with reference to FIG. 5 . In FIG. 5 , the first row is a randomly selected original image in the data set, the second row is a class activation map obtained by the global branch input from the original image, and the third row is a class activation map obtained by the suppression branch. It can be seen that in the global branch, the network learns the most salient regions of the image, such as bird's beak, car headlights, etc., and in the suppression branch, the network learns subtle features that are conducive to fine-grained classification, such as bird's beak, car headlights, etc. Torso, wheels, etc. The combination of multiple perspectives makes the judgment basis of the network model more comprehensive, which can capture both salient regions and fine-grained features subtly.

The fine-grained image recognition method described in this embodiment introduces a new data enhancement method, which suppresses component regions in the image through attention map guidance, thereby distracting attention and enabling the network to learn more complementary region feature information. A homogenous comparison module is introduced, which fuses feature information from multiple images of the same category to make the representations of images of the same category as close as possible in the embedding space, thereby improving the classification performance.

The technical features of the above-described embodiments can be combined arbitrarily. For the sake of brevity, all possible combinations of the technical features in the above-described embodiments are not described. However, as long as there is no contradiction between the combinations of these technical features, All should be regarded as the scope described in this specification.

The above-mentioned embodiments only represent several embodiments of the present invention, and the descriptions thereof are specific and detailed, but should not be construed as a limitation on the scope of the invention patent. It should be pointed out that for those skilled in the art, without departing from the concept of the present invention, several modifications and improvements can be made, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the appended claims.

Claims (4)

1. a fine-grained image recognition method based on multi-view feature fusion, is characterized in that: the method is realized by the following steps: Step 1. Bilinear feature extraction; The original image is input into the bilinear feature extraction network, and the feature maps output by different convolutional layers are fused to obtain bilinear feature vectors; the feature extraction network adopts the network structure pre-trained in the dataset ImageNet; Step 2: Suppress branch learning, the specific process is: Step 21: Generate an attention map according to the size and threshold of the feature maps output by different convolutional layers of the feature extraction network described in step 1; Step 22, generating a suppression mask according to the attention map described in step 21, and overlaying the suppression mask on the original image to generate a suppressed image in which the local area is masked; Step 2 and 3: Use step 1 to perform bilinear feature extraction on the suppressed image described in step 2, obtain bilinear feature vector, input the bilinear feature vector to the fully connected layer, and obtain the predicted class probability value, And calculate the multi-class cross entropy for the predicted class probability value; Step 3. Similar comparison module learning; Step 31: Randomly select other N images in the same category as the original image as positive sample images; Step 32: The target image and the positive sample image described in step 31 are sent to the feature extraction network described in step 1 for bilinear feature vector fusion, and the fusion feature is obtained. Feature vector; Step 33: Average the bilinear eigenvectors of multiple images of the same category obtained in step 32 to obtain a fused feature vector, and input the fused feature vector into the fully connected layer to obtain the predicted probability. Calculate the multi-class cross entropy of the predicted probability of the same category; Step 4: Calculate the central loss function _LC ; Let v _i be the bilinear feature of the ith sample, c _i be the average feature of all samples in the corresponding category of sample i, that is, the class center, and N be the number of samples in the current batch, then the formula of the center loss function L _C is as follows:

Step 5. Model optimization loss function calculation; The cross-entropy loss function of the bilinear feature vector of the original image, the cross-entropy loss function of the bilinear feature vector of the suppressed image, the cross-entropy loss function of the fusion feature, and the central loss function are weighted and summed to obtain the loss function of the model optimization. . 2. a kind of fine-grained image recognition method based on multi-view feature fusion according to claim 1, is characterized in that: in step 21, the specific process of generating attention map is: Feature map output from the last convolutional layer in the feature extraction network

Calculate the average value p _d of each channel of , where D is the number of channels of the feature, H and W are the height and width of the feature map, respectively; according to the average value, the formula for obtaining the entropy E is:

Construct an attention map A by comparing the magnitude of entropy and threshold δ:

In the formula, F _k is the two-dimensional feature map corresponding to each channel sorted by channel; In step 22, the specific process of generating the suppression mask is as follows: Enlarge the attention map in step 21 to the original image size, calculate the average value m of the attention map, set a threshold θ between 0 and 1, take m*θ as the threshold, and set the attention map larger than the threshold Elements of m*θ are set to 0, other elements are set to 1, and the suppression mask M is obtained:

In the formula, A(x, y) is the value of the (x, y) position in the attention map A; Covering the suppression mask to the original image to obtain the suppressed image Is ( _x , y) where the local area is masked; Is (x,y)=I( _x ,y)*M(x,y) In the formula, I(x,y) is the value of the (x,y) position in I in the original image. 3. a kind of fine-grained image recognition method based on multi-view feature fusion according to claim 1 is characterized in that: in step 33, the bilinear feature vectors of multiple images under the same category are averaged to obtain a The fused feature vector is expressed as:

In the formula, j is the position of the feature vector, V(j) is the value of the feature vector at the jth position, T is the number of positive samples selected; V _r (j) is the rth positive sample at the jth position. the value of; 4. a kind of fine-grained image recognition method based on multi-view feature fusion according to claim 1, is characterized in that: in step 5, the cross entropy loss function of the bilinear feature vector of the original image is L _CE1 , suppressing the image The cross-entropy loss function of the bilinear feature vector is L _CE2 , the cross-entropy loss function of the fusion feature is L _CE3 , and the central loss function is L _C , which are weighted and summed to obtain the model-optimized loss function L, and finally realize Fine-grained image recognition; expressed as:

In the formula, λ is the weight of the central loss function.

Images