CN108805029B - A ground-based cloud image recognition method based on significant dual activation coding - Google Patents

Abstract

The embodiment of the present invention discloses a ground-based cloud image recognition method based on salient dual activation coding, which includes: inputting a training ground-based cloud image into a convolutional neural network to obtain a convolution activation map; using a shallow convolution activation map to obtain a local area of a saliency image , extract its features to obtain salient feature vectors; obtain the image area corresponding to the deep convolution activation map, and learn the corresponding weights; based on the salient feature vectors and weights, obtain a set of salient feature vectors for the weight of the training ground cloud image, and perform significant dual activation coding on them. , obtain the weighted salient feature vector; obtain the weighted salient feature vector of the test ground cloud image, and classify it to obtain the identification result. The invention uses the convolutional neural network shallow and deep convolutional activation maps to extract features, mines significant structure and texture information and features containing high semantic information, and further obtains the most representative feature representation ground-based cloud map through significant dual activation coding , which improves the accuracy of ground-based cloud image classification.

Description

A ground-based cloud image recognition method based on significant dual activation coding

technical field

The invention belongs to the technical fields of pattern recognition and artificial intelligence, and in particular relates to a ground-based cloud image recognition method based on significant dual activation coding.

Background technique

In the field of atmospheric science, cloud formation, shape characteristics, and amount of cloud reflect the movement of the atmosphere, indicating one of the important signs of future weather changes, and playing a crucial role in weather forecasting and early warning. An important method of cloud observation is ground-based cloud observation. The automatic classification of ground-based cloud images is of great significance to climate and meteorological research. At present, domestic and foreign experts have carried out research work in related fields. Isosalo et al. used local texture information, such as Local Binary Patterns (LBP) and Local Edge Patterns (LEPs), for 5 sky types including Altocumulus, Cirrus, Stratus, Cumulus and Clear Sky. judge. Calbo et al. extracted the Fourier transform image information and the statistical information of the image to describe the ground-based cloud map, and classified five sky types including altocumulus, cirrus, stratus, cumulus and clear sky. Heinle et al. also used spectral, texture and color information to describe ground-based cloud images for ground-based cloud image classification. Xiao et al. further extracted texture, structure and color information in a dense sampling manner for classification of different sky types. Based on Rotation Invariant LBP (Rotation Invariant LBP), Wang et al. proposed Stable LBPs (The Stable LBPs) to classify different cloud types. With the great achievements of Convolutional Neural Networks (CNNs) in pattern recognition, image processing and other fields, CNNs have also begun to be applied to ground-based cloud image classification, and the classification effect of CNNs is better than that based on hand-craft features. ) of the traditional ground-based cloud image classification method. Ye et al. used Convolutional Neural Networks (CNNs) for ground-based cloud image classification for the first time, and the classification accuracy was significantly improved. Zhang et al. improved the performance of cross-domain ground-based cloud classification by encoding local features on convolutional activation maps. Furthermore, Shi et al. argue that features on deep convolutional activation maps outperform traditional hand-craft features in terms of ground-based cloud map representation. The above ground-based cloud image classification methods based on CNNs all perform feature extraction in a single convolution layer, and cannot obtain more complete ground-based cloud image information. In terms of feature representation of ground-based cloud images, these methods usually use max pooling, average pooling, etc. to aggregate the extracted features into a feature vector to represent ground-based cloud images. Such feature vectors are usually missing. discriminative. Therefore, in terms of feature representation of ground-based cloud images, further innovative methods are needed to improve the accuracy of ground-based cloud image classification.

SUMMARY OF THE INVENTION

The purpose of the present invention is to solve the problem of ground-based cloud image classification, and for this purpose, the present invention provides a ground-based cloud image identification method based on significant dual activation coding.

In order to achieve the purpose, the present invention proposes a ground-based cloud image recognition method based on significant dual activation coding, the method comprising the following steps:

Step S1, preprocessing multiple input ground-based cloud images to obtain training ground-based cloud images;

Step S2, inputting the training ground cloud image into a convolutional neural network to obtain a convolution activation map;

Step S3, using the shallow convolution activation map to obtain the saliency image local area of the training ground cloud map;

Step S4, perform feature extraction for each saliency image local area to obtain a corresponding salient feature vector;

Step S5, using the saliency image local area of the shallow layer, obtain the image area corresponding to the deep convolution activation map, and learn the corresponding weight of the image area;

Step S6, based on the significant feature vector and the weight, obtain the weighted significant feature vector set corresponding to the training ground cloud image;

Step S7, performing significant dual activation coding on the weight significant feature vector set to obtain the weight significant feature vector corresponding to the training ground cloud image;

In step S8, the weighted salient feature vector of the test ground cloud image is obtained, and the test ground cloud image is classified based on the weighted salient feature vector to obtain the ground foundation cloud image identification result.

Optionally, the step S1 includes the following steps:

Step S11, normalize the size of the input ground cloud image to H×W to obtain a training ground cloud image, where H and W represent the height and width of the training ground cloud image, respectively;

In step S12, the category label of each training ground cloud image is obtained.

Optionally, the step S2 includes the following steps:

Step S21, determining the convolutional neural network, and initializing it, and modifying the number of outputs at the end of the convolutional neural network to the number D of categories of the ground-based cloud map;

Step S22, inputting the training ground cloud image into the initialized convolutional neural network to obtain a convolution activation map.

Optionally, the step S3 includes the following steps:

Step S31, obtaining a set of shallow convolution activation maps corresponding to a preset shallow convolution layer, the set of shallow convolution activation maps can be expressed as a tensor with a size of H _s ×W _s ×N _s , where , the subscript s represents the number of layers where the shallow layer is located, H _s and W _s represent the height and width of the convolution activation map of this layer, respectively, and N _s represents the number of convolution activation maps of this layer;

Step S32, connecting the activation responses at each same position on all convolution activation maps corresponding to the shallow convolutional layers in turn to obtain an N _s -dimensional local feature vector;

Step S33, using a sliding window to densely sample all the convolution activation maps corresponding to the shallow convolutional layers, and obtain the activation response saliency value S _k of each sliding window based on the local feature vector, where the subscript k represents The kth sliding window;

Step S34, sort the activation response saliency values S _k in descending order, select the sliding windows corresponding to the first K activation response salient values as the saliency image local area, and obtain K saliency image local areas of the training ground cloud image.

Optionally, the size of the sliding window is a×a, and the step size of dense sampling is a/2.

Optionally, the activation response saliency S _k of the kth sliding window on the shallow convolution activation map is expressed as:

表示第i个位置处的局部特征向量，a²＝a×a表示第k个滑动窗口内的局部特征向量数目，

表示第k个滑动窗口的均值特征向量，即滑动窗口内所有局部特征向量的平均值，表示为：where PgP ² represents the two-norm of the vector,

represents the local feature vector at the ith position, a ² =a×a represents the number of local feature vectors in the kth sliding window,

Represents the mean eigenvector of the kth sliding window, that is, the average of all local eigenvectors in the sliding window, expressed as:

Optionally, in the step S4, each of the saliency image local regions is represented as a saliency feature vector m _k .

Optionally, the step S5 includes the following steps:

Step S51, obtain a set of deep convolution activation maps corresponding to a preset deep convolution layer, the set of deep convolution activation maps can be expressed as a tensor with a size of H _d ×W _d ×N _d , where the subscript d represents the number of layers where the deep layer is located, H _d and W _d represent the height and width of the convolution activation map of this layer, respectively, and N _d represents the number of convolution activation maps of this layer;

Wherein, the deep convolutional layer can be selected from the convolutional layers in the second half of the convolutional neural network.

Step S52, connecting the activation responses at each same position on all convolution activation maps corresponding to the deep convolutional layers in turn to obtain an N _d -dimensional local feature vector;

Step S53, according to the saliency image local area corresponding to the shallow convolution layer, obtain K corresponding image areas of size b×b in the deep convolution activation map;

Step S54, calculating the weight corresponding to the image area, expressed as:

表示第k个图像区域的权重，

表示第j个位置处的局部特征向量，b²＝b×b表示第k个图像区域的局部特征向量数目。in,

represents the weight of the kth image region,

represents the local feature vector at the jth position, and b ² =b×b represents the number of local feature vectors in the kth image region.

Optionally, in the step S6, according to the salient feature vectors m _k of the K salient image local regions of the shallow convolution activation map, and the weights w _k of the K image regions of the deep convolution activation map, each is obtained. The weighted salient feature vector set χ of the training ground-based cloud images:

χ={w ₁ m ₁ , w ₂ m ₂ , . . . , w _K m _K }.

Optionally, in the step S7, the weight significant feature vector is expressed as:

h=(ue m)((ue m) ^T (ue m)) ^-1 C

e表示矩阵元素对应相乘，

是一个元素均为c的常数向量。in,

e represents the corresponding multiplication of matrix elements,

is a constant vector with all elements c.

The beneficial effects of the present invention are as follows: the present invention utilizes the shallow and deep convolution activation maps of the convolutional neural network to perform feature extraction, can mine features with significant structure and texture information and contain high semantic information, and further encode through significant dual activation. , to obtain the most representative features representing the ground-based cloud map, thereby improving the accuracy of the ground-based cloud map classification.

It should be noted that the present invention has obtained the National Natural Science Foundation of China Project No. 61501327, No. 61711530240, the Tianjin Natural Science Foundation Key Project No. 17JCZDJC30600, and the Tianjin Applied Basic and Frontier Technology Research Program Youth Fund Project No. 15JCQNJC01700, Tianjin Normal University "Youth Scientific Research Top Talents Cultivation Program" No.135202RC1703, State Key Laboratory of Pattern Recognition Open Project Fund No.201700001, No.201800002, China National Scholarship Fund No.201708120040, No.201708120039 and Tianjin Higher Education Innovation Team Fund project funding.

Description of drawings

FIG. 1 is a flowchart of a ground-based cloud image identification method based on salient dual activation coding proposed according to an embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the specific embodiments and the accompanying drawings. It should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. Also, in the following description, descriptions of well-known structures and techniques are omitted to avoid unnecessarily obscuring the concepts of the present invention.

FIG. 1 is a flowchart of a ground-based cloud image identification method based on significant dual activation coding proposed according to an embodiment of the present invention, and some specific implementation processes of the present invention are described below by taking FIG. 1 as an example. The method of the present invention is a ground-based cloud image identification method based on significant dual activation coding, and its specific steps include:

Step S1, preprocessing multiple input ground-based cloud images to obtain training ground-based cloud images;

The preprocessing of multiple input ground-based cloud images includes the following steps:

Step S11, normalize the size of the input ground cloud image to H×W to obtain a training ground cloud image, where H and W represent the height and width of the training ground cloud image, respectively;

In an embodiment of the present invention, H×W=224×224.

In step S12, the category label of each training ground cloud image is obtained.

Step S2, inputting the training ground cloud image into a convolutional neural network to obtain a convolution activation map;

Further, the step S2 includes the following steps:

Step S21, select a typical convolutional neural network in deep learning to initialize, and modify the output number of the end of the convolutional neural network to the number of categories D of the ground-based cloud map;

In an embodiment of the present invention, the convolutional neural network is VGG19, which will classify 7 types of ground-based cloud images, so D=7.

Step S22, inputting the training ground cloud image into the initialized convolutional neural network to obtain a convolution activation map.

Step S3, using the shallow convolution activation map to obtain the saliency image local area of the training ground cloud map;

Further, the step S3 includes the following steps:

Step S31, obtaining a set of shallow convolution activation maps corresponding to a preset shallow convolution layer, the set of shallow convolution activation maps can be expressed as a tensor with a size of H _s ×W _s ×N _s , where , the subscript s represents the number of layers where the shallow layer is located, H _s and W _s represent the height and width of the convolution activation map of this layer, respectively, and N _s represents the number of convolution activation maps of this layer;

Wherein, the shallow convolutional layer can be selected from the convolutional layers in the first half of the convolutional neural network.

In an embodiment of the present invention, H _s ×W _s ×N _s =224×224×64.

Step S32, connecting the activation responses at each same position on all convolution activation maps corresponding to the shallow convolutional layers in turn to obtain an N _s -dimensional local feature vector;

Step S33, using a sliding window to densely sample all the convolution activation maps corresponding to the shallow convolutional layers, and obtain the activation response saliency value S _k of each sliding window based on the local feature vector, where the subscript k represents The kth sliding window;

The size of the sliding window is a×a, and the step size of dense sampling is a/2.

Among them, the activation response saliency value S _k of the kth sliding window on the shallow convolution activation map is expressed as:

表示第i个位置处的局部特征向量，a²＝a×a表示第k个滑动窗口内的局部特征向量数目，

表示第k个滑动窗口的均值特征向量，即滑动窗口内所有局部特征向量的平均值，表示为：where PgP ² represents the two-norm of the vector,

represents the local feature vector at the ith position, a ² =a×a represents the number of local feature vectors in the kth sliding window,

Represents the mean eigenvector of the kth sliding window, that is, the average of all local eigenvectors in the sliding window, expressed as:

Note that m _k is also referred to as a salient feature vector.

In an embodiment of the present invention, a×a=12×12, and the step size is 6.

Step S34, sorting the activation response salient values S _k in descending order, selecting the sliding windows corresponding to the first K activation response salient values as the saliency image local area, and obtaining K salient image local areas of the training ground cloud image;

In an embodiment of the present invention, K is taken as 200.

Step S4, perform feature extraction for each saliency image local area to obtain a corresponding salient feature vector;

Wherein, each of the saliency image local regions is represented by a saliency feature vector m _k .

Step S5, using the saliency image local area of the shallow layer, obtain the image area corresponding to the deep convolution activation map, and learn the corresponding weight of the image area;

Further, the step S5 includes the following steps:

Step S51 , obtain a set of deep convolution activation maps corresponding to a preset deep convolution layer, and the set of deep convolution activation maps can be expressed as a tensor with a size of H _d ×W _d ×N _d , where the subscript d represents the number of layers where the deep layer is located, H _d and W _d represent the height and width of the convolution activation map of this layer, respectively, and N _d represents the number of convolution activation maps of this layer;

Wherein, the deep convolutional layer can be selected from the convolutional layers in the second half of the convolutional neural network.

In an embodiment of the present invention, H _d ×W _d ×N _d =56×56×256.

Step S52, connecting the activation responses at each same position on all the convolution activation maps corresponding to the deep convolutional layers in turn to obtain an N _d -dimensional local feature vector;

Step S53, according to the saliency image local area corresponding to the shallow convolution layer, obtain K corresponding image areas of size b×b in the deep convolution activation map;

In an embodiment of the present invention, b×b=3×3.

Step S54, calculating the weight corresponding to the image area, expressed as:

表示第k个图像区域的权重，

表示第j个位置处的局部特征向量，b²＝b×b表示第k个图像区域的局部特征向量数目。in,

represents the weight of the kth image region,

represents the local feature vector at the jth position, and b ² =b×b represents the number of local feature vectors in the kth image region.

Step S6, based on the significant feature vector and the weight, obtain the weighted significant feature vector set corresponding to the training ground cloud image;

Among them, according to the salient feature vector m _k of the K saliency image local regions of the shallow convolution activation map, and the weight w _k of the K image regions of the deep convolution activation map, the weighted salient features of each training ground cloud image are obtained. Set of vectors χ:

χ={w ₁ m ₁ , w ₂ m ₂ , . . . , w _K m _K }.

Step S7, performing significant dual activation coding on the weight significant feature vector set to obtain the weight significant feature vector corresponding to the training ground cloud image;

Further, the step S7 includes the following steps:

作为图像的最终表示，即所述权重显著特征向量，基于：Step S71, utilize the following objective function to learn a feature vector

As the final representation of the image, i.e. the weighted salient feature vector, based on:

(w _k m _k ) ^T h=c, (k=1,2,...,K),

where c represents a constant.

Then the objective function can be expressed as:

(ue m) ^Th = C,

是一个元素均为c的常数向量。Among them, e represents the corresponding multiplication of matrix elements,

is a constant vector with all elements c.

In an embodiment of the present invention, c is taken as 1.

提出：Step S72, in order to obtain the optimal

propose:

minP(ue m) ^Th -CP ² ,

Use the pseudo-inverse to solve, and get the minimal norm solution, that is, the optimal h:

h=(ue m)((ue m) ^T (ue m)) ^- 1C.

Step S8, according to the steps S1-S7, obtain the weight significant feature vector of the test ground cloud image, classify the test ground cloud image based on the weight significant feature vector of the test ground cloud image, and obtain the ground foundation cloud image identification result.

In an embodiment of the present invention, based on the weighted salient feature vector of the test ground cloud image, a nearest neighbor classifier is used to classify the test ground cloud image.

Taking the ground-based cloud image database collected by the Chinese Academy of Meteorological Sciences as an example, the correct rate of ground-based cloud image recognition is 91.24%, which shows the effectiveness of the method of the present invention.

It should be understood that the above-mentioned specific embodiments of the present invention are only used to illustrate or explain the principle of the present invention, but not to limit the present invention. Therefore, any modifications, equivalent replacements, improvements, etc. made without departing from the spirit and scope of the present invention should be included within the protection scope of the present invention. Furthermore, the appended claims of this invention are intended to cover all changes and modifications that fall within the scope and boundaries of the appended claims, or the equivalents of such scope and boundaries.

Claims (10)

1. A foundation cloud picture identification method based on significant dual activation coding is characterized by comprising the following steps:

step S1, preprocessing the multiple input foundation cloud pictures to obtain training foundation cloud pictures;

step S2, inputting the training foundation cloud picture into a convolutional neural network to obtain a convolutional activation picture;

step S3, obtaining a saliency image local area of the training foundation cloud picture by utilizing the shallow convolution activation picture;

step S4, extracting the characteristics of each local area of the saliency image to obtain corresponding saliency characteristic vectors;

step S5, acquiring an image area corresponding to the deep convolution activation map by using the superficial saliency image local area, and learning the weight corresponding to the image area;

step S6, based on the significant feature vector and the weight, obtaining a weight significant feature vector set corresponding to the training foundation cloud picture;

step S7, carrying out significant dual activation coding on the weight significant feature vector set to obtain a weight significant feature vector corresponding to the training foundation cloud picture, namely learning a feature vector as the weight significant feature vector by using a target function;

and step S8, acquiring the weight significant feature vector of the test foundation cloud picture, and classifying the test foundation cloud picture based on the weight significant feature vector to obtain a foundation cloud picture identification result.

2. The method according to claim 1, wherein the step S1 comprises the steps of:

step S11, normalizing the size of the input foundation cloud picture into H multiplied by W to obtain a training foundation cloud picture, wherein H and W respectively represent the height and width of the training foundation cloud picture;

and step S12, obtaining the category label of each training foundation cloud picture.

3. The method according to claim 1, wherein the step S2 comprises the steps of:

step S21, determining a convolutional neural network, initializing the convolutional neural network, and modifying the output number of the tail end of the convolutional neural network into the class number D of the foundation cloud picture;

and step S22, inputting the training foundation cloud picture into the initialized convolutional neural network to obtain a convolutional activation picture.

4. The method according to claim 1, wherein the step S3 comprises the steps of:

step S31, obtaining a set of shallow convolution activation maps corresponding to a preset shallow convolution layer, where the set of shallow convolution activation maps can be expressed as a tensor with a size H_s×W_s×N_sWherein the subscript s denotes the number of shallow layers, H_sAnd W_sRespectively representing the height and width, N, of the layer of convolution activation map_sRepresenting the number of the layer of convolution activation maps;

step S32, sequentially connecting the activation responses at each same position on all convolution activation graphs corresponding to the shallow layer convolution layer to obtain N_sA local feature vector of the dimension;

step S33, performing dense sampling on all convolution activation graphs corresponding to the shallow convolution layer by using sliding windows, and acquiring an activation response significant value S of each sliding window based on the local feature vectors_kWherein the subscript k denotes the kth sliding window;

step S34, the activation response significant value S_kAnd performing descending sorting, and selecting sliding windows corresponding to the first K activation response significant values as significant image local areas to obtain K significant image local areas of the training foundation cloud pictures.

5. The method of claim 4, wherein the sliding window has a size of a x a and the step size of the dense samples is a/2.

6. The method of claim 5, wherein the activation response saliency value S for the kth sliding window on the shallow convolution activation map_kExpressed as:

wherein | · | purple sweet²Which represents the two-norm of the vector,

representing a local feature vector at the ith position, a²A x a denotes the number of local feature vectors within the kth sliding window,

the mean feature vector representing the kth sliding window, i.e. the mean of all local feature vectors within the sliding window, is represented as:

7. the method according to claim 1, wherein in step S4, each local region of the significant image is represented as a significant feature vector m_k。

8. The method according to claim 1, wherein the step S5 comprises the steps of:

step S51, obtaining a set of deep convolution activation maps corresponding to a preset deep convolution layer, where the set of deep convolution activation maps can be expressed as a tensor with a size of H_d×W_d×N_dWherein the subscript d represents the number of layers in which the deep layer is located, H_dAnd W_dRespectively representing the height and width, N, of the layer of convolution activation map_dRepresenting the number of the layer of convolution activation maps;

wherein the deep convolutional layer can be selected from convolutional layers of the latter half of the convolutional neural network;

step S52, sequentially connecting the activation responses at each identical position on all convolution activation graphs corresponding to the deep convolution layer to obtain N_dA local feature vector of the dimension;

step S53, acquiring K corresponding image areas with b × b size in the deep convolution activation map according to the local area of the saliency image corresponding to the shallow convolution layer;

step S54, calculating a weight corresponding to the image region, expressed as:

wherein,

the weight representing the k-th image region,

representing local feature vectors at the j-th position, b²B × b denotes the number of local feature vectors of the k-th image region.

9. The method according to claim 4, wherein in step S6, the salient feature vectors m of the local areas of K salient images of the activation map are convolved according to a shallow layer_kAnd weights w of K image regions of the deep convolution activation map_kAnd obtaining a weight significant feature vector set χ of each training foundation cloud picture:

χ＝{w₁m₁,w₂m₂,...,w_Km_K}。

10. the method according to claim 9, wherein in step S7, the weighted significant feature vector is represented as:

h＝(u⊙m)((u⊙m)^T(u⊙m))^-1C，

wherein,

an indication of a corresponding multiplication of matrix elements,

is a constant vector whose elements are c.

Images