CN112668403A - Fine-grained ship image target identification method for multi-feature area - Google Patents

Abstract

The invention provides a fine-grained ship image target identification method for a multi-feature area, which specifically comprises the steps of preprocessing ship image data to obtain the image size required by a convolutional neural network; inputting the processed image into an improved RA-CNN network for detection and identification, wherein the image is sent to a classification network VGG-SDP network of a first scale layer for feature extraction and classification, and meanwhile, a fifth pooling layer of the VGG-SDP is input into a positioning network JCMR-APN, and is sent into a second scale layer after being cut; sending the cut four feature area images to a VGG-SDP network for feature extraction and classification, inputting a fifth pooling layer of the VGG-SDP to a positioning network APN, cutting and sending to a third scale layer; sending the four feature area images with smaller sizes to a VGG-SDP network for feature extraction and classification, and outputting feature areas by the APN network; and finally, fusing the results of the three scale layers to obtain the characteristic region position and the category of the ship.

Description

Fine-grained ship image target identification method for multi-feature area

Technical Field

The invention relates to detection and identification of ship targets, in particular to a fine-grained ship image target identification method for a multi-feature area.

Background

The construction of a set of ship target identification system becomes an important work, and the strengthening of marine ship target identification is a research direction with military value and economic value. The traditional radar target tracking equipment and Automatic Identification System (AIS) of the ship mainly take positioning as a main part, and have many defects in the aspect of identifying the specific type of the ship. With the development of deep learning, a vessel target identification technology based on a Convolutional Neural Network (CNN) becomes a key point of research in the field of marine navigation, and Network models such as VGG-16 and VGG-19 are widely applied to vessel target identification; in terms of the CNN framework based on the proposed regions, fast-CNN has better performance in this area. At the present stage, ships are various, different models are derived from the same ship class, and the difference between the models is very small, so that the identification of the ship type is difficult in the aspects of time and accuracy, and the various CNN networks mentioned above are generally only suitable for the scenes with large differences between the ships. Aiming at the problem, a fine-grained image classification network (RA-CNN) is widely applied, but the research in the field of ship target identification is less, and the method is still in the primary stage at present. In 2019, a fine-grained ship identification method of a single-feature area is provided by Huolihao et al based on an RA-CNN network, and is verified on a photoelectric ship data set. However, because the RA-CNN network has 3 scale layers, based on the characteristics of the network itself, as the scale layers become deeper, the effect of the single feature region becomes worse, specifically, global information cannot be fully utilized, so that robustness becomes worse and the identification accuracy is also reduced.

Disclosure of Invention

The invention aims to provide a fine-grained ship image target identification method for a multi-feature area, and solves the actual problem that the similarity of an existing ship is too high. The identification performance is improved by adding the Scale Dependent Pooling (SDP) in the VGG-19 classification network, a plurality of feature regions are generated by adding the joint clustering (joint clustering) algorithm in the attention-directed suggestion network (APN), the overlapping rate among the plurality of feature regions is reduced by designing the optimization algorithm, a new loss function is defined to cross train the VGG-19 and the APN, and the accuracy and the robustness of the whole model are effectively improved.

The invention has the following advantages:

1. by introducing the SDP algorithm into the VGG-19 network, the problem of excessive pooling of small targets is solved, and the classification performance of the network is improved;

2. the multi-feature area positioning network can better utilize global information, so that the target identification has higher robustness;

3. the characteristic region optimization algorithm reduces the overlapping rate of a plurality of characteristic regions, so that each positioning region can be mutually independent;

4. a channel loss function is introduced to carry out common optimization on the network, so that the robustness of the image to noise can be improved, and the ship target identification rate is improved.

Specifically, the present invention achieves the above object by the following scheme:

the ship target identification method combining the multi-feature area and the fine-grained image is characterized by being more detailed

Identifying the target, comprising the steps of:

s1, preprocessing a ship data set;

s2, inputting the preprocessed ship image into the trained improved RA-CNN network;

s3, framing a ship characteristic region by fusing results of 3 scale layers of the improved RA-CNN network and displaying the classification of the ship characteristic region at the upper left corner;

the step S1 further includes the steps of:

s1.1, uniformly converting an input ship image into a ship image with a resolution of 224 multiplied by 224;

s1.2, uniformly mapping each pixel value of the image from [0, 255] to [ 1,1] according to the formula, wherein the calculation formula is as follows:

wherein x is_i,jAnd x'_i,jRespectively represent the pixel values before and after the pre-processing, i ∈ [0,223 ]]，j∈[2,223]；

The step S2 further includes the steps of:

s2.1, firstly, in a first scale layer, an input image passes through a VGG-SDP network v₁Extraction of features, output P of 5 th pooling layer₅Will be the input of the JCMR-APN network;

s2.2, JCMR-APN network m₁Will be according to P₅Clustering the generated channels and selecting a proper part to generate a plurality of independent characteristic areas;

S2.3、v₁the network selects the pixel number N of the minimum characteristic area in a plurality of characteristic areas as the input of the self-adaptive pooling criterion, selects a proper pooling layer to generate a classification confidence vector Y of a first scale⁽¹⁾；

S2.4, evaluating the coverage rate among the characteristic areas, and adjusting the areas;

s2.5, intercepting and amplifying the adjusted characteristic region to be used as the input of the next scale layer;

s2.6, inputting the characteristics of S2.5 into VGG-SDP network v of the second scale layer₂Extracting features, repeating S2.2 and S2.3 to generate a classification confidence vector Y of a second scale⁽²⁾. The positioning network m of the second layer is different from the first layer₂Intercepting and amplifying each input characteristic region by adopting an APN (access point name) network consistent with RA-CNN (random access network-common network) and inputting the input characteristic region to a third scale layer;

s2.7, repeating S2.6 in the third scale layer to generate a classification confidence vector Y of the third scale layer⁽³⁾APN network m in this layer₃Only generating corresponding characteristic regions, not intercepting any more, and determining final positioning according to each characteristic region generated by the first scale layer;

step S2.1 comprises the following steps:

s2.1-1, when VGG-SDP inputs image I, the image is firstly subjected to feature extraction through 5 convolution blocks and a fifth pooling layer P₅The output of the network is sent to a JCMR-APN network;

s2.1-2, the JCMR-APN network inputs a proper feature map selected from the last three pooling layers to a full connection layer for classification according to the size N of a plurality of feature area statistical areas and the VGG-SDP network, and the self-adaptive pooling selection standard is as follows:

wherein, P₃(I) And P₄(I) Respectively representing the output of the third and fourth pooling layers, selecting the appropriate output of the pooling layer according to the size of N by the M (I) function, and selecting the fifth pooling layer P by the network when the characteristic region N is too large₅To represent a feature of the target; when N is too small, the network then selects the third pooling layer P that undergoes less convolutional pooling₃To characterize to retain more information;

step S2.2 comprises the following steps:

s2.2-1, setting a characteristic diagram channel sample set S ═ S₁,s₂,…s_kConstructing an adjacency matrix W (matrix of weight among samples) and a degree matrix D (sum of weight of each sample and all connected samples) from the sample set S;

s2.2-2, obtaining a laplacian matrix L ═ D-W from W, D;

s2.2-3, standardizing L:

s2.2-4, mixing L_normThe eigenvalues of the system are arranged from large to small, and the first K eigenvalues are taken to calculate the eigenvectors of the eigenvalues;

s2.2-5, standardizing each eigenvector and forming an eigenvector matrix L_f；

S2.2-6, taking L_fGenerating a new sample set S 'for each row vector, carrying out K-means clustering on S' to generate K clusters corresponding to the characteristic regions of K ship images;

step S2.4 comprises the following steps:

s2.4-1, calculating the pixel value of each characteristic region from the coordinate parameter sequence, taking the position of the maximum characteristic region as a reference region, and recording the region into a fixed region sequence;

s2.4-2, calculating the overlapping area N of the second characteristic area and the reference area_olThe calculation formula is as follows:

N_ol＝max(t_x(br)-t_x(ul),0)×max(t_y(br)-t_y(ul),0) (2)

s2.4-3, when the ratio of the overlapping area to the characteristic area is larger than a certain threshold value, adjusting the characteristic area until the ratio is lower than the threshold value;

s2.4-4, counting the second characteristic region into a fixed region sequence after adjustment, and sequentially selecting comparison regions from the fixed region sequence from large to small for S2.4-2 and S2.4-3 operations in subsequent characteristic regions.

Drawings

FIG. 1 is an overall block diagram of a multi-feature area ship target identification system of the present invention;

FIG. 2 is a diagram of an improved RA-CNN network vessel target identification architecture of the present invention;

FIG. 3 is a block diagram of a VGG-19 network formed in conjunction with an SDP algorithm in accordance with the present invention;

FIG. 4 is a flow chart of the federated clustering algorithm of the present invention;

FIG. 5 is a schematic diagram of a JCMR-APN network overfitting according to the present invention;

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

As shown in fig. 1 to 5, the present invention provides a fine-grained ship image target identification method for a multi-feature area, the method includes the following steps, and the overall block diagram of the system is shown in fig. 1:

s1, preprocessing a ship data set;

s2, inputting the new image formed in step S1 to the improved RA-CNN network;

s3, performing cross training on two parts of networks in the improved RA-CNN network, and finally framing out the ship

The region of the target and the category is displayed in the upper left corner;

in the step S2, the conventional RA-CNN is an identification method based on a single feature region, and as the scale layers become deeper, extracting a single feature may make the original network unable to use global information well, thereby causing a decrease in identification rate, and for this problem, this research designs a novel network for identifying a ship target in which an RA-CNN network and a multi-feature region are fused, the novel RA-CNN network still maintains 3 original scale layers, but a classification network in each scale layer is changed from VGG-19 to VGG-SDP, a positioning network APN in a first scale layer is changed to a JCMR-APN network, other two scale layers still maintain APN structures, and the improved RA-CNN network ship target identification is as shown in fig. 2, and the specific steps may be divided into:

s2.1, firstly, in a first scale layer, an input image passes through a VGG-SDP network v₁Extraction of features, output P of 5 th pooling layer₅Will be the input of the JCMR-APN network;

s2.2, JCMR-APN network m₁Will be according to P₅Clustering the generated channels and selecting a proper part to generate a plurality of independent characteristic areas;

S2.3、v₁the network selects the pixel number N of the minimum characteristic area in a plurality of characteristic areas as the input of the self-adaptive pooling criterion, selects a proper pooling layer to generate a classification confidence vector Y of a first scale⁽¹⁾；

S2.4, evaluating the coverage rate among the characteristic areas, and adjusting the areas;

s2.5, intercepting and amplifying the adjusted characteristic region to be used as the input of the next scale layer;

s2.6, inputting the characteristics of S2.5 into VGG-SDP network v of the second scale layer₂Extracting features, repeating S2.2 and S2.3 to generate a classification confidence vector Y of a second scale⁽²⁾. The second layer being different from the first layerPositioning network m₂Intercepting and amplifying each input characteristic region by adopting an APN (access point name) network consistent with RA-CNN (random access network-common network) and inputting the input characteristic region to a third scale layer;

s2.7, repeating S2.6 in the third scale layer to generate a classification confidence vector Y of the third scale layer⁽³⁾APN network m in this layer₃Only generating corresponding characteristic regions, not intercepting any more, and determining final positioning according to each characteristic region generated by the first scale layer;

in the context of figure 2, it is shown,

representing the prediction probability of the first scale layer truth, Y⁽¹⁾Representing a prediction class label. Is provided with Y_truthRepresenting a true category label, Y⁽²⁾And Y⁽³⁾Respectively representing the prediction class labels of the second and third scale layers, then the classification loss L is obtained_inner＝(Y⁽ⁱ⁾,Y_truth) It is a real label Y_truthAnd a predictive label Y⁽ⁱ⁾And performing a cross entropy operation result. In addition to this, the present invention is,

respectively representing the true prediction probabilities of the second and third scales, with the loss between the first and second scales being

The second and third inter-scale losses are

The step S2.1, aiming at the problem of excessive pooling of small-sized targets, adds the SDP algorithm to the VGG-19 network, so that the network intelligently selects a proper convolution block output for classification, thereby improving the identification rate of the network on small ships, and the VGG-19 forms a VGG-SDP network structure diagram by combining with the SDP algorithm as shown in fig. 3, and the specific steps can be divided into:

s2.1-1, when VGG-SDP inputs image I, the image is firstly subjected to feature extraction through 5 convolution blocks and a fifth pooling layer P₅Output of (2) into JCMR-APNA network;

s2.1-2, the JCMR-APN network selects a proper feature map from the last three pooling layers according to the size of N according to the size of a plurality of feature area statistical areas, the VGG-SDP network inputs the feature map into a full connection layer for classification (here, the VGG-SDP network selects the pixel number N of the minimum feature area in a plurality of independent feature areas), and the self-adaptive pooling selection standard is as follows:

wherein, P₃(I) And P₄(I) Respectively representing the output of the third and fourth pooling layers, selecting the appropriate output of the pooling layer according to the size of N by the M (I) function, and selecting the fifth pooling layer P by the network when the characteristic region N is too large₅To represent a feature of the target; when N is too small, the network then selects the third pooling layer P that undergoes less convolutional pooling₃To characterize to retain more information;

in the step 2.2, the flow chart of the joint clustering algorithm is shown in fig. 4, and the specific steps can be divided into:

s2.2-1, setting a characteristic diagram channel sample set S ═ S₁,s₂,…s_kConstructing an adjacency matrix W (matrix of weight among samples) and a degree matrix D (sum of weight of each sample and all connected samples) from the sample set S;

s2.2-2, obtaining a laplacian matrix L ═ D-W from W, D;

s2.2-3, standardizing L:

s2.2-4, mixing L_normThe eigenvalues of the system are arranged from large to small, and the first K eigenvalues are taken to calculate the eigenvectors of the eigenvalues;

s2.2-5, standardizing each eigenvector and forming an eigenvector matrix L_f；

S2.2-6, taking L_fGenerating a new sample set S 'for each row vector, performing K-means clustering on S' to generate K clusters corresponding to K shipsCharacteristic regions of the ship image;

in the step 2.4, a schematic diagram of the JCMR-APN network overfitting is shown in fig. 5, and the specific steps can be divided into:

s2.4-1, calculating the pixel value of each characteristic region from the coordinate parameter sequence, taking the position of the maximum characteristic region as a reference region, and recording the region into a fixed region sequence;

s2.4-2, calculating the overlapping area N of the second characteristic area and the reference area_olThe calculation formula is as follows:

N_ol＝max(t_x(br)-t_x(ul),0)×max(t_y(br)-t_y(ul),0) (2)

s2.4-3, when the ratio of the overlapping area to the characteristic area is larger than a certain threshold value, adjusting the characteristic area until the ratio is lower than the threshold value;

s2.4-4, counting the second characteristic region into a fixed region sequence after adjustment, and sequentially selecting comparison regions from the fixed region sequence from large to small for S2.4-2 and S2.4-3 operations in subsequent characteristic regions.

Claims (1)

1. The fine-grained ship image target identification method for the multi-feature area is characterized by comprising the following steps of:

s1, preprocessing a ship data set;

s2, inputting the preprocessed ship image into the trained improved RA-CNN network;

s3, marking a ship feature area by fusing the results of 3 scale layers of the improved RA-CNN network and displaying the category of the ship feature area at the upper left corner;

the step S1 includes the steps of:

s1.1, uniformly converting an input ship image into a ship image with a resolution of 224 multiplied by 224;

s1.2, uniformly mapping each pixel value of the image from [0, 255] to [ 1,1] according to the formula, wherein the calculation formula is as follows:

wherein x is_i,jAnd x'_i,jRespectively represent the pixel values before and after the pre-processing, i ∈ [0,223 ]]，j∈[2,223]；

The step S2 includes the steps of:

s2.1, firstly, in a first scale layer, an input image passes through a VGG-SDP network v₁Extraction of features, output P of 5 th pooling layer₅Will be the input of the JCMR-APN network;

s2.2, JCMR-APN network m₁Will be according to P₅Clustering the generated channels and selecting a proper part to generate a plurality of independent characteristic areas;

S2.3、v₁the network selects the pixel number N of the minimum characteristic area in a plurality of characteristic areas as the input of the self-adaptive pooling criterion, selects a proper pooling layer to generate a classification confidence vector Y of a first scale⁽¹⁾；

S2.4, evaluating the coverage rate among the characteristic areas, and adjusting the areas;

s2.5, intercepting and amplifying the adjusted characteristic region to be used as the input of the next scale layer;

s2.6, inputting the characteristics of S2.5 into VGG-SDP network v of the second scale layer₂Extracting features, repeating S2.2 and S2.3 to generate a classification confidence vector Y of a second scale⁽²⁾(ii) a The positioning network m of the second layer is different from the first layer₂Intercepting and amplifying each input characteristic region by adopting an APN (access point name) network consistent with RA-CNN (random access network-common network) and inputting the input characteristic region to a third scale layer;

s2.7, repeating S2.6 in the third scale layer to generate a classification confidence vector Y of the third scale layer⁽³⁾APN network m in this layer₃Only generating corresponding characteristic regions, not intercepting any more, and determining final positioning according to each characteristic region generated by the first scale layer;

step S2.1 comprises the following steps:

s2.1-1, when VGG-SDP inputs image I, the image is firstly subjected to feature extraction through 5 convolution blocks and a fifth pooling layer P₅The output of the network is sent to a JCMR-APN network;

s2.1-2, the JCMR-APN network inputs a proper feature map selected from the last three pooling layers to a full connection layer for classification according to the size N of a plurality of feature area statistical areas and the VGG-SDP network, and the self-adaptive pooling selection standard is as follows:

wherein, P₃(I) And P₄(I) Respectively representing the output of the third and fourth pooling layers, selecting the appropriate output of the pooling layer according to the size of N by the M (I) function, and selecting the fifth pooling layer P by the network when the characteristic region N is too large₅To represent a feature of the target; when N is too small, the network then selects the third pooling layer P that undergoes less convolutional pooling₃To characterize to retain more information;

step S2.2 comprises the following steps:

s2.2-1, setting a characteristic diagram channel sample set S ═ S₁,s₂,…s_kConstructing an adjacent matrix W and a degree matrix D from the sample set S;

s2.2-2, obtaining a laplacian matrix L ═ D-W from W, D;

s2.2-3, standardizing L:

s2.2-4, mixing L_normThe eigenvalues of the system are arranged from large to small, and the first K eigenvalues are taken to calculate the eigenvectors of the eigenvalues;

s2.2-5, standardizing each eigenvector and forming an eigenvector matrix L_f；

S2.2-6, taking L_fGenerating a new sample set S 'for each row vector, carrying out K-means clustering on S' to generate K clusters corresponding to the characteristic regions of K ship images;

step S2.4 comprises the following steps:

s2.4-1, calculating the pixel value of each characteristic region from the coordinate parameter sequence, taking the position of the maximum characteristic region as a reference region, and recording the region into a fixed region sequence;

s2.4-2, calculating the overlapping area N of the second characteristic area and the reference area_olThe calculation formula is as follows:

N_ol＝max(t_x(br)-t_x(ul),0)×max(t_y(br)-t_y(ul),0) (2)

s2.4-3, when the ratio of the overlapping area to the characteristic area is larger than a certain threshold value, adjusting the characteristic area until the ratio is lower than the threshold value;

s2.4-4, counting the second characteristic region into a fixed region sequence after adjustment, and sequentially selecting comparison regions from the fixed region sequence from large to small for subsequent characteristic regions to carry out the operations of the steps S2.4-2 and S2.4-3.

Images