CN107609601B - Ship target identification method based on multilayer convolutional neural network - Google Patents

Abstract

The invention discloses a ship target identification method based on a multilayer convolutional neural network, which comprises the following steps: s1, constructing a ship sample library by using the existing images, parameters and model data, and continuously enriching the data through detecting the target data in the using process; s2, carrying out ship target feature training, namely forming a visible light/infrared and two-dimensional/three-dimensional fusion ship feature knowledge base for carrying out ship target classification and identification by carrying out identification training on a ship sample base under the framework of a convolutional neural network; s3, collecting ship target data, wherein the ship target data are used for collecting visible light or infrared video data of a marine ship target in real time with high resolution; s4, detecting the marine ship target; s5, roughly classifying the ship target images; s6, developing fine classification recognition work of the ship target based on the deep neural network model completed by the ship target feature training, and accurately recognizing the type of the ship. The problem of ship target identification is solved.

Description

Ship target identification method based on multilayer convolutional neural network

Technical Field

The invention relates to a target identification method, in particular to a ship target identification method based on a multilayer convolutional neural network.

Background

China has wide coastlines, sea areas and abundant ocean resources, with the continuous development of economy, the number of ships on the sea is more and more, and the ship detection has urgent practical requirements; and ships, civil ships and the like in peripheral countries and regions often illegally enter legal sea areas of China to carry out activities such as measurement, monitoring, fishing and the like, so that the legal ocean rights of the country cannot be effectively guaranteed, the ocean right maintaining condition is more and more complicated, the offshore safety is seriously threatened, and the detection and identification of the targets such as the ships and the like in the legal sea area of China are enhanced to have very important practical significance.

At present, the image reconnaissance capability to the sea area of China is weak, and the intelligent degree is low. The time period from aerial or satellite photography to accurate information acquisition is long, the result is not satisfactory, along with the increasing of the types and the number of sea surface ships, civil ships and other interference targets, the complexity of the sea surface environment is higher, the shapes of the ships and the civil ships are updated at high speed, the current ship identification system faces a plurality of problems, and the identification accuracy and the availability are difficult to meet the requirements. The accuracy of ship target identification algorithms such as Constant False Alarm Rate (CFAR) algorithm, wavelet transform and the like which are widely applied at present is 80-85 percent; the modern sea surface environment requires that the accuracy of ship and civil ship target identification reaches more than 95%, the traditional ship target identification algorithm has a single application range, the identification accuracy is low when complex targets are processed, and the requirement of the modern sea surface environment on higher identification accuracy cannot be met. At present, domestic ship target detection and identification are mostly based on satellite remote sensing image data such as SAR, IR and the like, but the method has serious defects. The satellite data has low transmission speed due to overlong distance with the ground, and the ship target data cannot be updated in real time and cannot find and identify ships appearing in a specified sea area in time; satellite remote sensing cannot clearly shoot sea surface images in thunderstorm and cloudy weather, so that accuracy of ship target identification in severe sea surface environment is reduced. Meanwhile, the shapes of various military and civil ships are updated quickly, and the ship identification image library generated by adopting the universal method needs to be frequently updated, so that the maintenance time and cost of the ships are increased, and the intelligent degree is low.

Disclosure of Invention

The invention aims to provide a ship target identification method based on a multilayer convolutional neural network, which is used for solving the problem of ship target identification.

The invention relates to a ship target identification method based on a multilayer convolutional neural network, which comprises the following steps: s1, constructing a ship sample library by using the existing images, parameters and model data, and improving the ship sample library by detecting target data acquisition; s2, forming a visible light/infrared and two-dimensional/three-dimensional integrated ship feature knowledge base for classification and identification of ship targets through identification training of a ship sample base under the framework of a convolutional neural network; the method comprises the following steps: establishing a ship target feature training architecture based on a deep convolutional neural network, wherein the ship target feature training architecture comprises a feature sharing CNN layer, a finely-adjustable CNN layer, a region extraction model, an ROI pooling layer and a classification regression full-connection layer; the shared characteristic CNN layer is a part of an existing deep convolutional neural network classification model with the last full connection layer removed, pre-training is carried out on a VOC classification data set or an ImageNet classification data set before the training of a ship target detection and identification model is carried out to improve the extraction capability of image characteristics, and then the parameter of the trained classification model with the full connection layer removed is utilized to carry out parameter initialization on the shared characteristic CNN layer; the fine-tuning CNN layer is used for online incremental learning; the region extraction model is a convolutional neural network, after a CNN layer and a finely-tunable CNN layer are shared, a convolutional layer and two parallel neural networks of full connection layers are constructed, and the extraction of a target region possibly existing in the ship to be detected is realized by sharing the characteristics of the CNN layer and the finely-tunable CNN layer; the ROI pooling layer is used for outputting a feature vector with fixed dimensionality after ROI pooling; the training of the ship target detection and recognition model comprises 4 steps; the first stage is as follows: initializing parameters of a shared characteristic CNN layer by using a classification model pre-trained on a sample library, initializing parameters of weights and bias items for a fine-tunable CNN layer and a region extraction model by using Gaussian normal distribution with zero-mean variance sigma, and fine-tuning the region extraction neural network model by using visible light or infrared training set image data; and a second stage: using the region extracted by the region extraction model in the first stage to extract and train a shared feature CNN layer and a fine-adjustable CNN layer, and simultaneously initializing parameters of the shared feature CNN layer by using a classification model pre-trained on a Voc2012 or ImageNet data set; and a third stage: using the shared characteristic CNN layer and the finely adjustable CNN layer parameters trained in the second stage, re-initializing the shared characteristic CNN layer and the finely adjustable CNN layer parameters of the region extraction model, then fixing the shared characteristic CNN layer and the finely adjustable CNN layer parameters and finely adjusting the region extraction neural network model; a fourth stage: freezing the parameter of the convolutional layer of the region extraction neural network model in the third stage, extracting the region, and then sharing the characteristic CNN layer and finely adjusting the parameter of the CNN layer; s3, collecting visible light or infrared video data of the marine ship target in real time with high resolution; s4, detecting the marine ship target; s5, carrying out rough classification on the ship target image aiming at simply classifying the large class of the ship and reducing the workload of subsequent fine classification and identification of the ship target; s6, developing fine classification recognition work of the ship target based on the deep neural network model completed by the ship target feature training, and accurately recognizing the type of the ship.

According to an embodiment of the ship target identification method based on the multilayer convolutional neural network, in the S4, the ship target detection specifically includes: calculating by adopting an SLIC image segmentation algorithm, wherein the SLIC algorithm comprises the following specific steps: initializing cluster center C by setting K super pixel point numbers_k＝[l_k，a_k，b_k，x_k，y_k]，k∈[1，K](ii) a Disturbing the center of the chaotic group in the neighborhood, and moving the clustering center to the position with the lowest gradient; for the center of each cluster group, distributing optimal matching pixel points in adjacent areas surrounding the center of one cluster group according to a distance calculation rule; calculating new population centers and residual errors; until convergence when the residual error E is less than the threshold.

According to an embodiment of the ship target identification method based on the multilayer convolutional neural network of the present invention, the image gradient calculation in S4 includes:

G(x，y)＝||I(x+1，y)-I(x-1，y)||²+||I(x，y+1)-I(x，y-1)||² (1)；

where I (x, y) is a lab vector, representing the color vector for the pixel spatial location (x, y), and | is the L2 norm; setting step length S to sqrt (N/K), N to total number of pixels and K to number of super pixels, distributing matching points to each clustering center in the neighborhood of 2S to 2S of the clustering center, calculating according to the set distance between each point and the clustering center, finally connecting each pixel point in the image with the nearest clustering center and covering the pixel point by the search domain of the clustering center, calculating new center after all the pixel points are connected with the nearest cluster center, wherein the new center is the average value of labxy vectors belonging to the cluster, and repeating the process of connecting the pixel points with the nearest cluster center and recalculating the cluster center until convergence.

According to an embodiment of the ship target identification method based on the multilayer convolutional neural network of the present invention, in S4, the core in the HOG extraction process includes: 1) calculating the gradients of the horizontal coordinate and the vertical coordinate of the image, and calculating the gradient direction value of each pixel position according to the gradients; 2) uniformly dividing an image into small blocks according to spatial positions, counting histograms in gradient directions in the small blocks according to a set quantization standard to obtain feature vectors corresponding to the small blocks, and then connecting the feature vectors of all cells in one large block in series to obtain large block HOG features, wherein each large block is composed of m × m small blocks, and each small block is composed of n × n pixels.

According to an embodiment of the ship target identification method based on the multilayer convolutional neural network, after extracting features of each region in training data, a proper classifier is trained to complete region classification, so that large regions of the sea surface and the sky are eliminated, and targets in an interested region are located.

According to an embodiment of the ship target identification method based on the multilayer convolutional neural network of the present invention, S6 includes: identification of warships and civil ships; let the training set be { (x)₁，t₁)，……，(x_N，t_N) In which x_iIs an input, t_iIs the target output, N is the number of samples in the training set, and the output of the deep neural network integration is defined as:

where m is the number of neural networks, F_i(n) is the output of the ith deep neural network on the nth sample, and F (n) is the output of the deep neural network integrated on the nth sample, and the ResNet deep neural network is used for integrated learning, so that incremental learning of a new image data set is achieved, and the method comprises four modules:initial ResNet deep neural network integration, current ResNet deep neural network integration, duplicate ResNet deep neural network integration, and new ResNet deep neural network integration; firstly, training initial ResNet deep neural network integration by utilizing a first warship and civil ship image data set to obtain current ResNet deep neural network integration; then, copying the current ResNet deep neural network integration to obtain a copied ResNet deep neural network integration; next, when a second warship and civil ship image data set arrives, training the copied ResNet deep neural network integration by using the data set to obtain a new ResNet deep neural network integration; finally, a selective negative correlation learning method is applied for selection.

According to an embodiment of the ship target identification method based on the multilayer convolutional neural network of the present invention, S6 further includes: when the nth sample is used for training, the error function of the ith neural network is defined as:

wherein p is_i(n) is a penalty term, λ ∈ [0, 1 ]]Is a control parameter, called penalty parameter, for controlling a balance between the mean square error and a penalty term, penalty term p_i(n) can be defined as: p is a radical of_i(n)＝(F_i(n)-F(n))∑_j≠iF_j(n) -f (n), the penalty term makes the ith neural network inversely related to the rest of the neural networks in the integration, so as to achieve the difference between the individual neural networks, and when the lambda is 0, the integration of the obtained neural networks is equivalent to independently training a group of neural networks; with the increasing of λ, the emphasis of training will be gradually adjusted to make the individuals have difference.

According to an embodiment of the ship target identification method based on the multilayer convolutional neural network of the present invention, S6 further includes: a) initializing a neural network integration ens1 of size m; b) training ens1 by negative correlation learning with S1; c) from the beginning to the end of the second sub data set, i.e., T2, …, T, the following loop process is performed: copying the current neural network integrated enst-1 to obtain a copy, and recording the copy as an enscopy; training the enscopy with St through negative correlation learning; combining the enst-1 and the enscopy, and recording the combined result as the enscomb; selecting m neural networks to form an integrated enst by using a selection process for the enscomb; d) outputting a final neural network integration ensT; aiming at the selection process in the step c), the genetic algorithm is used for selecting the enscomb, according to the selective negative correlation ensemble learning method, m neural networks are selected from the enscomb with the size of 2m in the selection process to form the enst, and the selection process can be formalized into the following optimization problem with constraints:

wherein, J (omega) is a predefined objective function, and the design quality of the objective function is directly related to the generalization performance of the selected neural network integration. Omega_iIs a 2 m-dimensional binary vector, ω _i1 means that the ith neural network is selected to form the enst, ω_iIf 0, the ith neural network is deleted; solving using a genetic algorithm includes: a) initialization algorithm setting: integration size m, population size of GA pop _ size, probability of crossover p_cProbability of mutation p_mAnd a fitness function J (ω); b) randomly generating an initial population consisting of omega; c) repeating the following steps until a specified number of times is reached: evaluating the fitness of each individual in the current population; selecting parent individuals using roulette and crossing with a single point with a probability p_cAnd the probability p of mutation_mGenerating offspring; repairing each individual omega, sigma by greedy strategy_iω_iM, an individual omega one of which is 1_iChange to 0 to minimize J (ω) and repeat the process until Σ_iω_iIf m, if e_iω_i< m, an individual omega, one of which is 0_iInstead, 1, J (ω) is minimized and the process is repeated until Σ_iω_iM; d) and outputting wopt as an optimal solution, and selecting a neural network corresponding to wopt which is 1 according to wopt to form final neural network integration.

An embodiment of the ship target identification method based on the multilayer convolutional neural network according to the present invention includes: the fine-tuning CNN layer is used for online incremental learning, when the target detection of the ship is performed online, if the detected ship target meets the labeling condition, the target detection model needs to be subjected to online incremental learning, the shared characteristic CNN layer, the region extraction model, the subsequent ROI pooling layer and the full connection layer of the fine-tuning CNN layer need to be fixed in the online incremental learning process, and then the fine-tuning CNN layer is subjected to parameter fine tuning through classification errors.

According to an embodiment of the ship target identification method based on the multilayer convolutional neural network of the present invention, S2 further includes: in the model inference stage, if the ship identified by the ship target detection and identification model meets the sample labeling condition, online incremental learning on the finely adjustable CNN layer of the ship target detection and identification model by using the sample is required, which includes: the first step is as follows: judging the sample by using a ship target identification threshold value predicted by a target detection identification model, and if the threshold value is higher than sigma₁Or below σ₂Online incremental learning is not required; conversely, if the threshold is lower than σ₁And is higher than sigma₂One online incremental learning is required; the second step is that: and performing primary parameter fine adjustment on the fine-adjustable CNN layer of the target detection model by using the image data meeting the sample labeling condition.

The ship target identification method based on the multilayer convolutional neural network utilizes the image characteristics of the ship target and massive sample image data to develop the artificial intelligent ship target identification technology research based on big data, and the ship target identification method based on the artificial intelligent image processing and ship target identification system improves the ship target identification capability and accuracy. The method comprises the steps of constructing a video image feature database of ships and civil ships, providing a fusion method for extracting and identifying the features of the ships and civil ships based on a deep convolutional neural network architecture, a classification method of the ships and civil ships based on deep learning, a parameter training method based on the deep learning, and establishing an image processing and ship target identification system based on artificial intelligence.

Drawings

FIG. 1 is a block diagram of an artificial intelligence based image processing and vessel target recognition system

FIG. 2 is a block diagram of an artificial intelligence based image processing and vessel target recognition system;

FIG. 3 is a diagram of a deep convolutional neural network-based ship target feature training architecture

Fig. 4 is a schematic diagram of a SLIC segmentation algorithm;

FIG. 5 is a diagram of the complete HOG feature extraction algorithm and process;

FIG. 6 is a schematic diagram of a rectangular cell;

FIG. 7 is a schematic diagram of 9 directional blocks of a HOG;

FIG. 8 is a schematic view of the HOG features of a ship, sky, and ocean;

FIG. 9 is a diagram illustrating an example of target location;

FIG. 10 is a flow chart illustrating a rough classification of visible light targets;

FIG. 11 is a SIFT descriptor feature extraction and description flow chart;

fig. 12 is a diagram showing a feature extraction result of SIFT descriptors;

FIG. 13 is a graph showing the result of feature extraction for dense-SIFT descriptors;

FIG. 14 is a graph illustrating feature extraction for a multi-scale dense-SIFT descriptor;

FIG. 15 is a diagram of a deep ResNet ensemble learning framework based on selective negative correlation learning;

FIG. 16 is a block diagram of a neural network integration selection algorithm.

Detailed Description

In order to make the objects, contents, and advantages of the present invention clearer, the following detailed description of the embodiments of the present invention will be made in conjunction with the accompanying drawings and examples.

The image processing and detection target identification system framework based on artificial intelligence comprises six parts, namely ship sample library construction, ship target feature training, ship target data acquisition, ship target detection, ship target rough classification and ship target fine classification identification.

Fig. 1 is a block diagram of an artificial intelligence based image processing and vessel target recognition system, and as shown in fig. 1, the artificial intelligence based image processing and vessel target recognition system includes: the system comprises a network image database 1, a ship target acquisition module 2, a ship sample library module 3, a ship target detection module 4, a ship target identification convolutional neural network model 5, a ship target coarse classification module 6 and a ship target fine classification module 7.

The invention relates to a ship target identification method based on a multilayer convolutional neural network, which comprises the following steps:

s1, constructing a ship sample library by using the existing images, parameters and model data, and continuously enriching the data through detecting the target data in the using process;

s2, carrying out ship target feature training, namely forming a visible light/infrared and two-dimensional/three-dimensional fusion ship feature knowledge base for carrying out ship target classification and identification by carrying out identification training on a ship sample base under the framework of a convolutional neural network;

s3, collecting ship target data, wherein the ship target data are used for collecting visible light or infrared video data of a marine ship target in real time with high resolution;

s4, detecting the marine ship target;

s5, roughly classifying the ship target images;

s6, developing fine classification recognition work of the ship target based on the deep neural network model completed by the ship target feature training, and accurately recognizing the type of the ship.

As shown in FIG. 1, the operation of an artificial intelligence based image processing and detection target recognition system includes three stages: the method comprises an image data acquisition stage, a model offline training stage and a model online reasoning stage, wherein each stage comprises different functional modules.

1) An image data acquisition phase comprising:

the method comprises two groups of data, namely a data set for training a ship target detection and identification model and a data set for classifying warships and civil ships, wherein the two types of image data comprise a training data set and a test data set of infrared images and visible light images. The system comprises an infrared image information acquisition module, an image preprocessing module and a sea-land separation module.

2) The model off-line training stage comprises the following steps:

the method is mainly used for offline training of a ship target detection and identification model and a warship and civil ship classification model.

3) The model online reasoning phase comprises the following steps:

fig. 2 is a frame diagram of an image processing and ship target recognition system based on artificial intelligence, and as shown in fig. 2, the stage includes online incremental detection and recognition of a ship target detection model in addition to detection and recognition of a ship target and classification tasks of warships and civil ships.

As shown in fig. 1 and 2, the method for constructing a ship sample library in S1 specifically includes:

the image data for training the target detection and classification model based on supervised learning comprises a training set, a verification set and a test set, wherein the verification set can be obtained by separating a part of the training set, and the proportion of the verification set to the training set is 50%: 25%: 25 percent. Training and testing image data sets for ship target detection and identification models are very different from training and testing image data sets for warships and civil ship classification models. Training and testing image data for ship target detection and identification need to perform label framing on ship targets and give labels to ship categories; and the training and testing image data for classifying the warships and the civil ships only need to give corresponding category labels to the images of the warships and the civil ships.

The collection and database establishment of visible light image samples are the basis of visible light target detection and identification, and the quantity, quality, target consistency and richness of images play a very key role in the establishment of a final detection and identification model.

Image data is acquired and compared through various channels, and ImageNet, Voc2012 and a Fleetmom image database are used in the invention. The ImageNet database is established by Stanford university in the United states, is the largest database in the field of image recognition at present, and comprises about 22000 types of targets and about 15 million types of calibration images.

According to the requirement analysis of scenes and projects, the establishment of the database is mainly established around ships near shore and on the sea. After analysis, three major types of target databases, namely military ships, civil ships and interference targets, mainly need to be established. Taking ImageNet as an example, the downloading of the database and the file information generation process are as follows:

1) obtain the ship class from ImageNet; xml, obtaining databseinfo using downloadstatus, releasedstate, structure _ freed information;

2) the waship large class is obtained from ImageNet; xml (generic) is obtained by using downloadstatus, releasedstate and structure _ released information as well;

3) xml (differentiated database) of novarship is generated using the database info generated in the above two steps;

4) the species aegeanisland, barrierrierreef, lighthouse, etc., are obtained from ImageNet.

Primary screening of data:

selecting principle: screening by name

For example, whether 30 classes of warship are all reserved or not, and whether 43 classes of nonwarship are all reserved or not are detailed in the reservation information of warship and the reservation information of nonwarhip (stored by xml text information);

using the above manual retention information, the folders are moved into warship V1 and Nonwashship V1, other V1.

Secondary screening of data:

selecting principle:

1) the side view is guaranteed as much as possible to ensure that the target feature is adequately captured.

2) The image object is to be centered.

3) The image objects cannot be too small, excluding images smaller than 400 x 400 pixels.

4) The background interference of the image is as little as possible;

5) the image object is to be completely presented in the image, and part of the image object outside the image view field is to be excluded.

6) The characteristics of the object itself should be apparent.

After two-stage screening, ship data with clear targets and moderate angles can be obtained. According to the analysis of project requirements and the assumption of scenes, three types of reef, lighthouse and island are mainly selected from the interference image data. The three major categories of image data are shown in table 1, with each category being subdivided into several subclasses.

TABLE 1

Categories	Before screening	After screening
			Warship	Class 30	Class 21
Nonwarship	Class 43	Class 11
			Other	Class 3	Class 3

S2, training ship target characteristics, comprising:

the ship target feature training module is a key link of ship identification, and mainly comprises the contents of ship target feature training architecture design based on a deep convolutional neural network model, two-dimensional and three-dimensional ship target fusion identification, model training, online incremental learning and the like.

Fig. 3 is a diagram illustrating a deep convolutional neural network-based ship target feature training architecture, and as shown in fig. 3, the deep convolutional neural network-based ship target feature training architecture includes a feature sharing CNN layer, a fine-tunable CNN layer, a region extraction model, an ROI pooling layer, a classification regression full-link layer, and the like.

The shared feature CNN layer may be part of an existing deep convolutional neural network classification model with the last fully-connected layer removed. Before carrying out training of a ship target detection and identification model, the part generally needs to be pre-trained on a VOC classification data set or an ImageNet classification data set to improve the extraction capability of image features, and then the trained classification model is used for removing parameters of a full connection layer part to carry out parameter initialization on a shared feature CNN layer. Common classification models for the shared feature CNN layer include AlexNet, GoogleNet, VGG16, VGG19, and ResNet, among others.

The fine-tunable CNN layer is mainly used for online incremental learning. The specific process is as follows: when the ship target detection is carried out on line, if the detected ship target meets the labeling condition, the target detection model needs to be subjected to online incremental learning. In the online incremental learning process, parameters of a shared feature CNN layer, a region extraction model, a subsequent ROI pooling layer of a fine-tunable CNN layer and a full-connection layer need to be fixed, and then the fine-tunable CNN layer is subjected to parameter fine tuning through classification errors, so that the purposes of learning new knowledge from new image data and keeping the ability of learning the knowledge before as much as possible are achieved.

The region extraction model is a convolutional neural network, and after a CNN layer and a fine-tunable CNN layer are shared, a neural network of a convolutional layer and two parallel full-connection layers is constructed. The model has the function of realizing the extraction of the target area possibly having the ship to be detected by sharing the CNN layer characteristics and finely adjusting the CNN layer characteristics. On the basis of the model, in order to solve the difficult problem of difficult detection of small targets, the information of the image Context can be added, so that the problem is effectively solved. At present, the commonly used region extraction models comprise an RPN region extraction model provided by fast RCNN based on an Attention mechanism, an AZ-Net model based on neighborhood information and an adaptive scaling Anchor strategy and an HAZN model capable of automatically adapting to an object and the scale of a component of the object, and the region extraction models solve the problems of low picture resolution and difficulty in detecting a small-size object by improving the quality of region extraction.

The design idea of the ROI pooling layer is derived from fast RCNN, and the design is to solve the problem that an input image with any scale can be accepted by a detection target detection recognition model. After ROI pooling, the model outputs a feature vector with fixed dimensionality, and then the feature vector can be connected with Softmax or a related algorithm capable of classifying, such as a Support Vector Machine (SVM), to classify whether the feature vector is a ship target.

The ship target detection and recognition model training comprises 4 stages. The first stage is as follows: initializing parameters of a shared characteristic CNN layer by using a classification model pre-trained on a sample library, initializing parameters of weight and bias items of a fine-tunable CNN layer and a region extraction model by using Gaussian normal distribution with zero mean variance (sigma) (generally taking 0.001), and fine-tuning a region extraction neural network model by using visible light or infrared training set image data; and a second stage: using the region extracted by the region extraction model in the first stage to extract and train a shared feature CNN layer and a fine-adjustable CNN layer, and simultaneously initializing parameters of the shared feature CNN layer by using a classification model pre-trained on a Voc2012 or ImageNet data set; and a third stage: using the shared characteristic CNN layer and the finely adjustable CNN layer parameters trained in the second stage, re-initializing the shared characteristic CNN layer and the finely adjustable CNN layer parameters of the region extraction model, then fixing the shared characteristic CNN layer and the finely adjustable CNN layer parameters and finely adjusting the region extraction neural network model; a fourth stage: and (3) freezing the parameter of the convolutional layer of the region extraction neural network model in the third stage, extracting the region by using the region extraction model, and then sharing the characteristic CNN layer and finely adjusting the parameter of the CNN layer.

The training process shows that the shared characteristic CNN layer, the finely adjustable CNN layer and the region extraction model of the ship target detection and recognition model need to be alternately trained, and meanwhile, one part of parameters need to be frozen in each training process, and then the other part of parameters need to be trained, so that the overall performance of the region extraction model and the classification regression part can be improved at the same time.

In the moldIn the type inference stage, if the ship identified by the ship target detection and identification model meets the sample labeling condition, online incremental learning on the finely-adjustable CNN layer of the ship target detection and identification model needs to be performed by using the sample. The specific method comprises two steps: judging the sample by using a ship target identification threshold value predicted by a target detection identification model, and if the threshold value is higher than sigma₁Or below σ₂(σ₁＞σ₂In general σ₁Take 0.9, σ₂Taking 0.6), online incremental learning is not needed; conversely, if the threshold is lower than σ₁And is higher than sigma₂One online incremental learning is required; the second step is that: and performing primary parameter fine adjustment on the fine-adjustable CNN layer of the target detection model by using the image data meeting the sample labeling condition.

S3, acquiring ship target data, including:

the ship target sample image acquired by the actual video sensor is closest to the actual target detection and identification environment, and the actual use scene can be better reflected. Therefore, the invention adopts the full high-definition visible light CMOS image sensor to convert the monitored target into an image signal, transmits the image signal to a special image processing system, and has the function of completing the acquisition of an original image through an image data acquisition channel and inputting the image signal into a processing computer for preprocessing.

S4, the ship target detection specifically comprises the following steps:

the detection of dynamic targets is the basis of the whole system, and subsequent target identification can be ensured only if targets appearing in a scene are timely and accurately detected and positioned. The target detection in the application of the invention not only ensures real-time performance, but also delays the fighter because the battlefield is changed instantly and caught and changed in time; and the accuracy of target detection and segmentation is ensured, because the accurate identification of the target is the core of the whole system.

Therefore, aiming at the requirements, the invention adopts a target detection method based on multi-algorithm fusion, exerts the advantages of each algorithm and completes the detection task.

(1) SLIC image segmentation algorithm

The SLIC (simple linear iterative clustering) algorithm is a super-pixel segmentation algorithm based on a clustering algorithm and is calculated by an LAB space and a 5-dimensional space consisting of x and y pixel coordinates. The algorithm provides a brand-new distance calculation method to enhance the regularity of the super-pixel shape, not only can divide a color image, but also can compatibly divide a gray image, and in addition, the number of the super-pixels needing to be divided can be set. SLICs can be significantly more efficient in producing similar or better segmentations.

The SLIC algorithm is started by setting the number of K super-pixels, uniformly distributing K clustering centers in an image according to the set number of the super-pixels, disordering the clustering centers in n x n (n is generally 3) neighborhoods, and moving the clustering centers to the positions with the minimum gradient in the neighborhoods. This operation not only can avoid placing the edge to the clustering center and can also reduce the probability of selecting noise pixel.

The image gradient calculation formula is as follows:

G(x，y)＝||I(x+1，y)-I(x-1，y)||²+||I(x，y+1)-I(x，y-1)||² (1)；

where I (x, y) is a lab vector, representing the color vector for the pixel spatial location (x, y), and | is the L2 norm. This contains color and intensity information.

The step size S is set to sqrt (N/K), N is the total number of pixels, and K is the number of super pixels. And distributing a matching point for each clustering center in the 2S-by-2S neighborhood of the clustering center, calculating according to the set distance between each point and the clustering center, and finally associating each pixel point in the image with the nearest clustering center and covering by the search domain of the clustering center. After all pixels are associated with the nearest cluster center, a new center is calculated, which is the average of all lab xy vectors belonging to the cluster. The process of associating the pixel points with the nearest population center and recalculating the population centers is repeated until convergence.

After this process is completed, a small amount of missing color exists, i.e., a small number of pixel points have the same color in the adjacent larger segment but are not associated with it. This is rare but may increase because the aggregation of SLIC algorithms does not explicitly enhance connectivity. The final stage of the algorithm requires that connectivity be enhanced to identify disjoint segments by maximum nearby clustering.

The time complexity of the SLIC algorithm is O (N), compared with other image segmentation algorithms, only linear calculated amount and memory amount are needed, the segmentation efficiency is greatly improved, and the algorithm is convenient to deploy and only needs 1 super-pixel number as an input parameter.

The SLIC algorithm comprises the following specific steps:

initializing cluster center C by setting K super pixel point numbers_k＝[l_k，a_k，b_k，x_k，y_k]，k∈[1，K]；

Disturbing the center of the chaotic group in the neighborhood, and moving the clustering center to the position with the lowest gradient;

for the center of each cluster group, distributing optimal matching pixel points in adjacent areas surrounding the center of one cluster group according to a distance calculation rule;

calculating new population centers and residual error E (distance between previous center and current center);

repeating the operations 3 and 4 until the residual error E is smaller than the threshold value, namely converging;

and forcing connectivity.

Fig. 4 is a schematic diagram of the SLIC segmentation algorithm, and as shown in fig. 4, the segmentation segments generated by the SLIC algorithm are uniform in size, and can specify the number of blocks, which is relatively flexible.

(2) Histogram of Oriented Gradients (HOG) feature representation comprising:

histogram of Oriented Gradient (HOG) features are a kind of feature descriptors used for object detection in computer vision and image processing. The method uses the gradient direction characteristics of the image, is similar to an edge direction histogram method, a SIFT descriptor and a context shape method, but is characterized in that the gradient direction characteristics are calculated on grid-dense grid cells with uniform size, and an overlapped local contrast normalization method is used for improving the accuracy.

The core idea of the histogram of directional gradients feature is that the appearance and shape of objects in an image can be well described by the directional density distribution of the gradients or edges. The realization method is that the image is divided into small connected areas called grid units; then collecting a gradient direction or edge direction histogram of each pixel point in the grid unit; finally, the histograms are combined to form a feature descriptor. In order to improve the accuracy, the local histograms may also be subjected to contrast normalization in a larger interval (block) of the image by first calculating the density of each histogram in this interval (block) and then normalizing each cell in the interval according to this density value. By this normalization, better stability to illumination variations and shadows can be obtained.

Fig. 5 shows a complete HOG feature extraction algorithm and process diagram, and fig. 6 shows a schematic diagram of a rectangular cell, as shown in fig. 5 and 6,

the core steps in the HOG extraction process may briefly include:

1) calculating the gradients of the horizontal coordinate and the vertical coordinate of the image, and calculating the gradient direction value of each pixel position according to the gradients;

2) the image is uniformly divided into small blocks according to spatial positions, namely the cells in fig. 6, a histogram in the gradient direction is counted in the cells according to a set quantization standard to obtain a feature vector corresponding to the cells, and then the feature vectors of all the cells in one block in fig. 6 are connected in series to obtain the HOG feature of the block.

The HOG feature is obtained by connecting directional gradient histograms of a plurality of rectangular cells in a rectangular block in series, both the block and the cells can be not rectangles in practical application, but the rectangles are more common ones, the HOG feature can be called as rectangular HOG, in fact, circular HOG and central surrounding HOG exist, and the HOG feature is rectangular according to project application requirements.

Fig. 7 is a schematic diagram of 9 directional blocks of the HOG, and fig. 8 is a schematic diagram of HOG features of a ship, the sky, and the ocean, as shown in fig. 7 and fig. 8, the rectangular HOG uses a rectangular cell grid which is repeatedly traversed, a Block is composed of cells which are densely and repeatedly traversed, and feature vector normalization is independently performed in each Block to reduce the influence of illumination. Typically each block consists of a grid of m x m cells, whereas a cell consists of n x n pixels. And the gradient direction of each cell is divided into z direction blocks, the z directions are subjected to weighted projection by using the gradient direction and amplitude in the cell, and finally, each cell generates a z-dimensional feature vector. The HOG used by Dalal et al for human detection chooses z to 9, i.e. divides 360 degrees into 9 direction blocks, and then projects the direction gradient.

After each cell is projected according to the graph division, a z-dimension feature vector is obtained, and the feature vectors corresponding to all the cells in the block are connected in series to form the HOG feature.

By comparing the texture features with the HOG features, it can be seen that the difference between the texture features of the ship, the sky and the ocean is not large, and the ship, the sky and the ocean are difficult to distinguish even if being seen by human eyes, so the HOG features are adopted.

(3) Training classifier to realize preliminary target positioning

After extracting features of each region in the training data, training a proper classifier to complete the classification of the regions so as to eliminate large blocks of regions of the sea surface and the sky and locate the target of the region of interest.

The Support Vector Machine (SVM) can be highlighted in the supervised classifier, can effectively solve the Machine learning problem under a small sample, can effectively solve the generalization problem and the nonlinear classification problem, and can be summarized into two main ideas:

1) the method is used for analyzing linear divisible conditions, and for linear inseparable conditions, a nonlinear mapping algorithm is used for converting linear inseparable samples of a low-dimensional input space into a high-dimensional feature space to enable the linear divisible samples to be linearly separable, so that the linear analysis of the nonlinear features of the samples by the high-dimensional feature space through the linear algorithm is possible;

2) it constructs the optimal segmentation hyperplane in the feature space based on the structure risk minimization theory, so that the classifier gets global optimization, and the expected risk in the whole sample space meets a certain upper bound with a certain probability.

Fig. 9 is a diagram illustrating an example of positioning at a target, and as shown in fig. 9, a diagram of a preliminary positioning result of a visible light target may be obtained by combining the SVM and the extracted features.

S5, roughly classifying the ship targets, including:

the final objective of the project is to hope to detect and identify the model of military ship from the visible light image, but there are several interference factors in the environment, such as civil ship, reef, island, etc., so these interference categories must be screened when identifying, so as to reduce the false alarm rate of identification.

If various types of military ships and warships are classified together with interference categories (such as various civil ships, lighthouses and small islands), the inter-category distances of the feature spaces are different. The distance between the military ships, the civil ships and the interference targets can be larger, the distance between the military ships of different types under the same large class can be smaller, the classes are mixed together for classification and identification, the accuracy of various types of division in the characteristic space can be influenced, and the accuracy of final classification is influenced.

Fig. 10 is a flowchart illustrating a rough classification of visible light targets, and as shown in fig. 10, comprehensive consideration is given to the fact that the visible light targets that have already been located need to be roughly classified, the identification of the large classes is completed, and on the basis, the classification of the fine classes is performed.

Aiming at the characteristic extraction of the rough classification of the visible light/infrared target, a complete processing flow is designed, and two technical schemes are compared:

1) adopting a multi-scale dense-SIFT and visual dictionary structure as feature representation, and using an SVM classifier as a modeling tool;

2) adopting multi-scale dense-SIFT and Fisher Vector as feature representation, and a linear SVM classifier as a modeling tool;

(1) SIFT descriptor

Fig. 11 is a flowchart illustrating the SIFT descriptor feature extraction and description, and as shown in fig. 11, the image feature extraction and description using the SIFT descriptor includes the following basic steps:

1) constructing a scale space and carrying out extreme value detection;

2) accurately positioning the key points;

3) determining the direction of the key point;

4) and generating a characteristic point descriptor.

The specific flow of the SIFT algorithm is briefly described as follows:

1) generating a scale space;

2) detecting a spatial extreme point;

3) determining the position of an extreme point;

4) removing the edge response;

5) distributing key point directions;

6) and generating a characteristic point descriptor.

(2) Multi-scale dense-SIFT descriptor

The SIFT descriptor is extracted and described only for stable feature points of an image, so that the problems of information loss and omission necessarily exist. When the SIFT descriptor is applied to an image, the image needs to have a more standard form, for example, the size of the image is large enough, and the proportion of key objects is large enough, so that enough feature points can be detected and then used for subsequent matching. In the process of detecting and describing the feature points, it is easy to see that the complexity is high, a large amount of calculation time is consumed, and the task of image identification and classification is not favorable. When the Fisher Vector is constructed, after the feature extraction link is carried out, a clustering method is applied to generate the codebook, so that if sufficient abundant information cannot be provided in the feature extraction link, the representativeness of the generated codebook is directly influenced, and the subsequent classification accuracy is further influenced. Therefore, on the basis of comprehensively considering the factors, the invention adopts an improved multi-scale dense-SIFT feature extraction method.

The dense-SIFT descriptor adopts a uniform sampling method, the image is extracted by the same pixel interval, the sampling density is controlled by a parameter 'step length', and the invention is expressed by step. This results in very dense feature points, thus ensuring better use of the rich information of the image. And because each key point does not need to be compared with 26 points in the neighborhood and the same-position areas of the upper layer and the lower layer to judge the extreme point, the complexity of calculation is greatly reduced. After the feature points are extracted at sampling intervals of a basic step size, a uniform scale S is distributed to each key point, the scale can be uniformly set according to the actual situation, and a large amount of complex operation for calculating the scale is avoided.

When describing key points, in order to ensure the rotation invariance of features, firstly adjusting the key points to 0 degree, then constructing a circular region by taking the key points as the circle centers and taking a pre-allocated uniform scale S as the radius, dividing pixel points falling in the circular region into 4 multiplied by 4 non-overlapping sub-regions, and calculating gradient accumulation values in 8 directions (0 degrees, 45 degrees, 90 degrees, 135 degrees, 180 degrees, 225 degrees, 270 degrees and 315 degrees) in each region. Different from SIFT, SIFT adopts the Gaussian window function to carry out gradient weighted accumulation, and in dense-SIFT, the Gaussian window function is replaced by a rectangular window, the neighborhood of key points is uniformly weighted instead of the Gaussian weighting, and the mean value of the Gaussian function of the unit where the key points are located is weighted after the gradient accumulation of the key points is completed. The approximation method not only improves the speed, but also ensures that the performance is not lost. Each feature region is still represented by a 128-dimensional vector.

Because the multiscale dense-SIFT adopts a method for uniformly extracting key points, the characteristic scale invariance performance of the multiscale dense-SIFT can be damaged to a certain extent, and in order to ensure the scale invariance, the multiscale extraction method is adopted to extract and describe each key point by 4 different scales. The large scale represents the profile features of the image and the small scale represents the detail features of the image. The features thus obtained also guarantee invariance of the scale.

Fig. 12 shows a feature extraction result graph of SIFT descriptor, fig. 13 shows a feature extraction result graph of dense-SIFT descriptor, fig. 14 shows a feature extraction graph of multi-scale dense-SIFT descriptor, as shown in fig. 12 to 14,

(3) fisher Vector construction of target rough classification

In the image classification problem, the most representative method for describing an image is to extract local features (SIFT) from the image, and then encode the local features into a high-dimensional vector, which is the global feature expression of the image. The most common encoding technique is to quantize a local feature set to an offline-trained visual dictionary, describe an image into a feature vector of a fixed length according to the frequency of occurrence of each feature, i.e., word frequency, and the visual dictionary can be obtained by training through a clustering method such as k-means (k-means) or Gaussian Mixture Model (GMM), which is called Bag-of-visual-words (BOV). However, the bag-of-words model is only one way of organizing discrete unordered feature point sets, and the quantization process is lossy, so the information that can be described is not comprehensive.

The item adopts a Fisher Kernel (Fisher Kernel) -based coding technology, and a Vector generated by coding is the Fisher Vector. The image global feature expression based on the Fisher Vector can be regarded as the expansion of a bag-of-words model (BOV), both the image global feature expression and the bag-of-words model are intermediate representations based on an image, and the Fisher Vector integrates the advantages of a discriminant model and a generator model, so that the appearance frequency of each visual word is reflected, and difference information about the distribution of local features on the visual words is encoded, so that the Fisher Vector can represent richer image features than the bag-of-words model, and the high-dimensional characteristics of the Fisher Vector enable the Fisher Vector to be combined with a simple and effective linear classifier to achieve a good effect.

The image Fisher Vector construction procedure is as follows:

let I ═ x₁，……，x_n) Is a set of D-dimensional feature vectors, such as extracted SIFT descriptors in the image. Let Θ be (μ ═ u_k，∑_k，π_k: k-1, … …, K) are parameters of a mixed gaussian model GMM, fitting the distribution of descriptors. GMM model for each feature vector x_iAnd each mixed state in GMM, assigned an intensity (a posteriori probability):

for each modality k, consider the mean and variance vectors:

where j is 1, 2, … … and D is the dimensional index of the feature vector. Using u of each mode in the Gaussian mixture model_kAnd v_kConnected in series, a Fisher Vector representation of the image is constructed:

if the length of the local feature descriptor is D and the number of the Gaussian mixture models is K (namely, a codebook of K is generated by applying a Gaussian mixture clustering algorithm), the visual bag of words model (BOV) can obtain a K-dimensional histogram Vector, and the constructed Fisher Vector is a (2D +1) K-1-dimensional histogram Vector. Therefore, for samples of the same size, the dimension of the Fisher Vector is generally much higher than the BOV, so that the Fisher Vector needs a smaller codebook, i.e., less computation cost, if histogram vectors of similar size are to be generated.

The Fisher Vector is improved, and an image classification method based on the spatial Fisher Vector is adopted. The implementation steps are summarized as follows:

1) and extracting the multi-scale dense-SIFT scale invariant feature transformation feature points of all the images in the image library.

2) And clustering the feature points in the feature point space of the image by using a Gaussian mixture clustering algorithm to obtain a codebook.

3) FisherVector for each image is generated using the gradient vectors and the Kelly components.

4) And 2, dividing each image into 2 multiplied by 2 spatial regions, and respectively counting the number and the coordinates of the characteristic points of each small block.

5) And splicing to generate a spatial Fisher Vector of each image by using the Fisher vectors of the small blocks.

After the Fisher vectors are constructed using the method described above, each image will be represented by one Fisher Vector, and the next step is to train a classifier to classify the image objects to complete the coarse classification of the visible light objects.

S6, the fine classification identification of the ship target comprises the following steps:

the classification of warships and civil ships is also required through the ship image obtained by image segmentation of the ship target. The image classification problem is to classify images into one of several categories by analyzing the images, and mainly emphasizes the judgment of the overall semantics of the images. Aiming at the classification tasks of warships and civil ships, a ResNet deep convolution neural network is adopted for realizing the classification tasks.

(1) Deep CNN ship classification recognition architecture negative correlation learning based on selective negative correlation ensemble learning is a method for training neural network integration, and a training set is represented by { (x)₁，t₁)，……，(x_N，t_N) In which x_iIs an input, t_iIs the target output and N is the number of samples in the training set. In negative correlation learning, the output of deep neural network integration is defined as:

where m is the number of neural networks, F_i(n) is the output of the ith deep neural network at the nth sample, and F (n) is the output of the deep neural network integrated at the nth sample. The ResNet deep neural network used in the project is used for ensemble learning, so that incremental learning of a new image data set is achieved. Fig. 15 is a diagram of a deep ResNet ensemble learning framework based on selective negative correlation learning, as shown in fig. 15, which mainly includes four modules: initial ResNet deep neural network integration, current ResNet deep neural network integration, replicated ResNet deep neural network integration, and new ResNet deep neural network integration.

As shown in fig. 15, the interaction relationship between the modules is: firstly, training initial ResNet deep neural network integration by utilizing a first warship and civil ship image data set to obtain current ResNet deep neural network integration; then, copying the current ResNet deep neural network integration to obtain a copied ResNet deep neural network integration; next, when a second warship and civil ship image data set arrives, training the copied ResNet deep neural network integration by using the data set to obtain a new ResNet deep neural network integration; finally, in order to enable the ResNet deep neural network integration to learn new image data knowledge while retaining the learned knowledge to a large extent and keeping the size of the deep integration model unchanged, a selective negative correlation learning method needs to be applied for selection, and a selection algorithm and a negative correlation learning algorithm used herein are explained in a model training section.

(2) Model training

Training for ResNet deep neural network integration involves a selection algorithm and an inverse correlation learning algorithm. The following will specifically describe the negative correlation learning algorithm and the selective negative correlation ensemble learning method.

The negative correlation learning algorithm trains each neural network individual in parallel by using a back propagation algorithm. The negative correlation learning can enable the neural network individuals to have differences, and meanwhile, the performance of each individual is guaranteed. The negative correlation learning may be characterized mainly in the design of the error function. The error function of the negative correlation learning consists of the mean square error and a penalty term. When the nth sample is used for training, the error function of the ith neural network is defined as:

wherein p is_i(n) is a penalty term, λ ∈ [0, 1 ]]Is a control parameter, called penalty parameter, for controlling a balance between the mean square error (affecting the accuracy of the individual) and the penalty term (affecting the difference between individuals). Penalty term p_i(n) can be defined as: p is a radical of_i(n)＝(F_i(n)-F(n))∑_j≠iF_j(n) -F (n). It can be seen that the penalty term makes the ith neural network negatively correlated with the rest of the neural networks in the ensemble, thereby achieving the goal of having a difference between the individual neural networks. It can also be seen that when λ ═ sWhen 0, the obtained neural network integration is equivalent to independently training a group of neural networks; with the increasing of λ, the emphasis of training will be gradually adjusted to make the individuals have difference.

Fig. 16 is a block diagram of a neural network integrated selection algorithm, and as shown in fig. 16, the selection algorithm is mainly applied to the selection link of fig. 15, and has two purposes: firstly, considering that not all individuals are beneficial to improving the generalization performance of the integrated model in the integrated learning; secondly, the neural network integration is not too large. The selective negative correlation ensemble learning method used in this project uses genetic algorithms to screen the ResNet deep neural network ensemble. The overall process of the method is as follows (assuming each sub data set is S1, S2 …, ST):

a) initializing a neural network integration ens1 of size m;

b) training ens1 by negative correlation learning with S1;

c) from the beginning to the end of the second sub data set, i.e., T2, …, T, the following loop process is performed: copying the current neural network integrated enst-1 to obtain a copy, and recording the copy as an enscopy; training the enscopy with St through negative correlation learning; combining the enst-1 and the enscopy, and recording the combined result as the enscomb; selecting m neural networks to form an integrated enst by using a selection process for the enscomb;

d) and outputting the final neural network integration ensT.

Aiming at the selection process in the step c), the item selects the enscomb by using a genetic algorithm. As known from the above selective negative correlation ensemble learning method, the selection process selects m neural networks from the enscomb of size 2m to constitute the enst. Thus, the selection process can be formulated as the following constrained optimization problem:

wherein, J (omega) is a predefined objective function, and the design quality of the objective function is directly related to the generalization performance of the selected neural network integration. Omega_iIs a 2 m-dimensional binary directionQuantity, omega _i1 means that the ith neural network is selected to form the enst, ω_i0 means that the ith neural network is deleted. Because the item only classifies warships and civil ships and the ResNet deep convolution neural network also has stronger classification capability, m can take a relatively small value, for example, m is 5.

For constrained optimization problems as above, a Genetic Algorithm (GA) can be used to solve. The integrated selection algorithm based on the genetic algorithm comprises the following steps:

a) initialization algorithm setting: integration size m, population size of GA pop _ size, probability of crossover p_cProbability of mutation p_mAnd a fitness function J (ω);

b) randomly generating an initial population consisting of omega;

c) repeating the following steps until a specified number of times is reached: evaluating the fitness of each individual (namely omega) in the current population; selecting parent individuals using roulette and crossing with a single point with a probability p_cAnd the probability p of mutation_mGenerating offspring; repairing each individual omega, sigma by greedy strategy_iω_iM, an individual omega one of which is 1_iChange to 0 to minimize J (ω) and repeat the process until Σ_iω_iIf m, if e_iω_i< m, an individual omega, one of which is 0_iInstead, 1, J (ω) is minimized and the process is repeated until Σ_iω_i＝m；

d) And outputting wopt as an optimal solution, and selecting a neural network corresponding to wopt which is 1 according to wopt to form final neural network integration.

The above description is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, several modifications and variations can be made without departing from the technical principle of the present invention, and these modifications and variations should also be regarded as the protection scope of the present invention.

Claims (9)

1. A ship target identification method based on a multilayer convolutional neural network is characterized by comprising the following steps:

s1, constructing a ship sample library by using the existing images, parameters and model data, and improving the ship sample library by detecting target data acquisition;

s2, forming a visible light/infrared and two-dimensional/three-dimensional integrated ship feature knowledge base for classification and identification of ship targets through identification training of a ship sample base under the framework of a convolutional neural network;

the method comprises the following steps: establishing a ship target feature training architecture based on a deep convolutional neural network, wherein the ship target feature training architecture comprises a feature sharing CNN layer, a finely-adjustable CNN layer, a region extraction model, an ROI pooling layer and a classification regression full-connection layer;

the shared characteristic CNN layer is a part of an existing deep convolutional neural network classification model with the last full connection layer removed, pre-training is carried out on a VOC classification data set or an ImageNet classification data set before the training of a ship target detection and identification model is carried out to improve the extraction capability of image characteristics, and then the parameter of the trained classification model with the full connection layer removed is utilized to carry out parameter initialization on the shared characteristic CNN layer;

the fine-tuning CNN layer is used for online incremental learning;

the region extraction model is a convolutional neural network, after a CNN layer and a finely-tunable CNN layer are shared, a convolutional layer and two parallel neural networks of full connection layers are constructed, and the extraction of a target region possibly existing in the ship to be detected is realized by sharing the characteristics of the CNN layer and the finely-tunable CNN layer;

the ROI pooling layer is used for outputting a feature vector with fixed dimensionality after ROI pooling;

the training of the ship target detection and recognition model comprises 4 steps; the first stage is as follows: initializing parameters of a shared characteristic CNN layer by using a classification model pre-trained on a sample library, initializing parameters of weights and bias items for a fine-tunable CNN layer and a region extraction model by using Gaussian normal distribution with zero-mean variance sigma, and fine-tuning the region extraction neural network model by using visible light or infrared training set image data; and a second stage: using the region extracted by the region extraction model in the first stage to extract and train a shared feature CNN layer and a fine-adjustable CNN layer, and simultaneously initializing parameters of the shared feature CNN layer by using a classification model pre-trained on a Voc2012 or ImageNet data set; and a third stage: using the shared characteristic CNN layer and the finely adjustable CNN layer parameters trained in the second stage, re-initializing the shared characteristic CNN layer and the finely adjustable CNN layer parameters of the region extraction model, then fixing the shared characteristic CNN layer and the finely adjustable CNN layer parameters and finely adjusting the region extraction neural network model; a fourth stage: freezing the parameter of the convolutional layer of the region extraction neural network model in the third stage, extracting the region, and then sharing the characteristic CNN layer and finely adjusting the parameter of the CNN layer;

s3, collecting visible light or infrared video data of the marine ship target in real time with high resolution;

s4, detecting the marine ship target;

s5, carrying out simple classification on the large class of the ship by the rough classification of the ship target image, and reducing the workload of subsequent fine classification and identification of the ship target;

s6, developing fine classification recognition work of the ship target based on the deep neural network model finished by the ship target characteristic training, and accurately recognizing the type of the ship;

the identification of warships and civilian ships includes:

let the training set be { (x)₁，t₁)，……，(x_N，t_N) In which x_iIs an input, t_iIs the target output, N is the number of samples in the training set, and the output of the deep neural network integration is defined as:

where m is the number of neural networks, F_i(n) is the output of the ith deep neural network on the nth sample, and F (n) is the output of the deep neural network integrated on the nth sample, and the ResNet deep neural network is used for integrated learning, so that incremental learning of a new image data set is achieved, and the method comprises four modules: initial ResNet deep neural network integration, current ResNet deep neural network integration, replicated ResNet deep neural network integration, and new ReIntegrating sNet deep neural networks;

firstly, training initial ResNet deep neural network integration by utilizing a first warship and civil ship image data set to obtain current ResNet deep neural network integration; then, copying the current ResNet deep neural network integration to obtain a copied ResNet deep neural network integration; next, when a second warship and civil ship image data set arrives, training the copied ResNet deep neural network integration by using the data set to obtain a new ResNet deep neural network integration; finally, a selective negative correlation learning method is applied for selection.

2. The ship target identification method based on the multilayer convolutional neural network as claimed in claim 1, wherein S4, the ship target detection specifically includes: calculating by adopting an SLIC image segmentation algorithm, wherein the SLIC algorithm comprises the following specific steps:

initializing cluster center C by setting K super pixel point numbers_k＝[l_k，a_k，b_k，x_k，y_k]，k∈[1，K]：

Disturbing the center of the chaotic group in the neighborhood, and moving the clustering center to the position with the lowest gradient;

for the center of each cluster group, distributing optimal matching pixel points in adjacent areas surrounding the center of one cluster group according to a distance calculation rule;

calculating new population centers and residual errors;

convergence occurs until the residual error E is less than the threshold.

3. The ship target identification method based on the multilayer convolutional neural network as claimed in claim 1, wherein the image gradient calculation in S4 comprises:

G(x，y)＝||I(x+1，y)-I(x-1，y)||²+||I(x，y+1)-I(x，y-1)||² (1)；

where I (x, y) is a lab vector, representing the color vector for the pixel spatial location (x, y), and | is the L2 norm;

setting step length S to sqrt (N/K), N to total number of pixels and K to number of super pixels, distributing matching points to each clustering center in the neighborhood of 2S to 2S of the clustering center, calculating according to the set distance between each point and the clustering center, finally connecting each pixel point in the image with the nearest clustering center and covering the pixel point by the search domain of the clustering center, calculating new center after all the pixel points are connected with the nearest cluster center, wherein the new center is the average value of lab xy vectors belonging to the cluster, and repeating the process of connecting the pixel points with the nearest cluster center and recalculating the cluster center until convergence.

4. The method of claim 1, wherein in S4, the kernel of the HOG extraction process includes:

1) calculating the gradients of the horizontal coordinate and the vertical coordinate of the image, and calculating the gradient direction value of each pixel position according to the gradients;

2) uniformly dividing an image into small blocks according to spatial positions, counting histograms in gradient directions in the small blocks according to a set quantization standard to obtain feature vectors corresponding to the small blocks, and then connecting the feature vectors of all cells in one large block in series to obtain large block HOG features, wherein each large block is composed of m × m small blocks, and each small block is composed of n × n pixels.

5. The method for ship target identification based on multilayer convolutional neural network of claim 4, wherein after extracting features for each region in the training data, training a suitable classifier to complete the classification of the region, so as to achieve the purpose of excluding large regions of the sea surface and sky and locating the target in the region of interest.

6. The ship target identification method based on the multilayer convolutional neural network of claim 5, wherein S6 further comprises:

when the nth sample is used for training, the error function of the ith neural network is defined as:

wherein p is_i(n) is a penalty term, λ ∈ [0, 1 ]]Is a control parameter, called penalty parameter, for controlling a balance between the mean square error and a penalty term, penalty term p_i(n) can be defined as: p is a radical of_i(n)＝(F_i(n)-F(n))∑_j≠iF_j(n) -f (n), the penalty term makes the ith neural network inversely related to the rest of the neural networks in the integration, so as to achieve the difference between the individual neural networks, and when the lambda is 0, the integration of the obtained neural networks is equivalent to independently training a group of neural networks; with the increasing of λ, the emphasis of training will be gradually adjusted to make the individuals have difference.

7. The ship target identification method based on the multilayer convolutional neural network of claim 5, wherein S6 further comprises:

a) initializing a neural network integration ens1 of size m;

b) training ens1 by negative correlation learning with S1;

c) from the beginning to the end of the second sub data set, i.e., T2, …, T, the following loop process is performed: copying the current neural network integrated enst-1 to obtain a copy, and recording the copy as an enscopy; training the enscopy with St through negative correlation learning; combining the enst-1 and the enscopy, and recording the combined result as the enscomb; selecting m neural networks to form an integrated enst by using a selection process for the enscomb;

d) outputting a final neural network integration ensT;

aiming at the selection process in the step c), the genetic algorithm is used for selecting the enscomb, according to the selective negative correlation ensemble learning method, m neural networks are selected from the enscomb with the size of 2m in the selection process to form the enst, and the selection process can be formalized into the following optimization problem with constraints:

wherein, J (ω)) Is a predefined objective function, and the quality of the design of the objective function is directly related to the generalization performance of the selected neural network integration, omega_iIs a 2 m-dimensional binary vector, ω_i1 means that the ith neural network is selected to form the enst, ω_iIf 0, the ith neural network is deleted;

solving using a genetic algorithm includes:

a) initialization algorithm setting: integration size m, population size of GA pop _ size, probability of crossover p_cProbability of mutation p_mAnd a fitness function J (ω);

b) randomly generating an initial population consisting of omega;

c) repeating the following steps until a specified number of times is reached: evaluating the fitness of each individual in the current population; selecting parent individuals using roulette and crossing with a single point with a probability p_cAnd the probability p of mutation_mGenerating offspring; repairing each individual omega, sigma by greedy strategy_iω_iM, an individual omega one of which is 1_iChange to 0 to minimize J (ω) and repeat the process until Σ_iω_iIf m, if e_iω_i< m, an individual omega, one of which is 0_iInstead, 1, J (ω) is minimized and the process is repeated until Σ_iω_i＝m；

d) And outputting wopt as an optimal solution, and selecting a neural network corresponding to wopt which is 1 according to wopt to form final neural network integration.

8. The ship target identification method based on the multilayer convolutional neural network as claimed in claim 1, comprising: the fine-tuning CNN layer is used for online incremental learning, when the target detection of the ship is performed online, if the detected ship target meets the labeling condition, the target detection model needs to be subjected to online incremental learning, the shared characteristic CNN layer, the region extraction model, the subsequent ROI pooling layer and the full connection layer of the fine-tuning CNN layer need to be fixed in the online incremental learning process, and then the fine-tuning CNN layer is subjected to parameter fine tuning through classification errors.

9. The ship target identification method based on the multilayer convolutional neural network of claim 1, wherein S2 further comprises:

in the model inference stage, if the ship identified by the ship target detection and identification model meets the sample labeling condition, online incremental learning on the finely adjustable CNN layer of the ship target detection and identification model by using the sample is required, which includes: the first step is as follows: judging the sample by using a ship target identification threshold value predicted by a target detection identification model, and if the threshold value is higher than sigma₁Or below σ₂Online incremental learning is not required; conversely, if the threshold is lower than σ₁And is higher than sigma₂One online incremental learning is required; the second step is that: and performing primary parameter fine adjustment on the fine-adjustable CNN layer of the target detection model by using the image data meeting the sample labeling condition.

Images