CN113705488A - Remote sensing image fine-grained airplane identification method based on local segmentation and feature fusion - Google Patents
Remote sensing image fine-grained airplane identification method based on local segmentation and feature fusion Download PDFInfo
- Publication number
- CN113705488A CN113705488A CN202111014277.6A CN202111014277A CN113705488A CN 113705488 A CN113705488 A CN 113705488A CN 202111014277 A CN202111014277 A CN 202111014277A CN 113705488 A CN113705488 A CN 113705488A
- Authority
- CN
- China
- Prior art keywords
- airplane
- image
- aircraft
- grained
- fine
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000004927 fusion Effects 0.000 title claims abstract description 64
- 238000000034 method Methods 0.000 title claims abstract description 50
- 230000011218 segmentation Effects 0.000 title claims abstract description 31
- 239000013598 vector Substances 0.000 claims description 42
- 230000000007 visual effect Effects 0.000 claims description 27
- 238000001514 detection method Methods 0.000 claims description 21
- 238000010586 diagram Methods 0.000 claims description 20
- 239000011159 matrix material Substances 0.000 claims description 18
- 238000012549 training Methods 0.000 claims description 18
- 238000000605 extraction Methods 0.000 claims description 14
- 238000005070 sampling Methods 0.000 claims description 8
- 230000006870 function Effects 0.000 claims description 5
- 238000012360 testing method Methods 0.000 claims description 5
- 238000013527 convolutional neural network Methods 0.000 claims description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000013145 classification model Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Computational Linguistics (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a remote sensing image fine-grained airplane identification method based on local segmentation and feature fusion, which comprises the steps of detecting key points in an airplane image, and calibrating the airplane direction in the airplane image according to the key points; cutting the calibrated aircraft image according to the key points to obtain an aircraft component view map; then, constructing a fine-grained identification model based on local feature fusion; and finally, combining the calibrated airplane image with the airplane component view map, classifying the airplane in a fine granularity mode through a fine granularity identification model based on local feature fusion, and outputting an airplane identification result. Compared with the prior art, the method has stronger generalization capability, can process the multi-resolution remote sensing image, has higher robustness, can process various noise interferences, and has higher identification accuracy.
Description
Technical Field
The invention relates to the technical field of digital image processing and artificial intelligence, relates to fine-grained target recognition and anomaly detection, aims at the problem of large-scale fine-grained airplane classification in a multi-resolution remote sensing image, and particularly relates to a remote sensing image fine-grained airplane recognition method based on local segmentation and feature fusion.
Background
The remote sensing image plays an important role in the real-time monitoring and management of the airport in the accurate detection and identification of the airplane in the airport.
In recent years, deep learning technology is rapidly developed in the field of computer vision, the fine-grained identification problem is paid more and more attention, and extremely high accuracy is achieved on some natural image data sets. However, at present, research on airplane targets in remote sensing images mainly focuses on airplane detection, a small amount of documents focus on airplane fine-grained classification tasks, the number of classes of the classification tasks is very limited, the difference between the number of classes of the classification tasks and the number of classes of targets in natural image data sets is large, and actual use requirements cannot be met.
Compared with the problem of fine-grained identification for a natural image data set, the fine-grained target classification for the satellite remote sensing image is more difficult. On one hand, the resolution difference of the remote sensing images is large, the resolution of a large number of images is low, and meanwhile, the size of the airplane target is small, so that the local features of the airplane in the images are lost. However, in the current fine-grained recognition model algorithms such as Trilinear Attention Sampling Network (TASN), Three Branch and Multi-Scale Learning Network (TBMSL-Net), fine-grained classification is usually implemented by extracting fine feature differences between targets in an image by using an Attention mechanism, which results in poor performance of the algorithms in remote sensing image data. On the other hand, the number of the airplane types is large, airplanes of the same model have a large number of sub-types, and the differences among the sub-types are small, so that extremely high requirements are provided for the capability of extracting effective features of a fine-grained classification model, and the realization is difficult.
Therefore, in order to solve the problem of low accuracy of fine-grained classification of the airplane target in the multi-resolution remote sensing image, a remote sensing image fine-grained airplane identification method based on local segmentation and feature fusion is provided.
Disclosure of Invention
The purpose of the invention is as follows: by utilizing the appearance shape characteristics of the airplane, a new fine-grained identification method for the remote sensing image is provided, and the accuracy of classifying large-scale fine-grained airplane targets in the multi-resolution remote sensing image is greatly improved.
The technical scheme is as follows: in order to achieve the above object, the present invention provides a fine-grained aircraft identification method based on local segmentation and feature fusion, comprising the following steps:
and 4, combining the calibrated airplane image and the airplane component visual field image, and performing fine-grained classification on the airplane through the fine-grained identification model based on local feature fusion to obtain an airplane identification result.
Further, in one implementation, the step 1 includes:
step 1-1, detecting key points in the airplane image by using a trained key point detection model, wherein the key points comprise: the aircraft comprises an aircraft head, an aircraft tail, a right wing end, a left wing end and a wing joint;
and 1-2, solving an included angle between the direction of the airplane and the right upper side according to the key points, generating a rotation matrix, calibrating the direction of the airplane head to the right upper side of the airplane image according to the rotation matrix, and obtaining the calibrated airplane image.
Further, in one implementation, the step 1-1 includes:
a Stacked Hourglass convolutional neural Network (Stacked Hourglass Network) is adopted as a key point detection model, the key point detection model adopts 4 stacks (Stack), and each Stack adopts 4 times of down sampling and 4 times of up sampling;
in the training stage of the key point detection model, converting the coordinate points of the key points into a thermodynamic diagram with Gaussian distribution as a training label; the coordinate points of the key points include: aircraft nose (x)1,y1) Right wing end (x)2,y2) Aircraft tail (x)3,y3) Left wing end (x)4,y4) And wing joint (x)5,y5)。
Further, in one implementation, the step 1-2 includes:
step 1-2-1, obtaining an airplane direction vector (x) according to coordinate points of the airplane head and the airplane tail in the key points3-x1,y3-y1) To obtain the included angle theta between the airplane direction and the vertical axis direction of the image coordinate system,
obtaining a rotation matrix:
wherein x isiTo rotate the front abscissa, yiIs a rotation front ordinate, xi' is the abscissa after rotation, yi' is the ordinate after rotation. Specifically, in the present invention, the vertical axis direction of the image coordinate system is the target direction for performing the image calibration of the airplane.
And 1-2-2, multiplying the aircraft image by a rotation matrix to obtain a calibrated aircraft image, so that the direction of the aircraft is consistent with the direction of a vertical axis of a coordinate system.
Further, in one implementation, the step 2 includes:
and cutting the calibrated airplane image into an airplane component visual field image according to the key points, wherein the airplane component visual field image comprises images of an airplane fuselage portion, an airplane wing portion, an airplane left upper portion, an airplane right upper portion, an airplane left lower portion and an airplane right lower portion, and the fuselage comprises an airplane tail.
Further, in an implementation manner, the step 2 includes:
the coordinates of the post-calibration key points include the post-calibration aircraft nose (x'1，y’1) Calibrated rear right wing (x'2，y'2) Calibrated aft aircraft tail (x'3,y'3) Calibrated rear aircraft left wing (x'4,y'4) And calibrated aft wing connection (x'5,y'5) Cutting the calibrated airplane image into an airplane fuselage part, an airplane wing part, an airplane left upper part, an airplane right upper part, an airplane left lower part and an airplane right lower part;
the coordinates of the visual field diagram of the aircraft fuselage portion are as follows:
[((x’1+x'4)/2,y’1),((x’1+x'2)/2,y’1),((x’1+x'2)/2,y'3),((x’1+x'4)/2,y'3)]，
the coordinates of the visual field diagram of the wing part of the airplane are as follows:
[(x'4,(y’1+y'4)/2),(x'2,(y’1+y'4)/2),(x'2,(y'3+y'4)/2),(x'4,(y'3+y'4)/2)]，
the coordinates of the upper left view map of the airplane are as follows:
[(x'4,y’1),(x’1,y’1),(x’1,y'4),(x'4,y'4)]
the coordinates of the upper right view of the aircraft are as follows:
[(x’1,y’1),(x'2,y’1),(x'2,y'2),(x’1,y'2)]
the coordinates of the aircraft lower left view map are as follows:
[(x'4,y'5),(x'5,y'5),(x'5,y'3),(x'4,y'3)]
the coordinates of the aircraft lower right view map are as follows:
[(x'5，y'5)，(x'2，y'5)，(x'2，y'3)，(x'5，y'3)]。
further, in one implementation, the step 3 includes:
step 3-1, adopting a ResNet152 convolution part as an image feature extraction basic model of the fine-grained identification model based on local feature fusion;
step 3-2, extracting the output of the last convolution layer output by the basic model through the image characteristics to be an image characteristic matrix, and flattening the image characteristic matrix to generate a characteristic vector corresponding to the image;
3-3, sequentially sending the calibrated airplane image and the airplane component view image into an image feature extraction basic model to obtain a feature vector of the calibrated image and a feature vector of the airplane component view image, and generating a first airplane category prediction vector through the feature vector of the calibrated image and a full-link layer;
step 3-4, splicing the characteristic vector of the visual field diagram of the aircraft component with the characteristic vector of the calibration image, obtaining a second aircraft category prediction vector through a full-link layer, and finally obtaining a fine-grained identification model based on local feature fusion;
3-5, pre-training the image feature extraction basic model convolution layer by using ImageNet, and initializing parameters by adopting random distribution in the full-link layer; in the invention, the convolution part parameters of the image feature extraction basic model are shared, and the parameters of two full-connection layers are not shared.
3-6, in the training stage, adopting different learning rates for the image feature extraction basic model and the full-connection layer part;
and 3-7, in the training stage, the loss function is the weighted sum of the cross entropies of the two parts, wherein the first part is the cross entropy of the first airplane type prediction vector and the label, and the second part is the cross entropy of the second airplane type prediction vector and the label.
And 3-8, selecting the second airplane category prediction vector prediction result as a final airplane classification result in the testing stage.
Further, in one implementation, the step 4 includes:
step 4-1, normalizing the size of the airplane image and the size of the airplane component visual field graph to obtain a normalized image; specifically, in the present invention, the sizes of the aircraft image and the aircraft component view map are normalized to 224 × 224, and the operation of step 4-2 is performed after the mean value is subtracted.
And 4-2, sequentially sending the normalized images to a fine-grained identification model based on local feature fusion to obtain a fine-grained classification result.
In the method, firstly, the key points of the airplane are positioned based on a key point detection model; calibrating the direction of the airplane according to the key points, and normalizing the direction of the airplane; cutting the calibrated aircraft image according to the key points to obtain a view map of the aircraft key components; sequentially sending the original aircraft image and the visual field image of the aircraft component into a fine-grained identification model based on local feature fusion to obtain a final fine-grained classification result; the method comprises the steps that image features of an original image and cut visual field images are respectively extracted by a fine-grained recognition model based on local feature fusion, the visual field image features are spliced to obtain fusion features, a model training stage adopts a weighted cross entropy loss function to simultaneously calculate an original image feature prediction result and a fusion feature to predict a classification result and a labeled cross entropy, and a testing stage adopts the fusion feature to predict the classification result as a final result. Compared with the prior art, the invention has the following beneficial effects:
(1) the fine-grained identification model based on local feature fusion constructed by the method has stronger generalization capability and can process multi-resolution remote sensing images; (2) the robustness of the fine-grained identification model based on local feature fusion constructed by the method is improved, and noise interference such as cloud and fog shielding, low resolution and the like in the remote sensing image can be processed; (3) the model has the capability of identifying large-scale fine-grained target categories, and compared with the existing fine-grained identification method, the identification accuracy can be remarkably improved.
Drawings
In order to more clearly illustrate the technical solution of the present invention, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious to those skilled in the art that other drawings can be obtained based on these drawings without creative efforts.
FIG. 1a is a schematic diagram of a workflow framework of a remote sensing image fine-grained aircraft identification method based on local segmentation and feature fusion according to an embodiment of the present invention;
FIG. 1b is a schematic diagram of a specific work flow of a remote sensing image fine-grained aircraft identification method based on local segmentation and feature fusion according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a structure of a keypoint detection model in a remote sensing image fine-grained aircraft identification method based on local segmentation and feature fusion, which is provided in the embodiment of the present invention;
FIG. 3 is a schematic diagram of image direction calibration and view map cutting in a remote sensing image fine-grained aircraft identification method based on local segmentation and feature fusion, which is provided in the embodiments of the present invention;
fig. 4 is a schematic diagram of a local feature fusion-based fine-grained identification model architecture of a local segmentation and feature fusion-based remote sensing image fine-grained aircraft identification method according to an embodiment of the present invention.
Detailed Description
In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in further detail below.
The embodiment of the invention discloses a remote sensing image fine-grained airplane identification method based on local segmentation and feature fusion.
As shown in fig. 1a, a schematic view of a workflow framework of a remote sensing image fine-grained aircraft identification method based on local segmentation and feature fusion provided in this embodiment is provided. The fine-grained aircraft identification method based on local segmentation and feature fusion provided by the embodiment comprises the following steps:
and 4, combining the calibrated airplane image and the airplane component visual field image, and performing fine-grained classification on the airplane through the fine-grained identification model based on local feature fusion to obtain an airplane identification result.
As shown in fig. 1b, a specific work flow diagram of the remote sensing image fine-grained aircraft identification method based on local segmentation and feature fusion provided in this embodiment is shown; each specific workflow diagram corresponds to the workflow frame diagram in fig. 1a one-to-one, that is, corresponds to the processing of the original image through steps 1 to 4 in the method.
In the fine-grained aircraft identification method based on local segmentation and feature fusion described in this embodiment, the step 1 includes: step 1-1, detecting key points in the airplane image by using a trained key point detection model, wherein the key points comprise: the aircraft comprises an aircraft head, an aircraft tail, a right wing end, a left wing end and a wing joint;
and 1-2, solving an included angle between the direction of the airplane and the right upper side according to the key points, generating a rotation matrix, calibrating the direction of the airplane to the right upper side of the airplane image according to the rotation matrix, and obtaining the calibrated airplane image.
In the fine-grained aircraft identification method based on local segmentation and feature fusion described in this embodiment, the step 1-1 includes: a Stacked Hourglass convolutional neural Network (Stacked Hourglass Network) is adopted as a key point detection model, the key point detection model adopts 4 stacks (Stack), 4 times of down sampling and 4 times of up sampling are adopted in each basic unit (basic Stacked Hourglass Network model architecture setting). The depth of the 2-stack Hourglass Network is insufficient, the key point detection is difficult to realize, the experimental effect of the 6-8-stack Network is basically consistent with that of the 4-stack Network, but the model deduces for a longer time, so the 4-stack Network is selected. The image size of the input model is 224 multiplied by 224, and better global characteristics can be obtained by 4 times of down sampling; as shown in fig. 2, the model structure of the keypoint detection model is shown.
In the training stage of the key point detection model, converting the coordinate points of the key points into a thermodynamic diagram with Gaussian distribution as a training label; the coordinate points of the key points include: aircraft nose (x)1,y1) Right wing end (x)2,y2) Aircraft tail (x)3,y3) Left wing end (x)4,y4) And wing joint (x)5,y5). As shown in fig. 3, 5 points in the airplane image before direction calibration are the key points, wherein the uppermost point is the airplane head (x)1，y1) The leftmost point, the left wing tip (x)4，y4) The rightmost point, the right wing end (x)2，y2) The left wing end (x)4，y4) And the right wing end (x)2，y2) Point in between, i.e. the wing joint (x)5，y5) The lowest point is the tail of the aircraft (x)3，y3)。
In the fine-grained aircraft identification method based on local segmentation and feature fusion described in this embodiment, the step 1-2 includes: step 1-2-1, obtaining an airplane direction vector (x) according to coordinate points of the airplane head and the airplane tail in the key points3-x1,y3-y1) To obtain the included angle theta between the airplane direction and the vertical axis direction of the image coordinate system,
obtaining a rotation matrix:
wherein x isiTo rotate the front abscissa, yiIs the ordinate, x 'before rotation'iIs the abscissa after rotation, y'iIs the ordinate after rotation. Specifically, in this embodiment, the vertical axis direction of the image coordinate system is the target direction for performing the image calibration of the airplane.
And 1-2-2, multiplying the aircraft image by a rotation matrix to obtain a calibrated aircraft image, so that the direction of the aircraft is consistent with the direction of a vertical axis of a coordinate system.
In the fine-grained aircraft identification method based on local segmentation and feature fusion described in this embodiment, the step 2 includes: and cutting the calibrated airplane image into an airplane component visual field image according to the key points, wherein the airplane component visual field image comprises images of an airplane fuselage portion, an airplane wing portion, an airplane left upper portion, an airplane right upper portion, an airplane left lower portion and an airplane right lower portion, and the fuselage comprises an airplane tail.
In the fine-grained aircraft identification method based on local segmentation and feature fusion described in this embodiment, the step 2 includes: the coordinates of the post-calibration key points include the post-calibration aircraft nose (x'1,y’1) Calibrated rear right wing (x'2,y'2) Calibrated aft aircraft tail (x'3,y'3) Calibrated rear aircraft left wing (x'4,y'4) And calibrated aft wing connection (x'5,y'5) As shown in FIG. 3, the top point in the aircraft image after the direction calibration is the calibrated aircraft head (x'1,y’1) The leftmost point is the calibrated rear aircraft left wing (x'4,y'4) The rightmost point is the calibrated rear right wing (x'2,y'2) The calibrated rear left wing (x'4,y'4) And calibrated rear right wing (x'2，y'2) The point in between is the calibrated aft wing connection (x'5,y'5) The lowest point is after calibrationAircraft tail (x'3,y'3). Cutting the calibrated airplane image into an airplane body part, an airplane wing part, an airplane left upper part, an airplane right upper part, an airplane left lower part and an airplane right lower part according to the key point coordinates; as shown in the cut-out view of the field of view in fig. 3, the cut views numbered 5, 6, 1, 2, 3 and 4 show, in sequence, an aircraft fuselage section, an aircraft wing section, an aircraft upper left section, an aircraft upper right section, an aircraft lower left section and an aircraft lower right section.
The coordinates of the visual field diagram of the aircraft fuselage portion are as follows:
[((x’1+x'4)/2,y’1),((x’1+x'2)/2,y’1),((x’1+x'2)/2,y'3),((x’1+x'4)/2,y'3)]，
the coordinates of the visual field diagram of the wing part of the airplane are as follows:
[(x'4,(y’1+y'4)/2),(x'2,(y’1+y'4)/2),(x'2,(y'3+y'4)/2),(x'4,(y'3+y'4)/2)]，
the coordinates of the upper left view map of the airplane are as follows:
[(x'4,y’1),(x’1,y’1),(x’1,y'4),(x'4,y'4)]
the coordinates of the upper right view of the aircraft are as follows:
[(x’1,y’1),(x'2,y’1),(x'2,y'2),(x’1,y'2)]
the coordinates of the aircraft lower left view map are as follows:
[(x'4,y'5),(x'5，y'5)，(x'5，y'3)，(x'4，y'3)]
the coordinates of the aircraft lower right view map are as follows:
[(x'5，y'5)，(x'2，y'5)，(x'2，y'3)，(x'5，y'3)]。
in the fine-grained aircraft identification method based on local segmentation and feature fusion described in this embodiment, the step 3 includes: step 3-1, adopting ResNet152 as an image feature extraction basic model of the fine-grained identification model based on local feature fusion;
step 3-2, extracting the output of the last convolution layer output by the basic model through the image characteristics to be an image characteristic matrix, and flattening the image characteristic matrix to generate a characteristic vector corresponding to the image;
3-3, sequentially sending the calibrated airplane image and the airplane component view image into an image feature extraction basic model to obtain a feature vector of the calibrated image and a feature vector of the airplane component view image, and generating a first airplane category prediction vector through the feature vector of the calibrated image and a full-link layer;
step 3-4, splicing the characteristic vector of the visual field diagram of the aircraft component with the characteristic vector of the calibration image, and obtaining a second aircraft category prediction vector through a full-link layer;
3-5, pre-training the image feature extraction basic model convolution layer by using ImageNet, and initializing parameters by adopting random distribution in the full-link layer; in this embodiment, the convolution part of the image feature extraction basic model is shared, and the two full-connected layer parameters are not shared.
3-6, in the training stage, adopting different learning rates for the image feature extraction basic model and the full-connection layer part; specifically, in this embodiment, the convolutional layer learning rate is 0.02, and the fully-connected layer learning rate is 0.04;
and 3-7, in a training stage, performing weighted summation on a loss function by using two parts of cross entropies, wherein the first part is the cross entropy of the first airplane type prediction vector and the label, and the second part is the cross entropy of the second airplane type prediction vector and the label, and finally obtaining a fine-grained identification model based on local feature fusion. Specifically, in this embodiment, the fine-grained identification model architecture based on local feature fusion is shown in fig. 4, where the calibration image and the aircraft component view map are sequentially sent to the same model convolution portion (ResNet152) to extract global features and component features, and then the features are flattened to generate feature vectors. And the global feature vector passes through the full-link layer separately to generate a second airplane category prediction vector. And splicing the global feature vector and the component feature vector, and generating a first airplane category prediction vector through a full-link layer.
And 3-8, selecting the second airplane category prediction vector prediction result as a final airplane classification result in the testing stage.
In the fine-grained aircraft identification method based on local segmentation and feature fusion described in this embodiment, the step 4 includes: step 4-1, normalizing the size of the airplane image and the size of the airplane component visual field graph to obtain a normalized image; specifically, in this embodiment, the sizes of the aircraft image and the aircraft component view map are normalized to 224 × 224, and the operation of step 4-2 is performed after the mean value of the training set images is subtracted.
And 4-2, sequentially sending the normalized images to a fine-grained identification model based on local feature fusion to obtain a fine-grained classification result.
In the method, firstly, the key points of the airplane are positioned based on a key point detection model; calibrating the direction of the airplane according to the key points, and normalizing the direction of the airplane; cutting the calibrated aircraft image according to the key points to obtain a view map of the aircraft key components; sequentially sending the original aircraft image and the visual field image of the aircraft component into a fine-grained identification model based on local feature fusion to obtain a final fine-grained classification result; the method comprises the steps that image features of an original image and cut visual field images are respectively extracted by a fine-grained recognition model based on local feature fusion, the visual field image features are spliced to obtain fusion features, a model training stage adopts a weighted cross entropy loss function to simultaneously calculate an original image feature prediction result and a fusion feature to predict a classification result and a labeled cross entropy, and a testing stage adopts the fusion feature to predict the classification result as a final result. Compared with the prior art, the invention has the following beneficial effects:
(1) the fine-grained identification model based on local feature fusion constructed by the method has stronger generalization capability and can process multi-resolution remote sensing images; (2) the robustness of the fine-grained identification model based on local feature fusion constructed by the method is improved, and noise interference such as cloud and fog shielding, low resolution and the like in the remote sensing image can be processed; (3) the model has the capability of identifying large-scale fine-grained target categories, and compared with the existing fine-grained identification method, the identification accuracy can be remarkably improved.
In a specific implementation, the present invention further provides a computer storage medium, where the computer storage medium may store a program, and when the program is executed, the program may include some or all of the steps in each embodiment of the remote sensing image fine-grained aircraft identification method based on local segmentation and feature fusion provided by the present invention. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), a Random Access Memory (RAM), or the like.
Those skilled in the art will readily appreciate that the techniques of the embodiments of the present invention may be implemented as software plus a required general purpose hardware platform. Based on such understanding, the technical solutions in the embodiments of the present invention may be essentially or partially implemented in the form of a software product, which may be stored in a storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method according to the embodiments or some parts of the embodiments.
The same and similar parts in the various embodiments in this specification may be referred to each other. The above-described embodiments of the present invention should not be construed as limiting the scope of the present invention.
Claims (8)
1. The remote sensing image fine-grained airplane identification method based on local segmentation and feature fusion is characterized by comprising the following steps of:
step 1, detecting key points in airplane images, and sequentially calibrating the airplane direction in each airplane image according to the key points;
step 2, cutting the calibrated aircraft image according to the key points to obtain an aircraft component view map;
step 3, constructing a fine-grained identification model based on local feature fusion;
and 4, combining the calibrated airplane image and the airplane component visual field image, and performing fine-grained classification on the airplane through the fine-grained identification model based on local feature fusion to obtain an airplane identification result.
2. The remote sensing image fine-grained aircraft identification method based on local segmentation and feature fusion according to claim 1, wherein the step 1 comprises the following steps:
step 1-1, detecting key points in the airplane image by using a trained key point detection model, wherein the key points comprise: the aircraft comprises an aircraft head, an aircraft tail, a right wing end, a left wing end and a wing joint;
and 1-2, solving an included angle between the direction of the airplane and the right upper side according to the key points, generating a rotation matrix, calibrating the direction of the airplane head to the right upper side of the airplane image according to the rotation matrix, and obtaining the calibrated airplane image.
3. The remote sensing image fine-grained aircraft identification method based on local segmentation and feature fusion as claimed in claim 2, wherein the step 1-1 comprises:
a stacked hourglass-shaped convolutional neural network is used as a key point detection model, the key point detection model adopts 4 stacks, each stack adopts 4 times of down sampling and 4 times of up sampling;
in the training stage of the key point detection model, converting the coordinate points of the key points into a thermodynamic diagram with Gaussian distribution as a training label; the coordinate points of the key points include: aircraft nose (x)1,y1) Right wing end (x)2,y2) Aircraft tail (x)3,y3) Left wing end (x)4,y4) And wing joint (x)5,y5)。
4. The remote sensing image fine-grained aircraft identification method based on local segmentation and feature fusion as claimed in claim 2, wherein the steps 1-2 comprise:
step 1-2-1, obtaining an airplane direction vector (x) according to coordinate points of the airplane head and the airplane tail in the key points3-x1,y3-y1) To obtain the included angle theta between the airplane direction and the vertical axis direction of the image coordinate system,
obtaining a rotation matrix:
wherein x isiTo rotate the front abscissa, yiIs the ordinate, x 'before rotation'iIs the abscissa after rotation, y'iIs the vertical coordinate after rotation;
and 1-2-2, multiplying the aircraft image by a rotation matrix to obtain a calibrated aircraft image, so that the direction of the aircraft is consistent with the direction of a vertical axis of a coordinate system.
5. The remote sensing image fine-grained aircraft identification method based on local segmentation and feature fusion as claimed in claim 1, wherein the step 2 comprises:
and cutting the calibrated airplane image into an airplane component visual field image according to the key points, wherein the airplane component visual field image comprises images of an airplane fuselage portion, an airplane wing portion, an airplane left upper portion, an airplane right upper portion, an airplane left lower portion and an airplane right lower portion, and the fuselage comprises an airplane tail.
6. The remote sensing image fine-grained aircraft identification method based on local segmentation and feature fusion as claimed in claim 1, wherein the step 2 comprises:
the coordinates of the calibrated keypoints includeQuasi-rear aircraft head (x'1,y′1) Calibrated rear right wing (x'2,y′2) Calibrated aft aircraft tail (x'3,y′3) Calibrated rear aircraft left wing (x'4,y′4) And calibrated aft wing connection (x'5,y′5) Cutting the calibrated airplane image into an airplane fuselage part, an airplane wing part, an airplane left upper part, an airplane right upper part, an airplane left lower part and an airplane right lower part;
the coordinates of the visual field diagram of the aircraft fuselage portion are as follows:
[((x′1+x′4)/2,y′1),((x′1+x′2)/2,y′1),((x′1+x′2)/2,y′3),((x′1+x′4)/2,y′3)]，
the coordinates of the visual field diagram of the wing part of the airplane are as follows:
[(x′4,(y′1+y′4)/2),(x′2,(y′1+y′4)/2),(x′2,(y′3+y′4)/2),(x′4,(y′3+y′4)/2)]，
the coordinates of the upper left view map of the airplane are as follows:
[(x′4,y′1),(x′1，y′1)，(x′1，y′4)，(x′4,y′4)]
the coordinates of the upper right view of the aircraft are as follows:
[(x′1,y′1),(x′2，y′1)，(x′2,y′2),(x′1,y′2)]
the coordinates of the aircraft lower left view map are as follows:
[(x′4,y′5),(x′5,y′5),(x′5,y′3),(x′4,y′3)]
the coordinates of the aircraft lower right view map are as follows:
[(x′5,y′5),(x′2,y′5),(x′2,y′3),(x′5,y′3)]。
7. the remote sensing image fine-grained aircraft identification method based on local segmentation and feature fusion as claimed in claim 1, wherein the step 3 comprises:
step 3-1, adopting a ResNet152 convolution part as an image feature extraction basic model of the fine-grained identification model based on local feature fusion;
step 3-2, extracting the output of the last convolution layer output by the basic model through the image characteristics to be an image characteristic matrix, and flattening the image characteristic matrix to generate a characteristic vector corresponding to the image;
3-3, sequentially sending the calibrated airplane image and the airplane component view image into an image feature extraction basic model to obtain a feature vector of the calibrated image and a feature vector of the airplane component view image, and generating a first airplane category prediction vector through the feature vector of the calibrated image and a full-link layer;
step 3-4, splicing the characteristic vector of the visual field diagram of the aircraft component with the characteristic vector of the calibration image, obtaining a second aircraft category prediction vector through a full-link layer, and finally obtaining a fine-grained identification model based on local feature fusion;
3-5, pre-training the image feature extraction basic model convolution layer by using ImageNet, and initializing parameters by adopting random distribution in the full-link layer;
3-6, in the training stage, adopting different learning rates for the image feature extraction basic model and the full-connection layer part;
and 3-7, in the training stage, the loss function is the weighted sum of the cross entropies of the two parts, wherein the first part is the cross entropy of the first airplane type prediction vector and the label, and the second part is the cross entropy of the second airplane type prediction vector and the label.
And 3-8, selecting the second airplane category prediction vector prediction result as a final airplane classification result in the testing stage.
8. The remote sensing image fine-grained aircraft identification method based on local segmentation and feature fusion as claimed in claim 1, wherein the step 4 comprises:
step 4-1, normalizing the size of the airplane image and the size of the airplane component visual field graph to obtain a normalized image;
and 4-2, sequentially sending the normalized images to a fine-grained identification model based on local feature fusion to obtain a fine-grained classification result.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111014277.6A CN113705488A (en) | 2021-08-31 | 2021-08-31 | Remote sensing image fine-grained airplane identification method based on local segmentation and feature fusion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111014277.6A CN113705488A (en) | 2021-08-31 | 2021-08-31 | Remote sensing image fine-grained airplane identification method based on local segmentation and feature fusion |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113705488A true CN113705488A (en) | 2021-11-26 |
Family
ID=78658170
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111014277.6A Pending CN113705488A (en) | 2021-08-31 | 2021-08-31 | Remote sensing image fine-grained airplane identification method based on local segmentation and feature fusion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113705488A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109726659A (en) * | 2018-12-21 | 2019-05-07 | 北京达佳互联信息技术有限公司 | Detection method, device, electronic equipment and the readable medium of skeleton key point |
| CN110459301A (en) * | 2019-07-29 | 2019-11-15 | 清华大学 | Brain neuroblastoma surgical navigation method for registering based on thermodynamic chart and facial key point |
| CN110598601A (en) * | 2019-08-30 | 2019-12-20 | 电子科技大学 | Face 3D key point detection method and system based on distributed thermodynamic diagram |
| CN111274893A (en) * | 2020-01-14 | 2020-06-12 | 中国人民解放军国防科技大学 | Aircraft image fine-grained identification method based on component segmentation and feature fusion |
| CN112966555A (en) * | 2021-02-02 | 2021-06-15 | 武汉大学 | Remote sensing image airplane identification method based on deep learning and component prior |
-
2021
- 2021-08-31 CN CN202111014277.6A patent/CN113705488A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109726659A (en) * | 2018-12-21 | 2019-05-07 | 北京达佳互联信息技术有限公司 | Detection method, device, electronic equipment and the readable medium of skeleton key point |
| CN110459301A (en) * | 2019-07-29 | 2019-11-15 | 清华大学 | Brain neuroblastoma surgical navigation method for registering based on thermodynamic chart and facial key point |
| CN110598601A (en) * | 2019-08-30 | 2019-12-20 | 电子科技大学 | Face 3D key point detection method and system based on distributed thermodynamic diagram |
| CN111274893A (en) * | 2020-01-14 | 2020-06-12 | 中国人民解放军国防科技大学 | Aircraft image fine-grained identification method based on component segmentation and feature fusion |
| CN112966555A (en) * | 2021-02-02 | 2021-06-15 | 武汉大学 | Remote sensing image airplane identification method based on deep learning and component prior |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Dvornik et al. | On the importance of visual context for data augmentation in scene understanding | |
| CN109344701B (en) | Kinect-based dynamic gesture recognition method | |
| Afzal et al. | Cutting the error by half: Investigation of very deep cnn and advanced training strategies for document image classification | |
| Xu et al. | Hierarchical semantic propagation for object detection in remote sensing imagery | |
| CN113033537B (en) | Method, apparatus, device, medium and program product for training a model | |
| Xue et al. | Complex ISAR target recognition using deep adaptive learning | |
| Wang et al. | Small-object detection based on yolo and dense block via image super-resolution | |
| Li et al. | Remote sensing airport detection based on end-to-end deep transferable convolutional neural networks | |
| CN112418330A (en) | Improved SSD (solid State drive) -based high-precision detection method for small target object | |
| CN111563550B (en) | Sperm morphology detection method and device based on image technology | |
| Sun et al. | Traffic Sign Recognition Method Integrating Multi-Layer Features and Kernel Extreme Learning Machine Classifier. | |
| CN113095333A (en) | Unsupervised feature point detection method and unsupervised feature point detection device | |
| Guo et al. | D3-Net: Integrated multi-task convolutional neural network for water surface deblurring, dehazing and object detection | |
| CN114187506A (en) | Remote sensing image scene classification method of viewpoint-aware dynamic routing capsule network | |
| JP2023528181A (en) | Method, non-transitory computer-readable storage medium, and apparatus for searching an image database | |
| Patil et al. | Techniques of deep learning for image recognition | |
| Gan et al. | Object detection in remote sensing images with mask R-CNN | |
| CN111832463A (en) | Deep learning-based traffic sign detection method | |
| CN114612802A (en) | System and method for classifying fine granularity of ship target based on MBCNN | |
| CN113705488A (en) | Remote sensing image fine-grained airplane identification method based on local segmentation and feature fusion | |
| Zhang et al. | Multi-scale image recognition strategy based on convolutional neural network | |
| Xu et al. | Compressed YOLOv5 for Oriented Object Detection with Integrated Network Slimming and Knowledge Distillation | |
| Hamasaki et al. | Discrimination between handwritten and computer-generated texts using a distribution of patch-wise font features | |
| HS et al. | A novel method to recognize object in Images using Convolution Neural Networks | |
| Uzair et al. | Electronet: An enhanced model for small-scale object detection in electrical schematic diagrams |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |