CN109522807B - Satellite image recognition system and method based on self-generated features and electronic equipment - Google Patents

Satellite image recognition system and method based on self-generated features and electronic equipment Download PDF

Info

Publication number
CN109522807B
CN109522807B CN201811227460.2A CN201811227460A CN109522807B CN 109522807 B CN109522807 B CN 109522807B CN 201811227460 A CN201811227460 A CN 201811227460A CN 109522807 B CN109522807 B CN 109522807B
Authority
CN
China
Prior art keywords
layer
network
self
feature
generated
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201811227460.2A
Other languages
Chinese (zh)
Other versions
CN109522807A (en
Inventor
张昱航
阳文斯
叶可江
须成忠
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Institute of Advanced Technology of CAS
Original Assignee
Shenzhen Institute of Advanced Technology of CAS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Institute of Advanced Technology of CAS filed Critical Shenzhen Institute of Advanced Technology of CAS
Priority to CN201811227460.2A priority Critical patent/CN109522807B/en
Publication of CN109522807A publication Critical patent/CN109522807A/en
Application granted granted Critical
Publication of CN109522807B publication Critical patent/CN109522807B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/30—Noise filtering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/10—Terrestrial scenes
    • G06V20/13—Satellite images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
    • G06V10/469—Contour-based spatial representations, e.g. vector-coding
    • G06V10/473—Contour-based spatial representations, e.g. vector-coding using gradient analysis

Abstract

The application relates to a satellite image identification system and method based on self-generated features and electronic equipment. The system comprises: an FCN network construction module: the device is used for changing a first full connection layer and a second full connection layer of the VGG network into a first layer of convolution layer and a second layer of convolution layer respectively to construct a new FCN network; the self-generation characteristic network construction module: the method comprises the steps of constructing a self-generating feature network based on a GAN network; a self-generated feature calculation module: the characteristic graph is transmitted to the first layer of convolutional layer and the second layer of convolutional layer, and the first layer of convolutional layer and the second layer of convolutional layer use the characteristic graph to obtain the identification result of the input image. According to the method and the device, the latest image features are continuously calculated and updated through the automatic feature network generation, so that the data features are enhanced and the pixels are identified.

Description

Satellite image recognition system and method based on self-generated features and electronic equipment
Technical Field
The present application relates to image recognition technologies, and in particular, to a satellite image recognition system and method based on self-generated features, and an electronic device.
Background
With the continuous development of national economy, the utilization of national resources in China is increasingly urgent, but the land state of the national resources needs to be accurately mastered, and the traditional feature extraction method and the traditional machine learning algorithm cannot accurately judge the large-scale complicated image of the national resources.
In satellite image recognition, objects in the image need to be accurately segmented, for example: agricultural land, forest vegetation, buildings, rivers, roads, mountains, and the like. Accurately delineating the boundary can provide accurate technical parameters for the next survey. In the prior art, edge features of a satellite image are extracted by using a traditional principal component analysis method, a Scale-invariant feature transform (SIFT) method, a Haar-like feature method and the like, so as to achieve an identification effect. However, under deep learning, the features can be regarded as shallow features, and more effective high-level features cannot be captured, so that much useful information is lost and cannot be utilized in the recognition stage.
Meanwhile, the existing document 1[ Lin M, Chen Q, Yan s.network in Network [ J ]. arXiv preprintiv: 1312.4400,2013 ] has proved that converting the last two fully-connected layers of the current Convolutional Neural Network (CNN) into convolutional layers can not only reduce more parameters to optimize the Network, but also realize the recognition effect of pixels to pixels (PTP hereinafter), which has very important significance in precise satellite image recognition.
In another part of technologies, CNN networks are used for identification, but CNN networks are more suitable for identification of object classes and are not suitable for such dense identification, that is, PTP-level identification, and the effect is poor. And the operation network occupies long time of GPU, and has low precision and the like. Therefore, an FCN (full Convolutional Network) Network is used, but even though dense pixel identification can be performed in the FCN Network, the FCN Network is not suitable for satellite image identification because satellite images are not optimized.
In addition, a great difficulty faced by geographic data is that the satellite images require pixel-specific identification, and because the contours are accurately marked, manual pixel labeling is performed before picture preprocessing, which takes a lot of time, and since the time elapses, the labeled pictures are rare, and the number of truly usable labeled pictures is very limited. Therefore, many industrial people use the traditional data expansion method to increase the volume of the existing image. Unfortunately, the performance improvement caused by such data expansion is very limited, and even in some scenarios, the influence of data capacity increase is very little.
Disclosure of Invention
The application provides a satellite image recognition system and method based on self-generated features and an electronic device, and aims to solve at least one of the technical problems in the prior art to a certain extent.
In order to solve the above problems, the present application provides the following technical solutions:
a satellite image recognition system based on self-generated features, comprising:
an FCN network construction module: the device is used for changing a first full connection layer and a second full connection layer of the VGG network into a first layer of convolution layer and a second layer of convolution layer respectively to construct a new FCN network;
the self-generation characteristic network construction module: the method comprises the steps of constructing a self-generating feature network based on a GAN network;
a self-generated feature calculation module: the characteristic graph is transmitted to the first layer of convolutional layer and the second layer of convolutional layer, and the first layer of convolutional layer and the second layer of convolutional layer use the characteristic graph to obtain the identification result of the input image.
The technical scheme adopted by the embodiment of the application further comprises an operator replacing module, wherein the operator replacing module is used for replacing the convolution kernel feature extraction operator of the original first convolution layer in the FCN network with an HOG operator, and the HOG operator is used for extracting the boundary feature of the input image.
The technical scheme adopted by the embodiment of the application further comprises the following steps: the method for extracting the boundary features by the HOG operator comprises the following steps:
the gradient in different directions is first calculated:
Gx(x,y)＝H(x+1,y)-H(x-1,y)
Gy(x,y)＝H(x,y+1)-H(x,y-1)
in the above formula, Gx(x, y) represents a gradient value in the horizontal direction, Gy(x, y) represents the gradient value in the vertical direction, and H (x, y) represents the gray value of the pixel point (x, y);
calculating the gradient amplitude of the current pixel point of the image:
Figure GDA0002631573290000031
calculating the gradient direction of the current pixel point:
Figure GDA0002631573290000032
the technical scheme adopted by the embodiment of the application further comprises the following steps: the calculation method for generating the satellite image feature map by the self-generated feature network comprises the following steps: after an image is input, extracting boundary characteristics of the image through the original five convolutional layers in the FCN network, transmitting the boundary characteristics to a self-generation characteristic network, learning the characteristics through the self-generation characteristic network once through the full convolutional layers, and transmitting the characteristics, which form a single channel, of the learned characteristics to the second convolutional layer; and the characteristics of the second layer of convolutional layer and the characteristics of the first layer of convolutional layer are jointly transmitted to a discriminator, the minimum maximization process is carried out in the discriminator, the result is fed back to a self-generated characteristic network by the discriminator again, so that the self-generated characteristic network continuously generates new characteristics, after the set iteration steps are reached, the latest generated characteristics are transmitted to the second layer of convolutional layer and are directly transmitted to a full connection layer, the full connection layer carries out attribute voting according to the characteristics, and finally the identification result of the input image is obtained.
The technical scheme adopted by the embodiment of the application further comprises the following steps: the embedding of the self-generated feature network into the FCN network comprises:
the characteristics after the fifth layer convolution are transmitted to the first layer convolution layer and the second layer convolution layer on one hand, and the characteristics are transmitted to a characteristic generator which is a first embedding point;
the features formed by the feature generator are transmitted to a second convolutional layer, which is a second embedding point;
and under the condition that the set threshold value is not reached, the output result of the second layer of convolution layer is only transmitted to a discriminator, which is a third embedding point.
Another technical scheme adopted by the embodiment of the application is as follows: a satellite image identification method based on self-generated features comprises the following steps:
step a: respectively changing a first full connection layer and a second full connection layer of the VGG network into a first layer of convolution layer and a second layer of convolution layer to construct a new FCN network;
step b: constructing a self-generating feature network based on the GAN network;
step c: embedding the self-generated feature network into the FCN, generating a feature map of an input image through the self-generated feature network, and transmitting the feature map to the first layer of convolutional layer and the second layer of convolutional layer, wherein the first layer of convolutional layer and the second layer of convolutional layer use the feature map to obtain an identification result of the input image.
The technical scheme adopted by the embodiment of the application further comprises the following steps: step a also includes: and replacing the convolution kernel feature extraction operator of the original first convolution layer in the FCN with an HOG operator, and extracting the boundary feature of the input image through the HOG operator.
The technical scheme adopted by the embodiment of the application further comprises the following steps: the method for extracting the boundary features by the HOG operator comprises the following steps:
the gradient in different directions is first calculated:
Gx(x,y)＝H(x+1,y)-H(x-1,y)
Gy(x,y)＝H(x,y+1)-H(x,y-1)
in the above formula, Gx(x, y) represents a gradient value in the horizontal direction, Gy(x, y) represents the gradient value in the vertical direction, and H (x, y) represents the gray value of the pixel point (x, y);
calculating the gradient amplitude of the current pixel point of the image:
Figure GDA0002631573290000051
calculating the gradient direction of the current pixel point:
Figure GDA0002631573290000052
the technical scheme adopted by the embodiment of the application further comprises the following steps: in step c, the calculation method for generating the satellite image feature map by the self-generated feature network is as follows: after an image is input, extracting boundary characteristics of the image through the original five convolutional layers in the FCN network, transmitting the boundary characteristics to a self-generation characteristic network, learning the characteristics through the self-generation characteristic network once through the full convolutional layers, and transmitting the characteristics, which form a single channel, of the learned characteristics to the second convolutional layer; and the characteristics of the second layer of convolutional layer and the characteristics of the first layer of convolutional layer are jointly transmitted to a discriminator, the minimum maximization process is carried out in the discriminator, the result is fed back to a self-generated characteristic network by the discriminator again, so that the self-generated characteristic network continuously generates new characteristics, after the set iteration steps are reached, the latest generated characteristics are transmitted to the second layer of convolutional layer and are directly transmitted to a full connection layer, the full connection layer carries out attribute voting according to the characteristics, and finally the identification result of the input image is obtained.
The technical scheme adopted by the embodiment of the application further comprises the following steps: in the step c, the embedding of the self-generated feature network into the FCN network includes:
the characteristics after the fifth layer convolution are transmitted to the first layer convolution layer and the second layer convolution layer on one hand, and the characteristics are transmitted to a characteristic generator which is a first embedding point;
the features formed by the feature generator are transmitted to a second convolutional layer, which is a second embedding point;
and under the condition that the set threshold value is not reached, the output result of the second layer of convolution layer is only transmitted to a discriminator, which is a third embedding point.
The embodiment of the application adopts another technical scheme that: an electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the following operations of the self-generated feature based satellite image recognition method described above:
step a: respectively changing a first full connection layer and a second full connection layer of the VGG network into a first layer of convolution layer and a second layer of convolution layer to construct a new FCN network;
step b: constructing a self-generating feature network based on the GAN network;
step c: embedding the self-generated feature network into the FCN, generating a feature map of an input image through the self-generated feature network, and transmitting the feature map to the first layer of convolutional layer and the second layer of convolutional layer, wherein the first layer of convolutional layer and the second layer of convolutional layer use the feature map to obtain an identification result of the input image.
Compared with the prior art, the embodiment of the application has the advantages that: according to the satellite image identification system, method and electronic device based on the self-generated features, the basic convolution kernel operator in the FCN is replaced by the multi-channel HOG operator, so that the system can generate a more sensitive detection effect in four major directions, namely, eight directions including the upper diagonal, the lower diagonal, the left diagonal and the right diagonal. The full-convolution network specially designed for satellite image detection is modified from the first and second full-connection layers of the VGG network, meanwhile, a self-generation feature network is added, and the automatic generation feature network is embedded into a new full-convolution network, so that the latest image features are continuously calculated and updated without generating a plurality of useless redundant data to accumulate the full-convolution network. Not only can the enhancement of data characteristics (which is the content of the phase-change extended data) be realized, but also the identification of pixels by pixels can be realized so as to meet the requirement of accurate identification. According to the method and the device, a large amount of computer resources are avoided being consumed to realize data expansion, the requirements of data redundancy and accurate boundaries can be well met, meanwhile, optimization is particularly carried out on satellite images which are rarely or never appeared in previous images, and the identification algorithm is enabled to be more robust.
Drawings
Fig. 1 is a schematic structural diagram of a satellite image recognition system based on self-generated features according to an embodiment of the present application;
FIG. 2 is a diagram of an FCN framework according to an embodiment of the present application;
FIG. 3 is a network framework diagram of a satellite image recognition system based on self-generated features according to an embodiment of the present application;
FIG. 4 is a flowchart of a satellite image recognition method based on self-generated features according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of hardware equipment of a satellite image identification method based on self-generated features according to an embodiment of the present disclosure.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
Aiming at the problems in the prior art, by means of the thought of a countermeasure generation network (generic adaptive Nets, hereinafter called GAN for short) [ Goodfellow I, Pouget-Abadie J, Mirza M, et al. genetic adaptive networks [ C ]// advanced in neural information processing systems.2014:2672 and 2680 ], the satellite image recognition system based on self-generated features is constructed, the features in the satellite images are automatically generated, then the neural network with the automatically generated features is embedded into a new full convolution network by utilizing the newly constructed neural network, so that the neural network is used for detecting the types of various ground objects in the satellite images, the enhancement of data features (which are the content of phase-changed extended data) can be realized, and the pixel-to-pixel recognition can be realized, so as to meet the requirement of accurate recognition.
Specifically, please refer to fig. 1, which is a schematic structural diagram of a satellite image recognition system based on self-generated features according to an embodiment of the present application. The satellite image identification system based on the self-generated features comprises an operator replacing module, an FCN network building module, a self-generated feature network building module and a self-generated feature calculating module.
An operator replacement module: the method is used for replacing a SOBEL (Sobel operator) operator and a SIFT (Scale Invariant Feature Transform) operator of a first layer convolutional layer in the original FCN network with an HOG (Histogram of Oriented Gradient) operator, and extracting points of other layers of features are unchanged; the SOBEL operator can only calculate the horizontal direction and the vertical direction, and the SOBEL operator has poor performance when facing some non-horizontal and vertical directions; the SIFT operator is mainly used for processing scale invariance, and the satellite image identification does not involve too much scale change and only needs to consider the detection of the image boundary. In the application, four directions of horizontal, vertical, 45 degrees to the east of the north and 45 degrees to the west of the north are used, and the directions are eight directions in total because of the up-down and left-right symmetry. Because the HOG operator can be expanded to a plurality of directions, the convolution kernel feature extraction operator of the first layer of convolution layer is replaced by the HOG operator, so that the HOG operator is more suitable for edge feature detection of the satellite image, and boundary feature extraction of the satellite image in a plurality of directions can be completed comprehensively.
In the embodiment of the present application, the direction of the HOG operator is determined by a matrix of 3 × 3, where the HOG operator calculates the features in the following manner:
1. the gradient in different directions is first calculated:
Gx(x,y)＝H(x+1,y)-H(x-1,y) (1)
Gy(x,y)＝H(x,y+1)-H(x,y-1) (2)
in the formulae (1) and (2), Gx(x, y) represents a gradient value in the horizontal direction, GyThe (x, y) represents the gradient value in the vertical direction, the H (x, y) represents the gray value of the pixel (x, y), and the gray values are all in the range of (0,255) in the image processing, wherein 0 represents black, and 255 represents white.
(2) Calculating the gradient amplitude of the current pixel point of the image:
Figure GDA0002631573290000091
(3) calculating the gradient direction of the current pixel point:
Figure GDA0002631573290000092
an FCN network construction module: an FCN network for constructing new dense pixel identification using the VGG16 network as an underlying framework under which the first fully-connected layer and the second fully-connected layer of the VGG network are all changed to convolutional layers (i.e., two convolutional layers are added); wherein, newly-increased two convolution layers's advantage lies in: if only one convolution layer is used for up-sampling, the up-sampling result is possibly insufficient, accurate identification is difficult to achieve under the condition that the satellite image is complex, and therefore one layer of connection is added, characteristics can be better represented, and the attributes of the final pixel points can be accurately represented. The FCN network structure description of the present embodiment is shown in table 1 below (CONV # in table 1 represents the number of convolutional layer):
Figure GDA0002631573290000093
compared with the existing VGG network, the method has the advantages that the original two fully-connected layers are completely changed into convolutional layers, and the image upsampling operation is executed through the two newly-added convolutional layers, so that the image is gradually improved. The last SOFT-MAX represents a classification method commonly used in imaging. Fig. 2 is a diagram of an FCN network framework according to an embodiment of the present application, in which a portion of a dashed frame indicates a new structure of the present application.
The self-generation characteristic network construction module: the method is used for constructing the self-generating feature network based on the GAN network. The GAN network is divided into a generator and a discriminator, and the basic principle of the GAN network is as follows: the generator is used for learning from the extracted features, further manufacturing a plurality of new images similar to the existing images, then transmitting the manufactured new images to the discriminator, judging whether the images are made pictures or not by the discriminator, under the continuous game of the two parties, feeding the results back to the generator by the discriminator, and continuously adjusting the learning of the generator according to the results, further manufacturing more vivid images until the discriminator is cheated. The method is based on the GAN network, but is different from the GAN network in that a self-generated feature network is constructed by means of GAN, so that more vivid features are generated, more robust feature maps are trained, the feature maps are transmitted to an upper sampling layer, and the upper sampling layer can identify more accurate satellite images by using the feature maps.
The self-generating feature network construction mode specifically comprises the following steps:
in the existing x, the generator learns a data distribution PgMeanwhile, because of the noise in the data distribution, a noise distribution function is defined: pZ(Z) ensures that the function is robust. Plus a parameter theta originally in the networkgThus defining G (Z, theta)g) A mapping of the original data. And a discriminator D (x) for representing the probability that the data comes from x, and training D (x) to identify whether the data comes from the self-training data set or G (x) with the maximum capability or the maximum probability. And simultaneously, the log (1-D (G (z)) represented by G is minimized, the innermost layer of the formula is nested with the generator, and if the formula is minimized, the D (G (z)) of the inner layer must be maximized, so that the discriminator maximizes the probability to accurately identify the content from the generator. Combining the two contents to obtain:
Figure GDA0002631573290000101
changing the data distribution into a characteristic distribution, and defining the original characteristic as OfThe generator generates a feature of NfEquation (5) is changed to:
Figure GDA0002631573290000111
in the formula (6), the characteristic Of、NfAre all single-channel, i.e. one-layer network characteristics.
A self-generated feature calculation module: the method is used for embedding a self-generated feature network into an FCN network of dense pixel identification to generate a satellite image identification system based on self-generated features.
Fig. 3 is a network framework diagram of a satellite image recognition system based on self-generated features according to an embodiment of the present application. The satellite image identification mode of the satellite image identification system based on the self-generated characteristics specifically comprises the following steps: after the satellite image is input, firstly, boundary characteristics of the satellite image in multiple directions are extracted through original 5 convolutional layers (CONV1 to CONV5) in an FCN network, then the boundary characteristics are transmitted to a self-generating characteristic network, a characteristic diagram of the satellite image is generated through the self-generating characteristic network, the characteristic diagram is transmitted to an upper sampling layer (namely newly-added NEWCONV1 and NEWCONV2), and the upper sampling layer obtains a satellite image identification result by utilizing the characteristic diagrams; the calculation method for generating the satellite image feature map from the self-generated feature network comprises the following steps: the characteristics of the original satellite image are transmitted to a self-generation characteristic network after fifth layer convolution (CONV5), and the self-generation characteristic network passes through a full convolution layer
Figure GDA0002631573290000112
Learning the FEATUREs and then transmitting the learned FEATUREs to a newly added second layer convolutional layer (NEWCONV2) to form a single channel FEATUREs (FEATURE MAP). The characteristics of the second layer of the convolutional layer (NEWCONV2) and the characteristics of the first layer of the convolutional layer (NEWCONV1) are jointly transmitted to the discriminator, then the minimum maximization process is carried out in the discriminator, and the discriminator feeds the result back to the self-generation characteristic network again, so that the self-generation characteristic network continuously generates characteristics which can be more false and true. And after the set iteration step number is reached (namely the loss function is smaller than a set threshold), transmitting the latest feature to a newly-added second layer convolution layer (NEWCONV2) and then directly transmitting the latest feature to a full-connection layer, and performing attribute voting by the full-connection layer according to the feature to obtain a satellite image identification result.
In the embodiment of the present application, the embedding of the self-generated feature network into the FCN network for dense pixel identification includes:
(1) the signature after the fifth layer convolution is transferred on the one hand to the newly added first layer convolution layer (NEWCONV1) and second layer convolution layer (NEWCONV2) on the original path, and the signature is transmitted to the signature generator, which is the first embedding point.
(2) The features formed by the feature generator are transferred to a newly added second layer convolutional layer (NEWCONV2), which is the second embedding point.
(3) And transmitting the output result of the second layer convolution layer (NEWCONV2) to a discriminator, namely a third embedding point, under the condition that the output result does not reach the set threshold value.
The three embedding points take account of the difference of the characteristics of each layer and also take account of the action mechanism of the discriminator and the generator under the mutual game.
Please refer to fig. 4, which is a flowchart illustrating a satellite image recognition method based on self-generated features according to an embodiment of the present disclosure. The satellite image identification method based on the self-generating characteristics comprises the following steps:
step 100: replacing the original SOBEL operator and SIFT operator of the first layer convolutional layer (CONV1) in the FCN network with HOG operator, and keeping the feature extraction points of the other layers unchanged;
in step 100, the SOBEL operator can only calculate the horizontal and vertical directions, and the performance is poor when the user faces some non-horizontal and vertical directions; the SIFT operator is mainly used for processing scale invariance, and the satellite image identification does not involve too much scale change and only needs to consider the detection of the image boundary. In the application, four directions of horizontal, vertical, 45 degrees to the east of the north and 45 degrees to the west of the north are used, and the directions are eight directions in total because of the up-down and left-right symmetry. Because the HOG operator can be expanded to a plurality of directions, the convolution kernel feature extraction operator of the first layer of convolution layer is replaced by the HOG operator, so that the HOG operator is more suitable for edge feature detection of the satellite image, and boundary feature extraction of the satellite image in a plurality of directions can be completed comprehensively.
In the embodiment of the present application, the direction of the HOG operator is determined by a matrix of 3 × 3, where the HOG operator calculates the features in the following manner:
1. the gradient in different directions is first calculated:
Gx(x,y)＝H(x+1,y)-H(x-1,y) (1)
Gy(x,y)＝H(x,y+1)-H(x,y-1) (2)
in the formulae (1) and (2), Gx(x, y) tableIndicating a gradient value in the horizontal direction, GyThe (x, y) represents the gradient value in the vertical direction, the H (x, y) represents the gray value of the pixel (x, y), and the gray values are all in the range of (0,255) in the image processing, wherein 0 represents black, and 255 represents white.
(2) Calculating the gradient amplitude of the current pixel point of the image:
Figure GDA0002631573290000131
(3) calculating the gradient direction of the current pixel point:
Figure GDA0002631573290000132
step 200: using the VGG16 network as an underlying framework under which the first fully-connected layer and the second fully-connected layer of the VGG network are all changed into convolutional layers (i.e. two convolutional layers are added), constructing a new FCN network with dense pixel identification;
in step 200, the advantage of adding two convolutional layers is: if only one convolution layer is used for up-sampling, the up-sampling result is possibly insufficient, accurate identification is difficult to achieve under the condition that the satellite image is complex, and therefore one layer of connection is added, characteristics can be better represented, and the attributes of the final pixel points can be accurately represented. The FCN network structure description of the present embodiment is shown in table 1 below (CONV # in table 1 represents the number of convolutional layer):
Figure GDA0002631573290000133
Figure GDA0002631573290000141
compared with the existing VGG network, the method has the advantages that the original two fully-connected layers are completely changed into convolutional layers, and the image upsampling operation is executed through the two newly-added convolutional layers, so that the image is gradually improved. The last SOFT-MAX represents a classification method commonly used in imaging.
Step 300: constructing a self-generating feature network based on the GAN network;
in step 300, the GAN network is divided into two parts, namely a generator and a discriminator, and the basic principle of the GAN network is as follows: the generator is used for learning from the extracted features, further manufacturing a plurality of new images similar to the existing images, then transmitting the manufactured new images to the discriminator, judging whether the images are made pictures or not by the discriminator, under the continuous game of the two parties, feeding the results back to the generator by the discriminator, and continuously adjusting the learning of the generator according to the results, further manufacturing more vivid images until the discriminator is cheated. The method is based on the GAN network, but is different from the GAN network in that a self-generated feature network is constructed by means of GAN, so that more vivid features are generated, more robust feature maps are trained, the feature maps are transmitted to an upper sampling layer, and the upper sampling layer can identify more accurate satellite images by using the feature maps.
The self-generating feature network construction mode specifically comprises the following steps:
in the existing x, the generator learns a data distribution PgMeanwhile, because of the noise in the data distribution, a noise distribution function is defined: pZ(Z) ensures that the function is robust. Plus a parameter theta originally in the networkgThus defining G (z, theta)g) A mapping of the original data. And a discriminator D (x) for representing the probability that the data comes from x, and training D (x) to identify whether the data comes from the self-training data set or G (x) with the maximum capability or the maximum probability. And simultaneously, the log (1-D (G (z)) represented by G is minimized, the innermost layer of the formula is nested with the generator, and if the formula is minimized, the D (G (z)) of the inner layer must be maximized, so that the discriminator maximizes the probability to accurately identify the content from the generator. Combining the two contents to obtain:
Figure GDA0002631573290000151
changing the data distribution into a characteristic distribution, and defining the original characteristic as OfThe generator generates a feature of NfEquation (5) is changed to:
Figure GDA0002631573290000152
in the formula (6), the characteristic Of、NfAre all single-channel, i.e. one-layer network characteristics.
Step 400: embedding a self-generated feature network into an FCN network for dense pixel identification, generating a satellite image identification system based on self-generated features, and identifying satellite images through the satellite image identification system based on the self-generated features;
in step 400, the satellite image recognition method of the satellite image recognition system based on the self-generated features specifically includes: the satellite image identification mode of the satellite image identification system based on the self-generated characteristics specifically comprises the following steps: after the satellite image is input, firstly, boundary characteristics of the satellite image in multiple directions are extracted through original 5 convolutional layers (CONV1 to CONV5) in an FCN network, then the boundary characteristics are transmitted to a self-generating characteristic network, a characteristic diagram of the satellite image is generated through the self-generating characteristic network, the characteristic diagram is transmitted to an upper sampling layer (namely newly-added NEWCONV1 and NEWCONV2), and the upper sampling layer obtains a satellite image identification result by utilizing the characteristic diagrams; the calculation method for generating the satellite image feature map from the self-generated feature network comprises the following steps: the characteristics of the original satellite image are transmitted to a self-generation characteristic network after fifth layer convolution (CONV5), and the self-generation characteristic network passes through a full convolution layer
Figure GDA0002631573290000161
Learning the FEATUREs and then transmitting the learned FEATUREs to a newly added second layer convolutional layer (NEWCONV2) to form a single channel FEATUREs (FEATURE MAP). The characteristics of the second layer (NEWCONV2) and the first layer (NEWCONV1) are transmitted together to a discriminator where the minimum is then performedIn the process of generalization, the result is fed back to the self-generated feature network again by the discriminator, so that the self-generated feature network can continuously generate features which can be more false and true. And after the set iteration step number is reached (namely the loss function is smaller than a set threshold), transmitting the latest feature to a newly-added second layer convolution layer (NEWCONV2) and then directly transmitting the latest feature to a full-connection layer, and performing attribute voting by the full-connection layer according to the feature to obtain a satellite image identification result.
Fig. 5 is a schematic structural diagram of hardware equipment of a satellite image identification method based on self-generated features according to an embodiment of the present disclosure. As shown in fig. 5, the device includes one or more processors and memory. Taking a processor as an example, the apparatus may further include: an input system and an output system.
The processor, memory, input system, and output system may be connected by a bus or other means, as exemplified by the bus connection in fig. 5.
The memory, which is a non-transitory computer readable storage medium, may be used to store non-transitory software programs, non-transitory computer executable programs, and modules. The processor executes various functional applications and data processing of the electronic device, i.e., implements the processing method of the above-described method embodiment, by executing the non-transitory software program, instructions and modules stored in the memory.
The memory may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data and the like. Further, the memory may include high speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory optionally includes memory located remotely from the processor, and these remote memories may be connected to the processing system over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input system may receive input numeric or character information and generate a signal input. The output system may include a display device such as a display screen.
The one or more modules are stored in the memory and, when executed by the one or more processors, perform the following for any of the above method embodiments:
step a: respectively changing a first full connection layer and a second full connection layer of the VGG network into a first layer of convolution layer and a second layer of convolution layer to construct a new FCN network;
step b: constructing a self-generating feature network based on the GAN network;
step c: embedding the self-generated feature network into the FCN, generating a feature map of an input image through the self-generated feature network, and transmitting the feature map to the first layer of convolutional layer and the second layer of convolutional layer, wherein the first layer of convolutional layer and the second layer of convolutional layer use the feature map to obtain an identification result of the input image.
The product can execute the method provided by the embodiment of the application, and has the corresponding functional modules and beneficial effects of the execution method. For technical details that are not described in detail in this embodiment, reference may be made to the methods provided in the embodiments of the present application.
Embodiments of the present application provide a non-transitory (non-volatile) computer storage medium having stored thereon computer-executable instructions that may perform the following operations:
step a: respectively changing a first full connection layer and a second full connection layer of the VGG network into a first layer of convolution layer and a second layer of convolution layer to construct a new FCN network;
step b: constructing a self-generating feature network based on the GAN network;
step c: embedding the self-generated feature network into the FCN, generating a feature map of an input image through the self-generated feature network, and transmitting the feature map to the first layer of convolutional layer and the second layer of convolutional layer, wherein the first layer of convolutional layer and the second layer of convolutional layer use the feature map to obtain an identification result of the input image.
Embodiments of the present application provide a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions that, when executed by a computer, cause the computer to perform the following:
step a: respectively changing a first full connection layer and a second full connection layer of the VGG network into a first layer of convolution layer and a second layer of convolution layer to construct a new FCN network;
step b: constructing a self-generating feature network based on the GAN network;
step c: embedding the self-generated feature network into the FCN, generating a feature map of an input image through the self-generated feature network, and transmitting the feature map to the first layer of convolutional layer and the second layer of convolutional layer, wherein the first layer of convolutional layer and the second layer of convolutional layer use the feature map to obtain an identification result of the input image.
According to the satellite image identification system, method and electronic device based on the self-generated features, the basic convolution kernel operator in the FCN is replaced by the multi-channel HOG operator, so that the system can generate a more sensitive detection effect in four major directions, namely, eight directions including the upper diagonal, the lower diagonal, the left diagonal and the right diagonal. The full-convolution network specially designed for satellite image detection is modified from the first and second full-connection layers of the VGG network, meanwhile, a self-generation feature network is added, and the automatic generation feature network is embedded into a new full-convolution network, so that the latest image features are continuously calculated and updated without generating a plurality of useless redundant data to accumulate the full-convolution network. Not only can the enhancement of data characteristics (which is the content of the phase-change extended data) be realized, but also the identification of pixels by pixels can be realized so as to meet the requirement of accurate identification. According to the method and the device, a large amount of computer resources are avoided being consumed to realize data expansion, the requirements of data redundancy and accurate boundaries can be well met, meanwhile, optimization is particularly carried out on satellite images which are rarely or never appeared in previous images, and the identification algorithm is enabled to be more robust.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (11)

1. A satellite image recognition system based on self-generated features, comprising:
an FCN network construction module: the device is used for changing a first full connection layer and a second full connection layer of the VGG network into a first layer of convolution layer and a second layer of convolution layer respectively to construct a new FCN network;
the self-generation characteristic network construction module: the method comprises the steps of constructing a self-generating feature network based on a GAN network;
a self-generated feature calculation module: the characteristic graph is transmitted to the first layer of convolutional layer and the second layer of convolutional layer, and the first layer of convolutional layer and the second layer of convolutional layer use the characteristic graph to obtain the identification result of the input image.
2. The self-generated feature based satellite image recognition system according to claim 1, further comprising an operator replacement module, wherein the operator replacement module is configured to replace a convolution kernel feature extraction operator of a first convolution layer existing in the FCN network with a HOG operator, and the HOG operator is used for performing boundary feature extraction of the input image.
3. The self-generated feature based satellite image recognition system of claim 2, wherein the HOG operator extracts the boundary features by:
the gradient in different directions is first calculated:
Gx(x，y)＝H(x+1，y)-H(x-1，y)
Gy(x，y)＝H(x，y+1)-H(x，y-1)
in the above formula, Gx(x, y) represents a gradient value in the horizontal direction, Gy(x, y) represents the gradient value in the vertical direction, and H (x, y) represents the gray value of the pixel point (x, y);
calculating the gradient amplitude of the current pixel point of the image:
Figure FDA0002631573280000011
calculating the gradient direction of the current pixel point:
Figure FDA0002631573280000021
4. the satellite image recognition system based on self-generated features of claim 3, wherein the self-generated feature network generates the satellite image feature map by a calculation method comprising: after an image is input, extracting boundary characteristics of the image through the original five convolutional layers in the FCN network, transmitting the boundary characteristics to a self-generation characteristic network, learning the characteristics through the self-generation characteristic network once through the full convolutional layers, and transmitting the characteristics, which form a single channel, of the learned characteristics to the second convolutional layer; and the characteristics of the second layer of convolutional layer and the characteristics of the first layer of convolutional layer are jointly transmitted to a discriminator, the minimum maximization process is carried out in the discriminator, the result is fed back to a self-generated characteristic network by the discriminator again, so that the self-generated characteristic network continuously generates new characteristics, after the set iteration steps are reached, the latest generated characteristics are transmitted to the second layer of convolutional layer and are directly transmitted to a full connection layer, the full connection layer carries out attribute voting according to the characteristics, and finally the identification result of the input image is obtained.
5. The self-generated feature based satellite image recognition system of claim 4, wherein the embedding of the self-generated feature network into the FCN network at an embedding point comprises:
the characteristics after the fifth layer convolution are transmitted to the first layer convolution layer and the second layer convolution layer on one hand, and the characteristics are transmitted to a characteristic generator which is a first embedding point;
the features formed by the feature generator are transmitted to a second convolutional layer, which is a second embedding point;
and under the condition that the set threshold value is not reached, the output result of the second layer of convolution layer is only transmitted to a discriminator, which is a third embedding point.
6. A satellite image identification method based on self-generated features is characterized by comprising the following steps:
step a: respectively changing a first full connection layer and a second full connection layer of the VGG network into a first layer of convolution layer and a second layer of convolution layer to construct a new FCN network;
step b: constructing a self-generating feature network based on the GAN network;
step c: embedding the self-generated feature network into the FCN, generating a feature map of an input image through the self-generated feature network, and transmitting the feature map to the first layer of convolutional layer and the second layer of convolutional layer, wherein the first layer of convolutional layer and the second layer of convolutional layer use the feature map to obtain an identification result of the input image.
7. The method for satellite image recognition based on self-generated features of claim 6, wherein step a further comprises: and replacing the convolution kernel feature extraction operator of the original first convolution layer in the FCN with an HOG operator, and extracting the boundary feature of the input image through the HOG operator.
8. The method of claim 7, wherein the HOG operator extracts the boundary features by:
the gradient in different directions is first calculated:
Gx(x，y)＝H(x+1，y)-H(x-1，y)
Gy(x，y)＝H(x，y+1)-H(x，y-1)
in the above formula, Gx(x, y) represents a gradient value in the horizontal direction, Gy(x, y) represents the gradient value in the vertical direction, and H (x, y) represents the gray value of the pixel point (x, y);
calculating the gradient amplitude of the current pixel point of the image:
Figure FDA0002631573280000031
calculating the gradient direction of the current pixel point:
Figure FDA0002631573280000032
9. the method for satellite image recognition based on self-generated features of claim 8, wherein in the step c, the self-generated feature network generates the satellite image feature map by a calculation method comprising: after an image is input, extracting boundary characteristics of the image through the original five convolutional layers in the FCN network, transmitting the boundary characteristics to a self-generation characteristic network, learning the characteristics through the self-generation characteristic network once through the full convolutional layers, and transmitting the characteristics, which form a single channel, of the learned characteristics to the second convolutional layer; and the characteristics of the second layer of convolutional layer and the characteristics of the first layer of convolutional layer are jointly transmitted to a discriminator, the minimum maximization process is carried out in the discriminator, the result is fed back to a self-generated characteristic network by the discriminator again, so that the self-generated characteristic network continuously generates new characteristics, after the set iteration steps are reached, the latest generated characteristics are transmitted to the second layer of convolutional layer and are directly transmitted to a full connection layer, the full connection layer carries out attribute voting according to the characteristics, and finally the identification result of the input image is obtained.
10. The satellite image recognition method based on self-generated features of claim 9, wherein in the step c, the embedding of the self-generated feature network into the FCN network comprises:
the characteristics after the fifth layer convolution are transmitted to the first layer convolution layer and the second layer convolution layer on one hand, and the characteristics are transmitted to a characteristic generator which is a first embedding point;
the features formed by the feature generator are transmitted to a second convolutional layer, which is a second embedding point;
and under the condition that the set threshold value is not reached, the output result of the second layer of convolution layer is only transmitted to a discriminator, which is a third embedding point.
11. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the following operations of the self-generated feature based satellite image recognition method according to any one of the above 6 to 10:
step a: respectively changing a first full connection layer and a second full connection layer of the VGG network into a first layer of convolution layer and a second layer of convolution layer to construct a new FCN network;
step b: constructing a self-generating feature network based on the GAN network;
step c: embedding the self-generated feature network into the FCN, generating a feature map of an input image through the self-generated feature network, and transmitting the feature map to the first layer of convolutional layer and the second layer of convolutional layer, wherein the first layer of convolutional layer and the second layer of convolutional layer use the feature map to obtain an identification result of the input image.
CN201811227460.2A 2018-10-22 2018-10-22 Satellite image recognition system and method based on self-generated features and electronic equipment Active CN109522807B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201811227460.2A CN109522807B (en) 2018-10-22 2018-10-22 Satellite image recognition system and method based on self-generated features and electronic equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201811227460.2A CN109522807B (en) 2018-10-22 2018-10-22 Satellite image recognition system and method based on self-generated features and electronic equipment

Publications (2)

Publication Number Publication Date
CN109522807A CN109522807A (en) 2019-03-26
CN109522807B true CN109522807B (en) 2020-11-27

Family

ID=65772178

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201811227460.2A Active CN109522807B (en) 2018-10-22 2018-10-22 Satellite image recognition system and method based on self-generated features and electronic equipment

Country Status (1)

Country Link
CN (1) CN109522807B (en)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3742346A3 (en) 2019-05-23 2021-06-16 HTC Corporation Method for training generative adversarial network (gan), method for generating images by using gan, and computer readable storage medium
CN112132155B (en) * 2019-06-25 2022-06-14 珠海一微半导体股份有限公司 SIFT algorithm image boundary expansion method and processing method based on hardware circuit
CN112446266B (en) * 2019-09-04 2024-03-29 北京君正集成电路股份有限公司 Face recognition network structure suitable for front end
CN110866908B (en) * 2019-11-12 2021-03-26 腾讯科技（深圳）有限公司 Image processing method, image processing apparatus, server, and storage medium
CN114648706B (en) * 2022-05-23 2022-07-29 四川嘉普信工程技术咨询有限公司 Forest tree species identification method, device and equipment based on satellite remote sensing image

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105631415A (en) * 2015-12-25 2016-06-01 中通服公众信息产业股份有限公司 Video pedestrian recognition method based on convolution neural network
CN108537807B (en) * 2017-03-02 2021-07-13 中国科学院遥感与数字地球研究所 Improved edge enhancement detection method based on high-resolution fourth-order satellite image
CN107016677B (en) * 2017-03-24 2020-01-17 北京工业大学 Cloud picture segmentation method based on FCN and CNN
CN108009472B (en) * 2017-10-25 2020-07-21 五邑大学 Finger back joint print recognition method based on convolutional neural network and Bayes classifier

Also Published As

Publication number Publication date
CN109522807A (en) 2019-03-26

Similar Documents

Publication Publication Date Title
CN109522807B (en) Satellite image recognition system and method based on self-generated features and electronic equipment
Kong et al. A federated learning-based license plate recognition scheme for 5G-enabled internet of vehicles
US11328401B2 (en) Stationary object detecting method, apparatus and electronic device
CN108734210B (en) Object detection method based on cross-modal multi-scale feature fusion
US10410354B1 (en) Method and apparatus for multi-model primitive fitting based on deep geometric boundary and instance aware segmentation
US11830103B2 (en) Method, apparatus, and computer program product for training a signature encoding module and a query processing module using augmented data
CN109919002B (en) Yellow stop line identification method and device, computer equipment and storage medium
CN114746898A (en) Method and system for generating trisection images of image matting
CN103632153A (en) Region-based image saliency map extracting method
CN104217459A (en) Spherical feature extraction method
EP3857449A1 (en) Apparatus and method for three-dimensional object recognition
CN110866900A (en) Water body color identification method and device
CN114677596A (en) Remote sensing image ship detection method and device based on attention model
CN113112542A (en) Visual positioning method and device, electronic equipment and storage medium
US20160125253A1 (en) Method and apparatus for image matching
CN110472632B (en) Character segmentation method and device based on character features and computer storage medium
WO2022247126A1 (en) Visual localization method and apparatus, and device, medium and program
CN115345895B (en) Image segmentation method and device for visual detection, computer equipment and medium
WO2024000728A1 (en) Monocular three-dimensional plane recovery method, device, and storage medium
Pang et al. PTRSegNet: A Patch-to-Region Bottom-Up Pyramid Framework for the Semantic Segmentation of Large-Format Remote Sensing Images
Shumeng et al. A semantic segmentation method for remote sensing images based on multiple contextual feature extraction
CN113569636B (en) Fisheye image feature processing method and system based on spherical features and electronic equipment
CN112101386B (en) Text detection method, device, computer equipment and storage medium
CN112686256B (en) Training-free and unsupervised target co-location method, system and device
CN115311573B (en) Site line detection and target positioning method, electronic equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant