CN113486894A - Semantic segmentation method for satellite image feature component - Google Patents

Semantic segmentation method for satellite image feature component Download PDF

Info

Publication number
CN113486894A
CN113486894A CN202110802773.1A CN202110802773A CN113486894A CN 113486894 A CN113486894 A CN 113486894A CN 202110802773 A CN202110802773 A CN 202110802773A CN 113486894 A CN113486894 A CN 113486894A
Authority
CN
China
Prior art keywords
satellite
semantic segmentation
image
model
feature component
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110802773.1A
Other languages
Chinese (zh)
Other versions
CN113486894B (en
Inventor
崔林艳
蔡敬艺
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beihang University
Original Assignee
Beihang University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beihang University filed Critical Beihang University
Priority to CN202110802773.1A priority Critical patent/CN113486894B/en
Publication of CN113486894A publication Critical patent/CN113486894A/en
Application granted granted Critical
Publication of CN113486894B publication Critical patent/CN113486894B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Molecular Biology (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Mathematical Physics (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)

Abstract

The invention relates to a semantic segmentation method for a satellite image feature component, which comprises the following steps of: (1) aiming at satellite images, based on three-dimensional simulation software and CAD software, information of satellites of various types, such as different angles, different distances and different brightness, is comprehensively considered, and a satellite image feature component segmentation data set is constructed; (2) aiming at the data set obtained in the step (1), combining the context information of the feature component area, and constructing a DeepLabV3+ semantic segmentation model based on the context information; (3) adding shape integrity prior information constraint to the model on the basis of the DeepLabV3+ semantic segmentation model based on the context information in the step (2); (4) and (3) adding a model after shape integrity prior constraint aiming at the step (3), training the model on the semantic segmentation data set of the satellite image feature component obtained in the step (1) to obtain a trained semantic segmentation model, and performing semantic segmentation on the satellite image by using the trained semantic segmentation model to obtain a predicted segmentation map.

Description

Semantic segmentation method for satellite image feature component
Technical Field
The invention relates to a semantic segmentation method for a satellite image feature component, which is a semantic segmentation model combining context information and shape integrity prior and is suitable for a semantic segmentation task of the satellite image feature component.
Background
From the computer vision field, the nature of satellite typical feature recognition is semantic segmentation of images. The semantic segmentation is a basic method for realizing image scene understanding by marking a pixel-level semantic information label on an image and simultaneously realizing two subtasks of segmentation and classification of the image. Image semantic segmentation is a technique and process that divides an image into several specific regions with unique properties and proposes an object of interest. It is a key step from image processing to image analysis. From a mathematical point of view, image segmentation is the process of dividing a digital image into mutually disjoint regions. The image segmentation process is also a labeling process, i.e. image indexes belonging to the same region are assigned with the same number. In the past decades, a great number of researchers have made great breakthrough in image segmentation research on specific application scenes such as medical images and remote sensing images, and image segmentation on satellites is very rare in the existing public documents. However, since the images such as the satellite image, the medical image, and the remote sensing image have features that can be distinguished by naked eyes, the image segmentation method widely applied to other application scenes can be shifted to the identification of the satellite typical feature component, and the identification of the feature component can be realized by semantic segmentation of the satellite image.
For the task of image semantic segmentation, various conventional machine learning techniques have been commonly used in the past. The traditional image segmentation method comprises a pixel clustering-based segmentation method, a graph theory partitioning-based segmentation method and the like, wherein the most important key technology for research is feature extraction and expression, the traditional method excessively depends on prior knowledge to carry out manual selection and design, image features are difficult to widely express, time and labor are consumed, and the precision of a final result is difficult to guarantee. The semantic segmentation task has many difficulties, for example, in an object level, the same object shows different appearance images under different shooting conditions; at the category level, the dissimilarity of similar objects and the similarity of heterogeneous objects, etc.; background level, the background in the actual scene is often complicated and changeable, etc.
Supervised learning through a deep CNN network becomes a main method for solving the image semantic segmentation task in recent years, such as FCN, U-Net, deep Lab series models and the like, end-to-end training is carried out to obtain a prediction segmentation graph. However, there are few methods for performing semantic segmentation task on satellite image feature components, and the difficulty of current research is mainly manifested in the following aspects: (1) real satellite images can hardly be obtained, the existing public data set does not focus on satellite characteristic components, and a great deal of effort is needed to construct the data set; (2) the satellite image needs to be subjected to operations of noise addition and blurring, and compared with a conventional image, the satellite image is poor in image quality, blurred in detail information such as edges and the like, and difficult to accurately segment; (3) the existing semantic segmentation method based on deep learning only aims at high-quality images of common scenes generally and is not suitable for satellite images.
Disclosure of Invention
The technical problem to be solved by the invention is as follows: the method is realized based on context information and shape integrity prior, improves the detail of the segmentation edge of the satellite image, improves the semantic recognition accuracy and improves the semantic segmentation accuracy.
The technical solution of the invention is as follows: a semantic segmentation method for satellite image feature components is realized based on context information and shape integrity prior, and comprises the following steps:
(1) acquiring an original satellite simulation image, and preprocessing the acquired original satellite simulation image so as to enable the original satellite simulation image to be closer to an on-orbit satellite real image, wherein the preprocessing comprises graying, illumination and noise adding and fuzzification processing operations on the original satellite simulation image; various satellite image characteristic components which are clearly visible are marked on the edge landmarks of the preprocessed satellite images, the satellite image characteristic components comprise six types of characteristic components including a sailboard, an antenna, a spray pipe, a lens cone, a KKV and a mechanical arm, and semantic segmentation data sets of the satellite image characteristic components of the six types of characteristic components are constructed;
(2) based on the data set obtained in the step (1), constructing a DeepLabV3+ semantic segmentation model based on the context information by utilizing the semantic segmentation result of the original DeepLabV3+ model and the feature map extracted by the original DeepLabV3+ basic network and combining the context information of the satellite image feature component, namely the region position information of the satellite image feature component;
(3) adding shape integrity constraint to the DeepLabV3+ semantic segmentation model based on the context information in the step (2) according to prior information that the satellite image feature component has similarity with a regular shape, and constructing a satellite image feature component semantic segmentation model based on the context information and the shape integrity prior;
(4) and (3) aiming at the model obtained in the step (3), training the model on the semantic segmentation data set of the satellite image feature component obtained in the step (1) to obtain a trained semantic segmentation model, and performing semantic segmentation on the satellite image by using the trained semantic segmentation model to obtain a semantic segmentation result.
In the step (1), the construction method of the semantic segmentation data set of the satellite image feature component is as follows:
(11) establishing an in-orbit scene of a simulation satellite by using a simulation software STK platform, setting orbit parameters, importing various types of satellite three-dimensional models, and demonstrating information such as changing background, angle, distance and the like while a target moves in space; simulating the in-orbit operation of the satellite by using STK software, recording a high-definition video with the width of 320-960, the height of 240-720 and 30-60 frames per second, reading the video by using a Matlab program, and intercepting and storing a color image in the in-orbit operation video of the satellite every 3-10 frames to obtain an original satellite simulation image;
(12) carrying out graying, illumination and noise adding and fuzzification preprocessing operations on the acquired original satellite simulation image to enable the image to be closer to an on-orbit satellite real image; the graying is to obtain a gray image by weighted average of RBG three channels, and the formula is as follows:
g(i,j)＝0.299×R(i,j)+0.578×G(i,j)+0.114×B(i,j)
wherein, (i, j) represents pixel coordinates in the image, R (i, j), G (i, j) and B (i, j) represent pixel values of RGB three channels of the image at the coordinates (i, j), respectively, and G (i, j) is a gray value of the calculated image at the coordinates (i, j);
the weighted average algorithm gives different weight coefficients to RGB components according to different sensitivities of human eyes to different colors, the highest sensitivity of the human eyes to green and the lowest sensitivity to blue;
after the image is grayed out, illumination is added to the image using the following formula:
Figure BDA0003165263550000031
wherein k is the central point illumination intensity, (i)0,j0) The center of the illumination is, r is the radius of the illumination, f (i, j) is the added illumination intensity at the coordinate (i, j), and the added illumination intensity is highest at the position of the center point and gradually reduced along the radius direction; simulating the real situation of the on-orbit satellite by adding Gaussian noise and Gaussian fuzzy operation to the picture after the illumination is added;
(13) and finally, processing the processed satellite image by using open source labeling software Labelme, labeling various clear visible feature components in the image along edge landmarks to obtain json files corresponding to the images, converting the json files into label graphs of usable satellite image feature component semantic segmentation data sets through python codes, and finally obtaining the satellite image feature component semantic segmentation data sets of the six feature components.
In the step (2), the method for constructing the DeepLabV3+ semantic segmentation model based on the context information comprises the following steps:
(21) firstly, a context information module is added to the rear end of an original DeepLabV3+ model, so that the semantic segmentation result of the original DeepLabV3+ model is obtained
Figure BDA0003165263550000032
Converting into a matrix with dimension of B × N × H × W, B representing Batchsize during training, H and W representing height and width of the image, N corresponding to the number of classes of the data set, and coordinates of points in the matrix are (B, N, H, W), wherein B, N, H, W represent training settings respectivelyThe low-to-high index of Batchsize, the category of the satellite image feature component, the height direction index of the satellite simulation image and the width direction index of the satellite simulation image, and the numerical value of each point represents the probability that the pixel at the corresponding position (h, w) in the satellite simulation image belongs to the category n;
(22) then, the matrix with the dimension of B multiplied by N multiplied by H multiplied by W obtained in the step (21) and the feature map F extracted by the original deep LabV3+ basic networkOMultiplying to obtain a matrix V with one dimension of B multiplied by N multiplied by K multiplied by 1, wherein K is a characteristic diagram FOEach Kx 1 vector correspondingly represents a category area and represents the position information of the satellite image feature component;
(23) using image feature maps FOCalculating a correlation matrix between the image pixels and the classification region with the matrix V, and carrying out weighted summation on the matrix V according to corresponding numerical values in the matrix correlation matrix to obtain a new characteristic diagram FCThe purpose of representing pixels through the category area is achieved, and the semantic segmentation precision is improved by enhancing the use of the model for the spatial information;
(24) finally F is putOAnd FCPerforming channel splicing to obtain a feature map with dimensions of Bx (2 x K) x H x W, performing 1 x 1 convolution, normalization and activation function operation based on the idea of None-local to obtain a feature map F, and performing convolution operation on the feature map F to obtain a final semantic segmentation result
Figure BDA0003165263550000041
The loss function of the DeepLabV3+ model based on the context information module consists of two parts: semantic segmentation result of original DeepLabV3+ model
Figure BDA0003165263550000042
Cross entropy loss of label graph of constructed satellite image feature component semantic segmentation data set with label truth value y
Figure BDA0003165263550000043
And semantic segmentation result of DeepLabV3+ model after adding context information module
Figure BDA0003165263550000044
Cross entropy loss with label truth y
Figure BDA0003165263550000045
Namely:
Figure BDA0003165263550000046
Figure BDA0003165263550000047
DeepLabV3+ model total loss L based on context informationtotThe expression of (a) is:
Ltot＝L1+αL2
wherein alpha is a weight coefficient used for adjusting the proportion of the context information and the single pixel information.
In the step (3), the method for constructing the semantic segmentation model based on the context information and the shape integrity prior comprises the following steps:
most of characteristic parts of the satellite images are regular in shape, including that satellite sailboards are approximately parallelogram, the antenna of the satellite is approximately round, the spray pipe of the satellite is approximately trapezoid, the lens barrel of the satellite is approximately rectangle, KKV and mechanical arms are approximately formed by splicing a plurality of rectangles, shape integrity constraints are added to DeepLabV3+ semantic segmentation based on context information by utilizing the prior information through a satellite image characteristic part shape integrity evaluation formula, and the evaluation formula of the shape integrity of the satellite image characteristic parts is as follows:
Figure BDA0003165263550000048
in the formula, S is the area of the satellite image feature component, C is the perimeter of the satellite image feature component, SC is a shape integrity coefficient, the value range is (0, 1), the closer the value is to 1, the more compact the shape of the satellite image feature component is, the more regular the shape is, and in actual use, the reciprocal of SC is taken as the shape integrity judgment standard;
firstly, semantic segmentation prediction results of DeepLabV3+ model satellite images added with context information modules
Figure BDA0003165263550000049
Converting the binary matrix into a binary matrix with dimensions of B multiplied by H multiplied by W multiplied by N through one-hot coding, and then calculating the gradient of the binary matrix with N B multiplied by H multiplied by W dimensions in the horizontal direction and the vertical direction
Figure BDA00031652635500000410
And
Figure BDA00031652635500000411
and calculating the sum of the absolute values of the values v (i, j) of each position in the matrix as the area of the characteristic component of the corresponding category of the matrix, wherein a loss function introduced by shape integrity prior is as follows:
Figure BDA0003165263550000051
wherein λ is a hyper-parameter used to calculate a stability prevention denominator of 0;
the loss function of the deplab v3+ semantic segmentation model based on context information and shape integrity prior is:
L＝Ltot+βLSCL＝L1+αL2+βLSCL
wherein beta is a weight coefficient, and the value is set to be 0.01-0.1.
In the step (4), the method for obtaining the semantic segmentation result comprises the following steps:
setting parameters and an optimization mode of a model training process, including a satellite image cutting size, a learning rate strategy, training steps and an output ratio;
the cutting size of the satellite image is set to be larger than one pixel of the original image in length and width; the learning rate strategy selects 'POLY', namely an exponential dynamic learning rate strategy, the initial learning rate is set to be 0.0005-0.0007, the initial training epoch is 280-300, the output ratio of the encoder is set to be 16, namely the output characteristic diagram of the encoder is 1/16 of the original image size, and the convolution expansion rate corresponding to the cavity is [12,24,36 ].
Compared with the prior art, the invention has the advantages that:
(1) according to the method, a mode of adding context information is adopted, an original DeepLabV3+ model is improved, the context information of the satellite image is utilized more fully, the mode further refines the segmentation result, and the segmentation precision is improved.
(2) The method adopts a mode of adding shape integrity prior constraint, adds a shape integrity loss function, and enhances the constraint effect of shape integrity prior information. The method can effectively improve the high-order inconsistency of the real label graph and the prediction segmentation graph, can improve the correctness of target semantic identification in the semantic segmentation graph, and integrally improves the semantic segmentation effect of the satellite image.
In a word, the method adopted by the invention has the advantages of simple principle and good semantic segmentation effect, and can achieve the purpose of accurate semantic segmentation of the satellite image feature components.
Drawings
FIG. 1 is a flow chart of a semantic segmentation method for a satellite image feature component according to the present invention;
FIG. 2 is an example of images before and after semantic segmentation obtained by the method of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be described clearly and completely with reference to the accompanying drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, rather than all embodiments, and all other embodiments obtained by a person skilled in the art based on the embodiments of the present invention belong to the protection scope of the present invention without creative efforts.
As shown in fig. 1, the specific implementation steps of the present invention are as follows:
step 1, (11) establishing an in-orbit scene of a simulation satellite by using a simulation software STK platform, setting orbit parameters, importing satellite three-dimensional models of various types, and demonstrating that the object moves in space and changes information such as background, angle, distance and the like; simulating the in-orbit operation of the satellite by using STK software, recording a high-definition video with the width of 320-960, the height of 240-720 and 30-60 frames per second, reading the video by using a Matlab program, and intercepting and storing a color image in the in-orbit operation video of the satellite every 3-10 frames to obtain an original satellite simulation image;
(12) carrying out graying, illumination and noise adding and fuzzification preprocessing operations on the acquired original satellite simulation image to enable the image to be closer to an on-orbit satellite real image; the graying is to obtain a gray image by weighted average of RBG three channels, and the formula is as follows:
g(i,j)＝0.299×R(i,j)+0.578×G(i,j)+0.114×B(i,j)
wherein, (i, j) represents pixel coordinates in the image, R (i, j), G (i, j) and B (i, j) represent pixel values of RGB three channels of the image at the coordinates (i, j), respectively, and G (i, j) is a gray value of the calculated image at the coordinates (i, j);
the weighted average algorithm gives different weight coefficients to RGB components according to different sensitivities of human eyes to different colors, the highest sensitivity of the human eyes to green and the lowest sensitivity to blue;
after the image is grayed out, illumination is added to the image using the following formula:
Figure BDA0003165263550000061
wherein k is the central point illumination intensity, (i)0,j0) The center of the illumination is, r is the radius of the illumination, f (i, j) is the added illumination intensity at the coordinate (i, j), and the added illumination intensity is highest at the position of the center point and gradually reduced along the radius direction; simulating the real situation of the on-orbit satellite by adding Gaussian noise and Gaussian fuzzy operation to the picture after the illumination is added;
(13) and finally, processing the processed satellite image by using open source labeling software Labelme, labeling various clear visible feature components in the image along edge landmarks to obtain json files corresponding to the images, converting the json files into label graphs of usable satellite image feature component semantic segmentation data sets through python codes, and finally obtaining the satellite image feature component semantic segmentation data sets of the six feature components.
Step 2, 21) firstly adding a context information module to the rear end of the original DeepLabV3+ model so as to segment the semantic meaning of the original DeepLabV3+ model
Figure BDA0003165263550000062
Converting into a matrix with dimension of B multiplied by N multiplied by H multiplied by W, wherein B represents Batchsize during training, H and W represent height and width of an image, N corresponds to the number of classes of a data set, coordinates of a midpoint of the matrix are (B, N, H, W), wherein B, N, H, W respectively represent an index of Batchsize from low to high in training setting, a class of a satellite image feature component, a satellite simulation image height direction index and a satellite simulation image width direction index, and a numerical value of each point represents the probability that a pixel at a corresponding position (H, W) in a satellite simulation image belongs to the class N;
(22) then, the matrix with the dimension of B multiplied by N multiplied by H multiplied by W obtained in the step (21) and the feature map F extracted by the original deep LabV3+ basic networkOMultiplying to obtain a matrix V with one dimension of B multiplied by N multiplied by K multiplied by 1, wherein K is a characteristic diagram FOEach Kx 1 vector correspondingly represents a category area and represents the position information of the satellite image feature component;
(23) using image feature maps FOCalculating a correlation matrix between the image pixels and the classification region with the matrix V, and carrying out weighted summation on the matrix V according to corresponding numerical values in the matrix correlation matrix to obtain a new characteristic diagram FCThe purpose of representing pixels through the category area is achieved, and the semantic segmentation precision is improved by enhancing the use of the model for the spatial information;
(24) finally F is putOAnd FCPerforming channel splicing to obtain a feature map with dimensions of Bx (2 x K) x H x W, performing 1 x 1 convolution, normalization and activation function operation based on the idea of None-local to obtain a feature map F, and performing convolution operation on the feature map F to obtain a final semantic segmentation result
Figure BDA0003165263550000071
The loss function of the DeepLabV3+ model based on the context information module consists of two parts: semantic segmentation result of original DeepLabV3+ model
Figure BDA0003165263550000072
Cross entropy loss of label graph of constructed satellite image feature component semantic segmentation data set with label truth value y
Figure BDA0003165263550000073
And semantic segmentation result of DeepLabV3+ model after adding context information module
Figure BDA0003165263550000074
Cross entropy loss with label truth y
Figure BDA0003165263550000075
Namely:
Figure BDA0003165263550000076
Figure BDA0003165263550000077
DeepLabV3+ model total loss L based on context informationtotThe expression of (a) is:
Ltot＝L1+αL2
wherein alpha is a weight coefficient used for adjusting the proportion of the context information and the single pixel information.
Step 3, most characteristic parts of the satellite image are regular in shape, the characteristic parts comprise satellite sailboards which are approximately parallelogram, the antenna of the satellite is approximately round, the spray pipe of the satellite is approximately trapezoid, the lens barrel of the satellite is approximately rectangle, the KKV and the mechanical arm are approximately formed by splicing a plurality of rectangles, the prior information is utilized to add shape integrity constraint to DeepLabV3+ semantic segmentation based on the context information through a satellite image characteristic part shape integrity evaluation formula, and the evaluation formula of the shape integrity of the satellite image characteristic parts is as follows:
Figure BDA0003165263550000078
in the formula, S is the area of the satellite image feature component, C is the perimeter of the satellite image feature component, SC is a shape integrity coefficient, the value range is (0, 1), the closer the value is to 1, the more compact the shape of the satellite image feature component is, the more regular the shape is, and in actual use, the reciprocal of SC is taken as the shape integrity judgment standard;
firstly, semantic segmentation prediction results of DeepLabV3+ model satellite images added with context information modules
Figure BDA0003165263550000081
Converting the binary matrix into a binary matrix with dimensions of B multiplied by H multiplied by W multiplied by N through one-hot coding, and then calculating the gradient of the binary matrix with N B multiplied by H multiplied by W dimensions in the horizontal direction and the vertical direction
Figure BDA0003165263550000082
And
Figure BDA0003165263550000083
and calculating the sum of the absolute values of the values v (i, j) of each position in the matrix as the area of the characteristic component of the corresponding category of the matrix, wherein a loss function introduced by shape integrity prior is as follows:
Figure BDA0003165263550000084
wherein λ is a hyper-parameter used to calculate a stability prevention denominator of 0;
the loss function of the deplab v3+ semantic segmentation model based on context information and shape integrity prior is:
L＝Ltot+βLSCL＝L1+αL2+βLSCL
wherein beta is a weight coefficient, and the value is set to be 0.01-0.1.
Step 4, setting parameters and an optimization mode of the model training process, including the cutting size of the satellite image, a learning rate strategy, training steps and an output ratio;
the cutting size of the satellite image is set to be larger than one pixel of the original image in length and width; the learning rate strategy selects 'POLY', namely an exponential dynamic learning rate strategy, the initial learning rate is set to be 0.0005-0.0007, the initial training epoch is 280-300, the output ratio of the encoder is set to be 16, namely the output characteristic diagram of the encoder is 1/16 of the original image size, and the convolution expansion rate corresponding to the cavity is [12,24,36 ].
As shown in fig. 2, the first column of the diagram is an original image, the second column is a true value image, and the third column is a semantic segmentation result of the original deep lab v3+ model, and it can be seen from the circled part in the diagram that the small-volume target mechanical arm is not recognized and segmented, and the antenna segmentation precision is poor; the fourth column is a semantic segmentation result of the satellite image obtained by the model provided by the invention, so that the small-volume target mechanical arm is identified and segmented, and the overall segmentation precision of the antenna is greatly improved. Therefore, the method can improve the overall details of the segmentation result and improve the semantic segmentation precision for the satellite image.
Those skilled in the art will appreciate that the invention may be practiced without these specific details. Although illustrative embodiments of the present invention have been described above to facilitate the understanding of the present invention by those skilled in the art, it should be understood that the present invention is not limited to the scope of the embodiments, but various changes may be apparent to those skilled in the art, and it is intended that all inventive concepts utilizing the inventive concepts set forth herein be protected without departing from the spirit and scope of the present invention as defined and limited by the appended claims.

Claims (5)

1. A semantic segmentation method for a satellite image feature component is realized based on context information and shape integrity prior, and comprises the following steps:
(1) acquiring an original satellite simulation image, and preprocessing the acquired original satellite simulation image so as to enable the original satellite simulation image to be closer to an on-orbit satellite real image, wherein the preprocessing comprises graying, illumination and noise adding and fuzzification processing operations on the original satellite simulation image; various satellite image characteristic components which are clearly visible are marked on the edge landmarks of the preprocessed satellite images, the satellite image characteristic components comprise six types of characteristic components including a sailboard, an antenna, a spray pipe, a lens cone, a KKV and a mechanical arm, and semantic segmentation data sets of the satellite image characteristic components of the six types of characteristic components are constructed;
(2) based on the data set obtained in the step (1), constructing a DeepLabV3+ semantic segmentation model based on the context information by utilizing the semantic segmentation result of the original DeepLabV3+ model and the feature map extracted by the original DeepLabV3+ basic network and combining the context information of the satellite image feature component, namely the region position information of the satellite image feature component;
(3) adding shape integrity constraint to the DeepLabV3+ semantic segmentation model based on the context information in the step (2) according to prior information that the satellite image feature component has similarity with a regular shape, and constructing a satellite image feature component semantic segmentation model based on the context information and the shape integrity prior;
(4) and (3) aiming at the model obtained in the step (3), training the model on the semantic segmentation data set of the satellite image feature component obtained in the step (1) to obtain a trained semantic segmentation model, and performing semantic segmentation on the satellite image by using the trained semantic segmentation model to obtain a semantic segmentation result.
2. The method for semantic segmentation of the satellite image feature component according to claim 1, characterized in that: in the step (1), the construction method of the semantic segmentation data set of the satellite image feature component is as follows:
(11) establishing an in-orbit scene of a simulation satellite by using a simulation software STK platform, setting orbit parameters, importing various types of satellite three-dimensional models, and demonstrating information such as changing background, angle, distance and the like while a target moves in space; simulating the in-orbit operation of the satellite by using STK software, recording a high-definition video with the width of 320-960, the height of 240-720 and 30-60 frames per second, reading the video by using a Matlab program, and intercepting and storing a color image in the in-orbit operation video of the satellite every 3-10 frames to obtain an original satellite simulation image;
(12) carrying out graying, illumination and noise adding and fuzzification preprocessing operations on the acquired original satellite simulation image to enable the image to be closer to an on-orbit satellite real image; the graying is to obtain a gray image by weighted average of RBG three channels, and the formula is as follows:
g(i,j)＝0.299×R(i,j)+0.578×G(i,j)+0.114×B(i,j)
wherein, (i, j) represents pixel coordinates in the image, R (i, j), G (i, j) and B (i, j) represent pixel values of RGB three channels of the image at the coordinates (i, j), respectively, and G (i, j) is a gray value of the calculated image at the coordinates (i, j);
the weighted average algorithm gives different weight coefficients to RGB components according to different sensitivities of human eyes to different colors, the highest sensitivity of the human eyes to green and the lowest sensitivity to blue;
after the image is grayed out, illumination is added to the image using the following formula:
Figure FDA0003165263540000021
wherein k is the central point illumination intensity, (i)0,j0) The center of the illumination is, r is the radius of the illumination, f (i, j) is the added illumination intensity at the coordinate (i, j), and the added illumination intensity is highest at the position of the center point and gradually reduced along the radius direction; simulating the real situation of the on-orbit satellite by adding Gaussian noise and Gaussian fuzzy operation to the picture after the illumination is added;
(13) and finally, processing the processed satellite image by using open source labeling software Labelme, labeling various clear visible feature components in the image along edge landmarks to obtain json files corresponding to the images, converting the json files into label graphs of usable satellite image feature component semantic segmentation data sets through python codes, and finally obtaining the satellite image feature component semantic segmentation data sets of the six feature components.
3. The method for semantic segmentation of the satellite image feature component according to claim 1, characterized in that: in the step (2), the method for constructing the DeepLabV3+ semantic segmentation model based on the context information comprises the following steps:
(21) firstly, a context information module is added to the rear end of an original DeepLabV3+ model, so that the semantic segmentation result of the original DeepLabV3+ model is obtained
Figure FDA0003165263540000022
Converting into a matrix with dimension of B multiplied by N multiplied by H multiplied by W, wherein B represents Batchsize during training, H and W represent height and width of an image, N corresponds to the number of classes of a data set, coordinates of a midpoint of the matrix are (B, N, H, W), wherein B, N, H, W respectively represent an index of Batchsize from low to high in training setting, a class of a satellite image feature component, a satellite simulation image height direction index and a satellite simulation image width direction index, and a numerical value of each point represents the probability that a pixel at a corresponding position (H, W) in a satellite simulation image belongs to the class N;
(22) then, the matrix with the dimension of B multiplied by N multiplied by H multiplied by W obtained in the step (21) and the feature map F extracted by the original deep LabV3+ basic networkOMultiplying to obtain a matrix V with one dimension of B multiplied by N multiplied by K multiplied by 1, wherein K is a characteristic diagram FOEach Kx 1 vector correspondingly represents a category area and represents the position information of the satellite image feature component;
(23) using image feature maps FOCalculating a correlation matrix between the image pixels and the classification region with the matrix V, and carrying out weighted summation on the matrix V according to corresponding numerical values in the matrix correlation matrix to obtain a new characteristic diagram FCThe purpose of representing pixels through the category area is achieved, and the semantic segmentation precision is improved by enhancing the use of the model for the spatial information;
(24) finally F is putOAnd FCPerforming channel splicing to obtain a feature map with dimensions of Bx (2 x K) x H x W, performing 1 x 1 convolution, normalization and activation function operation based on the idea of None-local to obtain a feature map F, and performing convolution operation on the feature map F to obtain a final semantic segmentation result
Figure FDA0003165263540000031
The loss function of the DeepLabV3+ model based on the context information module consists of two parts: semantic segmentation result of original DeepLabV3+ model
Figure FDA0003165263540000032
Cross entropy loss of label graph of constructed satellite image feature component semantic segmentation data set with label truth value y
Figure FDA0003165263540000033
And semantic segmentation result of DeepLabV3+ model after adding context information module
Figure FDA0003165263540000034
Cross entropy loss with label truth y
Figure FDA0003165263540000035
Namely:
Figure FDA0003165263540000036
Figure FDA0003165263540000037
DeepLabV3+ model total loss L based on context informationtotThe expression of (a) is:
Ltot＝L1+αL2
wherein alpha is a weight coefficient used for adjusting the proportion of the context information and the single pixel information.
4. The method for semantic segmentation of the satellite image feature component according to claim 1, characterized in that: in the step (3), the method for constructing the semantic segmentation model based on the context information and the shape integrity prior comprises the following steps:
the evaluation formula of the shape integrity of the satellite image feature part is as follows:
Figure FDA0003165263540000038
in the formula, S is the area of the satellite image feature component, C is the perimeter of the satellite image feature component, SC is a shape integrity coefficient, the value range is (0, 1), the closer the value is to 1, the more compact the shape of the satellite image feature component is, the more regular the shape is, and in actual use, the reciprocal of SC is taken as the shape integrity judgment standard;
firstly, semantic segmentation prediction results of DeepLabV3+ model satellite images added with context information modules
Figure FDA0003165263540000039
Converting the binary matrix into a binary matrix with dimensions of B multiplied by H multiplied by W multiplied by N through one-hot coding, and then calculating the gradient of the binary matrix with N B multiplied by H multiplied by W dimensions in the horizontal direction and the vertical direction
Figure FDA00031652635400000310
And
Figure FDA00031652635400000311
and calculating the sum of the absolute values of the values v (i, j) of each position in the matrix as the area of the characteristic component of the corresponding category of the matrix, wherein a loss function introduced by shape integrity prior is as follows:
Figure FDA00031652635400000312
wherein λ is a hyper-parameter used to calculate a stability prevention denominator of 0;
the loss function of the deplab v3+ semantic segmentation model based on context information and shape integrity prior is:
L＝Ltot+βLSCL＝L1+αL2+βLSCL
wherein beta is a weight coefficient, and the value is set to be 0.01-0.1.
5. The method for semantic segmentation of the satellite image feature component according to claim 1, characterized in that: in the step (4), the method for obtaining the semantic segmentation result comprises the following steps:
setting parameters and an optimization mode of a model training process, including a satellite image cutting size, a learning rate strategy, training steps and an output ratio;
the cutting size of the satellite image is set to be larger than one pixel of the original image in length and width; the learning rate strategy selects 'POLY', namely an exponential dynamic learning rate strategy, the initial learning rate is set to be 0.0005-0.0007, the initial training epoch is 280-300, the output ratio of the encoder is set to be 16, namely the output characteristic diagram of the encoder is 1/16 of the original image size, and the convolution expansion rate corresponding to the cavity is [12,24,36 ].
CN202110802773.1A 2021-07-15 2021-07-15 Semantic segmentation method for satellite image feature parts Active CN113486894B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110802773.1A CN113486894B (en) 2021-07-15 2021-07-15 Semantic segmentation method for satellite image feature parts

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110802773.1A CN113486894B (en) 2021-07-15 2021-07-15 Semantic segmentation method for satellite image feature parts

Publications (2)

Publication Number Publication Date
CN113486894A true CN113486894A (en) 2021-10-08
CN113486894B CN113486894B (en) 2023-07-25

Family

ID=77938919

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110802773.1A Active CN113486894B (en) 2021-07-15 2021-07-15 Semantic segmentation method for satellite image feature parts

Country Status (1)

Country Link
CN (1) CN113486894B (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114491694A (en) * 2022-01-17 2022-05-13 北京航空航天大学 Spatial target data set construction method based on illusion engine
WO2023092302A1 (en) * 2021-11-23 2023-06-01 宁德时代新能源科技股份有限公司 Image recognition method and apparatus, and computer-readable storage medium
CN116469014A (en) * 2023-01-10 2023-07-21 南京航空航天大学 Small sample satellite radar image sailboard identification and segmentation method based on optimized Mask R-CNN

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111310582A (en) * 2020-01-19 2020-06-19 北京航空航天大学 Turbulence degradation image semantic segmentation method based on boundary perception and counterstudy
AU2020103901A4 (en) * 2020-12-04 2021-02-11 Chongqing Normal University Image Semantic Segmentation Method Based on Deep Full Convolutional Network and Conditional Random Field

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111310582A (en) * 2020-01-19 2020-06-19 北京航空航天大学 Turbulence degradation image semantic segmentation method based on boundary perception and counterstudy
AU2020103901A4 (en) * 2020-12-04 2021-02-11 Chongqing Normal University Image Semantic Segmentation Method Based on Deep Full Convolutional Network and Conditional Random Field

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
陈天华;郑司群;于峻川;: "采用改进DeepLab网络的遥感图像分割", 测控技术, no. 11 *

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023092302A1 (en) * 2021-11-23 2023-06-01 宁德时代新能源科技股份有限公司 Image recognition method and apparatus, and computer-readable storage medium
US11823457B2 (en) 2021-11-23 2023-11-21 Contemporary Amperex Technology Co., Limited Image recognition method and apparatus, based on context representation, and computer-readable storage medium
CN114491694A (en) * 2022-01-17 2022-05-13 北京航空航天大学 Spatial target data set construction method based on illusion engine
CN116469014A (en) * 2023-01-10 2023-07-21 南京航空航天大学 Small sample satellite radar image sailboard identification and segmentation method based on optimized Mask R-CNN
CN116469014B (en) * 2023-01-10 2024-04-30 南京航空航天大学 Small sample satellite radar image sailboard identification and segmentation method based on optimized Mask R-CNN

Also Published As

Publication number Publication date
CN113486894B (en) 2023-07-25

Similar Documents

Publication Publication Date Title
CN110119728B (en) Remote sensing image cloud detection method based on multi-scale fusion semantic segmentation network
CN111325203B (en) American license plate recognition method and system based on image correction
CN111723585B (en) Style-controllable image text real-time translation and conversion method
CN109886066B (en) Rapid target detection method based on multi-scale and multi-layer feature fusion
CN113486894B (en) Semantic segmentation method for satellite image feature parts
CN111950453A (en) Optional-shape text recognition method based on selective attention mechanism
CN111191583A (en) Space target identification system and method based on convolutional neural network
CN111259828A (en) High-resolution remote sensing image multi-feature-based identification method
CN111401380B (en) RGB-D image semantic segmentation method based on depth feature enhancement and edge optimization
CN114187450A (en) Remote sensing image semantic segmentation method based on deep learning
CN113850324B (en) Multispectral target detection method based on Yolov4
CN115661144A (en) Self-adaptive medical image segmentation method based on deformable U-Net
CN111242026B (en) Remote sensing image target detection method based on spatial hierarchy perception module and metric learning
CN114022408A (en) Remote sensing image cloud detection method based on multi-scale convolution neural network
CN109977834B (en) Method and device for segmenting human hand and interactive object from depth image
CN114048822A (en) Attention mechanism feature fusion segmentation method for image
CN112270697B (en) Satellite sequence image moving target detection method combined with super-resolution reconstruction
CN111666852A (en) Micro-expression double-flow network identification method based on convolutional neural network
CN114038006A (en) Matting network training method and matting method
CN114724155A (en) Scene text detection method, system and equipment based on deep convolutional neural network
CN110648331A (en) Detection method for medical image segmentation, medical image segmentation method and device
CN114037640A (en) Image generation method and device
CN114943876A (en) Cloud and cloud shadow detection method and device for multi-level semantic fusion and storage medium
CN115205672A (en) Remote sensing building semantic segmentation method and system based on multi-scale regional attention
CN116958827A (en) Deep learning-based abandoned land area extraction method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant