CN113743450A - Hyperspectral image segmentation method based on non-local feature fusion - Google Patents

Hyperspectral image segmentation method based on non-local feature fusion Download PDF

Info

Publication number
CN113743450A
CN113743450A CN202110818767.5A CN202110818767A CN113743450A CN 113743450 A CN113743450 A CN 113743450A CN 202110818767 A CN202110818767 A CN 202110818767A CN 113743450 A CN113743450 A CN 113743450A
Authority
CN
China
Prior art keywords
output
feature map
multiplied
hyperspectral image
characteristic diagram
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110818767.5A
Other languages
Chinese (zh)
Inventor
郑建炜
刘豪
冯宇超
刘宇
李鹏飞
吴杰
许金山
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhejiang University of Technology ZJUT
Original Assignee
Zhejiang University of Technology ZJUT
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zhejiang University of Technology ZJUT filed Critical Zhejiang University of Technology ZJUT
Priority to CN202110818767.5A priority Critical patent/CN113743450A/en
Publication of CN113743450A publication Critical patent/CN113743450A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/253—Fusion techniques of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformation in the plane of the image
    • G06T3/40—Scaling the whole image or part thereof
    • G06T3/4084—Transform-based scaling, e.g. FFT domain scaling
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection

Abstract

The invention discloses a hyperspectral image segmentation method based on Non-local feature fusion, which utilizes a standard residual block to perform spectrum dimensionality reduction on a hyperspectral image, then performs operation of two times of downsampling feature fusion, namely two times of downsampling units and Non-local modules to output feature maps with the same size, effectively extracts long-distance information through a Non-local attention mechanism, performs feature fusion through adding operation, reduces the image to one fourth of the original image, and quadruples the number of channels to extract the features of the spectral image, performs upsampling through two times of upsampling, performs residual connection operation to obtain an output feature map, can fully utilize spectrum and space feature information to obtain more pixel features and global information, and finally utilizes the standard residual block to output a final probability distribution map so as to fully utilize the convolution advantages, the segmentation network is simplified, the parameter quantity is reduced, and the accuracy and the speed of pixel segmentation are obviously improved.

Description

Hyperspectral image segmentation method based on non-local feature fusion
Technical Field
The application belongs to the technical field of image processing, and particularly relates to a hyperspectral image segmentation method based on non-local feature fusion.
Background
The hyperspectral image classification is an important means for earth observation, and is widely applied to the aspects of refined agriculture, military detection, environmental monitoring and the like. With the continuous deepening of the technology, the capability of the hyperspectral image to express different objects is continuously enhanced by acquiring spectral information from continuous spectral bands of hundreds of surface objects, the spatial resolution of image data acquired by a ground observation mode including hyperspectral imaging is continuously improved, and the traditional pixel-by-pixel processing mode is not suitable for data of a high spatial resolution type.
Inspired by the internal structure of a visual system, with the emergence of a new deep learning technology, the classification of hyperspectral images is developed in a breakthrough manner, and a typical method is a Convolutional Neural Network (CNN) and is used for extracting a spectrum-space feature map with discriminability from the hyperspectral images. However, the two-dimensional convolution can only extract spatial information from the hyperspectral image, and information among spectral bands and global information are lost, which brings obvious influence on the segmentation of the hyperspectral image.
Disclosure of Invention
The application aims to provide a hyperspectral image segmentation method based on non-local feature fusion, and the accuracy of image pixel segmentation is remarkably improved.
In order to achieve the purpose, the technical scheme adopted by the application is as follows:
a hyperspectral image segmentation method based on non-local feature fusion comprises the following steps:
s1, acquiring a hyperspectral image to be segmented, wherein the size of the hyperspectral image to be segmented is DxHxWxN;
s2, reducing the dimension of the input hyperspectral image by using a standard residual error block, wherein the dimension of the hyperspectral image after dimension reduction is DxHxWxC;
step S3, respectively inputting the dimensionality reduced hyperspectral images into two branches of a down-sampling unit and a Non-local module, and adding feature maps output by the two branches;
step S4, inputting the feature map subjected to the adding operation in step S3 into two branches of a down-sampling unit and a Non-local module respectively, and adding the feature maps output by the two branches;
step S5, the feature map after the adding operation in the step S4 passes through an up-sampling unit, and the output of the up-sampling unit and the output of the step S3 are added;
step S6, the characteristic diagram obtained after the adding operation in the step S5 passes through an up-sampling unit, and the output of the up-sampling unit and the output of the step S2 are subjected to the adding operation;
and step S7, passing the output feature map with the size of D multiplied by H multiplied by W multiplied by C obtained after the addition operation in the step S6 through a standard residual block with the same structure as the standard residual block in the step S2 to obtain a probability distribution map with the dimension of D multiplied by H multiplied by W multiplied by N, and completing the segmentation of the hyperspectral image, wherein D represents the number of image groups, H represents the height of the image, W represents the width of the image, C represents the number of image channels, and N represents the preset number of categories for the segmentation of the hyperspectral image.
Several alternatives are provided below, but not as an additional limitation to the above general solution, but merely as a further addition or preference, each alternative being combinable individually for the above general solution or among several alternatives without technical or logical contradictions.
Preferably, the step S2, performing dimension reduction on the input hyperspectral image by using a standard residual block, includes:
s201, acquiring a hyperspectral image to be segmented, marking the hyperspectral image as F, wherein the dimension of the hyperspectral image is DxHxWxN;
s202, obtaining a characteristic diagram F' with the size of D multiplied by H multiplied by W multiplied by C by convolving the acquired hyperspectral image by a branch of two layers of BN + ReLu6 and 3 multiplied by 3;
s203, adding the feature maps F and F 'to obtain a feature map F', and then remolding the feature map F into a feature map F with dimensions of D multiplied by H multiplied by W multiplied by C0。
Preferably, in step S3, the method for inputting the reduced hyperspectral image into two branches, namely, a downsampling unit and a Non-local module, and adding the feature maps output by the two branches includes:
s301, acquiring the hyperspectral image after dimensionality reduction, and marking as F0The size is DxHxWxC;
s302, obtaining a hyperspectral image F0Passing through a down-sampling unit, namely a branch convolved by a convolution branch of 3x3x3 and a branch convolved by BN + ReLu6 and 3x3x3 of two layers respectively, and adding the outputs of the two branches to obtain the value of
Figure BDA0003171245100000021
Characteristic diagram F of1；
S303, performing hyperspectral image F0Through a Non-local module, namely, firstly, a residual block is passed through, 3D convolution and reshape operation of 1 multiplied by 1 are carried out in the residual block, and the characteristics output by the residual block and the original hyperspectral image F0Fusing to obtain a feature map F with the size of DxHxWxCoo；
S304, and then, converting the feature map FooPerforming Attention operation through three branches of 1 × 1 × 1 3D convolution, respectively converting into D sizeQ×HQ×WQ×CKThe feature map QueryMatrix of D × H × W × CKThe feature map Keyvector and the size of D multiplied by H multiplied by W multiplied by CVThen splitting the channel numbers of the feature maps QueryMatrix, KeyVector and ValueMasx, and obtaining tensors Q, K and V by using Unfold operation according to dimension expansion, wherein the formula is as follows:
Figure BDA0003171245100000034
Q＝Unfold(QueryMatrix)
Figure BDA0003171245100000035
K＝Unfold(KeyVector)
Figure BDA0003171245100000036
V＝Unfold(Valuematrix)
where Query Transform is a 1x1x1 convolution operation, Unfold (·) is a dimension expansion operation, Conv is a 1x1 3D convolution operation, CKAnd CVThe method comprises the steps of setting a preset hyper-parameter related to an image channel;
s305, calculating attention matrices a and O based on tensors Q, K and V as follows:
Figure BDA0003171245100000031
O＝AV
wherein Softmax (-) is a Softmax function, KTIs the transpose of tensor K, and the dimension of the attention matrix A is DQ×HQ×WQThe dimension of the attention matrix O is (D)Q×HQ×WQ)×CVWherein D isQ、HQAnd WQNumber of channels C determined by tensor QVDetermined by the matrix V;
s306, conducting Multi-Head Attention operation on the tensors Q, K and V, obtaining n groups Q, K and V by Q, K and V through n times of linear transformation respectively, wherein n is preset n-Head hyper-parameter, and then each group Qi、KiAnd ViObtaining corresponding output Head through Attention operationiI 1,2,3 … n, and finally splicing Head1…HeadnObtaining an output:
Headi＝Atttention(QWi Q,KWi K,VWi V)
MultiHead(Q,K,V)＝Concat(Head1,…,Headn)Wi O
in the formula (I), the compound is shown in the specification,
Figure BDA0003171245100000032
is a trained parameter matrix, and Attention () is an Attenttion operation;
s307, reshaping the obtained MultiHead (Q, K, V) into a matrix O with a constant size through Fold operation, and changing the number of output channels into C through 1 × 1 × 1 3D convolutionOObtaining an output Y:
Y＝Conv_1CO(Fold(MultiHead(Q,K,V)))
in the formula, the dimension of Y is the dimension D of the characteristic diagram QueryMatrixQ×HQ×WQ×CKDetermine and set CK＝CV＝COFold (. cndot.) is the number of passes by Fold operation, and the dimension of Y in this step is
Figure BDA0003171245100000033
S308, the characteristic diagram F obtained in the step S3021And the Y obtained in the step S307 is subjected to addition operation to obtain the dimension of
Figure BDA0003171245100000041
Characteristic diagram F of2。
Preferably, in step S4, the adding the feature map obtained in step S3 into two branches, i.e., the down-sampling unit and the Non-local module, and adding the feature maps obtained by the two branches includes:
s401, obtaining the size output by the step S3 as
Figure BDA0003171245100000042
Characteristic diagram F of2；
S402, setting the size to be
Figure BDA0003171245100000043
Characteristic diagram F of2Respectively inputting two branches, and obtaining the size of the down-sampling unit by the first branch
Figure BDA0003171245100000044
Characteristic diagram F of21The second branch of the Non-local module is used to obtain the size of
Figure BDA0003171245100000045
Characteristic diagram F of22；
S403, comparing feature maps F21And feature map F22Performing an addition operation to obtain a size of
Figure BDA0003171245100000046
Characteristic diagram output F of3。
Preferably, in step S5, the passing the feature map after the adding operation in step S4 through the upsampling unit, and adding the output of the upsampling unit and the output of step S3 includes:
s501, obtaining the size output in the step S4 as
Figure BDA0003171245100000047
Characteristic diagram F of3；
S502, converting the characteristic diagram F3By means of an up-sampling unit, i.e. two branches are input separately, the size is obtained by means of the first branch of the Non-local module
Figure BDA0003171245100000048
Characteristic diagram F of31The second branch of the transposed convolution by 3x3x3 yields a size of
Figure BDA0003171245100000049
Characteristic diagram F of32；
S503, comparing the feature map F31And feature map F32Performing a machining operation to obtain a dimension of
Figure BDA00031712451000000410
Characteristic diagram output F of33；
S504, feature map F33And the dimension in step S3 is
Figure BDA00031712451000000411
Characteristic diagram F of2Doing an add operation to obtain a dimension of
Figure BDA00031712451000000412
Characteristic diagram F of4。
Preferably, in step S6, the processing of passing the feature map obtained by the adding operation in step S5 through an up-sampling unit and adding the output of the up-sampling unit and the output of step S2 includes:
s601, obtaining the feature graph F with the dimension of D multiplied by H multiplied by W multiplied by C output by the step S20And the size output of step S5 is
Figure BDA0003171245100000051
Characteristic diagram F of4；
S602, feature map F4By means of an upsampling unit, i.e. by inputting two branches separately, a profile F of size DxHxWxC is obtained by means of the first branch of the Non-local module41After the second branch of the transposed convolution of 3x3x3, a feature map F with dimensions D × H × W × C is obtained42；
S603, matching the feature map F41And feature map F42Performing an addition operation to obtain a feature map output F with dimensions D × H × W × C43；
S604, converting the characteristic diagram F43And a feature map F with dimensions D × H × W × C in step S20Adding operation is carried out to obtain a feature map F with dimensions of D multiplied by H multiplied by W multiplied by C5。
Preferably, in step S7, the step of passing the output feature map with the size of D × H × W × C obtained by the addition operation in step S6 through the standard residual block with the same structure as the standard residual block in step S2 to obtain a probability distribution map with the dimension of D × H × W × N, so as to complete the hyperspectral image segmentation, includes:
s701, obtaining a characteristic diagram F after the addition operation of the step S65The size is DxHxWxC;
s702, obtaining a characteristic diagram F5Obtaining a feature map F with the size of D × H × W × C through standard residual block, namely through a branch of two-layer BN + ReLu6 and 3 × 3 convolution51；
S703, adding F51Remodelling into a probability distribution map F with dimensions D × H × W × Nout。
The hyperspectral image segmentation method based on Non-local feature fusion provided by the application utilizes a standard residual Block to perform spectrum dimensionality reduction on a hyperspectral image, then performs Down-Sampling feature fusion twice, namely outputs feature maps with the same size through two branches of a Down-Sampling Block and a Non-local module twice, effectively extracts long-distance information through a Non-local attention mechanism, then performs feature fusion through adding operation, reduces the image to one fourth of the original image, and changes the channel number to four times of the original image to extract the features of the spectral image, then performs Up-Sampling twice through an Up-Sampling Block, performs residual connection operation through skip-connection to obtain an output feature map, can fully utilize spectrum and spatial feature information to obtain more pixel features and global information, and finally utilizes the standard residual Block to output a final probability distribution map, therefore, the advantages of convolution are fully utilized, the segmentation network is simplified, the parameter quantity is reduced, and the accuracy and the speed of pixel segmentation are obviously improved.
Drawings
FIG. 1 is a flow chart of a hyperspectral image segmentation method based on non-local feature fusion according to the application;
FIG. 2 is a schematic diagram of a standard residual block structure of the present application;
FIG. 3 is a schematic diagram of the structure of a downsampling unit of the present application;
FIG. 4 is a schematic diagram of the present application utilizing Non-local module processing features;
FIG. 5 is a schematic diagram of the structure of a Non-local module of the present application;
fig. 6 is a schematic structural diagram of an upsampling unit of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description of the present application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.
In one embodiment, a hyperspectral image segmentation method based on non-local feature fusion is provided, long-distance information and attention mechanism comprehensive global information of a hyperspectral image are extracted through a non-local feature fusion U-Nets segmentation network, and then non-local features of down-sampling comprehensive global information of a CNN neural network are fused to achieve hyperspectral image pixel segmentation. The method fully utilizes the advantages of three-dimensional convolution, feature fusion and Non-local attention mechanism Non-local modules, and realizes the acquisition of global information so as to obtain a pixel segmentation result with high accuracy.
Specifically, as shown in fig. 1, the hyperspectral image segmentation method based on non-local feature fusion in the embodiment includes the following steps:
and step S1, acquiring the hyperspectral image to be segmented, wherein the size of the acquired hyperspectral image to be segmented is DxHxWxN.
In order to improve the data processing speed and accuracy, the acquired hyperspectral image needs to be preprocessed first, and the specific preprocessing step is shown in step S2.
And step S2, reducing the dimension of the input hyperspectral image by using a standard residual error module.
In this embodiment, the input hyperspectral image is reduced in dimension through a standard residual error module to achieve the purposes of reducing parameters and improving reasoning speed, as shown in fig. 2, the dimension reduction in this embodiment specifically includes the following steps:
s201, acquiring a hyperspectral image to be segmented, and marking the hyperspectral image as F, wherein the dimension of the hyperspectral image is D multiplied by H multiplied by W multiplied by N.
S202, obtaining a feature map F' with the size of D multiplied by H multiplied by W multiplied by C through a branch formed by convolution of BN (batch normalization) + ReLu6 (activation function) and 3 multiplied by 3.
S203, adding the feature maps F and F 'to obtain a feature map F', and reshaping (reshape) the feature map F into a feature map F with dimensions of D × H × W × C0。
And step 3, inputting the dimensionality reduced hyperspectral image into two branches of a Down-sampling Block and a Non-local module respectively, and adding the feature maps output by the two branches.
S301, obtaining the hyperspectral image F after dimensionality reduction0And the size is D × H × W × C.
S302, obtaining a hyperspectral image F0Passing through a down-sampling unit, as shown in FIG. 3, namely, respectively passing through a convolution branch of 3x3x3 and branches of two-layer convolution of BN + ReLu6 and 3x3x3, and adding the outputs of the two branches to obtain the output with the size of
Figure BDA0003171245100000071
Characteristic diagram F of1。
S303, as shown in FIG. 4, the hyperspectral image F0After passing through a Non-local Block (Non-local Block), as shown in fig. 5, a 1 × 1 × 1 3D convolution and reshape operation are performed in the residual Block once through a residual Block, and the output features of the residual Block are compared with the original hyperspectral image F0Fusing to obtain a feature map F with the size of DxHxWxCoo。
S304, and then, converting the feature map FooPerforming Attention operation through three branches of 1 × 1 × 1 3D convolution, respectively converting into D sizeQ×HQ×WQ×CKThe feature map QueryMatrix of D × H × W × CKThe feature map Keyvector and the size of D multiplied by H multiplied by W multiplied by CVThe feature map Valuemtrix, then the feature maps QueryMatrix, KeyVector and ValThe channel number of the uematrix is split, tensors Q, K and V are obtained by using Unfold operation according to dimension expansion, and the formula is as follows:
Figure BDA0003171245100000072
Q＝Unfold(QueryMatrix)
Figure BDA0003171245100000073
K＝Unfold(KeyVector)
Figure BDA0003171245100000074
V＝Unfold(Valuematrix)
where Query Transform is a 1x1x1 convolution operation, Unfold (·) is a dimension expansion operation, Conv is a 1x1 3D convolution operation, CKAnd CVIs a preset hyper-parameter related to the image channel.
S305, calculating attention matrices a and O based on tensors Q, K and V as follows:
Figure BDA0003171245100000075
O＝AV
wherein Softmax (-) is a Softmax function, KTIs the transpose of tensor K, and the dimension of the attention matrix A is DQ×HQ×WQThe dimension of the attention matrix O is (D)Q×HQ×WQ)×CVWherein D isQ、HQAnd WQNumber of channels C determined by tensor QVDetermined by the matrix V.
S306, conducting Multi-Head orientation operation on the tensors Q, K and V, obtaining n groups Q, K and V by Q, K and V through n times of linear transformation, wherein n is preset n-Head hyper-parameters, n corresponds to the manually set n-Head hyper-parameters, and then each group Qi、KiAnd ViObtaining corresponding output Head through Attention operationiI 1,2,3 … n, and finally splicing Head1…HeadnObtaining an output:
Headi＝Atttention(QWi Q,KWi K,VWi V)
MultiHead(Q,K,V)＝Concat(Head1,…,Headn)Wi O
in the formula (I), the compound is shown in the specification,
Figure BDA0003171245100000081
is a trained parameter matrix, Attention (. cndot.) is the Attenttion operation.
S307, the obtained Multihead (Q, K, V) is reshaped into a size-invariant (i.e. still being (D) by the Fold operationQ×HQ×WQ) ) is processed by a 3D convolution of 1 × 1 × 1 to change the number of output channels to COObtaining an output Y:
Figure BDA0003171245100000082
in the formula, the dimension of Y is the dimension D of the characteristic diagram QueryMatrixQ×HQ×WQ×CKDetermine and set CK＝CV＝COFold (. cndot.) is the number of passes by Fold operation, and the dimension of Y in this step is
Figure BDA0003171245100000083
S308, the characteristic diagram F obtained in the step S3021And the Y obtained in the step S307 is subjected to addition operation to obtain the dimension of
Figure BDA0003171245100000084
Characteristic diagram F of2。
And step S4, inputting the feature map subjected to feature fusion in step S3 into two branches of a downsampling unit and a Non-local module respectively, and adding the feature maps output by the two branches.
S401, obtaining the size output by the step S3 as
Figure BDA0003171245100000085
Characteristic diagram F of2。
S402, setting the size to be
Figure BDA0003171245100000086
Characteristic diagram F of2Respectively inputting two branches, and obtaining the size of the down-sampling unit by the first branch
Figure BDA0003171245100000087
Characteristic diagram F of21The second branch of the Non-local module is used to obtain the size of
Figure BDA0003171245100000088
Characteristic diagram F of22(ii) a Wherein the structures of the down-sampling unit and the Non-local module are identical to those of step S3.
S403, comparing feature maps F21And feature map F22Performing an addition operation to obtain a size of
Figure BDA0003171245100000089
Characteristic diagram output F of3。
In the implementation, the spectrum-space features are extracted by three-dimensional convolution and divided into three linear spaces, then a channel of a spectrum image is expanded by a multi-head attention mechanism and Unfold operation, and the global information of a feature map with any dimension is fused along the dimension of the channel by using a similar self-attention mechanism, so that long-distance information is extracted, and the parameter number and the reasoning speed are reduced.
The Non-local attention module of the Non-local module is operated to focus on global information in a given image to extract hyperspectral image information, for three-dimensional convolution operation and down-sampling operation, besides obtaining information by obtaining a spectrum-space feature map, the capability of feature representation can be improved by utilizing correlation among different channels, and then a feature map containing global information and local information and high-abundance information is obtained by adding the down-sampled feature map and the global feature map of the Non-local module.
The embodiment can fully acquire global information and long-distance information to achieve image pixel segmentation characteristics through a Non-local feature fused U-Nets network and a Non-local attention module of a Non-local module.
And step S5, passing the feature map subjected to feature fusion in the step S4 through an Up-Sampling Block (Up-Sampling Block), and adding the feature map and the output feature map in the step 3.
S501, obtaining the size output in the step S4 as
Figure BDA0003171245100000091
Characteristic diagram F of3。
S502, converting the characteristic diagram F3By means of an up-sampling unit, as shown in fig. 6, i.e. two branches are input separately, the first branch of the Non-local module being dimensioned to
Figure BDA0003171245100000092
Characteristic diagram F of31The second branch of the transposed convolution by 3x3x3 yields a size of
Figure BDA0003171245100000093
Characteristic diagram F of32。
S503, comparing the feature map F31And feature map F32Performing a machining operation to obtain a dimension of
Figure BDA0003171245100000094
Characteristic diagram output F of33。
S504, feature map F33And the dimension in step S3 is
Figure BDA0003171245100000095
Characteristic diagram F of2Doing an add operation to obtain a dimension of
Figure BDA0003171245100000096
Characteristic diagram F of4。
And step S6, passing the feature map obtained after the adding operation in step S5 through an Up-Sampling Block (Up-Sampling Block), and adding the output of the Up-Sampling Block and the output of step S2.
S601, obtaining the feature graph F with the dimension of D multiplied by H multiplied by W multiplied by C output by the step S20And the size output of step S5 is
Figure BDA0003171245100000097
Characteristic diagram F of4。
S602, feature map F4By means of an upsampling unit, i.e. by inputting two branches separately, a profile F of size DxHxWxC is obtained by means of the first branch of the Non-local module41After the second branch of the transposed convolution of 3x3x3, a feature map F with dimensions D × H × W × C is obtained42。
S603, matching the feature map F41And feature map F42Performing an addition operation to obtain a feature map output F with dimensions D × H × W × C43。
S604, converting the characteristic diagram F43And a feature map F with dimensions D × H × W × C in step S20Adding operation is carried out to obtain a feature map F with dimensions of D multiplied by H multiplied by W multiplied by C5。
In order to restore the pixel characteristics of the original image, Up-Sampling is carried out through two times of Up-Sampling blocks, so that the purpose of reducing the dimension can be achieved, and the specified image effect can be restored through the manually set hyper-parameters.
And S7, passing the output feature map with the size of D multiplied by H multiplied by W multiplied by C obtained after the addition operation in the step S6 through a standard residual block with the same structure as the standard residual block in the step S2 to obtain a probability distribution map with the dimension of D multiplied by H multiplied by W multiplied by N, and completing the segmentation of the hyperspectral image.
S701, obtaining a characteristic diagram F after the addition operation of the step S65And the size is D × H × W × C.
S702, obtaining a characteristic diagram F5Obtaining a feature map F with the size of D × H × W × C through standard residual block, namely through a branch of two-layer BN + ReLu6 and 3 × 3 convolution51。
S703, adding F51Remodelling into a probability distribution map F with dimensions D × H × W × Nout. Wherein D represents the number of image groupsH represents the image height, W represents the image width, C represents the number of image channels, and N represents the number of categories preset for hyperspectral image segmentation.
In the embodiment, the input image size D × H × W × C is changed into the final probability distribution map with the dimension D × H × W × N by using the standard residual block, where N is the number of classes of the segmentation task. And combining the class information of the segmentation task to output a final class segmentation probability map. It should be noted that the add operation in this embodiment includes add and concat, and the specific execution may distinguish the specific selected add operation mode according to the addition result of the feature, which is a conventional technique in the feature processing and is not specifically described here.
The technical features of the embodiments described above may be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the embodiments described above are not described, but should be considered as being within the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims (7)

1. A hyperspectral image segmentation method based on non-local feature fusion is characterized by comprising the following steps:
s1, acquiring a hyperspectral image to be segmented, wherein the size of the hyperspectral image to be segmented is DxHxWxN;
s2, reducing the dimension of the input hyperspectral image by using a standard residual error block, wherein the dimension of the hyperspectral image after dimension reduction is DxHxWxC;
step S3, respectively inputting the dimensionality reduced hyperspectral images into two branches of a down-sampling unit and a Non-local module, and adding feature maps output by the two branches;
step S4, inputting the feature map subjected to the adding operation in step S3 into two branches of a down-sampling unit and a Non-local module respectively, and adding the feature maps output by the two branches;
step S5, the feature map after the adding operation in the step S4 passes through an up-sampling unit, and the output of the up-sampling unit and the output of the step S3 are added;
step S6, the characteristic diagram obtained after the adding operation in the step S5 passes through an up-sampling unit, and the output of the up-sampling unit and the output of the step S2 are subjected to the adding operation;
and step S7, passing the output feature map with the size of D multiplied by H multiplied by W multiplied by C obtained after the addition operation in the step S6 through a standard residual block with the same structure as the standard residual block in the step S2 to obtain a probability distribution map with the dimension of D multiplied by H multiplied by W multiplied by N, and completing the segmentation of the hyperspectral image, wherein D represents the number of image groups, H represents the height of the image, W represents the width of the image, C represents the number of image channels, and N represents the preset number of categories for the segmentation of the hyperspectral image.
2. The non-local feature fusion-based hyperspectral image segmentation method according to claim 1, wherein the step S2 of performing dimensionality reduction on the input hyperspectral image by using a standard residual block comprises:
s201, acquiring a hyperspectral image to be segmented, marking the hyperspectral image as F, wherein the dimension of the hyperspectral image is DxHxWxN;
s202, obtaining a characteristic diagram F' with the size of D multiplied by H multiplied by W multiplied by C by convolving the acquired hyperspectral image by a branch of two layers of BN + ReLu6 and 3 multiplied by 3;
s203, adding the feature maps F and F 'to obtain a feature map F', and then remolding the feature map F into a feature map F with dimensions of D multiplied by H multiplied by W multiplied by C0。
3. The method for segmenting the hyperspectral image based on Non-local feature fusion according to claim 1, wherein in the step S3, the hyperspectral image after dimensionality reduction is respectively input into two branches of a downsampling unit and a Non-local module, and feature maps output by the two branches are added, and the method comprises:
s301, acquiring the hyperspectral image after dimensionality reduction, and marking as F0The size is DxHxWxC;
s302, obtaining a hyperspectral image F0Passing through a down-sampling unit, namely a branch convolved by a convolution branch of 3x3x3 and a branch convolved by BN + ReLu6 and 3x3x3 of two layers respectively, and adding the outputs of the two branches to obtain the value of
Figure FDA0003171245090000021
Characteristic diagram F of1；
S303, performing hyperspectral image F0Through a Non-local module, namely, firstly, a residual block is passed through, 3D convolution and reshape operation of 1 multiplied by 1 are carried out in the residual block, and the characteristics output by the residual block and the original hyperspectral image F0Fusing to obtain a feature map F with the size of DxHxWxCoo；
S304, and then, converting the feature map FooPerforming Attention operation through three branches of 1 × 1 × 1 3D convolution, respectively converting into D sizeQ×HQ×WQ×CKThe feature map QueryMatrix of D × H × W × CKThe feature map Keyvector and the size of D multiplied by H multiplied by W multiplied by CVThen splitting the channel numbers of the feature maps QueryMatrix, KeyVector and ValueMasx, and obtaining tensors Q, K and V by using Unfold operation according to dimension expansion, wherein the formula is as follows:
Figure FDA0003171245090000024
Q＝Unfold(QueryMatrix)
Figure FDA0003171245090000023
K＝Unfold(KeyVector)
Figure FDA0003171245090000025
V＝Unfold(Valuematrix)
where Query Transform is a 1x1x1 convolution operation, Unfold (·) is a dimension expansion operation, Conv is a 1x1 3D convolution operation, CKAnd CVThe method comprises the steps of setting a preset hyper-parameter related to an image channel;
s305, calculating attention matrices a and O based on tensors Q, K and V as follows:
Figure FDA0003171245090000022
O＝AV
wherein Softmax (-) is a Softmax function, KTIs the transpose of tensor K, and the dimension of the attention matrix A is DQ×HQ×WQThe dimension of the attention matrix O is (D)Q×HQ×WQ)×CVWherein D isQ、HQAnd WQNumber of channels C determined by tensor QVDetermined by the matrix V;
s306, conducting Multi-Head Attention operation on the tensors Q, K and V, obtaining n groups Q, K and V by Q, K and V through n times of linear transformation respectively, wherein n is preset n-Head hyper-parameter, and then each group Qi、KiAnd ViObtaining corresponding output Head through Attention operationiI 1,2,3 … n, and finally splicing Head1…HeadnObtaining an output:
Headi＝Atttention(QWi Q,KWi K,VWi V)
MultiHead(Q,K,V)＝Concat(Head1,…,Headn)Wi O
in the formula, Wi Q,Wi K，Wi V，Wi OIs a trained parameter matrix, and Attention () is an Attenttion operation;
s307, the obtained MultiHead (Q, K, V) is reshaped into a matrix O with a constant size through Fold operation, and then the matrix O is processed by 13D convolution by 1X1 changes the output channel number to COObtaining an output Y:
Figure FDA00031712450900000311
in the formula, the dimension of Y is the dimension D of the characteristic diagram QueryMatrixQ×HQ×WQ×CKDetermine and set CK＝CV＝COFold (. cndot.) is the number of passes by Fold operation, and the dimension of Y in this step is
Figure FDA0003171245090000031
S308, the characteristic diagram F obtained in the step S3021And the Y obtained in the step S307 is subjected to addition operation to obtain the dimension of
Figure FDA0003171245090000032
Characteristic diagram F of2。
4. The hyperspectral image segmentation method based on Non-local feature fusion according to claim 3, wherein in the step S4, the feature map after the adding operation in the step S3 is respectively input into two branches of a down-sampling unit and a Non-local module, and the feature map output by the two branches is added, and the method comprises the following steps:
s401, obtaining the size output by the step S3 as
Figure FDA0003171245090000033
Characteristic diagram F of2；
S402, setting the size to be
Figure FDA0003171245090000034
Characteristic diagram F of2Respectively inputting two branches, and obtaining the size of the down-sampling unit by the first branch
Figure FDA0003171245090000035
Characteristic diagram F of21The second branch of the Non-local module is used to obtain the size of
Figure FDA0003171245090000036
Characteristic diagram F of22；
S403, comparing feature maps F21And feature map F22Performing an addition operation to obtain a size of
Figure FDA0003171245090000037
Characteristic diagram output F of3。
5. The hyperspectral image segmentation method based on non-local feature fusion according to claim 4, wherein in the step S5, the feature map after the adding operation in the step S4 passes through an upsampling unit, and the adding operation is performed on the output of the upsampling unit and the output of the step S3, and the method comprises:
s501, obtaining the size output in the step S4 as
Figure FDA0003171245090000038
Characteristic diagram F of3；
S502, converting the characteristic diagram F3By means of an up-sampling unit, i.e. two branches are input separately, the size is obtained by means of the first branch of the Non-local module
Figure FDA0003171245090000039
Characteristic diagram F of31The second branch of the transposed convolution by 3x3x3 yields a size of
Figure FDA00031712450900000310
Characteristic diagram F of32；
S503, comparing the feature map F31And feature map F32Performing a machining operation to obtain a dimension of
Figure FDA0003171245090000041
Characteristic diagram output F of33；
S504, feature map F33And the dimension in step S3 is
Figure FDA0003171245090000042
Characteristic diagram F of2Doing an add operation to obtain a dimension of
Figure FDA0003171245090000043
Characteristic diagram F of4。
6. The hyperspectral image segmentation method based on non-local feature fusion according to claim 5, wherein in the step S6, the feature map obtained by the addition operation in the step S5 passes through an upsampling unit, and the addition operation is performed on the output of the upsampling unit and the output of the step S2, and the method comprises:
s601, obtaining the feature graph F with the dimension of D multiplied by H multiplied by W multiplied by C output by the step S20And the size output of step S5 is
Figure FDA0003171245090000044
Characteristic diagram F of4；
S602, feature map F4By means of an upsampling unit, i.e. by inputting two branches separately, a profile F of size DxHxWxC is obtained by means of the first branch of the Non-local module41After the second branch of the transposed convolution of 3x3x3, a feature map F with dimensions D × H × W × C is obtained42；
S603, matching the feature map F41And feature map F42Performing an addition operation to obtain a feature map output F with dimensions D × H × W × C43；
S604, converting the characteristic diagram F43And a feature map F with dimensions D × H × W × C in step S20Adding operation is carried out to obtain a feature map F with dimensions of D multiplied by H multiplied by W multiplied by C5。
7. The hyperspectral image segmentation method based on non-local feature fusion of claim 1, wherein in the step S7, the step S6 is performed to obtain the output feature map with the size of D × H × W × C, and the output feature map passes through the standard residual block with the same structure as the standard residual block in the step S2 to obtain the probability distribution map with the dimension of D × H × W × N, so as to complete hyperspectral image segmentation, and the method comprises:
s701, obtaining a characteristic diagram F after the addition operation of the step S65The size is DxHxWxC;
s702, obtaining a characteristic diagram F5Obtaining a feature map F with the size of D × H × W × C through standard residual block, namely through a branch of two-layer BN + ReLu6 and 3 × 3 convolution51；
S703, adding F51Remodelling into a probability distribution map F with dimensions D × H × W × Nout。
CN202110818767.5A 2021-07-20 2021-07-20 Hyperspectral image segmentation method based on non-local feature fusion Pending CN113743450A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110818767.5A CN113743450A (en) 2021-07-20 2021-07-20 Hyperspectral image segmentation method based on non-local feature fusion

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110818767.5A CN113743450A (en) 2021-07-20 2021-07-20 Hyperspectral image segmentation method based on non-local feature fusion

Publications (1)

Publication Number Publication Date
CN113743450A true CN113743450A (en) 2021-12-03

Family

ID=78728809

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110818767.5A Pending CN113743450A (en) 2021-07-20 2021-07-20 Hyperspectral image segmentation method based on non-local feature fusion

Country Status (1)

Country Link
CN (1) CN113743450A (en)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110874842A (en) * 2019-10-10 2020-03-10 浙江大学 Chest cavity multi-organ segmentation method based on cascade residual full convolution network
CN112541918A (en) * 2020-12-23 2021-03-23 山东师范大学 Three-dimensional medical image segmentation method based on self-attention mechanism neural network

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110874842A (en) * 2019-10-10 2020-03-10 浙江大学 Chest cavity multi-organ segmentation method based on cascade residual full convolution network
CN112541918A (en) * 2020-12-23 2021-03-23 山东师范大学 Three-dimensional medical image segmentation method based on self-attention mechanism neural network

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
ZHENGYANG WANG 等: "《Non-Local U-Nets for Biomedical Image Segmentation》", 《THE THIRTY-FOURTH AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE (AAAI-20)》, vol. 34, no. 04, pages 6316 - 6318 *
卫佳杰: "《基于U-Net和注意力机制的双源遥感图像语义分割方法研究》", 《中国优秀硕士学位论文全文数据库》, pages 1 - 94 *
郑欣: "《基于显微高光谱成像的乳腺肿瘤组织微阵列识别分析方法研究》", 《中国优秀硕士学位论文全文数据库》, pages 44 - 49 *

Similar Documents

Publication Publication Date Title
CN112396607B (en) Deformable convolution fusion enhanced street view image semantic segmentation method
CN111178316B (en) High-resolution remote sensing image land coverage classification method
CN111612807A (en) Small target image segmentation method based on scale and edge information
CN111259828A (en) High-resolution remote sensing image multi-feature-based identification method
CN113902915A (en) Semantic segmentation method and system based on low-illumination complex road scene
CN113673590B (en) Rain removing method, system and medium based on multi-scale hourglass dense connection network
CN116797787B (en) Remote sensing image semantic segmentation method based on cross-modal fusion and graph neural network
CN114862731B (en) Multi-hyperspectral image fusion method guided by low-rank priori and spatial spectrum information
CN114724155A (en) Scene text detection method, system and equipment based on deep convolutional neural network
CN113239736A (en) Land cover classification annotation graph obtaining method, storage medium and system based on multi-source remote sensing data
CN114266957A (en) Hyperspectral image super-resolution restoration method based on multi-degradation mode data augmentation
CN114998167A (en) Hyperspectral and multispectral image fusion method based on space-spectrum combined low rank
CN115471757A (en) Hyperspectral image classification method based on convolutional neural network and attention mechanism
CN114937206A (en) Hyperspectral image target detection method based on transfer learning and semantic segmentation
CN114708455A (en) Hyperspectral image and LiDAR data collaborative classification method
CN116977747B (en) Small sample hyperspectral classification method based on multipath multi-scale feature twin network
Kazimi et al. Semantic segmentation of manmade landscape structures in digital terrain models
Shi et al. Improvement of deep learning Method for water body segmentation of remote sensing images based on attention modules
Jiang et al. Semantic segmentation network combined with edge detection for building extraction in remote sensing images
CN114723951B (en) Method for RGB-D image segmentation
CN113743450A (en) Hyperspectral image segmentation method based on non-local feature fusion
CN115909077A (en) Hyperspectral image change detection method based on unsupervised spectrum unmixing neural network
CN115424140A (en) Satellite-borne mass image data distributed cooperative rapid high-precision processing system and method
CN113780305A (en) Saliency target detection method based on interaction of two clues
Kanthi et al. A 3D-Inception CNN for Hyperspectral Image Classification

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination