CN112348786B - One-shot brain image segmentation method based on bidirectional correlation - Google Patents
One-shot brain image segmentation method based on bidirectional correlation Download PDFInfo
- Publication number
- CN112348786B CN112348786B CN202011186634.2A CN202011186634A CN112348786B CN 112348786 B CN112348786 B CN 112348786B CN 202011186634 A CN202011186634 A CN 202011186634A CN 112348786 B CN112348786 B CN 112348786B
- Authority
- CN
- China
- Prior art keywords
- image
- generator
- atlas
- reconstructed
- discriminator
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30016—Brain
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Biophysics (AREA)
- Molecular Biology (AREA)
- Biomedical Technology (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Medical Informatics (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Radiology & Medical Imaging (AREA)
- Quality & Reliability (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Magnetic Resonance Imaging Apparatus (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a one-shot brain image segmentation method based on bidirectional correlation, which comprises the following steps of: constructing an image transformation model comprising a generator G F Generator G B And two discriminators D for inputting the atlas x and the unlabelled image y into the generator G F Shunting processing to obtain forward mapping delta p F And the reconstructed images are distinguished by a discriminator D
Obtaining a reconstructed image with the unmarked image y
Will reconstruct the image
And atlas x input generator G B Get backward mapping Δ p B And the reconstructed images are distinguished by a discriminator D
And the image set x to obtain a reconstructed image
Through generator G F Discriminator D and generator G B Mutually constraining to obtain final forward mapping delta pF and obtaining marked reconstructed image through warp operation
The method simultaneously learns the forward mapping from the image set x to the unlabelled image y and the backward mapping from the unlabelled image y to the image set x through the image transformation model, and restricts the forward mapping through the backward mapping, so that the accuracy of the forward mapping is improved.
Description
Technical Field
The invention relates to the technical field of medical image segmentation, in particular to a one-shot brain image segmentation method based on bidirectional correlation.
Background
A common method for segmenting brain anatomical structures is segmentation by conventional machine learning, which relies on manually extracted features having limited feature representation capability and generalization capability, so Convolutional Neural Network (CNN) learning has been developed, which is completely data-driven and can automatically retrieve hierarchical features using self-learned advanced features, eliminating the limitations of manual features in conventional machine learning methods, with the help of fully labeled data, the convolutional neural network has a better effect in a fully supervised segmentation task, a segmentation algorithm based on forward relevance is used, i.e. the segmentation network is improved to learn the forward mapping of an atlas x to an unlabeled image y, and the learned relevance mapping is applied to the labeling of the atlas, so that the labeling of the unlabeled image y can be obtained, but such method learns only the forward mapping of the atlas x to the unlabeled image y, the forward mapping is constrained only by similarity loss and smoothness loss, and the mapping is highly difficult to control, so that the accuracy of mapping learning is low.
Disclosure of Invention
The invention aims to provide a one-shot brain image segmentation method based on bidirectional correlation, which is used for constructing an image transformation model, learning forward mapping from an atlas x to an unlabelled image y and backward mapping from the unlabelled image y to the atlas x through the image transformation model at the same time, and constraining the forward mapping through the backward mapping to improve the accuracy of the forward mapping.
In order to realize the purpose, the invention adopts the following technical scheme:
a one-shot brain image segmentation method based on bidirectional correlation comprises the following steps:
s1, acquiring and classifying brain anatomical structure images to obtain labeled images and unlabeled images y, and dividing the labeled images into a picture set x;
s2, constructing an image transformation model, wherein the image transformation model comprises a generator G F G, generator B And two discriminators D, generator G F Generator G B All match a discriminator D, generator G F And generator G B The structure is the same and comprises a twin coder and a decoder;
s3, input generator G for image set x and unlabelled image y F Shunting processing by generator G F Extracting relevant characteristic graphs by the twin encoder, fusing the characteristic graphs and inputting the characteristic graphs into a decoder, and matching the decoder with the twin encoder to obtain forward mapping delta p from an image set x to an unmarked image y F ；
S4, obtaining a reconstructed image by the aid of warp operation of the atlas x
Differentiating the reconstructed images by a discriminator D
With the unmarked image y, the discriminator D and the generator G F Make a countermeasure, so that the generator G F Generating a reconstructed image similar to the unmarked image y
S5, reconstructing the image
And atlas x input generator G B By means of a generator G B Extracting relevant characteristic graphs from the obtained twin encoder, fusing the characteristic graphs, inputting the fused characteristic graphs into a decoder, and matching the decoder with the twin encoder to obtain a reconstructed image
Backward mapping Δ p to atlas x B ；
S6, reconstructing the image
Obtaining reconstructed images by warp operation
Distinguishing reconstructed images by discriminator D
And atlas x, discriminator D and generator G B Make a contrast so that the generator G B Generating a reconstructed image similar to atlas x
S7, reconstructing the image
Similarity to atlas x, so that Generator G F Discriminator D and generator G B Mutually constraining to obtain final forward mapping delta pF, applying the forward mapping delta pF to the label of the atlas x through warp operation to obtain a labeled reconstructed image
Further, the generator G F And generator G B The twin encoder comprises a plurality of encoding sub-modules for extracting shallow features of the image, and the image set x and the unlabeled image y are processed in a shunting way through the encoding sub-modules or the image is reconstructed in a shunting way through the encoding sub-modules
And an atlas x, inputting the extracted relevant characteristic diagram into a double-attention module, respectively learning the spatial information and the channel information of the relevant characteristic diagram through the double-attention module, transmitting the spatial information and the channel information to a decoder, and decodingThe device comprises decoding sub-modules matched with the number of the encoder sub-modules.
Furthermore, the encoding sub-module has 5 encoding sub-modules, which are respectively a first encoding sub-module, a second encoding sub-module, a third encoding sub-module, a fourth encoding sub-module and a fifth encoding sub-module, and forms 1 processing stream, and processes the atlas x and the unlabeled image y through 2 processing streams, respectively, or processes the reconstructed image through 2 processing streams
And the x, 2 processing streams of the graph set are simultaneously connected with a fifth coding submodule, and the fifth coding submodule is connected with the double-attention module; the decoder submodule is provided with 5 first decoding submodules, a second decoding submodule, a third decoding submodule, a fourth decoding submodule and a fifth decoding submodule respectively, the first decoding submodule is connected with the double-notice module, the second decoding submodule receives the first decoding submodule and is in long connection with the fourth coding submodule of 2 processing streams respectively, the third decoding submodule receives the second decoding submodule and is in long connection with the third coding submodule of 2 processing streams respectively, the fourth decoding submodule receives the third decoding submodule and is in long connection with the second coding submodule of 2 processing streams respectively, the fifth decoding submodule receives the fourth decoding submodule, the fifth decoding submodule outputs forward mapping delta p from the image set x to the unmarked image y F Or the fifth decoding sub-module outputs the reconstructed image
Backward mapping Δ p to atlas x B 。
Further, the dual attention module includes a space attention module and a channel module, which capture information in space and channel dimensions respectively, and add the results of the space attention module and the channel attention module to obtain a new feature map.
Further, the coding sub-module is composed of ResNet-34 stacked by basic residual modules.
Further, the arbiter D adopts a PatchGAN arbiter.
Further, the image transformation model also comprises a loss module for supervising the image transformation model, wherein the loss module comprises similarity loss, smooth loss, space cycle consistency loss and antagonism loss, and the generator G is constrained through the similarity loss F To obtain similar reconstructed images
And an unlabelled image y; constraining generator G by smoothing loss F To obtain a smoothed forward mapping Δ p F And backward mapping Δ p B (ii) a Constraint generator G through spatial cyclic consistency loss B To obtain similar reconstructed images
And atlas x; the discriminator D is constrained by the penalty on antagonism.
Further, the similarity loss employs a local normalized correlation loss for ensuring local consistency, and the formula is as follows:
where t represents a voxel point in the image, f y (t) and
respectively representing the calculation of the unmarked image y and the reconstruction image
Local mean intensity function:
t i denotes the volume around t is l 3 Coordinates within the range.
After adopting the technical scheme, compared with the background technology, the invention has the following advantages:
1. the invention classifies by acquiring images of brain anatomyObtaining an image set x with labels and an unlabelled image y, and constructing an image transformation model with 2 generators and 2 discriminators D, wherein the 2 generators are generators G respectively F Generator G B Inputting the atlas x and the unlabelled image y into a generator G F The twin encoder and decoder performs forward mapping to obtain a forward mapping Δ p F By means of a discriminator D and a generator G F The countermeasure is carried out, so that the atlas x is subjected to warp operation to obtain a reconstructed image similar to the unlabelled image y
Will reconstruct the image
Input generator G B The twin encoder and decoder performs backward mapping to obtain backward mapping delta p B By means of a discriminator D and a generator G B Confrontation is carried out to obtain a reconstructed image similar to the atlas x
From the reconstructed image
And (3) carrying out circulation to enable backward mapping to restrict forward mapping to obtain final forward mapping delta pF, applying the forward mapping delta pF to the label of the atlas x through warp operation to obtain a labeled reconstructed image
Through generator G F Generator G B Respectively confronted with a discriminator D, so that the image change model obtains forward mapping delta p with the highest accuracy F And reconstructing the image with the label
2. The invention introduces loss modules, including similarity loss, smoothing loss, spatial cycle consistency loss and countermeasureSexual losses, through differential pairs of losses F G, generator B And 2 discriminators D carry out constraint to improve the accuracy of the image transformation model.
3. The discriminator D selects the PatchGAN discriminator, the PatchGAN discriminator can better discriminate the local part of the image, each patch is judged whether the image is true or false by dividing the image into a plurality of patches, and finally the judgment of the image level is obtained, so that the accuracy and the performance are superior to those of a common discriminator.
Drawings
FIG. 1 is a schematic flow chart of the present invention;
FIG. 2 is a schematic diagram of the overall structure of an image transformation model according to the present invention;
FIG. 3 is a schematic diagram of the operation of a twin encoder and decoder according to the present invention;
FIG. 4 is a schematic diagram of the decoder operation of the present invention;
FIG. 5 is a schematic structural diagram of a dual-note module of the present invention;
FIG. 6 is a schematic diagram of a discriminator D according to the invention;
FIG. 7 is a diagram illustrating the segmentation result of ICGAN forward mapping according to the present invention;
FIG. 8 is a schematic diagram showing the comparison of the segmentation results of the Siamenet and the ICGAN according to the present invention;
FIG. 9 is a schematic diagram of the segmentation results of the ICGAN forward mapping and backward mapping according to the present invention;
FIG. 10 is a graph comparing the results of the visual segmentation of the Simeneet, ICGAN and RCGAN of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
Examples
With reference to fig. 1 to 7, the present invention discloses a one-shot brain image segmentation method based on bidirectional correlation, which includes the following steps:
and S1, acquiring and classifying the brain anatomical structure image to obtain a labeled image and an unlabeled image y, and dividing the labeled image into an atlas x.
S2, constructing an image transformation model which comprises a generator G F G, generator B And two discriminators D, generators G F Generator G B All match a discriminator D, generator G F And generator G B All structurally identical comprise a twin encoder and a decoder.
S3, input generator G for image set x and unlabelled image y F Shunting treatment by generator G F Extracting relevant characteristic graphs by the twin encoder, fusing the characteristic graphs and inputting the characteristic graphs into a decoder, and matching the decoder with the twin encoder to obtain forward mapping delta p from an image set x to an unmarked image y F 。
S4, obtaining a reconstructed image by the aid of warp operation of the atlas x
Differentiating the reconstructed images by a discriminator D
With the unmarked image y, the discriminator D and the generator G F Make a countermeasure, so that the generator G F Generating a reconstructed image similar to the unmarked image y
S5, reconstructing the image
And atlas x input generator G B By means of a generator G B Extracting relevant characteristic graphs from the obtained twin encoder, fusing the characteristic graphs, inputting the fused characteristic graphs into a decoder, and matching the decoder with the twin encoder to obtain a reconstructed image
Backward mapping Δ p to atlas x B 。
S6, reconstructing the image
Obtaining reconstructed images by warp operation
Distinguishing reconstructed images by discriminator D
And atlas x, discriminator D and generator G B Make a contrast so that the generator G B Generating a reconstructed image similar to atlas x
S7, reconstructing the image
Similarity to atlas x, so that Generator G F Discriminator D and generator G B Mutually constraining to obtain final forward mapping delta pF, applying the forward mapping delta pF to the label of the atlas x through warp operation to obtain a labeled reconstructed image
Referring to fig. 2, ICGAN adds antagonism to the basis of the conventional GAN model, and enables generators and discriminators in the GAN model to compete to produce the best result; the image transformation model of this embodiment adopts CycleGAN as a basic frame, and adds cycle consistency on the basis of ICGAN, and proposes rcgan (reversible coresponsence gan), that is, the image transformation model, and learns the mapping from an atlas x to an unlabeled image y and the backward mapping from the unlabeled image y to the atlas x, and the backward mapping can be used for constraining the forward mapping, so that the final forward mapping is applied to the label of the atlas, thereby obtaining the label of the unlabeled image.
As shown in fig. 3 to 5, the generator G F And generator G B Also comprises a double attention module, wherein the twin encoder comprises a plurality of modules for extracting imagesThe coding submodule for shallow feature processes the image set x and the unmarked image y in a shunting way through the coding submodule, and reconstructs the image in a shunting way through the coding submodule
And an image set x, inputting the extracted relevant characteristic diagram into a double-attention module, respectively learning the spatial information and the channel information of the relevant characteristic diagram through the double-attention module, and transmitting the spatial information and the channel information to a decoder, wherein the decoder comprises decoding sub-modules matched with the number of the encoder sub-modules.
The coding sub-module comprises 5 first coding sub-modules, second coding sub-modules, third coding sub-modules, fourth coding sub-modules and fifth coding sub-modules which are respectively a first coding sub-module, a second coding sub-module, a third coding sub-module, a fourth coding sub-module and a fifth coding sub-module, 1 processing stream is formed, the picture set x and the unmarked image y are respectively processed through 2 processing streams, or the image is respectively processed and reconstructed through 2 processing streams
And the x, 2 processing streams of the graph set are simultaneously connected with a fifth coding submodule, and the fifth coding submodule is connected with the double-attention module; the decoder submodule is provided with 5 first decoding submodules, a second decoding submodule, a third decoding submodule, a fourth decoding submodule and a fifth decoding submodule, wherein the first decoding submodule is connected with the double attention module, the second decoding submodule receives the first decoding submodule and is respectively in long connection with the fourth coding submodules of 2 processing streams, the third decoding submodule receives the second decoding submodule and is respectively in long connection with the third coding submodules of 2 processing streams, the fourth decoding submodule receives the third decoding submodule and is respectively in long connection with the second coding submodules of 2 processing streams, the fifth decoding submodule receives the fourth decoding submodule, and the fifth decoding submodule outputs forward mapping delta p from an image set x to an image labeled y without the submodules F Or the fifth decoding sub-module outputs the reconstructed image
Backward mapping Δ p to atlas x B 。
The double attention module comprises a space attention module and a channel module, information is captured in the space dimension and the channel dimension respectively, and the results of the space attention module and the channel attention module are added to obtain a new characteristic diagram.
The coding sub-module is composed of ResNet-34 stacked by basic residual modules.
Referring to fig. 6, the discriminator D adopts a patch gan discriminator, and a general discriminator directly judges whether an input image is a real image or a reconstructed image at an image level, and directly outputs a vector representative, which is or is not, but the high-frequency part of the image has poor recovery capability, and in order to better locally judge the image, the patch gan discriminator divides the image into N × N patches and judges whether each patch is true or false; inputting a 160 × 160 × 128 three-dimensional image, generating a feature map with a size of 10 × 10 × 8 after passing through another convolution block with a convolution kernel size of 4 × 4 × 4 and a step size of 2, wherein each pixel represents a patch with a size of 16 × 16 × 16 on an original image, after passing through another convolution layer with a convolution kernel size of 4 × 4 × 4 and a step size of 1, judging the authenticity of each patch by using an activation function Sigmoid, a normalization layer of a PatchGAN discriminator uses BatchNormalization, and the activation functions of the other layers use LeakyReLU except the activation function of the last layer.
The image transformation model also comprises a loss module for supervising the image transformation model, wherein the loss module comprises similarity loss, smooth loss, space cycle consistency loss and antagonism loss, and a generator G is constrained through the similarity loss F To obtain similar reconstructed images
And an unlabelled image y; constraining generator G by smoothing loss F To obtain a smoothed forward mapping Δ p F And backward mapping Δ p B (ii) a Consistency loss constraint generator G through spatial loop B To obtain similar reconstructed images
And atlas x; the discriminator D is constrained by the penalty on antagonism.
The similarity loss adopts a local normalized correlation loss for ensuring local consistency, and the formula is as follows:
where t represents a voxel point in the image, f y (t) and
respectively representing the calculation of the unmarked image y and the reconstruction image
Local mean intensity function:
t i denotes the volume around t is l 3 Coordinates within the range, where l is preferably 3.
The smoothing loss that different generators have respectively is defined as:
wherein t ∈ Ω denotes Δ P F Or Δ P B All position spaces in, L smooth Approximation using spatial gradients between adjacent pixels along the x, y, z directions, respectively
finally, the smoothness penalty for the image variation model is:
L smooth (Δp F ,Δp B )＝L smooth (Δp F )+L smooth (Δp B )
in addition, the image change model can reconstruct the image, and the reconstruction effect is similar to the original atlas x or the unlabeled image y, so the embodiment adopts the L1 loss to strengthen the real atlas x and the reconstructed atlas x
The consistency between them is defined as:
the resistance loss is defined as:
wherein D (y) and
judging the unmarked image y and the reconstructed image for the discriminator D respectively
And (4) judging a result.
The cycleGAN is mapped through two reversible forward and backward learning processes, and a generator G B Learning x → y mappings and generating reconstructed images
The discriminator D reconstructs the image by training
The same distribution as the unlabelled imagery y, but since the supervision in the learning process is set-level, learning-only forward conversion will map the input images all to the same output image, hence by another generator G F Learn the y → x mapping, generator G B And generator G F The learned mapping is an idea, and there is a cyclic consistency F (G (x) ≈ x,the image transformation model learns the invertible mapping once
Similarly, first pass through generator G F Learn the mapping of y → x, then pass through the generator G B Learning the x → y mapping, once the invertible mapping learned by the image transformation model is
Wherein, the first and the second end of the pipe are connected with each other,
and
namely two sets of reversible mapping, the CycleGAN introduces the following cycle consistency loss formula of the image:
in summary, the loss module of the image transformation model can be expressed as:
wherein λ is 1 ＝1，λ 2 ＝3，λ 3 ＝10。
Evaluation of experiments
As shown in fig. 8-11, data collected from The Child and Adolescent neurodevelopmental program (CANDI) at The medical institute of massachusetts university disclose a series of brain structure images as images of experimental examples and MRBrainS18 data published by The MICCAI2018 race.
The evaluation index of the experimental evaluation adopts a Dice similarity coefficient to evaluate the segmentation accuracy of the model, and the accuracy is used for measuring the similarity between the manual labeling and the prediction result:
wherein y is s A manual annotation representing the test data is shown,
the experimental evaluation takes the average Dice coefficient and the standard deviation of Dice as an evaluation standard, reflects the discrete degree of the prediction result of the measured data and is defined as:
where n denotes the number of test data, dice i The Dice value representing the ith test datum,
the average Dice of all test data is shown, and the smaller the standard deviation is, the more stable the performance of the model is.
Verifying the effectiveness of the countermeasure thought of the image transformation model, comparing the segmentation results of the SimENet and the ICGAN, wherein the SimENet is a segmentation model without antagonism and cycle consistency, the main structure of the SimENet is the same as that of the image transformation model, the ICGAN is based on the traditional GAN model and antagonism, and the GAN model generator and the discriminator are subjected to countermeasure, and the result is shown in Table 1:
table 1 comparison of the results of the segmentation of the SiamENet and IGGAN network models
The average partition Dice of ICGAN on CANDI test set was 78.1%, which is 1.7% higher than SiamENet, while the variance was also increased from 5.2 to 3.1. It is noted that the Dice index of the worst case in the test set is improved from 70.4% to 72.4%, and the best case is also improved to some extent. On the MRBrains18 data set, the average Dice is improved from 76.8% to 79%, the result shows the positive effect of the countermeasures in learning the correlation mapping, and it can be seen that after the countermeasures are added, the network can learn the segmentation result more accurately, and the countermeasures can be verified to restrict the learning of the correlation.
As shown in fig. 7 and 8, the validity of the bidirectional reversible correlation mapping of the image transformation model is verified, the image transformation model is RCGAN, and the result of comparing RCGAN with ICGAN can directly indicate whether the learning backward mapping is valid, and the results are shown in tables 2 and 3:
TABLE 2 comparison of RCGAN and ICGAN results
TABLE 3 average Dice coefficient tables for ICGAN and RCGAN
As shown in tables 2 and 3, in the CANDI data set, compared with the segmentation result of ICGAN, the average Dice of RCGAN on the test set is 1.1% higher than that of ICGAN, and the variance is also increased from 3.1 to 2.8; in addition, compared with the SimENet, the average Dice of RCGAN is improved by 79.2% from 76.4%, is improved by 2.8 percentage points, and the variance is also improved by 2.4; on the mrbrain dataset, the average Dice of RCGAN was 1.2% higher than ICGAN and 3.4% higher than SiamENet; the Dice comparison of the SiamENet, the RCGAN and the RCGAN for segmenting each type of brain anatomical structure is detailed in table 3, and as can be seen from the table, the RCGAN can segment most of brain anatomical structures more accurately, and the segmentation result is more accurate due to mutual constraint of bidirectional mapping.
Referring to FIG. 9, an intermediate result of RCGAN training is shown, wherein the first group of pictures represents the forward mapping, and the four columns show the atlas x, the unlabeled image y, and the forward mapping Δ P B And reconstructing the image
The second group of pictures represent backward mapping and respectively show a picture set x, an unmarked image y and backward mapping delta P F And reconstructing the image
The training goal of the forward mapping process is to map the image set x to the unlabeled image y, and the goal of the backward training is to map the unlabeled image y to the image set x.
Referring to fig. 10, the results of the brain anatomy segmentation by using SiamENet, ICGAN and RCGAN are visualized respectively, and compared with SiamENet and ICGAN, the results of the RCGAN image transformation model are smoother and the segmentation results are more accurate.
The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.
Claims (6)
1. A one-shot brain image segmentation method based on bidirectional correlation is characterized by comprising the following steps:
s1, acquiring and classifying brain anatomical structure images to obtain labeled images and unlabeled images y, and dividing the labeled images into a picture set x;
s2, constructing an image transformation model, wherein the image transformation model comprises a generator G F Generator G B And two discriminators D, generator G F Generator G B All match a discriminator D, generator G F And generator G B The structure is the same and comprises a twin coder and a decoder;
the generator G F And generator G B The twin encoder comprises a plurality of encoding sub-modules for extracting shallow features of the image, and the positions of the encoding sub-modules are shuntedThe atlas x and the unmarked image y or the reconstructed image through the split processing of the coding sub-module
And an image set x, inputting the extracted relevant feature map into a double-attention module, respectively learning the spatial information and the channel information of the relevant feature map through the double-attention module, and transmitting the spatial information and the channel information to a decoder, wherein the decoder comprises decoding sub-modules matched with the number of the encoding sub-modules;
the coding sub-module comprises 5 first coding sub-modules, 5 second coding sub-modules, 5 third coding sub-modules, 5 fourth coding sub-modules and 5 fifth coding sub-modules which form 1 processing stream, the picture set x and the unmarked image y are respectively processed through 2 processing streams, or the image is respectively processed and reconstructed through 2 processing streams
And the x, 2 processing streams of the graph set are simultaneously connected with a fifth coding submodule, and the fifth coding submodule is connected with the double-attention module; the decoding submodule comprises 5 first decoding submodules, a second decoding submodule, a third decoding submodule, a fourth decoding submodule and a fifth decoding submodule, wherein the first decoding submodule is connected with the double attention module, the second decoding submodule receives the first decoding submodule and is respectively in long connection with the fourth coding submodules of 2 processing streams, the third decoding submodule receives the second decoding submodule and is respectively in long connection with the third coding submodules of 2 processing streams, the fourth decoding submodule receives the third decoding submodule and is respectively in long connection with the second coding submodules of 2 processing streams, the fifth decoding submodule receives the fourth decoding submodule, and the fifth decoding submodule outputs forward mapping delta p from an image set x to an unmarked image y F Or the fifth decoding sub-module outputs the reconstructed image
Backward mapping Δ p to atlas x B ；
S3, input the image set x and the unlabelled image y into the image generatorFinished device G F Shunting treatment by generator G F The twin encoder extracts the relevant characteristic diagram and inputs the characteristic diagram into the decoder after fusion, and the decoder is matched with the twin encoder to obtain the forward mapping delta p from the image set x to the unmarked image y F ；
S4, obtaining a reconstructed image by the aid of warp operation of the atlas x
Differentiating the reconstructed images by a discriminator D
With the unmarked image y, the discriminator D and the generator G F Make a countermeasure, so that the generator G F Generating a reconstructed image similar to the unmarked image y
S5, reconstructing the image
And atlas x input generator G B By means of a generator G B Extracting relevant characteristic graphs from the obtained twin encoder, fusing the characteristic graphs, inputting the fused characteristic graphs into a decoder, and matching the decoder with the twin encoder to obtain a reconstructed image
Backward mapping Δ p to atlas x B ；
S6, reconstructing the image
Obtaining a reconstructed image through warp operation
Differentiating the reconstructed images by a discriminator D
And atlas x, discriminator D and generator G B Make a contrast so that the generator G B Generating a reconstructed image similar to atlas x
S7, reconstructing the image
Similarity to atlas x, so that Generator G F Discriminator D and generator G B Constrained with each other to obtain the final forward mapping Δ p F Forward mapping of Δ p F Applying warp operation on labels of the atlas x to obtain labeled reconstructed images
2. The one-shot brain image segmentation method based on the bidirectional correlation as claimed in claim 1, wherein: the double attention module comprises a space attention module and a channel module, information is captured in the space dimension and the channel dimension respectively, and the results of the space attention module and the channel attention module are added to obtain a new characteristic diagram.
3. The one-shot brain image segmentation method based on the bidirectional correlation as claimed in claim 2, wherein: the coding sub-module is composed of ResNet-34 stacked by basic residual modules.
4. The one-shot brain image segmentation method based on the bidirectional correlation as claimed in claim 1, wherein: the discriminator D adopts a PatchGAN discriminator.
5. The one-shot brain image segmentation method based on the bidirectional correlation as claimed in claim 1, wherein: the image transformation model further comprises a model for supervising the imageTransforming the loss module of the model, wherein the loss module comprises similarity loss, smooth loss, space cycle consistency loss and antagonism loss, and the generator G is constrained through the similarity loss F To obtain similar reconstructed images
And an unlabelled image y; constraining generator G by smoothing loss F To obtain a smoothed forward mapping Δ p F And backward mapping Δ p B (ii) a Constraint generator G through spatial cyclic consistency loss B To obtain similar reconstructed images
And atlas x; the discriminator D is constrained by the penalty on antagonism.
6. The one-shot brain image segmentation method based on the bidirectional correlation as claimed in claim 5, wherein: the similarity loss employs a local normalized correlation loss for ensuring local consistency, and the formula is as follows:
where t represents a voxel point in the image, f y (t) and
respectively representing the calculation of the unmarked image y and the reconstruction image
Local mean intensity function:
t i denotes t surrounding volume as l 3 Coordinates within the range.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011186634.2A CN112348786B (en) | 2020-10-29 | 2020-10-29 | One-shot brain image segmentation method based on bidirectional correlation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011186634.2A CN112348786B (en) | 2020-10-29 | 2020-10-29 | One-shot brain image segmentation method based on bidirectional correlation |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112348786A CN112348786A (en) | 2021-02-09 |
| CN112348786B true CN112348786B (en) | 2022-09-13 |
Family
ID=74356592
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011186634.2A Active CN112348786B (en) | 2020-10-29 | 2020-10-29 | One-shot brain image segmentation method based on bidirectional correlation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112348786B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108062753A (en) * | 2017-12-29 | 2018-05-22 | 重庆理工大学 | The adaptive brain tumor semantic segmentation method in unsupervised domain based on depth confrontation study |
| CN109559332A (en) * | 2018-10-31 | 2019-04-02 | 浙江工业大学 | A kind of sight tracing of the two-way LSTM and Itracker of combination |
| CN110084863A (en) * | 2019-04-25 | 2019-08-02 | 中山大学 | A kind of multiple domain image conversion method and system based on generation confrontation network |
| CN111047594A (en) * | 2019-11-06 | 2020-04-21 | 安徽医科大学 | Tumor MRI weak supervised learning analysis modeling method and model thereof |
| CN111402259A (en) * | 2020-03-23 | 2020-07-10 | 杭州健培科技有限公司 | Brain tumor segmentation method based on multi-level structure relation learning network |
-
2020
- 2020-10-29 CN CN202011186634.2A patent/CN112348786B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108062753A (en) * | 2017-12-29 | 2018-05-22 | 重庆理工大学 | The adaptive brain tumor semantic segmentation method in unsupervised domain based on depth confrontation study |
| CN109559332A (en) * | 2018-10-31 | 2019-04-02 | 浙江工业大学 | A kind of sight tracing of the two-way LSTM and Itracker of combination |
| CN110084863A (en) * | 2019-04-25 | 2019-08-02 | 中山大学 | A kind of multiple domain image conversion method and system based on generation confrontation network |
| CN111047594A (en) * | 2019-11-06 | 2020-04-21 | 安徽医科大学 | Tumor MRI weak supervised learning analysis modeling method and model thereof |
| CN111402259A (en) * | 2020-03-23 | 2020-07-10 | 杭州健培科技有限公司 | Brain tumor segmentation method based on multi-level structure relation learning network |
Non-Patent Citations (4)
| Title |
|---|
| Conquering Data Variations in Resolution: A Slice-Aware Multi-Branch Decoder Network;Shuxin Wang et al.;《IEEE Transactions on Medical Imaging》;20200805;第39卷(第12期);第4174-4185页 * |
| LT-Net: Label Transfer by Learning Reversible Voxel-Wise Correspondence for One-Shot Medical Image Segmentation;Shuxin Wang et al.;《2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)》;20200805;第9159-9168页 * |
| VoxelMorph: A Learning Framework for Deformable Medical Image Registration;Guha Balakrishnan et al.;《IEEE Transactions on Medical Imaging》;20190204;第38卷(第8期);第1788-1800页 * |
| 基于改进的CycleGAN模型非配对的图像到图像转换;何剑华等;《玉林师范学院学报》;20180401;第39卷(第02期);第122-126页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112348786A (en) | 2021-02-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Song et al. | Region-based quality estimation network for large-scale person re-identification | |
| Zhang et al. | Learning 3d human shape and pose from dense body parts | |
| Liu et al. | Context-aware network for semantic segmentation toward large-scale point clouds in urban environments | |
| CN113283444B (en) | Heterogeneous image migration method based on generation countermeasure network | |
| CN111445548B (en) | Multi-view face image generation method based on non-paired images | |
| CN113128424B (en) | Method for identifying action of graph convolution neural network based on attention mechanism | |
| Qin et al. | Geotransformer: Fast and robust point cloud registration with geometric transformer | |
| Liu et al. | Pgfnet: Preference-guided filtering network for two-view correspondence learning | |
| EP4246458A1 (en) | System for three-dimensional geometric guided student-teacher feature matching (3dg-stfm) | |
| Wang et al. | Relation-attention networks for remote sensing scene classification | |
| Bartoccioni et al. | Lara: Latents and rays for multi-camera bird’s-eye-view semantic segmentation | |
| Yoa et al. | Self-supervised learning for anomaly detection with dynamic local augmentation | |
| Li et al. | ConvTransNet: A CNN-transformer network for change detection with multi-scale global-local representations | |
| Chen et al. | Transformer-based 3d face reconstruction with end-to-end shape-preserved domain transfer | |
| CN113592769B (en) | Abnormal image detection and model training method, device, equipment and medium | |
| Wang et al. | Distillbev: Boosting multi-camera 3d object detection with cross-modal knowledge distillation | |
| CN112329662B (en) | Multi-view saliency estimation method based on unsupervised learning | |
| CN112348786B (en) | One-shot brain image segmentation method based on bidirectional correlation | |
| Xu et al. | Unsupervised learning of depth estimation and camera pose with multi-scale GANs | |
| Zhou et al. | Human interaction recognition with skeletal attention and shift graph convolution | |
| CN112308833B (en) | One-shot brain image segmentation method based on circular consistent correlation | |
| Xu et al. | ReA-Net: A Multiscale Region Attention Network With Neighborhood Consistency Supervision for Building Extraction From Remote Sensing Image | |
| Li et al. | Geometry to the Rescue: 3D Instance Reconstruction from a Cluttered Scene | |
| Nan et al. | Multi-scale attention and structural relation graph for local feature matching | |
| Yang et al. | Improving Skeleton-based Action Recognitionwith Robust Spatial and Temporal Features |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |