CN115578559A - Ultrasonic breast nodule end-to-end segmentation method based on multi-scale and cross-space fusion - Google Patents
Ultrasonic breast nodule end-to-end segmentation method based on multi-scale and cross-space fusion Download PDFInfo
- Publication number
- CN115578559A CN115578559A CN202211177184.XA CN202211177184A CN115578559A CN 115578559 A CN115578559 A CN 115578559A CN 202211177184 A CN202211177184 A CN 202211177184A CN 115578559 A CN115578559 A CN 115578559A
- Authority
- CN
- China
- Prior art keywords
- convolution
- scale
- fusion
- module
- cross
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/52—Scale-space analysis, e.g. wavelet analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/771—Feature selection, e.g. selecting representative features from a multi-dimensional feature space
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10132—Ultrasound image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30068—Mammography; Breast
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/03—Recognition of patterns in medical or anatomical images
- G06V2201/032—Recognition of patterns in medical or anatomical images of protuberances, polyps nodules, etc.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Health & Medical Sciences (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Medical Informatics (AREA)
- Databases & Information Systems (AREA)
- Molecular Biology (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Radiology & Medical Imaging (AREA)
- Quality & Reliability (AREA)
- Ultra Sonic Daignosis Equipment (AREA)
Abstract
The invention provides an ultrasonic breast nodule end-to-end segmentation method based on multi-scale and cross-space fusion, which comprises the following steps: the encoding-decoding structure of the traditional U-Net is taken as a main frame, wherein the encoder replaces the convolution operation of the traditional Unet through a multi-scale feature extraction and fusion module to acquire context information of different receptive fields; on a bottleneck layer, fusing information under different receptive fields acquired by the multi-scale feature extraction and fusion module through a receptive field self-adaptive aggregation module to generate a weight matrix, and performing feature screening on deep semantic channels through weights to highlight semantic features related to a segmentation result; on a jump connection between an encoder and a decoder, relieving semantic difference between peer layers of the encoder and the decoder by adding a cross-space residual error fusion module; and verifying the segmentation effect through an ablation experiment.
Description
Technical Field
The invention relates to the technical field, in particular to an ultrasonic breast nodule end-to-end segmentation method based on multi-scale and cross-space fusion.
Background
According to the statistics of the world health organization, breast cancer becomes one of the most common malignant tumors of women in the world, and seriously threatens the physical and psychological health of women. Clinical experience shows that the exact pathogenesis of the breast cancer is not clear, related high-risk factors are difficult to control, and primary etiology prevention is difficult to realize, so that the prevention and control of the breast cancer are mainly based on secondary prevention of 'early discovery, early diagnosis and early treatment'. Therefore, early screening and diagnosis are key factors in reducing breast cancer mortality. Nowadays, noninvasive breast diagnosis is rapidly developed, including technologies such as X-ray, magnetic resonance imaging, ultrasonic imaging, and the like. The ultrasonic imaging has the advantages of no radiation damage, easy use, capability of observing and imaging at any angle, high imaging speed, low price and the like, and becomes the most important early diagnosis mode and means of the breast cancer. However, the ultrasonic imaging has the defects of high noise, uneven gray scale, low contrast ratio and the like, and the breast nodules have different shapes and textures, and good and malignant nodules are difficult to distinguish visually, so that certain difficulty is brought to the ultrasonic breast nodule detection. In order to solve the above problems, a Computer Aided Diagnosis (CAD) system based on an artificial intelligence algorithm is used in diagnosis of an ultrasound breast image.
In early algorithmic studies, learners mostly used conventional machinesThe learning method segments the ultrasound breast nodules. For example, 2012, jiang et al [1] Firstly, detecting an initial ultrasonic breast nodule region by adopting an Adaboost + Haar frame, then further screening a detected nodule set by using a support vector machine, and finally thinning and segmenting the nodule by using a random walk algorithm. Shan et al [2] An NLM (neural L-means) generalized clustering method is provided and used for fuzzy boundary segmentation of single nodules of the ultrasonic mammary gland. 2016, luo et al [3] The particle swarm algorithm and the optimization graph theory algorithm are combined to realize the segmentation of the ultrasonic breast nodules, but two diagonal lines are required to be manually selected to determine the nodule area. 2018, liu et al [4] The nodule contour is positioned and initialized by adopting an adaptive threshold method and a morphological filtering method, and is further refined by improving an active contour method. Lotfollahi et al [5] An improved active contour method is provided, and ultrasonic mammary image segmentation is carried out by combining with a neutrosophic theory, so that inherent speckle noise of an ultrasonic image is overcome, and the defect that a nodule contour needs manual operation is overcome. All the above methods can realize the segmentation of the ultrasonic breast nodules, but do not get rid of the tedious process of multiple steps or artificial initial contours, so researchers consider a new approach to seek a relatively simple and efficient segmentation method.
In recent years, as deep learning develops in the field of image processing, its role in the medical field is gradually emerging, and in the task of ultrasound breast nodule segmentation, many scholars begin to study methods using deep learning in view of the shortcomings of the conventional machine methods. 2019, han et al [6] A multi-scale feature extraction network (BUS-S) and a double-attention fusion network (BUS-E) are used for segmenting the ultrasonic breast nodules, but the training load is too large. Zhuang et al [7] On the basis of an original UNet model, each common convolution layer is introduced into cross-layer connection to relieve gradient which gradually disappears, expansion convolution is used in a bottleneck layer to obtain more characteristics, meanwhile, an attention gate module is used for replacing cross-layer connection of coding and decoding, background information is suppressed, and therefore segmentation of breast nodules is improvedAnd (4) performance. In 2020 Byra et al [8] The standard convolution on the coding and decoding path is replaced by the selective kernel convolution, the convolution can self-adaptively adjust the receptive field, and the segmentation problem of the breast nodule with changeable morphology is solved. Vakanski et al [9] Attention is added to the pooling operation for focusing on the ultrasound breast nodule region, but the segmentation of the nodule fuzzy boundary is not effective. 2021, xue et al [10] And combining the multi-layer context information with the breast lesion boundary detection, and refining the boundary quality to improve the segmentation result. Iqbal et al [11] The method is based on a multi-scale double-attention network and used for ultrasonic breast nodule image segmentation, all standard convolutions in Unet are replaced by the multi-scale convolutions and used for feature extraction, double-attention adaptive learning advanced features are introduced after upsampling operation, and the nodule segmentation effect on fuzzy boundaries is poor. 2022 Punn et al [12] The method is characterized in that ultrasonic breast nodule regions of different sizes are adapted by fusing four different scale convolutions, more nodule feature information is reserved by using mixed pooling, and in addition, coding path adjacent spatial features and decoding path peer-level spatial features are combined with attention to pay attention to correlation in different spatial dimensions so as to better identify the nodule regions. Chen et al [13] The hybrid adaptive attention is provided to improve the Unet network, so that the network has the capability of adaptively adjusting the receptive field on a channel and a space to capture the characteristics of different dimensions, and the method obtains a better segmentation effect.
Although the deep learning method has better segmentation effect than the traditional machine learning method, the clinical requirements of the sonographer cannot be met in the segmentation accuracy.
Reference documents:
[1]JIANG P,PENG J,ZHANG G,etal.Learning-based automatic breast tumor detection and segmentation in ultrasound images[C]//2012 9th IEEE International Symposium on Biomedical Imaging(ISBI).IEEE,2012:1587-1590.
[2]SHAN J,CHENG H D,WANG Y.A novel segme-ntation method for breast ultrasound images based on neutrosophic l-means clustering[J].Me-dicalphysics,2012,39(9):5669-5682.
[3]LUO Y,HAN S,HUANG Q.A novel graph-based segmentation method for breast ultrasound images[C]//2016 International Conference on Digital Image Computing: Techniques and Applications(DICTA).IEEE,2016:1-6.
[4]LIU L,LI K,QIN W,et al.Automated breast tumor detection and segmentation with a novel computati-onal framework of whole ultrasound images[J].Medical&biological engineering& computing,2018,56(2):183-199.
[5]LOTFOLLAHI M,GITY M,YE J Y,etal.Segme-ntation of breast ultrasound images based on active contours using neutrosophic theory[J].Journal of Medical Ultrasonics,2018,45(2):205-212.
[6]HAN L,HUANG Y,DOU H,et al.Semi-supervised segmentation of lesion from breast ultrasound images with attentional generative adversarial network[J].Computer methods and programs in biomedicine,2020,189:105275.
[7]ZHUANG Z,LI N,JOSEPH RAJ A N,et al.An RDAU-NET model for lesion segmentation in breast ultrasound images[J].PloS one,2019,14(8):e0221535.
[8]BYRA M,JAROSIK P,SZUBERT A,et al.Breast mass segmentation in ultrasound with selective kernel U-Ne t convolutional neural network[J].Biomedical Signal Processing and Control,2020,61:102027.
[9]VAKANSKI A,XIAN M,FREER P E.Attention-enriched deep learning model for breast tumor segmentation in ultrasound images[J]. Ultrasoun-d in Medicine&Biology,2020,46(10):2819-2833.
[10]XUE C,ZHU L,Fu H,et al.Global guidance network for breast lesion segmentation in ultraso-und images[J].Medical image analysis, 2021,70:101989.
[11]IQBAL A,SHARIF M.MDA-Net:Multiscale dual attention-based network for breast lesion segmentation using ultrasound images[J].Journal of King Saud University-Computer and Informa-tion Sciences,2021.
[12]PUNN N S,AGARWAL S.RCA-IUnet:a residual cross spatial attention-guided inception U-Net model for tumor segmentation in breast ultrasou-nd imaging[J].Machine Vision and Applications,2022,33(2):1-10.
[13]CHEN G,DAI Y,ZHANG J,et al.AAU-net:An Adaptive Attention U-net for Breast Lesions Segmentation in Ultrasound Images[J].arXiv preprint arXiv:2204.12077,2022.
[14]HAN K,WANG Y,TIAN Q,et al.Ghostnet:More features from cheap operations[C]//Procee-dings of the IEEE/CVF conference on computer vision and pattern recognition.2020:1580-1589.
[15]Wu H,Wang W,Zhong J,et al.Scs-net:A scale and context sensitive network for retinal vessel segmentation[J].Medical Image Analysis,2021,70:102025.
[18]OKTAY O,SCHLEMPER J,FOLGOC L L,et al.Attention u-net: Learning where to look for the pancreas[J].arXiv preprint arXiv: 1804.03999,2018.
[19]JHA D,SMEDSRUD P H,RIEGLER M A,et al.Resunet++:An advanced architecture for medical image segmentation[C]//2019 IEEE International Symposium on Multimedia(ISM).IEEE,2019:225-2255.
[20]LI X,WANG W,HU X,et al.Selective kernel networks[C] //Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.2019:510-519.
[21]LOU A,GUAN S,LOEW M.Cfpnet-m:A light-weight encoder-decoder based network for multi-modal biomedical image real-time segmentation[J].arXiv preprint arXiv:2105.04075,2021.
[22]NING Z,WANG K,ZHONG S,et al.CF2-Net:Coarse-to-fine fusion convolutional network for breast ultrasound image segmentation[J].arXiv preprint arXiv:2003.10144,2020.
[23]SHAREEF B,VAKANSKI A,Xian M,et al.Estan: Enhanced small tumor-aware network for breast ultrasound image segmentation[J].arXiv preprint arXiv:2009.12894,2020.
[24]PI J,QI Y,LOU M,et al.FS-UNet:Mass segment-ation in mammograms using an encoder-decoder architecture with feature strengthening[J].Comp-uters in Biology and Medicine,2021,137:104800
disclosure of Invention
According to the proposal, the clinical requirements of the sonographer still cannot be met in the aspect of segmentation accuracy, in addition, the imaging characteristics of the ultrasonic image and the diversification of the shape of the breast nodule objectively increase the technical problem of the segmentation difficulty of the breast nodule, and the ultrasonic breast nodule end-to-end segmentation method based on multi-scale and cross-space fusion is provided. The invention mainly adopts a traditional U-Net coding-decoding structure as a main frame, and a coder of the traditional U-Net coding-decoding structure replaces the convolution operation of the traditional U-Net through a Multi-scale Feature Extraction and Fusion Module (MFEF) to obtain context information of different receptive fields so as to improve the Extraction and expression capability of the network shallow Feature; on a bottleneck layer, a Receptive-field Adaptive Aggregation module (RAA) is adopted to fuse information under a multi-scale Receptive field to generate a weight matrix, and a deep semantic channel is subjected to feature screening through weight, so that semantic features related to a segmentation result are highlighted, and the extraction and expression capacity of deep features in a coding stage is enhanced; a Cross-spatial residual Fusion module (CRF) is added on the jump connection between the encoder and the decoder to alleviate the semantic difference between the peer layers of the codec, thereby better compensating the information loss in the decoding phase. Through ablation experiments, the optimal segmentation effect can be obtained by jointly using the three modules.
The technical means adopted by the invention are as follows:
an ultrasonic breast nodule end-to-end segmentation method based on multi-scale and cross-space fusion comprises the following steps:
step 1: the encoding-decoding structure of the traditional U-Net is taken as a main frame, wherein the encoder replaces the convolution operation of the traditional Unet through a multi-scale feature extraction and fusion module to acquire context information of different receptive fields;
step 2: on a bottleneck layer, fusing information under different receptive fields acquired by the multi-scale feature extraction and fusion module through a receptive field self-adaptive aggregation module to generate a weight matrix, and performing feature screening on deep semantic channels through weights to highlight semantic features related to a segmentation result;
and 3, step 3: on the jump connection between the encoder and the decoder, relieving semantic difference between peer layers of the encoder and the decoder by adding a cross-space residual error fusion module;
and 4, step 4: and verifying the segmentation effect through an ablation experiment.
Compared with the prior art, the invention has the following advantages:
the invention discloses an ultrasonic breast nodule end-to-end segmentation method based on multi-scale and cross-space fusion, which is innovatively improved in the following aspects on the basis of U-Net:
firstly, by adding a multi-scale feature extraction and fusion module, the feature extraction receptive field is expanded, the nonlinearity of the module is enhanced, and the feature capture capability of a network on a target is enhanced;
secondly, the bottleneck layer is at the deep stage of the encoder, and contains higher level context information, so the feature extraction should consider the importance difference between different semantic features, but the existing method considers this less. According to the method, a reception field self-adaptive aggregation module is adopted in a bottleneck layer, and the convolution characteristics under different reception fields are subjected to weight screening, so that important semantic characteristics play a more remarkable role in a segmentation process, and the segmentation effect is further improved;
finally, semantic feature differences exist among the layers of a coder, a decoder and the like of the traditional U-Net network, a cross-space residual error fusion module is added, the direct connection and splicing type jump connection mode in the U-Net network is changed, a nonlinear jump connection mode is constructed between a coding path and a decoding path, information complementation among different coding layers is realized, and the semantic difference among the layers of the coder, the decoder and the like is relieved.
The method can automatically extract the nodule focus area of the ultrasonic breast image, has higher segmentation accuracy and more accurate segmentation boundary, can assist doctors to quickly and accurately diagnose the nodule focus, reduces misdiagnosis rate and missed diagnosis rate, and relieves the current situation that excellent ultrasonic doctors in primary hospitals are relatively insufficient. The extracted region can also be used as the basis for automatically judging the quality and the malignancy of the nodal focus in the follow-up process, and has very important research value and application prospect for promoting computer-aided ultrasonic medical diagnosis.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the description of the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a schematic view of the overall structure of the segmentation method of the present invention.
FIG. 2 is a schematic diagram of a multi-scale feature extraction and fusion module according to the present invention.
FIG. 3 is a schematic diagram of a receptor field adaptive aggregation module according to the present invention.
FIG. 4 is a schematic diagram of a cross-space residual fusion module according to the present invention.
Fig. 5 is a comparative schematic diagram of an ablation experiment of the present invention. Wherein, (a) is the original picture; (b) only adopting the original Unet network for segmentation; (c) An MFEF module, a RAA module and a CRF module are respectively and independently added on the basis of an Unet network architecture; (d) Adding two of the components in a mixed way on the basis of the Unet network architecture; (e) The three modules are added on the basis of the Unet network architecture.
FIG. 6 is a diagram illustrating segmentation results of different models according to the present invention. Wherein, (a) is the original picture; (b) AttUnet; (c) is ResUNet + +; (d) AKUNet; (e) is CFPNet; (f) is the process of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
As shown in fig. 1-4, the present invention provides an ultrasonic breast nodule end-to-end segmentation method based on multi-scale and cross-space fusion, comprising:
step 1: the encoding-decoding structure of the traditional U-Net is used as a main frame, wherein the encoder replaces the convolution operation of the traditional Unet through a multi-scale feature extraction and fusion module to acquire context information of different receptive fields.
Breast nodules in ultrasound images often exhibit complex morphological features such as varying sizes, various shapes (malignant nodules often exhibit various irregular shapes), unclear boundaries between the edges and surrounding tissues of some nodules, and low contrast and artifact interference of ultrasound images, which all pose great challenges to the precise segmentation of nodules. The coding layer of the traditional UNet network adopts 2 continuous 3 multiplied by 3 convolutions when carrying out feature extraction, so that only feature information under a single scale receptive field can be extracted, and the irregular change strain of the knotting features is poor. As an embodiment of the present application, a Multi-scale Feature Extraction and Fusion Module (MFEF) in the present application takes 4 convolution branches with different receptive fields as a main line, and the structure is shown in fig. 2.
Wherein, the first branch circuit is subjected to feature extraction for 1 × 1 convolution and has a minimum receptive field of 1 × 1; the second to the fourth branches are subjected to feature extraction by convolution with 3 multiplied by 3, and have 3 multiplied by 3 receptive fields;
the third branch and the fourth branch are provided with multi-scale rolling blocks, namely split modules; and the multi-scale feature extraction and fusion module respectively splices and fuses the rear ends of the other three branches two by two under the condition of keeping the convolution result of the first branch unchanged. The design enables the whole module to form a nesting effect of multi-scale convolution, thereby ensuring multi-scale feature extraction and enhancing the nonlinearity of the module. In addition, the module splices and fuses the rear ends of the other three branches pairwise under the condition of keeping the convolution result of the first branch unchanged, and the purpose is to keep more characteristic information under adjacent receptive fields.
Preferably, the SplitB module is inspired by Ghost-net to perform feature extraction and expansion of receptive fields on input features through grouping convolution and expansion convolution; the split module performs feature extraction and expansion of receptive fields on input features through grouping convolution and expansion convolution; the split B module uniformly divides the input characteristic channels into two 2 groups of group1 and group2 for multiple times, as shown in the figure; grouping any one of the two groups, namely group1 or group2, again, namely grouping 3 and group4; and then, performing feature extraction on any group after secondary grouping, namely group3 or group4, through expansion convolution, such as group3 and group4 in the graph, splicing group4 with group2, performing feature extraction by adopting expansion convolution, finally splicing the obtained feature map with group3, and finally splicing the obtained feature map with a group which is not subjected to secondary grouping, namely one group of group2 or group1, to form the final module output. The form of convolution and recombination after grouping can effectively reduce model parameters and reduce the calculated amount, and simultaneously increases the complexity of model connection and promotes the nonlinearity of a module.
The multi-scale feature extraction and fusion module in section 1 is only suitable for shallow feature extraction, because the model performs equal weight splicing on channel features of different scales when performing feature extraction, and the processing mode can focus on the details of each shallow feature. However, after entering the bottleneck layer, the shallow features are further abstracted into high-level semantic features, and the abstraction of the features is enhanced, so that the influence of the semantic features of different channels on the final segmentation result is different, and therefore, the feature extraction of the bottleneck layer needs to consider the importance difference among different semantic features besides the multi-scale receptive field.
Thus, step 2: and on a bottleneck layer, fusing information under different receptive fields acquired by the multi-scale feature extraction and fusion module through a receptive field self-adaptive aggregation module to generate a weight matrix, and performing feature screening on deep semantic channels through weights to highlight semantic features related to a segmentation result. The reception field self-adaptive aggregation module in the step 2 performs weight screening on convolution characteristics under different reception fields; the perception field self-adaptive aggregation module is provided with 4 convolution branches, the perception fields of each convolution branch are respectively 1 × 1, 3 × 3, 5 × 5 and 7 × 7, pairwise cascading and fusion are carried out on feature maps F1, F2 and F3 of the 3 × 3, 5 × 5 and 7 × 7 convolution branches, fused information generates a weight matrix through a softmax function and then is subjected to channel-by-channel weighted addition with the feature maps F1, F2 and F3, and therefore self-adaptive selection of convolution features under different perception fields is achieved, and self-adaptive selection of convolution features under different perception fields is achieved. The essence of the self-adaptive selection is that useful information in a multi-scale receptive field is highlighted through a weight matrix, useless information is weakened, and the selected result is superposed with the original 1 x 1 convolution branch characteristics, so that the aim of highlighting the strengthened information is achieved while information is not lost.
In order to supplement the information loss brought by the decoding stage, the classic U-Net network adopts jump connection to directly transmit and splice the low-level spatial information of the coding part to the decoding part. However, the decoder is characterized in that after multiple convolutions and pooling in the encoding stage, higher level context information is formed, which causes large semantic difference between the encoder and the decoder in the characteristics, and semantic gap is generated if the lower level characteristics are directly transmitted to the higher level through jump connection.
And step 3: on the jump connection between the encoder and the decoder, relieving semantic difference between peer layers of the encoder and the decoder by adding a cross-space residual error fusion module; the structure of parallel connection, series connection and parallel connection enables information on one side of the encoder to be fully fused when jumping transmission is carried out on one side of the decoder, and the fusion realizes information complementation between different encoding layers on one hand and further characteristic extraction on the information of the encoding layers on the other hand, thereby relieving semantic difference between encoding and decoding peer layers. Furthermore, the introduction of the residual branch makes the network easier to optimize. The cross-space residual error fusion module comprises five double-space fusion modules; the five double-space fusion modules form a cross fusion network structure in a series connection and cross parallel connection mode;
features F of four different coding layers s S is belonged to {1,2,3,4} and is respectively input into the first two DSFs arranged in parallel in the CRF according to a group of two adjacent codes in a neighboring sequence so as to enable the front and back adjacent codes in the encoder to carry out information fusion; and the fused output is respectively input into the three DSF modules connected in series and parallel to be fused continuously. The dual space fusion module includes: 4 convolution branches, wherein the first branch adopts 3 multiplied by 3 convolution to extract the characteristics of F3, the second branch adopts 3 multiplied by 3 and stride of 2 to extract the characteristics of F3, meanwhile, the third branch adopts an up-sampling operation to extract the characteristics of F4, and the fourth branch adopts 3 multiplied by 3 convolution to extract the characteristics of F4;
the first branch and the third branch are cascaded and fused, and the second branch and the fourth branch are cascaded and fused.
And 4, step 4: and verifying the segmentation effect through an ablation experiment.
The first embodiment is as follows:
as an embodiment of the application, the effectiveness of each designed module and the combination thereof on an ultrasonic breast nodule segmentation task is verified through an ablation experiment. For full verification, 4 groups of experiments are carried out in the ablation experiment, and the 1 st group is only segmented by using an original Unet network; group2 is to add MFEF module, RAA module, CRF module separately on the basis of the Unet network architecture; group3 is to add pairwise mixture based on the Unet network architecture; group4 is the addition of all three modules on the basis of the Unet network architecture. The results of the ablation experiments are shown in table 1.
As can be seen from the table, the index values of the model in the group2 are all improved on the basis model UNet, which indicates that the use of each module is effective. However, compared with the 3 rd group of models, the index value is improved more by adding two modules than by singly adopting one module, which shows that the mixed use of the modules can generate the advantage accumulation, and the advantage accumulation effect is more obvious in the 4 th group of experiments. The experimental results in the 4 th group are remarkably improved compared with the results in the 3 rd group except for precision indexes, and Dice, recall and IOU are respectively and averagely improved by 0.258, 0.307 and 0.243. Although the precision index of group4 was slightly lower than that of groups 2 and 3, the difference was not very significant and the overall improvement of all the indexes was more important. In addition, in the segmentation process of the medical image, the recall rate is a very important index which indicates how much lesion tissues are found, so that the experiment is more concerned about the promotion of the recall rate under the condition that the precision indexes are similar. The above analysis shows that the index evaluation obtained by simultaneously adopting the MFEF module, the RAA module, and the CRF module on the basis of the Unet network architecture is the best, which is also reflected in the visual effect comparison of the segmentation results, as shown in fig. 5.
Fig. 5 is a comparison of the results of the experimental segmentations of groups 1-4 in terms of visual effect. The green contour is a nodule region label given by an expert, and the red contour represents the segmentation result of the model. As can be seen from the figure, the segmentation result using the Unet model alone has many false positives, such as the regions a, b, and c in the figure. An MFEF module is added on the basis of the Unet network, false positives obviously disappear, and a certain gap exists between a segmentation boundary and an expert labeling boundary; after the MFEF module, the RAA module and the CRF module are added, a more complete contour is formed, more detailed information of the breast nodule boundary is reserved, the segmentation result is closer to a labeled graph in shape, and better performance is achieved.
TABLE 1 network structure ablation contrast
Table 1 Comparison of network structure ablation
To further verify the validity of the proposed method, the model proposed herein is first combined with an existing classical segmentation model, such as AttUNet [18] 、ResUnet++ [19] 、SKUnet [20] 、CFPNet [21] And so on for comparison. Segmentation of ultrasound breast nodules was achieved using these classical models and the segmentation results were evaluated as shown in table 2. As can be seen from the table, the evaluation index of the method is the best in all comparison models, particularly the indexes of Dice and IOU are obviously improved, and the segmentation result of the method is the closest to the gold standard given by a doctor. This is also shown in the comparison of the visual effect of the segmentation result, as shown in fig. 6, the green contour is the nodule region label given by the expert, and the red color represents the segmentation result of the model. As can be seen from the figure, the method can locate the nodule region compared with other contrast methods, identify more lesion regions, and segment the boundary closer to the boundary calibrated by an expert for the nodule with blurred boundary. In view of comprehensive comparison, the method provided by the invention is better than other comparison networks in both the capability of distinguishing the nodules from the background and the morphology of the segmentation result.
In addition, a comparison is made with 4 documents on ultrasound breast nodule segmentation, and the comparison results are shown in table 3. The references involved in the comparison are all as described herein, using the published data set of ultrasound breast nodules provided by Yap et al. As can be seen from Table 3, the evaluation indexes of the proposed method are higher than those of other literature methods, wherein the literature methods [24] The index of (2) is only second to the text method, the dice index and the IOU index of the two are very similar, but the recall rate of the text is obviously higher than that of the literature [24] Description of the invention [24] Herein, theFalse negative segmentation areas are reduced and more lesions are segmented.
TABLE 2 comparison of different segmentation algorithms
Table 2 Comparison of different segmentation algorithms
TABLE 3 comparison of different segmentation algorithms for the same dataset
Table 3 Comparison of different segmentation algorithms for the same dataset
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments. In the above embodiments of the present invention, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments. In the embodiments provided in the present application, it should be understood that the disclosed technology can be implemented in other ways.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.
Claims (6)
1. An ultrasonic breast nodule end-to-end segmentation method based on multi-scale and cross-space fusion is characterized by comprising the following steps of:
step 1: the encoding-decoding structure of the traditional U-Net is used as a main frame, wherein the encoder replaces the convolution operation of the traditional Unet through a multi-scale feature extraction and fusion module to acquire context information of different receptive fields;
step 2: on a bottleneck layer, fusing information under different receptive fields acquired by the multi-scale feature extraction and fusion module through a receptive field self-adaptive aggregation module to generate a weight matrix, and performing feature screening on deep semantic channels through weights to highlight semantic features related to a segmentation result;
and step 3: on the jump connection between the encoder and the decoder, relieving semantic difference between peer layers of the encoder and the decoder by adding a cross-space residual error fusion module;
and 4, step 4: and verifying the segmentation effect through an ablation experiment.
2. The method for ultrasonic breast nodule end-to-end segmentation based on multi-scale and cross-space fusion according to claim 1,
the multi-scale feature extraction and fusion module comprises 4 convolution branches with different receptive fields; wherein, the first branch circuit is subjected to feature extraction for 1 × 1 convolution and has a minimum receptive field of 1 × 1; the second to the fourth branches are subjected to feature extraction by convolution with 3 multiplied by 3, and have 3 multiplied by 3 receptive fields;
after the 3 x 3 convolution of the third branch and the fourth branch, a multi-scale convolution block, namely a split module, is arranged; and the multi-scale feature extraction and fusion module respectively splices and fuses the rear ends of the other three branches two by two under the condition of keeping the convolution result of the first branch unchanged.
3. The method for ultrasonic breast nodule end-to-end segmentation based on multi-scale and cross-space fusion according to claim 2,
the split module performs feature extraction and expansion of receptive fields on input features through grouping convolution and expansion convolution; the split B module divides the input characteristic channels into two groups, namely group1 and group2; grouping any one of the two groups, namely group1 or group2, again, namely grouping 3 and group4; and performing feature extraction on any group, namely group3 or group4, subjected to secondary grouping through expansion convolution, and finally splicing the obtained feature diagram with a group which is not subjected to secondary grouping, namely group2 or group1 to form final module output.
4. The method for ultrasonic breast nodule end-to-end segmentation based on multi-scale and cross-space fusion according to claim 1,
the reception field self-adaptive aggregation module in the step 2 performs weight screening on convolution characteristics under different reception fields; the perception field self-adaptive aggregation module is provided with 4 convolution branches, the perception fields of each convolution branch are respectively 1 × 1, 3 × 3, 5 × 5 and 7 × 7, pairwise cascading and fusion are carried out on feature maps F1, F2 and F3 of the 3 × 3, 5 × 5 and 7 × 7 convolution branches, fused information generates a weight matrix through a softmax function and then is subjected to channel-by-channel weighted addition with the feature maps F1, F2 and F3, and therefore self-adaptive selection of convolution features under different perception fields is achieved.
5. The method for ultrasonic breast nodule end-to-end segmentation based on multi-scale and cross-space fusion according to claim 1,
the cross-space residual error fusion module comprises five double-space fusion modules; the five double-space fusion modules form a cross fusion network structure in a series connection and cross parallel connection mode;
feature F for four different coding layers s S is belonged to {1,2,3,4} and is respectively input into the first two DSFs arranged in parallel in the CRF according to a group of two adjacent codes in a neighboring sequence so as to enable the front and back adjacent codes in the encoder to carry out information fusion; and the fused output is respectively input into the three DSF modules connected in series and parallel to be fused continuously.
6. The method for ultrasonic breast nodule end-to-end segmentation based on multi-scale and cross-space fusion according to claim 5,
the dual space fusion module includes: 4 convolution branches, wherein the first branch adopts 3 multiplied by 3 convolution to extract the characteristics of F3, the second branch adopts 3 multiplied by 3 and stride of 2 to extract the characteristics of F3, meanwhile, the third branch adopts an up-sampling operation to extract the characteristics of F4, and the fourth branch adopts 3 multiplied by 3 convolution to extract the characteristics of F4;
the first branch and the third branch are cascaded and fused, and the second branch and the fourth branch are cascaded and fused.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211177184.XA CN115578559A (en) | 2022-09-26 | 2022-09-26 | Ultrasonic breast nodule end-to-end segmentation method based on multi-scale and cross-space fusion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211177184.XA CN115578559A (en) | 2022-09-26 | 2022-09-26 | Ultrasonic breast nodule end-to-end segmentation method based on multi-scale and cross-space fusion |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115578559A true CN115578559A (en) | 2023-01-06 |
Family
ID=84582469
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211177184.XA Pending CN115578559A (en) | 2022-09-26 | 2022-09-26 | Ultrasonic breast nodule end-to-end segmentation method based on multi-scale and cross-space fusion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115578559A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115984293A (en) * | 2023-02-09 | 2023-04-18 | 中国科学院空天信息创新研究院 | Spatial target segmentation network and method based on edge perception attention mechanism |
| CN116934738A (en) * | 2023-08-14 | 2023-10-24 | 威朋（苏州）医疗器械有限公司 | Organ and nodule joint segmentation method and system based on ultrasonic image |
| CN117092612A (en) * | 2023-10-18 | 2023-11-21 | 湘潭大学 | Automatic driving navigation method based on laser radar |
| CN117218127A (en) * | 2023-11-09 | 2023-12-12 | 吉林大学 | Ultrasonic endoscope auxiliary monitoring system and method |
| CN118644678A (en) * | 2024-08-12 | 2024-09-13 | 四川大学 | Lightweight medical image segmentation method based on cascade multi-receptive field |
| CN118644678B (en) * | 2024-08-12 | 2024-10-22 | 四川大学 | Lightweight medical image segmentation method based on cascade multi-receptive field |
-
2022
- 2022-09-26 CN CN202211177184.XA patent/CN115578559A/en active Pending
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115984293A (en) * | 2023-02-09 | 2023-04-18 | 中国科学院空天信息创新研究院 | Spatial target segmentation network and method based on edge perception attention mechanism |
| CN115984293B (en) * | 2023-02-09 | 2023-11-07 | 中国科学院空天信息创新研究院 | Spatial target segmentation network and method based on edge perception attention mechanism |
| CN116934738A (en) * | 2023-08-14 | 2023-10-24 | 威朋（苏州）医疗器械有限公司 | Organ and nodule joint segmentation method and system based on ultrasonic image |
| CN116934738B (en) * | 2023-08-14 | 2024-03-22 | 威朋（苏州）医疗器械有限公司 | Organ and nodule joint segmentation method and system based on ultrasonic image |
| CN117092612A (en) * | 2023-10-18 | 2023-11-21 | 湘潭大学 | Automatic driving navigation method based on laser radar |
| CN117092612B (en) * | 2023-10-18 | 2024-01-26 | 湘潭大学 | Automatic driving navigation method based on laser radar |
| CN117218127A (en) * | 2023-11-09 | 2023-12-12 | 吉林大学 | Ultrasonic endoscope auxiliary monitoring system and method |
| CN117218127B (en) * | 2023-11-09 | 2024-01-26 | 吉林大学 | Ultrasonic endoscope auxiliary monitoring system and method |
| CN118644678A (en) * | 2024-08-12 | 2024-09-13 | 四川大学 | Lightweight medical image segmentation method based on cascade multi-receptive field |
| CN118644678B (en) * | 2024-08-12 | 2024-10-22 | 四川大学 | Lightweight medical image segmentation method based on cascade multi-receptive field |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Adegun et al. | Deep learning techniques for skin lesion analysis and melanoma cancer detection: a survey of state-of-the-art | |
| Saini et al. | ColpoNet for automated cervical cancer screening using colposcopy images | |
| CN115578559A (en) | Ultrasonic breast nodule end-to-end segmentation method based on multi-scale and cross-space fusion | |
| Song et al. | Attention based multi-scale parallel network for polyp segmentation | |
| CN111709952B (en) | MRI brain tumor automatic segmentation method based on edge feature optimization and double-flow decoding convolutional neural network | |
| CN116309648A (en) | Medical image segmentation model construction method based on multi-attention fusion | |
| Xu et al. | Dual-channel asymmetric convolutional neural network for an efficient retinal blood vessel segmentation in eye fundus images | |
| Guo | DPN: detail-preserving network with high resolution representation for efficient segmentation of retinal vessels | |
| Feng et al. | SSN: A stair-shape network for real-time polyp segmentation in colonoscopy images | |
| Shu et al. | Medical image segmentation based on active fusion-transduction of multi-stream features | |
| Yamanakkanavar et al. | MF2-Net: A multipath feature fusion network for medical image segmentation | |
| Shao et al. | Application of U-Net and Optimized Clustering in Medical Image Segmentation: A Review. | |
| Zhang et al. | Attention-guided feature extraction and multiscale feature fusion 3d resnet for automated pulmonary nodule detection | |
| CN117746045B (en) | Method and system for segmenting medical image by fusion of transducer and convolution | |
| Wu et al. | Continuous refinement-based digital pathology image assistance scheme in medical decision-making systems | |
| Guan et al. | NCDCN: multi-focus image fusion via nest connection and dilated convolution network | |
| Du et al. | AGMR-Net: Attention-guided multiscale recovery framework for stroke segmentation | |
| Ahmed et al. | COMA-Net: Towards generalized medical image segmentation using complementary attention guided bipolar refinement modules | |
| Khan et al. | ESDMR-Net: A lightweight network with expand-squeeze and dual multiscale residual connections for medical image segmentation | |
| Zhang et al. | ETUNet: Exploring efficient transformer enhanced UNet for 3D brain tumor segmentation | |
| Lin et al. | Polyp-LVT: Polyp segmentation with lightweight vision transformers | |
| Li et al. | A pathology image segmentation framework based on deblurring and region proxy in medical decision-making system | |
| Samanta et al. | Context aggregation network for semantic labeling in histopathology images | |
| Jagadeesh et al. | Brain Tumour Classification using CNN Algorithm | |
| Kang et al. | Cross-patch feature interactive net with edge refinement for retinal vessel segmentation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |