CN116563217A - Cervical vertebra segmentation method and device based on fusion of edge pyramid and cross feature - Google Patents
Cervical vertebra segmentation method and device based on fusion of edge pyramid and cross feature Download PDFInfo
- Publication number
- CN116563217A CN116563217A CN202310340206.8A CN202310340206A CN116563217A CN 116563217 A CN116563217 A CN 116563217A CN 202310340206 A CN202310340206 A CN 202310340206A CN 116563217 A CN116563217 A CN 116563217A
- Authority
- CN
- China
- Prior art keywords
- images
- edge
- layer
- fusion
- coded
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000004927 fusion Effects 0.000 title claims abstract description 90
- 230000011218 segmentation Effects 0.000 title claims abstract description 53
- 238000000034 method Methods 0.000 title claims abstract description 45
- 238000013135 deep learning Methods 0.000 claims abstract description 32
- 238000000605 extraction Methods 0.000 claims abstract description 30
- 238000002047 photoemission electron microscopy Methods 0.000 claims abstract description 15
- 230000006870 function Effects 0.000 claims description 25
- 238000011176 pooling Methods 0.000 claims description 12
- 238000012545 processing Methods 0.000 claims description 12
- 238000004364 calculation method Methods 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 10
- 230000000694 effects Effects 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 20
- 230000008569 process Effects 0.000 description 11
- 238000004891 communication Methods 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000003491 array Methods 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000001902 propagating effect Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- RYGMFSIKBFXOCR-UHFFFAOYSA-N Copper Chemical compound [Cu] RYGMFSIKBFXOCR-UHFFFAOYSA-N 0.000 description 1
- 206010041591 Spinal osteoarthritis Diseases 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 208000036319 cervical spondylosis Diseases 0.000 description 1
- 229910052802 copper Inorganic materials 0.000 description 1
- 239000010949 copper Substances 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 208000005801 spondylosis Diseases 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30008—Bone
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Radiology & Medical Imaging (AREA)
- Quality & Reliability (AREA)
- Image Processing (AREA)
Abstract
The invention provides a cervical vertebra segmentation method and a cervical vertebra segmentation device based on fusion of an edge pyramid and cross features, wherein the method comprises the following steps: acquiring a cervical vertebra medical image to be segmented; inputting the cervical vertebra medical image to be segmented into a deep learning network model for coding operation to obtain a plurality of first coding images with different sizes; based on a plurality of PEEM modules in the deep learning network model, performing edge extraction operation on the first coded image respectively to obtain a plurality of second coded images with different sizes; based on a plurality of CFM modules in the deep learning network model, respectively performing cross feature fusion operation on the plurality of first coded images and the plurality of second coded images to obtain a plurality of third coded images with different sizes; and carrying out fusion operation on the basis of a plurality of third coding images with different sizes and a plurality of decoding images with different sizes, and outputting a segmentation result of the cervical vertebra medical image. The method can enable the feature map obtained by fusion to be more accurate, reduce loss of detail features and improve segmentation effect.
Description
Technical Field
The invention relates to the medical field, in particular to a cervical vertebra segmentation method and device based on fusion of an edge pyramid and cross features.
Background
With the continuous development of technology, artificial intelligence technology is increasingly applied to the medical field. Taking cervical vertebra as an example, the introduction of artificial intelligence technology saves a lot of time for doctors and improves the operation efficiency. However, due to the diversity of cervical spondylosis, the accuracy of cervical vertebra segmentation is not high easily, and especially in segmentation detail characteristics, segmentation deviation is large, so that poor experience is caused.
Therefore, how to solve the above-mentioned problems is considered.
Disclosure of Invention
The invention provides a cervical vertebra segmentation method and device based on fusion of an edge pyramid and cross features, which are used for solving the problems.
In a first aspect, the present invention provides a cervical vertebra segmentation method based on fusion of an edge pyramid and a cross feature, including:
acquiring a cervical vertebra medical image to be segmented;
inputting the cervical vertebra medical image to be segmented into a deep learning network model for coding operation to obtain a plurality of first coding images with different sizes;
based on a plurality of pyramid feature extraction PEEM modules in the deep learning network model, respectively performing edge extraction operation on the first coded image to obtain a plurality of second coded images with different sizes;
based on a plurality of cross feature fusion CFM modules in the deep learning network model, cross feature fusion operation is respectively carried out on the plurality of first coded images and the plurality of second coded images, so that a plurality of third coded images with different sizes are obtained;
based on the fusion operation of the third coded images with different sizes and the decoding images with different sizes, outputting and obtaining a segmentation result of the cervical vertebra medical image;
wherein the plurality of decoded images corresponds to the plurality of first encoded images in one-to-one correspondence in size.
Optionally, the deep learning network model includes a five-layer network structure, the first layer network structure is a network layer inputting the cervical vertebra medical image to be segmented, and the sizes of the first coding images from the first layer network structure to the fifth layer network structure are sequentially reduced;
the pyramid feature extraction module and the cross feature fusion module are arranged on the first-layer network structure to the fourth-layer network structure;
the cross feature fusion module in the current network layer is used for carrying out cross feature fusion operation on the second coded image of the current network layer and the first coded images of other network layers to obtain a third coded image of the current network layer;
the current network layer is any one of the first layer network structure to the fourth layer network structure, and the other network layers comprise a plurality of network layers from the first layer network structure to the fourth layer network structure except the current network layer.
Optionally, the PEEM module comprises a four-layer sub-network structure;
each layer of sub-network structure is used for carrying out edge extraction operation, average pooling operation, addition operation and splicing operation on the first coded image of the current network layer to obtain a second coded image;
wherein the sub-network structures of different layers have different sizes for performing the average pooling operation.
Optionally, the calculating process of the edge extraction operation is as follows:
G(x,y)＝x(i,j)+y(i,j)
wherein G is a gradient calculation result of the first encoded image, edge is an Edge calculation result, x (i, j) is an Edge of the first encoded image in the x direction, and y (i, j) is an Edge of the first encoded image in the y direction.
Optionally, x (i, j) and t (i, j) are calculated based on the following:
where img is the edge extracted image and (i, j) is the coordinates of the first encoded image.
Optionally, the cross feature fusion CFM module includes:
a first branch structure and a second branch structure;
the first branch structure is used for performing Softmax operation on a second coded image of the current network layer to obtain a first characteristic parameter;
the second branch structure is used for carrying out feature fusion operation on a plurality of first coded images in the other network layers to obtain a first feature map;
and performing feature fusion operation on the first feature map, the first feature parameters and the second coded image of the current network layer to obtain a third coded image corresponding to the current network layer.
Optionally, the segmentation loss function adopted by the deep learning network model includes at least one of the following:
CELoss loss function; diceLoss loss function.
In a second aspect, the present invention provides a cervical vertebra segmentation apparatus based on fusion of an edge pyramid and a cross feature, comprising:
the acquisition module is used for acquiring the cervical vertebra medical image to be segmented;
the input module is used for inputting the cervical vertebra medical image to be segmented into a deep learning network model for coding operation to obtain a plurality of first coding images with different sizes;
the extraction module is used for extracting PEEM modules based on a plurality of pyramid features in the deep learning network model, and respectively carrying out edge extraction operation on the first coded image to obtain a plurality of second coded images with different sizes;
the fusion module is used for respectively carrying out cross feature fusion operation on the plurality of first coded images and the plurality of second coded images based on the plurality of cross feature fusion CFM modules in the deep learning network model to obtain a plurality of third coded images with different sizes;
the processing module is used for carrying out fusion operation on the basis of the plurality of third coding images with different sizes and the plurality of decoding images with different sizes and outputting and obtaining a segmentation result of the cervical vertebra medical image;
wherein the plurality of decoded images corresponds to the plurality of first encoded images in one-to-one correspondence in size.
In a third aspect, the present invention provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing a cervical segmentation method based on fusion of edge pyramids and cross features as described above when executing the program.
In a fourth aspect, the present invention provides a non-transitory computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when executed by a processor, implements a cervical segmentation method based on fusion of edge pyramids and intersection features as described above.
The technical scheme of the invention has at least the following beneficial effects:
according to the cervical vertebra segmentation method based on the edge pyramid and the cross feature fusion, edge extraction operation is carried out on the first coded images with different sizes through the PEEM module, so that edge feature information of the obtained first coded images is more accurate, cross feature fusion operation is carried out on a plurality of first coded images and second coded images through the CFM module, and edge feature information and overall feature information of the obtained third coded images are more accurate. In addition, because more accurate edge characteristic information and overall characteristic information are reserved in the obtained third coding image, fusion is carried out on the basis of the third coding image and the decoding image, so that the characteristic image obtained by fusion is more accurate, an accurate segmentation result of the cervical vertebra medical image is obtained, loss of detail characteristics is reduced, and the segmentation effect of the cervical vertebra medical image is improved.
Drawings
In order to more clearly illustrate the invention or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of a cervical vertebra segmentation method based on fusion of edge pyramids and cross features;
FIG. 2 is a schematic diagram of a deep learning network model according to the present invention;
FIG. 3 is a schematic diagram of a PEEM module according to the present invention;
fig. 4 is a schematic structural diagram of a cross feature fusion CFM module according to the present invention;
FIG. 5 is a schematic block diagram of a cervical vertebra segmentation apparatus based on fusion of edge pyramids and intersecting features according to the present invention;
fig. 6 is a schematic structural diagram of an electronic device according to the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The terms "first," "second," "third," "fourth" and the like in the description and in the claims and in the above drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein.
It should be understood that, in various embodiments of the present invention, the sequence number of each process does not mean that the execution sequence of each process should be determined by its functions and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
It should be understood that in the present invention, "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements that are expressly listed or inherent to such process, method, article, or apparatus.
It should be understood that in the present invention, "plurality" means two or more. "and/or" is merely an association relationship describing an association object, and means that three relationships may exist, for example, and/or B may mean: a exists alone, A and B exist together, and B exists alone. The character "/" generally indicates that the context-dependent object is an "or" relationship. "comprising A, B and C", "comprising A, B, C" means that all three of A, B, C comprise, "comprising A, B or C" means that one of the three comprises A, B, C, and "comprising A, B and/or C" means that any 1 or any 2 or 3 of the three comprises A, B, C.
It should be understood that in the present invention, "B corresponding to a", "a corresponding to B", or "B corresponding to a" means that B is associated with a, from which B can be determined. Determining B from a does not mean determining B from a alone, but may also determine B from a and/or other information. The matching of A and B is that the similarity of A and B is larger than or equal to a preset threshold value.
As used herein, "if" may be interpreted as "at … …" or "at … …" or "in response to a determination" or "in response to detection" depending on the context.
The technical scheme of the invention is described in detail below by specific examples. The following embodiments may be combined with each other, and some embodiments may not be repeated for the same or similar concepts or processes.
Referring to fig. 1, a flow chart of a cervical vertebra segmentation method based on fusion of an edge pyramid and a cross feature is provided, and the cervical vertebra segmentation method comprises the following steps:
s11: and acquiring a cervical vertebra medical image to be segmented.
The cervical vertebra medical image adopts DICOM format, and 2.5D image is input, and the corresponding relation between different slices of CT image is fully considered.
S12: and inputting the cervical vertebra medical image to be segmented into a deep learning network model for coding operation, so as to obtain a plurality of first coding images with different sizes.
It should be noted that the encoding operation includes a convolution Conv3x3 operation, a batch normalization BN operation, a nonlinear Relu operation, and a Max Pooling operation, and downsampling is performed multiple times, so as to obtain multiple first encoded images with different sizes.
S13: and respectively carrying out edge extraction operation on the first coded image based on a plurality of pyramid feature extraction PEEM modules in the deep learning network model to obtain a plurality of second coded images with different sizes.
By performing edge extraction operation on the first encoded images with different sizes, edge feature information of the first encoded images with different sizes can be reserved.
S14: and based on a plurality of cross feature fusion CFM modules in the deep learning network model, respectively performing cross feature fusion operation on the plurality of first coded images and the plurality of second coded images to obtain a plurality of third coded images with different sizes.
The CFM module is used for carrying out cross feature fusion operation on the plurality of first coded images and the plurality of second coded images, so that feature information reserved in the obtained third coded image is richer, and loss of detail features is avoided.
S15: based on the fusion operation of the third coded images with different sizes and the decoding images with different sizes, outputting and obtaining a segmentation result of the cervical vertebra medical image;
wherein the plurality of decoded images corresponds to the plurality of first encoded images in one-to-one correspondence in size.
According to the cervical vertebra segmentation method based on the edge pyramid and the cross feature fusion, edge extraction operation is carried out on the first coded images with different sizes through the PEEM module, so that edge feature information of the obtained first coded images is more accurate, cross feature fusion operation is carried out on a plurality of first coded images and second coded images through the CFM module, and edge feature information and overall feature information of the obtained third coded images are more accurate. In addition, because more accurate edge characteristic information and overall characteristic information are reserved in the obtained third coding image, fusion is carried out on the basis of the third coding image and the decoding image, so that the characteristic image obtained by fusion is more accurate, an accurate segmentation result of the cervical vertebra medical image is obtained, loss of detail characteristics is reduced, and the segmentation effect of the cervical vertebra medical image is improved.
For example, as shown in fig. 2, a schematic structural diagram of a deep learning network model is provided in the present invention. The deep learning network model comprises five layers of network structures, wherein a first layer of network structure is a network layer for inputting the cervical vertebra medical image to be segmented, and the sizes of first coding images from the first layer of network structure to a fifth layer of network structure are sequentially reduced.
It should be noted that, the first encoded images corresponding to the first to fifth layer network structures are denoted by E1 to E5, respectively.
The pyramid feature extraction module and the cross feature fusion module are arranged on the first-layer network structure to the fourth-layer network structure, wherein the cross feature fusion module in the current network layer is used for carrying out cross feature fusion operation on the second coded image of the current network layer and the first coded images of other network layers to obtain a third coded image of the current network layer.
It should be noted that, the second encoded images corresponding to the first layer network structure to the fourth layer network structure are denoted by E1 'to E4', respectively. Third encoded images corresponding to the first to fourth layer network structures are denoted by E1 to E4 respectively.
The current network layer is any one of the first layer network structure to the fourth layer network structure, and the other network layers comprise a plurality of network layers from the first layer network structure to the fourth layer network structure except the current network layer.
For example, the pyramid feature extraction module and the cross feature fusion module are located in the first layer network structure, where the pyramid feature extraction module performs edge extraction operation on only the first encoded image in the first layer network structure to obtain a second encoded image E1', and the cross feature fusion module is configured to perform cross feature fusion operation on the first encoded image E2, the first encoded image E3, and the first encoded image E4 in the second layer network structure to the fourth layer network structure to obtain a third encoded image E1.
For example, referring to fig. 3, a schematic structural diagram of a PEEM module provided in the present invention is shown. The PEEM module comprises a four-layer sub-network structure; each layer of sub-network structure is used for carrying out edge extraction operation, average pooling operation, addition operation and splicing operation on the first coded image of the current network layer to obtain a second coded image; wherein the sub-network structures of different layers have different sizes for performing the average pooling operation.
In fig. 3, the first encoded image is denoted by Ei, and i has values of 1,2,3, and 4. Alternatively, description is given taking i=1 as an example. Firstly, the first encoded image E1 is subjected to edge extraction operations through different sub-network structures, and then the obtained results are subjected to average pooling operations of different sizes, wherein the average pooling operations of the first-layer sub-network structure to the fourth-layer sub-network structure are denoted by avg_1, avg_2, avg_3, and avg_4, respectively. Avg_1 was pooled 2 x 2, avg_2 was pooled 4*4, avg_3 was pooled 8 x 8, and avg_4 was pooled 16 x 16. The edge information under multiple layers can be extracted to the maximum extent through the average pooling operation in different sub-network structures. And then, performing add operation on the obtained average pooled result and the first encoded image E1 respectively, and finally performing splice operation on the result obtained by performing add operation on the four-layer sub-network structure respectively to obtain a second encoded image E1'.
Specifically, the calculation process of the edge extraction operation is as follows:
G(x,y)＝x(i,j)+y(i,j)
wherein G is a gradient calculation result of the first encoded image, edge is an Edge calculation result, x (i, j) is an Edge of the first encoded image in the x direction, and y (i, j) is an Edge of the first encoded image in the y direction.
Further, x (i, j) and y (i, j) are calculated based on the following:
where img is the edge extracted image and (i, j) is the coordinates of the first encoded image.
Next, referring to fig. 4, a schematic structural diagram of a cross feature fusion CFM module according to the present invention is provided. The cross feature fusion CFM module comprises:
a first branch structure and a second branch structure; the first branch structure is used for performing Softmax operation on the second coded image of the current network layer to obtain a first characteristic parameter.
Specifically, after Softmax operation is performed on the second encoded image, a characteristic parameter sigma is obtained, and then 1-sigma operation is performed to obtain sigma ', wherein sigma' is the first characteristic parameter.
And the second branch structure is used for carrying out feature fusion operation on the plurality of first coded images in the other network layers to obtain a first feature map.
Specifically, the second branch structure includes a plurality of sub-branch structures, and each sub-branch structure is respectively used for performing Softmax operation on the first encoded image to respectively obtain the characteristic parameter sigma. And performing point multiplication operation on the characteristic parameter sigma and the first coding image in each sub-branch structure respectively to obtain a plurality of coding characteristic images. And adding the plurality of coded characteristic images to obtain a first characteristic image EE. The encoded feature image EE and the first feature parameter σ'
And performing feature fusion operation on the first feature map, the first feature parameters and the second coded image of the current network layer to obtain a third coded image corresponding to the current network layer.
Specifically, the first feature map EE performs a dot product operation with the first feature parameter σ' to obtain a second feature map EET, and the second feature map EET performs an addition operation with the second encoded image to perform feature fusion, so as to finally obtain a third encoded image Ei.
Further, the calculation process for obtaining the third encoded image Ei is as follows:
σ′＝1-Softmax(Ei′)， Ei * =eet+ei'. Wherein the variables in the formula correspond to the respective feature graphs and intermediate-valued variables in fig. 4, i represents the number of corresponding feature layers, i=1, 2,3,4, j=1, 2,3,4, and j+.i>Representing dot product.
Illustratively, the segmentation loss function employed by the deep-learning network model includes at least one of:
CELoss loss function; diceLoss loss function.
Wherein, the expression of the CELoss loss function is:
CELoss＝-[ylogy′+(1-y)log(1-y′)]
the expression of the DiceLoss loss function is:
if the deep learning network model adopts the two loss functions, the expression of the loss functions is:
Loss＝α·CELoss+(1-α)·DiceLoss
wherein y is a label value, y' is a predicted value, and alpha is a loss weight coefficient
Based on the same technical conception as the cervical vertebra segmentation method based on the fusion of the edge pyramid and the cross feature, the invention provides a cervical vertebra segmentation device based on the fusion of the edge pyramid and the cross feature. The cervical vertebra segmentation device has the same function as the cervical vertebra segmentation method, and is not described in detail herein.
Referring to fig. 5, a schematic block diagram of a cervical vertebra segmentation apparatus based on fusion of an edge pyramid and a cross feature according to the present invention is provided, where the cervical vertebra segmentation apparatus includes:
an acquisition module 51 for acquiring a cervical vertebra medical image to be segmented;
the input module 52 is configured to input the cervical vertebra medical image to be segmented into a deep learning network model for encoding operation, so as to obtain a plurality of first encoded images with different sizes;
the extracting module 53 is configured to extract a PEEM module based on a plurality of pyramid features in the deep learning network model, and perform an edge extracting operation on the first encoded image to obtain a plurality of second encoded images with different sizes;
the fusion module 54 is configured to perform cross feature fusion operation on the plurality of first encoded images and the plurality of second encoded images based on the plurality of cross feature fusion CFM modules in the deep learning network model, so as to obtain a plurality of third encoded images with different sizes;
the processing module 55 is configured to perform a fusion operation based on the third encoded images with different sizes and the decoded images with different sizes, and output a segmentation result of the cervical vertebra medical image;
wherein the plurality of decoded images corresponds to the plurality of first encoded images in one-to-one correspondence in size.
Optionally, the deep learning network model includes a five-layer network structure, the first layer network structure is a network layer inputting the cervical vertebra medical image to be segmented, and the sizes of the first coding images from the first layer network structure to the fifth layer network structure are sequentially reduced;
the pyramid feature extraction module and the cross feature fusion module are arranged on the first-layer network structure to the fourth-layer network structure;
the cross feature fusion module in the current network layer is used for carrying out cross feature fusion operation on the second coded image of the current network layer and the first coded images of other network layers to obtain a third coded image of the current network layer;
the current network layer is any one of the first layer network structure to the fourth layer network structure, and the other network layers comprise a plurality of network layers from the first layer network structure to the fourth layer network structure except the current network layer.
Optionally, the PEEM module comprises a four-layer sub-network structure;
each layer of sub-network structure is used for carrying out edge extraction operation, average pooling operation, addition operation and splicing operation on the first coded image of the current network layer to obtain a second coded image;
wherein the sub-network structures of different layers have different sizes for performing the average pooling operation.
Optionally, the calculating process of the edge extraction operation is as follows:
G(x,y)＝x(i,j)+y(i,j)
wherein G is a gradient calculation result of the first encoded image, edge is an Edge calculation result, x (i, j) is an Edge of the first encoded image in the x direction, and y (i, j) is an Edge of the first encoded image in the y direction.
Optionally, x (i, j) and y (i, j) are calculated based on the following:
where img is the edge extracted image and (i, j) is the coordinates of the first encoded image.
Optionally, the cross feature fusion CFM module includes:
a first branch structure and a second branch structure;
the first branch structure is used for performing Softmax operation on a second coded image of the current network layer to obtain a first characteristic parameter;
the second branch structure is used for carrying out feature fusion operation on a plurality of first coded images in the other network layers to obtain a first feature map;
and performing feature fusion operation on the first feature map, the first feature parameters and the second coded image of the current network layer to obtain a third coded image corresponding to the current network layer.
Optionally, the segmentation loss function adopted by the deep learning network model includes at least one of the following:
CELoss loss function; diceLoss loss function.
Referring next to fig. 6, a schematic structural diagram of an electronic device according to the present invention is provided.
The electronic device may include: processor 610, communication interface (Communications Interface) 620, memory 630, and communication bus 640, wherein processor 610, communication interface 620, and memory 630 communicate with each other via communication bus 640. The processor 610 may invoke logic instructions in the memory 630 to perform the cervical segmentation method based on fusion of edge pyramids and cross features provided by the methods described above.
Further, the logic instructions in the memory 630 may be implemented in the form of software functional units and stored in a computer-readable storage medium when sold or used as a stand-alone product. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
Another aspect of the invention provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a cervical spine segmentation method based on edge pyramid and intersection feature fusion as described above.
The computer readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: portable computer disks, hard disks, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static Random Access Memory (SRAM), portable compact disk read-only memory (CD-ROM), digital Versatile Disks (DVD), memory sticks, floppy disks, mechanical coding devices, punch cards or in-groove structures such as punch cards or grooves having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media, as used herein, are not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., optical pulses through fiber optic cables), or electrical signals transmitted through wires.
The computer readable program instructions described herein may be downloaded from a computer readable storage medium to a respective computing/processing device or to an external computer or external storage device over a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers and/or edge servers. The network interface card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium in the respective computing/processing device.
Computer program instructions for carrying out operations of the present invention may be assembly instructions, instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, c++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, aspects of the present invention are implemented by personalizing electronic circuitry, such as programmable logic circuitry, field Programmable Gate Arrays (FPGAs), or Programmable Logic Arrays (PLAs), with state information for computer readable program instructions, which can execute the computer readable program instructions.
Various aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.
These computer readable program instructions may be provided to a processing unit of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable medium having the instructions stored therein includes an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
Note that all features disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise. Thus, unless expressly stated otherwise, each feature disclosed is one example only of a generic set of equivalent or similar features. Where used, further, preferably, still further and preferably, the brief description of the other embodiment is provided on the basis of the foregoing embodiment, and further, preferably, further or more preferably, the combination of the contents of the rear band with the foregoing embodiment is provided as a complete construct of the other embodiment. A further embodiment is composed of several further, preferably, still further or preferably arrangements of the strips after the same embodiment, which may be combined arbitrarily.
It will be appreciated by persons skilled in the art that the embodiments of the invention described above and shown in the drawings are by way of example only and are not limiting. The objects of the present invention have been fully and effectively achieved. The functional and structural principles of the present invention have been shown and described in the examples and embodiments of the invention may be modified or practiced without departing from the principles described.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present disclosure, and not for limiting the same; although the present disclosure has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present disclosure.
Claims (10)
1. The cervical vertebra segmentation method based on the fusion of the edge pyramid and the cross feature is characterized by comprising the following steps:
acquiring a cervical vertebra medical image to be segmented;
inputting the cervical vertebra medical image to be segmented into a deep learning network model for coding operation to obtain a plurality of first coding images with different sizes;
based on a plurality of pyramid feature extraction PEEM modules in the deep learning network model, respectively performing edge extraction operation on the first coded image to obtain a plurality of second coded images with different sizes;
based on a plurality of cross feature fusion CFM modules in the deep learning network model, cross feature fusion operation is respectively carried out on the plurality of first coded images and the plurality of second coded images, so that a plurality of third coded images with different sizes are obtained;
based on the fusion operation of the third coded images with different sizes and the decoding images with different sizes, outputting and obtaining a segmentation result of the cervical vertebra medical image;
wherein the plurality of decoded images corresponds to the plurality of first encoded images in one-to-one correspondence in size.
2. The cervical vertebra segmentation method based on the fusion of the edge pyramid and the cross feature according to claim 1, wherein the deep learning network model comprises five layers of network structures, a first layer of network structure is a network layer for inputting the cervical vertebra medical image to be segmented, and the sizes of first coding images of the first layer of network structure to the fifth layer of network structure are sequentially reduced;
the pyramid feature extraction module and the cross feature fusion module are arranged on the first-layer network structure to the fourth-layer network structure;
the cross feature fusion module in the current network layer is used for carrying out cross feature fusion operation on the second coded image of the current network layer and the first coded images of other network layers to obtain a third coded image of the current network layer;
the current network layer is any one of the first layer network structure to the fourth layer network structure, and the other network layers comprise a plurality of network layers from the first layer network structure to the fourth layer network structure except the current network layer.
3. The cervical spine segmentation method based on edge pyramid and intersection feature fusion of claim 2, wherein the PEEM module comprises a four-layer sub-network structure;
each layer of sub-network structure is used for carrying out edge extraction operation, average pooling operation, addition operation and splicing operation on the first coded image of the current network layer to obtain a second coded image;
wherein the sub-network structures of different layers have different sizes for performing the average pooling operation.
4. The cervical spine segmentation method based on fusion of edge pyramids and cross features according to claim 3, wherein the edge extraction operation is calculated as follows:
G(x,y)＝x(i,j)+y(i,j)
wherein G is a gradient calculation result of the first encoded image, edge is an Edge calculation result, x (i, j) is an Edge of the first encoded image in the x direction, and y (i, j) is an Edge of the first encoded image in the y direction.
5. The cervical spine segmentation method based on edge pyramid and intersection feature fusion of claim 4, wherein x (i, j) and y (i, j) are calculated based on the following:
where img is the edge extracted image and (i, j) is the coordinates of the first encoded image.
6. The cervical spine segmentation method based on edge pyramid and cross feature fusion of claim 2, wherein the cross feature fusion CFM module comprises:
a first branch structure and a second branch structure;
the first branch structure is used for performing Softmax operation on a second coded image of the current network layer to obtain a first characteristic parameter;
the second branch structure is used for carrying out feature fusion operation on a plurality of first coded images in the other network layers to obtain a first feature map;
and performing feature fusion operation on the first feature map, the first feature parameters and the second coded image of the current network layer to obtain a third coded image corresponding to the current network layer.
7. The cervical spine segmentation method based on edge pyramid and cross feature fusion of claim 2, wherein the segmentation loss function adopted by the deep learning network model comprises at least one of the following:
CELoss loss function; diceLoss loss function.
8. Cervical vertebra segmentation apparatus based on edge pyramid and cross feature fusion, characterized by comprising:
the acquisition module is used for acquiring the cervical vertebra medical image to be segmented;
the input module is used for inputting the cervical vertebra medical image to be segmented into a deep learning network model for coding operation to obtain a plurality of first coding images with different sizes;
the extraction module is used for extracting PEEM modules based on a plurality of pyramid features in the deep learning network model, and respectively carrying out edge extraction operation on the first coded image to obtain a plurality of second coded images with different sizes;
the fusion module is used for respectively carrying out cross feature fusion operation on the plurality of first coded images and the plurality of second coded images based on the plurality of cross feature fusion CFM modules in the deep learning network model to obtain a plurality of third coded images with different sizes;
the processing module is used for carrying out fusion operation on the basis of the plurality of third coding images with different sizes and the plurality of decoding images with different sizes and outputting and obtaining a segmentation result of the cervical vertebra medical image;
wherein the plurality of decoded images corresponds to the plurality of first encoded images in one-to-one correspondence in size.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the cervical segmentation method based on fusion of edge pyramids with intersecting features as claimed in any one of claims 1 to 7 when the program is executed by the processor.
10. A non-transitory computer readable storage medium having stored thereon a computer program, wherein the computer program when executed by a processor implements the cervical spine segmentation method based on fusion of edge pyramids with intersecting features as claimed in any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310340206.8A CN116563217A (en) | 2023-03-31 | 2023-03-31 | Cervical vertebra segmentation method and device based on fusion of edge pyramid and cross feature |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310340206.8A CN116563217A (en) | 2023-03-31 | 2023-03-31 | Cervical vertebra segmentation method and device based on fusion of edge pyramid and cross feature |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116563217A true CN116563217A (en) | 2023-08-08 |
Family
ID=87488764
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310340206.8A Pending CN116563217A (en) | 2023-03-31 | 2023-03-31 | Cervical vertebra segmentation method and device based on fusion of edge pyramid and cross feature |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116563217A (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112561937A (en) * | 2020-12-18 | 2021-03-26 | 深圳先进技术研究院 | Context-aware medical image segmentation method based on lesion boundary information response |
| WO2022032824A1 (en) * | 2020-08-10 | 2022-02-17 | 中国科学院深圳先进技术研究院 | Image segmentation method and apparatus, device, and storage medium |
| CN114943876A (en) * | 2022-06-20 | 2022-08-26 | 南京信息工程大学 | Cloud and cloud shadow detection method and device for multi-level semantic fusion and storage medium |
-
2023
- 2023-03-31 CN CN202310340206.8A patent/CN116563217A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022032824A1 (en) * | 2020-08-10 | 2022-02-17 | 中国科学院深圳先进技术研究院 | Image segmentation method and apparatus, device, and storage medium |
| CN112561937A (en) * | 2020-12-18 | 2021-03-26 | 深圳先进技术研究院 | Context-aware medical image segmentation method based on lesion boundary information response |
| CN114943876A (en) * | 2022-06-20 | 2022-08-26 | 南京信息工程大学 | Cloud and cloud shadow detection method and device for multi-level semantic fusion and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| RUXIN WANG 等: "Boundary-aware context neural network for medical image segmentation", 《MEDICAL IMAGE ANALYSIS 》, 14 February 2022 (2022-02-14), pages 1 - 13 * |
| 姜世浩;齐苏敏;王来花;贾惠;: "基于Mask R-CNN和多特征融合的实例分割", 计算机技术与发展, no. 09, 10 September 2020 (2020-09-10) * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112699991B (en) | Method, electronic device, and computer-readable medium for accelerating information processing for neural network training | |
| CN108664981B (en) | Salient image extraction method and device | |
| CN109165660B (en) | Significant object detection method based on convolutional neural network | |
| CN111488985B (en) | Deep neural network model compression training method, device, equipment and medium | |
| CN109559363B (en) | Image stylization processing method and device, medium and electronic equipment | |
| CN111832570A (en) | Image semantic segmentation model training method and system | |
| CN108629414B (en) | Deep hash learning method and device | |
| CN113379627A (en) | Training method of image enhancement model and method for enhancing image | |
| CN111914654B (en) | Text layout analysis method, device, equipment and medium | |
| CN114581462A (en) | Image segmentation method, device, equipment and storage medium | |
| CN115731214A (en) | Medical image segmentation method and device based on artificial intelligence | |
| CN115471718A (en) | Construction and detection method of lightweight significance target detection model based on multi-scale learning | |
| CN109241930B (en) | Method and apparatus for processing eyebrow image | |
| CN114781499A (en) | Method for constructing ViT model-based intensive prediction task adapter | |
| CN114078097A (en) | Method and device for acquiring image defogging model and electronic equipment | |
| CN116402914B (en) | Method, device and product for determining stylized image generation model | |
| CN111582284A (en) | Privacy protection method and device for image recognition and electronic equipment | |
| CN116363429A (en) | Training method of image recognition model, image recognition method, device and equipment | |
| CN117333428A (en) | CT image segmentation method, device, equipment and storage medium | |
| CN116563217A (en) | Cervical vertebra segmentation method and device based on fusion of edge pyramid and cross feature | |
| CN115810152A (en) | Remote sensing image change detection method and device based on graph convolution and computer equipment | |
| CN113139617B (en) | Power transmission line autonomous positioning method and device and terminal equipment | |
| CN115565186A (en) | Method and device for training character recognition model, electronic equipment and storage medium | |
| CN116402780B (en) | Thoracic vertebra image segmentation method and device based on double self-attention and deep learning | |
| CN113283241B (en) | Text recognition method and device, electronic equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |