CN117830261A - Physiological channel labeling method, system, device and nonvolatile storage medium - Google Patents
Physiological channel labeling method, system, device and nonvolatile storage medium Download PDFInfo
- Publication number
- CN117830261A CN117830261A CN202311869032.0A CN202311869032A CN117830261A CN 117830261 A CN117830261 A CN 117830261A CN 202311869032 A CN202311869032 A CN 202311869032A CN 117830261 A CN117830261 A CN 117830261A
- Authority
- CN
- China
- Prior art keywords
- feature
- point
- point cloud
- channel
- layer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000002372 labelling Methods 0.000 title claims abstract description 158
- 238000003860 storage Methods 0.000 title claims abstract description 19
- 238000012545 processing Methods 0.000 claims abstract description 94
- 238000005070 sampling Methods 0.000 claims description 75
- 238000000034 method Methods 0.000 claims description 33
- 238000010586 diagram Methods 0.000 claims description 21
- 238000004590 computer program Methods 0.000 claims description 12
- 238000000605 extraction Methods 0.000 claims description 10
- 230000008569 process Effects 0.000 description 10
- 238000009826 distribution Methods 0.000 description 9
- 238000012549 training Methods 0.000 description 9
- 230000011218 segmentation Effects 0.000 description 7
- 239000013598 vector Substances 0.000 description 7
- 238000013459 approach Methods 0.000 description 5
- 238000013528 artificial neural network Methods 0.000 description 5
- 238000013527 convolutional neural network Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 210000004204 blood vessel Anatomy 0.000 description 3
- 210000000621 bronchi Anatomy 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000001965 increasing effect Effects 0.000 description 3
- 230000009191 jumping Effects 0.000 description 3
- 239000011159 matrix material Substances 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 3
- 208000006545 Chronic Obstructive Pulmonary Disease Diseases 0.000 description 2
- 230000009471 action Effects 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000013170 computed tomography imaging Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 238000004445 quantitative analysis Methods 0.000 description 2
- 208000017667 Chronic Disease Diseases 0.000 description 1
- 208000019693 Lung disease Diseases 0.000 description 1
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 201000010099 disease Diseases 0.000 description 1
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 238000003709 image segmentation Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000001939 inductive effect Effects 0.000 description 1
- 210000004072 lung Anatomy 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 231100000915 pathological change Toxicity 0.000 description 1
- 230000036285 pathological change Effects 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 230000000644 propagated effect Effects 0.000 description 1
- 238000007634 remodeling Methods 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Landscapes
- Measuring And Recording Apparatus For Diagnosis (AREA)
Abstract
The embodiment of the application discloses a physiological channel labeling method, a physiological channel labeling system, a physiological channel labeling device and a non-volatile storage medium. The physiological channel labeling method comprises the following steps: extracting a first feature set from the physiological channel point cloud data according to distance information among points in the physiological channel point cloud data and density information in the point cloud data through a first multi-scale point cloud set abstract layer; extracting a second feature set from the first feature point set according to the distance information and the density information of each point in the first feature point set through a second multi-scale point cloud set abstract layer of the physiological channel labeling model; extracting a third feature set from the second feature point set according to the distance information of each point in the second feature point set through a single-scale point cloud set abstract layer of the physiological channel labeling model; and sequentially processing the first feature set, the second feature set and the third feature set through a plurality of feature merging layers of the physiological channel labeling model to obtain labeling results corresponding to the physiological channel point cloud data.
Description
Technical Field
The present disclosure relates to the field of medical images, and in particular, to a method, a system, a device, and a non-volatile storage medium for labeling a physiological channel.
Background
In the related art, when marking and dividing the physiological channels such as the airway tree, the common technical means is to mark the unlabeled physiological channel model based on the marked physiological channel model, and define the corresponding relation between the two physiological channel models by extracting the vision and topological features on branches and nodes in the physiological channel model, thereby finishing the marking of the unlabeled physiological channel model. The problem of this approach is that there may be differences in physiological channel structures among different people, and there may be changes in physiological channel structures due to case changes, so the approach of labeling unlabeled physiological channel models based on labeled physiological channel models has a narrow application range and is not universal.
Disclosure of Invention
The embodiment of the application provides a physiological channel labeling method, a system, a device and a nonvolatile storage medium, which are used for solving the technical problem of narrow application range caused by labeling an unlabeled physiological channel model by adopting the labeled physiological channel model in the related technology.
The embodiment of the application provides a physiological channel labeling method, which comprises the following steps: extracting a first feature set from the physiological channel point cloud data according to distance information among points in the physiological channel point cloud data to be marked and density information in the point cloud data through a first multi-scale point cloud set abstract layer of a physiological channel labeling model, wherein the first feature set comprises a first feature point set and corresponding first point cloud features, and a channel space attention module is included in the first multi-scale point cloud set abstract layer; extracting a second feature set from the first feature point set according to the distance information and the density information of each point in the first feature point set through a second multi-scale point cloud set abstract layer of the physiological channel labeling model, wherein the second feature set comprises the second feature point set and second point cloud features, and the second multi-scale point cloud set abstract layer comprises a spatial attention module; extracting a third feature set from the second feature point set according to the distance information of each point in the second feature point set through a single-scale point cloud set abstraction layer of the physiological channel labeling model, wherein the third feature set comprises the third feature point set and third point cloud features; and sequentially processing the first feature set, the second feature set and the third feature set through a plurality of feature merging layers of the physiological channel labeling model to obtain labeling results corresponding to the physiological channel point cloud data, wherein the labeling results comprise label information corresponding to each point in the physiological channel point cloud data.
Optionally, the first multi-scale point cloud aggregation abstract layer further includes a first sampling layer, a first grouping layer, and a first point network layer, and the channel spatial attention module includes a spatial attention module and a channel attention module; the step of extracting a first feature set from the physiological channel point cloud data through a first multi-scale point cloud set abstract layer of the physiological channel labeling model according to distance information among points in the physiological channel point cloud data to be labeled and density information in the point cloud data comprises the following steps: determining a first center point set from the physiological channel point cloud data by adopting a furthest point sampling mode through a first sampling layer; determining point set groups corresponding to all central points in the first central point set in the physiological channel point cloud data according to the first central point set by adopting a ball query mode through the first grouping layer; extracting the characteristics of each point set group through the first point network layer to obtain local area characteristics corresponding to each point set group; and processing the local region features through the channel space attention module to obtain a first feature set.
Optionally, the first set of center points includes points in the physiological channel point cloud data that are arbitrarily located at end points of the physiological channel.
Optionally, the step of processing the local region features by the channel spatial attention module to obtain the first feature set includes: processing each local area characteristic by adopting a channel attention module in the channel space attention module to obtain a channel attention characteristic diagram corresponding to each local area characteristic; superposing the local area features and the channel attention feature map to obtain a first processing result, wherein the first processing result is a feature set with the feature weights of the local area features adjusted in the channel dimension according to the channel attention feature map; processing the first processing result by adopting a spatial attention module to obtain a spatial attention characteristic diagram corresponding to the first processing result; and carrying out superposition processing on the spatial attention feature map and the first processing result to obtain a first point cloud feature contained in the first feature set, wherein the first point cloud feature is a feature set with feature weights of the first processing result adjusted in a spatial dimension according to the spatial attention feature map.
Optionally, the second multi-scale point cloud collection abstraction layer includes a second sampling layer, a second grouping layer, a second point mesh layer, and a spatial attention module; the step of extracting the second feature set from the first feature point set by the second multi-scale point cloud set abstraction layer of the physiological channel labeling model according to distance information among points in the first feature point set and density information in the point cloud data comprises the following steps: determining a second center point set from the first characteristic point set by adopting a furthest point sampling mode through a second sampling layer; determining point set groups corresponding to all center points in the second center point set in the first characteristic point set according to the second center point set by adopting a ball query mode through the second grouping layer; extracting the characteristics of each point set group through the second point network layer to obtain local area characteristics corresponding to each point set group; and processing the local region features through the spatial attention module to obtain a second feature set.
Optionally, the single-scale point cloud set abstract layer comprises a third sampling layer, a third grouping layer and a third point network layer; the step of extracting a third feature set from the second feature point set according to the distance information of each point in the second feature point set through the single-scale point cloud set abstract layer of the physiological channel labeling model comprises the following steps: determining a third center point set from the first characteristic point set by a third sampling layer in a most distant point sampling mode; determining point set groups corresponding to all center points in the third center point set in the first characteristic point set according to the center point set by adopting a ball query mode through the third grouping layer; and extracting the characteristics of each point set group through a third point network layer to obtain local area characteristics corresponding to each point set group, wherein the local area characteristics corresponding to each point set group are a third characteristic set.
Optionally, the step of splicing the first feature set, the second feature set and the third feature set through the feature merging layer of the physiological channel labeling model to obtain a labeling result includes: upsampling the third feature set to obtain a first upsampling result; upsampling the second feature set and the first upsampling result to obtain a second upsampling processing result; and upsampling the first feature set and the second upsampling result to obtain a third upsampling processing result, and obtaining a labeling result based on the third upsampling processing result.
Optionally, the physiological channel labeling model comprises a feature merging layer, wherein the feature merging layer comprises a first distance interpolation feature layering propagation module, a second distance interpolation feature layering propagation module and a third distance interpolation feature layering propagation module, the first distance interpolation feature layering propagation module is connected with the second distance interpolation feature layering propagation module, and the second distance interpolation feature layering propagation module is connected with the third distance interpolation feature layering propagation module; the first multi-scale point cloud set abstract layer is connected with the third distance interpolation characteristic layering propagation module, the second multi-scale point cloud set abstract layer is connected with the second distance interpolation characteristic layering propagation module, and the single-scale point cloud set abstract layer is connected with the third distance interpolation characteristic layering propagation module.
Optionally, the first distance interpolation feature layered propagation module, the second distance interpolation feature layered propagation module and the third distance interpolation feature layered propagation module each include a distance interpolation layer and a feature extraction layer which are sequentially connected.
The embodiment of the application also provides a physiological channel labeling system, which comprises an image acquisition module, a processor and a display module, wherein the image acquisition module is used for acquiring physiological channel images of a physiological channel and generating physiological channel point cloud data to be labeled according to the physiological channel images; the processor is used for extracting a first feature set from the physiological channel point cloud data according to the distance information between each point in the physiological channel point cloud data to be marked and the density information in the point cloud data through a first multi-scale point cloud set abstract layer of the physiological channel labeling model, wherein the first feature set comprises a first feature point set and corresponding first point cloud features, and a channel space attention module is included in the first multi-scale point cloud set abstract layer; extracting a second feature set from the first feature point set according to the distance information and the density information of each point in the first feature point set through a second multi-scale point cloud set abstract layer of the physiological channel labeling model, wherein the second feature set comprises the second feature point set and second point cloud features, and the second multi-scale point cloud set abstract layer comprises a spatial attention module; extracting a third feature set from the second feature point set according to the distance information of each point in the second feature point set through a single-scale point cloud set abstraction layer of the physiological channel labeling model, wherein the third feature set comprises the third feature point set and third point cloud features; sequentially processing the first feature set, the second feature set and the third feature set through a plurality of feature merging layers of the physiological channel labeling model to obtain labeling results corresponding to the physiological channel point cloud data, wherein the labeling results comprise label information corresponding to each point in the physiological channel point cloud data; and the display module is used for displaying the labeling result.
The embodiment of the application also provides a physiological channel labeling device, which comprises: the first processing module is used for extracting a first feature set from the physiological channel point cloud data according to the distance information between each point in the physiological channel point cloud data to be marked and the density information in the point cloud data through a first multi-scale point cloud set abstract layer of the physiological channel labeling model, wherein the first feature set comprises a first feature point set and corresponding first point cloud features, and the first multi-scale point cloud set abstract layer comprises a channel space attention module; the second processing module is used for extracting a second feature set from the first feature point set according to the distance information and the density information of each point in the first feature point set through a second multi-scale point cloud set abstract layer of the physiological channel labeling model, wherein the second feature set comprises the second feature point set and second point cloud features, and the second multi-scale point cloud set abstract layer comprises a spatial attention module; the third processing module is used for extracting a third feature set from the second feature point set according to the distance information of each point in the second feature point set through the single-scale point cloud set abstract layer of the physiological channel labeling model, wherein the third feature set comprises a third feature point set and third point cloud features; and the fourth processing module is used for sequentially processing the first feature set, the second feature set and the third feature set through a plurality of feature merging layers of the physiological channel labeling model to obtain labeling results corresponding to the physiological channel point cloud data, wherein the labeling results comprise label information corresponding to each point in the physiological channel point cloud data.
The embodiment of the application also provides a nonvolatile storage medium, wherein the nonvolatile storage medium stores a computer program, and the computer program realizes the physiological channel labeling method when being executed by a processor.
The embodiment of the application provides electronic equipment, which comprises a memory, a processor and a computer program stored on the memory and capable of being operated on the processor, wherein the processor realizes a physiological channel labeling method when executing the computer program.
Based on the above scheme, the method and the device extract feature sets of different levels for multiple times according to the distance information of each point in the point cloud data of the physiological channel and the density information in the point cloud data, and fuse the feature sets of different levels to determine the labeling result of the point cloud data of the physiological channel, so that the labeling result of the point cloud data of the unlabeled physiological channel is obtained without comparing the existing labeling result, the automatic labeling of the point cloud data of the physiological channel is realized, the accuracy and the reliability of the obtained labeling result are improved, and the technical problem that the application range of labeling the unlabeled physiological channel model by adopting the labeled physiological channel model in the related technology is narrower is solved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the related art, the following description will briefly describe the drawings that are required to be used in the embodiments or the related technical descriptions, and it is obvious that, in the following description, the drawings are some embodiments of the present application, and other drawings may be obtained according to the drawings without any inventive effort to those skilled in the art.
FIG. 1 is a schematic diagram of a physiological channel labeling system according to an exemplary embodiment of the present application;
FIG. 2 is a flow chart of a method of labeling physiological channels in accordance with an exemplary embodiment of the present application;
FIG. 3 is a schematic diagram of the structure of a physiological channel labeling model in accordance with an exemplary embodiment of the present application;
FIG. 4 is a schematic diagram of the architecture of a first multi-scale point cloud collection abstraction layer in accordance with an example embodiment of the subject application;
FIG. 5 is a schematic diagram of a multi-scale sampling scheme in accordance with an exemplary embodiment of the present application;
FIG. 6 is a schematic diagram of a channel space attention module in accordance with an exemplary embodiment of the present application;
FIG. 7 is a schematic diagram of the architecture of a second multi-scale point cloud collection abstraction layer in accordance with an example embodiment of the subject application;
FIG. 8 is a schematic diagram of another multi-scale sampling approach in accordance with an exemplary embodiment of the present application;
FIG. 9 is a schematic diagram of a structure of a single-scale point cloud collection abstraction layer in accordance with an example embodiment of the subject application;
FIG. 10 is a schematic illustration of a physiological channel segmentation result in accordance with an exemplary embodiment of the present application;
FIG. 11 is a schematic diagram of point cloud data labeling results and labels in accordance with an exemplary embodiment of the present application;
FIG. 12 is a flow diagram of a physiological channel labeling model training flow in accordance with an exemplary embodiment of the present application;
FIG. 13 is a schematic structural view of a physiological channel labeling apparatus according to an exemplary embodiment of the present application;
fig. 14 is a schematic structural view of an electronic device according to an exemplary embodiment of the present application.
Detailed Description
For the purposes of making the objects, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that embodiments of the present application described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
For better understanding of the embodiments of the present application, technical terms related in the embodiments of the present application are explained below:
max: maximum pooling;
channel Attention: a channel attention module mechanism;
spatial Attention: a spatial attention module mechanism;
interpolation of distance: distance interpolation (the basic principle of distance interpolation is that the distance between two points can be estimated by the distance between the points closest to them, by calculating the distance between two points and then using this value to estimate the distance between two points that are not directly connected.)
Unit Point Net: similar to the 1x1 convolution in a convolutional neural network.
The technical scheme of the present application is described in detail below with specific examples. The following embodiments may be combined with each other, and some embodiments may not be repeated for the same or similar concepts or processes.
Chronic Obstructive Pulmonary Disease (COPD) is one of the most prevalent pulmonary diseases worldwide and is also the leading cause of inducing chronic disease and death. Among them, airway narrowing, obstruction and reconstruction are typical COPD features. Airway tree remodeling is therefore critical to assessing the severity and progression of disease. Chest CT imaging is a commonly used tool for quantitative analysis of living airways. However, when chest CT imaging is used for quantitative analysis of living airways, it is necessary to automatically segment scanned physiological channel model data such as airway tree. And after the airway tree is automatically segmented, the airway tree may be applied to specific regional anatomic branches. Automatic airway marking may expedite this process. And airway markers also have a great role in planning bronchoscopic interventions.
The airway automatic marking algorithm in the related art generally matches marked trees with unmarked trees, and defines the corresponding relation between the two trees by extracting visual and topological features on branches and nodes, thereby completing airway tree marking. Specifically, in the related art, according to the anatomical prior knowledge (the difference of the length, the spatial direction and the angle between branches) of the airway, the optimal match between the two trees is searched on the association graph as the maximum clique so as to measure the association between the branch points. In addition, a supervised machine learning based approach may also be employed to model the probability distribution of branch labels based on feature pairs given their features on the training data. For example, it may be assumed that the different anatomical branches are characterized by independent gaussian distributions, and the airways are labeled with learned gaussian distributions of direction, average radius, angle relative to the parent branch, and so on.
However, the topology structure of each airway tree is different, and the influence of noise and the structural change in the image acquisition process cause false branches or partial branches of the segmented airway tree to be absent, and in addition, the structure of the airway tree may also change due to pathological changes, so that the traditional airway tree automatic marking method has no universality.
With the continuous development of deep learning technology, convolutional neural networks and graph neural networks exert strong feature extraction and learning capabilities in various image segmentation tasks, and have better accuracy and robustness. The representative 3DU-Net airway tree segmentation algorithm can automatically learn the airway tree characteristics, thereby realizing the automatic labeling of the airway tree. However, due to sparse distribution (class imbalance) of third-level bronchi and no obvious interface between adjacent segments (voxels around the interface have similar local appearance and gray distribution), the semantic segmentation result has errors, and the processing of three-dimensional data has higher hardware requirements.
A method for inferring relationships between sampling points based on a graph neural network to label the airway tree is also provided in the related art, but the method is not end-to-end optimized, but relies on hand-made features as input.
The related technology also provides a labeling method for performing semantic segmentation on CTA images of head and neck blood vessels based on a graph neural network point cloud method, wherein the binary segmentation of the blood vessels is performed based on a convolution neural network and is converted into point cloud, and then the graph neural network is adopted to label the blood vessels. However, tree structures are not fully utilized a priori in this approach.
In order to solve the above problems, the present application provides a method, a system, a device and a non-volatile storage medium for labeling a physiological channel, which are described in detail below.
The following describes in detail the technical solutions provided by the embodiments of the present specification with reference to the accompanying drawings.
Fig. 1 is a schematic structural diagram of a physiological channel labeling system according to an embodiment of the present disclosure, which may be used to perform a physiological channel labeling method. Specifically, as shown in fig. 1, the system includes: the system comprises an image acquisition module 10, a processor 12 and a display module 14, wherein the image acquisition module 10 is used for acquiring a physiological channel image of a physiological channel and generating physiological channel point cloud data to be marked according to the physiological channel image; the processor 12 is configured to extract, by using a first multi-scale point cloud set abstraction layer of the physiological channel labeling model, a first feature set from the physiological channel point cloud data according to distance information between points in the physiological channel point cloud data to be labeled and density information in the point cloud data, where the first feature set includes a first feature point set and corresponding first point cloud features, and the first multi-scale point cloud set abstraction layer includes a channel space attention module; extracting a second feature set from the first feature point set according to the distance information and the density information of each point in the first feature point set through a second multi-scale point cloud set abstract layer of the physiological channel labeling model, wherein the second feature set comprises the second feature point set and second point cloud features, and the second multi-scale point cloud set abstract layer comprises a spatial attention module; extracting a third feature set from the second feature point set according to the distance information of each point in the second feature point set through a single-scale point cloud set abstraction layer of the physiological channel labeling model, wherein the third feature set comprises the third feature point set and third point cloud features; sequentially processing the first feature set, the second feature set and the third feature set through a plurality of feature merging layers of the physiological channel labeling model to obtain labeling results corresponding to the physiological channel point cloud data, wherein the labeling results comprise label information corresponding to each point in the physiological channel point cloud data; and a display module 14 for displaying the labeling result.
Fig. 2 is a flowchart of a physiological channel labeling method according to an embodiment of the present disclosure, which specifically includes the following steps:
step 202, extracting a first feature set from physiological channel point cloud data according to distance information among points in the physiological channel point cloud data to be marked and density information in the point cloud data through a first multi-scale point cloud set abstract layer of a physiological channel labeling model, wherein the first feature set comprises a first feature point set and corresponding first point cloud features, and a channel space attention module is included in the first multi-scale point cloud set abstract layer;
in some embodiments of the present application, the structure of the physiological channel labeling model is shown in fig. 3, and may be divided into a feature extraction layer and a feature merging layer, where the feature extraction layer includes a first multi-scale point cloud collection abstract layer, a second multi-scale point cloud collection abstract layer, and a single-scale point cloud collection abstract layer that are sequentially connected, and a channel spatial attention module is provided in the first multi-scale point cloud collection abstract layer, and a spatial attention module is provided in the second multi-scale point cloud collection abstract layer.
The feature merging layer comprises a first distance interpolation feature layering propagation module, a second distance interpolation feature layering propagation module and a third distance interpolation feature layering propagation module, wherein the first distance interpolation feature layering propagation module is connected with the second distance interpolation feature layering propagation module, and the second distance interpolation feature layering propagation module is connected with the third distance interpolation feature layering propagation module.
In addition, as can be seen from fig. 3, the first multi-scale point cloud set abstract layer is connected with the third distance interpolation characteristic layering propagation module, the second multi-scale point cloud set abstract layer is connected with the second distance interpolation characteristic layering propagation module, and the single-scale point cloud set abstract layer is connected with the third distance interpolation characteristic layering propagation module. The first distance interpolation feature layered propagation module, the second distance interpolation feature layered propagation module and the third distance interpolation feature layered propagation module comprise a distance interpolation layer and a feature extraction layer which are sequentially connected.
In the technical solution provided in step 202, as shown in fig. 4, the first multi-scale point cloud collection abstract layer further includes a first sampling layer, a first grouping layer, and a first point network layer, and the channel spatial attention module includes a spatial attention module and a channel attention module; the step of extracting a first feature set from the physiological channel point cloud data through a first multi-scale point cloud set abstract layer of the physiological channel labeling model according to distance information among points in the physiological channel point cloud data to be labeled and density information in the point cloud data comprises the following steps: determining a first center point set from the physiological channel point cloud data by adopting a furthest point sampling mode through a first sampling layer; determining point set groups corresponding to all central points in the first central point set in the physiological channel point cloud data according to the first central point set by adopting a ball query mode through the first grouping layer; extracting the characteristics of each point set group through the first point network layer to obtain local area characteristics corresponding to each point set group; and processing the local region features through the channel space attention module to obtain a first feature set.
In some embodiments of the present application, the first sampling layer may determine a set of points from the set of point clouds input into the sampling layer as centroids of the local regions, that is, the central points in the first set of central points. The first grouping layer may then construct a local region set, i.e., a group of point sets, by finding neighboring points around the centroid selected by the sampling layer. A micro point network is used in the first point network layer to encode the point set groups into feature vectors. Assume that the input to the first multi-scale point-cloud abstraction layer is an n× (d+c) matrix consisting of N points with d-dimensional coordinates and C-dimensional point features. The result of the first netlayer output is an N 'x (d + C') matrix of N 'downsampled points with d-dimensional coordinates and a new C' dimensional feature vector in which the local context features are converged. Where N is the number of points, d is the dimension of the point coordinates, and C is the dimension of the point feature. N 'is the number of sub-sample points, d is the dimension of the sub-sample point coordinates, and C' is the new dimension of the feature vector summarizing the local context.
Specifically, the specific processing flows of the first sampling layer, the first grouping layer and the first point network layer are as follows:
For the first sampling layer, assume that the set of point clouds input to the first sampling layer is { x ] 1 ,x 2 ,x 3 ,......,x n The set of point clouds is the above described n× (d+c) matrix, after which the first sampling layer may use an iterative Furthest Point Sampling (FPS) method from the input point cloudSelecting a subset of points from the collectionAnd continuing to process the point subset, wherein the output result of the obtained sampling layer is a coordinate set of the central point of N' x d. Wherein (1)>Is->The furthest apart points may be calculated here by means of euclidean distance measures.
It should be noted that, the first center point set includes any point located at an end point of the physiological channel in the physiological channel point cloud data.
The advantage of using such a distance measure is that the structural information between samples can be better preserved compared to random sampling, where the number of centroids is the same. Random sampling may cause the relationships between samples to be broken, making it difficult to distinguish between different samples in subsequent tasks. The distance measurement can be used for sampling by considering the distance relation between samples, so that the structural information between the samples is better preserved, and meanwhile, deviation possibly caused by random sampling can be avoided. And unlike CNNs, which do not consider data distribution in vector space, the sampling strategy generates an acceptance field in a data dependent manner.
For the first packet layer, it is assumed that the inputs of the first packet layer are a point set of size n× (d+c) and a center point coordinate set of size N' ×d. The output is groups of sets of points of size N' x K x (d+c), where each group corresponds to a local area and K is the number of center point attachment neighbors. In addition, K varies from group to group, but the following PointNet Layer can convert a different number of points into a fixed length local area feature vector. Wherein, the grouping layer adopts a Ball query mode, and can find all points (the upper limit of K is set in the implementation) within a certain radius with the query point.
In addition, K nearest neighbor search KNN can be adopted when range query is carried out, and a fixed number of adjacent points can be found in the mode. However, compared with KNN, the local neighborhood of Ball query can guarantee a fixed region scale, so that the local region features have more spatial versatility.
For the point network layer, assuming that the input of the point network layer is a point set group of N ' x k× (d+c), the output data of the point network layer is N ' x k× (d+c '), where the local area corresponding to the point set group in each output is abstractly represented by the local features of its center and the neighborhood of the encoding center.
Specifically, in the first point mesh layer, the coordinates of the points in each point set group are first translated into a local coordinate system relative to the centroid point: for i=1, 2, & gt.Wherein->Is the coordinates of the centroid. In addition, pointNet can be used as a basic building block for local pattern learning, and the relation between points in a local area can be captured by using relative coordinates and point characteristics. Wherein PointNet is expressed as: />(x 1 ,x 2 ,......,x n ) Is a disordered point set, x 1 ∈R d . Gamma and h are multi-layer perceptron networks, h is spatial encoding. The function can be expressed as f: χ→r, that is, the function maps a set of points to a vector and approximates an arbitrary continuous function.
In some embodiments of the present application, since one point cloud set typically has non-uniform density point sets in different areas, the extracted features are very rich in sampling in dense areas, but rich information cannot be extracted in low density areas, and dense sampling in areas where the point set density is low may occur, and there may be uncertainty in the boundaries of the different density point sets. The characteristics learned in different areas are promoted, and great challenges are brought to the characteristic learning of the point set. In this case, it is necessary to learn feature information of different scales in different ranges, and pay attention to boundary information of different density point sets.
In order to avoid the influence of the uneven distribution of the points in the point cloud set on the feature extraction process, as shown in fig. 5, multiple groups of sampling scales can be determined according to the density information of the point cloud data of the physiological channel, and the sampling radius and the sampling point number in each group of sampling scales are different. R in fig. 5 represents a sampling radius, and nsample represents the number of sampling points corresponding to the sampling radius. It should be noted that the specific values of the sampling radius and the sampling point number shown in fig. 5 are only illustrative, and do not represent that the sampling radius and the sampling point number in fig. 5 must be used for sampling in the present application.
Specifically, when sampling is performed, different number of point sets can be selected in the increasing radius to learn detail information, meanwhile, the boundary area of the point sets is focused through the channel space attention module, and the feature weight of the boundary area is improved, so that the extracted features are richer.
In some embodiments of the present application, the channel space attention module is as shown in fig. 6. As can be seen from fig. 6, the step of the channel space attention module processing the local area features by the channel space attention module to obtain the first feature set includes: processing each local area characteristic by adopting a channel attention module in the channel space attention module to obtain a channel attention characteristic diagram corresponding to each local area characteristic; superposing the local area features and the channel attention feature map to obtain a first processing result, wherein the first processing result is a feature set with the feature weights of the local area features adjusted in the channel dimension according to the channel attention feature map; processing the first processing result by adopting a spatial attention module to obtain a spatial attention characteristic diagram corresponding to the first processing result; and carrying out superposition processing on the spatial attention feature map and the first processing result to obtain first point cloud features contained in the first feature set, wherein the first point cloud features are feature sets with feature weights of the corresponding first processing result adjusted according to the spatial attention feature map. And the first point cloud feature is a feature in which weights are adjusted in both the channel dimension and the space dimension.
It should be noted that after the feature weights of the local region features are adjusted, the weights corresponding to the boundary features between the segments of the physiological channel model are increased.
Step 204, extracting a second feature set from the first feature point set according to the distance information and the density information of each point in the first feature point set through a second multi-scale point cloud set abstract layer of the physiological channel labeling model, wherein the second feature set comprises the second feature point set and second point cloud features, and the second multi-scale point cloud set abstract layer comprises a spatial attention module;
in the technical solution provided in step 204, as shown in fig. 7, the second multi-scale point cloud collection abstract layer includes a second sampling layer, a second packet layer, a second point network layer, and a spatial attention module; the step of extracting the second feature set from the first feature point set through the second multi-scale point cloud set abstract layer of the physiological channel labeling model according to distance information among points in the physiological channel point cloud data to be labeled and density information in the point cloud data comprises the following steps: determining a second center point set from the first characteristic point set by adopting a furthest point sampling mode through a second sampling layer; determining point set groups corresponding to all center points in the second center point set in the first characteristic point set according to the second center point set by adopting a ball query mode through the second grouping layer; extracting the characteristics of each point set group through the second point network layer to obtain local area characteristics corresponding to each point set group; and processing the local region features through the spatial attention module to obtain a second feature set.
Specifically, since the workflows of the second sampling layer, the second grouping layer, and the second point network layer are identical to those of the first sampling layer, the first grouping layer, and the first point network layer, and the workflow of the spatial attention module in the second multi-scale point cloud collection abstraction layer is identical to that of the spatial attention module in the first multi-scale point cloud collection abstraction layer, the description thereof will not be repeated here.
It should be noted that, as shown in fig. 8, the sampling scale set in the second multi-scale point cloud collection feature layer is different from the sampling scale corresponding to the first multi-scale point cloud collection feature layer. Likewise, the particular sample scale presented in FIG. 8 is also for illustration only and is not representative of the sample scale that must be employed in FIG. 8.
Different sampling scales are set in the second multiscale point cloud collection characteristic layer, and the fact that the deeper the network is, the larger the receptive field is, the wider the multiscale information is extracted, the larger the range of image characteristics is captured, and therefore more multiscale information is extracted. As depth increases, networks may learn increasingly more abstract and complex features that may contain information from different scales, thus setting larger query radii to extract broader multi-scale information.
In addition, as in deeper networks, features are more abstract, and in more abstract features, there are very rich spatial relationships between points or point sets, and these spatial structural relationships have a great influence on the accuracy of labeling results. At this time, the weight distribution is performed according to the spatial information collected by the spatial attention module, which is more favorable for improving the network performance, so that only the spatial attention module is used for focusing on the spatial distribution information in the second multi-scale point cloud collection abstract layer, thereby increasing the weight of the boundary information. As an alternative, an attention module may also be added in the second multi-scale point cloud collection abstraction layer.
Step 206, extracting a third feature set from the second feature point set according to the distance information of each point in the second feature point set through the single-scale point cloud set abstraction layer of the physiological channel labeling model, wherein the third feature set comprises the third feature point set and third point cloud features;
in the technical solution provided in step 206, as shown in fig. 9, the single-scale point cloud collection abstraction layer includes a third sampling layer, a third grouping layer and a third point network layer; the step of extracting a third feature set from the second feature point set according to the distance information of each point in the second feature point set through the single-scale point cloud set abstract layer of the physiological channel labeling model comprises the following steps: determining a third center point set from the first characteristic point set by a third sampling layer in a most distant point sampling mode; determining point set groups corresponding to all center points in the third center point set in the first characteristic point set according to the center point set by adopting a ball query mode through the third grouping layer; and extracting the characteristics of each point set group through a third point network layer to obtain local area characteristics corresponding to each point set group, wherein the local area characteristics corresponding to each point set group are a third characteristic set.
Specifically, the working flows of the third sampling layer, the third grouping layer and the third point network layer are identical to the working flows of the first sampling layer, the first grouping layer and the first point network layer in the first multi-scale point cloud collection abstract layer, so that the detailed description is omitted.
And step 208, sequentially processing the first feature set, the second feature set and the third feature set through a plurality of feature merging layers of the physiological channel labeling model to obtain labeling results corresponding to the physiological channel point cloud data, wherein the labeling results comprise label information corresponding to each point in the physiological channel point cloud data.
In the technical solution provided in step 208, the step of splicing the first feature set, the second feature set and the third feature set through the feature merging layer of the physiological channel labeling model to obtain a labeling result includes: upsampling the third feature set to obtain a first upsampling result; upsampling the second feature set and the first upsampling result to obtain a second upsampling processing result; and upsampling the first feature set and the second upsampling result to obtain a third upsampling processing result, and obtaining a labeling result based on the third upsampling processing result.
Specifically, in order to fully utilize the features collected in each point cloud set abstract layer, a mode of transmitting the point cloud features collected by each point cloud set abstract layer to the input point cloud set can be adopted to enable all points to obtain corresponding category information.
As an alternative implementationIn this way, a hierarchical propagation strategy based on distance interpolation and jump connection may be employed. Specifically, in the distance interpolation layer of each distance interpolation feature layered propagation module, the point features can be extracted from N l X (d+C) points (i.e. point cloud features collected by the respective point cloud collection abstraction layer) are directed to N l-1 Propagation of point features by points (i.e., point sets input into corresponding point-set abstraction layers), where N l-1 And N l (wherein N l ≤N l-1 ) Is the size of the point set input into the point-cloud-set abstraction layer/and the point set output from the point-cloud-set abstraction layer. By combining in the input point cloud with the output point cloud characteristics N l The corresponding point at the coordinates of the midpoint interpolates the point feature value f to achieve feature propagation to N l-1 A point.
In interpolation, a reciprocal distance weighted average based on k-nearest neighbors may be used. Then N is added l-1 Interpolation features on the points are connected with cross-level jump chain point features of the set abstraction layer. The characteristics of the connection are then passed through the "Unit PointNet". The delivery process is similar to one-to-one convolution in CNN, with several shared full-join and ReLU layers applied to update the feature vector for each point. This process is repeated until the feature is propagated to the original set of points. The specific calculation formula of the interpolation process is as follows:
Wherein,
in the above formula, f is an interpolated eigenvalue, d (x, x i ) Is the point x, x i P is d (x, x i ) Index, k is the number of neighborhood points, w i Is x, x i The reciprocal of the distance, j is the index number of the point cloud, C is an arbitrary constant, f i (j) Representing the value of the j nearest neighbor. In the above formula, P is typically 2 and k is typically 3.
The above point x i And representing a point in the input point cloud, which is consistent with the point coordinates in the output point cloud characteristics, and replacing the characteristics of the point with the characteristics of the corresponding point in the output point cloud characteristics. x represents the input point cloud set except x i Any point beyond.
In addition, in each distance interpolation feature layered propagation module, the output result of the previous distance interpolation feature layered propagation module is fused with the feature obtained after the distance interpolation is carried out, and then the feature is input into a feature extraction layer. If the last distance interpolation feature layering propagation module is not available, the interpolation result is directly input into the feature extraction layer.
Specifically, in the case where the physiological channel is a bronchus, the segmentation result of the airway tree model corresponding to the bronchus is shown in fig. 10, and the tag information may include lung segment names such as RB1, RB2, RB3, and RB 4. The labeling result determined by the physiological channel labeling method provided by the embodiment of the application is shown in fig. 11, wherein fig. 11 shows labeling results and corresponding labels under various viewing angles. The left side of each group of images is a labeling result output by adopting a physiological channel labeling model, and the right side of each group of images is a preset sample label.
As can be seen from comparing fig. 10 and fig. 11, the accuracy of labeling and segmentation results can be effectively improved by adopting the physiological channel labeling method provided by the application.
The embodiment of the application also provides a model training method shown in fig. 12 for training the physiological channel labeling model, which specifically comprises the following steps:
step 120, acquiring a 3D physiological channel model dataset with a sample tag, and converting the acquired 3D physiological channel model dataset into corresponding point cloud data;
step 122, preprocessing the obtained point cloud data;
the preprocessing operation may be implemented by ToTensor, rotate, scale, translate, jitter, etc.
Step 124, inputting the preprocessed point cloud data into a physiological channel labeling model to obtain a prediction labeling result for the preprocessed point cloud data, which is output by the physiological channel labeling model;
step 126, adjusting the value of the preset parameter contained in the physiological channel labeling model according to the difference information between the prediction labeling result and the sample label;
step 128, determining whether the value of the preset parameter contained in the physiological channel labeling model obtained after training is better than the value of the preset parameter contained in the physiological channel labeling model after the previous training, if so, jumping to step 1210, otherwise jumping to step 1212;
Step 1210, saving the value of the preset parameter of the labeling model of the optimal physiological channel;
step 1212, determining whether the model update reaches the preset training number, if so, ending the training, otherwise, jumping to step 122.
In summary, by adopting the physiological channel labeling method provided by the embodiment of the application, a labeling result with high accuracy can be obtained without complex network structure design, complex manual feature manufacturing and higher hardware equipment, and the labeling speed can be improved. In the embodiment of the application, the PointNet++ is used as a basic network to improve the system, different attention mechanisms are adopted to strengthen the attention of the model to the boundary of each branch, the distinguishing capability of each branch boundary is improved, the problem of wrong label separation is solved, and different strategies such as data enhancement, learning rate optimization, training time improvement and the like are adopted to improve the accuracy and speed. Specifically, taking the physiological channel model as an airway tree model as an example, the physiological channel model labeling method provided by the embodiment of the application can improve the automatic labeling classification accuracy from 65% to 75% in 32 different sections of the airway tree, and the time of automatic labeling of a single airway tree is improved by 2 times compared with that of a related method.
Fig. 13 is a block diagram of a physiological channel labeling apparatus according to an embodiment of the present disclosure, where the apparatus includes:
the first processing module 130 is configured to extract, by using a first multi-scale point cloud set abstraction layer of the physiological channel labeling model, a first feature set from the physiological channel point cloud data according to distance information between points in the physiological channel point cloud data to be labeled and density information in the point cloud data, where the first feature set includes a first feature point set and corresponding first point cloud features, and the first multi-scale point cloud set abstraction layer includes a channel space attention module;
the second processing module 132 is configured to extract, from the first feature point set, a second feature set according to distance information and density information of each point in the first feature point set by using a second multi-scale point cloud set abstraction layer of the physiological channel labeling model, where the second multi-scale point cloud set abstraction layer includes a spatial attention module;
a third processing module 134, configured to extract a third feature set from the second feature point set according to the distance information of each point in the second feature point set through the single-scale point cloud set abstraction layer of the physiological channel labeling model, where the third feature set includes a third feature point set and a third point cloud feature;
The fourth processing module 136 is configured to sequentially process the first feature set, the second feature set, and the third feature set through a plurality of feature merging layers of the physiological channel labeling model to obtain a labeling result corresponding to the physiological channel point cloud data, where the labeling result includes label information corresponding to each point in the physiological channel point cloud data.
In some embodiments of the present application, the physiological channel labeling model includes a feature merging layer, wherein the feature merging layer includes a first distance interpolation feature hierarchical propagation module, a second distance interpolation feature hierarchical propagation module, and a third distance interpolation feature hierarchical propagation module, wherein the first distance interpolation feature hierarchical propagation module is connected with the second distance interpolation feature hierarchical propagation module, and the second distance interpolation feature hierarchical propagation module is connected with the third distance interpolation feature hierarchical propagation module; the first multi-scale point cloud set abstract layer is connected with the third distance interpolation characteristic layering propagation module, the second multi-scale point cloud set abstract layer is connected with the second distance interpolation characteristic layering propagation module, and the single-scale point cloud set abstract layer is connected with the third distance interpolation characteristic layering propagation module.
In some embodiments of the present application, the first distance interpolation feature hierarchical propagation module, the second distance interpolation feature hierarchical propagation module, and the third distance interpolation feature hierarchical propagation module each include a distance interpolation layer and a feature extraction layer that are sequentially connected.
In some embodiments of the present application, the first multi-scale point cloud collection abstraction layer further includes a first sampling layer, a first grouping layer, and a first point mesh layer, the channel spatial attention module includes a spatial attention module and a channel attention module; the step of extracting, by the first processing module 130, the first feature set from the physiological channel point cloud data according to the distance information between the points in the physiological channel point cloud data to be marked and the density information in the point cloud data through the first multi-scale point cloud set abstraction layer of the physiological channel labeling model includes: determining a first center point set from the physiological channel point cloud data by adopting a furthest point sampling mode through a first sampling layer; determining point set groups corresponding to all central points in the first central point set in the physiological channel point cloud data according to the first central point set by adopting a ball query mode through the first grouping layer; extracting the characteristics of each point set group through the first point network layer to obtain local area characteristics corresponding to each point set group; and processing the local region features through the channel space attention module to obtain a first feature set.
In some embodiments of the present application, the first set of center points includes any point in the physiological channel point cloud data that is located at an endpoint of the physiological channel.
In some embodiments of the present application, the step of the first processing module 130 processing the local area feature by the channel space attention module to obtain the first feature set includes: processing each local area characteristic by adopting a channel attention module in the channel space attention module to obtain a channel attention characteristic diagram corresponding to each local area characteristic; superposing the local area features and the channel attention feature map to obtain a first processing result, wherein the first processing result is a feature set with the feature weights of the local area features adjusted in the channel dimension according to the channel attention feature map; processing the first processing result by adopting a spatial attention module to obtain a spatial attention characteristic diagram corresponding to the first processing result; and carrying out superposition processing on the spatial attention feature map and the first processing result to obtain a first point cloud feature contained in the first feature set, wherein the first point cloud feature is a feature set with feature weights of the first processing result adjusted in a spatial dimension according to the spatial attention feature map.
In some embodiments of the present application, the second multi-scale point cloud collection abstraction layer includes a second sampling layer, a second packet layer, a second point mesh layer, and a spatial attention module; the step of extracting, by the second processing module 132, the second feature set from the first feature point set according to the distance information between the points in the first feature point set and the density information in the point cloud data through the second multi-scale point cloud set abstraction layer of the physiological channel labeling model includes: determining a second center point set from the first characteristic point set by adopting a furthest point sampling mode through a second sampling layer; determining point set groups corresponding to all center points in the second center point set in the first characteristic point set according to the second center point set by adopting a ball query mode through the second grouping layer; extracting the characteristics of each point set group through the second point network layer to obtain local area characteristics corresponding to each point set group; and processing the local region features through the spatial attention module to obtain a second feature set.
In some embodiments of the present application, the single-scale point cloud set abstraction layer includes a third sampling layer, a third grouping layer, and a third point mesh layer; the step of extracting the third feature set from the second feature point set by the third processing module 134 through the single-scale point cloud set abstraction layer of the physiological channel labeling model according to the distance information of each point in the second feature point set includes: determining a third center point set from the first characteristic point set by a third sampling layer in a most distant point sampling mode; determining point set groups corresponding to all center points in the third center point set in the first characteristic point set according to the center point set by adopting a ball query mode through the third grouping layer; and extracting the characteristics of each point set group through a third point network layer to obtain local area characteristics corresponding to each point set group, wherein the local area characteristics corresponding to each point set group are a third characteristic set.
In some embodiments of the present application, the fourth processing module 136 concatenates the first feature set, the second feature set, and the third feature set through the feature merging layer of the physiological channel labeling model, and the step of obtaining the labeling result includes: upsampling the third feature set to obtain a first upsampling result; upsampling the second feature set and the first upsampling result to obtain a second upsampling processing result; and upsampling the first feature set and the second upsampling result to obtain a third upsampling processing result, and obtaining a labeling result based on the third upsampling processing result.
Referring to fig. 14, an embodiment of the present application provides a schematic structural diagram of an electronic device. As shown in fig. 14, the electronic device includes: a memory 1401, and a processor 1402.
The memory 1401 stores an executable computer program 1403 therein. The processor 1402 coupled to the memory 1401 invokes the executable computer program 1403 stored in the memory to perform the physiological channel labeling method provided by the above embodiments.
By way of example, the computer program 1403 may be partitioned into one or more modules/units that are stored in the memory 1401 and executed by the processor 1402 to complete the present application. The one or more modules/units may include the various modules in the physiological channel labeling apparatus in the above embodiments, such as: a first processing module 130, a second processing module 132, a third processing module 134, and a fourth processing module 136.
Further, the apparatus further comprises: at least one input device and at least one output device.
The processor 1402, memory 1401, input devices, and output devices described above may be connected by buses.
The input device may specifically be a camera, a touch panel, a physical button, a mouse, or the like. The output device may in particular be a display screen.
Further, the apparatus may also include more components than illustrated, or may combine certain components, or may be different components, such as network access devices, sensors, etc.
The processor 1402 may be a central processing unit (Central Processing Unit, CPU), but may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
Memory 1401 may be, for example, a hard drive memory, a non-volatile memory (e.g., flash memory or other electronically programmable limited delete memory used to form a solid state drive, etc.), a volatile memory (e.g., static or dynamic random access memory, etc.), and the like, as embodiments of the present application are not limited. Specifically, the memory 1401 may be an internal storage unit of the electronic device, for example: the hard disk or the memory of the electronic device. The memory 1401 may be an external storage device of the electronic apparatus, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card) or the like, which are provided in the electronic apparatus. Further, the memory 1401 may also include both an internal storage unit and an external storage device of the electronic apparatus. The memory 1401 is used to store a computer program and other programs and data required for the terminal. The memory 1401 may also be used to temporarily store data that has been output or is to be output.
Further, the embodiments of the present application also provide a non-volatile computer readable storage medium, which may be provided in the electronic device in the above embodiments, and the computer readable storage medium may be the memory 1401 in the embodiment shown in fig. 14. The computer readable storage medium stores a computer program which, when executed by a processor, implements the physiological channel labeling method described in the foregoing embodiments. Further, the computer-readable medium may be any medium capable of storing a program code, such as a usb (universal serial bus), a removable hard disk, a Read-Only Memory (ROM), a RAM, a magnetic disk, or an optical disk.
In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative, and for example, the division of the modules is merely a logical function division, and there may be additional divisions when actually implemented, for example, multiple modules or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or modules, which may be in electrical, mechanical, or other forms.
The modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical modules, i.e., may be located in one place, or may be distributed over a plurality of network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist alone physically, or two or more modules may be integrated into one module. The integrated modules may be implemented in hardware or in software functional modules.
The integrated modules, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application, or a portion or all or part of the technical solution contributing to the related art, may be embodied in the form of a software product, which is stored in a readable storage medium, and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned readable storage medium includes: a usb disk, a removable hard disk, a ROM, a RAM, a magnetic disk, or an optical disk, etc.
It should be noted that, for the sake of simplicity of description, the foregoing method embodiments are all expressed as a series of combinations of actions, but it should be understood by those skilled in the art that the present application is not limited by the order of actions described, as some steps may be performed in other order or simultaneously in accordance with the present application. Further, those skilled in the art will also appreciate that the embodiments described in the specification are all preferred embodiments, and that the acts and modules referred to are not necessarily all necessary for the present application.
In the foregoing embodiments, the descriptions of the embodiments are emphasized, and for parts of one embodiment that are not described in detail, reference may be made to the related descriptions of other embodiments.
In the description of the present specification, reference to the description of the terms "one embodiment," "some embodiments," "examples," "particular examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present application. In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, the different embodiments or examples described in this specification and the features of the different embodiments or examples may be combined and combined by those skilled in the art without contradiction.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.
Claims (13)
1. A method for labeling a physiological channel, comprising:
extracting a first feature set from physiological channel point cloud data according to distance information among points in the physiological channel point cloud data to be marked and density information in the point cloud data through a first multi-scale point cloud set abstraction layer of a physiological channel labeling model, wherein the first feature set comprises a first feature point set and corresponding first point cloud features, and a channel space attention module is included in the first multi-scale point cloud set abstraction layer;
extracting a second feature set from the first feature point set according to the distance information and the density information of each point in the first feature point set through a second multi-scale point cloud set abstract layer of the physiological channel labeling model, wherein the second feature set comprises a second feature point set and a second point cloud feature, and the second multi-scale point cloud set abstract layer comprises a spatial attention module;
Extracting a third feature set from the second feature point set according to the distance information of each point in the second feature point set through a single-scale point cloud set abstraction layer of the physiological channel labeling model, wherein the third feature set comprises a third feature point set and third point cloud features;
and sequentially processing the first feature set, the second feature set and the third feature set through a plurality of feature merging layers of the physiological channel labeling model to obtain labeling results corresponding to the physiological channel point cloud data, wherein the labeling results comprise label information corresponding to each point in the physiological channel point cloud data.
2. The physiological channel labeling method of claim 1, wherein the first multi-scale point cloud collection abstraction layer further comprises a first sampling layer, a first grouping layer, and a first point mesh layer, the channel spatial attention module comprising a spatial attention module and a channel attention module; the step of extracting a first feature set from the physiological channel point cloud data by the first multi-scale point cloud set abstract layer of the physiological channel labeling model according to distance information among points in the physiological channel point cloud data to be labeled and density information in the point cloud data comprises the following steps:
Determining a first center point set from the physiological channel point cloud data by adopting a furthest point sampling mode through the first sampling layer;
determining point set groups corresponding to all central points in a first central point set in the physiological channel point cloud data according to the first central point set by adopting a ball query mode through the first grouping layer;
extracting the characteristics of each point set group through the first point network layer to obtain local area characteristics corresponding to each point set group;
and processing the local region features through the channel space attention module to obtain the first feature set.
3. The method of claim 2, wherein the first set of center points includes any point in the physiological channel point cloud data that is located at an endpoint of a physiological channel.
4. The method for labeling a physiological channel according to claim 2, wherein said step of processing said local region features by said channel spatial attention module to obtain said first feature set comprises:
processing each local area characteristic by adopting a channel attention module in the channel space attention module to obtain a channel attention characteristic diagram corresponding to each local area characteristic;
Superposing the local area features and the channel attention feature map to obtain a first processing result, wherein the first processing result is a feature set with feature weights of the local area features adjusted in a channel dimension according to the channel attention feature map;
processing the first processing result by adopting a spatial attention module to obtain a spatial attention characteristic diagram corresponding to the first processing result;
and superposing the spatial attention feature map and the first processing result to obtain the first point cloud feature contained in the first feature set, wherein the first point cloud feature is a feature set with feature weights of the first processing result adjusted in a spatial dimension according to the spatial attention feature map.
5. The physiological channel labeling method of claim 1, wherein the second multi-scale point cloud collection abstraction layer comprises a second sampling layer, a second grouping layer, a second point mesh layer, and a spatial attention module; the step of extracting the second feature set from the first feature point set according to the distance information between the points in the first feature point set and the density information in the point cloud data by the second multi-scale point cloud set abstract layer of the physiological channel labeling model comprises the following steps:
Determining a second center point set from the first characteristic point set by the second sampling layer in a furthest point sampling mode;
determining point set groups corresponding to all center points in the second center point set in the first characteristic point set according to the second center point set by adopting a ball query mode through the second grouping layer;
extracting the characteristics of each point set group through the second point network layer to obtain local area characteristics corresponding to each point set group;
and processing the local region features through the spatial attention module to obtain the second feature set.
6. The physiological channel labeling method of claim 1, wherein the single-scale point cloud collection abstraction layer comprises a third sampling layer, a third grouping layer, and a third point mesh layer; the step of extracting a third feature set from the second feature point set according to the distance information of each point in the second feature point set by the single-scale point cloud set abstraction layer of the physiological channel labeling model comprises the following steps:
determining a third center point set from the first characteristic point set by the third sampling layer in a furthest point sampling mode;
Determining point set groups corresponding to all central points in a third central point set in the first characteristic point set according to the central point set by adopting a ball query mode through the third grouping layer;
and extracting the characteristics of each point set group through the third point network layer to obtain local area characteristics corresponding to each point set group, wherein the local area characteristics corresponding to each point set group are the third characteristic set.
7. The method for labeling a physiological channel according to claim 1, wherein the step of concatenating the first feature set, the second feature set and the third feature set through the feature merging layer of the physiological channel labeling model to obtain a labeling result includes:
upsampling the third feature set to obtain a first upsampling result;
upsampling the second feature set and the first upsampling result to obtain a second upsampling processing result;
and upsampling the first feature set and the second upsampling result to obtain a third upsampling processing result, and obtaining the labeling result based on the third upsampling processing result.
8. The method of claim 1, wherein the physiological channel labeling model comprises a feature merge layer, wherein,
the feature merging layer comprises a first distance interpolation feature layering propagation module, a second distance interpolation feature layering propagation module and a third distance interpolation feature layering propagation module, wherein the first distance interpolation feature layering propagation module is connected with the second distance interpolation feature layering propagation module, and the second distance interpolation feature layering propagation module is connected with the third distance interpolation feature layering propagation module;
the first multi-scale point cloud set abstract layer is connected with the third distance interpolation characteristic layered propagation module, the second multi-scale point cloud set abstract layer is connected with the second distance interpolation characteristic layered propagation module, and the single-scale point cloud set abstract layer is connected with the third distance interpolation characteristic layered propagation module.
9. The physiological channel labeling method according to claim 8, wherein the first distance interpolation feature layered propagation module, the second distance interpolation feature layered propagation module and the third distance interpolation feature layered propagation module each comprise a distance interpolation layer and a feature extraction layer which are sequentially connected.
10. The physiological channel labeling system comprises an image acquisition module, a processor and a display module, wherein,
the image acquisition module is used for acquiring a physiological channel image of a physiological channel and generating physiological channel point cloud data to be marked according to the physiological channel image;
the processor is used for extracting a first feature set from the physiological channel point cloud data according to distance information among points in the physiological channel point cloud data to be marked and density information in the point cloud data through a first multi-scale point cloud set abstraction layer of the physiological channel labeling model, wherein the first feature set comprises a first feature point set and corresponding first point cloud features, and the first multi-scale point cloud set abstraction layer comprises a channel space attention module; extracting a second feature set from the first feature point set according to the distance information and the density information of each point in the first feature point set through a second multi-scale point cloud set abstract layer of the physiological channel labeling model, wherein the second feature set comprises a second feature point set and a second point cloud feature, and the second multi-scale point cloud set abstract layer comprises a spatial attention module; extracting a third feature set from the second feature point set according to the distance information of each point in the second feature point set through a single-scale point cloud set abstraction layer of the physiological channel labeling model, wherein the third feature set comprises a third feature point set and third point cloud features; sequentially processing the first feature set, the second feature set and the third feature set through a plurality of feature merging layers of the physiological channel labeling model to obtain labeling results corresponding to the physiological channel point cloud data, wherein the labeling results comprise label information corresponding to each point in the physiological channel point cloud data;
The display module is used for displaying the labeling result.
11. A physiological channel labeling apparatus, comprising:
the first processing module is used for extracting a first feature set from the physiological channel point cloud data according to distance information among points in the physiological channel point cloud data to be marked and density information in the point cloud data through a first multi-scale point cloud set abstract layer of the physiological channel labeling model, wherein the first feature set comprises a first feature point set and corresponding first point cloud features, and the first multi-scale point cloud set abstract layer comprises a channel space attention module;
the second processing module is used for extracting a second feature set from the first feature point set according to the distance information and the density information of each point in the first feature point set through a second multi-scale point cloud set abstract layer of the physiological channel labeling model, wherein the second feature set comprises a second feature point set and second point cloud features, and the second multi-scale point cloud set abstract layer comprises a spatial attention module;
the third processing module is used for extracting a third feature set from the second feature point set according to the distance information of each point in the second feature point set through the single-scale point cloud set abstraction layer of the physiological channel labeling model, wherein the third feature set comprises a third feature point set and third point cloud features;
And the fourth processing module is used for sequentially processing the first feature set, the second feature set and the third feature set through a plurality of feature merging layers of the physiological channel labeling model to obtain labeling results corresponding to the physiological channel point cloud data, wherein the labeling results comprise label information corresponding to each point in the physiological channel point cloud data.
12. A non-volatile storage medium storing a computer program which, when executed by a processor, implements the method of any one of the preceding claims 1 to 9.
13. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any one of claims 1 to 9 when executing the computer program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311869032.0A CN117830261A (en) | 2023-12-29 | 2023-12-29 | Physiological channel labeling method, system, device and nonvolatile storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311869032.0A CN117830261A (en) | 2023-12-29 | 2023-12-29 | Physiological channel labeling method, system, device and nonvolatile storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117830261A true CN117830261A (en) | 2024-04-05 |
Family
ID=90518702
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311869032.0A Pending CN117830261A (en) | 2023-12-29 | 2023-12-29 | Physiological channel labeling method, system, device and nonvolatile storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117830261A (en) |
-
2023
- 2023-12-29 CN CN202311869032.0A patent/CN117830261A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Zhang et al. | A review of deep learning-based semantic segmentation for point cloud | |
| Lian et al. | Road extraction methods in high-resolution remote sensing images: A comprehensive review | |
| US10984659B2 (en) | Vehicle parking availability map systems and methods | |
| CN107506761B (en) | Brain image segmentation method and system based on significance learning convolutional neural network | |
| CN110059697B (en) | Automatic lung nodule segmentation method based on deep learning | |
| KR102177568B1 (en) | Method for semi supervised reinforcement learning using data with label and data without label together and apparatus using the same | |
| Zhang et al. | Hybrid region merging method for segmentation of high-resolution remote sensing images | |
| CN109165540B (en) | Pedestrian searching method and device based on prior candidate box selection strategy | |
| US11640714B2 (en) | Video panoptic segmentation | |
| US20130070997A1 (en) | Systems, methods, and media for on-line boosting of a classifier | |
| Mosinska et al. | Joint segmentation and path classification of curvilinear structures | |
| CN111754472A (en) | Pulmonary nodule detection method and system | |
| Xu et al. | Consistency-regularized region-growing network for semantic segmentation of urban scenes with point-level annotations | |
| Li et al. | Hierarchical semantic parsing for object pose estimation in densely cluttered scenes | |
| JP2019185787A (en) | Remote determination of containers in geographical region | |
| CN112364881B (en) | Advanced sampling consistency image matching method | |
| Dong et al. | A novel loop closure detection method using line features | |
| Zhou et al. | Attention transfer network for nature image matting | |
| Wang et al. | Small vehicle classification in the wild using generative adversarial network | |
| CN115147644A (en) | Method, system, device and storage medium for training and describing image description model | |
| Xie et al. | Automatic neuron tracing in volumetric microscopy images with anisotropic path searching | |
| Kalyani et al. | Deep learning-based detection and classification of adenocarcinoma cell nuclei | |
| Shekar et al. | An efficient stacked ensemble model for the detection of COVID-19 and skin cancer using fused feature of transfer learning and handcrafted methods | |
| CN117830261A (en) | Physiological channel labeling method, system, device and nonvolatile storage medium | |
| WO2023274599A1 (en) | Methods and systems for automated follow-up reading of medical image data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |