CN115082909A - Lung lesion identification method and system - Google Patents
Lung lesion identification method and system Download PDFInfo
- Publication number
- CN115082909A CN115082909A CN202111296412.0A CN202111296412A CN115082909A CN 115082909 A CN115082909 A CN 115082909A CN 202111296412 A CN202111296412 A CN 202111296412A CN 115082909 A CN115082909 A CN 115082909A
- Authority
- CN
- China
- Prior art keywords
- lesion
- network
- features
- lung
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000003902 lesion Effects 0.000 title claims abstract description 118
- 210000004072 lung Anatomy 0.000 title claims abstract description 55
- 238000000034 method Methods 0.000 title claims abstract description 24
- 230000011218 segmentation Effects 0.000 claims abstract description 30
- 238000001514 detection method Methods 0.000 claims abstract description 26
- 201000009267 bronchiectasis Diseases 0.000 claims abstract description 10
- 239000005338 frosted glass Substances 0.000 claims abstract description 6
- 238000012549 training Methods 0.000 claims description 34
- 238000000605 extraction Methods 0.000 claims description 19
- 238000013527 convolutional neural network Methods 0.000 claims description 12
- 239000005337 ground glass Substances 0.000 claims description 7
- 238000005070 sampling Methods 0.000 claims description 6
- 230000004927 fusion Effects 0.000 claims description 5
- 230000003993 interaction Effects 0.000 claims description 5
- 238000010276 construction Methods 0.000 claims description 3
- 230000007883 bronchodilation Effects 0.000 claims 2
- 238000013528 artificial neural network Methods 0.000 abstract description 2
- 238000012805 post-processing Methods 0.000 abstract description 2
- 230000000007 visual effect Effects 0.000 abstract description 2
- 230000001629 suppression Effects 0.000 abstract 1
- 230000006870 function Effects 0.000 description 10
- 238000013135 deep learning Methods 0.000 description 5
- 230000000694 effects Effects 0.000 description 5
- 238000010606 normalization Methods 0.000 description 3
- 210000000621 bronchi Anatomy 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000002059 diagnostic imaging Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 230000005764 inhibitory process Effects 0.000 description 1
- 210000000056 organ Anatomy 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Abstract
The invention provides a lung lesion identification method and a lung lesion identification system. And detecting a frosted glass area and a bronchiectasis area in the lung medical image through a target detection and example segmentation algorithm so as to be used for identifying the lung lesion. The method uses a multi-scale and multi-depth convolution neural network structure to extract basic visual features of the lung medical image, and fuses features of different scales and depths through a feature pyramid network. The characteristics of the possibly target area are extracted through the lesion candidate area identification network, and useless background information is filtered. And then, the extracted target area features pass through a classification head network, a detection head network and a segmentation head network to obtain a detection segmentation result, and finally, a final prediction result is obtained through non-maximum value suppression post-processing. The whole network structure adopts an end-to-end multi-task model, so that the network can effectively learn the characteristic information of the medical image, and the robustness and generalization capability of the algorithm are enhanced.
Description
Technical Field
The invention belongs to the technical field of target classification in image processing, and particularly relates to a lung lesion identification method and system by combining a target detection example segmentation network structure of a medical image for identifying lung lesion areas.
Background
In recent years, deep learning has been in the medical field with slow burning. Due to the strong nonlinear modeling capability of the deep learning network and the characteristics of large information amount, rich characteristics and multiple modal types of the medical images, the deep learning network is more and more widely applied to the medical images.
The two-stage target Detection structure based on deep learning is originally published in NIPS 2015, and is used for detecting common classes on MS-COCO natural pictures, namely, simultaneously classifying and positioning human, vehicle, animal and other targets. In 2017, improvement is carried out on Kaiming He on ICCV based on Faster R-CNN, and a paper MASK R-CNN is published for example segmentation of predefined class targets in natural images, so that the optimal result at that time is obtained. The principle of the method is mainly that a flow of firstly detecting and then segmenting is adopted, each target instance is detected firstly, then each instance is segmented, and another important contribution is that experiments show that the addition of segmentation tasks is of great help to the improvement of detection precision.
With the continuous progress and development of deep learning, target detection and example segmentation algorithms also start to be gradually applied to various industries, such as industrial flaw detection, pedestrian detection and the like, but the application in the field of medical imaging has great potential to be mined.
Disclosure of Invention
The present invention is directed to solving the above problems of the prior art. A target detection and instance segmentation network structure method combined with medical image multitask learning is provided.
In order to overcome the defects in the prior art, the invention provides a lung lesion identification method, which comprises the following steps:
step 1, constructing a lesion identification model comprising a convolutional neural network, a characteristic pyramid network, a lesion candidate region extraction network and a joint identification network, wherein the joint identification network comprises a classification head network, a detection head network and a segmentation head network; acquiring a lung medical image as training data, wherein the training data are marked with lesion areas and lesion types;
step 2, extracting multi-scale image features of the training data through the convolutional neural network; the feature pyramid network performs up-sampling and summation on adjacent features in the image features, and fuses features of different scale depths to obtain enhanced features; inputting the enhanced features into the lesion candidate region extraction network to obtain all candidate region features containing lesions in the enhanced features;
and 3, inputting all the candidate region characteristics into the joint recognition network to obtain the lesion type and lesion region of the training data, constructing a loss function by combining the labeled lesion region and lesion type of the training data, iteratively training the lesion recognition model through the loss function until the loss function is converged or reaches a preset iteration number, and recognizing by using the current lesion recognition model to obtain the lesion region of the specified lung medical image.
The lung lesion identification method is characterized in that the lung medical image is a CT image.
The lung lesion identification method, wherein the step 2 comprises: and performing information interaction fusion on the features with different scales or depths in the image features through a multi-scale feature pyramid network.
The method for identifying the lung lesion, wherein the lesion category comprises frosted glass and bronchiectasis;
the step 3 comprises the following steps: and respectively executing corresponding classification, coordinate frame regression and semantic segmentation tasks by a classification head, a detection head and a segmentation head network in the combined recognition network to obtain recognition results of the ground glass and the bronchiectasis associated regions in the lung medical image.
The invention also provides a lung lesion identification system, which comprises:
the model construction module is used for constructing a lesion identification model comprising a convolutional neural network, a characteristic pyramid network, a lesion candidate region extraction network and a joint identification network, wherein the joint identification network comprises a classification head network, a detection head network and a segmentation head network; acquiring a lung medical image as training data, wherein the training data are marked with lesion areas and lesion types;
the image feature extraction module is used for extracting multi-scale image features of the training data through the convolutional neural network; the feature pyramid network performs up-sampling and summation on adjacent features in the image features, and fuses features of different scale depths to obtain enhanced features; inputting the enhanced features into the lesion candidate region extraction network to obtain all candidate region features containing lesions in the enhanced features;
and the lesion area identification module is used for inputting all the candidate area characteristics into the joint identification network to obtain the lesion type and lesion area of the training data, constructing a loss function by combining the labeled lesion area and lesion type of the training data, iteratively training the lesion identification model through the loss function until the loss function is converged or reaches a preset iteration number, and identifying and obtaining the lesion area of the specified lung medical image by using the current lesion identification model.
The lung lesion identification system is characterized in that the lung medical image is a CT image.
The lung lesion recognition system, wherein the image feature extraction module comprises: and performing information interaction fusion on the features with different scales or depths in the image features through a multi-scale feature pyramid network.
The lung lesion recognition system, wherein the lesion category comprises frosted glass and concomitant bronchiectasis;
the lesion area identification module includes: and respectively executing corresponding classification, coordinate frame regression and semantic segmentation tasks by a classification head, a detection head and a segmentation head network in the combined recognition network to obtain recognition results of the ground glass and the bronchiectasis associated regions in the lung medical image.
The invention also proposes a storage medium for storing a program for executing the method for identifying a lung lesion.
The invention also provides a client used for the lung lesion identification system.
According to the scheme, the invention has the advantages that:
the invention identifies lung lesions using medical images by target detection instance segmentation network structures. And extracting basic visual features of the lung medical image by using a multi-scale and multi-depth convolution neural network structure, and fusing the features of different scales and depths through a feature pyramid network. The characteristics of the possibly target area are extracted through the lesion candidate area identification network, and useless background information is filtered. And then, the extracted target area features are subjected to classification head network, detection network and segmentation network to obtain a detection segmentation result, and finally, a final prediction result is obtained through non-maximum value inhibition post-processing. The whole network structure adopts an end-to-end multi-task model, so that the network can effectively learn the characteristic information of the medical image, and the robustness and generalization capability of the algorithm are enhanced.
Drawings
FIG. 1 is a flow chart of a preferred embodiment method provided by the present invention;
fig. 2 is an overall network structure.
Detailed Description
In order to make the aforementioned features and effects of the present invention more comprehensible, embodiments accompanied with figures are described in detail below.
The core technology of the invention is that firstly, the characteristics of the multi-scale medical image are fused through the characteristic pyramid network, and the targets with different sizes are effectively detected. Meanwhile, the overall recognition effect is improved by adopting a classification, detection and segmentation three task combined learning mode. Therefore, the medical image features are effectively extracted by using the feature pyramid network structure, the final recognition effect is improved through multitask learning, and the method has a wide application prospect in the medical field.
The invention provides a lung lesion identification method combined with medical images, which comprises the following steps:
and S1, acquiring an original image of the medical image, and then carrying out corresponding normalization on the image by using a fixed mean value and a fixed variance.
And S2, inputting the medical image after the normalization processing into a basic network to extract a basic feature map.
And S3, inputting the basic feature map into the feature pyramid network to extract and fuse the multi-scale medical image feature map, wherein the feature map can be used for simultaneously identifying lesion targets with different sizes.
And S4, inputting the multi-scale features obtained in the last step into a lesion candidate region extraction network to obtain a plurality of features which can be lesions.
And S5, sending all lesion candidate features into a classification head network, a detection head network and a segmentation head network for corresponding classification, performing coordinate frame regression and semantic segmentation tasks, and adopting a joint learning mode to mutually improve the effect. The joint learning refers to that three networks are classified, detected and segmented, have respective corresponding loss functions, and three networks are optimized simultaneously during training.
In order to make the aforementioned features and effects of the present invention more comprehensible, embodiments accompanied with figures are described in detail below.
The technical scheme for solving the technical problems is as follows:
the process flow of the present invention is shown in FIG. 1.
The present invention relates to a feature attention network architecture such as that shown in figure 2.
The method comprises the following specific steps:
step S1: first, a lung medical image of a person to be tested is obtained. Because medical image pictures are more, the medical image pictures need to be screened, and the medical image pictures with obvious lung lesion characteristics are selected as a training set. Because the image images have different numerical value distribution ranges, the image images need to be normalized by adopting a fixed mean variance for better training the network, and then the image images are put into the network for training and testing.
Step S2: and transmitting the medical image after normalization to a basic feature extraction network to extract a basic feature map. Medical images are single-channel gray level images generally, and because the structures of organs are fixed and semantic information is not particularly rich, high-level semantic information and low-level features are very important, and ResNet-50 is used as a basic network structure to extract the semantic information of the images.
Step S3: the medical image basic feature map is transmitted to a feature pyramid network structure, feature information of different scales is effectively fused by sampling and summing feature maps of adjacent scales, and lesion areas of different sizes are conveniently identified at the same time.
Step S4: and (4) inputting the multi-scale medical image feature map obtained in the S3 into a lesion candidate region extraction network to obtain features of a plurality of lesion candidate regions, wherein the potential lesion regions can be preliminarily screened out in this step, and irrelevant backgrounds are filtered out.
Step S5: and inputting the medical image characteristic diagram of the potential lesion obtained in the step S4 into a classification head network, a detection network and a segmentation network to obtain recognition results of a ground glass region and a known bronchiectasis region. The classification is to distinguish whether the glass is ground or background or whether the bronchus is background, the specific positions of the ground glass and the bronchus are detected, and the division is to perform detailed pixel level distinguishing in the obtained specific positions.
The following are system examples corresponding to the above method examples, and this embodiment can be implemented in cooperation with the above embodiments. The related technical details mentioned in the above embodiments are still valid in this embodiment, and are not described herein again in order to reduce repetition. Accordingly, the related-art details mentioned in the present embodiment can also be applied to the above-described embodiments.
The invention also provides a system for identifying lung lesions, which comprises:
the model construction module is used for constructing a lesion identification model comprising a convolutional neural network, a characteristic pyramid network, a lesion candidate region extraction network and a joint identification network, wherein the joint identification network comprises a classification head network, a detection head network and a segmentation head network; acquiring a lung medical image as training data, wherein the training data are marked with lesion areas and lesion types;
the image feature extraction module is used for extracting multi-scale image features of the training data through the convolutional neural network; the feature pyramid network performs up-sampling and summation on adjacent features in the image features, and fuses features of different scale depths to obtain enhanced features; inputting the enhanced features into the lesion candidate region extraction network to obtain all candidate region features containing lesions in the enhanced features;
and the lesion area identification module is used for inputting all the candidate area characteristics into the joint identification network to obtain the lesion type and lesion area of the training data, constructing a loss function by combining the labeled lesion area and lesion type of the training data, iteratively training the lesion identification model through the loss function until the loss function is converged or reaches a preset iteration number, and identifying and obtaining the lesion area of the specified lung medical image by using the current lesion identification model.
The lung lesion identification system is characterized in that the lung medical image is a CT image.
The lung lesion recognition system, wherein the image feature extraction module comprises: and performing information interaction fusion on the features with different scales or depths in the image features through a multi-scale feature pyramid network.
The lung lesion recognition system, wherein the lesion category comprises frosted glass and concomitant bronchiectasis;
the lesion area identification module includes: and respectively executing corresponding classification, coordinate frame regression and semantic segmentation tasks by a classification head, a detection head and a segmentation head network in the combined recognition network to obtain recognition results of the ground glass and the bronchiectasis associated regions in the lung medical image.
The invention also proposes a storage medium for storing a program for executing the method for identifying a lung lesion.
The invention also provides a client used for the lung lesion identification system.
Claims (10)
1. A method for identifying a lung lesion, comprising:
step 1, constructing a lesion identification model comprising a convolutional neural network, a characteristic pyramid network, a lesion candidate region extraction network and a joint identification network, wherein the joint identification network comprises a classification head network, a detection head network and a segmentation head network; acquiring a lung medical image as training data, wherein the training data are marked with lesion areas and lesion types;
step 2, extracting multi-scale image features of the training data through the convolutional neural network; the feature pyramid network performs up-sampling and summation on adjacent features in the image features, and fuses features of different scale depths to obtain enhanced features; inputting the enhanced features into the lesion candidate region extraction network to obtain all candidate region features containing lesions in the enhanced features;
and 3, inputting the characteristics of all the candidate regions into the joint recognition network to obtain the lesion type and the lesion region of the training data, constructing a loss function by combining the labeled lesion region and the labeled lesion type of the training data, iteratively training the lesion recognition model through the loss function until the loss function is converged or reaches a preset iteration number, and recognizing by using the current lesion recognition model to obtain the lesion region of the specified lung medical image.
2. The method of claim 1, wherein the medical image of the lung is a CT image.
3. The method of claim 1, wherein the step 2 comprises: and performing information interaction fusion on the features with different scales or depths in the image features through a multi-scale feature pyramid network.
4. The method of claim 1, wherein the lesion category includes frosted glass and concomitant bronchodilation;
the step 3 comprises the following steps: and respectively executing corresponding classification, coordinate frame regression and semantic segmentation tasks by a classification head, a detection head and a segmentation head network in the combined recognition network to obtain recognition results of the ground glass and the bronchiectasis associated regions in the lung medical image.
5. A lung lesion identification system, comprising:
the model construction module is used for constructing a lesion identification model comprising a convolutional neural network, a characteristic pyramid network, a lesion candidate region extraction network and a joint identification network, wherein the joint identification network comprises a classification head network, a detection head network and a segmentation head network; acquiring a lung medical image as training data, wherein the training data are marked with lesion areas and lesion types;
the image feature extraction module is used for extracting multi-scale image features of the training data through the convolutional neural network; the feature pyramid network performs up-sampling and summation on adjacent features in the image features, and fuses features of different scale depths to obtain enhanced features; inputting the enhanced features into the lesion candidate region extraction network to obtain all candidate region features containing lesions in the enhanced features;
and the lesion area identification module is used for inputting all the candidate area characteristics into the joint identification network to obtain the lesion type and lesion area of the training data, constructing a loss function by combining the labeled lesion area and lesion type of the training data, iteratively training the lesion identification model through the loss function until the loss function is converged or reaches a preset iteration number, and identifying and obtaining the lesion area of the specified lung medical image by using the current lesion identification model.
6. The system of claim 1, wherein the medical image of the lung is a CT image.
7. The system of claim 1, wherein the image feature extraction module comprises: and performing information interaction fusion on the features with different scales or depths in the image features through a multi-scale feature pyramid network.
8. The lung lesion recognition system of claim 1, wherein the lesion categories include frosted glass and concomitant bronchodilation;
the lesion area identification module includes: and respectively executing corresponding classification, coordinate frame regression and semantic segmentation tasks by a classification head, a detection head and a segmentation head network in the combined recognition network to obtain recognition results of the ground glass and the bronchiectasis associated regions in the lung medical image.
9. A storage medium storing a program for executing the lung lesion recognition method according to any one of claims 1 to 4.
10. A client for use in a lung lesion identification system as claimed in any one of claims 5 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111296412.0A CN115082909B (en) | 2021-11-03 | 2021-11-03 | Method and system for identifying lung lesions |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111296412.0A CN115082909B (en) | 2021-11-03 | 2021-11-03 | Method and system for identifying lung lesions |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115082909A true CN115082909A (en) | 2022-09-20 |
| CN115082909B CN115082909B (en) | 2024-04-12 |
Family
ID=83245545
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111296412.0A Active CN115082909B (en) | 2021-11-03 | 2021-11-03 | Method and system for identifying lung lesions |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115082909B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3171297A1 (en) * | 2015-11-18 | 2017-05-24 | CentraleSupélec | Joint boundary detection image segmentation and object recognition using deep learning |
| WO2019144575A1 (en) * | 2018-01-24 | 2019-08-01 | 中山大学 | Fast pedestrian detection method and device |
| CN110599448A (en) * | 2019-07-31 | 2019-12-20 | 浙江工业大学 | Migratory learning lung lesion tissue detection system based on MaskScoring R-CNN network |
| CN111126202A (en) * | 2019-12-12 | 2020-05-08 | 天津大学 | Optical remote sensing image target detection method based on void feature pyramid network |
-
2021
- 2021-11-03 CN CN202111296412.0A patent/CN115082909B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3171297A1 (en) * | 2015-11-18 | 2017-05-24 | CentraleSupélec | Joint boundary detection image segmentation and object recognition using deep learning |
| WO2019144575A1 (en) * | 2018-01-24 | 2019-08-01 | 中山大学 | Fast pedestrian detection method and device |
| CN110599448A (en) * | 2019-07-31 | 2019-12-20 | 浙江工业大学 | Migratory learning lung lesion tissue detection system based on MaskScoring R-CNN network |
| CN111126202A (en) * | 2019-12-12 | 2020-05-08 | 天津大学 | Optical remote sensing image target detection method based on void feature pyramid network |
Non-Patent Citations (3)
| Title |
|---|
| 姜世浩;齐苏敏;王来花;贾惠;: "基于Mask R-CNN和多特征融合的实例分割", 计算机技术与发展, no. 09, 10 September 2020 (2020-09-10) * |
| 李维嘉;陈爽;张雷;吴正灏;: "基于深度学习图像处理的肺部造影检测研究", 自动化与仪器仪表, no. 12, 25 December 2019 (2019-12-25) * |
| 高智勇;黄金镇;杜程刚;: "基于特征金字塔网络的肺结节检测", 计算机应用, no. 09, 31 December 2020 (2020-12-31) * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115082909B (en) | 2024-04-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Tong et al. | Salient object detection via bootstrap learning | |
| EP3739513A1 (en) | Surface defect identification method and apparatus | |
| CN110852316B (en) | Image tampering detection and positioning method adopting convolution network with dense structure | |
| Dong et al. | A deep-learning-based multiple defect detection method for tunnel lining damages | |
| CN111274926B (en) | Image data screening method, device, computer equipment and storage medium | |
| CN113011357A (en) | Depth fake face video positioning method based on space-time fusion | |
| CN111257341A (en) | Underwater building crack detection method based on multi-scale features and stacked full convolution network | |
| CN113052295B (en) | Training method of neural network, object detection method, device and equipment | |
| CN112257665A (en) | Image content recognition method, image recognition model training method, and medium | |
| CN114612472A (en) | SegNet improvement-based leather defect segmentation network algorithm | |
| CN115239644A (en) | Concrete defect identification method and device, computer equipment and storage medium | |
| Xu et al. | Concrete crack segmentation based on convolution–deconvolution feature fusion with holistically nested networks | |
| CN111652181B (en) | Target tracking method and device and electronic equipment | |
| JP5137805B2 (en) | Inspection system, control method therefor, and computer program | |
| CN112861881A (en) | Honeycomb lung recognition method based on improved MobileNet model | |
| CN116468690B (en) | Subtype analysis system of invasive non-mucous lung adenocarcinoma based on deep learning | |
| Shit et al. | An encoder‐decoder based CNN architecture using end to end dehaze and detection network for proper image visualization and detection | |
| CN116580232A (en) | Automatic image labeling method and system and electronic equipment | |
| CN115082909B (en) | Method and system for identifying lung lesions | |
| CN116977249A (en) | Defect detection method, model training method and device | |
| Ali et al. | Asphalt Pavement Potholes Localization and Segmentation using Deep Retina Net and Conditional Random Fields | |
| Nguyen et al. | Pavement crack detection and segmentation based on deep neural network | |
| CN114387496A (en) | Target detection method and electronic equipment | |
| Kee et al. | Cracks identification using mask region-based denoised deformable convolutional network | |
| CN114639013A (en) | Remote sensing image airplane target detection and identification method based on improved Orient RCNN model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |