CN116883760A - Medical image classification and segmentation method and system - Google Patents
Medical image classification and segmentation method and system Download PDFInfo
- Publication number
- CN116883760A CN116883760A CN202310941308.5A CN202310941308A CN116883760A CN 116883760 A CN116883760 A CN 116883760A CN 202310941308 A CN202310941308 A CN 202310941308A CN 116883760 A CN116883760 A CN 116883760A
- Authority
- CN
- China
- Prior art keywords
- medical
- image
- segmentation
- medical image
- instance
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000011218 segmentation Effects 0.000 title claims abstract description 98
- 238000000034 method Methods 0.000 title claims abstract description 47
- 238000003709 image segmentation Methods 0.000 claims abstract description 33
- 238000009826 distribution Methods 0.000 claims abstract description 26
- 238000003860 storage Methods 0.000 claims description 19
- 238000004590 computer program Methods 0.000 claims description 14
- 238000012545 processing Methods 0.000 claims description 14
- 238000013135 deep learning Methods 0.000 claims description 9
- 239000003623 enhancer Substances 0.000 claims description 6
- 238000012805 post-processing Methods 0.000 claims description 6
- 230000008569 process Effects 0.000 claims description 5
- 238000004458 analytical method Methods 0.000 claims description 4
- 238000013145 classification model Methods 0.000 claims description 4
- 239000000284 extract Substances 0.000 claims description 4
- 230000004927 fusion Effects 0.000 claims description 4
- 238000010801 machine learning Methods 0.000 claims description 4
- 230000002902 bimodal effect Effects 0.000 claims description 3
- 230000009977 dual effect Effects 0.000 claims description 3
- 230000002708 enhancing effect Effects 0.000 claims description 3
- 238000013507 mapping Methods 0.000 claims description 2
- 238000007781 pre-processing Methods 0.000 claims 1
- 230000007547 defect Effects 0.000 abstract description 3
- 238000005457 optimization Methods 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 10
- 230000000694 effects Effects 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 238000013527 convolutional neural network Methods 0.000 description 5
- 201000010099 disease Diseases 0.000 description 3
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000012549 training Methods 0.000 description 3
- 238000011282 treatment Methods 0.000 description 3
- 239000013598 vector Substances 0.000 description 3
- 241000282326 Felis catus Species 0.000 description 2
- 238000013136 deep learning model Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 238000009499 grossing Methods 0.000 description 2
- 230000003902 lesion Effects 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000003909 pattern recognition Methods 0.000 description 2
- 238000011176 pooling Methods 0.000 description 2
- 101100121955 Tanacetum cinerariifolium GLIP gene Proteins 0.000 description 1
- 210000001015 abdomen Anatomy 0.000 description 1
- 230000005856 abnormality Effects 0.000 description 1
- 210000003484 anatomy Anatomy 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 238000002591 computed tomography Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000006378 damage Effects 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000002059 diagnostic imaging Methods 0.000 description 1
- 238000010191 image analysis Methods 0.000 description 1
- 238000011337 individualized treatment Methods 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000001537 neural effect Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 230000001575 pathological effect Effects 0.000 description 1
- 230000000306 recurrent effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
- G06N3/0455—Auto-encoder networks; Encoder-decoder networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/03—Recognition of patterns in medical or anatomical images
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Image Analysis (AREA)
- Medical Treatment And Welfare Office Work (AREA)
Abstract
The invention discloses a medical image classification and segmentation method and a system, comprising the following steps: a complete medical image and a medical tissue name of interest are entered. And identifying and classifying the input medical images to determine the medical tissue types in the images. And selecting a corresponding medical example main body template according to the identification picture frame and the classification result of the medical image. Image segmentation is performed on the weight distribution of the instance image using the identified medical tissue instance image. The medical tissue subject matter instance of interest is segmented from the complete medical image by a segmentation unit or a segmentation algorithm. The invention has the advantages that: the method realizes specific optimization in the fields of medical image classification and segmentation, overcomes the defect of unsupervised learning, improves the accuracy of medical image segmentation classification, and optimizes the segmentation result.
Description
Technical Field
The invention relates to the technical field of medical image processing, in particular to a medical image classification and segmentation method and a medical image classification and segmentation system.
Background
Medical image classification and segmentation is important in the medical field, mainly in the following aspects: medical image classification and segmentation can help doctors to more accurately segment out structures or lesion areas of interest, and provide quantitative anatomical and pathological information; medical image classification and segmentation provides doctors with more detailed and accurate anatomical information, which is helpful for assisting doctors in diagnosing and planning treatment of diseases; medical image classification and segmentation provide the basis for individualized medical treatment, through accurate segmentation of anatomy and lesion area of each patient, doctor can formulate individualized treatment plan according to individual characteristic and demand, improves treatment to the maximum extent to reduce unnecessary risk and damage.
Although the need for intelligent medical image segmentation is great, in the past medical image segmentation field manual segmentation is generally performed, which is a time-consuming and tedious task. And the intelligent segmentation model can accomplish this automatically or semi-automatically. This can significantly improve the efficiency of the physician and medical imaging professionals, lessening their burden and enabling them to better focus on other important clinical tasks.
Intelligent medical image segmentation often has faster speeds and more accurate effects than traditional manual segmentation and is therefore widely recognized as being useful for better medical image segmentation. There is also a focus of researchers on the field of intelligently segmenting images. For example, in 2015, jonathan Long, evan Shellhamer and Trevor Darrell et al proposed FCN (Fully Convolutional Network). They first introduced the concept and architecture of FCNs in paper Fully Convolutiona l Networks for Semantic Segmentation. It is a deep learning model for image segmentation. The conventional Convolutional Neural Network (CNN) is mainly used for an image classification task, takes an entire image as an input, and outputs a class label of a corresponding image. The FCN is improved on the basis of the CNN, so that the FCN can perform pixel-level image segmentation, namely each pixel in the image is distributed to different categories, thus the accuracy of a segmentation result can be improved, and the problems of space information loss and pixel boundary blurring caused by pooling operation are reduced. The FCN has good effects in image segmentation tasks and is widely applied to the fields of medical image segmentation, automatic driving, remote sensing image analysis and the like. With the development of deep learning, FCNs have also derived many variants and modified versions, such as U-Net, segNet, etc., to further enhance the performance and effectiveness of segmentation.
While intelligent medical image classification is also of great importance for medical development and research, deep learning is one of the most popular and effective methods in medical image classification at present. Deep learning models, such as Convolutional Neural Networks (CNNs), recurrent Neural Networks (RNNs), and transducers (transformers), can automatically learn features from large-scale medical image data and classify them. These models can be trained on different medical image types (e.g., X-ray, MRI, CT, etc.) and diseases, and can identify disease features and abnormalities.
Although FCNs and their subsequent variants have made significant progress in image segmentation tasks, there are still some limitations such as pixel boundary blurring: FCNs tend to obscure the pixel boundaries of the segmentation result due to the convolution and pooling operations. The method is characterized in that the size of a segmented image output by a network is not completely matched with that of an original image after upsampling, so that the pixel position in a prediction result is inaccurate; small object segmentation is difficult: FCNs and variants thereof may experience difficulties in handling segmentation of small objects. Spatial information of small objects is often easily lost, resulting in inaccurate or incomplete segmentation results.
Regarding the segmentation classification as a whole, no good technology is used for combining the segmentation classification, but the combination of the segmentation classification of the medical images can greatly improve the efficiency of medical workers, so that the medical workers can see the full view of the medical images, and simultaneously, the interested segmentation positions can be quickly found, so that the medical workers can pay more attention to more meaningful work.
Reference to the literature
1.Kirillov,A.,Mintun,E.,Ravi,N.,Mao,H.,Rolland,C.,Gustafson,L.,...&Girshick,R.(2023).Segment anything.arXiv preprint arXiv:2304.02643；
2.Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez,A.N.,...&Polosukhin,I.(2017).Attention is all you need.Advances in neural information processing systems,30；
3.He,K.,Chen,X.,Xie,S.,Li,Y.,Dollár,P.,&Girshick,R.(2022).Mas ked autoencoders are scalable vision learners.In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition(pp.16000-16009)；
4.Dosovitskiy,A.,Beyer,L.,Kolesnikov,A.,Weissenborn,D.,Zhai,X.,Unterthiner,T.,...&Houlsby,N.(2020).An image is worth 16x16 words:Transf ormers for image recognition at scale.arXiv preprint arXiv:2010.11929；
5.Long,J.,Shelhamer,E.,&Darrell,T.(2015).Fully convolutional networ ks for semantic segmentation.In Proceedings of the IEEE conference on comput er vision and pattern recognition(pp.3431-3440)。
Disclosure of Invention
The invention provides a medical image classification and segmentation method and a medical image classification and segmentation system aiming at the defects of the prior art.
In order to achieve the above object, the present invention adopts the following technical scheme:
a method of classifying and segmenting medical images, comprising the steps of:
s1: inputting a complete medical image and inputting a medical tissue name of interest;
s2: and identifying and classifying the input medical image by using the main body position information corresponding to the medical tissue name of interest. Medical images are classified by machine learning or deep learning methods to determine the subject categories in the images.
S3: and selecting corresponding medical example main body templates for medical image classification frames of different categories according to the medical image identification picture frame and the classification result. The medical instance subject templates are predefined, each template containing different instance parameters, which in turn correspond to different weight distributions. The weight distribution reflects the probability that each instance in the medical image belongs to a different medical tissue class.
S4: image segmentation is performed on the weight distribution of the identified instance images using the classified and identified medical tissue instance images. The medical tissue subject matter instance of interest is segmented out of the complete medical image by a segmentation unit or a segmentation algorithm.
Further, in S2, classification identification of the medical image is performed using a deep learning method. A dual encoder-single decoder architecture is employed, comprising: an image backbone, a text backbone, a feature enhancer, a language-guided query selection module, and a cross-modality decoder.
The specific substeps of S2 are as follows:
s21: for each text and image pair, common image features and text features are extracted using the image backbone and the text backbone, respectively.
S22: and sending the image features and the text features to a feature enhancer for cross-modal feature fusion.
S23: after obtaining the cross-modal text and image features, a cross-modal query is selected from the image features using a language-guided query selection module.
S24: the cross-modal query is fed into a cross-modal decoder, which detects the required features from the bimodal features and updates itself.
The output query of the last decoder layer is used to predict the object box and extract the corresponding phrase S25.
Further, in S3, it includes: an input section, a dividing section, and an output section.
The input section needs to provide two inputs: image input and prompt input. A medical image is input and a prompt description is then made of the tissue site of the medical image that needs to be segmented. The promts fall into three categories: point promt, box promt, and text promt. The medical image box pr ompt identified by the previous stage classification identity model is used as input.
The segmentation is performed by a segmentation model consisting of an image encoder, a prompt encoder and a medical image mask decoder.
The image encoder uses a model ViT pre-trained by MAE. The image encoder runs once for each image and applies before the sample model.
A campt encoder uses position coding to represent the boxes and adds the learning embedded and free form text for each campt type to the ready text coding in CLIP.
The medical image decoder is responsible for mapping the image embedding, sample embedding and output token to a mask, thereby generating a segmentation mask.
Further, S4 comprises the sub-steps of:
s41: generating a classification identification frame: according to the classification identification result in step S2, the region of the medical image classified as the medical tissue of interest is marked as a classification identification frame. This identification frame may be rectangular or of any shape.
S42: segmentation of medical tissue: for each classified identification frame, image segmentation is performed on the medical tissue contained in the classified identification frame according to the weight distribution of each medical instance subject template in step S3. This may use image segmentation algorithms such as semantic segmentation or instance segmentation, etc.
S43: outputting a segmentation result: the segmented medical tissue image is processed to have the same dimensions as the original medical image. This can be achieved by adjusting the size of the segmented image or by interpolation or the like. Finally, a medical image instance with the medical tissue of interest is output.
The invention also discloses a medical image classifying and segmenting system which can be used for implementing the medical image classifying and segmenting method, and specifically comprises the following functional modules:
an image loading module: is responsible for loading medical image data and inputting it into the system for processing.
And a pretreatment module: the loaded medical image is preprocessed, including denoising, enhancing, resizing and the like, so as to improve the accuracy and effect of subsequent processing.
And the classification identification module is used for: the pre-trained classification model is used to classify and identify the pre-processed medical images. The module processes the image according to the prompt of the interested subject in the medical image, and outputs the category and the position of the medical tissue.
An example template selection module: and selecting a corresponding medical tissue image main body example template according to the classification identification result. Each subject instance template contains predefined instance parameters and weight distributions for representing the probability that each instance in the medical image belongs to a different medical tissue.
An image segmentation module: and performing image segmentation on the medical image based on the example template and the classification identification result. The medical tissue subject instance of interest is segmented from the original image using a suitable segmentation algorithm, such as semantic segmentation or instance segmentation.
And a post-processing module: post-processing is performed on the segmented medical tissue body instance, including resizing, interpolation, edge smoothing, etc., to obtain a more accurate and clear segmentation result.
And an output module: the final segmentation results are output to a display device or saved to a storage device for viewing, analysis and application by the user.
The invention also discloses a computer device, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor realizes the classification and segmentation method of the medical image when executing the program.
The invention also discloses a computer readable storage medium having stored thereon a computer program which when executed by a processor implements a method of classifying and segmenting medical images as described above.
Compared with the prior art, the invention has the advantages that:
the method realizes specific optimization in the fields of medical image classification and segmentation, overcomes the defect of unsupervised learning, and further improves the accuracy of medical image segmentation classification. The segmentation result is optimized after the picture classification.
Drawings
FIG. 1 is a flow chart of a method of classifying and segmenting medical images in accordance with an embodiment of the present invention;
FIG. 2 is a schematic diagram of a medical image classification identification model structure according to an embodiment of the invention;
FIG. 3 is a schematic view of a medical image segmentation model structure according to an embodiment of the present invention;
FIG. 4 is a frame diagram of the overall solution of an embodiment of the invention;
FIG. 5 is a diagram illustrating the difference or error level between the predicted value and the true value of the segmentation model according to an embodiment of the present invention.
Detailed Description
The invention will be described in further detail below with reference to the accompanying drawings and by way of examples in order to make the objects, technical solutions and advantages of the invention more apparent.
As shown in fig. 1, a method of classifying and segmenting medical images. The method for dividing the medical image example main body according to the result of the picture classification mark comprises the following steps:
1. in step S101, an image classification identification is performed on an input medical image according to a subject position in the medical image
2. Step S102 is performed after that, according to the result of the medical image classification identification, corresponding medical example main body templates are selected for the medical images of different categories, wherein the predefined example parameters in each main body template are different, and each example main body template is provided with different weight distribution according to the predefined example parameters, and the weight distribution reflects the probability that each example in the medical image belongs to different medical tissue categories.
3. According to the classified and identified medical tissue subject instance images, the segmentation unit performs picture segmentation on the weight distribution of the identified instance images, and segments the medical tissue subject instance of interest from the complete medical image.
The flow is ended thereafter.
According to the steps, before medical image segmentation, medical images are classified according to the example categories, and compared with the existing technologies for directly segmenting the images, the method optimizes the segmentation result after the image classification.
In the step S101, the picture classification and identification are performed by a deep learning method, so that the classification and identification are reasonable and the classification and identification effect is good. Of course, other methods of identifying the picture category may be applied herein, and are not limited thereto.
The medical image classification identifies a plurality of pairs of object boxes and noun phrases for a given (medical image, text sample) pair. For example, as shown in FIG. 2, the model locates a cat and a table from the input image and extracts words cat and table from the input text as corresponding tags. Both the target detection and REC tasks may be aligned with the pipeline. According to GLIP (a text and image correspondence algorithm), the present embodiment concatenates names of all categories as input text for an object detection task. REC requires a bounding box for each text entry. The present embodiment uses the output object with the largest score as the output of the REC task.
The medical image classification identification model is a dual encoder-single decoder architecture. The system comprises an image backbone for image feature extraction, a text backbone for text feature extraction, a feature enhancer for image and text feature fusion, a language-guided query selection module for query initialization and a cross-modal decoder for box refinement. The overall frame is shown in fig. 2.
For each (image, text) pair, first the image backbone and the text backbone are used to extract the normal image features and normal text features, respectively. These two common features are fed into a feature enhancer module for cross-modal feature fusion. After obtaining the cross-modal text and image features, the present embodiment uses a language-guided query selection module to select a cross-modal query from the image features. As with the object queries in most DETR class models, these cross-modal queries will be sent to a cross-modal decoder to detect the desired features from the bimodal features and update themselves. The output query of the last decoder layer will be used to predict the object box and extract the corresponding phrase.
The above step S102 is specifically described below. In step S102, corresponding subject instance templates are selected for medical images of different categories according to the result of the picture classification identifier frame, wherein the predefined segmentation parameters in each subject instance template are different, and each subject instance template sets different weight distribution according to the predefined segmentation parameters, and the weight distribution reflects the probability that each instance in the medical image belongs to different medical tissues.
The segmentation model shows the technical scheme of an unsupervised large model applied to medical image segmentation, and the technical scheme mainly comprises three parts: medical image and prompt inputs, segmentation model and output.
Wherein the input part requires two inputs, image input and prompt input, firstly, medical image input is required, and then, the prompt description is carried out on the tissue part needing to divide the medical image, and the prompt can be roughly divided into three types: the point template, box template and text template are used as input in the present method by the embodiment of the present invention for the medical image box template identified by the previous stage classification identification model.
Following is a segmentation of the technique, the segmentation model has three plates: image encoder, sample encoder, and medical image mask decoder. For image encoders, the present embodiment uses MAE pre-trained ViT, minimally adapted to handle high resolution inputs. The image encoder runs each image once and applies the images before the sample model; next is a promtt encoder, which only considers one promtt: block prompt. The present embodiment represents the box by position coding and adds the learning embedded and free form text for each sample type to the ready text coding in CLIP; finally, is a medical image decoder, which effectively maps image embedding, hint embedding, and output token to a mask. The design is flexible from the DETR, adopts the modification of the Transformer decoder module (with a dynamic mask pre-measuring head), and simultaneously adopts the medical image data set as the data set used in the model retraining process, so that the segmentation accuracy of the decoder is higher.
In particular, the medical image decoder receives as input embedded vectors from the image encoder and the prompt encoder. It uses these embedded vectors and the previously learned weight distributions to generate a segmentation mask for the medical image by using a modified transducer decoder module.
The decoder generates an output sequence step by step based on the input embedded vector and previous context information. At the same time, it uses specific attention mechanisms and self-attention mechanisms to capture the association information between the image and the prot. By continually iterating and generating an output, the decoder classifies each pixel in the image, judging to which medical organization or instance it belongs.
Fig. 3 depicts the construction of a medical image segmentation model. The large image model can input any picture to obtain an interested image segmentation mask through inputting proper prompt, and the supervised medical image segmentation decoding model compensates the suitability and accuracy of the unsupervised large image model in the medical field, so that the generalization and accuracy of segmentation can be realized simultaneously through the combination of the two models.
The above step S103 is specifically described below. In step S101, according to the input picture and the input corresponding classification result, medical images of different categories are in one-to-one correspondence, and finally, the prediction result of the maximum score is used as the output of the classification identifier; in step S102, the pre-defined segmentation parameters in each main body instance template are different, each instance main body template sets different weight distribution according to the pre-defined segmentation parameters, the weight distribution reflects the probability that each instance in the medical image belongs to different medical tissues, and finally, the segmented medical tissue image with the highest probability is output; and S103 is to combine the steps of S101 and S102, firstly generate a classification identification frame, then divide the medical tissue in the classification identification frame, and finally output the medical tissue in the same size as the original image.
Fig. 3 is a frame diagram of the whole technical scheme, firstly, a medical image to be classified and segmented is put into a classification original model trained by the medical image, and a text template, such as a river, is given to a meta model. The classification model can identify the corresponding object of the text sample and uses a square block for classification identification, then the original image and the square block which is subjected to classification identification are put into the segmentation model, the segmentation model can accurately segment the part which needs segmentation, and finally the functions of classification identification and segmentation are realized.
Fig. 4 illustrates a complete medical image processing flow for text prompt input, medical image classification identification model, medical image segmentation model and final medical image output. The main part of the technique is a supervised medical image segmentation decoding model and an unsupervised large image model, wherein the medical tissue text to be classified is input first, here a river is input, and the image is input next, and the image is an abdomen image of CT scanning, wherein the river is included. Placing the two inputs into a classification identification model, and generating an image which frames a river in the image through model prediction, wherein the size of the image is the same as that of the original image; then the image with the identification frame is put into the segmentation model and is input into the image and the frame on the image, at the moment, the segmentation model detects the image in the frame and segments the image in the frame, and the recognition rate is high because the segmentation model is pre-trained in the field of medical images, and finally, the image output of the medical tissue with classification, identification and segmentation is directly generated.
Taking fig. 5 as an example, the abscissa represents epoch (time, how many rounds are trained) of the segmentation model training, and the ordinate represents loss, which represents the difference or error degree between the predicted value and the true value of the segmentation model. It can be seen that the error level is continuously decreasing, and in the figure, the Dice and Cross Entropy Loss are depicted, both of which mean a similar level to the ground truth (artificially labeled true value). I.e. the prediction accuracy after model training is higher and higher. From the analysis of the final quantitative result, my ideas are also verified, and the model segmentation effect after medical image training is greatly improved compared with that of the original model.
In a further embodiment of the present invention, a medical image classification and segmentation system is provided, which can be used to implement a medical image classification and segmentation method as described above, and specifically includes the following functional modules:
an image loading module: is responsible for loading medical image data and inputting it into the system for processing.
And a pretreatment module: the loaded medical image is preprocessed, including denoising, enhancing, resizing and the like, so as to improve the accuracy and effect of subsequent processing.
And the classification identification module is used for: the pre-trained classification model is used to classify and identify the pre-processed medical images. The module processes the image according to the prompt of the interested subject in the medical image, and outputs the category and the position of the medical tissue.
An example template selection module: and selecting a corresponding medical tissue image main body example template according to the classification identification result. Each subject instance template contains predefined instance parameters and weight distributions for representing the probability that each instance in the medical image belongs to a different medical tissue.
An image segmentation module: and performing image segmentation on the medical image based on the example template and the classification identification result. The medical tissue subject instance of interest is segmented from the original image using a suitable segmentation algorithm, such as semantic segmentation or instance segmentation.
And a post-processing module: post-processing is performed on the segmented medical tissue body instance, including resizing, interpolation, edge smoothing, etc., to obtain a more accurate and clear segmentation result.
And an output module: the final segmentation results are output to a display device or saved to a storage device for viewing, analysis and application by the user.
In yet another embodiment of the present invention, a terminal device is provided, the terminal device including a processor and a memory, the memory for storing a computer program, the computer program including program instructions, the processor for executing the program instructions stored by the computer storage medium. The processor may be a central processing unit (Central Processing Unit, CPU), but may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circ uit, ASIC), off-the-shelf Programmable gate arrays (FPGAs) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., which are the computational core and control core of the terminal adapted to implement one or more instructions, in particular adapted to load and execute one or more instructions to implement a corresponding method flow or a corresponding function; the processor according to the embodiment of the invention can be used for the operation of a medical image classification and segmentation method, and comprises the following steps:
s1: inputting a complete medical image and inputting a medical tissue name of interest;
s2: and identifying and classifying the input medical image by using the main body position information corresponding to the medical tissue name of interest. Medical images are classified by machine learning or deep learning methods to determine the subject categories in the images.
S3: and selecting corresponding medical example main body templates for medical image classification frames of different categories according to the medical image identification picture frame and the classification result. The medical instance subject templates are predefined, each template containing different instance parameters, which in turn correspond to different weight distributions. The weight distribution reflects the probability that each instance in the medical image belongs to a different medical tissue class.
S4: image segmentation is performed on the weight distribution of the identified instance images using the classified and identified medical tissue instance images. The medical tissue subject matter instance of interest is segmented out of the complete medical image by a segmentation unit or a segmentation algorithm.
In a further embodiment of the present invention, the present invention also provides a storage medium, in particular, a computer readable storage medium (Memory), which is a Memory device in a terminal device, for storing programs and data. It will be appreciated that the computer readable storage medium herein may include both a built-in storage medium in the terminal device and an extended storage medium supported by the terminal device. The computer-readable storage medium provides a storage space storing an operating system of the terminal. Also stored in the memory space are one or more instructions, which may be one or more computer programs (including program code), adapted to be loaded and executed by the processor. The computer readable storage medium herein may be a high-speed RAM memory or a non-volatile memory (non-volatile memory), such as at least one magnetic disk memory.
One or more instructions stored in a computer-readable storage medium may be loaded and executed by a processor to implement the respective steps of the classification and segmentation method in relation to one medical image in the above-described embodiments; one or more instructions in a computer-readable storage medium are loaded by a processor and perform the steps of:
s1: inputting a complete medical image and inputting a medical tissue name of interest;
s2: and identifying and classifying the input medical image by using the main body position information corresponding to the medical tissue name of interest. Medical images are classified by machine learning or deep learning methods to determine the subject categories in the images.
S3: and selecting corresponding medical example main body templates for medical image classification frames of different categories according to the medical image identification picture frame and the classification result. The medical instance subject templates are predefined, each template containing different instance parameters, which in turn correspond to different weight distributions. The weight distribution reflects the probability that each instance in the medical image belongs to a different medical tissue class.
S4: image segmentation is performed on the weight distribution of the identified instance images using the classified and identified medical tissue instance images. The medical tissue subject matter instance of interest is segmented out of the complete medical image by a segmentation unit or a segmentation algorithm.
It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Those of ordinary skill in the art will appreciate that the embodiments described herein are intended to aid the reader in understanding the practice of the invention and that the scope of the invention is not limited to such specific statements and embodiments. Those of ordinary skill in the art can make various other specific modifications and combinations from the teachings of the present disclosure without departing from the spirit thereof, and such modifications and combinations remain within the scope of the present disclosure.
Claims (7)
1. A method of classifying and segmenting medical images, comprising the steps of:
s1: inputting a complete medical image and inputting a medical tissue name of interest;
s2: the method comprises the steps of utilizing main body position information corresponding to medical tissue names of interest to identify and classify input medical images; classifying the medical images by a machine learning or deep learning method to determine topic categories in the images;
s3: selecting corresponding medical example main body templates for medical image classification frames of different categories according to medical image identification frames and classification results; the medical example main body templates are predefined, each template comprises different example parameters, and the example parameters correspond to different weight distributions; the weight distribution reflects the probability that each instance in the medical image belongs to a different medical tissue class;
s4: image segmentation of the weight distribution of the identified instance images using the classified and identified medical tissue instance images; the medical tissue subject matter instance of interest is segmented out of the complete medical image by a segmentation unit or a segmentation algorithm.
2. A method of classifying and segmenting medical images according to claim 1, characterized in that: in S2, classifying and identifying medical images by using a deep learning method; a dual encoder-single decoder architecture is employed, comprising: the system comprises an image backbone, a text backbone, a feature enhancer, a language guide query selection module and a cross-modal decoder;
the specific substeps of S2 are as follows:
s21: for each text and image pair, extracting common image features and text features by using an image backbone and a text backbone respectively;
s22: sending the image features and the text features into a feature enhancer for cross-modal feature fusion;
s23: after obtaining the cross-modal text and image features, selecting a cross-modal query from the image features using a language-guided query selection module;
s24: the cross-modal query is sent to a cross-modal decoder, and the required characteristics are detected from the bimodal characteristics and updated;
the output query of the last decoder layer is used to predict the object box and extract the corresponding phrase S25.
3. A method of classifying and segmenting medical images according to claim 1, characterized in that: s3, including: an input section, a dividing section, and an output section;
the input section needs to provide two inputs: image input and prompt input; inputting a medical image, and then performing a prompt description on a tissue part of the medical image to be segmented; the promts fall into three categories: point prompt, box promt, and text promt; using the medical image frame pr ompt identified by the previous stage classification identification model as input;
the segmentation part is performed by a segmentation model, and the segmentation model consists of an image encoder, a prompt encoder and a medical image mask decoder;
the image encoder used a MAE pre-trained ViT model; the image encoder runs each image once and applies the images before the sample model;
a campt encoder that uses position coding to represent the boxes and adds the learning embedded and free form text for each campt type to the ready text coding in CLIP;
the medical image decoder is responsible for mapping the image embedding, sample embedding and output token to a mask, thereby generating a segmentation mask.
4. A method of classifying and segmenting medical images according to claim 1, characterized in that: s4 comprises the following substeps:
s41: generating a classification identification frame: marking the region classified as the medical tissue of interest in the medical image as a classification identification frame according to the classification identification result in the step S2; this identification frame may be rectangular or of any shape;
s42: segmentation of medical tissue: for each classified identification frame, according to the weight distribution of each medical instance main body template in the step S3, carrying out image segmentation on medical tissues contained in the classified identification frames;
s43: outputting a segmentation result: processing the segmented medical tissue image to have the same size as the original medical image; finally, a medical image instance with the medical tissue of interest is output.
5. A medical image classification and segmentation system, characterized by: the system can be used to implement a method of classifying and segmenting medical images according to one of claims 1 to 4, comprising the following functional modules:
an image loading module: the medical image data are loaded and input into the system for processing;
and a pretreatment module: preprocessing the loaded medical image, including denoising, enhancing and resizing;
and the classification identification module is used for: classifying and identifying the preprocessed medical image by using a pre-trained classification model; the module processes the image according to the prompt of the interested subject in the medical image and outputs the category and the position of the medical tissue;
an example template selection module: selecting a corresponding medical tissue image main body example template according to the classification identification result; each subject instance template contains predefined instance parameters and weight distributions for representing the probability that each instance in the medical image belongs to a different medical tissue;
an image segmentation module: based on the example template and the classification identification result, performing image segmentation on the medical image; using a suitable segmentation algorithm, segmenting the medical tissue body instance of interest from the original image;
and a post-processing module: post-processing the segmented medical tissue body instance;
and an output module: the final segmentation results are output to a display device or saved to a storage device for viewing, analysis and application by the user.
6. A computer device, characterized by: comprising a memory, a processor and a computer program stored on the memory and executable on the processor, said processor implementing a method of classifying and segmenting a medical image according to one of claims 1 to 4 when said program is executed.
7. A computer-readable storage medium, characterized by: a computer program stored thereon, which when executed by a processor, implements a method of classifying and segmenting medical images according to one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310941308.5A CN116883760A (en) | 2023-07-28 | 2023-07-28 | Medical image classification and segmentation method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310941308.5A CN116883760A (en) | 2023-07-28 | 2023-07-28 | Medical image classification and segmentation method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116883760A true CN116883760A (en) | 2023-10-13 |
Family
ID=88271358
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310941308.5A Pending CN116883760A (en) | 2023-07-28 | 2023-07-28 | Medical image classification and segmentation method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116883760A (en) |
-
2023
- 2023-07-28 CN CN202310941308.5A patent/CN116883760A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107895367B (en) | Bone age identification method and system and electronic equipment | |
| WO2021017372A1 (en) | Medical image segmentation method and system based on generative adversarial network, and electronic equipment | |
| Solovyev et al. | 3D convolutional neural networks for stalled brain capillary detection | |
| US20220277572A1 (en) | Artificial intelligence-based image processing method, apparatus, device, and storage medium | |
| Tang et al. | CT image enhancement using stacked generative adversarial networks and transfer learning for lesion segmentation improvement | |
| CN110648331B (en) | Detection method for medical image segmentation, medical image segmentation method and device | |
| CN114549470B (en) | Hand bone critical area acquisition method based on convolutional neural network and multi-granularity attention | |
| CN115423754A (en) | Image classification method, device, equipment and storage medium | |
| Cui et al. | A gender classification method for Chinese mitten crab using deep convolutional neural network | |
| Yang et al. | RADCU-Net: Residual attention and dual-supervision cascaded U-Net for retinal blood vessel segmentation | |
| Deb et al. | LS-Net: A convolutional neural network for leaf segmentation of rosette plants | |
| CN115546466A (en) | Weak supervision image target positioning method based on multi-scale significant feature fusion | |
| Yu et al. | An improved YOLOX for detection in urine sediment images | |
| Kundur et al. | Pneumonia detection in chest x-rays using transfer learning and TPUs | |
| Wang et al. | A novel automatic acne detection and severity quantification scheme using deep learning | |
| CN112070181A (en) | Image stream-based cooperative detection method and device and storage medium | |
| Liu et al. | Interactive Dual Network With Adaptive Density Map for Automatic Cell Counting | |
| CN115762721A (en) | Medical image quality control method and system based on computer vision technology | |
| CN116228660A (en) | Method and device for detecting abnormal parts of chest film | |
| CN116883760A (en) | Medical image classification and segmentation method and system | |
| Guo et al. | Deep multi-task framework for optic disc and fovea detection | |
| Kalaivani et al. | A Deep Ensemble Model for Automated Multiclass Classification Using Dermoscopy Images | |
| Hussain et al. | RecU-Net++: Improved utilization of receptive fields in U-Net++ for skin lesion segmentation | |
| Sun et al. | Semi-supervised breast cancer pathology image segmentation based on fine-grained classification guidance | |
| Gou et al. | Artificial intelligence multiprocessing scheme for pathology images based on transformer for nuclei segmentation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |