CN113469962B - Feature extraction and image-text fusion method and system for cancer lesion detection - Google Patents
Feature extraction and image-text fusion method and system for cancer lesion detection Download PDFInfo
- Publication number
- CN113469962B CN113469962B CN202110705588.0A CN202110705588A CN113469962B CN 113469962 B CN113469962 B CN 113469962B CN 202110705588 A CN202110705588 A CN 202110705588A CN 113469962 B CN113469962 B CN 113469962B
- Authority
- CN
- China
- Prior art keywords
- image
- feature extraction
- text
- feature
- detection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000605 extraction Methods 0.000 title claims abstract description 116
- 238000001514 detection method Methods 0.000 title claims abstract description 69
- 230000003902 lesion Effects 0.000 title claims abstract description 61
- 206010028980 Neoplasm Diseases 0.000 title claims abstract description 36
- 201000011510 cancer Diseases 0.000 title claims abstract description 36
- 238000007500 overflow downdraw method Methods 0.000 title claims abstract description 13
- 230000004927 fusion Effects 0.000 claims abstract description 23
- 238000010606 normalization Methods 0.000 claims abstract description 14
- 238000012545 processing Methods 0.000 claims abstract description 11
- 238000007781 pre-processing Methods 0.000 claims abstract description 6
- 238000005070 sampling Methods 0.000 claims abstract description 6
- 239000013598 vector Substances 0.000 claims description 35
- 230000007246 mechanism Effects 0.000 claims description 20
- 238000011176 pooling Methods 0.000 claims description 19
- 238000000034 method Methods 0.000 claims description 18
- 230000006870 function Effects 0.000 claims description 12
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 claims description 10
- 230000004913 activation Effects 0.000 claims description 10
- 238000006243 chemical reaction Methods 0.000 claims description 9
- 238000010586 diagram Methods 0.000 claims description 8
- 230000008569 process Effects 0.000 claims description 8
- 201000010099 disease Diseases 0.000 claims description 5
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 claims description 5
- 230000003862 health status Effects 0.000 claims description 4
- 238000012216 screening Methods 0.000 claims description 4
- 230000001502 supplementing effect Effects 0.000 claims description 4
- 238000013528 artificial neural network Methods 0.000 claims description 3
- 238000004364 calculation method Methods 0.000 claims description 3
- 238000009432 framing Methods 0.000 claims description 3
- 238000012549 training Methods 0.000 claims description 3
- 230000002457 bidirectional effect Effects 0.000 claims description 2
- 210000004185 liver Anatomy 0.000 description 5
- 238000013527 convolutional neural network Methods 0.000 description 3
- 210000004027 cell Anatomy 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 210000000056 organ Anatomy 0.000 description 2
- 210000001015 abdomen Anatomy 0.000 description 1
- 230000003213 activating effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 210000002569 neuron Anatomy 0.000 description 1
- 238000002600 positron emission tomography Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10072—Tomographic images
- G06T2207/10081—Computed x-ray tomography [CT]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30096—Tumor; Lesion
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Radiology & Medical Imaging (AREA)
- Quality & Reliability (AREA)
- Apparatus For Radiation Diagnosis (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a feature extraction and image-text fusion method and a system for cancer lesion detection, which are used for acquiring CT image data and text data of a patient; preprocessing CT image data, and carrying out convolution operation and normalization processing on the CT image data in sequence to obtain an initial feature map F; adopting Darknet-53 network as basic network, and extracting features of CT image data; extracting text data characteristics of text data of a patient by adopting a BERT model; the feature map and the text feature are spliced, so that the fusion of the text feature and the image feature is realized, and the detection of the cancer lesion by the network is guided; and carrying out up-sampling operation and lateral connection on the fused image-text information to form different characteristic levels like a pyramid, applying corresponding detection heads on the different levels, and calculating the position and confidence value of a lesion area in the characteristic map to obtain the specific position of the lesion area in an original image so as to realize detection of the cancerous area.
Description
Technical Field
The invention belongs to the technical field of computer vision, and particularly relates to a feature extraction and image-text fusion method and system for cancer lesion detection.
Background
In the process of cancer lesion detection, the medical image can be used as an auxiliary means to effectively help detection personnel identify a lesion area so as to provide timely treatment for patients. The conventional medical image detection method needs medical knowledge and abundant experience of a clinician to analyze the medical image, but the problems of missed detection, false detection and the like are easy to exist. Furthermore, manually labeling a lesion in a medical image often requires a significant amount of time and effort. Therefore, the target detection method introduced into the computer vision technology detects the cancer lesion area, so that not only can the interference of human factors be effectively avoided, but also the detection precision of the cancer lesion area can be improved.
The traditional machine learning method mainly extracts relevant characteristics through probability density functions or gradient histograms and other information, so that the identification accuracy of the detector on a lesion area in a medical image is improved. However, the conventional machine learning method not only requires prior knowledge to perform manual intervention on the whole detection model, but also is difficult to effectively mine deep semantic features in the image.
The deep learning-based cancer lesion detection method mainly utilizes a Convolutional Neural Network (CNN) to extract characteristic information about cancer lesions, and detects cancer according to the characteristic information. On the one hand, the conventional convolutional neural network extracts feature information only according to the pixel difference, and the feature information of the lesion area cannot be fully mined due to the large difference among the sizes, shapes and densities of various lesion areas. On the other hand, a great number of cancer lesion detection methods based on deep learning only focus on detection of medical image layers, but neglect various lesion stages and have larger characteristic differences.
Disclosure of Invention
In order to solve the defects in the prior art, the invention provides a feature extraction and image-text fusion method and system for cancer lesion detection, which are used for solving the problems of insufficient image features extracted by a detection network and low detection precision in the prior art.
The technical scheme adopted by the invention is as follows:
a feature extraction and image-text fusion method for cancer lesion detection comprises the following steps:
S1, acquiring patient personal information recorded by a clinician, CT image data of a patient and a text report corresponding to the CT image data;
s2, preprocessing CT image data so that the format of the CT image data can be directly fed into a neural network as input for training or prediction; performing convolution operation and normalization processing on CT image data in sequence to obtain an initial feature map F;
s3, adopting Darknet-53 network as a basic network, and extracting features of CT image data;
S4, extracting text data characteristics of a text report form corresponding to the patient personal information and the CT image data by adopting a BERT model;
S5, splicing the feature map obtained in the S3 with the text features to realize fusion of the text features and the image features, and guiding a network to detect cancer lesions;
s6, carrying out up-sampling operation and lateral connection on the fused image-text information to form different feature levels like a pyramid, applying corresponding detection heads on the different feature levels, and calculating the position and confidence value of a lesion area in the feature map to obtain the specific position of the lesion area in an original image so as to realize detection of the cancerous area.
Further, the Darknet-53 network is formed by alternately forming a plurality of downsampling layers and Residual blocks, wherein one downsampling layer and one Residual Block are called a feature extraction Block, and a CT image data feature extraction network is formed by 5 feature extraction blocks; the input of each feature extraction block is the output of the previous feature extraction block, and the current output is the input of the next feature extraction block.
Further, the times of performing the feature extraction block operation in the 5 feature extraction blocks are sequentially 1,2, 8 and 4;
further, the feature extraction block operation of each feature extraction block is as follows:
S3.1, performing downsampling operation on the feature map in a downsampling layer, and gradually reducing the size of the feature map;
s3.2, screening and weighting the feature images in the channel dimension, and highlighting information in the effective area by using pooling operation along the channel direction; expressed as: M c (F) represents performing feature extraction operation on F in the channel dimension;
S3.3, supplementing position information in a space dimension, learning a space weight graph by utilizing the relation among different space positions, and modeling by utilizing the relation of features in space, wherein the position relation information which cannot be better acquired by a channel attention mechanism is supplemented; expressed as: M s (F ') denotes performing feature extraction operation on F' in spatial dimension,/> Representing element-by-element multiplication;
S3.4, carrying out convolution operation on the feature map F' obtained in the step S3.3, wherein the convolution kernel size is 3*3, the step length is 1, the normalization is carried out by adopting Batch Normalization, and the activation function is LeakyReLu.
Further, the BERT model in S4 is a multi-layer bi-directional transform encoder, which includes an encoder mechanism and a decoder mechanism, the encoder receiving text as input, the decoder being primarily responsible for the prediction; the transducer builds up an encoder and decoder of 6 layers each, for a total of 12 layers of encoder-decoders.
Further, the transducer does not need to loop, but processes all words or symbols in the sequence in parallel, and combines the context with the far words by using a self-attention mechanism, so that the context information is fully considered; the calculation rule of the attention mechanism is as follows: for each vocabulary, there are 3 different vectors, namely a Query vector Q, a Key vector K and a Value vector V; calculating a score value for each vector, wherein score = Q x K; next, the score is subjected to a softmax activation function, and the vector V is multiplied by the softmax point to obtain a weighted score V of each vector; several attention mechanisms are overlapped to obtain a Multi-Head Ateention module, an output result Z= ΣVis obtained after addition, and then normalization processing is carried out. Wherein the input of the decoder comprises the output of the decoder and the output of the decoder of the upper layer, and the final result is respectively output after being activated by Linear and Softmax.
Further, 3 detection frames are set in each unit of the feature map, so the prediction vector length of each unit is: 3 (4+4+1) =27, where the first 4 corresponds to 4 classes of the dataset, 3 represents the number of detection frames per cell, the second 4 is the 4 position offsets per detection frame, 1 is the confidence value that each detection frame contains the target, and the final image feature is represented as a vector of 1x 27 dimensions. And the acquired text features are expressed as a 1*N-dimensional vector; finally, a vector with 1 (27+N) dimension is obtained and is the image-text information after fusion and is used for the subsequent detection network.
A feature extraction and image-text fusion system for cancer lesion detection comprises a data acquisition module, a feature extraction module for CT images and text data, an image-text fusion module and an image-text information detection module;
the data acquisition module is used for acquiring personal information of a patient, a CT image of the patient and a text report corresponding to the CT image; patient personal information includes information such as disease information, health status, etc. of the patient;
The feature extraction module of the CT image and the text data comprises a CT image format conversion unit, a CT image feature extraction unit and a text data feature extraction unit;
The characteristic diagram and the text characteristic are spliced in the image-text fusion module;
The image-text information detection module calculates coordinates of a lesion area and a confidence value according to the guidance of the text characteristics, wherein the coordinates are used for framing the position of the lesion area in an original image, the confidence value is used for judging the type of the lesion, and a final result is output;
Further, the CT image format conversion unit is image format conversion software and is used for converting the CT image format acquired by the scanner so as to facilitate subsequent processing; the CT image feature extraction unit is internally provided with a CT image data feature extraction network, the CT image data feature extraction network adopts Darknet-53 networks as a basic network, the Darknet-53 networks are composed of 5 feature extraction blocks, and each feature extraction Block comprises a downsampling layer and a Residual Block; taking the CT image after format conversion as the input of a first feature extraction block, and taking the output of the previous feature extraction block in the five feature extraction blocks as the input of the next feature extraction block; the output of the last feature extraction block is a feature map used for fusing with text features; the text data feature extraction unit adopts a BERT model, inputs text data into the BERT model, establishes a context relation between words, and acquires text features based on sentence level.
The invention has the beneficial effects that:
The feature extraction and image-text fusion method and the system for cancer lesion detection introduce feature description information about each lesion stage in the cancer lesion detection method, so that the identification accuracy of a cancer lesion region can be improved, and the severity of the cancer lesion can be effectively divided.
In addition, the feature extraction and image-text fusion method and system for cancer lesion detection adopt an extraction method combining channel dimension and space dimension to improve the identification capability of lesion areas with large differences, and use BERT network to perform feature fusion of images and texts by combining text description information of each lesion stage, so that effective detection of cancer lesion areas and effective judgment of cancer stages are performed.
Drawings
FIG. 1 is an overall flow chart of the method of the present invention;
FIG. 2 is a schematic workflow diagram of a CT image feature extraction module according to the method of the present invention;
FIG. 3 is a schematic diagram of a network structure of a text feature extraction module of the method of the present invention;
FIG. 4 is a schematic diagram of an encoder-decoder structure of the method of the present invention;
FIG. 5 is an expanded schematic view of the encoder-decoder structure of FIG. 4;
FIG. 6 is a schematic diagram of the structure of a detection module according to the method of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
The following describes in detail a feature extraction and image-text fusion method for cancer lesion detection according to the present invention with reference to fig. 1 to 5, and the CT image data provided by the present invention are all examples of liver CT images.
S1, a data acquisition stage, which specifically comprises acquisition of CT image data and text data, wherein a high-precision German Siemens 52-loop scanner (Positron Emission Tomography, PET-CT) is adopted to generate a large number of abdomen flat scan slices; in the process of generating CT image data, a text report is generated, and the text report and the personal information (including disease information, health condition, etc.) of the patient recorded by the clinician form a required text data set. CT image data and text data of each patient are in one-to-one correspondence and stored in a database so as to be used by a subsequent image feature extraction module and a text feature extraction module.
S2, preprocessing CT image data, namely preprocessing the CT image data into a format which accords with the acceptance of the image feature extraction network. The scan slice format directly generated by the machine is DICOM (. Dcm), which cannot be directly fed as input into the neural network for training or prediction, which is converted into the. Jpg or. Png image format by means of a Python correlation library, or into a two-dimensional array; during the transformation process, attention should be paid to the limitation of window level and window width (i.e. limitation of Hunter unit HU value) corresponding to the liver (or other organs and tissues) so as to obtain the contrast ratio with the surrounding organs to the greatest extent (make the liver appear as much as possible); for the liver, according to the experience of the professional radiologist, the HU value is limited in the range of [ -100, 240], -100 and 240 are window levels, and the distance between two end points is window width; the defined formula is shown below:
Where HU i,j represents the HU value at pixel (i, j) and PV i,j represents the pixel value of the original scan slice at pixel (i, j) where slope and intercept are the slope and intercept, respectively, of the corresponding linear operation.
S3, in a CT image data feature extraction stage, a Darknet-53 network is adopted as a basic network, and the Darknet-53 network is formed by alternately a plurality of downsampling layers and Residual blocks; specifically, a downsampling layer and a Residual Block are called a feature extraction Block, and a CT image data feature extraction network is formed by 5 feature extraction blocks. The downsampling layer is responsible for gradually reducing the size of a feature map, the Residual Block is responsible for carrying out feature extraction on an input image, extracting high-level semantic information with more expressive capacity and robustness in the image and used for detecting a subsequent module, the input of each feature extraction Block is the output of a previous feature extraction Block, and the current output is used as the input of a next feature extraction Block. The specific structure is shown in fig. 2.
In this embodiment, a convolution operation is performed on the CT image data with the size of 512×512×3 after preprocessing, the convolution kernel size is 3*3, the step size is 1, the normalization processing is performed by Batch Normalization, the activation function is LeakyReLu, the number of output channels is 32, and the feature map F size is 512×512×32. The above feature map is input into a first feature extraction block, and the specific operation steps in the feature extraction block are as follows:
s3.1, performing downsampling operation on the feature map in a downsampling layer, wherein the process is as follows: the convolution kernel size is 3*3, the step size is 2, the filling is 1, the output channel number is 64 channels, the feature map size is reduced to 1/2, the size is 256×256×64, and then the convolution operation of 1*1 is performed.
And S3.2, screening and weighting the feature map in the channel dimension. Information in the active area can be highlighted using a pooling operation along the channel direction. First, a pooling operation is required on the acquired feature map to pay attention to the main features and filter redundant information. In the pooling operation, the global average pooling has feedback on each characteristic value on the characteristic map, so that the image background information can be effectively learned; maximum pooling can collect salient feature value information. Thus, simultaneous global average pooling and global maximum pooling may be a good aid for feature screening. The specific formula is as follows:
Wherein M c (F) represents a feature extraction operation on F in the channel dimension, sigma is a Sigmoid activation function, avgpool (F) is an average pooling operation, maxpool (F) is a maximum pooling operation, F c avg, And the one-dimensional vectors obtained by global average pooling and maximum pooling of each channel of the feature map F are respectively obtained.
The adopted multi-layer perceptron MLP (Multilayer Perceptron) is a three-layer perceptron network shared by vectors F c avg、Fc max, the number of neurons in the middle layer is C/r, C is the channel number of the feature map, r is the compression rate, r=16 is set, W 0∈RC/r×C,W1∈RC×C/r is the weight of the three-layer perceptron, sigma is a Sigmoid activation function, and a ReLU (RECTIFIED LINEAR Unit) activation function is used for improving the nonlinearity degree of the network.
And S3.3, supplementing the position information in the space dimension. The spatial weight map is learned by utilizing the relation among different spatial positions, so that the method can be used for supplementing the position relation information which cannot be better acquired by a channel attention mechanism, and modeling can be performed by utilizing the relation of the characteristics in space. The location information of deep features can be further enhanced by introducing a spatial attention mechanism in the form of a residual network. Taking the feature map enhanced by the channel attention mechanism as input, carrying out 1-dimensional average pooling and 1-dimensional maximum pooling on the channel to obtain two feature descriptions of 1 XH XW, splicing the two feature descriptions together, activating the feature descriptions through a sigmoid function, obtaining space weights through a convolution kernel of 7X 7, and multiplying the space weights with the input feature map F 'by using a jump connection mode of a residual network to obtain a feature map F'. The formula is as follows:
Wherein M s (F ') represents a feature extraction operation performed on F' in a spatial dimension, avgpool (F ') is an average pooling operation, maxpool (F') is a maximum pooling operation, and F 'c avg、F'x max∈R1×H×W is a two-dimensional feature map of two single channels obtained by global average pooling and maximum pooling of the feature map F', respectively. f 7×7 denotes a convolution operation with a convolution kernel size of 7 x 7. F' is the characteristic diagram after S3.2 treatment.
According to steps 3.2 and 3.3, the two steps can be summarized as follows:
Wherein F epsilon R C×H×W is an input feature map, R is a real number set, and H, W, C respectively represents the length, width and channel number of the feature map; f' is a feature map after channel dimension improvement; f' is a feature map after the spatial dimension is lifted; Representing element-wise multiplication. M c (F) represents a feature extraction operation on F in the channel dimension. M s (F ') represents a feature extraction operation on F' in the spatial dimension.
And S3.4, carrying out convolution operation on the feature map F' obtained in the step. The convolution kernel size is 3*3, the step length is 1, the normalization is carried out by adopting Batch Normalization, the activation function is LeakyReLu, the output channel number is 64 channels, and the feature map size is 256×256×64.
The above operations are repeated by the 5 feature extraction blocks, and the number of times the 5 feature extraction blocks are operated is sequentially 1, 2, 8, 4. And obtaining a feature map through the last feature extraction, namely a feature map used for fusing with text features.
S4, extracting text data characteristics. In order to splice and fuse with the feature map, the text information needs to be converted into corresponding text features. The text data includes disease information, health status, and examination report of CT image of the corresponding patient. The text feature extraction module consists of an input layer E1-EN, a plurality of encoder Transformer layers Trm and an output layer T1-TN. The specific structure is shown in fig. 3.
In this embodiment, the text feature extraction uses BERT (Bidirectional Encoder Representations from Transformers) model, which is a multi-layer bi-directional transducer encoder. The transducer encoder comprises an encoder decoder mechanism and a decoder mechanism, wherein the encoder receives text as input and the decoder is mainly responsible for the prediction result. The transducer builds up an encoder and decoder of 6 layers each, for a total of 12 layers of encoder-decoders. The specific structure of the encoder-decoder is shown in FIG. 4.
In the above embodiment, the transducer does not need to loop, but instead processes all words or symbols in the sequence in parallel, while combining context with far words using a self-attention mechanism, taking full account of the context information. A pair of encoder-decoder implementation steps are shown in fig. 5. The structure mainly comprises a plurality of attention modules, wherein the calculation rule of the attention mechanism is as follows: for each vocabulary, there are 3 different vectors, namely a Query vector Q, a Key vector K and a Value vector V; calculating a score value for each vector, wherein score = Q x K; next, the score is subjected to a softmax activation function, and the vector V is multiplied by the softmax point to obtain a weighted score V of each vector; several attention mechanisms (e.g., 8) are superimposed to obtain a Multi-Head Ateention module, and after addition, an output result z= Σvis obtained, and then normalization processing is performed. Wherein the input of the decoder comprises the output of the decoder and the output of the decoder of the upper layer, and the final result is respectively output after being activated by Linear and Softmax. The whole procedure involves several jump connections as indicated by the dashed arrow in fig. 5. Text data is input into the BERT model, a context relation between words is established, text features are obtained based on sentence levels, and generalization capability of the text feature extraction model is further improved.
S5, a picture-text fusion stage. And (3) performing simple splicing operation on the feature map and the text features obtained in the step (S3) to realize fusion of the text features and the image features and guide the network to detect liver lesions. The specific process is as follows: 3 detection frames are arranged in each unit of the feature map, so that the length of the prediction vector of each unit is as follows: 3 (4+4+1) =27, where the first 4 corresponds to 4 classes of the dataset, 3 represents the number of detection frames per cell, the second 4 is the 4 position offsets per detection frame, 1 is the confidence value that each detection frame contains the target, and the final image feature is represented as a vector of 1 x 27 dimensions. And the acquired text feature is represented as a 1*N-dimensional vector. Finally, a vector with 1 (27+N) dimension is obtained and is the image-text information after fusion and is used for the subsequent detection network.
And S6, detecting based on the fused graphic information. And after the image-text fusion stage is finished, carrying out up-sampling operation and lateral connection on the fused image-text information to form different feature levels similar to a pyramid, wherein the up-sampling operation corresponds to a down-sampling stage during feature extraction.
Applying corresponding detection heads on different levels, and calculating the position and confidence value of a lesion region in the feature map to obtain the specific position of the lesion region in the original image so as to realize detection of the cancerous region; aiming at adapting to cancerous regions with different sizes, traversing to wider and more suspected cancerous regions; fig. 6 is an illustration of the internal operation of each pyramid level. The detection head mentioned in the application can adopt the scheme disclosed in the sections 21-2.2 of ' YOLOv3: AN INCREMENTAL Improvement ', joseph Redmon, ALI FARHADI, university of Washington '. In order to realize the method, the application also provides a feature extraction and image-text fusion system for cancer lesion detection, which comprises a data acquisition module, a feature extraction module for CT images and text data, an image-text fusion module and an image-text information detection module;
the data acquisition module is used for acquiring personal information of a patient, a CT image of the patient and a text report corresponding to the CT image; patient personal information includes information such as disease information, health status, etc. of the patient; CT images and text reports of patients may be acquired using a germany siemens 52-ring scanner.
The feature extraction module of the CT image and the text data comprises a CT image format conversion unit, a CT image feature extraction unit and a text data feature extraction unit; specifically, the CT image format conversion unit is image format conversion software and is used for converting the CT image format acquired by the scanner so as to facilitate subsequent processing. The CT image feature extraction unit is internally provided with a CT image data feature extraction network, the CT image data feature extraction network adopts Darknet-53 networks as basic networks, the Darknet-53 networks are composed of 5 feature extraction blocks, and each feature extraction Block comprises a downsampling layer and a Residual Block; taking the CT image after format conversion as the input of a first feature extraction block, and taking the output of the previous feature extraction block in the five feature extraction blocks as the input of the next feature extraction block; the output of the last feature extraction block is the feature map used for fusing with the text features. The text data feature extraction unit adopts a BERT model, inputs text data into the BERT model, establishes a context relation between words, and acquires text features based on sentence level.
The characteristic diagram and the text characteristic are spliced in the image-text fusion module;
And the image-text information detection module calculates coordinates of the lesion area and a confidence value according to the guidance of the text characteristics, wherein the coordinates are used for framing the position of the lesion area in the original image, the confidence value is used for judging the type of the lesion area, and a final result is output.
The detection of the cancer lesions can be completed through the processing of the modules, the feature extraction of CT images and text data and the fusion of the two features are completed, the detection of the cancer lesions is finally completed, the detection accuracy is improved, and the diagnosis of doctors is assisted.
The above embodiments are merely for illustrating the design concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement the same, the scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications according to the principles and design ideas of the present invention are within the scope of the present invention.
Claims (7)
1. The feature extraction and image-text fusion method for cancer lesion detection is characterized by comprising the following steps of:
S1, acquiring patient personal information recorded by a clinician, CT image data of a patient and a text report corresponding to the CT image data;
s2, preprocessing CT image data so that the format of the CT image data can be directly fed into a neural network as input for training or prediction; performing convolution operation and normalization processing on CT image data in sequence to obtain an initial feature map F;
s3, adopting Darknet-53 network as a basic network, and extracting features of CT image data;
S4, extracting text data characteristics of a text report form corresponding to the patient personal information and the CT image data by adopting a BERT model; the BERT model in S4 is a multi-layer bidirectional transducer encoder, the transducer encoder comprises an encoder mechanism and a decoder mechanism, the encoder is used as an input to receive text, and the decoder is mainly responsible for predicting results; the transducer builds up an encoder and a decoder of 6 layers each, and a total of 12 layers of encoder-encoders; the Transformer does not need to circulate, but processes all words or symbols in the sequence in parallel, combines the context with the far words by using a self-attention mechanism, and fully considers the context information; the calculation rule of the attention mechanism is as follows: for each vocabulary, there are 3 different vectors, namely a Query vector Q, a Key vector K and a Value vector V; calculating a score value for each vector, wherein score = Q x K; next, the score is subjected to a softmax activation function, and the vector V is multiplied by the softmax point to obtain a weighted score V of each vector; several attention mechanisms are overlapped to obtain a Multi-Head Ateention module, an output result Z=ΣV is obtained after addition, and then normalization processing is carried out. Wherein the input of the decoder comprises the output of the decoder and the output of the decoder of the last layer, and the final result is respectively activated by Linear and Softmax and then output;
S5, splicing the feature map obtained in the S3 with the text features to realize fusion of the text features and the image features, and guiding a network to detect cancer lesions;
s6, carrying out up-sampling operation and lateral connection on the fused image-text information to form different feature levels like a pyramid, applying corresponding detection heads on the different feature levels, and calculating the position and confidence value of a lesion area in the feature map to obtain the specific position of the lesion area in an original image so as to realize detection of the cancerous area.
2. The feature extraction and image-text fusion method for cancer lesion detection according to claim 1, wherein the Darknet-53 network is composed of a plurality of downsampling layers and Residual blocks, wherein one downsampling layer and one Residual Block are called one feature extraction Block, and the 5 feature extraction blocks form a CT image data feature extraction network; the input of each feature extraction block is the output of the previous feature extraction block, and the current output is the input of the next feature extraction block.
3. The feature extraction and image-text fusion method for cancer lesion detection according to claim 2, wherein the number of feature extraction block operations performed in 5 feature extraction blocks is sequentially 1,2, 8, 4.
4. A method of feature extraction and fusion for cancer lesion detection according to claim 2 or 3, wherein each feature extraction block operation of each feature extraction block comprises the steps of:
S3.1, performing downsampling operation on the feature map in a downsampling layer, and gradually reducing the size of the feature map;
s3.2, screening and weighting the feature images in the channel dimension, and highlighting information in the effective area by using pooling operation along the channel direction; expressed as: M c (F) represents performing feature extraction operation on F in the channel dimension;
S3.3, supplementing position information in a space dimension, learning a space weight graph by utilizing the relation among different space positions, and modeling by utilizing the relation of features in space, wherein the position relation information which cannot be better acquired by a channel attention mechanism is supplemented; expressed as: M s (F ') denotes performing feature extraction operation on F' in spatial dimension,/> Representing element-by-element multiplication;
S3.4, carrying out convolution operation on the feature map F' obtained in the step S3.3, wherein the convolution kernel size is 3*3, the step length is 1, the normalization is carried out by adopting Batch Normalization, and the activation function is LeakyReLu.
5. The method for feature extraction and image-text fusion for cancer lesion detection according to claim 1, wherein 3 detection frames are set in each unit of the feature map, so that the length of the prediction vector of each unit is as follows: 3 (4+4+1) =27, where the first 4 corresponds to 4 classes of the dataset, 3 represents the number of detection frames per cell, the second 4 is the 4 position offsets per detection frame, 1 is the confidence value that each detection frame contains the target, and the final image feature is represented as a vector of 1 x 27 dimensions. And the acquired text features are expressed as a 1*N-dimensional vector; finally, a vector with 1 (27+N) dimension is obtained and is the image-text information after fusion and is used for the subsequent detection network.
6. A feature extraction and image-text fusion system for cancer lesion detection based on the feature extraction and image-text fusion method for cancer lesion detection of claim 1, which is characterized by comprising a data acquisition module, a feature extraction module for CT images and text data, an image-text fusion module and an image-text information detection module;
the data acquisition module is used for acquiring personal information of a patient, a CT image of the patient and a text report corresponding to the CT image; patient personal information includes information such as disease information, health status, etc. of the patient;
The feature extraction module of the CT image and the text data comprises a CT image format conversion unit, a CT image feature extraction unit and a text data feature extraction unit;
The characteristic diagram and the text characteristic are spliced in the image-text fusion module;
And the image-text information detection module calculates coordinates of the lesion area and a confidence value according to the guidance of the text characteristics, wherein the coordinates are used for framing the position of the lesion area in the original image, the confidence value is used for judging the type of the lesion area, and a final result is output.
7. The feature extraction and image-text fusion system for cancer lesion detection according to claim 6, wherein the CT image format conversion unit is image format conversion software for converting a CT image format acquired by a scanner for subsequent processing; the CT image feature extraction unit is internally provided with a CT image data feature extraction network, the CT image data feature extraction network adopts Darknet-53 networks as a basic network, the Darknet-53 networks are composed of 5 feature extraction blocks, and each feature extraction Block comprises a downsampling layer and a Residual Block; taking the CT image after format conversion as the input of a first feature extraction block, and taking the output of the previous feature extraction block in the five feature extraction blocks as the input of the next feature extraction block; the output of the last feature extraction block is a feature map used for fusing with text features; the text data feature extraction unit adopts a BERT model, inputs text data into the BERT model, establishes a context relation between words, and acquires text features based on sentence level.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110705588.0A CN113469962B (en) | 2021-06-24 | 2021-06-24 | Feature extraction and image-text fusion method and system for cancer lesion detection |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110705588.0A CN113469962B (en) | 2021-06-24 | 2021-06-24 | Feature extraction and image-text fusion method and system for cancer lesion detection |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113469962A CN113469962A (en) | 2021-10-01 |
| CN113469962B true CN113469962B (en) | 2024-05-14 |
Family
ID=77872618
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110705588.0A Active CN113469962B (en) | 2021-06-24 | 2021-06-24 | Feature extraction and image-text fusion method and system for cancer lesion detection |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113469962B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20190131206A (en) * | 2018-05-16 | 2019-11-26 | 한양대학교 산학협력단 | Deep learning-based object detection method and system utilizing global context feature of image |
| CN111667489A (en) * | 2020-04-30 | 2020-09-15 | 华东师范大学 | Cancer hyperspectral image segmentation method and system based on double-branch attention deep learning |
| CN112801168A (en) * | 2021-01-25 | 2021-05-14 | 江苏大学 | Tumor image focal region prediction analysis method and system and terminal equipment |
-
2021
- 2021-06-24 CN CN202110705588.0A patent/CN113469962B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20190131206A (en) * | 2018-05-16 | 2019-11-26 | 한양대학교 산학협력단 | Deep learning-based object detection method and system utilizing global context feature of image |
| CN111667489A (en) * | 2020-04-30 | 2020-09-15 | 华东师范大学 | Cancer hyperspectral image segmentation method and system based on double-branch attention deep learning |
| CN112801168A (en) * | 2021-01-25 | 2021-05-14 | 江苏大学 | Tumor image focal region prediction analysis method and system and terminal equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113469962A (en) | 2021-10-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110110808B (en) | Method and device for performing target labeling on image and computer recording medium | |
| CN113223005B (en) | Thyroid nodule automatic segmentation and grading intelligent system | |
| CN110648331B (en) | Detection method for medical image segmentation, medical image segmentation method and device | |
| CN112132878B (en) | End-to-end brain nuclear magnetic resonance image registration method based on convolutional neural network | |
| CN111738363A (en) | Alzheimer disease classification method based on improved 3D CNN network | |
| Liu et al. | Recent progress in transformer-based medical image analysis | |
| CN115018809A (en) | Target area segmentation and identification method and system of CT image | |
| CN112819831A (en) | Segmentation model generation method and device based on convolution Lstm and multi-model fusion | |
| CN116563533A (en) | Medical image segmentation method and system based on target position priori information | |
| US11494908B2 (en) | Medical image analysis using navigation processing | |
| CN113538363A (en) | Lung medical image segmentation method and device based on improved U-Net | |
| CN112420170B (en) | Method for improving image classification accuracy of computer aided diagnosis system | |
| CN111340807B (en) | Nidus positioning core data extraction method, system, electronic equipment and storage medium | |
| CN113469962B (en) | Feature extraction and image-text fusion method and system for cancer lesion detection | |
| CN117314823A (en) | Attention mechanism-based knee joint MRI intelligent analysis method and device | |
| CN115937590A (en) | Skin disease image classification method with CNN and Transformer fused in parallel | |
| CN115471512A (en) | Medical image segmentation method based on self-supervision contrast learning | |
| CN116258732A (en) | Esophageal cancer tumor target region segmentation method based on cross-modal feature fusion of PET/CT images | |
| CN116524178A (en) | MRI image tissue segmentation method and imaging method based on semi-supervision | |
| CN114119558B (en) | Method for automatically generating nasopharyngeal carcinoma image diagnosis structured report | |
| Li et al. | Image analysis and diagnosis of skin diseases-a review | |
| Wu et al. | Swin Transformer based benign and malignant pulmonary nodule classification | |
| Lokhande et al. | Lung CT image segmentation: a convolutional neural network approach | |
| Dai et al. | A Generative Data Augmentation Trained by Low-quality Annotations for Cholangiocarcinoma Hyperspectral Image Segmentation | |
| Zhang et al. | UTSN-net: medical image semantic segmentation model based on skip non-local attention module |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |