CN114463583B - Deep hashing method for pneumonia CT image classification - Google Patents
Deep hashing method for pneumonia CT image classification Download PDFInfo
- Publication number
- CN114463583B CN114463583B CN202210093092.7A CN202210093092A CN114463583B CN 114463583 B CN114463583 B CN 114463583B CN 202210093092 A CN202210093092 A CN 202210093092A CN 114463583 B CN114463583 B CN 114463583B
- Authority
- CN
- China
- Prior art keywords
- image
- pneumonia
- hash
- bilinear
- branch
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 206010035664 Pneumonia Diseases 0.000 title claims abstract description 113
- 238000000034 method Methods 0.000 title claims abstract description 27
- 238000002591 computed tomography Methods 0.000 claims abstract description 153
- 210000004072 lung Anatomy 0.000 claims abstract description 41
- 230000006870 function Effects 0.000 claims abstract description 31
- 238000012549 training Methods 0.000 claims abstract description 30
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 19
- 238000012360 testing method Methods 0.000 claims abstract description 11
- 239000013598 vector Substances 0.000 claims description 41
- 239000011159 matrix material Substances 0.000 claims description 36
- 238000011176 pooling Methods 0.000 claims description 22
- 238000013507 mapping Methods 0.000 claims description 6
- 208000025721 COVID-19 Diseases 0.000 claims description 4
- 238000010606 normalization Methods 0.000 claims description 3
- 238000007781 pre-processing Methods 0.000 claims description 3
- 238000012545 processing Methods 0.000 abstract description 3
- 239000000284 extract Substances 0.000 description 5
- 244000052616 bacterial pathogen Species 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 208000024891 symptom Diseases 0.000 description 2
- 206010001889 Alveolitis Diseases 0.000 description 1
- 201000004813 Bronchopneumonia Diseases 0.000 description 1
- 102000009123 Fibrin Human genes 0.000 description 1
- 108010073385 Fibrin Proteins 0.000 description 1
- BWGVNKXGVNDBDI-UHFFFAOYSA-N Fibrin monomer Chemical compound CNC(=O)CNC(=O)CN BWGVNKXGVNDBDI-UHFFFAOYSA-N 0.000 description 1
- 206010048612 Hydrothorax Diseases 0.000 description 1
- 208000032923 Lobar pneumonia Diseases 0.000 description 1
- 206010030113 Oedema Diseases 0.000 description 1
- 206010057190 Respiratory tract infections Diseases 0.000 description 1
- 208000027418 Wounds and injury Diseases 0.000 description 1
- 230000003187 abdominal effect Effects 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000002146 bilateral effect Effects 0.000 description 1
- 210000000621 bronchi Anatomy 0.000 description 1
- 210000003123 bronchiole Anatomy 0.000 description 1
- 230000006378 damage Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 1
- 208000035475 disorder Diseases 0.000 description 1
- 201000001155 extrinsic allergic alveolitis Diseases 0.000 description 1
- 210000000416 exudates and transudate Anatomy 0.000 description 1
- 229950003499 fibrin Drugs 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- PCHJSUWPFVWCPO-UHFFFAOYSA-N gold Chemical compound [Au] PCHJSUWPFVWCPO-UHFFFAOYSA-N 0.000 description 1
- 208000022098 hypersensitivity pneumonitis Diseases 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000008595 infiltration Effects 0.000 description 1
- 238000001764 infiltration Methods 0.000 description 1
- 230000002757 inflammatory effect Effects 0.000 description 1
- 208000014674 injury Diseases 0.000 description 1
- 230000003902 lesion Effects 0.000 description 1
- 210000003097 mucus Anatomy 0.000 description 1
- 231100000915 pathological change Toxicity 0.000 description 1
- 230000036285 pathological change Effects 0.000 description 1
- 210000004911 serous fluid Anatomy 0.000 description 1
- 230000008719 thickening Effects 0.000 description 1
- 238000002834 transmittance Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2413—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on distances to training or reference patterns
- G06F18/24133—Distances to prototypes
- G06F18/24137—Distances to cluster centroïds
- G06F18/2414—Smoothing the distance, e.g. radial basis function networks [RBFN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10072—Tomographic images
- G06T2207/10081—Computed x-ray tomography [CT]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30061—Lung
Abstract
The invention relates to the technical field of medical image processing, in particular to a deep hash method for classifying CT (computed tomography) images of pneumonia. The invention firstly establishes a pneumonia CT image data set, unifies the size of the pneumonia CT image, and divides the pneumonia CT image data set into a training set T r And test set T e The method comprises the steps of carrying out a first treatment on the surface of the Then constructing a deep hash network model, and calculating the similarity loss L according to the hash codes S And contrast loss L cl Constructing a total loss function L; secondly, introducing a multi-task hash training strategy, optimizing a loss function L by using an alternate learning algorithm, and storing a deep hash network model; and finally, reading the test set to classify CT images. The deep hashing method has the advantages that the method can accurately find out the tiny differences among CT images of different lungs, so that the training model greatly reduces the storage space and the training time, effectively improves the classifying efficiency of large-scale pneumonia CT images, fully plays the characteristic extracting advantage of the bilinear convolutional neural network on fine-granularity CT images, and effectively improves the identifying accuracy and the generalization robustness.
Description
Technical Field
The invention relates to the technical field of medical image processing, in particular to a deep hash method for classifying CT (computed tomography) images of pneumonia.
Background
The lung CT image is a gold standard for diagnosing pneumonia, and different types of pneumonia appear differently on the lung CT image. Bilateral chest of a lung CT image of a normal person is symmetrical, lung transmittance and lung texture are normal, thickening and disorder are avoided, and obvious abnormal density shadows are avoided in double lung fields; the CT image of the lung of the patient with COVID-19 has small patch-shaped shadows of the outer belt of the lung and changes of the interstitial mass of the lung at early stage, so that the images develop into glass grinding shadows and infiltration shadows of two lungs, and lung real changes, hydrothorax and abdominal dropsy, pathological changes of the lung and real changes of the lung with different degrees can also occur when the real changes mainly show diffuse alveolar injury and exudative alveolitis, serous fluid, fibrin exudates and transparent films can be formed in alveolar cavities, and bronchi and bronchioles also see mucus plugs; for the lung CT image of a patient suffering from common pneumonia, if the common community acquired pathogenic bacteria cause lobar pneumonia, the common community acquired pathogenic bacteria are often represented as localized patch-like density heightened images distributed in lung lobes or lung segments in imaging, the patient often has clear respiratory tract infection history, and if the common community acquired pathogenic bacteria cause inflammatory exudation of the lung of the patient suffering from lobular pneumonia, the patients often take the patch-like exudation as the main part.
However, in the current large data environment, tens of thousands of CT images are produced daily, and how to effectively classify them is a major challenge to be solved. Traditionally, the subjective judgment of doctors is usually needed, so that a great deal of manpower is consumed, a great deal of time is wasted, misjudgment and misjudgment are caused more seriously, and the life health of patients is affected to a certain extent.
The rise of deep learning has driven the development of the medical image processing field, especially in the field of lung CT image classification. At present, the classification of lung CT images mostly adopts convolutional neural networks CNNs to extract deep features of the images, so as to distinguish different types. However, differences between CT images of the lung tend to occur in the lesion region, while other regions are substantially similar, that is, the similarity of medical images of the same region tends to be very large, which has a certain influence on the accuracy of classification.
Under the background, the invention proposes that fine granularity features of lung CT images are extracted by adopting a bilinear convolutional neural network BCNN, so as to find out tiny differences among different lung CT images, and further, the extracted depth features are mapped into a binary hamming space, thereby classifying the depth features. Therefore, not only the classification accuracy is improved, but also the classification time is saved.
Disclosure of Invention
The invention aims to solve the problems and provides a deep hash method for classifying CT images of pneumonia.
In order to achieve the above purpose, the present invention adopts the following technical scheme:
a deep hashing method for pneumonia CT image classification, comprising the steps of:
s10: establishing a pneumonia CT image data set;
s20: data preprocessing, namely firstly carrying out data enhancement and expansion on a data set, and dividing the data set into training sets T according to the proportion of 80 percent and 20 percent r ＝(x 1 ,x 2 ,...,x n ) N=1, 2, where, N and test set T e ＝(y 1 ,y 2 ,...,y m ) M=1, 2, m, the size of the pneumonia CT image is then uniformly adjusted to 224 x 224, the number of channels is 1, and 3 types of CT images are contained in the dataset, namely the lung CT image of a normal person, the lung CT image of a COVID-19 patient and the lung CT image of a common pneumonia patient, and finally constructing a training set T r Similarity matrix S of (1), wherein
And S is ij ∈R N×N ,i,j＝1,2,...,N；
S30: constructing a deep hash network model, wherein the model comprises two modules of bilinear feature learning and hash code learning, and when the model is trained, firstly, fine-granularity features of a pneumonia CT image are extracted by using a bilinear convolutional neural network BCNN, and then the extracted fine-granularity features are input into a hash code learning module, so that the fine-granularity features of the corresponding pneumonia CT image are mapped into binary hash codes;
s40: the hash obtained according to step S30The Highway code calculates 2 kinds of losses, namely similarity loss L S And contrast loss L cl And defining the total loss function as: l=l S +αL cl Wherein α=0.1 is a weight factor;
s50: introducing a multi-task hash training strategy, repeatedly using a bilinear feature learning module, and extracting bilinear feature vectors v' (x) of the CT (computed tomography) image of the pneumonia i )∈R 262144×1 ,v'(x j )∈R 262144×1 I, j=1, 2,..n, and i+.j pass through a hash code learning module comprising 4 branches, respectively, and each branch comprises 2 fully-connected layers and 1 fully-connected hash layer, so that the model can learn 12,24,32, 48-bit hash codes simultaneously;
s60: using an alternating learning algorithm on an objective functionThe deep hash network model parameters theta, the hash coding matrix B, the weight matrix W and the bias vector V are optimized and updated, and the model is stored;
s70: first read test set T using a pre-trained model e CT image y of pneumonia of (2) k K=1, 2,..m, resulting in its hash codeC= 12,24,32,48, then will +.>And Hash code matrix B epsilon R c×N Comparing each column of c= 12,24,32,48, comparing the first 5 columns with smaller hamming distance, and if the number of the categories is more, then y k And dividing the classification into the categories, and finally calculating the average accuracy of the classification of the test set.
As a preferable technical scheme of the invention: in step S30, the bilinear feature learning module mainly includes two branches a and B, and the branches a and B are composed of two identical VGG16 models, the convolution layer conv of each branch is divided into 5 segments, 13 convolution layers are altogether, the convolution kernel of each convolution layer has a size of 3×3, step size stride and padding are all set to 1, there is one maximum pooling layer maxpool after the first 4 segments of convolution layers, and the pooling frame has a size of 2×2, step size stride is set to 2, taking the branch a as an example, the designed network structure specifically includes the following steps:
s31: training set T of CT image of pneumonia r Randomly divided into image pairs (x i ,x j ) I, j=1, 2,..n, and i+.j), and reads the image pair and the similarity matrix S, then after passing through the segment 1 convolutional layers conv1, conv2 with a number of filter filters of 64, the size of the extracted pneumonia CT image characteristic map is 224 multiplied by 64, and then the pneumonia CT image characteristic map passes through the maximum pooling layer maxpool1, and the final output size of the pneumonia CT image characteristic map is 112 multiplied by 64;
s32: after the output of the maxpool1 passes through 2 nd stage convolution layers conv3 and conv4 with the filter number of 128, the size of the extracted pneumonia CT image characteristic map is 112 multiplied by 128, and then the output of the maxpool1 passes through the maximum pooling layer maxpool2, and the final output size of the pneumonia CT image characteristic map is 56 multiplied by 128;
s33: after the output of the maxpool2 passes through the 3 rd section convolution layers conv5, conv6 and conv7 with the filter number of 256, the size of the extracted pneumonia CT image characteristic image is 56 multiplied by 256, and then the output of the maxpool2 passes through the maximum pooling layer maxpool3, and the final output size of the pneumonia CT image characteristic image is 28 multiplied by 256;
s34: after the output of the maxpool3 passes through 4 th section convolution layers conv8, conv9 and conv10 with the filter number of 512, the size of the extracted pneumonia CT image characteristic image is 28 multiplied by 512, and then the output of the maxpool3 passes through the maximum pooling layer maxpool4, and the final output size of the pneumonia CT image characteristic image is 14 multiplied by 512;
s35: after the output of maxpool4 passes through the 5 th section convolution layers conv11, conv12 and conv13 with the filter number of 512, the output size of the extracted pneumonia CT image feature map is 14 multiplied by 512;
at this time, let the CT image of pneumonia extracted by branch A be characterized as F A (x i )∈R 14×14×512 、F A (x j )∈R 14 ×14×512 The CT image of pneumonia extracted by branch B is characterized by F B (x i )∈R 14×14×512 、F B (x j )∈R 14×14×512 ；
S36: the outputs of branch A and branch B were processed through bilinear pooling to obtain CT image pairs (x i ,x j ) I, j=1, 2,.. and i+.j deep features f at position l A (l,x i )∈R 1×512 、f B (l,x i )∈R 1 ×512 And f A (l,x j )∈R 1×512 、f B (l,x j )∈R 1×512 The following operations are performed:
(1) Computing a CT image pair (x) i ,x j ) I, j=1, 2,.. and i+.j bilinear features at position l
(2) CT image pairs (x) were pooled by summing pooled sampling i ,x j ) I, j=1, 2,.. and i +.j bilinear features at all locations to obtain global bilinear features
(3) Global bilinear feature matrix xi (x i ) And xi (x) j ) And tensed into a vector to obtain v (x i )＝vec(ξ(x i ))∈R 262144×1 、v(x j )＝vec(ξ(x j ))∈R 262144×1 Where vec (·) represents the operation of tensioning the matrix into vectors;
(4) Feature vector v (x) i ) And v (x) j ) Normalization operation is carried out to obtain Wherein I II 2 An L2 norm representing the vector;
s37: feature vector v' (x) of normalized CT image of pneumonia i )∈R 262144×1 、v'(x j )∈R 262144×1 After passing through the 2 full connection layers FC1, FC2 in turn, v "(x) is obtained i )、v”(x j )∈R c×1 C=12, 24,32,48, c represents the number of bits of the hash code;
s38: feature vector v "(x) of CT image of pneumonia after passing through full connection layer i )、v”(x j ) Mapping to a binary hash code b according to a hash function h () i ,b j ∈R c×1 C= 12,24,32,48, finally obtaining the training set T r Hash encoding matrix B e R of (C) c×N ，
Where h (·) represents the hash function and sign (·) represents the sign function.
As a preferable technical scheme of the invention: the step S40 is to obtain hash codes B E R according to the step S38 c×N C= 12,24,32,48 calculates 2 kinds of loss, i.e., similarity loss L S And contrast loss L cl The method comprises the following specific steps:
s41: for the input pair of CT images of pneumonia (x i ,x j ) I, j=1, 2,..n, and i+.j, with similarity loss L S To optimize the distance between similar samples and to enlarge the distance between dissimilar samples, the formula is as follows:
wherein the method comprises the steps ofHash code b i 、b j ∈R c×1 ,c＝12,24,32,48；
S42: in CT image x of pneumonia n N=1, 2,.. n ) And corresponding hash code b n Solving contrast loss L cl The formula is as follows:
wherein v (x) n )＝W T Φ(x n The method comprises the steps of carrying out a first treatment on the surface of the θ) +v, θ represents all parameters of the deep hash network model, Φ (x) n ；θ)∈R 262144×1 Representing input into fully connected layer FC1, W ε R 262144×c Is a weight matrix, V.epsilon.R c×1 Is the offset vector of the reference signal, I.I 2 An L2 norm representing the vector;
s43: the total loss function is defined as: l=l S +αL cl Where α=0.1 is a weight factor, so when optimizing the network model, the objective function should be set to minimize the loss function as follows:
wherein S is ij In the form of a similarity matrix,and->b j For CT image x of pneumonia i Transpose of hash code, CT image x of pneumonia j Hash coding of b) n CT image x representing pneumonia n N=1, 2,.. 262144×c Is a weight matrix, Φ (x n ；θ)∈R 262144×1 Representing input into fully connected layer FC1, V.epsilon.R c×1 Is the offset vector of the reference signal, I.I 2 Representative vectorIs a L2 norm of (c).
As a preferable technical scheme of the invention: step S50 introduces a multi-task hash training strategy, repeatedly uses a bilinear feature learning module, and extracts bilinear feature vectors v' (x) of the CT image of the pneumonia i )∈R 262144×1 ,v'(x j )∈R 262144×1 I, j=1, 2,..n, and i+.j pass through hash code learning modules of 4 branches, respectively, and each branch includes 2 fully-connected layers and 1 fully-connected hash layer, so that the model can learn 12,24,32, 48-bit hash codes simultaneously, taking branch 1 as an example, where c=12, and the specific steps are as follows:
s51: first, feature vector v' (x) of CT image of pneumonia is calculated i )∈R 262144×1 ,v'(x j )∈R 262144×1 V "(x) is obtained by the fully connected layers FC11, FC12 of branch 1 i )∈R 12×1 、v”(x j )∈R 12×1 ；
S52: the eigenvector v "(x) of the CT image of pneumonia is then determined i )、v”(x j ) Mapping to binary hash code b according to hash function h (.) i 、b j ∈R 12×1 Finally form training set T r Hash encoding matrix B e R of (C) 12×N ，
Where h (·) represents the hash function and sign (·) represents the sign function.
Compared with other lung CT image classification methods, the method has the beneficial effects that:
1. although convolutional neural network CNN can extract deep features of images, since the differences between the lung CT images are not too large, the accuracy of classification of the lung CT images is not very high. Therefore, the invention uses bilinear convolutional neural network BCNN to effectively extract the fine differences among different symptoms of the lung CT image, fully exerts the extraction advantages of the bilinear convolutional neural network on the fine-grained image characteristics, and effectively improves the recognition accuracy and generalization robustness of the model;
2. if the real-valued features extracted by the bilinear convolutional neural network BCNN are directly classified, the problems of overlarge storage space and overlong training time can be caused. Therefore, the patent introduces hash code learning, and maps real-value features extracted from the bilinear convolutional neural network BCNN to a binary hamming space, so that the storage space is greatly reduced and the training time is shortened when a model is trained;
3. the invention can reuse bilinear feature learning module, so that the model can generate hash codes with different code lengths simultaneously with lung CT images, thereby further saving calculation time and storage space.
Drawings
FIG. 1 is a flow chart of the invention for data enhancement and expansion of a collected CT image dataset of pneumonia;
FIG. 2 is a block diagram of a depth hashing method for CT image classification of pneumonia according to the present invention;
fig. 3 is a final frame diagram of the deep hashing method for pneumonia CT image classification of the present invention after introducing a multi-task hash training strategy.
Detailed Description
The present invention will be described in further detail with reference to the following examples in order to make the objects, technical solutions and advantages of the present invention more apparent. Of course, the examples described by reference to the drawings are only for the purpose of explaining the present invention and are not to be construed as limiting the invention.
As shown in fig. 1-3, a deep hashing method for pneumonia CT image classification includes the steps of:
s10: establishing a pneumonia CT image data set;
s20: data preprocessing, namely firstly carrying out data enhancement and expansion on a data set, and dividing the data set into training sets T according to the proportion of 80 percent and 20 percent r ＝(x 1 ,x 2 ,...,x n ) N=1, 2, where, N and test set T e ＝(y 1 ,y 2 ,...,y m ) M=1, 2, then the size of the pneumonia CT image is uniformly adjusted to 224×224, and the whole is doneThe number of the channels is 1, the data set comprises 3 types of CT images, namely a lung CT image of a normal person, a lung CT image of a COVID-19 patient and a lung CT image of a common pneumonia patient, and finally a training set T is constructed r Similarity matrix S of (1), wherein
And S is ij ∈R N×N ,i,j＝1,2,...,N；
S30: constructing a deep hash network model, wherein the model comprises two modules of bilinear feature learning and hash code learning, and when the model is trained, firstly, fine-granularity features of a pneumonia CT image are extracted by using a bilinear convolutional neural network BCNN, and then the extracted fine-granularity features are input into a hash code learning module, so that the fine-granularity features of the corresponding pneumonia CT image are mapped into binary hash codes;
s40: calculating 2 kinds of loss, namely similarity loss L according to the hash code obtained in the step S30 S And contrast loss L cl And defining the total loss function as: l=l S +αL cl Wherein α=0.1 is a weight factor;
s50: introducing a multi-task hash training strategy, repeatedly using a bilinear feature learning module, and extracting bilinear feature vectors v' (x) of the CT (computed tomography) image of the pneumonia i )∈R 262144×1 ,v'(x j )∈R 262144×1 I, j=1, 2,..n, and i+.j pass through a hash code learning module comprising 4 branches, respectively, and each branch comprises 2 fully-connected layers and 1 fully-connected hash layer, so that the model can learn 12,24,32, 48-bit hash codes simultaneously;
s60: using an alternating learning algorithm on an objective functionThe deep hash network model parameters theta, the hash coding matrix B, the weight matrix W and the bias vector V are optimized and updated, and the model is stored;
s70: firstReading test set T using a pre-trained model e CT image y of pneumonia of (2) k K=1, 2,..m, resulting in its hash codeC= 12,24,32,48, then will +.>And Hash code matrix B epsilon R c×N Comparing each column of c= 12,24,32,48, comparing the first 5 columns with smaller hamming distance, and if the number of the categories is more, then y k And dividing the classification into the categories, and finally calculating the average accuracy of the classification of the test set.
In step S30, the bilinear feature learning module mainly includes two branches a and B, and the branches a and B are composed of two identical VGG16 models, the convolution layer conv of each branch is divided into 5 segments, 13 convolution layers are altogether, the convolution kernel of each convolution layer has a size of 3×3, step size stride and padding are all set to 1, there is one maximum pooling layer maxpool after the first 4 segments of convolution layers, and the pooling frame has a size of 2×2, step size stride is set to 2, taking the branch a as an example, the designed network structure specifically includes the following steps:
s31: training set T of CT image of pneumonia r Randomly divided into image pairs (x i ,x j ) I, j=1, 2,..n, and i+.j), and reads the image pair and the similarity matrix S, then after passing through the segment 1 convolutional layers conv1, conv2 with a number of filter filters of 64, the size of the extracted pneumonia CT image characteristic map is 224 multiplied by 64, and then the pneumonia CT image characteristic map passes through the maximum pooling layer maxpool1, and the final output size of the pneumonia CT image characteristic map is 112 multiplied by 64;
s32: after the output of the maxpool1 passes through 2 nd stage convolution layers conv3 and conv4 with the filter number of 128, the size of the extracted pneumonia CT image characteristic map is 112 multiplied by 128, and then the output of the maxpool1 passes through the maximum pooling layer maxpool2, and the final output size of the pneumonia CT image characteristic map is 56 multiplied by 128;
s33: after the output of the maxpool2 passes through the 3 rd section convolution layers conv5, conv6 and conv7 with the filter number of 256, the size of the extracted pneumonia CT image characteristic image is 56 multiplied by 256, and then the output of the maxpool2 passes through the maximum pooling layer maxpool3, and the final output size of the pneumonia CT image characteristic image is 28 multiplied by 256;
s34: after the output of the maxpool3 passes through 4 th section convolution layers conv8, conv9 and conv10 with the filter number of 512, the size of the extracted pneumonia CT image characteristic image is 28 multiplied by 512, and then the output of the maxpool3 passes through the maximum pooling layer maxpool4, and the final output size of the pneumonia CT image characteristic image is 14 multiplied by 512;
s35: after the output of maxpool4 passes through the 5 th section convolution layers conv11, conv12 and conv13 with the filter number of 512, the output size of the extracted pneumonia CT image feature map is 14 multiplied by 512;
at this time, let the CT image of pneumonia extracted by branch A be characterized as F A (x i )∈R 14×14×512 、F A (x j )∈R 14 ×14×512 The CT image of pneumonia extracted by branch B is characterized by F B (x i )∈R 14×14×512 、F B (x j )∈R 14×14×512 ；
S36: the outputs of branch A and branch B were processed through bilinear pooling to obtain CT image pairs (x i ,x j ) I, j=1, 2,.. and i+.j deep features f at position l A (l,x i )∈R 1×512 、f B (l,x i )∈R 1 ×512 And f A (l,x j )∈R 1×512 、f B (l,x j )∈R 1×512 The following operations are performed:
(1) Computing a CT image pair (x) i ,x j ) I, j=1, 2,.. and i+.j bilinear features at position l
(2) CT image pairs (x) were pooled by summing pooled sampling i ,x j ) I, j=1, 2,.. and i +.j bilinear features at all locations to obtain global bilinear features
(3) Global bilinear feature matrix xi (x i ) And xi (x) j ) And tensed into a vector to obtain v (x i )＝vec(ξ(x i ))∈R 262144×1 、v(x j )＝vec(ξ(x j ))∈R 262144×1 Where vec (·) represents the operation of tensioning the matrix into vectors;
(4) Feature vector v (x) i ) And v (x) j ) Normalization operation is carried out to obtain Wherein I II 2 An L2 norm representing the vector;
s37: feature vector v' (x) of normalized CT image of pneumonia i )∈R 262144×1 、v'(x j )∈R 262144×1 After passing through the 2 full connection layers FC1, FC2 in turn, v "(x) is obtained i )、v”(x j )∈R c×1 C=12, 24,32,48, c represents the number of bits of the hash code;
s38: feature vector v "(x) of CT image of pneumonia after passing through full connection layer i )、v”(x j ) Mapping to a binary hash code b according to a hash function h () i ,b j ∈R c×1 C= 12,24,32,48, finally obtaining the training set T r Hash encoding matrix B e R of (C) c×N ，
Where h (·) represents the hash function and sign (·) represents the sign function.
The step S40 is to obtain hash codes B E R according to the step S38 c×N C= 12,24,32,48 calculates 2 kinds of loss, i.e., similarity loss L S And contrast loss L cl The method comprises the following specific steps:
s41: for the input pair of CT images of pneumonia (x i ,x j ) I, j=1, 2,..n, and i+.j, with similarity loss L S To optimize the distance between similar samples and to enlarge the distance between dissimilar samples, the formula is as follows:
wherein the method comprises the steps ofHash code b i 、b j ∈R c×1 ,c＝12,24,32,48；
S42: in CT image x of pneumonia n N=1, 2,.. n ) And corresponding hash code b n Solving contrast loss L cl The formula is as follows:
wherein v (x) n )＝W T Φ(x n The method comprises the steps of carrying out a first treatment on the surface of the θ) +v, θ represents all parameters of the deep hash network model, Φ (x) n ；θ)∈R 262144×1 Representing input into fully connected layer FC1, W ε R 262144×c Is a weight matrix, V.epsilon.R c×1 Is the offset vector of the reference signal, I.I 2 An L2 norm representing the vector;
s43: the total loss function is defined as: l=l S +αL cl Where α=0.1 is a weight factor, so when optimizing the network model, the objective function should be set to minimize the loss functionThe formula is as follows:
wherein S is ij In the form of a similarity matrix,and->b j For CT image x of pneumonia i Transpose of hash code, CT image x of pneumonia j Hash coding of b) n CT image x representing pneumonia n N=1, 2,.. 262144×c Is a weight matrix, Φ (x n ；θ)∈R 262144×1 Representing input into fully connected layer FC1, V.epsilon.R c×1 Is the offset vector of the reference signal, I.I 2 Representing the L2 norm of the vector.
Step S50, a multi-task hash training strategy is introduced, a bilinear feature learning module is repeatedly used, and the bilinear feature vector v' (x) of the extracted pneumonia CT image is obtained i )∈R 262144×1 ,v'(x j )∈R 262144×1 I, j=1, 2,..n, and i+.j pass through hash code learning modules of 4 branches, respectively, and each branch includes 2 fully-connected layers and 1 fully-connected hash layer, so that the model can learn 12,24,32, 48-bit hash codes simultaneously, taking branch 1 as an example, where c=12, and the specific steps are as follows:
s51: first, feature vector v' (x) of CT image of pneumonia is calculated i )∈R 262144×1 ,v'(x j )∈R 262144×1 V "(x) is obtained by the fully connected layers FC11, FC12 of branch 1 i )∈R 12×1 、v”(x j )∈R 12×1 ；
S52: the eigenvector v "(x) of the CT image of pneumonia is then determined i )、v”(x j ) Mapping to binary hash code b according to hash function h (.) i 、b j ∈R 12×1 Finally form training set T r Is of (1)The Highway code matrix B epsilon R 12×N ，
Where h (·) represents the hash function and sign (·) represents the sign function.
Although convolutional neural network CNN can extract deep features of images, since the differences between the lung CT images are not too large, the accuracy of classification of the lung CT images is not very high. Therefore, the invention uses bilinear convolutional neural network BCNN to effectively extract the fine differences among different symptoms of the lung CT image, fully exerts the extraction advantages of the bilinear convolutional neural network on the fine-grained image characteristics, and effectively improves the recognition accuracy and generalization robustness of the model; if the real-valued features extracted by the bilinear convolutional neural network BCNN are directly classified, the problems of overlarge storage space and overlong training time can be caused. Therefore, the patent introduces hash code learning, and maps real-value features extracted from the bilinear convolutional neural network BCNN to a binary hamming space, so that the storage space is greatly reduced and the training time is shortened when a model is trained; the invention can reuse bilinear feature learning module, so that the model can generate hash codes with different code lengths simultaneously with lung CT images, thereby further saving calculation time and storage space.
While the foregoing is directed to embodiments of the present invention, other and further details of the invention may be devised without departing from the basic scope thereof, and the scope thereof is determined by the claims that follow.
Claims (4)
1. A deep hashing method for pneumonia CT image classification, comprising the steps of:
s10: establishing a pneumonia CT image data set;
s20: data preprocessing, namely firstly carrying out data enhancement and expansion on a data set, and dividing the data set into training sets T according to the proportion of 80 percent and 20 percent r ＝(x 1 ,x 2 ,...,x n ) N=1, 2, where, N and test set T e ＝(y 1 ,y 2 ,...,y m ) M=1, 2, m, the size of the pneumonia CT image is then uniformly adjusted to 224 x 224, the number of channels is 1, and 3 types of CT images are contained in the dataset, namely the lung CT image of a normal person, the lung CT image of a COVID-19 patient and the lung CT image of a common pneumonia patient, and finally constructing a training set T r Similarity matrix S of (1), wherein
And S is ij ∈R N×N ,i,j＝1,2,...,N；
S30: constructing a deep hash network model, wherein the model comprises two modules of bilinear feature learning and hash code learning, and when the model is trained, firstly, fine-granularity features of a pneumonia CT image are extracted by using a bilinear convolutional neural network BCNN, and then the extracted fine-granularity features are input into a hash code learning module, so that the fine-granularity features of the corresponding pneumonia CT image are mapped into binary hash codes;
s40: calculating 2 kinds of loss, namely similarity loss L according to the hash code obtained in the step S30 S And contrast loss L cl And defining the total loss function as: l=l S +αL cl Wherein α=0.1 is a weight factor;
s50: introducing a multi-task hash training strategy, repeatedly using a bilinear feature learning module, and extracting bilinear feature vectors v' (x) of the CT (computed tomography) image of the pneumonia i )∈R 262144×1 ,v'(x j )∈R 262144×1 I, j=1, 2,..n, and i+.j pass through a hash code learning module containing 4 branches, respectively, and each branch contains 2 fully-connected layers and 1 fully-connected hash layer, so that the model can learn Ha Xibian of 12,24,32,48 bits simultaneouslyA code;
s60: using an alternating learning algorithm on an objective functionThe deep hash network model parameters theta, the hash coding matrix B, the weight matrix W and the bias vector V in alpha=0.1 are optimized and updated, and the model is stored;
s70: first read test set T using a pre-trained model e CT image y of pneumonia of (2) k K=1, 2,..m, resulting in its hash codec= 12,24,32,48, then will ∈ ->And Hash code matrix B epsilon R c×N Comparing each column of c= 12,24,32,48, comparing the first 5 columns with smaller hamming distance, and if the number of the categories is more, then y k And dividing the classification into the categories, and finally calculating the average accuracy of the classification of the test set.
2. The deep hashing method for pneumonia CT image classification according to claim 1, wherein in step S30, the bilinear feature learning module mainly includes two branches a and B, and branch a and branch B are composed of two identical VGG16 models, the convolution layer conv of each branch is divided into 5 segments, 13 convolution layers are total, the convolution kernel of each convolution layer has a size of 3×3, step size stride and padding are all set to 1, there is one maximum pooling layer maxpool after the first 4 segments of convolution layers, and the pooling frame has a size of 2×2, step size stride is set to 2, and the network structure designed by branch a specifically includes the following steps:
s31: training set T of CT image of pneumonia r Randomly divided into image pairs (x i ,x j ) I, j=1, 2,..n, and i+.j), and reads the image pair and the similarity matrix S, then passes through the segment 1 convolutional layers conv1, conv2 with a number of filter filters of 64Then, the size of the extracted CT image feature map of the pneumonia is 224 multiplied by 64, and then the CT image feature map of the pneumonia passes through the max pooling layer maxpool1, and the final output size of the CT image feature map of the pneumonia is 112 multiplied by 64;
s32: after the output of the maxpool1 passes through 2 nd stage convolution layers conv3 and conv4 with the filter number of 128, the size of the extracted pneumonia CT image characteristic map is 112 multiplied by 128, and then the output of the maxpool1 passes through the maximum pooling layer maxpool2, and the final output size of the pneumonia CT image characteristic map is 56 multiplied by 128;
s33: after the output of the maxpool2 passes through the 3 rd section convolution layers conv5, conv6 and conv7 with the filter number of 256, the size of the extracted pneumonia CT image characteristic image is 56 multiplied by 256, and then the output of the maxpool2 passes through the maximum pooling layer maxpool3, and the final output size of the pneumonia CT image characteristic image is 28 multiplied by 256;
s34: after the output of the maxpool3 passes through 4 th section convolution layers conv8, conv9 and conv10 with the filter number of 512, the size of the extracted pneumonia CT image characteristic image is 28 multiplied by 512, and then the output of the maxpool3 passes through the maximum pooling layer maxpool4, and the final output size of the pneumonia CT image characteristic image is 14 multiplied by 512;
s35: after the output of maxpool4 passes through the 5 th section convolution layers conv11, conv12 and conv13 with the filter number of 512, the output size of the extracted pneumonia CT image feature map is 14 multiplied by 512;
at this time, let the CT image of pneumonia extracted by branch A be characterized as F A (x i )∈R 14×14×512 、F A (x j )∈R 14×14×512 The CT image of pneumonia extracted by branch B is characterized by F B (x i )∈R 14×14×512 、F B (x j )∈R 14×14×512 ；
S36: the outputs of branch A and branch B were passed through bilinear pooling to image the CT image pair (x i ,x j ) I, j=1, 2,.. and i+.j deep features f at position l A (l,x i )∈R 1×512 、f B (l,x i )∈R 1×512 And f A (l,x j )∈R 1×512 、f B (l,x j )∈R 1×512 The following operations are performed:
(1) Computing a CT image pair (x) i ,x j ) I, j=1, 2,.. and i+.j bilinear features at position l
(2) CT image pair (x) of sumpooling pneumonia by summing pooling i ,x j ) I, j=1, 2,.. and i +.j bilinear features at all locations to obtain global bilinear features
(3) Global bilinear feature matrix xi (x i ) And xi (x) j ) And tensed into a vector to obtain v (x i )＝vec(ξ(x i ))∈R 262144×1 、v(x j )＝vec(ξ(x j ))∈R 262144×1 Where vec (·) represents the operation of tensioning the matrix into vectors;
(4) Feature vector v (x) i ) And v (x) j ) Normalization operation is carried out to obtain Wherein I II 2 An L2 norm representing the vector;
s37: feature vector v' (x) of normalized CT image of pneumonia i )∈R 262144×1 、v'(x j )∈R 262144×1 After passing through the 2 full connection layers FC1, FC2 in turn, v "(x) is obtained i )、v”(x j )∈R c×1 C=12, 24,32,48, c represents the number of bits of the hash code;
s38: feature vector v "(x) of CT image of pneumonia after passing through full connection layer i )、v”(x j ) Mapping to a binary hash code b according to a hash function h () i ,b j ∈R c×1 C= 12,24,32,48, finally obtaining the training set T r Hash encoding matrix B e R of (C) c×N ，
Where h (·) represents the hash function and sign (·) represents the sign function.
3. The deep hashing method for pneumonia CT image classification according to claim 2 wherein said step S40 is based on the hash code B e R obtained in S38 c×N C= 12,24,32,48 calculates 2 kinds of loss, i.e., similarity loss L S And contrast loss L cl The method comprises the following specific steps:
s41: for the input pair of CT images of pneumonia (x i ,x j ) I, j=1, 2,..n, and i+.j, with similarity loss L S To optimize the distance between similar samples and to enlarge the distance between dissimilar samples, the formula is as follows:
wherein the method comprises the steps ofHash code b i 、b j ∈R c×1 ,c＝12,24,32,48；
S42: in CT image x of pneumonia n N=1, 2,.. n ) And corresponding hash code b n Solving contrast loss L cl The formula is as follows:
wherein v (x) n )＝W T Φ(x n The method comprises the steps of carrying out a first treatment on the surface of the θ) +v, θ represents all parameters of the deep hash network model, Φ (x) n ；θ)∈R 262144×1 Representing input into fully connected layer FC1, W ε R 262144×c Is a weight matrix, V.epsilon.R c×1 Is the offset vector of the reference signal, I.I 2 An L2 norm representing the vector;
s43: the total loss function is defined as: l=l S +αL cl Where α=0.1 is a weight factor, L S For similarity loss, L cl To compare losses, the objective function should therefore be set to minimize the loss function when optimizing the network model, with the following formula:
wherein S is ij In the form of a similarity matrix,and->b j For CT image x of pneumonia i Transpose of hash code, CT image x of pneumonia j Hash coding of b) n CT image x representing pneumonia n N=1, 2,.. 262144×c Is a weight matrix, Φ (x n ；θ)∈R 262144×1 Representing input into fully connected layer FC1, V.epsilon.R c×1 Is the offset vector of the reference signal, I.I 2 Representing the L2 norm of the vector.
4. The deep hashing method for pneumonia CT image classification according to claim 1, wherein said step S50 introduces multipleTask hash training strategy, repeatedly using bilinear feature learning module, and extracting bilinear feature vector v' (x) of the CT image of pneumonia i )∈R 262144×1 ,v'(x j )∈R 262144×1 I, j=1, 2,..n, and i+.j pass through hash code learning modules of 4 branches, respectively, and each branch includes 2 fully-connected layers and 1 fully-connected hash layer, so that the model can learn 12,24,32, 48-bit hash codes simultaneously, and c=12 in branch 1, which specifically comprises the following steps:
s51: first, feature vector v' (x) of CT image of pneumonia is calculated i )∈R 262144×1 ,v'(x j )∈R 262144×1 V "(x) is obtained by the fully connected layers FC11, FC12 of branch 1 i )∈R 12×1 、v”(x j )∈R 12×1 ；
S52: the eigenvector v "(x) of the CT image of pneumonia is then determined i )、v”(x j ) Mapping to binary hash code b according to hash function h (.) i 、b j ∈R 12×1 Finally form training set T r Hash encoding matrix B e R of (C) 12×N ，
Where h (·) represents the hash function and sign (·) represents the sign function.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210093092.7A CN114463583B (en) | 2022-01-26 | 2022-01-26 | Deep hashing method for pneumonia CT image classification |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210093092.7A CN114463583B (en) | 2022-01-26 | 2022-01-26 | Deep hashing method for pneumonia CT image classification |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114463583A CN114463583A (en) | 2022-05-10 |
| CN114463583B true CN114463583B (en) | 2024-03-19 |
Family
ID=81411989
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210093092.7A Active CN114463583B (en) | 2022-01-26 | 2022-01-26 | Deep hashing method for pneumonia CT image classification |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114463583B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116128846B (en) * | 2023-02-01 | 2023-08-22 | 南通大学 | Visual transducer hash method for lung X-ray image retrieval |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109783682A (en) * | 2019-01-19 | 2019-05-21 | 北京工业大学 | It is a kind of based on putting non-to the depth of similarity loose hashing image search method |
| JP2019219712A (en) * | 2018-06-15 | 2019-12-26 | 日本電信電話株式会社 | Image feature learning device, image feature learning method, image feature extraction device, image feature extraction method, and program |
| CN112395438A (en) * | 2020-11-05 | 2021-02-23 | 华中科技大学 | Hash code generation method and system for multi-label image |
| CN112597324A (en) * | 2020-12-15 | 2021-04-02 | 武汉工程大学 | Image hash index construction method, system and equipment based on correlation filtering |
-
2022
- 2022-01-26 CN CN202210093092.7A patent/CN114463583B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2019219712A (en) * | 2018-06-15 | 2019-12-26 | 日本電信電話株式会社 | Image feature learning device, image feature learning method, image feature extraction device, image feature extraction method, and program |
| CN109783682A (en) * | 2019-01-19 | 2019-05-21 | 北京工业大学 | It is a kind of based on putting non-to the depth of similarity loose hashing image search method |
| CN112395438A (en) * | 2020-11-05 | 2021-02-23 | 华中科技大学 | Hash code generation method and system for multi-label image |
| CN112597324A (en) * | 2020-12-15 | 2021-04-02 | 武汉工程大学 | Image hash index construction method, system and equipment based on correlation filtering |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114463583A (en) | 2022-05-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Li et al. | Efficient and effective training of COVID-19 classification networks with self-supervised dual-track learning to rank | |
| CN111145170A (en) | Medical image segmentation method based on deep learning | |
| CN113256641B (en) | Skin lesion image segmentation method based on deep learning | |
| Liu et al. | Deep spatio-temporal representation and ensemble classification for attention deficit/hyperactivity disorder | |
| CN113674253A (en) | Rectal cancer CT image automatic segmentation method based on U-transducer | |
| Kong et al. | Automated maxillofacial segmentation in panoramic dental x-ray images using an efficient encoder-decoder network | |
| CN112884788B (en) | Cup optic disk segmentation method and imaging method based on rich context network | |
| CN114463583B (en) | Deep hashing method for pneumonia CT image classification | |
| CN113284136A (en) | Medical image classification method of residual error network and XGboost of double-loss function training | |
| CN115375711A (en) | Image segmentation method of global context attention network based on multi-scale fusion | |
| CN111242949B (en) | Fundus image blood vessel segmentation method based on full convolution neural network multi-scale features | |
| Lin et al. | Batformer: Towards boundary-aware lightweight transformer for efficient medical image segmentation | |
| CN115035127A (en) | Retinal vessel segmentation method based on generative confrontation network | |
| Zhuang et al. | APRNet: A 3D anisotropic pyramidal reversible network with multi-modal cross-dimension attention for brain tissue segmentation in MR images | |
| CN116912253B (en) | Lung cancer pathological image classification method based on multi-scale mixed neural network | |
| CN114820450A (en) | CT angiography image classification method suitable for Li's artificial liver treatment | |
| CN113344933A (en) | Glandular cell segmentation method based on multi-level feature fusion network | |
| CN117036288A (en) | Tumor subtype diagnosis method for full-slice pathological image | |
| CN116091446A (en) | Method, system, medium and equipment for detecting abnormality of esophageal endoscope image | |
| CN116309754A (en) | Brain medical image registration method and system based on local-global information collaboration | |
| CN116416452A (en) | Lung adenocarcinoma invasive intelligent classification system based on two-stage deep learning model | |
| Hu et al. | Classification of fissured tongue images using deep neural networks | |
| Mathina Kani et al. | Classification of skin lesion images using modified Inception V3 model with transfer learning and augmentation techniques | |
| CN114419000A (en) | Femoral head necrosis index prediction system based on multi-scale geometric embedded convolutional neural network | |
| Zhang et al. | Multimodal 2.5 D Convolutional Neural Network for Diagnosis of Alzheimer's Disease with Magnetic Resonance Imaging and Positron Emission Tomography. |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |