CN113763332A - Pulmonary nodule analysis method and device based on ternary capsule network algorithm and storage medium - Google Patents

Pulmonary nodule analysis method and device based on ternary capsule network algorithm and storage medium Download PDF

Info

Publication number
CN113763332A
CN113763332A CN202110946389.9A CN202110946389A CN113763332A CN 113763332 A CN113763332 A CN 113763332A CN 202110946389 A CN202110946389 A CN 202110946389A CN 113763332 A CN113763332 A CN 113763332A
Authority
CN
China
Prior art keywords
capsule network
image
ternary
result
lung nodule
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110946389.9A
Other languages
Chinese (zh)
Other versions
CN113763332B (en
Inventor
郑光远
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Jian Qiao University
Original Assignee
Shanghai Jian Qiao University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Jian Qiao University filed Critical Shanghai Jian Qiao University
Priority to CN202110946389.9A priority Critical patent/CN113763332B/en
Publication of CN113763332A publication Critical patent/CN113763332A/en
Application granted granted Critical
Publication of CN113763332B publication Critical patent/CN113763332B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10072—Tomographic images
    • G06T2207/10081—Computed x-ray tomography [CT]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30061—Lung
    • G06T2207/30064—Lung nodule

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Radiology & Medical Imaging (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Medical Informatics (AREA)
  • General Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Quality & Reliability (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • Image Analysis (AREA)

Abstract

The invention relates to a lung nodule analysis method, a device and a storage medium based on a ternary capsule network algorithm, wherein the method comprises the following steps: constructing a lung nodule analysis model based on a ternary capsule network algorithm; acquiring an image to be analyzed; learning a single capsule network module, and training a capsule network intelligent agent; performing ternary capsule network module learning, and optimizing a capsule network intelligent agent based on the ternary capsule network loss; analyzing the CT image based on the learnt and optimized capsule network intelligent agent to give a classification result of the lung nodules, namely acquiring the signs of the lung nodules suffered by the patient; and determining a data analysis result of the image to be analyzed according to the classification result, and outputting the data analysis result through a result determination module. Compared with the prior art, the method has the advantages of improving the analysis accuracy of the pulmonary nodules and the like.

Description

Pulmonary nodule analysis method and device based on ternary capsule network algorithm and storage medium
Technical Field
The invention relates to the technical field of machine learning, in particular to a lung nodule analysis method and device based on a ternary capsule network algorithm and a storage medium.
Background
With the development of computer-aided technology, the medical field also increasingly uses computer-aided equipment to reduce the work intensity of doctors. The doctor observes and analyzes the body image of the patient artificially, and the lung image examination and diagnosis needs increase year by year along with the enhancement of the health consciousness of people. In the field of medical image analysis, major difficulties currently include the following: firstly, the sample labeling is complex, the symptom expression of the focus in the medical image is complex and various, the difficulty in distinguishing different types of nodules is high, and professional knowledge and rich experience are needed for identifying and labeling; the complicated labeling process causes fatigue to readers, which leads to unstable diagnosis results, and therefore, each focus needs a plurality of experts to perform the comparison in different time periods, which is expensive. And (II) the medical radiology department doctors are insufficient, the daily film reading task of the doctors is heavy, and the doctors cannot extract special time to label the samples, so that the samples in the field of medical image analysis and research are insufficient. In the field of lung nodule classification and its related medical image classification, capsule network reinforcement learning has begun to attract interest. Silva used a set of 3D geometric features in combination with a reinforcement learning method to classify lung nodules as benign/malignant, with an accuracy of 81%. The heuristic of Ali in Alphago system proposes a lung nodule detection algorithm based on deep reinforcement learning, wherein an agent adopts a convolutional neural network. This method was trained and tested on the LIDC-IDRI database, resulting in an overall accuracy of 64.4% (sensitivity 58.9%, specificity 55.3%, PPV 54.2%, and NPV 60%). As for the ternary loss function, no relevant application in the field of lung nodule classification has been found at present. In the field of medical images, Puch proposes a small sample learning model based on a depth ternary network to classify brain MRI images. The result shows that under the condition of limited samples, the model can more accurately identify images of different modes than a traditional convolutional neural network classifier. Medela in one research work demonstrated that the use of triple network techniques can migrate knowledge from well-defined source data (colon tissue images) to a more general domain consisting of colon, lung and breast tissue images by using few training images.
The image analysis technology based on reinforcement learning gradually obtains the reliability and the adoption of doctors, particularly the detection task of lung nodules; image analysis techniques based on reinforcement learning can greatly reduce the workload of the doctor. However, the experts in the image department cannot additionally extract a large amount of time to perform sample labeling work, so that the sample in the field of medical image analysis and research is insufficient and the accuracy is not high.
Disclosure of Invention
The present invention aims to overcome the defects of the prior art, and provides a lung nodule analysis method, device and storage medium based on a ternary capsule network algorithm, which are used for improving the accuracy of lung nodule analysis.
The purpose of the invention can be realized by the following technical scheme:
a lung nodule analysis method based on a ternary capsule network algorithm comprises the following steps:
s1: and constructing a lung nodule analysis model based on a ternary capsule network algorithm, wherein the lung nodule analysis model comprises a memory pool module, a single capsule network module, a ternary capsule network module, an identification module and a result determination module.
S2: and acquiring an image to be analyzed.
S3: and (5) learning a single capsule network module, and training a capsule network intelligent agent. Specifically, the size of a sliding window on a CT image is kept unchanged, the number of image blocks extracted by each CT is determined, the state number M of each round is determined in the reinforcement learning process, the difference between the single capsule network diagnosis result and the calibration result is compared, and the action return on each image block is correspondingly calculated.
The specific method for comparing the dissimilarity of the single-capsule network diagnosis result and the calibration result comprises the following steps:
for an image block stIf the single capsule network diagnosis is correct, a reward value, namely a return value r, is given; otherwise, giving the reward value of 0; the method for judging whether the single-capsule network diagnosis result is correct comprises the following steps:
a: if image block stIf the region is overlapped with a focus region of the calibration result, further checking whether the classification result of the image block is consistent with the symptom category label of the calibration result, if so, judging that the diagnosis result of the single capsule network is correct, otherwise, judging that the diagnosis result is wrong;
b: if image block stAnd if the area is not overlapped with any focus area of the calibration result and the classification result of the image block is negative, judging that the single capsule network diagnosis result is correct, otherwise, judging that the single capsule network diagnosis result is wrong.
S4: and (4) learning a ternary capsule network module, and optimizing the capsule network intelligent agent based on the ternary capsule network loss.
S5: and inputting lung CT images into the learned capsule network, and performing sliding window processing on the CT images by the model to obtain a series of states.
S6: and determining a data analysis result of the image to be analyzed according to the classification result, and outputting the data analysis result through a result determination module.
Further, in S1, the specific content of constructing the lung nodule analysis model based on the ternary capsule network algorithm is as follows:
11) inputting a plurality of pictures, setting initial learning parameters including the memory space N of a memory pool D in a memory pool module and a temporary memory pool DTCapacity H, return value r, interval factor α, number of small batches B, number of training iterations EPOCHS, and capsule network initial parameter ωc；
12) For a CT image with 512 x 512 pixels, a sliding window with K x K pixels is adopted to obtain a 32 x 32 pixel image block on the CT image, and the coordinates (x) at the upper left corner of the image block are recordedi,yi) (ii) a Setting each image block to a state stFurther, all image blocks extracted from the CT image form a set S of all states;
13) state S in set S of capsule network pairstSelecting corresponding classification action a from action space AtOutput the class attribute c of this statej；
14) All the segments s taken from a CT image in the capsule networkiPerform the corresponding action atAnd after the detection is finished, displaying the lung nodule and the symptom classification result thereof detected by the capsule network back to the CT image according to the coordinate position of the image block region, thereby constructing a lung nodule analysis model based on a ternary capsule network algorithm.
Further, in S3, a fully-connected layer is added to the single-capsule network after the feature expression to output the accumulated reward Q corresponding to each categorytargetThe value:
Qtarget＝M×r
the corresponding single step loss function is:
Figure BDA0003216703240000031
in the formula, L (omega)c) For the name of the loss function, R (x) is the return function,
Figure BDA0003216703240000032
is a capsule network agent and is a capsule network agent,
Figure BDA0003216703240000033
as anchor image, ajFor corresponding actions, ωcIs radix Ginseng;
and learning the parameters of the single-capsule network by adopting the corresponding single-step loss function of the single-capsule network until the loss function is smaller than a preset value.
Further, in S4, the specific content of learning the ternary capsule network module is as follows:
41) inputting three images at a time, respectively being anchor images
Figure BDA0003216703240000034
Positive sample image
Figure BDA0003216703240000035
And negative sample image
Figure BDA0003216703240000036
Respectively extracting features from the three images by utilizing the capsule network, and further calculating the Euclidean distance d between the anchor point image and the positive sample image+And the Euclidean distance d between the anchor point image and the negative sample image-And normalizing the data; calculating a ternary loss function on the basis;
42) converting the loss function of the single-layer capsule network into a ternary loss function;
43) the ternary capsule network receives a state from the state set S and calculates rewards Q corresponding to different categoriestargetValue, select corresponding QtargetExecuting the action with the maximum value, correspondingly entering the next state, and simultaneously storing elements such as the current state, the action and the like; the process is iterated until the agent respectively executes corresponding actions on all states acquired from a CT image by a sliding window method; the environment compares the classification result of each image block by the ternary capsule network with the calibrated classification result, and correspondingly calculates and stores a return value;
44) the system performs experience playback, randomly extracts a batch of data from the state, action and return combination stored in the memory pool, calculates the loss of the capsule network and updates the network weight.
The calculation formula of the loss calculation and the network weight update of the capsule network is as follows:
Figure BDA0003216703240000041
in the formula, LcomposeGenerating a composite loss value; b is the number of data selected by experience playback in reinforcement learning, namely the number of previously-placed experience data is randomly selected from an experience pool; λ is two lost weight adjustment factors;
Figure BDA0003216703240000042
is a capsule network agent; alpha represents the minimum difference of the embedding expression distances of the anchor sample and the negative sample; l isi(d+,d-) Is a ternary loss function.
Further, in S6, the specific process of determining the data analysis result of the image to be analyzed according to the classification result is as follows:
suppose there is a sample s to be measuredAC is the number of all the classes considered, from the existing labeled sample set, N is randomly selected for each class of samplesPA total of N is obtained for each samplePX C samples; each sample sPRespectively associated with a sample s to be measuredAAfter combination form NPX C group of sample pairs
Figure BDA0003216703240000043
i∈[0,C-1]And after the distances corresponding to each group of sample pairs are calculated, calculating the average distance corresponding to each category, and taking the category with the minimum average distance as a classification result.
A pulmonary nodule analysis apparatus based on a ternary capsule network algorithm, the apparatus comprising:
a data storage module storing one or more programs;
the construction module is used for constructing a lung nodule analysis model based on a ternary capsule network algorithm;
the acquisition module is used for acquiring a lung image of a patient and taking the lung image as an image to be analyzed;
the analysis module divides the image to be analyzed into regions to form image data, inputs the data into a lung nodule analysis model which is trained in advance, and obtains a classification result output by the lung nodule analysis model;
the analysis result determining module is used for determining the analysis result of the image to be analyzed according to the classification result;
the output module is used for outputting data of the analysis result and the lung nodule image;
a central processor executing one or more programs that implement the lung nodule analysis method based on a ternary capsule network algorithm as described above.
A computer-readable storage medium having stored thereon a computer program for execution by a processor for implementing a lung nodule analysis method based on a ternary capsule network algorithm as described above.
Compared with the prior art, the lung nodule analysis method, the device and the storage medium based on the ternary capsule network algorithm provided by the invention at least have the following beneficial effects:
1) according to the invention, the intelligent agent is trained and learned in an early stage by using the loss of the single capsule network, when the performance of the single capsule network is improved and stagnated gradually to the best, the intelligent agent is further optimized based on the loss of the ternary capsule network, so that the characteristic distance between classes is relatively larger, the distance in the classes is smaller, and thus the samples in different classes can be distinguished in a finer granularity, and the final performance is obviously improved compared with the single capsule network;
2) the method is based on the ternary loss measurement method, and can obtain higher accuracy of analyzing the pulmonary nodules than single-point loss and pairwise loss (twin network).
Drawings
Fig. 1 is a diagram of a single capsule network architecture of a lung nodule analysis method based on a ternary capsule network algorithm according to an embodiment;
fig. 2 is a diagram of a ternary capsule network architecture of a lung nodule analysis method based on a ternary capsule network algorithm according to an embodiment;
fig. 3 is a schematic flowchart of a lung nodule analysis method based on a ternary capsule network algorithm according to an embodiment;
FIG. 4 is a graph comparing the lung nodule analysis method based on the ternary capsule network algorithm and the DQN average classification performance in the embodiment;
FIG. 5 is a lung nodule analysis method and DQN classification accuracy difference confusion matrix diagram based on the ternary capsule network algorithm in the embodiment;
fig. 6 is a comparison graph of a symptom map generated by a lung nodule analysis method based on a ternary capsule network algorithm and DQN reconstruction in the embodiment.
Detailed Description
The invention is described in detail below with reference to the figures and specific embodiments. It is to be understood that the embodiments described are only a few embodiments of the present invention, and not all embodiments. All other embodiments, which can be obtained by a person skilled in the art without any inventive step based on the embodiments of the present invention, shall fall within the scope of protection of the present invention.
Examples
The invention relates to a lung nodule analysis method based on a ternary capsule network algorithm, which can be applied to the condition of analyzing lung nodules according to lung images. The method may be performed by a lung nodule analyzing apparatus, which may be implemented in software and/or hardware, for example, which may be configured in a computer device. As shown in fig. 3, the lung nodule analysis method based on the ternary capsule network algorithm (hereinafter referred to as TriCaps-RL) specifically includes the following steps:
the lung nodule analysis model based on the ternary capsule network algorithm is constructed and comprises a memory pool module, a single capsule network module, a ternary capsule network module, an identification module and a result determination module. The concrete contents are as follows:
firstly, inputting a large number of pictures, performing multi-stage convolution on the input pictures by a capsule network, then encapsulating and learning image characteristics by using capsules, and generating an algorithm scheme; in this embodiment, as a preferred scheme, after two layers of convolution operations are performed on an adopted capsule network, the features extracted by the convolution layers are sent to initial capsule layers (PrimaryCaps) for encapsulation, so as to generate 8-dimensional feature combinations, and then 8-dimensional input space is mapped to 16 dimensions through 8 × 16 weight matrices, so as to obtain final feature expression of an image.
Setting initial learning parameters: memory space N of memory pool D in memory pool module and memory space D of temporary memory poolTCapacity H, return value r, interval factor α, number of small batches B, number of training iterations EPOCHS, and capsule network initial parameter ωc。
For a 512 x 512 pixel CT image, the algorithm first uses K x K (K)<512) The sliding window of the pixel acquires different image blocks on the CT image one by one, and simultaneously records the coordinates (x) of the upper left corner of the image blocksi,yi) (ii) a Each image block being regarded as a state stAll image blocks extracted from the CT image then constitute a set S of all states.
S in capsule network pair StSelecting corresponding classification action a from action space AtOutput the class attribute c of this statej(ii) a When the capsule network is used for all image blocks s obtained from a CT imageiThe corresponding action a is executedtThen, the lung nodule and the symptom classification result obtained by the capsule network detection are displayed back to the CT image according to the coordinate position of the image block area; therefore, a lung nodule analysis model based on a ternary capsule network algorithm is constructed.
And step two, acquiring an image to be analyzed.
Step three, learning a single-capsule network module, wherein the single-capsule network architecture diagram is shown in figure 1:
the size of a sliding window on one CT image is unchanged, so that the number of image blocks extracted from each CT image is determined, the state number M of each round is determined in the reinforcement learning process, the difference between the single-capsule network diagnosis result and the calibration result is compared, and the action return on each image block is correspondingly calculated, wherein the specific method comprises the following steps: for an image block stIf the single capsule network diagnosis is correct, a reward value r is given (which is determined experimentally); otherwise, giving the reward value of 0; the method for judging whether the single-capsule network diagnosis result is correct is as follows:
1) if image block stIf the region is overlapped with a certain focus region of the calibration result, further checking whether the classification result of the image block is consistent with the symptom category label of the calibration result, if so, determining that the single-capsule network diagnosis result is correct, otherwise, determining that the diagnosis result is wrong;
2) if image block stThe area is not overlapped with any focus area of the calibration result, and the classification result of the image block is negative, the single capsule network diagnosis result is considered to be correct; otherwise, the single capsule network diagnosis result is considered to be wrong.
Adding a fully-connected layer to output the accumulated reward Q corresponding to each category after the characteristics are expressed by the single-capsule networktargetThe value:
Qtarget＝M×r (1)
the corresponding single step loss function is:
Figure BDA0003216703240000071
wherein L isj(ωc) Is the name of the loss function and,
Figure BDA0003216703240000072
for the capsule network agent, R (x) is a reporting function,
Figure BDA0003216703240000073
representing one of the states in a round, a, for the anchor imagejRepresenting the corresponding movement, ωcThe super reference is shown.
And (3) learning the parameters of the single-capsule network by adopting the loss function formula (2) of the single-capsule network until the performance cannot be improved.
Step four, learning a ternary capsule network module, wherein the ternary capsule network architecture diagram is shown in fig. 2:
inputting three images at a time, respectively being anchor images
Figure BDA0003216703240000074
Positive sample image
Figure BDA0003216703240000075
And negative sample image
Figure BDA0003216703240000076
Respectively extracting features from the three images by utilizing the capsule network, and further calculating the Euclidean distance d between the anchor point image and the positive sample image+And the Euclidean distance d between the anchor point image and the negative sample image-And normalizing it to obtain:
Figure BDA0003216703240000077
Figure BDA0003216703240000078
in the formula (I), the compound is shown in the specification,
Figure BDA0003216703240000079
a capsule-embedded representation of the anchor point sample,
Figure BDA00032167032400000710
representing an embedded representation of the positive sample,
Figure BDA00032167032400000711
representing the embedded expression of the negative examples.
On this basis, the ternary loss function is calculated as:
Figure BDA00032167032400000712
in the formula: alpha represents the minimum difference between the anchor sample and the negative sample, and the anchor sample and the positive sample embed the expression distance.
The loss function of the single-layer capsule network is transited into a ternary loss function (5);
the ternary capsule network receives a state from state set S and computes Q corresponding to different classestargetValue, select corresponding QtargetExecuting the action with the maximum value, correspondingly entering the next state, and simultaneously storing elements such as the current state, the action and the like; the process is iterated until the agent respectively executes corresponding actions on all states acquired from a CT image by a sliding window method; and the environment compares the classification result of each image block by the ternary capsule network with the calibrated classification result, and automatically calculates and stores a return value (reward value r).
The system executes experience playback, randomly extracts a batch from the state, action and return combination stored in the memory pool, calculates the loss of the capsule network according to the following formula (6) and updates the network weight:
Figure BDA0003216703240000081
wherein L iscomposeRepresents the integrated loss value generated by equation (6); b is the number of data selected by experience playback in reinforcement learning, namely the number of previously-placed experience data is randomly selected from an experience pool; λ is two lost weight adjustment factors, which gradually increase with the increase of the training iteration number n, and the calculation formula is:
Figure BDA0003216703240000082
in equation (7), EPOCHS is a preset iteration value.
The use method of the formula (7) is as follows: when the iteration number n of the algorithm is less than EPOCHS, the algorithm mainly adopts the formula (8):
Figure BDA0003216703240000083
when the iteration number n is increased to exceed EPOCHS, the ternary loss shown in the formula (5) is mainly adopted, so that the classification accuracy of the intelligent agent is further finely adjusted, and the class distinction with finer granularity is realized for the sample. The effect of the unit loss stated above is to make the network converge faster.
And step five, inputting lung CT images into the learned capsule network, and performing sliding window processing on the CT images by the model to obtain a series of states.
And step six, selecting corresponding actions with maximum return for the states by the intelligent agents in the model according to the learned strategies, and outputting an analysis result, wherein the specific contents are as follows:
suppose there is a sample s to be measuredAC is the number of all the classes considered, from the existing labeled sample set, N is randomly selected for each class of samplesPA total of N is obtained for each samplePX C samples; each sample sPRespectively associated with a sample s to be measuredAAfter combination to form NPX C group of sample pairs
Figure BDA0003216703240000084
i∈[0,C-1]Calculating the distance corresponding to each group of sample pairs, then calculating the average distance corresponding to each category, and taking the category with the minimum average distance as a classification result, namely:
Figure BDA0003216703240000091
wherein, y*Is expressed as the mostClass of final output, sAFor the sample to be tested, yiIn the case of the (i) th category,
Figure BDA0003216703240000092
to test the group yiJ ∈ [1, N) of the jth sample of (c)P]。
The invention determines the analysis result of the image data to be analyzed according to the classification result and outputs the analysis result, and the lung nodule analysis result of the patient is output as the sign of the lung nodule suffered by the patient.
To demonstrate the effectiveness of the method of the invention, this example was verified experimentally. The experiments were performed on a sample set selected from the LIDC-IDRI library. The edge part of each nodule in the LIDC-IDRI library is sketched in detail by a plurality of doctors, and the nodules are evaluated according to the quality and the malignancy, and the labeling information is a record of the diagnosis operation of the real film reading process of the doctors. Thus, the label data in the LIDC-IDRI is used in this embodiment as a basis for the environment (simulating the doctor) to evaluate and feedback rewards for the actions selected and performed by the agent in the method of the invention.
For the convenience of experimental comparison, the node features provided by the radiologist during labeling are graded, the CT images of the positive non-central calcium feature, the lobular feature, the burr and the GGO feature 4-type node features are subjected to sample size increase through rotation and overturning operations, 12000 CT images are generated, and the experimental scheme is a 5-fold cross validation experiment.
For each CT image, a 32 × 32 pixel sliding window block acquisition is performed, with a sliding interval of 11 pixels, and accordingly 46 × 46 — 2116 blocks of pictures can be taken from each CT. In this experiment, for each agent identification, the correct return is given as r ═ 1, and Q corresponds to each imagetargetIs 2116. The maximum capacity N of the playback memory pool D is set to 3 k. Each time 50 small batches of samples were taken from D for training of the agent. For the epsilon greedy strategy, epsilon is the greedy weight, and the experiment sets epsilon to be linearly reduced from 1 to 0.1 in 3000 iteration cycles. Further, according to the experiment, for EPOCHS in equations (5-8), it is set to 22000.
In the classification stage of the symptoms in the experiment, the categoriesThe number C is 5. Randomly selecting P from each type of sample at a timenTotal 10 individuals, get PnAnd x C is 50 samples, and the x C and the samples to be classified form 50 sample pairs, and are classified according to the formula (9).
The accuracy rate of the Tricaps-RL changes dramatically before 2600 steps of the learning process, the process should be that the data in the experience memory pool is not completely updated, the value of the loss function at this stage is unstable and does not become smaller gradually with the increase of the iteration number, and the accuracy rate gradually increases steadily with the accumulation of the experience after 2600 steps. When training is carried out to 14000 iterations, the accuracy of the algorithm gradually reaches a stable value, and small-amplitude improvement is carried out after 22000 steps. This is because EPOCHS in equation (7) was set to 22000 in the experiment, after which the loss of the algorithm gradually transitions to be calculated by the ternary loss function shown in (5).
With the continuous increase of the training iteration number, the feature embedded expressions of the same feature class sample are gradually aggregated, and the feature embedded expressions of different feature class samples are gradually separated.
A DQN (classical deep reinforcement learning algorithm) network is constructed according to the structure shown in Table 1, and comprises three convolutional layers, then the data obtained from LIDC-IDRI is used for training the convolutional layers, and the final test result is calculated and compared with the classification performance of Tricaps-RL.
Table 1 CNN network architecture for comparison
Figure BDA0003216703240000101
The difference between DQN and the Tricaps-RL method of the invention is that: a) a CNN network is employed instead of a capsule network; 2) only the Q-learning objective is employed and no triplet learning objective is considered.
The average classification performance of both methods is shown in fig. 4. As can be seen from fig. 4, the classification effect of tricas-RL is significantly better than that of DQN classification method in terms of sensitivity, specificity and accuracy, which indicates that the capsule network and triplet learning objective employed herein are reasonably effective.
In order to perform finer-grained analysis and check the performance difference of the two methods in different categories, a difference confusion matrix of the classification accuracy of the TriCaps-RL and the DQN is constructed in the embodiment, as shown in fig. 5. The diagonal floating point numbers represent the difference in classification accuracy of the corresponding classes for TriCaps-RL and DQN, while the non-diagonal floating point numbers represent their difference in classification error rate, showing the rate of error classification to each of the other respective classes.
The value on the diagonal line is the difference of the classification accuracy of the Tricaps-RL and the DQN; the elements that are not on the diagonal are the difference in the classification error rate. As shown in fig. 5, the values of the elements on the diagonal of the difference confusion matrix are all positive, and the average value of the number of the diagonals is 0.1047, which indicates that the tricas-RL algorithm has higher accuracy on each class. The overall classification performance is better than DQN. No positive values appear in the off-diagonal elements, which means that the misclassification rate of Tricaps-RL is lower than DQN on all classes. Table 2 is a record of the average time of 5 trains and the average predicted time of 20 samples for both the TriCaps-RL method and the DQN method.
TABLE 2 comparison of training time and sort time for the Tricaps-RL and DQN methods
Figure BDA0003216703240000102
Figure BDA0003216703240000111
From table 2, it can be seen that the training time of the tricas-RL method is much longer than that of the DQN method, mainly because when the classification performance of the method cannot be improved any more in training the tricas-RL method, the strategy is gradually switched to the strategy optimization using the ternary loss metric. The ternary loss metric strategy increases the number of computations and also makes the gradient change small at each iteration, so it takes longer when the classification performance reaches a new balance. Meanwhile, compared with DQN, the time of the Tricaps-RL method is longer in the classification stage, because the Tricaps-RL method needs to be subjected to averaging after multiple sample similarity comparisons are carried out. However, the classification time is still less than 0.1 second, and the real-time interaction requirement can be fully met in the interaction process with the doctor.
And respectively reconstructing images on the two network output characteristics so as to further compare and analyze the quality of the characteristics acquired by the two networks. For the DQN network, a symmetrical deconvolution network is added on the basis of the CNN network shown in Table 1 to realize reconstruction; for the algorithm, reconstruction is realized by adding a reconstruction network behind a capsule network according to the scheme in the document Dynamic Routing Between Capsules. Both were re-generated after 30000 training sessions with the single capsule network of TriCaps-RL and CNN in DQN, respectively, for the incoming symptom samples. Fig. 6 shows a comparison of the two results, the first row being the original image and the second and third rows being the image blocks generated by the capsule network and CNN, respectively. It can be seen from the figure that the image blocks of the signs generated by the capsule network are obviously finer and more remarkable than those generated by the CNN; the image generated by the CNN obviously loses many details, especially for the No. 1 and No. 9 symbolic graphs with unobvious texture contrast or the No. 14 and No. 18 symbolic graphs with large lesion areas, the CNN can not successfully express the characteristics, and only a blank graph is generated. The reconstruction of the capsule network is also not very good for the 9 th image, but we can find that it is also difficult for the human eye to discern the texture of its symptoms for this sample.
Correspondingly, the invention also provides a lung nodule analysis device based on the ternary capsule network algorithm, which comprises:
a data storage module for storing one or more programs;
the construction module is used for constructing a lung nodule analysis model based on a ternary capsule network algorithm;
the acquisition module is used for acquiring a lung image of a patient as an image to be analyzed;
the analysis module is used for dividing the image to be analyzed into regions to form image data, inputting the data into a pre-trained lung nodule analysis model and obtaining a classification result output by the analysis model;
the analysis result determining module is used for determining the analysis result of the image to be analyzed according to the classification result;
the output module is used for outputting data of the analysis result and the lung nodule image;
a central processor for executing the one or more programs that implement the method for lung nodule analysis provided herein above.
Further, the present invention also provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the lung nodule analyzing method as provided above.
While the invention has been described with reference to specific embodiments, the invention is not limited thereto, and those skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope of the invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims (10)

1. A lung nodule analysis method based on a ternary capsule network algorithm is characterized by comprising the following steps:
1) constructing a lung nodule analysis model based on a ternary capsule network algorithm, wherein the lung nodule analysis model comprises a memory pool module, a single capsule network module, a ternary capsule network module, an identification module and a result determination module;
2) acquiring an image to be analyzed;
3) learning a single capsule network module, and training a capsule network intelligent agent;
4) performing ternary capsule network module learning, and optimizing a capsule network intelligent agent based on the ternary capsule network loss;
5) inputting lung CT images into the learned capsule network, and performing sliding window processing on the CT images by the model to obtain a series of states;
6) and determining a data analysis result of the image to be analyzed according to the classification result, and outputting the data analysis result through a result determination module.
2. The lung nodule analysis method based on the ternary capsule network algorithm as claimed in claim 1, wherein in the step 1), the specific contents for constructing the lung nodule analysis model based on the ternary capsule network algorithm are as follows:
11) inputting a plurality of pictures, setting initial learning parameters including the memory space N of a memory pool D in a memory pool module and a temporary memory pool DTCapacity H, return value r, interval factor α, number of small batches B, number of training iterations EPOCHS, and capsule network initial parameter ωc；
12) For a CT image with 512 x 512 pixels, a sliding window with K x K pixels is adopted to obtain a 32 x 32 pixel image block on the CT image, and the coordinates (x) at the upper left corner of the image block are recordedi，yi) (ii) a Setting each image block to a state stFurther, all image blocks extracted from the CT image form a set S of all states;
13) state S in set S of capsule network pairstSelecting corresponding classification action a from action space AtOutput the class attribute c of this statej；
14) All the segments s taken from a CT image in the capsule networkiPerform the corresponding action atAnd after the detection is finished, displaying the lung nodule and the symptom classification result thereof detected by the capsule network back to the CT image according to the coordinate position of the image block region, thereby constructing a lung nodule analysis model based on a ternary capsule network algorithm.
3. The lung nodule analysis method based on the ternary capsule network algorithm as claimed in claim 2, wherein in the step 3), the specific contents for learning the single capsule network module are as follows:
keeping the size of a sliding window on the CT image unchanged, determining the number of image blocks extracted by each CT, determining the state number M of each round in the reinforcement learning process, comparing the difference between the single-capsule network diagnosis result and the calibration result, and correspondingly calculating the action return on each image block.
4. The lung nodule analysis method based on the ternary capsule network algorithm as claimed in claim 3, wherein the specific method for comparing the dissimilarity of the diagnosis result and the calibration result of the single capsule network is as follows:
for an image block stIf the single capsule network diagnosis is correct, a reward value, namely a return value r, is given; otherwise, giving the reward value of 0; the method for judging whether the single-capsule network diagnosis result is correct comprises the following steps:
a: if image block stIf the region is overlapped with a focus region of the calibration result, further checking whether the classification result of the image block is consistent with the symptom category label of the calibration result, if so, judging that the diagnosis result of the single capsule network is correct, otherwise, judging that the diagnosis result is wrong;
b: if image block stAnd if the area is not overlapped with any focus area of the calibration result and the classification result of the image block is negative, judging that the single capsule network diagnosis result is correct, otherwise, judging that the single capsule network diagnosis result is wrong.
5. The lung nodule analysis method based on the ternary capsule network algorithm as claimed in claim 4, wherein in the step 3), a fully-connected layer is added to the single capsule network after the feature expression to output the accumulated reward Q corresponding to each categorytargetThe value:
Qtarget＝M×r
the corresponding single step loss function is:
Figure FDA0003216703230000021
in the formula, L (omega)c) For the name of the loss function, R (x) is the return function,
Figure FDA0003216703230000022
is a capsule network agent and is a capsule network agent,
Figure FDA0003216703230000023
as anchor image, ajFor corresponding actions, ωcIs radix Ginseng;
and learning the parameters of the single-capsule network by adopting the corresponding single-step loss function of the single-capsule network until the loss function is smaller than a preset value.
6. The lung nodule analysis method based on the ternary capsule network algorithm according to claim 5, wherein in the step 4), the specific contents for learning the ternary capsule network module are as follows:
41) inputting three images at a time, respectively being anchor images
Figure FDA0003216703230000024
Positive sample image
Figure FDA0003216703230000025
And negative sample image
Figure FDA0003216703230000026
Respectively extracting features from the three images by utilizing the capsule network, and further calculating the Euclidean distance d between the anchor point image and the positive sample image+And the Euclidean distance d between the anchor point image and the negative sample image-And normalizing the data; calculating a ternary loss function on the basis;
42) converting the loss function of the single-layer capsule network into a ternary loss function;
43) the ternary capsule network receives a state from the state set S and calculates rewards Q corresponding to different categoriestargetValue, select corresponding QtargetExecuting the action with the maximum value, correspondingly entering the next state, and simultaneously storing elements such as the current state, the action and the like; the process is iterated until the agent respectively executes corresponding actions on all states acquired from a CT image by a sliding window method; the environment carries out classification results of the ternary capsule network on each image block and the calibrated classification resultsComparing, correspondingly calculating and storing the return value;
44) the system performs experience playback, randomly extracts a batch of data from the state, action and return combination stored in the memory pool, calculates the loss of the capsule network and updates the network weight.
7. The lung nodule analysis method based on the ternary capsule network algorithm as claimed in claim 6, wherein the calculation formula of the loss calculation and the network weight update of the capsule network is as follows:
Figure FDA0003216703230000031
in the formula, LcomposeGenerating a composite loss value; b is the number of data selected by experience playback in reinforcement learning, namely the number of previously-placed experience data is randomly selected from an experience pool; λ is two lost weight adjustment factors;
Figure FDA0003216703230000032
is a capsule network agent; alpha represents the minimum difference of the embedding expression distances of the anchor sample and the negative sample; l isi(d+，d-) Is a ternary loss function.
8. The lung nodule analysis method based on the ternary capsule network algorithm as claimed in claim 5, wherein in the step 6), the specific process of determining the data analysis result of the image to be analyzed according to the classification result is as follows:
suppose there is a sample s to be measuredAC is the number of all the classes considered, from the existing labeled sample set, N is randomly selected for each class of samplesPA total of N is obtained for each samplePX C samples; each sample sPRespectively associated with a sample s to be measuredAAfter combination form NPX C group of sample pairs
Figure FDA0003216703230000033
And after the distance corresponding to each group of sample pairs is calculated, calculating the average distance corresponding to each category, and taking the category with the minimum average distance as a classification result.
9. A pulmonary nodule analysis apparatus based on a ternary capsule network algorithm, comprising:
a data storage module storing one or more programs;
the construction module is used for constructing a lung nodule analysis model based on a ternary capsule network algorithm;
the acquisition module is used for acquiring a lung image of a patient and taking the lung image as an image to be analyzed;
the analysis module divides the image to be analyzed into regions to form image data, inputs the data into a lung nodule analysis model which is trained in advance, and obtains a classification result output by the lung nodule analysis model;
the analysis result determining module is used for determining the analysis result of the image to be analyzed according to the classification result;
the output module is used for outputting data of the analysis result and the lung nodule image;
a central processor executing one or more programs implementing the lung nodule analysis method based on the ternary capsule network algorithm of any one of claims 1 to 8.
10. A computer-readable storage medium having stored thereon a computer program for execution by a processor to implement the lung nodule analysis method based on a ternary capsule network algorithm according to any one of claims 1 to 8.
CN202110946389.9A 2021-08-18 2021-08-18 Pulmonary nodule analysis method and device based on ternary capsule network algorithm and storage medium Active CN113763332B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110946389.9A CN113763332B (en) 2021-08-18 2021-08-18 Pulmonary nodule analysis method and device based on ternary capsule network algorithm and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110946389.9A CN113763332B (en) 2021-08-18 2021-08-18 Pulmonary nodule analysis method and device based on ternary capsule network algorithm and storage medium

Publications (2)

Publication Number Publication Date
CN113763332A true CN113763332A (en) 2021-12-07
CN113763332B CN113763332B (en) 2024-05-31

Family

ID=78790266

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110946389.9A Active CN113763332B (en) 2021-08-18 2021-08-18 Pulmonary nodule analysis method and device based on ternary capsule network algorithm and storage medium

Country Status (1)

Country Link
CN (1) CN113763332B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117726992A (en) * 2024-02-07 2024-03-19 吉林大学 Nursing skill training auxiliary system and method

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108830826A (en) * 2018-04-28 2018-11-16 四川大学 A kind of system and method detecting Lung neoplasm
CN110570425A (en) * 2019-10-18 2019-12-13 北京理工大学 Lung nodule analysis method and device based on deep reinforcement learning algorithm
CN111325169A (en) * 2020-02-26 2020-06-23 河南理工大学 Deep video fingerprint algorithm based on capsule network
CN111767952A (en) * 2020-06-30 2020-10-13 重庆大学 Interpretable classification method for benign and malignant pulmonary nodules
CN112733701A (en) * 2021-01-07 2021-04-30 中国电子科技集团公司信息科学研究院 Robust scene recognition method and system based on capsule network
CN113208641A (en) * 2021-05-10 2021-08-06 山东大学 Pulmonary nodule auxiliary diagnosis method based on three-dimensional multi-resolution attention capsule network

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108830826A (en) * 2018-04-28 2018-11-16 四川大学 A kind of system and method detecting Lung neoplasm
CN110570425A (en) * 2019-10-18 2019-12-13 北京理工大学 Lung nodule analysis method and device based on deep reinforcement learning algorithm
CN111325169A (en) * 2020-02-26 2020-06-23 河南理工大学 Deep video fingerprint algorithm based on capsule network
CN111767952A (en) * 2020-06-30 2020-10-13 重庆大学 Interpretable classification method for benign and malignant pulmonary nodules
CN112733701A (en) * 2021-01-07 2021-04-30 中国电子科技集团公司信息科学研究院 Robust scene recognition method and system based on capsule network
CN113208641A (en) * 2021-05-10 2021-08-06 山东大学 Pulmonary nodule auxiliary diagnosis method based on three-dimensional multi-resolution attention capsule network

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117726992A (en) * 2024-02-07 2024-03-19 吉林大学 Nursing skill training auxiliary system and method
CN117726992B (en) * 2024-02-07 2024-04-16 吉林大学 Nursing skill training auxiliary system and method

Also Published As

Publication number Publication date
CN113763332B (en) 2024-05-31

Similar Documents

Publication Publication Date Title
Sahu et al. FINE_DENSEIGANET: Automatic medical image classification in chest CT scan using Hybrid Deep Learning Framework
Aggarwal et al. COVID-19 image classification using deep learning: Advances, challenges and opportunities
US9959615B2 (en) System and method for automatic pulmonary embolism detection
Tian et al. Multi-path convolutional neural network in fundus segmentation of blood vessels
US9687199B2 (en) Medical imaging system providing disease prognosis
WO2020070519A1 (en) Method for detecting adverse cardiac events
Shakeri et al. Deep spectral-based shape features for Alzheimer’s disease classification
Yao et al. Pneumonia Detection Using an Improved Algorithm Based on Faster R‐CNN
Bhuvaneswari et al. A new fusion model for classification of the lung diseases using genetic algorithm
Yoon et al. Medical image analysis using artificial intelligence
CN113095382B (en) Interpretable tuberculosis classification network identification method based on CT image
Chen et al. Alzheimer's disease diagnosis with brain structural mri using multiview-slice attention and 3D convolution neural network
CN113096137B (en) Adaptive segmentation method and system for OCT (optical coherence tomography) retinal image field
CN110570425B (en) Pulmonary nodule analysis method and device based on deep reinforcement learning algorithm
Yerukalareddy et al. Brain tumor classification based on mr images using GAN as a pre-trained model
Zhang et al. Learning from multiple annotators for medical image segmentation
Beddiar et al. A Deep learning based data augmentation method to improve COVID-19 detection from medical imaging
CN113763332B (en) Pulmonary nodule analysis method and device based on ternary capsule network algorithm and storage medium
Prodhan et al. Combination of the features of pre-trained Xception and VGG16 models to identify childhood pneumonia from chest X-ray images
Sameer et al. Brain tumor segmentation and classification approach for MR images based on convolutional neural networks
Sünkel et al. Hybrid quantum machine learning assisted classification of COVID-19 from computed tomography scans
JP7091475B2 (en) Similarity determination device, method and program
Wijerathna et al. Brain Tumor Detection Using Image Processing
Arman et al. Intracranial hemorrhage classification from ct scan using deep learning and bayesian optimization
Paul et al. Computer-Aided Diagnosis Using Hybrid Technique for Fastened and Accurate Analysis of Tuberculosis Detection with Adaboost and Learning Vector Quantization

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant