CN110717518B - Continuous lung nodule recognition method and device based on 3D convolutional neural network - Google Patents
Continuous lung nodule recognition method and device based on 3D convolutional neural network Download PDFInfo
- Publication number
- CN110717518B CN110717518B CN201910857918.0A CN201910857918A CN110717518B CN 110717518 B CN110717518 B CN 110717518B CN 201910857918 A CN201910857918 A CN 201910857918A CN 110717518 B CN110717518 B CN 110717518B
- Authority
- CN
- China
- Prior art keywords
- data
- feature vector
- clinical
- image
- inputting
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/03—Recognition of patterns in medical or anatomical images
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
- Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation
Abstract
The application discloses a method and a device for identifying persistent lung nodules based on a 3D convolutional neural network. The method comprises the following steps: acquiring feature data of a lung nodule, wherein the feature data comprises image data and clinical data; inputting the image data and the clinical data into a branch network respectively to extract image feature vectors and clinical feature vectors respectively; fusing the image feature vector and the clinical feature vector to obtain a fused feature vector; and respectively inputting the image feature vector and the fusion feature vector into the 3D convolutional neural network to obtain the probability that the lung nodule is a persistent lung nodule. According to the method and the device, the purpose of accurately judging the probability that the lung nodule is the persistent lung nodule according to the persistent lung nodule recognition model is achieved, so that the technical effect of improving the accuracy of the persistent lung nodule recognition is achieved, and the technical problem that the detection accuracy of the detection method of the persistent lung nodule in the related technology is not high enough is solved.
Description
Technical Field
The application relates to the technical field of deep learning, in particular to a method and a device for continuously identifying lung nodules based on a 3D convolutional neural network.
Background
Lung cancer is the malignant tumor with highest incidence in the world at present, and presents a trend of obviously increasing incidence, which seriously threatens the health of people. The lung nodule is a tiny lesion in the lung, has close relation with the formation of lung tumor, and plays a vital role in early screening of lung cancer. In recent years, with the development of CT technology and its increasingly wide application in clinic, the number of accidentally discovered lung nodules has increased significantly. Among the various types of nodules, the probability of sub-solid nodules (part-solid nodules) turning into malignancy is much higher than that of solid nodules (solid nodules) and pure ground glass nodules (ground glass nodules), about 62.5% -89.6%, however, of these accidentally detected sub-solid nodules, about 49-70% of the nodules will gradually decrease or disappear over the next 3 months. Therefore, accurately finding persistent nodules from detected nodules can prevent unnecessary radiation exposure and invasive examination for patients, reduce economic burden and mental stress for patients, and have important clinical significance.
At present, the existing lung nodule persistence research algorithm mostly relies on a manual mode to manually extract features, an experienced imaging doctor is required to manually outline the ROI of the nodule, then the image features in the ROI are extracted, the level of the extracted features of the method has larger difference due to subjective reasons such as doctor experience, and the like, so that great difficulty is brought to the recognition of the persistence nodule; in addition, feature extraction and target classification are realized as two processes, the dependence degree of the classification result on the features is high, and the classification accuracy is easily limited by the quality level of the features.
In recent years, with the development of a deep learning algorithm, a lung nodule detection and classification algorithm based on deep learning is widely applied, however, in the aspect of lung nodule persistence judgment, the current method has more difficulty in solving the problem of lung nodule persistence classification. Firstly, the size change range of the nodules is larger, and the nodules are mainly distributed between 3mm and 30 mm; secondly, the characteristics required to be extracted for classifying the nodules not only comprise the integral characteristics of the nodules, but also comprise the local characteristics of the nodules; finally, the determination of the persistence of the nodule is based on clinical features and lung background features, such as age, sex, non-mechanized inflammation, emphysema, multiple nodules, and other prior information, in addition to the imaging features.
Aiming at the problem that the detection accuracy of the lung nodule detection method in the related art is not high enough, no effective solution is proposed at present.
Disclosure of Invention
The main purpose of the application is to provide a method and a device for identifying persistent pulmonary nodules based on a 3D convolutional neural network, so as to achieve the technical effect of improving the accuracy of identifying the persistent pulmonary nodules, and further solve the technical problem that the detection accuracy of the detection method of the persistent pulmonary nodules in the related technology is not high enough.
To achieve the above object, according to one aspect of the present application, there is provided a persistent lung nodule recognition method based on a 3D convolutional neural network.
The persistent lung nodule identification method based on the 3D convolutional neural network comprises the following steps: acquiring feature data of a lung nodule, wherein the feature data comprises image data and clinical data; inputting the image data and the clinical data into a branch network respectively to extract image feature vectors and clinical feature vectors respectively; fusing the image feature vector and the clinical feature vector to obtain a fused feature vector; and respectively inputting the image feature vector and the fusion feature vector into the 3D convolutional neural network to obtain the probability that the lung nodule is a persistent lung nodule.
Further, the inputting the image data and the clinical data into a branch network to extract image feature vectors and clinical feature vectors, respectively, includes: acquiring a region of interest based on the image data; determining the position information of the lung nodule in the image according to the region of interest; assigning a value to the lung nodule in the image data according to a first preset rule based on the position information; and inputting the assigned image data into the branch network to extract the image feature vector.
Further, the inputting the image data and the clinical data into a branch network to extract image feature vectors and clinical feature vectors, respectively, includes: constructing an N-dimensional feature vector based on the clinical data, wherein N is not less than 1; establishing a corresponding relation between clinical data and the dimension of the N-dimensional feature vector; assigning values to the clinical data according to a second preset rule; inputting the assigned clinical data into the branch network to extract the clinical feature vector.
Further, the acquiring the feature data of the lung nodule, wherein the feature data comprises image data and clinical data, and then further comprises: randomly extracting M groups of data based on the image data and inputting the M groups of data into the branch network to extract image feature vectors, wherein M is not less than 1; inputting the image feature vector into the 3D convolutional neural network for first-stage training so as to extract initial model parameters; reversely updating the initial model parameters according to training results; and stopping updating the model parameters after the branch network converges so as to output first characteristic parameters.
Further, the inputting the image feature vector and the fusion feature vector into the 3D convolutional neural network, respectively, to obtain the probability that the lung nodule is a persistent lung nodule includes: inputting the fusion feature vector into the 3D convolutional neural network for second-stage training so as to output a second feature parameter; adjusting the first characteristic parameter and the second characteristic parameter to output an identification model of the persistent lung nodule; and inputting the characteristic data of the lung nodules into an identification model of the persistent lung nodules to obtain the probability that the lung nodules are persistent lung nodules.
To achieve the above object, according to another aspect of the present application, there is provided a persistent pulmonary nodule recognition device based on a 3D convolutional neural network.
A persistent pulmonary nodule recognition device based on a 3D convolutional neural network according to the present application includes: the acquisition module is used for acquiring the characteristic data of the lung nodule, wherein the characteristic data comprises image data and clinical data; the extraction module is used for respectively inputting the image data and the clinical data into a branch network so as to respectively extract image feature vectors and clinical feature vectors; the fusion module is used for fusing the image feature vector and the clinical feature vector to obtain a fusion feature vector; and the identification module is used for respectively inputting the image feature vector and the fusion feature vector into the 3D convolutional neural network to obtain the probability that the lung nodule is a persistent lung nodule.
Further, the extraction module includes: an acquisition unit for acquiring a region of interest based on the image data; a determining unit, configured to determine location information of the lung nodule in an image according to the region of interest; the first assignment unit is used for assigning values to lung nodules in the image data according to a first preset rule based on the position information; and the first extraction unit is used for inputting the assigned image data into the branch network so as to extract the image feature vector.
Further, the extraction module further comprises: a construction unit configured to construct an N-dimensional feature vector based on the clinical data, wherein N is not less than 1; the establishing unit is used for establishing the corresponding relation between the clinical data and the dimension of the N-dimensional feature vector; the second assignment unit is used for assigning values to the clinical data according to a second preset rule; and the second extraction unit is used for inputting the assigned clinical data into the branch network so as to extract the clinical feature vector.
Further, the apparatus further comprises: the input module is used for randomly extracting M groups of data based on the image data and inputting the M groups of data into the branch network so as to extract image feature vectors, wherein M is not less than 1; the training module is used for inputting the image feature vector into the 3D convolutional neural network to perform first-stage training so as to extract initial model parameters; the updating module is used for reversely updating the initial model parameters according to the training result; and the output module is used for stopping updating the model parameters after the branch network converges so as to output the first characteristic parameters.
Further, the identification module includes: the input unit is used for inputting the fusion feature vector into the 3D convolutional neural network to perform second-stage training so as to output a second feature parameter; the adjusting unit is used for adjusting the first characteristic parameters and the second characteristic parameters to output an identification model of the persistent lung nodule; and the output unit is used for inputting the characteristic data of the lung nodules into the recognition model of the persistent lung nodules so as to obtain the probability that the lung nodules are persistent lung nodules.
In the embodiment of the application, a mode of acquiring feature data of a lung nodule, wherein the feature data comprises image data and clinical data is adopted, and the image data and the clinical data are respectively input into a branch network to respectively extract image feature vectors and clinical feature vectors; fusing the image feature vector and the clinical feature vector to obtain a fused feature vector; the image feature vector and the fusion feature vector are respectively input into the 3D convolutional neural network, so that the purpose of accurately judging the probability of a lung nodule as a persistent lung nodule according to a persistent lung nodule recognition model is achieved, the technical effect of improving the accuracy of the persistent lung nodule recognition is achieved, and the technical problem that the detection accuracy of a lung nodule detection method in the related art is not high enough is solved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this application, are included to provide a further understanding of the application and to provide a further understanding of the application with regard to the other features, objects and advantages of the application. The drawings of the illustrative embodiments of the present application and their descriptions are for the purpose of illustrating the present application and are not to be construed as unduly limiting the present application. In the drawings:
FIG. 1 is a flow chart of a method for persistent lung nodule identification based on a 3D convolutional neural network according to a first embodiment of the present application;
FIG. 2 is a schematic flow chart of extracting feature vectors based on a 3D convolutional neural network according to an embodiment of the present application;
FIG. 3 is a flow chart of a method for persistent lung nodule identification based on a 3D convolutional neural network according to a second embodiment of the present application;
FIG. 4 is a flow chart of a method for persistent lung nodule identification based on a 3D convolutional neural network according to a third embodiment of the present application;
FIG. 5 is a flow chart of a method for persistent lung nodule identification based on a 3D convolutional neural network according to a fourth embodiment of the present application;
FIG. 6 is a flow chart of a method for persistent lung nodule identification based on a 3D convolutional neural network according to a fifth embodiment of the present application;
FIG. 7 is a schematic diagram of the composition and structure of a persistent pulmonary nodule recognition device based on a 3D convolutional neural network according to a first embodiment of the present application;
FIG. 8 is a schematic diagram of the composition and structure of a persistent pulmonary nodule recognition device based on a 3D convolutional neural network according to a second embodiment of the present application;
FIG. 9 is a schematic diagram of the composition and structure of a persistent pulmonary nodule recognition device based on a 3D convolutional neural network according to a third embodiment of the present application; and
fig. 10 is a schematic diagram of the composition and structure of a persistent pulmonary nodule recognition apparatus based on a 3D convolutional neural network according to a fourth embodiment of the present application.
Detailed Description
In order to make the present application solution better understood by those skilled in the art, the following description will be made in detail and with reference to the accompanying drawings in the embodiments of the present application, it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, shall fall within the scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate in order to describe the embodiments of the present application described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
It should be noted that, in the case of no conflict, the embodiments and features in the embodiments may be combined with each other. The present application will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
According to an embodiment of the present invention, there is provided a method for identifying persistent lung nodules based on a 3D convolutional neural network, as shown in fig. 1, including the following steps S101 to S104:
in step S101, feature data of the lung nodule is acquired, the feature data including image data and clinical data.
In specific implementation, the main network adopted in the embodiment of the application is a 3D convolutional neural network, the 3D convolutional neural network comprises two branch networks, wherein the input of one branch network is image data of a lung nodule, namely, a 48 x 48 voxel size cut block can be obtained by cutting according to the position of the lung nodule in a CT image, and the input of the other branch network is clinical data of the lung nodule, namely, prior information such as clinical characteristics of a patient, lung background characteristics and the like can be obtained according to the data such as a case report of the patient.
Step S102, inputting the image data and the clinical data into a branch network respectively to extract image feature vectors and clinical feature vectors respectively.
In specific implementation, the two branch networks included in the 3D convolutional neural network are an image feature extraction branch network and a clinical feature extraction branch network, and specifically, as shown in fig. 2, the lung nodule image feature extraction branch network is mainly based on the thought of a feature pyramid, and is divided into four downsampling processes and two upsampling processes, where the downsampling processes and the upsampling processes are both based on currently mainstream convolutional modules, including, but not limited to, a sensitivity module, an acceptance module, a Res module, and the like. The down sampling process adds a layer of maximum pooling layer after the convolution module, the process can reduce the data dimension, retain the key information of the image features, enable the convolution module of the next layer to have a larger local perception field of view, extract the image features of higher latitude, and lose some bottom layer features and local features of the data at the same time. Therefore, two up-sampling processes are added after the down-sampling process, wherein the up-sampling process is to add a deconvolution layer after the convolution module, and splice the feature vector obtained after each up-sampling with the feature vector obtained after the corresponding down-sampling process, so as to realize the fusion of the high-level features and the low-level features of the data. Finally, the 3 x 64, 6 x 64 and 12 x 64 features extracted by the model are respectively extracted to serve as integral features, high-dimensional local features and low-dimensional local features of the image data, and the features are spliced together to form a group of 192-dimensional image feature vectors to serve as a part of 3D convolutional neural network input.
The other branch of the 3D convolutional neural network is a clinical feature extraction branch network, the input data of the branch may be a 16-dimensional feature vector containing clinical features and pulmonary background features of the patient, each dimension of the vector corresponds to one clinical feature and pulmonary background feature of the patient, and the clinical feature vector is extracted based on the input clinical data.
Step S103, fusing the image feature vector and the clinical feature vector to obtain a fused feature vector.
In the implementation, after the 16-dimensional clinical feature vector is extracted, the 16-dimensional clinical feature vector is spliced with the 192-dimensional image feature vector extracted by the image feature extraction branch network to serve as another part of the input of the 3D convolutional neural network.
Step S104, the image feature vector and the fusion feature vector are respectively input into the 3D convolutional neural network, so that the probability that the lung nodule is a persistent lung nodule is obtained.
In specific implementation, the image feature vector is input into a 3D convolutional neural network to perform first-stage training, then the fused feature vector obtained by fusing the image feature vector and the clinical feature vector is input into the 3D convolutional neural network to perform second-stage training on a full connection layer in the model, and after 10 epochs (each epoch refers to iterate one round by using all data) are completed, the whole model can be finely tuned by using a smaller learning rate until loss of the model converges. And finally, testing on a test set by using a trained model, wherein the output result of the model is the probability of judging a nodule to be a persistent nodule.
As a preferred implementation manner of the embodiment of the present application, as shown in fig. 3, the inputting the image data and the clinical data into the branch network to extract the image feature vector and the clinical feature vector respectively includes the following steps S201 to S204:
step S201, acquiring a region of interest based on the image data.
In practice, the data needs to be preprocessed, including image data processing and clinical data processing, before the image data and clinical data are input into the branch network for training. The image data processing process firstly extracts the ROI (region of interest), and the specific method can cut the ROI containing the nodule and the peripheral tissues by taking the coordinates of the center point of the nodule as the center and taking the length of 64 pixel points as the side length.
Step S202, determining the position information of the lung nodule in the image according to the region of interest.
In the specific implementation, after the ROI area is acquired, the coordinates of the center point of the lung nodule in the image are calculated according to the position information of the center point of the lung nodule in the data, the sampling layer thickness of the image and the initial position information.
And step S203, assigning values to the lung nodules in the image data according to a first preset rule based on the position information.
In the specific implementation, the three-dimensional lung CT image is cut according to the position information of the center point of the lung nodule, the sampling layer thickness of the image, the initial position information and the like, and the corresponding label is marked, and the specific assignment rule can be that the temporary nodule is assigned to 0 and the persistent nodule is assigned to 1.
Step S204, inputting the assigned image data into the branch network to extract the image feature vector.
In the specific implementation, the proportion of the positive samples (persistent nodules) to the negative samples (temporary nodules) in the data set is unbalanced, so that the data set is balanced by adopting a positive sample amplification mode, and specific methods comprise data rotation, offset, noise increase and the like; and then performing data normalization operation, namely performing normalization operation on HU values of the ROI region, and linearly normalizing and adjusting data with HU values in the range of [ -1000,400] to the interval of [0,1] according to the reference range of the normal human CT values and the image data distribution, wherein the data with HU values larger than 400 are uniformly converted into 1, and the values with HU values smaller than-1000 are uniformly converted into 0. And finally, inputting the image data obtained after the processing into a branch network to extract the image feature vector.
As a preferred implementation manner of the embodiment of the present application, as shown in fig. 4, the inputting the image data and the clinical data into the branch network to extract the image feature vector and the clinical feature vector respectively includes the following steps S301 to S304:
step S301, constructing an N-dimensional feature vector based on the clinical data, wherein N is not less than 1.
In practice, the clinical data processing process is to construct a 16-dimensional feature vector according to the clinical feature information of the patient and the priori knowledge of the doctor.
Step S302, establishing a corresponding relation between clinical data and the dimension of the N-dimensional feature vector.
In specific implementation, based on the constructed 16-dimensional feature vector, a corresponding relation between each clinical feature in the clinical data and one or more dimensions in the 16-dimensional feature vector is established.
And step S303, assigning values to the clinical data according to a second preset rule.
In the specific implementation, the clinical feature corresponding to each dimension in the 16-dimensional feature vector is assigned based on the corresponding relation, and a specific assignment rule can be that-1 indicates that the clinical feature is not present, and 1 indicates that the clinical feature is present.
And step S304, inputting the assigned clinical data into the branch network to extract the clinical feature vector.
In specific implementation, the clinical data obtained after assignment is input into another branch network for training so as to extract clinical feature vectors.
As a preferred implementation manner of the embodiment of the present application, as shown in fig. 5, the acquiring feature data of the lung nodule, where the feature data includes image data and clinical data, further includes the following steps S401 to S404:
step S401, randomly extracting M sets of data based on the image data, and inputting the M sets of data into the branch network to extract image feature vectors, wherein M is not less than 1.
In specific implementation, 16 or 32 groups of data and labels corresponding to the data are randomly extracted from the image data in each training process, and the data are uniformly input into an image feature extraction branch network to extract image feature vectors.
Step S402, inputting the image feature vector into the 3D convolutional neural network for a first stage training to extract initial model parameters.
In specific implementation, the image data is transmitted forward in the image feature extraction branch network to obtain a predicted result, namely an initial model parameter, and a loss value of the iteration is calculated based on a label affecting the data.
And step S403, reversely updating the initial model parameters according to the training result.
In particular, the initial model parameters in the model are updated inversely based on the loss value to correct the model parameters.
And step S404, stopping updating the model parameters after the branch network converges so as to output first characteristic parameters.
When the method is implemented, after the loss value iteratively output by the branch network converges, the model training is higher in completion, namely updating of the model parameters is stopped, the first-stage training is completed, and the first characteristic parameters are output according to the trained model.
As a preferred implementation manner of the embodiment of the present application, as shown in fig. 6, the inputting the image feature vector and the fusion feature vector into the 3D convolutional neural network respectively to obtain the probability that the lung nodule is a persistent lung nodule includes the following steps S501 to S503:
step S501, inputting the fusion feature vector into the 3D convolutional neural network for performing a second stage training, so as to output a second feature parameter.
In specific implementation, the embodiment of the application fuses the 192-dimensional image feature vector extracted by the image feature extraction branch network and the 16-dimensional clinical feature vector extracted by the clinical feature extraction branch network together, and inputs the fused feature vector into the 3D convolutional neural network for second-stage training, and outputs second feature parameters according to a model obtained by the second-stage training.
Preferably, before the fused feature vector is input into the 3D convolutional neural network to perform the second stage training to output the second feature parameter, the first feature parameter obtained after the first stage training is further required to be loaded, the parameters of the full-connection layer are adjusted from 192 dimensions to 208 dimensions, and then the first feature parameter of the image feature extraction part in the model is frozen.
Step S502, adjusting the first feature parameter and the second feature parameter to output an identification model of the persistent lung nodule.
In the specific implementation, after all data are iterated for 10 times, the first characteristic parameters are unfrozen, then the first characteristic parameters and the second characteristic parameters of the model obtained after the iteration are finely adjusted by using a smaller learning rate until the loss value of the model is converged, and a final recognition model of the persistent lung nodule is output.
Step S503, inputting the feature data of the lung nodule into the recognition model of the persistent lung nodule to obtain the probability that the lung nodule is the persistent lung nodule.
In the specific implementation, after the recognition model of the persistent lung nodule is obtained, the recognition model is used for testing on a test set, and the output result of the recognition model is the probability of judging that one lung nodule is the persistent nodule.
From the above description, it can be seen that the following technical effects are achieved: the method comprises the steps that characteristic data of lung nodules are obtained, wherein the characteristic data comprise image data and clinical data, and the image data and the clinical data are respectively input into a branch network to respectively extract image characteristic vectors and clinical characteristic vectors; fusing the image feature vector and the clinical feature vector to obtain a fused feature vector; the image feature vector and the fusion feature vector are respectively input into the 3D convolutional neural network, so that the purpose of accurately judging the probability of the lung nodule as the persistent lung nodule according to the persistent lung nodule identification model is achieved, and the technical effect of improving the accuracy of the persistent lung nodule identification is achieved.
It should be noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system such as a set of computer executable instructions, and that although a logical order is illustrated in the flowcharts, in some cases the steps illustrated or described may be performed in an order other than that illustrated herein.
There is further provided, according to an embodiment of the present invention, an apparatus for implementing the persistent pulmonary nodule identification method based on a 3D convolutional neural network, as shown in fig. 7, where the apparatus includes: the device comprises an acquisition module 1, an extraction module 2, a fusion module 3 and an identification module 4.
The acquiring module 1 of the embodiment of the present application is configured to acquire feature data of a lung nodule, where the feature data includes image data and clinical data.
In specific implementation, the main network adopted in the embodiment of the application is a 3D convolutional neural network, the 3D convolutional neural network comprises two branch networks, wherein the input of one branch network is image data of a lung nodule, namely, a 48 x 48 voxel size cut block can be obtained by cutting according to the position of the lung nodule in a CT image, the input of the other branch network is clinical data of the lung nodule, namely, prior information such as clinical characteristics of a patient and pulmonary background characteristics can be obtained according to data such as a case report of the patient.
The extraction module 2 of the embodiment of the present application is configured to input the image data and the clinical data into a branch network respectively, so as to extract an image feature vector and a clinical feature vector respectively.
In the implementation, the two branch networks included in the 3D convolutional neural network are an image feature extraction branch network and a clinical feature extraction branch network, and the image data and the clinical data are respectively input into the image feature extraction branch network and the clinical feature extraction branch network through the extraction module so as to respectively extract an image feature vector and a clinical feature vector.
The fusion module 3 of the embodiment of the present application is configured to fuse the image feature vector with the clinical feature vector, so as to obtain a fused feature vector.
In the implementation, after the 16-dimensional clinical feature vector is extracted, the 16-dimensional clinical feature vector is spliced with the 192-dimensional image feature vector extracted by the image feature extraction branch network through the fusion module to serve as the other part of the input of the 3D convolutional neural network.
The identification module 4 of the embodiment of the present application is configured to input the image feature vector and the fusion feature vector into the 3D convolutional neural network respectively, so as to obtain a probability that the lung nodule is a persistent lung nodule.
In specific implementation, the image feature vector is input into the 3D convolutional neural network through the identification module to perform first-stage training, then the fusion feature vector obtained by fusing the image feature vector and the clinical feature vector is input into the 3D convolutional neural network to perform second-stage training on the full connection layer in the model, and after 10 epochs (each epoch refers to iterating one round by using all data) are completed, the whole model can be finely tuned by using a smaller learning rate until loss of the model converges. And finally, testing on a test set by using a trained model, wherein the output result of the model is the probability of judging a nodule to be a persistent nodule.
As a preferred implementation of the embodiment of the present application, as shown in fig. 8, the extraction module 2 includes: the acquisition unit 21, the determination unit 22, the first assignment unit 23, and the first extraction unit 24.
The acquiring unit 21 of the embodiment of the present application is configured to acquire a region of interest based on the image data.
In practice, the data needs to be preprocessed, including image data processing and clinical data processing, before the image data and clinical data are input into the branch network for training. The image data processing process firstly extracts the ROI (region of interest) through an acquisition unit, and the specific method can cut the ROI containing the nodule and the peripheral tissues by taking the coordinates of the center point of the nodule as the center and taking the length of 64 pixel points as the side length.
The determining unit 22 of the present embodiment is configured to determine, according to the region of interest, location information of the lung nodule in the image.
In the specific implementation, after the ROI area is acquired, the center point coordinate of the lung nodule in the image is calculated by a determining unit according to the position information of the center point of the lung nodule in the data, the sampling layer thickness of the image and the initial position information.
The first assigning unit 23 of the present embodiment is configured to assign a value to a lung nodule in the image data according to a first preset rule based on the location information.
In the specific implementation, the first assignment unit performs cutting operation on the three-dimensional lung CT image according to the position information of the center point of the lung nodule, the sampling layer thickness of the image, the initial position information and the like, and marks the corresponding label, and the specific assignment rule can be that the temporary nodule is assigned to 0 and the persistent nodule is assigned to 1.
The first extracting unit 24 of the present embodiment is configured to input the assigned image data into the branch network to extract the image feature vector.
In the specific implementation, the proportion of the positive samples (persistent nodules) to the negative samples (temporary nodules) in the data set is unbalanced, so that the data set is balanced by adopting a positive sample amplification mode, and specific methods comprise data rotation, offset, noise increase and the like; and then performing data normalization operation, namely performing normalization operation on HU values of the ROI region, linearly normalizing and adjusting data with HU values within the range of < -1000 > and 400 > to a [0,1] interval according to the normal human body CT value reference range and the image data distribution, uniformly converting data with HU values larger than 400 into 1, and uniformly converting values with HU values smaller than-1000 into 0. And finally, inputting the image data obtained after the processing into a branch network through a first extraction unit to extract the image feature vector.
As a preferred implementation of the embodiment of the present application, as shown in fig. 8, the extraction module 2 further includes: the construction unit 25, the establishment unit 26, the second assignment unit 27 and the second extraction unit 28.
The construction unit 25 of the embodiment of the present application is configured to construct an N-dimensional feature vector based on the clinical data, where N is not less than 1.
In practice, the clinical data processing process is first to construct a 16-dimensional feature vector by a construction unit according to the clinical feature information of the patient and the priori knowledge of the doctor.
The establishing unit 26 of the embodiment of the present application is configured to establish a correspondence between clinical data and dimensions of the N-dimensional feature vector.
In specific implementation, based on the constructed 16-dimensional feature vector, a corresponding relation between each clinical feature in the clinical data and one or more dimensions in the 16-dimensional feature vector is established through an establishing unit.
The second assigning unit 27 of the embodiment is configured to assign values to the clinical data according to a second preset rule.
In the specific implementation, the second assignment unit is used for assigning the clinical feature corresponding to each dimension in the 16-dimensional feature vector based on the corresponding relation, and the specific assignment rule can be that-1 represents that the clinical feature does not exist, and 1 represents that the clinical feature exists.
The second extracting unit 28 of the embodiment is configured to input the assigned clinical data into the branch network to extract the clinical feature vector.
In specific implementation, the clinical data obtained after assignment is input into another branch network for training through a second extraction unit, so that clinical feature vectors are extracted.
As a preferred implementation of the embodiment of the present application, as shown in fig. 9, the apparatus further includes: input module 5, training module 6, update module 7 and output module 8.
The input module 5 of the embodiment of the present application is configured to randomly extract M groups of data based on the image data, and input the M groups of data into the branch network, so as to extract an image feature vector, where M is not less than 1.
In specific implementation, 16 or 32 groups of data and labels corresponding to the data are randomly extracted from the image data in each training process, and the data are uniformly input into an image feature extraction branch network through an input module to extract image feature vectors.
The training module 6 of the embodiment of the present application is configured to input the image feature vector into the 3D convolutional neural network for performing a first stage training to extract initial model parameters.
In specific implementation, the image data is transmitted forward in the image feature extraction branch network through the training module to obtain a predicted result, namely an initial model parameter, and a loss value of the iteration is calculated based on the label affecting the data.
The updating module 7 of the embodiment of the present application is configured to reversely update the initial model parameter according to a training result.
In specific implementation, the initial model parameters in the model are reversely updated through the updating module based on the loss value so as to correct the model parameters.
The output module 8 of the embodiment of the present application is configured to stop updating the model parameter after the branch network converges, so as to output a first feature parameter.
When the method is implemented, after the loss value iteratively output by the branch network converges, the model training is higher in completion, namely updating of the model parameters is stopped, the first-stage training is completed, and the first characteristic parameters are output through the output module according to the trained model.
As a preferred implementation of the embodiment of the present application, as shown in fig. 10, the identification module 4 includes: an input unit 41, an adjustment unit 42 and an output unit 43.
The input unit 41 of the embodiment of the present application is configured to input the fusion feature vector into the 3D convolutional neural network for performing a second stage training, so as to output a second feature parameter.
In specific implementation, the embodiment of the application fuses the 192-dimensional image feature vector extracted by the image feature extraction branch network and the 16-dimensional clinical feature vector extracted by the clinical feature extraction branch network together, and inputs the 192-dimensional image feature vector and the 16-dimensional clinical feature vector as fused feature vectors to the 3D convolutional neural network through the input unit for second-stage training, and outputs second feature parameters according to a model obtained by the second-stage training.
Preferably, before the fused feature vector is input into the 3D convolutional neural network to perform the second stage training to output the second feature parameter, the first feature parameter obtained after the first stage training is further required to be loaded, the parameters of the full-connection layer are adjusted from 192 dimensions to 208 dimensions, and then the first feature parameter of the image feature extraction part in the model is frozen.
The adjusting unit 42 of the embodiment is configured to adjust the first feature parameter and the second feature parameter to output an identification model of the persistent lung nodule.
In the specific implementation, after all data are iterated for 10 times, the first characteristic parameters are unfrozen, then the first characteristic parameters and the second characteristic parameters of the model obtained after the iteration are finely adjusted by the adjusting unit by using a smaller learning rate until the loss value of the model is converged, and a final recognition model of the persistent lung nodule is output.
The output unit 43 of the embodiment of the present application is configured to input the feature data of the lung nodule into the recognition model of persistent lung nodule to obtain the probability that the lung nodule is persistent lung nodule.
In specific implementation, after the recognition model of the persistent lung nodule is obtained, the recognition model is used for testing on a test set, and the output result of the recognition model is that the probability that one lung nodule is the persistent nodule is judged through the output unit.
It will be apparent to those skilled in the art that the modules or steps of the invention described above may be implemented in a general purpose computing device, they may be concentrated on a single computing device, or distributed across a network of computing devices, or they may alternatively be implemented in program code executable by computing devices, such that they may be stored in a memory device for execution by the computing devices, or they may be separately fabricated into individual integrated circuit modules, or multiple modules or steps within them may be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.
The foregoing description is only of the preferred embodiments of the present application and is not intended to limit the same, but rather, various modifications and variations may be made by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.
Claims (4)
1. A method for persistent pulmonary nodule identification based on a 3D convolutional neural network, wherein the 3D convolutional neural network comprises a first branch network, a second branch network, and a classifier, the method comprising:
acquiring feature data of a lung nodule, wherein the feature data comprises image data and clinical data;
inputting the image data into a first branch network to extract image feature vectors;
inputting the clinical data into a second branch network as a clinical feature vector;
fusing the image feature vector and the clinical feature vector to obtain a fused feature vector;
inputting the fusion feature vector into a classifier to obtain the probability that the lung nodule is a persistent lung nodule;
wherein, the inputting the image data into the first branch network, extracting the image feature vector includes:
acquiring a region of interest based on the image data;
determining the position information of the lung nodule in the image according to the region of interest;
assigning a value to the lung nodule in the image data according to a first preset rule based on the position information;
training the first branch network according to the assigned image data to extract the image feature vector;
the method for acquiring the characteristic data of the lung nodule comprises the following steps of:
randomly extracting M groups of data based on the image data and inputting the M groups of data into the first branch network to extract image feature vectors, wherein M is not less than 1;
inputting the image feature vector into the classifier for first-stage training so as to extract initial model parameters;
reversely updating the initial model parameters according to training results;
stopping updating the model parameters after the first branch network converges so as to output first characteristic parameters;
the inputting the fusion feature vector into the classifier to obtain a probability that the lung nodule is a persistent lung nodule comprises:
inputting the fusion feature vector into the 3D convolutional neural network for second-stage training so as to output a second feature parameter;
adjusting the first characteristic parameter and the second characteristic parameter to output an identification model of the persistent lung nodule; and inputting the characteristic data of the lung nodules into an identification model of the persistent lung nodules to obtain the probability that the lung nodules are persistent lung nodules.
2. The 3D convolutional neural network-based persistent pulmonary nodule recognition method of claim 1, wherein the inputting the clinical data into the second branch network as a clinical feature vector comprises:
constructing an N-dimensional feature vector based on the clinical data, wherein N is not less than 1;
establishing a corresponding relation between clinical data and the dimension of the N-dimensional feature vector;
assigning values to the clinical data according to a second preset rule;
and inputting the assigned clinical data into the second branch network to serve as the clinical feature vector.
3. A sustained pulmonary nodule recognition apparatus based on a 3D convolutional neural network, the 3D convolutional neural network comprising a first branch network, and a second branch network and a classifier, the apparatus comprising:
the acquisition module is used for acquiring the characteristic data of the lung nodule, wherein the characteristic data comprises image data and clinical data;
the extraction module is used for inputting the image data into a first branch network to extract image feature vectors;
inputting the clinical data into a second branch network as a clinical feature vector;
the fusion module is used for fusing the image feature vector and the clinical feature vector to obtain a fusion feature vector;
the recognition module is used for inputting the fusion feature vector into the classifier to obtain the probability that the lung nodule is a persistent lung nodule;
wherein, the extraction module includes:
an acquisition unit for acquiring a region of interest based on the image data;
a determining unit, configured to determine location information of the lung nodule in an image according to the region of interest;
the first assignment unit is used for assigning values to lung nodules in the image data according to a first preset rule based on the position information;
the first extraction unit is used for training the first branch network according to the assigned image data so as to extract the image feature vector;
the input module is used for randomly extracting M groups of data based on the image data and inputting the M groups of data into the first branch network so as to extract image feature vectors, wherein M is not less than 1;
the training module is used for inputting the image feature vector into the 3D convolutional neural network to perform first-stage training so as to extract initial model parameters;
the updating module is used for reversely updating the initial model parameters according to the training result;
the output module is used for stopping updating the model parameters after the first branch network converges so as to output first characteristic parameters;
the identification module comprises:
the input unit is used for inputting the fusion feature vector into the classifier for second-stage training so as to output a second feature parameter;
the adjusting unit is used for adjusting the first characteristic parameters and the second characteristic parameters to output an identification model of the persistent lung nodule;
and the output unit is used for inputting the characteristic data of the lung nodules into the recognition model of the persistent lung nodules so as to obtain the probability that the lung nodules are persistent lung nodules.
4. The 3D convolutional neural network-based persistent lung nodule recognition device of claim 3, wherein the extraction module further comprises:
a construction unit configured to construct an N-dimensional feature vector based on the clinical data, wherein N is not less than 1;
the establishing unit is used for establishing the corresponding relation between the clinical data and the dimension of the N-dimensional feature vector;
the second assignment unit is used for assigning values to the clinical data according to a second preset rule;
and the second extraction unit is used for inputting the assigned clinical data into the second branch network to serve as the clinical feature vector.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910857918.0A CN110717518B (en) | 2019-09-10 | 2019-09-10 | Continuous lung nodule recognition method and device based on 3D convolutional neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910857918.0A CN110717518B (en) | 2019-09-10 | 2019-09-10 | Continuous lung nodule recognition method and device based on 3D convolutional neural network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110717518A CN110717518A (en) | 2020-01-21 |
| CN110717518B true CN110717518B (en) | 2023-06-30 |
Family
ID=69210376
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910857918.0A Active CN110717518B (en) | 2019-09-10 | 2019-09-10 | Continuous lung nodule recognition method and device based on 3D convolutional neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110717518B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11610316B2 (en) * | 2020-03-06 | 2023-03-21 | Siemens Healthcare Gmbh | Method of computing a boundary |
| CN111382801B (en) * | 2020-03-16 | 2021-09-10 | 推想医疗科技股份有限公司 | Medical image classification method, device, equipment and storage medium |
| CN111127467B (en) * | 2020-03-31 | 2020-07-14 | 上海联影智能医疗科技有限公司 | Image quantization method, computer device, and storage medium |
| CN112927217B (en) * | 2021-03-23 | 2022-05-03 | 内蒙古大学 | Thyroid nodule invasiveness prediction method based on target detection |
| CN113743414B (en) * | 2021-08-02 | 2022-08-05 | 清华大学 | Method, device and system for identifying focus based on semantic coding |
| CN113724200A (en) * | 2021-08-02 | 2021-11-30 | 武汉大学 | Lung nodule evolution prediction method, device and equipment based on heterogeneous data |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108921195A (en) * | 2018-05-31 | 2018-11-30 | 沈阳东软医疗系统有限公司 | A kind of Lung neoplasm image-recognizing method neural network based and device |
| CN110084237A (en) * | 2019-05-09 | 2019-08-02 | 北京化工大学 | Detection model construction method, detection method and the device of Lung neoplasm |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107280697A (en) * | 2017-05-15 | 2017-10-24 | 北京市计算中心 | Lung neoplasm grading determination method and system based on deep learning and data fusion |
| CN107273925B (en) * | 2017-06-12 | 2020-10-09 | 太原理工大学 | Lung parenchyma CT image processing device based on local receptive field and semi-supervised depth self-coding |
| CN109544510B (en) * | 2018-10-24 | 2021-10-26 | 广州大学 | Three-dimensional lung nodule identification method based on convolutional neural network |
| CN109671055B (en) * | 2018-11-27 | 2021-04-02 | 杭州深睿博联科技有限公司 | Pulmonary nodule detection method and device |
-
2019
- 2019-09-10 CN CN201910857918.0A patent/CN110717518B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108921195A (en) * | 2018-05-31 | 2018-11-30 | 沈阳东软医疗系统有限公司 | A kind of Lung neoplasm image-recognizing method neural network based and device |
| CN110084237A (en) * | 2019-05-09 | 2019-08-02 | 北京化工大学 | Detection model construction method, detection method and the device of Lung neoplasm |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110717518A (en) | 2020-01-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110717518B (en) | Continuous lung nodule recognition method and device based on 3D convolutional neural network | |
| CN110956635B (en) | Lung segment segmentation method, device, equipment and storage medium | |
| CN109376636B (en) | Capsule network-based eye fundus retina image classification method | |
| CN112270660B (en) | Nasopharyngeal carcinoma radiotherapy target area automatic segmentation method based on deep neural network | |
| CN109584209B (en) | Vascular wall plaque recognition apparatus, system, method, and storage medium | |
| CN105574859B (en) | A kind of liver neoplasm dividing method and device based on CT images | |
| CN110889826B (en) | Eye OCT image focus region segmentation method, device and terminal equipment | |
| CN109872325B (en) | Full-automatic liver tumor segmentation method based on two-way three-dimensional convolutional neural network | |
| CN107563434B (en) | Brain MRI image classification method and device based on three-dimensional convolutional neural network | |
| CN113256641B (en) | Skin lesion image segmentation method based on deep learning | |
| CN108764342B (en) | Semantic segmentation method for optic discs and optic cups in fundus image | |
| Bulten et al. | Unsupervised prostate cancer detection on H&E using convolutional adversarial autoencoders | |
| Rehman et al. | Microscopic retinal blood vessels detection and segmentation using support vector machine and K‐nearest neighbors | |
| CN112927799B (en) | Life analysis system integrating multi-example learning and multi-task depth image histology | |
| CN113239755B (en) | Medical hyperspectral image classification method based on space-spectrum fusion deep learning | |
| CN105608687A (en) | Medical image processing method and medical image processing device | |
| CN113450305B (en) | Medical image processing method, system, equipment and readable storage medium | |
| Fernandez-Granero et al. | Automatic CDR estimation for early glaucoma diagnosis | |
| WO2020110774A1 (en) | Image processing device, image processing method, and program | |
| CN115471470A (en) | Esophageal cancer CT image segmentation method | |
| CN110458842B (en) | Brain tumor segmentation method based on two-channel three-dimensional dense connection network | |
| CN117612711B (en) | Multi-mode prediction model construction method and system for analyzing liver cancer recurrence data | |
| US11521323B2 (en) | Systems and methods for generating bullseye plots | |
| CN111814832A (en) | Target detection method, device and storage medium | |
| EP4167184A1 (en) | Systems and methods for plaque identification, plaque composition analysis, and plaque stability detection |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |