CN111798424B

CN111798424B - Medical image-based nodule detection method and device and electronic equipment

Info

Publication number: CN111798424B
Application number: CN202010621972.8A
Authority: CN
Inventors: 张佳琦; 吕晨翀; 丁佳; 王子腾; 孙安澜; 胡阳
Original assignee: Guangxi Yizhun Intelligent Technology Co ltd
Current assignee: Guangxi Yizhun Intelligent Technology Co ltd
Priority date: 2020-06-30
Filing date: 2020-06-30
Publication date: 2021-02-09
Anticipated expiration: 2040-06-30
Also published as: CN111798424A

Abstract

A nodule detection method based on medical images takes an image sequence based on medical images as input, positions candidate nodules through a coarse detection network and outputs a first prediction score; carrying out classification training on an image sequence corresponding to the candidate nodule positioned by the coarse detection network through a false positive inhibition network to obtain the undetermined nodule and a second prediction score thereof; and constructing a single feature of the undetermined nodule, constructing a global feature based on the single features of the undetermined nodules, splicing the single feature and the global feature into a fusion feature, performing regression training, and outputting a third prediction score to complete a nodule detection result. The problem of among the prior art to the not good effect of grinding glass nodule and special position is solved.

Description

Medical image-based nodule detection method and device and electronic equipment

Technical Field

The present invention relates to the field of computer technologies, and in particular, to a medical image-based nodule detection method and apparatus, a computer-readable storage medium, and an electronic device.

Background

Lung cancer is one of the most serious malignant tumors threatening human life and health today. Early diagnosis and treatment of lung cancer can increase the five-year survival rate of a patient from 14% to 49%, and therefore, early detection and diagnosis of lung cancer is the key for improving the survival probability of the patient. It is believed that nodules are one of the most important early signs of cancer, lesion characteristics can be inferred from lesion features of nodules, and CT examination is considered to be the best diagnostic tool for finding and diagnosing lung nodules. Due to the uncertainty of the characteristics such as the size, the shape and the density of the nodule, the traditional manual identification cannot meet the detection requirements in terms of workload and precision. The need for Computer Aided Detection (CAD) has become acute.

Early researchers primarily extracted features manually, and then classified the characterized vectors using various machine learning methods. However, the feature expression capability of the manual extraction is low, and additional preprocessing is required to reduce the influence of the extraneous region, so that the application to the actual environment is not possible. In recent years, with the development of artificial intelligence and deep learning algorithms, image processing in medical diagnosis is increasingly involved, and researchers are trying to extract features from original CT images by using CNNs, which are influenced by the breakthrough of deep neural networks represented by CNNs in image classification tasks, so that the characteristics of the images themselves can be more objectively characterized, and the methods are becoming the mainstream of nodule detection methods.

The conventional lung nodule detection algorithm based on the lung CT image has a good detection effect on isolated lung nodules at present, but the detection effect on nodules in special positions, frosted glass shadows and other types of nodules is not ideal. The isolated lung nodule has clear boundary and obvious position and is easy to detect; the nodes in the lung portal region are mixed with blood vessels, bronchi and lymph node tissues, so that false positive judgment is easy to occur. And for another category of lung nodules, such as Ground Glass Nodules (GGNs) which are mainly characterized by ground glass shadows, the detection effect is poor by using a conventional detection algorithm. From the histopathological point of view, the appearance of GGN indicates that the lesion is still in early stage, active stage or progressive stage, so that the timely and correct judgment of the morphology and properties of GGN is very important for guiding treatment. On the other hand, the lung nodules are various in types and shapes, and other tissues in the lung, such as thickened blood vessels, also present a sphere-like shape, similar to the presentation state of the lung nodules in the CT image, and also one of the difficulties that the detection algorithm should solve.

In some currently common CAD methods for deep learning, for example, an algorithm for extracting an object from an original CT image and then detecting the object by using a Region-based Convolutional neural network (RCNN) series, there are common problems that the object detection is easily affected by a large number of negative samples, the effect of suppressing the false positive result is not good when classifying ground glass nodules and other similar morphological tissues, and the effect of detecting a junction located at a special position is not good. How to solve the problems in the detection algorithm has great significance for fully utilizing medical resources and relieving the diagnosis pressure of doctors.

Disclosure of Invention

Aiming at solving the problem of poor effect of grinding glass nodules and special positions in the related art, the invention provides a nodule detection method based on a medical image, which mainly comprises the following three stages:

s1, in a coarse detection stage, positioning a candidate nodule through a coarse detection network by taking an image sequence based on a medical image as input, and outputting a first prediction score;

in the stage of S2 false positive inhibition, classification training is carried out on an image sequence corresponding to the candidate nodule positioned by the coarse detection network through the false positive inhibition network to obtain the undetermined nodule and a second prediction score thereof;

s3 global judgment stage, constructing single features of the undetermined nodules, constructing global features based on the single features of the undetermined nodules, splicing the single features and the global features into fusion features, performing regression training, and outputting third prediction scores to complete nodule detection; wherein the individual features of the undetermined nodule are constructed from their characteristic parameters in a false positive inhibition network in combination with artificial features.

In the coarse detection stage, a feature pyramid network is used for integrating multi-scale information. The characteristic graph output by the shallow layer network of the characteristic pyramid network contains more geometric information, and the target position is more accurate; the feature map of the deep network contains more semantic information, but the smaller size is coarse with the target location. The feature pyramid network is combined with semantic information and geometric information/detail information, and is suitable for actual conditions of different sizes of nodules in the medical image scene.

In addition, another sample is characterized by a larger absolute number of simple negative samples and more easily separable samples than hardly separable samples in the negative samples. By adopting the Focal Loss function (Focal local), the method aims at the situation that the positive and negative samples are extremely unbalanced in the target detection task, and the Loss of target detection is easily influenced by a large amount of simple negative samples. The function may make the model focus more on the hard-to-divide samples during training by reducing the weights of the hard-to-divide samples. The difficult-to-sample mining mechanism focuses more on difficult-to-distinguish samples, and detection accuracy of complex types of ground glass nodules and the like is improved.

And in the stage of false positive inhibition, the nodule candidates obtained by the coarse detection network are subjected to fine classification. The 3D convolutional neural network can further play a good false positive inhibition effect on the output result of the coarse detection stage, and the non-nodule area in the output result of the coarse detection stage is eliminated as far as possible. A typical 3D convolutional neural network includes an input layer, a convolutional layer, a pooling layer, a Dropout layer, a fully-connected layer, and an output layer, where the convolutional layer is a 3D convolutional kernel. After the candidate nodules are subjected to the 3D convolutional neural network, a second output score is given to represent the judgment result of the pulmonary nodules, and compared with the candidate nodules determined in the coarse detection stage, the non-nodule areas of the candidate nodules selected through judgment of the second output score are obviously reduced.

The design of the first two stage frames meets the requirements of high detection rate and low false positive of lung nodules to a greater extent, but detection deviation may occur when result judgment is carried out according to the characteristics of a single nodule. The problem that the false positive inhibition stage only depends on a single nodule to judge is further solved by combining the global information of all nodules in a case through the global information.

A structure of ContextNet may be used to implement comprehensive determination using global information, and a specific implementation of a typical ContextNet is as follows:

s301: extracting characteristic parameters of the undetermined nodule in a false positive inhibition stage, and combining artificial characteristics to form a single characteristic of the undetermined nodule; the characteristic parameters of the false positive inhibition stage can select the characteristics of the full connection layer in the 3D convolutional neural network adopted by the stage to form a basic characteristic vector of a single characteristic. The artificial features can be composed of one or more parameters of a first output score of a coarse detection network of the undetermined nodule, a second output score of a false positive inhibition network and nodule morphological features, wherein the morphological features can comprise a long diameter, a short diameter, a density and the like.

S302: constructing global features of a plurality of nodes to be determined; the global feature is constructed by the following method: and selecting the first n single features of all the undetermined nodules for splicing to form a global feature, and if the undetermined nodules of the case are less than n, filling the vacancy with 0 to ensure that the length of the vacancy is equal to the length of the global feature.

S303: fusing the single feature and the global feature; and for each nodule of the undetermined region, combining the single feature constructed in the S301 and the global feature constructed in the S302 to form a combined feature of the nodule.

S304: and classifying the fusion features to obtain a third prediction score, wherein one classification method is to perform regression training by adopting a Gradient Boosting Decision Tree (GBDT).

And after classification is carried out according to the third prediction score, the nodule detection through the three-stage framework is completed.

In another aspect of the present invention, extracting contour information from the detected nodule image using a segmentation network to compute morphological features, including one or more of a long diameter, a short diameter, an equivalent diameter, a volume, and a Hu value of the nodule, for use by a physician in an actual scene. The split network can use any model that works well, such as 3D U-Net.

In another aspect of the invention, a follow-up matching step is also included. Because the posture and the breathing of the patient can obviously influence the angle and the shape of image acquisition during each scanning, and medical images in different periods cannot be matched through pixel positions, the image registration is carried out through a 3D full convolution network. On the basis of image registration, by calculating Euclidean distances among lung nodules in different periods, a graph is built by connecting edges among the nodules meeting the threshold setting, the nodule pair with the minimum distance among the connected components is selected as a final matching result, the corresponding relation of the nodules in the images in different periods can be built, and a doctor is assisted in identifying and analyzing the change of the old nodules and the new nodules.

Typically, the method of the invention is used to process CT images of the lungs, with the detection target being a lung lesion.

The method for detecting a nodule based on a medical image according to the present invention may be applied to the purpose of auxiliary diagnosis of a disease, or may be applied to medical statistical research for the purpose of non-disease diagnosis. In addition, when used as an aid to diagnosis, the method is often not sufficient to diagnose the disease directly, and is only used as an intermediate result for reference by the physician.

The present disclosure also includes a nodule detection apparatus based on medical images, comprising: an acquisition unit that acquires a medical image to be processed; and the detection module is used for carrying out nodule detection on the medical image to be processed according to the medical image-based nodule detection method.

A computer-readable medium, in which a computer program is stored which, when being executed by a processor, carries out the aforementioned medical image-based nodule detection method.

An electronic device comprising a processor and a memory, wherein the memory stores thereon one or more computer readable instructions, and wherein the one or more computer readable instructions, when executed by the processor, implement the medical image-based nodule detection method.

The invention has the beneficial technical effects that:

the invention provides a three-stage pulmonary nodule detection method, which aims to solve the problem of poor effect on ground glass nodules and special positions in the related art by optimizing an algorithm frame, transplanting false positives and judging comprehensive global information. In addition, after the nodule is detected, quantitative morphological indexes of the nodule are analyzed by utilizing a segmentation network, and lesion pairing, analysis and comparison of multiple examinations of the same patient are performed by combining a lesion follow-up module. The computer-aided establishment of the corresponding relation of the nodules in the multi-detection image of the patient is realized, and doctors are assisted to make a better judgment.

The method has the advantages of high detection accuracy, low false positive rate, capability of quantifying nodules, capability of follow-up visit and tracking and the like.

Drawings

FIG. 1 is a schematic diagram of a three-stage detection method according to the present invention;

FIG. 2 is a schematic diagram of an actual scene of multi-scale information detection of a lung portal region nodule;

FIG. 3 is a schematic diagram of a coarse detection stage using a feature pyramid network;

FIG. 4 is a schematic diagram of a false positive suppression network;

FIG. 5 is a diagram of a typical architecture for ContextNet;

FIG. 6 is a graph of a second output and a third output score for a lung nodule candidate region;

FIG. 7 is a schematic view of ground glass nodule detection;

FIG. 8 is a schematic diagram of segmentation network extraction nodule contours;

FIG. 9 is a schematic diagram of a 3D U-Net network structure FIG. 10 is a schematic diagram of an image registration network;

FIG. 11 is a schematic representation of the establishment of a relationship to a nodule through a follow-up procedure.

Detailed Description

The present invention will be described in further detail with reference to the accompanying drawings and examples. It should be understood that the example embodiments may be embodied in many different forms and should not be construed as limited to the examples set forth herein.

The present application describes nodule detection apparatus/methods based on medical images, where the medical images may include projection image data obtained by various imaging systems. The imaging system may be a single mode imaging system, such as a Computed Tomography (CT) system, an ultrasound imaging system, an X-ray optical imaging system, a Positron Emission Tomography (PET) system, and the like. The imaging system may also be a multi-modality imaging system, such as a positron emission tomography-computed tomography (PET-CT), positron emission tomography-magnetic resonance imaging (PET-MRI) system, a single photon emission tomography-computed tomography (SPECT-CT) system, a digital subtraction angiography-computed tomography (DSA-CT) system, or the like. The medical image may comprise a reconstructed image obtained by reconstructing the projection data.

Computers may be used to implement particular methods and apparatus disclosed in some embodiments of the invention. In some embodiments, a computer may implement embodiments of the invention by its hardware devices, software programs, firmware, and combinations thereof. In some embodiments, the computer may be a general purpose computer, or a specific purpose computer. In some embodiments, the computer may be a mobile terminal, a personal computer, a server, a cloud computing platform, and the like.

The computer may include an internal communication bus, a processor, read-only memory, random access memory, communication ports, input/output components, a hard disk, and a user interface. An internal communication bus may enable data communication between computer components. The processor may make the determination and issue the prompt. In some embodiments, a processor may be comprised of one or more processors. The communication port may implement a computer and other components such as: and the external equipment, the image acquisition equipment, the database, the external storage, the image processing workstation and the like are in data communication. In some embodiments, a computer may send and receive information and data from a network through a communication port. The input/output components support the flow of input/output data between the computer and other components. By way of example, the input/output components may include one or more of the following: a mouse, a trackball, a keyboard, a touch-sensitive component, a sound receiver, etc. The user interface may enable interaction and information exchange between the computer and the user. The computer may also include various forms of program storage units and data storage units, such as a hard disk, Read Only Memory (ROM) and Random Access Memory (RAM), capable of storing various data files used in the processing and/or communication of the computer, and possibly program instructions executed by the processor.

It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

In another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or may exist separately without being assembled into the electronic device. The computer readable medium carries one or more programs which, when executed by an electronic device, cause the electronic device to implement the method as described in the embodiments below.

Before explaining the technical solution of the embodiment of the present invention in detail, a relevant principle about a Convolutional Neural Network (CNN) is first introduced.

CNN is a multi-layered supervised learning neural network, which is commonly used to handle image-related machine learning problems. A typical CNN consists of a convolutional layer (Convolution), a Pooling layer (firing), and a fully connected layer (FullyConnection). The lower layer generally consists of convolution layers and pooling layers, wherein the convolution layers are used for enhancing the original signal characteristics of the image and reducing noise through convolution operation, and the pooling layers are used for reducing the calculation amount while keeping the image rotation invariance according to the principle of image local correlation. The fully connected layer is located at the upper layer of the CNN, the input of the fully connected layer is a feature image obtained by feature extraction of the convolutional layer and the pooling layer, the output of the fully connected layer is a connectable classifier, and the input image is classified by adopting logistic regression, Softmax regression or Support Vector Machine (SVM).

The CNN training process generally adopts a gradient descent method to minimize a loss function, weight parameters of all layers in the network are reversely adjusted layer by layer through a loss layer connected behind a full connection layer, and the accuracy of the network is improved through frequent iterative training.

In an exemplary embodiment of the invention, the detection of lung nodules is performed in three stages, the principle of which is schematically illustrated in fig. 1:

s1 coarse detection stage

Taking lung nodule detection in a CT image as an example, as a detection target in a medical image, the size and the morphological difference of the lung nodule detection are large, it is difficult to cover all scales by a data set, and the robustness of the conventional CNN algorithm is challenged due to different acquisition equipment sources and parameters. Therefore, integrating multi-scale information is a problem to be solved in the coarse detection stage, and in the embodiment of the present invention, a Feature Pyramid Network (FPN) is used to solve the problem.

In a conventional image processing method, a pyramid is a relatively common means, such as Scale-invariant Feature Transform (SIFT), multi-layer Feature acquisition is performed based on the pyramid, and for a deep network, the native convolutional network features determine a natural pyramid structure. In the process of continuous convolution, the smaller the Size (Size) of the deep Feature Map (Feature Map), the more semantic information it contains, but due to the characteristic of the smaller Size, it is difficult to detect a small target and it is time-consuming. On the other hand, if the Feature Map is up-sampled and then restored step by step, the Size (Size) of the Feature Map (Feature Map) becomes large, and more detailed information can be focused, so that the problem that a small target is difficult to detect is solved.

Thus, a feature pyramid network is a network that consists of two parts, bottom-up and top-down. From bottom to top, the traditional convolution network is used for feature extraction, and with the deepening of convolution, the spatial resolution is reduced, the spatial information is lost, and more semantic information is detected; the FPN also provides a top-down path, a layer which simultaneously retains high precision is constructed, lateral connection is provided, original feature information is integrated into an up-sampling feature layer, the original feature represents accurate position information, and path results in two directions are fused together, so that feature graphs containing semantic information with different sizes or resolutions are obtained, and distribution of detection frames in the detector is adaptively adjusted according to characteristics of the nodule lesions.

In one embodiment, to eliminate extreme Scale target regions (too large or too small), training is only performed within the desired Scale range, i.e., only candidates within a certain size range are selected, and the remaining out-of-Scale candidates are ignored when propagating back, thereby generating detection targets when training and regressing, i.e., using a concept similar to Scale Normalization for Image Pyramides (SNIP) algorithm.

Fig. 2 illustrates the detection effect of the nodules in the lung portal region at different scales: the characteristic diagram of the shallow network contains more geometric information, and the target position is more accurate; the feature map of the deep network contains more semantic information, but the smaller size is coarse with the target location. Therefore, the characteristic pyramid network is adopted in the coarse detection stage, and the requirements of detection and classification can be better met at the same time.

On the other hand, the absolute number of simple negative samples for nodule detection is large, the number of easily-separable samples in the negative samples is more than that of difficultly-separable samples, and a difficultly-separable mining mechanism is introduced in the embodiment, so that the model focuses more on the difficultly-separable samples, and the detection accuracy of complex types of nodules such as ground glass nodules is improved.

In the model training process, a Loss Function (Loss Function) is used for evaluating the degree of inconsistency between a predicted value and a true value of the model, and the process of neural network training or optimization is the process of minimizing the Loss Function. Typical loss function in training classification tasks using CNN, the loss layer of a full connection layer post-connection often uses a cross-entropy loss function. Cross entropy loss function:

L₁(p_y)＝-logp_y

however, in order to perform difficult mining, an exponential coefficient is added into a typical cross entropy loss function, so that the contribution of positive and negative samples to the loss function is automatically adjusted, and the focusing on difficult samples is realized.

The introduction of the Focal Loss function (Focal local) is to solve the problem of serious imbalance of class sample proportion in target detection, and aims to solve the class imbalance problem by reducing internal weighting (simple samples), so that even if the number of simple samples is large, the contribution of the simple samples to the total Loss is small. The function can enable the model to be more concentrated in the samples difficult to be distinguished during training by reducing the weight of the samples easy to be distinguished, and the problem that the loss of target detection is easily controlled by a large number of simple negative samples is avoided.

For the two classes of focus Loss function (Focal Loss):

L₂(p_y)＝-α_y(1-p_y)^γlogp_y

wherein p is_yIs a model probability prediction value with the value between 0 and 1 and alpha_yThe balance of the number of positive and negative samples is controlled to be between 0 and 1. This is an improved loss function based on the cross-entropy loss function, the modulation term (1-P)_y)^γThe loss of the easily separable samples can be reduced, the loss of the difficultly separable samples can be increased, and the model can be focused on learning the difficultly separable samples. Alpha can be dynamically adjusted according to the performance of the standard data set on the neural network model_yValue of (d), detection rate of modelAnd the average false positive rate reaches an acceptable balance, then alpha under this result can be selected_y. Meanwhile, whether a higher gamma value needs to be set or not can be determined according to the detection rate and the trend of the false positive rate, the gamma value under the result can be kept unchanged under the condition that the detection rate is high and the average false positive rate is low, and if the detection rate and the average false positive rate are smaller, the gamma value is increased appropriately.

In one embodiment of the invention, α is selected_y＝1，γ＝2。

A typical coarse detection stage network is shown in figure 3. In the coarse detection stage, a CT image is used as input, a characteristic pyramid is used as a characteristic extraction network, candidates outside a specified range are omitted during back propagation, a focus loss function and a Smooth L1 model are used for regression training in training classification, and the positioning position of a candidate nodule and a first prediction score of the candidate nodule are output.

S2 false positive inhibition phase

The stage aims to further perform false positive inhibition on the output result of the coarse detection stage through the convolutional neural network, eliminate non-nodule areas in the output result of the coarse detection stage, and achieve fine classification.

The false positive inhibition network can adopt a typical CNN network model structure. In the traditional CNN method, a two-dimensional model is adopted, firstly, 2D visual angles of 3D structures are obtained around nodules in various slicing modes, then, the CNN model is used for carrying out feature extraction, and features are extracted by adopting the 2D CNN model, so that detection results are obtained. This method is based on a method of converting slices into 2D views, underutilizes scanned 3D environmental information, and has limitations in both detection accuracy and effect. And the features are directly extracted from a stereo space based on a 3D CNN model, and the steps and the thought of the method are basically consistent with those of a 2D CNN, and the difference is that a two-dimensional convolution kernel used by the former is replaced by a three-dimensional convolution kernel.

In this embodiment, a typical 3D CNN network is adopted, such as the false positive suppression network shown in fig. 4, which includes an input layer, a convolutional layer, a pooling layer, a Dropout layer, a full link layer, and a Softmax layer. In a typical embodiment, after a 3D image is input, a plurality of convolution kernels of 3 × 3 × 3 are used in a training process, and then an activation function such as a modified linear unit (ReLU) may be used to perform network activation. The convolutional layer, the pooling layer, the Droupout layer, the fully-connected layer and the Softmax layer are all classic network model units in a classic CNN network, and are not described in detail in this embodiment.

In one embodiment, the acquired multiple nodule images may be used directly as training samples; or using a public case database, such as a public database of lung CT images, to construct training samples. In another embodiment, a pre-trained 3D CNN is used as a parameter for the false positive cancellation network.

After the candidate nodules are subjected to the 3D convolutional neural network, a second output score is given to represent the judgment result of the pulmonary nodules, and compared with the candidate nodules determined in the coarse detection stage, the non-nodule areas of the candidate nodules selected through judgment of the second output score are obviously reduced.

S3 Global decision phase

The design of the frames in the first two stages meets the requirements of high detection rate and low false positive of pulmonary nodules to a greater extent, but in the diagnosis process of an actual doctor, more than one lesion part often exists in one case, the false positive inhibition stage eliminates the characteristics input by the network and judges the result as the characteristics of a single nodule, and the condition that the manifestation shape of the lesion is very similar to the nodule can cause deviation. If the overall characteristics of the case image are combined, the global information is integrated, the detected nodules are prevented from being looked at in an isolated mode, and the false positive nodules are further restrained.

Therefore, in the present embodiment, after the first two stages are ended, the global decision stage is added, and the main idea is to combine global information of all nodes in a case, further make up the bias of the S2 false positive suppression stage that only depends on a single node for decision, and help a doctor to eliminate false positive nodes that are difficult to decide.

In order to solve the above problem, the structure of ContextNet may be adopted to realize comprehensive determination using global information. A typical embodiment of ContextNet is shown in fig. 5. It basically comprises the following steps:

s301: extracting characteristic parameters of the undetermined nodule in a false positive inhibition stage, and combining artificial characteristics to form a single characteristic of the undetermined nodule;

the characteristic parameters of the false positive inhibition stage can select the characteristics of the 3D CNN full-connection layer to form a basic characteristic vector of a single characteristic.

The artificial features can be composed of one or more parameters of a first output score of a coarse detection network of the undetermined nodule, a second output score of a false positive inhibition network and nodule morphological features, wherein the morphological features can comprise a long diameter, a short diameter, a density and the like. This is to consider that introducing personal experience of the doctor in the problem of judging the nodule, such as parameters of the long diameter, the short diameter, the average density and the like of the nodule as the constituent elements of the artificial features, and introducing the first prediction score in the coarse detection stage and the second prediction score in the false positive suppression stage is also beneficial to constructing a single feature with rich hierarchy.

S302: constructing a global feature based on the single features of the plurality of undetermined nodules;

wherein the global features are constructed in the following way: and selecting the first n single features of all the undetermined nodules for splicing to form a global feature, and if the undetermined nodules of the case are less than n, filling 0 in the vacancy to ensure that the lengths of the global features are equal.

S303: fusing the single feature and the global feature;

and for each nodule of the undetermined region, the single feature in the S302 and the global feature constructed in the S302 are fused to form a fusion feature of the nodule.

S304: the fusion features are classified, and in one embodiment, a Gradient Boosting Decision Tree (GBDT) is used for regression training to obtain a third prediction score.

After the ContextNet network is used, the third prediction score of the undetermined candidate area is corrected according to the second prediction score, and the false positive candidate area is further restrained. In practical application, one of a plurality of undetermined nodules in one case may have a higher second prediction score, but after global judgment, the third prediction score is lower, and a doctor judges that the blood vessel thickening shadow is similar to a spherical shape and classifies the blood vessel thickening shadow into a false positive focus. The global information judgment stage has a good inhibition effect on false positive tissues similar to the nodule morphology.

An exemplary procedure for identifying nodules in CT images of the lungs using the three-phase method described above is described in the following example.

In the rough detection stage, the preprocessed image sequence is used as input, the feature pyramid is used as a feature extraction network, the propofol outside the specified range is omitted during back propagation, a focus loss function is used for classification, Smooth L1 is used for regression training, and a first prediction score is output. The network establishes a mapping relation between a feature map and an original CT image, wherein the position of a candidate nodule region in a corresponding CT image is marked as (x, y, z), each point on the feature map corresponds to the output probability of predicting whether the point is a nodule or not, and a 4-dimensional offset vector representing the point relative to the center of the marked nodule.

The three-dimensional image blocks are classified by a false positive suppression network, and the three-dimensional image blocks, for example, image blocks of 64 × 64 × 64 or 36 × 36 × 36 size, are clipped in the original CT image with the center of the nodule candidate region output in the coarse detection stage as a base point. In one embodiment, the 3D CNN network may be composed of six convolutional layers, three pooling layers, three fully-connected layers, and a final Softmax output layer; wherein, a ReLU activation function can be connected after each convolution layer, and a Dropout layer is input after each pooling layer and output layer to avoid overfitting. And finally, taking the second output score as a classification prediction result of the second-stage nodules.

And a global judgment stage: and taking the characteristics of the last full-connection layer of the false positive inhibition network of each undetermined nodule as a basic vector, outputting scores in a coarse detection stage, outputting scores in a false positive inhibition stage, and taking the long diameter, the short diameter, the average density and the like of the nodule as artificial characteristics to jointly construct a single characteristic of the undetermined nodule. And taking the first n second prediction scores of each case output in the false positive suppression network, wherein n is 20-30, preferably 24, splicing the single features of the corresponding undetermined nodules, and if the number of the undetermined nodules classified in the false positive suppression stage of the case is less than 24, filling the gaps with 0 to form the global feature. And splicing the single feature and the global feature, and performing regression training on the fusion feature of each node to be predicted by using GBDT (Gradient Boosting Decision Tree) to obtain a third prediction score of the node to be predicted. And giving a final nodule detection result through the third prediction score.

Fig. 6 illustrates a second prediction score for a nodule over a false positive suppression phase and a third prediction score corrected over a global decision phase for one case, where the outcome of the third prediction score is more focused by the physician and has a higher accuracy than its second prediction score.

In order to evaluate the effect of the above method. The test was selected on the public data set LUNA16 and the results were compared to both public methods.

The method comprises the following steps: DingJ, Li, A., Hu, Z., Wang, L: Accurate pure non-detection In computed tomography images using reliable connected networks In MICCAI. (2017)559-567

The method 2 comprises the following steps: dou, Q., Chen, H., Jin, e, Automated public non-product detection via 3d connected with online sample filtering and hybrid-less residual filtering in MICCAI (2017)630-638

The evaluation is performed using the FROC (Free-Response driver Operating characterization) criterion commonly used in machine learning algorithms. Specifically, the FROC criterion characterizes the relationship between recall (number of detected nodules in all CT data in the test set/number of nodules in all CT data in the test set) and the average number of false positives per CT scan (number of all non-actual nodules predicted as nodules in the test set/number of scans in the test set, FPs/scan).

Table 1 discloses data set test results

	0.125	0.250	0.500	1.000	2.000	4.000	8.000	Mean
									Method
1	0.659	0.745	0.819	0.865	0.906	0.933	0.946	0.839
									Method 2	0.748	0.853	0.887	0.922	0.938	0.944	0.946	0.891
The invention	0.793	0.853	0.902	0.944	0.958	0.972	0.975	0.9138

It can be seen that the nodule detection method of the present invention achieves good results in terms of nodule recall when the average number of false positives per CT scan is one or more.

The feature pyramid, the difficult case mining mechanism and the global information judgment stage introduced in the method optimize the algorithm aiming at the complex type of the nodules, so that the accuracy of the difficult case detection such as the ground glass nodules is improved.

An example of an actual scene is illustrated in fig. 7, and it can be seen from the marked boxes in the figure that the method can realize accurate identification of the ground glass nodules. The ground glass nodule is less prominent than the surrounding background region and has a blurred boundary, which makes detection difficult in the prior art.

In another embodiment, the morphological characteristics of the nodule detected in the present invention are calculated, and the morphological characteristic result is given, that is, the long diameter, the short diameter, the equivalent diameter, the volume, the Hu value, etc. of the nodule are calculated according to the contour of the identified nodule, and the effect diagram is shown in fig. 8.

The nodule contour identification can be carried out by adopting a better segmentation network, such as U-Net, FCN, deep Lab V1 and the like, and the invention adopts a 3D U-Net segmentation network. 3D U-Net principle As shown in FIG. 9, the left side can be considered an encoder and the right side can be considered a decoder. The encoder progressively pools the spatial dimensions of the layers and the decoder progressively restores the details and spatial dimensions of the object. A fast connection exists between the encoder and the decoder, thus helping the decoder to better repair the details of the target.

And outputting mask information of the region by the segmented network, wherein the pixel value of the mask belonging to the nodule region is 1, and the pixel value of the mask belonging to the background region is 0, so as to obtain the fine contour of the nodule candidate region. On this basis, the longest diameter, equivalent diameter, average HU value, maximum HU value, minimum HU value, volume value, and the like are performed, and the above-described indices are provided to the doctor to assist the doctor in determining the nodule.

In another aspect of the invention, the step of follow-up registration is further designed to match lung nodule correspondence in CT images of the lungs of the patient at different times. Due to the patient's posture, breathing, and the like during each scan, the angle and shape of the lungs are significantly affected and cannot be matched by pixel location.

Calibration and correspondence of the images can be performed using an image registration network to learn shape and position differences between two similar images. As shown in fig. 10, a CNN network is used to learn a parameterized registration deformation field (parametric function), and then registration between a changing image (Moving image) and a Fixed image (Fixed image) is achieved by spatial transformation. And for a given image pair to be registered, directly utilizing the learned parameter function to perform registration. The input end of the registration network is a group of image pairs consisting of a change image and a fixed image, and the output end is a registration area based on a parameter theta (kernel function of a convolution layer in a CNN network)

The registration region is a field of displacement vectors that specify for each voxel (the smallest unit that can be segmented in the three-dimensional image) the displacement offset from the Fixed image to the Moving image. The changing image (Moving image) is then transformed into a changed image registration result (Moved image) by a spatial transformation. The loss function in the network training process comprises two parts: loss of similarity and loss of smoothness, wherein the loss of similarity is a measure of facies between a Moved image and a Fixed imageThe smoothness loss is to ensure the smoothness of the registration area.

The loss function can be described as:

wherein I (F, M, phi) is I_sim(F,M(φ))+λI_smooth(φ), where F is fixed image, M is moving image, M (φ) is moving image, I_sim(.) loss of similarity, I_smooth(.) or smoothing loss, λ is the regularization coefficient.

Therefore, in one embodiment, the image registration is performed by a 3D full convolution network, specifically, the embodiment employs a volume-Morph network, the network structure of which is a full convolution structure similar to 3DUnet, the input is two images with the size H × W × D, which are respectively a change image and a fixed image, and the network output is a transformation matrix of H × W × D3. And (3) the 3-tuple (dh, dw, dd) at the (i, j, k) th position of the transformation matrix corresponds to the pixel at the (i, j, k) th position of the changed image one by one, and the registration changed image with the shape close to that of the fixed image can be generated by translating the pixel at each position of the changed image in the direction of the corresponding triplet.

On the basis of registration of two scanning images, by calculating the Euclidean distance between nodes obtained by a detection algorithm in two different scanning periods, a graph is built on the connected edges between the nodes meeting threshold setting (for example, 1cm), the node pair with the minimum distance in the connected components is selected as a final pairing result, and the corresponding relation of the nodes scanned at different times can be obtained.

Fig. 11 shows the correspondence of the same nodule established by CT images of the same patient taken at different times under the follow-up method based on the method of the present invention.

Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. The block diagrams shown in the figures are functional entities only and do not necessarily correspond to physically separate entities. I.e. these functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor means and/or microcontroller means. Therefore, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiment of the present invention.

It will be understood that the invention is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The flowcharts shown in the drawings are only exemplary and do not necessarily include all the contents and operations/steps, nor are they necessarily performed in the described order, and the order of actual execution may possibly vary depending on the actual situation. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the invention and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains.

The scope of the invention is limited only by the appended claims. The above embodiments are merely illustrative, and not restrictive, of the present disclosure, and various changes and modifications may be made by those skilled in the relevant art without departing from the spirit and scope of the present disclosure, and therefore all equivalent technical solutions also fall within the scope of the present disclosure.

Claims

1. A nodule detection method based on medical images is characterized by comprising the following steps:

taking an image sequence based on a medical image as an input, positioning a candidate nodule through a coarse detection network and outputting a first prediction score;

carrying out classification training on an image sequence corresponding to the candidate nodule positioned by the coarse detection network through a false positive inhibition network to obtain the undetermined nodule and a second prediction score thereof;

constructing a single feature of a to-be-determined nodule, constructing a global feature based on the single features of a plurality of to-be-determined nodules, splicing the single feature and the global feature into a fusion feature, performing regression training, and outputting a third prediction score;

the global feature is constructed by the following method: selecting the first n single features of all the nodules to be determined for splicing to form a global feature, and if the number of the nodules to be determined in a case is less than n, filling the vacancy with 0 to ensure that the length of the vacancy is equal to the length of the global feature;

constructing a single characteristic of the undetermined nodule by combining artificial characteristics with characteristic parameters of the single characteristic in a false positive inhibition network;

the artificial features of the pending nodule include one or more parameters of the first prediction score of a pending nodule over a coarse detection network, the second prediction score over a false positive suppression network, a nodule morphology feature.

2. The nodule detection method based on medical images as claimed in claim 1, wherein the rough detection network adopts a feature pyramid network for feature extraction, and a focus loss function is adopted as a loss function in classification training.

3. The medical image-based nodule detection method of claim 1 wherein the false positive suppression network employs a 3D convolutional neural network.

4. A medical image based nodule detection method according to any one of claims 1 to 3 wherein the construction of global features is performed by: selecting the first n single features of all the nodules to be determined for splicing, and filling the vacancy with 0 if the nodules to be determined of the case are less than n; and performing regression training on the fusion features by adopting a gradient lifting decision tree.

5. The medical image-based nodule detection method of claim 4, wherein n is 20-30.

6. A medical image-based nodule detection method according to any one of claims 1 to 3 further comprising extracting contour information for the detected nodule using a segmentation network to compute morphological features including one or more of a long diameter, a short diameter, an equivalent diameter, a volume, a Hu value of the nodule.

7. The medical image-based nodule detection method of claim 6 wherein the segmentation network is a 3D U-Net segmentation network.

8. The nodule detection method based on medical images as claimed in any one of claims 1 to 3, further comprising a follow-up registration step, wherein the follow-up matching step is implemented by performing image registration on medical images of the same case in different periods, calculating Euclidean distances between lung nodules in different periods, establishing a graph of connected edges between nodules meeting threshold setting, selecting a nodule pair with the minimum distance among connected components as a final pairing result, establishing a corresponding relation between nodules in images in different periods, and assisting a doctor in identifying and analyzing changes of old nodules and new nodules.

9. The medical image-based nodule detection method of any one of claims 1 to 3 wherein the medical image is from a lung CT image.

10. The medical image-based nodule detection method of claim 4 wherein the medical image is from a pulmonary CT image.

11. The medical image-based nodule detection method of claim 6, wherein the medical image is from a lung CT image.

12. The medical image-based nodule detection method of claim 8, wherein the medical image is from a lung CT image.

13. A medical image-based nodule detection apparatus includes an image acquisition unit that acquires a medical image to be processed; a detection unit for performing nodule detection on the medical image to be processed according to the medical image-based nodule detection method of any one of claims 1 to 12.

14. A computer-readable medium, in which a computer program is stored which, when being executed by a processor, carries out the medical image-based nodule detection method according to any one of claims 1 to 12.

15. An electronic device comprising a processor and a memory, the memory having stored thereon one or more readable instructions which, when executed by the processor, implement the medical image-based nodule detection method of any one of claims 1 to 12.