TWI802510B - Interpretation assistance system and method for pulmonary nodule - Google Patents

Abstract

The present invention provides an interpretation assistance system for pulmonary nodules, which comprises: an inference server, an image server and a reporting auxiliary information subsystem. The inference server receives a set of lung computed tomography (CT) images and respectively inputs them into a plurality of inference models to obtain a plurality of nodule data, deletes the nodule data corresponding to the overlapped bounding box information, and outputs an inferred nodule data and the set of Lung CT images. The image server marks the nodule detection position on the set of lung CT images according to the inferred nodule data. The reporting auxiliary information subsystem displays the marked lung CT images and the inferred nodule data, receives a confirmed nodule data edited by a user for the inferred nodule data, and finally copies the confirmed nodule data to a report system.

Description

Pulmonary nodule auxiliary interpretation system and method

The present invention relates to a pulmonary nodule auxiliary interpretation system and method thereof, in particular to a pulmonary nodule auxiliary interpretation system and method of low-dose computed tomography images.

According to the latest data from the Ministry of Health and Welfare, lung cancer continues to be the top cause of cancer death in the country. Related large-scale clinical trials in recent years have shown that low-dose computed tomography screening for high-risk groups can detect pulmonary nodules smaller than 1 cm early, thereby reducing mortality.

However, there are dozens of computed tomography images of the lungs in a single case, and radiologists need to be highly focused to examine them one by one, and the average case interpretation time is 10-15 minutes. In particular, small blood vessels and small nodules appear very similar in images. In addition, ground glass opacity (GGO) has irregular cloud-like thin shadows on the edge or inside, which is not only difficult to identify, but may even require adjustment of the window frame conditions to find it.

For radiologists, the interpretation of low-dose lung computed tomography is not only time-consuming but also harmful to the eyes. In addition to the heavy workload of daily clinical reports, there is an urgent need for high-accuracy detection models, as well as simplification of workflow and shortening of printing time. An auxiliary tool for reporting time to improve diagnostic efficiency.

The present invention provides an auxiliary interpretation system for pulmonary nodules, comprising: an inference server, which receives a group of computerized tomographic images of the lungs and inputs them to a plurality of inference models to obtain a plurality of nodule data, each of which includes a Prediction frame information and a corresponding prediction probability information, delete the nodule data corresponding to the overlapping prediction frame information to output a deduced nodule data and the group of lung computed tomography images; an image server, which is based on the inference The nodule data marks the nodule detection position in the group of lung computed tomography images; and a reporting auxiliary information subsystem is operated in a computer host and connected to the image server, including the following modules: a display module, displaying The set of marked lung computed tomography images and the deduced nodule data; an input module receiving confirmed nodule data edited by a user on the deduced nodule data; and an output module receiving the confirmed nodule data Copy to a reporting system.

The present invention provides an auxiliary interpretation method for pulmonary nodules, which includes: receiving a set of computed tomographic images of the lungs and inputting them into multiple inference models to obtain multiple nodule data, each of which contains a prediction frame information and a corresponding A prediction probability information; delete the nodule data corresponding to the repeated prediction frame information to output a deduced nodule data; mark the nodule detection position in the group of lung computer tomography images according to the deduced nodule data; and display the marked nodule data A set of computed tomography images of the lungs and the deduced nodule data, receiving confirmed nodule data edited by a user on the deduced nodule data, and copying the confirmed nodule data to a reporting system.

In some embodiments, the inference models include: a first inference model is a region proposal network model with preset anchor boxes, trained using computed tomography images of labeled solid pulmonary nodules; and a second inference model The model is a region proposal network model without preset anchor boxes and trained using computed tomography images of labeled ground-glass pulmonary nodules and semi-solid pulmonary nodules.

In some embodiments, the inference models adopt a UNet backbone structure, and include a residual module and a compressed excitation module in each upsampling and downsampling structure.

In some embodiments, the residual module includes two normalization layers.

In some embodiments, the group of lung computed tomography images may further perform a window frame setting, a resampling process and a cropping process before being input into the inference models.

In some embodiments, the inference models further include a lobe cut model, and each of the nodule data further includes lobe information.

In some embodiments, the residual module further includes an activation layer and a convolutional layer.

In some embodiments, the compressed excitation module includes a global average pooling layer and two linear layer sub-modules.

In some embodiments, each of the upsampling structures includes a variable dimension and upsampling layer.

In some embodiments, each of the downsampling structures includes a variable dimension layer and at least one fixed dimension layer.

The pulmonary nodule auxiliary interpretation system and method provided by the present invention combine multiple inference models to detect nodules in pulmonary computerized tomography images, improve the diagnostic sensitivity of pulmonary nodules, and list the detection results to radiologists for reference , so that radiologists can shorten the diagnosis time, and can further confirm the detected nodule images, and check and export the nodule information to be recorded in the report, which greatly reduces the diagnosis time and report input time, and improves the diagnostic capacity and accuracy.

Other technical contents, features and effects of the present invention will be clearly presented in the following detailed description of preferred embodiments with reference to the drawings.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

As used herein, the articles "a", "an" and "any" refer to one or more than one (ie, at least one) of the grammatical items of the item. For example, "an element" means one element or more than one element.

As used herein, the term "about", "approximately" or "approximately" means that the stated value or range of values is within 20%, preferably within 10%, and more preferably within 5% float within. Numerical quantities provided herein are approximations and are intended to be inferred if the terms "about", "approximately" or "approximately" are not used.

As used herein, "connect", "connect" or "connect" means to use any means of wires, circuit boards, cables, network cables, adapters, converters, Bluetooth or wireless networks to connect to power or network Combination of communications.

As used herein, "computed tomography of the lung" means an image obtained by scanning the chest cavity using a computerized tomography scanner (Computed Tomography, CT), usually from above the lung apex to the diaphragm, not limited to using Philips or Toshiba CT scanner. "Lung computed tomography" includes low-dose lung computed tomography (Low-Dose CT, LDCT), which uses X-ray helical scanning with a dose of about 0.8~1.2mSv, and the original image matrix size includes but is not limited to 512 x 512 pixels , the thickness of the original image includes but is not limited to 3mm, 2mm, and 1mm.

As shown in FIG. 1 , an embodiment of the present invention provides an auxiliary pulmonary nodule interpretation system 1 , including: an inference server 10 , an image server 20 and a reporting auxiliary information subsystem 30 . The inference server 10 and the image server 20 can be web servers or application program servers, and run in the same server host or separately in separate server hosts.

The report auxiliary information subsystem 30 is connected to the inference server 10, the image server 20, and the Radiological Information System 4 (Radiological Information System, RIS) to serve as the pulmonary nodule auxiliary interpretation system 1 with the Radiological Information System 4 and the original computer tomography system. The image storage device 2 is an application programming interface for accessing data, and at the same time uses a hospital intranet connection to ensure data security.

The radiology department information system 4 receives the patient information from the medical information system 5 (Healthcare Information System, HIS) to create a CT checklist, and the computerized tomography scanner 3 stores the scanned 3D lung computer tomography original image according to the checklist information to the storage device 2, wherein the patient information includes at least the medical record number and name, and the examination information includes at least the examination order number and examination items. When the radiologist wants to print a lung computer tomography diagnosis report, he can select the CT examination order number from the report auxiliary information subsystem 30 to input the corresponding original lung computer tomography image in the storage device 2 to the inference server 10 .

Please refer to FIG. 2 at the same time. The inference server 10 receives a group of lung computed tomography images from the storage device 2 (step S10) and uses deep learning technology to automatically identify lung nodules (about 20 seconds) to provide to the user (for example Radiologist) as a reference for image interpretation. At least the following image preprocessing steps (step S20) can be performed before inputting a plurality of inference models: a window frame setting (step S21), for example, a lung window frame (lung window), and its preset window level (window level, WL ) is -600, and the window width (window width, WW) is 1500. A resampling process (step S22 ), resampling the three dimensions of the three-dimensional lung computed tomography image to 1mm, so that the original images with different pixel spacings can be input into each inference model. A cropping process (step S23 ), the three-dimensional lung computed tomography image is divided into, for example, approximately 128 mm cubic image blocks for input into the inference model.

五個數值，其中預測框資訊

代表一球形預測框之球心( X, Y, Z)及直徑d， p為相對應之球形預測框內含肺部結節的預測機率資訊。理想上，單一球形預測框應盡其所能的將單一結節緊密包含，

與 d的單位為mm。若進一步輸入肺葉分割模型，該結節資料進一步包含一肺葉資訊

，肺葉資訊代表預測框資訊所在位置的肺葉，例如右上葉、右中葉、右下葉、左上葉、左下葉及非肺葉，並刪除

為非肺葉所對應的結節資料。 The inference server 10 includes a first inference model 11 and a second inference model 12 , and may further include a lung lobe segmentation model 13 . The inference server 10 inputs the above computed tomography images of the lungs to the first inference model 11 and the second inference model 12 for inference (step S30 ), and a plurality of nodule data can be obtained. Each nodule data contains at least

Five values, among which the prediction frame information

Represents the center ( X, Y, Z ) and diameter d of a spherical prediction frame, and p is the prediction probability information of pulmonary nodules contained in the corresponding spherical prediction frame. Ideally, a single spherical prediction box should try its best to tightly contain a single nodule,

The unit of d and d is mm. If the lung lobe segmentation model is further input, the nodule data further includes a lung lobe information

, the lung lobe information represents the lung lobe where the prediction box information is located, such as the upper right lobe, the middle right lobe, the lower right lobe, the upper left lobe, the lower left lobe and the non-lobe, and delete

It is the nodule data corresponding to the non-lobe.

的單位刻度轉換為像素。 Next, use non-maximum suppression (Non-Maximum Suppression, NMS) to delete duplicate nodule data (step S40). Non-maximum suppression is used to calculate the three-dimensional overlap (intersection over union, IoU) between multiple prediction frames. The principle is to calculate the overlap between each prediction frame and other prediction frames. It can be set that the overlap degree is greater than 0.5, which means that the prediction frame overlaps. If a prediction frame overlaps with other prediction frames, the nodule data corresponding to the overlapped prediction frame information will be deleted, and only the nodule data with the highest prediction probability will be kept. In this way, a set of inferred nodule data and a set of lung computed tomography images that contain a high probability of prediction and are not repeated can be output. Before sending the inference nodule data to the image server 20, the inference server 10 may further perform a unit conversion step to convert

The unit scale is converted to pixels.

及其相對應之預測機率資訊。其中，

為該輸入影像區塊內所推論出的結節個數，

為第

個結節的預測中心位置，

為第

個結節的預測直徑，

為第

個結節所在的肺葉資訊。 The image server 20 receives the set of computed tomography images of the lungs and the deduced nodule data, and marks nodule detection positions in the set of computed tomography images of the lungs according to the deduced nodule data. For each input image block, the resulting inferred nodule data contains at least the set

And its corresponding predicted probability information. in,

is the number of nodules deduced in the input image block,

for the first

The predicted center position of a nodule,

for the first

predicted diameter of nodules,

for the first

Information about the lung lobe where a nodule is located.

The reporting auxiliary information subsystem 30 operates in a computer host and is connected to the image server 20 . The reporting auxiliary information subsystem 30 includes the following modules: an input module 31 , a display module 32 , and an output module 33 .

)且像素邊長為

的一個二維非實心方框A將可在第

張軸切影像上框選出第

個結節之結節偵測位置。 As shown in FIG. 3A , the display module 32 receives the deduced nodule data and displays the group of marked CT images of the lungs on the display interface operated by the user. The center of the pixel (

) and the pixel side length is

A two-dimensional non-solid box A will be available at the

Select the first frame on the axis-cut image

The nodule detection position of nodules.

Referring to FIG. 3B at the same time, the display module 32 also lists the inferred nodule data, and the inferred nodule data can be sorted according to nodule prediction probability, nodule diameter (size), and number of nodule sheets. The user (radiologist) can quickly find and confirm the possible location of the pulmonary nodule with the assistance of the reporting auxiliary information subsystem 30 . When one of the nodule data B is selected in the inferred nodule data list, the image window can jump to the nodule detection position A on the corresponding computed tomography image, and the doctor can quickly confirm and reduce the probability of misdiagnosis.

Then, the doctor can check which one is confirmed as pulmonary nodule in the input box next to the inferred nodule data, and can even further add other confirmed nodule locations. The input module 31 may receive one of the confirmed nodule data obtained by editing (for example, checking, adding or deleting) the inferred nodule data by the above-mentioned user. The output module 33 uses the web API to copy and/or return the confirmed nodule data to a reporting system in the radiology department information system, and confirms that the nodule data at least includes nodule position, nodule diameter, and lung lobe where the nodule is located , so that the doctor does not need to manually input relevant information in the report, saving a lot of time for typing the report.

。 Finally, the training method and module structure of the first inference model and the second inference model are introduced. The first training data set and the second training data set were provided by Cathay Pacific Hospital, using the computed tomography images of the lungs of more than 3,000 patients with pulmonary nodules, and the radiologists circled the nodule locations on the axial section of the lung tomographic images Annotation, circled nodule positions in multiple slices can form a three-dimensional pulmonary nodule annotation image, in which pulmonary nodules include solid pulmonary nodules, semi-solid pulmonary nodules (Pulmonary subsolid nodules, SSNs) and ground glass pulmonary nodules. The computed tomography image of the lung is also subjected to image pre-processing in step S20, and cut into three-dimensional image blocks of about 128 mm cubic

.

The first training data set includes a plurality of 3D image blocks, including solid pulmonary nodules, ground glass pulmonary nodules, semi-solid pulmonary nodules or 3D image blocks without any annotation (representing normal lung tissue). The second training data set includes a plurality of 3D image blocks, including 3D image blocks marked with ground glass pulmonary nodules, semi-solid pulmonary nodules (with ground glass components) or unlabeled (normal lung tissue). The second training data set is a subset of the first training data set. Since the nodule training data contains ground glass components, it is difficult to define the diameter or center position of the nodule accurately. Therefore, the second training data set should be more difficult to learn data set. Accommodating various types of nodule training data helps to improve the sensitivity of the inference model and avoid missed nodule grasping.

As shown in FIG. 4A and FIG. 4B , both the first inference model in FIG. 4A and the second inference model in FIG. 4B adopt a 3D-UNet backbone structure, which is composed of several down-sampling and several up-sampling. Downsampling can condense image features and increase the receptive field. Each downsampling includes a variable dimension layer B and at least one fixed dimension layer A. Each upsampling includes a variable dimension and upsampling layer C. Each up-sampling and down-sampling structure includes a residual module and a compressed excitation module to enhance the extraction of image features.

The gray-scale 3D image blocks (N, 1, D, H, W) of N images with length H, width W, and height D will be down-sampled four times after inputting the first inference model or the second inference model respectively. In the downsampling stage, the features will be continuously condensed to obtain more abstract and streamlined image features. Therefore, each downsampling will successively generate 32, 64, 96, and 128 3D feature maps, and the data size will be halved accordingly. Then two upsamplings are performed, and a trilinear upsampling operation (trilinear upsampling operation) is performed for each upsampling. In the upsampling stage, the condensed features will be gradually enlarged, and finally the nodule data predicted by the model will be output at the final target scale. Between the upsampling and downsampling stages, there are two wires. These two connections will concatenate (concatenate, represented by [] in the figure) the upsampling input of the same scale and the downsampling output of the same scale. This method can avoid the loss of information that may be helpful for prediction and increase model learning. the ease of.

)，目的是學習將採樣到的影像特徵經過分類和回歸以轉換成預測到的結節資訊。 Next, input the data upsampled three times into the back-end module of the 3D Region Proposal Network (3D RPN) (hereafter referred to as Head), and the Head only contains n variable-dimensional layers (

), the purpose is to learn to convert the sampled image features into predicted nodule information through classification and regression.

As shown in FIG. 4A, the first inference model is a 3D object region proposal network model (3D RPN) with a preset anchor frame, and is trained using the first training data set. The default anchor frame of the first inference model is artificially sprinkled in advance on the image space, several spherical frames of different sizes and equally spaced distribution. The model will try to detect if there are covered objects within the scattered preset spherical boxes.

。將第二次上採樣後的特徵圖(N, 128, D/4, H/4, W/4)輸入第一推論模型的Head之後得到形狀為

，意即N張影像區塊中，一共設有錨框數量

個，每個(第i個)錨框對應之五個預測目標包含：每個錨框中心位置

與直徑

的校正資訊

以及該錨框包含節結的機率有多少

。這些校正資訊包含可以用來把第

個預設錨框中心

微調至預測結節中心之偏移校正資訊，還包含可以用來將錨框直徑

縮放至預測結節直徑之縮放校正資訊。需要注意的是，由於是在長寬高皆縮小四倍的影像尺度上去作出預測，因此在輸出預測的結節資料之前，我們的座標預測需要在各座標軸方向乘以四倍，以還原至原來的影像尺度。 5 different sizes of preset anchor boxes (Na=5) are used in the 3D RPN of the first inference model. Anchor frame diameters are 5, 10, 15, 20, 25

. After inputting the feature map (N, 128, D/4, H/4, W/4) after the second upsampling into the head of the first inference model, the shape is

, which means that in N image blocks, there are a total of anchor frames

, each (i-th) anchor box corresponds to five prediction targets including: the center position of each anchor box

with diameter

Calibration information for

and what is the probability that the anchor box contains a nodule

. These calibration information contains information that can be used to convert the

default anchor box centers

Deskew information fine-tuned to the center of the predicted nodule, also contains the anchor box diameter that can be used to

Scale correction information for scaling to predicted nodule diameter. It should be noted that since the prediction is made on an image scale that is reduced by four times in length, width and height, before outputting the predicted nodule data, our coordinate prediction needs to be multiplied by four times in the direction of each coordinate axis to restore to the original Image scale.

一共預測出num_pred_nodules個結節。 The trained first inference model may further include a first inference post-processing. Input the 3D image block for inference by the first inference model. After obtaining the correction information of the center position and diameter of the anchor frame, the first inference post-processing calculates the information of the post-correction anchor frame, and performs non-maximum suppression to calculate multiple corrections. The three-dimensional overlap (intersection over union, IoU) between the anchor boxes, when the overlap is greater than 0.5, it can be regarded as the repetition of the corrected anchor box information. Delete the repeated corrected anchor frame information in order according to the size of the predicted probability information, so as to output a set of nodule data with a higher probability and no repetition in N images

A total of num_pred_nodules nodules are predicted.

In the first training mode, different image blocks will be input to the model multiple times to obtain offset correction information, scale correction information, and the probability of anchor boxes containing nodules for learning whether each anchor box is matched to an object, and How should the anchor box paired to the object be: (1) correct to a single nodule position; (2) correct its size so that it is just big enough to contain a whole nodule. Each input image block has a corresponding physician mark as the ground truth. The ground truth information includes the number of num_nodule nodules in N images and the corresponding nodule centers (X, Y, Z) and Nodule diameter d.

The preset anchor boxes and the reference fact information are subjected to box matching processing (box matching) to determine whether each preset anchor box matches a nodule. If there is a match, such matching anchor boxes will pass through the RPN target generator (RPN target generator) to obtain the actual anchor box offset and scaling correction. This real correction amount combined with the correction information produced by the model in the inference stage will evaluate the correction regression error (regression loss) predicted by the model. In addition, the model will predict the probability of containing nodules for each anchor frame, combined with whether the preset anchor frame does contain nodules, and the probability of the existence of nodules predicted by the model in the anchor frame, the anchor frame classification error of the model can be evaluated ( classification loss). The parameter optimization process of the model can finally reduce the regression error of the anchor box correction amount and the anchor box classification error respectively and gradually converge to obtain a trained and mature inference model.

As shown in FIG. 4B , the second inference model is a region proposal network model (Object as points) without default anchor boxes, and is trained using the second training data set. The RPN of the second inference model does not use preset anchor boxes. Its principle is to estimate the possible location of nodules in the form of a three-dimensional probability density map (3D probability density map). Once the largest possible location of the nodule has been found, the assessment of the likely size of the nodule at that location will continue. Since the second inference model has no anchor frame, it is less likely to artificially limit the possible nodule detection locations and ranges. Since there is a high labeling uncertainty in the center position and size of the ground glass-like nodules, the embodiment of the present invention combined with the second inference model with an unlimited detection range may increase the detection probability of the ground-glass-like pulmonary nodules.

輸入第二推論模型的Head之後得到預測目標資訊

，其輸出可拆分成以下五張形狀為

張量：(1) N張三維機率密度圖，用以預測各影像的像素位置有多少機率為結節中心；(2)~(4)N張三個X, Y, Z位置偏移量預測張量，用以預測各像素若為結節中心，是否位置還需要微調；(5) N張結節直徑預測張量，用以預測各像素若為結節中心，其結節直徑應當為何。需要注意的是，由於是在長寬高皆縮小四倍的影像尺度上去作出預測，因此在預測輸出之前，我們的座標預測需要在各座標軸方向乘以四倍，以還原至原來的影像尺度。 The N feature images after the second upsampling

After inputting the Head of the second inference model, the predicted target information is obtained

, its output can be split into the following five shapes:

Tensor: (1) N three-dimensional probability density maps, used to predict the probability of the pixel position of each image being the nodule center; (2)~(4) N three X, Y, Z position offset prediction sheets (5) N nodule diameter prediction tensors, used to predict what the nodule diameter should be if each pixel is the nodule center. It should be noted that since the prediction is made on an image scale in which the length, width, and height are reduced by four times, our coordinate prediction needs to be multiplied by four times in the direction of each coordinate axis before the prediction output, so as to return to the original image scale.

The trained second inference model may further include a second inference post-processing, input the 3D image block of the second inference model for inference, after obtaining the offset information of the nodule center position, the second inference post-processing calculates the correct inference Nodule center location information to output a set of nodule data.

。其中，

為部分卷積層的步長(可為1 或2)。當

時，表示其輸出的特徵圖之長、寬、高皆會減半。圖5C則是在最後包含三線性上採樣層進行升維。 In the second training mode, the base fact data can be obtained after being processed by the RPN training target generator: (1) the diameter of the training target, (2) the offset correction amount of the center position of the training target, (3) the probability density of the training target graph to learn the possible positions of the training objects. By calculating the difference between the values of (1), (2), and (3) and the target information predicted by the model, the overall error (Total Loss) of the model prediction can be evaluated. The parameter optimization process of the model can finally reduce the overall error and gradually converge, and a trained and mature inference model can be obtained. As shown in FIGS. 5A to 5C , the variable dimension layer, the fixed dimension layer, and the variable dimension and upsampling layer all include a residual block and a squeeze-excitation block. The residual module consists of two normalization layers, two or three convolutional layers (depending on whether downsampling is actually required), and two ReLU activation layers. The compressed excitation module helps to organize images by filtering unimportant features in a weighted manner. It includes a global average pooling layer (Global Average Pooling, GAP), two linear layer sub-modules, and the linear layer sub-module can further include linear layers and activations. layer, the activation layer can be a ReLU activation layer or a Sigmoid activation layer. Figure 5A is a fixed-dimensional layer, and the data after this processing will not change the dimension. The variable dimension layer in Figure 5B changes the dimension by adjusting the number of convolutional layers and the step size. The shape of the tensor output by this variable dimensionality layer is

. in,

is the stride of the partial convolution layer (can be 1 or 2). when

When , the length, width, and height of the output feature map will be halved. Figure 5C includes a trilinear upsampling layer at the end for upscaling.

張

的特徵影像)。壓縮激發模塊是用來學習並給予

張三維特徵影像不同的權重，以便於保留較有助於預測的三維特徵影像。 The convolution layer (3D Conv) and the activation layer are used to learn and generate high-level image features. The normalization layer is used to reshape the feature distribution. In this way, the image features will not have too extreme values or too wide distribution after multiple convolutions or activations, causing the training process to be unstable. The normalization layer used in the embodiment of the present invention is GroupNorm or SyncBatchNorm. These two normalization layers can alleviate the problem of unstable training on limited graphics card memory resources when the model is too hypertrophy, and improve the training results. The output of the compressed excitation module will be equal to the input image (

open

feature image). Compression excitation modules are used to learn and give

Different weights of the three-dimensional feature images are used to preserve the three-dimensional feature images that are more helpful for prediction.

The present invention has been disclosed through the above-mentioned embodiments, which are only part of the preferred embodiments of the present invention, but they are not intended to limit the present invention. Any person familiar with this technical field who has ordinary knowledge can understand the aforementioned technology of the present invention Features and embodiments, and equivalent changes or modifications made without departing from the spirit and scope of the present invention still fall within the scope of the present invention, and the scope of patent protection of the present invention must be defined by the appended claims of this specification Whichever prevails.

1: Pulmonary nodule auxiliary interpretation system 10: Inference Server 11: First inference model 12: The second inference model 13: Lung Lobe Segmentation Model 20: Image server 30: Report auxiliary information subsystem 31: Input module 32: Display module 33: Output module 2: storage device 3:Computed tomography scanner 4: Radiology Information System 5: Medical Information System A: Fixed dimension layer B: variable dimension layer C: variable dimension and upsampling layer

Fig. 1 is a structure diagram of an auxiliary interpretation system for pulmonary nodules according to an embodiment of the present invention.

FIG. 2 is a flow chart of image inference processing in an inference server according to an embodiment of the present invention.

3A is a schematic diagram of the nodule auxiliary detection interface of the report auxiliary information subsystem of the embodiment of the present invention. FIG. 3B is a schematic diagram of a list of inferred nodule data according to an embodiment of the present invention.

FIG. 4A is a schematic flowchart of the data structure of the first inference model of the embodiment of the present invention. FIG. 4B is a schematic flowchart of the data structure of the second inference model of the embodiment of the present invention.

FIG. 5A is a schematic flow diagram of a fixed dimension layer according to an embodiment of the present invention. FIG. 5B is a schematic flow diagram of a variable dimension layer according to an embodiment of the present invention. FIG. 5C is a schematic flow diagram of the variable dimension and upsampling layer according to the embodiment of the present invention.

1: Pulmonary nodule auxiliary interpretation system

10: Inference Server

11: First inference model

12: The second inference model

13: Lung Lobe Segmentation Model

20: Image server

30: Report auxiliary information subsystem

31: Input module

32: Display module

33: Output module

2: storage device

3:Computed tomography scanner

4: Radiology Information System

5: Medical Information System

Claims (10)

A pulmonary nodule aided interpretation system, comprising: an inference server, which receives a group of lung computed tomography images and inputs them to a plurality of inference models to obtain a plurality of nodule data, each of which contains a prediction frame information And a corresponding prediction probability information, delete the nodule data corresponding to the overlapping prediction frame information to output a deduced nodule data and the group of lung computed tomography images, wherein the deduced models include: a first deduced model is A region proposal network model with default anchor boxes and trained using computed tomography images of labeled solid lung nodules; and a second inference model is a region proposal network model without default anchor boxes and using labeled ground glass lung CT imaging training of nodules and semi-solid pulmonary nodules; an image server that marks nodule detection locations in the set of lung CT images according to the inferred nodule data; and a reporting auxiliary information subsystem that operates on a The host computer is connected to the image server, and includes the following modules: a display module, which displays the marked group of lung computed tomography images and the inferred nodule data; an input module, which receives a user's inference A confirmed nodule data obtained by editing the nodule data; and an output module for copying the confirmed nodule data to a reporting system. The auxiliary interpretation system as described in Claim 1, wherein the inference models adopt a UNet backbone structure, and each up-sampling and down-sampling structure includes a residual module and a compressed excitation module. The auxiliary interpretation system as claimed in claim 2, wherein the residual module includes two normalization layers. The auxiliary interpretation system as described in Claim 1, wherein the group of lung computed tomography images can further perform a window frame setting, a resampling process and a cropping process before inputting the inference models. The auxiliary interpretation system as described in Claim 1, wherein the inference models further include a lung lobe cut model, and each of the nodule data further includes a lung lobe information. A method for assisting pulmonary nodule interpretation, comprising: a server host receives a group of lung computed tomography images and inputs them to a plurality of inference models to obtain a plurality of nodule data, each of which includes a prediction frame information and a corresponding One prediction probability information; delete the nodule data corresponding to the repeated prediction frame information to output a deduced nodule data; and mark the nodule detection position in the group of lung computed tomography images according to the deduced nodule data; and a computer host Displaying the group of marked CT images of the lungs and the deduced nodule data, receiving confirmed nodule data edited by a user on the deduced nodule data, and copying the confirmed nodule data to a reporting system; The inference model includes: a first inference model is a region proposal network model with default anchor boxes, and is trained using computed tomography images of labeled solid pulmonary nodules; and a second inference model is a region proposal without default anchor boxes network model and trained using computed tomography images of labeled ground-glass pulmonary nodules and semi-solid pulmonary nodules. The auxiliary interpretation method as described in Claim 6, wherein the inference models adopt a UNet backbone structure, and a residual module and a compression excitation module are added to each upsampling and downsampling structure. The auxiliary interpretation method as claimed in claim 7, wherein the residual module includes two normalization layers. The auxiliary interpretation method as described in Claim 6, further comprising a window frame setting, a re-sampling process and a cropping process before the group of lung computed tomography images are input into the inference models. The auxiliary interpretation method as described in Claim 6, wherein the inference models further include a lung lobe cut model, and each of the nodule data further includes a lung lobe information.

Images