JP7352261B2 - Learning device, learning method, program, trained model, and bone metastasis detection device - Google Patents

Description

The present invention relates to a technique for detecting a bone metastasis region from a scintigram of a subject.

Related research on bone metastasis detection on bone scintigrams includes research by Kazuki Kawakami et al. (Non-Patent Document 1). In Non-Patent Document 1, after performing segmentation (classification of the whole body skeleton), high-accumulation sites are detected in each skeletal region (8 regions) using information such as the average and standard deviation. In addition, in Non-Patent Document 2, bone scintigrams of whole body images were analyzed using the CAD system “BONENAVI version 2.1.7” (FUJIFILM RI Pharma Co., Ltd., Tokyo, Japan) to examine 225 patients with prostate cancer. Patients (bone metastasis cases: 124 cases, normal cases: 101 cases) were analyzed using artificial neural networks (ANN), and the analysis results are shown. This BONENAVI system outputs two imaging markers: ANN and BSI (Bone Scan Index). The ANN value indicates the possibility of bone metastasis, and the range of ANN is a continuous value of 0-1, where "0" means no possibility of bone metastasis, and "1" means suspected bone metastasis. means high. BSI indicates the amount of metastatic tumor (composition ratio of bone metastatic area to the whole body skeleton). The detection results are 82% (102/124) for Sensitivity and 83% (84/101) for Specificity.

Kazuki Kawakami et al. “Introduction of bone scintigraphy diagnostic support software “BONENAVI”” Journal of Nuclear Medicine, 63(0):41-51, 2011 M. Koizumi, K. Motegi, M. Koyama, T. Terauchi, T. Yuasa, and J. Yonese, “Diagnostic performance of a computer-assisted diagnosis system for bone scintigraphy of newly developed skeletal metastasis in prostate cancer patients: search for “Low-sensitivity subgroups” Annals of Nuclear Medicine, 31(7):521-528, 2017

The detection support system for bone metastases on bone scintigrams mainly consists of anatomical structure recognition processing, abnormal region emphasis (feature extraction), and detection processing. Ultimately, these processes are combined to determine the site suspected of bone metastasis, and the results are presented to the doctor.

In the above-mentioned conventional technology, bone metastasis areas are displayed on the subject's scintigram, but non-bone metastasis areas (non-malignant lesion areas such as fractures and inflammation) that have density values similar to bone metastasis areas are classified as bone metastasis areas. There were false positives (so-called "too much detection") and a decrease in detection accuracy. Therefore, the present invention provides a technique for reducing excessive detection while maintaining the detection rate of bone metastatic regions.

One aspect of the present invention is a learning device that generates a neural network model for detecting bone metastasis from scintigrams of a subject, the learning device including scintigrams of a plurality of subjects, bone metastasis areas in each scintigram, and non-bone metastasis. The apparatus includes an input section that inputs a correct label of a region as training data, and a learning section that uses the training data to train a neural network model used for detecting a bone metastasis region in a bone scintigram.

Another aspect of the present invention is a learning method for generating a neural network model for detecting a bone metastasis region from scintigrams of a subject, the method comprising: and a step of inputting a correct label of a non-bone metastatic region as training data, and a step of using the training data to learn a neural network model used for detecting a bone metastatic region in a bone scintigram. .

Another aspect of the present invention is a program for generating a neural network model for detecting a bone metastasis region from scintigrams of a subject, the program comprising: a plurality of scintigrams of subjects and bone metastasis in each scintigram; A program that executes the steps of inputting correct labels of regions and non-bone metastasis regions as training data, and learning a neural network model used for detecting bone metastasis regions in bone scintigrams using the training data. be.

Another aspect of the present invention is a trained model for operating a computer to detect a bone metastasis region from a scintigram of a subject, the model having a convolutional layer and a deconvolutional layer. It is composed of a neural network that inputs the feature map obtained in 1 to the deconvolution layer, and is trained using scintigrams of multiple subjects and the correct labels of bone metastasis areas and non-bone metastasis areas in each scintigram as training data. This is a trained model that causes a computer to function to detect bone metastasis areas from the subject's scintigrams input to the neural network.

By learning the neural network model using the correct labels of the bone metastasis region and non-bone metastasis region in this way, the bone metastasis region can be appropriately detected from the subject's scintigram using this model.

FIG. 1 is a diagram showing the configuration of a learning device according to a first embodiment. FIG. 2A is a diagram showing the subject's scintigram and correct label. FIG. 2B is a diagram showing an example of a patch image. FIG. 2C is a diagram showing another example of a patch image. FIG. 3 is a diagram showing the configuration of a neural network model. FIG. 4 is a diagram showing the configuration of the bone metastasis detection device according to the first embodiment. FIG. 5 is a diagram showing an example of a patch image cut out from a scintigram. FIG. 6 is a diagram showing the operation of the learning device according to the first embodiment. FIG. 7 is a diagram showing the operation of the bone metastasis detection device according to the first embodiment. FIG. 8 is a diagram showing the configuration of a learning device according to the second embodiment. FIG. 9 is a diagram showing the configuration of a learning device according to a modified example. FIG. 10 is a FROC curve showing the relationship between sensitivity and FP (P) obtained through experiments. FIG. 11 is a diagram showing the configuration of a learning device according to the third embodiment. FIG. 12 is a diagram showing a scintigram of a subject input into the learning device of the third embodiment. FIG. 13 is a diagram showing the configuration of a neural network model used in the learning device of the third embodiment. FIG. 14 is a diagram showing the configuration of a bone metastasis detection device according to the third embodiment. FIG. 15 is a diagram showing the operation of the learning device according to the third embodiment.

Hereinafter, a learning device and a bone metastasis detection device according to embodiments of the present invention will be described with reference to the drawings. In the following description, the numerical values described as conditions such as size are merely examples of preferred embodiments, and are not intended to limit the present invention.

A learning device according to an embodiment is a learning device that generates a neural network model for detecting bone metastasis from scintigrams of a subject, and is a learning device that generates a neural network model for detecting bone metastasis from scintigrams of a plurality of subjects. The apparatus includes an input section that inputs correct labels of metastatic regions as teaching data, and a learning section that uses the teaching data to train a neural network model used for detecting bone metastatic regions in bone scintigrams. Here, the non-bone metastasis area is an area that has a density value similar to that of the bone metastasis area, but is an area that has not caused bone metastasis. Non-bone metastasis areas include non-malignant disease areas (fractures, inflammation, etc.). Note that the bone metastasis region is also referred to as "abnormal accumulation."

By learning the neural network model using the correct labels of the bone metastasis region and non-bone metastasis region in this way, the bone metastasis region can be appropriately detected from the subject's scintigram using this model.

The learning device of the embodiment includes a patch image creation unit that generates a patch image by cutting out a region in which bones of a subject are shown from scintigrams of a plurality of subjects, and the learning unit generates a patch image and a correct answer corresponding to the patch image. Learning may be performed using labels as training data.

The memory size required for learning in a neural network model increases as the image size increases. According to the configuration of the embodiment, the memory size required for learning can be reduced by generating a patch image in which a region in which the subject's bones are shown is cut out and performing learning using the patch image. Furthermore, since the appearance of the bone metastasis region does not depend much on the shape of the organ, learning can be performed even when the entire organ is not photographed. Therefore, patch images are suitable as training data for learning a neural network model for detecting bone metastasis regions.

In the learning device of the embodiment, the patch image creation unit scans a window of a predetermined size on the scintigram of the subject, and when bones of the subject are included in the window, converts the area of the window into a patch image. You can also cut it out as By scanning the window and cutting out patch images in this way, patch images can be evenly cut out from the subject's scintigram.

The learning device of the embodiment is configured such that, in the patch images created by the patch image creation unit, a patch image that includes a bone metastasis region or a non-bone metastasis region and a patch image that does not include either a bone metastasis region or a non-bone metastasis region are configured. It may also include a teacher data analysis unit that calculates the ratio.

The inventors performed inference using a model generated by learning with various training data, and investigated conditions under which a neural network model that can appropriately detect bone metastasis areas can be generated. As a result, the content of the patch images that make up the training data (composition ratio of patch images that include bone metastasis areas or non-bone metastasis areas to patch images that do not include either bone metastasis areas or non-bone metastasis areas) is determined by the neural network. We found that this is related to the accuracy of the model. According to the embodiment, by analyzing the teacher data used for learning and displaying the contents of the teacher data, it becomes possible to adjust the teacher data so that appropriate learning can be performed.

The learning device of the embodiment calculates bone metastasis areas and non-bone metastasis areas from patch images created by the patch image creation unit so that the composition ratio determined by the teacher data analysis unit falls within a predetermined range. A patch image selection unit may be provided that thins out patch images that do not include any of the above.

If the composition ratio of patch images containing bone metastasis regions is too small, the accuracy of the model obtained by learning may deteriorate. Therefore, according to the configuration of the embodiment, the composition ratio of patch images containing bone metastasis regions is increased. .

The learning device of the embodiment may include a patch image reversing unit that horizontally or vertically inverts at least part of the patch images created by the patch image creating unit.

By inverting the patch images in this way, variations in the training data are increased, and a highly accurate model can be generated through learning. Note that when the patch image is inverted, the inverted patch image may be used, or both the inverted patch image and the patch image before inversion may be used as the teacher data.

In the learning device of the embodiment, the neural network has an encoder-decoder structure, and may include a structure for inputting a feature map obtained in the encoder structure to the decoder structure.

With this configuration, global features of an image are captured by the encoder structure, and local features are also learned by inputting the feature map obtained in the encoding process to the decoder structure. By capturing the spatial spread of the bone metastasis region, positional information on the bone metastasis region can be appropriately obtained.

In the bone metastasis detection device of the embodiment, the neural network has a structure in which a first network part having an encoder-decoder structure and a second network part having an encoder-decoder structure are combined, and the input part is , for each subject, the scintigram taken from the front and back and its correct label are input, and the learning unit inputs the scintigram taken from the front of the subject into the input layer of the first network part, and inputs the scintigram taken from the front of the subject into the input layer of the first network part. Learning may be performed by inputting a scintigram photographed from the rear of the subject into the input layer.

Further, in the bone metastasis detection device of the embodiment, the neural network has a structure in which a first network part having an encoder-decoder structure and a second network part having an encoder-decoder structure are combined, and The learning section inputs the scintigrams taken from the front and back of each subject and their correct labels, and the learning section inputs the first patch image cut out from the scintigram taken from the front of the subject into the input layer of the first network section. At the same time, learning may be performed by inputting, into the input layer of the second network part, a second patch image corresponding to the first patch image cut out from a scintigram photographed from the rear of the subject.

Instead of processing the scintigrams taken from the front and the scintigrams taken from the back independently, they are processed simultaneously by a neural network that combines two network parts with an encoder-decoder structure, thereby detecting the bone metastasis area. It is possible to generate a neural network model that improves the accuracy of differentiation of non-bone metastatic areas.

In the bone metastasis detection device according to the embodiment, the non-bone metastasis region includes a non-malignant lesion region, and the input section is configured to perform scintigraphy of a plurality of subjects with respective correct labels for the bone metastasis region and the non-malignant lesion region. gram as training data, and the learning unit may use the training data to learn a neural network model for detecting each of the bone metastasis region and the non-malignant lesion region.

The bone metastasis detection device of the embodiment includes a storage unit that stores a trained model of a neural network trained by the above-described learning device, an input unit that inputs a scintigram of a subject, and a patch image that is created from the scintigram. a patch image creation unit; an inference unit that inputs the patch image to the input layer of the learned model read from the storage unit and calculates the bone metastasis area included in the patch image; and an output unit that outputs data indicating the bone metastasis area. Equipped with With this configuration, it is possible to reduce excessive detection while maintaining the detection rate of bone metastasis regions.

The program of the embodiment is a program for detecting a bone metastasis region from a scintigram of a subject, and includes the steps of inputting the scintigram of the subject into a computer, creating a patch image from the scintigram, and the steps described above. reading the trained model from a storage unit that stores the trained model of the neural network trained by the learning device, inputting the patch image to the input layer of the trained model, and calculating the bone metastasis area included in the patch image; , and outputting data indicating the bone metastasis area.

The program of the embodiment is a program for detecting a bone metastasis area from a scintigram of a subject, and includes the steps of inputting two scintigrams taken from the front and back of the subject into a computer, and The trained model is read from the storage unit that stores the trained model generated in advance by learning using the teacher data, and the two sheets are added to the input layer of the trained model. inputting a scintigram of the scintigram, calculating the bone metastasis region included in the scintigram, and outputting data indicating the bone metastasis region. In this way, by inverting one of the two scintigrams taken from the front and back so that they are in the same direction, and then inputting the two scintigrams into the input layer of the trained model and performing inference, the bone metastasis area can be determined. can be detected with high accuracy.

Hereinafter, a learning device and a bone metastasis detection device according to an embodiment will be described with reference to the drawings.
(First embodiment)
FIG. 1 is a diagram showing the configuration of a learning device 1 according to the first embodiment. A learning device 1 according to the first embodiment is a device that generates, through learning, a neural network model for detecting a bone metastasis region from a scintigram of a subject. The neural network model generated by the learning device 1 of this embodiment is a model for classifying regions of a subject's scintigram into three classes: bone metastasis region, non-bone metastasis region, and background. In this embodiment, the class of non-bone metastatic regions includes physiological accumulation regions such as the kidney and bladder in addition to non-malignant lesion regions.

The learning device 1 includes an input unit 10 for inputting teacher data, a control unit 11 for learning a neural network model based on the teacher data, a storage unit 16 for storing a model generated by learning, and a storage unit 16 for storing a model generated by learning. It has an output section 17 that outputs the stored model to the outside.

FIG. 2A is a diagram showing an example of teacher data input to the input unit 10. The training data consists of the subject's scintigram and correct label data given to the subject's scintigram. In this example, the size of the scintigram is 512×1024 [pixels]. The correct label specifies, for each pixel, whether the pixel of interest is a pixel corresponding to an accumulation or a background (other than accumulation) pixel. In the case of a pixel corresponding to an accumulation, it is further specified whether it is a bone metastasis area, an injection leak or urine leak, or a non-bone metastasis area. Note that injection leakage or urine leakage is excluded from detection by the bone metastasis detection apparatus 20 of this embodiment.

Next, the control section 11 will be explained. The control section 11 includes a density normalization processing section 12 , a patch image creation section 13 , a patch image inversion section 14 , and a learning section 15 .

The density normalization processing unit 12 has a function of normalizing density values in order to suppress variations in density values of normal bone regions that differ from subject to subject. The density normalization processing unit 12 normalizes density values by processing density range adjustment, normal bone level identification, and gray scale normalization. To adjust the density range, for example, linear conversion is performed so that the cumulative top 0.2% pixel values of the density histogram excluding the density value 0 of the input image become 1023, and the cumulative top 98% pixel values become 0.

Identification of the normal bone level uses a multiple threshold method for the density histogram excluding the density value 0 of the image after density range adjustment. The threshold value is set in 1% increments from the cumulative top 1% of the histogram to the 25% pixel value. After binarizing the range-adjusted image using each threshold value, 4-concatenated labeling is performed. From the results, regions having an area of 10 [pixels] or more and less than 4900 [pixels] are extracted. Next, the average density values within the determined regions are arranged in descending order, and a transition point (border between the normal region and the abnormal region) is determined. A transition point is defined as a point where two consecutive average density values are 3% or less of the peak pixel. The peak pixel is the maximum value of the average density value for each region.

Subsequently, in gray scale normalization, the average value P of the average density values of five consecutive points including the transition point is determined. Finally, normalization is performed by multiplying the image after density range adjustment by a normalization coefficient F=k/P. Here, the constant k was set to 358.4, but this value was determined experimentally (Tatsuya Ito, "Development of abnormal accumulation detection processing on bone scintigrams", Tokyo University of Agriculture and Technology Bachelor's thesis, 2015).

The patch image creation unit 13 has a function of cutting out and creating patch images from the subject's scintigram. In this example, the size of the patch image is 64×64 [pixels]. A window of 64 x 64 [pixels] is scanned over the subject's scintigram (512 x 1024 [pixels]) at intervals of 2 [pixels], and (1) the accumulated labels (bone metastasis area or non-bone metastasis area) are displayed in the window. ) is included, or (2) a bone region is included and no accumulation is included, the image is extracted as an image patch. The patch image creation unit 13 cuts out image patches from the input bone scintigrams of a plurality of subjects. FIGS. 2B and 2C are diagrams illustrating examples of image patches cut out from the subject's scintigram and their corresponding correct labels.

The patch image reversing unit 14 has a function of horizontally reversing some of the created patch images.

The learning unit 15 has a function of learning a neural network model for detecting bone metastasis regions from scintigrams using patch images. In this embodiment, U-Net, which is one of FCN (Fully Convolutional Network), is used as a neural network model.

FIG. 3 is a diagram showing an example of a neural network model used in this embodiment. FIG. 3 shows an example of the structure when a patch with a patch size of 64×64 [pixel] is input. The neural network model used in this embodiment has an encoder-decoder structure. The encoder structure repeatedly performs convolution and pooling to extract global features of the image. The decoder structure restores the global structure to the original size image, but in the process, local features are also learned by combining the features obtained in the encode process.

In addition, in order to extract more advanced features, the neural network model used in this embodiment uses one of the residual blocks, Bottleneck (K. He, X. Zhang, S. Ren, and J. Sun ``Deep residual learning for image recognition”arXiv:1512.03385, 2015).

The structure of the neural network model shown in FIG. 3 will be explained in detail. In this example, the input image is grayscale and the dimensions of the input are 64x64x1. First, the number of channels is set to 32 in the convolutional layer and passed through Bottleneck. After that, 2×2 MAX pooling is performed and the signals are passed through Bottleneck so that the number of channels is doubled. These layers are repeated a total of four times, and the final feature map size of the encoder is 4×4×512.

Next, we use a deconvolution layer to double the size of the feature map. Then, the output of the deconvolution layer and the feature map of the encoder are concatenated and passed through Bottleneck. Similar to the Encoder, these layers are repeated a total of four times, and the final feature map size of the Decoder is 64×64×32. Finally, the number of output classes is 3 channels (background, bone metastasis region, and non-bone metastasis region) using a 1×1 convolution layer, which is 64×64×3. In addition, all 3×3 convolutional layers are zero-padded, and after the convolutional layers, Batch Normalization (S. Ioffe, and C. Szegedy, “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift” arXiv:1502.03167 , 2015) and the ReLU function.

The learning unit 15 performs learning of a neural network model using patch images (including those that have been horizontally inverted by the patch image inverting unit 14) and their correct labels. Learning is performed by evaluating the error (loss function) between the probability p _i of the correct answer and the probability p i obtained by converting the output when a patch image is input to the neural network model using a softmax function. The softmax function and loss function are shown below.

The learning unit 15 also verifies the learned network using a verification data set (validation set). A learning model that has been trained an arbitrary number of times using teacher data is saved, and the parameters of the learning model are searched using a validation set for all learning models. The number of repetitions of learning is determined using FP(P)+FN(P), which is the sum of pixel-by-pixel overpicking FP(P) and pixel-by-pixel oversight FN(P), as an evaluation value. The learning unit 15 stores the model generated through learning in the storage unit 16.

The configuration of the learning device 1 according to the present embodiment has been described above, but an example of the hardware of the learning device 1 described above is a computer equipped with a CPU, RAM, ROM, hard disk, display, keyboard, mouse, communication interface, etc. It is. The learning device 1 described above is realized by storing a program having modules for realizing each of the functions described above in the RAM or ROM, and executing the program by the CPU. Such programs are also included within the scope of the present invention.

FIG. 4 is a diagram showing the configuration of the bone metastasis detection device 20. The bone metastasis detection device 20 includes an input unit 21 that inputs a scintigram of a subject, a control unit 22 that detects a bone metastasis region from the scintigram of a subject, and a memory that stores a trained model learned by the learning device 1 described above. section 26, and an output section 27 that outputs data on the detected bone metastasis region.

The control section 22 includes a density normalization processing section 23, a patch image creation section 24, and an inference section 25. The density normalization processing unit 23 is the same as the density normalization processing unit 12 included in the learning device 1. The patch image creation unit 24 has a function of cutting out a 64×64 [pixels] patch image from the input scintigram of the subject. The basic configuration is the same as that of the patch image creation unit 13 included in the learning device 1, but the intervals at which patch images are cut out are different. That is, in the learning device 1, patch images are cut out at intervals of 2 [pixels], but in the bone metastasis detection device 20, patch images are cut out at intervals of 32 [pixels].

The inference unit 25 reads out the trained model from the trained model storage unit 26 and inputs the patch image to the input layer of the trained model, so that each pixel of the patch image corresponds to each of the background, bone metastasis area, and non-bone metastasis area. Find the probability of falling into a class.

FIG. 5 is a diagram showing an example of a patch image cut out from a scintigram. As shown in FIG. 5, the patch images are cut out from the subject's scintigram in such a way that adjacent patch images overlap each other by half. Therefore, for example, in region R, patch images A to D overlap, and the feature map of pixels in region R is obtained for each of the four patch images A to D. The inference unit 25 averages the feature maps obtained for each of the four patch images. Then, the inference unit 25 converts the reconstructed output into a probability using a softmax function, determines the class with the highest probability for each pixel, and sets it as the final output.

FIG. 6 is a diagram showing the operation of the learning device 1. The learning device 1 inputs scintigrams of a plurality of subjects and their corresponding correct labels (background, bone metastasis region, non-bone metastasis region) as training data (S10). The learning device 1 performs density normalization on the input scintigram (S11), and creates a patch image from the normalized scintigram (S12). The learning device 1 horizontally inverts some of the created patch images (S13). Subsequently, the learning device 1 performs learning of the neural network model using the patch images and the correct labels corresponding to the patch images (S14), and stores the neural network model obtained through the learning in the storage unit 16 (S15). Note that when a trained model is used in the bone metastasis detection device 20, the learned model stored in the storage unit 16 is read out and output to another device or the like.

FIG. 7 is a diagram showing the operation of the bone metastasis detection device 20. The bone metastasis detection device 20 inputs a scintigram of a subject to be examined (S20). The bone metastasis detection apparatus 20 performs density normalization of the input scintigram (S21), and creates a patch image from the normalized scintigram (S22). The bone metastasis detection device 20 reads out the learned neural network model from the storage unit 26, inputs the patch image to the input layer of the read neural network model, and detects the bone metastasis region of each pixel included in the patch image. (S23). The bone metastasis detection device 20 integrates detection results for pixels in an area where a plurality of patch images overlap (S24). The bone metastasis detection device 20 outputs the final result of the determined bone metastasis area (S25).

The learning device 1 of the first embodiment learns a neural network model using a subject's scintigram and its corresponding correct answer label. By using this trained model, so-called "overpicking" can be reduced and bone metastasis regions can be appropriately detected.

Moreover, the learning device 1 of the first embodiment can reduce the memory size required during learning by performing learning using patch images cut out from the subject's scintigram. Further, since the location of the bone metastasis region does not depend on the shape of the organ, appropriate learning can be performed even if learning is performed by dividing into patch images.

In addition, the learning device 1 of the first embodiment increases variations in teacher data by horizontally inverting some of the patch images among a large number of patch images, thereby making it possible to obtain robust learning results. Note that in this embodiment, an example is given in which the patch image is horizontally reversed, but the patch image may be vertically reversed. The method using a vertically inverted patch image is suitable when the anatomical structure of the background bone is vertically symmetrical (for example, when examining an accumulation of limbs extending in the vertical direction).

(Second embodiment)
FIG. 8 is a diagram showing the configuration of the learning device 2 according to the second embodiment. Similar to the first embodiment, the neural network model generated by the learning device 2 of the second embodiment divides the region of the subject's scintigram into three classes: bone metastasis region, non-bone metastasis region, and background. This is a model for classifying. The basic configuration of the learning device 2 of the second embodiment is the same as the learning device 1 of the first embodiment, but the learning device 2 of the second embodiment has a large number of teaching data. The computer includes a teacher data analysis unit 18 that analyzes the content of the image patch. The large number of image patches include patch images that include bone metastasis areas or non-bone metastasis areas, and patch images that do not include either bone metastasis areas or non-bone metastasis areas. The teacher data analysis unit 18 analyzes patch images that include a bone metastasis area or non-bone metastasis area and patch images that do not include either a bone metastasis area or a non-bone metastasis area, which are included in a large number of patch images that are teacher data. Find the composition ratio. The output unit 17 outputs data on the composition ratio of the patch image that generated the trained model stored in the storage unit 16.

By outputting the composition ratio of the patch images used to generate the trained model in this way, the accuracy of detecting bone metastasis areas using the trained model will not be high, and when generating a new trained model, , you can get hints on how to change the training data to perform learning. In this embodiment, an example is given in which a user who sees the composition ratio of a patch image changes the teacher data. You may.

FIG. 9 is a diagram showing a learning device 3 that is a modification of the second embodiment. The learning device 3 according to the modified example includes a patch image selection unit 19 in addition to the configuration of the learning device 2 of the second embodiment. The patch image selection unit 19 has a function of selecting patch images to be used for learning based on the analysis result by the teacher data analysis unit 18. According to the research of the present inventors, it is considered that if there are too many patch images that do not include either a bone metastasis region or a non-bone metastasis region, an appropriate model cannot be generated. Therefore, in the learning device 3 according to the modified example, when the composition ratio of patch images that do not include bone metastasis areas or non-bone metastasis areas is equal to or higher than a predetermined threshold, the learning device 3 does not include bone metastasis areas or non-bone metastasis areas. Select which patch images to use for training instead of using all available patch images. This increases the possibility of generating a model with high accuracy in detecting bone metastasis regions.

(Third embodiment)
FIG. 11 is a diagram showing the learning device 4 according to the third embodiment. The learning device 4 of the third embodiment uses Butterfly-Net as a model of a neural network to be learned. Butterfly-Net has a structure in which two network parts having an encoder-decoder structure are connected. Butterfly-Net is described in detail in "Btrfly Net: Vertebrae Labeling with Energybased Adversarial Learning of Local Spine Prior" by Anjany Sekuboyina et al., MICCAI 2018.

The learning device 4 includes an input section 40 for inputting teacher data, a control section 41 for learning a neural network model based on the teacher data, a storage section 47 for storing a model generated by learning, and a storage section 47 for storing a model generated by learning. It has an output section 48 that outputs the stored model to the outside. The learning device 4 of the third embodiment divides the subject's scintigram area into bone metastasis areas, non-malignant lesion areas (fractures, inflammation, etc.), other areas (physiological accumulation areas such as kidneys, bladder, etc.) A model is generated for classification into three classes: injection leakage, urine leakage, and background). In this embodiment, physiological accumulation regions are included in the class of other regions and classified into a different class from non-malignant lesion regions.

The learning device 4 of the present embodiment uses, as training data, a scintigram of the subject taken from the front (hereinafter referred to as "front image") and a scintigram of the subject taken from the rear (hereinafter referred to as "back image"). , using the ground truth label given to each scintigram. FIG. 12 is a diagram showing an example of a front image and a rear image. Note that the rear image is horizontally reversed.

The control section 41 includes an image inversion section 42 , a front and back image alignment section 43 , a density normalization processing section 44 , a patch image creation section 45 , and a learning section 46 .

The image reversing unit 42 has a function of reversing the rear image. When the image reversing unit 42 performs reversal, the correct label given to the backward image is also reversed. The front and rear image alignment unit 43 aligns the front image and the reversed rear image. Although an example is given here in which the rear image is inverted and aligned with the front image, it is of course possible to invert the front image and align it with the rear image.

The density normalization processing unit 44 has a function of normalizing density values in order to suppress variations in density values of normal bone regions that differ from subject to subject. The density normalization processing unit 44 normalizes density values by processing density range adjustment, normal bone level identification, and gray scale normalization. The density normalization processing unit 44 converts the density I _in of the input scintigram into a normalized density I _normalized using the following equation (3).

The patch image creation unit 45 has a function of cutting out and creating patch images from the subject's scintigram. In the present embodiment, the patch image creation unit 45 cuts out patch images at corresponding locations from the front image and the rear image to generate a pair of front and rear patch images. In FIG. 12, patch image A obtained from the front image and patch image A' obtained from the rear image are paired patch images. Furthermore, patch image B and patch image B' are also a pair of patch images.

In this example, the size of the patch image is 64×64 [pixels]. A window of 64 x 64 [pixels] is scanned over the subject's scintigram (512 x 1024 [pixels]) at intervals of 2 [pixels], and (1) the accumulated labels (bone metastasis area or non-bone metastasis area) are displayed in the window. ) is included, or (2) a bone region is included and no accumulation is included, the image is extracted as an image patch. If either the front image or the rear image meets the conditions (1) and (2) above and a patch image is cut out, a paired patch image is cut out from the other of the front image or the back image.

The learning unit 46 has a function of learning a neural network model for detecting bone metastasis regions from scintigrams using patch images. In this embodiment, Butterfly-Net, which has a structure in which two U-Nets are connected, is used as a neural network model.

FIG. 13 is a diagram showing an example of Butterfly-Net used in this embodiment. The upper side of the Butterfly-Net has a downward convex configuration, and has almost the same structure as the network shown in FIG. 3. The lower side of the Butterfly-Net has an upwardly convex configuration, and has the same structure as the network shown in FIG. 3 (it is only drawn upside down). In Butterfly-Net, two U-Nets are combined at 128 8×8 feature maps.

In addition, in order to extract more advanced features, the neural network model used in this embodiment uses one of the residual blocks, Bottleneck (K. He, X. Zhang, S. Ren, and J. Sun ``Deep residual learning for image recognition” arXiv:1512.03385, 2015). Butterfly-Net improved in this way is referred to as "ResButterfly-Net" in this book.

In this example, the input image is grayscale and the dimensions of the input are 64x64x1. First, the number of channels is set to 32 in the convolutional layer and passed through Bottleneck. After that, 2×2 MAX pooling is performed and the signals are passed through Bottleneck so that the number of channels is doubled. A process of repeating these layers three times in total is performed on each of the upper and lower U-Nets. Then, when a feature map with a size of 8 x 8 x 128 was obtained, the upper and lower two U-Net feature maps were combined, Bottleneck and MAX pooling were performed twice, and finally 2 A feature map of size ×2×512 is obtained.

Next, after passing through Bottleneck, deconvolution is performed to double the size of the feature map. Then, the output of the deconvolution and the feature map of the encoder are concatenated and passed through Bottleneck. After performing these layers twice in the same way as the encoder, the results are duplicated, the feature maps of the upper and lower encoders are connected, and the process of deconvolution through Bottleneck is repeated three times. Finally, the number of output classes is set to 3 channels (bone metastasis region, non-malignant lesion region, and other regions) using a 1×1 convolution layer, which is 64×64×3. Note that although FIG. 13 shows a bone metastatic region and a non-malignant lesion region, the portions other than the bone metastatic region and non-malignant lesion region are other regions.

ここで、ｗ_ｃは、ピクセル数の違いの影響を低減するためのクラスｃの重みである。

The learning unit 46 performs learning of the neural network model using the pair of previous and subsequent patch images and their correct labels. Learning is performed by evaluating the error (loss function) between the probability p _i obtained by converting the output when a pair of patch images is input into the neural network model using a softmax function and the probability of a correct answer. The loss function is shown below.

Here, w _c is the weight of class c to reduce the influence of the difference in the number of pixels.

FIG. 14 is a diagram showing the configuration of a bone metastasis detection device 50 according to the third embodiment. The bone metastasis detection device 50 includes an input unit 51 that inputs a scintigram of a subject, a control unit 52 that detects a bone metastasis region from the scintigram of a subject, and a memory that stores a trained model learned by the learning device 4 described above. section 58, and an output section 59 that outputs data on the detected bone metastasis region.

The control section 52 includes an image inversion section 53, a front and rear image alignment section 54, a density normalization processing section 55, a patch image creation section 56, and an inference section 57. The image inversion unit 53, the front and back image alignment unit 54, and the density normalization processing unit 55 are the same as the image inversion unit 42, the front and back image alignment unit 43, and the density normalization processing unit 44 included in the learning device 4. The patch image creation unit 56 has a function of cutting out a patch image from the input scintigram (front image and rear image) of the subject. The patch image creation unit 56 cuts out corresponding areas from the preceding and following scintigrams to generate a pair of patch images. Note that the patch image creation unit 56 may differ from the patch image creation unit 45 included in the learning device 4 in the interval at which patch images are cut out.

The inference unit 57 reads out the trained model from the trained model storage unit 58, inputs a pair of patch images to the input layer of the trained model, and determines whether each pixel of the patch image is a bone metastasis area, a non-malignant lesion area, or another area. Find the probability of each class in the region.

FIG. 15 is a diagram showing the operation of the learning device 4. The learning device 4 inputs scintigrams (front images and back images) of a plurality of subjects and their corresponding correct labels (bone metastasis area, non-malignant lesion area, other areas) as training data (S30). The learning device 4 inverts the rear image (S31) and aligns the front image and the inverted rear image (S32). Next, the learning device 4 normalizes the density of the input front image and rear image (S33), cuts out corresponding regions of the front and rear images, and generates a plurality of pairs of patch images (S34).

Subsequently, the learning device 4 performs learning of the neural network model using the patch images and the corresponding correct labels (S35). As described above, in this learning, a pair of previous and subsequent patch images are input to the input layer of Butterfly-Net, and learning is performed based on the class and correct answer data output from the output layer. The learning device 4 stores the neural network model obtained through learning in the storage unit 47 (S36). Note that when a trained model is used in the bone metastasis detection device 50, the learned model stored in the storage unit 47 is read out and output to another device or the like.

The learning device 4 of the third embodiment uses the Butterfly-Net neural network model as a learning model, and has a configuration in which learning is performed by inputting a pair of previous and subsequent patch images to its input layer, so that the learning device 4 has a high correlation. By simultaneously processing patch images, it is possible to generate a neural network model that can accurately detect bone metastasis areas.

Note that in the learning device 4 according to the third embodiment, as in the first embodiment, patch images are inverted, and some of the patch images among a large number of patch images are horizontally inverted. , the variation of training data may be increased. However, in this embodiment, the patch images that are the teacher data are a pair of front and rear images, so when the front or rear patch image is reversed, the other patch image is also reversed in the same direction. Robust learning can be performed by inverting patch images and increasing training data in this way.

Further, in the learning device 4 according to the third embodiment described above, an example was given in which a trained model is generated that is classified into a different class from that in the first embodiment, but the learning device 4 is the same as the first embodiment. Of course, it is also possible to generate trained models that classify into classes. Conversely, in the first and second embodiments described above, injection leakage and urine leakage were excluded, but physiological accumulation in the kidneys and bladder, injection leakage/urinary leakage, and the background were excluded from other areas. It is also possible to generate a model that classifies the data into the same class as in the third embodiment.

(Example 1)
An example will be described in which a bone metastasis region is detected using a learned model generated using the learning device 1 of the first embodiment. Bone metastasis areas were detected using trained models that were generated using training data with inverted patch images and trained models created using training data that did not invert patch images.
(Sample used in experiment)
・Frontal bone scintigram density value normalized image: 103 cases ・Image size: 512 x 1024 [pixels]
・Resolution: 2.8 x 2.8 [mm/pixel]
・Patch size: 64×64 [pixels]

(Evaluation method)
・3-fold cross validation (learning: 68 cases, validation: 17 cases, test: 17-18 cases)
Note that the verification data is data for determining the number of repetitions of learning.

(Evaluation value)
・FP(P): Too much detection in pixel units ・FN(P): Missing pixel units in bone metastasis areas ・Sensitivity: Detection rate in area units in bone metastasis areas
= (Number of bone metastasis areas detected) / (Number of bone metastasis areas)
・FROC curve: Sensitivity vs. FP(P) or FP(R)
・FP(P)/Background+FN(P)/Bone metastasis area

(Model learning conditions)
・Optimizer: Adam (α=0.001, β ₁ =0.9, β ₂ =0.999)
・Batch size 64
・Number of iterations: 10,000 times ・Select the network with the minimum FP (P) + FN (P) during verification

(Comparative example)
As a comparative example, MadaBoost (C. Domingo and O. Watanabe, "MadaBoost: A modification of AdaBoost", Proc. Thirteenth Annual Conference on Computational Learning Theory, pp.180-189) performs detection based on the classification results of many weak classifiers. , 2000). The MadaBoost algorithm used a method developed in the inventors' laboratory (Yuta Minami, "Improvement of bone metastasis detection processing on bone scintigrams", 3rd Oncology Nuclear Medicine Image Analysis Software Development Conference).

(Experimental result)
FIG. 10 is a FROC curve showing the relationship between sensitivity and FP (P) obtained through experiments. The FROC curve shown in FIG. 10 shows that as the sensitivity on the vertical axis increases, the number of "too much pick-up" in which non-bone metastasis areas are picked up as bone metastasis areas increases. In the method according to the embodiment, for example, if Sensitivity is set to 0.8, the excessive pick-up is 200 pixels or less. In contrast to the conventional method using MadaBoost, in which over-pickup of 500 pixels or more occurred, in the embodiment, it was possible to suppress over-pickup. In Figure 10, the graph of U-Net (Flip) shows the result of detection using a trained model generated using training data in which some patch images are inverted; , is a diagram showing the results of detection using a trained model generated without inverting patch images.

(Example 2)
An example in which a bone metastasis region is detected using a learned model generated using the learning device 4 of the third embodiment will be described. As the trained models, ResButterfly-Net described in the third embodiment and Butterfly-Net in which Bottleneck of ResButterfly-Net was replaced with a convolution layer were used. Furthermore, U-Net was used as a comparative example.

(Sample used in experiment)
・Japanese men aged 52 to 95 with prostate cancer: 246 cases (evaluation method)
・3-fold cross validation (learning: 164 cases, validation: 41 cases, test: 41 cases)
Note that the verification data is data for determining the optimal number of repetitions of learning.
(Evaluation value)
・FP(P): Picking up too much in pixel units ・FP(R): Picking up too much in region units ・FP(P)+FN(P): Picking up too much in pixel units and overlooking pixel units in bone metastasis areas (model learning conditions)
・Optimizer: Adam (α=0.001, β ₁ =0.9, β ₂ =0.999)
・Batch size 256
- Number of iterations: Maximum of 50,000 times, and the optimal number of iterations was set when the total number of misclassified pixels was minimized.

表１に見られるように、学習モデルとしてＲｅｓＢｕｔｔｅｒｆｌｙ－Ｎｅｔ、Ｂｕｔｔｅｒｆｌｙ－Ｎｅｔを用いると、Ｕ－Ｎｅｔを用いたモデルよりも、ホットスポット検出時の誤りが少ないことが、多くの指標において確認できた。(Experimental result)
Table 1 shows each evaluation value when the sensitivity of the bone metastasis area is 0.9. The upper part is the result of the front image, and the lower part is the result of the rear image.

As shown in Table 1, it was confirmed that using ResButterfly-Net and Butterfly-Net as learning models resulted in fewer errors when detecting hot spots than the model using U-Net in many indicators. .

The above embodiments and examples include the technical ideas shown in (1) to (11) below.
(1) A learning device that generates a neural network model for detecting abnormal accumulation from a subject's scintigram,
an input unit for inputting scintigrams of a plurality of subjects and correct labels of normal accumulation and abnormal accumulation in each scintigram as training data;
a learning unit that uses the training data to learn a neural network model used to detect abnormal accumulation in bone scintigrams;
A learning device equipped with.
(2) comprising a patch image creation unit that creates a patch image by cutting out a region in which the bones of the subject are shown from the scintigrams of the plurality of subjects;
The learning device according to (1), wherein the learning unit performs learning using the patch image and the correct label corresponding thereto as teacher data.
(3) The patch image creation unit scans a window of a predetermined size on the scintigram of the subject, and when bones of the subject are included in the window, the area of the window is used as the patch image. The learning device described in section (2).
(4) A teacher data analysis unit that calculates the composition ratio of patch images that include normal accumulation or abnormal accumulation and patch images that do not include either normal accumulation or abnormal accumulation in the patch images created by the patch image generation unit; The learning device according to 2) or (3).
(5) A patch image that does not include either normal accumulation or abnormal accumulation from the patch images created by the patch image creation unit so that the composition ratio determined by the training data analysis unit falls within a predetermined range. The learning device according to claim 4, further comprising a patch image selection unit that thins out the patch images.
(6) The learning device according to any one of (1) to (5), including a patch image reversing unit that horizontally or vertically inverts at least part of the patch images created by the patch image creating unit. .
(7) The learning device according to any one of (1) to (6), wherein the neural network has an encoder-decoder structure and includes a structure for inputting a feature map obtained from the encoder structure to the decoder structure.
(8) A learning method for generating a neural network model for detecting abnormal accumulation from a subject's scintigram, comprising:
inputting scintigrams of a plurality of subjects and correct labels of normal accumulation and abnormal accumulation in each scintigram as training data;
a step of learning a neural network model used for detecting abnormal accumulation in bone scintigrams using the training data;
A learning method that prepares you.
(9) A program for generating a neural network model for detecting abnormal accumulation from a subject's scintigram,
inputting scintigrams of a plurality of subjects and correct labels of normal accumulation and abnormal accumulation in each scintigram as training data;
a step of learning a neural network model used for detecting abnormal accumulation in bone scintigrams using the training data;
A program to run.
(10) A trained model for operating a computer to detect abnormal accumulation from a subject's scintigram,
It is composed of a neural network that has a convolutional layer and a deconvolutional layer, and includes a structure in which the feature map obtained in the convolutional layer is input to the deconvolutional layer. A trained model that is trained using correct labels of abnormal accumulations as training data and causes a computer to function to detect abnormal accumulations from scintigrams of subjects input to the neural network.
(11) A storage unit that stores a trained model of a neural network trained by the learning device according to any one of (2) to (7);
an input section for inputting a scintigram of a subject;
a patch image creation unit that creates a patch image from the scintigram;
an inference unit that inputs the patch image to an input layer of the learned model read from the storage unit and calculates an area of abnormal accumulation included in the patch image;
an output unit that outputs data indicating the abnormal accumulation area;
An abnormality accumulation detection device comprising:

This application claims priority based on Japanese Patent Application No. 2018-096186 filed on May 18, 2018, and the entire disclosure thereof is incorporated herein.

Claims (17)

A learning device that generates a neural network model for detecting bone metastasis areas from a subject's scintigram,
an input unit for inputting scintigrams of a plurality of subjects and correct labels of bone metastasis regions and non-bone metastasis regions in each scintigram as training data;
a learning unit that uses the training data to learn a neural network model used for detecting bone metastasis regions in bone scintigrams;
Equipped with
The neural network has a structure that combines a first network part and a second network part ,
The input unit inputs scintigrams taken from the front and back and their correct labels for each subject ,
The learning unit inputs a scintigram photographed from the front of the subject to an input layer of the first network portion, and inputs a scintigram photographed from the rear of the subject to an input layer of the second network portion. A learning device that performs learning . comprising a patch image creation unit that creates a patch image by cutting out a region in which the bones of the subject are shown from the scintigrams of the plurality of subjects;
The learning device according to claim 1, wherein the learning unit performs learning using the patch image and its corresponding correct label as teacher data. The patch image creation unit scans a window of a predetermined size on the scintigram of the subject, and when bones of the subject are included in the window, cuts out an area of the window as the patch image. 2. The learning device according to 2. A teacher data analysis unit that calculates the composition ratio of a patch image that includes a bone metastasis area or a non-bone metastasis area and a patch image that does not include either a bone metastasis area or a non-bone metastasis area in the patch images created by the patch image creation unit. The learning device according to claim 2 or 3, comprising: A patch image that does not include either a bone metastasis area or a non-bone metastasis area is selected from the patch images created by the patch image creation unit so that the composition ratio determined by the teacher data analysis unit is within a predetermined range. The learning device according to claim 4, further comprising a patch image selection unit that thins out the patch images. The learning device according to any one of claims 2 to 5, further comprising a patch image reversing unit that horizontally or vertically inverts at least part of the patch images created by the patch image creating unit. 7. The learning device according to claim 1, wherein the neural network has an encoder-decoder structure and includes a structure for inputting a feature map obtained from the encoder structure to the decoder structure. The learning device according to any one of claims 1 to 6 , wherein the first network part and the second network part have an encoder-decoder structure . the first network part and the second network part have an Encoder-D encoder structure;
The learning unit inputs a first patch image cut out from a scintigram photographed from the front of the subject into an input layer of the first network part, and inputs a first patch image cut out from a scintigram photographed from the front of the subject into an input layer of the second network part. performing learning by inputting a second patch image corresponding to the first patch image cut out from a scintigram taken from
The learning device according to any one of claims 2 to 6. The non-bone metastatic area includes a non-malignant lesion area,
The input unit receives scintigrams of a plurality of subjects with correct labels for bone metastasis regions and non-malignant lesion regions as training data;
The learning unit uses the teacher data to learn a neural network model for detecting each of a bone metastasis region and a non-malignant lesion region.
A learning device according to any one of claims 1 to 9. A learning method for generating a neural network model for detecting a bone metastasis region from a subject's scintigram, the method comprising:
inputting scintigrams of a plurality of subjects and correct labels of bone metastasis areas and non-bone metastasis areas in each scintigram as training data;
a step of learning a neural network model used for detecting a bone metastasis region in a bone scintigram using the training data;
Equipped with
The neural network has a structure that combines a first network part and a second network part ,
In the inputting step, for each subject, input the scintigram taken from the front and back and its correct answer label,
In the step of performing the learning, a scintigram photographed from the front of the subject is input to the input layer of the first network portion, and a scintigram photographed from the rear of the subject is input to the input layer of the second network portion. A learning method where you learn by inputting information . A program for generating a neural network model for detecting bone metastatic areas from a subject's scintigram, the program comprising:
inputting scintigrams of a plurality of subjects and correct labels of bone metastasis areas and non-bone metastasis areas in each scintigram as training data;
a step of learning a neural network model used for detecting a bone metastasis region in a bone scintigram using the training data;
run the
The neural network has a structure that combines a first network part and a second network part ,
The inputting step includes inputting scintigrams taken from the front and back and their correct labels for each subject ;
The step of performing the learning includes inputting a scintigram photographed from the front of the subject into the input layer of the first network part, and inputting a scintigram photographed from the rear of the subject into the input layer of the second network part. a step of learning by
A program to run. This is a trained model for operating a computer to detect bone metastasis areas from scintigrams of subjects, and it has a convolution layer and a deconvolution layer, and the feature map obtained from the convolution layer is transferred to the deconvolution layer. It is composed of a neural network that includes a structure input to The neural network is trained by inputting two scintigrams with one of the scintigrams flipped horizontally, and the neural network is designed to detect bone metastasis areas from the subject's scintigrams input to the neural network. , a trained model that makes the computer work. A storage unit storing a trained model of a neural network trained by the learning device according to any one of claims 2 to 10;
an input section for inputting a scintigram of a subject;
a patch image creation unit that creates a patch image from the scintigram;
an inference unit that inputs the patch image to an input layer of the trained model read from the storage unit and calculates a bone metastasis area included in the patch image;
an output unit that outputs data indicating the bone metastasis area;
A bone metastasis detection device comprising: A program for detecting bone metastasis areas from a subject's scintigram, the program comprising:
inputting a scintigram of the subject;
creating a patch image from the scintigram;
The learned model is read from a storage unit that stores the learned model of the neural network trained by the learning device according to any one of claims 2 to 10, and the patch image is added to the input layer of the learned model. inputting and determining a bone metastasis area included in the patch image;
outputting data indicating the bone metastasis area;
A program to run. A program for detecting bone metastasis areas from a subject's scintigram, the program comprising:
a step of inputting two scintigrams taken from the front and back of the subject;
horizontally flipping one of the two scintigrams;
The learned model is read from the storage unit that stores the learned model generated in advance by learning using teacher data, and two scintigrams are input to the input layer of the learned model, and the scintigram is determining the included bone metastasis area;
outputting data indicating the bone metastasis area;
A program to run. Neural network model for detecting bone metastasis areas from subject's scintigrams A learning device that generates a
Scintigrams of multiple subjects and bone metastatic areas and non-bone metastatic areas in each scintigram an input section for inputting correct answer labels as training data;
Neural network used to detect bone metastasis areas in bone scintigrams using the above training data A learning department that learns work models;
From the scintigrams of the multiple subjects, the area where the bones of the subjects are shown is cut out and packed. a patch image creation section that creates a patch image;
In the patch image created by the patch image creation section, bone metastasis area or non-bone metastasis area Composition ratio of patch images that include bone metastasis areas and patch images that do not include bone metastasis areas or non-bone metastasis areas The teacher data analysis department, which seeks
Equipped with
The learning unit uses the patch image and its corresponding correct answer label as training data. A learning device that allows you to learn using

Images