JP2021111076A - Diagnostic device using ai, diagnostic system, and program - Google Patents

Abstract

To provide a diagnostic device, a diagnostic system, and a program, which accurately determine a specific symptom on the basis of an image.SOLUTION: In a diagnostic system, the function configuration of a diagnostic device comprises: a first medical image input section for inputting a first medical image obtained by projecting a first region of a patient; a second medical image input section for inputting a second medical image obtained by projecting a second region which is a region different from the first region; a learning section for learning a network and building a learning model on the basis of the first medical image and the second medical image; and a determination section for determining a symptom on the basis of an execution medical image obtained by projecting the first region and the second region using the learning model.SELECTED DRAWING: Figure 13

Description

The present invention relates to diagnostic devices, diagnostic systems, and programs that use AI.

Conventionally, there is known a method of performing diagnosis based on an image by AI (Artificial Intelligence) such as CNN (Convolutional Neural Network).

FIG. 1 is a diagram showing a model of a symptomatology classifier architecture in the prior art. For example, as shown in the figure, a method of applying a CNN having a network structure as shown in the figure to a diagnostic image generated by MR (Magnetic Resonance) or PET (Positron Emission Tomography) or the like is known. Has been done.

FIG. 2 is a diagram showing a model of a symptomatology classifier architecture using MR images and PET images. Further, as shown in the figure, first, CNN, which is a network structure of VGG (Visual Geometry Group) -11, is separately applied to MR images and PET images, and the results are extracted. Then, each result is concatenated. As described above, CNNs and the like for inputting a plurality of diagnostic images are known (for example, Non-Patent Document 1 and the like).

Original Research ARTICLE Front. Neurosci. , 31 May 2019, "Diagnosis of Alzheimer's Disease via Multi-Modality 3D Convolutional Neural Network", [online], May 31, 2019, [online], May 31, 2019, [2019] /www.frontiersin.org/articles/10.3389/fnins.2019.00509/full>

However, in the conventional method, in many cases, the diagnosis is made only by projecting an image of only one type of organ or a part around the organ (hereinafter, simply referred to as "site") in the whole body of the patient. Therefore, specific symptoms such as vasculitis (including "large vasculitis", "Takayasu's arteritis", "vasculitis syndrome" or "systemic vasculitis"; hereinafter referred to as "large vasculitis") or leukoencephalopathy. In many cases, the accuracy of judgment is poor.

Therefore, one embodiment of the present invention has been made to solve the above-mentioned problems, and a diagnostic device, a diagnostic system, and a diagnostic system capable of accurately determining a specific symptomatism based on an image. , The purpose is to provide a program.

The diagnostic apparatus according to the embodiment of the present invention is
A first medical image input unit that inputs a first medical image that projects the first part of the patient,
A second medical image input unit that inputs a second medical image that projects a second part that is a part different from the first part, and a second medical image input unit.
A learning unit that learns a network and builds a learning model based on the first medical image and the second medical image.
It includes a judgment unit that makes a judgment about a symptom based on an image for an execution doctor that projects the first part and the second part using the learning model.

It is possible to provide a diagnostic device, a diagnostic system, and a program that can accurately judge a specific symptomatism based on an image.

It is a figure which models and shows the symptomatology classifier architecture. It is a figure which models and shows the classifier architecture of the symptomatology using MR image and PET image. It is a figure which shows the whole configuration example of a diagnostic system. It is a figure which shows the hardware configuration example of a diagnostic apparatus. It is a figure which shows the whole processing example. It is a figure which shows the generation example of the 1st medical image and the 2nd medical image. It is a figure which shows the example of a network structure. It is a figure which shows the example of the convolution processing. It is a figure which shows the example of the pooling process. It is a figure which shows the example of the full combination processing. It is a figure which shows the example of the dropout process. It is a figure which shows the example of the network used for execution processing. It is a figure which shows the functional structure example. It is a figure which shows the whole configuration example of the diagnostic system using a slice medical image group. It is a figure which shows the example of the network structure for white matter encephalopathy. It is a figure which shows the modification of the 1st part.

Hereinafter, each embodiment will be described with reference to the accompanying drawings. In the present specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals, so that duplicate description will be omitted.

<Overall configuration example>
FIG. 3 is a diagram showing an overall configuration example of the diagnostic system. For example, the diagnostic system 2 includes a medical image generation system 21 and a PC (Personal Computer) (hereinafter, simply referred to as “PC20”) which is an example of a diagnostic device.

The medical image generation system 21 is, for example, an X-ray apparatus or an MRI. The medical image generation system 21 may be any device that can generate a medical image.

A medical image is an image obtained by photographing or transmitting and projecting an internal organ or the like in the human body. Therefore, medical images are generated by a method using radiation, MRI (Magnetic Resonance Imaging), PET (Positron Emission Tomography), CT (Computed Tomography), etc. It is an image. Further, the medical image may be referred to as a radiographic image, a diagnostic image, a medical image, or the like.

The medical image is, for example, an image generated by angiography or the like, an image generated by using ultrasonic waves, an image generated by using thermography, an image generated by using an endoscope, and the like. Alternatively, an image generated by a combination of these and the above method may be used.

The medical image may be a monochrome image or a color image.

Hereinafter, the medical image and the feature map generated based on the medical image may be simply referred to as an "image".

The medical image generation system 21 may have a configuration having a PACS (Picture Archiving and Communication Systems).

Further, the medical image generation system 21 is not an essential configuration in the diagnostic system 2. That is, the medical image may be generated by another external device that is not connected to the PC 20. In this case, the medical image is input to the PC 20 via, for example, a communication line or a recording medium.

The PC 20 is, for example, the following information processing device.

FIG. 4 is a diagram showing a hardware configuration example of the diagnostic device. For example, the diagnostic device 20 has a hardware configuration including a CPU (Central Processing Unit, hereinafter simply referred to as “CPU 20H1”), a storage device 20H2, a communication device 20H3, an input device 20H4, an output device 20H5, an interface 20H6, and the like. ..

The CPU 20H1 is an example of an arithmetic unit and a control unit.

The storage device 20H2 is a main storage device such as a so-called memory. Further, the storage device 20H2 may further have an auxiliary storage device such as a hard disk or an SSD (Solid State Drive).

The communication device 20H3 communicates with the external device wirelessly or by wire. For example, the communication device 20H3 is a connector, an antenna, or the like.

The input device 20H4 is a device for inputting an operation by the user. For example, the input device 20H4 is a keyboard, a mouse, or the like.

The output device 20H5 is a device that outputs a processing result or the like to the user. For example, the output device 20H5 is a display or the like.

The interface 20H6 connects to an external device by wire or wirelessly to input / output data. For example, the interface 20H6 is a connector or the like.

The diagnostic device is not limited to the hardware configuration as shown in the figure. For example, the diagnostic device may have a hardware configuration further including an arithmetic device, a control device, a storage device, an input device, an output device, an auxiliary device, and the like.

When the first medical image IMG1 and the second medical image IMG2, which are learning data (sometimes referred to as "teacher data"), are input to the PC 20 as described above, the PC 20 is the one medical image. Network learning is performed based on IMG1 and the second medical image IMG2. Hereinafter, the network in which the learning process is completed is referred to as "learning model 22". In this way, the PC 20 builds the learning model 22 in advance by the learning process.

It is desirable that the network and learning model 22 have a CNN network structure. Hereinafter, an example in which the network and learning model 22 is a CNN will be described. However, the network and the learning model 22 may be an RNN (Recurrent Neural Network) or the like.

After the learning model 22 is constructed, when a medical image to be determined for symptomatology (hereinafter referred to as "execution medical image IMG3") is input, the PC 20 has a specific symptomatology based on the learning model 22. A judgment result 3 indicating whether or not the judgment can be made can be output.

The judgment result 3 does not have to be binary data indicating whether or not the patient 1 has a specific symptom. For example, the judgment result 3 may numerically indicate the probability of having a specific symptomatism. Further, the judgment result 3 may indicate the basis of the judgment (an image having a great influence on the judgment, a part of the image, or the like). In addition, the judgment result 3 may suggest not only the present situation but also the possibility of becoming a symptom or a tendency in the future.

<Overall processing example>
For example, the diagnostic device performs the following overall processing to determine a specific symptom.

FIG. 5 is a diagram showing an example of overall processing. Hereinafter, as shown in the figure, it is assumed that the "learning process" of steps S1 to S4 and the like is executed in advance. Then, it is assumed that the learning process is an overall process in which the "execution process" such as step S5 and step S6 is performed after the learning is completed.

<Learning process example>
(Example of inputting the first medical image)
In step S1, the diagnostic device inputs a first medical image.

(Example of inputting a second medical image)
In step S2, the diagnostic device inputs a second medical image.

For example, step S1 and step S2 are performed as follows.

<Input example of first medical image and second medical image>
Hereinafter, an example will be described in which a specific symptom is defined as "major vasculitis". However, the target symptom may be a symptom other than macroangiitis.

For example, the first medical image IMG1 and the second medical image IMG2 are images generated as follows.

FIG. 6 is a diagram showing an example of generating a first medical image and a second medical image.

For example, the first medical image and the second medical image are the first medical image IMG1 and the second medical image from the medical image showing the whole body of the patient 1 as shown in FIG. 6 (A) (hereinafter referred to as “whole body medical image”). It is generated by cutting out a region to be a medical image IMG2. Specifically, for example, the first medical image IMG1 and the second medical image IMG2 are generated as follows.

FIG. 6B is a diagram showing an example of the first medical image IMG1.

The first medical image IMG1 is a medical image on which a part of the patient's part where a symptom occurs (hereinafter referred to as “first part P1”) is projected. Specifically, as illustrated, in the case of large vasculitis, the first medical image IMG1 is a medical image projected to include a site of the mediastinum.

The mediastinum is the area of the chest surrounded by the left and right lungs, the thoracic spine, and the sternum. Therefore, it is desirable that the first medical image IMG1 is an image including a range surrounded by the left and right lungs, the thoracic vertebrae, and the sternum in the chest. Hereinafter, a case where the first portion P1 is a mediastinum will be described as an example. However, since the first site P1 differs depending on the type of symptom, it is set for each type of symptom.

FIG. 6C is a diagram showing an example of the second medical image IMG2.

The second medical image IMG2 is a reference image for determining the symptom based on the first site P1. The second medical image IMG2 is a medical image obtained by projecting a part of a part different from the first part P1 (hereinafter referred to as "second part P2"). Specifically, as illustrated, in the case of macroangiitis, the second medical image IMG2 is a medical image projected to include a portion of the liver. However, since the second site P2 differs depending on the type of symptom, it is set for each type of symptom.

It is desirable that the first medical image IMG1 is an image having a higher resolution than the second medical image IMG2. That is, it is desirable that the first medical image IMG1 is an image having a larger number of pixels than the second medical image IMG2.

Since the first medical image IMG1 is an image directly showing the first site P1, it is desirable that the first medical image IMG1 is an image in which details can be understood. Therefore, it is desirable that the first medical image IMG1 has at least a higher resolution than the second medical image IMG2 and is an image showing the details of the first portion P1.

Further, as shown in FIG. 6C, it is desirable that the second medical image IMG2 is an image obtained by cutting out a part of the liver instead of the entire liver.

The second medical image IMG2 is an image showing a region including a group of pixels showing the same color, pixels showing the same pixel value, or pixels within the reference value among the pixels showing the liver. Is desirable. In a medical image, the pixel value is often a value indicating lightness or darkness, that is, a so-called luminance value. However, the medical image may be a color image. For example, when projecting using thermography, ultrasound, or an endoscope, the medical image is often a color image. In such a case, the brightness value of the color image (for example, RGB format or the like) may be calculated based on the pixel value of each color. The RGB format value can be converted into a format including "Y" by a calculation such as so-called YUV conversion. The value of "Y" calculated in this way may be used as the pixel value. Alternatively, the pixel value may be used as the pixel value on behalf of a value indicating any one of a plurality of colors constituting the color image.

For example, when image processing such as so-called labeling is performed on a region including a liver, a region in which pixels showing the same color or the same pixel value are continuously extracted is extracted. Specifically, first, a process of comparing the pixel values of a plurality of pixels indicating the liver is performed. For example, among a plurality of pixels indicating the liver, pixels having a difference between the pixel values within a preset reference value are extracted as "pixels exhibiting the same color". Then, when the extracted pixels are adjacent to each other, the pixels are labeled as being in the same group.

In this way, there are cases where a plurality of similar pixel groups can be extracted by labeling or the like. In such a case, it is desirable that the second medical image IMG2 is generated so that the group having the largest number of pixels among the groups is the center of the image. In this way, a medical image that best shows the typical color of the liver can be generated.

However, as shown in FIG. 6C, the second medical image IMG2 does not have to be a medical image showing only the pixels of the same group. That is, the second medical image IMG2 may be a medical image showing a group of pixels showing a typical color or pixel value as the center among the pixels showing the liver, and includes peripheral pixels as a part thereof. It may be a medical image.

Further, it is desirable that the second medical image IMG2 is an image obtained by cutting out the center of an organ, not the entire liver, as in the illustrated second site P2. That is, it is desirable that the second medical image IMG2 is a medical image focused on the inside of the organ so as not to include the outline of the organ.

When the second medical image IMG2 shows the contour of the liver, the shape of the liver or the like tends to be a learning target, a so-called feature amount, based on the second medical image IMG2. On the other hand, when large vasculitis is determined, the contour of the organ, that is, the shape, etc., is often not helpful in diagnosis, and it is desirable that the symptom is determined based on the color or pixel value.

In order to judge large vasculitis, it is possible to judge accurately by comparing the color of the mediastinum with the color of the liver. Therefore, it is desirable that the second medical image IMG2 is an image that magnifies and shows the vicinity of the center of the liver so that the shape of the liver and the like are not learned so much based on the second medical image IMG2. When the second medical image IMG2 showing such a range is used, it becomes easy to adopt a color or a pixel value as a feature amount, and a specific symptomatology can be determined more accurately.

The first medical image IMG1 and the second medical image IMG2 do not have to be input in the order of the first medical image IMG1 and the second medical image IMG2, as in steps S1 and S2. For example, in the first medical image IMG1 and the second medical image IMG2, the second medical image IMG2 may be input first or may be input in parallel. That is, the first medical image IMG1 and the second medical image IMG2 may be input in any order.

Further, in the first medical image IMG1 and the second medical image IMG2, a whole body medical image is input, and each medical image generated by cutting out or enlarging based on the input whole body medical image is the first medical image IMG1. And the second medical image IMG2 may be input. On the other hand, the first medical image IMG1 and the second medical image IMG2 may be cut out from the whole body medical image in advance by another device and input.

Alternatively, the first medical image IMG1 and the second medical image IMG2 are not generated from the whole-body medical image, but are separately generated by the medical image generation system 21 or the like focusing on the first part P1 and the second part P2. It may be a medical image. Hereinafter, a case where the first medical image IMG1 has a resolution of “128 × 128” (pixel) and the second medical image IMG2 has a resolution of “32 × 32” (pixel) will be described as an example. However, the first medical image IMG1 and the second medical image IMG2 may have resolutions other than the above.

(Example of generating merged image)
In step S3, the diagnostic apparatus merges the first medical image and the image based on the second medical image to generate the merged image.

(Learning example based on merged images)
In step S4, the diagnostic apparatus learns based on the merged image.

For example, the above learning process is performed in a network having the following structure.

FIG. 7 is a diagram showing an example of a network structure. First, as described above, when steps S1 and S2 are performed, the first medical image IMG1 and the second medical image IMG2 are input to the PC 20.

For example, it is desirable that the network NW has a network structure of CNN as shown in the figure.

In the learning process, the network NW preferably has a network structure having a hidden layer with respect to the first medical image IMG1 and the second medical image IMG2 in order to generate the merged image IMG4.

Hereinafter, the hidden layer for the first medical image IMG1 is referred to as "first hidden layer L1". On the other hand, the hidden layer for the second medical image IMG2 is called "second hidden layer L2".

The first hidden layer L1 is, for example, a layer composed of a combination of a convolution layer and a pooling layer. Specifically, the first convolutional layer L1 is configured in the order of, for example, the 11th convolutional layer LC11, the 11th convolutional layer LP11, the 12th convolutional layer LC12, the 12th convolutional layer LP12, and the 13th convolutional layer LC13. Such a first hidden layer L1 generates a first feature map MP1 based on the first medical image IMG1.

In this example, the eleventh convolution layer LC11 generates 64 "128 × 128" feature maps by the convolution process. The details of the convolution process will be described later.

In this example, the eleventh pooling layer generates a “64 × 64” feature map by performing a pooling process on the feature map generated by the eleventh convolution layer LC11. The details of the pooling process will be described later.

In this example, the 12th convolution layer LC12 generates 128 “64 × 64” feature maps by the convolution process.

In this example, the twelfth pooling layer LP12 generates a “32 × 32” feature map by performing a pooling process on the feature map generated by the twelfth convolution layer LC12.

The thirteenth convolution layer LC13 generates 128 “32 × 32” feature maps by the convolution process.

For example, when the first medical image IMG1 is processed based on the first hidden layer L1 as described above, the first feature map MP1 can be generated. On the other hand, apart from the process based on the first hidden layer L1, the process based on the second hidden layer L2 is performed. In addition, each process may be performed in parallel or may be performed in order.

The second hidden layer L2 is, for example, a layer composed of a convolution layer. Specifically, the second hidden layer L2 is composed of a second convolution layer LC2. Such a second hidden layer L2 generates a second feature map MP2 based on the second medical image IMG2.

The second convolution layer LC2 generates 128 "32 × 32" feature maps by the convolution process.

Next, in the network NW, the merged image IMG4 is generated based on the first feature map MP1 and the second feature map MP2.

The merged image IMG4 is an image generated by a merge process for the first feature map MP1 and the second feature map MP2.

As described above, the merged image IMG4 generated by the merge process shows the result of considering the typical color or pixel value indicated by the second medical image IMG2 in determining the symptom based on the first medical image IMG1. ..

In the example of the network structure shown in FIG. 7, first, the first feature map MP1 and the second feature map MP2 are generated based on the first medical image IMG1 and the second medical image IMG2 by the processing based on the hidden layer in advance. Will be done. After that, the merged image IMG4 is generated based on the first feature map MP1 and the second feature map MP2, which is a network structure. Even with such a network structure, the first feature map MP1 and the second feature map MP2 are images showing the features of the first medical image IMG1 and the second medical image IMG2. Therefore, in the following description, in order to simplify the description, even when the first feature map MP1 and the second feature map MP2 are merged as in the example of the network structure shown in FIG. The process will be described as processing for the first medical image IMG1 and the second medical image IMG2. The network NW may have a structure in which the first hidden layer L1, the second hidden layer L2, or both are not provided.

For example, the merge process is a process of adding, subtracting, or multiplying pixel values.

Specifically, when the merged image IMG4 is generated by a merge process that calculates the difference between the pixel values of the pixels of the first medical image IMG1 and the second medical image IMG2, the merged image IMG4 is used for the first medical use. It is an image showing the difference between the pixel values of the feature map generated based on the image IMG1 and the second medical image IMG2.

Therefore, the merged image IMG4 can be an image that shows the difference in brightness of the first medical image IMG1 with respect to the color indicated by the second medical image IMG2. As described above, it is desirable that the merged image IMG4 is an image that determines the first medical image IMG1 based on the color indicated by the second medical image IMG2.

If there is an auxiliary image showing a reference color such as the second medical image IMG2, the symptomatology is more accurate than the case where the judgment is made by focusing on only the first medical image IMG1, that is, only one part. I can judge well. Therefore, it is desirable that the merged image IMG4 is generated so as to be based on the color or the like indicated by the second medical image IMG2.

In the merging process, for example, when the pixel value of the second medical image IMG2 is subtracted from the pixel value of the first medical image IMG1, the pixel value of the second medical image IMG2 is multiplied by a preset coefficient and then subtracted. May be done. That is, in performing the merge process, the pixel value of either one of the medical images may be weighted.

Based on the merged image IMG4 generated by the merge process as described above, for example, the process is performed based on the third hidden layer L3 (step S4).

The third hidden layer L3 is, for example, the 31st pooling layer LP31, the 31st convolution layer LC31, the 32nd pooling layer LP32, the 32nd convolution layer LC32, the 33rd pooling layer LP33, the 33rd convolution layer LC33, and the 34th pooling. It is composed of layer LP34 and the like. Further, the third hidden layer L3 has, for example, a normalized layer LN3, a first fully connected layer LF31, a first dropout layer LD31, a second fully connected layer LF32, a second dropout layer LD32, and the like. be.

In this example, the 31st pooling layer LP31 generates a “16 × 16” feature map by performing a pooling process on the merged image IMG4.

In this example, the 31st convolution layer LC31 performs a convolution process on the feature map generated by the 31st pooling layer LP31 to generate 256 "16 × 16" feature maps.

In this example, the 32nd pooling layer LP32 generates an “8 × 8” feature map by performing a pooling process on the feature map generated by the 31st convolution layer LC31.

In this example, the 32nd convolution layer LC32 performs a convolution process on the feature map generated by the 32nd pooling layer LP32 to generate 256 “8 × 8” feature maps.

In this example, the 33rd pooling layer LP33 generates a “4 × 4” feature map by performing a pooling process on the feature map generated by the 32nd convolution layer LC32.

In this example, the 33rd convolution layer LC33 performs a convolution process on the feature map generated by the 33rd pooling layer LP33 to generate 256 “4 × 4” feature maps.

In this example, the 34th pooling layer LP34 generates a “1 × 1” feature map by performing a pooling process on the feature map generated by the 33rd convolution layer LC33.

The normalization layer LN3 normalizes the feature map generated by the 34th pooling layer LP34. The details of the normalization process will be described later.

The first fully-bonded layer LF31 and the second fully-bonded layer LF32 are fully bonded. The details of the full combination process will be described later.

The first dropout layer LD31 and the second dropout layer LD32 perform dropout processing. The details of the dropout process will be described later.

In the network NW configuration, the number of pixels (which is also the vertical and horizontal size of the image) and the number of feature maps to be generated are not limited to the above examples. That is, it may be a network structure in which the number of pixels and the number of feature maps different from the above example are generated.

Further, the layer structure and the like are not limited to the structure of the example shown above. That is, it may have a layer structure having layers other than those shown above. Further, the layer structure may be changed by a learning process or the like.

In the learning process, the learning model may be evaluated. For example, in order to improve the accuracy, adjustments such as changing the projection range of at least one of the first medical image IMG1 and the second medical image IMG2 may be made.

<Example of convolution processing> (Convolution)
The convolution process is a process of extracting the features of the input image or feature map.

In the convolution process, first, based on the filter (sometimes called "kernel"), the filter coefficient (sometimes called "weight") that constitutes the filter is multiplied by the pixel value, and each pixel value is multiplied. Is processed to calculate the multiplication result. Next, a process is performed in which the respective multiplication results are summed to calculate each value constituting the feature map. As described above, the convolution process is a process of generating a feature map by a so-called filter process.

FIG. 8 is a diagram showing an example of the convolution process. For example, it is assumed that an image as shown in FIG. 8A (medical image, feature map, etc., hereinafter referred to as “first input image IN1”) is the target of the convolution process.

As shown in FIG. 8A, it is assumed that the first input image IN1 is an image having M rows horizontally and N columns vertically, that is, an image having M × N pixels. Further, in the figure, each pixel value is indicated as "A00" (in the figure, the last digit indicates a row and the last second column indicates a column).

FIG. 8B is a diagram showing the first filter F1. The first filter F1 is an example of a size of "3 x 3". However, the size of the filter may be other than "3x3". Hereinafter, in the figure, the filter coefficients constituting the first filter F1 are shown in the same format as the pixel values of the first input image IN1.

In the convolution process, for example, the calculation is performed by multiplying the upper left of the first input image IN1, that is, "A11" by the filter coefficient. Specifically, multiplication results are calculated as "A00 x B00", "A01 x B01", ..., And when these multiplication results are summed up, it is one of the feature values constituting the feature map. "C00" is generated.

FIG. 8C is a diagram showing a first output feature map OUT1. Hereinafter, in the figure, the feature values constituting the first output feature map OUT1 showing the convolution processing result are shown in the same format as the pixel values of the first input image IN1. Then, as shown in the figure, when the position where the first filter F1 is applied is changed, the feature values in the first output feature map OUT1 are generated as “C01”, “C02”, and so on.

Then, the filter coefficient is changed by the learning process. In this way, the filter coefficient is changed by so-called error back propagation, and the filter used for the convolution process is generated by the learning process. Further, the processing result of the convolution processing, that is, the feature map will be a different processing result if at least one of the filter coefficient of the filter used for the convolution processing or the input image is different.

When such a convolution process is performed, for example, even if the pixels showing the features in the input medical image or feature map are shifted vertically and horizontally from other images, or even if there is a difference in rotation or the like, the same applies. Features can be extracted. That is, so-called migration invariance can be ensured like the local receptive field of humans.

In the convolution process, pre-processing, so-called padding, in which pixels having pixel values set in advance are added to the periphery of the image to be processed may be performed. When padding is done, the convolution process prevents the size from shrinking. Further, when padding is performed, filtering can be performed up to the end.

Further, in the filter processing in the convolution process, the interval may be not one pixel but two or more pixels. That is, in the convolution process, so-called stride may be performed. Once stride is done, a small feature map can be generated.

<Example of pooling process> (Polling)
The pooling process is a process of making the position of the feature point robust.

The pooling process is a process of generating a feature map in which the input image is downsized by performing a preset operation. The pooling process may be called a downsampling process, a subsampling process, or the like.

FIG. 9 is a diagram showing an example of the pooling process. For example, it is assumed that an image as shown in FIG. 9A (medical image, feature map, etc., hereinafter referred to as “second input image IN2”) is the target of the pooling process.

The pooling process is, for example, a process of extracting the value having the largest pixel value among the pixels in the set range (sometimes called “max sampling” or the like). In the illustrated example, the pooling process is a process of extracting the maximum value for each range of “2 × 2”.

As shown in FIG. 9A, for example, at the upper left position of the second input image IN2, in the range of “2 × 2”, “3”, “5”, “4”, and “-1”. When there is a pixel having a pixel value of "5", the pixel having the pixel value of "5", which has the largest pixel value among these pixel values, is extracted by the pooling process. When such processing is performed, "5", which is one of the feature values constituting the output feature map, is generated.

FIG. 9B is a diagram showing a second output feature map OUT2. Hereinafter, the feature values constituting the second output feature map OUT2 showing the pooling processing result are shown in the same format as the pixel values of the first input image IN1. Then, as shown in the figure, when the range to be processed is changed, the feature values in the second output feature map OUT2 are generated as “5”, “5”, and so on.

The pooling process is not limited to the process of extracting the maximum value. That is, the pooling process may be a process of determining a representative value based on the pixel value in the set range. For example, the pooling process may be a process of extracting statistical values such as an average value or a median value.

Further, the pooling process may be padded.

<Example of normalization processing> (Normalization)
The normalization process is a process of aligning the variance and the mean value. For example, the normalization process is a process of changing the data so that the variance of the data is “1” and the average value is “0”. The normalization process may include a process such as setting the maximum value, the minimum value, or both of them to a value within the set value.

When the normalization process is performed in this way, the learning coefficient can be increased. Further, when the normalization process is performed, the values can be made to converge even if there are multiple layers.

Further, the normalization process may be performed for each layer. That is, the normalization process may be so-called batch normalization or the like.

<Example of full connection processing> (Full connected)
The full combination process is a process of dropping the feature map generated by the previously performed convolution process or the like into the output layer.

FIG. 10 is a diagram showing an example of a fully combined process. For example, as shown in the figure, the fully combined process is a process of associating each feature map with the output layer LO when a plurality of "feature maps" are generated by the process performed before the fully combined process. ..

The illustrated example is an example in which the output layer LO is set to two final output formats such as “Yes” or “No”. Therefore, the full coupling process is a process of determining which of the output formats is preset in the output layer LO by an activation function (so-called "softmax" or the like) or the like based on each feature map. Is.

Note that each feature map may be weighted.

Further, a network structure in which there is no output layer after the fully connected layer, that is, a configuration in which the output is a feature map may be used. Specifically, a so-called GAP (Global Average Pooling) layer or the like may be adopted. In this way, the GAP layer may be used to reduce the number of parameters.

<Example of dropout processing> (Dropout)
FIG. 11 is a diagram showing an example of dropout processing. For example, the dropout layer DP that performs the dropout process is set before and after the fully connected layer LF as shown in the figure.

In the figure, a circle (○) indicates a perceptron (sometimes referred to as a “node” or the like). In addition, in the figure, the front and rear perceptrons are shown in association with each other by connecting them with a line.

The dropout process is a process of losing the association with a predetermined probability in the learning process. In this figure, the dropout layer DP is a line connecting the front fully connected layer LF and the rear fully connected layer LF, and some of the lines passing through the dropout layer DP are predetermined. It is a process to erase with probability.

The dropout process is a process in which, for example, output and error backpropagation are performed in the learning process without using nodes at random. Dropout processing can reduce the dependencies between nodes. Therefore, overfitting and the like can be prevented.

<Execution processing example>
(Example of inputting image for execution doctor)
In step S5, the diagnostic device inputs an image for the performing doctor.

(Example of judgment based on medical image)
In step S6, the diagnostic apparatus makes a judgment based on the image for the executing doctor.

For example, the above-mentioned execution process is executed as follows based on the network generated by the learning process performed in advance.

FIG. 12 is a diagram showing an example of a network used for execution processing. For example, the diagnostic device learns the network by learning processing performed by inputting learning data in advance.

After the learning process is executed, the diagnostic apparatus inputs the execution medical image IMG3 to the learning model 22 and performs the execution process.

The performing medical image IMG3 is, for example, a whole-body medical image or the like. However, the medical image IMG3 for execution may be a medical image showing a range including both the first site P1 and the second site P2, and may not be a medical image including the whole body.

When the execution process is performed, the judgment result 3 for judging whether or not the symptom is a symptom such as large vasculitis is output. The determination result 3 is output by, for example, an output device so that a doctor or the like can see it.

For example, the doctor makes a final diagnosis by referring to the judgment result 3 output in this way for diagnosing the symptom.

The execution medical image IMG3 may be a combination of two medical images, a first medical image IMG1 and a second medical image IMG2, as in the learning process.

Further, the diagnostic apparatus may input parameters in addition to the images such as the first medical image IMG1 and the second medical image IMG2.

The second site P2 does not have to be one site as shown in the above example. That is, the second medical image IMG2 may be an image showing two or more types of parts. Further, the second medical image IMG2 may be two or more types of images.

Further, in the above example, the position where the merged image IMG4 is generated may be a position other than the position shown in the drawing.

The gradation or contrast of the first medical image IMG1, the second medical image IMG2, the executing medical image IMG3, or a combination of two or more of these images may be adjusted at the input.

<Function configuration example>
FIG. 13 is a diagram showing a functional configuration example. For example, the diagnostic device or diagnostic system has a functional configuration including a first medical image input unit FN1, a second medical image input unit FN2, a learning unit FN3, an execution image input unit FN4, a determination unit FN5, and the like.

The first medical image input unit FN1 performs a first medical image input procedure for inputting a first medical image IMG1 that is a projection of the patient's first site P1. For example, the first medical image input unit FN1 is realized by the communication device 20H3, the interface 20H6, or the like.

The second medical image input unit FN2 performs a second medical image input procedure for inputting a second medical image IMG2 that is a projection of the second portion P2. For example, the second medical image input unit FN2 is realized by the communication device 20H3, the interface 20H6, or the like.

The learning unit FN3 learns the network NW based on the first medical image IMG1 and the second medical image IMG2, and performs a learning procedure for constructing the learning model 22. For example, the learning unit FN3 is realized by the CPU 20H1 or the like.

The execution image input unit FN4 performs an execution doctor image input procedure for inputting the execution doctor image IMG3. For example, the execution image input unit FN4 is realized by the communication device 20H3, the interface 20H6, or the like.

The judgment unit FN5 uses the learning model 22 to perform a judgment procedure for judging a symptom based on the image IMG3 for an executing doctor. For example, the determination unit FN5 is realized by the CPU 20H1 or the like.

The first medical image input unit FN1, the second medical image input unit FN2, and the learning unit FN3 may exist when the learning process is performed. Therefore, after the learning process is completed and the learning model 22 is constructed, that is, when the execution process is performed, the diagnostic device or the diagnostic system uses the first medical image input unit FN1 and the second medical image input unit FN2. , And a functional configuration without the learning unit FN3 may be used.

Similarly, the execution image input unit FN4 may exist when the execution process is performed after the learning model 22 is constructed. Therefore, before the learning process is completed, that is, when the learning process is performed, the diagnostic device or the diagnostic system may have a functional configuration without the execution image input unit FN4.

Therefore, the learning process and the execution process may be performed separately by different devices.

As shown in FIG. 7, the diagnostic device or the diagnostic system preferably has a functional configuration further including a processing unit that performs a convolution process, a pooling process, and the like based on the hidden layer.

With the above functional configuration, the diagnostic device or diagnostic system can accurately determine cases based on images even for rare symptoms such as large vasculitis, in which the number of cases is several hundred or less. can.

Rare symptoms are, for example, symptoms designated as intractable diseases by the "Act on Medical Care for Patients with Intractable Diseases" (Act No. 50 of 2006), the so-called Intractable Diseases Act (hereinafter referred to as the "Intractable Diseases Act"). be. In the Act on Medical Care for Patients, macroangiitis is a symptom designated by the name of Takayasu's arteritis. Therefore, in this embodiment, Takayasu's arteritis in the Act on Medical Care for Patients may be targeted.

<Examples of specific target symptoms>
Specific symptoms are those with few cases. For learning in AI and the like, it is often desirable that there are tens of thousands or more images as learning data. On the other hand, symptoms such as large vasculitis are symptoms that do not occur so much. In such a symptom, it is often difficult to prepare an image of a case so that it can be sufficiently learned by machine learning or the like.

To prepare the training data, there is a method of processing one image in order to increase the number of training data. For example, there is a method in which the original image is rotated, inverted, deformed, moved, or processed such as a combination thereof to generate another image and increase the number of images. This is a so-called "Data Augmentation" method.

In the judgment based on the medical image, the accuracy is often not improved in the image generated by a method such as data expansion. For example, if the image is generated by data expansion, the format is far from the image generally used for diagnosis by doctors and the like, which may hinder learning.

In addition, images of premised cases (for example, images that serve as examples described in medical books and the like) often show similar shapes and the like of organs in which symptoms have occurred. Therefore, when such an image is used as learning data, it may be learned that only the shape or the like is learned and the judgment is made only by the shape or the like. Therefore, for a specific symptom, the accuracy may not be improved even if the training data is increased and trained by a method such as data expansion.

In addition, learning models specialized for medical images are often not published. In particular, there are many cases where a learning model using medical images related to radiology as learning data is not open to the public. In many cases, a learning model using a general image as training data cannot achieve high accuracy. For example, in a learning model trained using images provided by ImageNet, FMD (Flickr Material Database), The KTH-TIPS and KTH-TIPS2 image data, etc. as training data, the accuracy may not be improved. Therefore, in a process in which a medical image is input and a specific symptomatology is determined, the accuracy may not be improved in so-called transfer learning, which uses a network trained with a general image.

The target symptom determined by the diagnostic device may be leukoencephalopathy.

FIG. 14 is a diagram showing an overall configuration example of a diagnostic system using a slice medical image group. For example, if the particular symptomatology is leukoencephalopathy, the medical image generation system 21 produces the sliced medical image group IMG10.

The sliced medical image group IMG10 is, for example, a plurality of medical images generated from the crown to the nostril of patient 1 as a target range. Specifically, each medical image constituting the sliced medical image group IMG10 is, for example, a medical image generated in a so-called slice shape projected at regular intervals in a target range. Therefore, each medical image constituting the sliced medical image group IMG10 is an image showing a portion shifted at a fixed interval.

The fixed interval is set in advance and is determined by the specifications of the medical image generation system 21 and the like.

For example, two or more arbitrarily selected medical images in the slice medical image group IMG10 generated in this way may be the first medical image IMG1 and the second medical image IMG2, that is, training data. .. Hereinafter, as in the case of large vasculitis, the whole treatment as shown in FIG. 5 may be performed.

When the specific case is leukoencephalopathy, the network structure may be, for example, the following structure.

FIG. 15 is a diagram showing an example of a network structure for leukoencephalopathy. For example, when the medical images constituting the sliced medical image group IMG10 are learning data and execution data, the network structure as shown in the figure may be used.

As described above, the diagnostic device or the diagnostic system is configured to input a plurality of medical images showing different parts and perform learning processing and execution processing.

<Modification example>
The first portion may include a portion other than the mediastinum. For example, the first portion may further include a range up to the neck, etc., in addition to the mediastinum. Specifically, the first portion may include the following ranges.

FIG. 16 is a diagram showing a modified example of the first portion. Specifically, the first site includes aortic arch branch blood vessel P11, ascending aorta P12, aortic arch P13, thoracic descending aorta P14 (sometimes simply referred to as "descending aorta"), abdominal aorta P15, and. Renal artery P16 (right renal artery and left renal artery) and the like may be included.

The aortic arch branch vessel P11 is, for example, the brachiocephalic artery, the right subclavian artery, the right common carotid artery, the left common carotid artery, the left subclavian artery, or a part thereof. Further, ascending aorta P12, aortic arch P13, thoracic descending aorta P14, abdominal aorta P15, renal artery P16 and the like may be added.

Large vasculitis includes, for example, aortic arch branching vessels (classification type 1), ascending aorta, aortic arch, and branching vessels (classification type 2a), ascending aorta, aortic arch, branching blood vessels, and, depending on the lesion site. , Chest descending aorta (classification type 2b), thoracic descending aorta, abdominal aorta, and renal artery (classification type 3), abdominal aorta, and / or renal artery (classification type 4), and ascending aorta, aortic arch, It is classified into 6 types: branched blood vessels, descending thoracic aorta, abdominal aortic arch, and / or renal artery (classification type 5).

That is, in large vasculitis, even if symptoms occur in the mediastinum, symptoms may occur in sites such as the descending thoracic aorta P14, the abdominal aorta P15, and the renal artery P16. Therefore, learning and execution may be performed by adding the descending thoracic aorta P14, the abdominal aorta P15, the renal artery P16, etc. to the first site. When such a site is added, it can be determined whether or not symptoms occur in sites such as the descending thoracic aorta P14, the abdominal aorta P15, and the renal artery P16. That is, the symptoms classified into type 1 to type 5 can also be determined. It should be noted that all of these parts need not be included in one medical image, and may be input in another medical image.

The data of the first medical image and the second medical image may be expanded. However, when expanding data for medical images, for example, it is desirable to set the following conditions.

It is desirable that the enlargement and reduction of the medical image be performed within the range considered by human common sense. For example, the heart, blood vessels, etc. are rarely twice as large as the general size. Therefore, it is desirable that the enlargement and reduction of the medical image be performed in the range of about 0.9 times to 1.1 times.

It is desirable that the contrast of the medical image is changed within a range in which a human can judge the shape or the like by looking at the medical image. That is, it is desirable to change the contrast of the medical image in a range in which the image is not darkened or brightened to a level where the shape, color, etc. cannot be understood.

In addition, the method shown above and other data expansion may be performed in combination.

Further, the image for generating the merged image may be an image showing the first part and an image based on the image showing the second part. Therefore, in the learning process, the first medical image and the second medical image may be used as the image used to generate the merged image, or an image generated by performing some processing on these images. It may be. On the other hand, in the execution process, the image used to generate the merged image may be an image for the execution doctor or an image generated by performing some processing on the image for the execution doctor.

<Other Embodiments>
Each device shown above does not have to be one device. That is, each device may be composed of a plurality of devices. In addition, each device may have a device other than those shown above connected internally or externally by wire or wirelessly.

In the embodiment of the present invention, the diagnostic device, the information processing device included in the diagnostic system, and the like may perform processing in parallel, distributed, or redundantly.

Further, the processing and data according to the present invention may be processed in parallel, distributed or redundantly by using a cloud computer or the like. Further, the data may be stored redundantly or distributed in a cloud computer, RAID, or the like.

Further, the present embodiment may be realized by a program or the like for causing a computer to execute a diagnostic method.

Therefore, when each process for realizing the diagnostic method is executed based on the program, the arithmetic unit and the control device of the computer perform the calculation and control based on the program in order to execute each process. In addition, the storage device of the computer stores the data used for the processing based on the program in order to execute each processing. In this way, the diagnostic method may be performed by the hardware resources collaborating to execute the process based on the program.

In addition, the program can be recorded and distributed on a computer-readable recording medium. The recording medium is a medium such as a magnetic tape, a flash memory, an optical disk, a magneto-optical disk, or a magnetic disk. In addition, the program can be distributed over telecommunication lines.

Although the exemplary embodiments of the present invention have been described above, the present invention is not limited to the specifically disclosed embodiments. That is, various modifications, improvements or changes can be made without departing from the scope of claims.

1 Patient 2 Diagnostic system 3 Judgment result 20 Diagnostic device 21 Medical image generation system 22 Learning model DP Dropout layer F1 1st filter FN1 1st medical image input unit FN2 2nd medical image input unit FN3 Learning unit FN4 Execution image input unit FN5 Judgment unit IMG1 1st medical image IMG2 2nd medical image IMG3 Execution medical image IMG4 Merged image IMG10 Slice medical image group IN1 1st input image IN2 2nd input image L1 1st hidden layer L2 2nd hidden layer L3 3rd hidden layer LC11 11th Folding Layer LC12 12th Folding Layer LC13 13th Folding Layer LC2 2nd Folding Layer LC31 31st Folding Layer LC32 32nd Folding Layer LC33 33rd Folding Layer LD31 1st Dropout Layer LD32 2nd Dropout Layer LF Fully Bonded Layer LF31 1st fully connected layer LF32 2nd fully bonded layer LN3 Normalized layer LO Output layer LP11 11th pooling layer LP12 12th pooling layer LP31 31st pooling layer LP32 32nd pooling layer LP33 33rd pooling layer LP34 34th pooling layer MP1 1st feature map MP2 2nd feature map NW network OUT1 1st output feature map OUT2 2nd output feature map P1 1st part P2 2nd part

Claims (14)

A first medical image input unit that inputs a first medical image that projects the first part of the patient,
A second medical image input unit that inputs a second medical image that projects a second part that is a part different from the first part, and a second medical image input unit.
A learning unit that learns a network and builds a learning model based on the first medical image and the second medical image.
A diagnostic device including a determination unit that determines a symptom based on an image for an execution doctor that projects the first portion and the second portion using the learning model. The first part is the mediastinum.
The diagnostic device according to claim 1, wherein the second site is the liver. The diagnostic device according to claim 1 or 2, wherein the symptom is macroangiitis. The diagnostic device according to any one of claims 1 to 3, wherein the first medical image is an image having a higher resolution than the second medical image. The second medical image is an image including a group of pixels showing the same color, pixels showing the same pixel value, or pixels within the reference value in the second part. The diagnostic apparatus according to any one of 4. The diagnostic device according to claim 5, wherein the second medical image shows a position in the second site that does not include the contour of an organ. The diagnostic device according to any one of claims 1 to 6, wherein the first medical image and the second medical image are generated by cutting out from a medical image showing both the first part and the second part. The diagnostic device according to any one of claims 1 to 7, wherein the learning unit generates a merged image based on the first medical image and the second medical image, and performs learning using the merged image. .. The diagnostic device according to claim 8, wherein the merged image is an image showing the result of comparing the color or pixel value of each of the first medical image and the second medical image. The diagnostic device according to claim 9, wherein the merged image is an image generated by calculating the difference between the pixel values. The network has a hidden layer
The first medical image and the second medical image further include a processing unit that performs convolution, pooling, or a combination thereof based on the hidden layer.
The merged image is any one of claims 8 to 10 generated based on the first feature map and the second feature map generated by the processing unit by processing the first medical image and the second medical image. The diagnostic device described in the section. The first site includes any of the aortic arch branch blood vessel, the ascending aorta, the aortic arch, the descending thoracic aorta, the abdominal aorta, and the renal artery, and the first site further includes the mediastinum.
The diagnostic device according to any one of claims 1 to 11, wherein the second site is the liver. A first medical image input unit that inputs a first medical image that projects the first part of the patient,
A second medical image input unit that inputs a second medical image that projects a second part that is a part different from the first part, and a second medical image input unit.
A learning unit that learns a network and builds a learning model based on the first medical image and the second medical image.
A diagnostic system including a judgment unit that makes a judgment about a symptom based on an image for an execution doctor that projects the first part and the second part using the learning model. A program that lets a computer perform diagnostic methods
A first medical image input procedure in which a computer inputs a first medical image that projects a first part of a patient,
A second medical image input procedure in which a computer inputs a second medical image that projects a second part that is a part different from the first part.
A learning procedure in which a computer learns a network based on the first medical image and the second medical image and builds a learning model.
A program for a computer to execute a judgment procedure for making a judgment about a symptom based on an execution medical image projected on the first part and the second part using the learning model.

Images