JP2020062340A - Image processing apparatus and program - Google Patents

Abstract

To improve the accuracy of determining a deeply learned model.SOLUTION: An image processing apparatus according to an embodiment includes a learning unit and a correction unit. The learning unit generates a first learned model composed of a neural network by using first data relating to a first region which is a region inside a lesion and second data relating to a second region which is a region influenced by the first region, as training data. The correction unit corrects weighting of an intermediate layer of the first learned model on the basis of information indicating correlation between the first data and the second data, to generate a second learned model for performing determination relating to input medical data.SELECTED DRAWING: Figure 1

Description

  Embodiments of the present invention relate to an image processing device and a program.

  In fields such as so-called Computer Assisted Detection / Diagnostics, in which diagnosis is supported using a computer, for example, learning is performed by using a feature amount of a lesion known in advance as teacher data to generate a learned model. Moreover, in recent years, even if a human does not give the feature amount as prior knowledge by deep learning or reinforcement learning, the program self-learns the feature amount itself, and the human being labels (signs) the feature amount. A trained model may be generated by giving.

  However, for example, when learning is performed using deep learning, if the weighting of the intermediate layer is not appropriate, the accuracy of determination of the learned model that has been learned may decrease.

JP, 2016-7270, A

  The problem to be solved by the present invention is to improve the determination accuracy of a deep-learned learned model.

  The image processing apparatus according to the embodiment includes a learning unit and a correction unit. The learning unit uses, as training data, first data relating to a first region which is a region inside the lesion and second data relating to a second region which is a region affected by the first region. Then, a first trained model composed of a neural network is generated. The correction unit corrects the weighting of the intermediate layer of the first learned model based on the information indicating the correlation between the first data and the second data, and determines the input medical data. Generate a second trained model.

FIG. 1 is a diagram showing an image processing apparatus according to an embodiment. FIG. 2 is a flowchart illustrating the flow of learning processing performed by the image processing apparatus according to the embodiment. FIG. 3 is a diagram illustrating a process performed by the image processing apparatus according to the embodiment. FIG. 4 is a diagram illustrating an example of a neural network used by the image processing apparatus according to the embodiment. FIG. 5 is a flowchart illustrating the flow of processing for executing a learned model performed by the image processing apparatus according to the embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. Here, components that are the same as each other are assigned the same reference numerals, and redundant description is omitted.

(Embodiment)
FIG. 1 is a block diagram showing an image processing apparatus 100 according to the embodiment.

  The medical image diagnostic apparatus 50 is an apparatus that captures or images a subject and generates medical data. The medical image diagnostic apparatus 50 is, for example, an X-ray CT (Computed Tomography) apparatus. In such a case, the medical image diagnostic apparatus 50 performs X-ray CT imaging on the subject to generate X-ray CT data or X-ray CT images. The medical image diagnostic apparatus 50 uses the generated X-ray CT data or X-ray CT image as input medical data to be a target for the learned model to make a determination, or for training medical data for generating the learned model. Is transmitted to the image processing apparatus 100. The image processing apparatus 100 uses the interface function 120x included in the first processing circuit 120 or the interface function 130x included in the second processing circuit 130, which will be described later, to transmit X-ray CT data or X-ray CT images from the medical image diagnostic apparatus 50. To get.

  The medical image diagnostic apparatus 50 may generate an X-ray CT image based on the X-ray CT data obtained by performing X-ray CT imaging on the subject, or the image processing apparatus 100 may An X-ray CT image may be generated based on the X-ray CT data acquired from the medical image diagnostic apparatus 50.

  Further, as another example, the medical image diagnostic apparatus 50 is, for example, a magnetic resonance imaging apparatus. In such a case, the medical image diagnostic apparatus 50 performs magnetic resonance imaging on the subject to generate magnetic resonance imaging data or magnetic resonance imaging images.

  The medical image diagnostic apparatus 50 may be other types of medical equipment such as an ultrasonic diagnostic apparatus, a nuclear medicine diagnostic apparatus such as a positron emission tomography apparatus, a digital X-ray diagnostic apparatus, or the like.

  The image processing apparatus 100 is connected to the medical image diagnostic apparatus 50, and generates a learned model, executes the learned model, and executes various image processes. The image processing apparatus 100 uses the medical data generated by the medical image diagnostic apparatus 50 as input medical data which is a target for which the learned model makes a determination, or as medical data for training for generating the learned model. To do. The image processing apparatus 100 uses the input medical data and the medical data for training in the form of an input medical image and a medical image for training, respectively.

  The image processing apparatus 100 includes a memory 132, an input device 134, a display 135, a first processing circuit 120, and a second processing circuit 130. The first processing circuit 120 includes a training data creation function 120a, a learning function 120b, a modification function 120c, an interface function 120x, a control function 120y, and a generation function 120z. The second processing circuit 130 also includes an interface function 130x, a control function 130y, and a generation function 130z.

  In the embodiment, the training data creation function 120a, the learning function 120b, the modification function 120c, the interface function 120x, the control function 120y, the processing functions performed by the generation function 120z, and the learned model are stored in a computer-executable program. It is stored in the memory 132 in the form. The first processing circuit 120 and the second processing circuit 130 are processors that read a program from the memory 132 and execute the program to realize a function corresponding to each program. In other words, the first processing circuit 120 in the state where each program is read out has each function shown in the first processing circuit 120 of FIG. Further, the second processing circuit 130 in the state where each program is read out has the respective functions shown in the second processing circuit 130 of FIG. Further, when the second processing circuit 130 in the state where the program corresponding to the learned model is read out, the processing according to the learned model is performed. Note that in FIG. 1, the first processing circuit 120 and the second processing circuit 130 are described as having these processing functions realized by a single processing circuit, respectively. The first processing circuit 120 and the second processing circuit 130 may be configured in combination to realize the function by each processor executing a program. In other words, each of the above functions may be configured as a program, and one processing circuit may execute each program. Further, both the first processing circuit 120 and the second processing circuit 130 may be realized by a single processing circuit. As another example, a specific function may be implemented in a dedicated independent program execution circuit. In FIG. 1, the training data creation function 120a, the learning function 120b, the modification function 120c, the determination function 130a, the interface functions 120x and 130x, the control functions 120y and 130y, and the generation functions 120z and 130z are the creation unit, the learning unit, and the learning unit, respectively. It is an example of a correction unit, a determination unit, a reception unit, a control unit, an image generation unit.

  The word "processor" used in the above description is, for example, a CPU (Central Processing Unit), a GPU (Graphical Processing Unit), or an application-specific integrated circuit (ASIC), a programmable logic device (for example, simple logic device). Programmable logic device (SPLD), complex programmable logic device (Complex Programmable Logic Device: CPLD), field programmable gate array (Field Programmable Gate Array: FPGA), and the like. Means. The processor realizes the function by reading and executing the program stored in the memory 132.

  Further, instead of storing the program in the memory 132, the program may be directly incorporated in the circuit of the processor. In this case, the processor realizes the function by reading and executing the program incorporated in the circuit. Therefore, for example, instead of storing the learned model in the memory 132, the program related to the learned model may be directly incorporated in the circuit of the processor.

  The first processing circuit 120 is a processing circuit mainly responsible for creation of training data and generation of a learned model by learning using the training data, and the second processing circuit 130 is generated. It is a processing circuit mainly responsible for the function of performing discrimination / discrimination by applying the learned model to the discrimination / discrimination target data.

  The first processing circuit 120 causes the training data creation function 120a to generate training data for performing learning by the learning function 120b, which will be described later, based on the image data acquired from the medical image diagnostic apparatus 50. The training data is training image data, which is image data, for example. The first processing circuit 120 performs learning using the learning function 120b on the training data generated by the training data creation function 120a and generates a learned model. In addition, the second processing circuit 130 makes a distinction on the generated learned model and makes a determination. The training data creation function 120a, the learning function 120b, the correction function 120c, and the determination function 130a will be described later.

  Further, the first processing circuit 120 transmits, to the medical image diagnostic apparatus 50, information related to control of imaging of the medical image diagnostic apparatus 50 related to acquisition of image data for training by the interface function 120x, and the medical image diagnostic apparatus 50. Image data for training obtained by photographing is acquired from the diagnostic device 50. In addition, the first processing circuit 120 having the interface function 120x stores the received training image data in the memory 132.

  Similarly, the second processing circuit 130 uses the interface function 130x to transmit to the medical image diagnostic apparatus 50 information related to the control of imaging of the medical image diagnostic apparatus 50 related to the acquisition of image data to be distinguished, and the medical image diagnostic apparatus 50. From the image diagnostic apparatus 50, the image data of the discrimination target obtained by photographing is acquired. Further, the first processing circuit 120 having the interface function 120x stores the received image data of the discrimination target in the memory 132.

  The first processing circuit 120 and the second processing circuit 130 perform overall control of the medical image diagnostic apparatus 50 and the image processing apparatus 100 by the control function 120y and the control function 130y, respectively, and perform imaging and image generation. , Control the display of images, etc.

  Further, the first processing circuit 120 uses the generation function 120z to generate image data for training based on the image data for training, or performs various image processing on the generated medical data for training. I do. Similarly, the second processing circuit 130 uses the generation function 120z to generate medical data to be discriminated based on the image data to be discriminated, or to generate various images for the generated medical data to be discriminated. Perform processing.

  The memory 132 stores various data such as the above-mentioned training image data, training data, and image data to be classified. The memory 132 is a RAM (Random Access Memory), a semiconductor memory device such as a flash memory, a hard disk, an optical disk, or the like.

  The input device 134 receives various instructions and information inputs from the operator. The input device 134 is, for example, a pointing device such as a mouse or a trackball, a selection device such as a mode changeover switch, or an input device such as a keyboard. Under the control of the control function 130y, the display 135 displays a GUI (Graphical User Interface) for receiving the input of the imaging condition, an image generated by the generation function 120z or 130z, and the like. The display 135 is, for example, a display device such as a liquid crystal display. The display 135 is an example of a display unit.

  Next, the background of the embodiment will be briefly described.

  In fields such as so-called Computer Assisted Detection / Diagnostics, in which diagnosis is supported using a computer, for example, learning is performed by using a feature amount of a lesion known in advance as teacher data to generate a learned model. Moreover, in recent years, even if a human does not give the feature amount as prior knowledge by deep learning or reinforcement learning, the program self-learns the feature amount itself and labels the feature amount with a label (symbol). May be given to generate a trained model.

  However, for example, when performing deep learning, it may not be possible to prepare a sufficient number of training data. In this case, for example, if the weighting of the intermediate layer is not appropriate, the determination accuracy of the deep-learned learned model may decrease.

  In view of this background, the image processing apparatus 100 according to the embodiment includes the first processing circuit 120. The first processing circuit 120 uses the learning function 120b to generate the first data related to the first area, which is an area inside the lesion, and the first data related to the second area, which is an area affected by the first area. The second trained data is used as training data to generate a first trained model composed of a neural network. Here, the training data is, for example, training image data that is image data, and the first data and the second data are, for example, image data. The first processing circuit 120 corrects the weighting of the intermediate layer of the first trained model based on the information indicating the correlation between the first data and the second data by the correction function 120c. , A second learned model for making a determination on the input medical data is generated.

  As described above, the first processing circuit 120 corrects the weighting of the intermediate layer of the generated first learned model based on the prior knowledge, thereby generating the second learned model with high determination accuracy. be able to.

  The above will be described in more detail with reference to FIGS.

  First, a procedure for generating a learned model, which is performed by the first processing circuit 120 by the learning function 120b and the correction function 120c, will be described with reference to FIG.

  First, the first processing circuit 120 acquires medical data for training from the medical image diagnostic apparatus 50 by the interface function 120x (step S100). For example, when the medical image diagnostic apparatus 50 is an X-ray CT apparatus, the first processing circuit 120 acquires X-ray CT data or X-ray CT image from the medical image diagnostic apparatus 50 by the interface function 120x. The medical image diagnostic apparatus 50 may generate an X-ray CT image based on the X-ray CT data obtained by performing X-ray CT imaging on the subject, or the image processing apparatus 100 may An X-ray CT image may be generated based on the X-ray CT data acquired from the medical image diagnostic apparatus 50.

  Further, as another example, the medical image diagnostic apparatus 50 is, for example, a magnetic resonance imaging apparatus. In such a case, the image processing apparatus 100 acquires the magnetic resonance imaging data or the magnetic resonance imaging image from the medical image diagnostic apparatus 50 by the interface function 120x. In such a case, the medical image diagnostic apparatus 50 may generate the magnetic resonance imaging image based on the magnetic resonance imaging data obtained by performing the magnetic resonance imaging imaging, or the image processing apparatus 100 may cause the medical image diagnostic apparatus 100 to generate the magnetic resonance imaging image. A magnetic resonance imaging image may be generated by the generation function 120z based on the magnetic resonance imaging data acquired from 50.

  When the medical image diagnostic apparatus 50 is another type of medical equipment such as an ultrasonic diagnostic apparatus, a nuclear medicine diagnostic apparatus such as a positron emission tomography apparatus, a digital X-ray diagnostic apparatus, the image processing apparatus 100 is The interface function 120x acquires data generated by performing imaging or imaging of the subject from the medical image diagnostic apparatus 50 as medical data for training for generating a learned model.

  In addition, the format of medical data represented by medical data for training is image data reconstructed based on volume data, image data reconstructed based on raw data, and image data can be further processed. It includes image data in various formats such as processed image data, processed image data such as data in which edges of image data are emphasized.

  Further, the embodiment is not limited to the case where the image processing apparatus 100 acquires the medical data for training directly from the medical image diagnostic apparatus 50. For example, the medical image diagnostic apparatus 50 transmits medical data to a storage device such as an image storage server that exists separately as training medical data, and the image processing apparatus 100 acquires the medical data stored in the storage device. May be. In this case, the medical data stored in the storage device is not limited to the one generated by a single medical image diagnostic device 50, and may be generated by a plurality of medical image diagnostic devices 50.

  Subsequently, the first processing circuit 120 causes the training data creation function 120a to generate training data based on the training medical data acquired in step S100 (step S110).

  Here, the first processing circuit 120 uses the training data creation function 120a to generate the first data relating to the first region, which is the region inside the lesion, and the region affected by the first region (influence region). And the second data relating to the second region, which is, is generated as training data.

  That is, the first processing circuit 120 uses the training data creation function 120a to directly relate the first data relating to the first region, which is a target for determining the presence or absence of a lesion, to the first region. And the second data relating to the second region, which is a region that can be supplementarily used in determining the presence or absence of a lesion and the like, are generated as training data.

  Here, examples of the “second region (influenced region) which is a region affected by the first region” include the following examples.

  For example, when a pathological image is determined, the extracellular microenvironmental area of the lesion in the first area is an example of the second area. That is, the first processing circuit 120 uses the training data creation function 120a to generate the first data relating to the first region, which is the region inside the lesion, and the second data relating to the extracellular microenvironment region of the lesion. Second data relating to the region is generated as training data.

  Here, the microenvironmental region refers to normal cells, molecules, blood vessels and the like that are present around tumors and the like and feed nutrients to the tumors and the like. Since the growth and spread rates of the tumor are affected by the presence of the tumor, the first processing circuit 120 generates data related to these areas as training data.

  Further, when a cancer-related image is determined, that is, when the first region is a region inside the cancer, the stromal cell region outside the tissue of the cancer or the metastatic property of the cancer is high. The tissue is an example of the second area. That is, the first processing circuit 120 uses the training data creation function 120a to generate the first data relating to the first region, which is the region inside the cancer, and the stromal cell region outside the tissue of the cancer, or Second data relating to at least one region of the tissue with high metastaticity of the cancer is generated as training data.

  In this way, for example, in order to utilize the information outside the cancer cell tissue as a supplementary decision material, the first processing circuit 120 uses the training data creation function 120a to convert the data related to these areas into Generate as training data.

  Further, the second region in the case where the first region is the functional region of the brain will be described with reference to FIG. FIG. 3 is a diagram illustrating a process performed by the image processing apparatus according to the embodiment.

  FIG. 3 is an example showing an image of the brain, and the lesion area 10 shows the first area. The first region is included in the functional region of the brain. In addition, the nerve function-related region 11 represents a gyrus unit that is the lesion region 10 or a function aggregation region thereof, and a region having a nerve function connection. The nerve function related area 11 is an example of a second area. That is, the first processing circuit 120 uses the training data creation function 120a to generate the first data relating to the lesion area 10, which is an area inside the lesion in the functional area of the brain, and the area having the neural function connection with the lesion area 10. Second data relating to a certain nerve function-related area 11 is generated as training data.

  Further, the innervation area 12 is an area other than the brain, and indicates an innervation area associated with the lesion area 10, that is, an area to which innervation from the lesion area 10 extends. The innervation area 12 is an example of a second area. That is, the first processing circuit 120 uses the training data creation function 120a to generate the first data relating to the lesion area 10, which is an area inside the lesion in the functional area of the brain, and the area other than the brain, which is the lesion area 10. The second data related to the innervation area 12 related to is generated as training data.

  In this way, for example, in order to utilize the information of the area related to the lesion area in the brain as a supplementary judgment material, the first processing circuit 120 causes the training data creation function 120a to perform data related to these areas. Are generated as training data.

  The first processing circuit 120 uses the training data creation function 120a to extract the first region and the second region from the training medical data acquired in step S100 using, for example, a contour extraction technique such as the Levelset method. A region is extracted, and training data is generated based on the extracted first region and second region.

  Here, the training data generated by the training data creation function 120a by the first processing circuit 120 is, for example, teacher data in which images related to the first region and the second region and the correct answer data of the determination result are associated with each other. Is. Here, as an example of the correct answer data of the determination result, for example, there is data which gives "1" if the image is an image relating to a lesion and "0" if the image is not an image relating to a lesion. Further, as an example of correct data of the determination result, for example, if the image is an image related to a benign tumor, the value “benign tumor” is given, and if the image is an image related to a malignant tumor, the value “malignant tumor” is given, If the image is not an image relating to a tumor, data that gives the value "no tumor" can be mentioned. Moreover, as an example of the correct answer data of the determination result, there is data that gives the value of the feature amount related to the image as a real value.

  In deep learning, learning using so-called self-encoding (auto encoder) may be performed before learning using teacher data. In such learning, the teacher data in which the images related to the first region and the second region and the correct answer data of the determination result are associated with each other may not always be necessary. In such a case, the first processing circuit 120 uses, for example, the image data itself of the first region or the second region as training data to perform learning by self-coding. In such a case, for example, the image data itself of the first region or the second region is also positioned as an example of the training data generated by the training data creation function 120a by the first processing circuit 120.

  Returning to FIG. 2, subsequently, the first processing circuit 120 uses the training data generated in step S110 for the first area (abnormal area) and the second area (influenced area) by the learning function 120b. Learning is performed to generate a first learned model (step S120).

As an example, the first processing circuit 120 uses the learning function 120b to perform learning on the first area (abnormal area) and the second area (influence area) using the training data generated in step S110, A first trained model composed of a network is generated. Here, examples of the neural network include, for example, a convolutional neural network (CNN) or a RNN.
(Recurrent Natural Network) and the like. Among the neural networks, one having a multi-layer structure is also called a DNN (Deep Neural Network).

  The first processing circuit 120 causes the learning function 120b to perform inverse error propagation on the training data, which is the supervised data generated in step S110, for the first area (abnormal area) and the second area (influence area). Learning is performed by a method such as a method, and a first learned model configured by a neural network is generated. Here, to perform learning or to generate a learned model means, more specifically, to determine the weighting between the nodes forming the neural network.

  When so-called deep learning is performed, the first processing circuit 120 may perform learning by an automatic encoder using the learning function 120b and the training data generated in step S110. In this case, the first processing circuit 120 causes the learning function 120b to generate the first learned model based on the result of the learning performed by the auto encoder.

  Hereinafter, a case where the first processing circuit 120 performs learning using a convolutional neural network (CNN) will be described with reference to FIG. FIG. 4 is a diagram showing an outline of CNN.

  In FIG. 4, an input layer 1 represents a layer to which data is input. In the case of CNN, the data input to the input layer 1 is, for example, data in which each element takes a real value and is represented by a two-dimensional array having a size of 32 × 32. The convolutional layers 2a, 2b, 2c and 3d and the pooling layers 3a, 3b, 3c and 3d are layers that perform predetermined operations called convolution and pooling on the data of the previous layer and output the results of the operations. . Here, usually, for one data input to the input layer 1, a plurality of data are generated in the convolutional layer 2a, which is the next layer of the input device 1, by slightly changing the convolution method, for example. It For example, regarding the data generated in the convolutional layer 2a, when the size of the data input to the input layer is 32 × 32, 20 pieces of data having a data size of 28 × 28 are generated. Subsequently, a pooling operation is performed on the data generated in the convolutional layer 2a, and, for example, 20 pieces of data having a size of 14 × 14 are generated in the pooling layer 3a. Then, a convolution operation is performed on the rule data generated in the pooling layer 3a, and 20 pieces of data having a size of 10 × 10, for example, are generated in the convolution layer 3b. Similarly, the data size of the data generated by the convolutional layer 2c, the pooling layer 3c, the convolutional layer 2d, and the pooling layer 3d is reduced, for example. For example, the data generated by the pooling layer 3c has a data size of 5 × 5 and the number of data is 20. Further, the data generated by the pooling layer 3d has a data size of 3 × 3 and the number of data is 20.

  The all-coupling layers 4a and 4b are layers that receive all the data immediately preceding the layer as input and output the data. That is, all the data before the fully connected layer are combined and the data is output. For example, in the fully connected layer 4a, for example, 20 pieces of data having a size of 1 × 1 are generated, and the output of each 20 pieces of 1 × 1 data is 20 pieces generated in the pooling layer 3d. Data depends on the value of all data. Similarly, in the fully-coupled layer 4b, for example, 20 pieces of data having a size of 1 × 1 are generated, and the output of each 20 pieces of 1 × 1 data is generated in the fully-joined layer 4a. It depends on the value of all 20 data.

  The output layer 5 is a layer in which the final output result is displayed based on the output result of the immediately previous layer. For example, in the output layer 5, six pieces of data having a size of 1 × 1 are generated. Each of these data is, for example, data indicating whether the image is a lesion by 0 or 1, data indicating whether the lesion is benign or malignant by 0 or 1, and malignancy of the lesion. The data is represented by real numbers.

  The intermediate layer 6, which is a layer other than the input layer 1 and the output layer 5, is also called a hidden layer.

  For simplicity of description, as the calculation performed in the intermediate layer 6, the case where the convolutional layers 2a, 2b, 2c, 3d for performing the convolutional operation and the pooling layers 3a, 3b, 3c, 3d are alternately arranged has been described. However, the embodiment is not limited to this, and for example, a layer that performs operations other than the convolution operation and the pooling operation may be included. The data input to the input layer 1 is the first data relating to the first region, which is the region inside the lesion, and the second data relating to the second region, which is the region affected by the first region. Thus, for example, the full connection layer for mixing the first data portion and the second data portion may be appropriately arranged as needed between the pooling layer and the convolutional layer.

  The first processing circuit 120 uses the learning function based on the training data generated in step S110, which is the supervised data composed of the data input to the input layer 1 and the data output to the output layer 5. Learning is performed by 120b, for example, by the back error propagation method. Here, the meaning of learning means determining the weighting in the neural network including the input layer 1, the intermediate layer 6, and the output layer 5. Here, to determine the weighting in the neural network from the input layer 1, the intermediate layer 6 and the output layer 5 means, specifically, a set of coefficients characterizing the coupling between the input layer 1 and the convolutional layer 2a, the convolutional layer 2a. To determine the set of coefficients representing the degree of coupling between each layer, such as the set of coefficients characterizing the coupling between the pooling layer 3a and the pooling layer 3a, and the set of coefficients characterizing the coupling between the pooling layer 3a and the convolutional layer 2b. means. As described above, the first processing circuit 120 is based on the training data, which is the supervised data composed of the data input to the input layer 1 and the data output to the output layer 5, generated in step S110. Then, the learning function 120b performs learning to determine the weighting between the layers, and generates a first learned model which is configured by a neural network and whose weighting is determined. Here, the training data includes first data relating to a first region which is a region inside a lesion and second data relating to a second region which is a region affected by the first region. Become.

  In deep learning, self-coding (auto encoder) can be used. The first processing circuit 120 is the first data related to the first region, which is the region inside the lesion, generated by the learning function 120b in step S110, and the region affected by the first region. Self-encoding may be performed based on the second data related to the second region. The data required for this learning does not have to be supervised data, that is, it may be unlabeled data.

  Note that, as a feature of CNN, it is possible to extract the feature amount corresponding to the node of the intermediate layer by cutting out the data of the intermediate layer of the learned model whose weighting is determined. That is, the first processing circuit 120 extracts the feature amount of the intermediate layer by extracting and labeling the data of the intermediate layer of the first learned model generated in step S120 by the learning function 120b. (Step S130). For example, the first processing circuit 120 extracts the feature amount in the pooling layer 3a by cutting out the data of the pooling layer 3a in the first learned model generated in step S120 by the learning function 120b. In addition, for example, the first processing circuit 120 causes the learning function 120b to extract the data of the pooling layer 3c in the first trained model generated in step S120 and label the data, so that the pooling layer 3c can be labeled. Extract feature quantities.

  When comparing the pooling layer 3a and the pooling layer 3c, for example, the pooling layer 3a is relatively close to the input layer. Therefore, the feature amount obtained by the pooling layer 3a is a low-order level such as a repeating structure appearing in the current image. It becomes the feature quantity of. On the other hand, since the pooling layer 3c is closer to the output than the pooling layer 3a, the feature amount obtained in the pooling layer 3c is a feature amount of a higher level related to the lesion structure.

  Since there is generally a plurality of data in each intermediate layer, the first processing circuit 120 can extract a plurality of feature quantities corresponding to the number of data in the intermediate layer. For example, in the above-mentioned example, since there are 20 pieces of data having a data size of 5 × 5 in the data relating to the pooling layer 3c, the first processing circuit 120 corresponds to these 20 pieces of data, The feature amount of can be extracted.

  Then, the first processing circuit 120 causes the correction function 120c to weight the intermediate layer of the first learned model based on the information indicating the correlation between the first data and the second data. A second learned model that is changed and is used to determine the input medical data is generated (step S140).

  First, the information indicating the correlation between the first data and the second data will be described.

  Information indicating the correlation between the first data relating to the first region, which is the region inside the lesion, and the second data relating to the second region, which is the region affected by the first region For example, when there is a lesion in the first area (abnormal area), information indicating the probability that an abnormality appears in the second area (influence area), or conversely, the second area (influence area) This is information indicating the probability that a lesion will appear in the first area (abnormal area) when an abnormality appears in. That is, as the information indicating the correlation between the first data and the second data, the feature amount regarding the degree to which the second region (influence region) is influenced by the first region (abnormal region) is included. , And the second region (influenced region) and the feature amount relating to the degree of affecting the first region (abnormal region).

  Then, the first processing circuit 120 causes the correction function 120c to weight the intermediate layer of the first learned model based on the information indicating the correlation between the first data and the second data. To change. That is, the first processing circuit 120 determines the correlation between the first data and the second data when the feature amount to be labeled is generated in the learning process of generating the first learned model. Based on the index representing the magnitude of the relationship, the weighting of the layer downstream or upstream of the intermediate layer in which the feature amount is generated is corrected.

  This point will be described below.

  For example, consider the case of the brain, as shown in FIG. In this case, when a lesion candidate is found in the lesion area 10, it is clinically known that the nerve function-related area 11, which is a functional area of the brain functionally related to the brain, and the brain on the opposite side are also affected. There is. For example, in the case of dementia or the like, when a lesion candidate is found in a specific lesion area 10, the hippocampus is affected. That is, when there is a lesion in the first region, an abnormality also appears in the second region. In such a case, a positive difference is found between the lesion in the first region and the abnormality in the second region. There is a correlation.

  Here, consider a case where the feature amount extracted in step S130 is a feature amount corresponding to a case where there is a positive correlation between the lesion in the first region and the abnormality in the second region.

  In this case, the first processing circuit 120 uses the correction function 120c to determine that the weighting related to the node from which the feature amount is extracted is information indicating the correlation between the first data and the second data. And the strength of the weighting is strengthened. That is, the first processing circuit 120 uses the correction function 120c to increase the weight from the preceding layer or the subsequent layer regarding the node from which the feature amount is extracted.

  On the other hand, consider a case where the feature amount extracted in step S130 is a feature amount corresponding to a case where there is a negative correlation between the lesion in the first region and the abnormality in the second region.

  In this case, the first processing circuit 120 causes the correction function 120c to weight the weight of the node from which the feature amount is extracted, consistent with the information indicating the correlation between the first data and the second data. If not, the weighting strength is weakened. That is, the first processing circuit 120 reduces the weight from the preceding layer or the subsequent layer related to the node from which the feature amount is extracted, or cuts the node, by the correction function 120c.

  In this way, the first processing circuit 120 changes the weighting of the intermediate layer of the first learned model by the correction function 120c and generates the second learned model.

  Note that the first processing circuit 120 uses existing clinical knowledge as the correlation between the first data and the second data, and clinically influences it based on, for example, knowledge based on verified pathological information. The information indicated by is used as prior information. Further, as another example, the first processing circuit 120 uses information obtained by using the result of the learning performed separately as the correlation between the first data and the second data. That is, the first processing circuit 120 uses the information obtained from the third learned model as the correlation between the first data and the second data. Note that the third learned model is not limited to the model related to the neural network, and the feature amount can be extracted, but an identification method that does not label the extracted feature amount, for example, Support vector It may be a model generated using machine learning such as machine.

  As described above, the first processing circuit 120 generates the first learned model by deep learning, and then uses, for example, the knowledge obtained from clinical knowledge to generate the first learned model. The weighting of the hidden layer is modified to generate a second learning product model. In this way, by combining deep learning and prior knowledge, the advantages of both can be taken in and the determination accuracy of the deep learned learned model can be improved. For example, even when there are few cases of training data, it is possible to improve the accuracy of lesion determination. Further, for example, in the case of a complex disease related to each other, the ability to discriminate a plurality of lesions is improved.

  Subsequently, the execution of the second learned model generated by the first processing circuit 120 based on the learning function 120b will be described. FIG. 7 is a flowchart illustrating the flow of execution of the second learned model performed by the image processing apparatus according to the embodiment.

  The second processing circuit 130 uses the determination function 130a to perform determination (identification of lesion) on the input medical data based on the second learned model generated in step S140 of FIG. 3 (step S200). For example, the second processing circuit 130 uses the determination function 130a to determine the presence / absence of a lesion in the image, whether the lesion is benign or malignant, based on the second learned model generated in step S140 of FIG. The likelihood of a specific lesion in the lesion area is calculated.

  Subsequently, the second processing circuit 130 causes the control function 130y to display the determination result performed by the determination function 130a in step S200 on the display 135 (step S210).

  In step S200, the format of the input medical data, the medical data represented by the medical data for training, and the like is the image data reconstructed based on the volume data or the image data reconstructed based on the raw data. The image data includes various types of image data such as image data obtained by further processing the image data, for example, processed image data such as data in which the edges of the image data are emphasized.

  Regarding acquisition of input medical data, for example, the image processing apparatus 100 acquires input medical data from the medical image diagnostic apparatus 50. However, the medical image diagnostic apparatus 50 transmits the medical data to a storage device such as an image storage server that is separately present as medical data for training or input medical data, and the medical data stored in the storage device is used as an image processing apparatus. 100 may acquire. In this case, the medical data stored in the storage device is not limited to the one generated by a single medical image diagnostic device 50, and may be generated by a plurality of medical image diagnostic devices 50.

  Further, the input medical data to be the determination target or the discrimination target is not limited to the image captured by the medical image diagnostic apparatus 50, and may be data such as a pathological image captured by a microscope by collecting a subject cell. .

(Other embodiments)
The embodiment is not limited to the above example.

  In the embodiment, the case where the second data is data related to the second area that is an area affected by the first area has been described, but the embodiment is not limited to this. For example, the second data is not limited to the information related to the specific area, and may be information having a relationship with the first area.

  In such a case, the first processing circuit 120 uses the learning function 120b to generate the first data relating to the first region, which is the region inside the lesion, and the second data, which is information related to the first region. Generate a first trained model using and as training data. The first processing circuit 120 corrects the first learned model on the basis of the information indicating the correlation between the first data and the second data by the correction function 120c, so as to correct the input medical data. A second learned model for making the determination is generated.

  For example, consider a case where the second data is, for example, genetic information of the subject.

  In such a case, the first processing circuit 120 uses the learning function 120b to use the first data relating to the first region, which is the region inside the lesion, and the genetic information of the subject as training data, Generate a trained model. The first processing circuit 120 corrects the first trained model based on the information indicating the correlation between the first data and the gene information of the subject by the correction function 120c to input the input medical data. Generate a second trained model that makes the determination of

  The information indicating the correlation between the first data and the gene information of the subject is, for example, the probability that the subject having the specific gene has the specific lesion in the first region. Information.

  Further, in the embodiment, the first processing circuit 120 corrects the first learned model by the correction function 120c is not limited to the intermediate convolutional layer, the pooling layer, or the like, and the output layer of the first learned model is used. Alternatively, the fully connected layer may be modified to generate the second trained model.

  Further, separate processing circuits are in charge of learning of the first area and learning of the second area, and a processing circuit that integrates the processing circuits performs weighting adjustment on the output results of the plurality of processing circuits. Then, the final output result may be output. Information indicating the correlation between the first data and the second data may be used at the stage of such weight adjustment.

  In addition, the user gives feedback to the determination made by the determination function 130a by the second processing circuit 130, and the first processing circuit 120 performs learning by the learning function 120b based on the feedback given by the user. The internal algorithm of the completed model may be updated. That is, the first processing circuit 120 may perform self-learning by continuing to update the learned model based on the feedback from the user by the learning function 120b.

  The program installed in the image processing apparatus 100 includes the first data related to the first area which is the area inside the lesion and the second data related to the second area which is the area affected by the first area. Using the data as training data, a first trained model composed of a neural network is generated, and the first trained model is generated based on the information indicating the correlation between the first data and the second data. The image processing apparatus 100, which is a computer, is caused to execute each process of correcting the weighting of the intermediate layer of the learned model and generating the second learned model for making a determination on the input medical data.

  According to the image processing device of at least one of the above-described embodiments, the learning determination accuracy can be increased.

  Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the invention described in the claims and the equivalents thereof as well as included in the scope and the gist of the invention.

120a Training data creation function 120b Learning function 130a Judgment function 100 Image processing device 120 First processing circuit 130 Second processing circuit

Claims (10)

Using the first data relating to the first region, which is the region inside the lesion, and the second data relating to the second region, which is the region affected by the first region, as training data, a neural network A learning unit for generating a first trained model composed of a network;
A method of correcting the weighting of the intermediate layer of the first trained model based on information indicating a correlation between the first data and the second data to make a determination on input medical data; And a correction unit for generating a trained model of 2,
Image processing device.   The image processing apparatus according to claim 1, further comprising a determination unit that determines the input medical data based on the second learned model.   The image processing device according to claim 1, wherein the learning unit generates the second learned model based on the information obtained from the third learned model.   The image processing apparatus according to claim 1, wherein the second region is an extracellular microenvironment region of the lesion.   The lesion is a cancer, and the second region is at least one of a stromal cell region outside the tissue of the cancer and a highly metastatic tissue of the cancer. Image processing device.   The image processing apparatus according to claim 1, wherein the first region is a functional region of the brain, and the second region is a region having a neural function connection with the first region.   The image according to claim 1, wherein the first region is a functional region of the brain, and the second region is a region other than the brain, which is a innervation region related to the first region. Processing equipment. Using the first data relating to the first region, which is the region inside the lesion, and the second data relating to the second region, which is the region affected by the first region, as training data, the first data Generate a trained model of
A method of correcting the weighting of the intermediate layer of the first trained model based on information indicating a correlation between the first data and the second data to make a determination on input medical data; A program that causes a computer to execute each process for generating the learned model of No. 2. Using the first data relating to the first region, which is the region inside the lesion, and the second data, which is information having a relationship with the first region, as training data, the first learned model is obtained. A learning unit to generate,
A second trained model that corrects the first trained model based on information indicating a correlation between the first data and the second data to make a determination on input medical data. And a correction unit for generating
Image processing device.   The image processing apparatus according to claim 9, wherein the second data is gene information of a subject.

Images