JP2022126373A - Information processing device, information processing method, computer program, and medical diagnostic system - Google Patents

Abstract

To provide an information processing device for supporting differential diagnosis for a medical image.SOLUTION: An information processing device for processing information on a medical image includes: an inference part for inferring a first disease that indicates a correct answer for the medical image, and a second disease which is an object of differential diagnosis related to the first disease using a learned machine learning model; and a ground calculation part for calculating each ground of the first disease and the second disease from the medical image. The inference part specifies the second disease, which is an object of the differential diagnosis related to the first disease inferred from the medical image on the basis of the differential label information indicating the disease for which differential diagnosis is to be executed for each disease.SELECTED DRAWING: Figure 1

Description

The technology disclosed in this specification (hereinafter referred to as "this disclosure") relates to an information processing apparatus and information processing method for processing medical image data such as pathological image data, a computer program, and a medical diagnosis system.

To treat diseased patients, it is necessary to identify the pathology. Here, the pathology means the reason, process, and grounds for becoming ill. A doctor who performs pathological diagnosis is called a pathologist. In pathological diagnosis, for example, a thin slice of a lesion taken from the body is processed by staining or the like, and a method of diagnosing the presence or absence of a lesion and the type of lesion while observing the slice with a microscope is common. Hereinafter, the term "pathological diagnosis" in this specification refers to this diagnostic method unless otherwise specified. An image obtained by observing a thinly sliced lesion with a microscope is called a "pathological image", and a digitized pathological image is called "pathological image data".

Pathological diagnosis plays a very important role when a serious disease such as cancer is suspected. The number of examinations using pathological diagnosis is on the rise, but the problem is the shortage of pathologists in charge of diagnosis. The shortage of pathologists causes an increase in the workload of pathologists and an increase in the burden on patients due to the lengthening of the period until the diagnosis results are obtained. For this reason, digitization of pathological images, pathological diagnosis using image analysis functions by artificial intelligence, remote diagnosis based on online pathology, etc. are being considered.

Artificial intelligence technology is also permeating the medical field. For example, based on image information obtained from pathological image data to be diagnosed and existing pathological image information with diagnostic results, the diagnostic probability for each disease is obtained using a diagnostic probability model, and pathological image diagnosis support for narrowing down the diseases with high probability. A device has been proposed (see Patent Document 1).

Pathological diagnosis is an absolute medical practice that can only be performed by a doctor because it greatly affects treatment methods and the like. Therefore, pathological diagnosis by artificial intelligence should be used as a tool to support diagnosis, such as reducing diagnostic man-hours through screening, and the final decision should be made by the pathologist. On the other hand, artificial intelligence is likened to a black box because the grounds for its judgments are difficult to understand. For this reason, pathologists cannot understand the basis of pathological diagnosis by artificial intelligence, and there is concern that artificial intelligence cannot be fully used as a diagnostic support tool.

JP 2012-179336 A Japanese Patent Application Laid-Open No. 2020-38600

Residuals and Influence in Regression, Cook, R.; D. and Weisberg, S <https://conservancy. umn. edu/handle/11299/37076> What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision, NIPS 2017, Alex Kendall and Yarin Gal <https://papers. nips. cc/paper/7141-what-uncertainties-do-we-need-in-bayesian-deep-learning-for-computer-vision. pdf> Generative Adversarial Networks, Ian J.; Goodfellow et al. <https://arxiv. org/abs/1406.2661> Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization <https://arxiv. org/abs/1610.02391> "Why Should I Trust You?": Explaining the Predictions of Any Classifier <https://arxiv. org/abs/1602.04938> Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV) <https://arxiv. org/pdf/1711.11279. pdf>

An object of the present disclosure is to provide an information processing apparatus and information processing method, a computer program, and a medical diagnosis system that process medical image data using artificial intelligence functions.

The present disclosure has been made in consideration of the above problems, and a first aspect thereof is an information processing device that processes information about medical images,
an inference unit that infers a first disease that is correct for the medical image and a second disease that is a target of differential diagnosis related to the first disease;
a basis calculation unit that calculates basis for the first disease and basis for the second disease from the medical image;
It is an information processing device comprising

The inference unit infers the first disease from the medical image using a trained machine learning model. The machine learning model is trained using learning data that is a data set that combines medical images and correct diseases.

The inference unit identifies a second disease to be subjected to differential diagnosis related to the first disease inferred from the medical image based on differential label information indicating a disease to be differentially diagnosed for each disease.

The inference unit infers a first disease and a second disease using trained neural network models. The basis calculation unit causes the neural network model to infer the basis of each of the first disease and the second disease.

For example, the basis calculation unit infers the portion of the original medical image that affects each class by tracing the gradient backward from the label that is the inference result of class classification in the output layer of the neural network model. Alternatively, the basis calculation unit calculates the basis for each of the first disease and the second disease based on an output variation amount when the feature amount of the medical image data input to the neural network model is perturbed. are inferred respectively.

In addition, a second aspect of the present disclosure is
An information processing method for processing information about medical images,
a first inference step of inferring a first disease that is correct for the medical image;
a second inference step of inferring a second disease of interest for differential diagnosis associated with said first disease;
a first basis calculation step of calculating basis of the first disease from the medical image;
a second basis calculation step of calculating the basis of the second disease from the medical image;
It is an information processing method having

In addition, a third aspect of the present disclosure is
A computer program written in computer readable form to process information relating to medical images on a computer, said computer comprising:
an inference unit that infers a first disease that is correct for the medical image and a second disease that is a target of differential diagnosis related to the first disease;
an evidence calculation unit that calculates the evidence of the first disease and the evidence of the second disease from the medical image;
It is a computer program that functions as

A computer program according to a third aspect of the present disclosure defines a computer program written in a computer-readable format so as to implement predetermined processing on a computer. In other words, by installing the computer program according to the third aspect of the present disclosure on the computer, cooperative action is exhibited on the computer, and the same action as the information processing apparatus according to the first aspect of the present disclosure effect can be obtained.

In addition, a fourth aspect of the present disclosure is
a learning unit in which the machine learning model learns so that the machine learning model infers a disease from medical image data;
Using the machine learning model learned by the learning unit, infer a first disease that is correct for the medical image, and a second disease that is a target for differential diagnosis related to the first disease an inference unit that infers a disease;
a basis calculation unit that calculates basis for the first disease and basis for the second disease from the medical image;
a display device;
a presentation unit that presents the inference result of the inference unit and the calculation result of the basis calculation unit on the display device;
A medical diagnostic system comprising:

However, the "system" referred to here refers to a logical assembly of multiple devices (or functional modules that implement specific functions), and each device or functional module is in a single housing. It does not matter whether or not

According to the present disclosure, it is possible to provide an information processing apparatus and information processing method, a computer program, and a medical diagnosis system that perform processing for supporting differential diagnosis of medical images using an artificial intelligence function.

Note that the effects described in this specification are merely examples, and the effects provided by the present disclosure are not limited to these. In addition, the present disclosure may have additional effects in addition to the effects described above.

Still other objects, features, and advantages of the present disclosure will become apparent from a more detailed description based on the embodiments described below and the accompanying drawings.

FIG. 1 is a diagram showing a functional configuration example of a medical diagnosis system 100. As shown in FIG. FIG. 2 is a diagram schematically showing a mechanism for constructing learning data. FIG. 3 is a diagram showing the operation of the medical diagnostic system 100 including data adjustment processing by the data adjustment device 200. As shown in FIG. FIG. 4 is a diagram showing a configuration example of the additional data generation unit 313 using GAN. FIG. 5 is a diagram showing a configuration example of a machine learning model 500 applied in the medical diagnosis system 100. As shown in FIG. FIG. 6 is a diagram showing an example of discrimination label information held by the discrimination label information holding unit 113. As shown in FIG. FIG. 7 is a diagram exemplifying the basis image calculated for the inference label. FIG. 8 is a diagram showing an example of a basis image calculated for a discrimination label. FIG. 9 is a flow chart showing processing operations in the inference phase of medical diagnostic system 100 . FIG. 10 is a diagram showing a configuration example of the information processing apparatus 1000. As shown in FIG. FIG. 11 is a diagram showing an example of screen transition. It is a figure which shows roughly the whole structure of a microscope system. It is a figure which shows the example of an imaging system. It is a figure which shows the example of an imaging system.

Hereinafter, the present disclosure will be described in the following order with reference to the drawings.

A. OverviewB. System configurationC. Construction of training data D. Constructing Machine Learning ModelsE. Configuration of identification label information F. Basis calculation G. Presentation of Inference ResultsH. Operations in the Inference PhaseI. Configuration Example of Information Processing ApparatusJ. microscope system

A. Abstract Pathological diagnosis by artificial intelligence can be used as a diagnostic support tool for pathologists, such as reducing diagnostic man-hours. A pathological diagnosis is an absolute word and action that greatly influences treatment methods, etc., and ultimately a pathological diagnosis must be made by a pathologist. However, if it is difficult to understand the basis of pathological diagnosis by artificial intelligence, it becomes difficult for pathologists to use AI for the final pathological diagnosis.

In addition, there is a diagnostic method called "differential diagnosis" that rationally identifies a disease by comparing multiple possible diseases based on the symptoms of the affected area and test results. A pathologist identifies a disease while comparing the disease with the highest probability and the disease with the highest probability (or the disease with the second highest probability) in the thought process of making a pathological diagnosis from one pathological image data. For example, when a pathologist is uncertain as to whether the disease is disease A or disease B, he or she denies disease B and makes a differential diagnosis of disease A. In the pathological diagnosis of cancer, it is also necessary to diagnose the grade (malignancy) of the cancer, and a differential diagnosis is made as a certain gray denial of other grades. However, when diagnosing pathological images using artificial intelligence, it is difficult for a pathologist to make a differential diagnosis from the diagnosis results of artificial intelligence because artificial intelligence is a black box and the grounds for its judgment are not clear.

Therefore, the present disclosure proposes a medical diagnosis system that uses artificial intelligence to support differential diagnosis of pathological image data. The medical diagnosis system according to the present disclosure presents the grounds for the diagnosis results by artificial intelligence, and also presents the grounds for the differential diagnosis corresponding to the diagnosis results. Therefore, the pathologist can appropriately evaluate the artificial intelligence-based diagnosis results and differential diagnosis based on the presented grounds, and make a highly accurate (or confident) pathological diagnosis.

B. System Configuration FIG. 1 schematically shows a functional configuration example of a medical diagnosis system 100 to which the present disclosure is applied. The medical diagnosis system 100 is configured to use artificial intelligence functions to perform differential diagnosis mainly on medical image data such as pathological images, or to assist pathologists in differential diagnosis. The artificial intelligence function specifically consists of a machine learning model such as a CNN (Convolutional Neural Network). The operation of the medical diagnostic system 100 is roughly divided into a learning phase and an inference phase. In the learning phase, a machine learning model is trained using learning data consisting of a data set combining input data (medical image data such as pathological images) and correct labels (correct diagnosis results). Also, in the inference phase, inference (diagnosis of pathology) of input data (medical image data such as pathological images) is performed using the learned machine learning model acquired through the learning phase. In this embodiment, it is assumed that deep learning is performed using a huge amount of learning data, and inference is performed using a DNN (Deep Neural Network) in the inference phase.

The operation of the learning phase is realized by each functional module of the learning data holding unit 101, the learning unit 102, and the model parameter holding unit 103. FIG.

The learning data holding unit 101 holds a huge amount of learning data. The learning data consists of a data set combining digitized pathological image data and accurate diagnostic results by a pathologist, which are correct labels for the pathological image data. For example, pathological image data and diagnosis results diagnosed by pathologists all over the country or all over the world are collected, shaped into learning data in a predetermined format, and stored in the learning data holding unit 101 . Here, the learning data may be labeled with correct answers for all pathological images, or may be labeled with partial pathological images. In the latter case, the pathologist may designate a partial region of the image and attach a correct label to the designated region. As a method of specifying the area, the area may be simply cut into a grid shape, or may be specified with a rectangle, a polygon, or an arbitrary shape.

The learning unit 102 sequentially reads the learning data from the learning data holding unit 101 and learns the machine learning model. A machine learning model, for example, consists of a neural network with a structure that mimics human neurons. A loss function based on the error between the label output by the machine learning model for input data and the correct label is calculated, and the learning process of the machine learning model is performed so as to minimize the loss function.

For example, the learning process of the machine learning model is performed by updating the model parameters so as to output a diagnostic result that is a correct label for the loaded pathological image data. A model parameter is a variable element that defines the behavior of a machine learning model, such as a weighting factor given to each neuron of a neural network. In the error backpropagation method, a loss function is defined based on the error between the value of the output layer of the neural network and the correct diagnosis result (correct label), and the loss function is minimized using the steepest descent method, etc. Model parameters are updated. Then, the learning unit 102 stores model parameters obtained as learning results in the model parameter holding unit 103 .

The operation of the inference phase is realized by each functional module of the image acquisition unit 111 , the inference unit 112 , the discrimination label information storage unit 113 , the basis calculation unit 114 , and the presentation processing unit 115 .

The image fetching unit 111 fetches medical images to be diagnosed from the outside and inputs them to the inference unit 112 . A medical image is, specifically, pathological image data obtained by digitizing a pathological image observed by a pathologist using a microscope with high definition. The pathological image referred to here includes an image obtained by observing a pathological specimen obtained by thinly slicing a biological sample such as a lesion and performing processing such as staining or labeling under a microscope. The image capturing unit 111 is, for example, a WSI (Whole Slide Imaging) scanner that digitizes and captures a microscope observation image of a glass slide on which a pathological specimen is placed. Also, the image capturing unit 111 may be a device that receives pathological image data from a remote WSI scanner via a network.

The inference unit 112 uses a model in which the model parameters read from the model parameter storage unit 103 are set, that is, a learned machine learning model, to infer the pathological image data acquired via the image acquisition unit 111. A pathological diagnosis result is output as an output label.

The differential label information holding unit 113 holds information for performing differential diagnosis related to the diagnostic result inferred by the inferring unit 112 . Specifically, for each disease that can be an output label of a trained machine learning model used in the inference unit 112, information of a differential label for a disease that can be dealt with in differential diagnosis is stored. Upon receiving the output label (diagnosis result) from the inference unit 112, the discrimination label information storage unit 113 returns a discrimination label specifying a disease related to the disease. Further, the discrimination label updating unit 116 updates the discrimination label information held in the discrimination label information holding unit 113 by user input or by curating data in a public database. For example, when the pathological diagnosis protocol of the Society of Pathology is updated, the user may manually input to update the inference label and the corresponding differential label. These label update operations may be automatically performed. The differential label update unit 116 periodically acquires a URL (Uniform Resource Locator) in which the pathological diagnosis protocol of the Society of Pathology is described, detects a differential label portion that has changed, and performs update processing. can be

The basis calculation unit 114 calculates the basis of the result of the diagnosis made by the inference unit 112 using the learned machine learning model (that is, the basis of the judgment of the output label by the machine learning model) and the differential diagnosis corresponding to the diagnosis result of the inference unit 112. (i.e., the basis when the machine learning model determines the discriminative label) when inferring .

The basis calculation unit 114, for example, Gradient-weighted Class Activation Mapping (Grad-CAM) (see, for example, Non-Patent Document 4), LIME (LOCAL Interpretable model-agnostic Explanations) (see, for example, Non-Patent Document 5 ), SHAP (SHApley Additive exPlanations), which is an advanced form of LIME, TCAV (Testing with Concept Activation Vectors) (for example, see Non-Patent Document 6), etc. Algorithms such as It is possible to calculate an image that visualizes the basis for each of the diagnosis and differential diagnosis using the machine learning model. However, the details of the basis calculation method using Grad-Cam, LIME/SHAP, and TCAV will be described later.

The presentation processing unit 115 performs processing for displaying on the display device 120 the diagnosis result of the pathological image data and the differential diagnosis by the inference unit 112 together with the grounds for each judgment. Basically, as the diagnosis result of the pathological image data and the basis for the differential diagnosis, the judgment basis is presented in the form of a superimposed display of a heat map showing the portion of the original pathological image that strongly affects the diagnosis. In addition, the presentation processing unit 115 responds to a user input via the user interface (UI) unit 130 so as to switch and display the diagnosis result of the pathological image data and its base information and the differential diagnosis and its base information. can be However, the details of the display method will be given later.

The display device 120 is a monitor used for observing images for diagnosis using digital pathological images, but from the viewpoint that it is a tool for making a diagnosis by direct visual observation by a pathologist, it is similar to a lens in an optical microscope. Equally important, a high-quality monitor with excellent color reproducibility and a fine pixel pitch is preferable.

A pathologist can confirm the diagnosis result for the pathological image data to be pathologically diagnosed and the differential diagnosis related to the diagnosis result, together with the basis information for each, on the screen of the display device 120 . Therefore, the pathologist can appropriately evaluate the artificial intelligence-based diagnosis results and differential diagnosis based on the presented grounds, and make a highly accurate (or confident) pathological diagnosis.

In addition, as a result of confirming the diagnostic results for the pathological image data to be subjected to pathological diagnosis and the differential diagnosis related to the diagnostic results together with each basis information, the pathologist confirms that the diagnostic results, which are the output labels in the inference unit 112, are not , the differential label can also be adopted in the final pathological diagnosis result. In this case, the pathologist can input the final diagnosis result to the medical diagnosis system 100 through the UI unit 130 or the like. Further, a data set obtained by combining the pathological image data and the differential diagnosis at this time may be stored as new learning data in the learning data holding unit 101 and used for re-learning of the machine learning model.

Note that the learning phase and the inference phase may be realized on individual information processing devices (personal computers, etc.). Alternatively, the learning phase and the inference phase may be realized on one information processing device.

C. The learning data for learning the machine learning model used in the learning data inference unit 112 is a combination of digitized pathological image data and an accurate diagnosis result by a pathologist, which is a correct label for the pathological image data. consists of a set. The learning data may be part of the pathological image data and a set of correct labels.

FIG. 2 schematically shows a mechanism for collecting pathological image data and diagnosis results diagnosed by pathologists scattered all over the country or the world and storing learning data in the learning data holding unit 101 . Each pathologist may perform pathological diagnosis of pathological image data using the medical system disclosed in Patent Document 2, for example. Then, a data set composed of combinations of pathological image data of pathological diagnosis made by each pathologist and the diagnosis results is collected on the cloud through a wide area network such as the Internet.

Deep learning of machine learning models requires a huge amount of training data. All data sets collected on the cloud may be used as learning data. However, among the collected data sets, the data adjustment device 200 performs data adjustment processing such as elimination of harmful data sets, such as data sets that contribute less to the learning of the machine learning model, and investigation of the uncertainty of the machine learning model. to build training data for deep learning.

FIG. 3 conceptually shows the operation of the medical diagnosis system 100 including data adjustment processing by the data adjustment device 200 .

The learning data storage unit 101 stores learning data such as a data set 302 in which pathological image data diagnosed by a pathologist and diagnostic results are combined. The learning unit 102 uses the data set 302 to perform learning processing (deep learning) of a machine learning model 301 configured by a neural network (CNN, etc.).

Test data (TD) such as pathological image data is input to the machine learning model 301 in the learning process, and the accuracy of the output label (diagnostic result inferred from the input pathological image data) from the machine learning model 301 is determined. If it is a misdiagnosis, the information is fed back and the machine learning model 301 learns.

The data adjustment device 200 includes an impact evaluation section 311 , a learning state determination section 312 and an additional data generation section 313 . The influence evaluation unit 311 evaluates the influence of each data set collected through a network or the like on the machine learning model 311 . A data set with a high degree of influence is useful learning data, but a data set with a low degree of influence is harmful as learning data and may be removed. In addition, the learning state determination unit 312 determines whether the learning state of the machine learning model 301, specifically, whether the accuracy cannot be improved further due to the limit of deep learning, or whether the accuracy is not obtained due to lack of learning data. Determine (whether re-learning can further improve accuracy). Further, the additional data generation unit 313 generates additional learning data from learning data that has already been acquired (stored in the learning data storage unit 101) without relying on collection of new data sets from the pathologist. do. The processing of each unit will be described in more detail below.

C-1. Influence Evaluation Here, a method of evaluating the influence of each data set collected through a network or the like on the machine learning model 301, which is performed in the influence evaluation unit 311, will be described.

A data set z is data in which an output label (diagnosis result) y is associated with an input (pathological image data) x. Suppose there are n data sets, as shown in equation (1) below.

When the model parameter of the machine learning model 301 is θεΘ, and the loss of the data set z is L(z, θ), the empirical loss in all n data sets is represented by the following formula (2). can be done.

Training the machine learning model 301 means finding the model parameters that minimize the experience loss. Therefore, model parameters obtained as a result of learning the machine learning model 301 using the n data sets shown in the above formula (1) can be expressed as in the following formula (3). However, as shown on the left side of the equation (3), when "^" is added above the parameter "?", it represents the predicted value of the parameter "?". Hereinafter, in the text, it is written as "θ^", which is "θ" followed by "^".

Next, let us consider the effect on the learning of the machine learning model 301 when there is no data set z of a certain training point. The model parameters of the machine learning model 301 when learning processing is performed by removing the data set z of this training point can be expressed as in the following equation (4).

The degree of influence of the training point data set z is the difference between the model parameters obtained by performing the learning process when the data set z is removed and when all n data sets including the data set z are used. be. This difference is represented by the following formula (5).

Re-learning the model parameters by removing the dataset z for a particular data point is very computationally expensive. Therefore, the influence evaluation unit 311 uses influence functions (see Non-Patent Document 1) to effectively approximate the influence z of the data set without recalculation. Specifically, assuming that the input data (images) of the data set z are weighted by a small value ε, the parameter changes are calculated. Here, using the following equation (6), a new parameter “θ _ε,z ̂” as shown on the left side is defined.

Then, the influence function corresponding to the data set z can be expressed using the following equations (7) and (8).

The above equation (7) is an influence function corresponding to the data set z, and represents, for example, the amount of change in the model parameter θ ̂ with respect to a minute weight ε. Also, the above equation (8) represents a Hessian (Hesse matrix). Here we assume that it is a Hessian matrix that is positive definite, and also has an inverse. Assuming that removing the dataset z at a training point is the same as being weighted by "ε = -1/n", the change in the model parameters when removing the dataset z is 9) can be approximated and expressed.

Therefore, the impact evaluation unit 311 can measure the impact of the data set z without re-learning.

Subsequently, the impact evaluation unit 311 measures the impact on loss at a certain test point z _test using the following equations (10-1) to (10-3).

Thus, we can formulate the influence of a weighted data set z at some test point z _test . Therefore, the impact evaluation unit 311 can measure the impact of the data set on the machine learning model 301 by this calculation. For example, the influence of a certain data set on the prediction (loss) of the model can be obtained by the above equation (10-3). The right side of the above equation (10-3) consists of the gradient for the loss of certain data, the inverse Hessian matrix, the gradient of the loss for certain learning data, and the like.

However, the influence degree evaluation method described in Section C-1 is an example, and the influence degree evaluation unit 311 may measure the degree of influence of the data set by other methods.

C-2. Determination of Learning State Here, a method of determining the learning state of the machine learning model 301 performed by the learning state determination unit 312 will be described.

Although DNN model inference is generally highly accurate, inference is limited. Understand the learning state of the model, that is, whether the accuracy cannot be improved further due to the limit of deep learning, or whether the accuracy is not due to lack of training data (whether the accuracy can be further improved by re-learning). This is very important in making full use of deep learning. However, it is difficult to completely eliminate the uncertainty of deep learning.

Uncertainty in deep learning can be divided into two types: oleatoric uncertainty and epistemic uncertainty. The former random uncertainty is due to observational noise and not to lack of data. For example, an image that is hidden and cannot be seen (occlusion) corresponds to accidental uncertainty. The mouth of the masked person's face is hidden by the mask in the first place, so it cannot be observed as data. On the other hand, the latter uncertainty in recognition is due to the lack of data, and if enough data exists, the uncertainty in recognition can be ameliorated.

The learning state determination unit 312 clarifies the uncertainty of the machine learning model 301 using Bayesian Deep Learning (see Non-Patent Document 2, for example). Bayesian deep learning uses dropout (random invalidation of some model parameters) not only during learning but also during inference to determine the uncertainty of inference results. Specifically, when data (pathological image data) is input to the machine learning model 301, it passes through neurons that are missing due to dropout and obtains output labels characterized by the weight of the pathway, but the same data , the output is dispersed because it passes through different paths and outputs. A large output variance means a large uncertainty in the inference of the machine learning model 301, and the uncertainty can be improved by performing learning with sufficient learning data.

Therefore, based on the result of the learning state determination unit 312 determining the learning state based on Bayesian deep learning, the learning unit 102 terminates learning of the machine learning model 301 or adds learning data to continue learning. You can do it.

C-3. Generating Additional Data Here, a method for generating additional learning data from existing learning data, which is performed by the additional data generating section 313, will be described. The additional data generation unit 313 generates additional learning data for re-learning the machine learning model 301, for example, in response to the result of the learning state determination unit 312 determining the uncertainty of the machine learning model 301. . Further, the additional data generation unit 313 may generate the additional data triggered by the output label being erroneously determined when the test data (TD) is input to the machine learning model 301 . The additional data generator 313 may generate additional data based on the test data at that time.

In this embodiment, it is assumed that the additional data generation unit 313 automatically generates additional learning data using a GAN (Generative Adversarial Network) algorithm (see Non-Patent Document 3, for example). GAN is an algorithm in which two networks compete to deepen their learning of input data.

FIG. 4 shows a configuration example of the additional data generation unit 313 using GAN. The additional data generation unit 313 shown in FIG. 4 includes a generator (Generator: G) 401 and a discriminator (Discriminator: D) 402 . The generator 401 and discriminator 402 are each composed of a neural network model.

The generator 401 adds noise to the pathological image data accumulated in the learning data accumulation unit 101 to generate fake pathological image data (Fake Data: FD). On the other hand, the discriminator 402 discriminates whether the genuine pathological image data and the pathological image data generated by the generator 401 are true or false. Then, the generator 401 learns while competing with each other so that the authenticity by the classifier 402 becomes difficult, and one classifier 402 can correctly classify the pathological image data generated by the generator 401. New pathological image data that cannot be authenticated can be generated. The process of mutual learning is represented by the following formula (11).

In the above equation (11), G corresponds to the generator 401 and D corresponds to the discriminator 402 . D determines whether G is real or fake and learns to maximize the probability D(x) of correctly labeling it. It learns to minimize the probability log(1?D(G(z))) of labeling it as fake.If D is labeled correctly, the value of D(x) increases and logD(x ) also increases, and by finding that G is fake, D(G(z)) decreases, resulting in log(1?D(G(z))) increasing and D On the other hand, if G can generate realistic data, the value of G(z) will be large, and the value of D(G(z)) will also be large, and D correctly labels Not being able to do so results in a smaller value of D(x), which in turn reduces the value of logD(x), resulting in a smaller log(1?D(G(z))) and a preponderance of G. By repeating such an operation, D and G can be alternately updated to deepen each learning.

Of course, the additional data generation unit 313 may generate additional learning data using an algorithm other than GAN, or acquire new learning data by newly collecting pathologist diagnosis results. You may do so.

D. Configuration of machine learning model As described in section A above, the learning unit 102 performs learning processing of a machine learning model for inferring pathological diagnosis results of lesions from pathological image data. A learning model is used to infer the pathological diagnosis result of the lesion from the loaded pathological image data. FIG. 5 conceptually shows a configuration example of a machine learning model 500 applied in the medical diagnosis system 100 . The illustrated machine learning model 500 is constructed using a multi-layer convolutional neural network (CNN). The CNN includes a feature extraction unit that extracts the feature of the input image, and an image classification unit that infers an output label (diagnostic label) corresponding to the input image based on the extracted feature. The former feature extractor consists of a "convolution layer" that extracts edges and features by convolving the input image by restricting connections between neurons and sharing weights, and removing positional information that is not important for image classification. It has a “pooling layer” that provides robustness to the features extracted by the convolutional layer.

In FIG. 5, reference number 501 indicates an image (pathological image data) that is input data to the CNN. Reference numerals 502, 504, 506 indicate the outputs of the convolutional layers. Reference numerals 503 and 505 indicate the output of the pooling layer. Reference number 507 indicates a state in which the outputs 506 of the convolutional layer are arranged one-dimensionally, reference number 508 indicates a fully connected layer, and reference number 509 indicates an output layer as an inference result of class classification.

In the CNN shown in FIG. 5, the area surrounded by a square indicated by reference number 520 is a feature quantity extraction unit, which performs a process of acquiring the image feature quantity of the input pathological image data. A range surrounded by a rectangle indicated by reference number 530 is an image classification unit, which specifies an output label based on the image feature amount (in this embodiment, the diagnosis label of the lesion included in the pathological image data is inferred). ).

Note that the stage of the inference process (the order of processing of each layer) is l, the output value in the l-th layer is Y ^l , and the processing in the l-th layer is expressed as Y ^l =F ^l (Y ^l-1 ). . Also, Y ¹ =F ¹ (X) for the first layer and Y=F ⁷ (Y ⁶ ) for the final layer.

E. The configuration inference unit 112 for differential label information uses a trained machine learning model configured by CNN (see FIG. 5) to extract lesions from the pathological image data acquired via the image acquisition unit 111. support the pathological diagnosis by a pathologist by inferring the pathological diagnosis result of

Furthermore, in the medical diagnosis system 100 according to the present embodiment, the inference unit 112 supports differential diagnosis related to diagnosis results inferred from the loaded pathological image data. Therefore, the differential label information holding unit 113 holds information for performing differential diagnosis related to the diagnostic result inferred by the inferring unit 112 . Specifically, for each disease (inference label) that can be the output label of the learned machine learning model used by the inference unit 112, information on the differential label of the disease that can be the target of the related differential diagnosis is held. Upon receiving the output label (diagnosis result) from the inference unit 112, the discrimination label information storage unit 113 returns a discrimination label specifying a disease related to the disease.

A pathologist's differential diagnosis generally consists of the thought process of identifying a disease while comparing the most probable disease to the rare (or second most probable) disease. For example, when a pathologist makes a differential diagnosis between a disease A and a disease B related to the disease A, the disease B is denied and the disease A is taken as the final diagnosis. In this case, the differential label corresponding to the inference label "Disease A" is "Disease B". Specifically, cancer and non-cancer are diagnosed differentially, non-cancer is denied, and finally cancer is diagnosed. In this case, the differential label corresponding to the inference label "cancer" is "non-cancer".

In the pathological diagnosis of cancer, it is also necessary to diagnose the grade (malignancy) of the cancer, and a differential diagnosis is made as a certain gray denial of other grades. Therefore, the differential label information holding unit 113 holds each grade of cancer as an inference label, and a grade to be subjected to differential diagnosis for each grade as a differential label.

FIG. 6 shows an example of discrimination label information held by the discrimination label information holding unit 113. As shown in FIG. The illustrated discrimination label information has a structure of a lookup table for searching for a discrimination label corresponding to the inference label output from the inference section 112 . For example, when the inference unit 112 infers cancer grade 1 from pathological image data and outputs it to the discrimination label information storage unit 113, grade 2 related to grade 1 is returned as a discrimination label. Also, when the inference unit 112 outputs a diagnosis result of cancer grade 2, the differential label information storage unit 113 returns grades 1 and 3 related to grade 2 as differential labels. Also, when the inference unit 112 outputs the diagnosis result of cancer grade 3, the differential label information storage unit 113 returns grade 2 and grade 4 related to grade 2 as differential labels. Also, when the inference unit 112 outputs a diagnosis result of cancer grade 4, the differential label information storage unit 113 returns grade 3 related to grade 2 as a differential label.

Note that in the system configuration example shown in FIG. Although configured, the method of deriving the identification label is not limited to this. A machine learning model (CNN) used by the inference unit 112 may be trained so as to output a discrimination label together with an inference label.

F. The basis calculation basis calculation unit 114 calculates the basis for inference by the inference unit 112 for each of the inference label output from the inference unit 112 and the discrimination label corresponding to the inference label. The basis calculation unit 114 uses an algorithm such as Grad-CAM, LIME/SHAP, or TCAV to calculate an image that visualizes the judgment basis for each of the inference label and the discrimination label.

Grad-CAM traces the gradient backward from the label that is the inference result of class classification in the output layer (calculates the contribution of each feature map up to class classification, and back-propagates using that weight) The method is an algorithm for estimating the locations that contributed to class classification in the input image data, and the locations that contributed to the classification can be visualized like a heat map. Alternatively, the positional information of the pixels of the input image data is retained up to the final convolutional layer, and by obtaining the degree of influence of the positional information on the final discrimination output, the part of the original input image with strong influence is displayed as a heat map. You may do so.

In a neural network model such as CNN, when image recognition is performed on the input image and class c is output, a method of calculating the basis for judgment based on the Grad-Cam algorithm (a method of generating a heat map) will be explained. .

Assuming that the gradient y _c of class c is the feature map activation A _k , the neuron importance weights are given as shown in equation (12) below.

The forward propagation output of the final convolutional layer is multiplied by the weight for each channel, and Grad-Cam is calculated as shown in the following equation (13) via the activation function ReLU.

In FIG. 7, the portion that is the basis of “cancer grade 1” is superimposed on the original pathological image data as an inference label (that is, the pathological diagnosis result by the inference unit 112) as a heat map 701 and displayed on the original pathological image data. An image 700 is illustrated. Image 700 also displays diagnostic labels 702 . Also, FIG. 8 illustrates an image 800 in which a portion serving as the basis for “cancer grade 2” as a differential label related to the inference label is superimposed on the original pathological image data as a heat map 801. . An identification label 802 is also displayed in the image 800 . A pathologist makes a differential diagnosis while comparing the pathological image data with heat map display shown in FIGS. can be adopted as a pathological diagnosis result.

LIME estimates that if the output result of the neural network is reversed or greatly fluctuates when a specific input data item (feature value) is changed, that item is "highly important in determination". For example, the basis calculation unit 114 generates another model (basis model) for local approximation to indicate the reason (basis) for inference in the machine learning model used by the inference unit 112 . The basis calculation unit 114 generates a basis model that locally approximates a combination of input information (pathological image data) and an output result corresponding to the input information. Then, when the diagnostic label and the differential label are output from the inference unit 112, the basis calculation unit 114 uses the basis model to generate basis information for each of the diagnostic label and the differentiation label, and can be similarly generated.

TCAV is an algorithm that calculates the importance of Concepts (concepts that can be easily understood by humans) to the predictions of a trained model. For example, the basis calculation unit 114 generates a plurality of pieces of input information by duplicating or modifying the input information (pathological image data), and assigns a plurality of input information to a model (description target model) for which basis information is to be generated. , and output a plurality of output information corresponding to each input information from the model to be explained. Then, the basis calculation unit 114 learns a basis model using a combination (pair) of each of the plurality of input information and each of the corresponding plurality of output information as learning data, and uses the target input information as a target. Generate a rationale model that is locally approximated by another interpretable model. Then, when the diagnostic label and the differential label are output from the inference unit 112, the basis calculation unit 114 uses the basis model to generate basis information for each of the diagnostic label and the differentiation label, and can be similarly generated.

Of course, the basis calculation unit 114 may calculate the basis for each of the diagnostic label and the differential label in the inference unit 112 based on algorithms other than Grad-Cam, LIME/SHAP, and TCAV described above.

G. Presentation of Inference Results The inference unit 112 outputs a diagnostic label inferred from the pathological image data acquired by the image acquisition unit 111 and a discrimination label corresponding to the diagnostic label. Further, the basis calculation unit 114 calculates the basis for inferring each of the diagnostic label and the differential label. ) is output. Then, the presentation processing unit 115 executes processing for displaying the outputs of the inference unit 112 and the basis calculation unit 114 on the screen of the display device 120 .

The presentation processing unit 115 divides the screen of the display device 120 into two, for example, and displays the diagnosis result of the pathological image data and its basis information (for example, see FIG. 7), the differential diagnosis and its Ground information (for example, see FIG. 8) may be displayed at the same time. In addition, the presentation processing unit 115 responds to a user input via the user interface (UI) unit 130, the diagnosis result of the pathological image data and its basis information (for example, see FIG. 7), the differential diagnosis and The basis information (for example, see FIG. 8) may be switched to be displayed.

On the screen of the display device 120, the pathologist displays the diagnosis result of the pathological image data and its basis information (for example, see FIG. 7), and the differential diagnosis and its basis information (for example, see FIG. 8). It is possible to make a differential diagnosis of the pathological image data by confirming the The display device 120 is a monitor used for observing images for diagnosis using digital pathological images, but from the viewpoint that it is a tool for making a diagnosis by direct visual observation by a pathologist, it is similar to a lens in an optical microscope. Equally important, a high-quality monitor with excellent color reproducibility and a fine pixel pitch is preferable.

FIG. 11 shows an example of screen transition for displaying pathological images on the display device 120 . In the example shown in FIG. 11, display mode 1 displays the original pathological image data (that is, as it is captured by the image capturing unit 111), and a heat map representing the basis of the diagnostic label is displayed on the original pathological image data. A display mode 2 for superimposed display, a display mode 3 for superimposed display of a heat map showing the basis of the differential label on the original pathological image data, and a heat map showing the basis of the diagnostic label and the differential label on the original pathological image data. It has a display mode 4 for simultaneous superimposition display. A user (such as a pathologist) may use the UI section 130 to select from a menu which of the display modes 1 to 4 is to be displayed on the screen. Alternatively, a toggle button or the like may be used to switch the display modes in order such as display mode 1→display mode 2→display mode 3→display mode 4→display mode 1→ .

H. Operation in the Inference Phase FIG. 9 shows the processing operation in the inference phase of the medical diagnostic system 100 in the form of a flow chart.

First, the image capturing unit 111 captures pathological image data acquired using a microscope system (described later) (step S901). The image acquisition unit 111 preprocesses the received pathological image data and outputs the preprocessed pathological image data to the inference unit 1112 .

The inference unit 112 uses the machine learning model trained by the learning unit 102 to perform a process of acquiring the image feature amount of the pathological image data (step S902). A pathological diagnosis of the included lesion is inferred (step S903), and a diagnosis label is output.

The reasoning unit 112 inquires the diagnostic label to the discrimination label information holding unit 113 and acquires the corresponding discrimination label from the discrimination label information (see FIG. 6, for example) (step S904). The inference unit 112 then outputs the diagnostic label inferred from the pathological image data and the discrimination label corresponding to the diagnostic label to the basis calculation unit 114 .

The basis calculation unit 114 calculates the basis for inference by the inference unit 112 for each of the inference label output from the inference unit 112 and the discrimination label corresponding to the inference label (step S905). The basis calculation unit 114 calculates an image (for example, see FIGS. 7 and 8) in which judgment basis for each of the inference label and the discrimination label is visualized, based on the Grad-CAM algorithm, for example.

Then, the presentation processing unit 115 executes processing for displaying the outputs of the inference unit 112 and the basis calculation unit 114 on the screen of the display device 120 (step S906).

On the screen of the display device 120, the pathologist displays the diagnosis result of the pathological image data and its basis information (for example, see FIG. 7), and the differential diagnosis and its basis information (for example, see FIG. 8). is observed to perform differential diagnosis of the pathological image data (step S907). At this time, the pathologist can alternately switch and display the image of the diagnosis result of the pathological image data and its basis information and the image of the differential diagnosis and its basis information through the UI unit 130 or the like. .

In the differential diagnosis, the pathologist compares the diagnosis result of the pathological image data and its basis information (for example, see FIG. 7) with the differential diagnosis and its basis information (for example, see FIG. 8). , negates either the diagnostic label or the differential label and accepts the other as the final diagnosis. The pathologist selects either the diagnostic label or the differential label through the UI unit 130 or the like and inputs the final diagnosis result to the medical diagnosis system 100 .

The medical diagnosis system 100 outputs the final diagnosis result by the pathologist (step S908), and ends this process. The medical diagnosis system 100 records the final diagnosis result by the pathologist in the corresponding patient's electronic medical record.

If the pathologist cannot accept either the diagnostic label or the differential label and cannot obtain a final diagnosis, the pathologist should collect the relevant patient's lesion and observe it under a microscope. It may be performed again to re-input the pathological image data to the medical diagnosis system 100 .

I. Configuration Example of Information Processing Apparatus In this Section I, an information processing apparatus capable of realizing one or both of the learning phase and the inference phase in the medical diagnosis system 100 will be described.

FIG. 10 shows a configuration example of the information processing apparatus 1000. As shown in FIG. The information processing apparatus 100 includes a CPU (Central Processing Unit) 1001, a RAM (Random Access Memory) 1002, a ROM (Read Only Memory) 1003, a large-capacity storage device 1004, a communication interface (IF) 1005, and an input/output device. An interface (IF) 1006 is provided. Each unit of information processing apparatus 1000 is interconnected by bus 1010 . The information processing apparatus 1000 is configured using, for example, a personal computer.

The CPU 1001 operates based on programs stored in the ROM 1003 or the large-capacity storage device 1004 to control the operation of each unit. For example, the CPU 1001 expands and executes various programs stored in the ROM 1003 or large-capacity storage device 1004 on the RAM 1002, and temporarily stores work data in the RAM 1002 during program execution.

The ROM 1003 nonvolatilely stores a boot program executed by the CPU 1001 when the information processing apparatus 1000 is started, and programs and data dependent on the hardware of the information processing apparatus 1000, such as BIOS (Basic Input Output System). .

The mass storage device 1004 is composed of a computer-readable recording medium such as a HDD (Hard Disk Drive) or an SSD (Solid State Drive). A large-capacity storage device 1004 nonvolatilely records programs executed by the CPU 1001 and data used by the programs in file format. Specifically, in the medical diagnosis system 100 shown in FIG. model parameters (e.g., neuron weighting coefficients) of the machine learning model that has been learned, a program that realizes a processing operation for the inference unit 112 to perform pathological diagnosis (differential diagnosis) of pathological image data using the learned machine learning model, differentiation Each data such as label information is recorded in a file format.

A communication interface 1005 is an interface for connecting the information processing apparatus 1000 to an external network 1050 (for example, the Internet). For example, the CPU 1001 receives data from another device or transmits data generated by the CPU 1001 to another device via the communication interface 1005 .

The input/output interface 1006 is an interface for connecting the input/output device 1060 to the information processing apparatus 1000 . For example, the CPU 1001 receives data from input devices such as a keyboard and mouse (none of which are shown) via the input/output interface 1006 . The CPU 1001 also transmits data to an output device such as a display, speaker, or printer (none of which is shown) via the input/output interface 1006 . Also, the input/output interface 1006 may function as a media interface for reading files such as programs and data recorded on a predetermined recording medium. The media referred to here include, for example, optical recording media such as DVD (Digital Versatile Disc) and PD (Phase change rewritable disc), magneto-optical recording media such as MO (Magneto-Optical disk), tape media, magnetic recording media, or Semiconductor memory and the like are included.

For example, when the information processing apparatus 1000 functions as the medical diagnosis system 100 in the learning phase and the inference phase, the CPU 1001 executes a program loaded on the RAM 1002 to perform , realizes the function of the presentation processing unit 115 . The large-capacity storage device 1004 also stores a program for realizing a processing operation for the learning unit 102 to learn a machine learning model, model parameters (neuron weighting coefficients) of the machine learning model learned by the learning unit 102, and so on. etc.), a program for realizing a processing operation for the inference unit 112 to perform pathological diagnosis (differential diagnosis) of pathological image data using a learned machine learning model, and various data such as differential label information. Note that the CPU 1001 reads files such as programs and data from the mass storage device 1004 and executes them. The data may be acquired or transferred to another device.

J. Microscope System FIG. 12 shows a configuration example of the microscope system of the present disclosure. A microscope system 5000 shown in FIG. 12 includes a microscope device 5100 , a control section 5110 and an information processing section 5120 . A microscope device 5100 includes a light irradiation section 5101 , an optical section 5102 , and a signal acquisition section 5103 . The microscope device 5100 may further include a sample placement section 5104 on which the biological sample S is placed. The configuration of the microscope apparatus is not limited to that shown in FIG. 12. For example, the light irradiation unit 5101 may exist outside the microscope apparatus 5100. It may be used as the unit 5101 . Further, the light irradiation section 5101 may be arranged such that the sample mounting section 5104 is sandwiched between the light irradiation section 5101 and the optical section 5102, and may be arranged on the side where the optical section 5102 exists, for example. The microscope apparatus 5100 may be configured for one or more of bright field observation, phase contrast observation, differential interference observation, polarization observation, fluorescence observation, and dark field observation.

The microscope system 5000 may be configured as a so-called WSI (Whole Slide Imaging) system or a digital pathology system, and may be used for pathological diagnosis. Microscope system 5000 may also be configured as a fluorescence imaging system, in particular a multiplex fluorescence imaging system.

For example, microscope system 5000 may be used to perform intraoperative pathology or remote pathology. In the intraoperative pathological diagnosis, while the surgery is being performed, the microscope device 5100 acquires data of the biological sample S obtained from the subject of the surgery, and transfers the data to the information processing unit 5120. can send. In the remote pathological diagnosis, the microscope device 5100 can transmit the acquired data of the biological sample S to the information processing unit 5120 located in a place (another room, building, or the like) away from the microscope device 5100 . In these diagnoses, the information processing section 5120 receives and outputs the data. A user of the information processing unit 5120 can make a pathological diagnosis based on the output data.

(Biological sample)
The biological sample S may be a sample containing a biological component. The biological components may be tissues, cells, liquid components of a living body (blood, urine, etc.), cultures, or living cells (cardiomyocytes, nerve cells, fertilized eggs, etc.).
The biological sample may be a solid, a specimen fixed with a fixative such as paraffin, or a solid formed by freezing. The biological sample can be a section of the solid. A specific example of the biological sample is a section of a biopsy sample.

The biological sample may have been subjected to processing such as staining or labeling. The treatment may be staining for indicating the morphology of biological components or for indicating substances (surface antigens, etc.) possessed by biological components, examples of which include HE (Hematoxylin-Eosin) staining and immunohistochemistry staining. be able to. The biological sample may be treated with one or more reagents, and the reagents may be fluorescent dyes, chromogenic reagents, fluorescent proteins, or fluorescently labeled antibodies.

The specimen may be prepared from a specimen or tissue sample collected from a human body for the purpose of pathological diagnosis, clinical examination, or the like. Moreover, the specimen is not limited to the human body, and may be derived from animals, plants, or other materials. The specimen may be the type of tissue used (such as an organ or cell), the type of target disease, the subject's attributes (such as age, sex, blood type, or race), or the subject's lifestyle. The properties differ depending on habits (for example, eating habits, exercise habits, smoking habits, etc.). The specimens may be managed with identification information (bar code information, QR code (trademark) information, etc.) that allows each specimen to be identified.

(light irradiation part)
The light irradiation unit 5101 is a light source for illuminating the biological sample S and an optical unit for guiding the light irradiated from the light source to the specimen. The light source may irradiate the biological sample with visible light, ultraviolet light, or infrared light, or a combination thereof. The light source may be one or more of halogen lamps, laser light sources, LED lamps, mercury lamps, and xenon lamps. A plurality of types and/or wavelengths of light sources may be used in fluorescence observation, and may be appropriately selected by those skilled in the art. The light irradiator may have a transmissive, reflective, or episcopic (coaxial or lateral) configuration.

(Optical part)
The optical section 5102 is configured to guide the light from the biological sample S to the signal acquisition section 5103 . The optical section can be configured to allow the microscope device 5100 to observe or image the biological sample S.
Optical section 5102 may include an objective lens. The type of objective lens may be appropriately selected by those skilled in the art according to the observation method. Also, the optical section may include a relay lens for relaying the image magnified by the objective lens to the signal acquisition section. The optical unit may further include optical components other than the objective lens and the relay lens, an eyepiece lens, a phase plate, a condenser lens, and the like.
In addition, the optical section 5102 may further include a wavelength separation section configured to separate light having a predetermined wavelength from the light from the biological sample S. The wavelength separation section can be configured to selectively allow light of a predetermined wavelength or wavelength range to reach the signal acquisition section. The wavelength separator may include, for example, one or more of a filter that selectively transmits light, a polarizing plate, a prism (Wollaston prism), and a diffraction grating. The optical components included in the wavelength separation section may be arranged, for example, on the optical path from the objective lens to the signal acquisition section. The wavelength separation unit is provided in the microscope apparatus when fluorescence observation is performed, particularly when an excitation light irradiation unit is included. The wavelength separator may be configured to separate fluorescent light from each other or white light and fluorescent light.

(Signal acquisition part)
The signal acquisition unit 5103 can be configured to receive light from the biological sample S and convert the light into an electrical signal, particularly a digital electrical signal. The signal acquisition unit may be configured to acquire data on the biological sample S based on the electrical signal. The signal acquisition unit may be configured to acquire data of an image (image, particularly a still image, a time-lapse image, or a moving image) of the biological sample S, particularly an image magnified by the optical unit. It can be configured to acquire data. The signal acquisition unit includes one or more imaging elements, such as CMOS or CCD, having a plurality of pixels arranged one-dimensionally or two-dimensionally. The signal acquisition unit may include an image sensor for acquiring a low-resolution image and an image sensor for acquiring a high-resolution image, or an image sensor for sensing such as AF and an image sensor for image output for observation. and may include In addition to the plurality of pixels, the image sensor includes a signal processing unit (including one, two, or three of CPU, DSP, and memory) that performs signal processing using pixel signals from each pixel, and an output control unit for controlling output of image data generated from the pixel signals and processed data generated by the signal processing unit. Furthermore, the imaging device may include an asynchronous event detection sensor that detects, as an event, a change in brightness of a pixel that photoelectrically converts incident light exceeding a predetermined threshold. An imaging device including the plurality of pixels, the signal processing section, and the output control section may preferably be configured as a one-chip semiconductor device.

(control part)
The control unit 5110 controls imaging by the microscope device 5100 . The control unit 5110 can drive the movement of the optical unit 5102 and/or the sample placement unit 5104 to adjust the positional relationship between the optical unit 5102 and the sample placement unit 5104 for imaging control. The control unit 5110 can move the optical unit 5102 and/or the sample placement unit 5104 in a direction toward or away from each other (for example, the optical axis direction of the objective lens). Also, the control section 5110 may move the optical section and/or the sample mounting section 5104 in any direction on a plane perpendicular to the optical axis direction. The control unit 5110 may control the light irradiation unit 5101 and/or the signal acquisition unit 5103 for imaging control.

(Sample placement section)
The sample mounting section 5104 may be configured such that the position of the biological sample on the sample mounting section 5104 can be fixed, and may be a so-called stage. The sample mounting section 5104 can be configured to move the position of the biological sample in the optical axis direction of the objective lens and/or in a direction perpendicular to the optical axis direction.

(Information processing department)
The information processing section 5120 can acquire data (such as imaging data) acquired by the microscope device 5100 from the microscope device 5100 . The information processing section 5120 can perform image processing on captured data. The image processing may include color separation processing. The color separation process is a process of extracting data of light components of a predetermined wavelength or wavelength range from the captured data to generate image data, or removing data of light components of a predetermined wavelength or wavelength range from the captured data. It can include processing and the like. Further, the image processing may include autofluorescence separation processing for separating the autofluorescence component and dye component of the tissue section, and fluorescence separation processing for separating the wavelengths between dyes having different fluorescence wavelengths. In the autofluorescence separation processing, out of the plurality of specimens having the same or similar properties, autofluorescence signals extracted from one may be used to remove autofluorescence components from image information of the other specimen. .
The information processing section 5120 may transmit data for imaging control to the control section 5110, and the control section 5110 receiving the data may control imaging by the microscope apparatus 5100 according to the data.

The information processing section 5120 may be configured as an information processing device such as a general-purpose computer, and may include a CPU, a RAM, and a ROM. The information processing section may be included in the housing of the microscope device 5100 or may be outside the housing. Various processing or functions by the information processing section 5120 may be realized by a server computer or cloud connected via a network.

A method of imaging the biological sample S by the microscope device 5100 may be appropriately selected by a person skilled in the art according to the type of the biological sample, the purpose of imaging, and the like. An example of the imaging method will be described below.

One example of an imaging scheme is as follows. The microscope device 5100 can first identify an imaging target region. The imaging target region may be specified so as to cover the entire region where the biological sample exists, or a target portion (target tissue section, target cell, or target lesion portion) of the biological sample. ) may be specified to cover Next, the microscope device 5100 divides the imaging target region into a plurality of divided regions of a predetermined size, and the microscope device 5100 sequentially images each divided region. As a result, an image of each divided area is acquired.
As shown in FIG. 13, the microscope device 5100 identifies an imaging target region R that covers the entire biological sample S. As shown in FIG. Then, the microscope device 5100 divides the imaging target region R into 16 divided regions. Then, the microscope device 5100 can image the divided region R1, and then any region included in the imaging target region R, such as a region adjacent to the divided region R1. Then, image capturing of the divided areas is performed until there are no unimaged divided areas. Areas other than the imaging target area R may also be imaged based on the captured image information of the divided areas.
After imaging a certain divided area, the positional relationship between the microscope device 5100 and the sample mounting section 5104 is adjusted in order to image the next divided area. The adjustment may be performed by moving the microscope device 5100, moving the sample placement section 5104, or moving both of them. In this example, the image capturing device that captures each divided area may be a two-dimensional image sensor (area sensor) or a one-dimensional image sensor (line sensor). The signal acquisition section may capture an image of each divided area via the optical section. In addition, the imaging of each divided region may be performed continuously while moving the microscope device 5100 and/or the sample mounting unit 5104, or when imaging each divided region, the microscope device 5100 and/or the sample mounting unit Movement of 5104 may be stopped. The imaging target area may be divided so that the divided areas partially overlap each other, or the imaging target area may be divided so that the divided areas do not overlap. Each divided area may be imaged multiple times by changing imaging conditions such as focal length and/or exposure time.
Further, the information processing section 5120 can generate image data of a wider area by synthesizing a plurality of adjacent divided areas. By performing the synthesis processing over the entire imaging target area, it is possible to obtain an image of a wider area of the imaging target area. Also, image data with lower resolution can be generated from the image of the divided area or the image subjected to the synthesis processing.

Other examples of imaging schemes are as follows. The microscope device 5100 can first identify an imaging target region. The imaging target region may be specified so as to cover the entire region where the biological sample exists, or the target portion (target tissue section or target cell-containing portion) of the biological sample. may be specified. Next, the microscope device 5100 scans a partial region (also referred to as a “divided scan region”) of the imaging target region in one direction (also referred to as a “scanning direction”) within a plane perpendicular to the optical axis. Take an image. After the scanning of the divided scan area is completed, the next divided scan area next to the scan area is scanned. These scanning operations are repeated until the entire imaging target area is imaged.
As shown in FIG. 14, the microscope device 5100 specifies a region (gray portion) in which a tissue section exists in the biological sample S as an imaging target region Sa. Then, the microscope device 5100 scans the divided scan area Rs in the imaging target area Sa in the Y-axis direction. After completing scanning of the divided scan region Rs, the microscope device 5100 next scans an adjacent divided scan region in the X-axis direction. This operation is repeated until scanning is completed for the entire imaging target area Sa.
The positional relationship between the microscope device 5100 and the sample placement section 5104 is adjusted for scanning each divided scan area and for imaging the next divided scan area after imaging a certain divided scan area. The adjustment may be performed by moving the microscope device 5100, moving the sample placement section 5104, or moving both of them. In this example, the imaging device that captures each divided scan area may be a one-dimensional imaging device (line sensor) or a two-dimensional imaging device (area sensor). The signal acquisition section may capture an image of each divided area via an enlarging optical system. Also, the imaging of each divided scan area may be performed continuously while moving the microscope device 5100 and/or the sample mounting section 5104 . The imaging target area may be divided so that the divided scan areas partially overlap each other, or the imaging target area may be divided so that the divided scan areas do not overlap. Each divided scan area may be imaged multiple times by changing imaging conditions such as focal length and/or exposure time.
In addition, the information processing section 5120 can generate image data of a wider area by synthesizing a plurality of adjacent divided scan areas. By performing the synthesizing process over the entire imaging target area, it is possible to obtain an image of a wider area of the imaging target area. Further, image data with lower resolution can be generated from the image of the divided scan area or the image subjected to the synthesis processing.

The information processing unit 5120 is basically a device that realizes the inference mode operation in the medical diagnosis system 100 shown in FIG. 1, and can be configured using the information processing device 1000 shown in FIG. Of course, the information processing section 5120 may also have a function of operating in a learning mode, and may perform re-learning or additional learning of the machine learning model to be used. The information processing unit 5120 infers a disease from the pathological image data captured by the microscope device 5100, outputs a diagnostic label and a differential label corresponding to the diagnostic label, and calculates the grounds for each of the diagnostic label and the differential label, Output information such as a heat map that represents each basis. Further, the information processing unit 5120 includes an input device corresponding to the UI unit 130, and the final diagnosis by the pathologist (for example, the pathologist's findings such as the selection result of one of the diagnostic label and the differential label) and observation data ( For example, an input of a pathologist's comment on a pathological image such as "highly diffuse" is accepted.

The information processing section 5120 records the pathological image data captured by the microscope device 5100 in the mass storage device 1004 . In addition, the information processing unit 5120 records the diagnosis result inferred from the pathological image data, and the pathologist's findings and observation data on the pathological image in association with the pathological image data. The information processing unit 5120 may store test values such as blood, pathological image data, pathologist's findings and observation data for each patient in the mass storage device 1004, for example, in the form of an electronic medical record.

The present disclosure has been described in detail above with reference to specific embodiments. However, it is obvious that those skilled in the art can modify or substitute the embodiments without departing from the gist of the present disclosure.

In the present specification, an embodiment in which the present disclosure is applied to analysis of pathological images has been mainly described, but the gist of the present disclosure is not limited to this. The present disclosure can be similarly applied to diagnosis of various medical images such as X-ray images, CT (Computed Tomography), MRI (Magnetic Resonance Imaging), and endoscopic images.

In short, the present disclosure has been described in the form of examples, and the contents of this specification should not be construed in a restrictive manner. In order to determine the gist of the present disclosure, the scope of the claims should be considered.

It should be noted that the present disclosure can also be configured as follows.

(1) An information processing device that processes information about medical images,
an inference unit that infers a first disease that is correct for the medical image and a second disease that is a target of differential diagnosis related to the first disease;
a basis calculation unit that calculates basis for the first disease and basis for the second disease from the medical image;
An information processing device comprising:

(2) the inference unit infers the first disease from the medical image using a trained machine learning model;
The information processing apparatus according to (1) above.

(3) The machine learning model is learned using learning data consisting of a data set combining medical images and correct diseases.
The information processing apparatus according to (2) above.

(4) The inference unit identifies a second disease to be subjected to differential diagnosis related to the first disease inferred from the medical image based on differential label information indicating a disease to be differentially diagnosed for each disease. ,
The information processing apparatus according to any one of (1) to (3) above.

(5) the inference unit infers a first disease and a second disease using a trained neural network model;
The basis calculation unit causes the neural network model to infer the basis of each of the first disease and the second disease.
The information processing apparatus according to any one of (1) to (4) above.

(6) The basis calculation unit infers a portion of the original medical image that affects each class by reversely tracing the gradient from the label that is the inference result of class classification in the output layer of the neural network model.
The information processing apparatus according to (5) above.

(7) The basis calculation unit calculates each of the first disease and the second disease based on the amount of change in the output when the feature amount of the medical image data input to the neural network model is perturbed. Infer each reason,
The information processing apparatus according to (5) above.

(8) further comprising a presentation unit that presents the inference result of the inference unit and the calculation result of the basis calculation unit;
The information processing apparatus according to any one of (1) to (7) above.

(9) The presentation unit superimposes and presents the inference result of the inference unit and the calculation result of the basis calculation unit on the medical image.
The information processing apparatus according to (8) above.

(10) The presentation unit switches and displays the grounds for inferring the first disease and the grounds for inferring the second disease based on user input.
The information processing apparatus according to any one of (8) and (9) above.

(11) The medical image is pathological image data obtained by microscopically observing a lesion.
The information processing apparatus according to any one of (1) to (10) above.

(12) An information processing method for processing information about medical images,
a first inference step of inferring a first disease that is correct for the medical image;
a second inference step of inferring a second disease of interest for differential diagnosis associated with said first disease;
a first basis calculation step of calculating basis of the first disease from the medical image;
a second basis calculation step of calculating the basis of the second disease from the medical image;
An information processing method comprising:

(13) A computer program written in computer readable form to process information about medical images on a computer, the computer comprising:
an inference unit that infers a first disease that is correct for the medical image and a second disease that is a target of differential diagnosis related to the first disease;
an evidence calculation unit that calculates the evidence of the first disease and the evidence of the second disease from the medical image;
A computer program that acts as a

(14) a learning unit in which the machine learning model learns such that the machine learning model infers a disease from medical image data;
Using the machine learning model learned by the learning unit, infer a first disease that is correct for the medical image, and a second disease that is a target for differential diagnosis related to the first disease an inference unit that infers a disease;
a basis calculation unit that calculates basis for the first disease and basis for the second disease from the medical image;
a display device;
a presentation unit that presents the inference result of the inference unit and the calculation result of the basis calculation unit on the display device;
A medical diagnostic system comprising:

(15) an inference unit that infers input data and outputs a first label using a trained machine learning model;
a holding unit holding information about a second label to be distinguished from the first label;
a calculation unit that calculates a basis for inferring the first label and the second label by the machine learning model;
An information processing device comprising:

DESCRIPTION OF SYMBOLS 100... Medical diagnosis system 101... Learning data accumulation part 102... Learning part 103... Model parameter holding part 111... Image acquisition part 112... Inference part 113... Differentiation label information holding part 114... Evidence calculation part 115... Presentation Processing unit 116 Identification label update unit 120 Display device 130 UI unit 200 Data adjustment device 301 Machine learning model 311 Influence evaluation unit 312 Learning state determination unit 313 Additional data generation unit 401 ... Generator 402 ... Discriminator 502, 504, 506 ... Convolutional layer output 503, 505 ... Pooling layer output 507 ... Convolutional layer output 508 ... Fully connected layer 509 ... Output layer 1000 ... Information processing device 1001 ... CPU, 1002 RAM
1003... ROM, 1004... Mass storage device 1005... Communication interface, 1006... Input/output interface 1010... Bus, 1050... External network 1060... Input/output device 1200... DP system, 1210... Image capture device 1211... Glass slide, 1212 ... Microscope 1213 ... Slide 1220 ... Image display device 1230 ... Diagnosis device 1240 ... Diagnosis device 5000 ... Microscope system 5100 ... Microscope device 5101 ... Light irradiation part 5102 ... Optical part 5103 ... Signal acquisition part 5104 ... Sample mount placing unit 5110 control unit 5120 information processing unit

Claims (14)

An information processing device for processing information related to medical images,
an inference unit that infers a first disease that is correct for the medical image and a second disease that is a target of differential diagnosis related to the first disease;
a basis calculation unit that calculates basis for the first disease and basis for the second disease from the medical image;
An information processing device comprising: The inference unit infers the first disease from the medical image using a trained machine learning model.
The information processing device according to claim 1 . The machine learning model is learned using learning data consisting of a data set combining medical images and correct diseases,
The information processing apparatus according to claim 2. The inference unit identifies a second disease to be subjected to differential diagnosis related to the first disease inferred from the medical image based on differential label information indicating a disease to be differentially diagnosed for each disease.
The information processing device according to claim 1 . The inference unit infers a first disease and a second disease using a trained neural network model,
The basis calculation unit causes the neural network model to infer the basis of each of the first disease and the second disease.
The information processing device according to claim 1 . The basis calculation unit infers a portion of the original medical image that affects each class by reversely tracing the gradient from the label that is the inference result of class classification in the output layer of the neural network model.
The information processing device according to claim 5 . The basis calculation unit calculates the basis for each of the first disease and the second disease based on an output variation amount when the feature amount of the medical image data input to the neural network model is perturbed. infer,
The information processing device according to claim 5 . further comprising a presentation unit that presents the inference result of the inference unit and the calculation result of the basis calculation unit;
The information processing device according to claim 1 . The presentation unit superimposes and presents the inference result of the inference unit and the calculation result of the basis calculation unit on the medical image.
The information processing apparatus according to claim 8 . The presentation unit switches and displays the grounds for inferring the first disease and the grounds for inferring the second disease based on user input.
The information processing apparatus according to claim 8 . The medical image is pathological image data obtained by microscopically observing a lesion,
The information processing device according to claim 1 . An information processing method for processing information about medical images,
a first inference step of inferring a first disease that is correct for the medical image;
a second inference step of inferring a second disease of interest for differential diagnosis associated with said first disease;
a first basis calculation step of calculating basis of the first disease from the medical image;
a second basis calculation step of calculating the basis of the second disease from the medical image;
An information processing method comprising: A computer program written in computer readable form to process information relating to medical images on a computer, said computer comprising:
an inference unit that infers a first disease that is correct for the medical image and a second disease that is a target of differential diagnosis related to the first disease;
an evidence calculation unit that calculates the evidence of the first disease and the evidence of the second disease from the medical image;
A computer program that acts as a a learning unit in which the machine learning model learns so that the machine learning model infers a disease from medical image data;
Using the machine learning model learned by the learning unit, infer a first disease that is correct for the medical image, and a second disease that is a target for differential diagnosis related to the first disease an inference unit that infers a disease;
a basis calculation unit that calculates basis for the first disease and basis for the second disease from the medical image;
a display device;
a presentation unit that presents the inference result of the inference unit and the calculation result of the basis calculation unit on the display device;
A medical diagnostic system comprising:

Images