JP7828751B2 - Information processing device and program - Google Patents

Description

The embodiments disclosed herein and in the drawings relate to information processing devices and programs.

Conventionally, in medical imaging diagnostic equipment, technologies are known that use trained models created through machine learning to support positioning, setting imaging parameters, and diagnosis. For example, it is possible to infer the probability of a specific diagnosis (disease name, etc.) based on medical information such as medical images, and present the inference along with its rationale. However, in this case, the validity of the inference result is judged by a human.

Special Publication No. 2013-48798

Christopher M. Bishop, "Pattern Recognition and Machine Learning," (USA), 1st edition, Springer, 2006, pp. 225-290.

One of the problems that the embodiments disclosed in this specification and drawings aim to solve is to provide an information processing device and program that can support the validity judgment of inference results from a trained model. However, the problems that the embodiments disclosed in this specification and drawings aim to solve are not limited to the above problem. Problems corresponding to the effects of each configuration shown in the embodiments described later can also be positioned as other problems.

The information processing device of this embodiment comprises an acquisition unit and a determination unit. The acquisition unit acquires the inference result of an inference unit that performs inference based on one or more input pieces of information, and first supporting information indicating the basis for the inference result. The determination unit determines the validity of the inference result based on the first supporting information.

Figure 1 is a block diagram showing an example of the configuration of a medical information processing device according to an embodiment. Figure 2 shows an example of a positioning image according to the embodiment. Figure 3 schematically shows an example of an inference result using a pre-trained model. Figure 4 schematically illustrates an example of inference processing using a pre-trained model. Figure 5 is a diagram illustrating an example of the operation of the interpretation function according to this embodiment. Figure 6 is a diagram illustrating an example of the operation of the judgment function according to the embodiment. Figure 7 shows an example of the calculation result of the proportion of background pixels in relation to the region of interest according to this embodiment. Figure 8 illustrates an example of the process for determining whether a region of interest other than the region of greatest interest according to the embodiment matches knowledge information. Figure 9 is a diagram illustrating an example of the operation of the modification function according to the embodiment. Figure 10 schematically illustrates an example of inference processing using a pre-trained model. Figure 11 is a diagram illustrating an example of the operation of the interpretation function according to the embodiment. Figure 12 is a diagram illustrating an example of the operation of the judgment function according to the embodiment. Figure 13 is a flowchart showing an example of a process performed by the medical information processing device according to the embodiment.

The following describes in detail embodiments of the information processing device and program with reference to the drawings. However, the information processing device and program according to this application are not limited to the embodiments shown below. Furthermore, the embodiments can be combined with other embodiments or prior art to the extent that there is no inconsistency in the processing content.

(First embodiment)
Figure 1 is a block diagram showing an example of the configuration of a medical information processing device 3 according to an embodiment. The medical information processing device 3 is an example of an information processing device. For example, as shown in Figure 1, the medical information processing device 3 according to the embodiment is included in a medical information processing system 100 that is communicably connected to a medical image diagnostic device 1 and a medical image storage device 2 via a network 200.

Here, each device included in the medical information processing system 100 is capable of communicating with each other directly or indirectly via, for example, a hospital LAN (Local Area Network) installed within the hospital. Note that devices other than those shown may also be connected to the medical information processing system 100 shown in Figure 1 in a communication-enabled manner.

For example, the medical information processing system 100 may include various systems such as a hospital information system (HIS), a radiology information system (RIS), a diagnostic report system, a picture archiving and communication system (PACS), and a laboratory information system (LIS).

The medical imaging diagnostic device 1 captures images of the subject and collects medical images. The medical imaging diagnostic device 1 then transmits the collected medical images to the medical image storage device 2 and the medical information processing device 3. For example, the medical imaging diagnostic device 1 may be an MRI (Magnetic Resonance Imaging) device, an X-ray diagnostic device, an X-ray CT (Computed Tomography) device, an ultrasound diagnostic device, a SPECT (Single Photon Emission Computed Tomography) device, a PET (Positron Emission Computed Tomography) device, etc.

The medical image storage device 2 stores various medical images related to the subject. Specifically, the medical image storage device 2 acquires medical images from the medical image diagnostic device 1 via the network 200 and stores these medical images in its internal memory circuit.

For example, the medical image storage device 2 is implemented using computer equipment such as a server or workstation. Alternatively, for example, the medical image storage device 2 may be implemented using PACS (Picture Archiving and Communication System) and store medical images in a format compliant with DICOM (Digital Imaging and Communications in Medicine).

The medical information processing device 3 acquires various types of information from the medical image diagnostic device 1 and the medical image storage device 2, and performs various information processing using the acquired information. For example, the medical information processing device 3 can be implemented using computer equipment such as a server, workstation, personal computer, or tablet terminal. Alternatively, the medical information processing device 3 may be a console device for controlling the medical image diagnostic device 1.

As shown in Figure 1, the medical information processing device 3 includes a communication interface 31, a storage circuit 32, an input interface 33, a display 34, and a processing circuit 35.

The communication interface 31 is connected to the processing circuit 35 and controls communication between the medical information processing system 100 and each device. Specifically, the communication interface 31 receives various types of information from each device and outputs the received information to the processing circuit 35. For example, the communication interface 31 can be implemented using a network card, network adapter, NIC (Network Interface Controller), etc.

The memory circuit 32 is connected to the processing circuit 35 and stores various types of data. For example, the memory circuit 32 can be implemented using semiconductor memory elements such as RAM (Random Access Memory) or flash memory, or by using a hard disk, optical disc, etc.

Specifically, the memory circuit 32 stores various programs that the processing circuit 35 reads and executes to realize various functions.

Furthermore, the memory circuit 32 stores various information received from the medical image diagnostic device 1 and the medical image storage device 2, information input via the input interface 33, and processing results from the medical information processing device 3. For example, as shown in Figure 1, the memory circuit 32 stores the learned model 321, the knowledge database 322, the interpretation table 323, the judgment table 324, and the modification table 325.

The trained model 321 is a model generated and trained through machine learning using information acquired from the medical image diagnostic device 1, the medical image storage device 2, etc., as training data. The medical information processing device 3 performs inference processing using the trained model 321.

The trained model 321 may be generated by the processing circuit 35, or by a device other than the medical information processing device 3. For example, the trained model 321 may be generated by an external device located outside the medical information processing system 100.

Knowledge DB322 is a database that stores information including findings based on clinical research results, various guidelines, and simulations based on that information. Knowledge DB322 contains knowledge information that serves as a criterion for judging the validity of inference results from the trained model 321.

The interpretation table 323 holds information used to interpret the rationale information that provides the basis for the inference results of the trained model 321. For example, the interpretation table 323 is a data table that associates the processing content performed to interpret each type of rationale information.

Furthermore, the information used to interpret the supporting information is not limited to tables. For example, a pre-trained model (including deep learning) that takes supporting information as input and outputs interpreted information based on that input may also be used.

The decision table 324 holds information used to judge the validity of the inference results from the trained model 321. For example, the decision table 324 is a data table that associates referenced knowledge information with each type of evidence information or interpretive information. Here, the associated knowledge information serves as an indicator for judging the validity of the inference results.

Furthermore, the aforementioned Knowledge DB 322 may store knowledge information in association with supporting information or interpretive information. In this case, the judgment table 324 may be unnecessary.

The correction table 325 holds information used to correct the inference results from the trained model 321. For example, the correction table 325 is a data table that associates pairs of evidence information, interpretation information, and knowledge information with correction processes for modifying the inference results related to those pairs.

Furthermore, the information used to correct invalid inference results is not limited to tables. For example, a trained model, trained through machine learning to output corrected inference results when given invalid inference results as input, may also be used.

Furthermore, some or all of the trained model 321, knowledge database 322, interpretation table 323, judgment table 324, and modification table 325 may be stored in other information processing devices accessible by the medical information processing device 3.

The input interface 33 is connected to the processing circuit 35 and receives various instructions and information inputs from the operator. Specifically, the input interface 33 converts the input operations received from the operator into electrical signals and outputs them to the processing circuit 35.

For example, the input interface 33 can be implemented using a trackball, switch buttons, a mouse, a keyboard, a touchpad for input operations by touching the operating surface, a touchscreen integrating a display screen and a touchpad, a non-contact input circuit using an optical sensor, and an audio input circuit, etc.

Furthermore, in this specification, the input interface 33 is not limited to those equipped with physical operating components such as a mouse or keyboard. For example, an electrical signal processing circuit that receives electrical signals corresponding to input operations from an external input device separate from the device and outputs these electrical signals to a control circuit is also included as an example of the input interface 33.

The display 34 is connected to the processing circuit 35 and displays various information and images. Specifically, the display 34 converts the information and image data sent from the processing circuit 35 into electrical signals for display and outputs them. For example, the display 34 can be implemented as an LCD monitor, a CRT (Cathode Ray Tube) monitor, a touch panel, etc.

The processing circuit 35 controls the operation of the medical information processing device 3 in response to input operations received from the operator via the input interface 33. For example, the processing circuit 35 is implemented by a processor. As shown in Figure 1, the processing circuit 35 performs control functions 351, inference functions 352, acquisition functions 353, interpretation functions 354, judgment functions 355, and correction functions 356.

Here, the control function 351 is an example of the presentation unit. The inference function 352 is an example of the inference unit. The acquisition function 353 is an example of the acquisition unit. The generation function 357 is an example of the generation unit. The interpretation function 354 and the judgment function 355 are examples of the judgment unit. The modification function 356 is an example of the modification unit.

The control function 351 controls the system to execute processing in response to various requests input via the input interface 33. For example, the control function 351 controls the transmission and reception of medical images via the communication interface 31, the storage of various information in the memory circuit 32, and the display of information (e.g., medical images, processing results from various functions, etc.) on the display 34.

For example, the control function 351 acquires data such as positioning image data and medical image data for determining the imaging position of the subject from the medical imaging diagnostic device 1 and stores it in the memory circuit 32. Furthermore, for example, the control function 351 controls the medical imaging diagnostic device 1 to execute processing via a GUI and to display the processing results of each function on the display 34.

The inference function 352 performs inference using the trained model 321. Specifically, the inference function 352 inputs input data into the trained model 321, causing the trained model 321 to perform inference processing.

For example, the inference function 352 inputs a positioning image (an example of input data) to the trained model 321. The trained model 321 then outputs the imaging position of the subject in the MRI device, based on the input positioning image.

In this case, the imaging position is the inference result of the pre-trained model 321. Thus, the inference function 352 can assist in determining the imaging position of the subject in the MRI device by having the pre-trained model 321 output the imaging position based on the positioning image.

Furthermore, for example, the inference function 352 inputs user input information (an example of input data), such as information regarding the desired image quality and information regarding the imaging area, for medical images acquired by the MRI device, into the trained model 321. Then, the trained model 321 outputs imaging parameters for determining the imaging conditions of the subject in the MRI device, in response to the user input information.

In this case, the imaging parameters are the inference results from the pre-trained model 321. Thus, the inference function 352 can assist in determining imaging parameters by having the pre-trained model 321 output imaging parameters based on user input information. Information regarding the imaging area may be obtained from sources such as RIS.

Furthermore, for example, the inference function 352 inputs medical image data (an example of input data) acquired by an MRI device into the trained model 321. The trained model 321 then outputs a diagnostic name indicating the symptoms, etc., present in the subject, in response to the input medical image data.

In this case, the diagnostic name is the inference result of the pre-trained model 321. Thus, the inference function 352 can provide diagnostic support by having the pre-trained model 321 output a diagnostic name based on medical image data. Furthermore, the above-described processing related to the MRI device, such as positioning support, imaging parameter determination support, and diagnostic support, can also be performed similarly for other medical imaging diagnostic devices 1, such as X-ray CT scanners.

The acquisition function 353 acquires supporting information that shows the basis for the inference result. Specifically, the acquisition function 353 acquires the inference result output by the trained model 321 and supporting information that shows the basis for the said inference result.

Here, the supporting information refers to, for example, information representing candidate inference results from the trained model 321 and an index indicating the likelihood of those candidates. The supporting information may also represent information representing features that contributed to the derivation of the candidate inference results and an index (hereinafter also referred to as the contribution level) indicating the degree to which those features contributed to the derivation of the candidate inference results. Furthermore, the supporting information may be a combination of the former and the latter. Note that the candidate inference results may also include the final inference result.

The method for obtaining the rationale information is not particularly limited. For example, if the trained model 321 outputs rationale information along with the inference results, the acquisition function 353 will acquire the rationale information obtained from the trained model 321. Alternatively, the acquisition function 353 may acquire the rationale information by performing analysis on the output results (inference results) of the trained model 321, such as identifying the contributing features. Furthermore, the acquisition function 353 may acquire multiple pieces of rationale information.

For example, when the trained model 321 performs inference to assist positioning in an MRI device, the acquisition function 353 acquires the imaging position derived as an inference result, along with supporting information indicating the basis for this imaging position, from the trained model 321.

In this case, the supporting information would be, for example, candidate imaging locations identified based on the region of interest on the positioning image in the MRI device, and the probability (an example of accuracy) that these candidates are the desired imaging locations. Alternatively, the supporting information in this case might also be candidate imaging locations identified based on a feature of interest (e.g., shape) on the positioning image in the MRI device, and the probability that these candidates are the desired imaging locations.

Furthermore, for example, when inference is performed using the trained model 321 to assist in determining imaging parameters in an MRI device, the imaging parameters derived as a result of the inference, along with supporting information indicating the basis for the derivation of these imaging parameters, are obtained from the trained model 321. In this case, the supporting information would include, for example, the conditions that contributed to the derivation of the imaging parameters, and the probability that these imaging parameters are necessary for achieving the desired image quality.

Furthermore, for example, when inference for diagnostic support is performed based on medical image data acquired by an MRI device using the trained model 321, the diagnostic name derived as an inference result and the supporting information indicating the basis for this diagnostic name are obtained from the trained model 321. In this case, the supporting information would include, for example, the image findings that contributed to the derivation of the diagnostic name, and the probability that the diagnosis of the subject being diagnosed is that particular diagnosis.

The interpretation function 354 performs interpretation processing to interpret the underlying information. Here, interpretation processing is the process of converting the acquired underlying information into a form that can be directly compared with the knowledge information, based on the underlying information, when direct comparison with the knowledge information is not possible. Note that interpretation processing is unnecessary if the underlying information can be directly compared with the knowledge information.

For example, the interpretation function 354 refers to the interpretation table 323 and executes processing associated with the underlying information acquired by the acquisition function 353. In other words, it executes processing to interpret the underlying information. The interpretation function 354 acquires the processing result of this process as interpretation information.

The judgment function 355 determines whether the inference result from the inference function 352 is valid, based on the judgment table 324. For example, the judgment function 355 refers to the judgment table 324 and identifies knowledge information from the knowledge database 322 that corresponds to the basis information acquired by the acquisition function 353 or the interpretation information acquired by the interpretation function 354. Then, the judgment function 355 determines the validity of the inference result based on the basis information or interpretation information and the identified knowledge information.

If the supporting information or interpretive information matches the identified knowledge information, the judgment function 355 determines that the inference result is valid. Conversely, if the supporting information or interpretive information does not match the identified knowledge information, the judgment function 355 determines that the inference result is invalid. Furthermore, the judgment function 355 may also determine the validity of the inference result after it has been corrected by the correction function 356, which will be described later.

The correction function 356 corrects the inference result that the judgment function 355 deemed invalid, based on the correction table 325. For example, the correction function 356 refers to the correction table 325 and corrects the inference result by executing a corresponding process for the set of evidence information acquired by the acquisition function 353, the interpretation information acquired by the interpretation function 354, and the knowledge information identified by the judgment function 355. The correction function 356 then outputs the processing result of this process as the corrected inference result.

The generation function 357 generates visualization information that visualizes the processes performed by the inference function 352, acquisition function 353, interpretation function 354, judgment function 355, and correction function 356. For example, the generation function 357 generates image data, graphs, text data, etc., which become visualization information of the underlying information or interpretation information, based on the underlying information or interpretation information. For example, the generated visualization information is presented to the user by outputting it to the display 34 via the control function 351.

This makes it easier for users to understand the basis on which the inference results were derived. For example, when inference is performed for positioning assistance in an MRI device, the visualized information, which visualizes the underlying information, becomes image data of a focus region map representing the area of interest on the positioning image.

Furthermore, the visualization information in this case may include, for example, a feature distribution graph, such as a bar graph, showing the distribution of the features of interest. Alternatively, the visualization information may also be text data describing the process from the supporting information to the inference result, such as, "The most noteworthy region with the highest degree of interest was identified on the positioning image, and it was inferred that the centroid of that region is the shooting location."

For example, when inference is performed to assist in determining imaging parameters in an MRI system, the visualized information, which visualizes the underlying information, would be text data explaining the relationship between the conditions that contributed to the derivation of the imaging parameters and the derived imaging parameters.

For example, when reasoning is performed to support a diagnosis using an MRI device, the visualized information, which visualizes the supporting evidence, would include text data explaining that the image findings that contributed to the derivation of the diagnosis are findings specifically observed in that particular diagnosis.

The following explanation will describe the functions of the inference function 352, acquisition function 353, interpretation function 354, judgment function 355, correction function 356, and generation function 357, using Figures 2 to 9 as an example, where the imaging position is determined based on a positioning image, which is an example of medical image data.

Figure 2 shows an example of a positioning image. Figure 2 is a positioning image of the subject's head taken with an MRI scanner. The trained model 321 used in this example derives the imaging position (cross-section of the head to be imaged) as an inference result from this positioning image. The positioning image data contains information indicating the desired imaging position. For example, the positioning image data in Figure 2 contains information indicating that the desired imaging position is a cross-section of the head including the bridge of the nose.

For example, the desired shooting position may be determined by the control function 351 by analyzing the positioning image data acquired by the medical imaging diagnostic device 1. Alternatively, the desired shooting position may be determined by the control function 351 after receiving input regarding the desired shooting position from the user. In this case, the control function 351 performs control to add information indicating the determined desired shooting position to the positioning image data acquired from the medical imaging diagnostic device 1.

Here, we will explain the pre-trained model 321 used in this example. The pre-trained model 321 is designed to output the shooting position on the positioning image by inputting the positioning image and medical information related to the subject. The shooting position refers to a specific location on the subject's body (hereinafter also referred to as a feature point) used to identify the cross-section to be photographed.

Specifically, when a positioning image or similar data is input, the trained model 321 outputs the shooting position on the positioning image and the accuracy (an indicator of the likelihood of inference) of that shooting position as inference results, based on the features contained in the positioning image or similar data. Furthermore, for example, the trained model 321 outputs the shooting position on the positioning image with the highest accuracy as the inference result.

The method for generating the pre-trained model 321 is not particularly specified. For example, the pre-trained model 321 can be generated by using a learning device that generates pre-trained models to perform machine learning on positioning images and the shooting positions on those positioning images as training data.

Furthermore, there are no specific restrictions on the machine learning engine used; publicly known technologies can be employed. For example, a neural network described in the publicly known non-patent document "Pattern Recognition and Machine Learning" by Christopher M. Bishop (USA), 1st edition, Springer, 2006, pp. 225-290, can be applied as a machine learning engine.

Furthermore, the machine learning engine may utilize various algorithms other than the neural networks mentioned above, such as deep learning, logistic regression analysis, nonlinear discriminant analysis, support vector machines (SVM), random forests, and naive Bayes.

The inference function 352 inputs the positioning image shown in Figure 2 to the pre-trained model 321 described above, causing the pre-trained model 321 to perform inference processing.

Figure 3 schematically shows an example of the inference result obtained by the trained model 321. Figure 3 shows the inference result obtained by inputting the positioning image shown in Figure 2 into the trained model 321. Figure 3 shows that the position corresponding to the bridge of the nose on the positioning image in Figure 2 was output as the inference result. Here, the inference result represents the shooting position with the highest accuracy.

Figure 4 schematically illustrates an example of inference processing using the trained model 321. Figure 4 is a map of the region of interest in the positioning image from Figure 2. Here, the region of interest map is an image representing the region of interest that serves as the basis for the inference information on the positioning image. In Figure 4, (1), (2), and (3) represent the region of interest. The trained model 321 outputs the shooting position based on the region of interest. For example, the trained model 321 outputs the position of the centroid of the region of interest as the shooting position.

Furthermore, Figure 4 shows that the areas of interest are in the order of (1), (2), and (3). The level of interest is a numerical value related to the accuracy of the shooting location. For example, the trained model 321 outputs the accuracy of the shooting location based on the ratio of the level of interest of each area to the total level of interest of all areas of interest. Also, in Figure 4, the trained model 321 outputs the tip of the nose, which is the centroid position of area (1), the area of interest with the highest accuracy, as the inference result.

The interpretation function 354 performs interpretation processing based, for example, on information regarding the area of interest on the positioning image. Specifically, the interpretation function 354 refers to the interpretation table 323 and performs processing corresponding to the information regarding the area of interest on the positioning image.

As an example, let's assume that the interpretation table 323 stores information about the area of interest on the positioning image, and that the process of "calculating the percentage of background pixels in the area of interest" is performed accordingly. In this case, the interpretation function 354 calculates the percentage of background pixels in the area of interest. This calculation result becomes the interpretation information.

Figure 5 illustrates an example of the operation of the interpretation function 354. In Figure 5, "(1)" is a number identifying the area of interest on the area of interest map. "85%" in Figure 5 represents the probability that area of interest (1) includes the shooting location. "Percentage of background pixels relative to the area of interest = 70%" in Figure 5 represents interpretation information. Based on this interpretation information, the judgment function 355 determines the validity of the inference result that the shooting location is the tip of the nose.

Specifically, the judgment function 355 refers to the judgment table 324 and identifies the reference location in the knowledge database 322 corresponding to the "ratio of background pixels to the area of interest." In this example, it is assumed that the "ratio of background pixels to the area of interest" and the "address in the knowledge database 322 where the knowledge information that the ratio of background pixels in the area of interest, including the position corresponding to the bridge of the nose, occupies 30% of the area of interest is stored" are associated and stored.

This allows the judgment function 355 to refer to the knowledge database 322 and identify the knowledge information used for determining validity: "The proportion of background pixels in the area of interest that includes the position corresponding to the bridge of the nose to the area of interest = 30%." Based on this knowledge information, the judgment function 355 determines whether the inference result is valid or not.

Figure 6 illustrates an example of the operation of the judgment function 355. As shown in Figure 6, the judgment function 355 determines that the inference result is invalid because the knowledge information "the proportion of background pixels in the area of interest that includes the position corresponding to the bridge of the nose = 30%" does not match the interpretation information "the proportion of background pixels in the area of interest = 70%". In this case, the correction function 356 performs a correction process on the inference result.

Specifically, the correction function 356 refers to the correction table 325 and executes a correction process corresponding to the set of information: "Information regarding the area of interest on the positioning image," "Interpretation information: 70% of the area of interest is occupied by background pixels," and "Knowledge information: 30% of the area of interest is occupied by background pixels including the position corresponding to the bridge of the nose."

In this example, it is assumed that the following set of information is stored in association with the following: "Reasoning information: Information about the area of interest on the positioning image," "Interpretation information: Ratio of background pixels to the area of interest = 70%," and "Knowledge information: Ratio of background pixels to the area of interest including the position corresponding to the bridge of the nose = 30%." This is then followed by a correction process: "Calculate the ratio of background pixels for each area other than the most important area of interest, and set the centroid of the area with the highest probability of containing the shooting position, which matches the knowledge information, as the shooting position."

The correction function 356 calculates the proportion of background pixels in the focus regions (2) and (3) of Figure 2, excluding the focus region (1), which is the region of greatest interest. Figure 7 shows an example of the calculation result of the proportion of background pixels relative to the focus region.

Figure 7 shows that the area of interest (2) on the area of interest map in Figure 2 has an 80% probability of containing the shooting location, and that background pixels occupy 30% of the area of interest. Furthermore, Figure 7 shows that the area of interest (3) on the area of interest map in Figure 2 has a 20% probability of containing the shooting location, and that background pixels occupy 30% of the area of interest.

Next, the correction function 356, in cooperation with the judgment function 355, determines whether the "percentage of background pixels in the area of interest (2) = 30%" and the "percentage of background pixels in the area of interest (3) = 30%" match the knowledge information. Specifically, the correction function 356, similar to the validity judgment process for the inference result, determines whether it matches the knowledge information "the percentage of background pixels in the area of interest that includes the position corresponding to the bridge of the nose = 30%".

Figure 8 illustrates an example of the process for determining whether a region of interest other than the most important region matches the knowledge information. As shown in Figure 8, for both region (2) and region (3), the calculated percentage of background pixels relative to the region of interest matches the knowledge information: "The percentage of background pixels in the region of interest containing the position corresponding to the bridge of the nose = 30%."

Figure 9 illustrates an example of the operation of the correction function 356. As shown in Figure 9, the correction function 356 outputs information from two areas of interest (2) and (3) that match the knowledge information "the ratio of background pixels in the area of interest containing the position corresponding to the bridge of the nose to the area of interest = 30%". The correction function outputs information from area of interest (2), which is most likely to contain the shooting position, as the inference result of the inference regarding the corrected shooting position.

The generation function 357 generates an image showing the position of the centroid of the region of interest (2) on the positioning image in Figure 2, based on the inference result output by the correction function 356, as visualization information. The generation function 357 may also generate the diagrams shown in Figures 3 to 9 as visualization information. The generated visualization information is output as an image to the display 34 by, for example, the control function 351.

In Figure 9, incorrect inference results are automatically corrected; however, the correction function 356 may also correct the inference results according to the instructions of a user, such as a physician. In this case, the user may give instructions to correct the inference results based on the visualization information generated by the generation function 357. Allowing manual correction of inference results in this way is useful, for example, in situations where the user can easily recall the correction method from the generated visualization information.

Furthermore, the judgment function 355 may perform a validity judgment again on the inference result after correction by the correction function 356. In this case, it is preferable to judge the validity based on knowledge information other than the knowledge information used to judge the validity of the inference result before correction. For example, when performing inference to support the determination of the shooting position as described above, it is preferable to judge the validity based on knowledge information related to the shooting position other than the knowledge information "the ratio of background pixels in the area of interest including the position corresponding to the bridge of the nose to the area of interest = 30%".

Furthermore, in this case, the control function 351 may output the final inference result to the display 34, etc., only if the corrected inference result is valid. This can improve the accuracy of the inference.

The above explanation described the case where the supporting information is "information about the area of interest on the positioning image," but the form of the supporting information is not limited to this. For example, the supporting information may be "information about the feature of interest on the positioning image." Below, using Figures 10 to 12, we will explain the process of determining the shooting position based on the positioning image when the supporting information is "information about the feature of interest on the positioning image."

In this example, the trained model 321 is assumed to output the tip of the subject's nose as the shooting location. Furthermore, in this example, the trained model 321 outputs the shooting location on the positioning image based on "information about the featured area on the positioning image," rather than "information about the region of interest on the positioning image."

Figure 10 schematically illustrates an example of inference processing using the trained model 321. Figure 10 shows the distribution of notable features in the positioning image from Figure 2, represented as a bar graph. Furthermore, Figure 10 indicates that in the positioning image from Figure 2, the most notable feature is "convex," the next most notable feature is "concave," and the third most notable feature is "flat."

As shown in Figure 10, the trained model 321 outputs an inference that the shooting location is the tip of the subject's nose, based on the fact that the most noteworthy feature in the positioning image in Figure 2 is "convex." To determine the validity of this inference, the interpretation function 354 first performs an interpretation process on the "information regarding the noteworthy feature on the positioning image."

The interpretation function 354 refers to the interpretation table 323 and performs processing corresponding to the information regarding the focus feature on the positioning image. In this example, it is assumed that "information regarding the focus feature on the positioning image" and "extract the most focus feature" are stored in association. The interpretation function 354 extracts the most focus feature in the positioning image shown in Figure 2. This extraction result becomes the interpretation information.

Figure 11 illustrates an example of the operation of the interpretation function 354. Figure 11 shows that the most notable feature in the positioning image shown in Figure 2 is "convex." Based on this interpretation information, the judgment function 355 determines the validity of the inference that the shooting location is the tip of the nose.

The judgment function 355 refers to the judgment table 324 and identifies the reference location in the knowledge database 322 corresponding to "extract the most noteworthy feature = convex." In this example, it is assumed that "extract the most noteworthy feature = convex" and "the address in the knowledge database 322 where the knowledge information that the area near the bridge of the nose is concave" are stored are associated and stored.

This allows the judgment function 355 to refer to the knowledge database 322 and identify the knowledge information used for determining validity as "the area near the nasal root is concave." Based on this knowledge information, the judgment function 355 determines whether the inference result is valid.

Figure 12 illustrates an example of the operation of the judgment function 355. As shown in Figure 12, the judgment function 355 determines that the inference result is invalid because the knowledge information "the area near the bridge of the nose is concave" does not match the interpretation information "the most noteworthy feature was extracted = convex." Since the inference result is invalid, the correction function 356 performs a correction process on the inference result. If the inference result is valid, the control function 351 outputs information representing the inference result to the display 34.

The correction function 356 refers to the correction table 325 and executes a correction process corresponding to the combination of the following: "Information regarding the notable feature on the positioning image," "Extraction of the most notable feature = convex," and "Knowledge information: The area near the nasal root is concave."

In this example, it is assumed that the following pairs of information are stored in association with each other: "Information about the featured feature," "Interpretation information (extracting the most featured feature = convex)," and "Knowledge information (the area near the bridge of the nose is concave)." The correction process, "Apply a filter to remove convex shapes to the positioning image and perform the same process again," is also stored.

The correction function 356 applies a filter to the positioning image to remove convex shapes, and inputs the filtered positioning image into the trained model 321. The correction function 356 then outputs the inference result derived by the trained model 321 as the corrected inference result.

The generation function 357 generates an image showing the position of the subject's nasal root on the positioning image in Figure 2, based on the inference results output by the correction function 356, as visualization information. The generation function 357 may also generate the images shown in Figures 10 to 12 as visualization information. The generated visualization information is output as an image to the display 34 by, for example, the control function 351.

Furthermore, the judgment function 355 may judge the validity of the corrected inference result output by the correction function 356 based on the knowledge information "the area near the bridge of the nose is concave." In this case, the control function 351 may output the final inference result to the display 34, etc., only if the corrected inference result is valid. This can improve the accuracy of the inference.

Furthermore, both the processing based on the supporting information explained using Figures 2 to 9 and the processing based on the supporting information explained using Figures 10 to 12 may be executed. In this case, the control function 351 may output the inference result as the final inference result only if the inference results from the two processes match.

Furthermore, in addition to the processing based on the supporting information explained using Figures 2 to 9 and the processing based on the supporting information explained using Figures 10 to 12, processing based on yet another set of supporting information may be performed. The control function 351 may output the final inference result only if two or more of the inference results from the three processes match.

Alternatively, the first processing step may involve performing processing based on the supporting information explained using Figures 2 to 9, followed by a second processing step based on the supporting information explained using Figures 10 to 12. The order of processing may be determined based on the contribution rate of each piece of supporting information to the inference result.

In this case, the control function 351 may output the inference result as the final inference result only if the inference result from the first process matches the correction result from the second process. This allows the validity of the inference result to be judged from multiple different perspectives.

Furthermore, the generation function 357 may generate visualization information that visualizes the inference results, as well as information related to the processing performed by each of the inference function 352, acquisition function 353, interpretation function 354, judgment function 355, and correction function 356. The generated visualization information is output to the display 34 or the like by the control function 351.

This allows users to easily understand the basis on which they judged the validity of the inference results, and how they modified the inference results if they were not valid.

Next, we will explain the processes performed by the medical information processing device 3. Figure 13 is a flowchart showing an example of the processes performed by the medical information processing device 3.

First, the inference function 352 inputs input data to the trained model 321 (step S1). For example, when determining the imaging position, the inference function 352 inputs image data of the positioning image captured by the MRI device to the trained model 321. Next, the acquisition function 353 acquires the inference result output from the trained model 321 (step S2).

For example, when determining the shooting location, the acquisition function 353 acquires the region most likely to contain the shooting location from the information of multiple regions output from the trained model 321 as the inference result for the shooting location inference. The likelihood of each region containing the shooting location is determined based on the probability that the region contains the shooting location, which is output from the trained model 321 along with the information of each region.

Next, the acquisition function 353 acquires the basis information for the inference result (step S3). For example, when determining the shooting position, the acquisition function 353 acquires "information regarding the area of interest on the positioning image" as the basis information.

Next, the interpretation function 354 performs interpretation processing of the underlying information and obtains the interpretation information (step S4). Note that if interpretation processing is not required, step S4 is omitted.

For example, if the acquisition function 353 acquires information about the area of interest on the positioning image as supporting information, the interpretation function 354 refers to the interpretation table 323 and performs interpretation processing corresponding to the "information about the area of interest on the positioning image." If the interpretation table 323 stores the "information about the area of interest on the positioning image" and "calculate the percentage of background pixels occupying the area of interest," the interpretation function 354 calculates the percentage of background pixels occupying the area of interest. This calculation result becomes the interpretation information.

Next, the judgment function 355 determines whether the inference result obtained by the inference function 352 is valid, based on the evidence information obtained by the acquisition function 353 or the interpretation information obtained by the interpretation function 354 (step S5). For example, if the interpretation function 354 obtains the percentage of background pixels occupying the most important region as interpretation information, the interpretation function 354 refers to the judgment table 324 and identifies the knowledge information for determining the validity of the inference result corresponding to "the percentage of background pixels occupying the most important region."

In the judgment table 324, if the "percentage of background pixels occupying the most important region" and the "address in the knowledge database 322 where knowledge information regarding the percentage of background pixels occupying the region of interest including the shooting location is stored" are associated and stored, the judgment function 355 identifies the "percentage of background pixels occupying the region of interest including the shooting location" as knowledge information for judging the validity of the inference result. The judgment function 355 then determines the validity of the inference result based on whether or not this knowledge information matches the interpretation information.

If the inference result is deemed valid (Step S5: Yes), the generation function 357 generates visualization information representing the inference result output by the trained model 321. Then, the control function 351 outputs the generated visualization information to the display 34 and terminates this process (Step S6). On the other hand, if the inference result is deemed invalid (Step S5: No), the correction function 356 performs correction processing for the invalid inference result (Step S7).

For example, if the judgment function 355 determines that the inference result is not valid based on the rationale information ("information about the area of interest on the positioning image"), the interpretation information ("the proportion of background pixels occupying the area of interest"), and the knowledge information ("the proportion of background pixels occupying the area of interest including the shooting position"), the judgment function 355 refers to the correction table 325 and performs correction processing corresponding to the rationale information ("information about the area of interest on the positioning image"), the interpretation information ("the proportion of background pixels occupying the area of interest including the shooting position"), and the knowledge information ("the proportion of background pixels occupying the area of interest including the shooting position").

In the correction table 325, if the following are stored in association: "Information regarding the area of interest on the positioning image," "Interpretation information (the percentage of background pixels in the area of interest)," and "Knowledge information (the percentage of background pixels in the area of interest including the shooting position)," and "Calculate the percentage of background pixels in each area other than the area of interest, and set the shooting position to the centroid of the area with the highest probability of containing the shooting position matching the knowledge information," then the correction function 356 calculates the percentage of background pixels in each area other than the area of interest.

The correction function 356 then outputs the position of the centroid of the region of interest with the highest probability of containing the shooting location, among the regions of interest that match the knowledge information "the proportion of background pixels in the region of interest including the shooting location," as the corrected inference result. Next, the generation function 357 generates visualization information that visualizes the inference result based on the outputted inference result.

Next, the control function 351 outputs the visualized information, which visualizes the generated inference results, to the display 34, and terminates this process (step S8).

As described above, the medical information processing device 3 according to this embodiment performs inference using a trained model 321 generated using machine learning, based on input data such as positioning image data, in order to support medical procedures such as determining the imaging position of a subject in an MRI device. It acquires the basis information for the inference and determines the validity of the inference based on this basis information and knowledge information stored in the knowledge database 322 for judging the inference result.

This automatically determines the validity of the inference, eliminating the need for users to manually assess the validity of the inference results from the pre-trained model.

Furthermore, the medical information processing device 3 according to this embodiment determines the validity of the inference result based on multiple pieces of supporting information. This allows for the determination of validity from multiple perspectives, thus improving the accuracy of inference using the trained model.

Furthermore, if the medical information processing device 3 according to this embodiment determines that the inference result of the machine learning-based inference is not valid, it corrects the invalid inference result based on the supporting information, the interpreted information derived from that supporting information, and the knowledge information. This allows the user to obtain the corrected inference result without having to check the information that forms the basis of the inference, even if the initial inference result was invalid.

Furthermore, the medical information processing device 3 according to this embodiment determines the validity of the corrected inference result based on evidence information different from the evidence information used to determine validity before correction, and knowledge information stored in the knowledge DB 322. By using evidence information different from the evidence information used to determine validity before correction, the validity of the corrected inference result can be determined from a different perspective than before correction, thus improving the accuracy of inference by the trained model.

Furthermore, the term "processor" used in the above explanation refers to circuits such as CPUs (Central Processing Units), GPUs (Graphics Processing Units), Application Specific Integrated Circuits (ASICs), and programmable logic devices (e.g., Simple Programmable Logic Devices (SPLDs), Complex Programmable Logic Devices (CPLDs), and Field Programmable Gate Arrays (FPGAs)).

The processor performs its functions by reading and executing the program stored in the memory circuit 32. Alternatively, instead of storing the program in the memory circuit 32, the program may be directly integrated into the processor's circuitry. In this case, the processor performs its functions by reading and executing the program integrated into the circuitry. Furthermore, the processor in this embodiment is not limited to being configured as a single circuit; it may be configured as a single processor by combining multiple independent circuits, and its functions may be achieved through this combination.

Here, the program executed by the processor (medical information processing program) is provided pre-installed in ROM (Read Only Memory) or memory circuits. This program may also be provided as a file in an installable or executable format on these devices, recorded on a computer-readable storage medium such as a CD (Compact Disc)-ROM, FD (Flexible Disk), CD-R (Recordable), or DVD (Digital Versatile Disc).

Furthermore, this program may be provided or distributed by being stored on a computer connected to a network such as the Internet and downloaded via the network. For example, this program consists of modules containing the functional units described above. In terms of actual hardware, the CPU reads the program from a storage medium such as ROM and executes it, thereby loading each module into main memory and creating it in main memory.

According to at least one embodiment described above, it is possible to support the validity assessment of inference results from a trained model.

While several embodiments have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations of embodiments are possible without departing from the spirit of the invention. These embodiments and their variations are included within the scope and spirit of the invention, as well as within the scope of the claims and its equivalents.

100 Medical Information Processing System 200 Network 1 Medical Image Diagnostic Device 2 Medical Image Storage Device 3 Medical Information Processing Device 31 Communication Interface 32 Memory Circuit 33 Input Interface 34 Display 35 Processing Circuit 351 Control Function 352 Inference Function 353 Acquisition Function 354 Interpretation Function 355 Judgment Function 356 Correction Function 357 Generation Function

Claims (9)

An acquisition unit that acquires the inference result of an inference unit that performs inference based on one or more input pieces of information, and first supporting information that shows the basis for the inference result,
A determination unit that determines the validity of the inference result based on the first basis information , comprising: a determination unit that determines the validity of the inference result based on the first basis information and the input information or knowledge information related to the inference result ;
If the aforementioned inference result is not valid, a correction unit modifies the inference result based on the knowledge information,
An information processing device equipped with the following features. The determination unit determines the validity of the inference result that has been corrected by the correction unit.
The information processing apparatus according to claim 1 . The acquisition unit acquires a second basis information that is different from the first basis information used by the judgment unit to determine validity before the correction by the modification unit.
The determination unit determines the validity of the inference result based on the second basis information and the knowledge information related to the modified inference result.
The information processing apparatus according to claim 2 . A generation unit that generates visualization information that visualizes the processing content related to the validity judgment of the judgment unit,
A display unit that presents the aforementioned visualization information to the user,
It also has,
The information processing apparatus according to any one of claims 1 to 3 . The inference unit takes medical information, which includes at least a medical image of the subject, as input information, and infers information to support medical procedures related to the subject.
The information processing apparatus according to any one of claims 1 to 4 . The inference unit, in a medical image diagnostic device, uses a positioning image for determining the shooting position as input information and infers information to support the determination of the shooting position.
The information processing apparatus according to claim 5 . The inference unit, in a medical imaging diagnostic device, uses information regarding the desired image quality as input information and infers information to assist in determining the imaging parameters for capturing medical images.
The information processing apparatus according to claim 5 . The inference unit, in a medical imaging diagnostic device, uses the medical images captured by the medical imaging diagnostic device as input information and infers information to support the diagnostic procedure related to the subject.
The information processing apparatus according to claim 5 . On the computer,
An acquisition step to acquire the inference result of an inference unit that performs inference based on one or more input pieces of information, and first supporting information that shows the basis for the inference result,
A judgment step for determining the validity of the inference result based on the first basis information , comprising: a judgment step for determining the validity of the inference result based on the first basis information and the input information or knowledge information related to the inference result ;
If the aforementioned inference result is not valid, a correction step is taken to correct the inference result based on the knowledge information.
A program that executes the command.