WO2023068330A1 - 情報処理装置、情報処理方法及びコンピュータ読み取り可能な記録媒体 - Google Patents

情報処理装置、情報処理方法及びコンピュータ読み取り可能な記録媒体 Download PDF

Info

Publication number
WO2023068330A1
WO2023068330A1 PCT/JP2022/039094 JP2022039094W WO2023068330A1 WO 2023068330 A1 WO2023068330 A1 WO 2023068330A1 JP 2022039094 W JP2022039094 W JP 2022039094W WO 2023068330 A1 WO2023068330 A1 WO 2023068330A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
unit
unhealthy
healthy
model
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2022/039094
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
佑亮 近藤
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Deepeyevision
Deepeyevision Inc
Original Assignee
Deepeyevision
Deepeyevision Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Deepeyevision, Deepeyevision Inc filed Critical Deepeyevision
Priority to JP2023554736A priority Critical patent/JPWO2023068330A1/ja
Priority to US18/699,072 priority patent/US20240428940A1/en
Publication of WO2023068330A1 publication Critical patent/WO2023068330A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H30/00—ICT specially adapted for the handling or processing of medical images
    • G16H30/20—ICT specially adapted for the handling or processing of medical images for handling medical images, e.g. DICOM, HL7 or PACS
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H30/00—ICT specially adapted for the handling or processing of medical images
    • G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems

Definitions

  • the present disclosure relates to an information processing device, an information processing method, and a computer-readable recording medium.
  • medical specialists and radiologists annotate abnormal areas in medical images by, for example, drawing contours, and train a machine learning model based on the medical images to which the annotation information has been added. Extracting the feature amount of the abnormal region is performed using the obtained trained model.
  • the relationship between the fundus image and the abnormal blood circulation region in the fundus image is determined based on the fundus image, which is an image of the fundus, and the abnormal blood circulation region specified based on the fluorescent fundus angiography image of the fundus.
  • a diagnosis support device is disclosed that identifies an abnormal blood circulation region in a fundus image using a trained model that has been learned.
  • the present disclosure is an information processing capable of presenting an area with a high probability of being an abnormal area in an image without requiring learning data to which annotation information that specifies an abnormal area in a medical image in units of pixels is added.
  • An object is to provide an apparatus, an information processing method, and a computer-readable recording medium.
  • An information processing apparatus includes an image acquisition unit that acquires a first image acquired by imaging an inspection target region of a patient; An image classification unit that acquires the confidence that the image is unhealthy, an image creation unit that creates a second image that visualizes the area that contributed to the classification of the image classification unit, and the confidence acquired by the image classification unit and the image creation unit. and an output unit for outputting an inference result based on the created second image.
  • the region in the image with a high probability of being an abnormal region can be presented. Furthermore, according to this aspect, there is a high probability that the abnormal region is an abnormal region in the image without generating a trained model trained using a large amount of learning data to which annotation information specifying the abnormal region in units of pixels is added. You can grasp the area.
  • the image classification unit acquires the confidence that the inspection target site is healthy and/or the confidence that the inspection target site is unhealthy by inputting the first image into the inference model, and the inference model is , a model for estimating whether the inspection target portion included in the first image is healthy or unhealthy. According to this aspect, it is possible to easily obtain a certainty factor with desired accuracy.
  • the image creating unit may create the second image by adjusting a value representing the degree of contribution to classification by the image classifying unit based on the degree of certainty that the inspection target region is unhealthy. According to this aspect, it is possible to accurately point out an abnormal region in an image that is predicted to be unhealthy.
  • a method includes acquiring a first image obtained by imaging a region to be examined of a patient; Acquiring a certain degree of certainty, creating a second image that visualizes a region that contributed to the certainty that the inspection target part is unhealthy, and making an inference result based on the acquired certainty and the second image and outputting.
  • a computer-readable recording medium provides processing for acquiring a first image acquired by imaging a patient's inspection target region, and A process of acquiring a confidence that is healthy and / or a confidence that it is unhealthy, a process of creating a second image that visualizes the area that contributed to the confidence that the inspection target part is unhealthy, and the acquired confidence
  • a program is recorded for executing a process of outputting an inference result based on the degree and the second image.
  • An information processing device includes a learning unit that inputs an image labeled as correct data as either healthy or unhealthy to a machine learning model to learn the machine learning model; and a model output unit that outputs the learned model that has been learned.
  • the images may include a plurality of images in which either healthy or unhealthy for a plurality of diseases including diffuse diseases is labeled as correct data. According to this aspect, it is possible to obtain a trained model that has learned about unhealthy features across a plurality of diseases, including diffuse diseases in which it is particularly difficult to point out abnormal regions.
  • a method includes obtaining a plurality of images labeled as either healthy or unhealthy as correct data, and using the images to train a machine learning model, comprising:
  • the machine learning model is a machine learning model that estimates whether the inspection target part contained in the image is healthy or unhealthy, and outputs the learned model obtained by learning the machine learning model. including.
  • information processing capable of presenting an area with a high probability of being an abnormal area in an image without requiring learning data to which annotation information specifying an abnormal area in a medical image in units of pixels is added.
  • An apparatus, an information processing method, and a computer-readable recording medium can be provided.
  • FIG. 1 is a diagram showing a network configuration of an information processing system according to one embodiment;
  • FIG. 4 is a schematic diagram illustrating processing of the learning device according to one embodiment;
  • FIG. 4 is a schematic diagram illustrating processing of an inference device according to an embodiment;
  • FIG. 1 is a block diagram of a learning device according to one embodiment;
  • FIG. 1 is a block diagram of an inference device according to an embodiment;
  • FIG. 6 is a flow chart showing learning processing of the learning device according to one embodiment.
  • 4 is a flowchart showing inference processing of an inference device according to an embodiment;
  • FIG. 5 is a diagram illustrating a function used to adjust the intensity of a heatmap according to one embodiment;
  • 4 is a schematic diagram illustrating processing of the learning device according to one embodiment;
  • FIG. 4 is a flowchart showing inference processing of an inference device according to an embodiment;
  • FIG. 1 is a diagram showing a network configuration of an information processing system according to one embodiment.
  • FIG. 2 is a schematic diagram illustrating processing of the learning device according to one embodiment.
  • FIG. 3 is a schematic diagram illustrating processing of an inference device according to an embodiment.
  • the information processing system includes a learning device 10, an inference device 20, and a storage device 30.
  • the learning device 10 is connected to the inference device 20 and the storage device 30 via the communication network N.
  • the communication network N may be either a wired communication network or a wireless communication network configured by wired or wireless circuits, and may be the Internet or a Local Area Network (LAN).
  • LAN Local Area Network
  • the learning device 10 learns a machine learning model based on the learning data stored in the storage device 30, and stores the learned model in the storage device 30.
  • the learning device 10 according to this embodiment includes a machine learning model, but the machine learning model may be provided in a separate device from the learning device 10 .
  • the machine learning model has a predetermined model structure and processing parameters that vary depending on the learning process, and the processing parameters are optimized based on the experience obtained from the learning data, thereby increasing the identification accuracy. It is a model that improves. That is, the machine learning model is a model that learns optimum processing parameters through learning processing.
  • the algorithm of the machine learning model can be, for example, support vector machine, logistic regression, neural network, etc., but the type is not particularly limited.
  • Machine learning models that perform the learning include those before learning and those that have already undergone some learning using learning data.
  • a trained model is a model that has been trained in advance using appropriate learning data for a machine learning model based on an arbitrary machine learning algorithm.
  • a trained model is not one that does not perform further learning, and can also perform additional learning.
  • the inference device 20 uses the learned model to output output data according to the features of the input data.
  • the inference device 20 performs inference using a trained model acquired from the storage device 30 .
  • obtaining a trained model means obtaining information necessary for reproducing the functions of the trained model in the inference device 20 .
  • obtaining a trained model includes at least the number of layers of the neural network, the number of nodes for each layer, the weight parameter for links connecting nodes, the bias parameter for each node, and each Obtaining information about the functional form of the activation function for a node.
  • the storage device 30 stores learning data used for learning the machine learning model.
  • the storage device 30 according to the present embodiment stores, as learning data, fundus images labeled with either "healthy” or “unhealthy” as correct data.
  • the storage device 30 also stores the learned model output by the learning device 10 .
  • FIG. 1 shows the storage device 30 as a single storage device, the storage device 30 may be configured by one or more file servers.
  • fundus images labeled as either “healthy” or “unhealthy” are used as an example of learning data. and labeled as either “healthy” or “unhealthy” can also be used as training data.
  • the learning device 10 includes a machine learning model that receives fundus images as input data and classifies them into healthy eye images and non-healthy eye images.
  • the learning device 10 classifies fundus images using a machine learning model, and trains the machine learning model so as to minimize the error between the predicted result and the correct data labeled in the learning data.
  • a fundus image which is an example of a medical image, will be described. can be used.
  • Fundus images labeled with only "healthy” or “non-healthy” for each image can be used as training data. can be compared and collected easily.
  • the inference device 20 performs forward calculation using the learned model and infers whether the fundus image belongs to a healthy eye or a non-healthy eye.
  • the inference device 20 performs error backpropagation from the output layer corresponding to the non-healthy eye to the convolution layer to be visualized, calculates the contribution of each feature map to the output of the non-healthy eye, and obtains by forward calculation.
  • a heat map is created by weighting and summing the contribution of the feature map.
  • Abnormal regions in non-healthy eyes that are difficult to define can be presented.
  • the inference device 20 without generating a trained model trained using a large amount of learning data to which annotation information specifying an abnormal region in units of pixels is added, A region with a high probability of being an abnormal region can be grasped.
  • an abnormal region in a fundus image which is an example of a medical image, is presented.
  • Abnormal regions in medical images, such as images, can be presented.
  • FIG. 4 is a block diagram of a learning device according to one embodiment. In FIG. 4, a single learning device 10 is assumed and only the necessary functional configuration is shown. can.
  • the learning device 10 includes an input unit 110, a control unit 120, a storage unit 130, and a communication unit 140.
  • the input unit 110 is configured to receive operations from the administrator of the learning device 10, and can be realized by a keyboard, mouse, touch panel, or the like.
  • the control unit 120 includes an arithmetic processing unit 121 such as a CPU or MPU corresponding to a processor, and a memory 122 such as a RAM.
  • the arithmetic processing unit 121 (processor) develops the program recorded in the storage unit 130 in the memory 122 based on various inputs and executes the program, thereby realizing functions and processes described later in the arithmetic processing unit 121 .
  • This program may be stored in a computer-readable non-temporary recording medium such as a CD-ROM, or distributed via a network and installed in a computer.
  • the memory 122 functions as a work memory required for program execution by the arithmetic processing unit 121 (processor).
  • the storage unit 130 is configured by a storage device such as a hard disk, and stores various programs necessary for executing processes in the control unit 120 and data necessary for executing various programs.
  • the storage unit 130 desirably has a learning data storage unit 131 .
  • the learning data storage unit 131 stores learning data used for learning the machine learning model M described later.
  • the learning data storage unit 131 stores fundus images labeled with either “healthy” or “unhealthy” as correct data.
  • the learning data storage unit 131 stores a plurality of fundus images labeled with either “healthy” or “unhealthy” as correct data for a plurality of diseases including diffuse diseases.
  • multiple fundus images labeled as either "healthy” or "unhealthy” as correct data for a single diffuse disease may be saved; A plurality of fundus images labeled with either "healthy” or "unhealthy” as correct data for a plurality of diseases that do not include diffuse disease may be saved.
  • the communication unit 140 is configured to connect the learning device 10 to the network.
  • the communication unit 140 can be implemented by a LAN card, an analog modem, an ISDN modem, etc., and an interface for connecting these to the processing unit via a transmission line such as a system bus.
  • the arithmetic processing unit 121 includes a learning data acquisition unit 123, a learning unit 124, an image classification unit 125, and a model output unit 126 as functional units.
  • the learning data acquisition unit 123 acquires learning data used for learning a machine learning model M, which will be described later, and stores it in the learning data storage unit 131 .
  • the learning data acquisition unit 123 acquires a fundus image labeled with either “healthy” or “unhealthy” as correct data from the storage device 30 and stores it in the learning data storage unit 131. .
  • the learning unit 124 makes the machine learning model M learn using the learning data acquired by the learning data acquiring unit 123 .
  • the learning unit 124 inputs fundus images labeled with either “healthy” or “non-healthy” as correct data to the image classification unit 125 described later, and makes the machine learning model M learn. .
  • the image classification unit 125 receives input of fundus images and classifies them into healthy eye images and non-healthy eye images.
  • the image classification unit 125 receives an input of a fundus image, and uses the machine learning model M to output confidence factors for a healthy eye and an unhealthy eye.
  • the machine learning model M is a machine learning model that receives fundus images as input data and classifies them into healthy eye images and unhealthy eye images.
  • the machine learning model M an example using a convolutional neural network (CNN) that receives a fundus image as input data and outputs image classification will be described.
  • CNN convolutional neural network
  • the CNN is only an example of the machine learning model M, and the learning device 10 may use other configurations as the machine learning model M.
  • the learning unit 124 learns the machine learning model M so as to minimize the error between the result predicted by the machine learning model M and the correct data labeled with the learning data.
  • the model output unit 126 outputs the learned model obtained by the learning of the machine learning model M to the storage device 30.
  • the learning unit 124 may, for example, complete learning after making the machine learning model M learn using a predetermined number of learning data, or the classification accuracy predicted using the machine learning model M may reach a predetermined level. Learning may be completed when the conditions of
  • FIG. 5 is a block diagram of an inference device according to one embodiment. Note that FIG. 5 assumes a single inference device 20 and shows only the necessary functional configuration, but the inference device 20 can also be configured as part of a multifunctional distributed system comprising a plurality of computer systems. can.
  • the inference device 20 includes an input unit 210 , a control unit 220 , a storage unit 230 and a communication unit 240 .
  • the input unit 210 is configured to receive operations from the administrator of the inference device 20, and can be realized by a keyboard, mouse, touch panel, or the like.
  • the control unit 220 includes an arithmetic processing unit 221 such as a CPU or MPU corresponding to a processor, and a memory 222 such as a RAM.
  • the arithmetic processing unit 221 (processor) develops the program recorded in the storage unit 230 in the memory 222 based on various inputs and executes the program, thereby realizing functions and processes described later in the arithmetic processing unit 221 .
  • This program may be stored in a computer-readable non-temporary recording medium such as a CD-ROM, or distributed via a network and installed in a computer.
  • the memory 222 functions as a work memory required for program execution by the arithmetic processing unit 221 (processor).
  • the storage unit 230 is configured by a storage device such as a hard disk, and stores various programs necessary for executing processes in the control unit 220, data necessary for executing various programs, and the like.
  • the storage unit 230 preferably has an image storage unit 231 and a learned model 232 .
  • the image storage unit 231 stores images to be inferred.
  • the image storage unit 231 stores fundus images for inference of diagnosis.
  • the trained model 232 stores the trained model used for inference.
  • the trained model 232 stores a trained model that receives a fundus image as input data and classifies the image into a healthy eye image and an unhealthy eye image.
  • CNN convolutional neural network
  • the trained model 232 an example using a convolutional neural network (CNN) that receives a fundus image as input data and classifies the image into a healthy eye image and a non-healthy eye image will be described.
  • CNN convolutional neural network
  • reasoning apparatus 20 may use other configurations for trained model 232 .
  • the communication unit 240 is configured to connect the inference device 20 to the network.
  • the communication unit 240 can be implemented by a LAN card, an analog modem, an ISDN modem, etc., and an interface for connecting these to the processing unit via a transmission line such as a system bus.
  • the arithmetic processing unit 221 includes, as functional units, a model acquisition unit 223, an image acquisition unit 224, an inference unit 225, an image classification unit 226, a heat map creation unit 227, and an output unit 228.
  • the model acquisition unit 223 acquires a trained model used for inference and stores it in the trained model 232 .
  • the model acquisition unit 223 acquires a trained model from the storage device 30 and stores it in the trained model 232 .
  • the image acquisition unit 224 acquires an inference target image.
  • the image acquisition unit 224 acquires a fundus image for inference of diagnosis from the image storage unit 231 .
  • the inference unit 225 infers a predicted diagnosis from the image acquired by the image acquisition unit 224 .
  • the inference unit 225 is composed of an image classification unit 226 and a heat map creation unit 227 .
  • the inference unit 225 first inputs the fundus image to the image classification unit 226 and acquires the confidence factors of the healthy eye and the unhealthy eye from the image classification unit 226 .
  • the image classification unit 226 receives an input of fundus images and classifies them into healthy eye images and non-healthy eye images. In this embodiment, as shown in FIG. 3, the image classification unit 226 receives an input of a fundus image, performs forward calculation using the trained model 232, and calculates the certainty of the healthy eye and the non-healthy eye. Output.
  • the heat map creation unit 227 creates a heat map that visualizes the regions that have contributed to the classification by the image classification unit 226.
  • the heat map creation unit 227 transfers the error from the output layer corresponding to the non-healthy eye to the convolution layer to be visualized.
  • To perform backpropagation and calculate the contribution of each feature map to the non-healthy eye's output compute the gradient of the feature map to the non-healthy eye's output and take the Global Max Pooling (GMP) of the gradient.
  • GMP Global Max Pooling
  • the heat map creation unit 227 weights the feature maps obtained by the forward calculation with GMP, and acquires a coefficient map by adding all the feature maps.
  • the heat map creation unit 227 adjusts the value of each element of the coefficient map based on the degree of certainty of the unhealthy eye. For example, as shown in FIG. 8, the heat map creation unit 227 sets the value of each element of the coefficient map to zero when the certainty factor of the unhealthy eye is 0.0 to 0.3, When the confidence is 0.3 to 0.6, the value of each element is adjusted in proportion to the confidence of the unhealthy eye, and when the confidence of the unhealthy eye is 0.6 to 1.0 The value of each element may be adjusted to 100%, that is, the value as it is. An image predicted to be an unhealthy eye by reducing the value of each element of the coefficient map when the confidence of the unhealthy eye is low, and maintaining the value of each element of the coefficient map when the confidence is high. It is possible to point out an abnormal region with high accuracy.
  • the heat map creation unit 227 creates a heat map by imaging the coefficient map with a color scale and resizing the obtained image to the size of the input image.
  • heat maps may be obtained using other visualization methods.
  • a heat map that expresses the level of contribution with a color scale is created, but a visualization map that expresses it in another format may be created.
  • the output unit 228 outputs an inference result based on the information acquired by the inference unit 225 .
  • the output unit 228 superimposes the heat map on the fundus image acquired by the image acquiring unit 224 by alpha blending and outputs the image.
  • the learning process of the learning device will be described in detail with reference to FIG.
  • the learning data is stored in the storage device 30 under the control of the administrator of the learning device 10 before the learning process described with reference to FIG. 6 is performed.
  • the processing shown in FIG. 6 is executed, for example, when the administrator inputs an instruction for executing processing for generating a learned model via the input unit 110 .
  • step S ⁇ b>601 the learning data acquisition unit 123 of the learning device 10 acquires learning data used for learning the machine learning model M and stores it in the learning data storage unit 131 .
  • the learning data acquisition unit 123 acquires a fundus image labeled with either “healthy” or “unhealthy” as correct data from the storage device 30 and stores it in the learning data storage unit 131.
  • the learning data acquisition unit 123 acquires a plurality of fundus images labeled with either “healthy” or “unhealthy” as correct data for a plurality of diseases including diffuse diseases. It is assumed that the data is stored in the learning data storage unit 131 . That is, if one or more of a plurality of diseases, including diffuse disease, apply, the fundus image is labeled as "unhealthy” as correct data.
  • step S602 the learning unit 124 of the learning device 10 uses the learning data acquired by the learning data acquiring unit 123 to make the machine learning model M learn.
  • the learning unit 124 inputs a fundus image labeled as correct data with either “healthy” or “non-healthy” to the image classification unit 125 of the learning device 10 to generate the machine learning model M. let them learn
  • the machine learning model M is a machine learning model that receives fundus images as input data and classifies them into healthy eye images and unhealthy eye images.
  • the machine learning model M an example using a convolutional neural network (CNN) that receives a fundus image as input data and outputs image classification will be described.
  • CNN convolutional neural network
  • the CNN is only an example of the machine learning model M, and the learning device 10 may use other configurations as the machine learning model M.
  • the learning unit 124 learns the machine learning model M so as to minimize the error between the result predicted by the machine learning model M and the correct data labeled with the learning data.
  • the model output unit 126 of the learning device 10 outputs the learned model obtained by the learning of the machine learning model M to the storage device 30 in step S603.
  • the learning unit 124 may, for example, complete learning after making the machine learning model M learn using a predetermined number of learning data, or the classification accuracy predicted using the machine learning model M may reach a predetermined level. Learning may be completed when the conditions of
  • the inference processing of the inference device will be described in detail with reference to FIG.
  • a trained model acquired from the storage device 30 is stored in the trained model 232 under the control of the administrator of the inference device 20 before performing the inference processing described with reference to FIG. do.
  • the image storage unit 231 of the inference device 20 stores an inference-target fundus image. Note that the processing shown in FIG. 7 is executed, for example, by the administrator inputting an instruction for executing the inference processing via the input unit 210 .
  • step S701 the image acquisition unit 224 of the inference device 20 acquires an inference target image.
  • the image acquisition unit 224 acquires a fundus image for inference of diagnosis from the image storage unit 231 .
  • the inference unit 225 of the inference device 20 acquires the certainty factors of the healthy eye and the non-healthy eye.
  • the inference unit 225 inputs the fundus image to the image classification unit 226 of the inference device 20 and acquires the confidence factors of the healthy eye and the non-healthy eye from the image classification unit 226 .
  • the image classification unit 226 receives an input of a fundus image, performs forward calculation using the trained model 232, and outputs confidence factors for a healthy eye and an unhealthy eye.
  • CNN convolutional neural network
  • step S703 the heat map creation unit 227 of the inference device 20 back-propagates the error from the output layer corresponding to the non-healthy eye to the convolution layer to be visualized, To calculate the contribution of each feature map to the unhealthy eye's output, compute the gradient of the feature map to the unhealthy eye's output and take the Global Max Pooling (GMP) of the gradient.
  • GMP Global Max Pooling
  • step S704 the heat map creation unit 227 weights the feature maps obtained by forward calculation with GMP, and acquires a coefficient map by adding all feature maps. Then, in step S705, the heat map creation unit 227 adjusts the value of each element of the coefficient map based on the certainty factor of the unhealthy eye.
  • the heat map creation unit 227 sets the value of each element of the coefficient map to zero when the certainty factor of the unhealthy eye is 0.0 to 0.3,
  • the value of each element is adjusted in proportion to the confidence of the unhealthy eye, and the confidence of the unhealthy eye is 0.6 to 1.0.
  • the value of each element is adjusted to 100%, that is, the value as it is.
  • An image predicted to be an unhealthy eye by reducing the value of each element of the coefficient map when the confidence of the unhealthy eye is low, and maintaining the value of each element of the coefficient map when the confidence is high. It is possible to point out an abnormal region with high accuracy.
  • the heat map creation unit 227 creates a heat map by imaging the adjusted coefficient map with a color scale and resizing the obtained image to the size of the input image.
  • heat maps may be obtained using other visualization methods.
  • a heat map that expresses the level of contribution with a color scale is created, but a visualization map that expresses it in another format may be created.
  • step S707 the output unit 228 of the inference device 20 outputs an inference result based on the information acquired by the inference unit 225.
  • the output unit 228 superimposes the heat map on the fundus image acquired by the image acquiring unit 224 by alpha blending and outputs the image.
  • the learning device 10 can obtain a trained model that can be used for inference of a diagnosis predicted from a fundus image.
  • fundus images labeled with only "healthy” or “unhealthy” in image units can be used as learning data, so annotation information specifying abnormal regions in units of pixels was required. It can be easily collected compared to conventional learning data.
  • the inference device 20 uses a trained model that has learned features of unhealthy eyes from a large number of images of healthy and unhealthy eyes, and greatly contributes to the output of unhealthy eyes. By visualizing the regions, it is possible to present abnormal regions in non-healthy eyes that are difficult to define clearly across multiple diseases. Furthermore, according to the inference device 20 according to the present embodiment, without generating a trained model trained using a large amount of learning data to which annotation information specifying an abnormal region in units of pixels is added, A region with a high probability of being an abnormal region can be grasped.
  • an abnormal region in a fundus image which is an example of a medical image
  • Abnormal regions in medical images such as images of the liver, can be presented.
  • one machine learning model is learned using images labeled as correct data for either healthy or unhealthy for a plurality of diseases.
  • An example of performing inference processing has been described.
  • one machine learning model for each of a plurality of diseases is trained using images labeled as correct data for either healthy or unhealthy as learning data.
  • An example of inference processing using the model M-1 to MN will be described.
  • the learning device 10 includes a plurality of machine learning models that receive fundus images as input data and classify them into images of healthy eyes and images of non-healthy eyes (disease). .
  • the learning device 10 classifies fundus images using each machine learning model, and trains each machine learning model so as to minimize the error between the predicted result and the correct data labeled in the learning data.
  • the image classification unit 125-1 is a machine learning model that learns about the characteristics of glaucoma, and classifies images into images of healthy eyes and images of non-healthy eyes (glaucoma).
  • the image classification unit 125-2 is a machine learning model that learns about the characteristics of diabetic retinopathy, and classifies the images into healthy eye images and non-healthy eye (diabetic retinopathy) images.
  • the learning data storage unit 131 stores each learning data used for learning a plurality of machine learning models M-1 to MN.
  • the learning data storage unit 131 stores a fundus image labeled with either “healthy” or “non-healthy (glaucoma)” as correct data used for learning the machine learning model M-1, machine Fundus images labeled with either “healthy” or “unhealthy (diabetic retinopathy)” as correct data used for learning the learning model M-2, ..., for learning the machine learning model MN
  • a fundus image labeled as correct data with either "Healthy” or "Unhealthy (Disease N)" to be used is saved.
  • the inference device 20 performs forward calculation using a plurality of trained models and infers whether the fundus image belongs to a healthy eye or a non-healthy eye.
  • the reasoning device 20 is equipped with a plurality of trained models 232-1 to 232-N that have learned the features of each disease of a plurality of diseases, and the confidence of a healthy eye and the confidence of an unhealthy eye (disease) for each disease. Get degrees.
  • the inference unit 225 is composed of a plurality of image classification units 226-1 to 226-N and a plurality of heat map creation units 227-1 to 227-N corresponding to the plurality of image classification units. be.
  • the inference unit 225 first inputs fundus images to a plurality of image classification units 226 - 1 to 226 -N, and acquires confidence factors of a healthy eye and an unhealthy eye (disease) from each image classification unit 226 .
  • the inference unit 225 classifies one or more images out of the plurality of image classification units 226-1 to 226-N based on the certainty obtained from the plurality of image classification units 226-1 to 226-N.
  • a unit 226 is selected, and the corresponding heat map creation unit 227 is instructed to create a heat map that visualizes the region that contributed to the classification of the selected image classification unit 226 .
  • the inference unit 225 has one image classifier that has the highest certainty for the unhealthy eye (disease) among the certainties for the healthy eye and the unhealthy eye (disease) acquired from each image classifier 226. 226 is selected, and the heat map creation unit 227 corresponding to the selected image classification unit 226 creates a heat map.
  • the inference unit 225 selects one image classification unit 226. In other embodiments, the inference unit 225 selects the number of unhealthy eyes (disease) above a predetermined threshold, such as 0.5. One or a plurality of image classifiers 226 having a certainty may be selected, or a predetermined number of image classifiers 226 having a higher certainty of an unhealthy eye (disease) may be selected.
  • a predetermined threshold such as 0.5.
  • each of the plurality of machine learning models M-1 to MN can be learned in the same flow as in FIG. 6, which explains the learning of a single machine learning model, so the explanation is omitted here. do.
  • the inference processing of the inference device will be described in detail with reference to FIG.
  • a plurality of trained models 232-1 to 232 acquired from the storage device 30 are stored in the trained model 232 under the control of the administrator of the inference device 20 before performing the inference processing described with reference to FIG. -N is stored.
  • the image storage unit 231 of the inference device 20 stores an inference-target fundus image. Note that the processing shown in FIG. 10 is executed, for example, by the administrator inputting an instruction for executing the inference processing via the input unit 210 .
  • step S1001 the image acquisition unit 224 acquires an inference target image.
  • the image acquisition unit 224 acquires a fundus image for inference of diagnosis from the image storage unit 231 .
  • step S1002 the inference unit 225 inputs fundus images to a plurality of image classification units 226-1 to 226-N, and acquires confidence levels of a healthy eye and an unhealthy eye (disease) from each image classification unit 226. .
  • the inference unit 225 selects a plurality of One or a plurality of image classifiers 226 are selected from image classifiers 226-1 to 226-N.
  • the inference unit 225 has one image classifier that has the highest certainty for the unhealthy eye (disease) among the certainties for the healthy eye and the unhealthy eye (disease) acquired from each image classifier 226. 226 is selected.
  • step S1004 the heat map creation unit 227 performs error back propagation from the selected output layer corresponding to the unhealthy eye (disease) of the image classification unit 226 to the convolution layer to be visualized, and outputs the unhealthy eye (disease).
  • the heat map creation unit 227 performs error back propagation from the selected output layer corresponding to the unhealthy eye (disease) of the image classification unit 226 to the convolution layer to be visualized, and outputs the unhealthy eye (disease).
  • GMP Global Max Pooling
  • step S1005 the heat map creation unit 227 weights the feature maps obtained by the forward calculation with GMP, and acquires a coefficient map by adding all the feature maps. Then, in step S1006, the heat map creation unit 227 adjusts the value of each element of the coefficient map based on the certainty of the unhealthy eye (disease).
  • the heat map creation unit 227 sets the value of each element of the coefficient map to zero when the certainty factor of an unhealthy eye (disease) is 0.0 to 0.3. And, when the confidence of the unhealthy eye (disease) is 0.3 to 0.6, the value of each element is adjusted in proportion to the confidence of the unhealthy eye (disease), and the unhealthy eye (disease) is 0.6 to 1.0, the value of each element is adjusted to 100%, that is, the value as it is.
  • the heat map creation unit 227 creates a heat map by imaging the adjusted coefficient map with a color scale and resizing the obtained image to the size of the input image.
  • step S1008 the output unit 228 outputs an inference result based on the information acquired by the inference unit 225.
  • the output unit 228 superimposes the heat map on the fundus image acquired by the image acquiring unit 224 by alpha blending and outputs the image.
  • the learning device 10 can obtain a plurality of trained models, each of which has learned the characteristics of a specific disease, and which can be used for inference of a diagnosis predicted from a fundus image.
  • fundus images labeled with only "healthy” or "unhealthy” in image units can be used as learning data, so annotation information specifying abnormal regions in units of pixels was required. It can be easily collected compared to conventional learning data.
  • the inference device 20 uses a plurality of trained models, each of which has learned about the characteristics of a specific disease, to identify the unhealthy eye (disease) of the image classification unit with a high degree of certainty of disease. By visualizing the regions that greatly contributed to the output, it is possible to present abnormal regions in non-healthy eyes that are difficult to define clearly. Furthermore, according to the inference device 20 according to the present embodiment, without generating a trained model trained using a large amount of learning data to which annotation information specifying an abnormal region in units of pixels is added, A region with a high probability of being an abnormal region can be grasped.
  • An information processing apparatus includes a plurality of image classifying units, selects one or more image classifying units based on certainty obtained from the plurality of image classifying units, and selects an image creating unit.
  • One or more second images may be created that visualize regions that contributed to the classification of the image classifier.
  • the information processing device uses a plurality of different degrees of certainty of unhealthy features to present an area in the image that is likely to be an abnormal area due to the selected unhealthy feature. can be done.
  • each of the plurality of image classifying units acquires the degree of certainty that the inspection target site is healthy and/or the degree of certainty that the examination target site has a specific disease by inputting the first image into the inference model.
  • the inference model of each image classifying unit may be a model for inferring whether the inspection target portion included in the first image is healthy or has a specific disease. According to this aspect, the information processing apparatus can easily acquire a certainty factor with a desired accuracy for each of a plurality of different disease certainty factors.

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Public Health (AREA)
  • General Health & Medical Sciences (AREA)
  • Primary Health Care (AREA)
  • Epidemiology (AREA)
  • Theoretical Computer Science (AREA)
  • Biomedical Technology (AREA)
  • Databases & Information Systems (AREA)
  • Radiology & Medical Imaging (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Software Systems (AREA)
  • Evolutionary Computation (AREA)
  • Computing Systems (AREA)
  • Artificial Intelligence (AREA)
  • Data Mining & Analysis (AREA)
  • Pathology (AREA)
  • Image Analysis (AREA)
  • Eye Examination Apparatus (AREA)
PCT/JP2022/039094 2021-10-20 2022-10-20 情報処理装置、情報処理方法及びコンピュータ読み取り可能な記録媒体 Ceased WO2023068330A1 (ja)

Priority Applications (2)

Application Number Priority Date Filing Date Title
JP2023554736A JPWO2023068330A1 (https=) 2021-10-20 2022-10-20
US18/699,072 US20240428940A1 (en) 2021-10-20 2022-10-20 Information processing apparatus, information processing method, and computer-readable recording medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2021-171543 2021-10-20
JP2021171543 2021-10-20

Publications (1)

Publication Number Publication Date
WO2023068330A1 true WO2023068330A1 (ja) 2023-04-27

Family

ID=86058329

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/039094 Ceased WO2023068330A1 (ja) 2021-10-20 2022-10-20 情報処理装置、情報処理方法及びコンピュータ読み取り可能な記録媒体

Country Status (3)

Country Link
US (1) US20240428940A1 (https=)
JP (1) JPWO2023068330A1 (https=)
WO (1) WO2023068330A1 (https=)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20230110918A (ko) * 2022-01-17 2023-07-25 현대자동차주식회사 영상 처리 방법, 장치 및 이를 포함하는 차량

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190286880A1 (en) * 2018-03-16 2019-09-19 Proscia Inc. Deep learning automated dermatopathology
JP2021039748A (ja) * 2019-08-30 2021-03-11 キヤノン株式会社 情報処理装置、情報処理方法、情報処理システム及びプログラム
CN113255766A (zh) * 2021-05-25 2021-08-13 平安科技（深圳）有限公司 一种图像分类方法、装置、设备和存储介质

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018045363A1 (en) * 2016-09-02 2018-03-08 Gargeya Rishab Screening method for automated detection of vision-degenerative diseases from color fundus images
JP7181230B2 (ja) * 2017-05-31 2022-11-30 コーニンクレッカ フィリップス エヌ ヴェ 臨床判断支援のための生医療画像データの機械学習
EP3477553B1 (en) * 2017-10-27 2023-08-30 Robert Bosch GmbH Method for detecting an anomalous image among a first dataset of images using an adversarial autoencoder
US20210042916A1 (en) * 2018-02-07 2021-02-11 Ai Technologies Inc. Deep learning-based diagnosis and referral of diseases and disorders
KR102100698B1 (ko) * 2019-05-29 2020-05-18 (주)제이엘케이 앙상블 학습 알고리즘을 이용한 인공지능 기반 진단 보조 시스템
US12135763B2 (en) * 2021-06-28 2024-11-05 Siemens Healthineers International Ag Automatic localized evaluation of contours with visual feedback

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190286880A1 (en) * 2018-03-16 2019-09-19 Proscia Inc. Deep learning automated dermatopathology
JP2021039748A (ja) * 2019-08-30 2021-03-11 キヤノン株式会社 情報処理装置、情報処理方法、情報処理システム及びプログラム
CN113255766A (zh) * 2021-05-25 2021-08-13 平安科技（深圳）有限公司 一种图像分类方法、装置、设备和存储介质

Also Published As

Publication number Publication date
JPWO2023068330A1 (https=) 2023-04-27
US20240428940A1 (en) 2024-12-26

Similar Documents

Publication Publication Date Title
JP7406758B2 (ja) 人工知能モデルを使用機関に特化させる学習方法、これを行う装置
JP7058373B2 (ja) 医療画像に対する病変の検出及び位置決め方法、装置、デバイス、及び記憶媒体
US20210174497A1 (en) Saliency mapping by feature reduction and perturbation modeling in medical imaging
WO2021114630A1 (zh) 医学图像样本筛查方法、装置、计算机设备和存储介质
CN112102266A (zh) 基于注意力机制的脑梗死医学影像分类模型的训练方法
JP7058988B2 (ja) 情報処理装置、情報処理方法およびプログラム
EP4293615A1 (en) Learning device, learning method, trained model, and program
Barnett et al. Interpretable mammographic image classification using case-based reasoning and deep learning
US20240087120A1 (en) Geographic atrophy progression prediction and differential gradient activation maps
Satapathy et al. A federated learning approach for classifying chest diseases from chest X-ray images
WO2023068330A1 (ja) 情報処理装置、情報処理方法及びコンピュータ読み取り可能な記録媒体
US12119116B2 (en) Apparatus, information processing method, and computer-readable recording medium for disease correlation and inferencing
Kumar et al. Spatial attention U-Net model with Harris hawks optimization for retinal blood vessel and optic disc segmentation in fundus images
US11263481B1 (en) Automated contrast phase based medical image selection/exclusion
US20240293024A1 (en) Methods and systems for biomarker identification and discovery
JP2021189962A (ja) 医用情報処理装置、医用情報処理システム、医用情報処理方法、およびプログラム
Wang et al. Diabetic retinopathy detection based on weakly supervised object localization and knowledge driven attribute mining
CN112967246B (zh) 用于临床决策支持系统的x射线影像辅助装置及方法
WO2024128108A1 (ja) 情報処理装置、情報処理方法及びコンピュータ読み取り可能な記録媒体
Chan et al. Identifying pediatric vascular anomalies with deep learning
CN118941575A (zh) 图像分割模型训练方法与电子装置
JPWO2023068330A5 (https=)
JP7672694B2 (ja) 情報処理装置、情報処理方法及びプログラム
JP7700201B2 (ja) 情報処理装置、情報処理方法およびプログラム
EP4682821A1 (en) Method of analyzing 3d data acquired by a 3d image-producing procedure in an area of inspection, computer-based clinical decision support system, computer program product and computer-readable medium

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22879641

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 18699072

Country of ref document: US

ENP Entry into the national phase

Ref document number: 2023554736

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22879641

Country of ref document: EP

Kind code of ref document: A1