US20210000343A1 - Diagnosis support device, learning device, diagnosis support method, learning method, and program - Google Patents

Diagnosis support device, learning device, diagnosis support method, learning method, and program Download PDF

Info

Publication number
US20210000343A1
US20210000343A1 US16/930,510 US202016930510A US2021000343A1 US 20210000343 A1 US20210000343 A1 US 20210000343A1 US 202016930510 A US202016930510 A US 202016930510A US 2021000343 A1 US2021000343 A1 US 2021000343A1
Authority
US
United States
Prior art keywords
information
image
fundus image
area
npa
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US16/930,510
Other languages
English (en)
Inventor
Toshihide KURIHARA
Yusaku Katada
Kazuo Tsubota
Kanato MASAYOSHI
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nikon Corp
Original Assignee
Nikon Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nikon Corp filed Critical Nikon Corp
Assigned to KEIO UNIVERSITY reassignment KEIO UNIVERSITY ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: KATADA, YUSAKU, KURIHARA, Toshihide, MASAYOSHI, Kanato, TSUBOTA, KAZUO
Assigned to NIKON CORPORATION reassignment NIKON CORPORATION ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: KEIO UNIVERSITY
Publication of US20210000343A1 publication Critical patent/US20210000343A1/en
Abandoned legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/047—Probabilistic or stochastic networks
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
    • A61B3/0016—Operational features thereof
    • A61B3/0025—Operational features thereof characterised by electronic signal processing, e.g. eye models
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
    • A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
    • A61B3/12—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions for looking at the eye fundus, e.g. ophthalmoscopes
    • A61B3/1241—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions for looking at the eye fundus, e.g. ophthalmoscopes specially adapted for observation of ocular blood flow, e.g. by fluorescein angiography
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
    • A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
    • A61B3/14—Arrangements specially adapted for eye photography
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
    • A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
    • A61B5/1126—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb using a particular sensing technique
    • A61B5/1128—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb using a particular sensing technique using image analysis
    • G06N3/0472—
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10064—Fluorescence image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30041—Eye; Retina; Ophthalmic

Definitions

  • the present invention relates to a diagnosis support device, a learning device, a diagnosis support method, a learning method, and a program.
  • Diabetic retinopathy is an important disease that is the second leading cause of blindness in Japan, but subjective symptoms do not appear until the stage of the disease has progressed considerably, and thus, early detection and early treatment at medical examinations, or the like, is important.
  • a fundus image analysis system that emphasizes a capillary blood vessel aneurysm that shows an initial change in diabetic retinopathy (see Patent Document 1) and an image analysis system that screens diabetic retinopathy from fundus images (see Patent Document 2) have been proposed.
  • Patent Document 1
  • Patent Document 2
  • Patent Document 1 and Patent Document 2 are aimed at discovery and screening of diseases as described above.
  • advancing one step from the disease screening of the diabetic retinopathy in the case of the stage of medical treatment or cure of diabetic retinopathy patients, it has become clear that the diabetic retinopathy is caused by an ischemic state due to vascular disorder associated with hyperglycemia so that the diabetic retinopathy is progressing, and thus, it is most important to understand retinal circulatory dynamics.
  • angiography of retina such as fundus fluorescein angiography is essential.
  • OCT angiography is a non-invasive device for understanding retinal circulation abnormality, but unlike the contrast examination, the OCT angiography cannot detect the blood flow dynamics and cannot detect the whole image of the retina due to a narrow angle of view while being effective for local diseases such as a macular disease. Accordingly, it is not suitable for detecting the retinal circulatory dynamics of ocular ischemic diseases such as diabetic retinopathy.
  • the present invention has been made in view of such circumstances, and an object of the present invention is to quickly and easily recognize abnormal circulatory findings from a fundus image without performing fundus fluorescein angiography.
  • a diagnosis support device including: a identification unit that is configured to recognize an area of abnormality in blood circulation in a fundus image that is an image of a fundus, using a trained model obtained by learning a relationship between the fundus image and the area of abnormality in blood circulation in the fundus image, on the basis of the fundus image and the area of abnormality in blood circulation specified on the basis of a fluorescein angiography image of the fundus; and an output unit output information indicating the fundus image of a patient and the area of abnormality in blood circulation in the fundus image of the patient identified by the identification unit using the trained model.
  • the area of abnormality in blood circulation is generated on the basis of the fluorescein angiography image and an ophthalmologist's diagnostic note regarding one or both of a retinal non-perfusion area and a neovascularization attached to the fluorescein angiography image.
  • the identification unit is configured to recognize one or both of a retinal non-perfusion area and an area corresponding to a neovascularization in the fundus image.
  • the output unit is configured to output an image in which the area of abnormality in blood circulation identified by the identification unit is overlaid on the fundus image.
  • a learning device includes: a learning unit that generates, on the basis of a fundus image that is an image of a fundus and an area of abnormality in blood circulation specified on the basis of a fluorescein angiography image of the fundus, a trained model indicating a relationship between the fundus image and the area of abnormality in blood circulation in the fundus image, through learning.
  • the area of abnormality in blood circulation is generated on the basis of the fluorescein angiography image, and an ophthalmologist's diagnostic note regarding one or both of the retinal non-perfusion area and the neovascularization attached to the fluorescein angiography image.
  • a diagnosis support method executed by a diagnosis support device including: recognizing an area of abnormality in blood circulation in a fundus image that is an image of a fundus, using a trained model obtained by learning a relationship between the fundus image and the area of abnormality in blood circulation in the fundus image, on the basis of the fundus image and the area of abnormality in blood circulation specified on the basis of a fluorescein angiography image of the fundus; and outputting information indicating the fundus image of a patient and the area of abnormality in blood circulation in the fundus image of the patient identified by the identification unit using the trained model.
  • a learning method executed by a learning device including: acquiring information indicating a fundus image that is an image of a fundus, and information indicating an area of abnormality in blood circulation specified on the basis of a fluorescein angiography image of the fundus; and generating, on the basis of the acquired fundus image and the acquired area of abnormality in blood circulation specified on the basis of the fluorescein angiography image of the fundus, a trained model indicating a relationship between the fundus image and the area of abnormality in blood circulation in the fundus image, through learning.
  • a program causing a computer of a diagnosis support device to execute: recognizing an area of abnormality in blood circulation in a fundus image that is an image of a fundus, using a trained model obtained by learning a relationship between the fundus image and the area of abnormality in blood circulation in the fundus image, on the basis of the fundus image and the area of abnormality in blood circulation specified on the basis of a fluorescein angiography image of the fundus; and outputting information indicating the fundus image of a patient and the area of abnormality in blood circulation in the fundus image of the patient identified using the trained model.
  • a program causing a computer of a learning device to execute: acquiring information indicating a fundus image that is an image of a fundus, and information indicating an area of abnormality in blood circulation specified on the basis of a fluorescein angiography image of the fundus; and generating, on the basis of the acquired fundus image and the acquired area of abnormality in blood circulation specified on the basis of the fluorescein angiography image of the fundus, a trained model indicating a relationship between the fundus image and the area of abnormality in blood circulation in the fundus image, through learning.
  • the “fundus image” means an image obtained by imaging the fundus of a patient with a fundus camera in order to make a diagnosis regarding a disease of an ophthalmological system
  • the “fluorescein angiography image” means an image obtained by continuously taking a photograph of the fundus using an excitation light and a filter corresponding to fluorescence dye while injecting the fluorescence dye through a vein of the patient's arm and taking a contrast image of a circulating state of the fundus.
  • FIG. 1 is a configuration diagram illustrating an information processing system including a server that is an embodiment of an information processing device of the present invention.
  • FIG. 2 is a block diagram illustrating a hardware configuration of the server in the information processing system shown in FIG. 1 .
  • FIG. 3 is a diagram illustrating an example of a functional configuration for realizing various processes such as an NPA/NV existence probability map generating process, an accompanying finding existence probability map generating process, an estimated NPA/NV recognizing process, and an estimated NPA/NV display process in a functional configuration of the server shown in FIG. 2 .
  • FIG. 4 is a flowchart illustrating a flow of various processes such as the NPA/NV existence probability map generating process, the accompanying finding existence probability map generating process, the estimated NPA/NV recognizing process, and the estimated NPA/NV display process executed by the server shown in FIG. 3 .
  • FIG. 5 is a diagram illustrating a flow of various kinds of information used in the processes executed by the server shown in FIG. 3 .
  • FIG. 6 is a diagram illustrating an example of fundus image information acquired in the processes executed by the server shown in FIG. 3 .
  • FIG. 7 is a diagram illustrating an example of fluorescein angiography image information acquired in the processes executed by the server shown in FIG. 3 .
  • FIG. 8 is a diagram illustrating an example of an NPA/NV existence probability map output in the NPA/NV existence probability map generating process executed by the server shown in FIG. 3 .
  • FIG. 9 is a diagram illustrating an estimated NPA and an estimated NV identified in the estimated NPA/NV recognizing process executed by the server shown in FIG. 3 .
  • FIG. 10 is a configuration diagram illustrating an information processing system according to a modified example of the embodiment of the present invention.
  • FIG. 11 is a block diagram illustrating an example of a learning device according to a modified example of the embodiment of the present invention.
  • FIG. 12 is a diagram illustrating an example of a fundus image.
  • FIG. 13 is a diagram illustrating an example of a fluorescein angiography image.
  • FIG. 14 is a diagram illustrating an example of an area of abnormality in blood circulation.
  • FIG. 15 is a diagram illustrating an example of a structure of a neural network.
  • FIG. 16 is a block diagram illustrating an example of a diagnosis support device according to a modified example of the embodiment of the present invention.
  • FIG. 17 is a flowchart illustrating an example of an operation of a learning device according to the modified example of the embodiment of the present invention.
  • FIG. 18 is a flowchart illustrating an example of an operation of the diagnosis support device according to the modified example of the embodiment of the present invention.
  • FIG. 1 is a configuration diagram illustrating an information processing system including a server 1 that is an embodiment of an information processing device of the present invention.
  • the information processing system shown in FIG. 1 is configured to include the server 1 , an ophthalmologist terminal 2 , and an examination device 3 .
  • the server 1 , the ophthalmologist terminal 2 , and the examination device 3 are connected to each other through a network N such as the Internet.
  • the server 1 is a server that manages the information processing system shown in FIG. 1 , and executes various processes such as an NPA/NV existence probability map generating process, an accompanying finding existence probability map generating process, and an estimated NPA/NV recognizing process.
  • the content of specific processes executed by the server 1 will be described later with reference to FIG. 3 .
  • the “NPA/NV existence probability map generating process” is a series of processes executed by the server 1 from generation of NPA/NV training information to generation of an NPA/NV existence probability map.
  • the “NPA/NV training information” refers to training information for calculating a retinal non-perfusion area (NPA/non-perfusion area) (hereinafter, referred to as “a retinal non-perfusion area” or “NPA”) existence probability (hereinafter, referred to as an “NPA existence probability”) and a neovascularization (NV/neovascularization) existence probability (hereinafter, referred to as an “NV existence probability”), in fundus image information of a patient.
  • a retinal non-perfusion area hereinafter, referred to as “a retinal non-perfusion area” or “NPA”
  • NPA existence probability a neovascularization
  • NV existence probability neovascularization
  • the “retinal non-perfusion area” refers to a poor circulatory area of the retina that results from retinal vessel occlusion in ocular ischemic diseases.
  • the “NPA/NV existence probability map” is image information in which the NPA existence probability and the NV existence probability are identifiably displayed in fundus image information.
  • the “fundus image information” refers to image information based on a fundus image.
  • fluorescein angiography image information refers to information based on a fluorescein angiography image.
  • the “NPA/NV annotation information” refers to a diagnostic note of an ophthalmologist D, attached to fluorescein angiography image information, regarding at least one of a retinal non-perfusion area (NPA) or a neovascularization (NV).
  • NPA retinal non-perfusion area
  • NV neovascularization
  • neovascularization means that poor retinal circulation and ischemia have further progressed in the retinal non-perfusion area. In a case where bleeding or retinal detachment through proliferative membrane formation occurs from the neovascularization, it eventually leads to blindness, and thus, it is very important to specify a poor circulatory area of the retina such as a retinal non-perfusion area or a neovascularization.
  • the “accompanying finding existence probability map generating process” refers to a series of processes executed by the server 1 from generation of the accompanying finding training information to generation of the accompanying finding existence probability map.
  • the “accompanying finding training information” refers to training information in calculating an accompanying finding existence probability in fundus image information of a patient. Specifically, the “accompanying finding training information” refers to training information generated on the basis of the fluorescein angiography image information and the fundus image information, and accompanying finding annotation information attached to these image information.
  • the “accompanying finding annotation information” means a diagnostic note other than the diagnostic note regarding a retinal non-perfusion area (NPA) or a neovascularization (NV), among diagnostic notes indicating that the ophthalmologist D incidentally makes a judgment “not normal” with respect to the fluorescein angiography image information and the fundus image information.
  • NPA retinal non-perfusion area
  • NV neovascularization
  • information such as a capillary aneurysm, fundus bleeding, vitiligo, soft, vitiligo, venous abnormality, intra-retinal microvascular abnormality, vitreous hemorrhage, a proliferative membrane, and retinal detachment is an example of the “accompanying finding annotation information”.
  • the “accompanying finding existence probability map” is image information (not shown) in which the existence probability of the accompanying finding is identifiably displayed in the fundus image information.
  • the “estimated NPA/NV recognizing process” refers to a series of processes from a process of recognizing an area estimated to correspond to the retinal non-perfusion area (NPA) in the fundus image information as an estimated NPA, on the basis of the NPA existence probability, and a process of recognizing an area estimated to correspond to the neovascularization (NV) in the fundus image information as an estimated NV, on the basis of the NV existence probability, in the processes executed by the server 1 .
  • NPA retinal non-perfusion area
  • NV neovascularization
  • the ophthalmologist terminal 2 is an information processing device operated by the ophthalmologist D, and is configured by, for example, a personal computer or the like.
  • the ophthalmologist terminal 2 transmits NPA/NV annotation information and accompanying finding annotation information to the server 1 , and acquires information on estimated NPA/NV identified by the server 1 , for example.
  • the various types of information acquired by the ophthalmologist terminal 2 is output from the ophthalmologist terminal 2 , and is used for examination by the ophthalmologist D.
  • the examination device 3 is configured by various devices used in eye examination of a patient.
  • the examination device 3 transmits each of fundus image information obtained by imaging in fundus examination and fluorescein angiography image information obtained by imaging in fundus fluorescein angiography to the server 1 .
  • FIG. 2 is a block diagram illustrating a hardware configuration of the server 1 in the information processing system of FIG. 1 .
  • the server 1 includes a central processing unit (CPU) 11 , a read only memory (ROM) 12 , a random access memory (RAM) 13 , a bus 14 , an input/output interface 15 , an output unit 16 , an input unit 17 , a storage unit 18 , a communication unit 19 , and a drive 20 .
  • CPU central processing unit
  • ROM read only memory
  • RAM random access memory
  • the CPU 11 executes various processes according to a program recorded on the ROM 12 or a program loaded from the storage unit 18 into the RAM 13 .
  • the RAM 13 also appropriately stores data and the like necessary in a case where the CPU 11 executes various processes.
  • the CPU 11 , ROM 12 , and RAM 13 are connected to each other through the bus 14 .
  • the input/output interface 15 is also connected to the bus 14 .
  • the output unit 16 , the input unit 17 , the storage unit 18 , the communication unit 19 , and the drive 20 are connected to the input/output interface 15 .
  • the output unit 16 is configured by various liquid crystal displays or the like, and outputs various types of information.
  • the input unit 17 is configured by various types of hardware, through which various types of information are input.
  • the storage unit 18 is configured by a dynamic random access memory (DRAM) or the like, and stores various types of data.
  • DRAM dynamic random access memory
  • the communication unit 19 controls communication with other devices through the network N including the Internet.
  • a drive 20 is provided as necessary.
  • a removable medium 30 including a magnetic disc, an optical disc, a magneto-optical disc, a semiconductor memory, or the like is appropriately mounted on the drive 20 .
  • a program read from the removable medium 30 by the drive 20 is installed in the storage unit 18 as necessary.
  • the removable medium 30 may also store various types of data stored in the storage unit 18 , similarly to the storage unit 18 .
  • FIG. 3 is a functional block diagram illustrating an example of a functional configuration for realizing the NPA/NV existence probability map generating process, the accompanying finding existence probability map generating process, the estimated NPA/NV recognizing process, and the estimated NPA/NV display process in the functional configuration of the server 1 of FIG. 2 , in the information processing system of FIG. 1 .
  • an image acquisition unit 101 In the CPU 11 ( FIG. 2 ) of the server 1 , in a case where the NPA/NV existence probability map generating process is executed, an image acquisition unit 101 , an annotation acquisition unit 102 , a training information generation unit 103 , and an arithmetic unit 104 perform their functions.
  • the image acquisition unit 101 the annotation acquisition unit 102 , the training information generation unit 103 , and the arithmetic unit 104 perform their functions.
  • an estimated NPA/NV identification unit 106 performs its function.
  • an estimated NPA/NV display control unit 107 performs its function.
  • An image DB 401 , an annotation DB 402 , and a training DB 403 are provided in one area of the storage unit 18 ( FIG. 2 ).
  • the storage unit 18 may be disposed in the ophthalmologist terminal 2 instead of the server 1 .
  • the image acquisition unit 101 acquires fluorescein angiography image information of a patient and fundus image information of the patient.
  • the fluorescein angiography image information and the fundus image information acquired by the image acquisition unit 101 are respectively stored in the image DB 401 for management.
  • fluorescein angiography image information based on the fluorescein angiography image is transmitted to the server 1 .
  • fundus image information based on the fundus image is transmitted to the server 1 .
  • the image acquisition unit 101 of the server 1 acquires the fluorescein angiography image information and the fundus image information transmitted from the examination device 3 to the server 1 , and stores the image information in the image DB 401 .
  • the server 1 can accurately manage the fluorescein angiography image information and the fundus image information of the patient without omission.
  • the annotation acquisition unit 102 acquires a diagnostic note of the ophthalmologist D, which is attached to the fluorescein angiography image information of the patient, regarding at least one of a retinal non-perfusion area (NPA) or a neovascularization (NV) as NPA/NV annotation information.
  • NPA retinal non-perfusion area
  • NV neovascularization
  • the ophthalmologist terminal 2 transmits the diagnostic note to the server 1 as the NPA/NV annotation information based on the operation of the ophthalmologist D.
  • the annotation acquisition unit 102 of the server 1 acquires the NPA/NV annotation information transmitted from the ophthalmologist terminal 2 , and stores the information in the annotation DB 402 .
  • the fluorescein angiography image information and the NPA/NV annotation information attached to the image information are managed in association with each other.
  • the server 1 can manage, without omission, the diagnostic note of the ophthalmologist D regarding at least one of the retinal non-perfusion area (NPA) or the neovascularization (NV) attached to the fluorescein angiography image information, as the NPA/NV annotation information.
  • NPA retinal non-perfusion area
  • NV neovascularization
  • the annotation acquisition unit 102 acquires the diagnostic note of the ophthalmologist D regarding an accompanying finding attached to the fluorescein angiography image and the fundus image, as accompanying finding annotation information. Specifically, in a case where the diagnostic note of the ophthalmologist D regarding the accompanying finding is attached to the fluorescein angiography image information and the fundus image information, the ophthalmologist terminal 2 transmits the diagnostic note to the server 1 as the accompanying finding annotation information on the basis of the operation of the ophthalmologist D.
  • the annotation acquisition unit 102 of the server 1 acquires the accompanying finding annotation information transmitted from the ophthalmologist terminal 2 , and stores this information in the annotation DB 402 .
  • the fluorescein angiography image information and the fundus image information, and the accompanying finding annotation information attached to these image information are managed in association with each other.
  • the server 1 can manage the diagnostic note of the ophthalmologist D regarding the accompanying finding attached to the fluorescein angiography image information and the fundus image information as the accompanying finding annotation information, without omission.
  • the training information generation unit 103 generates NPA/NV training information that serves as training information for calculating an NPA existence probability and an NV existence probability on the basis of the fluorescein angiography image information and the NPA/NV annotation information corresponding to the fluorescein angiography image information.
  • the image DB 401 stores fluorescein angiography image information obtained from a plurality of patients
  • the annotation DB 402 stores NPA/NV annotation information.
  • the training information generation unit 103 generates NPA/NV training information which serves as training information in calculating the NPA existence probability and the NV existence probability in the fundus image information on the basis of the information stored in these databases.
  • the server 1 can generate and store the training information for calculating the NPA existence probability and the NV existence probability in the fundus image information of the patients.
  • the training information generation unit 103 generates the accompanying finding training information on the basis of the fluorescein angiography image information and the fundus image information, and the accompanying finding annotation information corresponding to these image information.
  • the image DB 401 stores the fundus image information and the fluorescein angiography image information obtained from a plurality of patients
  • the annotation DB 402 stores the accompanying finding annotation information.
  • the training information generation unit 103 generates the accompanying finding training information that serves as training information in calculating the existence probability of the accompanying findings in the fundus image information, on the basis of the information stored in these databases.
  • the server 1 can store the training information for calculating the existence probability of the accompanying findings in the fundus image information of the patients.
  • the arithmetic unit 104 calculates the NPA existence probability and the NV existence probability on the basis of at least the NPA/NV training information. In addition, in a case where an accompanying finding existence probability map (which will be described later) is generated, the arithmetic unit 104 calculates the NPA existence probability and the NV existence probability on the basis of the accompanying finding existence probability map and the NPA/NV training information.
  • a specific method for calculating the NPA existence probability and the NV existence probability is not particularly limited.
  • a method for calculating the NPA existence probability and the NV existence probability by extracting a feature common in the fundus images having the NPA/NV from the NPA/NV training information and normalizing the matching degree with the feature with respect to whether or not the fundus image information of the patient has the feature may be used.
  • the NPA existence probability and the NV existence probability may be calculated using a deep learning technique.
  • NPA retinal non-perfusion area
  • NV neovascularization
  • the arithmetic unit 104 calculates the existence probability of the accompanying finding in the fundus image information on the basis of the accompanying finding training information.
  • a specific method of calculating the existence probability of the accompanying findings is not particularly limited.
  • a method for calculating the accompanying finding existence probability by extracting a feature common in the fundus images having the accompanying finding from the accompanying finding training information and normalizing the matching degree with the feature with respect to whether or not the fundus image information of the patient has the feature may be used.
  • the accompanying finding existence probability may be calculated using a deep learning technique.
  • NPA retinal non-perfusion area
  • NV neovascularization
  • the map generation unit 105 generates an NPA/NV existence probability map as image information in which the NPA existence probability and the NV existence probability are identifiably displayed in the fundus image information. Specifically, image information such as an NPA/NV existence probability map E illustrated in FIG. 8 is generated.
  • NPA retinal non-perfusion area
  • NV neovascularization
  • a method of identifiably displaying the NPA existence probability and the NV existence probability in the fundus image information is not particularly limited.
  • the NPA existence probability and the NV existence probability may be identifiably displayed according to a color difference, or the NPA existence probability and the NV existence probability may be identifiably displayed according to color shades.
  • the map generation unit 105 also generates an accompanying finding existence probability map (not shown) as image information in which the accompanying finding existence probability is identifiably displayed the fundus image information.
  • NPA retinal non-perfusion area
  • NV neovascularization
  • a method for identifiably displaying the accompanying finding existence probability in the fundus image information is not particularly limited.
  • the accompanying finding existence probability may be identifiably displayed according to a color difference, or the accompanying finding existence probability may be identifiably displayed according to color shades.
  • the estimated NPA/NV identification unit 106 identifies an area of the fundus image information that is estimated to correspond to the retinal non-perfusion area (NPA), as an estimated NPA, on the basis of the NPA existence probability and the NV existence probability, and identifies an area that is estimated to correspond to the neovascularization (NV), as an estimated NV.
  • NPA retinal non-perfusion area
  • NV neovascularization
  • an area in which the NPA existence probability and the NV existence probability exceed a predetermined threshold value is identified as an area that is estimated to correspond to the retinal non-perfusion area (NPA) (estimated NPA), or an area that is estimated to correspond to the neovascularization (estimated NV).
  • the threshold value may be discretionally changed according to judgment of the ophthalmologist D.
  • NPA retinal non-perfusion area
  • NV neovascularization
  • the estimated NPA/NV display control unit 107 executes a control for displaying the estimated NPA area and the estimated NV area on the fundus image information.
  • the area where the existence of the retinal non-perfusion is estimated (estimated NPA) and the area where the existence of the neovascularization (NV) is estimated (estimated NV) are displayed to be overlaid on the image information based on the fundus image.
  • FIG. 4 is a flowchart illustrating a flow of a series of processes executed by the server 1 of FIG. 3 .
  • the server 1 executes the following series of processes.
  • step S 1 the image acquisition unit 101 determines whether or not fluorescein angiography image information has been transmitted from the examination device 3 .
  • step S 1 In a case where the fluorescein angiography image information has been transmitted, the determination in step S 1 is affirmative, and the procedure proceeds to step S 2 . On the other hand, in a case where the fluorescein angiography image information has not been transmitted, the determination in step S 1 is negative, and the procedure returns to step S 1 . That is, the determination process of step S 1 is repeated until the fluorescein angiography image information is transmitted. Thereafter, in a case where the fluorescein angiography image information is transmitted, the determination in step S 1 becomes affirmative, and the procedure proceeds to step S 2 .
  • step S 2 the image acquisition unit 101 acquires the transmitted fluorescein angiography image information.
  • step S 3 the annotation acquisition unit 102 determines whether or not NPA/NV annotation information has been transmitted from the ophthalmologist terminal 2 .
  • step S 3 In a case where the NPA/NV annotation information has been transmitted, the determination in step S 3 is affirmative, and the procedure proceeds to step S 4 . On the other hand, in a case where the NPA/NV annotation information has not been transmitted, the determination in step S 3 is negative, and the procedure returns to step S 3 . That is, the determination process of step S 3 is repeated until the NPA/NV annotation information is transmitted. Thereafter, in a case where the NPA/NV annotation information is transmitted, the determination in step S 3 becomes affirmative, and the procedure proceeds to step S 4 .
  • step S 4 the annotation acquisition unit 102 acquires the transmitted NPA/NV annotation information.
  • step S 5 the training information generation unit 103 generates NPA/NV training information on the basis of the fluorescein angiography image information and the NPA/NV annotation information corresponding to the fluorescein angiography image information.
  • step S 6 the image acquisition unit 101 determines whether or not fundus image information has been transmitted from the examination device 3 .
  • step S 6 In a case where the fundus image information has been transmitted, the determination in step S 6 is affirmative, and the procedure proceeds to step S 7 . On the other hand, in a case where the fundus image information has not been transmitted, the determination in step S 6 is negative, and the procedure returns to step S 6 . That is, the determination process of step S 6 is repeated until the fundus image information is transmitted. Thereafter, in a case where the fundus image information is transmitted, the determination in step S 6 becomes affirmative, and the procedure proceeds to step S 7 .
  • step S 7 the image acquisition unit 101 acquires the transmitted fundus image information.
  • step S 8 the annotation acquisition unit 102 determines whether or not accompanying finding annotation information has been transmitted from the ophthalmologist terminal 2 .
  • step S 8 In a case where the accompanying finding annotation information has been transmitted, the determination in step S 8 is affirmative, and the procedure proceeds to Step S 9 . On the other hand, in a case where the accompanying finding annotation information has not been transmitted, the determination in step S 8 is negative, and the procedure skips steps S 9 to S 11 and proceeds to step S 12 .
  • step S 9 the annotation acquisition unit 102 acquires the accompanying finding annotation information.
  • step S 10 the training information generation unit 103 generates the accompanying finding training information on the basis of the fluorescein angiography image information and the NPA/NV annotation information corresponding to the fluorescein angiography image information.
  • step S 11 the arithmetic unit 104 calculates an existence probability of an accompanying finding in the fundus image information on the basis of the accompanying finding training information.
  • step S 12 the map generation unit 105 generates an accompanying finding existence probability map as image information in which the accompanying finding existence probability is identifiably displayed in the fundus image information.
  • step S 13 the arithmetic unit 104 calculates the NPA existence probability and the NV existence probability on the basis of at least the NPA/NV training information. Further, in a case where the accompanying finding existence probability map is generated, the arithmetic unit 104 calculates the NPA existence probability and the NV existence probability on the basis of the NPA/NV training information and the accompanying finding existence probability map.
  • step S 14 the map generation unit 105 generates an NPA/NV existence probability map as image information in which the NPA existence probability and the NV existence probability are identifiably displayed in the fundus image information.
  • step S 15 the estimated NPA/NV identification unit 106 identifies an area that is estimated to correspond to the retinal non-perfusion area (NPA) in the fundus image information, as an estimated NPA, and identifies an area that is estimated to correspond to the neovascularization (NV), as an estimated NV, on the basis of the NPA existence probability and the NV existence probability.
  • NPA retinal non-perfusion area
  • NV neovascularization
  • step S 16 the estimated NPA/NV display control unit 107 executes a control for displaying the estimated NPA and the NV area on the fundus image information.
  • step S 17 the server 1 determines whether or not there is a process ending command.
  • step S 17 In a case where there is no process ending command, the determination in step S 17 is negative, and the procedure returns to step S 1 . On the other hand, in a case where there is the process ending command, the determination in step S 17 is affirmative, and the procedure ends.
  • the estimated NPA area and the estimated NV area are displayed in the fundus image information.
  • FIG. 5 is a diagram illustrating a flow of various types of information in the processes executed by the server 1 .
  • the server 1 acquires fluorescein angiography image information based on the fluorescein angiography image.
  • a diagnostic note of the ophthalmologist D regarding at least one of the retinal non-perfusion area (NPA) or the neovascularization (NV) is attached to this fluorescein angiography image information.
  • the diagnostic note forms the NPA/NV training information together with the fluorescein angiography image information as the NPA/NV annotation information.
  • fundus image information based on the fundus image is acquired by the server 1 .
  • the fundus image information and the NPA/NV training information are used for a calculation process based on a NPA/NV annotation program of the arithmetic unit 104 .
  • an NPA/NV existence probability map is generated.
  • An estimated NPA and an estimated NV in the fundus image information are identified on the basis of the NPA/NV existence probability map.
  • a diagnostic note of the ophthalmologist D regarding an accompanying finding may be attached to the fluorescein angiography image information and the fundus image information.
  • the diagnostic note in this case forms the accompanying finding training information together with the fluorescein angiography image information and the fundus image information as the accompanying finding annotation information.
  • the fundus image information and the NPA/NV training information are used for a calculation process based on an accompanying finding determination program of the arithmetic unit 104 .
  • an accompanying finding existence probability map is generated.
  • the estimated NPA and the estimated NV in the fundus image information are identified on the basis of the accompanying finding existence probability map and the NPA/NV training information.
  • FIG. 6 is a diagram illustrating an example of fundus image information acquired in the process executed by the server 1 .
  • FIG. 7 is a diagram illustrating an example of fluorescein angiography image information acquired in the process executed by the server 1 .
  • fundus image information as shown in FIG. 6 is obtained.
  • the ophthalmologist D performs diagnosis with reference to the fundus image information as shown in FIG. 6 .
  • the fundus image information only shows findings resulting from abnormal perfusion such as bleeding and vitiligo, and it is difficult to read circulatory dynamics and an abnormal circulatory site. Therefore, fundus fluorescein angiography is performed, and accordingly, fluorescein angiography image information as shown in FIG. 7 is obtained.
  • retinal circulatory dynamics can be clearly discriminated from the fluorescein angiography image information.
  • the fundus fluorescein angiography is a burdensome examination for both patients and medical staffs due to the risk of the examination, its invasiveness, and geographical restrictions such as limitation to large hospitals, which causes hesitancy to take pictures of patients with good eyesight, patients with reduced physical functions, to perform repeated imaging, or the like, and thus, there are many cases in that detection of symptom appearances or disease states is delayed and treatment is delayed.
  • the server 1 performs an NPA/NV existence probability map generating process using the stored fluorescein angiography image information obtained from a plurality of patients and the NPA/NV training information generated on the basis of the NPA/NV annotation information attached to each piece of fluorescein angiography image information.
  • FIG. 8 is a diagram illustrating an example of the NPA/NV existence probability map output in the NPA/NV existence probability map generating process executed by the server 1 .
  • An NPA/NV existence probability map E shown in FIG. 8 is image information in which the NPA existence probability and the NV existence probability are identifiably displayed in the fundus image information.
  • the NPA existence probability and the NV existence probability may be identifiably displayed according to color differences or color shades.
  • the identification display may be performed in such a manner that an area indicated by a warm color has a high NPA existence probability and a high NV existence probability and an area indicated by a cold color has a low NPA existence probability and a low NV existence probability.
  • the identification display may be performed in such a manner that a dark color area has a lower NPA existence probability and a lower NV existence probability than a light color area.
  • NPA retinal non-perfusion area
  • NV neovascularization
  • FIG. 9 is a diagram illustrating an example of an estimated NPA and an estimated NV identified in the estimated NPA/NV recognizing process executed by the server 1 .
  • the server 1 On the basis of the content of the NPA/NV existence probability map, the server 1 identifies an area of the fundus image information that is estimated to correspond to the retinal non-perfusion area (NPA), as an estimated NPA, and identifies an area that is estimated to correspond to the neovascularization (NV), as an estimated NV.
  • NPA retinal non-perfusion area
  • NV neovascularization
  • an area A indicated by a broken line may be set as the estimated NPA
  • an area B may be set as the estimated NV.
  • the estimated NPA and the estimated NV may be displayed to be superimposed on the fundus image information.
  • a configuration in which the annotation acquisition unit 102 acquires, in a case where various types of annotation information are transmitted from the ophthalmologist terminal 2 , the various types of annotation information is shown, but a configuration in which in a case where the various types of annotation information is input to the ophthalmologist terminal 2 , the annotation acquisition unit 102 voluntarily goes to obtain the various types of annotation information may be used.
  • FIG. 2 The hardware configurations shown in FIG. 2 are merely examples for achieving the object of the present invention, and are not particularly limited.
  • the functional block diagram shown in FIG. 3 is merely an example, and is not particularly limited. That is, it is sufficient if the information processing system has functions capable of executing the above-described series of processes as a whole, and what kind of functional block is used to realize the function is not particularly limited to the example of FIG. 3 .
  • existence locations of the functional blocks are not limited to locations shown in FIG. 3 , and may be appropriately modified.
  • at least a part of the functional blocks of the server 1 may be provided in the ophthalmologist terminal 2 or the examination device 3 .
  • one functional block may be configured by a hardware unit, or may be configured by a combination with a software unit.
  • a program that forms the software is installed in a computer or the like from a network or a recording medium.
  • the computer may be a computer embedded in dedicated hardware.
  • the computer may be a computer capable of executing various functions by installation of various programs, for example, a general-purpose smartphone or a personal computer as well as a server.
  • the recording medium that stores such a program is not only configured by a removable medium that is distributed separately from a device body in order to provide the program to each user, but is also configured by a recording medium or the like provided to each user in a state of being incorporated in the device body in advance.
  • steps of describing the program recorded on the recording medium are not only processes sequentially performed in a time series manner, but may also include processes performed in parallel or individually although the processes are not necessarily performed in a time series manner.
  • step S 6 of FIG. 4 the image acquisition unit 101 determines whether or not the fundus image information has been transmitted from the examination device 3 , and acquires the fundus image information in step S 7 .
  • the fundus image information has only to be appropriately managed at a time point when the accompanying finding training information is generated.
  • the accompanying finding existence probability map generating process is not executed, the fundus image information has only to be appropriately managed at a time point when the NPA existence probability and the NV existence probability are calculated.
  • the fundus image information may be stored for management in the image DB 401 from any time point before the accompanying finding training information is generated in step S 10 . Further, in a case where the accompanying finding training information is not generated, the fundus image information may be stored for management in the image DB 401 from any time point before the NPA existence probability and the NV existence probability are calculated in step S 13 .
  • a program to which the present invention is applied may have the following configuration, and may have various embodiments.
  • the program to which the present invention is applied causes a computer that controls the information processing device to execute: a fluorescein angiography image acquisition step (for example, step S 2 in FIG. 4 ) of acquiring fluorescein angiography image information (for example, fluorescein angiography image information C in FIG. 7 ); an NPA/NV annotation acquisition step (for example, step S 4 in FIG. 4 ) of acquiring a diagnostic note of an ophthalmologist regarding at least one of a retinal non-perfusion area (NPA) or a neovascularization (NV), attached to the fluorescein angiography image information, as NPA/NV annotation information; an NPA/NV training information generation step (for example, step S 5 in FIG.
  • a fluorescein angiography image acquisition step for example, step S 2 in FIG. 4
  • fluorescein angiography image information for example, fluorescein angiography image information C in FIG. 7
  • NPA/NV annotation acquisition step for example
  • NPA/NV training information that is training information for calculating an existence probability of the retinal non-perfusion area (NPA) and an existence probability of the neovascularization (NV), on the basis of the fluorescein angiography image information and the NPA/NV annotation information corresponding to the fluorescein angiography image information;
  • a fundus image acquisition step for example, step S 7 in FIG. 4
  • fundus image information for example, fundus image information F in FIG. 6
  • NPA/NV existence probability calculation step for example, step S 13 in FIG.
  • an estimated NPA/NV identification step (for example, step S 15 in FIG. 4 ) of recognizing an area of the fundus image information that is estimated to correspond to the retinal non-perfusion area (NPA), as an estimated NPA, and recognizing an area that is estimated to correspond to the neovascularization (NV), as an estimated NV, on the basis of the existence probability of the retinal non-perfusion area and the existence probability of the neovascularization.
  • the program may cause the computer to execute the control process further including: an NPA/NV existence probability map generation step (for example, step S 14 in FIG. 4 ) of generating an NPA/NV existence probability map (for example, the NPA/NV existence probability map E in FIG. 8 ) in which the NPA existence probability and the NV existence probability are identifiably displayed in the fundus image information.
  • an NPA/NV existence probability map generation step for example, step S 14 in FIG. 4
  • an NPA/NV existence probability map for example, the NPA/NV existence probability map E in FIG. 8
  • NPA retinal non-perfusion area
  • NV neovascularization
  • the program may cause the computer to further execute: an accompanying annotation acquisition step (for example, step S 9 in FIG. 4 ) of acquiring, as accompanying finding annotation information, a diagnostic note of the ophthalmologist regarding an accompanying finding, attached to the fluorescein angiography image information and the fundus image information; an accompanying finding training information generation step (for example, step S 10 in FIG.
  • an accompanying annotation acquisition step for example, step S 9 in FIG. 4
  • an accompanying finding training information generation step for example, step S 10 in FIG.
  • a control process of calculating the existence probability of the retinal non-perfusion area (NPA) and the existence probability of the neovascularization (NV) in the fundus image information on the basis of the accompanying finding existence probability and the NPA/NV training information may be executed.
  • the program may cause the computer to further execute: an accompanying finding existence probability map generation step (for example, step S 12 in FIG. 4 ) of generating an accompanying finding existence probability map in which the existence probability of the accompanying finding is identifiably displayed in the fundus image information.
  • an accompanying finding existence probability map generation step for example, step S 12 in FIG. 4 .
  • FIG. 10 is a configuration diagram of an information processing system of a modified example according to the embodiment of the present invention.
  • the information processing system of the modified example according to the embodiment of the present invention includes the ophthalmologist terminal 2 , the examination device 3 , a learning device 200 , and a diagnosis support device 300 . These devices are connected to each other through a network N.
  • the learning device 200 generates, on the basis of a fundus image that is an image of the fundus and an area of abnormality in blood circulation specified on the basis of a fluorescein angiography image of the fundus, a trained model indicating a relationship between the fundus image and the area of abnormality in blood circulation in the fundus image through learning.
  • the area of abnormality in blood circulation is an area in which blood circulation is abnormal due to blood damage on the retina that occurs in ocular ischemic diseases such as diabetic retinopathy.
  • the learning device 200 acquires information indicating a fundus image of a patient and information indicating a fluorescein angiography image of the patient, and stores the information indicating the fundus image and the information indicating the fluorescein angiography image acquired above in association with each other. Specifically, in a fundus examination of a patient, the examination device 3 captures a fundus image of the patient, creates fundus image notification information including a patient ID and information indicating the captured fundus image, to be addressed to the learning device 200 , and transmits the created fundus image notification information to the learning device 200 .
  • the examination device 3 captures a fluorescein angiography image of the patient, creates fluorescein angiography image notification information including the patient ID and information indicating the captured fluorescein angiography image, to be addressed to the learning device 200 , and transmits the created fluorescein angiography image notification information to the learning device 200 .
  • the learning device 200 acquires the patient ID and the information indicating the fundus image included in the fundus image notification information transmitted from the examination device 3 to the learning device 200 , the patient ID and the information indicating the fluorescein angiography image included in the fluorescein angiography image notification information, and stores the patient ID, the information indicating the fundus image, and the information indicating the fluorescein angiography image acquired above in association with each other.
  • the learning device 200 acquires a diagnostic note of the ophthalmologist D regarding any one or both of the retinal non-perfusion area (NPA) and the neovascularization (NV), attached to the fluorescein angiography image of the patient, as NPA/NV annotation information.
  • NPA retinal non-perfusion area
  • NV neovascularization
  • the ophthalmologist terminal 2 creates NPA/NV annotation notification information including the patient ID and the diagnostic note, to be addressed to the learning device 200 , on the basis of an operation of the ophthalmologist D, and transmits the created NPA/NV annotation notification information to the learning device 200 .
  • the learning device 200 receives the NPA/NV annotation notification information transmitted by the ophthalmologist terminal 2 , and stores the NPA/NV annotation information included in the received NPA/NV annotation notification information.
  • the information indicating the fluorescein angiography image and the NPA/NV annotation information attached to the fluorescein angiography image are stored in association with each other.
  • the learning device 200 acquires a diagnostic note of the ophthalmologist D regarding the accompanying finding, attached to the fundus image and the fluorescein angiography image, as accompanying finding annotation information. Specifically, in a case where the diagnostic note of the ophthalmologist D regarding the accompanying findings is attached to the information indicating the fluorescein angiography image and the information indicating the fundus image, the ophthalmologist terminal 2 creates accompanying finding annotation notification information including the patient ID and the diagnostic note, to be addressed to the learning device 200 , on the basis of an operation of the ophthalmologist D, and transmits the created accompanying finding annotation notification information to the learning device 200 .
  • the learning device 200 receives the accompanying finding annotation notification information transmitted by the ophthalmologist terminal 2 , acquires the patient ID and the accompanying finding annotation information included in the received accompanying finding annotation notification information, and stores the acquired patient ID and the accompanying finding annotation information.
  • the information indicating the fundus image, the information indicating the fluorescein angiography image, and the accompanying finding annotation information are stored in association with each other.
  • the learning device 200 acquires the information indicating the fundus image, the information indicating the fluorescein angiography image associated with the information indicating the fundus image, the NPA/NV annotation information, and the accompanying finding annotation information, and specifies an area of abnormality in blood circulation on the basis of the information indicating the fluorescein angiography image, the NPA/NV annotation information, and the accompanying finding annotation information acquired above.
  • the learning device 200 generates NPA/NV learning information in which the information correlating the fundus image and the area of abnormality in blood circulation specified on the basis of the fluorescein angiography image corresponding to the fundus image, and stores the generated NPA/NV learning information.
  • the learning device 200 generates, using the information indicating the fundus image included in the NPA/NV learning information as input information, and using the area of abnormality in blood circulation specified on the basis of the fluorescein angiography image corresponding to the fundus image as training information, a trained model indicating a relationship between the fundus image and the area of abnormality in blood circulation in the fundus image through learning.
  • a specific method of generating the trained model is not particularly limited. For example, a method for extracting a feature common in the fundus images having the area of abnormality in blood circulation from the NPA/NV learning information, and deriving a relationship between the extracted common feature and the area of abnormality in blood circulation in the fundus image may be used.
  • the relationship between the fundus image and the area of abnormality in blood circulation in the fundus image may be derived using a neural network or a deep learning technique.
  • the learning device 200 stores the generated trained model, creates trained model notification information including the trained model, to be addressed to the diagnosis support device 300 , and transmits the created trained model notification information to the diagnosis support device 300 .
  • the diagnosis support device 300 receives the trained model transmitted by the learning device 200 , and stores the received trained model.
  • the ophthalmologist terminal 2 creates patient information including the patient ID and the information indicating the fundus image of the patient, to be addressed to the diagnosis support device 300 , and transmits the created patient information to the diagnosis support device 300 .
  • the diagnosis support device 300 receives the patient information transmitted by the ophthalmologist terminal 2 , and acquires the patient ID and the information indicating the fundus image of the patient included in the received patient information.
  • the diagnosis support device 300 specifies the area of abnormality in blood circulation in the fundus image on the basis of the acquired information indicating the fundus image, using the stored trained model.
  • the diagnosis support device 300 creates a diagnosis result including the information indicating an area of abnormality in blood circulation in the fundus image identified using the patient fundus image and the trained model and the patient ID, to be addressed to the ophthalmologist terminal 2 , and transmits the created diagnosis result to the ophthalmologist terminal 2 .
  • the learning device 200 and the diagnosis support device 300 included in the information processing system will be described.
  • FIG. 11 is a block diagram illustrating an example of a learning device according to a modified example of the embodiment of the present invention.
  • the learning device 200 includes a communication unit 205 , a storage unit 210 , an operation unit 220 , an information processing unit 230 , a display unit 240 , and a bus line 250 such as an address bus or a data bus for electrically connecting the respective components as shown in FIG. 11 .
  • the communication unit 205 is realized by a communication module.
  • the communication unit 205 communicates with external communication devices such as the ophthalmologist terminal 2 , the examination device 3 , or the diagnosis support device 300 through the network N.
  • the communication unit 205 receives fundus image notification information transmitted by the examination device 3 , and outputs the received fundus image notification information to the information processing unit 230 .
  • the communication unit 205 receives fluorescein angiography image notification information transmitted by the examination device 3 , and outputs the received fluorescein angiography image notification information to the information processing unit 230 .
  • the communication unit 205 receives NPA/NV annotation notification information transmitted by the ophthalmologist terminal 2 , and outputs the received NPA/NV annotation notification information to the information processing unit 230 .
  • the communication unit 205 receives accompanying finding annotation notification information transmitted by the ophthalmologist terminal 2 , and outputs the received accompanying finding annotation notification information to the information processing unit 230 .
  • the communication unit 205 acquires trained model notification information output by the information processing unit 230 , and transmits the acquired trained model notification information to the diagnosis support device 300 .
  • the storage unit 210 is realized by, for example, a random access memory (RAM), a read only memory (ROM), a hard disk drive (HDD), a flash memory, or a hybrid storage device in which a plurality of these memories are combined.
  • the storage unit 210 stores a program 211 executed by the information processing unit 230 , an application 212 , an image DB 213 , an annotation DB 214 , learning information storage 215 , and a trained model 216 .
  • the program 211 is, for example, an operating system, which is located between a user or an application program and hardware, provides a standard interface for the user or the application program, and performs efficient management for each resource such as hardware.
  • the application 212 causes the learning device 200 to receive the fundus image notification information transmitted by the examination device 3 , and to store the patient ID and the information indicating the fundus image included in the received fundus image notification information in association with each other.
  • the application 212 causes the learning device 200 to receive the fluorescein angiography image notification information transmitted by the examination device 3 , and to store the patient ID and the information indicating the fluorescein angiography image included in the received fluorescein angiography image notification information in association with each other.
  • the application 212 causes the learning device 200 to receive the NPA/NV annotation notification information transmitted by the ophthalmologist terminal 2 , and to store the patient ID and the NPA/NV annotation information included in the received NPA/NV annotation notification information in association with each other.
  • the application 212 causes the learning device 200 to receive the accompanying finding annotation notification information transmitted by the ophthalmologist terminal 2 , and to store the patient ID and the accompanying finding annotation information included in the received accompanying finding annotation notification information in association with each other.
  • the application 212 causes the learning device 200 to acquire the information indicating the fundus image, the information indicating the fluorescein angiography image, the NPA/NV annotation information, and the accompanying finding annotation information associated with the patient ID.
  • the application 212 causes the learning device 200 to specify the area of abnormality in blood circulation on the basis of the information indicating the fluorescein angiography image, the NPA/NV annotation information, and the accompanying finding annotation information acquired above.
  • the application 212 causes the learning device 200 to generate NPA/NV learning information in which the information indicating the fundus image and the area of abnormality in blood circulation specified on the basis of the fluorescein angiography image corresponding to the fundus image are associated with each other, and to store the generated NPA/NV learning information.
  • the application 212 causes the learning device 200 to generate, using the information indicating the fundus image included in the NPA/NV learning information as input information, and using the area of abnormality in blood circulation specified on the basis of the fluorescein angiography image corresponding to the fundus image as training information, a trained model indicating a relationship between the fundus image and the area of abnormality in blood circulation in the fundus image through learning.
  • the application 212 causes the learning device 200 to store the generated trained model, and to transmit the trained model to the diagnosis support device 300 .
  • the image DB 213 stores the patient ID, the information indicating the fundus image, and the information indicating the fluorescein angiography image in association with each other.
  • the annotation DB 214 stores the patient ID, the NPA/NV annotation information, and the accompanying finding annotation information in association with each other.
  • the learning information storage 215 stores the NPA/NV learning information in which the information indicating the fundus image and the area of abnormality in blood circulation specified on the basis of the fluorescein angiography image corresponding to the fundus image in association with each other.
  • the trained model 216 stores, using the information indicating the fundus image included in the NPA/NV learning information as input information, and using the area of abnormality in blood circulation specified on the basis of the fluorescein angiography image corresponding to the fundus image as training information, the trained model indicating the relationship between the fundus image and the area of abnormality in blood circulation in the fundus image.
  • the operation unit 220 is configured by, for example, a touch panel, which detects a touch operation on a screen displayed on the display unit 240 , and outputs a detection result of the touch operation to the information processing unit 230 .
  • the display unit 240 is configured by, for example, a touch panel, which displays a screen for receiving the information indicating the fundus image received by the learning device 200 and the information indicating the fluorescein angiography image. Further, the display unit 240 displays a screen for receiving an operation of processing the NPA/NV annotation information received by the learning device 200 and the accompanying finding annotation information.
  • the entirety or a part of the information processing unit 230 is, for example, a function unit (hereinafter, referred to as a software function unit) realized as a processor such as a central processing unit (CPU) executes the program 211 stored in the storage unit 210 and the application 212 .
  • the entirety or a part of the information processing unit 230 may be realized by hardware such as a large scale integration (LSI), an application specific integrated circuit (ASIC), or a field-programmable gate array (FPGA), or may be realized by a combination of a software function unit and hardware.
  • the information processing unit 230 includes, for example, an image acquisition unit 231 , an annotation acquisition unit 232 , a learning information generation unit 233 , and a learning unit 234 .
  • the image acquisition unit 231 acquires the fundus image notification information output by the communication unit 205 , and acquires the patient ID and the information indicating the fundus image included in the acquired fundus image notification information.
  • the image acquisition unit 231 stores the patient ID and the information indicating the fundus image acquired above in association with each other in the image DB 213 .
  • the image acquisition unit 231 acquires the fluorescein angiography image notification information output by the communication unit 205 , and acquires the patient ID and the information indicating the fluorescein angiography image included in the acquired fluorescein angiography image notification information.
  • the image acquisition unit 231 stores the patient ID and the information indicating the fluorescein angiography image acquired above in association with each other in the image DB 213 .
  • the annotation acquisition unit 232 acquires the NPA/NV annotation notification information output by the communication unit 205 , and acquires the patient ID and the NPA/NV annotation information included in the acquired NPA/NV annotation notification information.
  • the annotation acquisition unit 232 stores the patient ID and the NPA/NV annotation information acquired above in association with each other in the annotation DB 214 .
  • the annotation acquisition unit 232 acquires the accompanying finding annotation notification information output by the communication unit 205 , and acquires the patient ID and the accompanying finding annotation information included in the acquired accompanying finding annotation notification information.
  • the annotation acquisition unit 232 stores the patient ID and the accompanying finding annotation information acquired above in association with each other in the annotation DB 214 .
  • the learning information generation unit 233 acquires the patient ID, the information indicating the fundus image associated with the patient ID, and the information indicating the fluorescein angiography image stored in the image DB 213 of the storage unit 210 .
  • the learning information generation unit 233 acquires the patient ID, the NPA/NV annotation information associated with the patient ID, and the accompanying finding annotation information stored in the annotation DB 214 of the storage unit 210 .
  • the learning information generation unit 233 specifies the area of abnormality in blood circulation on the basis of the information indicating the fluorescein angiography image, the NPA/NV annotation information, and the accompanying finding annotation information acquired above.
  • FIG. 12 is a diagram illustrating an example of a fundus image
  • FIG. 13 is a diagram illustrating an example of a fluorescein angiography image.
  • the learning information generation unit 233 extracts a green component of the fundus image, and removes a noise component from a green component extraction fundus image that is a fundus image obtained by the extraction of the green component.
  • the learning information generation unit 233 divides the green component extraction fundus image from which the noise component is removed into rectangles.
  • the learning information generation unit 233 resizes the green component extraction fundus image from which the noise component is removed to a predetermined size on the basis of the image divided into rectangles. In resizing the green component extraction fundus image into the predetermined size, interpolation is performed by an interpolation method such as bicubic interpolation.
  • the learning information generation unit 233 extracts a green component of the fluorescein angiography image, and removes a noise component from a green component extraction fluorescein angiography image which is a fluorescein angiography image obtained by the extraction of the green component.
  • the learning information generation unit 233 divides the green component extraction fluorescein angiography image from which the noise component is removed into rectangles.
  • the learning information generation unit 233 resizes the green component extraction fluorescein angiography image from which the noise component is removed to a predetermined size on the basis of the image divided into rectangles. In resizing the green component extraction fluorescein angiography image into the predetermined size, interpolation is performed by an interpolation method such as bicubic interpolation.
  • the learning information generation unit 233 corrects rotational components of the green component extraction fundus image from which the noise component is removed and the green component extraction fluorescein angiography image from which the noise component is removed so that positions of eyeballs coincide with each other, and performs resizing to a predetermined size.
  • the learning information generation unit 233 corrects the rotational components so that the positions of the eyeballs coincide with each other, and specifies an area of abnormality in blood circulation on the basis of the green component extraction fundus image from which the noise component is removed and the green component extraction fluorescein angiography image from which the noise component is removed, resized to the predetermined size.
  • FIG. 14 is a diagram illustrating an example of an area of abnormality in blood circulation.
  • the learning information generation unit 233 generates NPA/NV learning information in which the information indicating the fundus image and the area of abnormality in blood circulation specified on the basis of the fluorescein angiography image corresponding to the fundus image are associated with each other, and stores the generated NPA/NV learning information in the learning information storage 215 of the storage unit 210 .
  • NPA/NV learning information in which the information indicating the fundus image and the area of abnormality in blood circulation specified on the basis of the fluorescein angiography image corresponding to the fundus image are associated with each other, and stores the generated NPA/NV learning information in the learning information storage 215 of the storage unit 210 .
  • the learning unit 234 acquires the NPA/NV learning information stored in the learning information storage 215 of the storage unit 210 .
  • the learning unit 234 acquires information indicating the fundus image and the information indicating the area of abnormality in blood circulation included in the acquired NPA/NV learning information.
  • the learning unit 234 generates, using the acquired information indicating the fundus image as input information, and using the area of abnormality in blood circulation specified on the basis of the fluorescein angiography image corresponding to the fundus image as training information, a trained model indicating a relationship between the fundus image and the area of abnormality in blood circulation in the fundus image through learning.
  • the size of the patch is set to 64 px ⁇ 64 px, and its stride (interval for moving a frame for patch extraction) is set to 2 px.
  • the generated patches are divided into two groups of a group containing a positive area and a group containing no positive area.
  • the patches are selected so that the proportions of both groups used for learning become equal.
  • over-fitting A phenomenon in which a neural network shows good performance only for learning data itself or an image very similar to the learning data and shows extremely low performance for unknown images is called over-fitting, which may be improved by collecting a larger amount of samples or applying a geometrical operation such as rotation to the learning data.
  • FIG. 15 is a diagram illustrating an example of a structure of a neural network.
  • ReLU activation function
  • arrows indicated by a 1 , a 2 , a 3 , and a 4 represent skip connection by concatenation. This is considered to contribute to restoration of image position information.
  • the learning unit 234 stores the generated trained model in the trained model 216 of the storage unit 210 .
  • the learning unit 234 creates trained model notification information including the created trained model, to be addressed to the diagnosis support device 300 , and outputs the created trained model notification information to the communication unit 205 .
  • FIG. 16 is a block diagram illustrating an example of a diagnosis support device according to a modified example of the embodiment of the present invention.
  • the diagnosis support device 300 includes a communication unit 305 , a storage unit 310 , an operation unit 320 , an information processing unit 330 , a display unit 340 , and a bus line 350 such as an address bus or a data bus for electrically connecting the respective components as shown in FIG. 16 .
  • the communication unit 305 is realized by a communication module.
  • the communication unit 305 communicates with external communication devices such as the ophthalmologist terminal 2 or the learning device 200 through the network N. Specifically, the communication unit 305 receives trained model notification information transmitted by the learning device 200 , and outputs the received trained model notification information to the information processing unit 330 .
  • the communication unit 305 receives patient information transmitted by the ophthalmologist terminal 2 , and outputs the received patient information to the information processing unit 330 .
  • the communication unit 305 acquires diagnostic information output by the information processing unit 230 , and transmits the acquired diagnostic information to the ophthalmologist terminal 2 .
  • the storage unit 310 is realized by, for example, a RAM, a ROM, an HDD, a flash memory, or a hybrid storage device in which a plurality of these memories are combined.
  • the storage unit 310 stores a program 311 executed by the information processing unit 330 , an application 312 , and a trained model 216 .
  • the program 311 is, for example, an operating system, which is located between a user or an application program and hardware, provides a standard interface for the user or the application program, and performs efficient management for each resource such as hardware.
  • the application 312 causes the diagnosis support device 300 to receive the trained model notification information transmitted by the learning device 200 , and to store the trained model included in the received trained model notification information.
  • the application 312 causes the diagnosis support device 300 to receive the patient information transmitted by the ophthalmologist terminal 2 , and to acquire a patient ID and a fundus image included in the received patient information.
  • the application 312 causes the diagnosis support device 300 to recognize an area of abnormality in blood circulation in the acquired fundus image using the stored trained model.
  • the application 312 causes the diagnosis support device 300 to create diagnostic information including the fundus image of the patient, information indicating the area of abnormality in blood circulation identified using the fundus image of the patient and the trained model, and the patient ID, to be addressed to the ophthalmologist terminal 2 , and to transmit the created diagnostic information to the ophthalmologist terminal 2 .
  • the operation unit 320 includes, for example, a touch panel, which detects a touch operation on a screen displayed on the display unit 340 and outputs a detection result of the touch operation to the information processing unit 330 .
  • the display unit 340 includes, for example, a touch panel, which displays a screen that receives information indicating the fundus image included in the patient information received by the diagnosis support device 300 .
  • the display unit 240 displays a result of the diagnosis made by the diagnosis support device 300 .
  • the entirety or a part of the information processing unit 330 is a software function unit realized as a processor such as a CPU executes the program 311 stored in the storage unit 310 and the application 312 , for example.
  • the entirety or a part of the information processing unit 330 may be realized by hardware such as LSI, ASIC, or FPGA, or may be realized by a combination of a software function unit and hardware.
  • the information processing unit 330 includes, for example, a reception unit 331 , a identification unit 332 , and a creation unit 333 .
  • the reception unit 331 acquires the trained model notification information output by the communication unit 305 , and acquires the trained model included in the acquired trained model notification information.
  • the reception unit 331 receives the acquired trained model, and stores the received trained model in the trained model 216 of the storage unit 310 .
  • the reception unit 331 acquires the patient information output by the communication unit 305 , and acquires the patient ID and the information indicating the fundus image included in the acquired patient information.
  • the reception unit 331 receives the acquired patient ID and the information indicating the fundus image, and outputs the patient ID and the information indicating the fundus image received above to the identification unit 332 .
  • the identification unit 332 acquires the patient ID and the information indicating the fundus image output by the reception unit 331 .
  • the identification unit 332 acquires the trained model stored in the trained model 216 of the storage unit 310 , and specifies the area of abnormality in blood circulation in the acquired fundus image using the acquired trained model.
  • the identification unit 332 outputs the information indicating the area of abnormality in blood circulation in the identified fundus image and the patient ID to the creation unit 333 .
  • the size of the patch is set to 64 px ⁇ 64 px, and its stride (interval for moving a frame for patch extraction) is set to 2 px. All of the generated patches are selected.
  • the identification unit 332 acquires an image of 64 ⁇ 64 ⁇ 1 on the basis of the trained model.
  • the identification unit 332 votes the acquired pixel values to corresponding pixels of an original image and performs averaging therefor.
  • the identification unit 332 may convert the acquired image into color display.
  • the creation unit 333 acquires the patient ID and the information indicating the area of abnormality in blood circulation in the fundus image output by the identification unit 332 .
  • the creation unit 333 creates the diagnostic information including the acquired patient ID and the information indicating the area of abnormality in blood circulation in the fundus image, to be addressed to the ophthalmologist terminal 2 .
  • the creation unit 333 outputs the created diagnostic information to the communication unit 305 .
  • FIG. 17 is a flowchart illustrating an example of an operation of a learning device included in the information processing system of the modified example of the present embodiment.
  • FIG. 17 illustrates an operation after the ophthalmologist terminal 2 transmits NPA/NV annotation notification information and accompanying finding annotation notification information to the learning device 200 and the examination device 3 transmits fundus image notification information and fluorescein angiography image notification information to the learning device 200 .
  • Step S 201
  • the communication unit 205 of the learning device 200 receives the fundus image notification information transmitted by the examination device 3 , and outputs the received fundus image notification information to the information processing unit 230 .
  • the image acquisition unit 231 of the information processing unit 230 acquires the fundus image notification information output by the communication unit 205 , stores the patient ID and the information indicating the fundus image included in the acquired fundus image notification information in association with each other in the image DB 213 of the storage unit 210 .
  • the communication unit 205 of the learning device 200 receives the fluorescein angiography image notification information transmitted by the examination device 3 , and outputs the received fluorescein angiography image notification information to the information processing unit 230 .
  • the image acquisition unit 231 of the information processing unit 230 acquires the fluorescein angiography image notification information output by the communication unit 205 , and stores the patient ID and the information indicating the fluorescein angiography image included in the acquired fluorescein angiography image notification information in association with each other in the image DB 213 of the storage unit 210 .
  • the communication unit 205 of the learning device 200 receives the NPA/NV annotation notification information transmitted by the ophthalmologist terminal 2 , and outputs the received NPA/NV annotation notification information to the information processing unit 230 .
  • the annotation acquisition unit 232 of the information processing unit 230 acquires the NPA/NV annotation notification information output by the communication unit 205 , and stores the patient ID and the NPA/NV annotation information included in the acquired NPA/NV annotation notification information in association with each other in the annotation DB 214 of the storage unit 210 .
  • the communication unit 205 of the learning device 200 receives the accompanying finding annotation notification information transmitted by the ophthalmologist terminal 2 , and outputs the received accompanying finding annotation notification information to the information processing unit 230 .
  • the annotation acquisition unit 232 of the information processing unit 230 acquires the accompanying finding annotation notification information output by the communication unit 205 , stores the patient ID and the accompanying finding annotation information included in the acquired accompanying finding annotation notification information in association with each other in the annotation DB 214 of the storage unit 210 .
  • the learning information generation unit 233 of the learning device 200 acquires the patient ID, the information indicating the fundus image associated with the patient ID, and the information indicating the fluorescein angiography image stored in the image DB 213 of the storage unit 210 .
  • the learning information generation unit 233 acquires the patient ID, the NPA/NV annotation information associated with the patient ID, and the accompanying finding annotation information stored in the annotation DB 214 of the storage unit 210 .
  • the learning information generation unit 233 specifies the area of abnormality in blood circulation on the basis of the information indicating the fluorescein angiography image, the NPA/NV annotation information, and the accompanying finding annotation information acquired above.
  • the learning information generation unit 233 generates NPA/NV learning information in which the information indicating the fundus image and the area of abnormality in blood circulation specified on the basis of the fluorescein angiography image corresponding to the fundus image are associated with each other, and stores the generated NPA/NV learning information in the learning information storage 215 of the storage unit 210 .
  • Step S 205
  • the learning unit 234 of the learning device 200 acquires the NPA/NV learning information stored in the learning information storage 215 of the storage unit 210 .
  • the learning unit 234 acquires information indicating the fundus image and the information indicating the area of abnormality in blood circulation included in the acquired NPA/NV learning information.
  • the learning unit 234 generates, using the acquired information indicating the fundus image as input information, and using the area of abnormality in blood circulation specified on the basis of the fluorescein angiography image corresponding to the fundus image as training information, a trained model indicating a relationship between the fundus image and the area of abnormality in blood circulation in the fundus image through learning.
  • the learning unit 234 stores the generated trained model in the trained model 216 of the storage unit 210 .
  • the learning device 200 may specify the area of abnormality in blood circulation on the basis of the information indicating the fluorescein angiography image, the NPA/NV annotation information, and the accompanying finding annotation information.
  • the learning device 200 may generate, using the information indicating the fundus image as input information, and using the area of abnormality in blood circulation specified on the basis of the fluorescein angiography image corresponding to the fundus image as training information, a trained model indicating a relationship between the fundus image and the area of abnormality in blood circulation in the fundus image through learning.
  • FIG. 18 is a flowchart illustrating an example of an operation of a diagnosis support device included in the information processing system of the modified example of the present embodiment.
  • FIG. 18 shows an operation after the learning device 200 transmits the trained model notification information to the diagnosis support device 300 and the ophthalmologist terminal 2 transmits the patient information to the diagnosis support device 300 .
  • Step S 301
  • the communication unit 305 of the diagnosis support device 300 receives the trained model notification information transmitted by the learning device 200 , and outputs the received trained model notification information to the information processing unit 330 .
  • the reception unit 331 acquires the trained model notification information output by the communication unit 305 , and acquires the trained model included in the acquired trained model notification information.
  • the reception unit 331 receives the acquired trained model, and stores the received trained model in the trained model 216 of the storage unit 310 .
  • the communication unit 305 receives patient information transmitted by the ophthalmologist terminal 2 , and outputs the received patient information to the information processing unit 330 .
  • the reception unit 331 acquires the patient information output by the communication unit 305 , and acquires the patient ID and the information indicating the fundus image included in the acquired patient information.
  • the reception unit 331 receives the acquired patient ID and the information indicating the fundus image, and outputs the patient ID and the information indicating the fundus image received above to the identification unit 332 .
  • the identification unit 332 acquires the patient ID and the information indicating the fundus image output by the reception unit 331 .
  • the identification unit 332 acquires the trained model stored in the trained model 216 of the storage unit 310 .
  • the identification unit 332 acquires the trained model stored in the trained model 216 of the storage unit 310 , and recognizes the area of abnormality in blood circulation in the fundus image that is a identification target using the acquired trained model.
  • the identification unit 332 outputs the information indicating the area of abnormality in blood circulation in the identified fundus image and the patient ID to the creation unit 333 .
  • the creation unit 333 acquires the patient ID and the information indicating the area of abnormality in blood circulation in the fundus image output by the identification unit 332 .
  • the creation unit 333 creates diagnostic information including the patient ID and the information indicating the area of abnormality in blood circulation in the acquired fundus image, to be addressed to the ophthalmologist terminal 2 .
  • the creation unit 333 outputs the created diagnostic information to the communication unit 305 .
  • the communication unit 305 acquires the diagnostic information output by the creation unit 333 , and transmits the acquired diagnostic information to the ophthalmologist terminal 2 .
  • the diagnosis support device 300 may recognize the area of abnormality in blood circulation in the fundus image that is a identification target, using the trained model indicating the relationship between the fundus image generated by using the information indicating the fundus image as input information and using the area of abnormality in blood circulation specified on the basis of the fluorescein angiography image corresponding to the fundus image as training information, and the area of abnormality in blood circulation in the fundus image.
  • the diagnosis support device 300 receives the trained model notification information transmitted by the learning device 200 , and stores the trained model included in the received trained model notification information in the storage unit 310 .
  • the diagnosis support device 300 specifies the area of abnormality in blood circulation in the fundus image using the fundus image included in the patient information transmitted by the ophthalmologist terminal 2 and the stored trained model and transmits the diagnostic result including information indicating the identified area of abnormality in blood circulation to the ophthalmologist terminal 2 has been described, but the present invention is not limited to this example.
  • a configuration in which the diagnosis support device 300 transmits the patient information to the learning device 200 may be used.
  • the learning device 200 receives the patient information transmitted by the diagnosis support device 300 , and specifies the area of abnormality in blood circulation in the fundus image using the fundus image included in the received patient information and the trained model.
  • the learning device 200 creates a diagnostic result including the information indicating the identified area of abnormality in blood circulation, and transmits the created diagnostic result to the diagnosis support device 300 .
  • the diagnosis support device 300 may receive the diagnostic result transmitted by the learning device 200 , and may transmit the received diagnostic result to the ophthalmologist terminal 2 .
  • the diagnosis support device 300 includes an identification unit that specifies an area of abnormality in blood circulation in a fundus image that is an image of the fundus, using a trained model obtained by learning a relationship between the fundus image and the area of abnormality in blood circulation in the fundus image, on the basis of the fundus image and the area of abnormality in blood circulation specified on the basis of a fluorescein angiography image of the fundus, and an output unit that outputs information indicating the fundus image of a patient and the area of abnormality in blood circulation in the fundus image of the patient identified by the identification unit using the trained model.
  • the diagnosis support device 300 can estimate an abnormal circulation area with high accuracy from a normal fundus image using a trained model in which abnormal circulation area information specified by a doctor from a fluorescein angiography image and a fundus image corresponding thereto are used as training information.
  • the area of abnormality in blood circulation is generated on the basis of the fluorescein angiography image and an ophthalmologist's diagnostic note regarding one or both of a retinal non-perfusion area and a neovascularization attached to the fluorescein angiography image.
  • an ophthalmologist's diagnostic note regarding one or both of a retinal non-perfusion area and a neovascularization attached to the fluorescein angiography image.
  • the identification unit recognizes one or both of the retinal non-perfusion area and an area corresponding to the neovascularization in the fundus image. With this configuration, it is possible to identify one or both of the retinal non-perfusion area and the area corresponding to the neovascularization with high accuracy from the normal fundus image.
  • the output unit outputs an image in which the area of abnormality in blood circulation identified by the identification unit is overlaid on the fundus image.
  • the learning device 200 includes a learning unit that generates, on the basis of a fundus image that is an image of the fundus and an area of abnormality in blood circulation specified on the basis of a fluorescein angiography image of the fundus, a trained model indicating a relationship between the fundus image and the area of abnormality in blood circulation in the fundus image, through learning.
  • the learning device 200 can generate the trained model indicating the relationship between the fundus image and the area of abnormality in blood circulation in the fundus image through machine learning.
  • the area of abnormality in blood circulation is generated on the basis of the fluorescein angiography image and an ophthalmologist's diagnostic note regarding one or both of a retinal non-perfusion area and a neovascularization attached to the fluorescein angiography image.
  • an ophthalmologist's diagnostic note regarding one or both of a retinal non-perfusion area and a neovascularization attached to the fluorescein angiography image.
  • the server 1 , the ophthalmologist terminal 2 , the examination device 3 , the learning device 200 , and the diagnosis support device 300 have a computer therein. Further, steps of the respective processes of each device described above are stored in a computer-readable recording medium in the form of a program, and the above processes are performed as the computer reads and executes the program.
  • the computer-readable recording medium includes a magnetic disc, a magneto-optical disc, a CD-ROM, a DVD-ROM, a semiconductor memory, or the like.
  • the computer program may be distributed to the computer through a communication line, and the computer that receives the distribution may execute the program.
  • the program may be a program for realizing some of the functions described above.
  • the program may be a so-called difference file (difference program) capable of realizing the above-mentioned functions in combination with a program already recorded in the computer system.
  • difference file difference program
  • a program that causes a computer that controls an information processing device to execute a control process including: a fluorescein angiography image acquisition step of acquiring fluorescein angiography image information; an NPA/NV annotation acquisition step of acquiring a diagnostic note of an ophthalmologist regarding at least one of a retinal non-perfusion area or a neovascularization, attached to the fluorescein angiography image information, as NPA/NV annotation information; an NPA/NV training information generation step of generating NPA/NV training information that is training information for calculating an existence probability of the retinal non-perfusion area and an existence probability of the neovascularization, on the basis of the fluorescein angiography image information and the NPA/NV annotation information corresponding to the fluorescein angiography image information; a fundus image acquisition step of acquiring fundus image information; an NPA/NV existence probability calculation step of calculating the existence probability of the retinal non-perfusion area and the existence probability of the neovascular
  • the program according to supplementary note 1 that causes the computer to execute the control process further including: an NPA/NV existence probability map generation step of generating an NPA/NV existence probability map in which the NPA existence probability and the NV existence probability are identifiably displayed in the fundus image information.
  • the program according to supplementary note 1 or 2 that causes the computer to execute the control process further including: an accompanying annotation acquisition step of acquiring, as accompanying finding annotation information, a diagnostic note of the ophthalmologist regarding an accompanying finding, attached to the fluorescein angiography image information and the fundus image information; an accompanying finding training information generation step of generating accompanying finding training information that serves as training information for calculating an existence probability of the accompanying finding in the fundus image information, on the basis of the fluorescein angiography image information and the fundus image information, and the accompanying finding annotation information corresponding to the fluorescein angiography image information and the fundus image information; and an accompanying finding existence probability calculation step of calculating the existence probability of the accompanying finding in the fundus image information on the basis of the accompanying finding training information, and in which in the NPA/NV existence probability calculation step, a control process of calculating the existence probability of the retinal non-perfusion area and the existence probability of the neovascularization in the fundus image information,
  • the program according to supplementary note 3 that causes the computer to execute the control process further including: an accompanying finding existence probability map generation step of generating an accompanying finding existence probability map in which the existence probability of the accompanying finding is identifiably displayed in the fundus image information.
  • An information processing method executed by an information processing device including: a fluorescein angiography image acquisition step of acquiring fluorescein angiography image information; an NPA/NV annotation acquisition step of acquiring a diagnostic note of an ophthalmologist regarding at least one of a retinal non-perfusion area or a neovascularization, attached to the fluorescein angiography image information, as NPA/NV annotation information; an NPA/NV training information generation step of generating NPA/NV training information that is training information for calculating an existence probability of the retinal non-perfusion area and an existence probability of the neovascularization, on the basis of the fluorescein angiography image information and the NPA/NV annotation information corresponding to the fluorescein angiography image information; a fundus image acquisition step of acquiring fundus image information; an NPA/NV existence probability calculation step of calculating the existence probability of the retinal non-perfusion area and the existence probability of the neovascularization in the fundus image information
  • An information processing device including: fluorescein angiography image acquisition means for acquiring fluorescein angiography image information; NPA/NV annotation acquisition means for acquiring a diagnostic note of an ophthalmologist regarding at least one of a retinal non-perfusion area or a neovascularization, attached to the fluorescein angiography image information, as NPA/NV annotation information; NPA/NV training information generation means for generating NPA/NV training information that is training information for calculating an existence probability of the retinal non-perfusion area and an existence probability of the neovascularization, on the basis of the fluorescein angiography image information and the NPA/NV annotation information corresponding to the fluorescein angiography image information; fundus image acquisition means for acquiring fundus image information; NPA/NV existence probability calculation means for calculating the existence probability of the retinal non-perfusion area and the existence probability of the neovascularization in the fundus image information, on the basis of the NPA/NV training information; and estimated

Landscapes

  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Medical Informatics (AREA)
  • Animal Behavior & Ethology (AREA)
  • Surgery (AREA)
  • Public Health (AREA)
  • Veterinary Medicine (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Ophthalmology & Optometry (AREA)
  • Radiology & Medical Imaging (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Theoretical Computer Science (AREA)
  • Hematology (AREA)
  • Vascular Medicine (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Physics & Mathematics (AREA)
  • Signal Processing (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Data Mining & Analysis (AREA)
  • Computational Linguistics (AREA)
  • Software Systems (AREA)
  • Probability & Statistics with Applications (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Physiology (AREA)
  • Dentistry (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Pathology (AREA)
  • Quality & Reliability (AREA)
  • Eye Examination Apparatus (AREA)
  • Image Analysis (AREA)
US16/930,510 2018-01-19 2020-07-16 Diagnosis support device, learning device, diagnosis support method, learning method, and program Abandoned US20210000343A1 (en)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
JP2018007585 2018-01-19
JP2018-007585 2018-01-19
PCT/JP2019/001470 WO2019142910A1 (fr) 2018-01-19 2019-01-18 Dispositif de support au diagnostic, dispositif d'apprentissage, procédé de support au diagnostic, procédé d'apprentissage et programme

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2019/001470 Continuation WO2019142910A1 (fr) 2018-01-19 2019-01-18 Dispositif de support au diagnostic, dispositif d'apprentissage, procédé de support au diagnostic, procédé d'apprentissage et programme

Publications (1)

Publication Number Publication Date
US20210000343A1 true US20210000343A1 (en) 2021-01-07

Family

ID=67302295

Family Applications (1)

Application Number Title Priority Date Filing Date
US16/930,510 Abandoned US20210000343A1 (en) 2018-01-19 2020-07-16 Diagnosis support device, learning device, diagnosis support method, learning method, and program

Country Status (5)

Country Link
US (1) US20210000343A1 (fr)
EP (1) EP3741288A4 (fr)
JP (2) JP7143862B2 (fr)
CN (1) CN111902071A (fr)
WO (1) WO2019142910A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022197157A1 (fr) * 2021-03-18 2022-09-22 가톨릭대학교 산학협력단 Dispositif et procédé basés sur un apprentissage profond permettant de détecter automatiquement une région de non-perfusion dans une image d'angiographie en fluorescence de fond d'œil
WO2024027046A1 (fr) * 2022-08-02 2024-02-08 中山大学中山眼科中心 Procédé de génération automatique d'images d'angiographie à la fluorescéine à l'aide d'images de fond d'œil en couleur

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019142910A1 (fr) * 2018-01-19 2019-07-25 学校法人慶應義塾 Dispositif de support au diagnostic, dispositif d'apprentissage, procédé de support au diagnostic, procédé d'apprentissage et programme
JP6916849B2 (ja) * 2019-09-13 2021-08-11 株式会社クレスコ 情報処理装置、情報処理方法および情報処理プログラム
JP2021176056A (ja) * 2020-05-01 2021-11-04 国立研究開発法人理化学研究所 医療システム及び医療情報処理装置
CN112967260B (zh) * 2021-03-17 2024-01-26 中国科学院苏州生物医学工程技术研究所 基于弱监督学习的眼底荧光造影图像渗漏点检测方法
WO2023032162A1 (fr) * 2021-09-03 2023-03-09 株式会社ニデック Système de traitement d'informations ophtalmiques, dispositif de photographie ophtalmique et procédé de commande
JP2023051045A (ja) * 2021-09-30 2023-04-11 ＤｅｅｐＥｙｅＶｉｓｉｏｎ株式会社 情報処理装置、情報処理方法及びプログラム

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7290880B1 (en) * 2005-07-27 2007-11-06 Visionsense Ltd. System and method for producing a stereoscopic image of an eye fundus
JP2008079682A (ja) * 2006-09-26 2008-04-10 Gifu Univ 画像解析装置及び画像解析プログラム
US9275283B2 (en) * 2010-11-09 2016-03-01 Kabushiki Kaisha Topcon Fundus image processing apparatus and fundus observation apparatus
US20160278627A1 (en) * 2015-03-25 2016-09-29 Oregon Health & Science University Optical coherence tomography angiography methods
JP6090505B1 (ja) * 2016-04-12 2017-03-08 株式会社網膜情報診断研究所 眼底画像解析システムとそのプログラム
EP3300654A2 (fr) * 2016-09-30 2018-04-04 Nidek Co., Ltd. Programme d'observation d'une image de fond d' il
EP3741288A1 (fr) * 2018-01-19 2020-11-25 Keio University Dispositif de support au diagnostic, dispositif d'apprentissage, procédé de support au diagnostic, procédé d'apprentissage et programme

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2010178802A (ja) 2009-02-03 2010-08-19 Gifu Univ 眼底画像解析システム、及び眼底画像解析プログラム
CN201624637U (zh) * 2009-08-21 2010-11-10 上海市第一人民医院 可调谐光源结合共焦激光同步眼底血管造影系统
US20140314288A1 (en) 2013-04-17 2014-10-23 Keshab K. Parhi Method and apparatus to detect lesions of diabetic retinopathy in fundus images
EP3192051B1 (fr) 2014-09-08 2021-12-08 The Cleveland Clinic Foundation Analyse automatisée d'images angiographiques
CN104835150B (zh) * 2015-04-23 2018-06-19 深圳大学 一种基于学习的眼底血管几何关键点图像处理方法及装置
JP6744625B2 (ja) 2016-07-11 2020-08-19 株式会社ペティオ ペット用サークル
CN107423571B (zh) * 2017-05-04 2018-07-06 深圳硅基仿生科技有限公司 基于眼底图像的糖尿病视网膜病变识别系统

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7290880B1 (en) * 2005-07-27 2007-11-06 Visionsense Ltd. System and method for producing a stereoscopic image of an eye fundus
JP2008079682A (ja) * 2006-09-26 2008-04-10 Gifu Univ 画像解析装置及び画像解析プログラム
US9275283B2 (en) * 2010-11-09 2016-03-01 Kabushiki Kaisha Topcon Fundus image processing apparatus and fundus observation apparatus
US20160278627A1 (en) * 2015-03-25 2016-09-29 Oregon Health & Science University Optical coherence tomography angiography methods
JP6090505B1 (ja) * 2016-04-12 2017-03-08 株式会社網膜情報診断研究所 眼底画像解析システムとそのプログラム
EP3300654A2 (fr) * 2016-09-30 2018-04-04 Nidek Co., Ltd. Programme d'observation d'une image de fond d' il
EP3741288A1 (fr) * 2018-01-19 2020-11-25 Keio University Dispositif de support au diagnostic, dispositif d'apprentissage, procédé de support au diagnostic, procédé d'apprentissage et programme

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022197157A1 (fr) * 2021-03-18 2022-09-22 가톨릭대학교 산학협력단 Dispositif et procédé basés sur un apprentissage profond permettant de détecter automatiquement une région de non-perfusion dans une image d'angiographie en fluorescence de fond d'œil
KR20220130511A (ko) * 2021-03-18 2022-09-27 가톨릭대학교 산학협력단 딥러닝 기반의 안저 형광 혈관 조영 영상 내 비관류 영역의 자동 탐지 장치 및 방법
KR102543937B1 (ko) * 2021-03-18 2023-06-14 가톨릭대학교 산학협력단 딥러닝 기반의 안저 형광 혈관 조영 영상 내 비관류 영역의 자동 탐지 장치 및 방법
WO2024027046A1 (fr) * 2022-08-02 2024-02-08 中山大学中山眼科中心 Procédé de génération automatique d'images d'angiographie à la fluorescéine à l'aide d'images de fond d'œil en couleur

Also Published As

Publication number Publication date
JPWO2019142910A1 (ja) 2021-01-07
JP2022180466A (ja) 2022-12-06
EP3741288A1 (fr) 2020-11-25
EP3741288A8 (fr) 2021-01-13
WO2019142910A1 (fr) 2019-07-25
EP3741288A4 (fr) 2021-11-24
CN111902071A (zh) 2020-11-06
JP7143862B2 (ja) 2022-09-29

Similar Documents

Publication Publication Date Title
US20210000343A1 (en) Diagnosis support device, learning device, diagnosis support method, learning method, and program
US11210789B2 (en) Diabetic retinopathy recognition system based on fundus image
US20230260126A1 (en) Processing fundus images using machine learning models
Pao et al. Detection of diabetic retinopathy using bichannel convolutional neural network
US20210327062A1 (en) Method and device for assisting heart disease diagnosis
WO2020200087A1 (fr) Détection basée sur une image de maladies ophtalmiques et systémiques
AU2018347610A1 (en) Deep learning-based diagnosis and referral of ophthalmic diseases and disorders
CN103717122A (zh) 眼科诊断支持设备和眼科诊断支持方法
KR20130042640A (ko) 화상 처리 장치
JP2021534948A (ja) 眼底画像定量分析の前置処理方法および記憶装置
US20210357696A1 (en) Processing fundus camera images using machine learning models trained using other modalities
Shi et al. One-shot retinal artery and vein segmentation via cross-modality pretraining
JP2006263127A (ja) 眼底画像診断支援システム、及び眼底画像診断支援プログラム
US11730364B2 (en) Apparatus and methods for supporting reading of fundus image
JP2010158279A (ja) 眼底画像解析システム、及び眼底画像解析プログラム
Czepita et al. Image processing pipeline for the detection of blood flow through retinal vessels with subpixel accuracy in fundus images
JP2014128448A (ja) 画像処理装置および画像処理方法
Wang et al. MEMO: dataset and methods for robust multimodal retinal image registration with large or small vessel density differences
JP6410498B2 (ja) 眼科装置及びその制御方法
WO2013041977A1 (fr) Procédé d'amélioration de l'image rétinienne et outil pour la mise en œuvre de ce procédé
CN114847871B (zh) 用于分析被检者的眼底变化趋势的方法、系统和相关产品
US20240032784A1 (en) Integrated analysis of multiple spectral information for ophthalmology applications
CN116703941A (zh) 眼底渗出病灶提取方法及装置
JP2020065695A (ja) 眼科情報処理システム、眼科情報処理方法及びプログラム
Putnam et al. Comparison of retinal image evaluation techniques in novice clinicians

Legal Events

Date Code Title Description
AS Assignment

Owner name: KEIO UNIVERSITY, JAPAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:KURIHARA, TOSHIHIDE;KATADA, YUSAKU;TSUBOTA, KAZUO;AND OTHERS;REEL/FRAME:053902/0849

Effective date: 20200903

Owner name: NIKON CORPORATION, JAPAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:KEIO UNIVERSITY;REEL/FRAME:053903/0027

Effective date: 20200923

STPP Information on status: patent application and granting procedure in general

Free format text: APPLICATION DISPATCHED FROM PREEXAM, NOT YET DOCKETED

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION