JP2021039748A - Information processor, information processing method, information processing system, and program - Google Patents

Abstract

To improve the estimation accuracy in disease risk estimation.SOLUTION: Disclose information processor includes: estimating means that estimates the risk that the subject develops the disease using a trained model that has learned the relationship between the feature amount acquired from the fundus image and the risk of developing the disease evaluated from the feature amount; and correction means that corrects the risk of developing the estimated disease based on the biological information of the subject.SELECTED DRAWING: Figure 1

Description

The disclosure of this specification relates to an information processing apparatus, an information processing method, an information processing system and a program.

Various methods such as general X-ray imaging, X-ray computed tomography, magnetic resonance imaging, ultrasonography, positron emission tomography, or single photon emission tomography are used for disease screening and diagnosis. Diagnostic imaging equipment is used to image various parts of the living body.

In particular, the eye is the only site where blood vessels can be directly observed from outside the body, and eye diseases such as diabetic retinopathy and age-related macular degeneration are diagnosed through eye examinations. In addition to eye diseases, application to early diagnosis of lifestyle-related diseases such as arteriosclerosis and diabetes and screening of various diseases such as risk determination of cerebral infarction and dementia is being studied. For example, a technique for determining the risk of a disease using an ophthalmologic examination device is known. Patent Document 1 discloses that risk information indicating a risk of a specific disease is generated by analyzing test data obtained from an ophthalmic test device.

Japanese Unexamined Patent Publication No. 2017-386

However, it has been difficult to obtain sufficient risk determination accuracy only from the information obtained from the biological images taken by the diagnostic imaging apparatus or the ophthalmologic examination apparatus.

One of the purposes of the disclosure of the present specification is to improve the determination accuracy in the risk determination of a disease in view of the above problems.

It should be noted that the disclosure of the present specification is not limited to the above-mentioned purpose, and is an action and effect derived by each configuration shown in the embodiment for carrying out the invention described later, and exerts an action and effect that cannot be obtained by the conventional technique. It can be positioned as one of the other purposes.

The information processing device disclosed in this specification is
An estimation means for estimating the risk of developing a disease by a subject using a learned model that learned the relationship between the feature amount acquired from the fundus image and the risk of developing the disease evaluated from the feature amount.
It is provided with a correction means for correcting the risk of developing the estimated disease based on the biological information of the subject.

According to the disclosure of the present specification, the determination accuracy in the risk determination of a disease can be improved.

The figure which shows an example of the whole structure of the inspection system which concerns on 1st Embodiment A block diagram showing an example of the configuration of the functions of the information processing apparatus according to the first embodiment. A flow chart showing an example of a processing procedure of the information processing apparatus according to the first embodiment. The figure which shows the display example of the risk estimation result of the disease which concerns on 1st Embodiment The figure which showed an example of the structure of the medical treatment appointment system which concerns on the modification of 1st Embodiment The figure which showed the flow of the risk estimation which concerns on the 2nd Embodiment The figure which shows an example of the whole structure of the inspection system which concerns on 1st Embodiment The figure which shows an example of the whole structure of the inspection system which concerns on 1st Embodiment The figure which shows an example of the structure of the neural network used as the machine learning model which concerns on modification 4. The figure which shows an example of the structure of the neural network used as the machine learning model which concerns on modification 8. The figure which shows an example of the structure of the neural network used as the machine learning model which concerns on modification 8.

The information processing device according to the present embodiment estimates the risk of a disease based on a biological image (medical image of a subject) taken by an diagnostic imaging device or an ophthalmic examination device and biological information obtained by other tests or the like. It is characterized by that.

Hereinafter, preferred embodiments of the information processing apparatus disclosed in the present specification will be described in detail with reference to the accompanying drawings. However, the components described in this embodiment are merely examples, and the technical scope of the information processing apparatus disclosed in the present specification is determined by the scope of claims, and the following individual embodiments are made. It is not limited by the form. Further, the disclosure of the present specification is not limited to the following embodiments, and various modifications (including organic combinations of the respective embodiments) are possible based on the purpose of the disclosure of the present specification. It is not excluded from the scope of disclosure herein. That is, all the configurations in which each embodiment described later and a modification thereof are combined are also included in the embodiment disclosed in the present specification.

In the following embodiment, a case where a fundus camera is used as an imaging device for capturing a biological image used for estimating the risk of a disease will be described as a typical example, but the present invention is not limited to this, and other diagnostic imaging is performed. Even an apparatus or an ophthalmologic examination apparatus can be suitably applied. For example, another ophthalmologic examination device such as an optical coherence tomography device (OCT device) capable of taking an image of the fundus or anterior segment of the eye may be used depending on the disease for which risk estimation is performed. In addition, diagnostic imaging used for disease screening and diagnosis such as general X-ray imaging, X-ray computed tomography, magnetic resonance imaging, ultrasonography, positron emission tomography, and single photon emission tomography. An apparatus may be used.

[First Embodiment]
FIG. 1 is a diagram showing an overall configuration of an information processing system 100 including an information processing device according to the present embodiment.

The information processing system 100 includes a fundus camera 101, a biological information inspection device 102, a cloud server 103, and an information processing device 104.

The fundus camera 101 captures a fundus image, which is a biological image used for estimating the risk of a disease.

For example, near-infrared light may be used for photographing with the fundus camera 101. In general, there are factors that deteriorate the quality of a still image, such as the position of the subject's eyes, body movement, blinking, and foreign matter mixed in the image during shooting. Therefore, if the image at the time of shooting is not suitable for calculating the image feature amount, it is necessary to take it again. However, in the flash shooting with visible light used in the conventional fundus camera, miosis occurs and it takes some time to re-shoot. You will need it. Therefore, by using near-infrared light, miosis can be avoided, and continuous and repeated imaging becomes possible. Further, by using near-infrared light, it becomes easy to take a moving image without miosis.

Alternatively, weak visible light may be used for photographing with the fundus camera 101. As described above, in general, when shooting, there are factors that deteriorate the quality of the still image, such as the position of the subject's eyes, body movement, blinking, and foreign matter mixed in the image, and the image at the time of shooting is an image feature. If it is not suitable for calculating the amount, it will need to be redone. However, in the flash photography with visible light used in the conventional fundus camera, miosis occurs, and it takes some time to re-photograph. Therefore, by using weak visible light, miosis can be avoided, and continuous and repeated shooting becomes possible. In addition, by using weak visible light, it is possible to take a moving image without miosis.

Alternatively, a light source having the same light source as the observation light used to search for the position of the anterior segment of the eye or a light source having the same illuminance may be used for photographing with the fundus camera 101.

It should be noted that the above is merely an example, and the photographing method is not limited to the above as long as the fundus image can be acquired.

In the present embodiment, the fundus image taken by the fundus camera 101 is once transmitted to the biometric information inspection device 102. As a transmission method, a wired communication method such as USB or a wireless communication means such as Wi-Fi (Wireless Fidelity) (registered trademark) or Bluetooth (registered trademark) is used. The fundus image may be transmitted to the cloud server 103 without going through the biometric information inspection device 102, or may be transmitted to the information processing device 104.

The biological information inspection device 102 acquires biological information used for estimating the risk of a disease. The biometric information test device 102 can measure biometric information such as height, weight, body fat percentage, systolic blood pressure, diastolic blood pressure, irregular pulse wave, heart rate, and body temperature of the subject, for example. It should be noted that the biological information does not necessarily have to be acquired from examinations and measurements, and for example, the presence or absence of smoking habits and medical history may be acquired by input from the user. Further, for example, it may have a blood test function for measuring blood glucose level, red blood cell count, hemoglobin, uric acid, etc. from the blood of the subject, a urine test function for testing the urine of the subject, and the like. In the case of a test using a biological sample such as a blood test or a urine test, as shown in FIG. 8, the biological information test kit 108 is provided to the subject by mail or the like. Then, the result of the inspection by the subject may be stored in the cloud server 103 via the personal computer 107 or the mobile information terminal 106. Further, for example, data such as blood pressure and body weight measured by a sphygmomanometer 109 or a weight scale 110 without using a sample may be stored in the cloud server 103 by the subject by the same method. The type of biopsy method and the flow of data transmission / reception are not limited to this as an example, and it is sufficient that necessary biometric information can be transmitted to the information processing apparatus 104.

The cloud server 103 stores and manages data such as images taken by the fundus camera 101 and biometric information acquired by the biometric information inspection device 102.

As shown in FIG. 2, the information processing apparatus 104 has a communication IF (Interface) 111, a ROM (Read Only Memory) 112, a RAM (Random Access Memory) 113, a storage unit 114, and an operation unit 115 as functional configurations thereof. , Display unit 116, and control unit 117.

The communication IF 111 is realized by a LAN card or the like, and controls communication between an external device (for example, a cloud server 103) and an information processing device 104. The ROM 112 is realized by a non-volatile memory or the like and stores various programs or the like. The RAM 113 is realized by a volatile memory or the like, and temporarily stores various information. The storage unit 114 is an example of a computer-reading storage medium, and is realized by a large-capacity information storage device typified by a hard disk drive (HDD) or a solid state drive (SSD), and stores various information. The operation unit 115 is realized by a keyboard, a mouse, or the like, and inputs an instruction from the user to the device. The display unit 116 is realized by a display or the like, and displays various information to the user. The control unit 117 is realized by a CPU (Central Processing Unit), a GPU (Graphical Processing Unit), or the like, and controls each process in the information processing device 104 in an integrated manner.

The control unit 117 includes an acquisition unit 118, an estimation unit 119, a correction unit 120, and a display control unit 121 as its functional configuration.

The acquisition unit 118 reads and acquires data such as the fundus image of the subject taken by the fundus camera 101 and the biometric information of the subject acquired by the biometric information inspection device 102 from the cloud server 103. The data does not necessarily have to be acquired from the cloud server 103, and for example, the data transmitted directly from the fundus camera 101 or the biometric information inspection device 102 may be acquired.

The estimation unit 119 estimates the risk of the subject's disease from the fundus image of the subject acquired by the acquisition unit 118. In the present embodiment, the risk of a disease indicates the probability of developing a disease within a certain period of time. The probability of developing a disease may be a quantitative expression such as a percentage display, or a qualitative expression such as a high risk or a low risk.

The correction unit 120 corrects the estimation result of the disease risk estimated by the estimation unit 119 and calculates the final disease risk. More specifically, the disease risk estimated by the trained model is estimated by using the weighting coefficient of the probability of developing each disease preset for each of the plurality of biometric information obtained by the biometric information test device 102. Correct the result. That is, the correction unit 120 corresponds to an example of correction means for correcting the risk of developing a disease by using a predetermined weighting coefficient determined for each biological information.

The display control unit 121 causes the display unit 116 to display the final disease risk estimation result corrected by the correction unit 120.

Next, a processing procedure for estimating the risk of the disease of the information processing apparatus 104 according to the present embodiment will be described with reference to the flowchart of FIG.

(S3000) (Reading of fundus image)
In S3000, the acquisition unit 118 reads and acquires the fundus image captured by the fundus camera 101 stored in the cloud server 103. Alternatively, the fundus image transmitted directly from the fundus camera 101 is acquired.

(S3010) (Calculation of image feature amount)
In S3010, the estimation unit 119 detects a feature amount for estimating the risk of disease from the fundus image acquired from the cloud server 103.

Alternatively, a moving image taken by the fundus camera 101 may be acquired from the cloud server 103, and a desired image feature amount may be detected from the moving image. Generally, when taking a picture, there are factors that deteriorate the quality of a still image, such as the position of the subject's eyes, body movement, blinking, and foreign matter mixed in the image. Therefore, there is no problem in image quality in the moving image, and the risk of the disease can be appropriately estimated by calculating the image feature amount from the portion suitable for calculating the image feature amount. For the selection of the optimum part in the moving image, the brightness, contrast, sharpness of the image, matching with a pre-registered pattern, and the like can be appropriately used.

Examples of the detected feature amount include morphological features and color information of the defective portion such as the diameter, curvature, angle or branch of the blood vessel. Alternatively, for example, it may be abnormal shape, image contrast, or color information other than the blood vessel portion due to bleeding or vitiligo. The feature amount used for risk estimation is not limited to this, and various feature amounts can be used.

(S3020) (Estimating the risk of disease)
In S3020, the estimation unit 119 estimates the risk of the disease by inputting the feature amount detected in S3010 into the trained model. That is, the estimation unit 119 uses a learned model that learns the relationship between the feature amount acquired from the fundus image and the risk of developing the disease evaluated from the feature amount, and the risk that the subject develops the disease. Corresponds to an example of an estimation means for estimating. More specifically, it corresponds to an example of an estimation means for estimating the probability that the subject develops a disease by inputting the feature amount obtained from the fundus image of the subject into the trained model.

Here, the trained model is a machine learning model that follows a machine learning algorithm such as a support vector machine, and indicates a machine learning model that has been trained using appropriate learning data in advance. It should be noted that the trained model does not require further learning, and additional learning can be performed.

The learning data is composed of one or more pairs of input data and output data (correct answer data). The trained model according to the present embodiment uses output data (data related to disease risk) as training data for input data (data related to a plurality of feature quantities detected from a biological image such as a fundus image) according to an arbitrary learning algorithm. I'm learning. Specifically, for example, the correlation between the abnormal shape other than the blood vessel portion due to bleeding or vitiligo detected from the fundus image and the probability of developing diabetic retinopathy is learned. Alternatively, for example, a feature amount relating to the morphology of a blood vessel showing arterial diameter, vein diameter, ratio of arterial diameter to vein diameter, branch angle of blood vessel, asymmetry of the branch, arterial vein stenosis or twist of blood vessel, and the feature. Learn the correlation with the risk of developing cardiovascular disease and stroke, which is evaluated from the amount. As described above, the risk of developing a disease may be output as a percentage, or may be identified and output in a plurality of qualitative classes such as high risk and low risk. Further, the correlation between the input data and the output data to be learned is not limited to the above combination, and various correlations related to disease risk estimation can be learned.

In addition, the trained model can iteratively perform training based on a data set including input data and output data.

In the present embodiment, the learned model for estimating the disease risk may be generated by the information processing device 104, or may be a model generated by an information processing device different from the information processing device 104. ..

When the information processing device 104 also generates a trained model for estimating the disease risk, the information processing device 104 further includes a generation unit (not shown).

As described above, the generation unit learns the output data for the input data as teacher data according to an arbitrary learning algorithm, and generates a trained model. Specific algorithms for machine learning include the nearest neighbor method, the naive Bayes method, a decision tree, and a support vector machine. In addition, deep learning (deep learning) in which features and coupling weighting coefficients for learning are generated by themselves using a neural network can also be mentioned. When deep learning using a neural network is performed, a trained model is obtained by learning a pair of a fundus image and a risk of developing a disease evaluated from the fundus image. For example, a blood vessel called a retinal arteriole has a higher risk of developing hypertension in a person with a small size than a person with a large size. Learning is performed by grouping high risks as learning data.

That is, the estimation unit 119 deeply learns the relationship between the fundus image and the risk of developing a disease evaluated from the fundus image, and the fundus image of the subject acquired by the acquisition means. Corresponds to an example of an estimation means for estimating the risk of developing a disease by the subject by inputting.

The relationship between the input data and the output data to be learned is not limited to the above combination, and various correlations related to disease risk estimation can be learned. As appropriate, any of the above algorithms that can be used can be applied to this embodiment.

It should be noted that a plurality of trained models may be generated according to the disease for which the risk is to be estimated, or one trained model may be generated so that the risk of a plurality of diseases can be estimated.

That is, the estimation unit 119 evaluates the relationship between the feature amount acquired from the fundus image and the risk of developing the first disease evaluated from the feature amount, and the feature amount acquired from the fundus image and the feature amount. An example of an estimation means for estimating the risk of the subject developing the first disease and the second disease by using a learned model that has learned the relationship with the risk of developing the second disease. Corresponds to.

Alternatively, the estimation unit 119 is acquired from the first trained model that has learned the relationship between the feature amount acquired from the fundus image and the risk of developing the first disease evaluated from the feature amount, and the fundus image. Using a second learned model that learned the relationship between the feature amount and the risk of developing the second disease evaluated from the feature amount, the subject can use the first disease and the second disease. It corresponds to an example of an estimation means for estimating the risk of developing a disease.

(S3030) (Saving of estimation result)
In S3030, the information processing apparatus 104 stores the estimated risk of the disease in the storage unit 114. Alternatively, it is transferred to the cloud server 103 via the communication IF 111 and stored in the cloud server 103. It may be stored in both the storage unit 114 and the cloud server 104.

(S3040) (Reading biological information)
In S3040, the information processing device 104 reads the biometric information obtained from the biometric information inspection device 102 stored in the cloud server 103. Alternatively, the biometric information transmitted directly from the biometric information inspection device 102 is acquired.

(S3050) (Reading of estimation result)
In S3050, the acquisition unit 118 reads the disease risk estimation result stored in the storage unit 114 or the cloud server 103.

(S3060) (Correction of estimation result)
In S3060, the correction unit 120 corrects the estimation result of the disease risk estimated by the estimation unit 119 and calculates the final disease risk. More specifically, using the biometric information test device 102, the probability of developing each disease using a weighting coefficient of the probability of developing each disease preset for each of a plurality of measured and inspected biometric information. To correct. For example, since the probability of developing a disease differs depending on the body mass index (BMI) obtained from the blood pressure, height, and weight of the subject, the probability calculated based on the feature amount obtained from the fundus image is used as the living body. Make corrections based on information. As the correction amount, a value calculated from the feature amount acquired from the fundus image of the diseased person and the healthy person as the subject and the evaluation result of the biological information is used.

Then, the corrected probability is calculated as the final risk estimation result. A plurality of threshold values may be set for the probability of onset, the threshold values may be used to classify the categories into a plurality of risk stage categories, and the classified categories may be used as the final risk estimation result. For example, the probability of developing a disease is divided into three stages of "0 to 33%, 34% to 66%, 67% to 100%", and each is classified into the categories of "low risk, medium risk, and high risk". .. The classification method is not limited to the above, and may be divided into, for example, two stages or four or more stages. Further, the threshold value is also an example and is not limited to this. That is, the output method of the output estimation result is not limited to the above, and it may be output so that the subject can recognize the degree of risk of developing the disease. Further, in the above, the corrected estimation results are classified into categories, but when the trained model is trained, the output data is classified into a plurality of classes such as "low risk, medium risk, and high risk" in advance and trained in S3020. The estimation result may be output in a classified form at the stage where the disease risk is estimated.

(S3070) (Saving of corrected estimation result)
In S3070, the information processing apparatus 104 stores the corrected risk of the disease in the storage unit 114. Alternatively, it is transferred to the cloud server 103 via the communication IF 111 and stored in the cloud server 103. It may be stored in both the storage unit 114 and the cloud server 104.

(S3080) (Result display / print output)
In S3080, the display control unit 121 causes the display unit 116 to display the estimated final disease risk. Alternatively, it is sent to a separate printer and output.

FIG. 4 shows a display example of the risk estimation result of the disease. FIG. 4A shows how the disease risk estimation results calculated as the disease risk stages are classified and displayed in three stages of high risk, medium risk, and low risk. That is, the display control means 121 classifies the risk of developing the corrected disease into a plurality of classes and displays it on the display unit. In addition, it displays a description of what each category is doing. For example, if the estimated category is "low risk", an explanation such as "low risk of developing the disease. Please continue a healthy lifestyle" is displayed. The above description is an example and is not limited to the above. Furthermore, it is not always necessary to include an explanation, and a tab or the like for supplementary explanation may be provided only when the subject wants to know more about the disease, such as when the risk is presumed to be high. Alternatively, it may be provided with a tab or the like in which only a brief explanation is written together and a supplementary explanation is given when a person wants to know in more detail.

Further, the risk estimation result of the disease may be displayed as a graph, the risk estimation result of a plurality of diseases may be shown using a radar chart as shown in FIG. 4 (b), or the risk estimation result of a plurality of diseases may be shown as shown in FIG. 4 (c). In addition, a bar graph may be used to show the risk estimation results of a plurality of diseases. The above is an example, and the form is not limited as long as the risk estimation result of the disease is visually represented in two dimensions.

Further, FIGS. 4 (d) to 4 (f) show a state in which the fundus image taken by the fundus camera 101 is displayed, and for example, by displaying in parallel with the results of FIGS. 4 (a) to 4 (c). , The estimation result of the disease risk and the fundus image may be viewed at the same time. That is, the display control unit 121 may display the corrected risk estimation result of developing the disease in parallel with the fundus image of the subject.

Specifically, FIG. 4D shows a state in which images of the left and right eyes are displayed in parallel. Further, FIG. 4E shows a state in which two images (current image and past image) captured at different times are displayed in parallel. Further, FIG. 4 (f) shows an example of a main image portion related to risk estimation and an explanation. That is, the part highly correlated with the disease in the fundus image is emphasized and displayed.

The display method is not limited to these, and various display methods of risk estimation results and other display methods using not only captured images but also biological information, their past information, progress, etc. can be performed. it can. Further, the estimation result may be displayed not only on the display unit 116 of the information processing device 104 but also on the display unit included in the biometric information inspection device 102. Alternatively, as shown in FIG. 7, application software dedicated to the inspection system is installed on the mobile information terminal 106. As a result, the received risk estimation result of the disease and information related to other inspection systems may be displayed on the mobile information terminal 106 via the application software. That is, the device that displays the estimation result does not necessarily have to be the information processing device 104.

Further, not only the estimation result of the disease risk may be displayed on the display unit, but also it may be transmitted to a separate printer and output.

As described above, the processing of the information processing apparatus 104 is carried out.

According to the above, when estimating the risk of a disease of a subject, the probability that the estimation result estimated by the feature amount acquired from the fundus image will develop in each disease preset for each of a plurality of biometric information. The accuracy of the estimation result can be improved by correcting using the weighting coefficient of. In addition, since no specialization is required for disease risk estimation, disease risk estimation can be easily performed regardless of the user. Furthermore, by classifying the risk of the disease into a plurality of stages and displaying the estimation result in a display form that is easy for the subject to understand, the risk of the disease can be intuitively recognized. In addition, by displaying a biological image such as a fundus image to be examined in parallel with the estimation result, it becomes easier to recognize the location of the disease.

(Modification example 1)
In the first embodiment, the risks of diseases were estimated, and they were displayed or printed out. In this modified example, the estimation result is transmitted to a medical institution or the like so that the subject can receive consultation such as additional examination, consultation with an appropriate medical institution or clinical department, and lifestyle-related guidance.

FIG. 5 shows the configuration of a system that cooperates with the medical institution 130 and gives advice to the subject.

In the examination system of the present embodiment, recommended medical institutions 130 and clinical departments are output together with the risk estimation results from a plurality of pre-registered medical institutions 130 according to the type of disease and the risk estimation result. The subject can confirm the above output result, for example, on the display unit of the biological information inspection device 102, a print, or the like, or on the portable information terminal 106 or the personal computer 107.

The cloud server 103 stores medical institutions 102 and relevant clinical departments according to the type of disease and the degree of risk of the disease. Then, by reading out these information from the cloud server 103 according to the risk estimation result, the medical institution 130 and the clinical department can be sent to the biometric information inspection device 102, the personal digital assistant 106, and the personal computer 107 and presented to the subject. ..

Further, the medical institution 130 can be reserved on the biometric information inspection device 102, the mobile information terminal 106, or the personal computer 107. Application software for reservation for making a reservation for a medical institution is installed in the biometric information inspection device 102.

When making a reservation on the mobile information terminal 106 or the personal computer 107, the application software for reservation is downloaded and installed.

The application software for reservation communicates with the medical treatment reservation system 132 of the medical institution 130 to inquire about the ID of the subject and the desired reservation date and time. Then, if it is possible to make a reservation by collating with the latest reservation status stored in the medical treatment reservation system, the medical treatment reservation system 132 registers the reservation by ID. In addition, when the reservation is completed, the photographed image, the biological information, and the risk estimation result can be transferred to the medical institution 130 at the time of consultation. Select the transfer of captured images, biometric information, and risk estimation results on the application software for reservation. By this operation, information such as captured images, biological information, and risk estimation results is read from the cloud server 103, transferred securely, and stored in the examinee information storage unit 133 in the medical treatment reservation system 132 of the medical institution 130.

Further, in the examination system of the present embodiment, a doctor's diagnosis and consultation can be received on the system according to the disease risk obtained by the examination system. For example, when the risk of fundus disease is high as a result of risk estimation using the image of the fundus camera 101, the fundus image is transferred to a doctor. Then, based on this, the doctor and the doctor face-to-face via the video communication function (system capable of video communication) incorporated in the biometric information inspection device 102, the mobile information terminal 106 possessed by the subject, and the personal computer 107. You can receive face diagnosis and consultation.

Similarly, if the risk of other diseases is high, you can receive consultations such as additional tests, consultations with appropriate medical institutions and clinical departments, and lifestyle-related guidance.

Further, in the test system of the present embodiment, based on the risk estimation result and the result obtained by the biological information system 102, it is possible to directly propose the improvement of lifestyle habits, the intake of supplements and general medicines by the test system.

At this time, the cloud server 103 stores lifestyle-related improvement contents, recommended supplements, and general medicines according to the type of disease and the degree of risk of the disease.

Then, by reading out these information from the cloud server 103 according to the risk estimation result, the above-mentioned proposal can be sent to the biometric information inspection device 102, the mobile information terminal 106, or the personal computer 107 and presented to the subject.

These services are not limited to the above, and various services can be provided according to the above risk estimation results.

[Second Embodiment]
In the first embodiment, the risk of the disease estimated by inputting a plurality of features obtained from the biometric image into the trained model is a weighting coefficient preset for each biometric information obtained by the biopsy. The accuracy of disease risk estimation was improved by correcting with.

On the other hand, in the present embodiment, the image feature amount obtained from the image taken by the fundus camera 101 and the probability of developing the disease, the biological information obtained by the biometric information inspection device 102, and the probability of developing the disease are learned. Estimate the risk of disease using a trained model that has been trained.

The overall configuration of the information processing system in this embodiment is the same as that in the first embodiment. Further, the fundus camera 101, the biological information inspection device 102, and the cloud server 103 used in the present embodiment are the same as those in the first embodiment.

Hereinafter, the processing process of the present embodiment will be described with reference to FIG. Since S6000 to S6040 and S6070 are the same as those in the first embodiment, description thereof will be omitted.

(S6050: Estimating the risk of disease (judgment))
In S6050, first, the acquisition unit 118 acquires the feature amount obtained from the fundus image of the subject stored in the cloud server 103 and the biometric information measured / inspected by using the biometric information inspection device 102. Then, the estimation unit 119 estimates the risk of the subject developing the disease by inputting the feature amount and the biological information acquired by the acquisition unit 118 into the pre-generated learned model.

Specifically, for example, feature quantities obtained from fundus images such as arterial diameter, vein diameter, ratio of arterial diameter to vein diameter, branch angle of blood vessel, asymmetry of the branch, arterial vein stenosis or twist of blood vessel. Using biological information such as blood pressure, BMI index, age, gender, medical history, or presence or absence of smoking habits as input data, the probability of developing a disease such as cardiovascular disease or cerebrovascular disease evaluated from the characteristic amount and biological information. Learn the correlation.

The correlation between the input data and the output data to be learned is not limited to the above combination, and various correlations related to disease risk estimation can be learned.

Further, in the present embodiment, the learned model for estimating the disease risk may be generated by the information processing device 104, or may be a model generated by an information processing device different from the information processing device 104. ..

(S6060: Saving of estimation results)
In S6060, the information processing apparatus 104 stores the estimation result of the risk of the disease in the storage unit 114. Alternatively, it is transferred to the cloud server 103 via the communication IF 111 and stored in the cloud server 103. It may be stored in both the storage unit 114 and the cloud server 104.

As described above, the processing of the information processing apparatus 104 is carried out.

According to the above, when estimating the risk of a disease of a subject, the correlation between the feature amount acquired from the fundus image and the probability of developing the disease evaluated from the biological information acquired by the biopsy was learned. The accuracy of disease risk estimation can be improved by using a trained model.

(Modification 2)
In the various embodiments and modifications described above, learning is performed to adjust (tune) the trained model (learned model for estimation) used for the estimation processing regarding the disease of the subject for each subject, and the subject is subjected to learning. A dedicated trained model may be generated. For example, using the tomographic images acquired in the past examination of the subject, transfer learning of a general-purpose trained model for estimating the disease of the subject is performed, and the trained model dedicated to the subject is obtained. Can be generated. By associating the trained model dedicated to the subject with the ID of the subject and storing it in an external device such as a storage unit 114 or a server, the control unit 117 can perform the current inspection of the subject. In addition, a trained model dedicated to the subject can be specified and used based on the ID of the subject. By using a trained model dedicated to the subject, it is possible to improve the estimation accuracy of the disease for each subject.

(Modification example 3)
In the various embodiments and modifications described above, the control unit 117 may perform various image processing using images or the like acquired by photographing. For example, the control unit 117 may generate a high-quality image with improved image quality by using a learned model for high image quality (high image quality model) for the image acquired by shooting. Here, the improvement of image quality includes reduction of noise, conversion to colors and gradations that make it easy to observe the object to be photographed, improvement of resolution and spatial resolution, and enlargement of image size while suppressing a decrease in resolution.

As a machine learning model for improving image quality, for example, CNN or the like can be used. Further, as the training data of the high image quality model, various images such as an anterior eye image and an SLO image are used as input data, and high quality images corresponding to the input images, for example, which have undergone high image quality processing, are output data. And. Here, the high image quality processing includes alignment of images taken at the same spatial position a plurality of times, and addition and averaging of the aligned images. The image quality improvement process is not limited to the addition averaging process, and may be, for example, a process using a smoothing filter, a maximum a posteriori probability estimation process (MAP estimation process), a gradation conversion process, or the like. The high-quality image may be, for example, an image that has undergone filter processing such as noise removal and edge enhancement, or an image whose contrast has been adjusted so as to change from a low-brightness image to a high-brightness image. May be used. Further, since the output data of the learning data related to the high image quality model may be a high quality image, it is photographed by using an OCT device having higher performance than the OCT device when the tomographic image which is the input data is taken. It may be an image taken or an image taken with a high load setting.

However, if machine learning is performed using an image that has not been properly image-enhanced as output data of training data, the image obtained by using the trained model trained using the training data is also appropriately high. There is a possibility that the image will not be image-enhanced. Therefore, by removing the pair containing such an image from the teacher data, it is possible to reduce the possibility that an inappropriate image is generated by using the trained model.

The control unit 117 can acquire an image with high image quality with high accuracy at a higher speed by performing high image quality processing using such a high image quality model.

The high image quality model may be prepared for each type of various images as input data. For example, a high image quality model for an anterior eye image, a high image quality model for an SLO image, a high image quality model for a tomographic image, a high image quality model for an OCTA front image, and the like may be prepared. Further, for the OCTA front image and the En-Face image, a high image quality model may be prepared for each depth range for generating an image. For example, a high image quality model for the surface layer, a high image quality model for the deep layer, and the like may be prepared. Further, the high image quality model may be one in which learning is performed on an image for each imaging site (for example, the center of the macula and the center of the optic nerve head), or may be one in which learning is performed regardless of the imaging site. ..

At this time, for example, using a high image quality model obtained by learning the fundus OCTA front image as training data, the fundus OCTA front image is improved in image quality, and further, the anterior eye OCTA front image is learned as training data. The image quality of the frontal image of the front eye OCTA may be improved by using the high image quality model. Further, the high image quality model may be one that has been learned regardless of the imaging region. Here, for example, the fundus OCTA frontal image and the anterior eye OCTA frontal image may have relatively similar distributions of blood vessels to be imaged. As described above, in a plurality of types of medical images in which the appearances of the objects to be imaged are relatively similar to each other, the feature amounts may be relatively similar to each other. Therefore, for example, by using a high image quality model obtained by learning the fundus OCTA front image as learning data, not only the fundus OCTA front image can be improved, but also the front eye OCTA front image can be improved. May be done. Further, for example, by using a high image quality model obtained by learning the front eye OCTA front image as learning data, it is possible not only to improve the image quality of the front eye OCTA front image but also to improve the image quality of the fundus OCTA front image. It may be configured. That is, using a high-quality model obtained by learning at least one type of frontal image of the fundus OCTA frontal image and the anterior eye OCTA frontal image as training data, the fundus OCTA frontal image and the anterior eye OCTA frontal image are combined. At least one type of front image may be configured to have high image quality.

Here, consider a case where the anterior eye can also be imaged in the OCT device capable of photographing the fundus. At this time, for example, the fundus OCTA front image may be applied to the En-Face image of OCTA in the fundus photography mode, and the anterior eye OCTA front image may be applied in the anterior segment imaging mode. At this time, when the high image quality button is pressed, for example, in the fundus photography mode, in the display area of the En-Face image of OCTA, among the low image quality fundus OCTA front image and the high image quality fundus OCTA front image. One display may be configured to change to the other display. When the high image quality button is pressed, for example, in the anterior segment imaging mode, a low image quality front eye OCTA front image and a high image quality front eye OCTA front image are displayed in the OCTA En-Face image display area. The display of one of the above may be changed to the display of the other.

In an OCT device capable of fundus photography, an anterior eye adapter may be attached when the anterior eye can also be imaged. Further, the optical system of the OCT device may be configured to be movable at a distance of about the axial length of the eye to be inspected without using the anterior eye adapter. At this time, the focus position of the OCT device may be configured to be largely changeable to the emmetropic side to the extent that an image is formed on the front eye.

Further, for example, a fundus OCT tomographic image may be applied to the tomographic image in the fundus photography mode, and an anterior ocular OCT tomographic image may be applied to the tomographic image in the anterior segment imaging mode. Further, the above-mentioned high image quality processing of the fundus OCTA front image and the anterior ocular OCTA front image can be applied as, for example, high image quality processing of the fundus OCTA tomographic image and the anterior ocular OCT tomographic image. At this time, when the high image quality button is pressed, for example, in the fundus photography mode, one of the low image quality fundus OCT tomographic image and the high image quality fundus OCT tomographic image is displayed in the tomographic image display area. It may be configured to change to the other display. When the high image quality button is pressed, for example, in the anterior segment imaging mode, one of a low image quality anterior ocular OCT tomographic image and a high image quality anterior ocular OCT tomographic image is displayed in the tomographic image display area. The display of is changed to the other display.

Further, for example, the fundus OCTA tomographic image may be applied to the tomographic image in the fundus photography mode, and the anterior ocular OCTA tomographic image may be applied to the tomographic image in the anterior segment imaging mode. Further, the above-mentioned high image quality processing of the fundus OCTA front image and the anterior ocular OCTA front image can be applied as, for example, high image quality processing of the fundus OCTA tomographic image and the anterior ocular OCTA tomographic image. At this time, for example, in the fundus photography mode, in the tomographic image display area, the information indicating the blood vessel region (for example, motion contrast data equal to or higher than the threshold value) in the fundus OCTA tomographic image is superimposed on the fundus OCT tomographic image at the corresponding position. It may be configured to be displayed. Further, for example, in the anterior segment imaging mode, in the tomographic image display region, information indicating a blood vessel region in the anterior ocular OCTA tomographic image may be superimposed and displayed on the anterior ocular OCT tomographic image at a corresponding position. ..

In this way, for example, when it is considered that the feature amounts (states of the objects to be photographed) of a plurality of types of medical images are relatively similar to each other, at least one type of the plurality of types of medical images may be used. Using a high image quality model obtained by learning a medical image as training data, at least one type of medical image of a plurality of types of medical images may be configured to have high image quality. Thereby, for example, a common trained model (common high image quality improvement model) can be used to enable execution of high image quality improvement of a plurality of types of medical images.

The display screen of the fundus photography mode and the display screen of the anterior segment photography mode may have the same display layout or may have a display layout corresponding to each photography mode. Various conditions such as imaging conditions and analysis conditions may be the same or different between the fundus imaging mode and the anterior segment imaging mode.

Here, the target image of the high image quality processing may be, for example, a plurality of OCTA front images (corresponding to a plurality of depth ranges) (OCTA En-Face image, motion contrast En-Face image). Further, the target image for the high image quality processing may be, for example, one OCTA front image corresponding to one depth range. Further, the target image of the high image quality processing is, for example, a front image of brightness (En-Face image of brightness), an OCT tom image which is a B scan image, or a tom image of motion contrast data (for example, instead of the OCTA front image. OCTA tomographic image) may be used. Further, the target image of the high image quality processing is not only the OCTA front image, but also various medical applications such as an OCTA tomographic image which is a brightness front image and a B scan image, and a tomographic image (OCTA tomographic image) of motion contrast data. It may be an image. That is, the target image for the high image quality processing may be, for example, at least one of various medical images displayed on the display screen of the display unit 116. At this time, for example, since the feature amount of the image may differ depending on the type of the image, a trained model for high image quality corresponding to each type of the target image for the high image quality processing may be used. For example, when the high image quality button is pressed in response to an instruction from the examiner, the OCTA front image is not only processed for high image quality by using the learned model for high image quality corresponding to the OCTA front image. The OCT tomographic image may also be configured to be high quality processed using a trained model for high image quality corresponding to the OCT tomographic image. Further, for example, when the high image quality button is pressed in response to an instruction from the examiner, the high image quality OCTA front image generated by using the learned model for high image quality corresponding to the OCTA front image is displayed. It may be configured to be changed to display a high-quality OCT tomographic image generated by using a trained model for high image quality corresponding to the OCT tomographic image. At this time, the line indicating the position of the OCT tomographic image may be configured to be superimposed and displayed on the OCTA front image. Further, the line may be configured to be movable on the OCTA front image according to an instruction from the examiner. Further, when the display of the high image quality button is in the active state, after the above line is moved, the high image quality OCT tomographic image obtained by performing high image quality processing on the OCT tomographic image corresponding to the position of the current line is performed. It may be configured to be modified to display an image. Further, by displaying the high image quality button for each target image of the high image quality processing, the high image quality processing may be independently enabled for each image.

Further, information indicating a blood vessel region (for example, motion contrast data equal to or higher than a threshold value) in the OCTA tomographic image may be superimposed and displayed on the OCT tomographic image which is a B scan image at a corresponding position. At this time, for example, when the image quality of the OCTA tomographic image is improved, the image quality of the OCTA tomographic image at the corresponding position may be improved. Then, the information indicating the blood vessel region in the OCTA tomographic image obtained by improving the image quality may be superimposed and displayed on the OCTA tomographic image obtained by improving the image quality. The information indicating the blood vessel region may be any information as long as it is identifiable information such as color. Further, the superimposed display and non-display of the information indicating the blood vessel region may be configured to be changeable according to an instruction from the examiner. Further, when the line indicating the position of the OCT tomographic image is moved on the OCTA front image, the display of the OCT tomographic image may be updated according to the position of the line. At this time, since the OCTA tomographic image at the corresponding position is also updated, the superimposed display of the information indicating the blood vessel region obtained from the OCTA tomographic image may be updated. Thereby, for example, the three-dimensional distribution and state of the blood vessel region can be effectively confirmed while easily confirming the positional relationship between the blood vessel region and the region of interest at an arbitrary position. Further, the image quality of the OCTA tomographic image may be improved by an addition averaging process or the like of a plurality of OCTA tomographic images acquired at the corresponding positions instead of using the trained model for the image quality improvement. .. Further, the OCT tomographic image may be a pseudo OCT tomographic image reconstructed as a cross section at an arbitrary position in the OCT volume data. Further, the OCTA tomographic image may be a pseudo OCTA tomographic image reconstructed as a cross section at an arbitrary position in the OCTA volume data. The arbitrary position may be at least one arbitrary position, and may be configured to be changeable according to an instruction from the examiner. At this time, a plurality of pseudo tomographic images corresponding to a plurality of positions may be reconstructed.

Only one tomographic image (for example, OCT tomographic image or OCTA tomographic image) may be displayed, or a plurality of tomographic images may be displayed. When a plurality of tomographic images are displayed, the tomographic images acquired at positions in different sub-scanning directions may be displayed, or the plurality of tomographic images obtained by, for example, cross-scanning may be displayed in high quality. When displaying, images in different scanning directions may be displayed. Further, when displaying a plurality of tomographic images obtained by, for example, a radial scan with high image quality, a plurality of partially selected tomographic images (for example, two tomographic images at positions symmetrical with respect to a reference line) are displayed. ) May be displayed respectively. Further, a plurality of tomographic images are displayed on the follow-up display screen (follow-up display screen), and instructions for improving the image quality and analysis results (for example, the thickness of a specific layer, etc.) are obtained by the same method as the above method. ) May be displayed. At this time, the plurality of tomographic images displayed may be a plurality of tomographic images obtained at different dates and times at a predetermined part of the eye to be inspected, or a plurality of tomographic images obtained at different times on the same examination day. May be good. Further, the tomographic image may be subjected to high image quality processing based on the information stored in the database by the same method as the above method.

Similarly, when displaying the SLO image with high image quality, for example, the SLO image displayed on the same display screen may be displayed with high image quality. Further, when displaying the front image of brightness with high image quality, for example, the front image of brightness displayed on the same display screen may be displayed with high image quality. Further, a plurality of SLO images and front images having brightness are displayed on the display screen for follow-up observation, and instructions for improving the image quality and analysis results (for example, the thickness of a specific layer) are displayed by the same method as the above method. May be done. Further, the image quality improving process may be executed on the SLO image or the front image having the brightness based on the information stored in the database by the same method as the above method. The display of the tomographic image, the SLO image, and the front image of the brightness is an example, and these images may be displayed in any mode depending on the desired configuration. Further, at least two or more of the OCTA front image, the tomographic image, the SLO image, and the brightness front image may be displayed with high image quality by one instruction.

With such a configuration, the display control unit 121 can display the high-quality image obtained by the high-quality processing on the display unit 116. If at least one of a plurality of conditions relating to the display of a high-quality image, the display of the analysis result, the depth range of the displayed front image, etc. is selected, the selection is made even if the display screen is changed. It may be configured so that the specified conditions are maintained. The display control unit 121 may control the display of various high-quality images, the above lines, information indicating the blood vessel region, and the like.

Further, the high image quality model may be used for at least one frame of the live moving image on the preview screen displayed on the display unit 116 by the display control unit 121. At this time, when a plurality of live moving images of different parts or different types are displayed on the preview screen, the trained model corresponding to each live moving image may be used. For example, as the front eye image used for the alignment process, an image whose image quality has been improved by using a high image quality model for the front eye image may be used. Similarly, for various images used for the detection process of a predetermined region in various images, images having high image quality using a high image quality model for each image may be used.

At this time, for example, when the high image quality button is pressed in response to an instruction from the examiner, a plurality of different types of live moving images (for example, anterior segment image, SLO image, tomographic image) are displayed (for example). (At the same time), it may be configured to be changed to display a high-quality moving image obtained by performing high-quality processing. At this time, the display of the high-quality moving image may be a continuous display of the high-quality image obtained by performing high-quality processing on each frame. Further, for example, since the feature amount of the image may differ depending on the type of the image, a trained model for high image quality corresponding to each type of the target image for the high image quality processing may be used. For example, when the high image quality button is pressed in response to an instruction from the examiner, the front eye image is not only processed for high image quality by using the high image quality model corresponding to the front eye image, but also the SLO image is supported. The SLO image may also be configured to be processed for high image quality by using the high image quality model. Further, for example, when the high image quality button is pressed in response to an instruction from the examiner, the display is changed to the high image quality front eye image generated by using the high image quality model corresponding to the front eye image. Not only that, the display of the high-quality SLO image generated by using the high-quality model corresponding to the SLO image may be changed. Further, for example, when the high image quality button is pressed in response to an instruction from the examiner, not only the high image quality processing of the SLO image is performed by using the high image quality model corresponding to the SLO image, but also the tomographic image is supported. The tomographic image may also be configured to be processed for high image quality by using the high image quality model. Further, for example, when the high image quality button is pressed in response to an instruction from the examiner, the display is simply changed to the high image quality SLO image generated by using the high image quality model corresponding to the SLO image. Instead, the display may be changed to a high-quality tomographic image generated by using a high-quality model corresponding to the tomographic image. At this time, the line indicating the position of the tomographic image may be superimposed and displayed on the SLO image. Further, the line may be configured to be movable on the SLO image according to an instruction from the examiner. Further, when the display of the high image quality button is in the active state, after the above line is moved, the tomographic image corresponding to the position of the current line is subjected to high image quality processing to obtain a high image quality tomographic image. It may be configured to change to display. Further, by displaying the high image quality button for each target image of the high image quality processing, the high image quality processing may be independently enabled for each image.

As a result, for example, even if it is a live moving image, the processing time can be shortened, so that the examiner can obtain highly accurate information before the start of shooting. Therefore, for example, when the operator corrects the alignment position while checking the preview screen, it is possible to reduce the failure of re-imaging and the like, so that the accuracy and efficiency of the diagnosis can be improved. Further, the control unit 117 scans (rescans) the partial area such as the artifact area obtained by the segmentation process or the like during the shooting or at the end of the shooting in response to the instruction regarding the start of the shooting. The means may be driven and controlled. Depending on the movement of the eye to be inspected, it may not be possible to take a good picture by one rescan. Therefore, the drive may be controlled so that the rescan is repeated a predetermined number of times. At this time, the rescan may be terminated in response to an instruction from the operator (for example, after pressing the shooting cancel button) even during a predetermined number of rescans. At this time, it may be configured to save the shooting data until the rescan is completed according to the instruction from the operator. For example, a confirmation dialog may be displayed after the shooting cancel button is pressed, and the shooting data may be saved or the shooting data may be discarded according to an instruction from the operator. Also, for example, after pressing the shooting cancel button, the next rescan is not executed (although the current rescan is executed until it is completed), and it waits until there is an instruction (input) from the operator in the confirmation dialog. It may be configured as follows. Further, for example, when the information indicating the certainty of the object recognition result regarding the region of interest (for example, the numerical value indicating the ratio) exceeds the threshold value, each adjustment, shooting start, etc. may be automatically performed. Good. Further, for example, when the information indicating the certainty of the object recognition result regarding the region of interest (for example, the numerical value indicating the ratio) exceeds the threshold value, each adjustment, the start of imaging, etc. can be executed according to the instruction from the examiner. It may be configured to change to the above state (release the execution prohibited state).

Here, during auto-alignment, it is possible that the imaged object such as the retina of the eye E to be inspected has not yet been successfully imaged. Therefore, since there is a large difference between the medical image input to the trained model and the medical image used as the training data, there is a possibility that a high-quality image cannot be obtained with high accuracy. Therefore, when the evaluation value such as the image quality evaluation of the tomographic image (B scan image) exceeds the threshold value, the display of the high-quality moving image (continuous display of the high-quality frame) may be automatically started. Further, when the evaluation value such as the image quality evaluation of the tomographic image exceeds the threshold value, the image quality enhancement button may be configured to be changed to a state (active state) that can be specified by the examiner. The high image quality button is a button for designating the execution of the high image quality processing. Of course, the high image quality button may be a button for instructing the display of a high image quality image.

Further, a different high image quality model may be prepared for each shooting mode having a different scan pattern or the like, and a learned model for high image quality corresponding to the selected shooting mode may be selected. Further, one high image quality model obtained by learning learning data including various medical images obtained in different imaging modes may be used.

Here, in an ophthalmic apparatus, for example, an OCT apparatus, the scan pattern of the luminous flux used for the measurement and the imaging portion differ depending on the imaging mode. Therefore, for the trained model that uses the tomographic image as input data, a trained model is prepared for each shooting mode, and the trained model corresponding to the shooting mode selected according to the instruction of the operator is selected. It may be configured. In this case, the imaging mode may include, for example, a retinal imaging mode, an anterior segment imaging mode, a vitreous imaging mode, a macula imaging mode, an optic disc imaging mode, an OCTA imaging mode, and the like. Further, the scan pattern may include 3D scan, radial scan, cross scan, circle scan, raster scan, Lissajous scan (scan along the Lissajous curve) and the like. In the OCTA photographing mode, the drive control unit (not shown) controls the scanning means described above so that the measurement light is scanned a plurality of times in the same region (same position) of the eye to be inspected. Even in the OCTA shooting mode, for example, raster scan, radial scan, cross scan, circle scan, Lissajous scan and the like can be set as the scan pattern. Further, with respect to the trained model using the tomographic image as input data, it is possible to perform training by using the tomographic image corresponding to the cross section in different directions as the training data. For example, learning may be performed using a tomographic image of a cross section in the xz direction, a tomographic image of a cross section in the yz direction, or the like as training data.

The necessity of executing the high image quality processing by the high image quality model (or displaying the high image quality image obtained by the high image quality processing) is determined by the operator regarding the high image quality button provided on the display screen. It may be performed according to an instruction, or may be performed according to a setting stored in the storage unit 114 in advance. It should be noted that the fact that the high image quality processing using the trained model (high image quality improvement model) may be displayed in the active state of the high image quality button, or that this may be displayed on the display screen as a message. Good. Further, the execution of the high image quality processing may maintain the execution state at the time of the previous examination of the ophthalmic apparatus, or may maintain the execution state at the time of the previous examination for each subject.

Further, the moving image to which various trained models such as the high image quality improvement model can be applied is not limited to the live moving image, and may be, for example, a moving image stored (stored) in the storage unit 114. At this time, for example, the moving image obtained by aligning each frame of the tomographic moving image of the fundus stored (stored) in the storage unit 114 may be displayed on the display screen. For example, when it is desired to observe the vitreous body preferably, first, a reference frame based on a condition such as the presence of the vitreous body on the frame as much as possible may be selected. At this time, each frame is a tomographic image (B scan image) in the XZ direction. Then, a moving image in which another frame is aligned in the XZ direction with respect to the selected reference frame may be displayed on the display screen. At this time, for example, a high-quality image (high-quality frame) sequentially generated by the high-quality engine may be continuously displayed for each at least one frame of the moving image.

As the above-mentioned alignment method between frames, the same method may be applied to the alignment method in the X direction and the alignment method in the Z direction (depth direction), and all different methods may be applied. May be applied. Further, the alignment in the same direction may be performed a plurality of times by different methods, and for example, a precise alignment may be performed after performing a rough alignment. Further, as a method of alignment, for example, a plurality of alignments obtained by segmenting a tomographic image (B scan image) using a retinal layer boundary obtained by performing segmentation processing (coarse in the Z direction) and dividing the tomographic image. Alignment (precise in the X and Z directions) using the correlation information (similarity) between the region and the reference image, and a one-dimensional projected image generated for each tomographic image (B scan image) was used (in the X direction). ) Alignment, etc. Alignment (in the X direction) using a two-dimensional front image. Further, it may be configured so that the alignment is roughly performed in pixel units and then the precise alignment is performed in subpixel units.

Further, the high image quality improvement model may be updated by additional learning using the value of the ratio set (changed) according to the instruction from the examiner as the learning data. For example, when the input image is relatively dark, if the examiner tends to set a high ratio of the input image to the high-quality image, the trained model will be additionally trained so as to have such a tendency. Thereby, for example, it can be customized as a trained model that can obtain the ratio of synthesis that suits the taste of the examiner. At this time, a button for deciding whether or not to use the set (changed) ratio value as the learning data for the additional learning according to the instruction from the examiner may be displayed on the display screen. Further, the ratio determined by using the trained model may be set as the default value, and then the ratio value may be changed from the default value according to the instruction from the examiner. Further, the high image quality model may be a trained model obtained by additionally learning learning data including at least one high image quality image generated by using the high image quality model. At this time, it may be configured so that whether or not to use the high-quality image as learning data for additional learning can be selected by an instruction from the examiner.

(Modification example 4)
In the various embodiments and modifications described above, the control unit 117 may generate a label image of the image acquired by photographing using a learned model for image segmentation, and perform image segmentation processing. Here, the label image means a label image in which a region is labeled for each pixel of the tomographic image. Specifically, it is an image in which an arbitrary region is divided by a identifiable pixel value (hereinafter, label value) group among the region groups drawn in the acquired image. Here, the specified arbitrary region includes a region of interest, a volume of interest (VOI: Volume Of Interest), and the like.

By specifying the coordinate group of the pixel having an arbitrary label value from the image, the coordinate group of the pixel that depicts the corresponding region such as the retinal layer in the image can be specified. Specifically, for example, when the label value indicating the ganglion cell layer constituting the retina is 1, the coordinate group having the pixel value of 1 among the pixel groups of the image is specified, and the coordinate group corresponds to the coordinate group from the image. Extract the pixel group to be used. Thereby, the region of the ganglion cell layer in the image can be identified.

The image segmentation process may include a process of reducing or enlarging the label image. At this time, the image complement processing method used for reducing or enlarging the label image shall use the nearest neighbor method or the like so as not to erroneously generate an undefined label value or a label value that should not exist at the corresponding coordinates. ..

The image segmentation process is a process for specifying an area called ROI (Region Of Interest) or VOI, such as an organ or a lesion depicted in an image, for use in image diagnosis or image analysis. For example, according to the image segmentation process, it is possible to identify the region group of the group group constituting the retina from the image acquired by the OCT imaging of the back eye portion as the imaging target. If the area to be specified is not drawn in the image, the number of specified areas is 0. Further, as long as a plurality of region groups to be specified are depicted in the image, the number of the specified regions may be a plurality, or even one region surrounding the region groups so as to include the region groups. Good.

The specified area group is output as information that can be used in other processing. Specifically, for example, the coordinate group of the pixel group constituting each of the specified region groups can be output as a numerical data group. Further, for example, a coordinate group indicating a rectangular region, an ellipsoid region, a rectangular parallelepiped region, an ellipsoid region, or the like including each of the specified region groups can be output as a numerical data group. Further, for example, a coordinate group indicating a straight line, a curve, a plane, a curved surface, or the like corresponding to the boundary of the specified region group can be output as a numerical data group. Further, for example, a label image showing the specified region group can be output.

Here, as a machine learning model for image segmentation, for example, a convolutional neural network (CNN) can be used. Here, with reference to FIG. 9, an example in which the machine learning model according to this modified example is configured by CNN will be described. FIG. 9 shows an example of the configuration of a trained model for image segmentation. In the example of the trained model, for example, when the tomographic image 1301 is input, the label image 1302 indicating the specified region group can be output.

The machine learning model shown in FIG. 9 is composed of a plurality of layers responsible for processing and outputting an input value group. The types of layers included in the configuration of the machine learning model include a convolution layer, a Downsampling layer, an Upsampling layer, and a Merger layer.

The convolution layer is a layer that performs convolution processing on an input value group according to parameters such as the kernel size of the set filter, the number of filters, the stride value, and the dilation value. The number of dimensions of the kernel size of the filter may be changed according to the number of dimensions of the input image.

The downsampling layer is a layer that performs processing to reduce the number of output value groups to be smaller than the number of input value groups by thinning out or synthesizing input value groups. Specifically, as such a process, for example, there is a Max Polling process.

The upsampling layer is a layer that performs processing to increase the number of output value groups to be larger than the number of input value groups by duplicating the input value group or adding the interpolated value from the input value group. Specifically, as such a process, for example, there is a linear interpolation process.

The composite layer is a layer in which a value group such as an output value group of a certain layer or a pixel value group constituting an image is input from a plurality of sources, and the processing is performed by concatenating or adding them.

Note that if the parameter settings for the layers and nodes that make up the neural network are different, the degree to which the tendency trained from the teacher data can be reproduced in the output data may differ. That is, in many cases, the appropriate parameters differ depending on the embodiment, and therefore, the values can be changed to preferable values as needed.

In addition to the method of changing the parameters as described above, there are cases where the CNN can obtain better characteristics by changing the configuration of the CNN. Better characteristics include, for example, more accurate alignment position information output, shorter processing time, shorter training time for machine learning models, and the like.

The CNN configuration used in this modification is a U-net having a function of an encoder composed of a plurality of layers including a plurality of downsampling layers and a function of a decoder composed of a plurality of layers including a plurality of upsampling layers. It is a type machine learning model. In the U-net type machine learning model, position information (spatial information) that is ambiguous in a plurality of layers configured as encoders is displayed in layers of the same dimension (layers corresponding to each other) in a plurality of layers configured as a decoder. ) (For example, using a skip connection).

Although not shown, as an example of changing the configuration of the CNN, for example, a batch normalization layer or an activation layer using a rectifier linear unit may be incorporated after the convolutional layer. Good. Through these steps of CNN, the features of the captured image can be extracted.

Examples of the machine learning model according to this modification include a CNN (U-net type machine learning model) as shown in FIG. 9, a model combining CNN and LSTM, an FCN (Full Convolutional Network), or an FCN (Full Convolutional Network). SegNet and the like can be used. Further, a machine learning model or the like that performs object recognition can also be used according to a desired configuration. As a machine learning model for performing object recognition, for example, RCNN (Region CNN), fastRCNN, or fasterRCNN can be used. Furthermore, a machine learning model that recognizes an object in units of regions can also be used. As a machine learning model that recognizes an object in a region unit, YOLO (You Only Look Object) or SSD (Single Shot Detector or Single Shot MultiBox Detector) can also be used.

Further, as the training data of the machine learning model for image segmentation, the tomographic image acquired by OCT is used as input data, and the label image in which the area is labeled for each pixel of the tomographic image is used as output data. Label images include, for example, the inner limiting membrane (ILM), the nerve fiber layer (NFL), the ganglion cell layer (GCL), the photoreceptor inner segment outer segment junction (ISOS), the retinal pigment epithelial layer (RPE), and Bruch. Labeled images with labels such as membrane (BM) and choroid can be used. Other regions include, for example, the vitreous body, sclera, outer plexiform layer (OPL), outer nuclear layer (ONL), inner nuclear layer (IPL), inner nuclear layer (INL), cornea, anterior chamber, and iris. And an image with a label such as a crystalline lens may be used.

Moreover, the input data of the machine learning model for image segmentation is not limited to the tomographic image. It may be an anterior segment image, an SLO image, an OCTA image, or the like. In this case, as the learning data, various images can be used as input data, and label images in which region names and the like are labeled for each pixel of various images can be used as output data. For example, when the input data of the training data is an SLO image, the output data may be an image labeled with a peripheral portion of the optic disc, Disc, Cup, or the like.

The label image used as the output data may be an image in which each region is labeled in the tomographic image by a doctor or the like, or an image in which each region is labeled by the rule-based region detection process. There may be. However, if machine learning is performed using a label image that is not properly labeled as the output data of the training data, the image obtained by using the trained model trained using the training data is also properly labeled. There is a possibility that the label image will not be used. Therefore, by removing the pair containing such a label image from the training data, it is possible to reduce the possibility that an inappropriate label image is generated by using the trained model. Here, the rule-based region detection process refers to a detection process that utilizes a known regularity such as the regularity of the shape of the retina.

The control unit 117 can be expected to detect a specific region of various images at high speed and with high accuracy by performing image segmentation processing using such a trained model for image segmentation. A trained model for image segmentation may also be prepared for each type of various images as input data. Further, for the OCTA front image and the En-Face image, a trained model may be prepared for each depth range for generating an image. Further, the trained model for image segmentation may also be one in which the image for each imaging site (for example, the center of the macula and the center of the optic nerve head) is trained, or the model is trained regardless of the imaging site. You may.

Further, for the trained model for image segmentation, additional learning may be performed using the data manually modified according to the instruction of the operator as training data. Further, the determination of the necessity of additional learning and the determination of whether or not to transmit the data to the server may be performed by the same method. In these cases as well, it can be expected that the accuracy of each process can be improved and the process can be performed according to the tendency of the examiner's preference.

Further, when the control unit 117 detects a partial region of the eye E to be inspected (for example, a region of interest, an artifact region, an abnormal region, etc.) using the trained model, the control unit 117 performs predetermined image processing for each detected partial region. Can also be applied. As an example, the case of detecting at least two partial regions of the vitreous region, the retinal region, and the choroid region will be described. In this case, when performing image processing such as contrast adjustment on at least two detected partial regions, adjustments suitable for each region can be performed by using different image processing parameters. By displaying an image adjusted suitable for each region, the operator can more appropriately diagnose a disease or the like in each partial region. The configuration using different image processing parameters for each detected partial region is similarly applied to the partial region of the eye E to be inspected obtained by detecting the partial region of the eye E to be inspected without using the trained model. You may.

(Modification 5)
The display control unit 121 in the various embodiments and modifications described above may display analysis results such as a layer thickness of a desired layer and various blood vessel densities on a report screen of a display screen after taking a tomographic image. In addition, the optic nerve head, macula, vascular region, capillary region, arterial region, venous region, nerve fiber bundle, vitreous region, macula region, choroidal region, scleral region, sciatic plate region, retinal layer boundary, retina Parameter values (distribution) for the site of interest, including at least one of the layer boundary edges, photoreceptors, blood cells, vascular wall, vascular inner wall boundary, vascular lateral boundary, ganglion cells, corneal region, angular region, Schlemm's canal, etc. May be displayed as the analysis result. Here, the site of interest may be, for example, a vortex vein or the like, which is an outlet of a blood vessel in the Haller layer (an example of a blood vessel in a partial depth range of the choroidal region) to the outside of the eye. At this time, the parameters related to the site of interest include, for example, the number of vortex veins (for example, the number for each region), the distance from the optic nerve head to each vortex vein, the angle at which each vortex vein centered on the optic nerve head is located, and the like. It may be. This makes it possible to accurately diagnose, for example, various diseases related to Pachychoroid (thickened choroid) (for example, choroidal neovascularization). Further, for example, by analyzing a medical image to which various artifact reduction processes are applied, the above-mentioned various analysis results can be displayed as accurate analysis results. The artifact is, for example, a false image region generated by light absorption by a blood vessel region or the like, a projection artifact, a band-shaped artifact in a front image generated in the main scanning direction of the measured light depending on the state of the eye to be inspected (movement, blinking, etc.), or the like. There may be. Further, the artifact may be any image loss region as long as it is randomly generated for each image taken on a medical image of a predetermined portion of the subject, for example. Further, the display control unit 121 may display the value (distribution) of the parameter relating to the region including at least one of the various artifacts (copy loss region) as described above on the display unit 116 as the analysis result. Further, the value (distribution) of the parameter relating to the region including at least one such as drusen, new blood vessel, vitiligo (hard vitiligo), and abnormal site such as pseudo-drusen may be displayed as the analysis result. Further, the comparison result obtained by comparing the standard value or standard range obtained by using the standard database with the analysis result may be displayed.

Further, the analysis result may be displayed in an analysis map, a sector showing statistical values corresponding to each divided area, or the like. The analysis result may be generated by using a trained model (analysis result generation engine, trained model for analysis result generation) obtained by learning the analysis result of the medical image as training data. .. At this time, the trained model is trained using training data including a medical image and an analysis result of the medical image, training data including a medical image and an analysis result of a medical image of a type different from the medical image, and the like. It may be obtained by.

Further, the training data for performing image analysis may include a label image generated by using the trained model for image segmentation processing and an analysis result of a medical image using the label image. In this case, the control unit 117 can function as an example of the analysis result generation unit that generates the analysis result of the tomographic image from the result of the image segmentation processing by using, for example, the trained model for the analysis result generation. .. Further, the trained model is a training data including input data in which a plurality of medical images of different types of predetermined parts are set as a set, such as an En-Face image and a motion contrast front image (En-Face image of OCTA) described later. It may be obtained by learning using.

Further, the analysis result obtained by using the high-quality image generated by using the high-quality model may be displayed. In this case, the input data included in the training data may be a high-quality image generated by using the trained model for high image quality, or may be a set of a low-quality image and a high-quality image. May be good. The training data may be an image in which at least a part of an image whose image quality has been improved by using the trained model has been manually or automatically modified.

Further, the training data includes, for example, at least analysis values (for example, average value, median value, etc.) obtained by analyzing the analysis area, a table including the analysis values, an analysis map, the position of the analysis area such as a sector in the image, and the like. The information including one may be the data labeled (annotated) with the input data as the correct answer data (for supervised learning). In addition, the analysis result obtained by using the trained model for generating the analysis result may be displayed according to the instruction from the operator.

Further, the estimation unit 119 in the above-described embodiment and modification can output an accurate estimation result by using, for example, an image to which various artifact reduction processes as described above are applied for the estimation process. it can. Further, the display control unit 121 may display the estimation result on the image at the position of the specified abnormal portion or the like, or may display the state or the like of the abnormal portion by characters or the like. Further, the display control unit 121 may display a classification result (for example, Curtin classification) of an abnormal site or the like as a diagnosis result separately from the estimation result for the disease. Further, as the classification result, for example, information indicating the certainty of each abnormal part (for example, a numerical value indicating a ratio) may be displayed. In addition, information necessary for the doctor to confirm the diagnosis may be displayed as a diagnosis result. As the necessary information, for example, advice such as additional shooting can be considered. For example, when an abnormal site is detected in the blood vessel region in the OCTA image, it may be displayed that fluorescence imaging using a contrast medium capable of observing the blood vessel in more detail than OCTA is performed. In addition, the diagnosis result may be information on the future medical treatment policy of the subject. In addition, the diagnosis result is, for example, the diagnosis name, the type and state (degree) of the lesion (abnormal site), the position of the lesion in the image, the position of the lesion with respect to the region of interest, the findings (interpretation findings, etc.), and the basis of the diagnosis name (affirmation). Medical support information, etc.) and grounds for denying the diagnosis name (negative medical support information, etc.) may be included in the information. At this time, for example, a diagnosis result that is more probable than the diagnosis result such as the diagnosis name input in response to the instruction from the examiner may be displayed as medical support information. Further, when a plurality of types of medical images are used, for example, the types of medical images that can be the basis of the diagnosis result may be displayed in an identifiable manner. In addition, the basis of the diagnosis result is a map (attention map, activation map) that visualizes the features extracted by the trained model, for example, a color map (heat map) that shows the features in color. Good. At this time, for example, the heat map may be superimposed and displayed on the medical image used as the input data. The heat map is, for example, a method for visualizing a region (a region having a large gradient) that contributes greatly to the output value of the predicted (estimated) class, such as Grad-CAM (Gradient-weighted Class Activation Mapping) or Guided Grade. -Can be obtained using CAM or the like.

The diagnosis result may be generated by using a trained model (diagnosis result generation engine, trained model for generation of diagnosis result) obtained by learning the diagnosis result of the medical image as training data. .. In addition, the trained model is based on training using training data including a medical image and a diagnosis result of the medical image, and training data including a medical image and a diagnosis result of a medical image of a type different from the medical image. It may be obtained.

Further, the training data may include a label image generated by using the trained model for image segmentation processing and a diagnosis result of a medical image using the label image. In this case, the control unit 117 can function as an example of a diagnosis result generation unit that generates a diagnosis result of a tomographic image from the result of image segmentation processing by using, for example, a learned model for generating a diagnosis result. ..

Further, the diagnostic result obtained by using the high-quality image generated by using the trained model for high image quality may be displayed. In this case, the input data included in the training data may be a high-quality image generated by using the trained model for high image quality, or may be a set of a low-quality image and a high-quality image. May be good. The training data may be an image in which at least a part of an image whose image quality has been improved by using the trained model has been manually or automatically modified.

In addition, the learning data includes, for example, the diagnosis name, the type and state (degree) of the lesion (abnormal site), the position of the lesion in the image, the position of the lesion with respect to the region of interest, the findings (interpretation findings, etc.), and the basis of the diagnosis name (affirmation). Information including at least one such as (general medical support information, etc.) and grounds for denying the diagnosis name (negative medical support information), etc. are labeled (annotated) in the input data as correct answer data (for supervised learning). Data may be used. In addition, according to the instruction from the examiner, the diagnosis result obtained by using the trained model for generating the diagnosis result may be displayed.

A trained model may be prepared for each information used as input data or for each type of information, and a diagnosis result may be acquired using the trained model. In this case, the information output from each trained model may be statistically processed to determine the final diagnostic result. For example, the ratio of the information output from each trained model may be added to each type of information, and the information having a higher total ratio than the other information may be determined as the final diagnosis result. The statistical processing is not limited to the calculation of the total, and may be the calculation of the average value or the median value. Further, for example, among the information output from each trained model, the diagnosis result may be determined using information having a higher ratio than other information (information having the highest ratio). Similarly, the diagnosis result may be determined using the information of the ratio of the information output from each trained model that is equal to or greater than the threshold value.

Further, it may be configured so that the quality of the determined diagnosis result can be determined (approved) according to the instruction (selection) of the operator. Further, the diagnosis result may be determined from the information output from each trained model according to the instruction (selection) of the operator. At this time, for example, the display control unit 121 may display the information output from each trained model and the ratio thereof side by side on the display unit 116. Then, the operator may be configured to determine the selected information as a diagnosis result, for example, by selecting information having a higher ratio than other information. Further, the diagnosis result may be determined by using the machine learning model from the information output from each trained model. In this case, the machine learning algorithm may be a different type of machine learning algorithm from the machine learning algorithm used to generate the diagnostic result, for example, a neural network, a support vector machine, an adaboost, a Bayesian network, or a random. A forest or the like may be used.

The learning of the various trained models described above may be not only supervised learning (learning with labeled learning data) but also semi-supervised learning. In semi-supervised learning, for example, after multiple classifiers (classifiers) perform supervised learning, unlabeled learning data is identified (classified) and the identification result (classification result) is determined according to the reliability. This is a method in which (for example, an identification result whose certainty is equal to or higher than a threshold value) is automatically labeled (annotated) and learning is performed using the labeled learning data. Semi-supervised learning may be, for example, co-training (or Multiview). At this time, the trained model for generating the diagnosis result uses, for example, a first discriminator that identifies a medical image of a normal subject and a second discriminator that discriminates a medical image including a specific lesion. It may be a trained model obtained by semi-supervised learning (for example, co-training). The purpose is not limited to diagnostic purposes, but may be, for example, photography support. In this case, the second classifier may, for example, identify a medical image including a partial area such as a region of interest or an artifact region.

In addition, the display control unit 121 according to the various embodiments and modifications described above has an object recognition result (object detection) of a partial region such as a region of interest, an artifact region, and an abnormal region as described above on the report screen of the display screen. The result) and the segmentation result may be displayed. At this time, for example, a rectangular frame or the like may be superimposed and displayed around the object on the image. Further, for example, colors and the like may be superimposed and displayed on the object in the image. The object recognition result and the segmentation result are learned models (object recognition engine, for object recognition) obtained by learning the learning data in which the information indicating the object recognition and the segmentation is used as the correct answer data and labeled (annotated) on the medical image. It may be generated using a trained model, a segmentation engine, a trained model for segmentation). The above-mentioned analysis result generation and diagnosis result generation may be obtained by using the above-mentioned object recognition result and segmentation result. For example, analysis result generation and diagnosis result generation processing may be performed on a region of interest obtained by object recognition or segmentation processing.

Further, when detecting an abnormal portion, the control unit 117 may use a hostile generation network (GAN: Generative Adversarial Networks) or a variational autoencoder (VAE: Variational Auto-Encoder). For example, a DCGAN (Deep Convolutional GAN) consisting of a generator obtained by learning the generation of a medical image and a classifier obtained by learning the discrimination between a new medical image generated by the generator and a real medical image. Can be used as a machine learning model.

When DCGAN is used, for example, the discriminator encodes the input medical image into a latent variable, and the generator generates a new medical image based on the latent variable. After that, the difference between the input medical image and the generated new medical image can be extracted (detected) as an abnormal part. When VAE is used, for example, the input medical image is encoded by an encoder to be a latent variable, and the latent variable is decoded by a decoder to generate a new medical image. After that, the difference between the input medical image and the generated new medical image can be extracted as an abnormal part.

Further, the control unit 117 may detect an abnormal portion by using a convolutional autoencoder (CAE). When CAE is used, the same medical image is learned as input data and output data at the time of learning. As a result, when a medical image having an abnormal part is input to the CAE at the time of estimation, a medical image having no abnormal part is output according to the learning tendency. After that, the difference between the medical image input to the CAE and the medical image output from the CAE can be extracted as an abnormal part.

In these cases, the control unit 117 uses information on the difference between the medical image obtained by using the hostile generation network or the autoencoder and the medical image input to the hostile generation network or the autoencoder as information on the abnormal part. Can be generated. As a result, the control unit 117 can be expected to detect the abnormal portion at high speed and with high accuracy. For example, even if it is difficult to collect many medical images including abnormal parts as learning data in order to improve the detection accuracy of abnormal parts, a relatively large number of medical images of normal subjects that are easy to collect are used as learning data. Can be used. Therefore, for example, learning for accurately detecting an abnormal portion can be performed efficiently. Here, the autoencoder includes VAE, CAE, and the like. In addition, at least a part of the generation part of the hostile generation network may be composed of VAE. Thereby, for example, it is possible to generate a relatively clear image while reducing the phenomenon of generating similar data. For example, the control unit 117 provides information on the difference between a medical image obtained from various medical images using a hostile generation network or an autoencoder and a medical image input to the hostile generation network or the autoencoder. It can be generated as information about the abnormal part. Further, for example, the display control unit 121 relates to a difference between a medical image obtained from various medical images using a hostile generation network or an autoencoder and a medical image input to the hostile generation network or the autoencoder. The information can be displayed on the display unit 116 as information on the abnormal portion.

Further, in particular, the trained model for generating the diagnosis result may be a trained model obtained by learning from the training data including the input data including a set of a plurality of medical images of different types of the predetermined part of the subject. Good. At this time, as the input data included in the training data, for example, input data in which a motion contrast front image of the fundus and a luminance front image (or a luminance tom image) are set can be considered. Further, as the input data included in the training data, for example, input data in which a tomographic image (B scan image) of the fundus and a color fundus image (or a fluorescent fundus image) are set can be considered. Further, the plurality of medical images of different types may be anything as long as they are acquired by different modality, different optical systems, different principles, or the like.

Further, the trained model for generating the diagnosis result may be a trained model obtained by learning from the training data including the input data including a plurality of medical images of different parts of the subject. At this time, as the input data included in the training data, for example, input data in which a tomographic image of the fundus (B scan image) and a tomographic image of the anterior segment of the eye (B scan image) are considered as a set can be considered. Further, as the input data included in the training data, for example, input data in which a three-dimensional OCT image (three-dimensional tomographic image) of the macula of the fundus and a circle scan (or raster scan) tomographic image of the optic nerve head of the fundus are set. Is also possible.

The input data included in the learning data may be different parts of the subject and a plurality of different types of medical images. At this time, the input data included in the training data may be, for example, input data in which a tomographic image of the anterior segment of the eye and a color fundus image are set. Further, the trained model described above may be a trained model obtained by learning from training data including input data including a set of a plurality of medical images having different shooting angles of view of a predetermined portion of the subject. Further, the input data included in the learning data may be a combination of a plurality of medical images obtained by time-dividing a predetermined portion into a plurality of regions, such as a panoramic image. At this time, by using a wide angle of view image such as a panoramic image as learning data, there is a possibility that the feature amount of the image can be accurately acquired because the amount of information is larger than that of the narrow angle of view image. The result of can be improved. Further, the input data included in the learning data may be input data in which a plurality of medical images of different dates and times of a predetermined part of the subject are set.

Further, the display screen on which at least one of the above-mentioned estimation result, analysis result, diagnosis result, object recognition result, and segmentation result is displayed is not limited to the report screen. Such a display screen is, for example, at least one display screen such as a shooting confirmation screen, a display screen for follow-up observation, and a preview screen for various adjustments before shooting (a display screen on which various live moving images are displayed). It may be displayed in. For example, by displaying at least one result obtained by using the above-mentioned trained model on the shooting confirmation screen, the operator can confirm the accurate result even immediately after shooting.

Further, for example, when a specific object is recognized, a frame surrounding the recognized object may be configured to be superimposed and displayed on the live moving image. At this time, if the information indicating the certainty of the object recognition result (for example, the numerical value indicating the ratio) exceeds the threshold value, the color of the frame surrounding the object may be changed or highlighted. Good. This allows the examiner to easily identify the object on the live video.

In addition, in the generation of the correct answer data used for learning the various trained models described above, the trained model for generating the correct answer data for generating the correct answer data such as labeling (annotation) may be used. At this time, the trained model for generating correct answer data may be obtained by (sequentially) additionally learning the correct answer data obtained by labeling (annotation) by the examiner. That is, the trained model for generating correct answer data may be obtained by additional training of training data in which the data before labeling is used as input data and the data after labeling is used as output data. Further, in a plurality of consecutive frames such as a moving image, the result of the frame judged to have low accuracy is corrected in consideration of the results of object recognition and segmentation of the preceding and following frames. May be good. At this time, according to the instruction from the examiner, the corrected result may be additionally learned as correct answer data. Further, for example, for a medical image with low accuracy of the result, the feature amount is an example of a map (attention map, activation map) in which the examiner visualizes the feature amount extracted by the trained model on the medical image. The image labeled (annotated) may be additionally learned as input data while checking the color map (heat map) showing the above in color. For example, in the heat map on the layer just before outputting the result in the trained model, if the part to be noted is different from the intent of the examiner, label the part that the examiner thinks to be noticed (annotation). You may additionally learn the medical image. Thereby, for example, the trained model is a partial region on the medical image, and the feature amount of the partial region having a relatively large influence on the output result of the trained model is prioritized over the other regions ( Additional learning can be done (with weight).

Here, the various trained models described above can be obtained by machine learning using the training data. Machine learning includes, for example, deep learning consisting of a multi-layer neural network. Further, for example, a convolutional neural network can be used for at least a part of the multi-layer neural network. Further, a technique related to an autoencoder (self-encoder) may be used for at least a part of a multi-layer neural network. Further, a technique related to backpropagation (error backpropagation method) may be used for learning. Further, for learning, a method (dropout) in which each unit (each neuron or each node) is randomly inactivated may be used. Further, for learning, a method (batch normalization) may be used in which the data transmitted to each layer of the multi-layer neural network is normalized before the activation function (for example, the ReLu function) is applied. However, the machine learning is not limited to deep learning, and any learning using a model capable of extracting (expressing) the features of learning data such as images by learning may be used. Here, the machine learning model refers to a learning model based on a machine learning algorithm such as deep learning. The trained model is a model in which a machine learning model by an arbitrary machine learning algorithm is trained (learned) in advance using appropriate learning data. However, the trained model does not require further learning, and additional learning can be performed. The learning data is composed of a pair of input data and output data (correct answer data). Here, the learning data may be referred to as teacher data, or the correct answer data may be referred to as teacher data.

The GPU can perform efficient calculations by processing more data in parallel. Therefore, when learning is performed a plurality of times using a learning model such as deep learning, it is effective to perform processing on the GPU. Therefore, in this modification, the GPU is used in addition to the CPU for the processing by the control unit 117, which is an example of the learning unit (not shown). Specifically, when executing a learning program including a learning model, learning is performed by the CPU and the GPU collaborating to perform calculations. The processing of the learning unit may be performed only by the CPU or GPU. Further, the processing unit (estimation unit 119) that executes the processing using the various trained models described above may also use the GPU in the same manner as the learning unit. Further, the learning unit may include an error detecting unit and an updating unit (not shown). The error detection unit obtains an error between the output data output from the output layer of the neural network and the correct answer data according to the input data input to the input layer. The error detection unit may use the loss function to calculate the error between the output data from the neural network and the correct answer data. Further, the update unit updates the coupling weighting coefficient between the nodes of the neural network based on the error obtained by the error detection unit so that the error becomes small. This updating unit updates the coupling weighting coefficient and the like by using, for example, the backpropagation method. The error backpropagation method is a method of adjusting the coupling weighting coefficient between the nodes of each neural network so that the above error becomes small.

Further, as the machine learning model used for the above-mentioned object recognition, segmentation, high image quality, etc., the function of an encoder composed of a plurality of layers including a plurality of downsampling layers and a plurality of layers including a plurality of upsampling layers A U-net type machine learning model having a decoder function consisting of is applicable. In the U-net type machine learning model, position information (spatial information) that is ambiguous in a plurality of layers configured as encoders is displayed in layers of the same dimension (layers corresponding to each other) in a plurality of layers configured as a decoder. ) (For example, using a skip connection).

Further, as the machine learning model used for the above-mentioned object recognition, segmentation, high image quality, etc., for example, FCN (Full Convolutional Network), SegNet, or the like can be used. Further, a machine learning model that recognizes an object in a region unit according to a desired configuration may be used. As a machine learning model for performing object recognition, for example, RCNN (Region CNN), fastRCNN, or fasterRCNN can be used. Further, as a machine learning model for recognizing an object in a region unit, YOLO (You Only Look Object) or SSD (Single Shot Detector or Single Shot MultiBox Detector) can also be used.

Further, the machine learning model may be, for example, a capsule network (Capsule Network; CapsNet). Here, in a general neural network, each unit (each neuron or each node) is configured to output a scalar value, for example, a spatial positional relationship (relative position) between features in an image. It is configured to reduce spatial information about. Thereby, for example, learning can be performed so as to reduce the influence of local distortion and translation of the image. On the other hand, in the capsule network, each unit (each capsule) is configured to output spatial information as a vector, so that, for example, spatial information is retained. Thereby, for example, learning can be performed in which the spatial positional relationship between the features in the image is taken into consideration.

(Modification 6)
In the preview screens in the various embodiments and modifications described above, the various trained models described above may be used for at least one frame of the live video. At this time, when a plurality of live moving images of different parts or different types are displayed on the preview screen, the trained model corresponding to each live moving image may be used. As a result, for example, even if it is a live moving image, the processing time can be shortened, so that the examiner can obtain highly accurate information before the start of shooting. Therefore, for example, the failure of re-imaging can be reduced, so that the accuracy and efficiency of diagnosis can be improved.

The plurality of live moving images may be, for example, a moving image of the anterior segment for alignment in the XYZ directions, and a frontal moving image of the fundus for focusing adjustment and OCT focus adjustment of the fundus observation optical system. Further, the plurality of live moving images may be, for example, a tomographic moving image of the fundus for coherence gate adjustment of OCT (adjustment of the optical path length difference between the measured optical path length and the reference optical path length). When such a preview image is displayed, the above-mentioned various adjustments are performed so that the region detected by using the above-mentioned trained model for object recognition and the above-mentioned trained model for segmentation satisfies a predetermined condition. The control unit 117 may be configured as described above. For example, a value (for example, a contrast value or an intensity value) related to a predetermined retinal layer such as a vitreous region or RPE detected by using a trained model for object recognition or a trained model for segmentation exceeds a threshold value (for example, a contrast value or an intensity value). Alternatively, various adjustments such as OCT focus adjustment may be performed so as to reach a peak value). Further, for example, the OCT so that a predetermined retinal layer such as a vitreous region or RPE detected by using a trained model for object recognition or a trained model for segmentation is at a predetermined position in the depth direction. The coherence gate adjustment may be configured to be performed.

In these cases, the control unit 117 can generate a high-quality moving image by performing high-quality processing on the moving image using the trained model. In addition, the drive control unit (not shown) is a reference mirror or the like so that a partial area such as a region of interest obtained by segmentation processing or the like is at a predetermined position in the display area while a high-quality moving image is displayed. It is possible to drive and control the optical member for changing the imaging range of. In such a case, the drive control unit can automatically perform the alignment process so that the desired region is at a predetermined position in the display region based on highly accurate information. The optical member that changes the photographing range may be, for example, an optical member that adjusts the coherence gate position, and specifically, a reference mirror that reflects reference light. Further, the coherence gate position can be adjusted by an optical member that changes the optical path length difference between the measurement optical path length and the reference optical path length, and the optical member can change, for example, the optical path length of the measurement light (not shown). It may be a mirror or the like. The optical member for changing the photographing range may be, for example, a stage portion (not shown). Further, according to the drive control unit and the instruction regarding the start of shooting, the scanning means is driven so that the partial area such as the artifact area obtained by the segmentation process or the like is photographed (rescanned) again during or at the end of the photographing. You may control it. Further, for example, when the information indicating the certainty of the object recognition result regarding the region of interest (for example, the numerical value indicating the ratio) exceeds the threshold value, various adjustments, shooting start, etc. may be automatically performed. Good. Further, for example, when the information indicating the certainty of the object recognition result regarding the region of interest (for example, the numerical value indicating the ratio) exceeds the threshold value, each adjustment, the start of imaging, etc. can be executed according to the instruction from the examiner. It may be configured to change to the above state (release the execution prohibited state).

Further, the moving image to which the various trained models described above can be applied is not limited to the live moving image, and may be, for example, a moving image stored (stored) in the storage unit 114. At this time, for example, the moving image obtained by aligning each frame of the tomographic moving image of the fundus stored (stored) in the storage unit 114 may be displayed on the display screen. For example, when it is desired to observe the vitreous body preferably, first, a reference frame based on a condition such as the presence of the vitreous body on the frame as much as possible may be selected. At this time, each frame is a tomographic image (B scan image) in the XZ direction. Then, a moving image in which another frame is aligned in the XZ direction with respect to the selected reference frame may be displayed on the display screen. At this time, for example, a high-quality image (high-quality frame) sequentially generated by the trained model for high image quality may be continuously displayed for each at least one frame of the moving image.

As the above-mentioned alignment method between frames, the same method may be applied to the alignment method in the X direction and the alignment method in the Z direction (depth direction), and all different methods may be applied. May be applied. Further, the alignment in the same direction may be performed a plurality of times by different methods, and for example, a precise alignment may be performed after performing a rough alignment. Further, as a method of alignment, for example, a plurality of alignments obtained by segmenting a tomographic image (B scan image) using a retinal layer boundary obtained by performing segmentation processing (coarse in the Z direction) and dividing the tomographic image. Alignment using the correlation information (similarity) between the region and the reference image (precise in the X and Z directions), and using the one-dimensional projected image generated for each tomographic image (B scan image) (in the X direction). ) Alignment, etc. (in the X direction) using a two-dimensional front image. Further, it may be configured so that the alignment is roughly performed in pixel units and then the precise alignment is performed in subpixel units.

Here, during various adjustments, it is possible that the imaged object such as the retina of the eye to be inspected has not yet been successfully imaged. Therefore, since there is a large difference between the medical image input to the trained model and the medical image used as the training data, there is a possibility that a high-quality image cannot be obtained with high accuracy. Therefore, when the evaluation value such as the image quality evaluation of the tomographic image (B scan) exceeds the threshold value, the display of the high-quality moving image (continuous display of the high-quality frame) may be automatically started. Further, when the evaluation value such as the image quality evaluation of the tomographic image (B scan) exceeds the threshold value, the image quality enhancement button may be configured to be changed to a state (active state) that can be specified by the examiner.

Further, for example, a trained model for high image quality is prepared for each shooting mode having a different scan pattern, and the trained model for high image quality corresponding to the selected shooting mode is selected. May be done. Further, one trained model for high image quality obtained by learning learning data including various medical images obtained in different imaging modes may be used.

(Modification 7)
In the above-described embodiments and modifications, when various trained models are executing additional learning, it may be difficult to output (infer / predict) using the trained model itself during the execution of additional learning. is there. Therefore, it is preferable to prohibit the input of medical images other than the training data to the trained model during the execution of the additional learning. Further, another trained model that is the same as the trained model before the execution of the additional learning may be prepared as another preliminary trained model. At this time, during the execution of the additional learning, it is preferable that the input of the medical image other than the training data to the preliminary trained model can be executed. Then, after the additional learning is completed, the trained model after the execution of the additional learning is evaluated, and if there is no problem, the preliminary trained model may be replaced with the trained model after the execution of the additional learning. Also, if there is a problem, a preliminary trained model may be used.

As an evaluation of the trained model after the execution of the additional learning, for example, a trained model for classification for classifying a high-quality image obtained by the trained model for high image quality with another type of image is used. It may be used. The trained model for classification uses, for example, a plurality of images including a high-quality image and a low-quality image obtained by the trained model for high image quality as input data, and the types of these images are labeled (annotation). It may be a trained model obtained by training training data including the obtained data as correct answer data. At this time, the image type of the input data at the time of estimation (prediction) is displayed together with the information (for example, a numerical value indicating the ratio) indicating the certainty of each type of image included in the correct answer data at the time of learning. May be good. In addition to the above images, the input data of the trained model for classification includes overlay processing of a plurality of low-quality images (for example, averaging processing of a plurality of low-quality images obtained by alignment) and the like. It may include a high-quality image in which high contrast, noise reduction, etc. are performed. Further, as the evaluation of the trained model after the execution of the additional learning, for example, the trained model after the execution of the additional learning and the trained model before the execution of the additional learning (preliminary trained model) are used and the same. A plurality of high-quality images obtained from the above images may be compared, or the analysis results of the plurality of high-quality images may be compared. At this time, for example, the comparison result of the plurality of high-quality images (an example of change due to additional learning) or the comparison result of the analysis result of the plurality of high-quality images (an example of change due to additional learning) is within a predetermined range. It may be determined whether or not, and the determination result may be displayed.

In addition, the trained model obtained by learning for each imaging site may be selectively used. Specifically, the first trained model obtained by using the learning data including the first imaging region (for example, the anterior segment, the posterior segment, etc.) and the second imaging region different from the first imaging region. It is possible to prepare a second trained model obtained by using the training data including the imaged portion, and a plurality of trained models including the trained model. Then, the control unit 117 may have a selection means for selecting one of the plurality of trained models. At this time, the control unit 117 may have a control means for executing additional learning on the selected trained model. The control means searches for data in which the captured part corresponding to the selected trained model and the captured image of the captured part are paired in response to an instruction from the examiner, and the data obtained by the search is learned data. Can be executed as additional learning for the selected trained model. The imaging site corresponding to the selected trained model may be acquired from the information in the header of the data or manually input by the examiner. Further, the data search may be performed from a server of an external facility such as a hospital or a research institute via a network, for example. As a result, additional learning can be efficiently performed for each imaged part by using the photographed image of the imaged part corresponding to the trained model.

The selection means and the control means may be composed of software modules executed by a processor such as a CPU or MPU of the control unit 117. Further, the selection means and the control means may be composed of a circuit that performs a specific function such as an ASIC, an independent device, or the like.

In addition, when acquiring learning data for additional learning from a server of an external facility such as a hospital or research institute via a network, it is necessary to reduce reliability deterioration due to falsification or system trouble during additional learning. Is useful. Therefore, the validity of the learning data for additional learning may be detected by confirming the consistency by digital signature or hashing. As a result, the learning data for additional learning can be protected. At this time, if the validity of the training data for additional learning cannot be detected as a result of confirming the consistency by digital signature or hashing, a warning to that effect is given and additional learning is performed using the training data. Make it not exist. The server may be in any form, for example, a cloud server, a fog server, an edge server, or the like, regardless of its installation location.

Further, the data protection by confirming the consistency as described above can be applied not only to the learning data for additional learning but also to the data including the medical image. Further, the image management system may be configured so that the transaction of data including medical images between servers of a plurality of facilities is managed by a distributed network. Further, the image management system may be configured to connect a plurality of blocks in which the transaction history and the hash value of the previous block are recorded together in a time series. As a technique for confirming consistency, even if a cipher that is difficult to calculate even using a quantum computer such as a quantum gate method (for example, lattice-based cryptography, quantum cryptography by quantum key distribution, etc.) is used. Good. Here, the image management system may be a device and a system that receives and stores an image taken by a photographing device or an image processed image. In addition, the image management system may transmit an image in response to a request from the connected device, perform image processing on the saved image, or request an image processing request from another device. it can. The image management system can include, for example, an image storage communication system (PACS). In addition, the image management system includes a database that can store various information such as subject information and shooting time associated with the received image. In addition, the image management system is connected to a network and can send and receive images, convert images, and send and receive various information associated with saved images in response to requests from other devices. ..

When additional learning is performed on various trained models, GPU can be used to perform high-speed processing. Since the GPU can perform efficient calculations by processing more data in parallel, it is possible to perform processing on the GPU when learning is performed multiple times using a learning model such as deep learning. It is valid. The additional learning process may be performed by the GPU, CPU, or the like in collaboration with each other.

(Modification 8)
In the various embodiments and modifications described above, the instruction from the examiner may be an instruction by voice or the like in addition to a manual instruction (for example, an instruction using a user interface or the like). At this time, for example, a machine learning model including a voice recognition model (speech recognition engine, trained model for voice recognition) obtained by machine learning may be used. Further, the manual instruction may be an instruction by character input or the like using a keyboard, a touch panel, or the like. At this time, for example, a machine learning model including a character recognition model (character recognition engine, trained model for character recognition) obtained by machine learning may be used. Further, the instruction from the examiner may be an instruction by a gesture or the like. At this time, a machine learning model including a gesture recognition model (gesture recognition engine, learned model for gesture recognition) obtained by machine learning may be used.

Further, the instruction from the examiner may be the line-of-sight detection result of the examiner on the display screen of the display unit 116 or the like. The line-of-sight detection result may be, for example, a pupil detection result using a moving image of the examiner obtained by photographing from the periphery of the display screen on the display unit 116. At this time, the object recognition engine as described above may be used for the pupil detection from the moving image. Further, the instruction from the examiner may be an instruction by an electroencephalogram, a weak electric signal flowing through the body, or the like.

In such a case, for example, as the training data, character data or voice data (waver data) indicating instructions for displaying the results of the processing of the various trained models as described above are used as input data, and various trained data have been trained. It may be learning data in which the execution instruction for actually displaying the result or the like obtained by the processing of the model on the display unit 116 is the correct answer data. Further, the learning data may be learning data in which, for example, an execution command for whether or not to automatically set the shooting parameters and an execution command for changing the button for the command to the active state are correct data. Good. The learning data may be any data as long as the instruction content and the execution instruction content indicated by the character data, the voice data, or the like correspond to each other. Further, the voice data may be converted into character data by using an acoustic model, a language model, or the like. Further, the waveform data obtained by the plurality of microphones may be used to perform a process of reducing the noise data superimposed on the voice data. Further, the instruction by characters or voice and the instruction by a mouse or a touch panel may be configured to be selectable according to the instruction from the examiner. Further, the on / off of the instruction by characters or voice may be selectably configured according to the instruction from the examiner.

Here, machine learning includes deep learning as described above, and for at least a part of a multi-layer neural network, for example, a recurrent neural network (RNN) can be used. Here, as an example of the machine learning model according to this modified example, RNN, which is a neural network that handles time series information, will be described with reference to FIGS. 10 (a) and 10 (b). Further, a Long short-term memory (hereinafter referred to as LSTM), which is a kind of RNN, will be described with reference to FIGS. 11 (a) and 11 (b).

FIG. 10A shows the structure of the RNN, which is a machine learning model. The RNN 3520 has a loop structure in the network, ^{inputs data x t} 3510 at time t, and outputs ^{data h t 3530.} Since the RNN3520 has a loop function in the network, the current state can be inherited to the next state, so that time-series information can be handled. FIG. 10B shows an example of input / output of the parameter vector at time t. The data x ^t 3510 contains N pieces of data (Params1 to ParamsN). Further, the data h ^t 3530 output from the RNN 3520 includes N data (Params1 to ParamsN) corresponding to the input data.

However, since RNN cannot handle long-term information at the time of error back propagation, LSTM may be used. The LSTM can learn long-term information by including a forgetting gate, an input gate, and an output gate. Here, FIG. 11A shows the structure of the LSTM. In the RSTM3540, the information that the network takes over at the next time t is the internal state c ^{t-1 of the} network called the cell and the output data h ^t-1 . The lowercase letters (c, h, x) in the figure represent vectors.

Next, FIG. 11B shows the details of RSTM3540. In FIG. 11B, FG indicates a forgetting gate network, IG indicates an input gate network, and OG indicates an output gate network, each of which is a sigmoid layer. Therefore, a vector in which each element has a value of 0 to 1 is output. The oblivion gate network FG determines how much past information is retained, and the input gate network IG determines which value to update. The CU is a cell update candidate network and is an activation function tanh layer. This creates a vector of new candidate values to be added to the cell. The output gate network OG selects the cell candidate element and selects how much information to convey at the next time.

Since the above-mentioned LSTM model is a basic form, it is not limited to the network shown here. You may change the coupling between the networks. QRNN (Quasi Recurrent Neural Network) may be used instead of RSTM. Further, the machine learning model is not limited to the neural network, and boosting, a support vector machine, or the like may be used. Further, when the instruction from the examiner is input by characters, voice, or the like, a technique related to natural language processing (for example, Sequence to Sequence) may be applied. At this time, as a technique related to natural language processing, for example, a model that is output for each input sentence may be applied. Further, the various trained models described above are not limited to the instructions from the examiner, and may be applied to the output to the examiner. Further, a dialogue engine (dialogue model, trained model for dialogue) that responds to the examiner with an output in characters or voice may be applied.

Further, as a technique related to natural language processing, a trained model obtained by pre-learning document data by unsupervised learning may be used. Further, as a technique related to natural language processing, a trained model obtained by further transfer learning (or fine tuning) of a trained model obtained by pre-learning may be used. Further, as a technique related to natural language processing, for example, BERT (Bidirectional Encoder Representations from Transformers) may be applied. Further, as a technique related to natural language processing, a model capable of extracting (expressing) a context (feature amount) by itself by predicting a specific word in a sentence from both the left and right contexts may be applied. Further, as a technique related to natural language processing, a model capable of determining the relationship (continuity) of two sequences (sentences) in the input time series data may be applied. Further, as a technique related to natural language processing, a Transformer Encoder is used for the hidden layer, and a model in which a vector sequence is input and output may be applied.

Here, the instruction from the examiner to which this modification is applicable is for changing the display of various images and analysis results as described in the various embodiments and modifications described above, and for generating an En-Face image. Selection of depth range, selection of whether to use as training data for additional learning, selection of trained model, output (display, transmission, etc.) and storage of results obtained using various trained models, etc. Any instruction may be used as long as it is at least one instruction. Further, the instruction from the examiner to which this modification is applicable may be an instruction before photography as well as an instruction after photography. For example, an instruction regarding various adjustments and an instruction regarding setting of various imaging conditions. , It may be an instruction regarding the start of shooting. Further, the instruction from the examiner to which this modification is applicable may be an instruction regarding a change (screen transition) of the display screen.

The machine learning model may be a machine learning model that combines a machine learning model related to images such as CNN and a machine learning model related to time series data such as RNN. In such a machine learning model, for example, it is possible to learn the relationship between the feature amount related to the image and the feature amount related to the time series data. When the input layer side of the machine learning model is CNN and the output layer side is RNN, for example, a medical image is used as input data, and sentences related to the medical image (for example, presence / absence of lesion, type of lesion, recommendation of next examination). Etc.) may be used as output data for training. As a result, for example, medical information related to medical images is automatically explained in sentences, so that even an examiner with little medical experience can easily grasp medical information related to medical images. When the input layer side of the machine learning model is RNN and the output layer side is CNN, for example, texts related to medical treatment such as lesions, findings, and diagnoses are used as input data, and medical images corresponding to the texts related to the medical treatment are output. Learning may be performed using the learning data as data. This makes it possible, for example, to easily search for medical images related to the case that the examiner wants to confirm.

In addition, even if a machine translation engine (machine translation model, trained model for machine translation) that machine translates sentences such as characters and voices into any language is used for instructions from the examiner and output to the examiner. Good. In addition, any language may be configured to be selectable according to an instruction from the examiner. Further, any language may be configured to be automatically selectable by using a trained model that automatically recognizes the type of language. Further, the automatically selected language type may be configured to be correctable according to an instruction from the examiner. For example, the above-mentioned techniques related to natural language processing (for example, Sequence to Sequence) may be applied to the machine translation engine. For example, after the sentence input to the machine translation engine is machine-translated, the machine-translated sentence may be input to the character recognition engine or the like. Further, for example, the sentences output from the various trained models described above may be input to the machine translation engine, and the sentences output from the machine translation engine may be output.

Moreover, the various trained models described above may be used in combination. For example, the characters corresponding to the instructions from the examiner are input to the character recognition engine, and the voice obtained from the input characters is input to another type of machine learning engine (for example, a machine translation engine). May be done. Further, for example, characters output from other types of machine learning engines may be input to the character recognition engine, and the voice obtained from the input characters may be output. Further, for example, the voice corresponding to the instruction from the examiner is input to the voice recognition engine, and the characters obtained from the input voice are input to another type of machine learning engine (for example, a machine translation engine). It may be configured in. Further, for example, the voice output from another type of machine learning engine may be input to the voice recognition engine, and the characters obtained from the input voice may be displayed on the display unit 116. At this time, it may be configured so that the output to the examiner can be selected from the output by characters and the output by voice according to the instruction from the examiner. Further, it may be configured so that the input from the examiner can be selected from the input by characters and the input by voice according to the instruction from the examiner. In addition, the various configurations described above may be adopted by selection according to an instruction from the examiner.

(Modification 9)
A label image, a high-quality image, or the like related to the image acquired by the main shooting may be stored in the storage unit 114 in response to an instruction from the operator. At this time, for example, after an instruction from the operator for saving a high-quality image, when registering the file name, as a recommended file name, any part of the file name (for example, the first part, or In the last part), a file name containing information (for example, characters) indicating that the image is generated by processing using a trained model for high image quality (high image quality processing) is given by the operator. It may be displayed in an editable state according to the instruction. Similarly, for the label image and the like, a file name including information that is an image generated by processing using the trained model may be displayed.

Further, when displaying a high-quality image on the display unit 116 on various display screens such as a report screen, the displayed image is a high-quality image generated by processing using a high-quality model. The display shown may be displayed together with a high-quality image. In this case, the operator can easily identify from the display that the displayed high-quality image is not the image itself acquired by shooting, thus reducing erroneous diagnosis and improving diagnostic efficiency. be able to. The display indicating that the image is a high-quality image generated by the process using the high-quality model is any mode as long as the input image and the high-quality image generated by the process can be distinguished from each other. It may be the one. Further, it is shown that not only the processing using the high image quality model but also the processing using various trained models as described above is the result generated by the processing using the trained model of that kind. The display may be displayed with the result. For example, when displaying the analysis result of the segmentation result using the trained model for image segmentation processing, the display indicating that the analysis result is based on the result using the trained model for image segmentation is analyzed. It may be displayed with the result.

At this time, the display screen such as the report screen may be saved in the storage unit 114 as image data in response to an instruction from the operator. For example, the report screen may be saved in the storage unit 114 as one image in which a high-quality image or the like and a display indicating that these images are images generated by processing using the trained model are arranged side by side. ..

In addition, regarding the display indicating that the image is a high-quality image generated by the processing using the high-quality model, the display unit shows what kind of learning data the high-quality model has learned. It may be displayed at 116. The display may include an explanation of the types of the input data and the correct answer data of the learning data, and an arbitrary display regarding the correct answer data such as the imaging part included in the input data and the correct answer data. For processing using the various trained models described above, such as image segmentation processing, the display unit 116 displays a display indicating what kind of training data the trained model of that type was trained by. It may be displayed.

In addition, information (for example, characters) indicating that the image is generated by processing using the trained model may be displayed or stored in a state of being superimposed on the image or the like. At this time, the portion to be superimposed on the image may be any region (for example, the edge of the image) that does not overlap with the region in which the region of interest to be photographed is displayed. Further, the non-overlapping areas may be determined and superimposed on the determined areas. It should be noted that not only the processing using the high image quality model but also the image obtained by the processing using the various trained models described above such as the image segmentation processing may be processed in the same manner.

In addition, if the initial display screen of the report screen is set by default so that the high image quality processing button or the like is in the active state (high image quality processing is on), it is set to high according to the instruction from the examiner. The report image corresponding to the report screen including the image quality image and the like may be configured to be transmitted to the server. In addition, if the button is set to the active state by default, at the end of the inspection (for example, when the shooting confirmation screen or preview screen is changed to the report screen according to the instruction from the inspector). In addition, the report image corresponding to the report screen including the high-quality image and the like may be configured to be (automatically) transmitted to the server. At this time, various settings in the default settings (for example, the depth range for generating the En-Face image on the initial display screen of the report screen, the presence / absence of superimposition of the analysis map, whether or not the image is high quality, and the display screen for follow-up observation. The report image generated based on (settings related to at least one such as whether or not) may be configured to be transmitted to the server. It should be noted that the same processing may be performed when the button represents the switching of the image segmentation processing.

(Modification example 10)
In the above-described embodiments and modifications, among the various trained models described above, an image obtained by the first type of trained model (for example, an image showing an analysis result such as a high-quality image or an analysis map). An image showing a predetermined region detection result, an image showing a segmentation result) may be input to a second type of trained model different from the first type. At this time, the result of processing the second type of trained model (for example, estimation result, analysis result, diagnosis result, predetermined region detection result, segmentation result) may be generated.

Further, among the various trained models as described above, the results obtained by processing the first type of trained model (for example, estimation result, analysis result, diagnosis result, predetermined region detection result, segmentation result) are used. From the image input to the trained model of the first type, an image to be input to the trained model of the second type different from the first type may be generated. At this time, the generated image is likely to be an image suitable as an image to be processed using the second type of trained model. Therefore, an image obtained by inputting the generated image into the second type of trained model (for example, a high-quality image, an image showing an analysis result such as an analysis map, an image showing a predetermined area detection result, a segmentation result). The accuracy of the image) can be improved.

By inputting a common image into the first type of trained model and the second type of trained model, each processing result using these trained models can be generated (or displayed). It may be configured to run. At this time, for example, in response to an instruction from the examiner, the generation (or display) of each processing result using these trained models may be collectively (interlockedly) executed. In addition, the type of image to be input (for example, high-quality image, object recognition result, segmentation result, similar case image), and the type of processing result to be generated (or displayed) (for example, high-quality image, estimation result, diagnosis result, analysis). The result, the object recognition result, the segmentation result, the similar case image), the input type and the output type (for example, characters, voice, language) and the like may be selectably configured according to the instruction from the examiner. Further, the input type may be configured to be automatically selectable by using a trained model that automatically recognizes the input type. Further, the output type may be configured to be automatically selectable so as to correspond to the input type (for example, the same type). Further, the automatically selected type may be configured to be modifiable according to an instruction from the examiner. At this time, at least one trained model may be configured to be selected according to the selected type. At this time, when a plurality of trained models are selected, how to combine the plurality of trained models (for example, the order in which data is input) may be determined according to the selected type. Note that, for example, the type of image to be input and the type of processing result to be generated (or displayed) may be configured to be differently selectable, or if they are the same, they may be selected differently. It may be configured to output prompting information to the examiner. Also, each trained model may be executed anywhere. For example, some of the plurality of trained models may be configured to be used by a cloud server, and others may be configured to be used by another server such as a fog server or an edge server. In addition, when the network in the facility, the site including the facility, the area including a plurality of facilities, etc. is configured to enable wireless communication, for example, it is assigned only to the facility, the site, the area, etc. The reliability of the network may be improved by configuring so as to use radio waves in a dedicated wavelength band. Further, the network may be configured by wireless communication capable of high speed, large capacity, low delay, and a large number of simultaneous connections. With these, for example, surgery such as vitreous body, cataract, glaucoma, corneal refraction correction, external eye, and treatment such as laser photocoagulation can be supported in real time even if it is remote. At this time, for example, information obtained by using at least one of various trained models by a fog server, an edge server, or the like that wirelessly receives at least one of various medical images obtained by these devices related to surgery or treatment. May be configured to wirelessly transmit to a device for surgery or treatment. Further, for example, the information wirelessly received by the device related to surgery or treatment may be the amount of movement (vector) of the optical system or optical member as described above. In this case, the device related to surgery or treatment is automatically controlled. It may be configured to be. Further, for example, for the purpose of supporting the operation by the examiner, it may be configured as automatic control (semi-automatic control) with the permission of the examiner.

Further, a similar case image search using an external database stored in a server or the like may be performed using the analysis result, the diagnosis result, etc. obtained by the processing of the learned model as described above as a search key. Further, a similar case image search using an external database stored in a server or the like may be performed using an object recognition result, a segmentation result, or the like obtained by processing various learned models as described above as a search key. If a plurality of medical images stored in the database are already managed by machine learning or the like with the feature amounts of the plurality of medical images attached as incidental information, the medical images themselves may be used. A similar case image search engine (similar case image search model, learned model for similar case image search) as a search key may be used. For example, the control unit 117 uses a trained model for searching for similar case images (different from the trained model for high image quality) to search various medical images for similar case images related to the medical images. It can be carried out. Further, for example, the display control unit 121 can display the similar case image obtained from various medical images by using the learned model for searching the similar case image on the display unit 116. At this time, the similar case image is, for example, an image having a feature amount similar to the feature amount of the medical image input to the trained model. Further, the similar case image is, for example, an image having a feature amount similar to the feature amount of the partial area such as the abnormal part when the medical image input to the trained model includes a partial area such as an abnormal part. .. Therefore, for example, not only can learning for accurately searching similar case images be performed, but also when an abnormal part is included in the medical image, the examiner efficiently diagnoses the abnormal part. be able to. Further, a plurality of similar case images may be searched, and a plurality of similar case images may be displayed so that the order in which the feature amounts are similar can be identified. In addition, a trained model for searching similar case images is additionally learned by using learning data including an image selected according to an instruction from an examiner and a feature amount of the image among a plurality of similar case images. It may be configured to be.

Further, the training data of various trained models is not limited to the data obtained by using the ophthalmic apparatus itself that actually performs the imaging, and the data obtained by using the same type of ophthalmic apparatus or the same type according to the desired configuration. It may be data obtained by using an ophthalmic apparatus or the like.

In addition, various trained models according to the above-described embodiment and modification can be provided in the control unit 117. The trained model may be composed of, for example, a CPU, a software module executed by a processor such as an MPU, GPU, or FPGA, or a circuit or the like that performs a specific function such as an ASIC. Further, these learned models may be provided in a device of another server connected to the control unit 117 or the like. In this case, the control unit 117 can use the trained model by connecting to a server or the like provided with the trained model via an arbitrary network such as the Internet. Here, the server provided with the trained model may be, for example, a cloud server, a fog server, an edge server, or the like. In addition, when the network in the facility, the site including the facility, the area including a plurality of facilities, etc. is configured to enable wireless communication, for example, it is assigned only to the facility, the site, the area, etc. The reliability of the network may be improved by configuring so as to use radio waves in a dedicated wavelength band. Further, the network may be configured by wireless communication capable of high speed, large capacity, low delay, and a large number of simultaneous connections.

(Modification 11)
The medical image processed by the control unit 117 according to the various embodiments and modifications described above includes images acquired using any modality (imaging device, imaging method). The medical image to be processed may include a medical image acquired by an arbitrary imaging device or the like, or an image created by a medical image processing device or a medical image processing method.

Further, the medical image to be processed is an image of a predetermined part of the subject (subject), and the image of the predetermined part includes at least a part of the predetermined part of the subject. In addition, the medical image may include other parts of the subject. Further, the medical image may be a still image or a moving image, and may be a black-and-white image or a color image. Further, the medical image may be an image showing the structure (morphology) of a predetermined part or an image showing the function thereof. The image showing the function includes, for example, an OCTA image, a Doppler OCT image, an fMRI image, and an image showing blood flow dynamics (blood flow volume, blood flow velocity, etc.) such as an ultrasonic Doppler image. The predetermined part of the subject may be determined according to the subject to be imaged, and the human eye (eye to be examined), brain, lung, intestine, heart, pancreas, kidney, liver and other organs, head, chest, etc. Includes any part such as legs and arms. In particular, in the various embodiments and modifications described above, the medical image relating to the eye to be inspected was used for the estimation process. In this regard, the subject relating to the medical image used for the estimation process in the various embodiments and modifications described above is not limited to the subject to be examined, and may be a subject having symmetry in the left-right direction, the up-down direction, or the left-right up-down direction. It may be another organ such as the lung. However, the subjects related to the various embodiments and modifications described above are not limited to the subjects having symmetry. When the subject is an organ such as a lung, the imaging device may have, for example, an endoscope or the like.

Further, the medical image may be a tomographic image of the subject or a frontal image. The frontal image is, for example, an SLO image of the fundus or anterior segment of the eye, a fundus image photographed by fluorescence, and data acquired by OCT (three-dimensional OCT data) in at least a part of the range in the depth direction of the imaged object. Includes En-Face images generated using. The En-Face image is an OCTA En-Face image (motion contrast front image) generated by using at least a part of the data in the depth direction of the shooting target for the three-dimensional OCTA data (three-dimensional motion contrast data). ) May be. Further, three-dimensional OCT data and three-dimensional motion contrast data are examples of three-dimensional medical image data.

Here, the motion contrast data is data indicating a change between a plurality of volume data obtained by controlling the measurement light to be scanned a plurality of times in the same region (same position) of the eye to be inspected. At this time, the volume data is composed of a plurality of tomographic images obtained at different positions. Then, motion contrast data can be obtained as volume data by obtaining data showing changes between a plurality of tomographic images obtained at substantially the same position at different positions. The motion contrast frontal image is also referred to as an OCTA frontal image (OCTA En-Face image) relating to OCTA angiography (OCTA) for measuring the movement of blood flow, and the motion contrast data is also referred to as OCTA data. The motion contrast data can be obtained, for example, as a decorrelation value, a variance value, or a maximum value divided by a minimum value (maximum value / minimum value) between two tomographic images or corresponding interference signals. , It may be obtained by any known method. At this time, the two tomographic images can be obtained, for example, by controlling the measurement light to be scanned a plurality of times in the same region (same position) of the eye to be inspected. When controlling the scanning means so that the measurement light is scanned a plurality of times at substantially the same position, the time interval (time interval) between one scan (one B scan) and the next scan (next B scan). ) May be modified (determined). Thereby, for example, even if the blood flow velocity may differ depending on the state of the blood vessel, the blood vessel region can be visualized with high accuracy. At this time, for example, the time interval may be changed so as to be instructed by the examiner. Further, for example, one of the motion contrast images may be selectably configured from a plurality of motion contrast images corresponding to a plurality of preset time intervals according to an instruction from the examiner. Further, for example, the time interval when the motion contrast data is acquired may be associated with the motion contrast data so that the motion contrast data can be stored in the storage unit 114. Further, for example, the display control unit 121 may display the time interval when the motion contrast data is acquired and the motion contrast image corresponding to the motion contrast data on the display unit 116. Further, for example, the time interval may be automatically determined, or at least one candidate for the time interval may be determined. At this time, for example, using a machine learning model, the time interval may be determined (output) from the motion contrast image. In such a machine learning model, for example, a plurality of motion contrast images corresponding to a plurality of time intervals are used as input data, and the difference from the plurality of time intervals to the time interval when a desired motion contrast image is acquired is correctly answered. Learning as data It can be obtained by learning the data.

The En-Face image is, for example, a front image generated by projecting data in the range between two layer boundaries in the XY directions. At this time, the front image is at least a part of the depth range of the volume data (three-dimensional tomographic image) obtained by using optical interference, and is the data corresponding to the depth range determined based on the two reference planes. Is projected or integrated on a two-dimensional plane. The En-Face image is a frontal image generated by projecting the data corresponding to the depth range determined based on the detected retinal layer among the volume data onto a two-dimensional plane. As a method of projecting data corresponding to a depth range determined based on two reference planes on a two-dimensional plane, for example, a representative value of data within the depth range is set as a pixel value on the two-dimensional plane. Techniques can be used. Here, the representative value can include a value such as an average value, a median value, or a maximum value of pixel values within a range in the depth direction of a region surrounded by two reference planes. Further, the depth range related to the En-Face image is, for example, a range including a predetermined number of pixels in a deeper direction or a shallower direction with respect to one of the two layer boundaries relating to the detected retinal layer. May be good. Further, the depth range related to the En-Face image may be, for example, a range changed (offset) according to an operator's instruction from a range between two layer boundaries related to the detected retinal layer. Good.

The imaging device is a device for capturing an image used for diagnosis. The photographing device detects, for example, a device that obtains an image of a predetermined part by irradiating a predetermined part of the subject with radiation such as light or X-rays, electromagnetic waves, ultrasonic waves, or the like, or radiation emitted from the subject. This includes a device for obtaining an image of a predetermined part. More specifically, the imaging devices according to the various embodiments and modifications described above include at least an X-ray imaging device, a CT device, an MRI device, a PET device, a SPECT device, an SLO device, an OCT device, an OCTA device, and a fundus. Includes cameras, endoscopes, etc. It should be noted that the configurations according to each of the above-described embodiments and modifications can be applied to these imaging devices. In this case, the movement of the subject corresponding to the above-mentioned movement of the eye to be predicted may be, for example, the movement of the face or body, the movement of the heart (heartbeat), or the like.

The OCT apparatus may include a time domain OCT (TD-OCT) apparatus and a Fourier domain OCT (FD-OCT) apparatus. Further, the Fourier domain OCT apparatus may include a spectral domain OCT (SD-OCT) apparatus and a wavelength sweep type OCT (SS-OCT) apparatus. Further, the OCT apparatus may include a Line-OCT apparatus (or SS-Line-OCT apparatus) using line light. Further, the OCT apparatus may include a Full Field-OCT apparatus (or SS-Full Field-OCT apparatus) using area light. In addition, the OCT apparatus may include a Doppler-OCT apparatus. Further, the SLO device and the OCT device may include a wave surface compensation SLO (AO-SLO) device using a wave surface compensation optical system, a wave surface compensation OCT (AO-OCT) device, and the like. Further, the SLO device and the OCT device may include a polarized SLO (PS-SLO) device, a polarized OCT (PS-OCT) device, and the like for visualizing information on polarization phase difference and polarization elimination. Further, the SLO device and the OCT device may include a pathological microscope SLO device, a pathological microscope OCT device, and the like. Further, the SLO device and the OCT device may include a handheld type SLO device, a handheld type OCT device, and the like. Further, the SLO device and the OCT device may include a catheter SLO device, a catheter OCT device and the like. Further, the SLO device and the OCT device may include a head-mounted SLO device, a head-mounted OCT device, and the like. Further, the SLO device and the OCT device may include a binocular type SLO device, a binocular type OCT device, and the like. Further, the SLO device and the OCT device may have a structure in which the shooting angle of view can be changed by a configuration capable of optical scaling. Further, the SLO device can acquire a color image or a fluorescence image by using each of the RGB light sources and having a configuration in which one light receiving element receives time-divisionally or a plurality of light receiving elements simultaneously receive light. May be good.

Further, in the above-described embodiment and modification, the control unit 117 may be configured as a part of the OCT device, or the control unit 117 may be configured as a separate body from the OCT device. In this case, the control unit 117 may be connected to an imaging device such as an OCT device via the Internet or the like. Further, the configuration of the OCT apparatus is not limited to the above configuration, and a part of the configuration included in the OCT apparatus may be a configuration in which, for example, an SLO photographing unit or the like is separate from the OCT apparatus.

In the trained models for voice recognition, character recognition, gesture recognition, etc. related to the above-mentioned modification, learning is performed using time-series data, so that continuous time-series data values to be input are used. It is considered that the slope between them is extracted as a part of the feature amount and used for the estimation process. It is expected that such a trained model can perform accurate estimation by using the influence of a specific numerical value over time in the estimation process. In addition, even in the trained models for estimation processing, high image quality, segmentation processing, image analysis, and diagnosis result generation related to the above-described embodiments and modifications, the magnitude and brightness of the tomographic image brightness value and the bright part It can be considered that the order, inclination, position, distribution, continuity, etc. of the dark part are extracted as a part of the feature amount and used for the estimation process.

[Other Embodiments]
Further, the technique disclosed in the present specification can be implemented as, for example, a system, an apparatus, a method, a program, a recording medium (storage medium), or the like. Specifically, it may be applied to a system composed of a plurality of devices (for example, a host computer, an interface device, an imaging device, a web application, etc.), or it may be applied to a device composed of one device. good.

Further, it goes without saying that the object of the technique disclosed in the present specification is achieved by doing the following. That is, a recording medium (or storage medium) in which a software program code (computer program) that realizes the functions of the above-described embodiment is recorded is supplied to the system or device. Needless to say, the storage medium is a computer-readable storage medium. Then, the computer (or CPU or MPU) of the system or device reads and executes the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the function of the above-described embodiment, and the recording medium on which the program code is recorded constitutes the technique disclosed in the present specification.

Further, the technique disclosed in the present specification supplies a program that realizes one or more functions of the above-described embodiments and modifications to a system or device via a network or storage medium, and the computer of the system or device supplies the program. It can also be realized by the process of reading and executing the program. A computer may have one or more processors or circuits and may include multiple separate computers or a network of separate processors or circuits to read and execute computer executable instructions.

The processor or circuit may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or a field programmable gateway (FPGA). Also, the processor or circuit may include a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

Claims (28)

An estimation means for estimating the risk of developing a disease by a subject using a learned model that learned the relationship between the feature amount acquired from the fundus image and the risk of developing the disease evaluated from the feature amount.
An information processing device including a correction means for correcting the risk of developing the estimated disease based on the biological information of the subject. The estimation means evaluates the relationship between the feature amount acquired from the fundus image and the risk of developing a first disease evaluated from the feature amount, and the feature amount acquired from the fundus image and the feature amount. According to claim 1, the risk of the subject developing the first disease and the second disease is estimated by using a learned model that has learned the relationship with the risk of developing the second disease. The information processing device described. The estimation means is acquired from the first trained model that learned the relationship between the feature amount acquired from the fundus image and the risk of developing the first disease evaluated from the feature amount, and the fundus image. Using a second trained model that learned the relationship between the feature amount and the risk of developing a second disease evaluated from the feature amount, the subject can use the first disease and the second disease. The information processing apparatus according to claim 1, wherein the risk of developing the disease is estimated. Further equipped with an acquisition means for acquiring a fundus image of the subject,
Claims 1 to 3 estimate the probability that the subject develops a disease by inputting the feature amount obtained from the fundus image of the subject into the trained model. The information processing apparatus according to any one of the above items. The trained model learns the relationship between the morphology of blood vessels obtained from the fundus image and the risk of developing cardiovascular disease evaluated from the morphology of blood vessels.
The correction means according to any one of claims 1 to 4, wherein the correction means is corrected by using at least one biological information of the blood pressure, BMI index, age, sex, medical history or smoking habit of the subject. Information processing device. The morphology of the blood vessel comprises at least one of a feature amount indicating arterial diameter, vein diameter, ratio of arterial diameter to vein diameter, branch angle of blood vessel, asymmetry of the branch, arterial vein stenosis or twist of blood vessel. The information processing apparatus according to claim 5. An acquisition means for acquiring a fundus image obtained by imaging the fundus of a subject, and
The risk that the subject develops a disease by inputting the acquired fundus image into a trained model in which the relationship between the fundus image and the risk of developing a disease evaluated from the fundus image is deeply learned. And the estimation method to estimate
An information processing device including a correction means for correcting the risk of developing the estimated disease based on the biological information of the subject. The information processing apparatus according to any one of claims 1 to 7, wherein the correction means uses a predetermined weighting coefficient determined for each biological information to correct the risk of developing the estimated disease. .. The information processing apparatus according to any one of claims 1 to 8, further comprising a display control means for displaying the risk of developing the corrected disease on a display unit. The information processing device according to claim 9, wherein the display control means displays the risk of developing the corrected disease on the display unit in a state of being classified into a plurality of classes. The information processing device according to claim 9 or 10, wherein the display control means displays the risk of developing the corrected disease on the display unit in a state of being parallel to the fundus image of the subject. The information processing device according to any one of claims 9 to 11, wherein the display control means displays a graph based on the probability corresponding to the risk of developing the corrected disease on the display unit. The information processing apparatus according to any one of claims 9 to 12, wherein the display control means displays a fundus image of the subject on the display unit in a state where a portion having a high correlation with the disease is emphasized. .. The information according to any one of claims 9 to 13, wherein the display control means displays a medical institution recommended according to the risk of developing the corrected disease and the type of the disease on the display unit. Processing equipment. When the appointment of the recommended medical institution is completed, the fundus image used for estimating the risk of developing the disease, the biological information used for correcting the risk of developing the estimated disease, and the biometric information. The information processing apparatus according to claim 14, wherein the risk of developing the estimated disease is transmitted to the recommended medical institution via the system of the recommended medical institution. Claims configured to be diagnosed or consulted by a physician at the recommended medical institution via a video-communicable system, depending on the risk of developing the corrected disease and the type of the disease. Item 14. The information processing apparatus according to Item 14 or 15. Estimating the risk of developing a disease by a subject using a learned model that learns the relationship between the features acquired from the fundus image and the biological information acquired by the test device and the risk of developing the disease. Means and
A display control means for displaying the estimated risk of developing the disease on the display unit, and
Information processing device equipped with. The display control means is an analysis result generated by using a learned model for generating an analysis result obtained by learning a fundus image, and is an analysis related to the fundus image used for estimating the risk of developing the disease. The information processing apparatus according to any one of claims 9 to 17, wherein the result is displayed on the display unit. The display control means is a diagnostic result generated by using a learned model for generating a diagnostic result obtained by learning the fundus image, and is a diagnosis related to the fundus image used for estimating the risk of developing the disease. The information processing apparatus according to any one of claims 9 to 18, wherein the result is displayed on the display unit. The display control means abnormally displays information regarding the difference between the image generated by using the hostile generation network or autoencoder into which the fundus image is input and the fundus image input to the hostile generation network or autoencoder. The information processing apparatus according to any one of claims 9 to 19, which is displayed on the display unit as information about a part. The display control means is a similar case image searched by using a learned model for searching a similar case image obtained by learning a fundus image, and is a fundus image used for estimating the risk of developing the disease. The information processing apparatus according to any one of claims 9 to 20, wherein a similar case image relating to the above is displayed on the display unit. The display control means is an object detection result or a segmentation result generated by using a learned model for object recognition or a learned model for segmentation obtained by learning a fundus image, and is a risk of developing the disease. The information processing apparatus according to any one of claims 9 to 21, wherein the object detection result or the segmentation result related to the fundus image used for the estimation is displayed on the display unit. The instruction from the examiner regarding the estimation of the risk of developing the disease uses at least one trained model of a trained model for character recognition, a trained model for voice recognition, and a trained model for gesture recognition. The information processing apparatus according to any one of claims 1 to 22, which is the information obtained. An ophthalmic device that captures a fundus image of the subject,
An inspection device that inspects the subject and acquires biological information,
The information processing apparatus according to any one of claims 1 to 23,
Information processing system equipped with. An estimation process for estimating the risk of developing a disease by a subject using a learned model that learned the relationship between the feature amount acquired from the fundus image and the risk of developing the disease evaluated from the feature amount.
A correction step for correcting the risk of developing the estimated disease based on the biological information of the subject, and a correction step.
Information processing methods including. Estimating the risk of developing a disease by a subject using a learned model that learns the relationship between the features acquired from the fundus image and the biological information acquired by the test device and the risk of developing the disease. Process and
A display control step for displaying the estimated risk of developing the disease on the display unit, and
Information processing methods including. The acquisition process of acquiring a fundus image of the subject's fundus,
By inputting the acquired fundus image into a trained model that has learned the relationship between the fundus image and the risk of developing the disease evaluated from the fundus image, the subject develops the disease. The estimation process for estimating risk and
A correction step for correcting the risk of developing the estimated disease based on the biological information of the subject, and a correction step.
Information processing methods including. A program for executing each means of the information processing apparatus according to any one of claims 1 to 23.