TW202222245A - Examination method, machine learning execution method, examination device, and machine learning execution method - Google Patents

Abstract

This examination method acquires image data for inference that represents an image for inference on which an eye of a subject for inference is drawn; inputs the image data for inference to a machine learning program learnt using teacher data in which image data for learning that represents an image for learning on which an eye of a subject for learning is drawn is given as a problem and condition data for learning that indicates the condition of the eye of the subject for learning is provided as an answer thereto; causes the machine learning program to estimate the condition of the eye of the subject for inference; and causes the machine learning program to output data for inference that indicates the condition of the eye of the subject for inference.

Description

Inspection method, machine learning execution method, inspection device, and machine learning execution method

The present invention relates to an inspection method, a machine learning execution method, an inspection device and a machine learning execution method. This application is based on Japanese Patent Application No. 2020-180593 filed in Japan on October 28, 2020, Japanese Patent Application No. 2020-217683 filed in Japan on December 25, 2020, and Japanese Patent Application No. 2020-217683 filed in Japan on December 25, 2020 Japanese Patent Application No. 2020-217684 for which it applied in Japan and Japanese Patent Application No. 2020-217685 for which it applied in Japan on December 25, 2020 claims priority, and the content of these is used here.

Currently, especially in developed countries, electronic devices equipped with displays, such as personal computers, smartphones, and tablets, are widely used, and the number of people who feel uncomfortable with their eyes has increased rapidly. Furthermore, due to the increase of on-line services via the Internet, the increase of companies promoting telework, etc., it is expected that the number of people who feel eye discomfort will further increase.

Examples of such relatively frequent eye discomfort due to lifestyle habits include eye fatigue, and at least one of corneal and conjunctival epithelial damage, mucus damage, instability of the tear layer, and tarsal line insufficiency. Dry eye. In order to identify the presence or absence of these, inspections using reagents, medical equipment, and the like are required. As an example of such an examination, the examination performed using the ophthalmology diagnosis support apparatus disclosed in patent document 1 is mentioned, for example.

The ophthalmic diagnosis support apparatus includes a case data storage unit, a machine learning unit, a patient data acquisition unit, a comparison determination unit, and a result display unit. The case data storage unit stores a plurality of case-based image data including a combination of ophthalmic diagnostic image data and diagnostic results. The machine learning section extracts characteristic image elements and classifies the case-based image data. The patient data acquisition unit acquires patient ophthalmological diagnostic image data of the patient. The comparison and determination unit compares the patient's ophthalmic diagnostic image data with the case-based image data, and executes the similarity determination between the data. The result display unit displays the result of the similarity determination. [Prior Art Literature] [Patent Literature]

[Patent Document 1] Japanese Patent Laid-Open No. 2020-36835

[The problem to be solved by the invention]

However, the ophthalmological diagnosis support apparatus needs to acquire ophthalmological diagnosis data using the examination equipment used in medical institutions, and thus cannot easily check the presence or absence of eye fatigue and the presence or absence of dry eye syndrome in people who perceive eye discomfort.

Therefore, an object of the present invention is to provide an inspection method, a machine learning execution method, an inspection apparatus, and a machine learning execution method capable of predicting an eye-related inspection result without actually performing the inspection. [Means of Solving Problems]

One aspect of the present invention is an inspection method that acquires image data for inference, the image data for inference representing an image for inference depicting the eyes of a subject for inference, and inputs the image data for inference To a machine learning program that uses, as a question, image data for learning representing an image for learning on which the eyes of the subject for learning are drawn, and learning that represents the state of the eyes of the subject for learning The inspection method causes the machine learning program to infer the eye state of the subject for inference, and causes the machine learning program to output a representation of the subject for inference after learning from the teacher data for which the state data is the answer Data for inferences about the state of the eye.

One aspect of the present invention is the inspection method as described above, wherein the inference data is output, and the inference data indicates that a predetermined condition is satisfied in the eyes of the inference subject depicted in the inference image. , and an image that differs from the display aspect of the area that does not satisfy the predetermined condition.

One aspect of the present invention is the above-described inspection method, wherein the machine learning program uses, as a part of the answer, learning measure data that further includes measures for improving the eye condition of the learning subject. The teacher data is learned, and the inspection method causes the machine learning program to output the inference data indicating measures to improve the eye state of the inference subject.

One aspect of the present invention is the inspection method described above, in which the inference image data representing the inference image in which at least a part of the cornea of the eye of the inference subject is drawn is acquired, and the inference image data is used. The machine learning program outputs the data for inference, and the machine learning program learns using the teacher data using the image data for learning as the problem, and the image data for learning indicates that there is a problem in the description. The image for learning is at least a part of the cornea of the eye of the subject for learning.

One aspect of the present invention is the above-described inspection method, wherein the machine learning program uses, as the inspection result data for learning, the inspection result further including an inspection result indicating the thickness of the tear oil layer of the eye of the subject for learning. The teacher data of the answer is learned, and the checking method causes the machine learning program to output the data for inference.

One aspect of the present invention is the above-described inspection method, in which the inference image data representing the inference image in which at least a part of the conjunctiva of the eye of the inference subject is drawn is acquired, and the inference image data is used. The machine learning program outputs the data for inference, and the machine learning program learns using the teacher data using the image data for learning as the problem, and the image data for learning indicates that there is a problem in the description. The image for learning of at least a part of the conjunctiva of the eye of the subject for learning.

One aspect of the present invention is the above-described inspection method, further acquiring inference hyperemia data indicating the degree of hyperemia of the eye of the inference subject, and the machine learning program further includes using the machine learning program to indicate the learning subject The hyperemia data for learning the degree of hyperemia of the eyes of the subject is included as a part of the question, and further includes the learning when an examination related to the symptoms of dry eye is performed on the eyes of the subject for learning is depicted. Learning is performed using at least one of examination image data for learning of images of the eyes of the subject and examination result data for learning showing the examination results as the teacher data for the answer, and the examination method The machine learning program is made to infer the symptoms of dry eye syndrome appearing in the eyes of the subject for inference, and symptom data representing the symptoms of dry eye syndrome appearing in the eyes of the subject for inference are output as the subject. Data for inferences.

One aspect of the present invention is that the above-mentioned examination method further acquires answer data for inference, the answer data for inference indicating the answer result of the inquiry about the subjective symptoms of the eye possessed by the subject for inference, The machine learning program uses, as a part of the question, response data for learning that further includes an answer result of an inquiry about the subjective symptoms of the eyes possessed by the subject for learning, and further includes a representation that depicts the subject. The learning test image data of images of the eyes of the learning subject when an examination related to the symptoms of dry eye is performed on the eyes of the learning subject, and the learning test showing the test results At least one of the result data is learned as the teacher data for the answer, the inspection method causes the machine learning program to infer the symptoms of dry eye appearing in the eyes of the inference subject, and Symptom data representing the symptoms of dry eye appearing in the eyes of the subject for inference are output as the data for inference.

One aspect of the present invention is the inspection method described above, and further acquires the eyelid opening data for inference indicating the maximum eyelid opening time when the inference subject can be continuously opened and photographed. The time of the eye of the image for inference, the machine learning program uses a learning tool that further includes a maximum eyelid opening time that indicates the time that the subject for learning can continuously open the eye of the image for which the learning image is taken. The eyelid data further includes, as a part of the question, an image representing the eyes of the learning subject when the eyes of the learning subject have been examined for symptoms of dry eye syndrome At least one of examination image data for learning and examination result data for learning showing examination results related to symptoms of dry eye is learned as the teacher data of the answer, and the examination method causes the The machine learning program infers the symptoms of dry eye symptoms that appear in the eyes of the subject for inference, and outputs symptom data representing the symptoms of dry eye symptoms that appear in the eyes of the subject for inference as the inference material.

One aspect of the present invention is the inspection method as described above, in which inference tear meniscus image data representing an inference tear meniscus image are acquired as the inference image data, and the inference tear meniscus is obtained. The surface image is obtained by cutting out the region in which the tear meniscus of the inference subject is drawn from the inference image in which the tear meniscus of the eye of the inference subject is drawn, The machine learning program uses learning tear meniscus image data representing a learning tear meniscus image as the problem, and further includes a representation that depicts the learning image data as the problem. The image data of the learning test image for learning the image of the eye of the test subject when the test related to the symptoms of dry eye is performed on the eye of the test subject, and the test results showing the test results related to the symptoms of dry eye At least one of the examination result data for learning has been studied as the teacher data of the answer, and the tear meniscus image for learning is a tear meniscus drawn from the eye of the subject for learning A region in which the tear meniscus of the subject for learning is drawn in the image for learning on the surface is cut out, and the inspection method causes the machine learning program to infer the eye of the subject for inference. Symptoms of dry eye symptoms appearing in the eyes of the subject for inference are output as the data for inferences.

One aspect of the present invention is the inspection method as described above, in which inference illumination image data representing an inference illumination image is acquired as the inference image data, and the inference illumination image is self-drawn and reflected The machine learning program is obtained by clipping and drawing a region of the illumination reflected on the cornea of the inference subject from the inference image to the illumination on the cornea of the inference subject, and the machine learning program Using the illumination image data for learning representing the illumination image for learning as the question, and further including a representation that depicts the eye of the subject for learning that has been treated with dry eye syndrome At least one of the examination image data for learning of the image of the eye of the subject to be examined and the examination result data for learning showing the examination result related to the symptoms of dry eye at the time of the examination related to the symptoms of The teacher profile of the answer has been learned, and the learning illumination image is cut out and drawn from the learning image in which the illumination reflected on the cornea of the learning subject is drawn. obtained by reflecting on the illuminated area on the cornea of the subject for learning, the test method causes the machine learning program to infer the symptoms of dry eye appearing in the eye of the subject for inference, and Symptom data representing the symptoms of dry eye appearing in the eyes of the subject for inference are output as the data for inference.

One aspect of the present invention is the above-described inspection method, which acquires the information indicating that the eye on which the inference subject is drawn is irradiated with light having a lower color temperature than the light irradiating the inference subject's eye. The inference image data of the inference image of the eye of the inference subject, and the inference image data of the inference subject's eye, using the machine learning program to represent that the eye of the learning subject's eye is irradiated with a relatively high color temperature. The image data for learning of the image for learning of the eye of the subject for learning when the light of the eye of the subject for learning is irradiated with low light is the question, and further includes an expression The test result data for learning that investigates the test result of the thickness of the tear oil layer of the eye of the test subject for learning is learned as the teacher data for the answer, and the test method causes the machine learning program to infer the The symptoms of dry eye appearing in the eyes of the subject for inference are inferred, and symptom data representing the symptoms of dry eye appearing in the eyes of the subject for inference are output as the data for inference.

One aspect of the present invention is a machine learning execution method that acquires, as a question, image data for learning representing an image for learning on which an eye of a subject for learning is drawn, and an eye that represents the subject for learning to be examined. The learning of the state uses the state data as the teacher data of the answer, and the teacher data is input into the machine learning program, and the machine learning program is made to learn.

One aspect of the present invention is an inspection apparatus including: a data acquisition unit that acquires image data for inference, the image data for inference representing an inference image depicting an eye of a subject for inference; and an inference unit , and input the image data for inference into a machine learning program, and the machine learning program uses the image data for learning representing the image for learning in which the eyes of the subject for learning are drawn as a problem, The learning state data for learning the eye state of the subject to be examined is learned as the teacher data for the answer, and the inference unit causes the machine learning program to infer the eye state of the subject for inference, and causes the machine to learn The program outputs data for inference indicating the eye state of the subject for inference.

One aspect of the present invention is a machine learning execution device comprising: a teacher data acquisition unit that acquires, as a question, image data for learning representing an image for learning in which the eyes of a subject for learning are drawn, and which represents the Teacher data for learning the state data for learning the eye state of the subject to be examined as an answer; and a machine learning execution unit for inputting the teacher data into a machine learning program and causing the machine learning program to learn. [Effect of invention]

According to the present invention, examination results related to the eyes can be predicted without actually carrying out the examination.

[embodiment] 1 to 72 , specific examples of the inspection method, the machine learning execution method, the inspection apparatus, and the machine learning execution apparatus according to the embodiment will be described.

First, with reference to FIGS. 1 to 3 , specific examples of the machine learning execution method and the machine learning execution apparatus according to the embodiment will be described. FIG. 1 is a diagram showing an example of the hardware configuration of the machine learning execution device according to the embodiment. The machine learning execution apparatus 10 shown in FIG. 1 is an apparatus that causes the machine learning apparatus 700 to perform machine learning in a learning phase of the machine learning apparatus 700 to be described later. In addition, as shown in FIG. 1 , the machine learning execution device 10 includes a processor 11 , a main storage device 12 , a communication interface 13 , an auxiliary storage device 14 , an input and output device 15 , and a bus bar 16 .

The processor 11 is, for example, a central processing unit (CPU), and reads out and executes the machine learning execution program 100 to be described later, so as to achieve each function of the machine learning execution program 100 . In addition, the processor 11 may also read out and execute programs other than the machine learning execution program 100 , so as to achieve necessary functions on the basis of achieving each function of the machine learning execution program 100 .

The main storage device 12 is, for example, a random access memory (Random Access Memory, RAM), and stores in advance a machine learning execution program 100 and other programs read and executed by the processor 11 .

The communication interface 13 is an interface circuit for performing communication with the machine learning device 700 and other devices via the network NW shown in FIG. 1 . In addition, the network NW is, for example, a local area network (LAN) or an intranet.

The auxiliary storage device 14 is, for example, a hard disk drive (HDD: Hard Disk Drive), a solid state drive (SSD: Solid State Drive), a flash memory (Flash Memory), and a read only memory (Read Only Memory, ROM).

The input/output device 15 is, for example, an input/output port. The input/output device 15 is connected to, for example, a keyboard 811 , a mouse 812 , and a display 910 shown in FIG. 1 . The keyboard 811 and the mouse 812 are used, for example, for inputting data necessary for operating the machine learning execution device 10 . The display 910 displays, for example, a Graphical User Interface (GUI: Graphical User Interface) of the machine learning execution device 10 .

The bus bar 16 connects the processor 11 , the main storage device 12 , the communication interface 13 , the auxiliary storage device 14 and the input/output device 15 , so that these can transmit and receive data to and from each other.

FIG. 2 is a diagram showing an example of a software configuration of the machine learning execution device according to the embodiment. The machine learning execution device 10 uses the processor 11 to read out and execute the machine learning execution program 100 to achieve the teacher data acquisition function 101 and the machine learning execution function 102 shown in FIG. 2 .

The teacher data acquisition function 101 acquires a teacher who sets the learning image data representing the learning image depicting the eyes of the learning subject as a question and the learning state data representing the eye state of the learning subject as the answer material.

The eye state of the subject for learning is reflected, for example, when the eyes of the subject for learning have undergone an examination related to symptoms of dry eye, an examination related to corneal epithelial damage, an examination related to mucin damage, and a study study. These inspection results are used in the inspection of the tear layer destruction time of the eye of the subject or the inspection to investigate the thickness of the tear oil layer.

The image data for learning is data representing at least a part of the questions constituting the teacher data and an image for learning in which the eyes of the subject for learning are drawn. The learning image is captured using, for example, a camera connected to an inspection apparatus described later, a camera mounted on a smartphone, or the like.

In addition, the teacher data may further include, as a part of the answer, data on measures for learning indicating measures to improve the eye condition of the subject for learning.

The machine learning execution function 102 inputs the teacher data into the machine learning program 750 installed in the machine learning device 700, and causes the machine learning program 750 to learn. For example, the machine learning execution function 102 makes the machine learning program 750 including a Convolutional Neural Network (CNN) learn by backpropagation.

Next, an example of processing executed by the machine learning execution program 100 of the embodiment will be described with reference to FIG. 3 . FIG. 3 is a flowchart showing an example of processing executed by the machine learning execution program according to the embodiment. The machine learning execution program 100 executes the process shown in FIG. 3 at least once.

In step S11, the teacher data acquisition function 101 acquires teacher data that uses, as a question, the learning image data representing the learning image in which the eyes of the learning subject are drawn, and the learning image data representing the learning subject. The learning of the eye state uses the state data as the answer.

In step S12, the machine learning execution function 102 inputs the teacher data into the machine learning program 750, so that the machine learning program 750 learns.

Next, specific examples of an inspection program, an inspection apparatus, and an inspection method according to the embodiment will be described with reference to FIGS. 4 to 17 .

FIG. 4 is a diagram showing an example of the hardware configuration of the inspection apparatus according to the embodiment. The inspection apparatus 20 shown in FIG. 4 uses the machine learning apparatus 700 to infer the eye state of the inference subject in the inference stage of the machine learning apparatus 700 that has been learned by the machine learning execution program 100 . In addition, as shown in FIG. 4 , the inspection device 20 includes a processor 21 , a main storage device 22 , a communication interface 23 , an auxiliary storage device 24 , a touch panel display 25 , and a bus bar 26 .

The processor 21 is, for example, a CPU, and reads out and executes the later-described inspection program 200 to achieve each function of the inspection program 200 . In addition, the processor 21 may read out and execute programs other than the check program 200 to achieve necessary functions in addition to each function of the check program 200 .

The main storage device 22 is, for example, a RAM, and stores the check program 200 and other programs read and executed by the processor 21 in advance.

The communication interface 23 is an interface circuit for performing communication with the machine learning device 700 and other devices via the network NW shown in FIG. 4 . In addition, the network NW is, for example, a LAN or an intranet.

The auxiliary storage device 24 is, for example, a hard disk drive, a solid state drive, a flash memory, and a ROM.

The touch panel display 25 is, for example, an input/output port. The touch-panel display 25 displays, for example, the graphical user interface of the inspection apparatus 20 and the content represented by the inference data to be described later.

The bus bar 26 connects the processor 21 , the main storage device 22 , the communication interface 23 , the auxiliary storage device 24 and the touch panel display 25 , so that these can send and receive data to and from each other.

FIG. 5 is a diagram showing an example of the appearance of the inspection apparatus according to the embodiment. As shown in FIG. 5, the inspection apparatus 20 is supported by the support|pillar P attached to the upper part of the base B. As shown in FIG. In addition, the inspection apparatus 20, the base B, and the support|pillar P are installed in a grocery store (drugstore) etc., for example.

FIG. 6 is a diagram showing an example of a software configuration of the inspection apparatus according to the embodiment. The inspection device 20 uses the processor 21 to read out and execute the inspection program 200 to achieve the data acquisition function 201 and the inference function 202 shown in FIG. 6 .

The data acquisition function 201 acquires image data for inference. The image data for inference is a data showing an image for inference in which the eyes of the subject for inference are drawn. The image for inference is captured using, for example, a camera connected to an inspection apparatus described later, or a camera mounted on a smartphone.

The inference function 202 inputs the image data for inference to the machine learning program 750 that has been learned by the machine learning execution function 102, and causes the machine learning program 750 to infer the eye state of the inference subject.

Then, the inference function 202 causes the machine learning program 750 to output inference data indicating the eye state of the inference subject. In this case, the data for inference is used to display, for example, on the touch panel display 25 a numerical value representing the state of the eyes of the subject for inference, or the like. In addition, the data for inference may represent, for example, the degree of fatigue of the eye determined based on the state of the cornea, tears, etc. of the eye of the subject for inference that can be read from the image data for inference.

In addition, the inference function 202 may cause the machine learning program 750 to output inference data indicating measures to improve the eye state of the inference subject.

FIG. 7 is a diagram showing an example of an image displayed by the inspection apparatus according to the embodiment. The image A7 shown in FIG. 7 is the inference subject photographed by the camera after the inference subject has recognized the face of the subject in the state in front of the camera connected to the inspection apparatus 20 . body facial image. The two rectangles drawn in the image A7 represent the left eye and the right eye of the subject for inference, respectively. In addition, the image A7 is an example of the image for inference.

FIG. 8 is a diagram showing an example of an image displayed by the inspection apparatus according to the embodiment. The image A8 shown in FIG. 8 is an image which is estimated based on the image data for inference showing the image A7 for inference, which is an example of the image for inference, and shows the estimation result of the eye state of the subject for inference. As shown in FIG. 8 , the image A8 includes a score indicating the corneal state of the subject for inference, and an icon “low” indicating that the score is low. In addition, as shown in FIG. 8 , the image A8 includes an image A81 in which a graph representing the damaged portion of the cornea is superimposed on the eye image of the subject for inference.

FIG. 9 is a diagram showing an example of an image displayed by the inspection apparatus according to the embodiment. The image A9 shown in FIG. 9 is an image which is estimated based on the image data for inference showing the image A7 which is an example of the image for inference, and shows the estimation result of the eye state of the subject for inference. The image A9 shows a radar chart showing the score related to the corneal state of the subject for inference, the score related to the water layer of tears, the score related to the oil layer of tears, and the score related to the mucus layer score. In addition, the image A9 may show numerical values representing the above-mentioned four scores, icons, etc., instead of the radar chart shown in FIG. 9 . In addition, the image A9 shows the degree of fatigue of the eyes of the subject for inference by five-stage display. The image A9 may also be displayed on the touch panel display 25 instead of the image A8.

FIG. 10 is a diagram showing an example of an image displayed by the inspection apparatus according to the embodiment. The image A11 shown in FIG. 10 shows the result estimated as the cause of the eye state of the subject for inference shown in the image A10. For example, after the images A8 and A9 are displayed on the touch panel display 25 , the image A11 is displayed on the touch panel display 25 .

FIG. 11 is a diagram showing an example of an image displayed by the inspection apparatus according to the embodiment. An image A13 shown in FIG. 11 shows an example of a countermeasure against the cause shown in the image A11. After the image A11 is displayed on the touch panel display 25 , the image A13 is displayed on the touch panel display 25 .

FIG. 12 is a diagram showing an example of an image displayed by the inspection apparatus according to the embodiment. The image A14 shown in FIG. 12 shows an example of the eye drops useful for the image A11 or the image A13. After the image A12 or the image A13 is displayed on the touch panel display 25 , the image A14 is displayed on the touch panel display 25 .

13 to 16 are diagrams showing an example of an eye image displayed instead of the eye image shown in FIG. 8 . The inference function 202 outputs data for inference, which indicates the display state of the area that satisfies the predetermined condition and the display state of the area that does not satisfy the predetermined condition in the eye of the subject for inference drawn by the inference image. different images. The prescribed conditions mentioned here are conditions in which the possibility of a reduction in the aqueous layer of tears exceeds a prescribed threshold, conditions in which the possibility of a reduction in the oil layer of tears exceeds a prescribed threshold, and the possibility of damage to at least one of the cornea and the conjunctiva exceeds the threshold. At least one of the conditions that specify the threshold.

For example, as shown in FIG. 13 , the inference function 202 may superimpose and display, on the image for inference, a hatched slanted line indicating an area where the water layer of the tear fluid is likely to decrease. The diagonal hatching can have different concentrations depending on the degree of reduction of the aqueous layer of the tear fluid.

Alternatively, as shown in FIG. 14 , the inference function 202 may superimpose and display a vertical hatched line indicating a region where the oil layer of tears is likely to decrease on the inference image. The vertical hatching can have different concentrations depending on the degree of reduction of the oil layer of the tear fluid.

Alternatively, as shown in FIG. 15 , the inference function 202 may superimpose and display dotted hatching indicating areas where the cornea and conjunctiva are likely to be damaged on the inference image. The dotted hatching can have different concentrations depending on the degree of damage.

Alternatively, as shown in FIG. 16 , the inference function 202 may superimpose and display the oblique hatching, vertical hatching, and dot hatching on the inference image.

Next, an example of processing executed by the inspection program 200 of the embodiment will be described with reference to FIG. 17 . FIG. 17 is a flowchart showing an example of processing performed by the inspection program of the embodiment.

In step S21, the data acquisition function 201 acquires image data for inference, the image data for inference representing the inference image in which the eyes of the subject for inference are drawn.

In step S22, the inference function 202 inputs the image data for inference into the machine learning program 750, causes the machine learning program 750 to infer the eye state of the inference subject, and causes the machine learning program 750 to output the data representing the inference subject. Data for inference of eye state.

The machine learning execution method, the inspection method, the machine learning execution apparatus, and the inspection apparatus according to the embodiments have been described above.

The machine learning execution program 100 includes a teacher data acquisition function 101 and a machine learning execution function 102 .

The teacher data acquisition function 101 acquires a teacher who sets the learning image data representing the learning image depicting the eyes of the learning subject as a question and the learning state data representing the eye state of the learning subject as the answer material.

The machine learning execution function 102 inputs the teacher data into the machine learning program 750, and causes the machine learning program 750 to learn.

Thereby, the machine learning execution program 100 can generate the machine learning program 750 for predicting the eye state based on the image data for learning.

The inspection program 200 includes a data acquisition function 201 and an inference function 202 .

The data acquisition function 201 acquires image data for inference. The image data for inference is data representing an image in which the eyes of the subject for inference are drawn.

The inference function 202 inputs the image data for inference to the machine learning program 750 that has been learned by the machine learning execution program 100 to infer the eye state of the subject for inference. Then, the inference function 202 causes the machine learning program 750 to output inference data indicating the eye state of the inference subject.

Thereby, the examination program 200 can predict the examination result related to the eye without actually carrying out the examination.

In addition, the examination program 200 outputs data for inference, which indicates the display state of the area that satisfies the predetermined condition in the eyes of the subject for inference drawn by the image for inference and the area that does not satisfy the predetermined condition. Displays images in different ways. In this way, the inspection program 200 can present each of the areas that satisfy the predetermined conditions and the areas that do not satisfy the predetermined conditions in a more easily recognizable form.

In addition, the examination program 200 outputs data for inference indicating measures to improve the eye state of the subject for inference based on the data for inference. In this way, the examination program 200 can notify the inference subject of measures to improve the state of the eyes.

Furthermore, at least a part of the functions of the machine learning execution program 100 can be realized by hardware including circuitry. Likewise, at least a part of the functions of the checking program 200 can be realized by hardware including a circuit portion. In addition, such hardware is, for example, a Large Scale Integration (LSI), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), Graphics Processing Unit (GPU).

In addition, at least a part of the functions of the machine learning execution program 100 can also be achieved by the cooperation of software and hardware. Likewise, at least a part of the functions of the checking program 200 can also be achieved through the cooperation of software and hardware. In addition, these hardwares may be integrated into one, or may be divided into multiple pieces.

In addition, in the embodiment, the case where the machine learning execution device 10 , the machine learning device 700 , and the inspection device 20 are mutually independent devices has been described as an example, but the present invention is not limited to this. The devices can also be implemented as one device.

In addition, the inspection apparatus 20 may be implemented in the aspect other than the above-mentioned aspect. For example, the inspection device 20 may be a learning tablet, a game machine, a computer, or the like. In this case, the examination apparatus 20 can predict the examination result concerning the eyes of a learner, a game player, and a computer user.

In addition, since the inspection apparatus 20 does not require special equipment such as medical equipment and medical instruments, it is possible to eliminate labor for inferring that the subject to be inspected goes to the hospital, and to achieve remote diagnosis and the like. For example, in the case where the inspection apparatus 20 is a smartphone, it is possible to predict the eye-related inspection result of the inference subject while saving the labor of going to the hospital for the inference subject. Alternatively, in the case where the inspection apparatus 20 is a smartphone, it is possible to predict the eye-related inspection result of the inference subject, while saving the labor of going to the hospital for the inference subject, and to make the inference subject to be inspected. You can actually feel the effects of the treatment, or prescribe contact lenses, etc. In addition, the effects of the treatment include, for example, reduction in damage to the cornea or conjunctiva, and moisturizing of the eyes by the aqueous or oily layers of tears. Furthermore, the subject for inference may be an animal such as a dog or a cat.

Next, a specific example of the first embodiment of the above-described embodiment will be described with reference to FIGS. 18 to 31 .

FIG. 18 is a diagram showing an example of the configuration of the inspection system according to the first embodiment. The inspection system E1 includes the imaging device 1 and the inspection device 2 .

The photographing apparatus 1 includes a digital camera to photograph the eyes of the user U1. As an example, the photographing apparatus 1 is a smartphone including a digital camera. The photographing apparatus 1 transmits the eye image of the user U1 captured by the digital camera, that is, the user image data P1 to the inspection apparatus 2 . As an example, the inspection apparatus 2 is a server.

The inspection apparatus 2 stores the learning result L1 learned by deep learning. The learning result L1 is obtained by deep learning between the subject's image data, which is the subject's eye image captured by the digital camera, and the subject's test data including the test result of the subject's eye state. the result of the relationship. The examination data of the examinee may be a measured value, a score value based on a predetermined standard, or image data. In the first embodiment, the so-called eye state is at least one of the tear state and the corneal conjunctiva state. The tear state includes at least one of the amount of tear liquid, the thickness of the tear oil layer, the damaged area of the mucus layer, the tear layer destruction time, and the non-invasive tear layer destruction time. The corneal and conjunctival state includes the damaged area of the corneal and conjunctiva. For the score value, for example, the non-patent document "Jun Shima ▲ Saki ▼, 2006 Dry Eye Diagnosis Criteria, "New Ophthalmology", published by Medical Aoi, Feb. 28, 2007, Vol. 24, Feb. issue , p.181-184".

The so-called examinee refers to the provider of the examinee's image data and the examinee's examination data used in the study.

The inspection apparatus 2 predicts inspection data including the result of inspecting the eye state of the user U1 based on the learning result learned by deep learning and the user image data P1. The inspection apparatus 2 transmits the prediction result A1, which is a result obtained by predicting inspection data, to the imaging apparatus 1 . The photographing apparatus 1 presents the prediction result A1 to the user U1.

[Structure of the photographing device] FIG. 19 is a diagram showing an example of the configuration of an imaging device according to a modification of the first embodiment. The photographing apparatus 1 includes a photographing apparatus control unit 100 , a photographing unit 110 , a photographing apparatus communication unit 120 , a display unit 130 , and an operation accepting unit 140 .

The imaging device control unit 100 includes an image acquisition unit 1000 , an imaging condition determination unit 1010 , a white eyeball and black eyeball part extraction unit 1020 , an image output unit 1030 , a prediction result acquisition unit 1040 , and a presentation unit 1050 . As an example, the imaging device control unit 100 is implemented by a CPU, an image acquisition unit 1000, an imaging condition determination unit 1010, a white eyeball and black eyeball part extraction unit 1020, an image output unit 1030, a prediction result acquisition unit 1040, and a presentation unit 1050 are respectively achieved by the CPU reading the program from the ROM and executing the processing.

The image acquisition unit 1000 acquires the photographed image P0 using the photographing unit 110 . The photographed image P0 is an image obtained by photographing the eyes of the user U1. As an example, the photographed image P0 is an image obtained by photographing the eye of the user U1 in a state in which the eye of the user U1 is not dyed with a dyeing agent or the like. As an example, the captured image P0 is one image. The photographed image P0 may be a set of a plurality of images. Furthermore, the photographing unit 110 may photograph the eyes of the user U1 in a state where the eyes are dyed. The term "staining" refers to, for example, luciferin staining or Lissamine green staining. The imaging condition determination unit 1010 determines whether or not predetermined imaging conditions are satisfied with respect to the captured image P0 acquired by the image acquisition unit 1000 .

Here, predetermined imaging conditions will be described with reference to FIG. 20 . FIG. 20 is a diagram showing an example of predetermined imaging conditions in a modification of the first embodiment. Before describing the predetermined imaging conditions, the black eyeball and the white eyeball in the first embodiment will be described. The so-called black eye refers to the part corresponding to the cornea when the eye is viewed from the front. When the eye is viewed from the front, the pupil and the iris are located inside the cornea. Therefore, in other words, the black eye also includes the pupil and the iris. On the other hand, the so-called white eye refers to the portion corresponding to the bulbar conjunctiva when the eye is viewed from the front.

In the first embodiment, the predetermined photographing conditions include the following three photographing conditions.

The first photography condition is to capture the entire black eyeball.

The second imaging condition includes that the ratio of the vertical length to the horizontal length (referred to as an aspect ratio in the following description) is within a predetermined range for the portion where the white eyeball and the black eyeball are combined. Regarding the aspect ratio, the so-called predetermined range is, as an example, the range of 0.62 to 0.75. In FIG. 20 , the captured image R1 , the captured image R2 , and the captured image R3 are examples of captured images for explaining the second imaging condition, respectively. Regarding each of the captured image R1 , the captured image R2 , and the captured image R3 , the aspect ratios are 0.61, 0.64, and 0.76, respectively. In the photographed image R1, the eyelids are not sufficiently opened, and the second photographing condition is not satisfied. In the photographed image R2, the eyelid is sufficiently opened to satisfy the second photographing condition. In the photographed image R3, the eyelid is excessively opened, and the second photographing condition is not satisfied.

The third photographing condition includes: the longitudinal length of the portion where the white eyeball and the black eyeball are combined, and the ratio of the longitudinal length of the black eyeball in the portion where the white eyeball and the black eyeball are combined (referred to as the white eyeball and the black eyeball in the following description). eyeball ratio) is within the specified range. Regarding the ratio of white eyeball to black eyeball, the so-called predetermined range is, as an example, the range of 1.20 to 1.40. In FIG. 20 , the captured image R4 , the captured image R5 , and the captured image R6 are examples of captured images for explaining the third imaging condition, respectively. For each of the captured image R4, the captured image R5, and the captured image R6, the ratios of white eyeballs and black eyeballs are 1.12, 1.31, and 1.51, respectively. In the photographed image R5, the third photographing condition is satisfied. In the photographed image R4 and the photographed image R6, the third photographing condition is not satisfied.

Returning to FIG. 19 , the description of the configuration of the photographing apparatus 1 will be continued.

The white eyeball and black eyeball portion extraction unit 1020 cuts out the portion of the eye in which the white eyeball and the black eyeball are combined from the captured image P0. The part of the eye in which the white eyeball and the black eyeball are combined refers to the part of the eye excluding the eyelid, the skin, and the like. An image obtained by cutting out the portion of the eye in which the white eyeball and the black eyeball are combined from the photographed image P0 is referred to as a user image material P1. An example of the user image data P1 is shown in FIG. 21 . In the user image material P1, the portion of the eyes other than the portion where the white eyeball and the black eyeball are combined is deleted. The portion of the eye other than the portion where the white eyeball and the black eyeball are combined includes the eyelid and the skin. In the first embodiment, the user image data P1 is an image obtained by cutting out the portion of the eye that is composed of the white eyeball and the black eyeball.

Furthermore, the captured image P0 may be directly used as the user image data P1 without the processing of cutting out the portion of the eye where the white eyeball and the black eyeball are combined in the captured image P0. In this case, the white eyeball and black eyeball portion extraction unit 1020 can be omitted from the configuration of the imaging device control unit 100 .

The image output unit 1030 outputs the user image data P1 to the inspection apparatus 2 .

The prediction result acquisition unit 1040 acquires the prediction result A1 from the inspection device 2 . The prediction result A1 represents the state of the eyes by, for example, a score. The prediction result A1 includes, for example, at least one or more of a score related to the tear state and a score related to the cornea and conjunctiva.

The presentation unit 1050 presents the prediction result A1. The presentation unit 1050 presents the prediction result A1 by displaying the prediction result A1 on the display unit 130 .

The imaging unit 110 includes a digital camera. The photographing unit 110 photographs the eyes of the user U1. Preferably, the eyes of the user U1 are photographed from the front. In addition, when the photographing unit 110 captures the eyes of the user U1, it is preferable that the focal length is appropriately adjusted even when the distance between the lens of the digital camera constituting the photographing unit 110 and the eyes of the user U1 is short.

The image captured by the photographing unit 110 may be a still image or a moving image. When the photographing unit 110 captures a moving image, the image acquiring unit 1000 automatically selects the photographed image P0 from a plurality of frames included in the moving image. For example, the image acquisition unit 1000 automatically selects a frame that satisfies the predetermined photographing conditions from a plurality of frames included in the animation. In this case, the image acquisition unit 1000 causes the imaging condition determination unit 1010 to determine whether or not a predetermined imaging condition is satisfied for each of a plurality of frames included in the animation.

The imaging device communication unit 120 communicates with the inspection device 2 via a wireless network. The camera communication unit 120 includes hardware for communicating via a wireless network.

The display unit 130 displays various kinds of information. The display unit 130 displays the photographed image P0 captured by the photographing unit 110 or the prediction result A1. The display unit 130 is a liquid crystal display or an organic electroluminescence (El: Electroluminescence) display.

The operation accepting unit 140 accepts an operation from the user U1. The operation accepting unit 140 includes, for example, a touch panel, and is configured integrally with the display unit 130 .

[Structure of Inspection Device] FIG. 22 is a diagram showing an example of the configuration of an inspection apparatus according to a modification of the first embodiment. The inspection apparatus 2 includes an inspection apparatus control unit 200 , a storage unit 210 , and an inspection apparatus communication unit 220 .

The inspection apparatus control unit 200 includes an image data acquisition unit 2000 , a prediction unit 2010 , a prediction result output unit 2020 , and a learning unit 2030 . As an example, the inspection apparatus control unit 200 is implemented by a CPU, and the image data acquisition unit 2000 , the prediction unit 2010 , the prediction result output unit 2020 , and the learning unit 2030 are implemented by the CPU reading a program from the ROM and executing the processing.

The image data acquisition unit 2000 acquires the user image data P1 from the photographing device 1 .

The prediction unit 2010 predicts the user inspection data based on the learning result L10 and the user image data P1. The user inspection data includes results when at least one or more of the tear state and the corneal conjunctiva state of the user U1 are inspected.

The prediction result output unit 2020 outputs the user inspection data predicted by the prediction unit 2010 to the imaging device 1 as the prediction result A1.

The learning unit 2030 learns the relationship between the subject image data and the subject examination data. The subject image data is an image of the subject's eyes captured by a digital camera. The digital camera may be different from the digital camera included in the photographing apparatus 1 .

The examinee examination data includes examination results of the eye state of the examinee. There can be multiple examinees. User U1 may or may not be included in the inspected persons. The photographing of the subject's eye image and the examination of the subject's eye state are performed on the same date and time.

The amount of tear fluid is examined using, for example, the height of the tear fluid meniscus using an optical interference tomography instrument. The tear oil layer thickness is checked, for example, with an optical interferometer. The damage area of the mucus layer is examined using, for example, the staining area or staining concentration of the lissamine green-stained eye as an index. The tear layer destruction time is examined, for example, with the time from the moment of blinking to the time when luciferin-stained tears are destroyed as an index. The non-invasive tear layer destruction time was examined as the time from the moment of blinking to the tear liquid, visualized by optical interference imaging, as an index. The corneal and conjunctival lesion area is examined by, for example, the staining area or staining concentration of the fluorescein-stained eye.

The learning unit 2030 performs learning based on a data set including a combination of subject image data and subject examination data. The data set is pre-stored in the storage part 210, the learning part 2030 can acquire the data set from the storage part 210, and the learning part 2030 can also acquire the data set from an external device such as a server. The learning unit 2030 stores the learned result in the storage unit 210 as the learning result L10.

As an example, the learning unit 2030 performs learning based on deep learning. The learning result L10 is information representing a neural network-like network whose parameters of weight and bias have been changed by performing learning.

The subject image data is an image obtained by photographing the eyes of the subject in a state where the predetermined imaging conditions are satisfied. Furthermore, the image data of the subject may not satisfy the predetermined photographing conditions. In addition, the subject image data may include both images satisfying predetermined imaging conditions and images not satisfying predetermined imaging conditions.

When an image satisfying predetermined imaging conditions is used as the user image data P1 as in the first embodiment, it is preferable that the subject image data satisfy the predetermined imaging conditions in order to improve the accuracy of prediction.

The image data of the examinee is an image obtained by cutting out the part of the eye where the white eyeball and the black eyeball are combined. Furthermore, the image data of the examinee may also be an image from which the portion is not cut out. In addition, the image data of the examinee may include both an image obtained by cutting out a portion of the eye where the white eyeball and the black eyeball are combined, and an image in which the portion is not cut out.

In the case where an image obtained by cutting out the portion of the eye that is combined with the white eyeball and the black eyeball is used as the user image data P1 as in the first embodiment, in order to improve the accuracy of prediction, the image data of the examinee is preferred. An image to crop out that part.

The image data of the examinee includes images of normal eyes and images of abnormal eyes. The image of the eye with an abnormal corneal and conjunctival lesion area refers to, for example, an eye image taken on the same date and time as when the corneal and conjunctival lesion area is 33% or more. The image of an eye with a normal corneal and conjunctival damage area refers to, for example, an eye image taken on the same date and time as when the corneal and conjunctival damage area is less than 33%. Similarly, the so-called mucus layer damaged area, tear layer destruction time, non-invasive tear layer destruction time, tear oil layer thickness, and abnormality in the amount of tear fluid refer to, for example, an image with an abnormal mucus layer damage area of 33% or more, Eye images taken at the same date and time when the tear layer destruction time is less than 5 seconds, the non-invasive tear layer destruction time is less than 10 seconds, the thickness of the tear oil layer is less than 80 μm, and the amount of tear fluid is less than 310 μm. The so-called mucus layer damage area, tear layer destruction time, non-invasive tear layer destruction time, tear oil layer thickness, and tear volume are normal images, for example, refers to the mucus layer damage area less than 33%, tear layer destruction time Eye images taken at the same date and time when more than 5 seconds, the non-invasive tear layer destruction time is more than 10 seconds, the thickness of the tear oil layer is more than 80 μm, and the amount of tear fluid is more than 310 μm.

In the first embodiment, the number of images of normal eyes included in the image data of the subject is equal to the number of images of abnormal eyes. The number of images of normal eyes and the number of images of abnormal eyes are, for example, 1000, respectively. In order to improve the accuracy of prediction, the number of images of normal eyes and the number of images of abnormal eyes are preferably equal. Furthermore, the number of images of normal eyes and the number of images of abnormal eyes included in the image data of the subject may be different.

The learning unit 2030 may also apply deformation to the subject image data before performing learning. The deformation includes, for example, deformation caused by enlargement and reduction, rotation, horizontal and vertical translation, or shear strain. For example, the learning unit 2030 applies deformation to the image data of the subject, for example, so that the part of the black eye is located in the center of the image, and the size of the part of the black eye is made the same as between multiple images. The learning unit 2030 may also combine the original image data of the examinee before deformation and a new image obtained by deforming the image data of the examinee, thereby increasing the number of image data of the examinee.

In the neural network-like network used in the deep learning by the learning unit 2030, the number of intermediate layers is three. Furthermore, the intermediate layers of a neural network-like network for deep learning can also have more than three layers. The learning unit 2030 executes deep learning a predetermined number of times. The predetermined number of times is, for example, five times. In addition, the predetermined number of times may be other than five times.

Furthermore, the learning unit 2030 may perform learning based on machine learning other than deep learning.

The result learned by the external device may be stored in the storage unit 210 in advance as the learning result L10. In this case, the learning unit 2030 can be omitted from the configuration of the inspection apparatus control unit 200 .

The storage unit 210 stores various kinds of information. The information stored in the storage unit 210 includes the learning result L10. The storage unit 210 is configured using a storage device such as a magnetic hard disk device or a semiconductor storage device.

The inspection device communication unit 220 communicates with the imaging device 1 via a wireless network. The inspection device communication section 220 includes hardware for communication via a wireless network.

[Eye state prediction processing of photographing device] Next, with reference to FIG. 23 , an eye state prediction process, which is a process in which the photographing apparatus 1 captures the eyes of the user U1 and presents the prediction result A1 , will be described. FIG. 23 is a diagram showing an example of an eye state prediction process in a modification of the first embodiment.

Step S10 : The photographing device control unit 100 displays the photographing screen on the display unit 130 . The photographing screen is a screen displayed on the display unit 130 when the photographing unit 110 photographs the eyes of the user U1. The face image of the user U1 photographed by the photographing unit 110 is displayed on the photographing screen in real time.

On the photographing screen, a photographing guide may be displayed in addition to the facial image of the user U1 photographed by the photographing unit 110 . The photography guide is text, icons, and the like for indicating the photography program to the user U1. The photographing guide may include text and icons for instructing the user U1 to photograph the eyes of the user U1 in a state where predetermined photographing conditions are satisfied. For example, the text is "Please open your eyes to the extent that your eyelids will not cover your black eyeballs." The icon is, for example, an arrow or the like which is displayed on the upper and lower sides of the eye image of the user U1 on the photographing screen and instructs to open the eyes.

Step S20: The photographing unit 110 photographs the eyes of the user U1. The photographing unit 110 generates a photographed image P0 obtained by photographing the eyes of the user U1.

Step S30 : the image acquisition unit 1000 acquires the photographed image P0 from the photographing unit 110 . The image acquisition unit 1000 supplies the acquired photographed image P0 to the photographing condition determination unit 1010 .

Step S40 : The imaging condition determination unit 1010 determines whether or not a predetermined imaging condition is satisfied with respect to the captured image P0 acquired by the image acquisition unit 1000 . The photographing condition determination unit 1010 extracts the portion of the black eyeball and the portion of the white eyeball in the photographed image P0. The imaging condition determination unit 1010 uses an image recognition technique for this extraction process. The imaging condition determination unit 1010 performs determination based on the extracted part of the black eyeball and the part of the white eyeball.

In the first embodiment, the imaging condition determination unit 1010 determines that the predetermined imaging conditions are satisfied when all of the three imaging conditions are satisfied as the predetermined imaging conditions. In addition, the imaging condition determination unit 1010 may determine that the predetermined imaging conditions are satisfied when one or more of the three imaging conditions as the predetermined imaging conditions are satisfied.

When the imaging condition determination unit 1010 determines that the predetermined imaging conditions are satisfied (step S40 ; YES), the imaging condition determination unit 1010 supplies the captured image P0 to the white eyeball and black eyeball portion extraction unit 1020 . After that, the imaging device control unit 100 executes the process of step S50. On the other hand, when the imaging condition determination unit 1010 determines that the predetermined imaging conditions are not satisfied (step S40 ; NO (NO)), the imaging device control unit 100 executes the process of step S90 .

Step S50 : The white eyeball and black eyeball portion extraction unit 1020 cuts out the portion of the eye that is composed of the white eyeball and the black eyeball from the photographed image P0 . The white eyeball and black eyeball portion extraction unit 1020 determines, from the photographed image P0, the portion of the eye where the white eyeball and the black eyeball are combined, and the other portions, based on the image recognition technology. The white eyeball and black eyeball portion extraction unit 1020 cuts out the portion of the eye that is combined with the white eyeball and the black eyeball by deleting the portion other than the combined portion of the white eyeball and the black eyeball from the photographed image P0.

The white eyeball and black eyeball portion extraction unit 1020 supplies an image obtained by cutting out the portion where the white eyeball and the black eyeball are combined from the captured image P0 to the image output unit 1030 as user image data P1.

Step S60 : the image output unit 1030 outputs the user image data P1 to the inspection apparatus 2 . Here, the image output unit 1030 transmits the user image data P1 to the inspection apparatus 2 via the photographing apparatus communication unit 120 .

Furthermore, before the image output unit 1030 outputs the user image data P1 to the inspection apparatus 2 , the photographing device control unit 100 may display the captured user image data P1 on the display unit 130 to remind the user U1 for confirmation. When the user U1 approves the user image data P1 , the image output unit 1030 may output the user image data P1 to the inspection apparatus 2 . If the user U1 does not approve the user image data P1, the photographing device control unit 100 may execute the process of step S10 again. The approval by the user U1 is performed as an operation from the operation accepting unit 140 .

Step S70 : the prediction result acquisition unit 1040 acquires the prediction result A1 from the inspection device 2 . Here, the prediction result acquisition unit 1040 receives the prediction result A1 from the inspection device 2 via the imaging device communication unit 120 .

Step S80: The presentation unit 1050 presents the prediction result A1. The presentation unit 1050 presents the prediction result A1 by displaying the prediction result A1 on the display unit 130 .

Step S90 : The photographing condition determination unit 1010 controls the display unit 130 so that the display unit 130 displays the photographing guide. The photographing guide displayed in step S90 includes, for example, content corresponding to a photographing condition that is not satisfied among the predetermined photographing conditions. For example, when the photographed image P0 does not satisfy the first photographing condition, a text such as "Please open your eyes to the extent that the eyelids do not cover the dark eyeballs." is displayed. In step S90, the same photography guide as that displayed in step S10 may also be displayed.

After the photographing condition determination unit 1010 completes the processing of step S90, the photographing apparatus 1 executes the processing of step S20 again.

As described above, the photographing apparatus 1 ends the eye state prediction process.

Furthermore, as described above, the user image data P1 is generated from the photographed image P0 that satisfies the predetermined photographing conditions. Therefore, the user image data P1 is an image obtained by photographing the eyes of the user U1 in a state in which predetermined photographing conditions are satisfied.

Furthermore, the user image data P1 may not satisfy the predetermined photographing conditions. In this case, the process of step S40 described above is omitted. In this case, the imaging condition determination unit 1010 may be omitted from the configuration of the imaging device control unit 100 .

Here, various screens displayed on the display unit 130 of the photographing apparatus 1 will be described with reference to FIGS. 24 to 27 .

FIG. 24 is a diagram showing an example of a photographing screen in a modification of the first embodiment. On the photographing screen G1, a facial image including the eyes of the user U1 and a photographing guide are displayed. On the photographing screen G1, a photographing procedure is displayed by text as a photographing guide.

FIG. 25 is a diagram showing an example of a photographing screen in a modification of the first embodiment. In the photographing screen G2, the photographing of the eyes of the user U1 is completed, and the photographed user image data P2 is displayed.

On the photographing screen G2, an "evaluation start" button and a "re-shooting" button are displayed. The "evaluation start" button is a button for the user U1 to approve the captured user image data P2. ” button is used to execute photography again.

FIG. 26 is a diagram showing an example of a prediction result display screen in a modification of the first embodiment. On the prediction result display screen G3, based on the prediction result A1, examination data including the result of examining the eye state of the user U1 are displayed. On the prediction result display screen G3, as an example, the use scores display the respective states of the tear oil layer thickness, the amount of tear liquid, the damaged area of the mucus layer, and the damaged area of the cornea and conjunctiva. In addition, on the prediction result display screen G3, based on the prediction result A1, a comprehensive prediction result for the health risk of the eyes of the user U1 is displayed.

[Predictive processing of inspection equipment] Next, the prediction processing of the inspection apparatus 2 will be described with reference to FIG. 27 . FIG. 27 is a diagram showing an example of prediction processing in a modification of the first embodiment. The prediction process shown in FIG. 27 is performed after the process of step S60 is performed in the eye state prediction process shown in FIG. 23 .

Step S100 : the image data acquisition unit 2000 acquires the user image data P1 from the photographing device 1 . The image data acquisition unit 2000 receives the user image data P1 from the photographing device 1 via the inspection device communication unit 220 . The image data acquisition unit 2000 supplies the acquired user image data P1 to the prediction unit 2010 .

Step S110 : the prediction unit 2010 acquires the learning result L10 from the storage unit 210 .

Step S120: The prediction unit 2010 predicts the user inspection data based on the learning result L10 and the user image data P1. The prediction unit 2010 performs prediction based on deep learning.

As described above, the learning result L10 refers to the subject's examination data including the subject's image data, which is the subject's eye image captured by the digital camera, and the examination result including the subject's eye state. The combined data set is the result of learning the relationship between the subject's image data and the subject's inspection data through deep learning. That is, the learning result L10 refers to a result of learning based on a data set including a combination of the subject image data and the subject examination data. Therefore, the prediction unit 2010 predicts the user examination data based on the data set including the combination of the subject image data and the subject examination data, and the user image data P1.

The prediction unit 2010 inputs each pixel value of the pixels constituting the user image data P1 to the input layer of the neural network-like network used in the deep learning. In the output layer of the neural network-like network, values corresponding to the input pixel values, as well as weights and biases can be output. The value output by the output layer is pre-established with the score about the eye state.

The prediction unit 2010 supplies the prediction result A1 to the prediction result output unit 2020 .

Step S130 : The prediction result output unit 2020 outputs the prediction result A1 to the imaging device 1 . Here, the prediction result output unit 2020 transmits the prediction result A1 to the imaging device 1 via the inspection device communication unit 220 .

As described above, the inspection apparatus 2 ends the prediction process.

[Summary of the first embodiment] As described above, the inspection apparatus 2 of the first embodiment includes the image data acquisition unit 2000 and the prediction unit 2010 .

The image data acquisition unit 2000 acquires user image data P1 , which is an eye image of the user U1 captured by a digital camera (the photographing device 1 in the first embodiment).

The prediction unit 2010 is based on a data set that includes subject image data, which is an image of the subject's eyes captured by a digital camera, and at least one or more ( The data set (in the first embodiment, the learning result L10) and the user image data P1, which are the combined data set of the inspection data of the examinee of the inspection result of the eye state in the first embodiment, are predicted to contain the data of the user U1. User inspection data of the results when at least one of the tear state and the corneal and conjunctiva state (the eye state in the first embodiment) is inspected.

With this configuration, the inspection apparatus 2 according to the first embodiment can predict the results of the inspection including at least one of the tear state and the corneal conjunctiva state of the user U1 based on the eye image of the user U1 captured by the digital camera. The user checks the data, so the eye condition can be checked by a simple check. In the first embodiment, the so-called eye state is at least one of the tear state and the corneal conjunctiva state. In the inspection apparatus 2, the eye state of the user U1 can be inspected by a simple inspection by using a digital camera attached to a smartphone or the like to photograph the eyes of the user U1 without using an eye inspection device.

In addition, in the inspection apparatus 2 of the first embodiment, the eye image of the subject and the eye image of the user U1 are images obtained by photographing the eye in an unstained state, respectively.

With this configuration, in the inspection apparatus 2 according to the first embodiment, a digital camera attached to a smartphone or the like is used to photograph the eyes of the user U1 without using a chemical for dyeing, thereby making it possible to perform a simple inspection. Check eye status.

In addition, in the inspection apparatus 2 of the first embodiment, the eye image of the subject and the eye image of the user U1 are images obtained by photographing the eyes in a state satisfying predetermined imaging conditions, respectively.

With this configuration, in the inspection apparatus 2 of the first embodiment, the accuracy of predicting user inspection data can be improved compared to the case where the predetermined imaging conditions are not satisfied.

In addition, in the inspection apparatus 2 according to the first embodiment, the eye image of the subject and the eye image of the user U1 are images obtained by clipping the portion of the eye where the white eyeball and the black eyeball are combined.

With this configuration, in the inspection apparatus 2 according to the first embodiment, it is possible to predict the eye image by excluding the feature quantities of the parts of the eye other than the eyeball such as the eyelid and the skin from the eye image of the subject and the eye image of the user U1. Since the user examines the data, the accuracy of predicting the user's inspection data can be improved compared to the case where the portion of the eye that is composed of the white eyeball and the black eyeball is not clipped.

In addition, in the inspection apparatus 2 of the first embodiment, the prediction unit 2010 predicts the user inspection data based on the learning result L10 based on the image data of the examinee and the user image data P1, and the learning result L10 is based on the image data of the examinee and the examinee. The data set of the combination of inspection data is obtained by learning the relationship between the image data of the subject and the inspection data of the subject by deep learning. With this configuration, in the inspection apparatus 2 of the first embodiment, since prediction is performed based on the learning result L10 learned by deep learning, the accuracy of prediction can be improved compared with the case where deep learning is not used.

Furthermore, in the first embodiment, an example of the case where the imaging device 1 and the inspection device 2 are included in the inspection system E1 as separate bodies has been described, but the present invention is not limited to this. The imaging device 1 and the inspection device 2 may also be an integrated device.

In addition, in the first embodiment, an example of the case where the imaging device 1 and the inspection device 2 communicate by wireless communication has been described, but the present invention is not limited to this. The photographing device 1 and the inspection device 2 can also communicate through wired communication.

The imaging device 1 may be provided with a part of the functions of the inspection device 2 . For example, the prediction unit 2010 may be included in the photographing apparatus 1 . In addition, the learning result L10 may also be stored in the photographing device 1 .

The inspection device 2 may be provided with a part of the functions of the imaging device 1 . For example, the inspection apparatus 2 may include an imaging condition determination unit 1010, a white eyeball and black eyeball portion extraction unit 1020, and the like. In the first embodiment, the photographing device 1 includes the photographing condition determining unit 1010 and the white eyeball and black eyeball part extraction unit 1020, and the photographing device 1 is used to perform the processing of determining photographing conditions and to cut out the white eyeball and the black eyeball. part of the processing. Therefore, the load of the inspection apparatus 2 (ie, the server) accompanying these processes can be reduced.

Furthermore, in the first embodiment, an example in which the photographing device 1 is a smartphone has been described, but the present invention is not limited to this. The photographing device 1 may be a terminal device such as a personal computer (Personal Computer: PC) that is installed in a store such as a grocery store or a pharmacy and includes a digital camera. In addition, the photographing device 1 may also be a tablet terminal including a digital camera. In addition, the photographing device 1 may be a game machine including a digital camera.

In addition, the photographing device 1 can also photograph the eyes of the user U1 while the user U1 is using the photographing device 1 , and based on the prediction result A1 , display an alarm or a warning sound to notify the eyestrain condition. In this case, the imaging device 1 images the eyes of the user U1 at a predetermined cycle.

(Variation of the first embodiment) Modifications of the first embodiment will be described with reference to FIGS. 28 to 31 . In the first embodiment, the case where the eye state is predicted based on the user's eye image captured by the digital camera, that is, the user's image data, has been described. In the first embodiment, the case where the eye state is predicted based on the user's image data and based on the user's answer to the questionnaire will be described. The imaging device of the first embodiment is referred to as imaging device 1a, and the inspection device is referred to as inspection device 2a. In addition, the same code|symbol is attached|subjected to the same structure as the said 1st Example, and the description is abbreviate|omitted about the same structure and operation.

The configuration of the photographing device 1a is the same as that of the photographing device 1 of the first embodiment except that the photographing device 1a acquires the user's answer result Q1, and thus the detailed description of the structure is omitted.

The photographing apparatus 1a acquires the user answer result Q1 by accepting the operation of the user U1 by the operation accepting unit 140 . The user answer result Q1 is the answer result of the user U1 to the questionnaire. This questionnaire is displayed, for example, on a photographing screen. This questionnaire includes, for example, questions related to subjective symptoms of the eyes of the user U1. In this questionnaire, for example, the subjective symptoms of the eyes of the user U1 when the eyes of the user U1 are photographed with the photographing device 1a are selected for reporting.

FIG. 28 is a diagram showing an example of the configuration of the inspection apparatus 2a according to this modification. When the inspection device 2 a ( FIG. 28 ) of the present modification is compared with the inspection device 2 ( FIG. 22 ) of the first embodiment, the inspection device control unit 200 a and the learning result L10 a are different. Here, other structural units have the same functions as the first embodiment. The description of the same functions as those of the first embodiment will be omitted, and in this modification, the description will be focused on the parts different from those of the first embodiment.

The inspection apparatus control unit 200a includes an image data acquisition unit 2000, a prediction unit 2010a, a prediction result output unit 2020, a learning unit 2030a, and an answer result acquisition unit 2040a. The learning unit 2030a learns the relationship between the subject's answer result, the subject's image data, and the subject's examination data. The result of the examinee's response is the result of the examinee's answer to the questionnaire.

The learning unit 2030a uses deep learning in learning. To the neural-like network used in the deep learning, each pixel value of the pixels constituting the subject's image data and the subject's answer result are input together.

The answer result acquisition unit 2040 a acquires the user answer result Q1 from the photographing device 1 . The prediction unit 2010a predicts the user inspection data based on the learning result L10a, the user answer result Q1, and the user image data P1. The prediction unit 2010a performs prediction based on deep learning.

The storage unit 210 stores the learning result L10a. The learning result L10a is based on the second data set of the examinee including the combination of the examinee's answer result, the examinee's image data, and the examinee's examination data, and the examinee's answer result and the examinee are learned by deep learning The results obtained from the relationship between the subject's image data and the subject's inspection data. That is, the learning result L10a refers to a result of learning based on the second data set of the examinee including a combination of the examinee's answer result, the examinee image data, and the examinee's examination data. Therefore, the predicting unit 2010a predicts the user based on the second data set of the examinee, the user image data P1, and the user answer result Q1, which includes the combination of the examinee's answer result, the examinee's image data, and the examinee's examination data. Check the information.

Before the photographing apparatus 1a outputs the user image data P1 and the user answer result Q1 to the inspection apparatus 2, the photographing screen G4 shown in FIG. 29 is displayed.

On the photographing screen G4, the user image data P2 and the questionnaire are displayed together.

The prediction result display screen G6 shown in FIG. 30 and the history screen G7 shown in FIG. 31 are displayed in order to present the prediction result A1 acquired by the imaging apparatus 1 a from the inspection apparatus 2 .

The prediction result display screen G6 shown in FIG. 30 displays suggestions for improving the eye condition based on the prediction result A1.

In the history screen G7 shown in FIG. 31, the history of the score regarding an eye state is displayed by a graph.

The inspection apparatus 2a of the present modification includes an answer result acquisition unit 2040a. The answer result acquisition unit 2040a acquires the user answer result Q1 which is the answer result of the user U1 to the questionnaire. The prediction unit 2010a is based on the second data set of the examinee, the user image data P1, As well as the user answer result Q1, predict the user inspection data.

With this configuration, in the inspection apparatus 2a of the present modification, the accuracy of prediction can be improved based on the user's answer result Q1. In addition, in the inspection apparatus 2a, a suggestion to the user U1 for improving the eye condition can be generated based on the prediction result A1.

Furthermore, in the first embodiment and the modified example, the examinee and the user U1 may be the same person. In the case where the subject and the user U1 are the same person, the subject image data is the eye of the user U1 captured by the digital camera before the image of the eye of the user U1 used as the user image data P1 was captured image. In addition, when the examinee and the user U1 are the same person, the examinee's answer result is the user U1's answer to the questionnaire obtained before the user U1's answer result to the questionnaire is obtained as the user's answer result Q1 result.

When the examinee and the user U1 are the same person, the inspection apparatus 2 or the inspection apparatus 2a can improve the accuracy of predicting the user's examination data compared with the case where the examinee and the user U1 are not the same person.

When the examinee and the user U1 are the same person, the examination apparatus 2 or the examination apparatus 2a appropriately uses the user U1 as a patient for remote treatment and/or remote diagnosis.

The user U1 who is a patient visits an ophthalmology department more than once, and acquires and accumulates image data and examination data (data set) of his own eyes. Even if the user U1 does not go to the hospital afterward, the tear cornea state can be determined only by taking an image of his own eye with a digital camera included in the photographing device 1 (eg, a smartphone) and sending the image to an ophthalmologist. By using the examination apparatus 2 or the examination apparatus 2a, the user U1 can reduce the burden of going to the hospital in a long distance or the like. In addition, the user U1 can observe the progress of the symptoms by himself without sending his own eye image to the ophthalmologist. Therefore, it is only necessary for the user U1 to go to the hospital when the symptoms worsen, so that medical expenses can be reduced.

Furthermore, the photographing device 1 , the photographing device 1 a or the inspection device 2 , and a part of the inspection device 2 a in the first embodiment can also be realized by using a computer, for example, the photographing device control unit 100 and the inspection device control unit 200 , and the inspection device control unit. Section 200a. In this case, it can be achieved by recording a program for realizing the control function in a computer-readable recording medium, and causing a computer system to read and execute the program recorded in the recording medium. In addition, the so-called "computer system" mentioned here is a computer system built in the photographing device 1, the photographing device 1a, or the inspection device 2, and the inspection device 2a, and is assumed to include an operating system (OS), peripherals. equipment and other hardware. The term "computer-readable recording medium" refers to portable media such as floppy disks, optical disks, ROMs, compact discs (CDs)-ROMs, and storage devices such as hard disks built into computer systems. Furthermore, the so-called "computer-readable recording medium" may also include those that dynamically hold the program for a short period of time, such as the communication line when the program is transmitted via a network such as the Internet or a communication line such as a telephone line; The volatile memory inside the computer system of the server or client below generally holds the program for a certain period of time. In addition, the program may be a program for realizing a part of the function, or a program that realizes the function by combining with a program already recorded in the computer system.

In addition, part or all of the imaging device 1 , imaging device 1 a , inspection device 2 , and inspection device 2 a in the first embodiment may be implemented in the form of an integrated circuit such as an LSI (Large Scale Integrated Circuit). Each functional block of the imaging device 1 , the imaging device 1 a , the inspection device 2 , and the inspection device 2 a may be individually processed, or a part or all of them may be integrated and processed. In addition, the method of integrating the circuit is not limited to the LSI, but can also be achieved by using a dedicated circuit or a general-purpose processor. In addition, in the case where a technology for forming an integrated circuit in place of the LSI has appeared due to the advancement of the semiconductor technology, an integrated circuit based on this technology can also be used.

Next, a specific example of the second example of the above-described embodiment will be described with reference to FIGS. 32 to 38 .

FIG. 32 is a diagram showing an example of the hardware configuration of the machine learning execution device according to the second embodiment. The machine learning execution device 10b shown in FIG. 32 is a device that causes the machine learning device 700b to execute machine learning in a learning phase of the machine learning device 700b to be described later. In addition, as shown in FIG. 32, the machine learning execution device 10b includes a processor 11b, a main storage device 12b, a communication interface 13b, an auxiliary storage device 14b, an input and output device 15b, and a bus bar 16b.

The processor 11b is, for example, a CPU, and reads out and executes the machine learning execution program 100b described later to achieve each function of the machine learning execution program 100b. In addition, the processor 11b may read out and execute programs other than the machine learning execution program 100b, so as to achieve necessary functions in addition to the functions of the machine learning execution program 100b.

The main storage device 12b is, for example, a RAM, and stores in advance a machine learning execution program 100b and other programs that are read and executed by the processor 11b.

The communication interface 13b is an interface circuit for performing communication with the machine learning device 700b and other devices via the network NW shown in FIG. 32 . In addition, the network NW is, for example, a LAN or an intranet.

The auxiliary storage device 14b is, for example, a hard disk drive, a solid state drive, a flash memory, and a ROM.

The input/output device 15b is, for example, an input/output port. The input/output device 15b is connected to, for example, a keyboard 811b, a mouse 812b, and a display 910b shown in FIG. 32 . The keyboard 811b and the mouse 812b are used, for example, for inputting data necessary for operating the machine learning execution device 10b. The display 910b, for example, displays a graphical user interface of the machine learning execution device 10b.

The bus bar 16b connects the processor 11b, the main storage device 12b, the communication interface 13b, the auxiliary storage device 14b, and the input/output device 15b, so that these can exchange data with each other.

FIG. 33 is a diagram showing an example of the software configuration of the machine learning execution program of the second embodiment. The machine learning execution device 10b uses the processor 11b to read out and execute the machine learning execution program 100b, so as to achieve the teacher data acquisition function 101b and the machine learning execution function 102b shown in FIG. 33 .

The teacher data acquisition function 101b acquires teacher data in which one image data for learning and congestion data for learning are used as questions, and at least one of one image data for learning and inspection result data for learning is an answer.

The image data for learning is a part of the problem constituting the teacher data, and represents the image for learning in which the eyes of the subject for learning are drawn. The learning image is captured by, for example, a camera mounted on a smartphone.

The hyperemia data for learning is a part of the questions constituting the teacher data, and is data showing the degree of hyperemia of the eyes of the subject for learning.

For example, the hyperemia data for learning may represent a value that is calculated based on the image data for learning and expresses the degree of hyperemia of the eyes of the subject for learning. Specifically, the hyperemia data for learning can represent a value obtained by converting an image for learning in which the eyes of the subject for learning are drawn in color into an image of only a green array, and assigning it to the image A value corresponding to the area of the region in which the blood vessels of the conjunctiva are drawn in white in the grayscale image in which the luminance of each of the included pixels is inverted.

Furthermore, the value can also be calculated using an image generated by clipping the region delineating the conjunctiva from the grayscale image prior to evaluating the area. This value is a value indicating that the larger the area of the area where the blood vessels of the conjunctiva are drawn, the greater the degree of hyperemia of the eye, and the smaller the area of the area where the blood vessels of the conjunctiva are drawn, the less the degree of hyperemia of the eye.

Alternatively, the hyperemia data for learning may represent a value that is input using a user interface and expresses the degree of hyperemia of the eyes of the subject for learning. The user interface is displayed, for example, on the display 910b shown in FIG. 32 . Alternatively, the user interface is displayed on a touch panel display mounted in a smartphone contracted by the subject for learning.

The examination image data for learning is a part of the answer that constitutes the teacher data, and it is used for learning that depicts the eyes of the subject for learning when an examination related to the symptoms of dry eye is performed on the eyes of the subject for learning. Check the profile of the image. Such tests include, for example, an inspection of the degree of corneal and conjunctival epithelial damage using a luciferin staining reagent, and an inspection of the degree of mucin damage using a lissamine green staining reagent. Therefore, for example, the examination image data for learning is data representing an image in which an eye stained with luciferin or lissamine green is drawn.

The test result data for learning is data that constitutes a part of the answers of the teacher data and shows test results related to symptoms of dry eye. As such test results, for example, a numerical value of 0 or more and 1 or less, which expresses the degree of corneal and conjunctival epithelial damage based on the test using a luciferin staining reagent, can be mentioned. Alternatively, as such an inspection result, for example, a numerical value of 0 or more and 1 or less, which expresses the degree of mucin damage based on an inspection using a lissamine green staining reagent, can be mentioned.

When the two values are 0 or more and less than 0.5, the area of corneal and conjunctival epithelial lesions or mucin lesions stained by luciferin staining reagent or Lissamine green staining reagent is less than 30% of the entire area of the eye, so It appears that the eyes of the subject for study are normal, and examination by an ophthalmologist is not required. In addition, when the two numerical values are 0.5 or more and 1 or less, the area of corneal and conjunctival epithelial damage or mucin damage stained with luciferin staining reagent or lissamine green staining reagent is 30% of the entire area of the eye As described above, the eyes of the subject for learning are abnormal, and therefore, an ophthalmologist is required to conduct an examination.

For example, the teacher profile acquisition function 101b acquires 1280 teacher profiles. In this case, the 1280 image data for learning represent 1280 images obtained by performing four sets of processing of photographing the two eyes of 8 subjects for learning 20 times using a camera mounted on a smartphone. Imagery for learning. In addition, in this case, the 1280 pieces of hyperemia data for learning represent 1280 values each of which is calculated based on the image data for learning and represents the degree of hyperemia of the eyes of the subject for learning.

In addition, in this case, the 1280 pieces of examination image data for learning respectively indicate when the examination related to the symptoms of dry eye is performed on the eyes of the examination subject for learning depicted in each of the 1280 pieces of learning images. The image of the examination for learning was taken. In addition, the 1280 pieces of examination result data for learning represent 1280 numerical values of 0 or more and 1 or less, which respectively represent the examination results related to the symptoms of dry eye performed when 1280 examination images for learning were taken.

In addition, in the following description, the following case is taken as an example for description: 1280 pieces of examination result data for learning include 640 examinations for learning which show that the eyes of the examination subject for learning are normal and have a value of 0 or more and less than 0.5 Result data, and 640 test result data for learning showing a numerical value of 0.5 or more and 1 or less indicating abnormality in the eyes of the subject for learning.

The machine learning execution function 102b inputs the teacher data into the machine learning program 750b installed in the machine learning device 700b, and causes the machine learning program 750b to learn. For example, the machine learning execution function 102b causes a machine learning program 750b including a convolutional neural network to learn by backpropagation.

For example, the machine learning execution function 102b inputs the following 560 teacher data into the machine learning program 750b, that is, the 560 teacher data includes a value of 0 or more and less than 0.5, which indicates that the eyes of the subject for performance learning are normal. Study test results data. The 560 teacher profiles are teacher profiles obtained from 7 selected from the 8 persons.

In addition, for example, the machine learning execution function 102b inputs the following 560 teacher data into the machine learning program 750b, that is, the 560 teacher data includes 0.5 or more and 1 or less, indicating that the eyes of the learning subject are abnormal. Numerical learning test result data. The 560 teacher profiles are teacher profiles obtained from 7 selected from the 8 persons.

Then, the machine learning execution function 102b uses the 1080 teacher data to make the machine learning program 750b learn.

In addition, for example, the machine learning execution function 102b can also input the following 80 teacher data as test data into the machine learning program 750b, that is, the 80 teacher data show that the eyes of the subject for learning are normal and are not used for machine learning The learning of the learning program 750b is in progress. The 80 teacher profiles are teacher profiles obtained from 1 person who was not selected as the 7 persons.

In addition, for example, the machine learning execution function 102b can also input the following 80 teacher data as test data into the machine learning program 750b, that is, the 80 teacher data represent abnormal eyes of the subject for learning and are not used for machine learning The learning of the learning program 750b is in progress. The 80 teacher profiles are teacher profiles obtained from 1 person who was not selected as the 7 persons.

Thereby, the machine learning execution function 102b can evaluate the characteristics of the machine learning program 750b obtained by learning the machine learning program 750b using the 1080 teacher data.

Furthermore, the machine learning execution function 102b may use, for example, Class Activation Mapping (CAM: Class Activation Mapping) when evaluating the characteristics of the machine learning program 750b using the test data. Class activation mapping is a technique for partially clarifying the data input to the neural network as the basis for the output of the neural network. Examples of the class activation map include gradient weighted class activation maps. Gradient-weighted class activation mapping is a technique that utilizes the gradient of the classification score relative to the features of the convolution performed by the convolutional neural network to determine that the image input to the convolutional neural network gives a certain the area of influence above the degree.

34 is a diagram showing an example of a portion to be considered in an eye image of a subject for learning when predicting an examination result related to corneal epithelial damage by the machine learning program of the second embodiment. For example, the machine learning execution function 102b uses a gradient weighted class activation map, and evaluates the comparison of the area surrounded by the ellipse C36 shown in FIG. 34 in the learning image shown in FIG. 34 with the cornea by the machine learning program 750b Prediction of test results related to epithelial damage gave more than a certain degree of influence. In addition, the gray scales of the three stages included in the area enclosed by the ellipse C36 shown in FIG. 34 indicate the degree of influence given to the prediction of the test result related to the corneal epithelial damage, and all of them are superimposed and displayed on the drawing of the test subject for learning. on the region of the blood vessels of the conjunctiva of the body of the eye.

Next, an example of processing executed by the machine learning execution program 100b of the second embodiment will be described with reference to FIG. 35 . FIG. 35 is a flowchart showing an example of processing executed by the machine learning execution program of the second embodiment. The machine learning execution program 100b executes the processing shown in FIG. 35 at least once.

In step S31, the teacher data acquisition function 101b acquires teacher data with at least one of the learning image data and the learning hyperemia data as the question and at least one of the learning inspection image data and the learning inspection result data as the answer.

In step S32, the machine learning execution function 102b inputs the teacher data into the machine learning program 750b, and makes the machine learning program 750b learn.

36 and 37 , specific examples of the dry eye syndrome inspection program, the dry eye syndrome inspection apparatus, and the dry eye syndrome inspection method according to the second embodiment will be described.

FIG. 36 is a diagram showing an example of the hardware configuration of the dry eye disease inspection apparatus according to the second embodiment. The dry eye disease inspection apparatus 20b shown in FIG. 36 uses the machine learning apparatus 700b in the inference stage of the machine learning apparatus 700b that has been learned by the machine learning execution program 100b to infer the dry eye that appears in the eyes of the subject for inference symptomatic device. In addition, as shown in FIG. 36 , the dry eye disease inspection device 20b includes a processor 21b, a main storage device 22b, a communication interface 23b, an auxiliary storage device 24b, an input/output device 25b, and a bus bar 26b.

The processor 21b is, for example, a CPU, and reads out and executes the dry eye syndrome inspection program 200b described later, so as to achieve each function of the dry eye syndrome inspection program 200b. In addition, the processor 21b may read out and execute programs other than the dry eye disease inspection program 200b, so as to achieve necessary functions in addition to the respective functions possessed by the dry eye disease inspection program 200b.

The main storage device 22b is, for example, a RAM, and stores in advance the dry eye disease examination program 200b and other programs read and executed by the processor 21b.

The communication interface 23b is an interface circuit for performing communication with the machine learning apparatus 700b and other devices via the network NW shown in FIG. 36 . In addition, the network NW is, for example, a LAN or an intranet.

The auxiliary storage device 24b is, for example, a hard disk drive, a solid state drive, a flash memory, and a ROM.

The input/output device 25b is, for example, an input/output port. The input/output device 25b is connected to, for example, a keyboard 821b, a mouse 822b, and a display 920b shown in FIG. 36 . The keyboard 821b and the mouse 822b are used, for example, in the operation of inputting data necessary to operate the dry eye disease inspection apparatus 20b. The display 920b displays, for example, a graphical user interface of the dry eye disease inspection apparatus 20b.

The bus bar 26b connects the processor 21b, the main storage device 22b, the communication interface 23b, the auxiliary storage device 24b and the input/output device 25b, so that these can send and receive data to and from each other.

FIG. 37 is a diagram showing an example of a software configuration of the dry eye syndrome examination program according to the second embodiment. The dry eye disease inspection apparatus 20 uses the processor 21b to read out and execute the dry eye disease inspection program 200b, so as to achieve the data acquisition function 201b and the symptom estimation function 202b shown in FIG. 37 .

The data acquisition function 201b acquires image data for inference and congestion data for inference.

The image data for inference is data representing an image in which the eyes of the subject for inference are drawn. The inference image is captured using, for example, a camera mounted on a smartphone.

The hyperemia data for inference are data showing the degree of hyperemia of the eyes of the subject for inference. For example, the hyperemia data for inference can represent a value that is calculated based on the image data for inference and expresses the degree of hyperemia of the eye of the subject for inference. Specifically, the hyperemia data for inference can represent a value in which an inference image in which an eye of a subject for inference is drawn in color is converted into an image of only a green array and assigned to the image A value corresponding to the area of the region in which the blood vessels of the conjunctiva are drawn in white in the grayscale image in which the luminance of each of the included pixels is inverted.

Furthermore, the value can also be calculated using an image generated by clipping the region delineating the conjunctiva from the grayscale image prior to evaluating the area. This value is a value indicating that the larger the area of the area where the blood vessels of the conjunctiva are drawn, the greater the degree of hyperemia of the eye, and the smaller the area of the area where the blood vessels of the conjunctiva are drawn, the less the degree of hyperemia of the eye.

Alternatively, the hyperemia data for inference may represent a value that is input using the user interface and expresses the degree of hyperemia of the eye of the subject for inference. The user interface is displayed, for example, on the display 920b shown in FIG. 36 . Alternatively, the user interface is displayed on a touch panel display mounted on a smartphone contracted by the subject for inference.

The symptom inference function 202a inputs the image data for inference and the hyperemia data for inference into the machine learning program 750 that has been learned by the machine learning executive function 102b, and causes the machine learning program 750b to infer the dryness appearing in the eyes of the subject for inference. Symptoms of eye disease. For example, the symptom inference function 202b inputs the two data into the machine learning program 750b to infer a numerical value representing the degree of corneal and conjunctival epithelial damage occurring in the eye of the subject for inference.

Then, the symptom estimating function 202b causes the machine learning program 750b to output symptom data indicating the symptoms of dry eye syndrome appearing in the eyes of the subject for inference. For example, the symptom estimation function 202b causes the machine learning program 750b to output symptom data representing a numerical value representing the degree of corneal and conjunctival epithelial damage occurring in the eye of the subject for estimation. In this case, the symptom data is used to display on the display 920b, for example, a numerical value that expresses itself and expresses the degree of corneal and conjunctival epithelial damage occurring in the eye of the subject for inference.

Next, with reference to FIG. 38 , an example of the processing executed by the dry eye syndrome examination program 200 b of the second embodiment will be described. FIG. 38 is a flowchart showing an example of processing performed by the dry eye syndrome examination program of the second embodiment.

In step S41, the data acquisition function 201b acquires the image data for inference and the hyperemia data for inference.

In step S42, the symptom inference function 202b inputs the image data for inference and the hyperemia data for inference into the machine learning program 750b that has already been learned to infer the symptoms of dry eye appearing in the eyes of the subject for inference, and The machine learning program 750b is caused to output symptom data indicating the symptoms of dry eye syndrome appearing in the eyes of the subject for inference.

The machine learning execution program, the dry eye disease inspection program, the machine learning execution device, the dry eye disease inspection device, the machine learning execution method, and the dry eye disease inspection method of the second embodiment have been described above.

The machine learning execution program 100b includes a teacher data acquisition function 101b and a machine learning execution function 102b.

The teacher data acquisition function 101b acquires teacher data in which the image data for learning and the hyperemia data for learning are the questions, and the answer is at least one of the image data for learning and the inspection result data for learning. The image data for learning is data representing an image in which the eyes of the subject for learning are drawn. The hyperemia data for learning are data showing the degree of hyperemia of the eyes of the subject for learning. The examination image data for learning is data representing an image in which the eyes of the subject for learning are drawn when an examination related to the symptoms of dry eye is performed on the eyes of the subject for learning. The test result data for learning are data showing test results related to symptoms of dry eye.

The machine learning execution function 102b inputs the teacher data into the machine learning program 750b, and causes the machine learning program 750b to learn.

Thereby, the machine learning execution program 100b can generate the machine learning program 750b that predicts the test results related to the symptoms of dry eye based on the learning image data and the learning hyperemia data.

In addition, the machine learning execution program 100b causes the machine learning program 750b to learn using the teacher data including not only the learning image data but also the learning hyperemia data as questions. Therefore, the machine learning execution program 100b can generate the machine learning program 750b capable of predicting the test results related to the symptoms of dry eye more accurately.

In addition, the machine learning execution program 100b acquires teacher data including hyperemia data for learning representing the degree of hyperemia of the eyes of the subject for learning calculated based on the image data for learning as a part of the problem value. Thereby, the machine learning execution program 100b can save the labor of visually checking the degree of hyperemia of the eyes of the learning subject and others.

In addition, the machine learning execution program 100b acquires teacher data that uses the hyperemia data for learning input using the user interface as a part of the problem. Thereby, the machine learning execution program 100b can omit the process of calculating the value representing the degree of hyperemia of the eyes of the subject for learning based on the image data for learning.

The dry eye syndrome examination program 200b includes a data acquisition function 201b and a symptom estimation function 202b.

The data acquisition function 201b acquires image data for inference and congestion data for inference. The image data for inference is data representing an image in which the eyes of the subject for inference are drawn. The hyperemia data for inference are data showing the degree of hyperemia of the eyes of the subject for inference.

The symptom inference function 202b inputs the inference image data and the inference hyperemia data to the machine learning program 750b that has been learned by the machine learning execution program 100b to infer the symptoms of dry eye appearing in the eyes of the inference subject. Then, the symptom estimating function 202b causes the machine learning program 750b to output symptom data indicating the symptoms of dry eye syndrome appearing in the eyes of the subject for inference.

Thereby, the dry eye disease examination program 200b can predict the examination result related to the symptoms of dry eye disease without actually carrying out the examination related to the symptoms of dry eye disease.

In addition, the dry eye examination program 200b acquires hyperemia data for inference, which represents a value that is calculated based on the image data for inference and expresses the degree of hyperemia of the eye of the subject for inference. Thereby, the dry eye examination program 200b can save labor for visually checking the degree of hyperemia of the eyes of the inference subject and others.

In addition, the dry eye examination program 200b acquires the hyperemia data for inference input using the user interface. Thereby, the dry eye examination program 200b can omit the process of calculating the value representing the degree of hyperemia of the eye of the subject for inference based on the image data for inference.

Next, an example in which the machine learning program 750b is actually learned to infer the symptoms of dry eye disease is given, and the effect of executing the program 100b by machine learning is described by comparing a comparative example with the example of the second embodiment. specific example.

First, a specific example of the effect obtained by executing the program 100b by machine learning in the case where the test is performed using the luciferin staining reagent will be described.

In the comparative example in this case, the data obtained by removing the hyperemia data for learning from the 560 teacher data showing normal eyes of the subject for learning and the 560 teacher data showing abnormal eyes of the subject for learning Examples of teacher data to make machine learning programs learn.

With regard to the machine learning program that has been learned in this way, if the characteristics are evaluated using the 80 test data representing the normal eye of the subject for learning and the 80 test data representing the abnormal eye of the subject for learning, the results are shown as follows: The prediction accuracy was 73% and the false negative rate was 34%.

On the other hand, the machine learning program 750b learned by the machine learning execution program 100b showed that the prediction accuracy was 78% and the false negative rate was 11% when the characteristics were evaluated using the same test data. This characteristic is superior to that of the machine learning program of the comparative example.

In addition, the prediction accuracy is the number of sheets X predicted to be abnormal by the machine learning program among the learning images included in the abnormal group, and the number of sheets predicted to be normal by the machine learning program among the learning images included in the normal group. The ratio of the total X+Y of the number Y to the total number of test data. Therefore, in the case of this comparative example, the prediction accuracy is calculated using ((X+Y)/160)×100.

In addition, the false negative rate is the ratio of the number of images for learning included in the abnormal group that are predicted to be normal by the machine learning program to the total number of images for learning included in the abnormal group. Therefore, in the case of this comparative example, the false negative rate was calculated by ((80-X)/80)×100.

Second, a specific example of the effect obtained by executing the program 100b by machine learning when an inspection is performed using the lissamine green staining reagent will be described.

In the comparative example in this case, the data obtained by removing the hyperemia data for learning from the 540 teacher data showing normal eyes of the subject for learning and the 580 teacher data showing abnormal eyes of the subject for learning Examples of teacher data to make machine learning programs learn.

With regard to the machine learning program that has been learned in this way, if the characteristics are evaluated using the 80 test data representing the normal eye of the subject for learning and the 80 test data representing the abnormal eye of the subject for learning, the results are shown as follows: The prediction accuracy was 73% and the false negative rate was 34%.

On the other hand, the machine learning program 750b learned by the machine learning execution program 100b showed that the prediction accuracy was 78% and the false negative rate was 11% when the characteristics were evaluated using the same test data. This characteristic is superior to that of the machine learning program of the comparative example.

Furthermore, at least a part of the functions of the machine learning execution program 100b can be realized by hardware including a circuit portion. Likewise, at least a part of the functions of the dry eye syndrome examination program 200b can be realized by hardware including a circuit unit. In addition, such hardware is, for example, LSI, ASIC, FPGA, and GPU.

In addition, at least a part of the functions of the machine learning execution program 100b can also be achieved by the cooperation of software and hardware. Similarly, at least a part of the functions of the dry eye syndrome checking program 200b can also be achieved by the cooperation of software and hardware. In addition, these hardwares may be integrated into one, or may be divided into multiple pieces.

In the second embodiment, the case where the machine learning execution device 10b, the machine learning device 700b, and the dry eye disease inspection device 20b are independent devices has been described as an example, but the present invention is not limited to this. The devices can also be implemented as one device.

Next, a specific example of the third example of the above-described embodiment will be described with reference to FIGS. 39 to 44 .

First, with reference to FIGS. 39 and 40 , specific examples of the machine learning execution program, the machine learning execution device, and the machine learning execution method of the third embodiment will be described. Different from the machine learning execution program, machine learning execution device, and machine learning execution method of the second embodiment, the machine learning execution program, machine learning execution device, and machine learning execution method of the third embodiment use the answer data for learning to be described later. Teacher data for non-learning use hyperemia data as questions.

FIG. 39 is a diagram showing an example of the hardware configuration of the machine learning execution device according to the third embodiment. The machine learning execution device 10c shown in FIG. 39 is a device that causes the machine learning device 700c to execute machine learning in a learning phase of the machine learning device 700c to be described later. In addition, as shown in FIG. 39, the machine learning execution device 10c includes a processor 11c, a main storage device 12c, a communication interface 13c, an auxiliary storage device 14c, an input/output device 15c, and a bus bar 16c.

The processor 11c is, for example, a CPU, and reads out and executes the machine learning execution program 100c described later to achieve each function of the machine learning execution program 100c. In addition, the processor 11c may read out and execute programs other than the machine learning execution program 100c, so as to achieve necessary functions in addition to the functions of the machine learning execution program 100c.

The main storage device 12c is, for example, a RAM, and stores in advance a machine learning execution program 100c and other programs that are read and executed by the processor 11c.

The communication interface 13c is an interface circuit for performing communication with the machine learning device 700c and other devices via the network NW shown in FIG. 39 . In addition, the network NW is, for example, a LAN or an intranet.

The auxiliary storage device 14c is, for example, a hard disk drive, a solid state drive, a flash memory, and a ROM.

The input/output device 15c is, for example, an input/output port. The input/output device 15c is connected to, for example, a keyboard 811c, a mouse 812c, and a display 910c shown in FIG. 1 . The keyboard 811c and the mouse 812c are used, for example, for inputting data necessary for operating the machine learning execution device 10c. The display 910c, for example, displays a graphical user interface of the machine learning execution device 10c.

The bus bar 16c connects the processor 11c, the main storage device 12c, the communication interface 13c, the auxiliary storage device 14c and the input/output device 15c, so that these can transmit and receive data with each other.

FIG. 40 is a diagram showing an example of the software configuration of the machine learning execution program of the third embodiment. The machine learning execution device 10c uses the processor 11c to read out and execute the machine learning execution program 100c, so as to achieve the teacher data acquisition function 101c and the machine learning execution function 102c shown in FIG. 40 .

The teacher data acquisition function 101c acquires teacher data with one image data for learning and answer data for learning as a question, and at least one of one image data for learning and inspection result data for learning as an answer.

The image data for learning is a part of the problem constituting the teacher data, and represents the image for learning in which the eyes of the subject for learning are drawn. The learning image is captured by, for example, a camera mounted on a smartphone.

The response data for learning is data showing the results of the responses to the inquiries about the subjective symptoms of the eyes possessed by the subject for learning.

For example, as the question represented by the learning response data, for example, the learning subject "blurred eyes", "dazzled eyes", "difficulty opening eyes", "feeling of foreign body in eyes", "feeling of eye discomfort" ” and other inquiries. Alternatively, as the inquiries expressed by the answer material for learning, inquiries such as "eyestrain", "dry eyes", "heavy eyes", "eye congestion", etc. may be mentioned.

In addition, for example, the answer to the inquiry about the subjective symptoms of the eyes may be selected from "0 points: no symptoms at all", "1 point: not very concerned", "2 points: slightly concerned", "3 points: concerned ” and “4 points: I care very much” in the five stages. Alternatively, the answer to the inquiry about the subjective symptoms of the eyes may be selected from both "Yes" and "No".

The examination image data for learning is a part of the answer that constitutes the teacher data, and it is used for learning that depicts the eyes of the subject for learning when an examination related to the symptoms of dry eye is performed on the eyes of the subject for learning. Check the profile of the image. Such tests include, for example, an inspection of the degree of corneal and conjunctival epithelial damage using a luciferin staining reagent, and an inspection of the degree of mucin damage using a lissamine green staining reagent. Therefore, for example, the examination image data for learning is data representing an image in which an eye stained with luciferin or lissamine green is drawn.

The test result data for learning is data that constitutes a part of the answers of the teacher data and shows test results related to symptoms of dry eye. As such test results, for example, a numerical value of 0 or more and 1 or less, which expresses the degree of corneal and conjunctival epithelial damage based on the test using a luciferin staining reagent, can be mentioned. Alternatively, as such an inspection result, for example, a numerical value of 0 or more and 1 or less, which expresses the degree of mucin damage based on an inspection using a lissamine green staining reagent, can be mentioned.

When the two values are 0 or more and less than 0.5, the area of corneal and conjunctival epithelial lesions or mucin lesions stained by luciferin staining reagent or Lissamine green staining reagent is less than 30% of the entire area of the eye, so It appears that the eyes of the subject for study are normal, and examination by an ophthalmologist is not required. In addition, when the two numerical values are 0.5 or more and 1 or less, the area of corneal and conjunctival epithelial damage or mucin damage stained with luciferin staining reagent or lissamine green staining reagent is 30% of the entire area of the eye As described above, the eyes of the subject for learning are abnormal, and therefore, an ophthalmologist is required to conduct an examination.

For example, the teacher profile acquisition function 101c acquires 1280 teacher profiles. In this case, the 1280 learning image data respectively represent 1280 obtained by performing four sets of processing of photographing the two eyes of 8 learning subjects 20 times using a camera mounted on a smartphone. Zhang learning images. In addition, in this case, the 1280 pieces of response data for learning respectively represent the results of 1280 kinds of responses to the inquiries about the subjective symptoms of the eyes possessed by the subject for learning.

In addition, in this case, the 1280 pieces of examination image data for learning respectively indicate the time when the examination related to the symptoms of dry eye is performed on the eyes of the examination subject for learning depicted in each of the 1280 pieces of learning images. The image taken for the examination of the study. In addition, the 1280 pieces of examination result data for learning represent 1280 numerical values of 0 or more and 1 or less, which respectively represent the examination results related to the symptoms of dry eye performed when 1280 examination images for learning were taken.

In addition, in the following description, the following case is taken as an example for description: 1280 pieces of examination result data for learning include 640 examinations for learning which show that the eyes of the examination subject for learning are normal and have a value of 0 or more and less than 0.5 Result data, and 640 test result data for learning showing a numerical value of 0.5 or more and 1 or less indicating abnormality in the eyes of the subject for learning.

The machine learning execution function 102c inputs the teacher data into the machine learning program 750c installed in the machine learning device 700c, and causes the machine learning program 750c to learn. For example, the machine learning execution function 102c causes a machine learning program 750c comprising a convolutional neural network to learn by backpropagation.

For example, the machine learning execution function 102c inputs the following 560 teacher data into the machine learning program 750c, that is, the 560 teacher data includes a value of 0 or more and less than 0.5, which indicates that the eyes of the subject for performance learning are normal. Study test results data. The 560 teacher profiles are teacher profiles obtained from 7 selected from the 8 persons.

In addition, for example, the machine learning execution function 102c inputs the following 560 teacher data into the machine learning program 750c, that is, the 560 teacher data includes 0.5 or more and 1 or less, indicating that the eyes of the learning subject are abnormal. Numerical learning test result data. The 560 teacher profiles are teacher profiles obtained from 7 selected from the 8 persons.

Then, the machine learning execution function 102c makes the machine learning program 750c learn by using the 1080 teacher data.

In addition, for example, the machine learning execution function 102c can also input the following 80 teacher data as test data into the machine learning program 750c, that is, the 80 teacher data show that the eyes of the subject for learning are normal, and are not used for machine learning The learning of the learning program 750c is in progress. The 80 teacher profiles are teacher profiles obtained from 1 person who was not selected as the 7 persons.

In addition, for example, the machine learning execution function 102c may also input the following 80 teacher data as test data into the machine learning program 750c, that is, the 80 teacher data express the abnormality of the eyes of the subject for learning and are not used in the machine learning program 750c. The learning of the learning program 750c is in progress. The 80 teacher profiles are teacher profiles obtained from 1 person who was not selected as the 7 persons.

Thereby, the machine learning execution function 102c can evaluate the characteristics of the machine learning program 750c obtained by learning the machine learning program 750c using the 1080 teacher data.

Furthermore, the machine learning execution function 102c may use, for example, a class activation map when evaluating the properties of the machine learning program 750c using test data. Class activation mapping is a technique for partially clarifying the data input to the neural network as the basis for the output of the neural network. Examples of the class activation map include gradient weighted class activation maps. Gradient-weighted class activation mapping is a technique that utilizes the gradient of the classification score relative to the features of the convolution performed by the convolutional neural network to determine that the image input to the convolutional neural network gives a certain the area of influence above the degree.

Next, an example of processing executed by the machine learning execution program 100c of the third embodiment will be described with reference to FIG. 41 . FIG. 41 is a flowchart showing an example of processing executed by the machine learning execution program of the third embodiment. The machine learning execution program 100c executes the processing shown in FIG. 41 at least once.

In step S51 , the teacher data acquisition function 101c acquires teacher data for which the image data for learning and the answer data for learning are the questions, and at least one of the image data for learning and the inspection result data for learning is the answer.

In step S52, the machine learning execution function 102a inputs the teacher data into the machine learning program 750c, and makes the machine learning program 750c learn.

42 and 43 , specific examples of the dry eye syndrome inspection program, the dry eye syndrome inspection apparatus, and the dry eye syndrome inspection method according to the third embodiment will be described. Unlike the dry eye inspection program, dry eye inspection device, and dry eye inspection method of the second embodiment, the dry eye inspection program, dry eye inspection device, and dry eye inspection method of the third embodiment do not acquire inferences Instead of using the hyperemia data, the answer data for the inference described later is obtained.

FIG. 42 is a diagram showing an example of the hardware configuration of the dry eye disease inspection apparatus according to the third embodiment. The dry eye disease inspection apparatus 20c shown in FIG. 42 uses the machine learning apparatus 700c to infer dry eye appearing in the eyes of the subject for inference in the inference stage of the machine learning apparatus 700c that has been learned by the machine learning execution program 100c. symptomatic device. In addition, as shown in FIG. 42 , the dry eye disease inspection device 20c includes a processor 21c, a main storage device 22c, a communication interface 23c, an auxiliary storage device 24c, an input/output device 25c, and a bus bar 26c.

The processor 21c is, for example, a CPU, and reads out and executes the dry eye syndrome inspection program 200c described later, so as to achieve each function of the dry eye syndrome inspection program 200c. In addition, the processor 21c may read out and execute programs other than the dry eye disease inspection program 200c, so as to achieve necessary functions in addition to each function of the dry eye disease inspection program 200c.

The main storage device 22c is, for example, a RAM, and stores in advance a dry eye syndrome examination program 200c and other programs read and executed by the processor 21c.

The communication interface 23c is an interface circuit for performing communication with the machine learning device 700c and other devices via the network NW shown in FIG. 42 . In addition, the network NW is, for example, a LAN or an intranet.

The auxiliary storage device 24c is, for example, a hard disk drive, a solid state drive, a flash memory, and a ROM.

The input/output device 25c is, for example, an input/output port. The input/output device 25c is connected to, for example, a keyboard 821c, a mouse 822c, and a display 920c shown in FIG. 42 . The keyboard 821c and the mouse 822c are used, for example, in the operation of inputting data necessary for operating the dry eye disease inspection apparatus 20c. The display 920c displays, for example, a graphical user interface of the dry eye disease inspection apparatus 20c.

The bus bar 26c connects the processor 21c, the main storage device 22c, the communication interface 23c, the auxiliary storage device 24c and the input/output device 25c, so that these can transmit and receive data to and from each other.

FIG. 43 is a diagram showing an example of a software configuration of the dry eye syndrome examination program according to the third embodiment. The dry eye disease inspection device 20c uses the processor 21c to read out and execute the dry eye disease inspection program 200c, so as to achieve the data acquisition function 201c and the symptom estimation function 202c shown in FIG. 43 .

The data acquisition function 201c acquires image data for inference and answer data for inference.

The image data for inference is data representing an image in which the eyes of the subject for inference are drawn. The inference image is captured using, for example, a camera mounted on a smartphone.

The response data for inference is data showing the result of an answer to an inquiry about the subjective symptoms of the eye possessed by the subject for inference.

For example, as a question represented by the answer data for inference, for example, the subject for inference "blurred eyes", "dazzled eyes", "difficulty opening eyes", "feeling of foreign body in eyes", "feeling of eye discomfort" ” and other inquiries. Alternatively, as the inquiries expressed by the answer data for inference, inquiries such as "eyestrain", "dry eyes", "heavy eyes", and "eye congestion" can be mentioned.

In addition, for example, the answer to the inquiry about the subjective symptoms of the eyes may be selected from "0 points: no symptoms at all", "1 point: not very concerned", "2 points: slightly concerned", "3 points: concerned ” and “4 points: I care very much” in the five stages. Alternatively, the answer to the question about the awareness of the eyes may be selected from both "yes" and "no".

The symptom inference function 202c inputs the image data for inference and the answer data for inference into the machine learning program 750c that has been learned by the machine learning executive function 102c, and causes the machine learning program 750c to infer the dryness appearing in the eyes of the subject for inference. Symptoms of eye disease. For example, the symptom inference function 202c inputs the two data into the machine learning program 750c to infer a numerical value representing the degree of corneal and conjunctival epithelial damage occurring in the eye of the subject for inference.

Then, the symptom estimating function 202c causes the machine learning program 750c to output symptom data indicating the symptoms of dry eye syndrome appearing in the eyes of the subject for inference. For example, the symptom estimation function 202c causes the machine learning program 750c to output symptom data representing a numerical value representing the degree of corneal and conjunctival epithelial damage occurring in the eye of the subject for estimation. In this case, the symptom data is used to display on the display 920c, for example, a numerical value that expresses itself and expresses the degree of corneal and conjunctival epithelial damage occurring in the eye of the subject for inference.

Next, with reference to FIG. 44 , an example of the processing executed by the dry eye syndrome examination program 200 c of the third embodiment will be described. FIG. 44 is a flowchart showing an example of processing performed by the dry eye syndrome examination program of the third embodiment.

In step S61, the data acquisition function 201c acquires image data for inference and answer data for inference.

In step S62, the symptom inference function 202c inputs the image data for inference and the answer data for inference into the machine learning program 750c that has been learned, so as to infer the symptoms of dry eye appearing in the eyes of the subject for inference, and The machine learning program 750c is caused to output symptom data indicating the symptoms of dry eye appearing in the eyes of the subject for inference.

The machine learning execution program, the dry eye disease inspection program, the machine learning execution device, the dry eye disease inspection device, the machine learning execution method, and the dry eye disease inspection method of the third embodiment have been described above.

The machine learning execution program 100c includes a teacher data acquisition function 101c and a machine learning execution function 102c.

The teacher data acquisition function 101c acquires teacher data in which the image data for learning and the answer data for learning are the questions, and the answer is at least one of the image data for learning and the inspection result data for learning. The image data for learning is data representing an image in which the eyes of the subject for learning are drawn. The response data for learning is data showing the results of the responses to the inquiries about the subjective symptoms of the eyes possessed by the subject for learning. The examination image data for learning is data representing an image in which the eyes of the subject for learning are drawn when an examination related to the symptoms of dry eye is performed on the eyes of the subject for learning. The test result data for learning are data showing test results related to symptoms of dry eye.

The machine learning execution function 102c inputs the teacher data into the machine learning program 750c, and causes the machine learning program 750c to learn.

Thereby, the machine learning execution program 100c can generate the machine learning program 750c for predicting the test results related to the symptoms of dry eye based on the image data for learning and the response data for learning.

In addition, the machine learning execution program 100c causes the machine learning program 750c to learn using the teacher data including not only the learning image data but also the learning hyperemia data as questions. Therefore, the machine learning execution program 100c can generate the machine learning program 750c capable of predicting the test results related to the symptoms of dry eye more accurately.

The dry eye syndrome examination program 200c includes a data acquisition function 201c and a symptom estimation function 202c.

The data acquisition function 201c acquires image data for inference and answer data for inference. The image data for inference is data representing an image in which the eyes of the subject for inference are drawn. The response data for inference is data showing the result of an answer to an inquiry about the subjective symptoms of the eye possessed by the subject for inference.

The symptom inference function 202c inputs the inference image data and inference answer data to the machine learning program 750c that has been learned by the machine learning execution program 100c, to infer the symptoms of dry eye appearing in the eyes of the inference subject. Then, the symptom estimating function 202c causes the machine learning program 750c to output symptom data indicating the symptoms of dry eye syndrome appearing in the eyes of the subject for inference.

Thereby, the dry eye disease examination program 200c can predict the examination result related to the symptoms of dry eye without actually carrying out the examination related to the symptoms of dry eye.

Next, an example in which the machine learning program 750c is actually learned to infer the symptoms of dry eye syndrome is given, and the effect of executing the program 100c by machine learning is described by comparing a comparative example with the example of the third embodiment. specific example.

First, a specific example of the effect obtained by executing the program 100c by machine learning when the test is performed using the luciferin staining reagent will be described.

In the comparative example in this case, the data obtained by removing the response data for learning from the above-mentioned 560 teacher data showing normal eyes of the subject for learning and 560 teacher data showing abnormal eyes of the subject for learning Examples of teacher data to make machine learning programs learn.

With regard to the machine learning program that has been learned in this way, if the characteristics are evaluated using the 80 test data representing the normal eye of the subject for learning and the 80 test data representing the abnormal eye of the subject for learning, the results are shown as follows: The prediction accuracy was 73% and the false negative rate was 34%.

On the other hand, the machine learning program 750b learned by the machine learning execution program 100b showed that the prediction accuracy was 77% and the false negative rate was 12% when the characteristics were evaluated using the same test data. This characteristic is superior to that of the machine learning program of the comparative example. In addition, in this case, the questions represented by the answering materials for learning are five questions: "blurred eyes", "dazzled eyes", "difficulty when opening eyes", "feeling of foreign body in eyes", and "feeling of eye discomfort".

In addition, the prediction accuracy is the number of sheets X predicted to be abnormal by the machine learning program among the learning images included in the abnormal group, and the number of sheets predicted to be normal by the machine learning program among the learning images included in the normal group. The ratio of the total X+Y of the number Y to the total number of test data. Therefore, in the case of this comparative example, the prediction accuracy is calculated using ((X+Y)/160)×100.

In addition, the false negative rate is the ratio of the number of images for learning included in the abnormal group that are predicted to be normal by the machine learning program to the total number of images for learning included in the abnormal group. Therefore, in the case of this comparative example, the false negative rate was calculated by ((80-X)/80)×100.

Second, a specific example of the effect obtained by executing the program 100c by machine learning when the test is performed using the lissamine green staining reagent will be described.

In the comparative example in this case, the data obtained by removing the response data for learning from the 540 teacher data showing normal eyes of the subject for learning and the 580 teacher data showing abnormal eyes of the subject for learning are taken as Examples of teacher data to make machine learning programs learn.

With regard to the machine learning program that has been learned in this way, if the characteristics are evaluated using the 80 test data representing the normal eye of the subject for learning and the 80 test data representing the abnormal eye of the subject for learning, the results are shown as follows: The prediction accuracy was 73% and the false negative rate was 34%.

On the other hand, the machine learning program 750b learned by the machine learning execution program 100b showed that the prediction accuracy was 77% and the false negative rate was 12% when the characteristics were evaluated using the same test data. This characteristic is superior to that of the machine learning program of the comparative example. In addition, in this case, the questions represented by the answering materials for learning are five questions: "blurred eyes", "dazzled eyes", "difficulty when opening eyes", "feeling of foreign body in eyes", and "feeling of eye discomfort".

Furthermore, at least a part of the functions of the machine learning execution program 100c can be realized by hardware including a circuit portion. Likewise, at least a part of the functions of the dry eye syndrome examination program 200b can be realized by hardware including a circuit unit. In addition, such hardware is, for example, LSI, ASIC, FPGA, and GPU.

In addition, at least a part of the functions of the machine learning execution program 100c can also be achieved by the cooperation of software and hardware. Similarly, at least a part of the functions of the dry eye disease checking program 200c can also be achieved by the cooperation of software and hardware. In addition, these hardwares may be integrated into one, or may be divided into multiple pieces.

In addition, in the third embodiment, the case where the machine learning execution device 10c, the machine learning device 700c, and the dry eye disease inspection device 20c are independent devices has been described as an example, but the present invention is not limited to this. The devices can also be implemented as one device.

[Variations of the second embodiment and the third embodiment] In the second embodiment, the case where the teacher data acquisition function 101b acquires the teacher data that does not include the learning answer material described in the third embodiment as a part of the question is exemplified, but it is not limited to this. The teacher data acquisition function 101b can also acquire teacher data that includes not only the study congestion data but also the study response data as questions. In addition, in this case, the data acquisition function 201b acquires answer data for inference in addition to the congestion data for inference. In this case, the symptom estimating function 202b causes the machine learning program 750b to infer the symptoms of dry eye appearing in the eyes of the subject for inference based on the response data for inference in addition to the congestion data for inference.

In addition, in the third embodiment, the case where the teacher data acquisition function 101c acquires the teacher data that does not include the hyperemia data for learning described in the second embodiment as a part of the problem is exemplified, but it is not limited to this. The teacher data acquisition function 101c can also acquire teacher data that includes not only the answer data for learning but also the congestion data for learning as a question. Moreover, in this case, the data acquisition function 201c acquires the hyperemia data for inference in addition to the answer data for inference. In this case, the symptom estimating function 202c causes the machine learning program 750c to infer the symptoms of dry eye appearing in the eyes of the subject for inference based on the inference hyperemia data in addition to the inference response data.

Next, study using teacher data including both the hyperemia data for learning and the response data for learning as questions, and infer the dryness appearing in the eyes of the subject for inference based on both the hyperemia data for inference and the response data for inference. A specific example of the effect obtained when the symptoms of eye disease are present will be described. In addition, in the following description, the case where the teacher data acquisition function 101b acquires the answer material for learning, and the data acquisition function 201b acquires the answer material for inference is taken as an example and demonstrated.

First, a specific example of the effect obtained by executing the program 100b by machine learning in the case where the test is performed using the luciferin staining reagent will be described.

In the comparative example in this case, the study hyperemia data and the study responses were removed from the 560 teacher data showing the normal eyes of the learning subject and the 560 teacher data showing the abnormal eyes of the learning subject. The data after the data is used as an example of teacher data to make the machine learning program learn.

With regard to the machine learning program that has been learned in this way, if the characteristics are evaluated using the 80 test data representing the normal eye of the subject for learning and the 80 test data representing the abnormal eye of the subject for learning, the results are shown as follows: The prediction accuracy was 73% and the false negative rate was 34%.

On the other hand, the machine learning program 750b learned by the machine learning execution program 100b shows that the prediction accuracy is 84% and the false negative rate is 5% when the characteristics are evaluated using the same test data. This characteristic is superior to that of the machine learning program of the comparative example. In addition, in this case, the questions represented by the answering materials for learning are five questions: "blurred eyes", "dazzled eyes", "difficulty when opening eyes", "feeling of foreign body in eyes", and "feeling of eye discomfort".

In addition, the prediction accuracy is the number of sheets X predicted to be abnormal by the machine learning program among the learning images included in the abnormal group, and the number of sheets predicted to be normal by the machine learning program among the learning images included in the normal group. The ratio of the total X+Y of the number Y to the total number of test data. Therefore, in the case of this comparative example, the prediction accuracy is calculated using ((X+Y)/160)×100.

In addition, the false negative rate is the ratio of the number of images for learning included in the abnormal group that are predicted to be normal by the machine learning program to the total number of images for learning included in the abnormal group. Therefore, in the case of this comparative example, the false negative rate was calculated by ((80-X)/80)×100.

Second, a specific example of the effect obtained by executing the program 100b by machine learning when an inspection is performed using the lissamine green staining reagent will be described.

In the comparative example in this case, the study hyperemia data and the study responses were removed from the 540 teacher data showing the normal eyes of the learning subject and the 580 teacher data showing the abnormal eyes of the learning subject. The data after the data is used as an example of teacher data to make the machine learning program learn.

With regard to the machine learning program that has been learned in this way, if the characteristics are evaluated using the 80 test data representing the normal eye of the subject for learning and the 80 test data representing the abnormal eye of the subject for learning, the results are shown as follows: The prediction accuracy was 73% and the false negative rate was 34%.

On the other hand, the machine learning program 750b learned by the machine learning execution program 100b shows that the prediction accuracy is 84% and the false negative rate is 5% when the characteristics are evaluated using the same test data. This characteristic is superior to that of the machine learning program of the comparative example. In addition, in this case, the questions represented by the answering materials for learning are five questions: "blurred eyes", "dazzled eyes", "difficulty when opening eyes", "feeling of foreign body in eyes", and "feeling of eye discomfort".

In addition, the above-mentioned dry eye examination program, dry eye examination apparatus, and dry eye examination method may be used before and after instillation of the dry eye eye drops into the eyes of the subject for inference. By using at such a timing, the dry eye test program, the dry eye test device, and the dry eye test method can also be used to verify the dry eye possessed by the test subject for inference with respect to the dry eye eye drops. Symptoms of the disease and have the effectiveness of the means.

Next, a specific example of the fourth example of the above-described embodiment will be described with reference to FIGS. 45 to 50 .

FIG. 45 is a diagram showing an example of the hardware configuration of the machine learning execution device according to the fourth embodiment. The machine learning execution device 10d shown in FIG. 45 is a device that causes the machine learning device 700d to execute machine learning in a learning phase of the machine learning device 700d to be described later. In addition, as shown in FIG. 45, the machine learning execution device 10d includes a processor 11d, a main storage device 12d, a communication interface 13d, an auxiliary storage device 14d, an input/output device 15d, and a bus bar 16d.

The processor 11d is, for example, a CPU, and reads out and executes the machine learning execution program 100d described later to achieve each function of the machine learning execution program 100d. In addition, the processor 11d may read out and execute programs other than the machine learning execution program 100d, so as to achieve necessary functions in addition to the functions of the machine learning execution program 100d.

The main storage device 12d is, for example, a RAM, and stores in advance a machine learning execution program 100d and other programs read and executed by the processor 11d.

The communication interface 13d is an interface circuit for performing communication with the machine learning apparatus 700d and other devices via the network NW shown in FIG. 45 . In addition, the network NW is, for example, a LAN or an intranet.

The auxiliary storage device 14d is, for example, a hard disk drive, a solid state drive, a flash memory, and a ROM.

The input/output device 15d is, for example, an input/output port. To the input/output device 15d, for example, a keyboard 811d, a mouse 812d, and a display 910d shown in FIG. 45 are connected. The keyboard 811d and the mouse 812d are used, for example, for inputting data necessary for operating the machine learning execution device 10d. The display 910d, for example, displays a graphical user interface of the machine learning execution device 10d.

The bus bar 16d connects the processor 11d, the main storage device 12d, the communication interface 13d, the auxiliary storage device 14d, and the input/output device 15d, so that these can exchange data with each other.

FIG. 46 is a diagram showing an example of the software configuration of the machine learning execution program of the fourth embodiment. The machine learning execution device 10d uses the processor 11d to read out and execute the machine learning execution program 100d, so as to achieve the teacher data acquisition function 101d and the machine learning execution function 102d shown in FIG. 46 .

The teacher data acquisition function 101d acquires teacher data with one image data for learning and eyelid opening data for learning as a question and at least one of one image data for learning and inspection result data for learning as an answer.

The image data for learning is a part of the problem constituting the teacher data, and represents the image for learning in which the eyes of the subject for learning are drawn. The learning image is captured by, for example, a camera mounted on a smartphone.

The eyelid opening data for learning is a part of the problem constituting the teacher data and indicates the maximum eyelid opening time, which is the time during which the subject for learning can continuously open the eyes of which the learning image is captured.

For example, the eyelid opening data for learning can represent the maximum eyelid opening time based on at least the period from the moment when the subject for learning to blink and open the eyes to the next blink by taking at least the image. It is calculated by the movement of the eyelid reflected in the animation.

Alternatively, the eyelid opening data for learning may represent the maximum eyelid opening time based on the self-report of the subject for learning and input using the user interface. The user interface is displayed, for example, on the display 910d shown in FIG. 45 . Alternatively, the user interface is displayed on a touch panel display mounted in a smartphone contracted by the subject for learning.

The examination image data for learning is a part of the answer that constitutes the teacher data, and it is used for learning that depicts the eyes of the subject for learning when an examination related to the symptoms of dry eye is performed on the eyes of the subject for learning. Check the profile of the image. As such an inspection, for example, a luciferin dyeing reagent is dropped into the eye, and the eye is observed with a slit lamp to examine the tear layer destruction time. Therefore, for example, the examination image data for learning is data representing an image or a moving image in which an eye that has been stained with fluorescein and observed with a slit lamp is drawn.

The test result data for learning is data that constitutes a part of the answers of the teacher data and shows test results related to symptoms of dry eye. As a result of such an examination, for example, the length of the tear layer destruction time is expressed as 0 based on an examination that investigates the tear layer destruction time by instilling a luciferin dyeing reagent into the eye and observing the eye with a slit lamp. A numerical value of more than 1 and less than 1.

When the numerical value is 0 or more and less than 0.5, the tear layer destruction time is 10 seconds or more, and therefore the eyes of the subject for study appear to be normal, and examination by an ophthalmologist is unnecessary. In addition, when the numerical value is 0.5 or more and 1 or less, the tear layer destruction time is less than 10 seconds, and therefore the eye of the subject for study is abnormal, and an ophthalmologist is required to examine it.

For example, the teacher profile acquisition function 101d acquires 1280 teacher profiles. In this case, the 1280 learning image data respectively represent 1280 obtained by performing four sets of processing of photographing the two eyes of 8 learning subjects 20 times using a camera mounted on a smartphone. Zhang learning images. In this case, the 1280 eyelid opening data for learning represent 1280 values each showing a value representing the maximum eyelid opening time of the subject for learning.

In addition, in this case, the 1280 pieces of examination image data for learning respectively indicate the time when the examination related to the symptoms of dry eye is performed on the eyes of the examination subject for learning depicted in each of the 1280 pieces of learning images. The image taken for the examination of the study. In addition, the 1280 pieces of examination result data for learning represent 1280 numerical values of 0 or more and 1 or less, which respectively represent the examination results related to the symptoms of dry eye performed when 1280 examination images for learning were taken.

In addition, in the following description, the following case is taken as an example for description: 1280 pieces of examination result data for learning include 620 examinations for learning which show that the eyes of the examination subject for learning are normal with a value of 0 or more and less than 0.5 Result data, and 660 test result data for learning showing a numerical value of 0.5 or more and 1 or less indicating abnormality of the eyes of the subject for learning.

The machine learning execution function 102d inputs the teacher data into the machine learning program 750d installed in the machine learning device 700d, and causes the machine learning program 750d to learn. For example, the machine learning execution function 102d causes a machine learning program 750d including a convolutional neural network to learn by backpropagation.

For example, the machine learning execution function 102d inputs the following 540 teacher data into the machine learning program 750d, that is, the 540 teacher data includes a numerical value of 0 or more and less than 0.5, which indicates that the eyes of the subject for performance learning are normal. Study test results data. The 540 teacher profiles are teacher profiles obtained from 7 selected from the 8 persons.

In addition, for example, the machine learning execution function 102d inputs the following 580 teacher data into the machine learning program 750d, that is, the 580 teacher data includes 0.5 or more and 1 or less indicating abnormality of the eyes of the subject for learning. Numerical learning test result data. The 580 teacher profiles are teacher profiles obtained from 7 selected from the 8 persons.

Then, the machine learning execution function 102d makes the machine learning program 750d learn by using the 1120 teacher data.

In addition, for example, the machine learning execution function 102d can also input the following 80 teacher data as test data into the machine learning program 750d, that is, the 80 teacher data show that the eyes of the subject for learning are normal and are not used for machine learning Learning program 750d is in progress. The 80 teacher profiles are teacher profiles obtained from 1 person who was not selected as the 7 persons.

In addition, for example, the machine learning execution function 102d can also input the following 80 teacher data as test data into the machine learning program 750d. That is, the 80 teacher data represent abnormal eyes of the subject for learning and are not used for machine learning. Learning program 750d is in progress. The 80 teacher profiles are teacher profiles obtained from 1 person who was not selected as the 7 persons.

Thereby, the machine learning execution function 102d can evaluate the characteristics of the machine learning program 750d obtained by learning the machine learning program 750d using the 1120 teacher data.

Furthermore, the machine learning execution function 102d may use, for example, a class activation map when evaluating the properties of the machine learning program 750d using test data. Class activation mapping is a technique for partially clarifying the data input to the neural network as the basis for the output of the neural network. Examples of the class activation map include gradient weighted class activation maps. Gradient-weighted class activation mapping is a technique that utilizes the gradient of the classification score relative to the features of the convolution performed by the convolutional neural network to determine that the image input to the convolutional neural network gives a certain the area of influence above the degree.

Next, an example of processing executed by the machine learning execution program 100d of the fourth embodiment will be described with reference to FIG. 47 . 47 is a flowchart showing an example of processing executed by the machine learning execution program of the fourth embodiment. The machine learning execution program 100d executes the processing shown in FIG. 47 at least once.

In step S71, the teacher data acquisition function 101d acquires teacher data with at least one of the learning image data and the learning eyelid opening data as the question and at least one of the learning inspection image data and the learning inspection result data as the answer.

In step S72, the machine learning execution function 102d inputs the teacher data into the machine learning program 750d, so that the machine learning program 750d learns.

48 to 50 , specific examples of the dry eye syndrome inspection program, the dry eye syndrome inspection apparatus, and the dry eye syndrome inspection method according to the fourth embodiment will be described.

FIG. 48 is a diagram showing an example of the hardware configuration of the dry eye disease inspection apparatus according to the fourth embodiment. The dry eye disease inspection apparatus 20d shown in FIG. 48 uses the machine learning apparatus 700d to infer dry eye appearing in the eyes of the subject for inference in the inference stage of the machine learning apparatus 700d that has been learned by the machine learning execution program 100d symptomatic device. In addition, as shown in FIG. 48, the dry eye disease inspection apparatus 20d includes a processor 21d, a main storage device 22d, a communication interface 23d, an auxiliary storage device 24d, an input/output device 25d, and a bus bar 26d.

The processor 21d is, for example, a CPU, and reads out and executes a dry eye disease inspection program 200d described later, so as to achieve each function of the dry eye disease inspection program 200d. In addition, the processor 21d may read out and execute programs other than the dry eye disease inspection program 200d, so as to achieve necessary functions in addition to the functions of the dry eye disease inspection program 200d.

The main storage device 22d is, for example, a RAM, and stores the dry eye examination program 200d and other programs read out and executed by the processor 21d in advance.

The communication interface 23d is an interface circuit for performing communication with the machine learning apparatus 700d and other devices via the network NW shown in FIG. 48 . In addition, the network NW is, for example, a LAN or an intranet.

The auxiliary storage device 24d is, for example, a hard disk drive, a solid state drive, a flash memory, and a ROM.

The input/output device 25d is, for example, an input/output port. The input/output device 25d is connected to, for example, a keyboard 821d, a mouse 822d, and a display 920d shown in FIG. 48 . The keyboard 821d and the mouse 822d are used, for example, in the operation of inputting data necessary to operate the dry eye examination apparatus 20d. The display 920d displays, for example, a graphical user interface of the dry eye disease inspection apparatus 20d.

The bus bar 26d connects the processor 21d, the main storage device 22d, the communication interface 23d, the auxiliary storage device 24d, and the input/output device 25d, so that these can exchange data with each other.

FIG. 49 is a diagram showing an example of a software configuration of a dry eye syndrome test program according to the fourth embodiment. The dry eye disease inspection device 20d uses the processor 21d to read out and execute the dry eye disease inspection program 200d, so as to achieve the data acquisition function 201d and the symptom estimation function 202d shown in FIG. 49 .

The data acquisition function 201d acquires image data for inference and eyelid opening data for inference.

The image data for inference is data representing an image in which the eyes of the subject for inference are drawn. The inference image is captured by, for example, a camera mounted on a smartphone.

The eyelid opening data for inference is data indicating the maximum eyelid opening time, which is the time during which the subject for inference can continuously open the eyes of which the inference image is captured.

For example, the eyelid opening data for inference can represent the maximum eyelid opening time based on at least the period from the moment when the subject for inference blinks and the eye is opened until the next blink is imaged. It is calculated by the movement of the eyelid reflected in the animation.

Alternatively, the eyelid opening data for inference may represent the maximum eyelid opening time based on the self-report of the subject for inference and input using the user interface. The user interface is displayed, for example, on the display 920d shown in FIG. 48 . Alternatively, the user interface is displayed on a touch panel display mounted on a smartphone contracted by the subject for inference.

The symptom inference function 202d inputs the image data for inference and the eyelid opening data for inference into the machine learning program 750d that has been learned by the machine learning executive function 102d, and causes the machine learning program 750d to infer the symptoms appearing in the eyes of the subject for inference. Symptoms of dry eye. For example, the symptom inference function 202d inputs the two data into the machine learning program 750d, and infers a numerical value representing the length of the tear layer destruction time of the eye of the subject for inference.

Then, the symptom estimating function 202d causes the machine learning program 750d to output symptom data indicating the symptoms of dry eye syndrome appearing in the eyes of the subject for inference. For example, the symptom estimation function 202d causes the machine learning program 750d to output symptom data representing a numerical value representing the length of the tear layer destruction time of the eye of the subject for estimation. In this case, the symptom data is used to display, on the display 920d, for example, a numerical value that expresses itself and expresses the length of the tear layer destruction time of the eye of the subject for inference.

Next, with reference to FIG. 50 , an example of the processing executed by the dry eye syndrome examination program 200d of the fourth embodiment will be described. FIG. 50 is a flowchart showing an example of processing performed by the dry eye syndrome examination program of the fourth embodiment.

In step S81, the data acquisition function 201d acquires image data for inference and eyelid opening data for inference.

In step S82, the symptom inference function 202d inputs the image data for inference and the eyelid opening data for inference into the machine learning program 750d that has been learned, so as to infer the symptoms of dry eye appearing in the eyes of the subject for inference, The machine learning program 750d is caused to output symptom data indicating the symptoms of dry eye appearing in the eyes of the subject for inference.

The machine learning execution program, the dry eye disease inspection program, the machine learning execution device, the dry eye disease inspection device, the machine learning execution method, and the dry eye disease inspection method of the fourth embodiment have been described above.

The machine learning execution program 100d includes a teacher data acquisition function 101d and a machine learning execution function 102d.

The teacher data acquisition function 101d acquires teacher data with at least one of the image data for learning and the eyelid opening data for learning as a question, and at least one of the image data for learning and the test result data for learning as an answer. The image data for learning is data representing an image in which the eyes of the subject for learning are drawn. The eyelid opening data for learning is data indicating the maximum eyelid opening time, which is the time during which the subject for learning can continuously open the eyes of which the image for learning is captured. The examination image data for learning is data representing an image in which the eyes of the subject for learning are drawn when an examination related to the symptoms of dry eye is performed on the eyes of the subject for learning. The test result data for learning are data showing test results related to symptoms of dry eye.

The machine learning execution function 102d inputs the teacher data into the machine learning program 750d, and causes the machine learning program 750d to learn.

Thereby, the machine learning execution program 100d can generate the machine learning program 750d for predicting the test result related to the symptoms of dry eye based on the image data for learning and the eyelid opening data for learning.

In addition, the machine learning execution program 100d causes the machine learning program 750d to learn using the teacher data including not only the image data for learning but also the eyelid opening data for learning as a problem. Therefore, the machine learning execution program 100d can generate the machine learning program 750d capable of predicting the test results related to the symptoms of dry eye more accurately.

The dry eye syndrome examination program 200d includes a data acquisition function 201d and a symptom estimation function 202d.

The data acquisition function 201d acquires image data for inference and eyelid opening data for inference. The image data for inference is data representing an image in which the eyes of the subject for inference are drawn. The eyelid opening data for inference is data indicating the maximum eyelid opening time, which is the time during which the subject for inference can continuously open the eyes of which the inference image is captured.

The symptom inference function 202d inputs the image data for inference and the eyelid opening data for inference into the machine learning program 750d that has been learned by the machine learning execution program 100d, so as to infer the symptoms of dry eye appearing in the eyes of the subject for inference . Then, the symptom estimating function 202d causes the machine learning program 750d to output symptom data indicating the symptoms of dry eye syndrome appearing in the eyes of the subject for inference.

Thereby, the dry eye syndrome test program 200d can predict the test results related to the symptoms of the dry eye syndrome without actually carrying out the test related to the symptoms of the dry eye syndrome.

Next, an example in which the machine learning program 750d is actually learned to infer the symptoms of dry eye syndrome is given, and the effect of executing the program 100d by machine learning is explained by comparing the comparative example with the example of the fourth embodiment. specific example.

The comparative example in this case is the data obtained by excluding the eyelid opening data for learning from the 540 teacher data showing normal eyes of the subject for learning and 580 teacher data showing abnormal eyes of the subject for learning. An example of making machine learning programs learn as teacher data.

With regard to the machine learning program that has been learned in this way, if the characteristics are evaluated using the 80 test data representing the normal eye of the subject for learning and the 80 test data representing the abnormal eye of the subject for learning, the results are shown as follows: The prediction accuracy was 72% and the false negative rate was 28%.

On the other hand, the machine learning program 750d learned by the machine learning execution program 100d shows that the prediction accuracy is 80% and the false negative rate is 20% when the characteristics are evaluated using the same test data. This characteristic is superior to that of the machine learning program of the comparative example.

In addition, the prediction accuracy is the number of sheets X predicted to be abnormal by the machine learning program among the learning images included in the abnormal group, and the number of sheets predicted to be normal by the machine learning program among the learning images included in the normal group. The ratio of the total X+Y of the number Y to the total number of test data. Therefore, in the case of this comparative example, the prediction accuracy is calculated using ((X+Y)/160)×100.

In addition, the false negative rate is the ratio of the number of images for learning included in the abnormal group that are predicted to be normal by the machine learning program to the total number of images for learning included in the abnormal group. Therefore, in the case of this comparative example, the false negative rate was calculated by ((80-X)/80)×100.

Furthermore, at least a part of the functions of the machine learning execution program 100d can be realized by hardware including a circuit portion. Likewise, at least a part of the functions of the dry eye disease examination program 200d can be realized by hardware including a circuit unit. In addition, such hardware is, for example, LSI, ASIC, FPGA, and GPU.

In addition, at least a part of the functions of the machine learning execution program 100d can also be achieved by the cooperation of software and hardware. Similarly, at least a part of the functions of the dry eye syndrome checking program 200d can also be achieved by the cooperation of software and hardware. In addition, these hardwares may be integrated into one, or may be divided into multiple pieces.

In addition, in the fourth embodiment, the case where the machine learning execution device 10d, the machine learning device 700d, and the dry eye disease inspection device 20d are independent devices has been described as an example, but the present invention is not limited to this. The devices can also be implemented as one device.

Next, a specific example of the fifth embodiment of the above-described embodiment will be described with reference to FIGS. 51 to 59 . Different from the machine learning execution program, machine learning execution device, and machine learning execution method of the fourth embodiment, the machine learning execution program, machine learning execution device, and machine learning execution method of the fifth embodiment use the following teacher data, that is, all The teacher data includes, as a question, at least one of learning tear meniscus image data and learning illumination image data cut out from the learning image data described later.

FIG. 51 is a diagram showing an example of the hardware configuration of the machine learning execution device according to the fifth embodiment. The machine learning execution device 10e shown in FIG. 51 is a device that causes the machine learning device 700e to execute machine learning in a learning phase of the machine learning device 700e to be described later. In addition, as shown in FIG. 51 , the machine learning execution device 10e includes a processor 11e, a main storage device 12e, a communication interface 13e, an auxiliary storage device 14e, an input/output device 15e, and a bus bar 16e.

The processor 11e is, for example, a CPU, and reads out and executes the machine learning execution program 100e described later to achieve each function of the machine learning execution program 100e. In addition, the processor 11e may read out and execute programs other than the machine learning execution program 100e, so as to achieve necessary functions in addition to each function possessed by the machine learning execution program 100e.

The main storage device 12e is, for example, a RAM, and stores in advance a machine learning execution program 100e and other programs read and executed by the processor 11e.

The communication interface 13e is an interface circuit for performing communication with the machine learning apparatus 700e and other devices via the network NW shown in FIG. 51 . In addition, the network NW is, for example, a LAN or an intranet.

The auxiliary storage device 14e is, for example, a hard disk drive, a solid state drive, a flash memory, and a ROM.

The input/output device 15e is, for example, an input/output port. The input/output device 15e is connected to, for example, a keyboard 811e, a mouse 812e, and a display 910e shown in FIG. 51 . The keyboard 811e and the mouse 812e are used, for example, for inputting data necessary for operating the machine learning execution device 10e. The display 910e, for example, displays a graphical user interface of the machine learning execution device 10e.

The bus bar 16e connects the processor 11e, the main storage device 12e, the communication interface 13e, the auxiliary storage device 14e, and the input/output device 15e, so that these can exchange data with each other.

FIG. 52 is a diagram showing an example of the software configuration of the machine learning execution program of the fifth embodiment. The machine learning execution device 10e uses the processor 11e to read out and execute the machine learning execution program 100e, so as to achieve the teacher data acquisition function 101e and the machine learning execution function 102e shown in FIG. 52 .

The teacher data acquisition function 101e acquires at least one of one tear meniscus image data for learning and one illumination image data for learning as a question, and at least one of one inspection image data for learning and inspection result data for learning. One as the teacher profile for the answer.

The learning image data is data representing a learning image in which the eyes of the subject for learning are drawn. The learning image is captured by, for example, a camera mounted on a smartphone.

The tear meniscus image data for learning is data representing a tear meniscus image for learning, which is a kind of image for learning, and the tear meniscus image for learning is a self-drawing subject for learning. It is obtained by cutting out the area of the tear meniscus of the subject for learning by cutting out the image for learning of the eye. The tear meniscus is the layer of tear formed between the cornea and the lower eyelid, and reflects the amount of tear fluid. When the tear meniscus is low, there are few tears, and when the tear meniscus is high, there are more tears. In addition, the tear meniscus image for learning is generated by, for example, cutting out a region in which the tear meniscus is drawn in the image for learning. FIG. 53 is a diagram showing an example of a learning tear meniscus image obtained by clipping a region in which the tear meniscus is drawn from the learning image of the fifth embodiment. The teacher data acquisition function 101e acquires, for example, learning tear meniscus image data representing the learning tear meniscus image shown in FIG. 53 .

Illumination image data for learning is data representing an illumination image for learning, which is a type of image for learning, and the illumination image for learning is clipped from an image for learning in which the eyes of the subject for learning are drawn. It is obtained by delineating the area of illumination reflected on the cornea of the subject for learning. Such lighting is, for example, a fluorescent lamp installed on the ceiling of a room in which images for learning are captured. In addition, the illumination image for learning is generated by, for example, clipping and drawing a region of illumination reflected on the cornea of the subject for learning in the image for learning. 54 is a diagram showing an example of an illumination image for learning obtained by clipping and drawing a region of illumination reflected on the cornea of the subject for learning from the image for learning in the fifth embodiment. The teacher data acquisition function 101e acquires, for example, learning illumination image data representing the learning illumination image shown in FIG. 54 .

The examination image data for learning is a part of the answer that constitutes the teacher data, and it is used for learning that depicts the eyes of the subject for learning when an examination related to the symptoms of dry eye is performed on the eyes of the subject for learning. Check the profile of the image. As such an examination, for example, a luciferin dyeing reagent is dropped into the eye, and the eye is observed with a slit lamp to examine the tear layer destruction time. Therefore, for example, the examination image data for learning is data representing an image or a moving image in which an eye that has been stained with fluorescein and observed with a slit lamp is drawn.

The test result data for learning is data that constitutes a part of the answers of the teacher data and shows test results related to symptoms of dry eye. As a result of such an examination, for example, the length of the tear layer destruction time is expressed as 0 based on an examination that investigates the tear layer destruction time by instilling a luciferin dyeing reagent into the eye and observing the eye with a slit lamp. A numerical value of more than 1 and less than 1.

When the numerical value is 0 or more and less than 0.5, the tear layer destruction time is 10 seconds or more, and therefore the eyes of the subject for study appear to be normal, and examination by an ophthalmologist is unnecessary. In addition, when the numerical value is 0.5 or more and 1 or less, the tear layer destruction time is less than 10 seconds, and therefore the eye of the subject for study is abnormal, and an ophthalmologist is required to examine it.

For example, the teacher profile acquisition function 101e acquires 1280 teacher profiles. In this case, the 1280 tear meniscus image data for learning and the illumination image data for learning respectively represent the following images. That is, by implementing four sets of images for 8 students for learning using a camera mounted on a smartphone The two eyes of the test subject were photographed 20 times to acquire 1,280 study images, and the area where the tear meniscus was drawn and the area reflected on the cornea were cut out of the 1,280 study images. image generated on the illuminated area.

In addition, in this case, the 1280 pieces of examination image data for learning respectively indicate the time when the examination related to the symptoms of dry eye is performed on the eyes of the examination subject for learning depicted in each of the 1280 pieces of learning images. The image taken for the examination of the study. In addition, the 1280 pieces of examination result data for learning represent 1280 numerical values of 0 or more and 1 or less, which respectively represent the examination results related to the symptoms of dry eye performed when 1280 examination images for learning were taken.

In addition, in the following description, the following case is taken as an example for description: 1280 pieces of examination result data for learning include 620 examinations for learning which show that the eyes of the examination subject for learning are normal with a value of 0 or more and less than 0.5 Result data, and 660 test result data for learning showing a numerical value of 0.5 or more and 1 or less indicating abnormality of the eyes of the subject for learning.

The machine learning execution function 102e inputs the teacher data into the machine learning program 750e installed in the machine learning device 700e, and causes the machine learning program 750e to learn. For example, the machine learning execution function 102e causes a machine learning program 750e comprising a convolutional neural network to learn by back propagation.

For example, the machine learning execution function 102e inputs, into the machine learning program 750e, 540 pieces of teacher data, that is, the 540 pieces of teacher data including a numerical value of 0 or more and less than 0.5 indicating that the eyes of the subject for performance learning are normal. Study test results data. The 540 teacher profiles are teacher profiles obtained from 7 selected from the 8 persons.

In addition, for example, the machine learning execution function 102e inputs the following 580 teacher data into the machine learning program 750e, that is, the 580 teacher data includes 0.5 or more and 1 or less, indicating that the eyes of the learning subject are abnormal. Numerical learning test result data. The 580 teacher profiles are teacher profiles obtained from 7 selected from the 8 persons.

Then, the machine learning execution function 102e uses the 1120 teacher data to make the machine learning program 750e learn.

In addition, for example, the machine learning execution function 102e can also input the following 80 teacher data as test data into the machine learning program 750e, that is, the 80 teacher data show that the eyes of the subject for learning are normal and are not used for the machine learning Learning program 750e is being learned. The 80 teacher profiles are teacher profiles obtained from 1 person who was not selected as the 7 persons.

In addition, for example, the machine learning execution function 102e may input the following 80 teacher data as test data into the machine learning program 750e, that is, the 80 teacher data representing the abnormality of the eyes of the subject for learning and not used for the machine learning Learning program 750e is being learned. The 80 teacher profiles are teacher profiles obtained from 1 person who was not selected as the 7 persons.

Thereby, the machine learning execution function 102e can evaluate the characteristics of the machine learning program 750e obtained by learning the machine learning program 750e using the 1120 teacher data.

Furthermore, the machine learning execution function 102e may use, for example, a class activation map when evaluating the properties of the machine learning program 750e using the test data. Class activation mapping is a technique for partially clarifying the data input to the neural network as the basis for the output of the neural network. Examples of the class activation map include gradient weighted class activation maps. Gradient-weighted class activation mapping is a technique that utilizes the gradient of the classification score relative to the features of the convolution performed by the convolutional neural network to determine that the image input to the convolutional neural network gives a certain the area of influence above the degree.

Next, an example of processing executed by the machine learning execution program 100e of the fifth embodiment will be described with reference to FIG. 55 . FIG. 55 is a flowchart showing an example of processing executed by the machine learning execution program of the fifth embodiment. The machine learning execution program 100e executes the processing shown in FIG. 55 at least once.

In step S91, the teacher data acquisition function 101e acquires at least one of the tear meniscus image data for learning and the illumination image data for learning as a question, and the inspection image data for learning and the inspection result data for learning. at least one of the teacher profiles for the answer.

In step S92, the machine learning execution function 102e inputs the teacher data into the machine learning program 750e, and makes the machine learning program 750e learn.

56 to 58 , specific examples of the dry eye syndrome inspection program, the dry eye syndrome inspection apparatus, and the dry eye syndrome inspection method according to the fifth embodiment will be described. Different from the dry eye inspection program, dry eye inspection device, and dry eye inspection method of the fourth embodiment, the dry eye inspection program, dry eye inspection device, and dry eye inspection method of the fifth embodiment are obtained from the following description. At least one of the tear meniscus image data for inference and the illumination image data for inference cut out from the image data for inference learning.

FIG. 56 is a diagram showing an example of the hardware configuration of the dry eye disease testing apparatus according to the fifth embodiment. The dry eye disease inspection apparatus 20e shown in FIG. 56 uses the machine learning apparatus 700e to infer dry eye appearing in the eyes of the subject for inference in the inference stage of the machine learning apparatus 700e that has been learned by the machine learning execution program 100e symptomatic device. In addition, as shown in FIG. 56, the dry eye disease inspection apparatus 20e includes a processor 21e, a main storage device 22e, a communication interface 23e, an auxiliary storage device 24e, an input/output device 25e, and a bus bar 26e.

The processor 21e is, for example, a CPU, and reads out and executes a dry eye disease inspection program 200e described later, so as to achieve each function of the dry eye disease inspection program 200e. In addition, the processor 21e may read out and execute programs other than the dry eye disease inspection program 200e, so as to achieve necessary functions in addition to the respective functions of the dry eye disease inspection program 200e.

The main storage device 22e is, for example, a RAM, and stores in advance the dry eye syndrome examination program 200e and other programs read and executed by the processor 21e.

The communication interface 23e is an interface circuit for performing communication with the machine learning apparatus 700e and other devices via the network NW shown in FIG. 56 . In addition, the network NW is, for example, a LAN or an intranet.

The auxiliary storage device 24e is, for example, a hard disk drive, a solid state drive, a flash memory, and a ROM.

The input/output device 25e is, for example, an input/output port. The input/output device 25e is connected to, for example, a keyboard 821e, a mouse 822e, and a display 920e shown in FIG. 56 . The keyboard 821e and the mouse 822e are used, for example, in the operation of inputting data necessary to operate the dry eye disease inspection apparatus 20e. The display 920e displays, for example, a graphical user interface of the dry eye examination apparatus 20e.

The bus bar 26e connects the processor 21e, the main storage device 22e, the communication interface 23e, the auxiliary storage device 24e, and the input/output device 25e, so that these can exchange data with each other.

FIG. 57 is a diagram showing an example of the software configuration of the dry eye syndrome examination program according to the fifth embodiment. The dry eye disease inspection apparatus 20e uses the processor 21e to read out and execute the dry eye disease inspection program 200e, so as to achieve the data acquisition function 201e and the symptom estimation function 202e shown in FIG. 57 .

The data acquisition function 201e acquires at least one of tear meniscus image data for inference and illumination image data for inference.

The image data for inference is data representing an image in which the eyes of the subject for inference are drawn. The inference image is captured by, for example, a camera mounted on a smartphone.

The tear meniscus image data for inference shows the tear meniscus for inference obtained by cutting out the area where the tear meniscus of the subject for inference is drawn from the image for which the eye of the subject for inference is drawn. Information on liquid level images. In addition, the tear meniscus image for inference is generated by, for example, clipping a region in which the tear meniscus is drawn in the image for inference.

The illumination image data for inference is an illumination map for inference obtained by cutting out the area of illumination reflected on the cornea of the inference subject from the inference image depicting the eye of the subject for inference. like data. In addition, the illumination image for inference is generated by, for example, clipping and drawing a region of illumination reflected on the cornea of the subject for inference in the image for inference.

The symptom inference function 202e inputs at least one of the tear meniscus image data for inference and the illumination image data for inference to the machine learning program 750e that has been learned by the machine learning executive function 102e, and causes the machine learning program 750e to infer Symptoms of dry eye appearing in the eyes of the subject are inferred. For example, the symptom inference function 202e inputs the data into the machine learning program 750e, and infers a numerical value representing the length of the tear layer destruction time of the eye of the subject for inference.

Then, the symptom estimating function 202e causes the machine learning program 750e to output symptom data indicating the symptoms of dry eye syndrome appearing in the eyes of the subject for inference. For example, the symptom estimation function 202e causes the machine learning program 750e to output symptom data representing a numerical value representing the length of the tear layer destruction time of the eye of the subject for estimation. In this case, the symptom data is used to display, for example, on the display 920e, a numerical value that expresses itself and expresses the length of the tear layer destruction time of the eye of the subject for inference.

Next, with reference to FIG. 58 , an example of the processing executed by the dry eye syndrome examination program 200e of the fifth embodiment will be described. 58 is a flowchart showing an example of processing performed by the dry eye syndrome examination program of the fifth embodiment.

In step S101, the data acquisition function 201e acquires at least one of tear meniscus image data for inference and illumination image data for inference.

In step S102, the symptom inference function 202e inputs at least one of the tear meniscus image data for inference and the illumination image data for inference into the machine learning program that has been learned, so as to infer the eye of the subject for inference. Symptoms of dry eye appearing in the test subject, and the machine learning program is caused to output symptom data indicating the symptoms of dry eye appearing in the eyes of the subject for inference.

The machine learning execution program, the dry eye disease inspection program, the machine learning execution device, the dry eye disease inspection device, the machine learning execution method, and the dry eye disease inspection method of the fifth embodiment have been described above.

The machine learning execution program 100e includes a teacher data acquisition function 101e and a machine learning execution function 102e.

The teacher data acquisition function 101e acquires at least one of the tear meniscus image data for learning and the illumination image data for learning as a question, and at least one of the inspection image data for learning and the inspection result data for learning as a question. Teacher profile for answers. The tear meniscus image data for learning is data representing a tear meniscus image for learning that depicts the tear meniscus of the subject for learning. The illumination image data for learning is data representing an illumination image for learning that depicts illumination reflected on the cornea of the subject for learning. The examination image data for learning is data representing an image in which the eyes of the subject for learning are drawn when an examination related to the symptoms of dry eye is performed on the eyes of the subject for learning. The test result data for learning are data showing test results related to symptoms of dry eye.

The machine learning execution function 102e inputs the teacher data into the machine learning program 750e, and causes the machine learning program 750e to learn.

Thereby, the machine learning execution program 100e can generate the machine learning program 750e for predicting the test result related to the symptoms of dry eye based on at least one of the tear meniscus image data for learning and the illumination image data for learning.

In addition, the machine learning execution program 100e uses the teacher data including at least one of the tear meniscus image data for learning and the illumination image data for learning cut out from the image data for learning as a question, and makes the machine learning program 750e for learning. Therefore, the machine learning execution program 100e can generate the machine learning program 750e capable of predicting the test results related to the symptoms of dry eye more accurately.

The dry eye syndrome examination program 200e includes a data acquisition function 201e and a symptom estimation function 202e.

The data acquisition function 201e acquires at least one of tear meniscus image data for inference and illumination image data for inference. The tear meniscus image data for inference is a data showing a tear meniscus image for inference that depicts the tear meniscus of the inference subject. The illumination image data for inference is data representing an illumination image for inference that depicts illumination reflected on the cornea of the inference subject.

The symptom inference function 202e inputs at least one of the tear meniscus image data for inference and the illumination image data for inference into the machine learning program 750e that has been learned by the machine learning execution program 100e to infer the inference subject Symptoms of dry eye in the eyes. Then, the symptom estimating function 202e causes the machine learning program 750e to output symptom data indicating the symptoms of dry eye syndrome appearing in the eyes of the subject for inference.

Thereby, the dry eye disease examination program 200e can predict the examination result related to the symptoms of dry eye without actually carrying out the examination related to the symptoms of dry eye.

Next, an example in which the machine learning program 750e is actually learned to infer the symptoms of dry eye syndrome will be described, and the effect of executing the program 100e by machine learning will be described by comparing a comparative example with the example of the fifth embodiment. specific example.

In the comparative example in this case, the learning tear meniscus was removed from the 540 teacher data representing the normal eyes of the learning subject and the 580 teacher data representing the abnormal eyes of the learning subject. An example of learning a machine learning program by adding the image data and the learning image data as the teacher data and adding the learning image data as the teacher data.

As for the machine learning program that has been learned in this way, if using the above-mentioned 80 test data showing normal eyes of the subject for learning, excluding the tear meniscus image data for learning and the illumination image data for learning, In addition, the data of the image data for learning and the 80 test data that express the abnormality of the eyes of the subject for learning are added. Evaluating the features with data from image data shows a prediction accuracy of 72% and a false negative rate of 28%.

On the other hand, with respect to the machine learning program 750e that executes the program 100e by machine learning and is learned using the tear meniscus image data for learning, the characteristics are evaluated using the test data including only the tear meniscus image data for learning. , it shows a prediction accuracy of 77% and a false negative rate of 23%. This characteristic is superior to that of the machine learning program of the comparative example.

In addition, about the machine learning program 750e that has been learned by using the machine learning execution program 100e and using the illumination image data for learning, when the characteristics are evaluated using the test data including only the illumination image data for learning, the prediction accuracy is 77%. % and the false negative rate was 23%. This characteristic is superior to that of the machine learning program of the comparative example.

In addition, with regard to the machine learning program 750e that executes the program 100e by machine learning and is learned using the tear meniscus image data for learning and the illumination image data for learning, if the machine learning program 750e is used that includes the tear meniscus image data for learning and the learning Evaluating the characteristics with test data of illumination image data shows a prediction accuracy of 82% and a false negative rate of 18%. This characteristic is superior to the characteristics of the machine learning program of the comparative example, the characteristics of the machine learning program 750e learned using the tear meniscus image data for learning, and the machine learning program 750e learned using the illumination image data for learning. characteristic.

In addition, the prediction accuracy is the number of sheets X predicted to be abnormal by the machine learning program among the learning images included in the abnormal group, and the number of sheets predicted to be normal by the machine learning program among the learning images included in the normal group. The ratio of the total X+Y of the number Y to the total number of test data. Therefore, in the case of this comparative example, the prediction accuracy is calculated using ((X+Y)/160)×100.

In addition, the false negative rate is the ratio of the number of images for learning included in the abnormal group that are predicted to be normal by the machine learning program to the total number of images for learning included in the abnormal group. Therefore, in the case of this comparative example, the false negative rate was calculated by ((80-X)/80)×100.

Furthermore, at least a part of the functions of the machine learning execution program 100e can be realized by hardware including a circuit portion. Similarly, at least a part of the functions of the dry eye syndrome examination program 200e can be realized by hardware including a circuit unit. In addition, such hardware is, for example, LSI, ASIC, FPGA, and GPU.

In addition, at least a part of the functions of the machine learning execution program 100e can also be achieved by the cooperation of software and hardware. Similarly, at least a part of the functions of the dry eye syndrome checking program 200e can also be achieved by the cooperation of software and hardware. In addition, these hardwares may be integrated into one, or may be divided into multiple pieces.

In addition, in the fifth embodiment, the case where the machine learning execution device 10e, the machine learning device 700e, and the dry eye disease inspection device 20e are mutually independent devices has been described as an example, but it is not limited to this. The devices can also be implemented as one device.

[Variations of the fourth embodiment and the fifth embodiment] In the fourth embodiment, it is listed that the teacher data acquisition function 101d acquires teacher data that does not include the tear meniscus image data for learning and the illumination image data for learning described in the fifth embodiment as part of the problem. For example, but not limited to this. The teacher data acquisition function 101d can also be acquired based on the eyelid opening data for learning, and at least one of the tear meniscus image data for learning and the illumination image data for learning is also used as the teacher of the image data for learning material. In addition, in this case, the data acquisition function 201d acquires at least one of the tear meniscus image data for inference and the illumination image data for inference as the inference image in addition to the eyelid opening data for inference. material. In this case, the symptom inference function 202d causes the machine learning program 750d to infer the inference based on at least one of the tear meniscus image data for inference and the illumination image data for inference in addition to the eyelid opening data for inference. Symptoms of dry eye appearing in the eyes of the subject.

In addition, in the fifth embodiment, the case where the teacher data acquisition function 101e acquires the teacher data that does not include the eyelid opening material for learning described in the fourth embodiment as a part of the problem is exemplified, but it is not limited to here. The teacher data acquisition function 101e can also acquire teacher data for which the eyelid opening data for learning is used as a problem based on the tear meniscus image data for learning and the illumination image data for learning. In addition, in this case, the data acquisition function 201e acquires eyelid opening data for inference in addition to at least one of the tear meniscus image data for inference and the illumination image data for inference. In this case, the symptom inference function 202e causes the machine learning program 750e to infer the inference based on the eyelid opening data for inference in addition to at least one of the tear meniscus image data for inference and the illumination image data for inference Symptoms of dry eye appearing in the eyes of the subject.

Furthermore, the machine learning execution function 102d of the fourth embodiment and the modification of the fifth embodiment may use, for example, a class activation map when evaluating the characteristics of the machine learning program 750d using test data. Similarly, the machine learning execution function 102e of the fourth embodiment and the modification of the fifth embodiment may use, for example, a class activation map when evaluating the characteristics of the machine learning program 750e using test data.

Fig. 59 is a diagram showing the important consideration in the learning image depicting the eye of the subject for learning when the machine learning program of the fourth embodiment and the modification of the fifth embodiment is used to predict the test result of the tear layer destruction time A diagram of an example of a part of . For example, the machine learning execution function 102d of the fourth embodiment and the modification of the fifth embodiment uses the gradient weighted class activation map, and evaluates the area surrounded by the ellipse C611 in the learning image shown in FIG. 59, the area surrounded by the ellipse C612 The enclosed area and the area enclosed by the ellipse C61 have more than a certain degree of influence on the prediction of the inspection to investigate the tear layer destruction time by the machine learning program 750d. In addition, the matters described with reference to FIG. 59 are also applied to the machine learning execution function 102e of the fourth embodiment and the modification of the fifth embodiment.

In addition, the gray scales of the three stages included in the area surrounded by these ellipses indicate the degree of influence given to the prediction of the test results related to corneal epithelial damage. Both the area enclosed by the ellipse C611 and the area enclosed by the ellipse C612 are displayed superimposed on the area in which the tear meniscus of the subject for learning is drawn. The area surrounded by the ellipse C61 is superimposed and displayed on the area in which the illumination reflected on the cornea of the subject for learning is drawn.

Next, learn using the teacher data including the eyelid opening data for learning, and at least one of the tear meniscus image data for learning and the illumination image data for learning as a problem, and based on the eyelid opening data for inference, And a specific example of the effect obtained when at least one of the tear meniscus image data for inference and the illumination image data for inference infers the symptoms of dry eye appearing in the eye of the subject will be described. In addition, in the following description, the case where the teacher data acquisition function 101d acquires the eyelid opening data for learning, and the data acquisition function 201d acquires the eyelid opening data for inference will be described as an example.

In the comparative example in this case, the eyelid opening materials for learning and the learning materials for learning are excluded from the 540 teacher data for expressing the normal eyes of the subject for learning and the 580 teacher data for expressing the abnormal eyes of the subject for learning. An example of learning a machine learning program using tear meniscus image data and learning illumination image data, and adding learning image data as teacher data.

As for the machine learning program that has been learned in this way, if the eyelid opening data for learning, the tear meniscus image data for learning, and the learning are removed from the 80 test data of normal eyes of the subject for performance learning described above Illumination image data, additional study image data, and 80 test data showing eye abnormalities of the study subject were excluded from the study eyelid opening data and the study tear meniscus image data. and the learning illumination image data, and the data of the learning image data are added to evaluate the characteristics, the prediction accuracy is 72% and the false negative rate is 28%.

On the other hand, with respect to the machine learning program 750d that executes the program 100d by machine learning and has learned using the eyelid opening data for learning, the tear meniscus image data for learning, and the illumination image data for learning The characteristics were evaluated by the test data of the meniscus image data and the illumination image data for learning, and the prediction accuracy was 90% and the false negative rate was 10%. This characteristic is superior to the characteristics of the machine learning program of the comparative example, the characteristics of the machine learning program 750d of the fourth embodiment, and the characteristics of the machine learning program 750e of the fifth embodiment.

In addition, the prediction accuracy is the number of sheets X predicted to be abnormal by the machine learning program among the learning images included in the abnormal group, and the number of sheets predicted to be normal by the machine learning program among the learning images included in the normal group. The ratio of the total X+Y of the number Y to the total number of test data. Therefore, in the case of this comparative example, the prediction accuracy is calculated using ((X+Y)/160)×100.

In addition, the false negative rate is the ratio of the number of images for learning included in the abnormal group that are predicted to be normal by the machine learning program to the total number of images for learning included in the abnormal group. Therefore, in the case of this comparative example, the false negative rate was calculated by ((80-X)/80)×100.

In addition, the above-mentioned dry eye examination program, dry eye examination apparatus, and dry eye examination method may be used before and after instillation of the dry eye eye drops into the eyes of the subject for inference. By using at such a timing, the dry eye test program, the dry eye test device, and the dry eye test method can also be used to verify the dry eye possessed by the test subject for inference with respect to the dry eye eye drops. Symptoms of the disease and have the effectiveness of the means.

Next, a specific example of the sixth embodiment of the above-described embodiment will be described with reference to FIGS. 60 to 66 .

FIG. 60 is a diagram showing an example of the hardware configuration of the machine learning execution device according to the sixth embodiment. The machine learning execution device 10f shown in FIG. 60 is a device that causes the machine learning device 700f to execute machine learning in a learning phase of the machine learning device 700f described later. In addition, as shown in FIG. 60, the machine learning execution device 10f includes a processor 11f, a main storage device 12f, a communication interface 13f, an auxiliary storage device 14f, an input/output device 15f, and a bus bar 16f.

The processor 11f is, for example, a CPU, and reads out and executes the machine learning execution program 100f described later, so as to achieve each function of the machine learning execution program 100f. In addition, the processor 11f may read out and execute programs other than the machine learning execution program 100f, so as to achieve necessary functions in addition to each function of the machine learning execution program 100f.

The main storage device 12f is, for example, a RAM, and stores in advance a machine learning execution program 100f and other programs read and executed by the processor 11f.

The communication interface 13f is an interface circuit for performing communication with the machine learning device 700f and other devices via the network NW shown in FIG. 1 . In addition, the network NW is, for example, a LAN or an intranet.

The auxiliary storage device 14f is, for example, a hard disk drive, a solid state drive, a flash memory, and a ROM.

The input/output device 15f is, for example, an input/output port. The input/output device 15f is connected to, for example, a keyboard 811f, a mouse 812f, and a display 910f shown in FIG. 1 . The keyboard 811f and the mouse 812f are used, for example, for inputting data necessary for operating the machine learning execution device 10f. The display 910f displays, for example, a graphical user interface of the machine learning execution device 10f.

The bus bar 16f connects the processor 11f, the main storage device 12f, the communication interface 13f, the auxiliary storage device 14f, and the input/output device 15f, so that these can exchange data with each other.

FIG. 61 is a diagram showing an example of the software configuration of the machine learning execution program of the sixth embodiment. The machine learning execution device 10f uses the processor 11f to read out and execute the machine learning execution program 100f, so as to achieve the teacher data acquisition function 101f and the machine learning execution function 102f shown in FIG. 61 .

The teacher data acquisition function 101f acquires teacher data with one image material for learning as the question and one examination result material for learning as the answer.

The image data for learning is data representing an image for learning in which it is assumed that the eyes of the subject for learning are irradiated with light having a lower color temperature than the eyes of the subject for learning. Learn to use the subject's eyes. Such a learning image is, for example, generated by adjusting the white balance of an image depicting an eye of a subject for learning actually irradiated with light having a color temperature of 4000 K, and is simulated by depicting an eye with a color temperature of 3000 K. Image of this eye illuminated by K light. In addition, the image for learning is photographed using, for example, a camera mounted on a smartphone.

The test result data for learning is data that constitutes a part of the answers of the teacher data and shows test results related to symptoms of dry eye. As a result of such an inspection, the numerical value which expresses the thickness of a tear oil layer based on the inspection which evaluates the thickness of a tear oil layer using an optical interferometer and is 0 or more and 1 or less is mentioned, for example.

When the numerical value is 0 or more and less than 0.5, the thickness of the tear oil layer evaluated using the optical interferometer is less than 75 μm, and therefore the eyes of the subject for study appear to be normal, and examination by an ophthalmologist is unnecessary. In addition, when the numerical value is 0.5 or more and 1 or less, the thickness of the tear oil layer evaluated using the optical interferometer is 75 μm or more, and therefore, the eye of the subject for study is abnormal, and an ophthalmologist is required to perform the evaluation. diagnosis.

For example, the teacher profile acquisition function 101f acquires 1280 teacher profiles. In this case, the 1280 learning image data respectively represent 1280 obtained by performing four sets of processing of photographing the two eyes of 8 learning subjects 20 times using a camera mounted on a smartphone. Zhang learning images. In addition, in this case, the 1280 pieces of inspection result data for learning represent 1280 numerical values of 0 or more and 1 or less, each representing the thickness of the tear oil layer evaluated using an optical interferometer.

In addition, in the following description, the following case is taken as an example for description: 1280 pieces of examination result data for learning include 600 examinations for learning which show that the eyes of the examination subject for learning are normal and have a value of 0 or more and less than 0.5 Result data, and 680 test result data for learning showing a numerical value of 0.5 or more and 1 or less indicating abnormality in the eyes of the subject for learning.

The machine learning execution function 102f inputs the teacher data into the machine learning program 750f installed in the machine learning device 700f, and causes the machine learning program 750f to learn. For example, the machine learning execution function 102f causes a machine learning program 750f including a convolutional neural network to learn by back-propagation.

For example, the machine learning execution function 102f inputs the following 520 teacher data into the machine learning program 750f, that is, the 540 teacher data includes a value of 0 or more and less than 0.5 which indicates that the eyes of the subject for performance learning are normal. Study test results data. The 520 teacher profiles are teacher profiles obtained from 7 selected from the 8 persons.

In addition, for example, the machine learning execution function 102f inputs the following 600 teacher data into the machine learning program 750f, that is, the 600 teacher data includes 0.5 or more and 1 or less indicating abnormality in the eyes of the subject for learning. Numerical learning test result data. The 600 teacher profiles are teacher profiles obtained from 7 of the 8 persons.

Then, the machine learning execution function 102f makes the machine learning program 750f learn by using the 1120 teacher data.

In addition, for example, the machine learning execution function 102f can also input the following 80 teacher data as test data into the machine learning program 750f, that is, the 80 teacher data show that the eyes of the subject for learning are normal, and are not used for machine learning Learning program 750f is being learned. The 80 teacher profiles are teacher profiles obtained from 1 person who was not selected as the 7 persons.

In addition, for example, the machine learning execution function 102e may also input the following 80 teacher data as test data into the machine learning program 750f, that is, the 80 teacher data representing the abnormality of the eyes of the subject for learning and not used for the machine learning Learning program 750f is being learned. The 80 teacher profiles are teacher profiles obtained from 1 person who was not selected as the 7 persons.

Thereby, the machine learning execution function 102f can evaluate the characteristics of the machine learning program 750f obtained by learning the machine learning program 750f using the 1080 teacher data.

Furthermore, the machine learning execution function 102f may use, for example, a class activation map when evaluating the characteristics of the machine learning program 750f using test data. Class activation mapping is a technique for partially clarifying the data input to the neural network as the basis for the output of the neural network. Examples of the class activation map include gradient weighted class activation maps. Gradient-weighted class activation mapping is a technique that utilizes the gradient of the classification score relative to the features of the convolution performed by the convolutional neural network to determine that the image input to the convolutional neural network gives a certain the area of influence above the degree.

62 is a diagram showing an example of a portion to be considered in the eye image of the subject for learning when the machine learning program of the sixth embodiment predicts the test result related to the thickness of the tear oil layer. For example, the machine learning execution function 102f uses a gradient weighted class activation map, and evaluates the pairing of the area surrounded by the ellipse C64 shown in FIG. 62 in the learning image shown in FIG. 62 with the tear fluid performed by the machine learning program 750f The prediction of the inspection results related to the thickness of the oil layer gives more than a certain degree of influence. In addition, the three-stage gray scales included in the area enclosed by the ellipse C64 shown in FIG. 62 indicate the degree of influence given to the prediction of the test result related to the thickness of the tear oil layer, and are superimposed and displayed on the drawing of the test subject for learning. on the area of the cornea of the human eye near the lower lid. Furthermore, this gray scale is superimposed on about half of the lower eyelid side of the cornea of the eye of the subject for learning.

Next, FIG. 63 is a flowchart showing an example of processing executed by the machine learning execution program of the sixth embodiment. 63 is a flowchart showing an example of processing executed by the machine learning execution program of the sixth embodiment. The machine learning execution routine 100f executes the processing shown in FIG. 63 at least once.

In step S111, the teacher data acquisition function 101f acquires teacher data with the image data for learning as the question and the examination result data for learning as the answer.

In step S112, the machine learning execution function 102f inputs the teacher data into the machine learning program 750f, so that the machine learning program 750f learns.

64 to 66 , specific examples of the dry eye syndrome inspection program, the dry eye syndrome inspection apparatus, and the dry eye syndrome inspection method according to the sixth embodiment will be described.

FIG. 64 is a diagram showing an example of the hardware configuration of the dry eye disease testing apparatus according to the sixth embodiment. The dry eye disease inspection apparatus 20f shown in FIG. 64 uses the machine learning apparatus 700f to infer dry eye appearing in the eyes of the subject for inference in the inference stage of the machine learning apparatus 700f that has been learned by the machine learning execution program 100f symptomatic device. In addition, as shown in FIG. 64, the dry eye disease inspection apparatus 20f includes a processor 21f, a main storage device 22f, a communication interface 23f, an auxiliary storage device 24f, an input/output device 25f, and a bus bar 26f.

The processor 21f is, for example, a CPU, and reads out and executes a dry eye syndrome inspection program 200f described later, so as to achieve each function of the dry eye syndrome inspection program 200f. In addition, the processor 21f may read out and execute programs other than the dry eye disease inspection program 200f, so as to achieve necessary functions in addition to each function of the dry eye disease inspection program 200f.

The main storage device 22f is, for example, a RAM, and stores in advance a dry eye syndrome examination program 200f and other programs read and executed by the processor 21f.

The communication interface 23f is an interface circuit for performing communication with the machine learning apparatus 700f and other devices via the network NW shown in FIG. 64 . In addition, the network NW is, for example, an LFN or an intranet.

The auxiliary storage device 24f is, for example, a hard disk drive, a solid state drive, a flash memory, and a ROM.

The input/output device 25f is, for example, an input/output port. The input/output device 25f is connected to, for example, a keyboard 821f, a mouse 822e, and a display 920f shown in FIG. 5 . The keyboard 821f and the mouse 822f are used, for example, in the operation of inputting data necessary to operate the dry eye disease inspection apparatus 20f. The display 920f displays, for example, a graphical user interface of the dry eye disease inspection apparatus 20f.

The bus bar 26f connects the processor 21f, the main storage device 22f, the communication interface 23f, the auxiliary storage device 24f and the input/output device 25f, so that these can transmit and receive data to and from each other.

FIG. 65 is a diagram showing an example of a software configuration of the dry eye syndrome examination program of the sixth embodiment. The dry eye disease inspection device 20f uses the processor 21f to read out and execute the dry eye disease inspection program 200f, so as to achieve the data acquisition function 201f and the symptom estimation function 202f shown in FIG. 65 .

The data acquisition function 201f acquires image data for inference. The image data for inference is data showing an image for inference that depicts a case where the eyes of the subject for inference are irradiated with light having a lower color temperature than the light irradiating the eyes of the subject for inference. The inference is made with the eyes of the subject. Such an inference image is, for example, generated by adjusting the white balance of an image depicting the eye of the inference subject actually irradiated with light having a color temperature of 4000 K, and is simulated to depict an eye with a color temperature of 3000 K. Image of this eye illuminated by K light. In addition, the image for inference is photographed using, for example, a camera mounted on a smartphone.

The symptom inference function 202f inputs the image data for inference to the machine learning program 750f that has been learned by the machine learning execution function 102f, and causes the machine learning program 750f to infer the symptoms of dry eye appearing in the eyes of the subject for inference. For example, the symptom estimation function 202f inputs the image data for inference into the machine learning program 750f, and infers a numerical value representing the thickness of the tear oil layer of the eye of the subject for inference.

Then, the symptom estimating function 202f causes the machine learning program 750f to output symptom data indicating the symptoms of dry eye appearing in the eyes of the subject for inference. For example, the symptom estimation function 202f causes the machine learning program 750f to output symptom data representing a numerical value representing the thickness of the tear oil layer of the eye of the subject for estimation. In this case, the symptom data is used to display on the display 920f, for example, a numerical value that expresses itself and expresses the thickness of the tear oil layer of the eye of the subject for inference.

Next, with reference to FIG. 66 , an example of the processing executed by the dry eye syndrome examination program 200f of the sixth embodiment will be described. FIG. 66 is a flowchart showing an example of processing performed by the dry eye syndrome examination program of the sixth embodiment.

In step S121, the data acquisition function 201f acquires image data for inference.

In step S122, the symptom inference function 202f inputs the image data for inference into the machine learning program 750f that has already been learned to infer the symptoms of dry eye appearing in the eyes of the subject for inference, and makes the machine learning program 750f. Symptom data showing symptoms of dry eye syndrome appearing in the eyes of the subject for inference is output.

The machine learning execution program, the dry eye disease inspection program, the machine learning execution device, the dry eye disease inspection device, the machine learning execution method, and the dry eye disease inspection method of the sixth embodiment have been described above.

The machine learning execution program 100f includes a teacher data acquisition function 101f and a machine learning execution function 102f.

The teacher data acquisition function 101f acquires teacher data with image data for learning as questions and examination result data for learning as answers. The image data for learning is data representing an image for learning in which it is assumed that the eyes of the subject for learning are irradiated with light having a lower color temperature than the eyes of the subject for learning. Learn to use the subject's eyes. The examination result data for learning is the data showing the examination result of investigating the thickness of the tear oil layer of the eye of the subject for examination.

The machine learning execution function 102f inputs the teacher data into the machine learning program 750f, and causes the machine learning program 750f to perform learning.

Thereby, the machine learning execution program 100f can generate the machine learning program 750f for predicting the test results related to the symptoms of dry eye based on the image data for learning.

The dry eye syndrome examination program 200f includes a data acquisition function 201f and a symptom estimation function 202f.

The data acquisition function 201f acquires image data for inference. The image data for inference is data showing an image for inference that depicts a case where the eyes of the subject for inference are irradiated with light having a lower color temperature than the light irradiating the eyes of the subject for inference. The inference is made with the eyes of the subject.

The symptom estimating function 202f inputs the image data for inference to the machine learning program 750f that has been learned by the machine learning execution program 100f, and thereby infers the symptoms of dry eye appearing in the eyes of the subject for inference. Then, the symptom estimating function 202f causes the machine learning program 750f to output symptom data indicating the symptoms of dry eye appearing in the eyes of the subject for inference.

Thereby, the dry eye syndrome test program 200f can predict the test results related to the symptoms of dry eye syndrome without actually carrying out the test related to the symptoms of dry eye syndrome.

Next, an example in which the machine learning program 750f is actually learned to infer the symptoms of dry eye disease is given, and the effect of executing the program 100f by machine learning is described by comparing the comparative example with the example of the sixth embodiment. specific example.

In the comparative example in this case, the problem contained in the teacher data is not the image data for learning, but the problem that the color temperature of the eyes of the subject for learning is drawn and the light irradiating the eyes of the subject for learning is equal. Image data of images of the eyes of the subject for study during light irradiation. In addition, the comparative example in this case is an example in which the machine learning program is learned using 520 teacher data for expressing the normal eyes of the subject for learning and 600 teacher data for expressing the abnormal eyes of the learning subject .

As for the machine learning program that has been learned in this way, if 80 test data including the image data and representing the normal eyes of the subject for learning are used, and the image data including the image data and representing the subject for learning are used. 80 test data of abnormal eyes were used to evaluate the characteristics, which showed a prediction accuracy of 50% and a false negative rate of 50%.

On the other hand, about the machine learning program 750f learned by the machine learning execution program 100f, when the characteristics were evaluated using the test data including the learning image data, the prediction accuracy was 74% and the false negative rate was 26%. . This characteristic is superior to that of the machine learning program of the comparative example.

In addition, the prediction accuracy is the number of sheets X predicted to be abnormal by the machine learning program among the learning images included in the abnormal group, and the number of sheets predicted to be normal by the machine learning program among the learning images included in the normal group. The ratio of the total X+Y of the number Y to the total number of test data. Therefore, in the case of this comparative example, the prediction accuracy is calculated using ((X+Y)/160)×100.

In addition, the false negative rate is the ratio of the number of images for learning included in the abnormal group that are predicted to be normal by the machine learning program to the total number of images for learning included in the abnormal group. Therefore, in the case of this comparative example, the false negative rate was calculated by ((80-X)/80)×100.

Furthermore, at least a part of the functions of the machine learning execution program 100f can be realized by hardware including a circuit portion. Likewise, at least a part of the functions of the dry eye syndrome examination program 200f can be realized by hardware including a circuit unit. In addition, such hardware is, for example, LSI, ASIC, FPGA, and GPU.

In addition, at least a part of the functions of the machine learning execution program 100f can also be achieved by the cooperation of software and hardware. Similarly, at least a part of the functions of the dry eye disease checking program 200f can also be achieved by the cooperation of software and hardware. In addition, these hardwares may be integrated into one, or may be divided into multiple pieces.

In addition, in the sixth embodiment, the case where the machine learning execution device 10f, the machine learning device 700f, and the dry eye disease inspection device 20f are independent devices has been described as an example, but the present invention is not limited to this. The devices can also be implemented as one device.

Next, a specific example of the seventh example of the above-described embodiment will be described with reference to FIGS. 67 to 72 . Different from the machine learning execution program, machine learning execution apparatus, and machine learning execution method of the sixth embodiment, the machine learning execution program, machine learning execution apparatus, and machine learning execution method of the seventh embodiment use corneal images including the later-described learning use. Data rather than teacher data for learning to use image data as questions.

FIG. 67 is a diagram showing an example of the hardware configuration of the machine learning execution device according to the seventh embodiment. The machine learning execution device 10g shown in FIG. 67 is a device for causing the machine learning device 700g to execute machine learning in a learning phase of the machine learning device 700g described later. In addition, as shown in FIG. 67, the machine learning execution device 10g includes a processor 11g, a main storage device 12g, a communication interface 13g, an auxiliary storage device 14g, an input/output device 15g, and a bus bar 16g.

The processor 11g is, for example, a CPU, and reads out and executes the machine learning execution program 100g described later to achieve each function of the machine learning execution program 100g. In addition, the processor 11g may read out and execute programs other than the machine learning execution program 100g, so as to achieve necessary functions in addition to each function of the machine learning execution program 100g.

The main storage device 12g is, for example, a RAM, and stores in advance a machine learning execution program 100g and other programs read and executed by the processor 11g.

The communication interface 13g is an interface circuit for performing communication with the machine learning apparatus 700g and other devices via the network NW shown in FIG. 8 . In addition, the network NW is, for example, a LAN or an intranet.

The auxiliary storage device 14g is, for example, a hard disk drive, a solid state drive, a flash memory, or a ROM.

The input/output device 15g is, for example, an input/output port. The input/output device 15g is connected to, for example, a keyboard 811g, a mouse 812g, and a display 910g shown in FIG. 1 . The keyboard 811g and the mouse 812g are used, for example, for inputting data necessary for operating the machine learning execution device 10g. The display 910g displays, for example, a graphical user interface of the machine learning execution device 10g.

The bus bar 16g connects the processor 11g, the main storage device 12g, the communication interface 13g, the auxiliary storage device 14g, and the input/output device 15g, so that these can send and receive data to and from each other.

FIG. 68 is a diagram showing an example of the software configuration of the machine learning execution program of the seventh embodiment. The machine learning execution device 10g uses the processor 11g to read out and execute the machine learning execution program 100g, so as to achieve the teacher data acquisition function 101g and the machine learning execution function 102g shown in FIG. 68 .

The teacher data acquisition function 101g acquires teacher data with a corneal image data for learning as a question and a learning examination result data as an answer.

The corneal image data for learning is data representing a corneal image for learning that depicts at least a part of the cornea of the eye of the subject for learning. The corneal image for learning may be an image that depicts only at least a part of the cornea of the eye of the subject for learning, or an image that also depicts parts other than the cornea of the eye of the subject for learning. In addition, the corneal image for learning may be an image obtained by cutting out an area in which at least a part of the cornea of the eye of the subject for learning is drawn from the image for learning in which the eye of the subject for learning is drawn.

In addition, with regard to the corneal image for learning, only the portion close to the upper eyelid of the subject for learning and the portion close to the lower eyelid of the eye of the subject for learning are drawn as compared The prediction accuracy of the machine learning program 750g can be improved when the lower eyelid part of the subject is learned. For example, when the learning corneal image depicts only half of the cornea on the lower eyelid side of the eye of the learning subject, the accuracy of predicting the symptoms of dry eye using the machine learning program 750g can be further improved. In addition, the corneal image for learning is photographed using, for example, a camera mounted on a smartphone.

The test result data for learning is data that constitutes a part of the answers of the teacher data and shows test results related to symptoms of dry eye. As a result of such an inspection, the numerical value which expresses the thickness of a tear oil layer based on the inspection which evaluates the thickness of a tear oil layer using an optical interferometer and is 0 or more and 1 or less is mentioned, for example.

When the numerical value is 0 or more and less than 0.5, the thickness of the tear oil layer evaluated using the optical interferometer is less than 75 μm, and therefore the eyes of the subject for study appear to be normal, and examination by an ophthalmologist is unnecessary. In addition, when the numerical value is 0.5 or more and 1 or less, the thickness of the tear oil layer evaluated using the optical interferometer is 75 μm or more, and therefore, the eye of the subject for study is abnormal, and an ophthalmologist is required to perform the evaluation. diagnosis.

For example, the teacher profile acquisition function 101f acquires 1280 teacher profiles. In this case, the 1280 corneal image data for learning are obtained by performing four sets of processing of photographing the two eyes of 8 subjects for learning 20 times using a camera mounted on a smartphone. 1280 images for learning. In addition, in this case, the 1280 pieces of inspection result data for learning represent 1280 numerical values of 0 or more and 1 or less, each representing the thickness of the tear oil layer evaluated using an optical interferometer.

In addition, in the following description, the following case is taken as an example for description: 1280 pieces of examination result data for learning include 600 examinations for learning which show that the eyes of the examination subject for learning are normal and have a value of 0 or more and less than 0.5 Result data, and 680 test result data for learning showing a numerical value of 0.5 or more and 1 or less indicating abnormality in the eyes of the subject for learning.

The machine learning execution function 102g inputs the teacher data into the machine learning program 750g installed in the machine learning device 700g, and causes the machine learning program 750g to learn. For example, the machine learning execution function 102g causes a machine learning program 750g including a convolutional neural network to learn by backpropagation.

For example, the machine learning execution function 102g inputs the following 520 teacher data into the machine learning program 750g, that is, the 520 teacher data includes a numerical value of 0 or more and less than 0.5, which indicates that the eyes of the subject for performance learning are normal. Study test results data. The 520 teacher profiles are teacher profiles obtained from 7 selected from the 8 persons.

In addition, for example, the machine learning execution function 102g inputs the following 600 teacher data into the machine learning program 750g, that is, the 600 teacher data includes 0.5 or more and 1 or less indicating abnormality in the eyes of the subject for learning. Numerical learning test result data. The 600 teacher profiles are teacher profiles obtained from 7 of the 8 persons.

Then, the machine learning execution function 102g makes the machine learning program 750g learn by using the 1120 teacher data.

In addition, in the machine learning execution function 102g, the more the teacher data including the following corneal image data for learning, that is, the corneal image data for learning representing the image that depicts the subject for learning, is preferentially input to the machine learning program. An image of the cornea for learning of the portion of the cornea of the eye that is close to the lower eyelid of the subject for learning. The reason for this is that, as described above, the corneal image for learning is different from the case where both the portion near the upper eyelid and the portion near the lower eyelid of the eye of the subject for learning are drawn in the cornea of the subject for learning. In contrast, when only the portion close to the lower eyelid of the subject for learning is drawn, the prediction accuracy of the machine learning program 750g can be improved.

In addition, for example, the machine learning execution function 102g can also input the following 80 teacher data as test data into the machine learning program 750g, that is, the 80 teacher data show that the eyes of the subject for learning are normal and are not used for the machine learning Learning program 750g is learning. The 80 teacher profiles are teacher profiles obtained from 1 person who was not selected as the 7 persons.

In addition, for example, the machine learning execution function 102g may also input the following 80 teacher data as test data into the machine learning program 750g, that is, the 80 teacher data representing the abnormality of the eyes of the subject for learning and not used for the machine learning Learning program 750g is learning. The 80 teacher profiles are teacher profiles obtained from 1 person who was not selected as the 7 persons.

Thereby, the machine learning execution function 102g can evaluate the characteristics of the machine learning program 750g obtained by learning the machine learning program 750g using the 1080 teacher data.

Furthermore, the machine learning execution function 102g may use, for example, a class activation map when evaluating the characteristics of the machine learning program 750g using test data. Class activation mapping is a technique for partially clarifying the data input to the neural network as the basis for the results output by the neural network. Examples of the class activation map include gradient weighted class activation maps. Gradient-weighted class activation mapping is a technique that utilizes the gradient of the classification score relative to the features of the convolution performed by the convolutional neural network to determine that the image input to the convolutional neural network gives a certain amount to the classification area of influence above the degree.

Next, an example of processing executed by the machine learning execution program 100g of the seventh embodiment will be described with reference to FIG. 69 . FIG. 69 is a flowchart showing an example of processing executed by the machine learning execution program of the seventh embodiment. The machine learning execution program 100g executes the processing shown in FIG. 69 at least once.

In step S131, the teacher data acquisition function 101g acquires teacher data with the corneal image data for learning as the question and the examination result data for learning as the answer.

In step S132, the machine learning execution function 102g inputs the teacher data into the machine learning program 750g, and makes the machine learning program 750g learn.

70 to 72 , specific examples of the dry eye syndrome inspection program, the dry eye syndrome inspection apparatus, and the dry eye syndrome inspection method according to the seventh embodiment will be described. Unlike the dry eye inspection program, dry eye inspection device, and dry eye inspection method of the sixth embodiment, the dry eye inspection program, dry eye inspection device, and dry eye inspection method of the seventh embodiment do not acquire inferences Instead of using the image data, the corneal image data for inference described later is acquired.

FIG. 70 is a diagram showing an example of the hardware configuration of the dry eye disease inspection apparatus according to the seventh embodiment. The dry eye disease inspection apparatus 20g shown in FIG. 70 uses the machine learning apparatus 700g in the inference stage of the machine learning apparatus 700g that has been learned by the machine learning execution program 100g to infer the dry eye that appears in the eyes of the subject for inference symptomatic device. In addition, as shown in FIG. 70 , the dry eye disease inspection apparatus 20g includes a processor 21g, a main storage device 22g, a communication interface 23g, an auxiliary storage device 24g, an input/output device 25g, and a bus bar 26g.

The processor 21f is, for example, a CPU, and reads out and executes the dry eye disease inspection program 200g described later, so as to achieve each function of the dry eye disease inspection program 200g. In addition, the processor 21g may read out and execute programs other than the dry eye disease inspection program 200g, so as to achieve necessary functions in addition to each function of the dry eye disease inspection program 200g.

The main storage device 22g is, for example, a RAM, and stores in advance the dry eye examination program 200g and other programs read and executed by the processor 21g.

The communication interface 23g is an interface circuit for performing communication with the machine learning apparatus 700g and other devices via the network NW shown in FIG. 70 . In addition, the network NW is, for example, a LAN or an intranet.

The auxiliary storage device 24g is, for example, a hard disk drive, a solid state drive, a flash memory, and a ROM.

The input/output device 25g is, for example, an input/output port. The input/output device 25g is connected to, for example, a keyboard 821g, a mouse 822g, and a display 920g shown in FIG. 70 . The keyboard 821g and the mouse 822g are used, for example, in the operation of inputting data necessary to operate the dry eye examination apparatus 20g. The display 920g displays, for example, a graphical user interface of the dry eye examination apparatus 20g.

The bus bar 26g connects the processor 21g, the main storage device 22g, the communication interface 23g, the auxiliary storage device 24g, and the input/output device 25g, so that these can send and receive data to and from each other.

FIG. 71 is a diagram showing an example of the software configuration of the dry eye syndrome examination program according to the seventh embodiment. The dry eye disease inspection apparatus 20g uses the processor 21g to read out and execute the dry eye disease inspection program 200g, so as to achieve the data acquisition function 201g and the symptom estimation function 202g shown in FIG. 71 .

The data acquisition function 201g acquires corneal image data for inference. The corneal image data for inference is data representing a corneal image for inference that depicts at least a part of the cornea of the eye of the subject for inference. The corneal image for inference may be an image that depicts only at least a part of the cornea of the eye of the subject for inference, or an image that also depicts a portion other than the cornea of the eye of the subject for inference. In addition, the cornea image for inference may be an image obtained by cutting out an area in which at least a part of the cornea of the eye of the subject for inference is drawn from the image for inference in which the eye of the subject for inference is drawn.

In addition, when the machine learning program 750g is used for learning using the teacher data that uses the learning corneal image data representing the learning corneal image depicting only the portion close to the lower eyelid of the learning subject as a problem, the inference cornea map is used for learning. Preferably, only the portion close to the lower eyelid of the subject for inference is drawn. Thereby, the dry eye syndrome examination program 200g can predict the symptoms of dry eye syndrome of the subject for inference with higher accuracy using the machine learning program 750g. For example, when the inference cornea image only depicts a half of the lower eyelid side of the cornea of the eye of the inference subject, the accuracy in predicting the symptoms of dry eye using the machine learning program 750g can be further improved. In addition, the corneal image for inference is captured using, for example, a camera mounted on a smartphone.

The symptom inference function 202g inputs the corneal image data for inference into the machine learning program 750g that has been learned by the machine learning executive function 102g, and causes the machine learning program 750g to infer the symptoms of dry eye appearing in the eyes of the subject for inference . For example, the symptom inference function 202g inputs the corneal image data for inference into the machine learning program 750g, and infers a numerical value representing the thickness of the tear oil layer of the eye of the subject for inference.

In addition, the symptom estimation function 202g can input the corneal image data for inference into a machine learning program, that is, the more the machine learning program includes the cornea indicating that the eye of the object for learning is drawn, the closer the learning program is. The teacher data of the corneal image data for learning of the corneal image for learning of the lower eyelid part of the subject is input with priority.

Then, the symptom estimation function 202g causes the machine learning program 750g to output symptom data indicating the symptoms of dry eye syndrome appearing in the eyes of the subject for estimation. For example, the symptom estimation function 202g causes the machine learning program 750g to output symptom data indicating the numerical value of the tear oil layer thickness of the eye of the subject for estimation. In this case, the symptom data is used to display, for example, on the display 920g, a numerical value representing the thickness of the tear oil layer of the eye of the subject for inference, which expresses itself.

Next, with reference to FIG. 72 , an example of the processing executed by the dry eye syndrome examination program 200g of the seventh embodiment will be described. FIG. 72 is a flowchart showing an example of processing performed by the dry eye syndrome examination program of the seventh embodiment.

In step S141, the data acquisition function 201g acquires corneal image data for inference.

In step S142, the symptom inference function 202g inputs the corneal image data for inference into the machine learning program 750g that has already been learned to infer the symptoms of dry eye appearing in the eyes of the subject to be inspected for inference, and makes the machine learning program. 750g output symptom data indicating the symptoms of dry eye syndrome appearing in the eyes of the subject for inference.

The machine learning execution program, the dry eye disease inspection program, the machine learning execution device, the dry eye disease inspection device, the machine learning execution method, and the dry eye disease inspection method of the seventh embodiment have been described above.

The machine learning execution program 100g includes a teacher data acquisition function 101g and a machine learning execution function 102g.

The teacher data acquisition function 101g acquires teacher data with the corneal image data for learning as the question and the examination result data for learning as the answer. The corneal image data for learning is data representing a corneal image for learning that depicts at least a part of the cornea of the eye of the subject for learning. The examination result data for learning is the data showing the examination result of investigating the thickness of the tear oil layer of the eye of the subject for examination.

The machine learning execution function 102g inputs the teacher data into the machine learning program 750g, and causes the machine learning program 750g to learn.

Thereby, the machine learning execution program 100g can generate the machine learning program 750g for predicting the test result related to the symptoms of dry eye based on the corneal image data for learning.

In addition, in the machine learning execution program 100g, the more the teacher data including the following corneal image data for learning, that is, the corneal image data for learning representing the image that depicts the subject for learning, is preferentially input to the machine learning program. An image of the cornea for learning of the portion of the cornea of the eye that is close to the lower eyelid of the subject for learning.

Thereby, the machine learning execution program 100g can generate the machine learning program 750g that can predict the test results related to the symptoms of dry eye more accurately based on the corneal image data for learning.

The dry eye syndrome examination program 200g includes a data acquisition function 201g and a symptom estimation function 202g.

The data acquisition function 201g acquires corneal image data for inference. The corneal image data for inference is data representing a corneal image for inference that depicts at least a part of the cornea of the eye of the subject for inference.

The symptom estimating function 202g inputs the corneal image data for inference into the machine learning program 750g that has been learned by the machine learning execution program 100g, and thereby infers the symptoms of dry eye appearing in the eyes of the subject for inference. Then, the symptom estimation function 202g causes the machine learning program 750g to output symptom data indicating the symptoms of dry eye syndrome appearing in the eyes of the subject for estimation.

Thereby, the dry eye syndrome test program 200g can predict the test results related to the symptoms of the dry eye syndrome without actually carrying out the test related to the symptoms of the dry eye syndrome.

In addition, the dry eye disease examination program 200g can input the corneal image data for inference into the following machine learning program 750g, that is, the more the machine learning program 750g includes the cornea representing the eye of the object to be examined for learning. The teacher data of the corneal image data for learning of the portion near the lower eyelid of the subject for learning is inputted with priority. In addition, the dry eye syndrome examination program 200g causes the machine learning program 750g to infer the symptoms of dry eye syndrome appearing in the eyes of the subject.

Thereby, the dry eye syndrome test program 200g can predict the test results related to the symptoms of dry eye syndrome with higher accuracy without actually carrying out the test related to the symptoms of dry eye syndrome.

Next, an example in which the machine learning program 750g is actually learned to infer the symptoms of dry eye syndrome is given, and the effect of executing the program 100g by machine learning is explained by comparing the comparative example with the example of the seventh embodiment. specific example.

In the comparative example in this case, the problem included in the teacher data is not the corneal image data for learning, but the image data representing the image in which the eye of the subject for learning is drawn. In addition, the comparative example in this case is an example in which the machine learning program is learned using 520 teacher data for expressing the normal eyes of the subject for learning and 600 teacher data for expressing the abnormal eyes of the learning subject .

As for the machine learning program that has been learned in this way, if 80 test data including the image data and representing the normal eyes of the subject for learning are used, and the image data including the image data and representing the subject for learning are used. 80 test data of abnormal eyes were used to evaluate the characteristics, which showed a prediction accuracy of 50% and a false negative rate of 50%.

On the other hand, with regard to the machine learning execution program 100g, the machine learning program 750g that has been learned using only the corneal image data for learning in which only the entire cornea of the eye of the subject for learning is drawn is used. The test data of the image data to evaluate the characteristics shows a prediction accuracy of 69% and a false negative rate of 31%. This characteristic is superior to that of the machine learning program of the comparative example.

In addition, with regard to the machine learning execution program 100g, the machine learning program 750g that has been learned using only the corneal image data for learning that depicts only half of the cornea of the eye of the subject for learning and the lower eyelid side is used. Learning to evaluate properties using test data of corneal image data showed a prediction accuracy of 78% and a false negative rate of 22%. This characteristic is superior to that of the machine learning program of the comparative example.

In addition, the prediction accuracy is the number of sheets X predicted to be abnormal by the machine learning program among the learning images included in the abnormal group, and the number of sheets predicted to be normal by the machine learning program among the learning images included in the normal group. The ratio of the total X+Y of the number Y to the total number of test data. Therefore, in the case of this comparative example, the prediction accuracy is calculated using ((X+Y)/160)×100.

In addition, the false negative rate is the ratio of the number of images for learning included in the abnormal group that are predicted to be normal by the machine learning program to the total number of images for learning included in the abnormal group. Therefore, in the case of this comparative example, the false negative rate was calculated by ((80-X)/80)×100.

Furthermore, at least a part of the functions of the machine learning execution program 100g can be realized by hardware including a circuit portion. Likewise, at least a part of the functions of the dry eye disease examination program 200g can be realized by hardware including a circuit unit. In addition, such hardware is, for example, LSI, ASIC, FPGA, and GPU.

In addition, at least a part of the functions of the machine learning execution program 100g can also be achieved by the cooperation of software and hardware. Similarly, at least a part of the functions of the dry eye syndrome checking program 200g can also be achieved by the cooperation of software and hardware. In addition, these hardwares may be integrated into one, or may be divided into multiple pieces.

In addition, in the seventh embodiment, the case where the machine learning execution device 10g, the machine learning device 700g, and the dry eye disease inspection device 20g are mutually independent devices has been described as an example, but it is not limited to this. The devices can also be implemented as one device.

[Modifications of the sixth embodiment and the seventh embodiment] In the sixth embodiment, the case where the teacher data acquisition function 101f acquires the teacher data that does not include the learning corneal image data described in the seventh embodiment as a question is given as an example, but it is not limited to this. The teacher data acquisition function 101f may also acquire teacher data including learning corneal image data instead of learning image data as a problem, the learning corneal image data depicting the eyes of the assumed learning subject whose eyes are irradiated with a higher color temperature. Learning to use the cornea of the subject to be examined when irradiated with light that is low in light of the subject's eye. In addition, in this case, the data acquisition function 201f acquires the inference corneal image data instead of the inference image data, the inference cornea image data depicts the eye of the hypothetical inference subject whose eyes are illuminated with a color temperature higher than that of the inference subject. The cornea of the test subject is used for inference when the light of the eye of the test subject is low in light irradiation. In this case, the symptom inference function 202f causes the machine learning program 750f to infer the symptoms of dry eye appearing in the eyes of the subject for inference based on the corneal image data for inference instead of the image data for inference. The corneal image data depicts the cornea of the subject for inference when it is assumed that the subject's eye for inference is irradiated with light having a lower color temperature than the light irradiating the eye of the subject for inference.

In addition, in the seventh embodiment, the case where the teacher data acquisition function 101g acquires teacher data that does not include the learning image data described in the sixth embodiment as a question is taken as an example. The data depicts the eyes of the subject for learning when it is assumed that the eyes of the subject for learning are irradiated with light having a lower color temperature than the eyes of the subject for learning, but the seventh embodiment is not limited to this. The teacher data acquisition function 101g may also acquire teacher data including the following corneal image data for learning in place of the corneal image data for learning as a problem, that is, the corneal image data for learning depicting a hypothetical subject for learning. The cornea of the subject for learning when the eye is irradiated with light having a lower color temperature than the light irradiating the eye of the subject for learning. In addition, in this case, the data acquisition function 201g acquires the corneal image data for inference, that is, the inference cornea image data in which the eye of the hypothetical inference subject is depicted is irradiated with a color temperature of the inference subject. The cornea of the subject is used for inference when the light of the eye is low in light irradiation. In this case, the symptom estimating function 202g infers the symptoms of dry eye appearing in the eye of the subject for inference based on the corneal image data for inference, that is, the corneal image data for inference depicting the hypothetical The cornea of the subject for inference when the eye of the subject for inference is irradiated with light having a lower color temperature than the light irradiating the eye of the subject for inference.

Next, when learning using the teacher data including the following corneal image data for learning as questions, and inferring the symptoms of dry eye appearing in the eye of the subject for inference based on the following corneal image data for inference A specific example of the effect obtained will be described. That is, the corneal image data for learning depicts a case where it is assumed that the eye of the subject for learning is irradiated with light having a lower color temperature than the light irradiating the eye of the subject for learning. The cornea of the object to be studied is studied, and the cornea image data for the inference is drawn when the eye of the object to be inspected for inference is irradiated with light having a lower color temperature than the eye of the object to be inspected for inference. body's cornea. In addition, in the following description, the case where the teacher data acquisition function 101f acquires the following corneal image data for learning, and the data acquisition function 201f acquires the following corneal image data for inference will be described as an example. The corneal image data depicts the cornea of the subject for learning when it is assumed that the eye of the subject for learning is irradiated with light having a lower color temperature than the eye of the subject for learning, and the corneal image data for inference is described. The cornea of the inference subject when the eye of the inference subject is assumed to be irradiated with light having a lower color temperature than the light irradiating the inference subject's eye is depicted.

In the comparative example in this case, the problem contained in the teacher data does not describe the learning test subject when the eyes of the learning subject are assumed to be irradiated with light having a lower color temperature than the light irradiating the eyes of the learning subject. It is the corneal image data for learning of the subject's cornea, but represents the cornea of the subject for learning when the eye of the subject for learning is depicted when irradiated with light having a color temperature equal to the light irradiating the eye of the subject for learning. image information of the image. In addition, the comparative example in this case is an example in which the machine learning program is learned using 520 teacher data for expressing the normal eyes of the subject for learning and 600 teacher data for expressing the abnormal eyes of the learning subject .

As for the machine learning program that has been learned in this way, if 80 test data including the image data and representing the normal eyes of the subject for learning are used, and the image data including the image data and representing the subject for learning are used. 80 test data of abnormal eyes were used to evaluate the characteristics, which showed a prediction accuracy of 50% and a false negative rate of 50%.

On the other hand, with regard to the machine learning execution program 100f, the eyes of the learning subject when only the eyes of the hypothetical learning subject are drawn are irradiated with light having a lower color temperature than the light irradiating the eyes of the learning subject. The machine learning program 750f, which has learned the corneal image data for learning of the entire cornea, evaluates the characteristics using the test data including the following corneal image data for learning, the prediction accuracy is 81%, and the false negative rate is 81%. 15%, that is, the learning test subject when the eye of the learning test subject is assumed to be irradiated with light having a lower color temperature than the light irradiating the eyes of the learning test subject in the learning corneal image data cornea. This characteristic is superior to that of the machine learning program of the comparative example.

In addition, regarding the machine learning execution program 100f, the cornea of the eye of the subject for learning is used when only the eye of the subject for learning is drawn with light having a lower color temperature than the light irradiating the eye of the subject for learning. The machine learning program 750f, which has learned half of the corneal image data for learning on the middle and lower eyelid sides, evaluated the characteristics using test data including the following corneal image data for learning, and showed that the prediction accuracy was 89% and false The negative rate is 5%, that is, the learning corneal image data depicts the learning use when the eyes of the same learning subject are irradiated with light having a lower color temperature than the light irradiating the eyes of the learning subject. The cornea of the subject being examined. This characteristic is superior to that of the machine learning program of the comparative example.

In addition, the prediction accuracy is the number of sheets X predicted to be abnormal by the machine learning program among the learning images included in the abnormal group, and the number of sheets predicted to be normal by the machine learning program among the learning images included in the normal group. The ratio of the total X+Y of the number Y to the total number of test data. Therefore, in the case of this comparative example, the prediction accuracy is calculated using ((X+Y)/160)×100.

In addition, the false negative rate is the ratio of the number of images for learning included in the abnormal group that are predicted to be normal by the machine learning program to the total number of images for learning included in the abnormal group. Therefore, in the case of this comparative example, the false negative rate was calculated by ((80-X)/80)×100.

In addition, the above-mentioned dry eye examination program, dry eye examination apparatus, and dry eye examination method may be used before and after instillation of the dry eye eye drops into the eyes of the subject for inference. By using at such a timing, the dry eye test program, the dry eye test device, and the dry eye test method can also be used to verify the dry eye possessed by the test subject for inference with respect to the dry eye eye drops. Symptoms of the disease and have the effectiveness of the means.

1: Photographic installation 2, 2a, 20: Inspection device 10, 10b, 10c, 10d, 10e, 10f, 10g: Machine learning actuators 11, 11b, 11c, 11d, 11e, 11f, 11g, 21, 21b, 21c, 21d, 21e, 21f, 21g: Processor 12, 12b, 12c, 12d, 12e, 12f, 12g, 22, 22b, 22c, 22d, 22e, 22f, 22g: Primary storage device 13, 13b, 13c, 13d, 13e, 13f, 13g, 23, 23b, 23c, 23d, 23e, 23f, 23g: Communication interface 14, 14b, 14c, 14d, 14e, 14f, 14g, 24, 24b, 24c, 24d, 24e, 24f, 24g: auxiliary storage device 15, 15b, 15c, 15d, 15e, 15f, 15g, 25b, 25c, 25d, 25e, 25f, 25g: Input and output devices 16, 16b, 16c, 16d, 16e, 16f, 16g, 26, 26b, 26c, 26d, 26e, 26f, 26g: Busbars 20b, 20c, 20d, 20e, 20f, 20g: Dry Eye Examination Device 25: Touch Panel Display 100: Machine Learning Execution Program/Camera Device Control Department 100b, 100c, 100d, 100e, 100f, 100g: Machine Learning Executive Program 101, 101b, 101c, 101d, 101e, 101f, 101g: Teacher data acquisition function 102, 102b, 102c, 102d, 102e, 102f, 102g: Machine Learning of Executive Functions 110: Photography Department 120:Camera Device Communication Department 130: Display part 140: Operation Reception Department 200: Check program/check device control section 200a: Inspection device control section 200b, 200c, 200d, 200e, 200f, 200g: Dry Eye Examination Program 201, 201b, 201c, 201d, 201e, 201f, 201g: Data acquisition function 202: Infer function 202b, 202c, 202d, 202e, 202f, 202g: Symptom inference function 210: Storage Department 220: Check Device Communication Department 700, 700b, 700c, 700d, 700e, 700f, 700g: Machine Learning Devices 750, 750b, 750c, 750d, 750e, 750f, 750g: Machine Learning Programs 811, 811b, 811c, 811d, 811e, 811f, 811g, 821b, 821c, 821d, 821e, 821f, 821g: Keyboard 812, 812b, 812c, 812d, 812e, 812f, 812g, 822b, 822c, 822d, 822e, 822f, 822g: Mouse 910, 910b, 910c, 910d, 910e, 910f, 910g, 920b, 920c, 920d, 920e, 920f, 920g: Monitor 1000: Image Acquisition Department 1010: Photography Condition Determination Department 1020: Parts of white eyeball and black eyeball extraction 1030: Image output section 1040: Prediction result acquisition department 1050: Tips Department 2000: Image data acquisition department 2010, 2010a: Forecasting Department 2020: Prediction result output section 2030, 2030a: Department of Learning 2040a: Answer Result Acquisition Department A1: Predicted results A7, A8, A9, A11, A13, A14, A81: Image B: base C36, C61, C64, C611, C612: Ellipse E1: Check the system G1, G2, G4: Photography screen G3, G6: Prediction result display screen G7: History screen L1, L10, L10a: Learning Results NW: Internet P: Pillar P0: Photographic image P1, P2: User image data R1, R2, R3, R4, R5, R6: Photographic images S10, S11, S12, S20, S21, S22, S30, S31, S32, S40, S41, S42, S50, S51, S52, S60, S61, S62, S70, S71, S72, S80, S81, S82, S90, S91, S92, S100, S101, S102, S110, S111, S112, S120, S121, S122, S130, S131, S132, S141, S142: Steps U1: User

FIG. 1 is a diagram showing an example of the hardware configuration of the machine learning execution device according to the embodiment. FIG. 2 is a diagram showing an example of a software configuration of the machine learning execution device according to the embodiment. FIG. 3 is a flowchart showing an example of processing executed by the machine learning execution program according to the embodiment. FIG. 4 is a diagram showing an example of the hardware configuration of the inspection apparatus according to the embodiment. FIG. 5 is a diagram showing an example of the appearance of the inspection apparatus according to the embodiment. FIG. 6 is a diagram showing an example of a software configuration of the inspection apparatus according to the embodiment. FIG. 7 is a diagram showing an example of an image displayed by the inspection apparatus according to the embodiment. FIG. 8 is a diagram showing an example of an image displayed by the inspection apparatus according to the embodiment. FIG. 9 is a diagram showing an example of an image displayed by the inspection apparatus according to the embodiment. FIG. 10 is a diagram showing an example of an image displayed by the inspection apparatus according to the embodiment. FIG. 11 is a diagram showing an example of an image displayed by the inspection apparatus according to the embodiment. FIG. 12 is a diagram showing an example of an image displayed by the inspection apparatus according to the embodiment. FIG. 13 is a diagram showing an example of an eye image displayed instead of the eye image shown in FIG. 8 . FIG. 14 is a diagram showing an example of an eye image displayed instead of the eye image shown in FIG. 8 . FIG. 15 is a diagram showing an example of an eye image displayed instead of the eye image shown in FIG. 8 . FIG. 16 is a diagram showing an example of an eye image displayed instead of the eye image shown in FIG. 8 . FIG. 17 is a flowchart showing an example of processing performed by the inspection program of the embodiment. FIG. 18 is a diagram showing an example of the configuration of the inspection system according to the first embodiment. FIG. 19 is a diagram showing an example of the configuration of the imaging device according to the first embodiment. FIG. 20 is a diagram showing an example of predetermined imaging conditions in the first embodiment. FIG. 21 is a diagram showing an example of user image data in the first embodiment. FIG. 22 is a diagram showing an example of the configuration of the inspection apparatus according to the first embodiment. FIG. 23 is a diagram showing an example of an eye state prediction process according to the first embodiment. FIG. 24 is a diagram showing an example of a photographing screen of the first embodiment. FIG. 25 is a diagram showing an example of a photographing screen of the first embodiment. FIG. 26 is a diagram showing an example of a prediction result display screen in the first embodiment. FIG. 27 is a diagram showing an example of prediction processing in the first embodiment. FIG. 28 is a diagram showing an example of the configuration of an apparatus according to a modification of the first embodiment. FIG. 29 is a diagram showing an example of a photographing screen in a modification of the first embodiment. FIG. 30 is a diagram showing an example of a prediction result display screen in a modification of the first embodiment. FIG. 31 is a diagram showing an example of a history screen in a modification of the first embodiment. FIG. 32 is a diagram showing an example of the hardware configuration of the machine learning execution device according to the second embodiment. FIG. 33 is a diagram showing an example of the software configuration of the machine learning execution program of the second embodiment. 34 is a diagram showing an example of a portion to be considered in an eye image of a subject for learning when predicting an examination result related to corneal epithelial damage by the machine learning program of the second embodiment. FIG. 35 is a flowchart showing an example of processing executed by the machine learning execution program of the second embodiment. FIG. 36 is a diagram showing an example of the hardware configuration of the dry eye disease inspection apparatus according to the second embodiment. FIG. 37 is a diagram showing an example of a software configuration of the dry eye syndrome examination program according to the second embodiment. FIG. 38 is a flowchart showing an example of processing performed by the dry eye syndrome examination program of the second embodiment. FIG. 39 is a diagram showing an example of the hardware configuration of the machine learning execution device according to the third embodiment. FIG. 40 is a diagram showing an example of the software configuration of the machine learning execution program of the third embodiment. FIG. 41 is a flowchart showing an example of processing executed by the machine learning execution program of the third embodiment. FIG. 42 is a diagram showing an example of the hardware configuration of the dry eye disease inspection apparatus according to the third embodiment. FIG. 43 is a diagram showing an example of a software configuration of the dry eye syndrome examination program according to the third embodiment. FIG. 44 is a flowchart showing an example of processing performed by the dry eye syndrome examination program of the third embodiment. FIG. 45 is a diagram showing an example of the hardware configuration of the machine learning execution device according to the fourth embodiment. FIG. 46 is a diagram showing an example of the software configuration of the machine learning execution program of the fourth embodiment. 47 is a flowchart showing an example of processing executed by the machine learning execution program of the fourth embodiment. FIG. 48 is a diagram showing an example of the hardware configuration of the dry eye disease inspection apparatus according to the fourth embodiment. FIG. 49 is a diagram showing an example of a software configuration of a dry eye syndrome test program according to the fourth embodiment. FIG. 50 is a flowchart showing an example of processing performed by the dry eye syndrome examination program of the fourth embodiment. FIG. 51 is a diagram showing an example of the hardware configuration of the machine learning execution device according to the fifth embodiment. FIG. 52 is a diagram showing an example of the software configuration of the machine learning execution program of the fifth embodiment. FIG. 53 is a diagram showing an example of a learning tear meniscus image obtained by clipping a region in which the tear meniscus is drawn from the learning image of the fifth embodiment. 54 is a diagram showing an example of an illumination image for learning obtained by clipping and drawing a region of illumination reflected on the cornea of the subject for learning from the image for learning in the fifth embodiment. FIG. 55 is a flowchart showing an example of processing executed by the machine learning execution program of the fifth embodiment. FIG. 56 is a diagram showing an example of the hardware configuration of the dry eye disease testing apparatus according to the fifth embodiment. FIG. 57 is a diagram showing an example of the software configuration of the dry eye syndrome examination program according to the fifth embodiment. 58 is a flowchart showing an example of processing performed by the dry eye syndrome examination program of the fifth embodiment. Fig. 59 is a diagram showing the important consideration in the learning image depicting the eye of the subject for learning when the machine learning program of the fourth embodiment and the modification of the fifth embodiment is used to predict the test result of the tear layer destruction time A diagram of an example of a part of . FIG. 60 is a diagram showing an example of the hardware configuration of the machine learning execution device according to the sixth embodiment. FIG. 61 is a diagram showing an example of the software configuration of the machine learning execution program of the sixth embodiment. 62 is a diagram showing an example of a portion to be considered in the eye image of the subject for learning when the machine learning program of the sixth embodiment predicts the test result related to the thickness of the tear oil layer. 63 is a flowchart showing an example of processing executed by the machine learning execution program of the sixth embodiment. FIG. 64 is a diagram showing an example of the hardware configuration of the dry eye disease testing apparatus according to the sixth embodiment. FIG. 65 is a diagram showing an example of a software configuration of the dry eye syndrome examination program of the sixth embodiment. FIG. 66 is a flowchart showing an example of processing performed by the dry eye syndrome examination program of the sixth embodiment. FIG. 67 is a diagram showing an example of the hardware configuration of the machine learning execution device according to the seventh embodiment. FIG. 68 is a diagram showing an example of the software configuration of the machine learning execution program of the seventh embodiment. FIG. 69 is a flowchart showing an example of processing executed by the machine learning execution program of the seventh embodiment. FIG. 70 is a diagram showing an example of the hardware configuration of the dry eye disease inspection apparatus according to the seventh embodiment. FIG. 71 is a diagram showing an example of the software configuration of the dry eye syndrome examination program according to the seventh embodiment. FIG. 72 is a flowchart showing an example of processing performed by the dry eye syndrome examination program of the seventh embodiment.

20: Inspection device

200: Check program

201: Data acquisition function

202: Infer function

700: Machine Learning Devices

750: Machine Learning Programs

Claims (15)

An inspection method that acquires image data for inference, the image data for inference representing an image for inference depicting an eye of a subject for inference, and The image data for inference is input into a machine learning program, and the machine learning program uses, as a question, the image data for learning representing the image for learning in which the eyes of the subject for learning are drawn, and the learning program is used to represent the learning. Learning is performed using the state data for learning of the eye state of the subject as an answer to the teacher data, the inspection method causes the machine learning program to infer the eye state of the inference subject, and causes the machine learning program to infer the eye state of the subject for inference Data for inference indicating the eye state of the subject for inference is output. The inspection method according to claim 1, wherein, Outputting the inference data indicating the display state of the area in the eye of the inference subject drawn by the inference image that satisfies the predetermined condition, and the display state of the area that does not satisfy the predetermined condition Different images are displayed in the area of the condition. The inspection method according to claim 1 or claim 2, wherein, The machine learning program has learned using the teacher data that further includes, as a part of the answer, a learning measure data indicating a measure to improve the eye state of the learning subject, and the inspection method makes the machine learn. The learning program outputs the inference data indicating measures to improve the eye state of the inference subject. The inspection method according to any one of claim 1 to claim 3, wherein, acquiring the inference image data representing the inference image in which at least a part of the cornea of the eye of the inference subject is depicted, and The machine learning program is caused to output the data for inference, and the machine learning program has learned using the teacher data that uses the image data for learning as the problem, and the image data for learning indicates that there are The image for learning of at least a part of the cornea of the eye of the subject for learning. The inspection method according to claim 4, wherein, The machine learning program is learned by using the teacher data further including, as the answer, the test result data for learning that shows the test result of the test result of investigating the thickness of the tear oil layer of the eye of the test subject, and the test method The machine learning program is caused to output the data for inference. The inspection method according to any one of claim 1 to claim 5, wherein, acquiring the inference image data representing the inference image in which at least a part of the conjunctiva of the eye of the inference subject is depicted, and The machine learning program is caused to output the data for inference, and the machine learning program has learned using the teacher data that uses the image data for learning as the problem, and the image data for learning indicates that there are The image for learning of at least a part of the conjunctiva of the eye of the subject for learning. The inspection method according to any one of claim 1 to claim 6, wherein, Furthermore, the hyperemia data for inference indicating the degree of hyperemia of the eyes of the subject for inference is acquired, The machine learning program uses, as a part of the question, the hyperemia data for learning which further includes the hyperemia degree of the eyes of the subject for learning, and further includes the data showing that the eyes of the subject for learning are drawn. At least one of the examination image data for learning of the image of the eye of the subject for learning and the examination result data for learning indicating the examination result at the time of the examination related to the symptoms of dry eye is used as the object. The teacher data of the answer is learned, and the inspection method causes the machine learning program to infer the symptoms of dry eye appearing in the eyes of the inference subject, and outputs an output indicating the inference subject The symptom data of the symptoms of dry eye appearing in the eyes of 2000 were used as the data for the inference. The inspection method according to any one of claim 1 to claim 7, wherein, Further, answer data for inference are obtained, and the answer data for inference indicates the answer result of the inquiry related to the subjective symptoms of the eyes possessed by the subject for inference, The machine learning program uses, as a part of the question, response data for learning that further includes an answer result of an inquiry about the subjective symptoms of the eyes possessed by the subject for learning, and further includes a representation that depicts the subject. The learning test image data of images of the eyes of the learning subject when an examination related to the symptoms of dry eye is performed on the eyes of the learning subject, and the learning test showing the test results At least one of the result data is learned as the teacher data for the answer, the inspection method causes the machine learning program to infer the symptoms of dry eye appearing in the eyes of the inference subject, and Symptom data representing the symptoms of dry eye appearing in the eyes of the subject for inference are output as the data for inference. The inspection method according to any one of claim 1 to claim 6, wherein, And then obtain the eyelid opening data for inference representing the maximum eyelid opening time, the maximum eyelid opening time is the time when the subject for inference can continuously open the eyes of which the image for inference is captured, The machine learning program uses, as a part of the problem, the eyelid opening data for learning that indicates the maximum eyelid opening time that the subject for learning can continuously open the eyes of which the learning image is captured, that is, the maximum eyelid opening time, Furthermore, it further includes examination image data for learning and a representation showing an image of the eye of the subject for learning when the eye of the subject for learning was subjected to an examination related to the symptoms of dry eye syndrome. At least one of the test result data for learning of test results related to symptoms of dry eye is learned as the teacher data for the answer, and the test method causes the machine learning program to infer the inference subject. Symptoms of dry eye syndrome appearing in the eyes of the test subject, and symptom data indicating the symptoms of dry eye syndrome appearing in the eyes of the subject for inference are output as the data for inference. The inspection method according to any one of claim 1 to claim 6 and claim 9, wherein, The inference tear meniscus image data representing the inference tear meniscus image is acquired as the inference image data, and the inference tear meniscus image is self-drawn with the inference subject. The tear meniscus of the eye is obtained by cutting out the area of the tear meniscus of the subject for inference in the image for the inference, The machine learning program uses learning tear meniscus image data representing a learning tear meniscus image as the problem, and further includes a representation that depicts the learning image data as the problem. The image data of the learning test image for learning the image of the eye of the test subject when the test related to the symptoms of dry eye is performed on the eye of the test subject, and the test results showing the test results related to the symptoms of dry eye At least one of the examination result data for learning has been studied as the teacher data of the answer, and the tear meniscus image for learning is a tear meniscus drawn from the eye of the subject for learning A region in which the tear meniscus of the subject for learning is drawn in the image for learning on the surface is cut out, and the inspection method causes the machine learning program to infer the eye of the subject for inference. Symptoms of dry eye symptoms appearing in the eyes of the subject for inference are output as the data for inferences. The inspection method according to any one of claim 1 to claim 6, claim 9 and claim 10, wherein, Acquiring illumination image data for inference representing an illumination image for inference as the image data for inference, the illumination image for inference self-depicting illumination reflected on the cornea of the subject for inference The inference image is obtained by clipping and drawing a region of illumination reflected on the cornea of the inference subject, The machine learning program uses the illumination image data for learning representing the illumination image for learning, the image data for learning is used as the question, and further includes a representation that depicts an eye performed on the subject for learning. In the examination image data for learning of the image of the eye of the subject for learning at the time of the examination related to the symptoms of dry eye, and the examination result data for learning showing the results of the examination related to the symptoms of dry eye At least one of the teacher data of the answer has been studied, and the illumination image for study is the image for study in which illumination reflected on the cornea of the subject for study is drawn. The middle cut is obtained by delineating a region of illumination reflected on the cornea of the subject for learning, and the inspection method causes the machine learning program to infer dry eye that appears in the eye of the subject for inference Symptoms of the disease, and the symptom data representing the symptoms of dry eye appearing in the eyes of the subject for inference are output as the data for inference. The inspection method according to any one of claim 1 to claim 6, wherein, Acquires the inference object representing the inference object's eye when it is assumed that the inference object's eye is irradiated with light having a lower color temperature than the light irradiating the inference object's eye. Image data for said inference of images, The machine learning program uses an image representing the learning subject when the eyes of the learning subject are drawn with light having a lower color temperature than the light irradiating the eyes of the learning subject. The learning image data of the learning image of the eye is used as the question, and further includes the learning test result data representing the test result of investigating the thickness of the tear oil layer of the eye of the learning subject as the question. The teacher data of the answer is learned, and the inspection method causes the machine learning program to infer the symptoms of dry eye appearing in the eyes of the inference subject, and outputs an output indicating that the inference subject is inspected. The symptom data of the symptoms of dry eye appearing in the eyes of the human body are used as the data for the inference. A machine learning execution method that acquires, as a question, image data for learning representing an image for learning in which the eyes of the subject for learning are drawn, and state data for learning representing the state of the eyes of the subject for learning as a problem. teacher profile for the answer, and The teacher data is input into a machine learning program, so that the machine learning program learns. An inspection device, comprising: A data acquisition unit that acquires image data for inference, the image data for inference representing an image for inference depicting the eyes of the subject for inference; and The inference unit inputs the image data for inference into a machine learning program that uses, as a question, the image data for learning representing the image for learning in which the eyes of the subject for learning are drawn, and expresses The learning state data of the eye state of the subject for learning is learned as the teacher data for the answer, and the inference unit causes the machine learning program to infer the eye state of the subject for inference, and causes the inference to be performed. The machine learning program outputs data for inference indicating the state of the eyes of the subject for inference. A machine learning execution device, comprising: A teacher data acquisition unit that acquires, as a question, learning image data representing a learning image in which the eyes of the learning subject are drawn, and learning state data representing the state of the eyes of the learning subject as an answer teacher information; and The machine learning execution unit inputs the teacher data into the machine learning program, and causes the machine learning program to learn.

Images