JP2021099559A - Information processing device, inference model generating method, information processing method, and program - Google Patents

Abstract

To provide a new technology to acquire higher accuracy output by machine learning using a small amount of teacher data.SOLUTION: An information processing device includes a first learning part, a second learning part, and an inference model generating part. The first learning part performs unsupervised machine learning by using training data so that the output data of a decoder match the input data of an encoder for an autoencoder including the encoder and the decoder to which the output data of the encoder are input. The second learning part performs a machine learn by using the output data corresponding to the labeled data input to the encoder and the label attached to the labeled data as the teacher data. The inference model generating part generates an inference model to infer the label corresponding to the input data to the encoder by using the encoder obtained by the unsupervised machine learning and the sorter obtained by the machine learning with a teacher.SELECTED DRAWING: Figure 10

Description

The present invention relates to an information processing device, an inference model generation method, an information processing method, and a program.

In recent years, due to the rapid progress of machine learning methods represented by deep learning, artificial intelligence technology has been put into practical use in various fields. For example, the safety of automatic driving can be improved by improving the detection accuracy of objects, pedestrians, etc. by deep learning. Further, for example, by improving the detection accuracy of the diseased part or tissue aspect in the diagnostic image by deep learning, accurate and highly accurate medical diagnosis can be performed quickly.

A convolutional neural network (CNN) is widely used for such deep learning (Patent Document 1, Patent Document 2, Patent Document 3).

Japanese Unexamined Patent Publication No. 2019-172010 Japanese Unexamined Patent Publication No. 2018-121886 JP-A-2019-152977

However, a convolutional neural network widely used for deep learning requires a large amount of teacher data in order to obtain a highly accurate output (for example, classification result or certainty). Therefore, it takes a lot of cost and labor to create teacher data.

The present invention has been made in view of such circumstances, and an object of the present invention is to provide a new technique for obtaining highly accurate output by machine learning using a small amount of teacher data.

A first aspect of some embodiments trains a self-encoder that includes an encoder and a decoder into which the output data of the encoder is input so that the output data of the decoder matches the input data of the encoder. A classifier using the first learning unit that executes unsupervised machine learning using data, the output data corresponding to the labeled data input to the encoder, and the label attached to the labeled data as teacher data. Using the second learning unit that executes supervised machine learning, the encoder obtained by the unsupervised machine learning, and the classifier obtained by the supervised machine learning, the input data of the encoder is used. It is an information processing apparatus including an inference model generation unit for generating an inference model for inferring a label corresponding to.

In a second aspect of some embodiments, in the first aspect, the training data is image data representing a frontal image of the fundus in which blood vessels in the choroid are depicted, and the classifier is rendered in the frontal image. It is inferred whether or not the traveling pattern of the blood vessel is symmetric.

A third aspect of some embodiments comprises an encoder that constitutes a self-encoder learned by unsupervised machine learning using training data, and outputs data corresponding to the input data of the encoder of the input data. It is learned by supervised machine learning using a feature amount extractor that outputs as a feature amount, output data corresponding to the labeled data input to the encoder, and a label attached to the labeled data as teacher data. It is an information processing apparatus including a classifier that infers a label corresponding to the input data based on the feature amount output by the feature amount extractor.

In a fourth aspect of some embodiments, in the third aspect, the training data is image data representing a frontal image of the fundus in which blood vessels in the choroid are depicted, and the labeled data is a running pattern of the blood vessels. Is image data representing the front image with a label indicating whether or not is symmetric.

In a fifth aspect of some embodiments, in the second or fourth aspect, the frontal image is a frontal image of the Haller layer in the choroid.

In a sixth aspect of some embodiments, in any of the first to fifth aspects, the labeled data includes data with a portion of the training data labeled.

In the seventh aspect of some embodiments, in any one of the first to sixth aspects, the training data includes image data of a plurality of front images at a plurality of positions within a predetermined depth range on the fundus. ..

In an eighth aspect of some embodiments, in the seventh aspect, the teacher data includes image data of a frontal image of the fundus at a predetermined depth position within the predetermined depth range.

A ninth aspect of some embodiments trains a self-encoder that includes an encoder and a decoder into which the output data of the encoder is input so that the output data of the decoder matches the input data of the encoder. A classifier using the first learning step of executing unsupervised machine learning using data, the output data corresponding to the labeled data input to the encoder, and the label attached to the labeled data as teacher data. Using the second learning step of executing supervised machine learning, the encoder obtained by the unsupervised machine learning, and the classifier obtained by the supervised machine learning, the input data of the encoder is used. A method for generating an inference model, which includes an inference model generation step for generating an inference model for inferring a label corresponding to.

In a tenth aspect of some embodiments, in the ninth aspect, the labeled data includes data in which a portion of the training data is labeled.

In the eleventh aspect of some embodiments, in the ninth or tenth aspect, the training data is image data representing a frontal image of the fundus in which blood vessels in the choroid are depicted, and the classifier is the frontal. It is inferred whether or not the traveling pattern of the blood vessel depicted in the image is symmetric.

In the twelfth aspect of some embodiments, in the eleventh aspect, the frontal image is a frontal image of the Haller layer in the choroid.

In the thirteenth aspect of some embodiments, in any of the ninth to twelfth aspects, the training data includes image data of a plurality of front images at a plurality of positions within a predetermined depth range on the fundus. ..

In the 14th aspect of some embodiments, in the 13th aspect, the teacher data includes image data of a frontal image of the fundus at a predetermined depth position within the predetermined depth range.

In the fifteenth aspect of some embodiments, the input data corresponds to the input data of the encoder by using the encoder constituting the self-encoder learned by unsupervised machine learning using the training data. It is learned by supervised machine learning using the feature amount extraction step to be output as the feature amount of the above, the output data corresponding to the labeled data input to the encoder, and the label attached to the labeled data as teacher data. This is an information processing method including an inference step of inferring a label corresponding to the input data based on the feature amount output in the feature amount extraction step using the classifier.

In the 16th aspect of some embodiments, in the 15th aspect, the labeled data includes data in which a part of the training data is labeled.

In the 17th aspect of some embodiments, in the 15th or 16th aspect, the training data is image data representing a frontal image of the fundus in which blood vessels in the choroid are depicted, and the labeled data is said. It is image data which represents the front image with the label which shows whether or not the running pattern of a blood vessel is symmetric.

In the eighteenth aspect of some embodiments, in the seventeenth aspect, the frontal image is a frontal image of the Haller layer in the choroid.

In the 19th aspect of some embodiments, in any of the 15th to 18th aspects, the training data includes image data of a plurality of front images at a plurality of positions within a predetermined depth range on the fundus. ..

In a twentieth aspect of some embodiments, in the nineteenth aspect, the teacher data includes image data of a frontal image of the fundus at a predetermined depth position within the predetermined depth range.

A twenty-first aspect of some embodiments is a program that causes a computer to perform each step of the inference model generation method according to any of the ninth to fourteenth aspects.

A 22nd aspect of some embodiments is a program that causes a computer to perform each step of the information processing method according to any of the 15th to 20th aspects.

In addition, it is possible to arbitrarily combine the configurations according to the above-mentioned plurality of aspects.

According to some embodiments of the present invention, it is possible to provide a new technique for obtaining highly accurate output by machine learning using a small amount of teacher data.

It is the schematic which shows an example of the structure of the ophthalmic system which concerns on embodiment. It is the schematic which shows an example of the structure of the ophthalmic apparatus which concerns on embodiment. It is the schematic which shows an example of the structure of the ophthalmic information processing apparatus which concerns on embodiment. It is the schematic which shows an example of the structure of the ophthalmic information processing apparatus which concerns on embodiment. It is the schematic which shows an example of the structure of the ophthalmic information processing apparatus which concerns on embodiment. It is the schematic which shows an example of the structure of the ophthalmic information processing apparatus which concerns on embodiment. It is the schematic which shows an example of the structure of the ophthalmic information processing apparatus which concerns on embodiment. It is the schematic which shows an example of the structure of the ophthalmic information processing apparatus which concerns on embodiment. It is the schematic which shows an example of the operation flow of the ophthalmic information processing apparatus which concerns on embodiment. It is the schematic for demonstrating the operation of the ophthalmic information processing apparatus which concerns on embodiment. It is the schematic for demonstrating the operation of the ophthalmic information processing apparatus which concerns on embodiment. It is the schematic for demonstrating the operation of the ophthalmic information processing apparatus which concerns on embodiment. It is the schematic which shows an example of the structure of the ophthalmic apparatus which concerns on the modification of embodiment.

An example of an information processing apparatus, an inference model generation method, an information processing method, and a program according to some embodiments of the present invention will be described in detail with reference to the drawings. In addition, the matters described in the document cited in this specification and any known technique can be incorporated into the embodiment.

Hereinafter, a case where the information processing apparatus, the inference model generation method, the information processing method, and the program according to the embodiment are applied to the field of ophthalmology will be described. However, the information processing apparatus, the inference model generation method, the information processing method, and the application fields of the program according to the embodiment are not limited to the ophthalmic field.

When the information processing device according to the embodiment is applied to the field of ophthalmology, the function of the information processing device according to the embodiment is realized by the ophthalmic information processing device. Similarly, when the information processing method according to the embodiment is applied to the field of ophthalmology, the function of the information processing method according to the embodiment is realized by the ophthalmic information processing method.

The ophthalmic information processing device according to the embodiment can perform a predetermined analysis process and a predetermined display process on the data of the fundus of the eye to be inspected optically acquired by the ophthalmic device. The ophthalmic apparatus according to some embodiments has a function of acquiring a frontal image of the fundus of the eye to be inspected. Ophthalmic devices equipped with a function to acquire a frontal image of the fundus of the eye to be inspected include an optical coherence tomography (OCT) device, a fundus camera, a scanning laser ophthalmoscope, a slit lamp microscope, and a surgical microscope. is there. The ophthalmic apparatus according to some embodiments has a function of measuring the optical characteristics of the eye to be inspected. Ophthalmic devices having the function of measuring the optical characteristics of the eye to be inspected include a reflex meter, a keratometer, a tonometer, a wavefront analyzer, a specular microscope, and a perimeter. The ophthalmic apparatus according to some embodiments comprises the function of a laser treatment apparatus used for laser treatment.

In the following embodiment, a case where the ophthalmic apparatus acquires OCT data by performing OCT on the eye to be inspected will be described. The ophthalmic apparatus or the ophthalmic information processing apparatus can form a frontal image of the fundus using the acquired OCT data.

Further, in the following embodiment, a case where the ophthalmologic information processing apparatus infers whether or not the running pattern of blood vessels in the choroid of the fundus of the eye to be inspected has symmetry will be described. The symmetry of the blood vessel running pattern in the choroid is judged with the line connecting the fovea centralis and the optic nerve head as the axis of symmetry. Cases in which the running pattern of blood vessels in the choroid is determined to be asymmetric are considered to be prone to develop certain diseases (eg, T. Hiroe, et al., "Dilation of Acute Vortex Vein in Central Serous Chorioretinopathy". ", American Academy of Ophthalmology, 2017 years, ISSN 2468-6530 / 17, or, M.C.SAVASTANO, et al.," CLASSIFICATION OF HALLER VESSEL ARRANGEMENTS IN ACUTE aND CHRONIC CENTRAL SEROUS CHORIORETINOPATHY IMAGED WITH EN FACE OPTICAL COHERENCE TOMOGRAPHY " , 2017, Ophthalmic Communications Society).

For example, in the case where the running pattern of blood vessels in the choroid is determined to be asymmetric, it has been pointed out that Pachychoroid may be associated with the pathological condition of age-related macular degeneration. Therefore, by inferring the symmetry of the above-mentioned running pattern, it is possible to assist in determining the treatment policy for Pachychoroid.

For example, "Intravitreal aflibercept and ranibizumab for pachychoroid neovasculopathy" (BJ Yung, et al., 2019, 9: 2055, considered to be effective in SCIENTIFIC REPORTS) ing. Therefore, a therapeutic agent for age-related macular degeneration can be selected from the inference result of the symmetry of the traveling pattern described above.

For example, "Efficacy of Photodynamic Therapy for Polypoidal Chroidal Vasculopathy Associated with and without Pachychoroid Phenotypes" (M.Hata, et al, 2018 years, Ophthalmology Retina.) The, polypoid choroidal angiopathy (Polypoidal Choroidal Vasculopathy: PCV) to rays It has been pointed out that photodynamic therapy (PDT) is effective. Therefore, a treatment method for age-related macular degeneration can be selected from the inference result of the symmetry of the running pattern described above.

The ophthalmologic information processing apparatus according to the embodiment uses an inference model obtained by machine learning to depict a blood vessel in the choroid on a frontal image of the fundus of the eye to be inspected (particularly, an OCT image obtained by using OCT). It is possible to infer whether or not the traveling pattern has symmetry. The inference model generation method according to the embodiment is a method of generating an inference model for inferring whether or not the above-mentioned traveling pattern has symmetry.

The ophthalmic system according to the embodiment includes an ophthalmic information processing device. The ophthalmic information processing method or the inference model generation method according to the embodiment is executed by the ophthalmic information processing apparatus. The program according to the embodiment causes a computer to execute each step of the ophthalmic information processing method or the inference model generation method.

[Ophthalmology system]
FIG. 1 shows a block diagram of a configuration example of an ophthalmic system according to an embodiment. The ophthalmology system 1 according to the embodiment includes an ophthalmology device 10, an ophthalmology information processing device (ophthalmology image processing device, ophthalmology analysis device) 100, an operation device 180, and a display device 190.

The ophthalmic apparatus 10 optically collects data of the eye to be inspected. The ophthalmologic apparatus 10 optically collects data on the fundus of the eye to be inspected by scanning the fundus of the eye to be inspected. In this embodiment, the ophthalmologic apparatus 10 uses OCT to acquire OCT data of the fundus of the eye to be inspected. The ophthalmic apparatus 10 can acquire an image of the fundus of the eye to be inspected from the acquired data of the eye to be inspected. Images of the fundus include tomographic images and frontal images of the fundus. The tomographic image of the fundus includes a B-scan image and the like. The frontal image of the fundus includes a C-scan image, an en-face image, a shadow gram, a projection image, and the like. The ophthalmic apparatus 10 transmits the acquired data of the eye to be inspected or the acquired image data to the ophthalmic information processing apparatus 100.

In some embodiments, the ophthalmic apparatus 10 and the ophthalmic information processing apparatus 100 are connected via a data communication network. The ophthalmic information processing device 100 according to some embodiments receives the above data from one of a plurality of ophthalmic devices 10 selectively connected via a data communication network.

The ophthalmic information processing apparatus 100 uses the inference model (learning model) obtained by machine learning to determine whether or not the running pattern (distribution) of the choroidal blood vessels drawn on the frontal image of the fundus of the eye to be inspected has symmetry. Infer. In some embodiments, the ophthalmologic information processing apparatus 100 comprises a fovea (specifically, the central position of the region corresponding to the fovea) and an optic nerve head (specifically, the optic nerve) visualized in a frontal image of the fundus. With the line connecting the center position of the region corresponding to the optic disc as the axis of symmetry, it is inferred whether or not the running pattern of the choroidal blood vessels is symmetric.

In some embodiments, the ophthalmic information processing apparatus 100 uses an externally constructed inference model to infer whether or not the running pattern of the choroidal blood vessels is symmetric. In some embodiments, the ophthalmic information processing apparatus 100 constructs the above inference model and infers whether or not the running pattern of the choroidal blood vessels is symmetric using the constructed inference model. In some embodiments, the ophthalmic information processing apparatus 100 builds the above inference model and outputs the constructed inference model to an external device.

Hereinafter, the ophthalmic information processing apparatus 100 shall perform the above inference with respect to the en-face image of the fundus. The choroid includes the Choriocapillaris layer, the Sattler layer, and the Haller layer. By analyzing the blood vessels in the Haller layer (macrovascular layer) of the choroid, it becomes easier to determine the symmetry of the running pattern of the choroidal blood vessels (for example, H. Shiihara, et al., "Automated segmentation for en face choroidal". images obtined by optical coherent tomography by machine learning ”, March 29, 2018, Japanese Journal of Ophthalmology). The ophthalmologic information processing apparatus 100 according to the embodiment can make the above inference with respect to the en-face image of the Haller layer of the choroid.

The ophthalmologic information processing apparatus 100 can output as an inference result a classification result (symmetrical or asymmetrical) as to whether or not the running pattern of the choroidal blood vessel has symmetry. In some embodiments, the ophthalmologic information processing apparatus 100 indicates, as an inference result, information (numerical value, image) indicating the degree of certainty indicating an index (degree) for determining that the running pattern of the choroidal blood vessel is symmetric (or asymmetric). Etc.) is output.

The operation device 180 and the display device 190 provide functions as a user interface unit for exchanging information between the ophthalmic information processing device 100 and its user, such as information display, information input, and operation instruction input. The operating device 180 includes operating devices such as levers, buttons, keys, and pointing devices. The operating device 180 according to some embodiments includes a microphone for inputting information by sound. The display device 190 includes a display device such as a flat panel display. In some embodiments, the functions of the operating device 180 and the display device 190 are realized by a device such as a touch panel display in which a device having an input function and a device having a display function are integrated. In some embodiments, the operating device 180 and the display device 190 include a graphical user interface (GUI) for inputting and outputting information.

[Ophthalmic equipment]
FIG. 2 shows a block diagram of a configuration example of the ophthalmic apparatus 10 according to the embodiment.

The ophthalmic apparatus 10 is provided with an optical system for acquiring OCT data of the eye to be inspected. The ophthalmic apparatus 10 has a function of performing a swept source OCT, but the embodiment is not limited to this. For example, the type of OCT is not limited to swept source OCT, and may be spectral domain OCT or the like. The swept source OCT divides the light from the wavelength sweep type (wavelength scanning type) light source into the measurement light and the reference light, and causes the return light of the measurement light passing through the object to be measured to interfere with the reference light to generate interference light. Then, this interference light is detected by a balanced photodiode or the like, and the detection data collected in response to the sweeping of the wavelength and the scanning of the measurement light is subjected to Fourier transform or the like to form an image. Spectral domain OCT divides the light from the low coherence light source into measurement light and reference light, and causes the return light of the measurement light that has passed through the object to be measured to interfere with the reference light to generate interference light. This is a method in which the spectral distribution is detected by a spectroscope and the detected spectral distribution is subjected to Fourier transform or the like to form an image.

The ophthalmic apparatus 10 includes a control unit 11, a data acquisition unit 12, an image forming unit 13, and a communication unit 14.

The control unit 11 controls each unit of the ophthalmic apparatus 10. In particular, the control unit 11 controls the data acquisition unit 12, the image forming unit 13, and the communication unit 14.

The data collection unit 12 collects data of the eye to be inspected (for example, three-dimensional OCT data) by scanning the eye to be inspected using OCT. The data acquisition unit 12 includes an interference optical system 12A and a scanning optical system 12B.

The interference optical system 12A divides the light from the light source (wavelength sweep type light source) into the measurement light and the reference light, and interferes with the return light of the measurement light passing through the eye to be inspected by interfering with the reference light passing through the reference optical path. Generates light and detects the generated interference light. The interference optical system 12A includes at least a fiber coupler and a receiver such as a balanced photodiode. The fiber coupler divides the light from the light source into measurement light and reference light, and causes interference light to be generated by interfering with the return light of the measurement light passing through the eye to be inspected and the reference light passing through the reference optical path. The receiver detects the interference light generated by the fiber coupler. The interference optical system 12A may include a light source.

The scan optical system 12B is controlled by the control unit 11 and changes the incident position of the measurement light on the fundus of the eye to be inspected by deflecting the measurement light generated by the interference optical system 12A. The scanning optical system 12B includes, for example, an optical scanner arranged at a position substantially conjugate with the pupil of the eye to be inspected. The optical scanner includes, for example, a first galvano mirror that deflects the measurement light in the horizontal direction, a second galvano mirror that deflects the measurement light in the vertical direction, and a mechanism that independently drives them. For example, the second galvanometer mirror is configured to further deflect the measurement light deflected by the first galvanometer mirror. Thereby, the measurement light can be scanned in any direction on the fundus plane.

The detection result (detection signal) of the interference light by the interference optical system 12A is an interference signal showing the spectrum of the interference light.

The image forming unit 13 receives control from the control unit 11 and forms image data of a tomographic image of the fundus of the eye to be inspected based on the data of the eye to be inspected collected by the data collecting unit 12. This process includes processing such as noise removal (noise reduction), filter processing, and FFT (Fast Fourier Transform). The image data acquired in this way is a data set including a group of image data formed by imaging reflection intensity profiles in a plurality of A lines (paths of each measurement light in the eye to be inspected). In order to improve the image quality, it is possible to superimpose (add and average) a plurality of data sets collected by repeating scanning with the same pattern a plurality of times.

The image forming unit 13 can form a B-scan image, a C-scan image, an en-face image, a projection image, a shadow gram, or the like by performing various renderings on the three-dimensional OCT data. An image of an arbitrary cross section such as a B scan image or a C scan image is formed by selecting pixels (pixels, voxels) on a designated cross section from three-dimensional OCT data (image data). The en-face image is formed by flattening a part of three-dimensional OCT data (image data). In some embodiments, the en-face image is formed as an OCT image at a predetermined position in the A scan direction (Z direction, depth direction). The projection image is formed by projecting three-dimensional OCT data in a predetermined direction (Z direction, depth direction, A scan direction). The shadow gram is formed by projecting a part of three-dimensional OCT data (for example, partial data corresponding to a specific layer) in a predetermined direction.

The ophthalmic apparatus 10 according to some embodiments is provided with a data processing unit that performs various data processing (image processing) and analysis processing on the image formed by the image forming unit 13. For example, the data processing unit executes correction processing such as image brightness correction and dispersion correction. The data processing unit can form volume data (voxel data) of the eye to be inspected by executing known image processing such as interpolation processing for interpolating pixels between tomographic images. When displaying an image based on volume data, the data processing unit performs rendering processing on the volume data to form a pseudo three-dimensional image when viewed from a specific line-of-sight direction.

Each of the control unit 11 and the image forming unit 13 includes a processor. The processor is, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), a programmable logic device (for example, a SPLD (Simple Programmable Computer)) (Field Programmable Gate Array)) and the like. The function of the image forming unit 13 is realized by the image forming processor. In some embodiments, the functions of both the control unit 11 and the image forming unit 13 are realized by one processor. In some embodiments, if the ophthalmic apparatus 10 is provided with a data processing unit, the function of the data processing unit is also realized by the processor.

The processor realizes the function according to the embodiment by reading and executing a program stored in a storage circuit or a storage device, for example. At least part of the storage circuit or storage device may be included in the processor. Further, at least a part of the storage circuit and the storage device may be provided outside the processor.

A storage device or the like stores various types of data. The data stored in the storage device or the like includes data (measurement data, imaging data, etc.) acquired by the data collection unit 12, information on the subject and the eye to be inspected, and the like. Various computer programs and data for operating each part of the ophthalmic apparatus 10 may be stored in the storage device or the like.

The communication unit 14 receives control from the control unit 11 and executes a communication interface process for transmitting or receiving information with the ophthalmic information processing device 100.

The ophthalmic apparatus 10 according to some embodiments transmits the image data of the eye to be inspected formed by the image forming unit 13 to the ophthalmic information processing apparatus 100.

The ophthalmologic apparatus 10 according to some embodiments is provided with a fundus camera, a scanning laser ophthalmoscope, and a slit lamp microscope for acquiring an image of the fundus of the eye to be inspected. In some embodiments, the fundus image acquired by the fundus camera is a fluorescein fluorescence fundus angiography image or a fundus autofluorescence test image.

[Ophthalmology information processing device]
3 to 5 show a block diagram of a configuration example of the ophthalmic information processing apparatus 100 according to the embodiment. FIG. 3 shows a functional block diagram of the ophthalmic information processing apparatus 100. FIG. 4 shows a functional block diagram of the analysis unit 200 of FIG. FIG. 5 shows a functional block diagram of the inference model construction unit 210 of FIG.

The ophthalmic information processing device 100 includes a control unit 110, an image forming unit 120, a data processing unit 130, and a communication unit 140.

The image forming unit 120 receives control from the control unit 110 and forms an OCT image from the three-dimensional OCT data acquired by the ophthalmic apparatus 10. The image forming unit 120 can form the above-mentioned image in the same manner as the image forming unit 13.

The data processing unit 130 performs various data processing (image processing) and analysis processing on the image formed by the image forming unit 120. For example, the data processing unit 130 executes correction processing such as image brightness correction and dispersion correction. The data processing unit 130 can form volume data (voxel data) of the eye to be inspected by executing known image processing such as interpolation processing for interpolating pixels between tomographic images. When displaying an image based on volume data, the data processing unit 130 performs rendering processing on the volume data to form a pseudo three-dimensional image when viewed from a specific line-of-sight direction.

Further, the data processing unit 130 (or the image forming unit 120) forms a B scan image, a C scan image, an en-face image, a projection image, a shadow gram, etc. from the OCT image formed based on the three-dimensional OCT data. It is possible to do.

The data processing unit 130 performs predetermined data processing on the formed image of the eye to be inspected. The data processing unit 130 includes an analysis unit 200, an inference model construction unit 210, and an inference unit 220.

The analysis unit 200 performs a predetermined analysis process on the image data of the fundus of the eye to be inspected formed by the image forming unit 120 (or the image data of the fundus of the eye to be inspected acquired by the ophthalmologic apparatus 10). The analysis process according to some embodiments includes a process of specifying a predetermined layer region (layer tissue) of the fundus, an image generation process of a predetermined layer region (layer tissue) of the fundus, and the like.

As shown in FIG. 4, the analysis unit 200 includes a segmentation processing unit 201 and a front image generation unit 202.

The segmentation processing unit 201 identifies a plurality of layer regions in the A scan direction based on the data of the eye to be inspected acquired by the ophthalmic apparatus 10. The segmentation processing unit 201 according to some embodiments analyzes a three-dimensional OCT data to identify a plurality of partial data sets corresponding to a plurality of tissues of the eye to be inspected. The segmentation process is an image process for identifying a specific tissue or tissue boundary. For example, the segmentation processing unit 201 obtains the gradient of the pixel value (luminance value) in each A scan image included in the OCT data, and specifies the position where the gradient is large as the tissue boundary. The A-scan image is one-dimensional image data extending in the depth direction of the fundus. The depth direction of the fundus is defined as, for example, the Z direction, the incident direction of the OCT measurement light, the axial direction, the optical axis direction of the interference optical system, and the like.

In a typical example, the segmentation processing unit 201 analyzes three-dimensional OCT data representing the fundus (retina, chorioretinal membrane, etc.) and the vitreous body to obtain a plurality of partial data sets corresponding to a plurality of layer tissues of the fundus. Identify. Each sub-dataset is defined by the boundaries of the stratified structure. An example of a layered tissue identified as a partial data set is the layered tissue that constitutes the retina. The layered tissues constituting the retina include the inner limiting membrane, the nerve fiber layer, the ganglion cell layer, the inner plexiform layer, the inner nuclear layer, the outer plexiform layer, the outer nuclear layer, the outer limiting membrane, the photoreceptor layer, and the RPE. Further, the segmentation processing unit 201 can identify a partial data set corresponding to a Bruch membrane, a choroid, a sclera, a vitreous body, and the like. The segmentation processing unit 201 can identify partial data sets corresponding to the Choriocapillaris layer, the Sattler layer, and the Haller layer in the choroid. In particular, the segmentation processing unit 201 is described in, for example, in the above-mentioned method of specifying the data in the above-mentioned "Automated segmentation for en face horoidal images obtined by optical coherent tomography tomography by machine learning". It is possible. The segmentation processing unit 201 according to some embodiments identifies a partial data set corresponding to a lesion. Examples of lesions include exfoliation, edema, bleeding, tumors, and drusen.

The segmentation processing unit 201 according to some embodiments specifies a layer structure corresponding to a predetermined number of pixels on the sclera side as a bruff film with respect to the RPE, and sets a partial data set corresponding to the layer structure as a portion of the bruff film. Get as a dataset.

The front image generation unit 202 generates the above-mentioned en-face image as the front image. Specifically, the front image generation unit 202 generates an en-face image at an arbitrary depth position in the layer region corresponding to the Haller layer of the choroid specified by the segmentation processing unit 201. For example, the front image generation unit 202 generates a plurality of en-face images at a plurality of depth positions in the deep layer direction with reference to the boundary position on the shallow layer side of the Haller layer. Assuming that the boundary position on the shallow side of the Haller layer is 0% depth position and the boundary position on the deep layer side is 100% depth position, the front image generation unit 202 starts from, for example, 10% depth position. It is possible to generate multiple en-face images in the range of 50% depth position.

As shown in FIG. 5, the inference model construction unit 210 includes a first learning unit 211, a second learning unit 212, and an inference model generation unit 213.

6 to 8 show an explanatory diagram of the operation of the inference model construction unit 210 of FIG. FIG. 6 schematically shows an operation explanatory diagram of the first learning unit 211 of FIG. FIG. 7 schematically shows an operation explanatory diagram of the second learning unit 212 of FIG. FIG. 8 schematically shows an operation explanatory diagram of the inference model generation unit 213 of FIG.

The first learning unit 211 executes unsupervised machine learning using training data on an autoencoder. In this embodiment, the case where the self-encoder is a variational autoencoder (VAE) will be described, but the self-encoder according to the embodiment is a self other than the variational autoencoder. It may be an encoder.

As shown in FIG. 6, the self-encoder 211B according to the embodiment includes an encoder EN and a decoder DE. The output data of the encoder EN is configured to be input to the input (input layer unit) of the decoder DE. The number of dimensions of the input of the encoder EN is larger than the number of dimensions of the output of the encoder EN. The encoder EN performs dimensional compression of the input data In1 by compressing the input data In1 into a latent space according to a predetermined probability distribution, and converts the input data In1 into a latent variable Va as a feature amount. The number of dimensions of the input of the decoder DE is smaller than the number of dimensions of the output of the decoder DE. The decoder DE performs a conversion opposite to that of the encoder EN, and outputs the output data Out1 corresponding to the latent variable Va.

Each of such an encoder EN and a decoder DE is configured by a neural network. For example, the self-encoder 211B includes an input layer, an intermediate layer, and an output layer. In this case, the encoder EN is composed of the first to first L (L is an integer of 2 or more) input layer unit in the input layer and the first to Mth (M is an integer smaller than L) intermediate layer unit in the intermediate layer. It is composed of a neural network between them. Similarly, the decoder DE is composed of a neural network between the first to Mth intermediate layer units in the intermediate layer and the first to Nth (N is an integer larger than M) output layer unit in the output layer. ..

Each of the first to first L input layer units is coupled to any of the first to Mth intermediate layer units, and a coupling coefficient is assigned between the input layer unit and the intermediate layer unit. In each intermediate layer unit, information obtained by calculating the information obtained from the information of one or more combined input layer units and the coupling coefficient between each bond by a predetermined activation function is input. For example, the information of the first to first L input layer units corresponds to the input data, and the information of the first to M first intermediate layer units corresponds to the latent variable Va.

Similarly, each of the first to Mth intermediate layer units is coupled to any of the first to Nth output layer units, and a coupling coefficient is assigned between the intermediate layer unit and the output layer unit. .. In each output layer unit, information obtained by calculating the information obtained from the information of one or more bonded intermediate layer units and the coupling coefficient between the couplings by a predetermined activation function is input. For example, the information of the first to Nth output layer units corresponds to the output data.

The first learning unit 211 includes the first learning processing unit 211A. The first learning processing unit 211A executes unsupervised machine learning using the training data so that the output data of the decoder DE matches the input data of the encoder EN. As a result, the coupling coefficient between the input layer unit and the intermediate layer unit and the coupling coefficient between the intermediate layer unit and the output layer unit are changed so that the output data of the decoder DE matches the input data of the encoder EN. Will be updated.

In this embodiment, the first learning unit 211 executes unsupervised machine learning using a plurality of en-face images generated by the front image generation unit 202 as training data. The plurality of en-face images are images in the range of 10% depth position to 50% depth position with respect to the boundary position on the shallow layer side of the Haller layer.

The second learning unit 212 executes supervised machine learning using supervised data for the classifier. In this embodiment, the case where the classifier is a support vector machine (SVM) will be described, but the classifier according to the embodiment may be a classifier other than the support vector machine such as a random forest. ..

As shown in FIG. 7, the output data (feature amount) of the encoder EN learned by the first learning unit 211 is input to the classifier CL. That is, the classifier CL is configured to output the inference result (classification result, certainty) corresponding to the feature amount of the input data In2 of the encoder EN as the output data Out2. The classifier CL can change the inference result, for example, by changing the (virtual) identification boundary line for the input data. Discrimination boundaries are updated by supervised machine learning.

The second learning unit 212 includes the second learning processing unit 212A. The second learning processing unit 212A executes supervised machine learning on the classifier CL using the teacher data. Here, the teacher data includes the output data output from the encoder EN corresponding to the labeled data input to the encoder EN, and the label attached to the labeled data. By executing known machine learning using such teacher data, the identification boundary line can be changed.

In this embodiment, the second learning unit 212 executes supervised machine learning using a part of a plurality of en-face images generated by the front image generation unit 202 as teacher data. The teacher data is based on the result of a doctor's judgment in advance whether or not the running pattern of the choroidal blood vessels is symmetric with respect to the image at a depth of 25% based on the boundary position on the shallow layer side of the Label layer. This is the data attached as a label. That is, the teacher data used by the second learning unit 212 includes data labeled with a part of the plurality of training data used by the first learning unit 211.

The inference model generation unit 213 generates an inference model for inferring a label (for example, classification result, certainty, in a broad sense, information) corresponding to the input data. As shown in FIG. 8, the inference model generation unit 213 is obtained by the encoder EN obtained by unsupervised machine learning executed by the first learning unit 211 and the supervised machine learning executed by the second learning unit 212. The above inference model is generated by using the classifier CL. Here, the inference model includes an encoder EN that accepts input data and a classifier CL in which the input is combined with the output of the encoder EN, and the output data of the classifier CL is output as an inference result for the input data.

The inference unit 220 uses the inference model generated by the inference model generation unit 213 to output the output data Out corresponding to the input data In of the inference model as the inference result.

In FIG. 3, the communication unit 140 receives control from the control unit 110 and executes a communication interface process for transmitting or receiving information with the communication unit 14 of the ophthalmic information processing apparatus 100.

The control unit 110 controls each unit of the ophthalmic information processing apparatus 100. In particular, the control unit 110 controls the image forming unit 120, the data processing unit 130, and the communication unit 140. The control unit 110 includes a main control unit 111 and a storage unit 112.

The control unit 110 controls each unit of the ophthalmic system 1 based on an operation instruction signal corresponding to the operation content of the user with respect to the operation device 180.

Each of the control unit 110, the image forming unit 120, and the data processing unit 130 includes a processor. The function of the image forming unit 120 is realized by the image forming processor. The function of the data processing unit 130 is realized by the data processing processor. In some embodiments, at least two functions of the control unit 110, the image forming unit 120, and the data processing unit 130 are realized by one processor.

The storage unit 112 stores various types of data. The data stored in the storage unit 112 includes data (measurement data, imaging data, etc.) acquired by the ophthalmic apparatus 10, image data formed by the image forming unit 120, data processing results by the data processing unit 130, and a test. There is information about the person and the eye to be examined. The storage unit 112 may store various computer programs and data for operating each unit of the ophthalmic information processing apparatus 100.

The ophthalmic information processing device 100 is an example of the "information processing device" according to the embodiment. The encoder EN is an example of the "feature amount extractor" according to the embodiment.

[Operation example]
An operation example of the ophthalmic information processing apparatus 100 according to some embodiments will be described.

9 and 10 show an operation explanatory diagram of the ophthalmic information processing apparatus 100 according to the embodiment. FIG. 9 shows a flow chart of an operation example of the ophthalmic information processing apparatus 100. A computer program for realizing the process shown in FIG. 9 is stored in the storage unit 112. The control unit 110 (main control unit 111) can execute the process shown in FIG. 9 by operating according to this computer program. FIG. 10 is an operation explanatory diagram corresponding to the flow diagram of the operation example of FIG. In FIG. 10, the same parts as those in FIGS. 6 to 8 are designated by the same reference numerals, and the description thereof will be omitted as appropriate.

In FIG. 9, OCT is performed on a plurality of eyes to be inspected in advance using the ophthalmic apparatus 10, and the depth position of 10% is based on the boundary position on the shallow layer side of the Haller layer of the choroid from the obtained OCT data. It is assumed that a plurality of en-face images (for example, 14760 images) in the range from to 50% depth position are formed. It is possible to preliminarily form a plurality of en-face images having different depth positions from the OCT data of one eye to be inspected. In addition, among the obtained plurality of en-face images, the plurality of en-face images at a depth of 25% are labeled in advance according to the determination result of the symmetry of the running pattern of the choroidal blood vessels. (For example, 216 sheets determined to be symmetrical, 236 sheets determined to be asymmetrical).

(S1: Unsupervised machine learning for VAE)
First, the control unit 110 controls the first learning unit 211 so that the output data of the decoder DE matches the input data of the encoder EN for the self-encoder 211B without a teacher using training data. Perform machine learning.

In step S1, a plurality of en-face images (for example, 14760 images) in the range from a depth position of 10% to a depth position of 50% are used as training data based on the boundary position on the shallow layer side of the Haller layer of the choroid. , Unsupervised machine learning is performed using each en-face image.

As a result, in step S1, as shown in FIG. 10, the encoder EN learned by unsupervised machine learning using a large amount of training data is obtained.

(S2: Perform supervised machine learning for SVM)
Subsequently, the control unit 110 controls the second learning unit 212 to attach the output data and the labeled data corresponding to the labeled data input to the encoder EN to the classifier (SVM) CL. Perform supervised machine learning using the above labels as supervised data.

In step S2, a plurality of en-face images (for example, 452 (= 216 + 236) images) at a depth of 25% based on the boundary position on the shallow layer side of the Haller layer of the choroid are used as supervised data for each en-face. Supervised machine learning is performed using images. Specifically, each en-face image is input to the encoder EN learned by unsupervised machine learning in step S1, the output data (feature amount) of the obtained encoder EN is input to the classifier CL, and the classifier Train the classifier CL so that the output data of CL matches the label of the teacher data.

As a result, in step S2, as shown in FIG. 10, a classifier CL learned by supervised machine learning using a small amount of teacher data is obtained.

(S3: Generate an inference model)
After that, the control unit 110 controls the inference model generation unit 213 to generate the inference model used in the inference unit 220. Specifically, the inference model generation unit 213 generates an inference model using the encoder EN obtained in step S1 and the classifier CL obtained in step S2.

As a result, in step S3, as shown in FIG. 10, an inference model capable of obtaining a highly accurate output by machine learning using a small amount of teacher data can be obtained.

Using the inference model generated in step S3, the output data corresponding to the input data of the encoder EN obtained in step S1 is output as the feature amount of the input data, and the above-mentioned classifier CL obtained in step S2 outputs the output data. It is possible to infer the label corresponding to the input data based on the features.

That is, the encoder EN constituting the inference model includes an encoder constituting a self-encoder learned by unsupervised machine learning using training data, and the output data corresponding to the input data of the encoder EN is a feature of the input data. Output as a quantity. Further, the classifier CL constituting the inference model is learned by supervised machine learning using the output data corresponding to the labeled data input to the encoder EN and the label attached to the labeled data as supervised data. The label corresponding to the input data is inferred based on the feature amount output by the encoder EN.

This completes the operation of the ophthalmic information processing device 100 (end).

As described above, according to the embodiment, it is possible to obtain a highly accurate output by machine learning using a small amount of teacher data.

Here, when the training data (here, the teacher data) is small, the area under the receiver operating characteristic curve (AREA Under Receiver Operating Character Curve: AUROC) is evaluated for the convolutional neural network and the inference model according to the embodiment.

11A and 11B show an example of the evaluation result of AUROC of the convolutional neural network and the inference model according to the embodiment. FIG. 11A shows an example of the evaluation result of AUROC of the convolutional neural network. FIG. 11B shows an example of the evaluation result of AUROC of the inference model according to the embodiment.

In FIGS. 11A and 11B, 45 cases, which are a part of the teacher data of 452 cases used in supervised machine learning in the embodiment, are used as training data, and the remaining 407 cases are used as validation data (that is, the learning data is small). The evaluation result of cross-validation of 10 divisions is expressed as AUROC.

In the evaluation result of the convolutional neural network shown in FIG. 11A, AUROC = 0.747 ± 0.064. In the evaluation result of the inference model according to the embodiment shown in FIG. 11B, AUROC = 0.841 ± 0.021. In Wilcoxon signed-rank sum test, where the p-value of the significance level is "0.05", the evaluation results of FIGS. 11A and 11B are significant because the p-value is 0.00503. It is considered that there is a difference. When the amount of training data is small, the data dependency becomes high, and the variance of AUROC becomes large in the convolutional neural network. That is, according to the embodiment, when the training data is small, the data dependency is low, and the label corresponding to the input data can be inferred with higher accuracy than the convolutional neural network.

<Modification example>
The configurations according to some embodiments are not limited to the above configurations.

The ophthalmic apparatus according to some embodiments includes at least one of the functions of the ophthalmic information processing apparatus 100, the operation device 180, and the display device 190, in addition to the functions of the ophthalmic apparatus 10.

Hereinafter, the ophthalmic apparatus according to the modified examples of some embodiments will be described focusing on the differences from the ophthalmic apparatus according to the above-described embodiment.

FIG. 12 shows a block diagram of a configuration example of the ophthalmic apparatus 10a according to the modified example of the embodiment. In FIG. 12, the same parts as those in FIG. 2 are designated by the same reference numerals, and the description thereof will be omitted as appropriate.

The configuration of the ophthalmic apparatus 10a according to the present modification is different from the configuration of the ophthalmic apparatus 10 according to the above embodiment in that the ophthalmic apparatus 10a has the functions of the ophthalmic information processing apparatus 100, the functions of the operating apparatus 180, and the display apparatus. The point is that it has 190 functions. The ophthalmic apparatus 10a includes a control unit 11a, a data acquisition unit 12, an image forming unit 13, an ophthalmic information processing unit 15a, an operation unit 16a, and a display unit 17a.

The control unit 11a controls each unit of the ophthalmic apparatus 10a. In particular, the control unit 11a controls the data acquisition unit 12, the image forming unit 13, the ophthalmic information processing unit 15a, the operation unit 16a, and the display unit 17a.

The ophthalmic information processing unit 15a has the same configuration as the ophthalmic information processing device 100, and has the same functions as the ophthalmic information processing device 100. The operation unit 16a has the same configuration as the operation device 180, and has the same functions as the operation device 180. The display unit 17a has the same configuration as the display device 190, and has the same functions as the display device 190.

According to this modification, it is possible to provide an ophthalmic apparatus capable of obtaining a highly accurate output by machine learning using a small amount of teacher data with a compact configuration.

<Other variants>
In the above embodiment or a modification thereof, a case where it is inferred whether or not the running pattern of the choroidal blood vessel is symmetric is described by using an inference model learned by using the training data and the teacher data of the Haller layer in the choroid. did. However, the configuration according to the embodiment is not limited to this. For example, it is possible to classify the running patterns of choroidal blood vessels into a plurality of predetermined pattern types by using an inference model learned in the same manner as in the embodiment using the training data and the teacher data of the Haller layer in the choroid. is there. Also, for example
It is possible to classify into multiple disease types included in the pachychoroid spectrum disease group (pachychoroid spectrum disease group) using an inference model learned in the same manner as in the embodiment using the training data and teacher data of the Haller layer in the choroid. Is.

Further, in the above embodiment or a modified example thereof, an inference model learned in the same manner as in the embodiment using training data and teacher data of a fundus photograph (for example, a color fundus photograph) acquired by using a fundus camera or the like is used. It can be used to classify whether the eye is normal or glaucoma. Further, for example, using the inference model learned in the same manner as in the embodiment using the training data and the teacher data of the fundus photograph (for example, the color fundus photograph) acquired by using the fundus camera or the like, a predetermined part (for example). It is possible to classify whether or not the optic disc) is bleeding. In addition, retinoblastoma and choroidal malignancy are also used by using an inference model learned in the same manner as in the embodiment using training data and teacher data of fundus photographs (for example, color fundus photographs) acquired using a fundus camera or the like. It is possible to classify the presence or absence of eye tumors such as melanoma.

<Effect>
Hereinafter, the information processing apparatus, the inference model generation method, the information processing method, and the program according to some embodiments will be described.

The information processing devices (ophthalmology information processing device 100, ophthalmology information processing unit 15a) according to some embodiments include a first learning unit (211), a second learning unit (212), and an inference model generation unit (213). And include. The first learning unit makes the output data of the decoder match the input data of the encoder with respect to the self-encoder (211B) including the encoder (EN) and the decoder (DE) into which the output data of the encoder is input. Perform unsupervised machine learning using training data. The second learning unit executes supervised machine learning on the classifier (CL) using the output data corresponding to the labeled data input to the encoder and the label attached to the labeled data as teacher data. .. The inference model generator generates an inference model for inferring the label corresponding to the input data of the encoder by using the encoder obtained by unsupervised machine learning and the classifier obtained by supervised machine learning.

With such a configuration, it is possible to generate an inference model that has low data dependency and can infer the label corresponding to the input data with high accuracy even with a small amount of teacher data as compared with the convolutional neural network. It will be possible.

In some embodiments, the training data is image data representing a frontal image (en-face image) of the fundus in which the blood vessels in the choroid are depicted, and the classifier is the running pattern of the blood vessels depicted in the frontal image. Infer whether it is symmetric or not.

With such a configuration, it becomes possible to generate an inference model capable of inferring with high accuracy whether or not the running pattern of blood vessels in the choroid is symmetric by using a small amount of teacher data. ..

The information processing apparatus (ophthalmology information processing apparatus 100, ophthalmology information processing unit 15a) according to some embodiments includes a feature amount extractor (encoder EN) and a classifier (CL). The feature extractor includes an encoder that constitutes a self-encoder (212b) learned by unsupervised machine learning using training data, and outputs output data corresponding to the input data of the encoder as the feature amount of the input data. To do. The classifier is learned by supervised machine learning using the output data corresponding to the labeled data input to the encoder and the label attached to the labeled data as supervised data, and the features output by the feature extractor. Infer the label corresponding to the input data based on the quantity.

According to such a configuration, even with a small amount of teacher data, the data dependency is low as compared with the convolutional neural network, and the label corresponding to the input data can be inferred with high accuracy.

In some embodiments, the training data is image data representing a frontal image (en-face image) of the fundus in which the blood vessels in the choroid are depicted, and the labeled data is whether or not the running pattern of the blood vessels is symmetric. This is image data representing a front image with a label indicating the above.

With such a configuration, it is possible to infer with high accuracy whether or not the running pattern of blood vessels in the choroid is symmetric by using a small amount of teacher data.

In some embodiments, the frontal image is a frontal image of the Haller layer in the choroid.

According to such a configuration, by using the front image of the Haller layer, which is considered to be easy to grasp the state of the choroidal blood vessels, it can be inferred with higher accuracy whether or not the running patterns of the blood vessels in the choroid are symmetrical. It becomes possible to do.

In some embodiments, the labeled data includes data with a portion of the training data labeled.

According to such a configuration, it is possible to infer the label corresponding to the input data with higher accuracy by preparing a small amount of labeled data.

In some embodiments, the training data includes image data of a plurality of frontal images at multiple positions within a predetermined depth range on the fundus.

With such a configuration, it becomes possible to create a plurality of training data from the measurement data (for example, OCT data) acquired for one eye to be inspected. As a result, the accuracy of learning can be further improved, and the label corresponding to the input data can be inferred with even higher accuracy.

In some embodiments, the teacher data includes image data of a frontal image of the fundus at a predetermined depth position within a predetermined depth range.

According to such a configuration, it is possible to use a front image at a predetermined depth position from a plurality of front images within a predetermined depth range for supervised machine learning. As a result, training data and teacher data can be efficiently collected, learning accuracy can be further improved, and labels corresponding to input data can be inferred with even higher accuracy.

The inference model generation method according to some embodiments includes a first learning step, a second learning step, and an inference model generation step. In the first learning step, the output data of the decoder matches the input data of the encoder with respect to the self-encoder (212b) including the encoder (EN) and the decoder (DE) to which the output data of the encoder is input. Unsupervised machine learning using training data is performed. In the second learning step, supervised machine learning is executed on the classifier (CL) using the output data corresponding to the labeled data input to the encoder and the label attached to the labeled data as teacher data. Will be done. In the inference model generation step, an inference model for inferring the label corresponding to the input data of the encoder is generated using the encoder obtained by unsupervised machine learning and the classifier obtained by supervised machine learning. ..

According to such a method, it is possible to generate an inference model that has low data dependency even with a small amount of teacher data and can infer the label corresponding to the input data with high accuracy as compared with the convolutional neural network. It will be possible.

In some embodiments, the labeled data includes data with a portion of the training data labeled.

According to such a method, it is possible to infer the label corresponding to the input data with higher accuracy by preparing a small amount of labeled data.

In some embodiments, the training data is image data representing a frontal image (en-face image) of the fundus in which the blood vessels in the choroid are depicted, and the classifier is the running pattern of the blood vessels depicted in the frontal image. Infer whether it is symmetric or not.

According to such a method, it is possible to infer with high accuracy whether or not the running pattern of blood vessels in the choroid is symmetric by using a small amount of teacher data.

In some embodiments, the frontal image is a frontal image of the Haller layer in the choroid.

According to such a method, by using a frontal image of the Haller layer, which is considered to be easy to grasp the state of choroidal blood vessels, it is more accurately inferred whether or not the running pattern of blood vessels in the choroid is symmetrical. It becomes possible to do.

In some embodiments, the training data includes image data of a plurality of frontal images at multiple positions within a predetermined depth range on the fundus.

According to such a method, a plurality of training data can be created from the measurement data (for example, OCT data) acquired for one eye to be inspected. As a result, the accuracy of learning can be further improved, and the label corresponding to the input data can be inferred with even higher accuracy.

In some embodiments, the teacher data includes image data of a frontal image of the fundus at a predetermined depth position within a predetermined depth range.

According to such a method, it is possible to use a front image at a predetermined depth position from a plurality of front images within a predetermined depth range for supervised machine learning. As a result, training data and teacher data can be efficiently collected, learning accuracy can be further improved, and labels corresponding to input data can be inferred with even higher accuracy.

The information processing method according to some embodiments includes a feature amount extraction step and an inference step. In the feature extraction step, the output data corresponding to the input data of the encoder is used as the input data by using the encoder (EN) constituting the self-encoder (212b) learned by unsupervised machine learning using the training data. It is output as a feature amount. In the inference step, a classifier (CL) learned by supervised machine learning using the output data corresponding to the labeled data input to the encoder and the label attached to the labeled data as supervised data is used. The label corresponding to the input data is inferred based on the feature amount output in the feature amount extraction step.

According to such a method, even with a small amount of teacher data, the data dependency is low as compared with the convolutional neural network, and the label corresponding to the input data can be inferred with high accuracy.

In some embodiments, the labeled data includes data with a portion of the training data labeled.

According to such a method, it is possible to infer the label corresponding to the input data with higher accuracy by preparing a small amount of labeled data.

In some embodiments, the training data is image data representing a frontal image (en-face image) of the fundus in which the blood vessels in the choroid are depicted, and the labeled data is whether or not the running pattern of the blood vessels is symmetric. This is image data representing a front image with a label indicating the above.

According to such a method, it is possible to infer with high accuracy whether or not the running pattern of blood vessels in the choroid is symmetric by using a small amount of teacher data.

In some embodiments, the frontal image is a frontal image of the Haller layer in the choroid.

According to such a method, by using a frontal image of the Haller layer, which is considered to be easy to grasp the state of choroidal blood vessels, it is more accurately inferred whether or not the running pattern of blood vessels in the choroid is symmetrical. It becomes possible to do.

In some embodiments, the training data includes image data of a plurality of frontal images at multiple positions within a predetermined depth range on the fundus.

According to such a method, a plurality of training data can be created from the measurement data (for example, OCT data) acquired for one eye to be inspected. As a result, the accuracy of learning can be further improved, and the label corresponding to the input data can be inferred with even higher accuracy.

In some embodiments, the teacher data includes image data of a frontal image of the fundus at a predetermined depth position within a predetermined depth range.

According to such a method, it is possible to use a front image at a predetermined depth position from a plurality of front images within a predetermined depth range for supervised machine learning. As a result, training data and teacher data can be efficiently collected, learning accuracy can be further improved, and labels corresponding to input data can be inferred with even higher accuracy.

The program according to some embodiments causes a computer to perform each step of the inference model generation method described in any of the above.

According to such a program, it is possible to generate an inference model that has low data dependency and can infer the label corresponding to the input data with high accuracy even with a small amount of teacher data as compared with the convolutional neural network. It will be possible to provide possible programs.

The program according to some embodiments causes a computer to perform each step of the information processing method described in any of the above.

According to such a program, it is possible to provide a program that has low data dependency even with a small amount of teacher data and can infer the label corresponding to the input data with high accuracy as compared with the convolutional neural network. become able to.

A program for realizing an inference model generation method or an information processing method according to some embodiments can be stored in any computer-readable non-transitory recording medium. The recording medium may be an electronic medium using magnetism, light, magneto-optical, semiconductor, or the like. Typically, the recording medium is a magnetic tape, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, a solid state drive, or the like.

It is also possible to send and receive computer programs through networks such as the Internet and LAN.

The embodiments described above are merely examples for carrying out the present invention. A person who intends to carry out the present invention can make arbitrary modifications (omission, substitution, addition, etc.) within the scope of the gist of the present invention.

1 Ophthalmology system 10, 10a Ophthalmology device 11, 11a, 110 Control unit 12 Data collection unit 12A Interference optical system 12B Scan optical system 13, 120 Image formation unit 14, 140 Communication unit 15a Ophthalmology information processing unit 16a Operation unit 17a Display unit 100 Ophthalmic information processing device 111 Main control unit 112 Storage unit 130 Data processing unit 180 Operation device 190 Display device 200 Analysis unit 201 Segmentation processing unit 202 Front image generation unit 210 Inference model construction unit 211 First learning unit 211A First learning processing unit 211B Self-encoder 212 2nd learning unit 212A 2nd learning processing unit 213 Inference model generation unit 220 Inference unit EN encoder DE decoder

Claims (22)

For a self-encoder including an encoder and a decoder to which the output data of the encoder is input, unsupervised machine learning using training data is performed so that the output data of the decoder matches the input data of the encoder. 1st learning department and
A second learning unit that performs supervised machine learning on a classifier using the output data corresponding to the labeled data input to the encoder and the label attached to the labeled data as teacher data.
An inference model that generates an inference model for inferring a label corresponding to the input data of the encoder by using the encoder obtained by the unsupervised machine learning and the classifier obtained by the supervised machine learning. Generator and
Information processing equipment, including. The training data is image data representing a frontal image of the fundus in which blood vessels in the choroid are depicted.
The information processing apparatus according to claim 1, wherein the classifier infers whether or not the traveling pattern of the blood vessel depicted in the front image is symmetrical. A feature amount extractor that includes an encoder that constitutes a self-encoder learned by unsupervised machine learning using training data and outputs output data corresponding to the input data of the encoder as a feature amount of the input data.
The feature that was learned by supervised machine learning using the output data corresponding to the labeled data input to the encoder and the label attached to the labeled data as supervised data and output by the feature extractor. A classifier that infers the label corresponding to the input data based on the quantity, and
Information processing equipment, including. The training data is image data representing a frontal image of the fundus in which blood vessels in the choroid are depicted.
The information processing apparatus according to claim 3, wherein the labeled data is image data representing the front image with a label indicating whether or not the traveling pattern of the blood vessel is symmetrical. The information processing apparatus according to claim 2 or 4, wherein the front image is a front image of the Haller layer in the choroid. The information processing apparatus according to any one of claims 1 to 5, wherein the labeled data includes data in which a part of the training data is labeled. The information processing apparatus according to any one of claims 1 to 6, wherein the training data includes image data of a plurality of front images at a plurality of positions within a predetermined depth range on the fundus. .. The information processing apparatus according to claim 7, wherein the teacher data includes image data of a front image of the fundus at a predetermined depth position within the predetermined depth range. For a self-encoder including an encoder and a decoder to which the output data of the encoder is input, unsupervised machine learning using training data is performed so that the output data of the decoder matches the input data of the encoder. The first learning step and
A second learning step of performing supervised machine learning on a classifier using the output data corresponding to the labeled data input to the encoder and the label attached to the labeled data as teacher data, and
An inference model that generates an inference model for inferring a label corresponding to the input data of the encoder by using the encoder obtained by the unsupervised machine learning and the classifier obtained by the supervised machine learning. Generation steps and
Inference model generation methods, including. The inference model generation method according to claim 9, wherein the labeled data includes data in which a part of the training data is labeled. The training data is image data representing a frontal image of the fundus in which blood vessels in the choroid are depicted.
The inference model generation method according to claim 9, wherein the classifier infers whether or not the traveling pattern of the blood vessel depicted in the front image is symmetrical. The inference model generation method according to claim 11, wherein the front image is a front image of the Haller layer in the choroid. The inference model generation according to any one of claims 9 to 12, wherein the training data includes image data of a plurality of front images at a plurality of positions within a predetermined depth range on the fundus. Method. The inference model generation method according to claim 13, wherein the teacher data includes image data of a front image of the fundus at a predetermined depth position within the predetermined depth range. A feature amount extraction step that outputs output data corresponding to the input data of the encoder as a feature amount of the input data by using an encoder that constitutes a self-encoder learned by unsupervised machine learning using training data. ,
The feature amount extraction step using a classifier learned by supervised machine learning using the output data corresponding to the labeled data input to the encoder and the label attached to the labeled data as supervised data. The inference step of inferring the label corresponding to the input data based on the feature amount output in
Information processing methods, including. The information processing method according to claim 15, wherein the labeled data includes data in which a part of the training data is labeled. The training data is image data representing a frontal image of the fundus in which blood vessels in the choroid are depicted.
The information according to claim 15 or 16, wherein the labeled data is image data representing the front image with a label indicating whether or not the traveling pattern of the blood vessel is symmetric. Processing method. The information processing method according to claim 17, wherein the front image is a front image of the Haller layer in the choroid. The information processing method according to any one of claims 15 to 18, wherein the training data includes image data of a plurality of front images at a plurality of positions within a predetermined depth range on the fundus. .. The information processing method according to claim 19, wherein the teacher data includes image data of a front image of the fundus at a predetermined depth position within the predetermined depth range. A program comprising causing a computer to execute each step of the inference model generation method according to any one of claims 9 to 14. A program comprising causing a computer to execute each step of the information processing method according to any one of claims 15 to 20.

Images