JP2019208852A - Ophthalmologic image processing apparatus and ophthalmologic image processing program - Google Patents

Abstract

To provide an ophthalmologic image processing apparatus capable of presenting useful information, and an ophthalmologic image processing program.SOLUTION: An ophthalmologic image processing apparatus for processing an ophthalmologic image includes: image acquisition means for acquiring the ophthalmologic image of a subject's eye, which is taken by photographing means; and image processing means for outputting a prediction image predicting the change of the subject's eye, by inputting the ophthalmologic image of the subject's eye into a mathematical model trained on the basis of a machine learning algorithm.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an ophthalmic image processing apparatus and an ophthalmic image processing program for processing an ophthalmic image of an eye to be examined.

  In recent years, neural networks have attracted attention in the field of image recognition (see Patent Document 1) and are expected to be applied to medical images.

  Deep learning based on neural networks is mainly used for image recognition and segmentation, and in recent years, image generation is also actively performed.

JP 2017-219960 A

  By the way, in the ophthalmology field, when explaining the state of the eye to a patient, a clinical image is shown for explanation. However, since only a typical eye disease image is shown, there is a case where information is insufficient for the patient himself to understand the disease state.

  In view of the conventional problems, it is an object of the present disclosure to provide an ophthalmic image processing apparatus and an ophthalmic image processing program capable of presenting useful information.

  In order to solve the above problems, the present disclosure is characterized by having the following configuration.

(1) An ophthalmologic image processing apparatus for processing an ophthalmologic image, wherein an ophthalmologic image of the eye to be examined is acquired by an image acquisition means for acquiring an ophthalmologic image of the eye to be examined photographed by the photographing means and a mathematical model trained by a machine learning algorithm Image processing means for outputting a predicted image obtained by predicting a change in the eye to be examined by inputting an image.
(2) An ophthalmic image processing program that is executed in an ophthalmic image processing apparatus that processes an ophthalmic image, and is executed by a processor of the ophthalmic image processing apparatus so that an ophthalmic image of an eye to be inspected captured by an imaging unit is obtained. An image acquisition step for acquiring an image processing step for outputting a predicted image in which a change in the eye to be examined is predicted by inputting the ophthalmic image of the eye to be examined into a mathematical model trained by a machine learning algorithm. The image processing apparatus is executed.

It is a figure which shows schematic structure of an ophthalmic image processing apparatus. It is a figure which shows the optical system of an imaging device. It is a figure which shows about a mathematical model and the relationship between an input and an output. It is a figure which shows a mode when the progress degree of a disease is input into the mathematical model. It is a flowchart for demonstrating control operation. It is a figure at the time of outputting the prediction image of a thickness map from a mathematical model. It is a figure at the time of outputting the prediction image of a blood vessel image from a mathematical model. It is a figure at the time of outputting the prediction image after a treatment from a mathematical model.

<Embodiment>
An embodiment according to the present disclosure will be described. The ophthalmic image processing apparatus (for example, ophthalmic image processing apparatus 100) of the embodiment processes an ophthalmic image. The ophthalmic image processing apparatus mainly includes, for example, an image acquisition unit (for example, image acquisition unit 171) and an image processing unit (for example, image processing unit 172). The image acquisition unit acquires, for example, an ophthalmic image captured by an imaging unit (for example, the ophthalmologic imaging apparatus 1). For example, the image processing unit inputs an ophthalmic image of the eye to be examined into a mathematical model (network) trained by a machine learning algorithm, and outputs a predicted image in which a change in the eye to be examined is predicted. The change of the eye to be examined is, for example, a tissue change or a shape change of the eye to be examined.

  Since the ophthalmologic image processing apparatus generates a predicted image based on the subject's own image, the examiner can easily image future changes in the subject's eye. Moreover, since it is possible to explain changes in eyes according to each subject, it is easy for the subject to understand. Furthermore, it can be used as a training tool for letting the trainee learn eye changes.

  Note that the predicted image may be an image in which a change when the eye to be examined becomes a disease is predicted. Thereby, the examiner can easily imagine the change when the eye to be examined becomes a disease. Moreover, since it is possible to explain the eye change due to a disease according to each subject using the predicted image of the subject himself, it is easy for the subject to understand.

  Note that the predicted image may be an image in which a change when a disease of the eye to be examined is treated is predicted. Thereby, the examiner can easily imagine the change when the eye to be examined is treated. Moreover, since the change of the eye by treatment can be explained according to each subject using the predicted image of the subject himself, it is easy for the subject to understand.

  In addition, you may further provide the information acquisition part (information acquisition part 177) which acquires additional information. In this case, the image processing unit may output the predicted image by inputting additional information together with the ophthalmic image into the mathematical model. Thus, the examiner can generate a predicted image corresponding to the additional information. For example, the examiner can generate a desired predicted image by designating additional information.

  The additional information may be a disease name, for example. The disease name is various diseases such as age-related macular degeneration (AMD), diabetic retinopathy, glaucoma and the like. By inputting a disease name together with an ophthalmic image into a mathematical model, a prediction image when a desired disease is obtained can be generated.

  The additional information may be a treatment method, for example. The treatment method is, for example, various operations or medications. By inputting the treatment method together with the ophthalmic image into the mathematical model, a predicted image when the desired treatment is performed can be generated.

  Note that the additional information may include the degree of progression of a disease or treatment. In this case, the image processing unit may generate a predicted image based on the degree of progress. As a result, a predicted image that shows changes in the eye to be examined in stages can be generated. Note that the degree of progress may be, for example, a level divided into a plurality of stages such as an initial stage, a middle period, and an end stage.

  Further, the additional information may include position information of a disease or treatment. In this case, the image processing unit may generate a predicted image based on the position information. For example, the information acquisition unit may acquire a position designated by the examiner as additional information. Thus, the examiner can create a predicted image in which a desired position of the ophthalmic image of the eye to be examined has changed.

  Note that the image processing unit may generate a moving image in which a change in the eye to be examined is predicted by generating a plurality of predicted images. For example, the image processing unit may generate a moving image by connecting predicted images generated for each degree of progress. This makes it easier for the examiner to visually recognize changes in the eye to be examined.

  The imaging unit that captures an ophthalmologic image may include, for example, any one of an optical system such as a fundus camera, OCT, SLO, or Shine-Pluke camera, or may be a multifunction device. The image acquisition unit may acquire an image captured by these optical systems. Note that the image processing unit may input a plurality of ophthalmic images into the mathematical model. The plurality of ophthalmic images may be images taken with the same modality, or may be images taken with a plurality of modalities. When inputting ophthalmic images taken in a plurality of modalities into a mathematical model, a plurality of predicted images corresponding to each modality may be output, or a predicted image corresponding to a modality representing a plurality of modalities may be output. Good. In this case, ophthalmic images other than the representative modalities are input to the mathematical model as additional information.

Note that the image processing unit may perform a process of extracting blood vessels on the ophthalmologic image, and input the extracted blood vessel image into the mathematical model. At this time, a blood vessel arteriovenous identification result may be given.
For example, the image processing unit may output a predicted fundus image when the normal eye becomes a disease by inputting a blood vessel extraction image of the normal eye into the mathematical model. In this case, an image obtained by extracting blood vessels from a diseased eye may be created, the blood vessel extraction image may be input, and a fundus image with a disease may be learned as a correct answer value by the mathematical model. Thereby, even when a normal fundus image and a disease image corresponding to the normal fundus image are not available, the mathematical model can be learned. The image processing unit may input both the blood vessel extraction image and the fundus image into the mathematical model. In this case, since color information can be acquired from the fundus image, generation of a fundus image having a different hue can be reduced.

  Note that the processor of the eye image processing apparatus may execute an ophthalmic image processing program stored in a storage unit or the like. The ophthalmic image processing program includes, for example, an image acquisition step and an image processing step. The image acquisition step is, for example, an image acquisition step for acquiring an ophthalmic image captured by the imaging unit. The image processing step is a step of generating a predicted image in which a tissue change of the eye to be examined is predicted by inputting an ophthalmic image of the eye to be examined into a mathematical model trained by a machine learning algorithm.

  Note that the image processing unit may output a thickness map indicating the thickness of the retinal layer as a predicted image. The thickness map is created based on data acquired by OCT, for example. By outputting the thickness map as a predicted image, the examiner can easily find a disease in which the thickness of the retina changes from the predicted image. The image processing unit may output a blood vessel image obtained by extracting blood vessels from the ophthalmic image as a predicted image. This makes it easier for the examiner to find a disease whose blood vessel shape or blood vessel number changes from the predicted image. Further, the image processing unit may output the thickness map and the blood vessel image as a predicted image, or may output an image obtained by combining the thickness map and the blood vessel image as a predicted image. Thus, the examiner can confirm the relationship between changes in the thickness of the retina and the state of the blood vessels. Thus, the predicted image is not limited to a specific image and may be various images. The image processing unit may output a predicted image of a type different from the input ophthalmic image. The image processing unit may output more predicted images than the input images, or may output fewer predicted images than the input images.

  Note that the image processing unit may indicate a changed position in the predicted image. In this case, the image processing unit may detect the changed position by comparing the ophthalmic image input to the mathematical model with the output predicted image. Further, the image processing unit may detect a changed position by performing a disease detection process on the predicted image. The image processing unit may highlight the changed position in the predicted image. For example, the image processing unit may add a specific color or a mark to the image. By indicating the changed position in the predicted image, the possibility that the examiner misses the change in the predicted image can be reduced.

  The image processing unit may analyze the ophthalmic image using a network trained by a machine learning algorithm. As a machine learning algorithm, for example, a neural network is generally known.

  A neural network is a network in which a plurality of units that output the result of performing some processing on an input signal are connected. A convolutional network, which is a kind of neural network, is an effective technique for image recognition problems. Typical image recognition problems include object identification, object detection, and segmentation, and various models have been proposed for each problem.

  Object identification is a problem of identifying what is shown in an image, and models such as AlexNet and GoogLeNet have been proposed. Object detection is a problem that identifies the position and orientation of an object as a rectangle, and models such as YOLO have been proposed. Semantic segmentation is a problem of identifying where an object is located at the pixel level, and a model such as U-Net has been proposed. In addition to this, a convolution network model has been proposed for various problems targeting images such as super-resolution that increases the resolution of an image and colorization that adds color to a monochrome image.

  An example of a neural network model other than a convolutional network is a recursive neural network. A recursive neural network is a model that can handle time-series data, and is an effective technique for natural language processing.

  The network refers to a structure that expresses a relationship between input data and output data, for example. The network is optimized based on the training data set. The training data set is a set of input training data and output training data. The input training data is sample data input to the network. For example, for the input training data, an image of the eye to be examined that was taken in the past is used. The output training data is sample data of values that the network should predict for the input training data. For example, for the output training data, an image when the eye to be examined becomes a disease, a segmentation result of the image, a map indicating a region where the disease exists in the image, or a label indicating whether the disease exists in the image is used. . The network is optimized so that the output when certain input training data is input approaches the corresponding output training data. For example, in a neural network, the weight applied when inputting the output of a unit to the next connected unit is updated.

  While there are object identification and segmentation using a neural network, a technique for generating an image using a neural network is also known. Although the network structure is arbitrary, image generation is composed of a network that generates an image called a generator and a network that distinguishes whether an image called a discriminator is a fake image generated from the generator or a real image. The generator grows by competing with each other so that the image generated by the generator can be mistaken for the real discriminator, and the discriminator can discriminate the image generated by the generator from the fake and the real image. By repeating learning in this way, the Generator can generate realistic images.

<Example>
Hereinafter, embodiments according to the present disclosure will be described with reference to the drawings. The ophthalmologic image processing apparatus according to the present embodiment generates a predicted image in which a change in the eye to be examined is predicted by processing the ophthalmic image using a mathematical model.

  As shown in FIG. 1, the ophthalmic image processing apparatus 100 includes, for example, an image acquisition unit 171, an image processing unit 172, a storage unit 173, a display control unit 174, a display unit 175, an operation unit 176, an information acquisition unit 177, and the like. . The image acquisition unit 171 acquires an image of the eye to be examined. The image acquisition unit 171 is connected to the ophthalmologic photographing apparatus 1 via a wired or wireless communication means. For example, the image acquisition unit 171 receives an ophthalmic image from the ophthalmologic photographing apparatus 1 via the communication unit, and stores it in the storage unit 73 or the like. Note that the image acquisition unit 171 may acquire an ophthalmic image from an external storage device such as an HDD or a USB memory connected via a communication unit.

  The image processing unit 172 performs image processing using a mathematical model trained by a machine learning algorithm. For example, the image processing unit 172 outputs a predicted image in which a change in the eye to be examined is predicted by inputting an ophthalmic image obtained by photographing the eye to be examined into a mathematical model. The processing result by the image processing unit 172 is sent to the display unit 175, the storage unit 173, or the like.

  The storage unit 173 stores various programs related to control of the ophthalmic image processing apparatus 100, various image data, processing results, and the like. The display control unit 174 controls display on the display unit 175. The display unit 175 displays the image acquired by the image acquisition unit 171 and the processing result by the image processing unit 172. The display unit 175 may be a touch panel display. In this case, the display unit 175 is also used as the operation unit 176.

  The information acquisition unit 177 acquires additional information. The additional information includes, for example, a disease name, a treatment method, a disease or progress of treatment, location information, and the like. For example, the information acquisition unit 177 may acquire additional information based on an operation signal output by the operation unit 176 being operated by the examiner. Further, the information acquisition unit 177 may automatically acquire additional information from an ophthalmologic imaging apparatus connected to the ophthalmologic image treatment apparatus 100 by a communication unit or the like, or an external storage unit.

  The image acquisition unit 171, the image processing unit 172, the storage unit 173, the display control unit 174, and the information acquisition unit 177 are, for example, a computer processor (for example, the control unit 170) used as the ophthalmic image processing apparatus 100. It may be realized by executing various programs, or may be provided as independent control boards.

  The ophthalmic image processing apparatus 100 may be a personal computer, for example. For example, a desktop PC, a notebook PC, or a tablet PC may be used as the ophthalmic image processing apparatus 100. Of course, it may be a server. The ophthalmic image processing apparatus 100 may be a computer stored inside the ophthalmic imaging apparatus 1.

<Ophthalmologic imaging device>
Next, the ophthalmologic photographing apparatus 1 will be described. The ophthalmologic photographing apparatus 1 photographs an image of an eye to be examined. The image acquisition unit 171 is connected to the ophthalmologic photographing apparatus 1 through a communication unit, and acquires an ophthalmic image photographed by the ophthalmic photographing apparatus 1. In this embodiment, a case where a fundus camera is used as the ophthalmologic photographing apparatus 1 will be described. Of course, it is not limited to the fundus camera, and may be an ophthalmologic image photographed by an OCT (Optical Coherence Tomography), SLO (Scanning Laser Ophthalmoscope), a Shine-Pluke camera, an anterior eye photographing device, a corneal endothelial cell photographing device, or the like.

  Hereinafter, an outline of the ophthalmologic photographing apparatus 1 will be described with reference to FIG. For example, the ophthalmologic photographing apparatus 1 acquires the fundus image of the eye to be examined by irradiating the eye E with measurement light and receiving the reflected light. The ophthalmologic photographing apparatus 1 mainly includes, for example, an illumination optical system 10 and a photographing optical system (front photographing optical system) 30. The photographing optical system 30 is used as a fundus camera optical system for obtaining an infrared fundus image, a color fundus image, and the like by photographing the fundus (for example, a non-mydriatic state). The ophthalmologic photographing apparatus 1 may include a focus index projection optical system 40, an alignment index projection optical system 50, and an anterior ocular segment observation optical system 60.

<Illumination optics>
The illumination optical system 10 includes, for example, an observation illumination optical system and a photographing illumination optical system. The photographing illumination optical system mainly includes a light source 14, a condenser lens 15, a ring slit 17, a relay lens 18, a mirror 19, a black spot plate 20, a relay lens 21, a perforated mirror 22, and an objective lens 25. The photographing light source 14 may be a flash lamp or the like. The black spot plate 20 has a black spot at the center.

  The observation illumination optical system mainly includes an optical system from the light source 11, the infrared filter 12, the condenser lens 13, the dichroic mirror 16, and the ring slit 17 to the objective lens 25. The light source 11 may be, for example, a halogen lamp. For example, the infrared filter 12 transmits near infrared light having a wavelength of 750 nm or more. The dichroic mirror 16 is disposed, for example, between the condenser lens 13 and the ring slit 17. Further, the dichroic mirror 16 has a characteristic of reflecting light from the light source 11 and transmitting light from the photographing light source 14, for example.

<Photographing optical system>
In the photographic optical system 30, for example, an objective lens 25, a photographing aperture 31, a focusing lens 32, an imaging lens 33, and an image sensor 35 are mainly disposed. The photographing aperture 31 is located in the vicinity of the aperture of the perforated mirror 22. The focusing lens 32 is movable in the optical axis direction. The image sensor 35 can be used for imaging having sensitivity in the visible range, for example. For example, the photographing aperture 31 is disposed at a position substantially conjugate with the pupil of the eye E with respect to the objective lens 25. The focusing lens 32 is moved in the optical axis direction by a moving mechanism 49 including a motor, for example.

  A dichroic mirror 37 having a characteristic of reflecting part of infrared light and visible light and transmitting most of visible light is disposed between the imaging lens 33 and the image sensor 35. In the reflection direction of the dichroic mirror 37, an imaging device for observation 38 having sensitivity in the infrared region is disposed. Instead of the dichroic mirror 37, a flip-up mirror may be used. For example, the flip-up mirror is inserted into the optical path during fundus observation, and is retracted from the optical path during fundus imaging.

  In addition, between the objective lens 25 and the perforated mirror 22, for example, an insertable / detachable dichroic mirror (wavelength selective mirror) 24 as an optical path branching member is provided obliquely. The dichroic mirror 24 reflects, for example, the wavelength light of the OCT measurement light and the wavelength light (center wavelength 940 nm) of the alignment index projection optical system 50 and the anterior ocular segment illumination light source 58.

  The dichroic mirror 24 has a characteristic of transmitting a wavelength of 900 nm or less including the light source wavelength (center wavelength 880 nm) of the wavelength light of the fundus observation illumination, for example. When photographing is performed by the photographing optical system 30, the dichroic mirror 24 is flipped up by the insertion / removal mechanism 66 and retracted out of the optical path. The insertion / removal mechanism 66 can be composed of a solenoid and a cam.

  In addition, the optical path correction glass 28 is disposed on the image pickup device 35 side of the dichroic mirror 24 so that it can be flipped up by driving the insertion / removal mechanism 66. When the optical path is inserted, the optical path correction glass 28 has a role of correcting the position of the optical axis L1 shifted by the dichroic mirror 24.

  The light beam emitted from the observation light source 11 is converted into an infrared light beam by the infrared filter 12 and reflected by the condenser lens 13 and the dichroic mirror 16 to illuminate the ring slit 17. The light transmitted through the ring slit 17 reaches the perforated mirror 22 through the relay lens 18, the mirror 19, the black spot plate 20, and the relay lens 21. The light reflected by the perforated mirror 22 passes through the correction glass 28 and the dichroic mirror 24, and once converges near the pupil of the eye E to be examined by the objective lens 25, and then diffuses to illuminate the fundus of the eye to be examined.

  Reflected light from the fundus is obtained through an imaging element 38 via an objective lens 25, a dichroic mirror 24, a correction glass 28, an aperture of a perforated mirror 22, an imaging aperture 31, a focusing lens 32, an imaging lens 33, and a dichroic mirror 37. To form an image. The output of the image sensor 38 is input to the control unit 70, and the control unit 70 displays a fundus observation image of the eye to be inspected captured by the image sensor 38 on the display unit 75.

  Further, the light beam emitted from the photographing light source 14 passes through the dichroic mirror 16 via the condenser lens 15. Thereafter, the fundus is illuminated with visible light through the same optical path as the illumination light for fundus observation. Then, the reflected light from the fundus is imaged on the image sensor 35 through the objective lens 25, the opening of the perforated mirror 22, the imaging aperture 31, the focusing lens 32, and the imaging lens 33.

<Focus index projection optical system>
The focus index projection optical system 40 mainly includes an infrared light source 41, a slit index plate 42, two declination prisms 43, a projection lens 47, and a spot mirror 44 obliquely provided in the optical path of the illumination optical system 10. The two declination prisms 43 are attached to the slit indicator plate 42. The spot mirror 44 is provided obliquely in the optical path of the illumination optical system 10. The spot mirror 44 is fixed to the tip of the lever 45. The spot mirror 44 is normally inclined to the optical axis, but is retracted out of the optical path by rotation of the rotary solenoid 46 at a predetermined timing before photographing.

  The spot mirror 44 is arranged at a position conjugate with the fundus of the eye E. The light source 41, the slit indicator plate 42, the deflection prism 43, the projection lens 47, the spot mirror 44 and the lever 45 are moved in the optical axis direction by the moving mechanism 49 in conjunction with the focusing lens 32. Further, the light flux of the slit index plate 42 of the focus index projection optical system 40 is reflected by the spot mirror 44 via the deflection prism 43 and the projection lens 47, and then the relay lens 21, the perforated mirror 22, the dichroic mirror 24, The light is projected onto the fundus of the eye E through the objective lens 25. When the fundus is out of focus, the two index images are projected onto the fundus in a state of being separated according to the shift direction and the shift amount. On the other hand, when the focus is achieved, the two index images are projected onto the fundus in a matched state. Then, the two index images are captured together with the fundus image by the image sensor 38.

<Alignment index projection optical system>
The alignment index projection optical system 50 projects an alignment index beam onto the eye E. In the alignment index projection optical system 50, as shown in the diagram in the lower left dotted line in FIG. 2, a plurality of infrared light sources are arranged at 45 degree intervals on a concentric circle with the photographing optical axis L1 as the center. The ophthalmologic photographing apparatus in the present embodiment mainly includes a first index projection optical system (0 degrees and 180) and a second index projection optical system.

  The first index projection optical system has an infrared light source 51 and a collimating lens 52. The second index projection optical system is arranged at a different position from the first index projection optical system and has six infrared light sources 53. The infrared light sources 51 are arranged symmetrically with respect to a vertical plane passing through the photographing optical axis L1.

  In this case, the first index projection optical system projects an index at infinity on the cornea of the eye E from the left-right direction. The second index projection optical system is configured to project a finite index on the cornea of the eye E from the vertical direction or the oblique direction. In FIG. 2, for convenience, the first index projection optical system (0 degrees and 180 degrees) and only a part of the second index projection optical system (45 degrees and 135 degrees) are shown. .

<Anterior segment observation optical system>
An anterior ocular segment observation (imaging) optical system 60 that images the anterior ocular segment of the eye E to be examined is provided on the reflection side of the dichroic mirror 24 with a diaphragm 63, a relay lens 64, a two-dimensional image sensor (hereinafter abbreviated as an image sensor 65). Mainly provided with 65). The image sensor 65 has infrared sensitivity. The imaging element 65 also serves as an imaging means for detecting the alignment index, and the anterior segment illuminated by the anterior segment illumination light source 58 that emits infrared light and the alignment index are imaged. The anterior segment illuminated by the anterior segment illumination light source 58 is received by the imaging element 65 from the objective lens 25 and the dichroic mirror 24 via the optical system of the relay lens 64. Further, the alignment light beam emitted from the light source of the alignment index projection optical system 50 is projected onto the eye cornea to be examined. The cornea reflection image is received (projected) on the image sensor 65 through the objective lens 25 to the relay lens 64.

  The output of the two-dimensional image sensor 65 is input to the control unit 70, and the anterior segment image captured by the two-dimensional image sensor 65 is displayed on the display unit 75. The anterior ocular segment observation optical system 60 also serves as a detection optical system for detecting the alignment state of the apparatus main body with respect to the eye to be examined.

  An infrared light source for forming an optical alignment index (working dot) on the cornea of the subject's eye around the hole of the perforated mirror 22 (two in this embodiment, but not limited to this) 55 is arranged. The light source 55 may have a configuration in which infrared light is guided to an optical fiber having an end face disposed in the vicinity of the perforated mirror 22. The corneal reflected light from the light source 55 is imaged on the imaging surface of the imaging element 38 when the working distance between the subject eye E and the imaging unit 3 (apparatus body) is appropriate. Thus, the examiner can finely adjust the alignment using the working dots formed by the light source 55 in a state where the fundus image is displayed on the display unit 75.

<Mathematical model>
In this embodiment, when an ophthalmologic image is processed by the image processing unit 172, a mathematical model trained by a machine learning algorithm is used. The mathematical model of the present embodiment generates and outputs a predicted image in which a change in the eye to be examined is predicted in response to an ophthalmic image input. The image processing unit 71 inputs an ophthalmic image to the mathematical model and outputs a predicted image.

  The mathematical model is composed of a plurality of convolution layers, for example. As the structure of the mathematical model, a network called U-net may be used, or a network optimized for ophthalmic images may be used. An arbitrary network structure may be used for the mathematical model of the present embodiment.

  FIG. 3 is a diagram showing the input / output relationship of the mathematical model 200 used in this embodiment. In the example of FIG. 3, the fundus image P <b> 1 is input to the mathematical model 200. The mathematical model 200 outputs, with respect to the fundus image P1 input by the image processing unit 172, a predicted image Q1 that predicts the state of the fundus when the eye to be examined becomes a disease.

  When learning the mathematical model 200 of FIG. 3, for example, an ophthalmic image of a normal eye and an image (disease image) when the normal eye has a disease are learned. For example, the error between the predicted image output from the mathematical model 200 and the correct answer value (disease image) with respect to the input ophthalmic image of normal eyes is calculated, and the weight of the mathematical model 200 is updated. By repeating this until the error becomes small, a weight suitable for the output of the predicted image is obtained.

  Note that the image processing unit 172 may input not only the ophthalmic image but also the additional information 210 acquired by the information acquisition unit 177 to the mathematical model 200. For example, the disease name 211 may be input as the additional information 210. For example, the image processing unit 172 inputs a label of a disease name 211 such as diabetic retinopathy, age-related macular degeneration, glaucoma and the like together with the fundus image P1 to the mathematical model 200. Thus, a predicted image Q1 when the eye to be examined becomes a disease corresponding to the disease name 211 is output from the mathematical model 200. For example, when a fundus image P1 and a label “diabetic retinopathy” are input, a predicted image Q1 when the subject's eye becomes diabetic retinopathy is output. Similarly, when the label “aged macular degeneration” is input together with the fundus oculi image P1, a predicted image Q1 when the eye to be examined becomes age-related macular degeneration is output, and the label “glaucoma” is output together with the fundus oculi image P1. When input, a predicted image Q1 when the eye to be examined has glaucoma is output.

  Note that the image processing unit 172 may input the degree of disease progression as the additional information 210. For example, the image processing unit 172 may input the degree of progress divided into a plurality of stages into the mathematical model 200. FIG. 4 is an example when the fundus image P1 and the degree of progression 212 are input to the mathematical model 200. In the example of FIG. 4, three stages of progress 212 of the initial stage, the middle period, and the final stage are input. For example, in FIG. 4A, together with the fundus oculi image P1, “diabetic retinopathy” is input as the disease name 211, and “initial” is input as the progression degree 212. In this case, an initial predicted image Q2 of diabetic retinopathy is output from the mathematical model 200. Similarly, in the example of FIG. 4B, “diabetic retinopathy” is input as the disease name 211, and “medium term” is input as the progression degree 212, and the mathematical model 200 predicts the middle stage of diabetic retinopathy. An image Q3 is output. In the example of FIG. 4C, “diabetic retinopathy” is input as the disease name 211, and “end stage” is input as the degree of progression 212, and the predicted image of the end stage of diabetic retinopathy is calculated from the mathematical model 200. Q4 is output. As described above, the mathematical model 200 outputs a predicted image corresponding to the input degree 212 of progress. As a result, the examiner can confirm the progression of the disease step by step.

  Note that when additional information 210 such as a disease name 211 or a degree of progression 212 is input, the mathematical model is learned in the same manner as described above. For example, an ophthalmological image of normal eyes, a disease name, a degree of progression, and an image (disease image) when the normal eye becomes a disease are learned. For example, an ophthalmic image of a normal eye, a label of a disease name, a prediction image output from the mathematical model 200 in response to an input of a progress label, and a correct value (disease specified by the label, disease image of the progress) And the weight of the mathematical model 200 is updated. By repeating this until the error becomes small, a weight suitable for the output of the predicted image is obtained.

<Control action>
The control operation of the ophthalmic image processing apparatus 1 according to this embodiment will be described with reference to FIG. In the following description, a case where a fundus image photographed by the ophthalmologic photographing apparatus 1 is processed is shown.

(Step S1: Ophthalmic image acquisition)
First, when a fundus image of the eye to be examined is photographed by the ophthalmologic photographing apparatus 1, the image acquisition unit 171 acquires the fundus image. The image acquisition unit 171 acquires a fundus image via, for example, a wireless or wired communication unit. For example, the image acquisition unit 171 may acquire a fundus image directly from the imaging device 35 of the ophthalmic imaging apparatus 1 or may acquire a fundus image stored in the storage unit 74 or the like of the ophthalmic imaging apparatus 1. The image acquisition unit 171 may acquire a fundus image from the ophthalmologic photographing apparatus 1 or may acquire a fundus image stored in an external storage device such as a USB memory.

(Step S2: Acquisition of additional information)
Subsequently, the information acquisition unit 177 acquires additional information 210 to be input to the mathematical model 200. For example, the information acquisition unit 177 acquires the additional information 210 based on the operation signal output from the operation unit 76 in accordance with the examiner's operation. The information acquisition unit 177 acquires, for example, a disease name, a disease progress level, and the like.

(Step S3: Input to mathematical model)
The image processing unit 172 processes the fundus image using the mathematical model 200. For example, the image processing unit 172 inputs the fundus image P1 acquired by the image acquisition unit 171 and the label of the additional information 210 acquired by the information acquisition unit 177 to the mathematical model 200.

(Step S4: Output of predicted image)
When the fundus oculi image P1, the additional information 210, and the like are input, calculation is performed by the mathematical model 200, and a predicted image Q1 corresponding to the additional information 210 is output. For example, a predicted image when the eye to be examined becomes a disease from the mathematical model 200 by inputting a fundus image P1 of a normal eye and a disease state such as a disease name and a degree of progression of disease to the mathematical model 200. Q1 is output.

  As described above, the ophthalmologic image processing apparatus 100 according to the present embodiment generates a predicted image when a disease occurs based on the subject's own eye image, and therefore, the examiner is suitable for each subject. An explanation of the disease can be made. In addition, since a predicted image is generated from the subject's own image, the examiner can obtain information useful for future diagnosis of each subject.

  Further, as described above, additional information can be input to the mathematical model together with the ophthalmic image, so that predicted images corresponding to various situations can be created. In addition, by creating a predicted image when a disease occurs based on an ophthalmologic image of normal eyes, it can also be used for training for residents.

  As additional information 210, disease position information may be input to the mathematical model 200. By inputting the position of a disease, it is possible to output a prediction image when a disease occurs at a specific position. In this case, the examiner inputs a position where a disease is likely to occur in the fundus image of the subject using the operation unit 176. The information acquisition unit 177 acquires disease position information based on an operation signal from the operation unit 176. The image processing unit 172 inputs disease position information together with the fundus image into the mathematical model 200. The mathematical model 200 outputs a predicted image when a disease occurs at that position.

  The ophthalmologic image processing apparatus 100 may generate a predicted image by predicting a therapeutic effect when symptoms are improved by treating from a disease state. For example, a diseased fundus image and a treatment method label may be input to the mathematical model 200 and a predicted image after treatment may be output. For example, as shown in FIG. 6, a fundus image P2 with a disease and a label of a treatment method 213 are input to the mathematical model 200 to output a predicted image Q5 when the disease is treated. Thus, the examiner can easily explain the treatment suitable for each patient. At this time, the progress of treatment may be input to the mathematical model 200. For example, a prediction image corresponding to the progress of the treatment may be output by inputting the progress of the treatment at a plurality of stages such as the initial stage, the middle stage, and the final stage. Thereby, the examiner can confirm the therapeutic effect in stages.

  The predicted image output from the mathematical model 200 is not only a normal color image but also a retinal layer thickness map Q6 as shown in FIG. 7 or a blood vessel image Q7 obtained by extracting blood vessels in the fundus as shown in FIG. There may be.

  Note that the image processing unit 172 may generate not only a still image as a predicted image but also a moving image. For example, a moving image in which a tissue change of the eye to be examined is predicted may be generated by generating a plurality of predicted images. For example, a moving image in which the progression of the disease gradually progresses may be generated. Thus, by generating the predicted moving image, it becomes easy to understand the tissue change of the eye to be examined.

  Note that the fundus image input to the mathematical model 200 may be an image as acquired by the image acquisition unit 171 or an image with noise added. By adding noise, the mathematical model can be learned efficiently. Alternatively, a blood vessel image may be once extracted from the fundus image and the blood vessel image may be used as the input image.

  Note that the ophthalmologic image input to the mathematical model 200 may be not only a fundus front image captured by a fundus camera, but also a fundus tomographic image captured by OCT or an En-face image. Further, it may be an anterior segment image.

DESCRIPTION OF SYMBOLS 1 Ophthalmic imaging device 70 Control part 74 Storage part 100 Ophthalmology image processing apparatus 170 Control part 171 Image acquisition part 172 Image processing part 173 Storage part 174 Display control part 175 Display part 177 Information acquisition part

Claims (5)

An ophthalmic image processing apparatus for processing an ophthalmic image,
Image obtaining means for obtaining an ophthalmic image of the eye to be examined photographed by the photographing means;
An ophthalmic image processing comprising: an image processing unit that outputs a predicted image in which a change in the subject eye is predicted by inputting the ophthalmic image of the subject eye into a mathematical model trained by a machine learning algorithm apparatus.   The ophthalmic image processing apparatus according to claim 1, wherein the predicted image is an image in which a change when the eye to be examined becomes a disease is predicted.   The ophthalmic image processing apparatus according to claim 1, wherein the predicted image is an image in which a change when the disease of the eye to be examined is treated is predicted. It further comprises information acquisition means for acquiring additional information,
The ophthalmic image processing apparatus according to claim 1, wherein the image processing unit outputs the predicted image by inputting the additional information together with the ophthalmic image into the mathematical model. An ophthalmic image processing program that is executed in an ophthalmic image processing apparatus that processes an ophthalmic image, and is executed by a processor of the ophthalmic image processing apparatus,
An image acquisition step of acquiring an ophthalmologic image of the eye to be examined imaged by the imaging means;
Causing the ophthalmic image processing apparatus to execute an image processing step of outputting a predicted image in which a change in the subject eye is predicted by inputting an ophthalmic image of the subject eye into a mathematical model trained by a machine learning algorithm An ophthalmologic image processing program.

Images