JP2020000678A - Ophthalmic image processing apparatus, oct apparatus, ophthalmic image processing program, and method of building mathematical model - Google Patents

Abstract

To provide an ophthalmic image processing apparatus, an OCT apparatus, an ophthalmic image processing program, and a method of building a mathematical model which allow suitable acquisition of high-resolution images from low-resolution images.SOLUTION: A control unit of the ophthalmic image processing apparatus obtains a base image captured by an ophthalmic imaging apparatus. The control unit obtains a target image that has a higher image resolution than the base image by inputting the base image into a mathematical model that is trained with a machine learning algorithm. The mathematical model uses a portion of the same tissues as an imaging target and is trained with low-resolution training images and high-resolution training images contained in a set of plurality of ophthalmic images captured by the same ophthalmic imaging apparatus.SELECTED DRAWING: Figure 2

Description

  The present disclosure relates to an ophthalmologic image processing apparatus that processes an ophthalmologic image of a subject's eye, an OCT apparatus, an ophthalmologic image processing program executed by the ophthalmologic image processing apparatus, and a mathematical model construction that constructs a mathematical model used by the ophthalmologic image processing apparatus About the method.

  In recent years, techniques for acquiring various medical information using a mathematical model trained by a machine learning algorithm have been proposed. For example, in the ophthalmologic apparatus described in Patent Literature 1, by inputting eye shape parameters to a mathematical model trained by a machine learning algorithm, IOL-related information (for example, a predicted predicate anterior chamber depth) of the subject's eye is acquired. You. The IOL frequency is calculated based on the acquired IOL-related information.

  It is also conceivable to obtain a high-resolution image from a low-resolution image using a mathematical model trained by a machine learning algorithm. In this case, a high-resolution image is output by inputting a low-resolution image to the mathematical model.

JP 2018-51223 A

  One aspect of the technology according to the present disclosure will be described. When acquiring a high-resolution image from a low-resolution image using a mathematical model trained by a machine learning algorithm, the mathematical model is obtained by using a low-resolution image and a high-resolution image of the same part. Must be pre-trained. That is, an operator who constructs a mathematical model has to prepare many sets of low-resolution images and high-resolution images of the same part and train the mathematical model. Preparing many sets of images requires a great deal of effort.

  Other aspects will be described. In order to properly train a mathematical model, it is necessary that the correspondence between a low-resolution image and a high-resolution image used for training is appropriate. When a mathematical model is trained using a set of images with improper correspondence, it is difficult to obtain good high-resolution images using the mathematical model.

  An object of the present disclosure is to solve at least one of the above aspects. That is, a typical object of the present disclosure is to provide an ophthalmologic image processing apparatus, an OCT apparatus, an ophthalmologic image processing program, and a mathematical model construction method for appropriately acquiring a high-resolution image from a low-resolution image. It is.

  An ophthalmologic image processing device provided by a typical embodiment of the present disclosure is an ophthalmologic image processing device that processes an ophthalmic image that is an image of a tissue of an eye to be examined, and a control unit of the ophthalmologic image processing device includes an ophthalmic image processing device. Obtaining a base image that is an ophthalmological image taken by an imaging device, and inputting the base image to a mathematical model trained by a machine learning algorithm, a target image whose image resolution is higher than the image resolution of the base image Acquiring the mathematical model, a low-resolution training image that is an at least one ophthalmologic image among a set of a plurality of ophthalmologic images captured by the same ophthalmologic image capturing apparatus, with a portion of the same tissue being a capturing target; Is used as input training data, and a high-resolution training image that is an ophthalmic image having a higher image resolution than the low-resolution training image is trained as output training data. That.

  An OCT apparatus provided in a typical embodiment of the present disclosure processes an OCT signal based on reference light and reflected light of measurement light applied to a tissue of an eye to be examined, thereby capturing an ophthalmologic image of the tissue. An OCT apparatus, wherein the control unit of the OCT apparatus obtains a captured ophthalmic image as a base image, and inputs the base image to a mathematical model trained by a machine learning algorithm, so that an image resolution is the base image. Acquiring a target image that is higher than the image resolution of, the mathematical model targets the same tissue site, and at least one of a set of a plurality of ophthalmologic images captured by the same ophthalmologic image capturing apparatus. A low-resolution training image that is one ophthalmic image is used as input training data, and a high-resolution training image that is an ophthalmic image having a higher image resolution than the low-resolution training image is used for output. They are trained as paste data.

  An ophthalmologic image processing program provided by a typical embodiment of the present disclosure is an ophthalmologic image processing program executed by an ophthalmologic image processing apparatus that processes an ophthalmologic image that is an image of a tissue of an eye to be inspected, wherein the ophthalmologic image processing program The program is executed by the control unit of the ophthalmologic image processing apparatus, so that a basic image obtaining step of obtaining a basic image that is an ophthalmic image captured by the ophthalmic image capturing apparatus, and a mathematical model trained by a machine learning algorithm. By inputting the base image, a target image obtaining step of obtaining a target image whose image resolution is higher than the image resolution of the base image, and causing the ophthalmologic image processing apparatus to execute the mathematical model, the mathematical model Of the set of a plurality of ophthalmic images captured by the same ophthalmologic image capturing apparatus, Without even the input training data to the low resolution training image which is one of ophthalmic images, the image resolution is trained as output training data of high-resolution training images, which is a high ophthalmologic image than the low-resolution training images.

  A mathematical model construction method provided by an exemplary embodiment according to the present disclosure is a mathematical model construction method that is constructed by being trained by a machine learning algorithm, and constructs a mathematical model that outputs a target image when a base image is input. A method for constructing a mathematical model executed by an apparatus, comprising: acquiring an image set for acquiring a set of a plurality of ophthalmic images captured by the same ophthalmic image capturing apparatus, wherein the same part of a tissue of an eye to be inspected is captured. And at least one low-resolution training image, which is at least one ophthalmic image, of the set as input training data, and outputting a high-resolution training image, which is an ophthalmic image having a higher image resolution than the low-resolution training image, as output training data. Training the mathematical model as a training step to build the mathematical model.

  According to the ophthalmologic image processing apparatus, the OCT apparatus, the ophthalmologic image processing program, and the mathematical model building method according to the present disclosure, a high-resolution image is appropriately acquired from a low-resolution image.

1 is a block diagram illustrating a schematic configuration of a mathematical model construction device 1, an ophthalmologic image processing device 21, and ophthalmologic image capturing devices 11A and 11B. 3 is a flowchart of a mathematical model construction process executed by the mathematical model construction device 1. FIG. 4 is an explanatory diagram for explaining a training data set 30 including a high-resolution training image 31 and a low-resolution training image 32. It is an explanatory view for explaining the contents of training of a mathematical model in this embodiment. 6 is a flowchart of ophthalmologic image processing executed by the ophthalmologic image processing device 21. FIG. 3 is a diagram illustrating an example of a composite image 40 of a target image 42 and a region of interest image 44.

<Overview>
The control unit of the ophthalmologic image processing device exemplified in the present disclosure acquires a base image that is an ophthalmologic image captured by the ophthalmologic image capturing device. The control unit acquires a target image whose image resolution is higher than the image resolution of the base image by inputting the base image to the mathematical model trained by the machine learning algorithm. The mathematical model is trained by a training data set which is a set of input training data and output training data. The training data set includes a plurality of ophthalmic images captured by the same ophthalmologic image capturing apparatus with the same tissue site as a capturing target. As the input training data, a low-resolution training image that is at least one ophthalmic image in the training data set is used. As the output training data, a high-resolution training image that is an ophthalmic image having a higher image resolution than the low-resolution training image in the training data set is used.

  That is, the mathematical model construction device that constructs the mathematical model executes the image set acquisition step and the training step. In the image set obtaining step, a set of a plurality of ophthalmologic images obtained by capturing the same site of the tissue of the eye to be imaged and being captured by the same ophthalmologic image capturing apparatus is obtained. In the training step, a low-resolution training image that is at least one ophthalmic image of the acquired set is used as input training data, and a high-resolution ophthalmic image that has a higher image resolution than the low-resolution training image in the set. A mathematical model is trained using the resolution training image as output training data.

  In this case, the low-resolution training image and the high-resolution training image included in the training data set are both captured by the same ophthalmologic image capturing apparatus. Therefore, the training data set is obtained more efficiently than when the low-resolution training image and the high-resolution training image are captured by different ophthalmic imaging apparatuses. In addition, the correspondence between the low-resolution training image and the high-resolution training image included in the training data set tends to be more appropriate than when the low-resolution training image and the high-resolution training image are captured by different ophthalmic imaging apparatuses. (For example, the color of the image, the amount of noise included in the image, the artifact, the optical resolution, the imaging angle of view, the state of the tissue at the time of imaging, and the like are easily approximated between the low-resolution training image and the high-resolution training image. Therefore, a better target image is obtained by using the trained mathematical model.

  In addition, the “image resolution” in the present disclosure means the fineness (that is, the fineness of an image) for describing a shooting target. In other words, the “image resolution” in the present disclosure is the number of pixels of an image corresponding to a unit area on a tissue to be imaged.

  Further, the low-resolution training image and the high-resolution training image included in the training data set are not limited to the ophthalmic image itself (that is, the original image) captured by the same ophthalmologic image capturing apparatus. For example, at least one of the low-resolution training image and the high-resolution training image may be an image generated based on an original image captured by an ophthalmologic image capturing apparatus. In other words, when at least one of the low-resolution training image and the high-resolution training image has been processed, the original image on which the low-resolution training image and the high-resolution training image are based is captured by the same ophthalmologic image capturing apparatus. It should just be done.

  The base image, the low-resolution training image, and the high-resolution training image may be OCT angio images of the fundus of the subject's eye captured by the OCT apparatus. The fundus blood vessels of the eye to be examined appear in the OCT angio image. Therefore, by using the OCT angiographic image, a high-resolution target image in which a fundus blood vessel appears is appropriately acquired based on the base image.

  Note that the OCT angio image may be a two-dimensional front image in which the fundus is viewed from the front (that is, the line of sight of the subject's eye). The OCT angio image may be, for example, a motion contrast image obtained by processing at least two OCT signals obtained at different times for the same position.

  However, it is also possible to use an image other than the OCT angio image of the fundus as the ophthalmologic image used as the training data set and the base image input to the mathematical model. For example, at least one of a tomographic image (two-dimensional tomographic image or three-dimensional tomographic image) captured by an OCT device, an image captured by a fundus camera, an image captured by a laser scanning optometry device (SLO), and the like, It may be used as a training data set and a base image. Further, the training data set and the base image may be images obtained by photographing tissues other than the fundus of the eye to be inspected (for example, the anterior segment of the eye to be inspected).

  The high-resolution training image used in at least one of the plurality of training data sets is a panoramic image generated by aligning a plurality of ophthalmic images captured in each of a plurality of imaging regions that do not completely overlap. You may. That is, the ophthalmologic image capturing apparatus may capture an ophthalmic image (hereinafter, referred to as a “unit image”) for each of a plurality of capturing regions that do not completely overlap. A panoramic image may be generated by aligning a plurality of captured unit images. The image data acquisition step executed by the mathematical model construction device may include a panorama image acquisition step of acquiring the generated panorama image as a high-resolution training image.

  In this case, by capturing a unit image having a high image resolution, a high-resolution image obtained by capturing a wide range of the tissue is used as a high-resolution training image of the training data set. Therefore, a high-resolution training image of the training data set is more appropriately acquired.

  The low-resolution training image used for at least one of the plurality of training data sets may be an image generated by reducing the image resolution of a panoramic image that is a high-resolution training image. That is, the image data obtaining step executed by the mathematical model construction device may include a low-resolution training image generating step of generating a low-resolution training image by reducing the image resolution of the panoramic image.

  In this case, in at least one of the training data sets, the shooting range, the shooting position, and the state of the tissue at the time of shooting match between the low-resolution training image and the high-resolution training image. Therefore, a better target image is obtained by using the trained mathematical model.

  When training a mathematical model, a plurality of training data sets are used for training. Here, when a panoramic image is used as the high-resolution training image of the training data set, not all of the plurality of high-resolution training images need to be panoramic images, and at least a part of the plurality of high-resolution training images is a panoramic image. I just need. When the ratio of the panoramic image to the plurality of high-resolution training images is large (preferably 50% or more, more preferably 70% or more), the high-resolution training image is more easily acquired. Similarly, when an image having a reduced image resolution of a panoramic image (hereinafter, referred to as a “low-resolution panoramic image”) is used for training, it is not necessary that all of the plurality of low-resolution training images are low-resolution panoramic images. When the ratio of the low-resolution panoramic image to the plurality of low-resolution training images is large (preferably 50% or more, more preferably 70% or more), the quality of the target image is further improved.

  However, it is also possible to change the specific method for generating the low-resolution training image and the high-resolution training image of the training data set. For example, an ophthalmic image that is not a panoramic image may be used as a high-resolution training image of the training data set. Further, a low-resolution training image of a training data set may be generated by reducing the image resolution of an ophthalmic image that is not a panoramic image. In addition, a plurality of ophthalmologic images generated by capturing the same tissue a plurality of times at different image resolutions may be used as a low-resolution training image and a high-resolution training image of a training data set. Further, when a panoramic image is used as a high-resolution training image, an image obtained by shooting a shooting range equivalent to the panoramic image at a low resolution may be used as a low-resolution training image of the training data set.

  The low-resolution training image used for at least one of the plurality of training data sets may have been subjected to at least a part of the image region to a deterioration process of deteriorating the image quality. That is, the mathematical model construction device may execute a deterioration step of deteriorating the image quality on at least a part of the image region of the low-resolution training image.

  In this case, even when the image quality of at least a part of the base image is not good, the mathematical model is trained to output the target image in which the influence of the poor image quality is reduced. Therefore, a more appropriate target image is obtained.

  The specific contents of the deterioration process can be appropriately selected. For example, the control unit of the mathematical model construction device may perform at least one of a noise imparting process for imparting noise and a blur imparting process for imparting blur.

  The control unit of the ophthalmologic image processing apparatus may execute at least one of the simultaneous display processing and the switching display processing. In the simultaneous display processing, the control unit causes the display device to simultaneously display the base image and the target image. In the switching display process, the control unit switches between the base image and the target image according to the instruction input by the user and displays the target image on the display device. In this case, the user can easily check both the target image having a high image resolution and the base image (original image) on which the target image is based. Therefore, the user can perform diagnosis and the like more appropriately. Note that the base image and the target image may be displayed on the same display device, or may be displayed on different display devices.

  The control unit may sequentially acquire the base images continuously photographed by the ophthalmologic photographing apparatus. The control unit may sequentially acquire the target images by sequentially inputting the acquired base images to the mathematical model. The control unit may generate moving image data of the organization based on the acquired target images. In this case, the user can confirm the moving image of the organization displayed at a high image resolution. Therefore, the user can perform diagnosis and the like more appropriately.

  In particular, when a high-resolution panoramic image is generated, a panoramic image is generated from a plurality of unit images. Therefore, when a moving image of a tissue is displayed using the panoramic image, the frame rate tends to be significantly reduced. In particular, when the photographing position changes with the passage of time, it is difficult to generate a favorable moving image using a panoramic image. On the other hand, the ophthalmologic image processing apparatus according to the present disclosure can generate a moving image that captures a wide range with high resolution while suppressing a decrease in the frame rate.

  Note that the control unit may display a high-resolution moving image of the tissue captured by the ophthalmologic image capturing apparatus on the display device in real time in parallel with the capturing. The control unit may cause the display device to simultaneously display the moving image of the base image and the moving image of the target image. In addition, the control unit may switch the moving image of the base image and the moving image of the target image to be displayed on the display device in accordance with an instruction input by the user. In this case, the user can easily check both the moving image of the target image and the moving image of the base image.

  The control unit of the ophthalmologic image processing apparatus may execute the attention area image obtaining step and the combining step. In the attention area image acquiring step, an attention area image photographed by the ophthalmologic image photographing apparatus is acquired. The attention area image is an image in which the attention area of the shooting area of the base image is shot at an image resolution higher than the image resolution of the base image. In the combining step, the original image of the attention area image is combined with the attention area in the target image. In this case, in the synthesized image, an attention area is formed by an original image having a high image resolution, and an area other than the attention area is formed by a target image. Therefore, the user can check the attention area with the original image, and can check the entire image at a high resolution.

<Embodiment>
(Device configuration)
Hereinafter, one of exemplary embodiments of the present disclosure will be described with reference to the drawings. As shown in FIG. 1, in this embodiment, a mathematical model construction device 1, an ophthalmologic image processing device 21, and ophthalmologic image capturing devices 11A and 11B are used. The mathematical model construction device 1 constructs a mathematical model by training the mathematical model by a machine learning algorithm. The constructed mathematical model outputs a target image having a higher image resolution than the base image based on the input base image. The ophthalmologic image processing device 21 acquires a target image from a base image using a mathematical model. The ophthalmologic image capturing apparatuses 11A and 11B capture ophthalmic images that are images of the tissue of the eye to be examined.

  As an example, a personal computer (hereinafter, referred to as “PC”) is used for the mathematical model construction device 1 of the present embodiment. Although details will be described later, the mathematical model construction device 1 constructs a mathematical model by training the mathematical model using the ophthalmic image acquired from the ophthalmologic image capturing device 11A. However, devices that can function as the mathematical model construction device 1 are not limited to PCs. For example, the ophthalmologic image photographing device 11A may function as the mathematical model construction device 1. Further, the control units of a plurality of devices (for example, the CPU of the PC and the CPU 13A of the ophthalmologic imaging apparatus 11A) may cooperate to construct a mathematical model.

  Further, a PC is used for the ophthalmologic image processing apparatus 21 of the present embodiment. However, a device that can function as the ophthalmologic image processing apparatus 21 is not limited to a PC. For example, the ophthalmologic image capturing apparatus 11B or a server may function as the ophthalmologic image processing apparatus 21. When the ophthalmologic image capturing apparatus (OCT apparatus in the present embodiment) 11B functions as the ophthalmologic image processing apparatus 21, the ophthalmologic image capturing apparatus 11B captures an ophthalmic image while generating a higher-quality ophthalmologic image than the captured ophthalmic image. Can be obtained properly. A portable terminal such as a tablet terminal or a smartphone may function as the ophthalmologic image processing device 21. The control units of a plurality of devices (for example, the CPU of the PC and the CPU 13B of the ophthalmologic imaging apparatus 11B) may perform various processes in cooperation with each other.

  In this embodiment, a case where a CPU is used as an example of a controller that performs various processes will be described. However, it goes without saying that a controller other than the CPU may be used for at least a part of the various devices. For example, the processing speed may be increased by adopting a GPU as a controller.

  The mathematical model construction device 1 will be described. The mathematical model construction device 1 is disposed, for example, at a manufacturer or the like that provides a user with an ophthalmologic image processing device 21 or an ophthalmologic image processing program. The mathematical model construction device 1 includes a control unit 2 that performs various control processes, and a communication I / F 5. The control unit 2 includes a CPU 3, which is a controller for controlling, and a storage device 4 capable of storing programs, data, and the like. The storage device 4 stores a mathematical model construction program for executing a mathematical model construction process (see FIG. 2) described later. The communication I / F 5 connects the mathematical model construction device 1 to another device (for example, the ophthalmologic image capturing device 11A and the ophthalmologic image processing device 21).

  The mathematical model construction device 1 is connected to the operation unit 7 and the display device 8. The operation unit 7 is operated by the user so that the user inputs various instructions to the mathematical model construction device 1. For the operation unit 7, for example, at least one of a keyboard, a mouse, and a touch panel can be used. Note that a microphone or the like for inputting various instructions may be used together with or instead of the operation unit 7. The display device 8 displays various images. As the display device 8, various devices capable of displaying an image (for example, at least one of a monitor, a display, a projector, and the like) can be used. Note that the “image” in the present disclosure includes both a still image and a moving image.

  The mathematical model construction device 1 can acquire the data of the ophthalmic image (hereinafter, may be simply referred to as “ophthalmic image”) from the ophthalmologic image capturing device 11A. The mathematical model construction device 1 may acquire the data of the ophthalmologic image from the ophthalmologic image capturing device 11A by, for example, at least one of a wired communication, a wireless communication, and a removable storage medium (for example, a USB memory).

  The ophthalmologic image processing device 21 will be described. The ophthalmologic image processing apparatus 21 is disposed, for example, in a facility (eg, a hospital or a health check-up facility) for performing a diagnosis or an examination of a subject. The ophthalmologic image processing device 21 includes a control unit 22 that performs various control processes, and a communication I / F 25. The control unit 22 includes a CPU 23, which is a controller for controlling, and a storage device 24 that can store programs, data, and the like. The storage device 24 stores an ophthalmologic image processing program for executing ophthalmologic image processing (see FIG. 5) described later. The ophthalmological image processing program includes a program for realizing the mathematical model constructed by the mathematical model construction device 1. The communication I / F 25 connects the ophthalmologic image processing device 21 to another device (for example, the ophthalmologic image capturing device 11B and the mathematical model construction device 1).

  The ophthalmologic image processing device 21 is connected to the operation unit 27 and the display device 28. Various devices can be used for the operation unit 27 and the display device 28, similarly to the operation unit 7 and the display device 8 described above.

  The ophthalmologic image processing device 21 can acquire an ophthalmologic image from the ophthalmologic image capturing device 11B. The ophthalmologic image processing device 21 may acquire the ophthalmologic image from the ophthalmologic image capturing device 11B by at least one of, for example, wired communication, wireless communication, and a removable storage medium (for example, a USB memory). Further, the ophthalmologic image processing apparatus 21 may acquire, via communication or the like, a program or the like for realizing the mathematical model constructed by the mathematical model construction apparatus 1.

  The ophthalmologic imaging devices 11A and 11B will be described. As an example, in the present embodiment, a case will be described in which an ophthalmic image capturing apparatus 11A that provides an ophthalmic image to the mathematical model construction apparatus 1 and an ophthalmic image capturing apparatus 11B that provides an ophthalmic image to the ophthalmic image processing apparatus 21 are used. . However, the number of ophthalmologic imaging devices used is not limited to two. For example, the mathematical model construction device 1 and the ophthalmologic image processing device 21 may acquire ophthalmologic images from a plurality of ophthalmologic image capturing devices. The mathematical model construction device 1 and the ophthalmologic image processing device 21 may acquire an ophthalmologic image from one common ophthalmologic image capturing device. Note that the two ophthalmologic image capturing apparatuses 11A and 11B exemplified in the present embodiment have the same configuration. Accordingly, hereinafter, the two ophthalmologic image capturing apparatuses 11A and 11B will be described together.

  In the present embodiment, an OCT device capable of capturing an OCT angiographic image of the fundus is used as the ophthalmologic image capturing device 11 (11A, 11B). The fundus blood vessels of the eye to be examined appear in the OCT angio image. Therefore, the ophthalmologic image processing apparatus 21 can acquire a high-resolution target image in which a fundus blood vessel is displayed. Note that the OCT angio image of the present embodiment is a two-dimensional front image when the fundus is viewed from the front (that is, the line of sight of the eye to be inspected).

  The ophthalmologic image capturing apparatus 11 (11A, 11B) includes a control unit 12 (12A, 12B) for performing various control processes, and an ophthalmologic image capturing unit 16 (16A, 16B). The control unit 12 includes a CPU 13 (13A, 13B), which is a controller for controlling, and a storage device 14 (14A, 14B) capable of storing programs, data, and the like.

  The ophthalmologic image capturing section 16 has various components necessary for capturing an ophthalmologic image of the eye to be examined. The ophthalmic image photographing unit 16 of the present embodiment includes an OCT light source, a branching optical element that branches the OCT light emitted from the OCT light source into measurement light and reference light, a scanning unit for scanning the measurement light, and a measurement light. An optical system for irradiating the optometry, a light receiving element for receiving a combined light of the light reflected by the tissue of the subject's eye and the reference light, and the like are included.

  The ophthalmologic image capturing apparatus 11 captures (generates) a motion contrast image, which is a type of OCT angio image. More specifically, the CPU 13 scans the same position in the tissue with the OCT light a plurality of times to acquire at least two frames of OCT signals (interference signals) having different measurement times. The CPU 13 obtains the data of the motion contrast image by performing an arithmetic process on the obtained OCT signal. Methods of processing the OCT signal to obtain motion contrast data include, for example, a method of calculating a phase difference of a complex OCT signal, a method of calculating a vector difference of a complex OCT signal, a decorrelation, a ratio, a difference, a variance, and the like. At least one of a method of calculating a change in amplitude or intensity and a method of multiplying a plurality of calculation results may be employed.

  Further, the ophthalmologic image photographing apparatus 11 can photograph (generate) a panoramic image of a tissue (the fundus in this embodiment). A panoramic image is generated by aligning a plurality of images (hereinafter, referred to as unit images) captured in each of a plurality of imaging regions that do not completely overlap each other. The shooting area of each unit image is smaller than the shooting area of the entire panoramic image. Therefore, the ophthalmologic image photographing apparatus 11 can photograph each unit image at a high image resolution. Therefore, the panoramic image is a high-resolution image in which a wide range of the tissue is captured. An example of a method for acquiring a panoramic image of an OCT angio image is described in, for example, JP-A-2017-47113.

  However, it is also possible to use an ophthalmologic image photographing device other than the OCT device. For example, a fundus camera, a laser scanning optometry device (SLO), a corneal endothelial cell imaging device (CEM), or the like may be used as the ophthalmologic imaging device. Further, the OCT apparatus may provide a two-dimensional tomographic image or a three-dimensional tomographic image to the mathematical model construction apparatus 1 or the ophthalmologic image processing apparatus 21.

(Mathematical model construction processing)
The mathematical model construction processing executed by the mathematical model construction apparatus 1 will be described with reference to FIGS. The mathematical model construction process is executed by the CPU 3 according to the mathematical model construction program stored in the storage device 4.

  In the mathematical model construction process, a mathematical model for outputting the target image from the base image is constructed by training the mathematical model with the training data set. As will be described later in detail, the training data set includes input training data and output training data. In the present embodiment, low-resolution training image data is used as input training data. Further, as output training data, data of a high-resolution training image having a higher image resolution than a low-resolution training image is used. The low-resolution training image and the high-resolution training image target the same tissue site. Further, the low-resolution training image and the high-resolution training image are images based on the images captured by the same ophthalmologic image capturing apparatus 11.

  As shown in FIG. 2, the CPU 3 acquires a high-resolution panoramic image (in this embodiment, a panoramic image of an OCT angio image) as a high-resolution training image 30 (S1). In the present embodiment, the CPU 3 acquires the panoramic image generated by the ophthalmologic image photographing apparatus 11A via communication or the like. However, the CPU 3 may acquire the panorama image by acquiring data (for example, motion contrast data) on which the panorama image is generated, and generating the panorama image based on the acquired data.

  FIG. 3 shows an example of the high-resolution training image 31 used in the present embodiment. As shown in FIG. 3, the high-resolution training image 31 of the present embodiment is a panoramic image of an OCT angio image, and is generated by aligning four unit images 311, 312, 313, and 314. As an example, in the present embodiment, each of the four unit images 311, 312, 313, and 314 is an image obtained by capturing a 3 mm-square area on the tissue. The resolution of each of the four unit images 311, 312, 313, and 314 is 256 pt × 256 pt. The imaging regions of the four unit images 311, 312, 313, and 314 are in contact with each other in a lattice pattern without any gap. Therefore, in the high-resolution training image 31 that is a panoramic image, the imaging area on the tissue is a square area with a side of 6 mm, and the resolution is 512 pt × 512 pt.

  Next, the CPU 3 generates a low-resolution training image 32 by reducing the image resolution of the high-resolution training image 31 that is a panoramic image (S2). As an example, the CPU 3 of the present embodiment generates the low-resolution training image 32 by thinning out a plurality of pixels constituting the high-resolution training image 31 regularly (in this embodiment, one by one). As a result, in the low-resolution training image 32, the imaging region on the tissue is a square region of 6 mm on a side, like the imaging region of the high-resolution training image 31. On the other hand, the resolution of the low-resolution training image 32 is reduced to 256 pt × 256 pt. As a result, the image resolution of the low-resolution training image 32 (the number of pixels of the image per unit area on the tissue) is lower than the image resolution of the high-resolution training image 31.

  As described above, the CPU 3 obtains the training data set 30 including the high-resolution training image 31 and the low-resolution training image 32 by reducing the image resolution of the high-resolution training image 31 and generating the low-resolution training image 32. I do. In this case, the imaging region, the imaging position, and the state of the tissue at the time of imaging match between the high-resolution training image 31 and the low-resolution training image 32. Therefore, the training of the mathematical model described later is performed more appropriately.

  A plurality of training data sets are used for training the mathematical model. In the present embodiment, training data including a high-resolution training image 31 that is a panoramic image and a low-resolution training image 32 generated by reducing the image resolution of the high-resolution training image 31 among a plurality of training data sets. The ratio of the set 30 is 50% or more (preferably 70% or more). As a result, the quality of the target image output by the mathematical model is further improved.

  Note that a method of acquiring a training data set that does not include the high-resolution training image 31 and the low-resolution training image 32 among the plurality of training data sets can be appropriately selected. For example, the CPU 3 may acquire an ophthalmologic image that is not a panoramic image as a high-resolution training image. The CPU 3 may generate a low-resolution training image by reducing the image resolution of a high-resolution training image that is not a panoramic image. Further, a plurality of ophthalmologic images generated by photographing the same tissue a plurality of times at different image resolutions may be used for the training data set. Note that the shooting region of the high-resolution training image and the shooting region of the low-resolution training image do not need to completely match. For example, the shooting region of the high-resolution training image and a part of the shooting region of the low-resolution training image may overlap.

  Next, the CPU 3 randomly adds noise to the low-resolution training image 32 (S3). Specifically, the CPU 3 randomly selects a low-resolution training image 32 to which noise is to be added, from the plurality of low-resolution training images 32. In addition, when performing the noise adding process, the CPU 3 randomly determines the amount of the noise to be added and the area to which the noise is to be added.

  Next, the CPU 3 randomly blurs the low-resolution training image 32 (S4). Specifically, the CPU 3 randomly selects a low-resolution training image 32 to which blur is to be applied, from the plurality of low-resolution training images 32. In addition, when performing the blurring process, the CPU 3 randomly determines the degree of blurring, the area to be blurred, and the like. Note that the noise imparting process (S3) and the blur imparting process (S4) are examples of a degradation process for degrading at least a part of the low-resolution training image 32. It is also possible to change the specific method of the deterioration process.

  Next, the CPU 3 executes the training of the mathematical model using the training data set by the machine learning algorithm (S5). As a machine learning algorithm, for example, a neural network, a random forest, boosting, a support vector machine (SVM), and the like are generally known.

  A neural network is a technique for imitating the behavior of a biological neural network. Examples of the neural network include a feedforward (forward propagation) neural network, an RBF network (radial basis function), a spiking neural network, a convolutional neural network, a recursive neural network (a recurrent neural network, a feedback neural network, etc.), a probability Neural network (Boltzmann machine, Bayesian network, etc.).

  The random forest is a method of generating a large number of decision trees by performing learning based on training data that is randomly sampled. When a random forest is used, a branch of a plurality of decision trees previously learned as a discriminator is traced, and an average (or majority decision) of the results obtained from each decision tree is calculated.

  Boosting is a method of generating a strong discriminator by combining a plurality of weak discriminators. A strong classifier is constructed by successively learning simple and weak classifiers.

  The SVM is a method of configuring two classes of pattern classifiers using linear input elements. The SVM learns the parameters of the linear input element based on, for example, a criterion (hyperplane separation theorem) for obtaining a margin-maximizing hyperplane that maximizes the distance to each data point from training data.

  The mathematical model refers to, for example, a data structure for predicting a relationship between input data and output data. The mathematical model is constructed by being trained using a training data set. The training data set is a set of input training data and output training data. The mathematical model is trained such that, when certain input training data is input, corresponding output training data is output. For example, the training updates the correlation data (eg, weights) for each input and output.

  As an example, in the present embodiment, a hostile generation network (GAN) using two competing neural networks is adopted as a machine learning algorithm. However, another machine learning algorithm may be used. For example, a multilayer neural network (for example, a convolutional neural network (CNN) or the like) may be used as the machine learning algorithm.

  Referring to FIG. 4, a method of training a mathematical model by a machine learning algorithm (GAN) employed in the present embodiment will be described. As described above, the GAN network includes two sub-networks (generator and discriminator) competing with each other. In the present embodiment, first, an image 32E obtained by enlarging the low-resolution training image 32 that is input training data is generated. As a result, the size of the enlarged low-resolution training image 32E matches the size of the high-resolution training image 31. In addition, the low-resolution training image 32 may be used as training input data without being enlarged, and the process of enlarging may be performed by a generator. Next, the generator outputs a fake image 32F that matches the high-resolution training image 31 as much as possible based on the low-resolution training image 32E. The discriminator attempts to determine whether the high-resolution training image 31 which is a real image and the fake image 32F are correct. The judgment result is reflected on both the generator and the discriminator. By repeating this training, the generator tries to generate a fake image 32F closer to the higher-resolution training image 31, and the discriminator tries to improve the accuracy of identification. A mathematical model (generator model) constructed as a result of the training is configured such that, by inputting an ophthalmic image having a low image resolution (for example, a base image described later), an ophthalmic image having a higher image resolution than the input image (for example, (A target image described later) can be generated.

  As described above, in the present embodiment, the high-resolution image and the low-resolution image included in the training data set are images captured by the same ophthalmologic image capturing device 11A (specifically, the same ophthalmologic image capturing device). 11A). Therefore, the colors of the high-resolution image and the low-resolution image, the amount of noise included in the images, artifacts, optical resolution, the angle of view, the state of the tissue at the time of imaging, and the like are easily approximated. Thus, the mathematical model is properly trained.

  Further, in the present embodiment, the low-resolution training image used for at least one of the plurality of training data sets is subjected to the deterioration process on at least a part of the image region. Therefore, even when the image quality of at least a part of the ophthalmic image (for example, a base image described later) input to the constructed mathematical model is not good, an image in which the influence of the poor image quality is reduced is output.

  Returning to the description of FIG. Until the construction of the mathematical model is completed (S6: NO), the processing of S1 to S5 is repeated. When the construction of the mathematical model is completed (S6: YES), the mathematical model construction processing ends. The program and data for realizing the constructed mathematical model are incorporated in the ophthalmologic image processing apparatus 21.

(Ophthalmological image processing)
The ophthalmologic image processing performed by the ophthalmologic image processing device 21 will be described with reference to FIG. The ophthalmologic image processing is executed by the CPU 23 in accordance with the ophthalmologic image processing program stored in the storage device 24.

  First, the CPU 23 acquires a base image which is an ophthalmologic image captured by the ophthalmologic image capturing apparatus 11B (S11). In the present embodiment, the ophthalmologic image photographing apparatus 11B continuously photographs the base image (that is, a moving image). The CPU 23 sequentially acquires the base images that have been continuously photographed.

  The base image in the present embodiment is an OCT angio image of the fundus of the subject's eye, like the high-resolution training image 31 and the low-resolution training image 32 described above. Further, the base image in the present embodiment is an image whose image resolution is lower than that of the high-resolution training image 31 described above. As an example, the ophthalmologic imaging apparatus 11 </ b> B of the present embodiment converts an imaging area (for example, an imaging area of 6 mm × 6 mm) having the same size as the high-resolution training image 31 with a resolution lower than the resolution of the high-resolution training image 31 ( For example, data of a base image is generated by shooting at 256 pt × 256 pt). Therefore, the photographing time for photographing the base image is shorter than the photographing time for photographing the high-resolution training image 31. Therefore, when capturing the base image, the effects of the movement of the subject's eye (for example, fine fixation and blinking), the burden of the tracking operation for following the movement of the subject's eye in the imaging area, and the like are caused by the high-resolution training image 31. Is lower than when photographing. When the target image is acquired, the ophthalmologic image photographing apparatus 11B automatically sets the photographing region of the base image to a photographing region equivalent to the high-resolution training image 31. As a result, the base image and the target image are obtained more smoothly.

  Next, the CPU 23 obtains a target image having a higher image resolution than the base image by inputting the base image into the mathematical model described above (S12). In the present embodiment, the CPU 23 successively acquires the target images by sequentially inputting the successively acquired base images to the mathematical model.

  Next, the CPU 23 determines whether or not an instruction to display the target image on the display device 28 has been input (S14). In the present embodiment, the user operates the operation unit 27 to issue a target image, a base image, and an instruction on which of the target image and the base image to display on the display device 28. Can be entered. If the instruction to display the target image has not been input (S14: NO), the process proceeds to S17. If the instruction to display the target image has been input (S14: YES), the CPU 23 displays the target image on the display device 28 (S15). More specifically, in S15 of the present embodiment, a plurality of target images continuously acquired in S12 are sequentially displayed on the display device 28, so that a moving image of the organization is displayed. That is, a high-resolution moving image of the tissue being captured is displayed on the display device 28 in real time based on the moving image of the base image (original image) captured by the ophthalmologic image capturing device 11B at that time. The CPU 23 may generate moving image data of the target image based on the plurality of target images continuously acquired in S12 and store the data in the storage device.

  Note that the ophthalmologic image processing apparatus 21 can input the target image, the base image, and an instruction on which of the target image and the base image to display on the display device 28 by various methods. For example, the CPU 23 may cause the display device 28 to display an icon for inputting a display instruction for the target image, an icon for inputting a display instruction for the base image, and an icon for inputting a display instruction for both images. The user may input a display instruction to the ophthalmologic image processing apparatus 21 by selecting a desired icon.

  Next, the CPU 23 determines whether or not an instruction to display the base image on the display device 28 has been input (S17). If it has not been input (S17: NO), the process proceeds directly to S20. If the instruction to display the base image has been input (S17: YES), the CPU 23 displays the base image on the display device 28 (S18). Specifically, in S18 of the present embodiment, a plurality of base images continuously acquired in S11 are sequentially displayed on the display device 28, so that a moving image of the tissue is displayed. As described above, the CPU 23 can switch between the target image and the base image to be displayed on the display device 28 according to the instruction input by the user. Therefore, the user can easily check both the target image and the base image.

  Next, the CPU 23 determines whether or not an instruction to simultaneously display the target image and the base image on the display device 28 has been input (S20). If not (S20: NO), the process proceeds to S23. When the simultaneous display instruction has been input (S20: YES), the CPU 23 causes the display device 28 to simultaneously display the target image and the base image. Specifically, in S20 of the present embodiment, both the moving image of the target image and the moving image of the base image are displayed on the display device 28.

  In S15, S18, and S21, a still image instead of a moving image may be displayed. The CPU 23 may switch between displaying a moving image and displaying a still image according to an instruction input by the user. Alternatively, the CPU 23 may cause the storage device 24 to store at least one data of the moving image and the still image without displaying the moving image and the still image on the display device 28.

  Next, the CPU 23 determines whether an instruction to end the display has been input (S23). If not (S23: NO), the process returns to S11, and the processes of S11 to S23 are repeated. When an instruction to end the display is input (S23: YES), the CPU 23 executes various analysis processes on the target image (in the present embodiment, at least one of a plurality of target images acquired continuously) ( S24). As described above, the image resolution of the target image is higher than the image resolution of the base image. Therefore, by performing the analysis process on the target image, the analysis accuracy is improved as compared with the case where the analysis process is performed on the base image. Note that the specific contents of the analysis processing can be appropriately selected. As an example, in S24 of the present embodiment, the blood vessel density of the fundus oculi (for example, the blood vessel density around the fovea non-perfusion region) is analyzed.

  Examples of the timing at which the target image is displayed include a timing at which the target image is displayed in real time during imaging by the ophthalmologic imaging apparatus 11B, a timing at which the user confirms whether or not the captured image is good, and an analysis result after imaging. Alternatively, there are a plurality of timings, such as timings to be displayed together with the report. The CPU 23 separately issues a target image, a base image, and an instruction on which of the target image and the base image to display on the display device 28 at each of a plurality of timings at which the target image can be displayed. Can be entered. That is, the user sets in advance whether the target image, the base image, or both the target image and the base image are to be displayed on the display device 28 in accordance with the timing at which the target image can be displayed. I can put it. As a result, user convenience is further improved.

(Synthetic image generation processing)
With reference to FIG. 6, a composite image generation process executed by the CPU 23 of the ophthalmologic image processing apparatus 21 will be described. As shown in FIG. 6, the composite image 40 includes a target image 42 and an attention area image 44. The target image is a high-resolution ophthalmic image acquired by inputting the base image to the mathematical model, as described in S12 of FIG. The attention area image 44 is an image in which a part of the attention area of the imaging area of the base image is captured by the ophthalmologic image capturing apparatus 11B. The attention area image 44 is captured at an image resolution higher than the image resolution of the base image. The attention area image 44 is the image itself (that is, the original image) captured by the ophthalmologic image capturing apparatus 11B. A method of setting the position, size, shape, and the like of the attention area with respect to the shooting area of the base image can be appropriately selected. For example, the ophthalmologic image photographing apparatus 11B may set the attention area in the photographing area of the base image according to an instruction input by the user. Further, the ophthalmologic imaging apparatus 11B performs image processing or the like on the base image to detect a characteristic portion (for example, at least one of a macula, an optic disc, a fundus blood vessel, and the like) in the base image, and detects the characteristic portion. The attention area may be automatically set on the basis of.

  The CPU 23 of the ophthalmologic image processing apparatus 21 acquires the base image and the attention area image 44. The CPU 23 acquires the target image 42 having a higher image resolution than the base image by inputting the base image into the mathematical model. The CPU 23 generates a combined image 40 by combining the attention area image 44 with the attention area in the target image 42. According to the composite image 40, the user can confirm the attention area with the original image, and also can confirm the entire image at a high resolution.

  The technology disclosed in the above embodiment is merely an example. Therefore, it is also possible to change the technique exemplified in the above embodiment. First, it is also possible to execute only a part of the plurality of techniques exemplified in the above embodiment. For example, instead of employing a technique using a panoramic image as a high-resolution training image, a technique of executing at least one of a noise adding process and a blur applying process on a low-resolution training image may be adopted.

  Note that the process of acquiring the base image in S11 of FIG. 5 is an example of a “base image acquisition step”. The process of acquiring the target image in S12 of FIG. 5 is an example of a “target image acquisition step”. The process of simultaneously displaying the target image and the base image in S21 of FIG. 5 is an example of the “simultaneous display process” and the “simultaneous display step”. The process of switching and displaying the base image and the target image in S14 to S18 in FIG. 5 is an example of “switching display process” and “switching display step”. The process of acquiring the training data set in S1 and S2 in FIG. 2 is an example of the “image set acquiring step”. The process of constructing a mathematical model in S5 of FIG. 2 is an example of a “training step”. The process of acquiring a panoramic image in S1 of FIG. 2 is an example of a “panoramic image acquiring step”. The process of generating the low-resolution training image 32 in S2 of FIG. 2 is an example of a “low-resolution training image generating step”. The process of giving noise to the low-resolution training image 32 in S3 of FIG. 2 and the process of giving blur to the low-resolution training image 32 in S4 of FIG. 2 are examples of “deterioration process” and “deterioration step”. .

1 Mathematical model construction device 3 CPU
4 Storage Device 11 Ophthalmologic Image Capturing Device 21 Ophthalmologic Image Processing Device 23 CPU
24 storage device 27 operation unit 28 display device 30 training data set 31 high-resolution training image 32 low-resolution training image 40 composite image 42 target image 44 area-of-interest image

Claims (11)

An ophthalmologic image processing apparatus that processes an ophthalmic image that is an image of a tissue of a subject's eye,
The control unit of the ophthalmologic image processing apparatus,
Obtain a base image that is an ophthalmic image taken by the ophthalmic image taking apparatus,
By inputting the base image to a mathematical model trained by a machine learning algorithm, to obtain a target image whose image resolution is higher than the image resolution of the base image,
The mathematical model is for inputting a low-resolution training image that is at least one ophthalmologic image among a set of a plurality of ophthalmologic images captured by the same ophthalmologic image capturing apparatus, with the same tissue portion as a capturing target. An ophthalmologic image processing apparatus, wherein training data is trained as a high-resolution training image, which is an ophthalmologic image having a higher image resolution than the low-resolution training image, as training data for output. The ophthalmologic image processing apparatus according to claim 1,
The ophthalmologic image processing apparatus, wherein the base image, the low-resolution training image, and the high-resolution training image are OCT angio images of the fundus of the subject's eye captured by an OCT apparatus. The ophthalmologic image processing apparatus according to claim 1 or 2,
At least one of the plurality of high-resolution training images is a panoramic image generated by aligning a plurality of ophthalmic images captured in each of a plurality of imaging regions that do not completely overlap, and Ophthalmological image processing device. The ophthalmologic image processing apparatus according to claim 3,
An ophthalmologic image processing apparatus, wherein at least one of the low-resolution training images is an image generated by reducing the image resolution of the panoramic image. The ophthalmologic image processing apparatus according to any one of claims 1 to 4,
An ophthalmologic image processing apparatus, characterized in that at least one of the low-resolution training images has been subjected to a deterioration process for deteriorating image quality on at least a part of an image region. The ophthalmologic image processing apparatus according to any one of claims 1 to 5,
The control unit performs a simultaneous display process of simultaneously displaying the base image and the target image on a display device, and a switching display process of switching between the base image and the target image according to an input instruction and displaying the target image on a display device. An ophthalmologic image processing apparatus that performs at least one of the following. The ophthalmologic image processing apparatus according to any one of claims 1 to 5,
The control unit includes:
Acquiring the base images sequentially taken by the ophthalmologic imaging apparatus sequentially,
By sequentially inputting the acquired base image to the mathematical model, the target image is continuously obtained,
An ophthalmologic image processing apparatus that generates moving image data of the tissue based on the acquired target images. The ophthalmologic image processing apparatus according to any one of claims 1 to 7,
The control unit includes:
The ophthalmologic image capturing apparatus is an ophthalmologic image in which a part of the attention area of the imaging area of the base image is captured, and obtains an attention area image in which the image resolution is higher than the image resolution of the base image.
An ophthalmologic image processing apparatus, wherein the attention area image is combined with the attention area in the target image. An OCT apparatus that captures an ophthalmologic image of the tissue by processing the OCT signal based on the reference light and the reflected light of the measurement light applied to the tissue of the subject's eye,
The control unit of the OCT apparatus includes:
Acquire the captured ophthalmic image as a base image,
By inputting the base image to a mathematical model trained by a machine learning algorithm, to obtain a target image whose image resolution is higher than the image resolution of the base image,
The mathematical model is for inputting a low-resolution training image that is at least one ophthalmologic image among a set of a plurality of ophthalmologic images captured by the same ophthalmologic image capturing apparatus, with the same tissue portion as a capturing target. An OCT apparatus wherein training data is trained, and a high-resolution training image, which is an ophthalmic image having a higher image resolution than the low-resolution training image, is trained as output training data. An ophthalmologic image processing program executed by an ophthalmologic image processing apparatus that processes an ophthalmologic image that is an image of a tissue of an eye to be examined,
The ophthalmologic image processing program is executed by the control unit of the ophthalmologic image processing apparatus,
A base image obtaining step of obtaining a base image which is an ophthalmologic image taken by the ophthalmologic imaging apparatus,
By inputting the base image into a mathematical model trained by a machine learning algorithm, a target image obtaining step of obtaining a target image whose image resolution is higher than the image resolution of the base image,
Is executed by the ophthalmic image processing apparatus,
The mathematical model is for inputting a low-resolution training image that is at least one ophthalmologic image among a set of a plurality of ophthalmologic images captured by the same ophthalmologic image capturing apparatus, with a portion of the same tissue being a capturing target. An ophthalmologic image processing program, wherein training data is trained as output training data, which is a high-resolution training image that is an ophthalmic image having a higher image resolution than the low-resolution training image. A mathematical model construction method that is constructed by being trained by a machine learning algorithm and executed by a mathematical model construction device that constructs a mathematical model that outputs a target image when a base image is input,
An image set obtaining step of obtaining a set of a plurality of ophthalmic images captured by the same ophthalmologic image capturing apparatus, with the same portion of the tissue of the subject's eye being captured, and
In the set, a low-resolution training image that is at least one ophthalmic image is used as input training data, and a high-resolution training image that is an ophthalmic image having a higher image resolution than the low-resolution training image is used as output training data. Training a model to train the mathematical model;
Mathematical model construction method including.