JP7183590B2 - Ophthalmic image processing device, OCT device, and ophthalmic image processing program - Google Patents

Description

The present disclosure relates to an ophthalmic image processing apparatus that processes an ophthalmic image of an eye to be examined, an OCT apparatus, and an ophthalmic image processing program executed in the ophthalmic image processing apparatus.

In recent years, techniques have been proposed for obtaining various medical information using mathematical models trained by machine learning algorithms. For example, in the ophthalmologic apparatus described in Patent Document 1, eye shape parameters are input to a mathematical model trained by a machine learning algorithm to obtain IOL-related information (for example, predicted predicate anterior chamber depth) of the eye to be examined. be. The IOL power is calculated based on the acquired IOL-related information.

It is also conceivable to obtain high resolution images from low resolution images using mathematical models trained by machine learning algorithms. In this case, a low-resolution image is input to the mathematical model and a high-resolution image is output.

JP 2018-51223 A

One aspect of the technology in the present disclosure will be described. When acquiring high-resolution images from low-resolution images using a mathematical model trained by a machine learning algorithm, using low-resolution images and high-resolution images of the same part, the mathematical model must be pre-trained. In other words, the operator who constructs the mathematical model must prepare a large number of sets of low-resolution images and high-resolution images of the same part to train the mathematical model. It takes a lot of effort to prepare a large set of images.

Another aspect will be described. In order to properly train a mathematical model, it is necessary to have an appropriate correspondence between low-resolution images and high-resolution images used for training. It is difficult to obtain good high-resolution images using a mathematical model when the mathematical model is trained using a set of images with poor correspondence.

An object of the present disclosure is to solve at least one of the above aspects. In other words, a typical object of the present disclosure is to provide an ophthalmic image processing apparatus, an OCT apparatus, and an ophthalmic image processing program for appropriately acquiring high-resolution images from low-resolution images.

An ophthalmic image processing apparatus provided by a typical embodiment of the present disclosure is an ophthalmic image processing apparatus that processes an ophthalmic image that is an image of a tissue of an eye to be examined, wherein a control unit of the ophthalmic image processing apparatus includes: Obtaining a base image, which is an ophthalmologic image captured by an imaging device, and inputting the base image to a mathematical model trained by a machine learning algorithm to obtain a target image having a higher image resolution than the base image. a low-resolution training image, wherein the mathematical model is at least one ophthalmologic image from a set of a plurality of ophthalmologic images captured by the same ophthalmologic image capturing device, wherein the same tissue region is captured; is training data for input, high-resolution training images, which are ophthalmologic images having a higher image resolution than the low-resolution training images , are trained as training data for output, and at least one of the plurality of high-resolution training images is 1 is a panoramic image generated by aligning multiple ophthalmic images captured in each of multiple non-overlapping imaging regions.

An OCT apparatus provided by a typical embodiment of the present disclosure processes an OCT signal of reference light and reflected light of measurement light applied to a tissue of an eye to be examined, thereby capturing an ophthalmologic image of the tissue. An OCT apparatus, wherein the control unit of the OCT apparatus obtains a photographed ophthalmologic image as a base image, and inputs the base image to a mathematical model trained by a machine learning algorithm so that the image resolution is the base image. and the mathematical model captures at least one of a plurality of sets of ophthalmologic images that target the same tissue region and are captured by the same ophthalmologic imaging device. Low-resolution training images, which are two ophthalmic images, are used as input training data, and high-resolution training images, which are ophthalmologic images having a higher image resolution than the low-resolution training images , are used as output training data. At least some of the resolution training images are panoramic images generated by registering multiple ophthalmic images taken in each of multiple non-overlapping fields of view.

An ophthalmic image processing program provided by a typical embodiment of the present disclosure is an ophthalmic image processing program executed by an ophthalmic image processing apparatus that processes an ophthalmic image that is an image of tissue of an eye to be examined, the ophthalmic image processing A program is executed by the control unit of the ophthalmologic image processing apparatus to acquire a base image, which is an ophthalmologic image captured by the ophthalmologic imaging apparatus, and a mathematical model trained by a machine learning algorithm. a target image obtaining step of obtaining a target image having a higher image resolution than the image resolution of the base image by inputting the base image, wherein of a set of a plurality of ophthalmologic images captured by the same ophthalmologic image capturing device, with a low-resolution training image that is at least one ophthalmologic image as input training data, and the low-resolution training High-resolution training images, which are ophthalmologic images having a higher image resolution than the images, are trained as output training data, and at least one of the plurality of high-resolution training images is each of a plurality of imaging regions that do not completely overlap. 1 is a panoramic image generated by aligning multiple ophthalmic images taken at .

According to the ophthalmologic image processing apparatus, OCT apparatus, and ophthalmologic image processing program according to the present disclosure, high-resolution images are appropriately obtained from low-resolution images.

1 is a block diagram showing schematic configurations of a mathematical model construction device 1, an ophthalmologic image processing device 21, and ophthalmologic imaging devices 11A and 11B; FIG. 4 is a flowchart of mathematical model building processing executed by the mathematical model building device 1; FIG. 3 is an explanatory diagram for explaining a training data set 30 including high-resolution training images 31 and low-resolution training images 32; FIG. 4 is an explanatory diagram for explaining the contents of training of a mathematical model in this embodiment; 4 is a flowchart of ophthalmologic image processing executed by the ophthalmologic image processing apparatus 21. FIG. 4 is a diagram showing an example of a synthesized image 40 of a target image 42 and an attention area image 44; FIG.

<Overview>
A control unit of an ophthalmologic image processing apparatus exemplified in the present disclosure acquires a base image, which is an ophthalmologic image captured by an ophthalmologic imaging apparatus. The control unit acquires a target image having an image resolution higher than that of the original image by inputting the original image into a mathematical model trained by a machine learning algorithm. A mathematical model is trained with a training data set, which is a set of input training data and output training data. The training data set includes a plurality of ophthalmologic images captured by the same ophthalmologic image capturing device and of the same tissue site. The input training data includes at least one ophthalmologic low-resolution training image of the training data set. High-resolution training images, which are ophthalmologic images with higher image resolution than low-resolution training images, are used as the output training data.

That is, a mathematical model builder that builds a mathematical model performs an image set acquisition step and a training step. In the image set acquisition step, a set of a plurality of ophthalmologic images captured by the same ophthalmologic image capturing device with the same site of the tissue of the subject's eye as the imaging target is obtained. In the training step, at least one low-resolution training image, which is an ophthalmologic image, in the obtained set is used as input training data, and a high-resolution ophthalmic image, which is an ophthalmologic image having a higher image resolution than the low-resolution training image, is used as training data for input. A mathematical model is trained using the resolution training images as training data for output.

In this case, both the low-resolution training images and the high-resolution training images contained in the training data set are captured by the same ophthalmic imaging device. Therefore, a training data set is obtained more efficiently than if the low-resolution training images and the high-resolution training images were captured by different ophthalmic imaging devices. In addition, the correspondence relationship between the low-resolution training images and the high-resolution training images included in the training data set is more likely to be appropriate than when the low-resolution training images and the high-resolution training images are captured by different ophthalmologic imaging devices. (For example, the color tone of the image, the amount of noise contained in the image, the artifact, the optical resolution, the imaging angle of view, the state of the tissue at the time of imaging, etc. are likely to be similar between the low-resolution training image and the high-resolution training image. ), the trained mathematical model is used to obtain a better target image.

It should be noted that “image resolution” in the present disclosure means the fineness of depicting an object to be photographed (that is, the fineness of an image). In other words, the “image resolution” in the present disclosure is the number of pixels of the image corresponding to the unit area on the tissue to be imaged.

Also, the low-resolution training images and high-resolution training images included in the training data set are not limited to ophthalmologic images themselves (that is, original images) captured by the same ophthalmologic image capturing device. For example, at least one of the low resolution training images and the high resolution training images may be images generated based on original images captured by an ophthalmic imaging device. In other words, if at least one of the low-resolution training images and the high-resolution training images has been processed, the original images on which the low-resolution training images and the high-resolution training images are based are captured by the same ophthalmologic imaging device. It is good if it is.

The base images, the low-resolution training images, and the high-resolution training images may be OCT angio images of the fundus of the subject's eye taken by an OCT device. The OCT angio image shows the fundus blood vessels of the eye to be examined. Therefore, by using the OCT angio image, a high-resolution target image showing the fundus blood vessels can be appropriately acquired based on the base image.

Note that the OCT angio image may be a two-dimensional front image of the fundus viewed from the front (that is, the line-of-sight direction of the subject's eye). An OCT angio image may be, for example, a motion contrast image obtained by processing at least two OCT signals obtained at different times with respect to the same location.

However, it is possible to use images other than OCT angio images of the fundus as ophthalmologic images used as training data sets and as base images input to the mathematical model. For example, at least one of a tomographic image (two-dimensional tomographic image or three-dimensional tomographic image) captured by an OCT device, an image captured by a fundus camera, an image captured by a laser scanning optometric device (SLO), etc. It may be used as a training dataset and a base image. Also, the training data set and the base image may be images of tissues other than the fundus of the subject's eye (for example, the anterior segment of the subject's eye, etc.).

The high-resolution training images used in at least one of the plurality of training data sets are panoramic images generated by registering ophthalmologic images taken in each of the plurality of non-overlapping imaging regions. may In other words, the ophthalmologic imaging apparatus may capture an ophthalmologic image (hereinafter referred to as a "unit image") for each of a plurality of imaging areas that do not completely overlap. A panoramic image may be generated by aligning a plurality of captured unit images. The image data acquisition step performed by the mathematical model construction device may include a panoramic image acquisition step of acquiring the generated panoramic images as high-resolution training images.

In this case, by capturing unit images with high image resolution, high-resolution images capturing a wide range of tissues are used as high-resolution training images for the training data set. Therefore, high-resolution training images of the training dataset are better obtained.

The low-resolution training images used for at least one of the plurality of training data sets may be images generated by reducing the image resolution of panoramic images, which are high-resolution training images. That is, the image data acquisition step executed by the mathematical model construction device may include a low-resolution training image generation step of generating low-resolution training images by reducing the image resolution of the panoramic image.

In this case, in at least one of the training data sets, the low-resolution training image and the high-resolution training image match in imaging range, imaging position, and tissue state at the time of imaging. Therefore, a better target image is obtained by using a trained mathematical model.

Note that when training a mathematical model, multiple training data sets are used for training. Here, when panoramic images are used as the high-resolution training images of the training dataset, not all of the multiple high-resolution training images need to be panoramic images, and at least some of the multiple high-resolution training images are panoramic images. I wish I had. Note that when the panoramic images account for a large proportion of the plurality of high-resolution training images (preferably 50% or more, more preferably 70% or more), the high-resolution training images are acquired more easily. Similarly, when images obtained by reducing the image resolution of panoramic images (hereinafter referred to as "low-resolution panoramic images") are used for training, not all of the plurality of low-resolution training images need to be low-resolution panoramic images. Note that when the low-resolution panoramic image accounts for a large proportion of the plurality of low-resolution training images (preferably 50% or more, more preferably 70% or more), the quality of the target image is further improved.

However, it is possible to vary the specific method for generating the low-resolution training images and the high-resolution training images of the training dataset. For example, non-panoramic ophthalmic images may be used as high-resolution training images for the training dataset. Low-resolution training images of the training dataset may also be generated by reducing the image resolution of non-panoramic ophthalmic images. Also, multiple ophthalmologic images generated by imaging the same tissue multiple times at different image resolutions may be used as the low-resolution training images and the high-resolution training images of the training data set. Further, when panoramic images are used as high-resolution training images, images obtained by photographing the same imaging range as the panoramic images at low resolution may be used as low-resolution training images for the training data set.

The low-resolution training images used for at least one of the plurality of training data sets may have undergone degradation processing that degrades image quality on at least part of the image region. That is, the mathematical model builder may perform a degradation step that degrades image quality for at least part of the image region of the low-resolution training images.

In this case, the mathematical model is trained so that even if at least a portion of the original image is of poor quality, it outputs a destination image that is less affected by the poor quality of the image. Therefore, a more suitable target image is acquired.

The specific contents of the deterioration process can be selected as appropriate. For example, the control unit of the mathematical model construction device may perform at least one of a noise addition process of adding noise and a blurring process of adding blurring.

The control unit of the ophthalmologic image processing apparatus may execute at least one of the simultaneous display process and the switching display process. In the simultaneous display process, the control unit causes the display device to display the base image and the target image at the same time. In the switching display process, the control unit causes the display device to switch between the base image and the target image in accordance with an instruction input by the user. In this case, the user can easily confirm both the target image with high image resolution and the base image (original image) on which the target image is based. Therefore, the user can more appropriately perform diagnosis and the like. Note that the base image and the target image may be displayed on the same display device, or may be displayed on separate display devices.

The control unit may sequentially acquire base images that are continuously captured by the ophthalmologic imaging device. The control unit may successively acquire target images by sequentially inputting the acquired base images into the mathematical model. The control unit may generate moving image data of the tissue based on the plurality of acquired target images. In this case, the user can see the animation of the tissue displayed in high image resolution. Therefore, the user can more appropriately perform diagnosis and the like.

In particular, when generating a high-resolution panoramic image, the panoramic image is generated from a plurality of unit images. Therefore, when a moving image of tissue is displayed using the panoramic image, the frame rate is likely to drop significantly. In particular, when the shooting position changes with the passage of time, it is difficult to generate a good moving image using a panoramic image. On the other hand, the ophthalmologic image processing apparatus of the present disclosure can generate a moving image in which a wide range is captured with high resolution while suppressing a decrease in frame rate.

Note that the control unit may cause the display device to display a high-resolution moving image of the tissue imaged by the ophthalmologic image capturing device in real time in parallel with the image capturing. Further, the control unit may display the moving image of the base image and the moving image of the target image at the same time on the display device. In addition, the control unit may switch between the moving image of the base image and the moving image of the target image and cause the display device to display the moving image according to the instruction input by the user. In this case, the user can easily confirm both the moving image of the target image and the moving image of the base image.

The control unit of the ophthalmologic image processing apparatus may execute the region-of-interest image acquiring step and the synthesizing step. In the region-of-interest image acquisition step, a region-of-interest image captured by an ophthalmologic imaging device is acquired. The attention area image is an image in which the attention area of the imaging area of the base image is captured with a higher image resolution than the image resolution of the base image. In the synthesizing step, the original image of the attention area image is synthesized with the attention area in the target image. In this case, in the synthesized image, the original image with high image resolution forms the attention area, and the areas other than the attention area are formed by the target image. Therefore, the user can confirm the attention area with the original image, and can also confirm the entire image with high resolution.

<Embodiment>
(Device configuration)
One typical embodiment of the present disclosure will be described below with reference to the drawings. As shown in FIG. 1, in this embodiment, a mathematical model construction device 1, an ophthalmologic image processing device 21, and ophthalmologic imaging devices 11A and 11B are used. The mathematical model building device 1 builds a mathematical model by training the mathematical model with a machine learning algorithm. The constructed mathematical model outputs a target image having a higher image resolution than the basic image based on the input basic image. The ophthalmic image processing device 21 uses a mathematical model to obtain a target image from the original image. The ophthalmologic image capturing devices 11A and 11B capture ophthalmologic images, which are images of tissues of the eye to be examined.

As an example, a personal computer (hereinafter referred to as “PC”) is used for the mathematical model building device 1 of this embodiment. Although the details will be described later, the mathematical model building device 1 builds a mathematical model by training the mathematical model using the ophthalmologic image acquired from the ophthalmologic imaging device 11A. However, a device that can function as the mathematical model construction device 1 is not limited to a PC. For example, the ophthalmologic imaging device 11A may function as the mathematical model building device 1 . Also, the control units of a plurality of devices (for example, the CPU of the PC and the CPU 13A of the ophthalmologic imaging apparatus 11A) may work together to construct the mathematical model.

A PC is used for the ophthalmologic image processing apparatus 21 of this embodiment. However, a device that can function as the ophthalmologic image processing apparatus 21 is not limited to a PC either. For example, the ophthalmologic imaging device 11B, a server, or the like may function as the ophthalmologic image processing device 21 . When the ophthalmic imaging device (OCT device in this embodiment) 11B functions as the ophthalmic image processing device 21, the ophthalmic imaging device 11B captures an ophthalmic image and produces an ophthalmic image with higher image quality than the captured ophthalmic image. can be obtained properly. A portable terminal such as a tablet terminal or a smartphone may function as the ophthalmologic image processing apparatus 21 . The controllers of a plurality of devices (for example, the CPU of the PC and the CPU 13B of the ophthalmologic imaging apparatus 11B) may work together to perform various processes.

Also, in this embodiment, a case where a CPU is used as an example of a controller that performs various processes will be described. However, it goes without saying that a controller other than the CPU may be used for at least some of the various devices. For example, a GPU may be employed as a controller to speed up processing.

The mathematical model construction device 1 will be described. The mathematical model building device 1 is located, for example, in a manufacturer or the like that provides the ophthalmic image processing device 21 or an ophthalmic image processing program to users. The mathematical model construction device 1 includes a control unit 2 that performs various control processes, and a communication I/F 5 . The control unit 2 includes a CPU 3, which is a controller for control, and a storage device 4 capable of storing programs, data, and the like. The storage device 4 stores a mathematical model building program for executing a later-described mathematical model building process (see FIG. 2). Also, the communication I/F 5 connects the mathematical model construction device 1 with other devices (for example, the ophthalmologic imaging device 11A and the ophthalmologic image processing device 21, etc.).

The mathematical model construction device 1 is connected to an operation section 7 and a display device 8 . The operation unit 7 is operated by the user to input various instructions to the mathematical model construction device 1 . For example, at least one of a keyboard, a mouse, a touch panel, and the like can be used as the operation unit 7 . A microphone or the like for inputting various instructions may be used together with the operation unit 7 or instead of the operation unit 7 . The display device 8 displays various images. Various devices capable of displaying images (for example, at least one of a monitor, a display, a projector, etc.) can be used as the display device 8 . It should be noted that the “image” in the present disclosure includes both still images and moving images.

The mathematical model construction device 1 can acquire ophthalmologic image data (hereinafter sometimes simply referred to as “ophthalmologic image”) from the ophthalmologic imaging device 11A. The mathematical model construction device 1 may acquire ophthalmologic image data from the ophthalmologic imaging device 11A, for example, by at least one of wired communication, wireless communication, and a removable storage medium (eg, USB memory).

The ophthalmologic image processing apparatus 21 will be described. The ophthalmologic image processing apparatus 21 is installed, for example, in a facility (for example, a hospital, a health checkup facility, or the like) for diagnosing or examining a subject. The ophthalmologic image processing apparatus 21 includes a control unit 22 that performs various control processes, and a communication I/F 25 . The control unit 22 includes a CPU 23 which is a controller for control, and a storage device 24 capable of storing programs, data, and the like. The storage device 24 stores an ophthalmologic image processing program for executing ophthalmologic image processing (see FIG. 5), which will be described later. The ophthalmologic image processing program includes a program for realizing the mathematical model constructed by the mathematical model construction device 1 . The communication I/F 25 connects the ophthalmologic image processing apparatus 21 with other devices (for example, the ophthalmic imaging apparatus 11B, the mathematical model construction apparatus 1, etc.).

The ophthalmologic image processing apparatus 21 is connected to an operation section 27 and a display device 28 . Various devices can be used for the operation unit 27 and the display device 28, similarly to the operation unit 7 and the display device 8 described above.

The ophthalmic image processing device 21 can acquire an ophthalmic image from the ophthalmic image capturing device 11B. The ophthalmologic image processing apparatus 21 may acquire an ophthalmologic image from the ophthalmologic imaging apparatus 11B, for example, by at least one of wired communication, wireless communication, and a removable storage medium (eg, USB memory). Further, the ophthalmologic image processing apparatus 21 may acquire, via communication or the like, a program or the like for realizing the mathematical model constructed by the mathematical model construction apparatus 1 .

The ophthalmologic imaging devices 11A and 11B will be described. As an example, in this embodiment, an ophthalmic imaging device 11A that provides an ophthalmic image to the mathematical model building device 1 and an ophthalmic imaging device 11B that provides an ophthalmic image to the ophthalmic image processing device 21 are used. . However, the number of ophthalmic imaging devices used is not limited to two. For example, the mathematical model construction device 1 and the ophthalmologic image processing device 21 may acquire ophthalmologic images from a plurality of ophthalmologic imaging devices. Further, the mathematical model construction device 1 and the ophthalmologic image processing device 21 may acquire ophthalmologic images from one common ophthalmologic imaging device. Note that the two ophthalmologic imaging apparatuses 11A and 11B exemplified in this embodiment have the same configuration. Therefore, the two ophthalmologic imaging devices 11A and 11B will be collectively described below.

Further, in this embodiment, an OCT apparatus capable of capturing an OCT angio image of the fundus is used as the ophthalmologic imaging apparatus 11 (11A, 11B). The OCT angio image shows the fundus blood vessels of the eye to be examined. Therefore, the ophthalmologic image processing apparatus 21 can acquire a high-resolution target image in which the fundus blood vessels appear. Note that the OCT angio image of the present embodiment is a two-dimensional front image when the fundus is viewed from the front (that is, the line-of-sight direction of the subject's eye).

The ophthalmologic imaging apparatus 11 (11A, 11B) includes a control unit 12 (12A, 12B) that performs various control processes, and an ophthalmologic imaging section 16 (16A, 16B). The control unit 12 includes a CPU 13 (13A, 13B), which is a controller for control, and a storage device 14 (14A, 14B) capable of storing programs, data, and the like.

The ophthalmologic image capturing unit 16 has various configurations necessary for capturing an ophthalmologic image of the subject's eye. The ophthalmologic image capturing unit 16 of the present embodiment includes an OCT light source, a branching optical element that branches the OCT light emitted from the OCT light source into measurement light and reference light, a scanning unit for scanning the measurement light, and a scanning unit for scanning the measurement light. It includes an optical system for irradiating an eye to be examined, a light receiving element for receiving the combined light of the light reflected by the tissue of the eye to be examined and the reference light, and the like.

The ophthalmologic image capturing device 11 captures (generates) a motion contrast image, which is a type of OCT angio image. Specifically, the CPU 13 acquires at least two frames of OCT signals (interference signals) with different measurement times by scanning the same position in the tissue with the OCT light multiple times. The CPU 13 acquires motion contrast image data by performing arithmetic processing on the acquired OCT signal. Methods for processing OCT signals to obtain motion contrast data include, for example, a method of calculating a phase difference of a complex OCT signal, a method of calculating a vector difference of a complex OCT signal, decorrelation, ratio, difference, variance, etc. At least one of a method of calculating changes in amplitude or intensity and a method of multiplying a plurality of calculation results may be adopted.

Furthermore, the ophthalmologic image capturing apparatus 11 can capture (generate) a panoramic image of a tissue (fundus in this embodiment). A panoramic image is generated by aligning a plurality of images (hereinafter referred to as unit images) photographed in each of a plurality of photographing regions that do not completely overlap each other. The imaging area of each unit image is narrower than the imaging area of the entire panoramic image. Therefore, the ophthalmologic image capturing apparatus 11 can capture each unit image with high image resolution. Therefore, a panoramic image is a high-resolution image in which a wide range of tissue is captured. Note that an example of a method of acquiring a panoramic image of an OCT angio image is described, for example, in Japanese Unexamined Patent Application Publication No. 2017-47113.

However, it is also possible to use an ophthalmologic imaging apparatus other than the OCT apparatus. For example, a fundus camera, a laser scanning optometric device (SLO), a corneal endothelial cell imaging device (CEM), or the like may be used as an ophthalmologic imaging device. Also, the OCT apparatus may provide a two-dimensional tomographic image or a three-dimensional tomographic image to the mathematical model building device 1 or the ophthalmic image processing device 21 .

(mathematical model construction processing)
The mathematical model building process executed by the mathematical model building device 1 will be described with reference to FIGS. 2 to 4. FIG. The mathematical model building process is executed by CPU 3 according to a mathematical model building program stored in storage device 4 .

In the mathematical model construction process, the mathematical model is trained using the training data set to construct the mathematical model for outputting the target image from the original image. Although the details will be described later, the training data set includes input training data and output training data. In this embodiment, low-resolution training image data is used as input training data. Also, as training data for output, data of high-resolution training images whose image resolution is higher than that of low-resolution training images is used. The low-resolution training image and the high-resolution training image target the same tissue site. Also, the low-resolution training image and the high-resolution training image are images based on images captured by the same ophthalmologic imaging apparatus 11 .

As shown in FIG. 2, the CPU 3 acquires high-resolution panoramic images (in this embodiment, panoramic images of OCT angio images) as high-resolution training images 30 (S1). In this embodiment, the CPU 3 acquires the panoramic image generated by the ophthalmologic imaging device 11A through communication or the like. However, the CPU 3 may acquire the panoramic image by acquiring data (for example, motion contrast data) that is the basis for generating the panoramic image and generating the panoramic image based on the acquired data.

FIG. 3 shows an example of a high-resolution training image 31 used in this embodiment. As shown in FIG. 3, the high-resolution training image 31 of this embodiment is a panorama image of OCT angio images, and is generated by aligning four unit images 311, 312, 313, and 314. FIG. As an example, in this embodiment, each of the four unit images 311, 312, 313, and 314 is an image of a square area of 3 mm on a side of tissue. Each of the four unit images 311, 312, 313, and 314 has a resolution of 256pt×256pt. The photographing areas of the four unit images 311, 312, 313, and 314 are in contact with each other in a grid pattern without gaps. Therefore, in the high-resolution training image 31, which is a panoramic image, the imaging area on the tissue is a square area with a side of 6 mm, and the resolution is 512pt×512pt.

Next, the CPU 3 generates low-resolution training images 32 by lowering the image resolution of the high-resolution training images 31, which are panoramic images (S2). As an example, the CPU 3 of the present embodiment generates the low-resolution training image 32 by regularly (one by one in the present embodiment) thinning out a plurality of pixels forming the high-resolution training image 31 . As a result, in the low-resolution training image 32, the imaging region on the tissue becomes a square region with a side of 6 mm, like the imaging region in the high-resolution training image 31. FIG. On the other hand, the resolution of the low resolution training images 32 is reduced to 256pt×256pt. As a result, the image resolution of the low-resolution training images 32 (the number of pixels in the image per unit tissue area) is lower than the image resolution of the high-resolution training images 31 .

As described above, the CPU 3 reduces the image resolution of the high-resolution training images 31 to generate the low-resolution training images 32, thereby acquiring the training data set 30 including the high-resolution training images 31 and the low-resolution training images 32. do. In this case, the high-resolution training image 31 and the low-resolution training image 32 match in imaging region, imaging position, and tissue state at the time of imaging. Therefore, the training of the mathematical model described below is performed more appropriately.

Also, multiple training data sets are used to train the mathematical model. In this embodiment, training data including high-resolution training images 31, which are panoramic images, and low-resolution training images 32 generated by lowering the image resolution of the high-resolution training images 31, among the plurality of training data sets. The percentage of sets 30 is 50% or more (preferably 70% or more). As a result, the quality of the target image output by the mathematical model is further improved.

It should be noted that a method of obtaining a training data set that does not include the above-described high-resolution training image 31 and low-resolution training image 32 from among the plurality of training data sets can be appropriately selected. For example, the CPU 3 may acquire non-panoramic ophthalmic images as high-resolution training images. The CPU 3 may generate low-resolution training images by lowering the image resolution of high-resolution training images that are not panorama images. Also, multiple ophthalmic images generated by imaging the same tissue multiple times at different image resolutions may be used in the training data set. Note that the high-resolution training image capturing area and the low-resolution training image capturing area may not completely match. For example, part of the captured area of the high-resolution training image and the captured area of the low-resolution training image may overlap.

Next, the CPU 3 randomly adds noise to the low-resolution training image 32 (S3). Specifically, the CPU 3 randomly selects the low-resolution training images 32 to which noise is to be applied from among the plurality of low-resolution training images 32 . Moreover, when performing the noise addition process, the CPU 3 randomly determines the amount of noise to be added, the area to which the noise is added, and the like.

Next, the CPU 3 randomly blurs the low-resolution training images 32 (S4). Specifically, the CPU 3 randomly selects the low-resolution training images 32 to be blurred out of the plurality of low-resolution training images 32 . Further, when performing the blurring process, the CPU 3 randomly determines the degree of blurring, the area to be blurred, and the like. Note that the noise addition process (S3) and the blurring process (S4) are examples of degradation processing that degrades at least a portion of the low-resolution training image 32. FIG. It is also possible to change the specific method of deterioration processing.

CPU 3 then performs training of the mathematical model using the training data set by a machine learning algorithm (S5). As machine learning algorithms, for example, neural networks, random forests, boosting, support vector machines (SVM), etc. are generally known.

A neural network is a technique that imitates the behavior of a neural network of living organisms. Neural networks include, for example, feedforward neural networks, RBF networks (radial basis functions), spiking neural networks, convolutional neural networks, recurrent neural networks (recurrent neural networks, feedback neural networks, etc.), probability neural networks (Boltzmann machine, Baysian network, etc.), etc.

Random forest is a method of learning based on randomly sampled training data to generate a large number of decision trees. When a random forest is used, branches of a plurality of decision trees learned in advance as discriminators are traced, and the average (or majority vote) of the results obtained from each decision tree is taken.

Boosting is a method of generating a strong classifier by combining a plurality of weak classifiers. A strong classifier is constructed by sequentially learning simple and weak classifiers.

SVM is a method of constructing a two-class pattern discriminator using linear input elements. The SVM learns the parameters of the linear input element, for example, based on the criterion of finding the margin-maximizing hyperplane that maximizes the distance to each data point from the training data (hyperplane separation theorem).

A mathematical model, for example, refers to a data structure for predicting the relationship between input data and output data. A mathematical model is built by being trained using a training data set. A training data set is a set of input training data and output training data. A mathematical model is trained such that when certain input training data is input, corresponding output training data is output. For example, training updates the correlation data (eg, weights) for each input and output.

As an example, in this embodiment, as a machine learning algorithm, a generative adversarial network (GAN) utilizing two competing neural networks is employed. However, other machine learning algorithms may be used. For example, a multilayer neural network (eg, a convolutional neural network (CNN), etc.) may be used as the machine learning algorithm.

A method of training a mathematical model by the machine learning algorithm (GAN) employed in this embodiment will be described with reference to FIG. As mentioned above, the network of GANs comprises two sub-networks (generator and discriminator) that compete with each other. In this embodiment, first, an image 32E is generated by enlarging the low-resolution training image 32, which is training data for input. As a result, the sizes of the enlarged low-resolution training image 32E and the high-resolution training image 31 match. Note that the low-resolution training image 32 may be used as training input data without being enlarged, and the enlargement process may be performed by the generator. A generator then outputs a fake image 32F that closely matches the high resolution training image 31 based on the low resolution training image 32E. The discriminator attempts to discriminate whether the high-resolution training image 31, which is the real image, and the fake image 32F are correct. The determination result is reflected in both the generator and the discriminator. By repeating this training, the generator tries to generate a fake image 32F that is closer to the high-resolution training image 31, and the discriminator tries to improve the accuracy of discrimination. A mathematical model (generator model) constructed as a result of training is input with an ophthalmologic image with a low image resolution (for example, a base image to be described later), and an ophthalmologic image with a higher image resolution than the input image (for example, A target image (to be described later) can be generated.

As described above, in the present embodiment, the high-resolution image and the low-resolution image included in the training data set are images captured by the same ophthalmologic imaging device 11A (specifically, images captured by the same ophthalmologic imaging device). 11A). Therefore, the color tone of the high-resolution image and the low-resolution image, the amount of noise contained in the image, the artifact, the optical resolution, the imaging angle of view, the state of the tissue at the time of imaging, etc. are likely to approximate each other. Thus the mathematical model is properly trained.

Further, in the present embodiment, degradation processing is performed on at least part of the image region of the low-resolution training images used for at least one of the plurality of training data sets. Therefore, even if the image quality of at least a part of the ophthalmologic image (for example, a base image described later) input to the constructed mathematical model is not good, an image in which the influence of the unfavorable image quality is reduced is output.

Returning to the description of FIG. The processes of S1 to S5 are repeated until the construction of the mathematical model is completed (S6: NO). When construction of the mathematical model is completed (S6: YES), the mathematical model construction process ends. Programs and data for realizing the constructed mathematical model are installed in the ophthalmologic image processing apparatus 21 .

(Ophthalmic image processing)
The ophthalmologic image processing executed by the ophthalmologic image processing apparatus 21 will be described with reference to FIG. The ophthalmologic image processing is executed by the CPU 23 according to the ophthalmologic image processing program stored in the storage device 24 .

First, the CPU 23 acquires a base image, which is an ophthalmologic image captured by the ophthalmic imaging device 11B (S11). Note that in the present embodiment, the ophthalmologic image capturing device 11B captures the base images continuously (that is, as a moving image). The CPU 23 sequentially acquires the base images that are continuously photographed.

The base image in this embodiment is an OCT angio image of the fundus of the subject's eye, similar to the high-resolution training image 31 and the low-resolution training image 32 described above. Further, the base image in this embodiment is an image having an image resolution lower than that of the high-resolution training image 31 described above. As an example, the ophthalmologic imaging apparatus 11B of the present embodiment captures an imaging area of the same size as the high-resolution training image 31 (for example, an imaging area of 6 mm×6 mm) with a resolution lower than the resolution of the high-resolution training image 31 ( For example, by shooting at 256pt×256pt), the base image data is generated. Therefore, the imaging time for capturing the base image is shorter than the imaging time for capturing the high-resolution training image 31 . Therefore, when capturing the base image, the influence of the movement of the subject's eye (for example, involuntary eye movement and blinking), the burden of the tracking operation for following the movement of the subject's eye, and the like are taken into account in the high-resolution training image 31. is lower than when shooting . Note that when the target image is acquired, the ophthalmologic imaging apparatus 11B automatically sets the imaging region of the base image to the same imaging region as the high-resolution training image 31 . As a result, the base image and the target image are acquired more smoothly.

Next, the CPU 23 acquires a target image having a higher image resolution than the original image by inputting the original image into the above-described mathematical model (S12). In this embodiment, the CPU 23 successively acquires target images by sequentially inputting successively acquired base images into the mathematical model.

Next, the CPU 23 determines whether or not an instruction to display the target image on the display device 28 has been input (S14). In this embodiment, the user operates the operation unit 27 to instruct the display device 28 to display the target image, the base image, or both the target image and the base image. can be entered in If the instruction to display the target image has not been input (S14: NO), the process proceeds directly to S17. If an instruction to display the target image has been input (S14: YES), the CPU 23 causes the display device 28 to display the target image (S15). Specifically, in S15 of the present embodiment, a moving image of the tissue is displayed by sequentially displaying a plurality of target images successively acquired in S12 on the display device 28 . That is, a high-resolution moving image of the tissue being imaged is displayed in real time on the display device 28 based on the moving image of the base image (original image) captured at that time by the ophthalmologic image capturing device 11B. Note that the CPU 23 may generate moving image data of the target images based on the plurality of target images successively acquired in S12, and store the data in the storage device.

The ophthalmologic image processing apparatus 21 can use various methods to input instructions as to which of the target image, the base image, and both the target image and the base image should be displayed on the display device 28 . For example, the CPU 23 may cause the display device 28 to display an icon for inputting a display instruction for the target image, an icon for inputting a display instruction for the base image, and an icon for inputting a display instruction for both images. The user may input a display instruction to the ophthalmologic image processing apparatus 21 by selecting a desired icon.

Next, the CPU 23 determines whether or not an instruction to display the base image on the display device 28 has been input (S17). If no input has been made (S17: NO), the process proceeds directly to S20. If an instruction to display the base image has been input (S17: YES), the CPU 23 causes the display device 28 to display the base image (S18). Specifically, in S18 of the present embodiment, a moving image of the tissue is displayed by sequentially displaying a plurality of base images successively acquired in S11 on the display device 28 . As described above, the CPU 23 can switch between the target image and the base image and cause the display device 28 to display them in accordance with an instruction input by the user. Therefore, the user can easily confirm both the target image and the base image.

Next, the CPU 23 determines whether or not an instruction to simultaneously display the target image and the base image on the display device 28 has been input (S20). If no input has been made (S20: NO), the process proceeds directly to S23. If a simultaneous display instruction has been input (S20: YES), the CPU 23 causes the display device 28 to display the target image and the base image simultaneously. Specifically, in S<b>20 of the present embodiment, both the moving image of the target image and the moving image of the base image are displayed on the display device 28 .

Note that in S15, S18, and S21, a still image may be displayed instead of the moving image. The CPU 23 may switch between moving image display and still image display according to an instruction input by the user. Alternatively, the CPU 23 may cause the storage device 24 to store data of at least one of the moving image and the still image without displaying the moving image and the still image on the display device 28 .

Next, the CPU 23 determines whether or not an instruction to end the display has been input (S23). If not (S23: NO), the process returns to S11, and the processes of S11 to S23 are repeated. When an instruction to end the display is input (S23: YES), the CPU 23 executes various analysis processes on the target image (in this embodiment, at least one of a plurality of target images acquired continuously) ( S24). As described above, the image resolution of the target image is higher than the image resolution of the base image. Therefore, by executing the analysis process on the target image, the analysis accuracy is improved compared to the case where the analysis process is executed on the base image. Note that the specific content of the analysis process can be selected as appropriate. As an example, in S24 of the present embodiment, the blood vessel density of the fundus (for example, the blood vessel density around the non-perfused foveal region, etc.) is analyzed.

The timing of displaying the target image includes, for example, the timing of displaying the target image in real time during imaging by the ophthalmologic imaging apparatus 11B, the timing of allowing the user to check whether the captured image is good, and the analysis result after imaging. Alternatively, there are multiple timings, such as the timing to display with the report. The CPU 23 separately instructs which of the target image, the base image, and both the target image and the base image should be displayed on the display device 28 for each of a plurality of timings at which the target image can be displayed. can be entered in That is, the user sets in advance which of the target image, the base image, and both the target image and the base image is to be displayed on the display device 28 according to the timing at which the target image can be displayed. can be kept As a result, user convenience is further improved.

(Synthetic image generation processing)
Synthetic image generation processing executed by the CPU 23 of the ophthalmologic image processing apparatus 21 will be described with reference to FIG. As shown in FIG. 6, the synthesized image 40 includes a target image 42 and an attention area image 44 . The target image is a high-resolution ophthalmic image obtained by inputting the base image into the mathematical model, as described in S12 of FIG. A region-of-interest image 44 is an image obtained by photographing a region of interest that is part of the photographing region of the base image by the ophthalmologic imaging apparatus 11B. The attention area image 44 is captured with an image resolution higher than that of the base image. Also, the attention area image 44 is the image itself (that is, the original image) captured by the ophthalmologic image capturing device 11B. The method of setting the position, size, shape, etc. of the attention area with respect to the photographing area of the base image can be appropriately selected. For example, the ophthalmologic image capturing apparatus 11B may set the attention area within the capturing area of the base image according to an instruction input by the user. Further, the ophthalmic imaging apparatus 11B performs image processing on the base image to detect a characteristic region (for example, at least one of the macula, the optic papilla, and the fundus blood vessel) in the base image, and detect the characteristic region. may be used as a reference to automatically set the attention area.

The CPU 23 of the ophthalmologic image processing apparatus 21 acquires the base image and the attention area image 44 . The CPU 23 acquires the target image 42 having a higher image resolution than the original image by inputting the original image into the mathematical model. The CPU 23 generates a synthesized image 40 by synthesizing the attention area image 44 with the attention area in the target image 42 . According to the synthesized image 40, the user can confirm the attention area from the original image, and also confirm the entire image with a high resolution.

The technology disclosed in the above embodiment is merely an example. Therefore, it is also possible to modify the techniques exemplified in the above embodiments. First, it is also possible to implement only some of the techniques exemplified in the above embodiments. For example, instead of using panoramic images as high-resolution training images, a technique of performing at least one of noise addition processing and blurring processing on low-resolution training images may be adopted.

The process of acquiring the base image in S11 of FIG. 5 is an example of the "base image obtaining step". The process of acquiring the target image in S12 of FIG. 5 is an example of the "target image acquisition step". The process of simultaneously displaying the target image and the base image in S21 of FIG. 5 is an example of the "simultaneous display process" and the "simultaneous display step." The process of switching and displaying the base image and the target image in S14 to S18 of FIG. 5 is an example of the "switching display process" and the "switching display step". The process of acquiring the training data set in S1 and S2 of FIG. 2 is an example of the "image set acquisition step". The process of building a mathematical model in S5 of FIG. 2 is an example of a "training step." The process of acquiring the panorama image in S1 of FIG. 2 is an example of the "panorama image acquisition step". The process of generating the low-resolution training images 32 in S2 of FIG. 2 is an example of the "low-resolution training image generation step." The process of adding noise to the low-resolution training images 32 in S3 of FIG. 2 and the process of adding blurring to the low-resolution training images 32 in S4 of FIG. 2 are examples of "degradation processing" and "degradation step." .

1 mathematical model building device 3 CPU
4 storage device 11 ophthalmic imaging device 21 ophthalmic image processing device 23 CPU
24 storage device 27 operation unit 28 display device 30 training data set 31 high resolution training image 32 low resolution training image 40 composite image 42 target image 44 attention area image

Claims (4)

An ophthalmic image processing apparatus for processing an ophthalmic image that is an image of tissue of an eye to be examined,
The control unit of the ophthalmologic image processing apparatus includes:
Acquiring a base image, which is an ophthalmologic image captured by an ophthalmic imaging device,
obtaining a target image whose image resolution is higher than that of the base image by inputting the base image into a mathematical model trained by a machine learning algorithm;
The mathematical model uses, as an input, a low-resolution training image that is at least one ophthalmologic image out of a set of a plurality of ophthalmologic images captured by the same ophthalmologic image capturing device, with the same tissue site as an imaging target. Training is performed using a high-resolution training image, which is an ophthalmologic image having a higher image resolution than the low-resolution training image, as training data for output ,
At least one of the plurality of high-resolution training images is a panoramic image generated by aligning a plurality of ophthalmologic images captured in each of a plurality of imaging regions that do not completely overlap. ophthalmic image processing device. The ophthalmic image processing apparatus according to claim 1 ,
An ophthalmologic image processing apparatus, wherein at least one of the low-resolution training images is an image generated by lowering the image resolution of the panoramic image. An OCT apparatus that captures an ophthalmologic image of a tissue of an eye to be inspected by processing an OCT signal from reference light and reflected light of measurement light applied to a tissue of an eye to be inspected,
The control unit of the OCT device
Acquire the captured ophthalmic image as a base image,
obtaining a target image whose image resolution is higher than that of the base image by inputting the base image into a mathematical model trained by a machine learning algorithm;
The mathematical model uses, as an input, a low-resolution training image that is at least one ophthalmologic image out of a set of a plurality of ophthalmologic images captured by the same ophthalmologic image capturing device, with the same tissue site as an imaging target. Training is performed using a high-resolution training image, which is an ophthalmologic image having a higher image resolution than the low-resolution training image, as training data for output ,
At least one of the plurality of high-resolution training images is a panoramic image generated by aligning a plurality of ophthalmologic images captured in each of a plurality of imaging regions that do not completely overlap. OCT device to do. An ophthalmic image processing program executed by an ophthalmic image processing device that processes an ophthalmic image that is an image of tissue of an eye to be examined,
By executing the ophthalmic image processing program by the control unit of the ophthalmic image processing apparatus,
a base image obtaining step of obtaining a base image, which is an ophthalmologic image captured by an ophthalmic imaging device;
a target image obtaining step of obtaining a target image having an image resolution higher than that of the base image by inputting the base image into a mathematical model trained by a machine learning algorithm;
causes the ophthalmic image processing apparatus to execute
The mathematical model uses, as an input, a low-resolution training image that is at least one ophthalmologic image out of a set of a plurality of ophthalmologic images captured by the same ophthalmologic image capturing device, with the same tissue site as an imaging target. Training is performed using a high-resolution training image, which is an ophthalmologic image having a higher image resolution than the low-resolution training image, as training data for output ,
At least one of the plurality of high-resolution training images is a panoramic image generated by aligning a plurality of ophthalmologic images captured in each of a plurality of imaging regions that do not completely overlap. ophthalmic image processing program.

Images