JP7180187B2 - Ophthalmic image processing device, OCT device, and ophthalmic image processing program - Google Patents

Description

The present disclosure relates to an ophthalmic image processing apparatus that processes an ophthalmic image of an eye to be examined, an OCT apparatus, and an ophthalmic image processing program executed in the ophthalmic image processing apparatus.

Conventionally, at least one of the boundary of multiple tissues (for example, multiple layers) captured in an ophthalmologic image and a specific site on the tissue captured in an ophthalmologic image (hereinafter simply referred to as "boundary/specific site") Various techniques have been proposed for detecting For example, in the technique disclosed in Non-Patent Document 1, first, to which layer each pixel belongs is mapped for each pixel. Then, the boundaries of each layer are detected by outputting the thickness of each layer based on the mapping result.

Yufan He, Aaron Carass, et al. "Topology guaranteed segmentation of the human retina from OCT using convolutional neural networks." arXiv: 1803.05120, 14 Mar 2018

In the method of Non-Patent Document 1, for example, when the tissue structure is destroyed due to disease, it is difficult to accurately map which tissue each pixel belongs to. As a result, the detection accuracy of the tissue boundary/specific site also decreases.

A typical object of the present disclosure is to provide an ophthalmic image processing apparatus, an OCT apparatus, and an ophthalmic image processing program capable of appropriately detecting at least one of a tissue boundary and a specific site appearing in an ophthalmic image. That is.

An ophthalmic image processing apparatus provided by a typical embodiment of the present disclosure is an ophthalmic image processing apparatus that processes an ophthalmic image that is an image of a tissue of an eye to be examined, wherein a control unit of the ophthalmic image processing apparatus includes: Acquiring an ophthalmologic image captured by an imaging device and inputting the ophthalmologic image to a mathematical model trained by a machine learning algorithm to detect at least a specific boundary and a specific region of the tissue within a region in the ophthalmologic image. A probability distribution is obtained in which coordinates of one or more dimensions in which any of them exist are used as random variables, and at least one of the specific boundary and the specific part is detected based on the obtained probability distribution.

An OCT apparatus provided by a typical embodiment of the present disclosure processes an OCT signal of reference light and reflected light of measurement light applied to a tissue of an eye to be examined, thereby capturing an ophthalmologic image of the tissue. An OCT apparatus, wherein a control unit of the OCT apparatus inputs the photographed ophthalmologic image to a mathematical model trained by a machine learning algorithm, thereby identifying a specific boundary of the tissue within a region in the ophthalmologic image. and at least one of the specific part , obtaining a probability distribution with coordinates of one or more dimensions as random variables , and detecting at least one of the specific boundary and the specific part based on the obtained probability distribution do.

An ophthalmic image processing program provided by a typical embodiment of the present disclosure is an ophthalmic image processing program executed by an ophthalmic image processing apparatus that processes an ophthalmic image that is an image of tissue of an eye to be examined, the ophthalmic image processing A program is executed by the control unit of the ophthalmic image processing device to obtain an image acquisition step of acquiring an ophthalmic image captured by an ophthalmic image capturing device, and inputting the ophthalmic image to a mathematical model trained by a machine learning algorithm. a probability distribution obtaining step of obtaining a probability distribution in which at least one of a specific boundary and a specific part of the tissue exists in the region in the ophthalmologic image , with coordinates of one or more dimensions as random variables; and a detection step of detecting at least one of the specific boundary and the specific part based on the acquired probability distribution.

According to the ophthalmic image processing apparatus, OCT apparatus, and ophthalmic image processing program according to the present disclosure, at least one of the tissue boundary and the specific site appearing in the ophthalmic image is appropriately detected.

1 is a block diagram showing schematic configurations of a mathematical model construction device 1, an ophthalmologic image processing device 21, and ophthalmologic imaging devices 11A and 11B; FIG. 4 is a flowchart of mathematical model building processing executed by the mathematical model building device 1; FIG. 3 is a diagram showing an example of a training ophthalmologic image 30; 4 is a diagram showing an example of training data 31. FIG. 4 is a flowchart of boundary detection processing executed by the ophthalmologic image processing apparatus 21. FIG. 4 is a diagram schematically showing the relationship between a two-dimensional tomographic image 40 input to a mathematical model and one-dimensional regions A1 to AN in the two-dimensional tomographic image 40; FIG. It is an example of the graph which shows the probability distribution of the coordinate in which boundary Bi exists by making the one-dimensional coordinate of one-dimensional area|region A1 into a random variable. FIG. 3 shows an example of a probability map of the boundary Bi, which is the inner limiting membrane (ILM). FIG. 4 is a diagram showing an example of a probability map of the boundary Bg between the nerve fiber layer (NFL) and the ganglion cell layer (GCL); 4 is a flowchart of specific part detection processing executed by the ophthalmologic image processing apparatus 21. FIG. 5 is a diagram schematically showing the relationship between an ophthalmologic image 50 input to a mathematical model and a coordinate system C of two-dimensional coordinates in the ophthalmologic image 50. FIG. It is an example of a graph showing a probability distribution of coordinates where a specific part exists, with two-dimensional coordinates as random variables.

<Overview>
A control unit of an ophthalmic image processing apparatus exemplified in the present disclosure acquires an ophthalmic image captured by an ophthalmic image capturing apparatus. By inputting an ophthalmologic image to a mathematical model trained by a machine learning algorithm, the control unit obtains the probability that the coordinates of at least one of a specific boundary and a specific part of the tissue in the region in the ophthalmologic image are random variables. Get the distribution. The controller detects at least one of the specific boundary and the specific part based on the acquired probability distribution.

According to the ophthalmologic image processing apparatus exemplified in the present disclosure, for example, even when the tissue structure is destroyed due to the influence of a disease, or when the image quality of at least a part of the ophthalmologic image is not good, the coordinates are calculated with probability. The boundary/specific part is appropriately and directly detected based on the probability distribution used as a variable.

In the conventional technology described above, after each pixel belongs to which layer is mapped, the layer thickness is output based on the mapping result, and finally the layer boundary is detected. In this case, it is difficult to reduce the amount of processing because multistage processing is required. On the other hand, in the ophthalmologic image processing apparatus exemplified in the present disclosure, the amount of processing for detecting a boundary/specific part based on a probability distribution with coordinates as random variables is small. Therefore, the amount of processing can also be easily reduced.

Note that the aforementioned "area" may be any one of a one-dimensional area, a two-dimensional area, and a three-dimensional area. If the "area" is a one-dimensional area, the coordinates are one-dimensional coordinates. Similarly, if the "area" is a two-dimensional area, the coordinates will be two-dimensional coordinates, and if the "area" is a three-dimensional area, the coordinates will be three-dimensional coordinates.

The "specific boundary" for which the probability distribution is to be acquired may be one boundary or a plurality of boundaries. When obtaining probability distributions for a plurality of boundaries, the control unit may separately obtain probability distributions for each of the plurality of boundaries. Similarly, the number of "specific parts" for which the probability distribution is to be acquired may be one or more.

A mathematical model may be used in the process of detecting the boundary/specific part based on the probability distribution, as in the process of obtaining the probability distribution. Alternatively, the control unit itself may detect the boundary/specific part based on the obtained probability distribution without using the mathematical model.

A variety of images can be used as the ophthalmological image input to the mathematical model. For example, the ophthalmic image may be a two-dimensional tomographic image or a three-dimensional tomographic image of the tissue of the subject's eye captured by an OCT apparatus. A tomographic image may be captured by a device other than an OCT device (for example, a Scheimpflug camera or the like). The ophthalmologic image may be a two-dimensional frontal image captured by a fundus camera, a two-dimensional frontal image captured by a laser scanning optometric device (SLO), or the like. The ophthalmologic image may be a two-dimensional front image (so-called "Enface image") generated based on data of a three-dimensional tomographic image captured by an OCT apparatus. Further, the ophthalmologic image may be a two-dimensional frontal image (so-called "motion contrast image") created from motion contrast data obtained by processing a plurality of OCT data acquired from the same position at different times. . A two-dimensional front image is a two-dimensional image of a tissue photographed from the direction of the optical axis of photographing light. Also, the tissue to be imaged can be selected as appropriate. For example, an image obtained by photographing any one of the fundus, anterior segment, angle, etc. of the subject's eye may be used as the ophthalmologic image.

The mathematical model has, on the input side, data data of an ophthalmologic image of the tissue of the subject's eye taken in the past, and on the output side, the position of at least one of a specific boundary and a specific portion of the tissue in the ophthalmologic image on the input side. It may have been trained using a training data set with the data shown. In this case, the trained mathematical model can appropriately output a probability distribution of a specific boundary/specific region with coordinates as random variables by inputting ophthalmologic images.

A mathematical model building method executed by a mathematical model building device for building a mathematical model can be expressed as follows. A mathematical model building method executed by a mathematical model building device that builds a mathematical model that is built by being trained by a machine learning algorithm and outputs data corresponding to the input ophthalmologic image when an ophthalmologic image is input. an input training data acquiring step of acquiring an ophthalmic image of a tissue of an eye to be inspected as a training ophthalmic image; and a specific boundary of the tissue in the training ophthalmic image acquired in the input training data acquiring step. and an output training data obtaining step of obtaining training data indicating at least one position of a specific part; using the training ophthalmologic image data as input training data, and using the training data as output training data; Build a mathematical model that outputs a probability distribution whose random variables are the coordinates of at least one of a specific boundary and a specific part of the tissue in the region in the input ophthalmologic image by training the mathematical model as a mathematical model building method comprising:

By inputting an ophthalmologic image into the mathematical model, the control unit uses the one-dimensional coordinates of the specific boundary in the one-dimensional region extending in the direction intersecting the specific boundary of the tissue in the ophthalmologic image as random variables. A probability distribution may be obtained. The control unit may detect a specific boundary based on the obtained probability distribution. In this case, it is easy to obtain an appropriately biased probability distribution, so the accuracy of detecting a specific boundary is easy to improve.

Note that the ophthalmologic image may be a two-dimensional tomographic image or a three-dimensional tomographic image captured by an OCT apparatus. In this case, for a one-dimensional region extending in the direction of the optical axis of measurement light from the OCT apparatus (so-called “A-scan direction”), a probability distribution with one-dimensional coordinates as random variables may be obtained. The direction of the tissue boundary tends to be nearly perpendicular to the A-scan direction. Therefore, by setting the direction in which the one-dimensional region extends as the A-scan direction, the probability distribution tends to be appropriately biased. It also reduces the possibility that the one-dimensional region and a specific boundary will have more than one intersection point. Therefore, detection accuracy of a specific boundary is improved. Also, the control unit may acquire a probability distribution with one-dimensional coordinates as random variables for a one-dimensional region extending perpendicularly to a specific boundary. In this case, the probability distribution becomes more likely to be biased.

The control unit may detect a two-dimensional or three-dimensional boundary based on a plurality of probability distributions obtained for each of a plurality of one-dimensional regions different from each other. In this case, 2D or 3D boundaries are properly captured.

Note that the two-dimensional or three-dimensional boundary may be detected after post-processing is performed on the probability distribution for each of the plurality of one-dimensional regions. For example, the boundary on the N-th one-dimensional region is detected based on the probability distribution on the N-th one-dimensional region and the probability distribution on the one-dimensional regions located near the N-th one-dimensional region. good too. In this case, for example, a technique such as a known graph cut can be adopted. Two or three dimensional boundaries may also be detected by utilizing the shortest path search theory of finding the path with the lowest weight.

The control unit may obtain a two-dimensional or three-dimensional map indicating a particular boundary likelihood generated based on the plurality of probability distributions obtained for each of the plurality of one-dimensional regions. That is, the ophthalmic image processing apparatus may perform a map acquisition step of acquiring a two-dimensional or three-dimensional map indicating a particular boundary-likeness. In this case, the location of a particular 2D or 3D boundary is well known based on the map.

Note that the control unit may display the acquired map on the display device. In this case, the user can appropriately grasp the position of a specific two-dimensional or three-dimensional boundary by looking at the displayed map. The map may be generated by the controller based on multiple probability distributions output from the mathematical model. A mathematical model may also output a map.

The ophthalmologic image may be a three-dimensional tomographic image captured by an OCT apparatus. Based on the three-dimensional tomographic image and the detected three-dimensional boundary, the controller generates a two-dimensional front image ( A so-called "Enface image") may be acquired. That is, the ophthalmologic image processing apparatus may execute an enface image acquisition step of acquiring an enface image based on the three-dimensional tomographic image and the detected three-dimensional boundary. In this case, a specific layer is specified based on the appropriately detected boundary, and then the Enface image of the specific layer is appropriately acquired. The Enface image may be generated by the controller. A mathematical model may also output an Enface image.

In addition, the control unit may obtain a thickness map showing two-dimensional thickness distribution of a specific layer included in the three-dimensional tomographic image based on the three-dimensional tomographic image and the detected three-dimensional boundary. . In this case, specific layers are identified based on well-detected boundaries to obtain a more accurate thickness map.

By inputting an ophthalmologic image into the mathematical model, the control unit acquires a probability distribution of the probability that a specific part exists in a two-dimensional or more-dimensional area in the ophthalmologic image, with two-dimensional or more-dimensional coordinates as random variables. good. The control unit may detect the specific part based on the obtained probability distribution. In this case, the specific site is appropriately detected in the two-dimensional area or the three-dimensional area.

<Embodiment>
(Device configuration)
One typical embodiment of the present disclosure will be described below with reference to the drawings. As shown in FIG. 1, in this embodiment, a mathematical model construction device 1, an ophthalmologic image processing device 21, and ophthalmologic imaging devices 11A and 11B are used. The mathematical model building device 1 builds a mathematical model by training the mathematical model with a machine learning algorithm. Based on the input ophthalmologic image, the constructed mathematical model outputs a probability distribution whose random variables are the coordinates at which a specific boundary/specific part exists within a region in the ophthalmologic image. The ophthalmologic image processing apparatus 21 acquires a probability distribution using a mathematical model, and detects a specific boundary/specific part based on the probability distribution. The ophthalmologic image capturing devices 11A and 11B capture ophthalmologic images, which are images of tissues of the eye to be examined.

As an example, a personal computer (hereinafter referred to as “PC”) is used for the mathematical model building device 1 of this embodiment. Although the details will be described later, the mathematical model construction device 1 generates an ophthalmologic image (hereinafter referred to as a “training ophthalmologic image”) obtained from the ophthalmologic image capturing device 11A, and a specific boundary/specific portion of a tissue in the training ophthalmologic image. A mathematical model is constructed by training the mathematical model using the training data shown. However, a device that can function as the mathematical model construction device 1 is not limited to a PC. For example, the ophthalmologic imaging device 11A may function as the mathematical model building device 1 . Also, the control units of a plurality of devices (for example, the CPU of the PC and the CPU 13A of the ophthalmologic imaging apparatus 11A) may work together to construct the mathematical model.

A PC is used for the ophthalmologic image processing apparatus 21 of this embodiment. However, a device that can function as the ophthalmologic image processing apparatus 21 is not limited to a PC either. For example, the ophthalmologic imaging device 11B, a server, or the like may function as the ophthalmologic image processing device 21 . When the ophthalmic imaging device (an OCT device in this embodiment) 11B functions as the ophthalmic image processing device 21, the ophthalmic imaging device 11B captures an ophthalmic image while capturing a specific boundary/specification in the tissue of the captured ophthalmic image. Sites can be detected appropriately. Also, a portable terminal such as a tablet terminal or a smartphone may function as the ophthalmologic image processing apparatus 21 . The controllers of a plurality of devices (for example, the CPU of the PC and the CPU 13B of the ophthalmologic imaging apparatus 11B) may work together to perform various processes.

Also, in this embodiment, a case where a CPU is used as an example of a controller that performs various processes will be described. However, it goes without saying that a controller other than the CPU may be used for at least some of the various devices. For example, a GPU may be employed as a controller to speed up processing.

The mathematical model construction device 1 will be described. The mathematical model building device 1 is located, for example, in a manufacturer or the like that provides the ophthalmic image processing device 21 or an ophthalmic image processing program to users. The mathematical model construction device 1 includes a control unit 2 that performs various control processes, and a communication I/F 5 . The control unit 2 includes a CPU 3, which is a controller for control, and a storage device 4 capable of storing programs, data, and the like. The storage device 4 stores a mathematical model building program for executing a later-described mathematical model building process (see FIG. 2). Also, the communication I/F 5 connects the mathematical model construction device 1 with other devices (for example, the ophthalmologic imaging device 11A and the ophthalmologic image processing device 21, etc.).

The mathematical model construction device 1 is connected to an operation section 7 and a display device 8 . The operation unit 7 is operated by the user to input various instructions to the mathematical model construction device 1 . For example, at least one of a keyboard, a mouse, a touch panel, and the like can be used as the operation unit 7 . A microphone or the like for inputting various instructions may be used together with the operation unit 7 or instead of the operation unit 7 . The display device 8 displays various images. Various devices capable of displaying images (for example, at least one of a monitor, a display, a projector, etc.) can be used as the display device 8 . It should be noted that the “image” in the present disclosure includes both still images and moving images.

The mathematical model construction device 1 can acquire ophthalmologic image data (hereinafter sometimes simply referred to as “ophthalmologic image”) from the ophthalmologic imaging device 11A. The mathematical model construction device 1 may acquire ophthalmologic image data from the ophthalmologic imaging device 11A, for example, by at least one of wired communication, wireless communication, and a removable storage medium (eg, USB memory).

The ophthalmologic image processing apparatus 21 will be described. The ophthalmologic image processing apparatus 21 is installed, for example, in a facility (for example, a hospital, a health checkup facility, or the like) for diagnosing or examining a subject. The ophthalmologic image processing apparatus 21 includes a control unit 22 that performs various control processes, and a communication I/F 25 . The control unit 22 includes a CPU 23 which is a controller for control, and a storage device 24 capable of storing programs, data, and the like. The storage device 24 stores an ophthalmologic image processing program for executing later-described ophthalmologic image processing (for example, boundary detection processing illustrated in FIG. 5, specific part detection processing illustrated in FIG. 10, etc.). The ophthalmologic image processing program includes a program for realizing the mathematical model constructed by the mathematical model construction device 1 . The communication I/F 25 connects the ophthalmologic image processing apparatus 21 with other devices (for example, the ophthalmic imaging apparatus 11B, the mathematical model construction apparatus 1, etc.).

The ophthalmologic image processing apparatus 21 is connected to an operation section 27 and a display device 28 . Various devices can be used for the operation unit 27 and the display device 28, similarly to the operation unit 7 and the display device 8 described above.

The ophthalmic image processing device 21 can acquire an ophthalmic image from the ophthalmic image capturing device 11B. The ophthalmologic image processing apparatus 21 may acquire an ophthalmologic image from the ophthalmologic imaging apparatus 11B, for example, by at least one of wired communication, wireless communication, and a removable storage medium (eg, USB memory). Further, the ophthalmologic image processing apparatus 21 may acquire, via communication or the like, a program or the like for realizing the mathematical model constructed by the mathematical model construction apparatus 1 .

The ophthalmologic imaging devices 11A and 11B will be described. As an example, in this embodiment, an ophthalmic imaging device 11A that provides an ophthalmic image to the mathematical model building device 1 and an ophthalmic imaging device 11B that provides an ophthalmic image to the ophthalmic image processing device 21 are used. . However, the number of ophthalmic imaging devices used is not limited to two. For example, the mathematical model construction device 1 and the ophthalmologic image processing device 21 may acquire ophthalmologic images from a plurality of ophthalmologic imaging devices. Further, the mathematical model construction device 1 and the ophthalmologic image processing device 21 may acquire ophthalmologic images from one common ophthalmologic imaging device. Note that the two ophthalmologic imaging apparatuses 11A and 11B exemplified in this embodiment have the same configuration. Therefore, the two ophthalmologic imaging devices 11A and 11B will be collectively described below.

Further, in this embodiment, an OCT apparatus is exemplified as the ophthalmic imaging apparatus 11 (11A, 11B). However, an ophthalmologic imaging device other than the OCT device (for example, a laser scanning optometric device (SLO), a fundus camera, a Scheimpflug camera, or a corneal endothelial cell imaging device (CEM)) may be used.

The ophthalmologic imaging apparatus 11 (11A, 11B) includes a control unit 12 (12A, 12B) that performs various control processes, and an ophthalmologic imaging section 16 (16A, 16B). The control unit 12 includes a CPU 13 (13A, 13B), which is a controller for control, and a storage device 14 (14A, 14B) capable of storing programs, data, and the like.

The ophthalmologic image capturing unit 16 has various configurations necessary for capturing an ophthalmologic image of the subject's eye. The ophthalmologic image capturing unit 16 of the present embodiment includes an OCT light source, a branching optical element that branches the OCT light emitted from the OCT light source into measurement light and reference light, a scanning unit for scanning the measurement light, and a scanning unit for scanning the measurement light. It includes an optical system for irradiating an eye to be examined, a light receiving element for receiving the combined light of the light reflected by the tissue of the eye to be examined and the reference light, and the like.

The ophthalmologic image capturing apparatus 11 can capture two-dimensional tomographic images and three-dimensional tomographic images of the fundus of the subject's eye. Specifically, the CPU 13 captures a two-dimensional tomographic image of a cross section that intersects the scan lines by scanning OCT light (measurement light) along the scan lines. The two-dimensional tomographic image may be an averaging image generated by averaging a plurality of tomographic images of the same region. Moreover, the CPU 13 can capture a three-dimensional tomographic image of a tissue by two-dimensionally scanning the OCT light. For example, the CPU 13 acquires a plurality of two-dimensional tomographic images by causing measurement light to scan each of a plurality of scan lines at different positions in a two-dimensional region when the tissue is viewed from the front. Next, the CPU 13 acquires a three-dimensional tomographic image by combining a plurality of captured two-dimensional tomographic images.

(mathematical model construction processing)
The mathematical model building process executed by the mathematical model building device 1 will be described with reference to FIGS. 2 to 4. FIG. The mathematical model building process is executed by CPU 3 according to a mathematical model building program stored in storage device 4 . In the mathematical model construction process, the mathematical model is trained using the training data set, and outputs a probability distribution whose random variables are the coordinates of at least one of a specific boundary and a specific part of the tissue within the region in the ophthalmologic image. A mathematical model is constructed that The training data set includes data on the input side (training data for input) and data on the output side (training data for output).

As shown in FIG. 2, the CPU 3 acquires training ophthalmologic image data, which is an ophthalmologic image captured by the ophthalmologic imaging device 11A, as input training data (S1). In this embodiment, the training ophthalmologic image data is acquired by the mathematical model construction device 1 after being generated by the ophthalmologic imaging device 11A. However, the CPU 3 acquires a signal (for example, an OCT signal) that is the basis for generating a training ophthalmic image from the ophthalmologic imaging device 11A, and generates a training ophthalmic image based on the acquired signal. Image data may be obtained.

In the present embodiment, when the later-described boundary detection process (see FIG. 5) is executed by the ophthalmologic image processing apparatus 21, in S1, a two-dimensional tomographic image of the tissue of the eye to be examined (the fundus as an example) is used for training. acquired as an ophthalmologic image. FIG. 3 shows an example of a training ophthalmologic image 30, which is a two-dimensional tomographic image of the fundus. A plurality of layers in the fundus appear in the training ophthalmic image 30 illustrated in FIG.

Further, when the later-described specific part detection processing (see FIG. 10) is executed by the ophthalmologic image processing apparatus 21, in S1, a two-dimensional front image (for example, an OCT apparatus An Enface image or the like captured by a certain ophthalmologic image capturing device 11A) is acquired as a training ophthalmologic image.

However, it is also possible to change the ophthalmic image used as the training ophthalmic image. For example, when a two-dimensional frontal image is used as a training ophthalmic image, the training ophthalmic image is a two-dimensional frontal image captured by a device other than an OCT device (for example, an SLO device or a fundus camera). may Two-dimensional images other than frontal images (for example, two-dimensional tomographic images) may be used as training ophthalmologic images. Further, in the specific part detection processing (see FIG. 10) described later, when the specific part is detected in three dimensions, in S1, a three-dimensional tomographic image may be acquired as the training ophthalmologic image.

Next, the CPU 3 acquires training data indicating the position of at least one of the specific boundary of the tissue and the specific site in the ophthalmologic training image (S2). FIG. 4 shows an example of training data 31 indicating the position of a specific boundary when a two-dimensional tomographic image of the fundus is used as the training ophthalmologic image 30 . The training data 31 illustrated in FIG. 4 includes data of labels 32A-32F indicating the positions of each of the six boundaries in the ophthalmic training image 30. FIG. In this embodiment, the data of the labels 32A to 32F in the training data 31 are generated by the operator operating the operation section 7 while looking at the boundaries in the ophthalmologic training image 30. FIG. However, it is also possible to change the method of generating label data.

It should be noted that when the later-described specific region detection process (see FIG. 10) is executed by the ophthalmologic image processing apparatus 21, the position of the specific region in the tissue of the two-dimensional tomographic image or the three-dimensional tomographic image, which is the training ophthalmologic image, is The data shown is obtained in S2. For example, the position of the fovea is detected in the specific part detection process to be described later. In this case, in S2, data indicating the position of the fovea in the two-dimensional tomographic image or the three-dimensional tomographic image is acquired.

CPU 3 then performs training of the mathematical model using the training data set by a machine learning algorithm (S3). As machine learning algorithms, for example, neural networks, random forests, boosting, support vector machines (SVM), etc. are generally known.

A neural network is a technique that imitates the behavior of a neural network of living organisms. Neural networks include, for example, feedforward neural networks, RBF networks (radial basis functions), spiking neural networks, convolutional neural networks, recurrent neural networks (recurrent neural networks, feedback neural networks, etc.), probability neural networks (Boltzmann machine, Baysian network, etc.), etc.

Random forest is a method of learning based on randomly sampled training data to generate a large number of decision trees. When a random forest is used, branches of a plurality of decision trees learned in advance as discriminators are traced, and the average (or majority vote) of the results obtained from each decision tree is taken.

Boosting is a method of generating a strong classifier by combining a plurality of weak classifiers. A strong classifier is constructed by sequentially learning simple and weak classifiers.

SVM is a method of constructing a two-class pattern discriminator using linear input elements. The SVM learns the parameters of the linear input element, for example, based on the criterion of finding the margin-maximizing hyperplane that maximizes the distance to each data point from the training data (hyperplane separation theorem).

A mathematical model, for example, refers to a data structure for predicting the relationship between input data and output data. A mathematical model is built by being trained using a training data set. As described above, the training data set is a set of input training data and output training data. For example, training updates the correlation data (eg, weights) for each input and output.

In this embodiment, a multilayer neural network is used as the machine learning algorithm. A neural network includes an input layer for inputting data, an output layer for generating data to be predicted, and one or more hidden layers between the input layer and the output layer. A plurality of nodes (also called units) are arranged in each layer. Specifically, in this embodiment, a convolutional neural network (CNN), which is a type of multilayer neural network, is used. In addition, the mathematical model constructed in this embodiment is at least one of a specific boundary and a specific part of tissue within a region (any one of a one-dimensional region, a two-dimensional region, and a three-dimensional region) in an ophthalmologic image. Outputs a probability distribution whose random variable is the coordinates (any one of one-dimensional, two-dimensional, and three-dimensional coordinates) where is present. In this embodiment, a softmax function is applied to force the mathematical model to output a probability distribution.

In this embodiment, when the later-described boundary detection process (see FIG. 5) is executed by the ophthalmologic image processing apparatus 21, the mathematical model constructed in S3 intersects a specific boundary in the two-dimensional tomographic image. A probability distribution of coordinates at which a specific boundary exists in a one-dimensional area on a line extending in a direction (the A-scan direction of OCT in this embodiment) is output.

Further, when the later-described specific region detection process (see FIG. 10) is executed by the ophthalmologic image processing device 21, the mathematical model constructed in S3 is a two-dimensional region in the two-dimensional ophthalmologic image. A probability distribution is output in which the two-dimensional coordinates at which the part (for example, the fovea in this embodiment) exists are random variables.

However, other machine learning algorithms may be used. For example, Generative adversarial networks (GAN), which utilize two competing neural networks, may be employed as machine learning algorithms.

The processes of S1 to S3 are repeated until the construction of the mathematical model is completed (S4: NO). When construction of the mathematical model is completed (S4: YES), the mathematical model construction process ends. Programs and data for realizing the constructed mathematical model are installed in the ophthalmologic image processing apparatus 21 .

(Ophthalmic image processing)
The ophthalmologic image processing executed by the ophthalmologic image processing apparatus 21 will be described with reference to FIGS. 5 to 12 . The ophthalmic image processing is executed by the CPU 23 according to the ophthalmic image processing program stored in the storage device 21 .

(Boundary detection processing)
Boundary detection processing, which is an example of ophthalmologic image processing, will be described with reference to FIGS. In the boundary detection process, in each of a plurality of one-dimensional regions, a specific boundary (in this embodiment, a two-dimensional boundary and a three-dimensional ) are detected. Further, an Enface image of a specific layer in the tissue is acquired based on the detected three-dimensional boundaries.

First, the CPU 23 acquires a three-dimensional tomographic image of the tissue of the subject's eye (the fundus in this embodiment) (S11). The three-dimensional tomographic image is captured by the ophthalmologic imaging device 11B and acquired by the ophthalmologic image processing device 21 . As described above, a three-dimensional tomographic image is formed by combining a plurality of two-dimensional tomographic images captured by scanning different scan lines with measurement light. Note that the CPU 23 may acquire a signal (for example, an OCT signal) that serves as a basis for generating a three-dimensional tomographic image from the ophthalmic imaging apparatus 11B and generate a three-dimensional tomographic image based on the acquired signal.

The CPU 23 extracts the T-th (initial value of T is "1") two-dimensional tomographic image among the plurality of two-dimensional tomographic images forming the acquired three-dimensional tomographic image (S12). FIG. 6 shows an example of a two-dimensional tomographic image 40. As shown in FIG. A plurality of boundaries of the fundus of the subject's eye appear in the two-dimensional tomographic image 40 . In the example shown in FIG. 6, a plurality of boundaries including a boundary Bi that is the inner limiting membrane (ILM) and a boundary Bg between the nerve fiber layer (NFL) and the ganglion cell layer (GCL) appear. Also, a plurality of one-dimensional regions A1 to AN are set in the two-dimensional tomographic image 40. FIG. In the present embodiment, the plurality of one-dimensional regions A1 to AN set in the two-dimensional tomographic image 40 are arranged on axes that intersect a specific boundary (in this embodiment, a plurality of boundaries including the boundary Bi and the boundary Bg). extend along. Specifically, the one-dimensional areas A1 to AN of the present embodiment correspond to the areas of each of the multiple (N) A-scans forming the two-dimensional tomographic image 40 captured by the OCT apparatus.

It is also possible to change the method of setting a plurality of one-dimensional regions. For example, the CPU 23 may set a plurality of one-dimensional areas such that the angle between the axis of each one-dimensional area and a specific boundary is as close to vertical as possible. In this case, the position and angle of each one-dimensional region may be set, for example, based on the general shape of the tissue of the subject's eye (in this embodiment, the fundus) so that the angle approaches the vertical.

By inputting the T-th two-dimensional tomographic image to the mathematical model, the CPU 23 obtains the coordinates of the M-th boundary (the initial value of M is "1") in each of the plurality of one-dimensional regions A1 to AN. A probability distribution is acquired (S14). FIG. 7 shows an example of a graph showing the probability distribution of the coordinates at which the boundary Bi exists, acquired from the one-dimensional area A1. In the example shown in FIG. 7, the probability distribution of the coordinates at which the boundary Bi exists is shown using the one-dimensional coordinates of the one-dimensional area A1 as random variables. That is, in the example shown in FIG. 7, the horizontal axis is the random variable, the vertical axis is the probability of the random variable, and the random variable is the coordinates of the boundary Bi in the one-dimensional area A1. In S14, probability distributions are acquired for each of the plurality of one-dimensional areas A1 to AN.

According to the graph shown in FIG. 7, it can be determined that the point P is the point at which the boundary Bi is most likely to exist among the points on the one-dimensional area A1. In addition, even if the structure of the tissue is destroyed due to the influence of the disease, or if the image quality of at least a part of the ophthalmologic image is not good, the probability distribution output by the mathematical model is There is a high probability of forming a peak at the position. Therefore, by using the probability distribution output in S14, the boundary is properly and directly detected.

The CPU 23 acquires a probability map two-dimensionally showing the ratio of the Mth boundary existing in the Tth two-dimensional tomographic image. The CPU 23 displays the obtained probability map on the display device 28 (S15). A probability map is generated by two-dimensionally arranging a plurality of probability distributions obtained for each of the plurality of one-dimensional regions A1 to AN. The probability map may be generated by the CPU 23 based on multiple probability distributions. A probability map may also be generated by the mathematical model.

FIG. 8 is an example of a probability map 41 of the boundary Bi, which is the inner limiting membrane (ILM). FIG. 9 is an example of a probability map 42 of the boundary Bg between the nerve fiber layer (NFL) and the ganglion cell layer (GCL). In the probability maps 41 and 42 illustrated in FIGS. 8 and 9, the value for each coordinate is represented by luminance, and the position that seems to be a specific boundary is brighter. Therefore, the user can appropriately grasp the positions of the boundaries Bi and Bg two-dimensionally by looking at the displayed probability maps 41 and 42 . Needless to say, the representation method of the probability map can be changed as appropriate.

Next, the CPU 23 detects the two-dimensional M-th boundary based on the probability distribution regarding the M-th boundary obtained for each of the plurality of one-dimensional areas A1 to AN (S16). In this embodiment, the CPU 23 detects a two-dimensional boundary by combining a plurality of probability distributions acquired for each of the plurality of one-dimensional areas A1 to AN. A two-dimensional boundary may be detected based on the plurality of probability distributions acquired in S14, or may be detected based on the probability map acquired in S15. Further, the mathematical model may output the boundary detection result, or the CPU 23 may perform the boundary detection processing.

Note that, in the present embodiment, the probability distributions regarding the plurality of one-dimensional areas A1 to AN are subjected to post-processing, and then the probability maps are generated and the boundaries are detected. As an example, the boundary on the n-th one-dimensional area is detected based on the probability distribution on the n-th one-dimensional area and the probability distribution on the one-dimensional area located near the n-th one-dimensional area. may In this case, for example, a method such as a known graph cut may be adopted. Boundaries may also be detected by utilizing shortest path search theory that finds the path with the lowest weight.

Further, the CPU 23 may superimpose the probability maps 41 and 42 or the detected specific boundaries on the two-dimensional tomographic image (that is, the raw image) whose boundaries are to be detected. In this case, the positional relationship between the actually photographed tissue and the boundary detected by the probability distribution is properly grasped. Further, the CPU 23 may input an instruction from the user for designating the position of the boundary in the tissue while the probability maps 41 and 42 or the detected boundary is superimposed on the raw image. In this case, the user can appropriately designate the position of the boundary while comparing the boundary detected by the probability distribution and the actually captured image of the tissue.

Next, the CPU 23 determines whether or not detection of all boundaries to be detected in the T-th two-dimensional tomographic image has been completed (S18). If the detection of some boundaries has not been completed (S18: NO), "1" is added to the boundary order M (S19), the process returns to S14, and the next boundary detection process is executed. (S14-S16). When detection of all boundaries is completed (S18: YES), the CPU 23 determines whether or not detection of boundaries of all two-dimensional tomographic images forming the three-dimensional tomographic image has been completed (S21). If the detection of the boundary of some of the two-dimensional tomographic images has not been completed (S21: NO), "1" is added to the order T of the two-dimensional tomographic images (S22), the process returns to S12, and the next A boundary of the two-dimensional tomographic image is detected (S12-S19).

When the detection of the boundaries of all the two-dimensional tomographic images is completed (S21: YES), the detection of the three-dimensional boundaries is completed. The CPU 23 acquires a three-dimensional boundary by integrating a plurality of two-dimensional boundaries detected in S16 regarding at least one of the M boundaries (S24). The CPU 23 can generate image data of the detected three-dimensional boundary and display it on the display device 28 . Therefore, the user can appropriately grasp the three-dimensional shape of the boundary. Further, a probability map indicating the probability distribution in the three-dimensional area may be generated based on the acquisition result of the probability distribution for T two-dimensional tomographic images.

Next, the CPU 23 acquires an Enface image of a specific layer (or boundary) included in the three-dimensional tomographic image based on the detected three-dimensional boundary and the three-dimensional tomographic image acquired in S11 ( S25). An enface image is a two-dimensional front image of a specific layer viewed in a direction along the optical axis of OCT measurement light. The Enface image acquired at S25 is of high quality because it is generated based on well-detected boundaries. Note that in S25, an Enface image for a specific one of a plurality of layers and boundaries included in the three-dimensional image may be acquired. Enface images may also be obtained for more than one of the layers and boundaries included in the three-dimensional image.

Further, the CPU 23 creates a thickness map two-dimensionally showing the thickness distribution of a specific layer included in the three-dimensional tomographic image based on the three-dimensional tomographic image acquired in S11 and the detected three-dimensional boundary. may be obtained. In this case, specific layers are identified based on well-detected boundaries to obtain a more accurate thickness map.

In the example shown in FIG. 5, after the three-dimensional tomographic image is acquired in S11, specific boundaries are detected two-dimensionally and three-dimensionally. Further, an Enface image is acquired based on the detected three-dimensional boundaries. However, only two-dimensional boundaries may be detected. In this case, a two-dimensional ophthalmologic image may be acquired in S11. Also, in the example shown in FIG. 5, in S14, a plurality of probability distributions are obtained for each of the plurality of one-dimensional areas A1 to AN. As a result, two-dimensional boundaries are detected. However, for example, when only one point on a specific boundary needs to be detected, in S14 only the probability distribution in one one-dimensional area may be obtained.

(Specific part detection processing)
Specific part detection processing, which is an example of ophthalmologic image processing, will be described with reference to FIGS. 10 to 12 . In the specific part detection process, the specific part is detected based on the probability distribution of the coordinates at which the specific part exists. A case of detecting a specific part from a two-dimensional ophthalmologic image will be exemplified below.

First, the CPU 23 acquires an ophthalmologic image of the tissue of the subject's eye (the fundus in this embodiment) (S31). As an example, in S31 of the present embodiment, a two-dimensional front image (for example, an Enface image or a motion contrast image) captured by an OCT device is acquired. However, the ophthalmologic image acquired in S31 can be changed as appropriate. For example, a two-dimensional front image captured by an SLO device, a fundus camera, or the like may be acquired in S31. Also, an ophthalmologic image in which tissues other than the fundus are photographed may be acquired in S31.

FIG. 11 shows an example of an ophthalmologic image 50 acquired in S31 of this embodiment. A two-dimensional ophthalmologic image 50 illustrated in FIG. 11 shows an optic papilla 51, a macula 52, and a fundus blood vessel 53 in the fundus. The fovea F is the center of the macula 52 . Also, a two-dimensional coordinate system (XY coordinates) is set in the two-dimensional area C in the two-dimensional ophthalmologic image 50 .

By inputting the ophthalmologic image 50 into the mathematical model, the CPU 23 acquires a probability distribution of the presence of a specific site in the two-dimensional region in the ophthalmologic image 50, with the two-dimensional coordinates as random variables (S32). As an example, the specific site in this embodiment is the fovea F. However, it goes without saying that the specific site is not limited to the fovea F. For example, the specific site may be the optic disc or the like. Also, the specific site may be one point, or may be a site having a certain area or volume. The CPU 23 detects specific regions in the two-dimensional ophthalmologic image 50 based on the probability distribution acquired in S32 (S33).

FIG. 12 shows an example of a graph showing a probability distribution obtained from a two-dimensional coordinate system and having coordinates at which a specific part exists as random variables. In the example shown in FIG. 12, the X-axis and Y-axis are random variables, the axis perpendicular to the X-axis and the Y-axis is the probability of the random variable, and the random variable is the fovea F in the two-dimensional area C. coordinates. According to the graph shown in FIG. 12, it can be determined that (X', Y') are the coordinates of the point in the two-dimensional region C where the fovea F is most likely to exist. In addition, even if the structure of the tissue is destroyed due to the influence of the disease, or if the image quality of at least a part of the ophthalmologic image is not good, the probability distribution output by the mathematical model is There is a high probability of forming a peak at the position. Therefore, the specific site is appropriately and directly detected.

In addition, in the specific part detection process, it is also possible to detect a specific part from a three-dimensional ophthalmologic image. In this case, in S31, a three-dimensional ophthalmologic image (for example, a three-dimensional tomographic image captured by an OCT apparatus, etc.) is acquired. In S32, in a three-dimensional area in the three-dimensional ophthalmologic image, a probability distribution is obtained with the three-dimensional coordinates of the specific site as random variables. In S33, a specific site in the three-dimensional ophthalmologic image is detected based on the probability distribution acquired in S32.

The technology disclosed in the above embodiment is merely an example. Therefore, it is also possible to modify the techniques exemplified in the above embodiments. First, it is also possible to implement only some of the techniques exemplified in the above embodiments. For example, the ophthalmologic image processing apparatus 21 may perform only one of the boundary detection process (see FIG. 5) and the specific part detection process (see FIG. 10).

Note that the process of acquiring the training ophthalmologic image in S1 of FIG. 2 is an example of the "input training data acquisition step". The process of acquiring the training data in S2 of FIG. 2 is an example of the "output training data acquisition step". The process of training the mathematical model in S3 of FIG. 2 is an example of a "training step." The process of acquiring an ophthalmologic image in S11 of FIG. 5 and S31 of FIG. 10 is an example of an "image acquisition step." The process of acquiring the probability distribution in S14 of FIG. 5 and S32 of FIG. 10 is an example of the "probability distribution acquisition step". The process of detecting a boundary or a specific part in S16, S24 of FIG. 5 and S33 of FIG. 10 is an example of a "detection step." The process of acquiring the probability map in S15 of FIG. 5 is an example of the "map acquisition step". The process of acquiring the Enface image in S25 of FIG. 5 is an example of the "enface image acquisition step".

1 mathematical model building device 3 CPU
11A, 11B ophthalmic imaging device 21 ophthalmic image processing device 23 CPU
24 storage device 28 display device 30 two-dimensional tomographic images 41 and 42 probability map 50 ophthalmic image

Claims (6)

An ophthalmic image processing apparatus for processing an ophthalmic image that is an image of tissue of an eye to be examined,
The control unit of the ophthalmologic image processing apparatus includes:
Acquiring an ophthalmic image captured by an ophthalmic imaging device,
By inputting the ophthalmic image into a mathematical model trained by a machine learning algorithm, one or more dimensional coordinates of at least one of a specific boundary and/or a specific portion of the tissue within a region in the ophthalmic image are obtained. Get the probability distribution as a random variable,
An ophthalmologic image processing apparatus, wherein at least one of the specific boundary and the specific part is detected based on the acquired probability distribution. The ophthalmic image processing apparatus according to claim 1,
The control unit
By inputting the ophthalmological image into the mathematical model, the one-dimensional coordinates of the specific boundary in the one-dimensional region extending in the direction intersecting the specific boundary of the tissue in the ophthalmic image are determined as random variables. Obtain the probability distribution for
An ophthalmologic image processing apparatus, wherein the specific boundary is detected based on the acquired probability distribution. The ophthalmic image processing apparatus according to claim 2,
The control unit
An ophthalmologic image processing apparatus that detects the two-dimensional or three-dimensional boundary based on a plurality of the probability distributions obtained for each of the plurality of one-dimensional regions that are different from each other. The ophthalmic image processing apparatus according to claim 1,
The control unit
By inputting the ophthalmologic image into the mathematical model, obtaining a probability distribution with the coordinates of the two or more dimensions where the specific part exists as a random variable in a region of two or more dimensions in the ophthalmologic image,
An ophthalmologic image processing apparatus, wherein the specific part is detected based on the acquired probability distribution. An OCT apparatus that captures an ophthalmologic image of a tissue of an eye to be inspected by processing an OCT signal from reference light and reflected light of measurement light applied to a tissue of an eye to be inspected,
The control unit of the OCT device
By inputting the captured ophthalmological image into a mathematical model trained by a machine learning algorithm, one or more dimensions in which at least one of a specific boundary and a specific part of the tissue exists within a region in the ophthalmic image. Obtain the probability distribution with the coordinates of as random variables,
An OCT apparatus that detects at least one of the specific boundary and the specific site based on the acquired probability distribution. An ophthalmic image processing program executed by an ophthalmic image processing device that processes an ophthalmic image that is an image of tissue of an eye to be examined,
By executing the ophthalmic image processing program by the control unit of the ophthalmic image processing apparatus,
an image acquisition step of acquiring an ophthalmic image captured by an ophthalmic imaging device;
By inputting the ophthalmic image into a mathematical model trained by a machine learning algorithm, one or more dimensional coordinates of at least one of a specific boundary and/or a specific portion of the tissue within a region in the ophthalmic image are obtained. a probability distribution obtaining step for obtaining a probability distribution as a random variable;
a detection step of detecting at least one of the specific boundary and the specific part based on the acquired probability distribution;
is executed by the ophthalmologic image processing apparatus.

Images