JP2023149742A - Ocular fundus image processing device and ocular fundus image processing program - Google Patents

Abstract

To provide on ocular fundus image processing device and an ocular fundus image processing program capable of more appropriately presenting, to a user, a degree of deviation when a site captured in an ocular fundus image is identified by a mathematical model.SOLUTION: In an image acquisition step, a control part acquires a three-dimensional image of the ocular fundus captured by an ocular fundus image capturing device. In a deviation degree acquisition step, the control part acquires a plurality of probability distributions for identifying each of a plurality of sites in an ocular fundus tissue captured in the three-dimensional image by inputting the three-dimensional image to a mathematical model trained by a machine learning algorithm, and acquires a deviation degree of the acquired probability distribution for the probability distribution when each of the plurality of sites is accurately identified. In a selection display step, the control part causes a display part to selectively display a map indicating a two-dimensional distribution of part of the plurality of acquired deviation degrees.SELECTED DRAWING: Figure 11

Description

The present disclosure relates to a fundus image processing device and a fundus image processing program used to process a fundus image of an eye to be examined.

In recent years, a technique has been proposed that uses a mathematical model trained by a machine learning algorithm to obtain medical information from a fundus image of an eye to be examined. For example, the image processing device described in Patent Document 1 obtains a probability distribution for identifying tissue in an ophthalmological image by inputting the ophthalmological image into a mathematical model trained by a machine learning algorithm. The image processing device obtains the degree of deviation of the obtained probability distribution from the probability distribution when the tissue is accurately identified as structural information indicating the degree of abnormality of the structure of the tissue.

Japanese Patent Application Publication No. 2020-18794

In order for a user (for example, a doctor, etc.) to use the degree of deviation more effectively for diagnosis, etc., it is desirable to be able to present the degree of deviation to the user in an appropriate manner according to various situations.

A typical object of the present disclosure is to provide a fundus image processing device and a fundus image processing program that can more appropriately present to a user the degree of deviation when a region shown in a fundus image is identified by a mathematical model. That's true.

A fundus image processing device provided by a typical embodiment of the present disclosure is a fundus image processing device that processes a fundus image of an eye to be examined, and a control unit of the fundus image processing device is configured to control a fundus image processing device that processes a fundus image of an eye to be examined. an image acquisition step of acquiring a three-dimensional image of the fundus; and inputting the three-dimensional image to a mathematical model trained by a machine learning algorithm to identify each of a plurality of parts of the fundus tissue shown in the three-dimensional image. a step of obtaining a degree of deviation of the obtained probability distributions with respect to the probability distribution when each of the plurality of parts is accurately identified; a selective display step of selectively displaying a map showing a two-dimensional distribution of some of the deviation degrees on the display unit;

A fundus image processing program provided by a typical embodiment of the present disclosure is a fundus image processing program executed by a fundus image processing device that processes a fundus image of an eye to be examined, wherein the fundus image processing program an image acquisition step executed by a control unit of the processing device to acquire a three-dimensional image of the fundus taken by the fundus image capturing device; and inputting the three-dimensional image to a mathematical model trained by a machine learning algorithm. By doing this, a plurality of probability distributions for identifying each of a plurality of parts in the fundus tissue shown in the three-dimensional image are obtained, and the obtained probability distribution for each of the plurality of parts is accurately identified. a deviation degree obtaining step of obtaining a deviation degree of a probability distribution obtained by the above-described deviation; and a selective display for selectively displaying a map showing a two-dimensional distribution of some of the deviation degrees among the obtained plurality of deviation degrees on a display unit. and causing the fundus image processing device to execute the steps.

According to the fundus image processing device and the fundus image processing program according to the present disclosure, the degree of deviation when a region appearing in a fundus image is identified by a mathematical model is more appropriately presented to the user.

1 is a block diagram showing a schematic configuration of a mathematical model construction device 101, a fundus image processing device 1, and OCT devices 10A and 10B. FIG. 2 is an explanatory diagram for explaining an example of a method of photographing a three-dimensional tomographic image. 4 is a diagram showing an example of a two-dimensional tomographic image 42. FIG. 4 is a diagram showing an example of a three-dimensional tomographic image 43. FIG. FIG. 2 is a diagram schematically showing the structure of layers and boundaries in the fundus of the eye. 2 is a flowchart of mathematical model construction processing executed by the mathematical model construction device 101. 2 is a flowchart of fundus image processing executed by the fundus image processing device 1. FIG. It is a flowchart of the deviation degree acquisition process performed during fundus oculi image processing. 4 is a diagram schematically showing the relationship between a two-dimensional tomographic image 42 input to a mathematical model and one-dimensional areas A1 to AN in the two-dimensional tomographic image 42. FIG. It is a figure which shows an example of several deviation degree maps. FIG. 3 is a diagram illustrating a display example of a two-dimensional tomographic image and a discrepancy map. It is a figure which shows the example of a display of a thickness map and a deviation degree map.

<Summary>
The control unit of the fundus image processing device exemplified in this disclosure executes an image acquisition step, a deviation degree acquisition step, and a selection display step. In the image acquisition step, the control unit acquires a three-dimensional image of the fundus taken by the fundus image photographing device. In the discrepancy degree acquisition step, the control unit inputs the three-dimensional image into a mathematical model trained by a machine learning algorithm, thereby obtaining a plurality of probabilities for identifying each of the plurality of parts of the fundus tissue shown in the three-dimensional image. A distribution is obtained, and the degree of deviation of the obtained probability distribution from the probability distribution when each of the plurality of parts is accurately identified is obtained. In the selective display step, the control unit selectively displays, on the display unit, a map showing a two-dimensional distribution of some of the acquired deviation degrees (hereinafter referred to as “deviation degree map”). let

According to the fundus image processing device exemplified in the present disclosure, a portion of the degree of deviation in the identification of each of a plurality of parts in the fundus tissue shown in the three-dimensional image (hereinafter referred to as "the degree of deviation in identification of multiple parts") is A two-dimensional distribution of the degree of deviation of is selectively displayed on the display unit. Therefore, the degree of deviation in region identification is presented to the user more appropriately.

The degree of deviation will be explained. For example, if there is no abnormality in the structure of the tissue, it is easy for the mathematical model to accurately identify the site, so the obtained probability distribution is likely to be biased. On the other hand, if there is an abnormality in the structure of the tissue, the obtained probability distribution will be less likely to be biased. Therefore, the degree of deviation between the probability distribution when a part is accurately identified and the actually obtained probability distribution often increases or decreases depending on the degree of abnormality of the structure. Therefore, the user can obtain information useful for diagnosis from the displayed deviation map. For example, the user can grasp the position on the fundus tissue where the degree of structural abnormality is estimated to be high based on the displayed discrepancy map, and create a detailed image (for example, a two-dimensional tomographic image) of the grasped position. By checking, it is also possible to improve the efficiency of diagnosis.

The degree of deviation may be output by a mathematical model. Alternatively, the control unit may calculate the degree of deviation based on a probability distribution output by a mathematical model.

The degree of deviation may include the entropy (average information amount) of the acquired probability distribution. Entropy represents the degree of uncertainty, randomness, and disorder. In the present disclosure, the entropy of the output probability distribution is 0 when the site is correctly identified. Furthermore, the more difficult it is to identify a region, the more entropy increases. Therefore, by using the entropy of the probability distribution as the degree of deviation, the degree of abnormality of the tissue structure can be more appropriately quantified. However, a value other than entropy may be employed as the degree of deviation. For example, at least one of the standard deviation, coefficient of variation, variance, etc. indicating the degree of dispersion of the obtained probability distribution may be used as the degree of deviation. KL divergence, which is a measure of the difference between probability distributions, may be used as the degree of deviation. Furthermore, the maximum value of the obtained probability distribution may be used as the degree of deviation.

Note that in the present disclosure, a three-dimensional image is configured by arranging a plurality of two-dimensional images. The degree of deviation is acquired for each of the plurality of two-dimensional images that make up the three-dimensional image, or for each pixel, each column, or each row that makes up the image. As a result, the degree of deviation when a specific region is identified by the mathematical model is obtained for the entire fundus tissue shown in the three-dimensional image. As an example, the discrepancy map of the present disclosure shows a two-dimensional distribution of discrepancies when the fundus tissue is viewed from the front (that is, a direction along the optical axis of light for capturing a three-dimensional image). However, the direction in which the two-dimensional distribution of the deviation degree is shown in the deviation degree map can be changed as appropriate.

Furthermore, there are multiple layers in the fundus tissue. The degree of deviation obtained in the present disclosure refers to a plurality of specific layers and boundaries among a plurality of layers in the fundus tissue and boundaries between mutually adjacent layers (hereinafter sometimes simply referred to as "layers/boundaries"). This is the degree of deviation when each boundary is identified by the mathematical model. However, the degree of deviation is obtained when parts of the fundus tissue other than layers/boundaries (for example, two or more parts of the optic disc, macula, fovea, fundus blood vessels, and diseased parts) are identified. may be done. Further, when displaying a deviation degree map of a specific layer, the control unit displays a two-dimensional distribution such as an average value of deviation degrees of at least two boundaries included in the specific layer as a deviation degree map of the specific layer. You may.

In the selective display step, the control unit may selectively display maps of a predetermined number of deviation degrees that are ranked high among the plurality of obtained deviation degrees on the display unit. In this case, a predetermined number of discrepancy maps with high ranks of discrepancies are automatically displayed. Therefore, the user can easily check the deviation degree map of the part whose identification by the mathematical model has a high degree of deviation among the plurality of parts. Therefore, the efficiency of diagnosis is improved.

Note that a specific method for displaying a predetermined number of deviation degree maps having a high ranking of deviation degrees can be selected as appropriate. For example, the control unit may display a predetermined number of deviation degree maps in descending order of the average value of deviation degrees. Further, the control unit may display a predetermined number of deviation degree maps in descending order of the local maximum value of the deviation degree within the two-dimensional map. Further, the control unit may display one degree of deviation map having the highest rank of degree of divergence, or may display a plurality of degree of divergence maps in descending order of rank of degree of divergence. Further, the control unit may display a composite map obtained by aligning and synthesizing a plurality of discrepancy maps having high ranks of discrepancies.

The control unit may further execute a two-dimensional tomographic image display step and a region designation receiving step. In the two-dimensional tomographic image display step, the control unit extracts a part of the two-dimensional tomographic image from the three-dimensional image (that is, the three-dimensional image input to the mathematical model) and causes the display unit to display the extracted two-dimensional tomographic image. A two-dimensional tomographic image is a two-dimensional image that extends in the depth direction of the fundus tissue. In the region designation receiving step, the control unit receives an instruction input from the user for designating at least one of the regions shown in the displayed two-dimensional tomographic image. In the selective display step, when the instruction input for specifying a region is received, the control section selectively displays a map of the degree of deviation of the specified region among the plurality of regions on the display section. Good too.

In this case, the user can display the deviation degree map of the designated region by confirming the inside of the fundus tissue shown in the two-dimensional tomographic image and then specifying the region of interest. Therefore, the user can easily check the deviation degree map of an appropriate part from among the deviation degree maps of a plurality of parts based on the internal state of the fundus tissue shown in the two-dimensional tomographic image. For example, by specifying a region where an abnormality is observed in a two-dimensional tomographic image and checking the deviation degree map for the specified region, the user can easily understand the degree of diffusion and location of the abnormal region on the two-dimensional plane. . Note that the number of regions that can be specified by the user (that is, the regions on which the deviation degree map is displayed) may be one or more.

The control unit may further execute a corresponding region display step of displaying, on the two-dimensional tomographic image, the region for which the map of the degree of deviation is displayed (hereinafter also referred to as "corresponding region"). In this case, the user can easily and appropriately grasp the region on which the discrepancy map is displayed on the two-dimensional tomographic image that extends in the depth direction. Therefore, the efficiency of diagnosis is further improved.

Note that the method for displaying the corresponding region on the two-dimensional tomographic image can be selected as appropriate. For example, the control unit may indicate the corresponding region by changing the color of the corresponding region on the two-dimensional tomographic image relative to the color of other regions. The control unit may indicate the corresponding region by displaying a frame surrounding the corresponding region on the two-dimensional tomographic image. The control unit may indicate the corresponding region by displaying a cursor or the like pointing to the corresponding region.

The control unit may further execute an extraction position display step of displaying a position from which the two-dimensional tomographic image displayed on the display unit is extracted on a map of the degree of deviation. In this case, the user can diagnose the fundus tissue of the eye to be examined while checking the extraction position of the displayed two-dimensional tomographic image on the discrepancy map. Therefore, it becomes easier to perform a diagnosis more appropriately.

Note that the method for displaying the extraction position of the two-dimensional tomographic image on the discrepancy map can be selected as appropriate. For example, the control unit may display a line indicating the extraction position of the two-dimensional tomographic image on a two-dimensional discrepancy map when the fundus tissue is viewed from the front.

The control unit may further execute an extraction position designation receiving step and a designated tomographic image display step. In the extraction position designation reception step, the control unit receives an instruction input from the user for designating the extraction position of the two-dimensional tomographic image on the displayed map of the degree of deviation. In the specified tomographic image display step, when an instruction input for specifying an extraction position is received, the control unit extracts and displays a two-dimensional tomographic image extending in the depth direction of the fundus tissue from the specified extraction position. section. In this case, the user can check the two-dimensional distribution of identification discrepancies regarding a specific region using the discrepancy map, and then directly specify the position of the two-dimensional tomographic image to be displayed on the discrepancy map. . Therefore, it becomes easier to perform a diagnosis more appropriately.

Note that a method of having the user specify the extraction position of the two-dimensional tomographic image on the discrepancy map can also be selected as appropriate. For example, the control unit allows the user to specify a line on the two-dimensional discrepancy map that indicates the extraction position of the two-dimensional tomographic image (that is, the position where the extracted two-dimensional tomographic image and the discrepancy map intersect). Good too. In this case, the shape of the line may be a straight line or a shape other than a straight line (for example, curved or annular). The size of the line and the angle of the line may also be changed according to instructions input by the user.

A related site within the fundus tissue may be associated with each of a plurality of diseases that may occur in the subject. In the selective display step, the control unit may cause the display unit to selectively display a map of deviation degrees for a region associated with the disease specified by the user among the plurality of obtained deviation degrees. good. In this case, the user can efficiently utilize the disparity map for diagnosis depending on the disease. For example, a user can easily check a degree of deviation map for a region related to a predicted disease of a subject simply by specifying the predicted disease. Furthermore, the user can also predict the disease of the subject by checking the disparity map of the parts associated with each disease while switching the specified disease.

Note that a method for having the user specify at least one of a plurality of diseases can also be selected as appropriate. For example, the control unit may cause the user to specify at least one of a plurality of disease names prepared in advance, or may cause the user to input the disease name. Further, each of a plurality of analysis modes for performing disease-related analysis on a fundus image may be associated with a relevant site within the fundus tissue. The control unit may selectively display a deviation degree map for a region associated with the analysis mode to be executed among the plurality of analysis modes. In this case, a deviation map of a region related to the analysis mode (for example, a region to be analyzed) is appropriately displayed. Further, the control unit may cause the user to specify a disease while displaying the deviation degree map (or all the deviation maps) for at least one of the plurality of parts on the display unit. Alternatively, the user may be allowed to specify a disease while displaying a composite map obtained by aligning and synthesizing a plurality of discrepancy maps on the display unit. In this case, regardless of the type of disease, regions with a high degree of discrepancy can be efficiently identified.

The control unit may further execute a thickness map display step of causing the display unit to display a thickness map indicating a two-dimensional distribution of the thickness of a specific layer among the plurality of layers included in the fundus tissue shown in the three-dimensional image. . In the selection display step, the control unit selects at least the layers and boundaries within the range of the layer whose thickness is indicated by the thickness map among the plurality of layers included in the fundus tissue and the boundaries between mutually adjacent layers. A map of the degree of deviation for either one may be selectively displayed on the display unit. In this case, the user can easily confirm both the thickness map and the deviation degree map for layers/boundaries within the range whose thickness is indicated by the thickness map. Therefore, it becomes easier to perform a diagnosis more appropriately.

Note that the specific layer whose thickness is shown by the thickness map may be one layer or a plurality of layers. When the thickness of a specific plurality of layers is indicated by a thickness map, the plurality of layers whose thicknesses are indicated may be successively stacked. In this case, the total thickness of the plurality of successively stacked layers may be indicated by the thickness map.

Further, a specific method for selectively displaying the deviation map for layers/boundaries within the range where the thickness is indicated can be selected as appropriate. For example, a deviation map may be displayed when one specific layer whose thickness is indicated by the thickness map is identified. Further, at least one of a deviation degree map when the boundary on the surface layer side of the retina is identified in one specific layer and a degree of deviation map when the boundary on the deep retina side is identified may be displayed. A discrepancy map for the boundary on the superficial retina side and a discrepancy map for the boundary on the deep retina side in a specific layer may be displayed side by side, or a composite map in which the two discrepancy maps are combined may be displayed. Good too. Further, deviation degree maps for all layers/boundaries within the range whose thickness is indicated may be displayed. In this case as well, a plurality of deviation degree maps may be displayed separately, or a composite map may be displayed.

<Embodiment>
(Device configuration)
One typical embodiment of the present disclosure will be described below with reference to the drawings. As shown in FIG. 1, in this embodiment, a mathematical model construction device 101, a fundus image processing device 1, and OCT devices (fundus image capturing devices) 10A and 10B are used. The mathematical model construction device 101 constructs a mathematical model by training the mathematical model using a machine learning algorithm. Based on the input fundus image, the constructed mathematical model identifies (detects) a specific region (in this embodiment, multiple layers/boundaries in the fundus tissue) appearing in the fundus image. The fundus image processing device 1 executes various processes using the results output by the mathematical model. The OCT apparatuses 10A and 10B function as fundus image capturing apparatuses that capture fundus images of the eye to be examined (in this embodiment, tomographic images of the fundus).

As an example, a personal computer (hereinafter referred to as "PC") is used as the mathematical model construction device 101 of this embodiment. Although the details will be described later, the mathematical model construction device 101 uses the data of the fundus image of the subject's eye obtained from the OCT device 10A (hereinafter referred to as the "training fundus image") and the data of the subject's eye from which the training fundus image was taken. A mathematical model is trained using data indicating a plurality of parts (in this embodiment, a plurality of layers/boundaries). As a result, a mathematical model is constructed. However, a device that can function as the mathematical model construction device 101 is not limited to a PC. For example, the OCT device 10A may function as the mathematical model construction device 101. Moreover, the control units of a plurality of devices (for example, the CPU of the PC and the CPU 13A of the OCT apparatus 10A) may cooperate to construct the mathematical model.

Further, a PC is used as the fundus image processing apparatus 1 of this embodiment. However, the device that can function as the fundus image processing device 1 is not limited to a PC either. For example, the OCT device 10B, a server, or the like may function as the fundus image processing device 1. When the OCT device 10B functions as the fundus image processing device 1, the OCT device 10B can process the captured fundus image while capturing the fundus image. Further, a mobile terminal such as a tablet terminal or a smartphone may function as the fundus image processing device 1. The control units of multiple devices (for example, the CPU of the PC and the CPU 13B of the OCT apparatus 10B) may cooperate to perform various processes.

Further, in this embodiment, a case will be described in which a CPU is used as an example of a controller that performs various processes. However, it goes without saying that a controller other than the CPU may be used for at least some of the various devices. For example, a GPU may be used as the controller to speed up the processing.

The mathematical model construction device 101 will be explained. The mathematical model construction device 101 is placed, for example, at a manufacturer that provides the fundus image processing device 1 or the fundus image processing program to users. The mathematical model construction device 101 includes a control unit 102 that performs various control processes, and a communication I/F 105. The control unit 102 includes a CPU 103 which is a controller in charge of control, and a storage device 104 capable of storing programs, data, and the like. The storage device 104 stores a mathematical model building program for executing a mathematical model building process (see FIG. 6), which will be described later. Further, the communication I/F 105 connects the mathematical model construction device 101 to other devices (for example, the OCT device 10A, the fundus image processing device 1, etc.).

The mathematical model construction device 101 is connected to an operation unit 107 and a display device 108. The operation unit 107 is operated by the user in order for the user to input various instructions to the mathematical model construction device 101. For the operation unit 107, for example, at least one of a keyboard, a mouse, a touch panel, etc. can be used. Note that a microphone or the like for inputting various instructions may be used together with or in place of the operation unit 107. The display device 108 displays various images. As the display device 108, various devices capable of displaying images (for example, at least one of a monitor, a display, a projector, etc.) can be used. Note that "image" in the present disclosure includes both still images and moving images.

The mathematical model construction device 101 can acquire fundus image data (hereinafter sometimes simply referred to as “fundus image”) from the OCT device 10A. The mathematical model construction device 101 may acquire fundus image data from the OCT device 10A through at least one of wired communication, wireless communication, a removable storage medium (for example, a USB memory), and the like.

The fundus image processing device 1 will be explained. The fundus image processing device 1 is placed, for example, in a facility (for example, a hospital or a medical examination facility) that performs diagnosis or examination of subjects. The fundus image processing device 1 includes a control unit 2 that performs various control processes, and a communication I/F 5. The control unit 2 includes a CPU 3, which is a controller that controls, and a storage device 4 that can store programs, data, and the like. The storage device 4 stores a fundus image processing program for executing fundus image processing (see FIG. 7), which will be described later. The fundus image processing program includes a program that realizes the mathematical model constructed by the mathematical model construction device 101. The communication I/F 5 connects the fundus image processing device 1 with other devices (for example, the OCT device 10B, the mathematical model construction device 101, etc.).

The fundus image processing device 1 is connected to an operation section 7 and a display device (display section) 8. Various devices can be used for the operation section 7 and the display device 8, similar to the operation section 107 and the display device 108 described above.

The fundus image processing device 1 can acquire a fundus image (in this embodiment, a three-dimensional tomographic image of fundus tissue) from the OCT device 10B. The fundus image processing device 1 may acquire a fundus image from the OCT device 10B, for example, by wired communication, wireless communication, a removable storage medium (eg, USB memory), or the like. Further, the fundus image processing device 1 may acquire a program or the like that realizes the mathematical model constructed by the mathematical model construction device 101 via communication or the like.

The OCT apparatus 10 (10A, 10B) will be explained. The OCT device 10 is an example of a fundus image capturing device that captures tomographic images of the fundus. In this embodiment, a case will be described in which an OCT device 10A that provides a fundus image to the mathematical model construction device 101 and an OCT device 10B that provides a fundus image to the fundus image processing device 1 are used. However, the number of OCT devices used is not limited to two. For example, the mathematical model construction device 101 and the fundus image processing device 1 may acquire fundus images from a plurality of OCT devices. Furthermore, the mathematical model construction device 101 and the fundus image processing device 1 may acquire fundus images from one common OCT device.

The OCT apparatus 10 is capable of capturing two-dimensional tomographic images and three-dimensional tomographic images of the fundus of the eye to be examined. An example of a photographing method will be explained. As shown in FIG. 2, the OCT apparatus 10 of this embodiment uses a linear scanning line (scan line) that scans a spot within a two-dimensional measurement area 40 that extends in a direction intersecting the optical axis of OCT measurement light. A plurality of 41 are set at equal intervals. The OCT apparatus 10 can capture a two-dimensional tomographic image 42 (see FIG. 3) of a cross section intersecting each scanning line 41 by scanning a spot of measurement light on each scanning line 41. The two-dimensional tomographic image 42 may be an average image generated by performing averaging processing on a plurality of two-dimensional tomographic images of the same region. In addition, the OCT apparatus 10 arranges a plurality of two-dimensional tomographic images 42 taken for a plurality of scanning lines 41 in a direction perpendicular to the image area, so that a three-dimensional image (three-dimensional tomographic image) 43 ( (see FIG. 4) can be acquired (photographed).

Returning to the explanation of FIG. The OCT device 10A connected to the mathematical model construction device 101 is capable of capturing at least a two-dimensional tomographic image 42 (see FIG. 3) of the fundus of the eye to be examined. Further, the OCT device 10B connected to the fundus image processing device 1 can capture a three-dimensional image 43 (see FIG. 4) of the fundus of the eye to be examined in addition to the two-dimensional tomographic image 42 described above.

(Structure of layers and boundaries of the fundus)
With reference to FIG. 5, the structure of the layers in the fundus of the eye to be examined and the boundaries between mutually adjacent layers will be described. FIG. 5 schematically shows the structure of layers and boundaries in the fundus. The upper side of FIG. 5 is the surface side (superficial layer side) of the retina of the fundus. In other words, the depth of the layer/boundary becomes larger toward the bottom of FIG. 5 . Further, in FIG. 5, the names of boundaries between adjacent layers are shown in parentheses.

The layers of the fundus will be explained. In the fundus, from the surface side (upper side of Figure 5), the ILM (internal limiting membrane), NFL (nerve fiber layer), GCL (ganglion cell layer), IPL ( inner plexiform layer, INL (inner nuclear layer), OPL (outer plexiform layer), ONL (outer nuclear layer), E LM (external limiting membrane) membrane), IS/OS (junction between photoreceptor inner and outer segment), RPE (retinal pigment epithelium), BM (Bruch's membrane), Choroid ) exists.

In addition, boundaries that tend to appear on tomographic images include, for example, NFL/GCL (boundary between NFL and GCL), IPL/INL (boundary between IPL and INL), and OPL/ONL (boundary between OPL and ONL). , RPE/BM (boundary between RPE and BM), BM/Choroid (boundary between BM and Choroid), etc.

(mathematical model construction process)
With reference to FIG. 6, the mathematical model construction process executed by the mathematical model construction device 101 will be described. The mathematical model construction process is executed by the CPU 103 according to the mathematical model construction program stored in the storage device 104.

Below, as an example, we will construct a mathematical model that outputs the identification results of multiple specific layers/boundaries among multiple layers/boundaries shown in a fundus image by analyzing an input two-dimensional tomographic image. exemplify. In this embodiment, each of ILM, NFL/GCL, IPL/INL, OPL/ONL, IS/OS, RPE/BM, and BM is identified by the mathematical model as a plurality of specific layers/boundaries. However, in the mathematical model construction process, it is also possible to construct a mathematical model that outputs a result different from the layer/boundary identification result. For example, a mathematical model that outputs the identification results of two or more parts of the optic disc, macula, fovea, fundus blood vessels, diseased parts, etc. that appear in the input fundus image is constructed by a mathematical model construction process. It's okay.

Further, the mathematical model exemplified in this embodiment outputs a probability distribution for identifying each specific region (specific layer/boundary in this embodiment) appearing in the input fundus image.

In the mathematical model construction process, a mathematical model is constructed by training the mathematical model using a training data set. The training data set includes input side data (input training data) and output side data (output training data).

As shown in FIG. 6, the CPU 103 acquires data of a fundus image (a two-dimensional tomographic image in this embodiment) captured by the OCT apparatus 10A as input training data (S1). Next, the CPU 103 acquires, as output training data, data indicating each of a plurality of specific parts of the eye to be examined in which the fundus image acquired in S1 was photographed (S2). The output training data in this embodiment includes label data indicating the positions of a plurality of specific layers/boundaries shown in the fundus image. The label data may be generated, for example, by the operator operating the operating unit 107 while looking at the layers/boundaries in the fundus image.

Next, the CPU 103 executes training of a mathematical model using the training data set using a machine learning algorithm (S3). As machine learning algorithms, for example, neural networks, random forests, boosting, support vector machines (SVM), etc. are generally known.

Neural networks are a method that imitates the behavior of biological nerve cell networks. Neural networks include, for example, feedforward neural networks, RBF networks (radial basis functions), spiking neural networks, convolutional neural networks, recurrent neural networks (recurrent neural networks, feedback neural networks, etc.), and stochastic neural networks. There are various types of neural networks (Boltzmann machines, Bassian networks, etc.).

Random forest is a method of generating a large number of decision trees by performing learning based on randomly sampled training data. When using random forest, branches of multiple decision trees trained in advance as a classifier are followed, and the average (or majority vote) of the results obtained from each decision tree is taken.

Boosting is a method of generating a strong classifier by combining multiple weak classifiers. A strong classifier is constructed by sequentially training simple and weak classifiers.

SVM is a method of constructing a two-class pattern discriminator using linear input elements. For example, the SVM learns the parameters of a linear input element based on the criterion (hyperplane separation theorem) of finding a margin-maximizing hyperplane that maximizes the distance to each data point from training data.

The mathematical model is, for example, input data (in this embodiment, two-dimensional tomographic image data similar to input training data) and output data (in this embodiment, data of identification results of a plurality of specific parts). Refers to a data structure for predicting relationships. A mathematical model is constructed by being trained using a training dataset. As described above, the training data set is a set of input training data and output training data. For example, training updates correlation data (eg, weights) between each input and output.

In this embodiment, a multilayer neural network is used as a machine learning algorithm. A neural network includes an input layer for inputting data, an output layer for generating data of an analysis result desired to be predicted, and one or more hidden layers between the input layer and the output layer. A plurality of nodes (also called units) are arranged in each layer. Specifically, in this embodiment, a convolutional neural network (CNN), which is a type of multilayer neural network, is used.

As an example, the mathematical model constructed in this embodiment is based on a specific area within a fundus image (one-dimensional area, two-dimensional area, three-dimensional area, or four-dimensional area including the time axis) Probability using the coordinates (one-dimensional coordinates, two-dimensional coordinates, three-dimensional coordinates, or four-dimensional coordinates) where each of a plurality of parts (in this embodiment, a plurality of specific layers/boundaries) exists as a random variable The distribution is output as a probability distribution for identifying each part. In this embodiment, a softmax function is applied to cause the mathematical model to output a probability distribution. In detail, the mathematical model constructed in S3 is constructed in the direction intersecting a specific layer/boundary in the two-dimensional tomographic image (in this embodiment, the depth direction is the direction along the optical axis of the OCT measurement light). A probability distribution is output in which the coordinates of a specific layer/boundary exist as random variables in a one-dimensional region extending in the vertical direction (in FIG. 3, vertical direction).

However, the specific method by which the mathematical model outputs a probability distribution for identifying a specific region can be changed as appropriate. For example, in a two-dimensional or three-dimensional region, a mathematical model is based on a probability distribution using two-dimensional coordinates or three-dimensional coordinates where a specific region (for example, a characteristic region, etc.) exists as a random variable, and a probability distribution for identifying the region. It may also be output as a distribution. In addition, the mathematical model outputs a probability distribution for each region (for example, for each pixel) of the input ophthalmological image, with the types of multiple parts (for example, multiple layers and boundaries) in the eye to be examined as random variables. Good too. Further, the ophthalmological image input to the mathematical model may be a moving image.

Note that other machine learning algorithms may be used. For example, generative adversarial networks (GAN), which utilize two competing neural networks, may be employed as the machine learning algorithm.

The processes of S1 to S3 are repeated until construction of the mathematical model is completed (S4: NO). When construction of the mathematical model is completed (S4: YES), the mathematical model construction process ends. In this embodiment, a program and data for realizing the constructed mathematical model are incorporated into the fundus image processing device 1.

(fundus image processing)
Referring to FIGS. 7 to 12, fundus image processing performed by the fundus image processing device 1 will be described. Fundus image processing is executed by the CPU 3 of the fundus image processing device 1 according to a fundus image processing program stored in the storage device 4.

First, the CPU 3 acquires a three-dimensional image (three-dimensional tomographic image) of the fundus of the eye to be examined (S11). The three-dimensional image is photographed by the OCT device 10B and acquired by the fundus image processing device 1. As described above, a three-dimensional image is constructed by combining a plurality of two-dimensional images (two-dimensional tomographic images) taken by scanning measurement light on different scan lines. Note that the CPU 3 may acquire a signal (for example, an OCT signal) that is a basis for generating a three-dimensional image from the OCT device 10B, and generate a three-dimensional image based on the acquired signal.

The CPU 3 executes a deviation degree acquisition process (S12). In the deviation degree acquisition process, the deviation degree of the three-dimensional image acquired in S11 is acquired. Discrepancy is the difference between the probability distribution for a mathematical model to identify each of multiple parts of the fundus tissue in a three-dimensional image, and the probability distribution when each of the multiple parts is accurately identified by the mathematical model. degree. The degree of deviation often increases or decreases depending on the degree of abnormality of the structure. Therefore, when the user understands the degree of deviation, it becomes easier to improve the efficiency of diagnosis.

As shown in FIG. 8, the CPU 3 extracts the T-th (initial value of T is "1") two-dimensional tomographic image from among the plurality of two-dimensional tomographic images forming the three-dimensional image acquired in S11 ( S36). FIG. 9 shows an example of the extracted two-dimensional tomographic image 42. The two-dimensional tomographic image 42 shows a plurality of layers and boundaries in the fundus of the subject's eye. Furthermore, a plurality of one-dimensional areas A1 to AN are set in the two-dimensional tomographic image 42. In this embodiment, the one-dimensional regions A1 to AN set in the two-dimensional tomographic image 42 extend along an axis that intersects a specific layer/boundary. Specifically, the one-dimensional regions A1 to AN of the present embodiment correspond to regions of each of a plurality of (N) A-scans that constitute the two-dimensional tomographic image 42 captured by the OCT apparatus 10.

By inputting the T-th two-dimensional tomographic image into the mathematical model, the CPU 3 calculates the M-th (initial value of M is "1") region (in this embodiment, the initial value of M is "1") in each of the plurality of one-dimensional regions A1 to AN. The probability distribution of the coordinates where the Mth layer/boundary exists is acquired as the probability distribution for identifying the Mth region (S37). The CPU 3 obtains the degree of deviation of the probability distribution regarding the Mth part (S38). The degree of deviation obtained in S38 is the difference between the probability distribution obtained in S37 and the probability distribution when the Mth region is correctly identified.

In this embodiment, the entropy of the probability distribution P is calculated as the degree of deviation. Entropy is given by the following (Equation 1). Entropy H(P) takes a value of 0≦H(P)≦log (number of events), and the value becomes smaller as the probability distribution P is biased. In other words, the smaller the entropy H(P), the higher the identification accuracy of the M-th region tends to be.
H(P)=-Σplog(p)...(Math. 1)

Next, the CPU 3 determines whether the degree of deviation of all parts (layers/boundaries) to be identified in the T-th two-dimensional tomographic image has been acquired (S40). If the degree of deviation for some parts has not yet been obtained (S40: NO), "1" is added to the order M of parts to be identified (S41), and the process returns to S37 to obtain the next part. The degree of deviation is acquired (S37, S38). When the degrees of deviation of all parts are acquired (S40: YES), the CPU 3 causes the storage device 4 to store the plurality of degrees of deviation acquired for each of the plurality of parts reflected in the T-th two-dimensional tomographic image. (S42).

Next, the CPU 3 determines whether the degrees of deviation of all the two-dimensional tomographic images constituting the three-dimensional tomographic image have been acquired (S44). If the degree of deviation of some two-dimensional tomographic images has not yet been acquired (S44: NO), "1" is added to the order T of the two-dimensional tomographic images (S45), and the process returns to S36 to obtain the next The degree of deviation of the two-dimensional tomographic images is acquired (S36 to S42).

When the degree of deviation of all two-dimensional tomographic images is acquired (S44: YES), the CPU 3 acquires a degree of deviation map for each of the plurality of parts to be identified (S47). Thereafter, the process returns to fundus image processing (see FIG. 7). The discrepancy map shows a two-dimensional distribution of the discrepancy when the fundus is viewed from the front (that is, the direction along the optical axis of the OCT measurement light). As described above, in this embodiment, a deviation degree map is obtained for each of ILM, NFL/GCL, IPL/INL, OPL/ONL, IS/OS, RPE/BM, and BM.

FIG. 10 shows an example of a deviation degree map for some parts. In the deviation degree map shown in FIG. 10, parts with large deviation degrees are expressed in bright colors. The deviation degree map shown in FIG. 10 is obtained from a three-dimensional image near the optic disc (hereinafter simply referred to as "papilla"). Where the papilla is present, the ILM and NFL are present, while layers and boundaries deeper than the NFL are absent. Therefore, at a position where a papilla is present, the degree of deviation regarding the identification of layers and boundaries deeper than the NFL is greater than at a position where a papilla is not present. In the example shown in FIG. 10, in the ILM deviation map, the deviation is small overall. Therefore, it can be seen that ILMs are identified with high probability in the entire two-dimensional region. On the other hand, in the IPL/INL and BM deviation maps, the deviation is large in the two-dimensional region near the center where the nipple is present.

Returning to the explanation of FIG. 7. The CPU 3 selectively selects a predetermined number of deviation degree maps having a high deviation degree rank (that is, a deviation degree size ranking) among a plurality of deviation degree maps for each of the plurality of parts to be identified. It is displayed on the display device 8 as a default (S13). Therefore, the user can easily check the deviation degree map of the part whose identification based on the mathematical model has a high degree of deviation among the plurality of parts to be identified. In addition, in this embodiment, the CPU 3 displays a predetermined number of deviation degree maps in descending order of the average value of the deviation degrees in the two-dimensional deviation degree maps. However, the method for determining the rank of deviation degree is not limited to the method using the average value. For example, the CPU 3 may display a predetermined number of deviation degree maps in descending order of the local maximum value of the deviation degree within the two-dimensional deviation degree map. Further, the CPU 3 may display one deviation degree map with the highest deviation degree ranking, or may display a plurality of deviation degree maps in descending order of deviation degree ranking. Further, the CPU 3 may display a composite map obtained by aligning and synthesizing a plurality of discrepancy maps having high ranks of discrepancies.

The CPU 3 extracts a part of the two-dimensional tomographic image extending in the depth direction of the fundus from the default position of the three-dimensional image acquired in S11, and displays it on the display device 8 (S14). FIG. 11 shows an example in which a two-dimensional tomographic image extracted from a three-dimensional image is displayed on the display device 8 together with the selected degree of deviation map. In this embodiment, as shown in FIG. 11, the imaging range of the three-dimensional image acquired in S11 is a two-dimensional fundus front view when viewed from the front (that is, a direction along the optical axis of the OCT measurement light). The image is also displayed together with the discrepancy map and the two-dimensional tomographic image. In this case, the fundus front image may be an image generated based on a three-dimensional image (so-called Enface image), or may be an image generated based on a three-dimensional image (a so-called Enface image), or a device different from the device that captured the three-dimensional image (for example, an SLO device or a fundus camera, etc.). ) may be an image taken by

Note that the default position for extracting a two-dimensional tomographic image from a three-dimensional image can be set as appropriate. For example, on a two-dimensional frontal image of a three-dimensional image viewed from the front, a line-shaped position crossing the center position left and right may be the default position from which the two-dimensional tomographic image is extracted in S14. In addition, characteristic parts (for example, papilla or macula) in the fundus of the eye are identified using image processing, etc., and the line-shaped position passing through the identified part is set as the default position for extracting the two-dimensional tomographic image in S14. good. Note that the line is not limited to a straight line.

The CPU 3 displays, on the two-dimensional tomographic image, the part for which the deviation degree map is displayed (sometimes referred to as "corresponding part") among the plurality of parts targeted for identification by the mathematical model (S15). Therefore, the user can easily and appropriately grasp the region on which the discrepancy map is displayed ("region A" in FIG. 11) on the two-dimensional tomographic image that spreads in the depth direction. Now, as shown in Fig. 11, by changing the color of region A (layer/boundary) where the discrepancy map is displayed among the multiple layers/boundaries shown in the two-dimensional tomographic image, region A is displayed. has been done.

The CPU 3 displays the position from which the two-dimensional tomographic image displayed on the display device 8 is extracted on the deviation degree map (S16). Therefore, the user can diagnose the fundus tissue while checking the extraction position of the displayed two-dimensional tomographic image on the discrepancy map. In this embodiment, as shown in FIG. 11, a line P indicating the extraction position of the two-dimensional tomographic image is displayed on the two-dimensional degree of deviation map when the fundus tissue is viewed from the front. Furthermore, in the present embodiment, a line P indicating the extraction position of the two-dimensional tomographic image is similarly displayed on the two-dimensional fundus frontal image.

Next, the CPU 3 determines whether or not the user has input an instruction to specify at least one of the regions shown in the two-dimensional tomographic image displayed on the display device 8 for displaying the discrepancy map. A judgment is made (S17). As an example, in the present embodiment, the user selects the region on which the discrepancy map is to be displayed by operating the operation unit 7 (for example, moving and clicking a cursor, specifying a position using a touch panel, etc.). specified on the two-dimensional image. The number of parts designated by the user may be one or more. If part A is not specified (S17: NO), the process directly advances to S21.

When the region on which the deviation degree map is to be displayed is specified (S17: YES), the CPU 3 selects the region A (corresponding part) on a two-dimensional tomographic image (S18). Therefore, the user can easily and appropriately grasp the region A where the discrepancy map is displayed on the two-dimensional tomographic image that extends in the depth direction. In the example shown in FIG. 11, IPL/INL is specified on the two-dimensional image as the region A on which the deviation degree map is displayed. In the present embodiment, the CPU 3 indicates the designated region by changing the color of IPL/INL, which is the designated region (corresponding region), on the two-dimensional image with respect to the color of other regions.

Further, the CPU 3 selectively causes the display device 8 to display a discrepancy map of a region (corresponding region) specified by the user among the plurality of regions to be identified by the mathematical model (S19). Therefore, the user can easily check the deviation degree map of an appropriate part from among the deviation degree maps of a plurality of parts based on the internal state of the fundus tissue shown in the two-dimensional tomographic image. After that, the process moves to S21. In the example shown in FIG. 11, the CPU 3 causes the display device 8 to display the IPL/INL deviation degree map specified by the user. Note that the CPU 3 causes the position where the two-dimensional tomographic image is extracted to be displayed on the IPL/INL deviation degree map displayed on the display device 8.

Next, the CPU 3 determines whether the user has input an instruction to specify a position P from which a two-dimensional tomographic image is to be extracted on the discrepancy map displayed on the display device 8 (S21). If the position P has not been designated (S21: NO), the process moves directly to S25. As an example, in the present embodiment, the user displays on the display device 8 a position P from which a two-dimensional tomographic image is to be extracted by operating the operation unit 7 (for example, moving and clicking a cursor, specifying a position using a touch panel, etc.). specified on the deviation map. In addition, in this embodiment, the CPU 3 can also similarly receive an instruction for specifying the position P from which a two-dimensional tomographic image is extracted on the two-dimensional fundus front image displayed on the display device 8. be. Furthermore, in the example shown in FIG. 11, the position P from which the two-dimensional tomographic image is extracted is specified by specifying the position of the straight line. However, it is also possible to change the method for specifying the extraction position P. For example, the line used to designate the position P is not limited to a straight line, but may be annular, curved, or the like.

When the position P from which a two-dimensional tomographic image is extracted is designated on the discrepancy map (S21: YES), the CPU 3 displays the designated position P on the discrepancy map (S22). Therefore, the user can diagnose the fundus tissue of the eye to be examined while checking the extraction position P of the displayed two-dimensional tomographic image on the discrepancy map. As described above, in the example shown in FIG. 11, a line P indicating the extraction position of the two-dimensional tomographic image is displayed on the two-dimensional discrepancy map when the fundus tissue is viewed from the front. Furthermore, in the present embodiment, a line P indicating the extraction position of the two-dimensional tomographic image is similarly displayed on the two-dimensional fundus frontal image. Note that even when the extraction position P of the two-dimensional tomographic image is specified on the fundus frontal image, a line P indicating the extraction position of the two-dimensional tomographic image is similarly displayed on the discrepancy map and the fundus frontal image. .

Furthermore, the CPU 3 converts a two-dimensional tomographic image that spreads in the depth direction of the fundus tissue from the specified position P (that is, a two-dimensional tomographic image that passes through the specified position P and spreads in the depth direction) into a three-dimensional image. , and display it on the display device 8 (S23). Therefore, the user confirms the two-dimensional distribution of the identification discrepancy regarding a specific region (in this embodiment, a specific layer/boundary) using the discrepancy map, and then determines the position of the two-dimensional tomographic image that he/she wishes to display. It can be specified directly on the deviation map. After that, the process moves to S25.

Next, the CPU 3 determines whether the user has input an instruction specifying any one of the plurality of diseases (S25). If no disease has been designated (S25: NO), the process moves directly to S28. In this embodiment, by specifying any disease, the user can display a disparity map for the region associated with the specified disease. Specifically, each of the plurality of diseases that may occur in the subject is associated with at least one of the related sites in the fundus tissue. For example, in this embodiment, ERM (preretinal membrane) is a site on the surface layer of the retina, diabetic retinopathy is a site on the inner retinal layer, and AMD (age-related macular degeneration) and CNV (choroidal neovascularization) is a site on the outer retinal layer. are associated. When a disease is specified by the user (S25: YES), the CPU 3 selects a deviation degree map for a region associated with the specified disease from among the plurality of regions targeted for identification by the mathematical model. displayed on the display device 8 (S26). Therefore, the user can easily check the degree of deviation map for the region related to the predicted disease of the subject simply by specifying the predicted disease. Furthermore, the user can also predict the disease of the subject by checking the disparity map of the parts associated with each disease while switching the specified disease. Note that in S26 as well, the region on which the deviation degree map is displayed may be displayed on the two-dimensional tomographic image.

Next, the CPU 3 determines whether an instruction to display the thickness map has been input (S28). The thickness map shows a two-dimensional distribution of the thickness of a specific layer among multiple layers included in the fundus tissue. The thickness map illustrated in FIG. 12 is a thickness map of the layers from ILM to NFL/GCL among the multiple layers included in the fundus tissue. However, it is also possible to display thickness maps of other layers. It is also possible to have the user specify the layer for which the thickness map is to be displayed.

When an instruction to display the thickness map is input (S28: YES), the CPU 3 causes the display device 8 to display the thickness map of the specific layer (S29). Furthermore, the CPU 3 causes the display device 8 to display a deviation degree map for the layers and boundaries within the range where the thickness map is shown among the layers and boundaries captured in the three-dimensional image acquired in S11 (S30). . Therefore, the user can easily confirm both the thickness map and the deviation degree map for layers/boundaries within the range whose thickness is indicated by the thickness map. In the example shown in FIG. 12, a thickness map of layers from ILM to NFL/GCL is displayed. Also displayed is a deviation degree map between ILM and NFL/GCL that exists within the range from ILM to NFL/GCL. Note that in S230 as well, the region on which the deviation degree map is displayed may be displayed on the two-dimensional tomographic image.

The CPU 3 determines whether a termination instruction has been input (S32). If it has not been input (S32: NO), the process returns to S17, and the processes from S17 to S32 are repeated. When the end instruction is input (S32: YES), the fundus image processing ends.

The techniques disclosed in the above embodiments are merely examples. Therefore, it is also possible to modify the techniques exemplified in the above embodiments. First, it is also possible to execute only some of the techniques exemplified in the above embodiments. For example, it is also possible to omit some of the processes of S17, S21, S25, and S28.

Note that the process of acquiring a three-dimensional image in S11 of FIG. 7 is an example of an "image acquisition step." The process of acquiring the degree of deviation in S12 of FIG. 7 (FIG. 8) is an example of a "degree of deviation acquisition step." The process of selectively displaying some of the deviation degree maps in S13, S19, S26, and S30 in FIG. 7 is an example of a "selective display step." The process of extracting and displaying a two-dimensional tomographic image in S14 and S23 of FIG. 7 is an example of a "two-dimensional tomographic image display step." The process of accepting the input of the region A designation instruction in S17 of FIG. 7 is an example of the "region designation reception step." The process of displaying the corresponding region on the two-dimensional tomographic image in S15 and S18 in FIG. 7 is an example of the "corresponding region display step." The process of displaying the extraction position of the two-dimensional tomographic image on the discrepancy map in S16 and S22 of FIG. 7 is an example of the "extraction position display step." The process of receiving an instruction input for the extraction position of a two-dimensional tomographic image in S21 of FIG. 7 is an example of the "extraction position designation reception step." The process of extracting and displaying a two-dimensional tomographic image in S23 of FIG. 7 is an example of a "designated tomographic image display step." The process of displaying the thickness map in S29 of FIG. 7 is an example of a "thickness map display step."

1 Fundus image processing device 3 CPU
4 Storage device 8 Display devices 10A, 10B OCT device 42 Two-dimensional tomographic image 43 Three-dimensional image

Claims (9)

A fundus image processing device that processes a fundus image of an eye to be examined,
The control unit of the fundus image processing device includes:
an image acquisition step of acquiring a three-dimensional image of the fundus taken by a fundus image capturing device;
By inputting the three-dimensional image into a mathematical model trained by a machine learning algorithm, a plurality of probability distributions for identifying each of the plurality of parts of the fundus tissue shown in the three-dimensional image are obtained, and a deviation degree obtaining step of obtaining a deviation degree of the obtained probability distribution with respect to a probability distribution when each part is accurately identified;
a selective display step of selectively displaying a map showing a two-dimensional distribution of some of the obtained deviation degrees on a display unit;
A fundus image processing device that performs the following. The fundus image processing device according to claim 1,
The fundus image processing is characterized in that, in the selective display step, a predetermined number of maps of the deviation degrees that are ranked high among the plurality of acquired deviation degrees are selectively displayed on the display unit. Device. The fundus image processing device according to claim 1 or 2,
a two-dimensional tomographic image display step of extracting a part of the two-dimensional tomographic image extending in the depth direction of the fundus tissue from the three-dimensional image and displaying it on a display unit;
a region designation receiving step of receiving an instruction input from a user to designate at least one of the regions shown in the displayed two-dimensional tomographic image;
further run
In the selective display step, when an instruction input for specifying a region is received, the map of the degree of deviation of the specified region among the plurality of regions is selectively displayed on the display unit. Characteristic fundus image processing device. The fundus image processing device according to claim 3,
The fundus oculi image processing device further comprises: displaying, on the two-dimensional tomographic image, a corresponding region displaying a region on which the map of the degree of deviation is displayed among the plurality of regions. The fundus image processing device according to claim 3 or 4,
A fundus oculi image processing apparatus, further comprising: displaying an extraction position display step of displaying a position from which the two-dimensional tomographic image displayed on the display unit is extracted on a map of the degree of deviation. The fundus image processing device according to any one of claims 1 to 5,
an extraction position designation reception step of accepting an instruction input from a user to designate an extraction position of the two-dimensional tomographic image on the displayed map of the degree of deviation;
When an instruction input for designating an extraction position is accepted, a designated tomographic image display step of extracting a two-dimensional tomographic image extending in the depth direction of the fundus tissue from the designated extraction position and displaying it on a display unit. and,
A fundus image processing device further comprising: The fundus image processing device according to any one of claims 1 to 6,
For each of the multiple diseases that may occur in the subject, a relevant site within the fundus tissue is associated.
The selective display step is characterized by selectively displaying on the display unit a map of the deviation degrees for a region associated with the disease specified by the user among the plurality of obtained deviation degrees. Fundus image processing device. The fundus image processing device according to any one of claims 1 to 7,
further performing a thickness map display step of displaying on a display unit a thickness map showing a two-dimensional distribution of thickness of a specific layer among a plurality of layers included in the fundus tissue shown in the three-dimensional image;
In the selection display step, at least one of a plurality of layers included in the fundus tissue and boundaries between mutually adjacent layers is within a range whose thickness is indicated by the thickness map. A fundus image processing device characterized in that the map of the degree of deviation of is selectively displayed on a display unit. A fundus image processing program executed by a fundus image processing device that processes a fundus image of an eye to be examined,
The fundus image processing program is executed by the control unit of the fundus image processing device,
an image acquisition step of acquiring a three-dimensional image of the fundus taken by a fundus image capturing device;
By inputting the three-dimensional image into a mathematical model trained by a machine learning algorithm, a plurality of probability distributions for identifying each of the plurality of parts of the fundus tissue shown in the three-dimensional image are obtained, and a deviation degree obtaining step of obtaining a deviation degree of the obtained probability distribution with respect to a probability distribution when each part is accurately identified;
a selective display step of selectively displaying a map showing a two-dimensional distribution of some of the obtained deviation degrees on a display unit;
A fundus image processing program that causes the fundus image processing device to execute.

Images