JP2023149095A - Ocular fundus image processing device and ocular fundus image processing program - Google Patents

Abstract

To provide an ocular fundus image processing device and an ocular fundus image processing program capable of appropriately acquiring information indicating a detailed running state of ocular fundus blood vessels.SOLUTION: A control part of an ocular fundus image processing device executes an ocular fundus image acquisition step and a detection result acquisition step. In the ocular fundus acquisition step, the control part acquires an ocular fundus image 30 including blood vessels of the ocular fundus of an eye to be examined captured by an ocular fundus image capturing device. In the detection result acquisition step, the control part acquires a detection result of the blood vessels present in the ocular fundus image 30, and a detection result of specific positions including a branch position and an end position of the blood vessels present in the ocular fundus image by inputting at least part of the ocular fundus image 30 to a mathematical model 20 that is trained by a machine learning algorithm.SELECTED DRAWING: Figure 2

Description

The present disclosure relates to a fundus image processing device that processes a fundus image of a subject's eye, and a fundus image processing program that is executed in the fundus image processing device.

In recent years, attempts have been made to obtain information on blood vessels in the fundus of the eye by processing fundus images of the eye to be examined. For example, the fundus image processing device described in Patent Document 1 inputs the fundus image into a mathematical model trained by a machine learning algorithm, and detects arteries and veins present in the fundus image (arteries and veins are classified). Acquire blood vessel images).

Japanese Patent Application Publication No. 2019-208851

In addition to the detection results of arteries and veins, it would be extremely useful for disease diagnosis and research if information indicating the running state of each of a plurality of blood vessels in more detail could be obtained. For example, it is conceivable to analyze the detailed running state of each blood vessel based on a blood vessel image (an image in which blood vessels are extracted from a fundus image) obtained using a machine learning algorithm or the like. However, after various trials and studies by the inventor, we found that there are cases where blood vessels that are actually connected are disconnected in blood vessel images, and it is difficult to determine the detailed running state of blood vessels based only on blood vessel images. It was difficult to analyze.

A typical object of the present disclosure is to provide a fundus image processing device and a fundus image processing program that can appropriately acquire information indicating detailed running states of fundus blood vessels.

A fundus image processing device provided by a typical embodiment of the present disclosure is a fundus image processing device that processes a fundus image of an eye to be examined, and a control unit of the fundus image processing device is configured to control a fundus image processing device that processes a fundus image of an eye to be examined. In addition, a fundus image acquisition step of acquiring the fundus image including blood vessels in the fundus of the eye to be examined, and inputting at least a part of the fundus image to a mathematical model trained by a machine learning algorithm, and a detection result acquisition step of acquiring a detection result of a blood vessel and a detection result of a specific position including a branch position and an end position of the blood vessel existing in the fundus image.

A fundus image processing program provided by a typical embodiment of the present disclosure is a fundus image processing program executed by a fundus image processing device that processes a fundus image of an eye to be examined, wherein the fundus image processing program a fundus image acquisition step executed by a control unit of the processing device to obtain a fundus image including blood vessels in the fundus of the eye to be examined, captured by a fundus image capturing device; and a mathematical model trained by a machine learning algorithm. By inputting at least a part of the fundus image into the fundus image, a detection result of blood vessels present in the fundus image and a detection result of a specific position including a branch position and an end position of the blood vessel present in the fundus image are obtained. The fundus oculi image processing device is caused to perform a detection result acquisition step.

According to the fundus image processing device and the fundus image processing program according to the present disclosure, information indicating detailed running states of fundus blood vessels is appropriately acquired.

1 is a block diagram showing a schematic configuration of a mathematical model construction device 101, a fundus image processing device 1, and fundus image photographing devices 10A and 10B. FIG. 2 is an explanatory diagram for explaining a mathematical model 20 used in the first embodiment. 2 is a flowchart of mathematical model construction processing executed by the mathematical model construction device 101. 2 is a flowchart of fundus image processing executed by the fundus image processing device 1. FIG. FIG. 2 is a diagram schematically showing an example of a complete graph. 3 is a diagram schematically showing an example of a minimum spanning tree of a subgraph G. FIG. FIG. 4 is an explanatory diagram for explaining a method for calculating the weights of sides of a branch position 52S and a branch position 52T on a blood vessel in a venous blood vessel image 41V. 6 is a diagram showing an example of blood vessel topology information 60. FIG. FIG. 3 is an explanatory diagram for explaining mathematical models 21, 22, and 23 used in the second embodiment.

<Summary>
The control unit of the fundus image processing device exemplified in this disclosure executes a fundus image acquisition step and a detection result acquisition step. In the fundus image acquisition step, the control unit obtains a fundus image including blood vessels in the fundus of the subject's eye, which is captured by the fundus image photographing device. In the detection result acquisition step, the control unit inputs at least a portion of the fundus image into a mathematical model trained by a machine learning algorithm, thereby obtaining detection results of blood vessels present in the fundus image and blood vessels present in the fundus image. Obtain detection results for specific positions including branch positions and end positions.

A blood vessel image (an image in which fundus blood vessels are extracted), which is an example of a detection result of blood vessels in the fundus, is useful for understanding the condition of the fundus. However, in a blood vessel image, there are cases where a part of the fundus blood vessels that are actually connected is not detected, and the blood vessels are detected in a state where they are interrupted. Therefore, for example, it is often difficult to determine whether the edge (cut) of a blood vessel shown in a blood vessel image is the actual end of a blood vessel or the boundary between a detected blood vessel and an undetected blood vessel. . Therefore, it is difficult to analyze the detailed running state of blood vessels using only blood vessel images. In contrast, according to the fundus image processing device according to the present disclosure, the photographed fundus image, not the blood vessel image, is input into the mathematical model, so that in addition to the blood vessel detection results, the branch position and end of the blood vessel are A detection result of a specific position including the position is obtained. Obtaining detection results for specific positions including branch positions and end positions of blood vessels makes it easier to understand the detailed running state of blood vessels, which is difficult to analyze using only blood vessel images. That is, according to the technology of the present disclosure, information indicating the detailed running state of the fundus blood vessels is appropriately acquired.

Note that the fundus image input into the mathematical model to obtain detection results for a specific position is not a blood vessel image in which some blood vessels may not be detected, but a fundus image taken by a fundus image capturing device. It is. For example, a fundus image before a blood vessel image is detected may be input into a mathematical model to obtain a detection result at a specific position. Further, the same fundus image that is input to the mathematical model to obtain the blood vessel detection result may be input to the mathematical model to obtain the detection result of a specific position.

Note that the blood vessel detection result obtained using the mathematical model may be a blood vessel image in which fundus blood vessels within the tissue reflected in the fundus image are extracted. In this case, the distribution of fundus blood vessels can be appropriately understood by the blood vessel image.

As the fundus image input to the mathematical model, images captured by various fundus image capturing devices can be used. For example, at least one of a fundus image taken by a fundus camera, a fundus image taken by a laser scanning ophthalmoscope (SLO), a fundus image acquired by an OCT device, etc. may be input into the mathematical model.

The blood vessel detection results acquired by the mathematical model in the detection result acquisition step may be the detection results of each of the arteries and veins (hereinafter sometimes referred to as "arteriovenous") present in the fundus image. When detection results of arteries and veins are obtained, the running state of fundus blood vessels can be grasped more appropriately than when detection results of blood vessels whose arteries and veins are not classified are obtained. In other words, by using the classification results of arteries and veins, the running states of arteries and veins can be grasped separately.

Note that the mathematical model may be trained to classify each pixel of the input fundus image into one of three types: arteries, veins, and other tissues. In this case, compared to the case where the detected blood vessels are classified into arteries and veins after the blood vessels are detected, arteries and veins can be detected appropriately with simpler processing.

Further, the detection result of the specific position of the blood vessel acquired by the mathematical model in the detection result acquisition step may be a result of classifying the specific position in the artery and the specific position in the vein. In this case, as described above, since the running states of the arteries and veins are grasped separately, the running states of the blood vessels can be grasped more appropriately.

However, even if the specific position in the artery and the specific position in the vein are not classified, the control unit can compare the detected specific position with the arteriovenous detection result, so that the detected specific position is By determining which specific position is located, it is also possible to grasp the running state of each artery and vein separately.

In the detection result acquisition step, the control unit may acquire, as the specific position, the detection result of the intersection position of the blood vessel when the fundus of the eye is viewed from the front, as well as the branch position and end position of the blood vessel. In this case, the running state of each of the plurality of blood vessels can be more accurately understood.

The mathematical model may be trained using a training data set that includes input data (input training data) and output data (output training data). The input training data may be data of a fundus image of the subject's eye taken in the past. The output training data may be data indicating a blood vessel in the fundus image of the input training data, and data indicating a specific position in the fundus image of the input training data. In the detection result acquisition step, one fundus image may be input to the mathematical model to obtain both blood vessel detection results and specific position detection results.

In this case, since the mathematical model is made to learn a plurality of related tasks at the same time, common factors occur in the plurality of tasks, and the accuracy of each task is likely to be improved. In other words, since the mathematical model can use common information for both blood vessel detection and specific position detection, the accuracy of each detection is likely to be improved. Therefore, detection accuracy can be improved more easily than when different mathematical models are used for blood vessel detection and specific position detection.

As mathematical models, a blood vessel detection model and a specific position detection model may be constructed separately. The blood vessel detection model is trained by using a fundus image of the subject's eye photographed in the past as input training data, and using data indicating blood vessels in the fundus image of the input training data as output training data. The specific position detection model is trained using a fundus image of the subject's eye photographed in the past as input training data, and using data indicating a specific position in the fundus image of the input training data as output training data. In the detection result acquisition step, the control unit acquires blood vessel detection results by inputting the fundus image into the blood vessel detection model, and acquires detection results at a specific position by inputting the fundus image into the specific position detection model. It's okay.

In this case, it becomes easy to adjust the parameters of each of the two mathematical models. In addition, since the blood vessel detection model and the specific position detection model are constructed separately, the number of man-hours required to train the mathematical model is reduced compared to building a mathematical model that outputs both blood vessel detection results and specific position detection results. It is easy to reduce In other words, when training a mathematical model, unlike the case where both data indicating blood vessels in the fundus image and data indicating a specific position are prepared as input training data, one of the data indicating blood vessels and the data indicating a specific position is prepared. Once prepared, a blood vessel detection model or a specific position detection model is constructed. Therefore, depending on various situations, it is also possible to construct either the blood vessel detection model or the specific position detection model first. Further, if either the blood vessel detection model or the specific position detection model has already been constructed, it is only necessary to newly construct the other mathematical model.

In the detection result acquisition step, the control unit inputs at least a portion of the fundus image into a mathematical model trained by a machine learning algorithm, thereby determining the optic nerves present in the fundus image together with blood vessel detection results and specific position detection results. A detection result of a nipple (hereinafter sometimes simply referred to as "nipple") may be obtained. There are various relationships between the fundus blood vessels and the papilla. For example, each of the plurality of fundus blood vessels spreads from the papilla to the periphery, so that each fundus blood vessel passes through the papilla. On the other hand, since the connections between blood vessels are unclear at the position of the papilla in the blood vessel image, it tends to be difficult to accurately extract the blood vessels. Therefore, by acquiring the papilla detection results together with the blood vessel detection results and the specific position detection results, it becomes easier to understand the detailed running state of the fundus blood vessels more appropriately.

Note that the same mathematical model may be trained to output blood vessel, specific location, and nipple detection results. In this case, the output training data used for training the mathematical model may include data indicating blood vessels, data indicating a specific position, and data indicating the position of the papilla in the fundus image of the input training data. As described above, when the same mathematical model is used, each detection accuracy tends to improve.

Furthermore, a mathematical model (nipple detection model) that outputs the detection results of the nipple may be constructed separately from the mathematical model that outputs the detection results of blood vessels and specific positions. The papilla detection model may be trained using a fundus image of the subject's eye taken in the past as input training data, and using data indicating the position of the papilla in the fundus image of the input training data as output training data. . As mentioned above, when different mathematical models are used, it is easy to reduce the man-hours required to train the mathematical models.

In addition, for nipple information, instead of obtaining nipple detection results output by a mathematical model, other methods (for example, detecting the nipple by image processing, or specifying the nipple area by the user) It is also possible to obtain the information by a method such as

The control unit may further execute a blood vessel topology information generation step of generating blood vessel topology information based on the blood vessel detection result and the specific position detection result obtained in the detection result acquisition step. Blood vessel topology information generation is information on the branching structure of each detected blood vessel from the branch position to the end position. Blood vessel topology information shows the branching structure (connection form: topology) of each blood vessel from its branching position to its end position, so the detailed running state of the blood vessel can be seen in more detail than when only blood vessel images are used. better understood.

In the blood vessel topology information generation step, the control unit may generate the blood vessel topology information by also referring to the detection results of the papilla in addition to the detection results of the blood vessel and the specific position. As mentioned above, there are various relationships between the fundus blood vessels and the papilla. Therefore, the blood vessel topology information is generated more appropriately by referring to the papilla detection results along with the blood vessel detection results and the specific position detection results.

As mentioned above, each of the plurality of fundus blood vessels extends from the papilla to the periphery. Therefore, for example, by generating blood vessel topology information starting from the position of the detected papilla, it becomes easier to appropriately generate blood vessel topology information indicating the branching structure of blood vessels extending from the papilla to the surroundings. Furthermore, since the connections between blood vessels are unclear at the position of the papilla in the blood vessel image, it tends to be difficult to accurately extract the blood vessels. Therefore, for example, by performing processing on the premise that arteries and veins are connected within the region of the detected papilla, it becomes easier to generate blood vessel topology information more appropriately.

A specific method for generating blood vessel topology information based on the detection results of blood vessels and specific positions can be selected as appropriate. For example, the control unit may generate blood vessel topology information by processing the detection results of blood vessels and specific positions according to graph theory. In this case, the control unit first processes one of the detected arteries and veins, and creates a complete graph (that is, any two Construct a graph with edges between all vertices. Next, a minimum spanning tree is searched using the shortest path length through the blood vessel between the two vertices as edge weights. A spanning tree is a subgraph that includes all vertices on the graph and has no cycles (loops). A minimum spanning tree is a spanning tree with the smallest sum of edge weights. Here, the control unit may search for the minimum spanning tree based on Prim's method or the like. In this case, the control unit (1) adds the vertex closest to the starting point to the vertex set, (2) searches for the closest vertex from the vertex included in the added vertex set, and (3) adds the searched vertex to the vertex set. Add to. The control unit searches for the minimum spanning tree by repeating the processes (2) and (3) until all vertices are included in the vertex set. Note that the control unit may perform the above processing using the position of the nipple acquired in the detection result acquisition step as a starting point. In this case, the process proceeds according to the shape of the blood vessel extending from the papilla to the surrounding area, so the searched minimum spanning tree directly serves as blood vessel topology information. By performing the above processing for each artery and vein, blood vessel topology information is appropriately generated.

The control unit may perform further analysis regarding the running state of the blood vessel using the blood vessel topology information. For example, the control unit identifies the region of the blood vessel based on the number of branch positions of the blood vessel from the papilla obtained from the blood vessel topology information, and performs various analyzes (for example, analysis of blood vessel density) on the blood vessel in the identified region. , analysis (for example, quantification) of the degree of tortuosity of the blood vessel, etc.). As an example, the control unit may specify a region where the number of branch positions from the papilla is N or more as a capillary region, and may analyze the degree of tortuosity of the blood vessel in the specified capillary region. The degree of tortuosity of blood vessels in the capillary region is useful for diagnosis and research of diabetic retinopathy and the like. Further, the control unit may specify a region where the number of branching positions from the papilla is less than N as a region of a large blood vessel, and perform blood vessel analysis on the specified region of the large blood vessel. Note that the control unit determines the blood vessel region (e.g., capillary region, etc.) identified based on the blood vessel topology information using an actually photographed fundus image (e.g., an OCT angiograph in which the fundus blood vessels appear) instead of using the aforementioned blood vessel image. image, etc.), and then various analyzes may be performed on a specific blood vessel region in an actually photographed fundus image. Further, the blood vessel topology information can also be used for diagnosis of branch retinal vein occlusion (BRVO), etc.

<Embodiment>
(Device configuration)
One typical embodiment of the present disclosure will be described below with reference to the drawings. As shown in FIG. 1, in this embodiment, a mathematical model construction device 101, a fundus image processing device 1, and fundus image photographing devices 10A and 10B are used. The mathematical model construction device 101 constructs a mathematical model by training the mathematical model using a machine learning algorithm. The constructed mathematical model outputs detection results for the input fundus image (in this embodiment, detection results of blood vessels present in the fundus image, detection results of specific positions of blood vessels, and detection results of papilla). . The fundus image processing device 1 executes various processes using the results output by the mathematical model. The fundus image capturing devices 10A and 10B capture a fundus image (in this embodiment, a two-dimensional front image showing fundus blood vessels) of the eye to be examined.

As an example, a personal computer (hereinafter referred to as "PC") is used as the mathematical model construction device 101 of this embodiment. Although the details will be described later, the mathematical model construction device 101 uses data of a fundus image of the subject's eye (hereinafter referred to as a "training fundus image") obtained from the fundus image capturing device 10A and the subject on which the training fundus image was captured. A mathematical model is trained using data indicating the optometric site (in this embodiment, a blood vessel, a specific position in the blood vessel, and a papilla). As a result, a mathematical model is constructed. However, a device that can function as the mathematical model construction device 101 is not limited to a PC. For example, the fundus image capturing device 10A may function as the mathematical model construction device 101. Moreover, the control units of a plurality of devices (for example, the CPU of the PC and the CPU of the fundus image capturing apparatus 10A) may cooperate to construct the mathematical model.

Further, a PC is used as the fundus image processing device 1 of this embodiment. However, the device that can function as the fundus image processing device 1 is not limited to a PC either. For example, the fundus image capturing device 10B, a server, or the like may function as the fundus image processing device 1. When the fundus image capturing device 10B functions as the fundus image processing device 1, the fundus image capturing device 10B can process the captured fundus image while capturing the fundus image. Further, a mobile terminal such as a tablet terminal or a smartphone may function as the fundus image processing device 1. The control units of a plurality of devices (for example, the CPU of the PC and the CPU of the fundus image capturing apparatus 10B) may cooperate to perform various processes.

Further, in this embodiment, a case will be described in which a CPU is used as an example of a controller that performs various processes. However, it goes without saying that a controller other than the CPU may be used for at least some of the various devices. For example, a GPU may be used as the controller to speed up the processing.

The mathematical model construction device 101 will be explained. The mathematical model construction device 101 is placed, for example, at a manufacturer that provides the fundus image processing device 1 or the fundus image processing program to users. The mathematical model construction device 101 includes a control unit 102 that performs various control processes, and a communication I/F 105. The control unit 102 includes a CPU 103 which is a controller in charge of control, and a storage device 104 capable of storing programs, data, and the like. The storage device 104 stores a mathematical model building program for executing a mathematical model building process (see FIG. 3), which will be described later. Further, the communication I/F 105 connects the mathematical model construction device 101 to other devices (for example, the fundus image capturing device 10A and the fundus image processing device 1, etc.).

The mathematical model construction device 101 is connected to an operation unit 107 and a display device 108. The operation unit 107 is operated by the user in order for the user to input various instructions to the mathematical model construction device 101. For the operation unit 107, for example, at least one of a keyboard, a mouse, a touch panel, etc. can be used. Note that a microphone or the like for inputting various instructions may be used together with or in place of the operation unit 107. The display device 108 displays various images. As the display device 108, various devices capable of displaying images (for example, at least one of a monitor, a display, a projector, etc.) can be used. Note that "image" in the present disclosure includes both still images and moving images.

The mathematical model construction device 101 can acquire fundus image data (hereinafter sometimes simply referred to as “fundus image”) from the fundus image capturing device 10A. The mathematical model construction device 101 may acquire fundus image data from the fundus image capturing device 10A, for example, by wired communication, wireless communication, a removable storage medium (eg, USB memory), or the like.

The fundus image processing device 1 will be explained. The fundus image processing device 1 is placed, for example, in a facility (for example, a hospital or a medical examination facility) that performs diagnosis or examination of subjects. The fundus image processing device 1 includes a control unit 2 that performs various control processes, and a communication I/F 5. The control unit 2 includes a CPU 3, which is a controller that controls, and a storage device 4 that can store programs, data, and the like. The storage device 4 stores a fundus image processing program for executing fundus image processing (see FIG. 4), which will be described later. The fundus image processing program includes a program that realizes the mathematical model constructed by the mathematical model construction device 101. The communication I/F 5 connects the fundus image processing device 1 with other devices (for example, the fundus image photographing device 10B and the mathematical model construction device 101).

The fundus image processing device 1 is connected to an operation section 7 and a display device (display section) 8. Various devices can be used for the operation section 7 and the display device 8, similar to the operation section 107 and the display device 108 described above.

The fundus image processing device 1 can acquire a fundus image from the fundus image capturing device 10B. The fundus image processing device 1 may acquire the fundus image from the fundus image capturing device 10B, for example, by wired communication, wireless communication, a removable storage medium (eg, USB memory), or the like. Further, the fundus image processing device 1 may acquire a program or the like that realizes the mathematical model constructed by the mathematical model construction device 101 via communication or the like.

As the fundus image photographing device 10 (10A, 10B), various devices that take a fundus image of the eye to be examined can be used. As an example, the fundus image photographing device 10 used in this embodiment is a fundus camera capable of photographing a color image (color two-dimensional front image) of the fundus using visible light. Therefore, arteriovenous detection processing, which will be described later, is appropriately performed based on the color fundus image. However, a fundus image capturing device other than a fundus camera (for example, at least one of an OCT device, a laser scanning ophthalmoscope (SLO), etc.) may be used. Although details will be described later, a plurality of ophthalmological images taken of the same subject's eye using different imaging methods (for example, by different devices) may be processed by the fundus image processing device 1.

In this embodiment, a case will be described in which a fundus image capturing device 10A that provides fundus images to the mathematical model construction device 101 and a fundus image capturing device 10B that provides fundus images to the fundus image processing device 1 are used. However, the number of fundus image capturing devices used is not limited to two. For example, the mathematical model construction device 101 and the fundus image processing device 1 may acquire fundus images from a plurality of fundus image capturing devices. Furthermore, the mathematical model construction device 101 and the fundus image processing device 1 may acquire fundus images from one common fundus image capturing device.

(mathematical model)
The mathematical model 20 used in the first embodiment will be described with reference to FIG. 2. The fundus image processing device 1 of the first embodiment inputs a fundus image 30 actually photographed by the fundus image photographing device 10B into one mathematical model 20 trained by a machine learning algorithm, so that the mathematical model 20 is Obtain the blood vessel detection results, specific position detection results, and nipple detection results to be output.

The blood vessel detection result is the detection result of blood vessels present in the fundus image 30 input to the mathematical model 20 (that is, present in the fundus tissue reflected in the fundus image 30). As an example, the mathematical model 20 of the present embodiment includes blood vessel images 41A and 41B in which fundus blood vessels are extracted from the fundus tissue shown in the fundus image 30 (specifically, a blood vessel probability map showing a two-dimensional distribution of the probability of being a blood vessel). is trained to output as blood vessel detection results. According to the blood vessel images 41A and 41B, the state of blood vessels in the fundus tissue can be appropriately understood. The mathematical model 20 of this embodiment outputs detection results for each of the arteries and veins (hereinafter also referred to as "arteries and veins") of the fundus tissue shown in the input fundus image 30. Therefore, it becomes easier to grasp the running state of the fundus blood vessels compared to the case where detection results of blood vessels whose arteries and veins are not classified are obtained. Specifically, the mathematical model 20 of the present embodiment separately outputs an artery blood vessel image 41A of the fundus tissue and a vein blood vessel image 41V that appear in the input fundus image 30. Therefore, the running state of each artery and vein can be more easily grasped. Note that the mathematical model 20 of this embodiment classifies each pixel of the input fundus image 30 into one of three types: arteries, veins, and other tissues, thereby dividing the arterial blood vessel image 41A and the venous blood vessel image 41A. Outputs 41V. Therefore, compared to the case where the detected blood vessels are classified into arteries and veins after the blood vessels are detected, arteries and veins can be detected appropriately with simple processing.

The specific position detection result is a detection result of a specific position of a blood vessel present in the fundus image 30 input to the mathematical model 20 (that is, present in the fundus tissue reflected in the fundus image 30). In this embodiment, the specific position includes a branch position and an end position of a blood vessel. In the blood vessel images 41A and 41V described above, there are cases where a part of the fundus blood vessels that are actually connected is not detected, and the blood vessels are detected in a state where they are interrupted. Therefore, it is difficult to analyze the detailed running state of blood vessels using only the blood vessel images 41A and 41V. In contrast, the mathematical model 20 of this embodiment outputs detection results of specific positions including the branch positions and end positions of blood vessels in addition to the blood vessel images 41A and 41V. Therefore, the detailed running state of the blood vessel, which is difficult to analyze using only the blood vessel images 41A and 41V, can be more appropriately understood.

Further, in this embodiment, in addition to the branching position and end position of the blood vessel, the specific position also includes the intersection position of the blood vessel when the fundus of the eye is viewed from the front. Therefore, by considering the crossing positions of the blood vessels, it becomes easier to understand the running state of the blood vessels more accurately.

Note that the mathematical model 20 of the present embodiment is based on a two-dimensional distribution of branch positions and end positions of arteries in the fundus tissue reflected in the input fundus image 30, a two-dimensional distribution of branch positions and end positions of veins, and Each two-dimensional distribution of blood vessel intersection positions is classified and detected, and these are collectively output as specific position information 42. Therefore, according to the specific location information 42, the attributes of each detected location can be appropriately grasped. Note that in this embodiment, branch points, end points, and intersections of blood vessels are detected as specific positions. However, an area may be detected as a specific position instead of a point. Further, in the example shown in FIG. 2, the branching positions and end positions of arteries are shown as circles, the branching positions and end positions of veins are shown as square symbols, and the crossing positions of blood vessels are shown as x symbols.

The papilla detection result is a detection result of a papilla present in the fundus image 30 input to the mathematical model 20 (that is, present in the fundus tissue reflected in the fundus image 30). There are various relationships between the fundus blood vessels and the papilla. For example, each of the plurality of fundus blood vessels spreads from the papilla to the periphery, so that each fundus blood vessel passes through the papilla. On the other hand, there is also a tendency for blood vessels to be difficult to extract at the position of the papilla in the blood vessel image. Therefore, by acquiring the papilla detection results together with the blood vessel detection results and the specific position detection results, it becomes easier to understand the detailed running state of the fundus blood vessels more appropriately. The mathematical model 20 of this embodiment is trained to output nipple information 43, which is a map (nipple map) indicating a two-dimensional region where a nipple exists and its position, as a nipple detection result.

The mathematical model 20 of the first embodiment is trained to output blood vessel images 41A, 41V, specific position information 42, and nipple information 43 when one fundus image 30 is input. In this case, since the mathematical model 20 is made to learn a plurality of related tasks at the same time, common factors occur in the plurality of tasks, and the accuracy of each task is likely to be improved. In other words, since the mathematical model 20 can use common information for blood vessel detection, specific position detection, and nipple detection, the accuracy of each detection can be easily improved.

(mathematical model construction process)
With reference to FIG. 3, the mathematical model construction process executed by the mathematical model construction device 101 will be described. The mathematical model construction process is executed by the CPU 103 according to the mathematical model construction program stored in the storage device 104.

In the mathematical model construction process, a mathematical model is constructed by training the mathematical model using a training data set. The training data set includes input side data (input training data) and output side data (output training data).

As shown in FIG. 3, the CPU 103 converts data of a fundus image (in this embodiment, a two-dimensional frontal image similar to the fundus image 30 shown in FIG. 2) captured by the fundus image photographing device 10A into input training data. (S1). Next, the CPU 103 generates data indicating the blood vessels of the fundus tissue (in this embodiment, arteries and veins) and specific positions of blood vessels (in this embodiment, arteriovenous branch positions and The data indicating the end point position and the intersection position) and the data indicating the nipple (in this embodiment, the area and position of the nipple) are acquired as training data for output (S2). For example, by operating the operation unit 107 while viewing blood vessels, specific positions, and papillae in the input training data (fundus image), an operator can perform output training according to operation instructions input to the mathematical model construction device 101. Data may be generated.

Next, the CPU 103 executes training of a mathematical model using the training data set using a machine learning algorithm (S3). As machine learning algorithms, for example, neural networks, random forests, boosting, support vector machines (SVM), etc. are generally known.

Neural networks are a method that imitates the behavior of biological nerve cell networks. Neural networks include, for example, feedforward neural networks, RBF networks (radial basis functions), spiking neural networks, convolutional neural networks, recurrent neural networks (recurrent neural networks, feedback neural networks, etc.), and stochastic neural networks. There are various types of neural networks (Boltzmann machines, Bassian networks, etc.).

Random forest is a method of generating a large number of decision trees by performing learning based on randomly sampled training data. When using random forest, branches of multiple decision trees trained in advance as a classifier are followed, and the average (or majority vote) of the results obtained from each decision tree is taken.

Boosting is a method of generating a strong classifier by combining multiple weak classifiers. A strong classifier is constructed by sequentially training simple and weak classifiers.

SVM is a method of constructing a two-class pattern discriminator using linear input elements. For example, the SVM learns the parameters of a linear input element based on the criterion (hyperplane separation theorem) of finding a margin-maximizing hyperplane that maximizes the distance to each data point from training data.

The mathematical model includes, for example, input data (in this embodiment, two-dimensional tomographic image data similar to the input training data) and output data (in this embodiment, data of blood vessel detection results, detection results of specific positions). refers to the data structure for predicting the relationship between A mathematical model is constructed by being trained using a training dataset. As described above, the training data set is a set of input training data and output training data. For example, training updates correlation data (eg, weights) between each input and output.

In this embodiment, a multilayer neural network is used as a machine learning algorithm. A neural network includes an input layer for inputting data, an output layer for generating data of an analysis result desired to be predicted, and one or more hidden layers between the input layer and the output layer. A plurality of nodes (also called units) are arranged in each layer. Specifically, in this embodiment, a convolutional neural network (CNN), which is a type of multilayer neural network, is used.

Note that other machine learning algorithms may be used. For example, generative adversarial networks (GAN), which utilize two competing neural networks, may be employed as the machine learning algorithm.

The processes of S1 to S3 are repeated until construction of the mathematical model is completed (S4: NO). When construction of the mathematical model is completed (S4: YES), the mathematical model construction process ends. In this embodiment, a program and data for realizing the constructed mathematical model are incorporated into the fundus image processing device 1.

(fundus image processing)
Referring to FIGS. 4 to 8, fundus image processing performed by the fundus image processing apparatus 1 of this embodiment will be described. Fundus image processing is executed by the CPU 3 of the fundus image processing device 1 according to a fundus image processing program stored in the storage device 4.

First, the CPU 3 acquires a fundus image 30 (in this embodiment, a color two-dimensional front image) including blood vessels in the fundus of the eye to be examined, which is captured by the fundus image capturing device 10B (S1). As shown in FIG. 2, the fundus image 30 acquired in S11 includes blood vessels (arteries and veins) included in the fundus tissue of the subject's eye and a papilla.

Next, the CPU 3 inputs at least a portion of the fundus image 30 acquired in S1 (in this embodiment, the entire fundus image 30) to the mathematical model 20 trained by the machine learning algorithm. The output blood vessel detection results, specific position detection results, and nipple detection results are obtained (S12). As shown in FIG. 2, in this embodiment, an arterial blood vessel image 41A and a venous blood vessel image 41V are used as blood vessel detection results, specific position information 42 is used as a specific position detection result, and nipple information 43 is used as a nipple detection result. Output from one mathematical model 20.

Next, the CPU 3 determines the branch position of the detected blood vessel based on the blood vessel detection results (arterial blood vessel image 41A and venous blood vessel image 41V) acquired in S12 and the specific position detection result (specific position information 42). Blood vessel topology information 60 (see FIG. 8), which is information on the branching structure, is generated until reaching the end position through the blood vessel (S13). As shown in FIG. 8, the blood vessel topology information 60 shows a branching structure (connection form: topology) in which each blood vessel passes through a branch position and reaches an end position. Therefore, compared to the case where only the blood vessel images 41A and 41V are used, the detailed running state of the blood vessel can be grasped more appropriately.

Note that in S13 of the present embodiment, the blood vessel topology information 60 is generated by referring to the nipple detection result (nipple information 43) in addition to the blood vessel detection result and the specific position detection result. As mentioned above, there are various relationships between the fundus blood vessels and the papilla. Therefore, the blood vessel topology information 60 is generated more appropriately by referring to the papilla detection results along with the blood vessel detection results and the specific position detection results.

Here, an example of a method for generating blood vessel topology information 60 will be described. In this embodiment, the CPU 3 generates blood vessel topology information 60 by processing blood vessel images 41A and 41V, specific position information 42, and nipple information 43 according to graph theory. Graph theory is a mathematical theory regarding graphs that are composed of a set of nodes (nodes, vertices, points) and a set of edges (branches, sides, lines).

First, the CPU 3 sets one of the detected arteries and veins as a processing target, and constructs a complete graph having all the branch positions and end positions as vertices in the specific position information 42 about the blood vessel to be processed. FIG. 5 schematically shows an example of a complete graph with 12 vertices. As shown in FIG. 5, a complete graph is a graph that has edges between any two vertices. As described above, in this embodiment, all the branch positions and end positions of the blood vessel to be processed are set as the vertices 52.

Next, the CPU 3 searches for a minimum spanning tree using the shortest path length between two vertices in the complete graph through the blood vessel to be processed as edge weights. Here, the spanning tree is a subgraph that includes all vertices on the graph and has no cycles (loops). A minimum spanning tree is a spanning tree with the smallest sum of edge weights. FIG. 6 schematically shows an example of a graph G and a minimum spanning tree of the graph G. The numerical value attached to the edge is an example of the weight of the edge between two vertices.

FIG. 7 is an explanatory diagram for explaining a method for calculating the weights of the sides of the branch position 52S and the branch position 52T on the blood vessel in the venous blood vessel image 41V. As shown in FIG. 7, the shortest path length on the blood vessel (vein) 52V extending between the branch position 52S and the branch position 52T, rather than the shortest distance connecting the branch position 52S and the branch position 52T with a straight line, is the shortest path length in the complete graph. This is the weight of the edge between the vertex at the branch position 52S and the vertex at the branch position 52T. Specifically, in a blood vessel image (venous blood vessel image 41V in FIG. 7), consider a grid graph in which each pixel is a vertex, and four vertices (top, bottom, left, and right) (eight vertices including diagonal ones may also be used) and sides are connected. In the grid graph, the weight w_ab of the edge ab between the connected vertex a and vertex b is set to "w_ab=2-(p(a)+p(b))+eps". p(x) is the probability that the pixel x is a blood vessel to be processed (venous blood vessel in FIG. 4), and eps is a small value (for example, 0.00001) to prevent the weight from becoming 0. This reduces the weight of the edges connecting blood vessel pixels. In the constructed grid graph, the distance between two designated points is determined using Dijkstra's method.

Here, since connections between blood vessels tend to be unclear within the nipple area, processing is performed on the assumption that the blood vessels to be processed are connected in the nipple area 53 indicated by the nipple information 43. For example, in constructing a grid graph in the shortest path search described above, the weight of the edge within the nipple area is set to 0. This makes it possible to pass through the nipple area at zero cost, and the blood vessels to be processed are treated as connected within the nipple area. Therefore, by referring to the nipple information 43, it becomes easier to perform appropriate processing than when only the blood vessel images 41A and 41V, in which detection results of blood vessels in the nipple are difficult to appear, are referred to.

In this embodiment, the Prim method, which is one of the algorithms for searching a minimum spanning tree, is used as an example. The CPU 3 (1) adds the vertex closest to the starting point to the vertex set, (2) searches for the vertex closest to the vertex included in the added vertex set, and (3) adds the searched vertex to the vertex set. The CPU 3 searches for the minimum spanning tree by repeating the processes (2) and (3) until all vertices are included in the vertex set.

Here, in this embodiment, the CPU 3 performs the above-described process for searching the minimum spanning tree, starting from the position of the nipple area 53 indicated by the nipple information 43. As a result, the process proceeds according to the shape of the blood vessel extending from the papilla to the surrounding area, so the searched minimum spanning tree directly serves as the blood vessel topology information of the blood vessel to be processed. By performing the above processing for each artery and vein, blood vessel topology information 60 (see FIG. 8) is appropriately generated.

Next, the CPU 3 performs further analysis regarding the running state of the blood vessel based on the generated blood vessel topology information 60 (S14). By using the blood vessel topology information 60, various useful analysis results can be obtained.

For example, the CPU 3 specifies a blood vessel region based on the number (order) of branch positions of blood vessels from the papilla obtained from the blood vessel topology information 60, and performs various analyzes (for example, blood vessel At least one of density analysis and blood vessel tortuosity analysis (for example, quantification) may be performed. As an example, the CPU 3 may specify a region where the number of branching positions from the papilla is N or more as a capillary region, and may analyze the degree of tortuosity of the blood vessel in the specified capillary region. The degree of tortuosity of blood vessels in the capillary region is useful for diagnosis and research of diabetic retinopathy and the like. Further, the CPU 3 may specify a region where the number of branching positions from the papilla is less than N as a region of a large blood vessel, and perform blood vessel analysis on the specified region of the large blood vessel. Note that the CPU 3 uses an actually photographed fundus image (for example, where the fundus blood vessels are In addition, various analyzes may be performed on a specific blood vessel region in an actually captured fundus image. In this case, the fundus image to be analyzed may be the fundus image 30 (see FIG. 2) input to the mathematical model 20. Further, it may be an ophthalmologic image different from the fundus image 30 inputted to the mathematical model 20 (for example, a fundus image photographed using a different modality than the fundus image 30). For example, the CPU 3 performs analysis by applying the blood vessel region specified based on the blood vessel topology information 60 to an OCT angio image in which fundus blood vessels are likely to appear, thereby providing useful information that is difficult to obtain by analyzing a single ophthalmological image. It is also possible to obtain information appropriately.

A second embodiment will be described with reference to FIG. 9. The second embodiment and the first embodiment described above are different in the configuration of the mathematical model used to obtain blood vessel detection results and specific position detection results, and in the processing when obtaining the detection results. Other configurations and processing may be the same in the first embodiment and the second embodiment. Therefore, in the following, only the configuration and processing that are different from the first embodiment will be explained, and other explanations will be omitted. As shown in FIG. 9, in the second embodiment, a blood vessel detection model 21, a specific position detection model 22, and a nipple detection model 23 are separately constructed as mathematical models.

The blood vessel detection model 21 uses a fundus image of the subject's eye photographed in the past as input training data, and outputs data indicating blood vessels (arteries and veins in this embodiment) in the fundus image of the input training data. trained as data. The blood vessel detection model 21 of the second embodiment outputs detection results (arterial blood vessel image 41A and venous blood vessel image 41V) of arteries and veins of the fundus tissue shown in the input fundus image 30.

The specific position detection model 22 uses a fundus image of the subject's eye photographed in the past as input training data, and uses a specific position in the fundus image of the input training data (in this embodiment, a branch position and an end position of a blood vessel). , and intersection position) as output training data. The specific position detection model 22 of the second embodiment includes a two-dimensional distribution of branch positions and end positions of arteries in the fundus tissue reflected in the input fundus image 30, a two-dimensional distribution of branch positions and end positions of veins, and , the two-dimensional distribution of blood vessel intersection positions are classified and detected, and these are collectively output as specific position information 42.

The papilla detection model 23 is trained using a fundus image of the subject's eye taken in the past as input training data, and using data indicating the region and position of the papilla in the fundus image of the input training data as output training data. There is. The papilla detection model 23 of the second embodiment outputs, as a papilla detection result, papilla information 43 indicating a two-dimensional region where a papilla exists and its position in the fundus tissue shown in the input fundus image 30.

In the second embodiment, the CPU 3 of the fundus image processing device 1 inputs the same fundus image 30 to each of the blood vessel detection model 21, the specific position detection model 22, and the papilla detection model 23 in S12 of FIG. Then, blood vessel images 41A, 41V, specific position information 42, and nipple information 43 output from each model are acquired.

In the second embodiment, it becomes easy to adjust the parameters of each of the plurality of mathematical models 21 to 23. Furthermore, since the plurality of mathematical models 21 to 23 are constructed separately, the number of man-hours required for training the mathematical models can be easily reduced compared to the case of constructing the mathematical model 20 of the first embodiment (see FIG. 2). It is. For example, when training a mathematical model, unlike the case where both data indicating blood vessels and data indicating a specific position in a fundus image are prepared as input training data, one of data indicating blood vessels and data indicating a specific position is prepared. Once prepared, a blood vessel detection model 21 or a specific position detection model 22 is constructed. Therefore, it is also possible to construct any one of a plurality of mathematical models first depending on various situations. Further, if any one of the plurality of mathematical models has already been constructed, it is sufficient to newly construct the other mathematical model.

The techniques disclosed in the above embodiments are merely examples. Therefore, it is also possible to modify the techniques exemplified in the above embodiments. For example, it is also possible to execute only some of the techniques exemplified in the above embodiments. Note that the process of acquiring a fundus image in S11 in FIG. 4 is an example of a "fundus image acquisition step." The process of acquiring the detection result in S12 of FIG. 4 is an example of a "detection result acquisition step." The process of generating the blood vessel topology information 60 in S13 of FIG. 4 is an example of a "blood vessel topology information generation step."

1 Fundus image processing device 3 CPU
4 Storage device 20 Mathematical model 21 Blood vessel detection model 22 Specific position detection model 23 Papilla detection model 30 Fundus images 41A, 41V Blood vessel image 42 Specific position information 43 Papilla information 60 Blood vessel topology information 10 (10A, 10B) Fundus image capturing device 101 Mathematics model building equipment

Claims (9)

A fundus image processing device that processes a fundus image of an eye to be examined,
The control unit of the fundus image processing device includes:
a fundus image acquisition step of acquiring the fundus image including blood vessels in the fundus of the eye to be examined, which is photographed by a fundus image photographing device;
By inputting at least a portion of the fundus image into a mathematical model trained by a machine learning algorithm, detection results of blood vessels present in the fundus image, and branch positions and end positions of blood vessels present in the fundus image are obtained. a detection result acquisition step of acquiring detection results of a specific position including;
A fundus image processing device that performs the following. The fundus image processing device according to claim 1,
The fundus oculi image processing device, wherein the blood vessel detection results acquired by the mathematical model in the detection result acquisition step are detection results for each of an artery and a vein present in the fundus image. The fundus image processing device according to claim 1 or 2,
In the detection result acquisition step, the control unit acquires, as the specific position, a detection result of a blood vessel intersection position when the fundus is viewed from the front, as well as a branch position and an end position of the blood vessel. Fundus image processing device. The fundus image processing device according to any one of claims 1 to 3,
The mathematical model uses, as input training data, a fundus image of the subject's eye photographed in the past, and data indicating blood vessels in the fundus image of the input training data, and the fundus image of the input training data. is trained using data indicating the specific position in as output training data,
In the fundus oculi image processing device, in the detection result acquisition step, one of the fundus images is input to the mathematical model, so that both the blood vessel detection result and the specific position detection result are obtained. The fundus image processing device according to any one of claims 1 to 3,
As the mathematical model,
A blood vessel detection model trained using a fundus image of a subject's eye photographed in the past as input training data, and using data indicating blood vessels in the fundus image of the input training data as output training data;
A specific position detection model trained using a fundus image of the subject's eye taken in the past as input training data, and using data indicating a specific position in the fundus image of the input training data as output training data;
has been constructed,
In the detection result acquisition step, the control unit inputs the fundus image into the blood vessel detection model to obtain the blood vessel detection result, and inputs the fundus image into the specific position detection model to obtain the blood vessel detection result. A fundus image processing device characterized by acquiring a detection result of a specific position. The fundus image processing device according to any one of claims 1 to 5,
In the detection result acquisition step, the control unit inputs at least a part of the fundus image into a mathematical model trained by a machine learning algorithm, thereby obtaining the blood vessel detection result and the specific position detection result together with the blood vessel detection result and the specific position detection result. A fundus image processing device characterized by acquiring a detection result of an optic disc present in a fundus image. The fundus image processing device according to any one of claims 1 to 6,
The control unit includes:
Information on the branching structure of each detected blood vessel from the branch position to the end position, based on the detection result of the blood vessel and the detection result of the specific position acquired in the detection result acquisition step. A fundus oculi image processing device further comprising a step of generating blood vessel topology information. The fundus image processing device according to claim 6,
The control unit includes:
Based on the detection result of the blood vessel, the detection result of the specific position, and the detection result of the papilla acquired in the detection result acquisition step, in each of the detected blood vessels, go to the end position through the branch position. A fundus oculi image processing device characterized by further executing a blood vessel topology information generation step of generating blood vessel topology information that is information on a branched structure up to the point. A fundus image processing program executed by a fundus image processing device that processes a fundus image of an eye to be examined,
The fundus image processing program is executed by the control unit of the fundus image processing device,
a fundus image acquisition step of acquiring the fundus image including blood vessels in the fundus of the eye to be examined, which is photographed by a fundus image photographing device;
By inputting at least a portion of the fundus image into a mathematical model trained by a machine learning algorithm, detection results of blood vessels present in the fundus image, and branch positions and end positions of blood vessels present in the fundus image are obtained. a detection result acquisition step of acquiring detection results of a specific position including;
A fundus image processing program that causes the fundus image processing device to execute.

Images