JP7328489B2 - Ophthalmic image processing device and ophthalmic photographing device - Google Patents

Description

The present invention relates to an ophthalmic image processing apparatus for processing an ophthalmic image and an ophthalmic photographing apparatus for capturing an ophthalmic image.

In recent years, image processing by deep learning has achieved results, and its application to medical imaging is expected.

JP 2017-162438 A

By the way, in order to create an image processing model using deep learning, it is necessary to create a large number of sets of images and correct data corresponding to them, and have the image processing model learn them. However, the annotation work of giving correct answers to a large number of images has been a heavy burden on the user. In addition, ophthalmologic images have complex tissue shapes and the like, which makes annotation difficult and imposes a heavy burden on the user.

In view of the conventional problems, the technical problem of the present disclosure is to provide an ophthalmologic image processing apparatus and an ophthalmologic imaging apparatus capable of efficiently annotating ophthalmologic images.

In order to solve the above problems, the present disclosure is characterized by having the following configuration.

(1) An ophthalmologic image processing apparatus for processing an ophthalmologic image of an eye to be examined, comprising: image acquisition means for acquiring the ophthalmologic image; and control means for processing the ophthalmologic image, wherein the control means processes the image Each pixel obtained when the ophthalmologic image is input to the image processing model, from among the ophthalmologic images obtained by the obtaining means, to be learned by an image processing model that performs segmentation of the retinal layer in the ophthalmologic image. is selected according to the entropy of the probability distribution indicating which retinal layer each pixel belongs to or the degree of dispersion of the probability distribution indicating which retinal layer each pixel belongs to .
(2) An ophthalmic photographing apparatus for photographing an ophthalmologic image of an eye to be examined, comprising photographing means for photographing the ophthalmic image, and control means for controlling the ophthalmic photographing apparatus, wherein the control means is the photographing means. From among the ophthalmologic images photographed by the above, a training image to be learned by an image processing model that performs segmentation of the retinal layer in the ophthalmologic image is input to the image processing model. The selection is made according to the entropy of the probability distribution indicating whether it is a retinal layer or the degree of scatter of the probability distribution indicating which retinal layer each pixel belongs to .

1 is a block diagram showing a schematic configuration of an image processing model construction device, an ophthalmologic image processing device, and an ophthalmologic photographing device; FIG. FIG. 4 is a diagram showing an example of input and output of an image processing model; 6 is a flowchart showing an example of image processing model construction processing; 6 is a flowchart showing an example of image processing model construction processing;

<Embodiment>
Embodiments according to the present disclosure will be described below. An ophthalmologic image processing apparatus of the present disclosure processes an ophthalmologic image of an eye to be examined. The ophthalmologic image processing apparatus includes, for example, an image acquisition unit (eg, control unit 21, etc.) and a control unit (eg, control unit 21, etc.). The image acquisition unit acquires, for example, an ophthalmologic image. The control unit processes ophthalmic images. In addition, the control unit selects learning images to be learned by the image processing model (for example, the image processing model 15) from among the ophthalmologic images acquired by the image acquisition unit.

Note that the control unit may select a learning image based on an evaluation value (for example, the degree of confusion) based on an output when the ophthalmologic image is input to the image processing model. The evaluation value may be, for example, entropy relating to the output of the image processing model, standard deviation, coefficient of variation, variance, or the like.

Note that the control unit may acquire annotation information (such as a label, for example) attached by the user's annotation to the learning image. That is, the control unit may acquire annotation information added by the user annotating the learning image. Annotation is, for example, assigning a label to each pixel of the learning image.

Note that the control unit may present annotation information candidates (for example, candidate labels) for the learning images. For example, the control unit may cause the display unit (for example, the display unit 28) to display candidates for annotation information. The annotation information candidates may be created based on the output when the learning image is input to the image processing model.

Note that the present disclosure may be applied to an ophthalmic imaging apparatus. For example, it may be applied to an ophthalmologic imaging apparatus (for example, ophthalmologic imaging apparatus 30) that captures an ophthalmologic image of an eye to be examined. In this case, the ophthalmologic imaging apparatus may include an imaging section (eg, ophthalmologic image imaging section 34) and a control section (eg, control section 31). The imaging unit captures an ophthalmologic image. The control unit controls the ophthalmic imaging apparatus. Further, the control unit may select a learning image to be learned by the image processing model from among the ophthalmologic images captured by the imaging unit. In this case, the control unit may select a learning image for learning by the image processing model each time an ophthalmologic image is captured by the imaging unit.

Note that the ophthalmologic image processing apparatus or ophthalmologic imaging apparatus may include an interface for annotating the learning image (for example, the operation unit 27, the display unit 28, the operation unit 37, the display unit 38, etc.). In this case, the control unit (for example, control unit 21 or control unit 31) may acquire annotation information based on an input to the interface. Note that the control unit (for example, the control unit 21 or the control unit 31) may construct or update the image processing model based on the learning image.

<Example>
One exemplary embodiment of the present disclosure will now be described. As shown in FIG. 1, the image processing model building system 1 of this embodiment includes an image processing model building device 10, an ophthalmologic image processing device 20, an ophthalmologic photographing device 30, an image database (hereinafter abbreviated as image DB) 40, and the like. Prepare. The image processing model building device 10 builds the image processing model 15 by training the image processing model 15 with a machine learning algorithm. The image processing model 15 outputs, for each region (for example, each pixel) of the input ophthalmologic image, a probability distribution with the type of each tissue in the eye to be inspected as a random variable. In this case, the tissue in the ophthalmic image is identified based on the probability distributions output by the image processing model 15 . Also, the image processing model 15 may output a mapping image showing the tissue identification result. For example, in the example of the image processing model 15 shown in FIG. 2, when a fundus tomographic image is input, a mapping image color-coded for each retinal layer is output. This is so-called retinal layer segmentation. The ophthalmologic photographing apparatus 30 photographs an ophthalmologic image, which is an image of tissue of an eye to be examined. The image DB 40 stores ophthalmologic images captured by the ophthalmic imaging device 30 . The image DB 40 is, for example, a server.

For example, a personal computer (hereinafter referred to as “PC”) is used for the image processing model construction device 10 of this embodiment. The image processing model construction device 10 trains the image processing model 15 using, for example, an ophthalmologic image captured by the ophthalmologic imaging device 30 and a label indicating at least one type of tissue in the ophthalmologic image. A processing model 15 is constructed. A device that can function as the image processing model construction device 10 is not limited to a PC. For example, the ophthalmologic image processing device 20 or the ophthalmologic imaging device 30 may function as the image processing model building device 10 . Also, multiple devices may work together to build an image processing model.

A PC is used for the ophthalmologic image processing apparatus 20 of this embodiment. However, a device that can function as the ophthalmologic image processing apparatus 20 is not limited to a PC either. For example, the ophthalmic photographing device 30 or a server may function as the ophthalmic image processing device 20 . Moreover, a portable terminal such as a tablet terminal or a smartphone may function as the ophthalmologic image processing apparatus 20 . The controllers of a plurality of devices (for example, the CPU of the PC and the CPU 32 of the ophthalmologic photographing apparatus 30) may work together to perform various processes.

Also, in this embodiment, a case in which a CPU is used as an example of a controller that performs various processes will be described. However, it goes without saying that a controller other than the CPU may be used for at least some of the various devices. For example, a GPU may be employed as a controller to speed up processing.

<Image processing model building device>
The image processing model construction device 10 will be described. The image processing model construction device 10 is installed, for example, in a manufacturer or the like that provides the ophthalmic image processing device 20 or an ophthalmic image processing program to users. The image processing model construction device 10 includes a control unit 11 that performs various control processes. The control unit 11 includes a CPU 12, which is a controller for control, and a storage device 13 capable of storing programs, data, and the like. The storage device 13 stores an image processing model construction program for executing image processing model construction processing. The storage device 13 also stores a labeled image database (hereinafter abbreviated as a labeled image DB) 14 in which labeled images are stored, and an image processing model 15 . Note that the image processing model construction device 10 is connected to other devices (for example, the ophthalmic image processing device 20 and the ophthalmic photographing device 30, etc.) by communication means (not shown).

The image processing model construction device 10 is connected to the operation section 17 and the display section 18 . The operation unit 17 is operated by the user U to input various instructions to the image processing model construction device 10 . For example, at least one of a keyboard, a mouse, a touch panel, and the like can be used as the operation unit 17 . A microphone or the like for inputting various instructions may be used together with the operation unit 17 or instead of the operation unit 17 . The display unit 18 displays various images. Various devices capable of displaying images (for example, at least one of a monitor, a display, a projector, etc.) can be used for the display unit 18 . It should be noted that the “image” in the present disclosure includes both still images and moving images.

In the image processing model building process, the image processing model 15 is trained using the training data set to build the image processing model 15 that outputs a probability distribution for identifying tissue in the ophthalmologic image. The training data set includes data on the input side (training data for input) and data on the output side (training data for output).

The image processing model 15 learns the training data set based on machine learning algorithms. As machine learning algorithms, for example, neural networks, random forests, boosting, support vector machines (SVM), etc. are generally known.

A neural network is a technique that imitates the behavior of a neural network of living organisms. Neural networks include, for example, feedforward neural networks, RBF networks (radial basis functions), spiking neural networks, convolutional neural networks, recurrent neural networks (recurrent neural networks, feedback neural networks, etc.), probability neural networks (Boltzmann machine, Baysian network, etc.), etc.

Random forest is a method of learning based on randomly sampled training data to generate a large number of decision trees. When a random forest is used, branches of a plurality of decision trees learned in advance as discriminators are traced, and the average (or majority vote) of the results obtained from each decision tree is taken.

Boosting is a method of generating a strong classifier by combining a plurality of weak classifiers. A strong classifier is constructed by sequentially learning simple and weak classifiers.

SVM is a method of constructing a two-class pattern discriminator using linear input elements. The SVM learns the parameters of the linear input element, for example, based on the criterion of finding the margin-maximizing hyperplane that maximizes the distance to each data point from the training data (hyperplane separation theorem).

An image processing model, for example, refers to a data structure for predicting the relationship between input data and output data. An image processing model is constructed by being trained using a training dataset. As described above, the training data set is a set of input training data and output training data. For example, training updates the correlation data (eg, weights) for each input and output.

In this embodiment, a multilayer neural network is used as the machine learning algorithm. A neural network includes an input layer for inputting data, an output layer for generating data to be predicted, and one or more hidden layers between the input layer and the output layer. A plurality of nodes (also called units) are arranged in each layer. Specifically, in this embodiment, a convolutional neural network (CNN), which is a type of multilayer neural network, is used.

As an example, the image processing model 15 constructed in this embodiment has a region in an ophthalmologic image (any one of a one-dimensional region, a two-dimensional region, a three-dimensional region, and a four-dimensional region including a time axis), Coordinates (any one of one-dimensional, two-dimensional, three-dimensional, and four-dimensional coordinates) at which a specific tissue (for example, a specific boundary, a specific layer, or a specific part) exists is a random variable Output the probability distribution as a probability distribution for identifying tissue. In this embodiment, a softmax function is applied to force the image processing model to output a probability distribution.

Other machine learning algorithms may also be used. For example, Generative adversarial networks (GAN), which utilize two competing neural networks, may be employed as machine learning algorithms.

<Ophthalmic image processing device>
The ophthalmologic image processing apparatus 20 will be described. The ophthalmologic image processing apparatus 20 is installed, for example, in a facility (for example, a hospital, a health checkup facility, or the like) for diagnosing or examining a subject. The ophthalmologic image processing apparatus 20 includes a control unit 21 that performs various control processes. The control unit 21 includes a CPU 22, which is a controller for control, and a storage device 23 capable of storing programs, data, and the like. The storage device 23 stores an ophthalmic image processing program for executing ophthalmic image processing. The ophthalmologic image processing program includes a program for realizing the image processing model constructed by the image processing model construction device 10 . The ophthalmologic image processing apparatus 20 is connected to the ophthalmologic image processing model building apparatus 10, the ophthalmologic photographing apparatus 30, the image DB 40, and the like by communication means (not shown).

The ophthalmologic image processing apparatus 20 is connected to an operation section 27 and a display section 28 . Various devices can be used for the operation unit 27 and the display unit 28, similarly to the operation unit 7 and the display unit 8 described above.

The ophthalmologic image processing apparatus 20 can acquire an ophthalmologic image from the ophthalmic photographing apparatus 30 or the image DB 40 . The ophthalmologic image processing apparatus 20 may acquire ophthalmologic images from the ophthalmologic photographing apparatus 30 or the image DB 40 by, for example, at least one of wired communication, wireless communication, and a removable storage medium (eg, USB memory). In this case, the control unit 21 may function as an image acquisition unit that acquires an ophthalmologic image. Further, the ophthalmologic image processing apparatus 20 may acquire a program or the like for realizing the image processing model 15 constructed by the image processing model construction apparatus 10 via communication means or the like.

<Ophthalmic imaging equipment>
The ophthalmologic imaging apparatus 30 will be described. In this embodiment, an OCT apparatus is exemplified as the ophthalmologic imaging apparatus 30 . However, an ophthalmologic imaging device other than the OCT device (for example, a laser scanning optometric device (SLO), a fundus camera, a Scheimpflug camera, or a corneal endothelial cell imaging device (CEM)) may be used.

The ophthalmologic imaging apparatus 30 includes a control unit 31 that performs various control processes, and an ophthalmologic imaging unit 34 . The control unit 31 includes a CPU 32, which is a controller for control, and a storage device 33 capable of storing programs, data, and the like.

The ophthalmologic image capturing unit 34 has various configurations necessary for capturing an ophthalmologic image of the subject's eye. The ophthalmologic image capturing unit 34 of the present embodiment includes an OCT light source, a branching optical element for branching the OCT light emitted from the OCT light source into measurement light and reference light, a scanning unit for scanning the measurement light, and a scanning unit for scanning the measurement light. It includes an optical system for irradiating an eye to be examined, a light receiving element for receiving the combined light of the light reflected by the tissue of the eye to be examined and the reference light, and the like.

The ophthalmologic imaging apparatus 30 can capture a two-dimensional tomographic image and a three-dimensional tomographic image of the fundus of the subject's eye. Specifically, the CPU 32 captures a two-dimensional tomographic image of a cross section that intersects the scan lines by scanning OCT light (measurement light) along the scan lines. The two-dimensional tomographic image may be an averaging image generated by averaging a plurality of tomographic images of the same region. Moreover, the CPU 32 can capture a three-dimensional tomographic image of a tissue by two-dimensionally scanning the OCT light. For example, the CPU 32 acquires a plurality of two-dimensional tomographic images by scanning measurement light on each of a plurality of scan lines at different positions in a two-dimensional region when the tissue is viewed from the front. Next, the CPU 32 acquires a three-dimensional tomographic image by combining a plurality of captured two-dimensional tomographic images.

<Image processing model construction processing>
The image processing model building system 1 creates an image processing model in the flow of steps S1 to S8, as shown in the flowchart of FIG. Each step will be described below.

(S1: Input of unlabeled image)
In step S<b>1 , a large amount of unlabeled images, which are unlabeled ophthalmologic images, are input from the ophthalmologic imaging apparatus 30 to the image DB 40 . For example, an unlabeled image captured by the ophthalmic imaging device 30 is automatically transmitted via the communication means. Alternatively, an unlabeled image selected by the user U may be input to the ophthalmic imaging device 30 . When an OCT apparatus is used as the ophthalmologic imaging apparatus 30, for example, a tomographic image in which retinal layer segmentation has not been performed is input to the image DB 40 as an unlabeled image.

(S2: random image selection)
In step S2, the control unit 21 of the ophthalmologic image processing apparatus 20 randomly selects some of the large number of unlabeled images stored in the image DB 40, and uses them as learning images for the image processing model 15 to learn. provide to U. For example, the control unit 21 may display the selected unlabeled image on the display unit 28 .

(S3: Labeling)
In step S<b>3 , the user U labels (annotates) the learning images provided by the control unit 21 of the ophthalmologic image processing apparatus 20 . A label (annotation information) is identification information of an organization or the like. The user U labels each pixel of the unlabeled image, for example. The user U, for example, uses a paint tool or the like to paint the tomographic image in each region of each retinal layer. The user U may perform labeling using the operation unit 27 and the display unit 28 of the ophthalmologic image processing apparatus 20, or may perform labeling using the operation unit 17 and the display unit 18 of the image processing model construction apparatus 10. may be performed. Moreover, if the ophthalmologic photographing apparatus 30 includes the operation unit 37 and the display unit 38, the user U may use them for labeling.

(S4: Input labeled image)
In step S4, the labeled image labeled by the user U is input to the labeled image DB14. For example, the ophthalmologic image processing apparatus 20 transmits labeled images labeled by the user U to the image processing model construction apparatus 10 . The CPU 12 of the image processing model construction device 10 stores the labeled image received from the ophthalmologic image processing device 20 in the labeled image DB 14 .

(S5: Learning labeled images)
In step S<b>5 , the image processing model construction device 10 causes the image processing model 15 to learn the labeled images stored in the labeled image DB 14 . For example, only labeled images are input, and the internal parameters (weights) of the image processing model 15 are adjusted so that the result approaches the label information.

(S6: model evaluation)
In step S<b>6 , the image processing model construction device 10 evaluates the accuracy of the image processing model 15 . For example, an ophthalmologic image prepared in advance with a correct answer is input to the image processing model 15, and the output at that time is compared with the correct answer for evaluation. Alternatively, an ophthalmologic image may be input to the image processing model 15, and the user U may check whether the output at that time is appropriate. If the evaluation of the image processing model 15 is good, the image processing model construction device 10 ends the image processing model construction processing, and if the evaluation is bad, the process proceeds to step S7.

(S7: Calculation of hesitation degree)
In step S7, the ophthalmologic image processing apparatus 20 inputs the unlabeled image of the image DB 40 to the image processing model 15 that is in the process of learning, and causes the image processing model 15 to calculate the degree of hesitation with respect to this image.

In this embodiment, the entropy (average amount of information) of the probability distribution P indicating which retinal layer each pixel belongs to is calculated as the degree of confusion. Entropy represents the degree of uncertainty, clutter, and disorder. Entropy is given by the following equation (1). The entropy H(P) takes a value of 0≦H(P)≦log (number of events), and becomes a smaller value as the probability distribution P is biased. That is, the smaller the entropy H(P), the more the image processing model 15 can identify the tissue without hesitation. The entropy of the probability distribution is zero when the tissue is correctly identified. Also, the more difficult it is to identify tissue, the greater the entropy H(P). Therefore, by using the entropy H(P) of the probability distribution P as the degree of hesitation, an image that is difficult for the image processing model 15 to identify (that is, an image to be learned) can be determined.

However, a value other than entropy may be adopted as the degree of hesitation. For example, at least one of the standard deviation, coefficient of variation, variance, etc. that indicates the degree of dispersion of the obtained probability distribution P may be used as the degree of hesitation. KL divergence or the like, which is a measure of the difference between the probability distributions P, may be used as the degree of hesitation. Also, the maximum value of the acquired probability distribution P may be used as the degree of hesitation. Also, the difference between the maximum value of the acquired probability distribution P and the second largest value may be used as the degree of hesitation.

(S8: Image selection based on hesitation)
In step S8, the ophthalmologic image processing apparatus 20 selects learning images for causing the image processing model 15 to learn images based on the degree of hesitation calculated in step S7. For example, the top few images with a high degree of confusion are selected and provided to the user U. The image processing model construction system 1 repeats steps S3 to S8 until the accuracy of the image processing model 15 is improved.

Note that the image processing model construction system 1 of the present embodiment may perform the labeling in step S3 in the following flow (S301 to S303) to simplify the labeling of the user U (see FIG. 4). ).

(S301: Candidate label creation)
In step S301, the ophthalmologic image processing apparatus 20 inputs learning images selected based on the degree of hesitation into the image processing model 15 to create candidate labels. That is, the output when a learning image (unlabeled image) is input to the image processing model 15 that is in the process of learning is used as the label of the learning image.

(S302: add candidate label)
In step S302, the ophthalmologic image processing apparatus 20 adds the candidate label created in step S301 to the learning image.

(S303: Candidate label confirmation/correction)
In step S303, the user U compares the learning image provided by the image processing model construction system 1 with its candidate label, and confirms whether the candidate label is appropriate for the learning image. If the candidate label is suitable, it is used as the label of the training image. If the candidate label is not suitable, modify the candidate label or create a new label.

As described above, the image processing model construction system 1 reduces the amount of annotations by learning only some images selected from a large number of unlabeled images, while efficiently creating an image processing model 15 with high accuracy. can be constructed. In addition, the image processing model building system 1 can simplify annotations by adding label candidates to learning images, thereby reducing the burden on the user. In particular, for ophthalmologic images where segmentation target areas are small, shapes are complex, and boundaries are difficult to distinguish, reducing the amount of annotations and simplifying the work reduces the possibility of omissions and mistakes in annotations. can be reduced.

By facilitating the creation of the image processing model 15, it becomes possible to create the image processing model 15 in small units such as hospitals and doctors, and the image processing model 15 suitable for each user can be created. It also facilitates the use of the image processing model 15 for research purposes.

In the above description, creation of a retinal layer segmentation model was taken as an example, but application to other image identification and regression models (for example, models that output numerical values) is also possible.

Note that when selecting a learning image, a part of the unlabeled image stored in the image DB 40 may be cut out and used as the learning image. For example, a characteristic portion (such as a lesion) detected from the image may be cut out, or a portion of the image with a large degree of confusion may be cut out.

Additional information added to the ophthalmologic image may be used as a selection criterion for selecting a learning image. The additional information includes, for example, the disease name of the subject's eye being photographed, the photographing conditions when photographing with the ophthalmologic photographing apparatus 30, and the like. For example, the ophthalmic image processing apparatus 20 selects learning images so that the image processing model 15 learns ophthalmologic images with different disease names or imaging conditions as much as possible. For example, the ophthalmologic image processing apparatus 20 may preferentially select, as learning images, ophthalmologic images with disease names or imaging conditions that are fewer in number among the images stored in the image DB 40 .

In addition, as a selection criterion for selecting learning images, the strength of the signal of the ophthalmologic image or an index indicating the quality of the signal (for example, SSI (Signal Strength Index) or QI (Quality Index), etc.) is used. may In this case, for example, when the acquired index is equal to or less than the threshold, the CPU 22 may determine that the image is not a proper image and exclude it from the learning image options.

Various images can be used as the ophthalmologic image input to the image processing model 15 . For example, the ophthalmic image may be a two-dimensional tomographic image or a three-dimensional tomographic image of the tissue of the subject's eye captured by an OCT apparatus. A tomographic image may be captured by a device other than an OCT device (for example, a Scheimpflug camera or the like). The ophthalmologic image may be a two-dimensional frontal image captured by a fundus camera, a two-dimensional frontal image captured by a laser scanning optometric device (SLO), or the like. The ophthalmologic image may be a two-dimensional front image (so-called "Enface image") generated based on data of a three-dimensional tomographic image captured by an OCT apparatus. Further, the ophthalmologic image may be a two-dimensional frontal image (so-called "motion contrast image") created from motion contrast data obtained by processing a plurality of OCT data acquired from the same position at different times. . A two-dimensional front image is a two-dimensional image of a tissue photographed from the direction of the optical axis of photographing light. Also, the tissue to be imaged can be selected as appropriate. For example, an image obtained by photographing any one of the fundus, anterior segment, angle, etc. of the subject's eye may be used as the ophthalmologic image.

1 image processing model building system 10 image processing model building device 20 ophthalmic image processing device 30 ophthalmic photographing device

Claims (2)

An ophthalmic image processing device for processing an ophthalmic image of an eye to be examined,
an image acquisition means for acquiring the ophthalmologic image;
a control means for processing the ophthalmic image;
The control means is configured to transfer, from the ophthalmologic images acquired by the image acquisition means, learning images to be learned by an image processing model that performs segmentation of a retinal layer in the ophthalmologic images to the image processing model. An ophthalmologic image processing apparatus, characterized in that selection is made according to the entropy of a probability distribution indicating which retinal layer each pixel belongs to when inputting, or the degree of dispersion of a probability distribution indicating which retinal layer each pixel belongs to . An ophthalmic imaging apparatus for capturing an ophthalmic image of an eye to be examined,
a photographing means for photographing the ophthalmologic image;
and a control means for controlling the ophthalmic imaging apparatus,
The control means is configured to transfer the ophthalmologic image to the image processing model from among the ophthalmologic images photographed by the photographing means as learning images to be learned by an image processing model that performs segmentation of a retinal layer in the ophthalmologic image. An ophthalmologic imaging apparatus, characterized in that selection is made according to the entropy of a probability distribution indicating which retinal layer each pixel belongs to when an input is made or the degree of dispersion of a probability distribution indicating which retinal layer each pixel belongs to .

Images