JP7435067B2 - Ophthalmology image processing device and ophthalmology image processing program - Google Patents

Description

The present disclosure relates to an ophthalmological image processing device and an ophthalmological image processing program that process an ophthalmological image of an eye to be examined.

Conventionally, various techniques for acquiring high-quality images have been proposed. For example, the ophthalmological image processing apparatus described in Patent Document 1 acquires a target image of higher quality than the base image by inputting the base image to a mathematical model trained by a machine learning algorithm. The mathematical model is trained in advance using low-resolution training images as input training data and high-resolution training images as output training data.

JP 2020-678 Publication

According to the ophthalmologic image processing apparatus described in Patent Document 1, a high-quality target image can be obtained even if a high-quality (high-resolution) ophthalmologic image is not actually captured. However, with conventional techniques, it has been difficult to obtain images that have effects other than high image quality from images that have actually been photographed.

A typical objective of the present disclosure is to provide an ophthalmological image processing device and an ophthalmic image processing program for acquiring useful images from images actually taken.

A first aspect of an ophthalmologic image processing apparatus provided by a typical embodiment of the present disclosure is an ophthalmologic image processing apparatus that processes an ophthalmologic image that is an image of a tissue of a subject's eye, the ophthalmologic image processing apparatus including a control unit of the ophthalmologic image processing apparatus. comprises a base image acquisition step of acquiring, as a base image, an ophthalmological image photographed by an OCT device, or an arithmetic average image obtained by adding and averaging a plurality of ophthalmological images photographed at the same site of tissue by the OCT device; an imaging condition input step that accepts input of an instruction to designate an imaging condition other than the number of additional images, which is the number of ophthalmological images used for (≧1), and a mathematical model trained by a machine learning algorithm in the imaging condition input step. executing a target image acquisition step of acquiring a target image corresponding to an image photographed under the photographing conditions other than the input additional number of images by inputting the input photographing conditions and the base image; The mathematical model uses the data of some of the training ophthalmology images taken of the same part of the tissue as the input training data, and sets the imaging conditions other than the number of images for input as the input training data. Training is performed using data of the training ophthalmological images that are different from the shooting conditions of the training data as output training data.
A second aspect of an ophthalmological image processing apparatus provided by a typical embodiment of the present disclosure is an ophthalmological image processing apparatus that processes an ophthalmological image that is an image of a tissue of an eye to be examined, the control unit of the ophthalmological image processing apparatus comprises a base image acquisition step of acquiring, as a base image, an ophthalmological image photographed by an OCT device, or an averaged image obtained by adding and averaging a plurality of ophthalmological images photographed at the same site of tissue by the OCT device; A base image photographing condition acquisition step of acquiring the photographing conditions under which the image was photographed by the OCT device, and a mathematical model trained by a machine learning algorithm that includes the base image acquired in the base image photographing condition acquisition step. By inputting the imaging conditions and the base image, the number of ophthalmological images used for the arithmetic average (≧1) is added to the imaging conditions under which the base image was captured by the OCT device. a target image acquisition step of acquiring target images corresponding to images having different imaging conditions other than the number of images; The data of the training ophthalmology image is used as input training data, and the data of the training ophthalmology image whose imaging conditions other than the number of images to be added are different from the imaging conditions of the input training data is trained as output training data. There is.
A third aspect of an ophthalmological image processing apparatus provided by a typical embodiment of the present disclosure is an ophthalmological image processing apparatus that processes an ophthalmological image that is an image of a tissue of an eye to be examined, the control unit of the ophthalmological image processing apparatus a base image acquisition step of acquiring, as a base image, an ophthalmologic image captured by an OCT device, or an averaged image obtained by adding and averaging a plurality of ophthalmologic images captured by the OCT device at the same site of tissue; and an image range. A range input step of accepting an input for specifying a part of the range, and inputting the base image to a mathematical model trained by a machine learning algorithm, when the base image is captured by the OCT device. Acquire a target image corresponding to a different image in the range input in the range input step for the imaging condition other than the number of ophthalmological images used for the arithmetic average (≧1). performing a target image acquisition step, and the mathematical model uses data of some of the training ophthalmology images, out of a plurality of training ophthalmology images taken of the same part of the tissue, as input training data; Training is performed using data of the training ophthalmological images whose imaging conditions other than the number of images to be added differ from those of the input training data as output training data.

A first aspect of an ophthalmological image processing program provided by a typical embodiment of the present disclosure is an ophthalmic image processing program executed by an ophthalmic image processing apparatus that processes an ophthalmological image that is an image of a tissue of a subject's eye. The ophthalmologic image processing program is executed by the control unit of the ophthalmologic image processing device, so that an ophthalmologic image captured by the OCT device or a plurality of ophthalmologic images captured by the OCT device of the same region of tissue are averaged. a base image acquisition step of acquiring the averaged image obtained as a base image; and a photographing condition input step of receiving an instruction to designate a photographing condition other than the number of additional images, which is the number of ophthalmological images used for the averaging (≧1). By inputting the photographing conditions input in the photographing condition input step and the base image to a mathematical model trained by a machine learning algorithm, images are taken under the photographing conditions other than the input additional number of images . the ophthalmological image processing apparatus executes a target image acquisition step of acquiring a target image corresponding to the image; The data of the training ophthalmology image is used as the input training data, and the data of the training ophthalmology image whose imaging conditions other than the number of images to be added are different from the imaging conditions of the input training data is used as the output training data. .
A second aspect of the ophthalmologic image processing program provided by the exemplary embodiment of the present disclosure is an ophthalmologic image processing program executed by an ophthalmologic image processing apparatus that processes an ophthalmologic image that is an image of a tissue of a subject's eye, The ophthalmologic image processing program is executed by the control unit of the ophthalmologic image processing device, so that an ophthalmologic image captured by the OCT device or a plurality of ophthalmologic images captured by the OCT device of the same region of tissue are averaged. a base image acquisition step of acquiring the averaged image obtained as a base image; a base image photography condition acquisition step of acquiring the photography conditions under which the base image was photographed by the OCT device; By inputting the photographing conditions of the base image acquired in the base image photographing condition acquisition step and the base image into a mathematical model, the photographing conditions under which the base image was photographed by the OCT device can be calculated. and causing the ophthalmological image processing device to execute a target image acquisition step of acquiring a target image corresponding to an image with different imaging conditions other than the number of added images, which is the number of ophthalmological images used for averaging (≧1). , the mathematical model uses the data of some of the training ophthalmology images taken of the same part of the tissue as input training data, and the imaging conditions other than the number of images to be added are the input training data. Training is performed using data of the training ophthalmological images that are different from the shooting conditions of the training data as output training data.
A third aspect of the ophthalmologic image processing program provided by the exemplary embodiment of the present disclosure is an ophthalmologic image processing program executed by an ophthalmologic image processing apparatus that processes an ophthalmologic image that is an image of a tissue of a subject's eye, The ophthalmologic image processing program is executed by the control unit of the ophthalmologic image processing device, so that an ophthalmologic image captured by the OCT device or a plurality of ophthalmologic images captured by the OCT device of the same region of tissue are averaged. a base image acquisition step of acquiring the calculated averaged image as a base image; a range input step of accepting an input for specifying a part of the entire image range; By inputting an image, the imaging conditions other than the number of ophthalmologic images used for averaging (≧1) are changed to the imaging conditions under which the base image was captured by the OCT device. the ophthalmological image processing device executes a target image acquisition step of acquiring target images corresponding to different images in the range input in the range input step; Among the training ophthalmology images, data of some of the training ophthalmology images are used as input training data, and data of the training ophthalmology images whose imaging conditions other than the number of additional images are different from the imaging conditions of the input training data. is trained as the output training data.

According to the ophthalmologic image processing device and the ophthalmologic image processing program according to the present disclosure, useful images are obtained from images that are actually photographed.

1 is a block diagram showing a schematic configuration of a mathematical model construction device 1, an ophthalmological image processing device 21, and ophthalmological image photographing devices 11A and 11B. 3 is a flowchart of a mathematical model construction process executed by the mathematical model construction device 1. FIG. It is an explanatory view for explaining an example of an acquisition pattern of training data for input and training data for output. It is a flowchart of the imaging condition change process executed by the ophthalmologic image processing device 21. 3 is a diagram showing an example of a condition specification screen 30 and a target image display screen 40. FIG.

The control unit of the ophthalmologic image processing apparatus exemplified in this disclosure executes a base image acquisition step and a target image acquisition step. In the base image acquisition step, the control unit acquires, as the base image, an ophthalmologic image captured by the OCT device, or an averaged image obtained by adding and averaging a plurality of ophthalmologic images captured by the OCT device at the same site of tissue. In the target image acquisition step, the control unit acquires the target image by inputting the base image to a mathematical model trained by a machine learning algorithm. The mathematical model uses the data of some of the training ophthalmological images taken of the same part of the tissue as the input training data, and also sets the imaging conditions other than the number of additional images as the input training data. Training is performed using training ophthalmological image data different from the imaging conditions as output training data. The target image to be acquired is an image with different imaging conditions than the imaging conditions under which the base image was taken by the OCT device, except for the number of additional images, which is the number of ophthalmological images used for averaging (≧1). Equivalent to.

According to the ophthalmological image processing device exemplified in the present disclosure, a target image corresponding to an image whose imaging conditions other than the number of added images are different from the imaging conditions of the base image is acquired from the base image actually captured by the OCT device. Therefore, an image corresponding to an image photographed under photographing conditions useful to the user is easily obtained.

Note that this disclosure exemplifies an ophthalmologic image processing apparatus that processes an ophthalmologic image captured by an OCT device. However, the technology exemplified in the present disclosure uses ophthalmologic images captured by an ophthalmologic imaging device other than an OCT device (for example, at least one of a scanning laser ophthalmoscope (SLO), a fundus camera, and a corneal endothelial cell imaging device). It can also be applied to an ophthalmological image processing device that processes images.

The imaging conditions other than the number of additional images include the optical path length of at least one of the measurement light and reference light in the OCT device, the polarization state of at least one of the measurement light and reference light, the exposure time of the light receiving element when acquiring the OCT signal, It may be at least one of the number of A-scan data acquired when scanning one scan line with measurement light and the resolution in the A-scan direction. In this case, a target image corresponding to an image photographed under photographing conditions different from the photographing conditions of the base image is appropriately acquired by the ophthalmological image processing apparatus.

Specifically, when the imaging condition to be changed by a mathematical model is the optical path length, the position clearly reflected in the image changes among the positions in the depth direction of the tissue. When the photographing condition to be changed is the polarization state, the difference in tissue structure appearing in the image changes depending on the polarization state. When the photographing condition to be changed is the exposure time, the brightness of the image due to the exposure time changes. When the photographing condition to be changed is the number of A-scan data for each scan line, the resolution of the image in the scan line direction (so-called B-scan direction) changes. When the imaging condition to be changed is the resolution in the A-scan direction, the resolution of the image in the depth direction of the tissue changes. Note that the photographing conditions changed by the mathematical model can be different from the optical path length, polarization state, exposure time, number of A-scan data, and resolution in the A-scan direction.

The input training data used for training the mathematical model may be data based on L (L≧1) training ophthalmology images among a plurality of training ophthalmology images taken of the same part of the tissue. In addition, the output training data used for training the mathematical model is an H The data may be an average image of training ophthalmological images (H>L). In this case, a target image in which the influence of noise is suppressed and corresponds to an image captured under different imaging conditions is easily obtained from a base image actually captured by the OCT device using a mathematical model.

The control unit may further execute a denoising step of suppressing the influence of noise on the base image or the acquired target image. In this case, in addition to changing the photographing conditions other than the number of added images, the influence of noise included in the images is suppressed. Therefore, more useful ophthalmological images are obtained.

Note that when the target image acquisition step and the denoising step are executed separately, the specific method of the denoising step can be selected as appropriate. For example, the control unit inputs the base image or the target image to a mathematical model trained in advance using the averaged image as an output training model, thereby obtaining an image in which the influence of noise on the base image or the target image is suppressed. You may. Further, the control unit may suppress the influence of noise on the base image or the target image using other methods that do not use a mathematical model (for example, averaging processing or filter processing). The control unit may execute the target image acquisition step after suppressing the influence of noise on the base image.

The control unit may further execute a photographing condition input step of accepting an input of an instruction to designate a photographing condition other than the number of images to be added. The control unit may acquire a target image corresponding to an image photographed under the input photographing conditions by inputting the photographing conditions input in the target image acquisition step and the base image into a mathematical model. That is, the control unit may obtain a target image obtained by converting the base image using the input photographing conditions as image conversion conditions. In this case, a target image corresponding to an image photographed under the photographing conditions desired by the user is appropriately acquired.

The input photographing condition may be the type of photographing condition to be changed, or may be a numerical value specifying the photographing condition of the target image. The type of photographing condition, numerical value, etc. may be input together. Alternatively, the shooting conditions may be input by the user selecting a desired shooting condition from among a plurality of shooting condition candidates.

However, the photographing conditions (at least one of the type and numerical value of the photographing conditions) to be changed may be determined in advance without the user specifying the photographing conditions.

The control unit may further execute a base image photographing condition acquisition step of acquiring photographing conditions under which the base image was photographed by the OCT apparatus. The control unit may acquire the target image by inputting the base image and the base image photographing conditions acquired in the base image photographing condition acquisition step into a mathematical model. In this case, the quality of the target image output from the mathematical model is improved compared to the case where the photographing conditions of the base image are not input to the mathematical model.

Further, the control unit may acquire the target image by inputting the base image and noise information indicating the degree of noise in the base image to a mathematical model. In this case, noise in the target image output from the mathematical model is more appropriately reduced than when noise information is not input to the mathematical model. Note that the noise information may be a value indicating the degree of noise in the base image. Furthermore, when the base image is an averaged image, the noise information may be the number of images averaged when generating the base image.

Note that the process of inputting the photographing conditions and noise information of the base image into the mathematical model may be omitted. Even in this case, a target image corresponding to an image photographed under photographing conditions different from the photographing conditions of the base image is appropriately acquired.

The control unit may execute a range input step of accepting an input of an instruction to designate a part of the entire image range. The control unit may acquire a target image in which the imaging conditions have been changed within the input range. In this case, the imaging conditions are changed in a range desired by the user out of the entire image range. Therefore, images that are more useful to the user are obtained.

In the following, by inputting the base image into a mathematical model, the influence of noise is suppressed compared to the base image, and the base image is taken under different shooting conditions from the one under which the base image was shot by the OCT device. A case in which a target image corresponding to a captured image is acquired will be exemplified. However, a mathematical model may also be used to obtain a target image that is not affected by noise from the base image and corresponds to an image photographed under different photographing conditions from those of the base image. In this case, the mathematical model uses, as input training data, an image based on one or more training ophthalmology images out of a plurality of training ophthalmology images taken of the same part of the tissue, and also uses input training data as input training data. The image may be trained using an image based on one or more training ophthalmology images taken under conditions different from those under which the image was taken as the output training data.

<Embodiment>
(Device configuration)
One typical embodiment of the present disclosure will be described below with reference to the drawings. As shown in FIG. 1, in this embodiment, a mathematical model construction device 1, an ophthalmological image processing device 21, and ophthalmological image capturing devices 11A and 11B are used. The mathematical model construction device 1 constructs a mathematical model by training the mathematical model using a machine learning algorithm. Based on the input base image, the constructed mathematical model generates a target image that corresponds to an image that was photographed under different photographing conditions (photographing conditions other than the number of additional images) when the base image was photographed. Output. The ophthalmological image processing device 21 uses a mathematical model to obtain a target image from a base image. The ophthalmologic image capturing devices 11A and 11B capture ophthalmologic images that are images of tissues of the eye to be examined.

As an example, a personal computer (hereinafter referred to as "PC") is used as the mathematical model construction device 1 of this embodiment. Although details will be described later, the mathematical model construction device 1 constructs a mathematical model by training the mathematical model using the ophthalmologic images acquired from the ophthalmologic image capturing device 11A. However, the device that can function as the mathematical model construction device 1 is not limited to a PC. For example, the ophthalmologic image capturing device 11A may function as the mathematical model construction device 1. Further, the control units of a plurality of devices (for example, the CPU of the PC and the CPU 13A of the ophthalmological image capturing apparatus 11A) may cooperate to construct the mathematical model.

Further, a PC is used as the ophthalmological image processing apparatus 21 of this embodiment. However, the device that can function as the ophthalmological image processing device 21 is not limited to a PC either. For example, the ophthalmologic image capturing device 11B, a server, or the like may function as the ophthalmologic image processing device 21. When the ophthalmologic image capturing device 11B (OCT device in this embodiment) functions as the ophthalmologic image processing device 21, the ophthalmologic image capturing device 11B captures an ophthalmologic image while changing the capturing conditions for the captured ophthalmologic image. ophthalmological images can be appropriately acquired. Further, a mobile terminal such as a tablet terminal or a smartphone may function as the ophthalmological image processing device 21. The control units of a plurality of devices (for example, the CPU of the PC and the CPU 13B of the ophthalmological image capturing apparatus 11B) may cooperate to perform various processes.

Further, in this embodiment, a case will be described in which a CPU is used as an example of a controller that performs various processes. However, it goes without saying that a controller other than the CPU may be used for at least some of the various devices. For example, a GPU may be used as the controller to speed up the processing.

The mathematical model construction device 1 will be explained. The mathematical model construction device 1 is placed, for example, at a manufacturer that provides the ophthalmologic image processing device 21 or the ophthalmologic image processing program to users. The mathematical model construction device 1 includes a control unit 2 that performs various control processes and a communication I/F 5. The control unit 2 includes a CPU 3, which is a controller that controls, and a storage device 4 that can store programs, data, and the like. The storage device 4 stores a mathematical model building program for executing a mathematical model building process (see FIG. 2), which will be described later. Further, the communication I/F 5 connects the mathematical model construction device 1 to other devices (for example, the ophthalmologic image capturing device 11A and the ophthalmologic image processing device 21, etc.).

The mathematical model construction device 1 is connected to an operation section 7 and a display device 8. The operation unit 7 is operated by the user in order for the user to input various instructions to the mathematical model construction device 1. As the operation unit 7, for example, at least one of a keyboard, a mouse, a touch panel, etc. can be used. Note that a microphone or the like for inputting various instructions may be used together with or in place of the operation section 7. The display device 8 displays various images. As the display device 8, various devices capable of displaying images (for example, at least one of a monitor, a display, a projector, etc.) can be used. Note that "image" in the present disclosure includes both still images and moving images.

The mathematical model construction device 1 can acquire ophthalmologic image data (hereinafter sometimes simply referred to as "ophthalmologic image") from the ophthalmologic image capturing device 11A. The mathematical model construction device 1 may acquire ophthalmologic image data from the ophthalmologic image capturing device 11A through at least one of wired communication, wireless communication, a removable storage medium (for example, a USB memory), and the like.

The ophthalmologic image processing device 21 will be explained. The ophthalmological image processing device 21 is placed, for example, in a facility (for example, a hospital or a medical examination facility) that performs diagnosis or examination of subjects. The ophthalmologic image processing device 21 includes a control unit 22 that performs various control processes, and a communication I/F 25. The control unit 22 includes a CPU 23 that is a controller that controls control, and a storage device 24 that can store programs, data, and the like. The storage device 24 stores an ophthalmological image processing program for executing ophthalmological image processing (for example, photographing condition changing processing illustrated in FIG. 4), which will be described later. The ophthalmologic image processing program includes a program that realizes the mathematical model constructed by the mathematical model construction device 1. The communication I/F 25 connects the ophthalmologic image processing device 21 to other devices (for example, the ophthalmologic image capturing device 11B and the mathematical model construction device 1).

The ophthalmological image processing device 21 is connected to an operation section 27 and a display device 28. As with the operation section 7 and display device 8 described above, various devices can be used for the operation section 27 and the display device 28.

The ophthalmologic image processing device 21 can acquire ophthalmologic images from the ophthalmologic image capturing device 11B. The ophthalmological image processing device 21 may acquire ophthalmological images from the ophthalmological image capturing device 11B, for example, by wired communication, wireless communication, a removable storage medium (for example, a USB memory), or the like. Further, the ophthalmological image processing device 21 may acquire a program or the like that realizes the mathematical model constructed by the mathematical model construction device 1 via communication or the like.

The ophthalmologic image capturing devices 11A and 11B will be explained. As an example, in this embodiment, a case will be described in which an ophthalmologic image capturing device 11A that provides ophthalmologic images to the mathematical model construction device 1 and an ophthalmologic image capturing device 11B that provides ophthalmologic images to the ophthalmologic image processing device 21 are used. . However, the number of ophthalmological imaging devices used is not limited to two. For example, the mathematical model construction device 1 and the ophthalmology image processing device 21 may acquire ophthalmology images from a plurality of ophthalmology image capturing devices. Further, the mathematical model construction device 1 and the ophthalmologic image processing device 21 may acquire ophthalmologic images from one common ophthalmologic image capturing device. Note that the two ophthalmologic image capturing apparatuses 11A and 11B illustrated in this embodiment have the same configuration. Therefore, below, the two ophthalmological image capturing apparatuses 11A and 11B will be collectively described.

Moreover, in this embodiment, an OCT device is illustrated as the ophthalmologic image capturing device 11 (11A, 11B). However, an ophthalmologic imaging device other than the OCT device (for example, a laser scanning ophthalmoscope (SLO), a fundus camera, or a corneal endothelial cell imaging device (CEM)) may be used.

The ophthalmologic image capturing device 11 (11A, 11B) includes a control unit 12 (12A, 12B) that performs various control processes, and an ophthalmologic image capturing section 16 (16A, 16B). The control unit 12 includes a CPU 13 (13A, 13B) that is a controller that controls control, and a storage device 14 (14A, 14B) that can store programs, data, and the like.

The ophthalmological image photographing unit 16 includes various configurations necessary for photographing an ophthalmological image of the eye to be examined. Although not shown, the ophthalmologic image capturing unit 16 includes an OCT light source and a branching optical element that branches the OCT light emitted from the OCT light source into measurement light and reference light. Furthermore, the ophthalmological image capturing section 16 of this embodiment includes a light receiving element 17 (17A, 17B), a scanner 18 (18A, 18B), an optical path length adjustment section 19 (19A, 19B), a polarization state adjustment section 20 (20A, 20B), and a polarization state adjustment section 20 (20A, 20B). ). The light receiving element 17 receives the combined light of the light and the reference light by the tissue of the eye to be examined. The scanner 18 scans the measurement light over the tissue of the eye to be examined. The optical path length adjustment section 19 adjusts the optical path length of at least one of the measurement light and the reference light. The polarization state adjustment unit 20 adjusts the polarization state of at least one of the measurement light and the reference light.

The ophthalmologic image capturing device 11 can capture two-dimensional tomographic images and three-dimensional tomographic images of the fundus of the eye to be examined. Specifically, the CPU 13 scans the scan line with OCT light (measuring light) to capture a two-dimensional tomographic image of a cross section intersecting the scan line. Further, the CPU 13 can capture a three-dimensional tomographic image of the tissue by two-dimensionally scanning the OCT light. For example, the CPU 13 acquires a plurality of two-dimensional tomographic images by scanning measurement light on each of a plurality of scan lines at different positions within a two-dimensional region when the tissue is viewed from the front. Next, the CPU 13 obtains a three-dimensional tomographic image by combining the plurality of captured two-dimensional tomographic images.

Further, the CPU 13 can photograph an ophthalmological image by changing various photographing conditions. For example, by adjusting the optical path length using the optical path length adjustment unit 19, the CPU 13 can change the position that is clearly reflected in the tomographic image among the positions in the depth direction of the tissue. By adjusting the polarization state using the polarization state adjustment unit 20, the CPU 13 can change the structure of the tissue appearing in the image according to the polarization state. By adjusting the exposure time of the light receiving element 17, the CPU 13 can change the brightness of the ophthalmological image due to the exposure time. The CPU 13 can change the resolution of the image in the scan line direction (B scan direction) by adjusting the number of A scan data (data in the depth direction) for each scan line. The CPU 13 can also change the resolution of the image in the A-scan direction.

Furthermore, the CPU 13 scans the measurement light multiple times over the same site on the tissue (in this embodiment, on the same scan line) to capture multiple ophthalmological images of the same site. The CPU 13 can obtain an average image in which the influence of speckle noise is suppressed by performing averaging processing on a plurality of ophthalmological images of the same site. The averaging process may be performed, for example, by averaging pixel values of pixels at the same position among a plurality of ophthalmological images. The larger the number of images subjected to the averaging process, the more likely it is to suppress the influence of speckle noise, but the longer the imaging time will be. Note that while the ophthalmologic image capturing apparatus 11 captures a plurality of ophthalmologic images of the same site, it executes a tracking process that causes the scanning position of the OCT light to follow the movement of the subject's eye.

(mathematical model construction process)
The mathematical model construction process executed by the mathematical model construction device 1 will be described with reference to FIGS. 2 and 3. The mathematical model construction process is executed by the CPU 3 according to the mathematical model construction program stored in the storage device 4.

In the mathematical model construction process, a mathematical model is trained using a training dataset, which corresponds to an image taken under shooting conditions different from the shooting conditions (shooting conditions other than the number of added images) when the base image was shot. A mathematical model is constructed for outputting a target image (that is, corresponding to an image with changed imaging conditions). Furthermore, the mathematical model exemplified in this disclosure outputs a target image in which the influence of noise is suppressed more than the target image. Although details will be described later, the training data set includes input training data and output training data. In this embodiment, data based on one to a plurality of ophthalmological images (in this embodiment, a two-dimensional tomographic image of the fundus) is used as input training data. In addition, as the output training data, data based on ophthalmological images taken under imaging conditions different from the imaging conditions other than the number of additional images when the input training data was taken is used. Furthermore, in this embodiment, data of an average image based on a larger number (H) of ophthalmological images than the number (L) of ophthalmological images used for input training data is used as output training data. Ru. The input training data and the output training data target the same part of the tissue.

As shown in FIG. 2, the CPU 3 acquires an ophthalmological image I for input training data (S1). In this embodiment, a two-dimensional tomographic image captured by scanning the fundus of the eye to be examined with OCT light is acquired from the ophthalmologic image capturing apparatus 11A as the ophthalmologic image I for input training data. However, the CPU 3 acquires a signal (for example, an OCT signal) that is the basis for generating a two-dimensional tomographic image from the ophthalmological image capturing device 11A, and generates a two-dimensional tomographic image based on the acquired signal. Images may also be acquired.

In the example shown in FIG. 3, data of one ophthalmological image 30I photographed under the first photographing condition is acquired as input training data. However, data of a plurality of ophthalmological images 30I taken of the same part of the tissue, or data of an average image of the plurality of ophthalmological images 30I taken of the same part of the tissue may be acquired as input training data. By acquiring the average image data as the input training data, the image quality of the target image output by the mathematical model is improved compared to the case where the data of one ophthalmological image 30I is acquired as the input training data. do.

Note that the input training data acquired in S1 of this embodiment includes the first imaging conditions (optical path length, polarization state, exposure time, scan line by scan line) other than the number of additional images when the ophthalmological image I was taken. At least one of the number of A-scan data and the resolution in the A-scan direction) and noise information indicating the degree of noise in the ophthalmological image I are included. The first imaging condition may be acquired from the ophthalmologic image capturing device 11A. Furthermore, information indicating the degree of noise in the ophthalmological image I may be automatically acquired from the ophthalmological image I, or may be input by the user.

Next, the CPU 3 captures the same part of the subject's eye that is the same as the input training data (ophthalmological image I) under imaging conditions (second imaging conditions) different from the imaging conditions (first imaging conditions) other than the number of additional images of the input training data. A plurality of ophthalmological images O taken under conditions) are acquired as ophthalmological images for output training data (S2). In other words, the CPU 3 captures a plurality of ophthalmologic images O captured by scanning the OCT light on the same scan line as the scan line scanned by the OCT light when capturing the input training data. It is acquired from the device 11A.

In the example shown in FIG. 3, H ophthalmological images O (30O1 to 30OH), which are larger than the number of ophthalmological images 30I (L images) used as input training data, are acquired as ophthalmological images for output training data. be done. For example, when an average image of L ophthalmological images 30I is acquired as input training data, H ophthalmological images are larger than the number (L) of ophthalmological images 30I used for the average image. O is obtained in S2.

In this embodiment, the optical path described above is determined between the photographing conditions for the ophthalmological image I for input training data (first photographing condition) and the photographing condition for the ophthalmological image O for output training data (second photographing condition). One or more of the length, polarization state, exposure time, number of A-scan data per scan line, and resolution in the A-scan direction are different.

Next, the CPU 3 acquires the average image 31 of the plurality of ophthalmological images O for output training data as output training data (S3). The number H of ophthalmological images O to be used is larger than the number L of ophthalmological images I used for input training data. Note that the average image 31 may be generated by the mathematical model construction device 1. Furthermore, the mathematical model construction device 1 may acquire the averaged image 31 generated by the ophthalmologic image capturing device 11A.

Note that the output training data acquired in S1 and S3 of this embodiment includes the second imaging conditions (optical path length, polarization state, exposure time, A At least one of the number of scan data and the resolution in the A-scan direction) and noise information indicating the degree of noise in the averaged image 31 are included. The second imaging condition may be acquired from the ophthalmologic image capturing device 11A. Furthermore, information indicating the degree of noise in the averaged image 31 may be automatically acquired from the averaged image 31, or may be input by the user. Information on the number of ophthalmological images O used in the averaged image 31 may be acquired as information indicating the degree of noise.

Next, the CPU 3 executes training of a mathematical model using the training data set using a machine learning algorithm (S4). As machine learning algorithms, for example, neural networks, random forests, boosting, support vector machines (SVM), etc. are generally known.

Neural networks are a method that imitates the behavior of biological nerve cell networks. Neural networks include, for example, feedforward neural networks, RBF networks (radial basis functions), spiking neural networks, convolutional neural networks, recurrent neural networks (recurrent neural networks, feedback neural networks, etc.), and stochastic neural networks. There are various types of neural networks (Boltzmann machines, Bassian networks, etc.).

Random forest is a method of generating a large number of decision trees by performing learning based on randomly sampled training data. When using random forest, branches of multiple decision trees trained in advance as a classifier are followed, and the average (or majority vote) of the results obtained from each decision tree is taken.

Boosting is a method of generating a strong classifier by combining multiple weak classifiers. A strong classifier is constructed by sequentially training simple and weak classifiers.

SVM is a method of constructing a two-class pattern discriminator using linear input elements. For example, the SVM learns the parameters of a linear input element based on the criterion (hyperplane separation theorem) of finding a margin-maximizing hyperplane that maximizes the distance to each data point from training data.

A mathematical model refers to, for example, a data structure for predicting the relationship between input data and output data. A mathematical model is constructed by being trained using a training dataset. The training data set is a set of input training data and output training data. A mathematical model is trained such that when certain input training data is input, corresponding output training data is output. For example, training updates correlation data (eg, weights) between each input and output.

In this embodiment, a multilayer neural network is used as a machine learning algorithm. A neural network includes an input layer for inputting data, an output layer for generating data to be predicted, and one or more hidden layers between the input layer and the output layer. A plurality of nodes (also called units) are arranged in each layer. Specifically, in this embodiment, a convolutional neural network (CNN), which is a type of multilayer neural network, is used. However, other machine learning algorithms may be used. For example, generative adversarial networks (GAN), which utilize two competing neural networks, may be employed as the machine learning algorithm.

In addition, in S4 of this embodiment, as described above, the first photographing condition under which the ophthalmological image I was photographed and information indicating the degree of noise in the ophthalmological image I are included in the input training data. Further, the second photographing conditions under which the plurality of ophthalmological images O were photographed and information indicating the degree of noise in the averaged image 31 are included in the output training data. As a result, the mathematical model can appropriately output the target image according to the first imaging condition and the degree of noise of the input base image.

In addition, among a plurality of imaging conditions other than the number of additional images (in this embodiment, optical path length, polarization state, exposure time, number of A-scan data for each scan line, and resolution in the A-scan direction), input training data Each of the plurality of mathematical models may be trained according to any of the imaging conditions that are different between the ophthalmological image I for use as an output training data and the ophthalmological image O for output training data. That is, a mathematical model may be trained separately for each imaging condition that is changed using the mathematical model. Furthermore, the mathematical model may be trained by changing a plurality of imaging conditions other than the number of images to be added between the ophthalmological image I for input training data and the ophthalmological image O for output training data. In this case, the mathematical model can output a target image in which a plurality of photographing conditions are changed with respect to the photographing conditions of the base image, based on the input base image.

The processes of S1 to S4 are repeated until construction of the mathematical model is completed (S5: NO). When construction of the mathematical model is completed (S5: YES), the mathematical model construction process ends. A program and data for realizing the constructed mathematical model are incorporated into the ophthalmological image processing device 21.

(Photographing condition change processing)
The photographing condition changing process executed by the ophthalmological image processing apparatus 21 will be described with reference to FIGS. 4 and 5. The photographing condition changing process is executed by the CPU 23 according to the ophthalmological image processing program stored in the storage device 24. In the photographing condition change process of this embodiment, by inputting the base image into the mathematical model, the influence of noise is suppressed compared to the base image, and the photographing conditions (other than the number of additional images) under which the base image was photographed are A target image corresponding to an image photographed under photographing conditions different from the photographing conditions) is acquired.

First, the CPU 23 acquires the base image 31 (see FIG. 5) captured by the ophthalmologic image capturing device (OCT device in this embodiment) 11B. The base image 31 of this embodiment, like the input training data described above, is a single ophthalmological image taken under the first imaging condition, or a plurality of ophthalmological images taken under the first imaging condition. This is an average image.

The CPU 23 acquires the photographing conditions other than the number of additional images (photographing conditions for the base image 31) when the base image 31 was photographed by the ophthalmological image photographing device 11B, and noise information indicating the degree of noise in the base image 31 ( S2). The photographing conditions for the base image 31 may be acquired via wired communication, wireless communication, a removable storage medium, or the like, or may be input by the user. Further, the noise information may be automatically acquired from the base image 31 or may be input by the user. When the base image 31 is an averaged image, the noise information may be the number of images that are averaged when generating the base image 31 (added number of images).

The CPU 23 causes the display device 28 to display the base image 31 and the condition specifying section 32 acquired in S1 (S3). FIG. 5 shows an example of the condition specification screen 30 on which the base image 31 and the condition specification section 32 are displayed. The condition specifying unit 32 stores a plurality of shooting conditions (in this embodiment, shooting conditions for the base image 31 and shooting conditions for the target image specified by the user (that is, shooting conditions to be changed from the shooting conditions for the base image 31). , optical path length, polarization state, exposure time, number of A-scan data per scan line, and resolution in the A-scan direction). Note that in this embodiment, the added number of base images 31 and target images is also displayed. The user can specify the shooting conditions for the target image as appropriate by specifying the field for the shooting conditions to be changed and inputting a desired value. Note that among the plurality of shooting conditions, a "-" is displayed in the column of the shooting conditions that the user does not specify to change.

The CPU 23 determines whether an instruction specifying photographing conditions has been input (S5). If it has not been input (S5: NO), the process moves directly to S8. When an instruction to designate photographing conditions is input (S5: YES), the CPU 23 sets the designated photographing conditions as the photographing conditions of the target image (that is, the image conversion conditions), and changes the display of the condition specifying section 32. Change (S6).

The CPU 23 determines whether a range designation instruction has been input by the user (S8). In this embodiment, the user can specify a change range 33 (see FIG. 5) in which to change the shooting conditions of the base image 31. When the change range 33 is specified, a target image in which the influence of noise is suppressed and the photographing conditions are changed is acquired within the specified change range out of the entire image range of the base image 31. If the change range 33 has not been specified (S8: NO), the process directly proceeds to S11. When the change range 33 is specified (S8: YES), the CPU 23 sets the specified change range 33 as the change range of the photographing conditions (S9).

The CPU 23 determines whether the user has input an instruction to determine photographing conditions (S11). If it has not been input (S11: NO), the process returns to S5, and the processes from S5 to S11 are repeated. When the user operates the decision button 34 (see FIG. 5) and inputs a decision instruction (S11: YES), the CPU 23 inputs the base image 31, the shooting conditions of the base image 31, the noise information of the base image 31, and the specified The photographing conditions for the target image are input into the mathematical model (S12). As a result, the influence of noise is suppressed more than in the base image 31, and the image is captured under different capture conditions (in this embodiment, specified capture conditions) than the capture conditions under which the base image 31 was captured. A target image corresponding to is output by the mathematical model. The CPU 23 causes the display device 28 to display the acquired target image. FIG. 5 shows an example of the target image display screen 40. In the displayed target image 41, the influence of noise is suppressed. Furthermore, the shooting conditions for the target image 41 have been changed to different shooting conditions from the shooting conditions for the base image 31.

Note that when the change range 33 is set in S9, in S12 of this embodiment, the image within the change range 33 set in S9 out of the entire image range of the base image 31 is input to the mathematical model. As a result, the influence of noise and the photographing conditions are changed in the range desired by the user among the entire image range of the base image 31.

The techniques disclosed in the above embodiments are merely examples. Therefore, it is also possible to modify the techniques exemplified in the above embodiments. First, it is also possible to execute only some of the techniques exemplified in the above embodiments. For example, by inputting the base image 31 into a mathematical model, the ophthalmological image processing device 21 can perform imaging under different photographing conditions than the one under which the base image 31 was photographed, without changing the influence of noise on the base image 31. You may acquire the target image 41 corresponding to the image photographed in . In this case, the ophthalmological image processing device 21 may perform a denoising process to suppress the influence of noise on the base image 31 or the target image 41, in addition to the process of acquiring the target image 41 with changed imaging conditions. The denoising process may be performed by using a mathematical model for suppressing the influence of image noise, which is provided separately from the mathematical model for acquiring the target image 41. Further, the denoising process may be performed by known processing (for example, averaging processing or filter processing) without using a mathematical model.

Furthermore, in the embodiment described above, the user can freely input desired imaging conditions into the ophthalmological image processing apparatus 21. The ophthalmological image processing device 21 obtains the target image 41 by converting the base image 31 using the input imaging conditions as conversion conditions. However, the ophthalmological image processing device 21 automatically sets a combination of imaging conditions that provides good image quality of the target image 41 from among a large number of combinations of imaging conditions, and uses the set combination of imaging conditions as a conversion condition for the purpose. An image 41 may also be acquired.

Note that the process of acquiring the base image 31 in S1 in FIG. 4 is an example of a "base image acquisition step." The process of acquiring the target image in S12 of FIG. 4 is an example of a "target image acquisition step." The process of accepting the input of the instruction to designate the imaging conditions in S5 of FIG. 4 is an example of the "imaging condition input step." The process of acquiring the photographing conditions for the base image 31 in S2 of FIG. 4 is an example of the "base image photographing condition acquisition step." The process of accepting the input of the change range 33 designation instruction in S8 of FIG. 4 is an example of a "range input step."

1 Mathematical model construction device 11A, 11B Ophthalmology image photographing device 17A, 17B Light receiving element 18A, 18B Scanner 19A, 19B Optical path length adjustment section 20A, 20B Polarization state adjustment section 21 Ophthalmology image processing device 23 CPU
24 Storage device 28 Display device 31 Base image 41 Target image

Claims (6)

An ophthalmological image processing device that processes an ophthalmological image that is an image of a tissue of an eye to be examined,
The control unit of the ophthalmological image processing device includes:
a base image acquisition step of acquiring, as a base image, an ophthalmological image taken by an OCT device, or an averaged image obtained by adding and averaging a plurality of ophthalmological images taken of the same part of tissue by the OCT device;
an imaging condition input step that receives an input of an instruction to designate an imaging condition other than the number of additional images, which is the number of ophthalmological images used for the averaging (≧1);
By inputting the photographing conditions input in the photographing condition input step and the base image to a mathematical model trained by a machine learning algorithm, images photographed under the photographing conditions other than the input additional number of images can be created. a target image acquisition step of acquiring a corresponding target image;
Run
The mathematical model uses the data of some of the training ophthalmology images taken of the same part of the tissue as the input training data, and sets the imaging conditions other than the number of images for input as the input training data. An ophthalmological image processing apparatus, characterized in that the ophthalmological image processing apparatus is trained using data of the training ophthalmological image that is different from photographing conditions of the training data as output training data. An ophthalmological image processing device that processes an ophthalmological image that is an image of a tissue of an eye to be examined,
The control unit of the ophthalmological image processing device includes:
a base image acquisition step of acquiring, as a base image, an ophthalmological image taken by an OCT device, or an averaged image obtained by adding and averaging a plurality of ophthalmological images taken of the same part of tissue by the OCT device;
a base image photographing condition acquisition step of acquiring photographing conditions when the base image was photographed by the OCT device;
By inputting the photographing conditions of the base image acquired in the base image photographing condition acquisition step and the base image into a mathematical model trained by a machine learning algorithm, the base image is photographed by the OCT device. a target image acquisition step of acquiring a target image corresponding to an image with different photographing conditions other than the number of added images, which is the number of ophthalmological images used for averaging (≧1), with respect to the photographing conditions at the time of averaging;
Run
The mathematical model uses the data of some of the training ophthalmology images taken of the same part of the tissue as the input training data, and sets the imaging conditions other than the number of images for input as the input training data. An ophthalmological image processing apparatus, characterized in that the ophthalmological image processing apparatus is trained using data of the training ophthalmological image that is different from photographing conditions of the training data as output training data. An ophthalmological image processing device that processes an ophthalmological image that is an image of a tissue of an eye to be examined,
The control unit of the ophthalmological image processing device includes:
a base image acquisition step of acquiring, as a base image, an ophthalmological image taken by an OCT device, or an averaged image obtained by adding and averaging a plurality of ophthalmological images taken of the same part of tissue by the OCT device;
a range input step of accepting input of an instruction to specify a part of the entire image range;
By inputting the base image into a mathematical model trained by a machine learning algorithm, the number of ophthalmological images used for averaging (≧ 1) a target image acquisition step of acquiring a target image corresponding to a different image in a range in which photographing conditions other than the number of additional images are input in the range input step;
Run
The mathematical model uses the data of some of the training ophthalmology images taken of the same part of the tissue as the input training data, and sets the imaging conditions other than the number of images for input as the input training data. An ophthalmological image processing apparatus, characterized in that the ophthalmological image processing apparatus is trained using data of the training ophthalmological image that is different from photographing conditions of the training data as output training data. An ophthalmologic image processing program executed by an ophthalmologic image processing device that processes an ophthalmologic image that is an image of a tissue of an eye to be examined,
The ophthalmology image processing program is executed by the control unit of the ophthalmology image processing device,
a base image acquisition step of acquiring, as a base image, an ophthalmological image taken by an OCT device, or an averaged image obtained by adding and averaging a plurality of ophthalmological images taken of the same part of tissue by the OCT device;
an imaging condition input step that receives an input of an instruction to designate an imaging condition other than the number of additional images, which is the number of ophthalmological images used for the averaging (≧1);
By inputting the photographing conditions input in the photographing condition input step and the base image to a mathematical model trained by a machine learning algorithm, images photographed under the photographing conditions other than the input additional number of images can be created. a target image acquisition step of acquiring a corresponding target image;
causing the ophthalmological image processing device to execute
The mathematical model uses the data of some of the training ophthalmology images taken of the same part of the tissue as the input training data, and sets the imaging conditions other than the number of images for input as the input training data. An ophthalmological image processing program characterized in that the program is trained using data of the training ophthalmological image different from the shooting conditions of the training data as output training data. An ophthalmologic image processing program executed by an ophthalmologic image processing device that processes an ophthalmologic image that is an image of a tissue of an eye to be examined,
The ophthalmology image processing program is executed by the control unit of the ophthalmology image processing device,
a base image acquisition step of acquiring, as a base image, an ophthalmological image taken by an OCT device, or an averaged image obtained by adding and averaging a plurality of ophthalmological images taken of the same part of tissue by the OCT device;
a base image photographing condition acquisition step of acquiring photographing conditions when the base image was photographed by the OCT device;
By inputting the photographing conditions of the base image acquired in the base image photographing condition acquisition step and the base image into a mathematical model trained by a machine learning algorithm, the base image is photographed by the OCT device. a target image acquisition step of acquiring a target image corresponding to an image with different photographing conditions other than the number of added images, which is the number of ophthalmological images used for averaging (≧1), with respect to the photographing conditions at the time of averaging;
causing the ophthalmological image processing device to execute
The mathematical model uses the data of some of the training ophthalmology images taken of the same part of the tissue as the input training data, and sets the imaging conditions other than the number of images for input as the input training data. An ophthalmological image processing program characterized in that the program is trained using data of the training ophthalmological image different from the shooting conditions of the training data as output training data. An ophthalmologic image processing program executed by an ophthalmologic image processing device that processes an ophthalmologic image that is an image of a tissue of an eye to be examined,
The ophthalmology image processing program is executed by the control unit of the ophthalmology image processing device,
a base image acquisition step of acquiring, as a base image, an ophthalmological image taken by an OCT device, or an averaged image obtained by adding and averaging a plurality of ophthalmological images taken of the same part of tissue by the OCT device;
a range input step of accepting input of an instruction to specify a part of the entire image range;
By inputting the base image into a mathematical model trained by a machine learning algorithm, the number of ophthalmological images used for averaging (≧ 1) a target image acquisition step of acquiring a target image corresponding to a different image in a range in which photographing conditions other than the number of additional images are input in the range input step;
causing the ophthalmological image processing device to execute
The mathematical model uses the data of some of the training ophthalmology images taken of the same part of the tissue as the input training data, and sets the imaging conditions other than the number of images for input as the input training data. An ophthalmological image processing program characterized in that the program is trained using data of the training ophthalmological image different from the shooting conditions of the training data as output training data.

Images