WO2021045019A1 - Ophthalmic image processing program and ophthalmic image processing device - Google Patents

Abstract

A control unit of this ophthalmic image processing device executes an image acquisition step (S11), a transformed image acquisition step (S12), and an evaluation information acquisition step (S13). In the image acquisition step, the control unit acquires an ophthalmic image captured by an ophthalmic image imaging device. In the transformed image acquisition step, the control unit enters, as an input image, the ophthalmic image acquired in the image acquisition step to a mathematical model that has been trained by a machine learning algorithm, so as to acquire a transformed image that is obtained by transforming the image quality of the input image. In the evaluation information acquisition step, the control unit acquires evaluation information for evaluating the validity of transformation from the input image into the transformed image by using the mathematical model.

Description

Ophthalmic image processing program and ophthalmic image processing device

The present disclosure relates to an ophthalmic image processing program and an ophthalmic image processing apparatus used for processing an ophthalmic image of an eye to be inspected.

A technique for acquiring various medical information using a mathematical model trained by a machine learning algorithm has been proposed. For example, in the ophthalmic apparatus described in Patent Document 1, IOL-related information (for example, expected postoperative anterior chamber depth) of the eye to be inspected is acquired by inputting an eye shape parameter into a mathematical model trained by a machine learning algorithm. Will be done. The IOL frequency is calculated based on the acquired IOL-related information.

Further, in Non-Patent Document 1, by inputting an ophthalmic image as an input image into a mathematical model trained by a machine learning algorithm, a converted image obtained by converting the image quality of the input image is obtained.

JP-A-2018-51223

The conversion from the input image to the converted image by the mathematical model may not be executed properly. For example, when the ophthalmic image used for learning the mathematical model and the ophthalmic image actually input to the mathematical model are significantly different, it is difficult to properly convert the input image to the converted image. If the converted image is presented to the user as it is even though the conversion from the input image to the converted image is not properly performed, the user may not be able to accurately perform various judgments based on the converted image.

A typical object of the present disclosure is to provide an ophthalmic image processing program and an ophthalmic image processing apparatus capable of presenting more appropriate information to a user.

The ophthalmic image processing program provided by the typical embodiment in the present disclosure is an ophthalmic image processing program executed by an ophthalmic image processing apparatus that processes an ophthalmic image which is an image of a tissue of an eye to be examined, and is the ophthalmic image processing. When the program is executed by the control unit of the ophthalmic image processing device, the image acquisition step of acquiring the ophthalmic image taken by the ophthalmic image capturing device and the image acquisition step of the mathematical model trained by the machine learning algorithm. In the conversion image acquisition step of acquiring the converted image obtained by converting the image quality of the input image by inputting the ophthalmic image acquired in the above as the input image, and the conversion of the input image to the converted image by the mathematical model. The evaluation information acquisition step of acquiring the evaluation information for evaluating the validity is executed by the ophthalmic image processing apparatus.

The ophthalmic image processing apparatus provided by the typical embodiment in the present disclosure is an ophthalmic image processing apparatus that processes an ophthalmic image which is an image of a tissue of an eye to be inspected, and a control unit of the ophthalmic image processing apparatus is an ophthalmic image. The input image is obtained by inputting the ophthalmic image acquired in the image acquisition step as an input image into an image acquisition step for acquiring an ophthalmic image captured by an imaging device and a mathematical model trained by a machine learning algorithm. The conversion image acquisition step of acquiring the converted image obtained by converting the image quality of the above, and the evaluation information acquisition step of acquiring the evaluation information for evaluating the validity of the conversion from the input image to the converted image by the mathematical model are executed. ..

According to the ophthalmic image processing program and the ophthalmic image processing apparatus according to the present disclosure, more appropriate information is presented to the user.

The control unit of the ophthalmic image processing apparatus exemplified in the present disclosure executes an image acquisition step, a converted image acquisition step, and an evaluation information acquisition step. In the image acquisition step, the control unit acquires an ophthalmic image taken by the ophthalmologic image capturing device. In the converted image acquisition step, the control unit acquires the converted image obtained by converting the image quality of the input image by inputting the ophthalmic image acquired in the image acquisition step as an input image into the mathematical model trained by the machine learning algorithm. To do. In the evaluation information acquisition step, the control unit acquires evaluation information for evaluating the validity of the conversion from the input image to the converted image by the mathematical model.

According to the ophthalmic image processing apparatus exemplified in the present disclosure, evaluation information for evaluating the validity of conversion from an input image to a converted image by a mathematical model is acquired. Therefore, the ophthalmic image processing apparatus can present appropriate information to the user by using the evaluation information.

Various ophthalmic images can be used as the input image. For example, tomographic images taken by an OCT device (two-dimensional tomographic image or three-dimensional tomographic image), images taken by a fundus camera, images taken by a laser scanning eye examination device (SLO), and corneal endothelial cell imaging. At least one of the images taken by the device may be used as the input image. Further, the ophthalmologic image may be an OCT angio image of the fundus of the eye to be inspected taken by the OCT apparatus. The OCT angio image may be a two-dimensional front image in which the fundus is viewed from the front (that is, the line-of-sight direction of the eye to be inspected). The OCT angio image may be, for example, a motion contrast image acquired by processing at least two OCT signals acquired at different times with respect to the same position. Further, the ophthalmologic image is an Enface image (OCT front image) when at least a part of the three-dimensional tomographic image taken by the OCT device is viewed from the direction (front direction) along the optical axis of the measurement light of the OCT device. There may be.

Also, the image quality converted by the mathematical model can be selected as appropriate. For example, the control unit may acquire a converted image in which at least one of the noise amount, contrast, and resolution of the input image is converted by using a mathematical model.

In the evaluation information acquisition step, the control unit may acquire the difference information of the pixel values between the corresponding pixels of the input image input to the mathematical model and the converted image output from the mathematical model as evaluation information. .. When the image quality of the input image is converted appropriately, the difference between the input image and the converted image becomes small. Therefore, by acquiring the difference information as the evaluation information, the validity of the conversion from the input image to the converted image is appropriately evaluated. The difference information may be the difference in pixel values between the corresponding pixels, or may be the ratio of the other pixel value to one pixel value.

Note that the control unit may acquire the difference information after executing the Fourier transform (for example, two-dimensional Fourier transform) for each of the input image and the converted image in the evaluation information acquisition step. Periodic artifacts may occur in the transformed image. When periodic artifacts occur in the converted image, the input image and the converted image show different frequency distributions. Therefore, by using the Fourier transform, the presence or absence of periodic artifacts is evaluated more appropriately.

The control unit evaluates that the conversion from the input image to the converted image is not appropriate when the value of the difference in an arbitrary region including a plurality of pixels is equal to or more than the threshold value in the difference image obtained by imaging the difference information. One evaluation step may be performed. The difference image may be interspersed with pixels with large values, even if the conversion is performed properly. On the other hand, when the conversion is not properly executed due to the presence of the diseased part or the like, a region in which pixels having a large value are densely appears appears in the difference image. Therefore, the control unit compares the difference value in an arbitrary region including a plurality of pixels with the threshold value in the difference image to suppress the influence of the scattered pixels even when the conversion is properly executed. Then, the validity of the conversion can be evaluated appropriately.

Note that the control unit may evaluate the validity of the conversion based on the difference image after performing the smoothing process on the pixel value of the difference image (the value of the difference corresponding to each pixel). In this case, the influence of the scattered pixels is more appropriately suppressed even when the conversion is properly executed.

In addition, a specific method for evaluating the validity of conversion based on the value of the difference in the region can be appropriately selected. For example, the control unit may evaluate that the conversion is not valid when the average value of the differences in the region is equal to or greater than the threshold value. Further, the control unit may evaluate whether or not the conversion is appropriate based on the number of pixels in the unit region whose difference value is equal to or greater than the threshold value. Further, the control unit may acquire information indicating the degree of validity of the conversion based on the difference information. Information indicating the degree of validity may be displayed on the display unit.

In the evaluation information acquisition step, the control unit acquires a difference image that is an image of the difference information of the pixel values between the corresponding pixels of the input image input to the mathematical model and the converted image output from the mathematical model. , The degree of similarity between the difference image and the input image (for example, correlation) may be acquired as evaluation information. When the image quality of the input image is appropriately converted, the difference between the input image and the converted image becomes small, so that the similarity between the difference image and the input image becomes small. On the other hand, if the conversion of the input image is not properly executed and the irregular part or the like in the input image affects the conversion, the position of the irregular part or the like in the input image and the difference value in the difference image become large. Since the positions are similar, the similarity between the input image and the difference image is increased. Therefore, the validity of the conversion is appropriately evaluated by acquiring the similarity between the difference image and the input image as evaluation information. The method of acquiring the degree of similarity can be appropriately selected. For example, a correlation diagram may be acquired, or a correlation coefficient may be acquired.

The control unit may further perform a second evaluation step of evaluating that the conversion from the input image to the converted image is not appropriate when the value indicating the similarity is equal to or higher than the threshold value. As mentioned above, when the conversion is performed properly, the similarity between the input image and the difference image becomes small. On the other hand, if the conversion is not performed properly, the similarity between the input image and the difference image becomes large. Therefore, the control unit can appropriately evaluate whether or not the conversion from the input image to the converted image is appropriate by determining whether or not the value indicating the similarity is equal to or greater than the threshold value. As the value indicating the degree of similarity, various values (for example, correlation coefficient and the like) can be appropriately used.

The control unit may further execute a difference image display step of displaying a difference image in which the difference information is imaged on the display unit. In the difference image, there is a difference between the region where the conversion is properly performed and the region where the conversion is not performed properly. Therefore, the user can confirm whether or not the conversion is properly performed by confirming the difference image. Further, the user can grasp the area where the conversion is not properly performed by checking the difference image.

If the input image contains an irregular part (for example, a diseased part), it is difficult to properly convert the image quality of the irregular part. Therefore, the user can appropriately grasp the irregular portion in the input image by checking the difference image.

The specific method for displaying the difference image on the display unit can be selected as appropriate. For example, the display unit may display at least one of the input image and the converted image at the same time as the difference image (for example, side by side). Further, the control unit may superimpose and display the difference image on at least one of the input image and the converted image. In this case, the user can easily compare at least one of the input image and the converted image with the converted image. Further, the control unit may independently display the difference image on the display unit.

In the evaluation information acquisition step, the control unit inputs the input image and the converted image into a mathematical model trained by a machine learning algorithm (a mathematical model for acquiring evaluation information different from the mathematical model for acquiring the converted image). Then, the evaluation information may be acquired. In this case, the validity of the conversion is appropriately evaluated even if the difference between the input image and the converted image is not acquired.

The mode of evaluation information output by the mathematical model for acquiring evaluation information can also be selected as appropriate. For example, the mathematical model for acquiring evaluation information may output evaluation information indicating whether or not the conversion from the input image to the converted image is appropriate. In this case, it is easy to evaluate whether the conversion is valid or not. Further, the mathematical model for acquiring the evaluation information may output information such as a numerical value indicating the degree of validity of the conversion as the evaluation information.

It is also possible to change the mode of evaluation information. For example, the control unit may acquire evaluation information by using various parameters related to the image quality of the image (at least the converted image). Parameters related to image quality include, for example, the signal strength of an ophthalmic image, an index indicating the goodness of the signal (for example, SSI (Signal Contrast Index) or SQI (SLO Quality Index)), and noise with respect to the signal level of the image. At least one of the level ratio (SNR (Signal to Noise Radio), background noise level, image contrast, etc.) may be used as a parameter related to image quality. The input image is converted to improve the image quality of the input image. When acquiring a transformed image, if the conversion is performed properly, the image quality of the converted image should be better than the image quality of the input image. Therefore, for example, a parameter indicating the image quality of the converted image may be used. It may be acquired as evaluation information. Further, the difference between the parameter indicating the image quality of the converted image and the parameter indicating the image quality of the input image may be acquired as evaluation information.

The control unit may further execute a warning step that warns the user when the conversion is evaluated as invalid by the evaluation information acquired in the evaluation information acquisition step. In this case, the user can easily grasp that the conversion from the input image to the converted image may not have been performed properly.

The specific method of warning processing can be selected as appropriate. For example, the control unit may warn the user by displaying at least one of a warning message, a warning image, and the like on the display unit. Further, the control unit may issue a warning to the user by generating at least one of a warning message and a warning sound from the speaker. Further, the control unit may execute the warning process while displaying the actually converted converted image on the display unit, or may execute the warning process without displaying the converted image.

The control unit stops the display processing of the converted image acquired in the converted image acquisition step on the display unit when the evaluation information acquired in the evaluation information acquisition step evaluates that the conversion is not valid. May be further executed. In this case, it is suppressed that the converted image that has not been properly converted from the input image is displayed on the display unit. Therefore, the possibility that the user cannot accurately perform various judgments based on the converted image is reduced.

It is also possible to change the method of using the evaluation information. For example, the control unit may display at least one of a numerical value indicating the acquired evaluation information, a graph, and the like on the display unit. In this case, the user can easily grasp whether or not the conversion from the input image to the converted image is properly performed based on the displayed evaluation information. Further, as described above, the difference image between the input image and the converted image may be displayed on the display unit as evaluation information.

In addition, when the conversion is evaluated as invalid by the evaluation information, the control unit inputs the input image to a mathematical model different from the mathematical model that performed the conversion evaluated as invalid, so that the converted image is input. You may get it. The characteristics of conversion by the mathematical model differ depending on the algorithm and training data used when training the mathematical model. Therefore, if the transformation is evaluated as invalid, the transformed image may be acquired appropriately by acquiring the transformed image by a different mathematical model.

As described above, when the input image contains an irregular part (for example, a diseased part), it is difficult to properly convert the image quality of the irregular part. In this case, the position of the irregular portion appears in the difference image between the input image and the converted image. Therefore, the control unit may display the difference image on the display unit regardless of the validity of the conversion from the input image to the converted image. As a result, the user can easily grasp the position of the irregular portion based on the difference image.

In this case, the ophthalmic image processing program can be expressed as follows. It is an ophthalmic image processing program executed by an ophthalmic image processing apparatus that processes an ophthalmic image which is an image of the tissue of the eye to be inspected, and the ophthalmic image processing program is executed by a control unit of the ophthalmic image processing apparatus. By inputting the ophthalmic image acquired in the image acquisition step as an input image into the image acquisition step of acquiring the ophthalmic image taken by the ophthalmic image capturing apparatus and the mathematical model trained by the machine learning algorithm, the said A conversion image acquisition step of acquiring a converted image obtained by converting the image quality of the input image, and a difference in pixel value between the corresponding pixels of the input image input to the mathematical model and the converted image output from the mathematical model. An ophthalmic image processing program characterized in that a difference image acquisition step of acquiring a difference image obtained by imaging information and a difference image display step of displaying the difference image on a display unit are executed by the ophthalmic image processing apparatus. ..

The device that executes the image acquisition step, the converted image acquisition step, the evaluation information acquisition step, etc. can be appropriately selected. For example, the control unit of a personal computer (hereinafter referred to as “PC”) may execute all of the converted image acquisition step, the evaluation information acquisition step, and the like. That is, the control unit of the PC may acquire an ophthalmologic image from the ophthalmology image capturing apparatus and perform conversion image acquisition processing or the like based on the acquired ophthalmology image. Further, the control unit of the ophthalmic imaging apparatus may execute all of the conversion image acquisition step, the evaluation information acquisition step, and the like. Further, the control units of a plurality of devices (for example, an ophthalmologic image capturing device and a PC, etc.) may cooperate to execute the converted image acquisition step, the evaluation information acquisition step, and the like.

It is a block diagram which shows the schematic structure of the mathematical model construction apparatus 1, the ophthalmologic image processing apparatus 21, and the ophthalmology imaging apparatus 11A, 11B. It is a figure which shows an example of the training data for input and the training data for output in the case of outputting a high-quality two-dimensional tomographic image as a conversion image to a mathematical model. It is a flowchart of the mathematical model construction process executed by the mathematical model construction apparatus 1. It is a flowchart of ophthalmic image processing executed by ophthalmic image processing apparatus 21. It is a figure which shows an example of an ophthalmic image used as an input image. It is a figure which shows an example of the converted image which converted the image quality of the input image shown in FIG. It is a figure which shows an example of the difference image of the input image shown in FIG. 5 and the conversion image shown in FIG. It is explanatory drawing for demonstrating an example of the method of evaluating the validity of conversion from an input image to a conversion image based on a difference image. It is a flowchart of ophthalmic image processing in a transformation example.

(Device configuration)
Hereinafter, one of the typical embodiments in the present disclosure will be described with reference to the drawings. As shown in FIG. 1, in this embodiment, the mathematical model construction device 1, the ophthalmic image processing device 21, and the ophthalmic imaging devices 11A and 11B are used. The mathematical model construction device 1 constructs a mathematical model by training the mathematical model by a machine learning algorithm. The program that realizes the constructed mathematical model is stored in the storage device 24 of the ophthalmic image processing device 21. The ophthalmic image processing device 21 inputs an ophthalmic image as an input image into a mathematical model to acquire a converted image in which the image quality of the input image is converted (in the present embodiment, the image quality is improved). In addition, the ophthalmic image processing device 21 acquires evaluation information for evaluating the validity of conversion of the converted image from the input image. The ophthalmic imaging devices 11A and 11B capture an ophthalmic image which is an image of the tissue of the eye to be inspected.

As an example, a personal computer (hereinafter referred to as "PC") is used for the mathematical model construction device 1 of the present embodiment. Although the details will be described later, the mathematical model building apparatus 1 uses an ophthalmic image acquired from the ophthalmic imaging apparatus 11A (hereinafter referred to as “training ophthalmic image”) and an image obtained by converting the image quality of the training ophthalmic image. Build a mathematical model by training it. However, the device that can function as the mathematical model construction device 1 is not limited to the PC. For example, the ophthalmologic imaging device 11A may function as the mathematical model building device 1. Further, the control units of the plurality of devices (for example, the CPU of the PC and the CPU 13A of the ophthalmologic imaging apparatus 11A) may collaborate to construct a mathematical model.

Further, in the present embodiment, a case where a CPU is used as an example of a controller that performs various processes will be illustrated. However, it goes without saying that a controller other than the CPU may be used for at least a part of various devices. For example, by adopting a GPU as a controller, the processing speed may be increased.

The mathematical model construction device 1 will be described. The mathematical model construction device 1 is arranged, for example, in an ophthalmic image processing device 21 or a manufacturer that provides an ophthalmic image processing program to a user. The mathematical model building apparatus 1 includes a control unit 2 that performs various control processes and a communication I / F5. The control unit 2 includes a CPU 3 which is a controller that controls control, and a storage device 4 that can store programs, data, and the like. The storage device 4 stores a mathematical model construction program for executing a mathematical model construction process (see FIG. 3) described later. Further, the communication I / F5 connects the mathematical model building device 1 to other devices (for example, an ophthalmic imaging device 11A and an ophthalmic image processing device 21).

The mathematical model construction device 1 is connected to the operation unit 7 and the display device 8. The operation unit 7 is operated by the user in order for the user to input various instructions to the mathematical model construction device 1. For the operation unit 7, for example, at least one of a keyboard, a mouse, a touch panel, and the like can be used. A microphone or the like for inputting various instructions may be used together with the operation unit 7 or instead of the operation unit 7. The display device 8 displays various images. As the display device 8, various devices capable of displaying an image (for example, at least one of a monitor, a display, a projector, and the like) can be used. The "image" in the present disclosure includes both a still image and a moving image.

The mathematical model construction device 1 can acquire data of an ophthalmic image (hereinafter, may be simply referred to as an “ophthalmic image”) from the ophthalmic imaging device 11A. The mathematical model building apparatus 1 may acquire ophthalmic image data from the ophthalmic imaging apparatus 11A by, for example, at least one of wired communication, wireless communication, a detachable storage medium (for example, a USB memory), and the like.

The ophthalmic image processing device 21 will be described. The ophthalmologic image processing device 21 is arranged, for example, in a facility (for example, a hospital or a health examination facility) for diagnosing or examining a subject. The ophthalmic image processing device 21 includes a control unit 22 that performs various control processes and a communication I / F 25. The control unit 22 includes a CPU 23, which is a controller that controls control, and a storage device 24 that can store programs, data, and the like. The storage device 24 stores an ophthalmic image processing program for executing ophthalmic image processing (see FIGS. 4 and 9) described later. The ophthalmic image processing program includes a program that realizes a mathematical model constructed by the mathematical model building apparatus 1. The communication I / F 25 connects the ophthalmic image processing device 21 to other devices (for example, the ophthalmic imaging device 11B and the mathematical model building device 1).

The ophthalmic image processing device 21 is connected to the operation unit 27 and the display device 28. As the operation unit 27 and the display device 28, various devices can be used in the same manner as the operation unit 7 and the display device 8 described above.

The ophthalmic image processing device 21 can acquire an ophthalmic image from the ophthalmic image capturing device 11B. The ophthalmic image processing device 21 may acquire an ophthalmic image from the ophthalmic image capturing device 11B by, for example, at least one of wired communication, wireless communication, a detachable storage medium (for example, a USB memory), and the like. Further, the ophthalmic image processing device 21 may acquire a program or the like for realizing the mathematical model constructed by the mathematical model building device 1 via communication or the like.

The ophthalmic imaging devices 11A and 11B will be described. As an example, in the present embodiment, a case where an ophthalmic image capturing device 11A for providing an ophthalmic image to the mathematical model building apparatus 1 and an ophthalmologic imaging device 11B for providing an ophthalmic image to the ophthalmic image processing device 21 will be described. .. However, the number of ophthalmic imaging devices used is not limited to two. For example, the mathematical model construction device 1 and the ophthalmic image processing device 21 may acquire ophthalmic images from a plurality of ophthalmic imaging devices. Further, the mathematical model construction device 1 and the ophthalmology image processing device 21 may acquire an ophthalmology image from one common ophthalmology image capturing device.

Further, in the present embodiment, the OCT device is exemplified as the ophthalmic imaging device 11 (11A, 11B). However, an ophthalmologic imaging device other than the OCT device (for example, a laser scanning optometry device (SLO), a fundus camera, a Scheimpflug camera, a corneal endothelial cell imaging device (CEM), etc.) may be used.

The ophthalmic imaging device 11 (11A, 11B) includes a control unit 12 (12A, 12B) that performs various control processes, and an ophthalmic imaging unit 16 (16A, 16B). The control unit 12 includes a CPU 13 (13A, 13B) which is a controller that controls control, and a storage device 14 (14A, 14B) capable of storing programs, data, and the like. When the ophthalmic image capturing apparatus 11 executes at least a part of the ophthalmic image processing (see FIGS. 4 and 9) described later, at least a part of the ophthalmic image processing program for executing the ophthalmic image processing is stored in the storage device 14. Needless to say, it is remembered in.

The ophthalmic imaging unit 16 includes various configurations necessary for capturing an ophthalmic image of the eye to be inspected. The ophthalmic imaging unit 16 of the present embodiment is provided with an OCT light source, a branched optical element that branches OCT light emitted from the OCT light source into measurement light and reference light, a scanning unit for scanning the measurement light, and measurement light. It includes an optical system for irradiating the eye examination, a light receiving element that receives the combined light of the light reflected by the tissue of the eye to be inspected and the reference light, and the like.

The ophthalmologic image capturing device 11 can capture a two-dimensional tomographic image and a three-dimensional tomographic image of the fundus of the eye to be inspected. Specifically, the CPU 13 scans the OCT light (measurement light) on the scan line to take a two-dimensional tomographic image (see FIG. 5) of the cross section intersecting the scan line. In addition, the CPU 13 can capture a three-dimensional tomographic image of the tissue by scanning the OCT light two-dimensionally. For example, the CPU 13 acquires a plurality of two-dimensional tomographic images by scanning measurement light on each of a plurality of scan lines having different positions in a two-dimensional region when the tissue is viewed from the front. Next, the CPU 13 acquires a three-dimensional tomographic image by combining a plurality of captured two-dimensional tomographic images.

Further, the CPU 13 can capture a plurality of ophthalmic images of the same site by scanning the measurement light a plurality of times on the same site on the tissue (in the present embodiment, on the same scan line). The CPU 13 can acquire an averaging image in which the influence of speckle noise is suppressed by performing an averaging process on a plurality of ophthalmic images of the same portion. The image quality of the two-dimensional tomographic image can be improved by performing the addition averaging processing on a plurality of two-dimensional tomographic images of the same part. The addition averaging process may be performed, for example, by averaging the pixel values of the pixels at the same position in a plurality of ophthalmic images. The larger the number of images to be subjected to the averaging process, the easier it is to suppress the influence of speckle noise, but the longer the shooting time. The ophthalmologic image capturing device 11 executes a tracking process for tracking the scanning position of the OCT light with the movement of the eye to be inspected while taking a plurality of ophthalmic images of the same site.

(Mathematical model construction process)
The mathematical model construction process executed by the mathematical model construction apparatus 1 will be described with reference to FIGS. 2 and 3. The mathematical model construction process is executed by the CPU 3 according to the mathematical model construction program stored in the storage device 4.

In the mathematical model construction process, the mathematical model is trained by the training data set, so that the mathematical model that outputs the converted image obtained by converting the image quality of the input image is constructed. The training data set includes input side data (input training data) and output side data (output training data). The mathematical model can convert various ophthalmic images into converted images. The type of training data set used to train the mathematical model is determined by the type of ophthalmic image to which the mathematical model transforms image quality. Hereinafter, a case where a two-dimensional tom image (high-quality image) in which the image quality of the input image is improved by inputting the two-dimensional tom image as an input image to the mathematical model is output to the mathematical model as a converted image will be described.

FIG. 2 shows an example of input training data and output training data when a high-quality two-dimensional tomographic image is output as a converted image to a mathematical model. In the example shown in FIG. 2, the CPU 3 acquires a set 40 of a plurality of two-dimensional tomographic images 400A to 400X in which the same part of the tissue is photographed. The CPU 3 uses a part of the plurality of two-dimensional tomographic images 400A to 400X in the set 40 (the number of images smaller than the number of images used for the addition averaging of the output training data described later) as the input training data. Further, the CPU 3 acquires the additional average image 41 of the plurality of two-dimensional tomographic images 400A to 400X in the set 40 as output training data. When a mathematical model is trained using the input training data and output training data illustrated in FIG. 2, the influence of speckle noise is suppressed by inputting a two-dimensional tomographic image as an input image to the trained mathematical model. The high-quality two-dimensional tomographic image is output as a converted image.

Note that the ophthalmic image that converts the image quality to the mathematical model is not limited to the two-dimensional tomographic image of the fundus. For example, the ophthalmologic image may be an image of a portion other than the fundus of the eye to be examined. Further, the ophthalmic image may be a three-dimensional tomographic image, an OCT angio image, an Enface image, or the like taken by an OCT apparatus. The OCT angio image may be a two-dimensional front image in which the fundus is viewed from the front (that is, the line-of-sight direction of the eye to be inspected). The OCT angio image may be, for example, a motion contrast image acquired by processing at least two OCT signals acquired at different times with respect to the same position. The Enface image is a two-dimensional front image when at least a part of the three-dimensional tomographic image taken by the OCT device is viewed from the direction (front direction) along the optical axis of the measurement light of the OCT device. Further, the ophthalmologic image may be an image taken by a fundus camera, an image taken by a laser scanning optometry device (SLO), an image taken by a corneal endothelial cell photography device, or the like.

It is also possible to change the method of generating high-quality ophthalmic images used as output training data. For example, the image quality may be improved by a process other than the addition averaging process.

The mathematical model construction process will be described with reference to FIG. The CPU 3 acquires at least a part of the ophthalmic image taken by the ophthalmic image capturing device 11A as input training data (S1). In the present embodiment, the data of the ophthalmic image is generated by the ophthalmic imaging apparatus 11A and then acquired by the mathematical model construction apparatus 1. However, the CPU 3 acquires the data of the ophthalmic image by acquiring the signal (for example, OCT signal) that is the basis for generating the ophthalmic image from the ophthalmic imaging apparatus 11A and generating the ophthalmic image based on the acquired signal. You may.

Next, the CPU 3 acquires the output training data corresponding to the input training data acquired in S1 (S3). An example of the correspondence between the input training data and the output training data is as described above.

Next, the CPU 3 executes the training of the mathematical model using the training data set by the machine learning algorithm (S3). As machine learning algorithms, for example, neural networks, random forests, boosting, support vector machines (SVMs), and the like are generally known.

Neural networks are a method of imitating the behavior of biological nerve cell networks. Neural networks include, for example, feed-forward (forward propagation) neural networks, RBF networks (radial basis functions), spiking neural networks, convolutional neural networks, recursive neural networks (recurrent neural networks, feedback neural networks, etc.), and probabilities. Neural networks (Boltzmann machines, Basian networks, etc.).

Random forest is a method of generating a large number of decision trees by learning based on randomly sampled training data. When using a random forest, the branches of a plurality of decision trees learned in advance as a discriminator are traced, and the average (or majority vote) of the results obtained from each decision tree is taken.

Boosting is a method of generating a strong classifier by combining multiple weak classifiers. A strong classifier is constructed by sequentially learning simple and weak classifiers.

SVM is a method of constructing a two-class pattern classifier using a linear input element. The SVM learns the parameters of the linear input element based on, for example, the criterion (hyperplane separation theorem) of obtaining the margin maximizing hyperplane that maximizes the distance from each data point from the training data.

A mathematical model refers to, for example, a data structure for predicting the relationship between input data and output data. Mathematical models are constructed by training with training datasets. As described above, the training data set is a set of training data for input and training data for output. For example, training updates the correlation data (eg, weights) for each input and output.

In this embodiment, a multi-layer neural network is used as a machine learning algorithm. A neural network includes an input layer for inputting data, an output layer for generating the data to be predicted, and one or more hidden layers between the input layer and the output layer. A plurality of nodes (also called units) are arranged in each layer. Specifically, in this embodiment, a convolutional neural network (CNN), which is a kind of multi-layer neural network, is used. However, other machine learning algorithms may be used. For example, a hostile generative network (GAN) that utilizes two competing neural networks may be adopted as a machine learning algorithm.

The processes of S1 to S3 are repeated until the construction of the mathematical model is completed (S5: NO). When the construction of the mathematical model is completed (S5: YES), the mathematical model construction process is completed. The program and data for realizing the constructed mathematical model are incorporated in the ophthalmologic image processing apparatus 21.

(Ophthalmic image processing)
An example of ophthalmic image processing executed by the ophthalmic image processing apparatus 21 will be described with reference to FIGS. 4 to 8. 4 to 8 show an example of evaluating the validity of image quality conversion by a mathematical model based on the difference information (difference image) between the input image and the converted image. The ophthalmic image processing illustrated in FIG. 4 is executed by the CPU 23 according to the ophthalmic image processing program stored in the storage device 24.

As shown in FIG. 4, the CPU 23 acquires an ophthalmologic image of the tissue of the eye to be inspected taken by the ophthalmologic imaging apparatus (OCT apparatus in this embodiment) 11B (S11). In S11 of the present embodiment, a two-dimensional tomographic image (see FIG. 5) of the fundus tissue of the eye to be inspected is acquired.

Next, the CPU 23 converted the image quality of the input image by inputting the ophthalmic image acquired in S11 as the input image into the mathematical model trained by the machine learning algorithm (in the present embodiment, the image quality of the input image is changed. The (improved) converted image is acquired (S12).

FIG. 5 shows an example of an ophthalmic image used as an input image. Further, FIG. 6 shows a converted image in which the image quality of the input image shown in FIG. 5 is converted (improved image quality). The image quality of the input image shown in FIG. 5 is lower than that of the converted image shown in FIG. However, the input image shown in FIG. 5 is generated without performing the addition averaging process or by performing the addition averaging process on a small number of ophthalmic images. Therefore, the input image shown in FIG. 5 can be captured in a short time. When an image with high image quality equivalent to the converted image shown in FIG. 6 is photographed by the ophthalmologic image photographing apparatus 11B, it is necessary to photograph the image of the same part a plurality of times and execute the addition averaging process, thus shortening the photographing time. Is difficult. In the present embodiment, a high-quality converted image is acquired by inputting an input image taken in a short time into a mathematical model. Therefore, a high-quality image can be acquired while suppressing a long shooting time.

Next, the CPU 23 obtains the difference information of the pixel values between the corresponding pixels of the input image (see FIG. 5) input to the mathematical model in S12 and the converted image (see FIG. 6) output from the mathematical model in S12. , Acquired as evaluation information (S13). The evaluation information is information for evaluating the validity of conversion from an input image to a converted image by a mathematical model. When the image quality of the input image is appropriately converted and the converted image is output, the difference between the input image and the converted image becomes small. On the other hand, the image quality conversion may not be properly executed depending on the state of the input image. For example, if a part (for example, an irregular part such as a lesion part) contained in the training data set (ophthalmic image) used for training a mathematical model is present in the input image, the irregular part is converted. Is difficult to execute properly. Therefore, if the irregular portion in the input image affects the conversion and the conversion of the input image is not properly executed, the difference between the input image and the converted image becomes large. Therefore, by acquiring the difference information as the evaluation information, the validity of the conversion from the input image to the converted image is appropriately evaluated.

Note that the CPU 23 may acquire the difference information after executing the Fourier transform (for example, two-dimensional Fourier transform) for each of the input image and the converted image. Periodic artifacts may occur in the transformed image. When periodic artifacts occur in the converted image, the input image and the converted image show different frequency distributions. Therefore, by using the Fourier transform, the presence or absence of periodic artifacts is evaluated more appropriately.

In S13 of the present embodiment, a difference image (see FIG. 7) showing the distribution of the difference values acquired for each pixel is acquired as the difference information. In the difference image illustrated in FIG. 7, the gray portion where the brightness is an intermediate value (for example, 128, which is an intermediate value when the brightness changes in the range of 1 to 256) is a pixel having a small difference value. It is displayed. In the difference image, there is a difference between the region where the conversion is properly performed and the region where the conversion is not performed properly. Therefore, the validity of the conversion from the input image to the converted image is appropriately evaluated based on the difference image.

Note that the difference information may be information other than the difference image. For example, in S13, the average value of the difference values between the plurality of pixels may be acquired as the difference information. Further, the difference information may be the difference in pixel values between the corresponding pixels, or may be the ratio of the other pixel value to one pixel value.

Next, the CPU 23 evaluates the validity of the conversion from the input image to the converted image in S12 based on the difference image (see FIG. 7) acquired in S13 (S14). Specifically, in S14 of this embodiment, it is evaluated whether or not the conversion in S12 was appropriate.

An example of a method for evaluating the validity of conversion based on a difference image will be described with reference to FIG. As shown in FIG. 8, even when the conversion from the input image to the converted image is properly executed, the pixels 51 having a large difference value are scattered in the image area 50 of the difference image. On the other hand, in the region where the conversion is not properly executed due to the presence of the irregular portion (for example, the disease portion), the pixels 51 having a large difference value are densely packed. Therefore, in the present embodiment, in the CPU 23, when the value of the difference in an arbitrary region including a plurality of pixels is equal to or greater than the threshold value, there is an region in which the conversion is not properly executed (that is, the conversion is not valid). ). In the example shown in FIG. 8, the value of the difference in the region 55 is equal to or greater than the threshold value. Therefore, the CPU 23 evaluates the region 55 as a region in which the conversion is not properly executed (a region in which an irregular portion exists).

Further, the CPU 23 executes a smoothing process on the pixel value of the difference image (the value of the difference corresponding to each pixel), and then evaluates the validity of the conversion based on the difference image. Therefore, even when the conversion is properly executed, the validity of the conversion is evaluated more appropriately in a state where the influence of the scattered pixels 51 is suppressed.

The specific method for evaluating the validity of the conversion based on the value of the difference in the region can be appropriately selected. As an example, in the present embodiment, when the average value of the difference values in the region is equal to or more than the threshold value, the CPU 23 evaluates that the conversion of the image quality in the region is not appropriate. However, the CPU 23 may evaluate whether or not the conversion is appropriate based on the number of pixels in the unit region whose difference value is equal to or greater than the threshold value.

Further, instead of evaluating whether or not the conversion in S12 is valid, the CPU 23 may acquire information (for example, a numerical value or a graph) indicating the degree of validity of the conversion in S12 based on the difference information. Good. The CPU 23 may notify the user (for example, display on the display device 28) of information indicating the degree of validity of the conversion as evaluation information.

Return to the explanation in Fig. 4. When the conversion from the input image to the converted image in S12 is appropriate (S15: YES), the CPU 23 causes the display device 28 to display the converted image acquired in S12 (S16). On the other hand, when the CPU 23 evaluates that the conversion from the input image to the converted image in S12 is not appropriate (S15: NO), the CPU 23 stops the process of displaying the converted image acquired in S12 on the display device 28 (S15: NO). That is, the process of S16 is not executed).

Further, when the CPU 23 evaluates that the conversion is not valid (S15: NO), the CPU 23 executes a warning process for the user (S17). As an example, the CPU 23 warns the user by displaying a warning message such as "image conversion was not properly executed" or a warning image on the display device 28. However, the warning method can be changed as appropriate. For example, the CPU 23 may issue a warning to the user by generating at least one of a warning message and a warning sound from the speaker.

Note that the CPU 23 can also execute warning processing while displaying the converted image that has not been properly converted on the display device 28. In this case, the CPU 23 may use a warning message such as "the displayed converted image may be inappropriate" in the process of S17. Further, when the CPU 23 evaluates that the conversion is not valid (S15: NO), the CPU 23 displays the ophthalmic image used as the input image while stopping the process of displaying the converted image acquired in S12 on the display device 28. It may be displayed on the device 28. In this case, the user can observe the desired site based on the ophthalmic image before the image quality is converted.

Next, the CPU 23 causes the display device 28 to display the difference image (see FIG. 7) acquired in S13 (S18). As described above, in the difference image, there is a difference between the region where the conversion is properly performed and the region where the conversion is not performed properly. Therefore, the user can confirm whether or not the conversion is properly performed by confirming the difference image. Further, the user can grasp the area where the conversion is not properly performed by checking the difference image. Further, when the input image contains an irregular portion (for example, a diseased portion), it is difficult to properly convert the image quality of the irregular portion. Therefore, the user can appropriately grasp the irregular portion in the input image by checking the difference image.

(Transformation example)
An example of transformation of the above embodiment will be described with reference to FIG. FIG. 9 is a flowchart of ophthalmic image processing in the transformation example. In the transformation example shown in FIG. 9, the similarity (for example, correlation) between the difference image and the input image is acquired as evaluation information, and the validity of the conversion is evaluated based on the similarity. In addition, at least a part of the ophthalmologic image processing (see FIG. 4) exemplified in the above embodiment can be similarly adopted in the ophthalmologic image processing of the transformation example shown in FIG. Therefore, for the processes that can adopt the same processes as those in the above embodiment, the same step numbers as those in the above embodiments are assigned, and the description thereof will be omitted or simplified.

In the ophthalmic image processing of the transformation example shown in FIG. 9, the CPU 23 acquires the difference image (see FIG. 7) between the input image and the converted image after executing the converted image acquisition process (S12) (S23). Next, the CPU 23 acquires the similarity between the input image and the difference image as evaluation information (S24). As described above, when the image quality of the input image is appropriately converted, the difference between the input image and the converted image becomes small, so that the similarity between the difference image and the input image becomes small. On the other hand, if the conversion of the input image is not properly executed and the irregular part or the like in the input image affects the conversion, the position (area) of the irregular part or the like in the input image and the difference value in the difference image Since the position (region) where is large is approximated, the degree of similarity between the input image and the difference image is large. Therefore, the validity of the conversion is appropriately evaluated by acquiring the similarity between the difference image and the input image as evaluation information.

Next, the CPU 23 evaluates the validity of the conversion from the input image to the converted image in S12 based on the similarity acquired in S24 (S25). Specifically, in S25 of the present embodiment, whether or not the conversion in S12 was appropriate is evaluated based on the degree of similarity. As mentioned above, when the conversion is performed properly, the similarity between the input image and the difference image becomes small. On the other hand, if the conversion is not performed properly, the similarity between the input image and the difference image becomes large. Therefore, the CPU 23 can appropriately evaluate whether or not the conversion from the input image to the converted image is appropriate by determining whether or not the value indicating the similarity is equal to or greater than the threshold value.

Various values (for example, correlation coefficient) can be appropriately used as values indicating the degree of similarity. Further, instead of evaluating whether or not the conversion in S12 is appropriate, the CPU 23 displays information (for example, numerical value, correlation diagram, graph, etc.) indicating the degree of validity of the conversion in S12 as evaluation information. It may be displayed on 28.

The technology disclosed in the above embodiments and transformation examples is only an example. Therefore, it is possible to modify the techniques exemplified in the above embodiments and transformation examples. First, in the above embodiment, the validity of the conversion from the input image to the converted image is evaluated based on the difference information (difference image). Further, in the above transformation example, the validity of the conversion is evaluated based on the similarity between the input image and the difference image. However, the method of acquiring the evaluation information for evaluating the validity of the conversion is not limited to the method exemplified in the above-described embodiment and the transformation example.

For example, the CPU 23 may acquire evaluation information using a mathematical model trained by a machine learning algorithm. In this case, the mathematical model (mathematical model for acquiring evaluation information) uses, for example, input images and converted images as input training data, and provides evaluation information indicating the validity of conversion in the input images and converted images of the input training data. It may be trained in advance as output training data. The training data for output may be generated by the user comparing the input image and the converted image. The CPU 23 may acquire the evaluation information output by the mathematical model by inputting the input image and the converted image into the mathematical model for acquiring the evaluation information. By acquiring the evaluation information by the mathematical model for acquiring the evaluation information, the validity of the conversion is appropriately evaluated even if the difference between the input image and the converted image is not acquired.

The mathematical model for acquiring evaluation information may output evaluation information indicating whether or not the conversion from the input image to the converted image is appropriate, or evaluates a numerical value or the like indicating the degree of validity of the conversion. Information may be output.

Also, the process to be executed when it is judged that the conversion from the input image to the converted image is not appropriate can be changed as appropriate. For example, when the CPU 23 evaluates that the conversion is not valid based on the evaluation information, the CPU 23 acquires the converted image by inputting the input image to a mathematical model different from the mathematical model that performed the conversion evaluated as invalid. May be good. The characteristics of conversion by the mathematical model differ depending on the algorithm and training data used when training the mathematical model. Therefore, if the transformation is evaluated as invalid, the transformed image may be acquired appropriately by acquiring the transformed image by a different mathematical model.

It is also possible to implement only some of the plurality of techniques exemplified in the above embodiments and transformation examples. For example, in the ophthalmic image processing shown in FIG. 4, when the conversion is evaluated to be invalid (S15: NO), the processing for stopping the display of the converted image and the warning processing are both executed. However, it is also possible to omit at least one of the process of stopping the display of the converted image and the process of warning.

The process of acquiring an ophthalmic image in S11 of FIGS. 4 and 9 is an example of the “image acquisition step”. The process of acquiring the converted image in S12 of FIGS. 4 and 9 is an example of the “converted image acquisition step”. The process of acquiring the evaluation information in S13 of FIG. 4 and S24 of FIG. 9 is an example of the “evaluation information acquisition step”. The process of evaluating the validity of the conversion in S14 of FIG. 4 is an example of the “first evaluation step”. The process of evaluating the validity of the conversion in S25 of FIG. 9 is an example of the “second evaluation step”. The process of displaying the difference image in S18 of FIGS. 4 and 9 is an example of the “difference image display step”. The warning process shown in S17 of FIGS. 4 and 9 is an example of the “warning step”. The process of stopping the display process of the converted image at S15: NO in FIGS. 4 and 9 is an example of the “display stop step”.

11A, 11B Ophthalmic Imaging Device 21 Ophthalmic Image Processing Device 23 CPU
24 Storage device 28 Display device

Claims (10)

1. An ophthalmic image processing program executed by an ophthalmic image processing device that processes an ophthalmic image that is an image of the tissue of the eye to be inspected.
   When the ophthalmic image processing program is executed by the control unit of the ophthalmic image processing device,
   An image acquisition step to acquire an ophthalmic image taken by an ophthalmic imaging device, and
   A conversion image acquisition step of acquiring a converted image obtained by converting the image quality of the input image by inputting the ophthalmic image acquired in the image acquisition step as an input image into a mathematical model trained by a machine learning algorithm.
   An evaluation information acquisition step for acquiring evaluation information for evaluating the validity of conversion from the input image to the conversion image by the mathematical model, and
   Is executed by the ophthalmic image processing apparatus.
2. The ophthalmic image processing program according to claim 1.
   In the evaluation information acquisition step, the difference information of the pixel values between the corresponding pixels of the input image input to the mathematical model and the converted image output from the mathematical model is acquired as the evaluation information. An ophthalmic image processing program characterized by.
3. The ophthalmic image processing program according to claim 2.
   In the difference image obtained by imaging the difference information, when the value of the difference in an arbitrary region including a plurality of pixels is equal to or larger than the threshold value, it is evaluated that the conversion from the input image to the converted image is not appropriate. 1 evaluation step,
   Is executed by the ophthalmic image processing apparatus.
4. The ophthalmic image processing program according to claim 1.
   In the evaluation information acquisition step, a difference image obtained by imaging the difference information of the pixel values between the corresponding pixels of the input image input to the mathematical model and the converted image output from the mathematical model is acquired. In addition, an ophthalmic image processing program characterized in that the similarity between the difference image and the input image is acquired as the evaluation information.
5. The ophthalmic image processing program according to claim 4.
   The second evaluation step of evaluating that the conversion from the input image to the converted image is not appropriate when the value indicating the similarity is equal to or greater than the threshold value.
   Is executed by the ophthalmic image processing apparatus.
6. The ophthalmic image processing program according to any one of claims 2 to 5.
   A difference image display step of displaying a difference image obtained by imaging the difference information on the display unit,
   Is executed by the ophthalmic image processing apparatus.
7. The ophthalmic image processing program according to claim 1.
   In the evaluation information acquisition step,
   By inputting the input image input to the mathematical model in the converted image acquisition step and the converted image output from the mathematical model into the mathematical model for acquiring evaluation information trained by a machine learning algorithm. , An ophthalmic image processing program characterized in that the evaluation information is acquired.
8. The ophthalmic image processing program according to any one of claims 1 to 7.
   A warning step that warns the user when the conversion is evaluated as invalid by the evaluation information acquired in the evaluation information acquisition step.
   Is executed by the ophthalmic image processing apparatus.
9. The ophthalmic image processing program according to any one of claims 1 to 8.
   A display stop step for stopping the display process of the converted image acquired in the converted image acquisition step on the display unit when the evaluation information acquired in the evaluation information acquisition step evaluates that the conversion is not valid. ,
   Is executed by the ophthalmic image processing apparatus.
10. An ophthalmic image processing device that processes an ophthalmic image, which is an image of the tissue of the eye to be inspected.
    The control unit of the ophthalmic image processing device
    An image acquisition step to acquire an ophthalmic image taken by an ophthalmic imaging device, and
    A conversion image acquisition step of acquiring a converted image obtained by converting the image quality of the input image by inputting the ophthalmic image acquired in the image acquisition step as an input image into a mathematical model trained by a machine learning algorithm.
    An evaluation information acquisition step for acquiring evaluation information for evaluating the validity of conversion from the input image to the converted image by the mathematical model, and
    An ophthalmic image processing apparatus characterized by performing.
    
    
    

Images