JP2021037177A - Ophthalmologic image processing program and ophthalmologic image processing device - Google Patents

Abstract

To provide an ophthalmologic image processing program and an ophthalmologic image processing device capable of appropriately presenting medical information desired by a user to the user.SOLUTION: A control unit of an ophthalmologic image processing device acquires an ophthalmologic image captured by an OCT device (S11). The control unit acquires a high quality image obtained by improving the quality of a base image as medical information by inputting the base image to a mathematical model trained by a machine learning algorithm with a partial image, which is a part of the acquired ophthalmologic image, or an image generated on the basis of the partial image as the base image. When an instruction to select the partial image is received, the control unit causes medical information acquired on the selected partial image to be displayed in a display unit (S28). The control unit executes preliminary acquisition processing for preliminarily acquiring and storing medical information on a partial image yet to be selected of the ophthalmologic image (S22).SELECTED DRAWING: Figure 5

Description

The present disclosure relates to an ophthalmic image processing program and an ophthalmic image processing apparatus used for processing an ophthalmic image of an eye to be inspected.

Techniques for acquiring various medical information using mathematical models trained by machine learning algorithms have been proposed. For example, in the ophthalmic apparatus described in Patent Document 1, IOL-related information (for example, expected postoperative anterior chamber depth) of the eye to be inspected is acquired by inputting an eye shape parameter into a mathematical model trained by a machine learning algorithm. Will be done. The IOL frequency is calculated based on the acquired IOL-related information.

It is also conceivable to acquire a high quality image from a low quality image using a mathematical model trained by a machine learning algorithm. In this case, a high-quality image is output by inputting a low-quality image into the mathematical model.

JP-A-2018-51223

The user may want to select a part of the captured image (hereinafter referred to as "partial image") of interest and check medical information (for example, a high-quality image) related to the selected partial image. .. In this case, it is conceivable that the image processing apparatus acquires medical information regarding the selected partial image by a mathematical model each time the partial image is selected by the user, and presents the acquired medical information to the user. However, since it takes time to acquire medical information using a mathematical model, it is difficult to shorten the user's work time by the method of acquiring and presenting medical information every time a partial image is selected. Is. Therefore, it is desirable to be able to appropriately present the medical information desired by the user to the user.

A typical object of the present disclosure is to provide an ophthalmic image processing program and an ophthalmic image processing apparatus capable of appropriately presenting medical information desired by the user to the user.

The ophthalmic image processing program provided by the typical embodiment in the present disclosure is an ophthalmic image processing program executed by an ophthalmic image processing apparatus that processes an ophthalmic image which is an image of a tissue of an eye to be inspected, and is the ophthalmic image processing. When the program is executed by the control unit of the ophthalmic image processing device, an image acquisition step of acquiring an ophthalmic image of the tissue of the eye to be inspected taken by the OCT device and a part of the acquired ophthalmic images are obtained. A partial image or an image generated based on the partial image is used as a base image, and the base image is input to a mathematical model trained by a machine learning algorithm to improve the image quality of the base image. A medical information acquisition step of acquiring an image as medical information and storing it in a storage device, an instruction receiving step of accepting an instruction to select the partial image in the acquired ophthalmic image, and a case where the instruction is received. , A medical information display step of displaying the medical information acquired with respect to the partial image selected by the instruction on the display unit is executed by the ophthalmic image processing apparatus, and in the medical information acquisition step, the ophthalmic image of the ophthalmic image is displayed. Among them, a pre-acquisition process is executed in which the medical information regarding the partial image that has not yet been selected by the instruction is acquired and stored in advance.

The ophthalmic image processing apparatus provided by the typical embodiment in the present disclosure is an ophthalmic image processing apparatus that processes an ophthalmic image which is an image of a tissue of an eye to be inspected, and a control unit of the ophthalmic image processing apparatus is an OCT apparatus. The image acquisition step of acquiring an ophthalmic image of the tissue of the eye to be inspected taken by, and a partial image of the acquired ophthalmic image, or an image generated based on the partial image is used as a base image. By inputting the basic image into a mathematical model trained by a machine learning algorithm, a high-quality image with improved image quality of the basic image is acquired as medical information and stored in a storage device. The instruction receiving step that accepts the input of the instruction to select the partial image in the acquired ophthalmic image and the medical information acquired with respect to the partial image selected by the instruction when the instruction is accepted are displayed. A medical information display step to be displayed on the unit is executed, and in the medical information acquisition step, the control unit displays the medical information related to the partial image of the ophthalmic images that has not yet been selected by the instruction. , Execute the pre-acquisition process to acquire and store in advance.

According to the ophthalmic image processing program and the ophthalmic image processing apparatus according to the present disclosure, the medical information desired by the user is appropriately presented to the user.

It is a block diagram which shows the schematic structure of the mathematical model construction apparatus 1, the ophthalmologic image processing apparatus 21, and the ophthalmology imaging apparatus 11A, 11B. It is a figure which shows an example of the training data for input and the training data for output in the case of outputting a high-quality two-dimensional tomographic image to a mathematical model. It is a figure which shows an example of the 3D tomographic image 50 and the Enface image 55 generated based on the partial image included in the 3D tomographic image 50. It is a flowchart of the mathematical model construction process executed by the mathematical model construction apparatus 1. It is a flowchart of ophthalmic image processing executed by ophthalmic image processing apparatus 21. This is an example of a display screen 60 displayed on the display device 28 during execution of ophthalmic image processing. It is a figure which shows an example of the partial image 66 selected by a user, and the high-quality image 67 acquired from the partial image 66. It is a figure which shows an example of the setting screen displayed on the display device when a user sets a pre-acquisition pattern. It is explanatory drawing for demonstrating an example of the method of setting the priority with respect to a plurality of partial images 66 with reference to a reference image BL. It is explanatory drawing for demonstrating an example of the method of setting the priority with respect to a plurality of partial images 66 along the direction in which the partial image 66 selected by a user is switched. It is a flowchart of the pre-acquisition process executed by the ophthalmologic image processing apparatus 21. This is an example of a display screen 80 displayed on the display device 28 during execution of ophthalmic image processing in the first transformation example.

<Overview>
The control unit of the ophthalmic image processing apparatus exemplified in the present disclosure executes an image acquisition step, a medical information acquisition step, an instruction reception step, and a medical information display step. In the image acquisition step, the control unit acquires an ophthalmologic image of the tissue of the eye to be inspected taken by the OCT apparatus. In the medical information acquisition step, the control unit uses a partial image of the acquired ophthalmic image or an image generated based on the partial image as a base image and turns it into a mathematical model trained by a machine learning algorithm. By inputting the base image, a high-quality image in which the image quality of the base image is improved is acquired as medical information and stored in the storage device. In the instruction receiving step, the control unit receives an input of an instruction for selecting a partial image in the acquired ophthalmic image. In the medical information display step, when the instruction to select the partial image is received, the control unit causes the display unit to display the medical information acquired with respect to the partial image selected by the instruction. In the medical information acquisition step, the control unit executes a pre-acquisition process of pre-acquiring and storing medical information related to a partial image of the ophthalmic image that has not yet been selected by instruction.

In the ophthalmologic image processing apparatus exemplified in the present disclosure, medical information (high-quality image) related to a partial image that has not yet been selected by the user is acquired and stored in advance by the pre-acquisition process. Therefore, the control unit can display the medical information desired by the user on the display unit in a short time when the medical information regarding the actually selected partial image is already stored in the storage device. Therefore, the user can appropriately confirm the desired medical information.

Various ophthalmic images can be adopted as the ophthalmic images processed by the ophthalmic image processing apparatus. For example, the ophthalmic image acquired in the image acquisition step may be a three-dimensional tomographic image including a plurality of two-dimensional tomographic images. In this case, the partial image may be one or more of a plurality of two-dimensional tomographic images included in the three-dimensional tomographic image. The control unit may use a two-dimensional tomographic image as a partial image as a base image and acquire an image with improved image quality of the base image as a high-quality image.

When the ophthalmic image is a three-dimensional tomographic image, the partial image is a partial three-dimensional tomographic image included in the entire image area of the three-dimensional tomographic image (for example, a tertiary in a region sandwiched between two boundary surfaces). It may be a former tomographic image, etc.). The control unit may use an Enface image (OCT front image) when the three-dimensional tomographic image, which is a partial image, is viewed from a direction (front direction) along the optical axis of the measurement light of the OCT device as a base image. The Enface image may be, for example, an integrated image obtained by integrating the pixel values of the partial images in the depth direction (Z direction) at each position in the XY direction intersecting the optical axis. Further, the Enface image may be generated by the integrated value of the spectral data of the partial image at each position in the XY direction. The control unit may acquire a high-quality image by inputting a base image, which is an Enface image, into the mathematical model. Further, the three-dimensional tomographic image itself, which is a partial image, may be used as the base image. Further, the base image may be an OCT angio image. The OCT angio image may be a two-dimensional front image in which the fundus is viewed from the front (that is, the line-of-sight direction of the eye to be inspected). The OCT angio image may be, for example, a motion contrast image acquired by processing at least two OCT signals acquired at different times with respect to the same position.

Further, the ophthalmologic image acquired in the image acquisition step may be a moving image including a plurality of still images acquired by intermittently photographing the same tissue a plurality of times in a time series. In this case, the partial image may be one or more of a plurality of still images constituting the moving image. The control unit may use a still image, which is a partial image, as a base image.

Further, the ophthalmic image acquired in the image acquisition step may be a two-dimensional image which is a still image. In this case, the partial image may be an image of a part of the area included in the area of the two-dimensional image. The control unit may use an image of a part of the region, which is a partial image, as a base image.

In the pre-acquisition process, the control unit may sequentially execute the acquisition process of medical information regarding each partial image in descending order of priority set for a plurality of partial images in the ophthalmic image. In this case, medical information is sequentially acquired in an appropriate order for each of the plurality of partial images.

In the pre-acquisition process, the control unit may use one or more of the plurality of partial images in the ophthalmic image as a reference image, and set a higher priority for each of the plurality of partial images as the image is closer to the reference image. .. In this case, medical information is preferentially acquired from a partial image close to the reference image. Therefore, by appropriately setting the reference image, medical information is sequentially acquired in a more appropriate order. Therefore, the work time of the user can be further shortened.

In addition, various "closeness" can be adopted as the "closeness" of the partial image with respect to the reference image, which is a reference when setting the priority. For example, when a two-dimensional tomographic image constituting a three-dimensional tomographic image is used as a partial image, the "closeness" of the distance to the two-dimensional tomographic image used as the reference image may be used as a reference.

When a part of the 3D tomographic image included in the image area of the 3D tomographic image is used as a partial image, the "closeness" of the distance to the 3D tomographic image used as the reference image, the "closeness" of the range, and the like. At least one of the "closeness" of the overlap ratio and the like may be used as a reference. Further, when a three-dimensional tomographic image in a region sandwiched between two boundary surfaces is used as a partial image, the "closeness" of the distance of the boundary surface of the partial image to the boundary surface of the reference image may be used as a reference. ..

Further, when the still image constituting the moving image is used as a partial image, the "closeness" of the shooting time to the still image used as the reference image may be used as a reference. Further, when the image of a part of the area in the two-dimensional image is used as a partial image, at least one of the "closeness" of the distance, the range, and the overlap ratio with respect to the reference image may be used as a reference.

The control unit may acquire a specific position of the tissue to be reflected in the ophthalmologic image in the pre-acquisition process, and may use a partial image at the acquired specific position as a reference image. In this case, medical information is preferentially acquired from a specific position of the organization. Therefore, the medical information at a specific position is appropriately grasped by the user.

The method by which the control unit acquires a specific position can be appropriately selected. For example, the control unit may detect a specific position of the tissue by performing well-known image processing on the ophthalmic image. Further, the control unit may acquire a specific position by inputting the acquired ophthalmic image into a mathematical model that inputs an ophthalmic image and outputs a specific position.

In addition, the type of the specific position acquired by the control unit can be appropriately selected. For example, when the ophthalmologic image is a fundus image of the eye to be examined, the specific position may be at least one of the fovea centralis, the optic nerve head, the lesion site, the fundus blood vessel, and the like in the fundus. When the fundus image is a three-dimensional tomographic image of the fundus, the specific position may be the position of a specific layer on the fundus.

Further, the specific position may be acquired by using various ophthalmologic examination devices. As an example, the specific position may be a position determined by a perimeter, which is a kind of ophthalmologic examination device (for example, a position on the fundus with low sensitivity). In this case, the control unit may acquire a specific position based on the image of the inspection result by the perimeter.

However, the reference image may be set regardless of the specific position. For example, the reference image may be a partial image including the center of the image area of the ophthalmic image.

When the instruction to select the partial image is received, the control unit may set the partial image selected by the instruction as the reference image. When a user selects a partial image, he / she often wants to confirm medical information about a partial image close to the selected partial image. By setting the partial image selected by the user as the reference image, the control unit can acquire and store medical information about the partial image close to the selected partial image by the pre-acquisition process. As a result, medical information of a partial image close to the selected partial image is displayed on the display unit in a short time. Therefore, the user can more appropriately confirm the desired medical information.

If the medical information related to the partial image selected by the instruction has not yet been acquired, the control unit acquires the medical information related to the selected partial image with the highest priority in the medical information acquisition step, and the medical information display step. May be displayed on the display unit. Further, when the medical information related to the partial image selected by the instruction is acquired in advance by the pre-acquisition process, the control unit causes the display unit to display the already acquired medical information in the medical information display step. You may. As a result, the medical information desired by the user is smoothly displayed on the display unit.

When the partial image selected by the user's instruction is changed, the control unit gives priority to a plurality of partial images along the direction of the partial image after the change with respect to the partial image before the change. It may be set in order. The user often switches the partial images in order along the same direction. The control unit acquires medical information of the partial image that is likely to be selected later by the user in advance by setting the priority order for the plurality of partial images in order along the direction in which the partial image is switched. It can be memorized. Therefore, the user can more appropriately confirm the desired medical information.

When the partial image selected by the user is set as the reference image, the control unit may store the position of the reference image in the ophthalmic image in the storage device. When processing an ophthalmic image whose imaging target is the same as an ophthalmic image in which the position of the reference image is stored in the past, the control unit sets a partial image of the position stored in the past as the reference image among the ophthalmic images. You may. When making a diagnosis or the like by taking ophthalmic images at different times for the same tissue of the eye to be inspected, the user often confirms medical information at the same position in the ophthalmic image. By setting a partial image of a position stored in the past as a reference image, the control unit can efficiently acquire medical information of a position that is likely to be confirmed by the user in advance.

When the reference image is newly set, the control unit may interrupt the pre-acquisition process that has been executed and restart the pre-acquisition process according to the priority based on the newly set reference image. In this case, the medical information of the partial image closer to the newly set reference image is more than that of the case where the pre-acquisition process based on the newly set reference image is executed after the executed pre-acquisition process is completed. Obtained in a short time. Therefore, the user can more appropriately confirm the desired medical information.

The control unit may restart the interrupted pre-acquisition process when the pre-acquisition process based on the newly set reference image is completed. Further, the control unit may omit the pre-acquisition process for the partial image for which medical information has already been acquired when the pre-acquisition process for each of the plurality of partial images is sequentially executed according to the priority order. In this case, medical information regarding each of the plurality of partial images is acquired more efficiently.

Of the plurality of partial images in the ophthalmic image, a pattern of the partial image from which medical information is acquired in advance may be provided in advance. The control unit may execute the acquisition process of medical information regarding the partial image corresponding to the pattern in the pre-acquisition process. In this case, medical information regarding the partial image corresponding to the pattern is acquired in advance. Therefore, the user can appropriately grasp the medical information corresponding to the pattern.

The specific mode of the pattern can be appropriately selected. For example, when a two-dimensional tomographic image constituting a three-dimensional tomographic image of the fundus is used as a partial image, the arrangement of a plurality of partial images when the fundus is viewed from the front is cross-shaped, radial (radial), or The pattern may be provided so as to be concentric, raster (parallel), or a combination thereof.

Further, when a three-dimensional tomographic image in a region sandwiched between two boundary surfaces is used as a partial image among the three-dimensional images of the fundus, a pattern of the partial image in which at least one position of the two boundary surfaces is preset. (For example, a pattern in which the layer between the two interface surfaces is at least one of a surface layer of the retina, a deep layer of the retina, a vitreous body, an outer layer of the retina, an ORCC, a capillary lamina plate, a choroid, and the like) may be provided. In particular, when a three-dimensional image in a region sandwiched between two boundary surfaces is used as a partial image, there are many (infinitely infinite) combinations of the two boundary surfaces. Therefore, it is very difficult to obtain medical information in advance for all the partial images included in the three-dimensional image before displaying the medical information. On the other hand, in the ophthalmic image processing apparatus of the present disclosure, medical information is acquired by a pre-acquisition process with respect to a partial image corresponding to one or a plurality of patterns provided in advance. Therefore, medical information useful to the user is presented to the user more efficiently.

Further, when the still image constituting the moving image is used as a partial image, a plurality of partial image patterns may be provided in which the intervals between the captured times are fixed time intervals.

The control unit may acquire medical information corresponding to a pattern selected by the user from a plurality of patterns provided in advance by a pre-acquisition process.

In addition, the control unit may automatically acquire medical information corresponding to a default pattern (for example, a two-dimensional tomographic image passing through the center of a three-dimensional tomographic image) by a pre-acquisition process. In this case, when the control unit starts displaying the medical information about the acquired ophthalmic image, the control unit may first display the medical information corresponding to the default pattern on the display unit. In this case, by setting the partial image that is likely to be selected by the user as the default pattern, the convenience of the user is further improved. Further, the control unit may perform a pre-acquisition process of medical information corresponding to the pattern before displaying the display screen for displaying the medical information about the acquired ophthalmic image on the display unit. In this case, when the display screen is displayed on the display unit, the medical information corresponding to the pattern is smoothly displayed.

The device that executes the image acquisition step, the medical information acquisition step, the instruction reception step, and the medical information display step can be appropriately selected. For example, the control unit of a personal computer (hereinafter referred to as “PC”) may execute all of the image acquisition step, the medical information acquisition step, the instruction reception step, and the medical information display step. That is, the control unit of the PC may acquire an ophthalmic image from the OCT device and acquire medical information based on the acquired ophthalmic image. Further, the control unit of the OCT device may execute all of the image acquisition step, the medical information acquisition step, the instruction reception step, and the medical information display step. Further, control units of a plurality of devices (for example, an OCT device and a PC) may cooperate to execute an image acquisition step, a medical information acquisition step, an instruction reception step, and a medical information display step.

The medical information acquired from the base image by using the mathematical model is not limited to the high-quality image in which the image quality of the base image is improved. For example, the mathematical model may output the analysis result of the structure of the tissue of the eye to be inspected by inputting the basic image. Specifically, the medical information may be the analysis result (segmentation result) of the tissue layer and boundary. In addition, the medical information may be the analysis result regarding the disease of the eye to be examined.

Also, the ophthalmic image is not limited to the ophthalmic image taken by the OCT apparatus. For example, at least a part of the technique exemplified in the present disclosure can be applied to ophthalmic images taken by a laser scanning eye examination device (SLO), a fundus camera, a Scheimpflug camera, a corneal endothelial cell imaging device (CEM), or the like. ..

<Embodiment>
(Device configuration)
Hereinafter, one of the typical embodiments in the present disclosure will be described with reference to the drawings. As shown in FIG. 1, in this embodiment, the mathematical model construction device 1, the ophthalmic image processing device 21, and the ophthalmic imaging devices 11A and 11B are used. The mathematical model construction device 1 constructs a mathematical model by training the mathematical model by a machine learning algorithm. The program that realizes the constructed mathematical model is stored in the storage device 24 of the ophthalmic image processing device 21. The ophthalmic image processing device 21 acquires medical information by inputting an image (base image) into a mathematical model. As an example, in the present embodiment, a high-quality image in which the image quality of the base image is improved (for example, the influence of noise in the base image is suppressed) is acquired as medical information. The ophthalmic imaging devices 11A and 11B capture an ophthalmic image which is an image of the tissue of the eye to be inspected.

As an example, a personal computer (hereinafter referred to as "PC") is used for the mathematical model building apparatus 1 of the present embodiment. Although the details will be described later, the mathematical model building apparatus 1 uses an image acquired from the ophthalmic imaging apparatus 11A (hereinafter referred to as “training ophthalmic image”) and medical information corresponding to the training ophthalmic image for mathematics. Build a mathematical model by training the model. However, the device that can function as the mathematical model construction device 1 is not limited to the PC. For example, the ophthalmologic imaging device 11A may function as the mathematical model building device 1. Further, the control units of the plurality of devices (for example, the CPU of the PC and the CPU 13A of the ophthalmologic imaging apparatus 11A) may collaborate to construct a mathematical model.

Further, a PC is used for the ophthalmic image processing device 21 of the present embodiment. However, the device that can function as the ophthalmic image processing device 21 is not limited to the PC. For example, the ophthalmic imaging device 11B or a server may function as the ophthalmic image processing device 21. When the ophthalmologic image capturing device (OCT device in this embodiment) 11B functions as the ophthalmologic image processing device 21, the ophthalmologic image capturing device 11B is appropriate for the partial image included in the captured ophthalmic image while capturing the ophthalmic image. Medical information can be obtained. Further, a mobile terminal such as a tablet terminal or a smartphone may function as the ophthalmic image processing device 21. The control units of the plurality of devices (for example, the CPU of the PC and the CPU 13B of the ophthalmologic image capturing device 11B) may cooperate to perform various processes.

Further, in the present embodiment, a case where a CPU is used as an example of a controller that performs various processes will be illustrated. However, it goes without saying that a controller other than the CPU may be used for at least a part of various devices. For example, by adopting a GPU as a controller, the processing speed may be increased.

The mathematical model construction device 1 will be described. The mathematical model construction device 1 is arranged, for example, in an ophthalmic image processing device 21 or a manufacturer that provides an ophthalmic image processing program to a user. The mathematical model building apparatus 1 includes a control unit 2 that performs various control processes and a communication I / F5. The control unit 2 includes a CPU 3 which is a controller that controls control, and a storage device 4 that can store programs, data, and the like. The storage device 4 stores a mathematical model construction program for executing a mathematical model construction process (see FIG. 4) described later. Further, the communication I / F5 connects the mathematical model building device 1 to other devices (for example, an ophthalmic imaging device 11A and an ophthalmic image processing device 21).

The mathematical model construction device 1 is connected to the operation unit 7 and the display device 8. The operation unit 7 is operated by the user in order for the user to input various instructions to the mathematical model construction device 1. For the operation unit 7, for example, at least one of a keyboard, a mouse, a touch panel, and the like can be used. A microphone or the like for inputting various instructions may be used together with the operation unit 7 or instead of the operation unit 7. The display device 8 displays various images. As the display device 8, various devices capable of displaying an image (for example, at least one of a monitor, a display, a projector, and the like) can be used. The "image" in the present disclosure includes both a still image and a moving image.

The mathematical model building apparatus 1 can acquire data of an ophthalmic image (hereinafter, may be simply referred to as an “ophthalmic image”) from the ophthalmic imaging apparatus 11A. The mathematical model building apparatus 1 may acquire ophthalmic image data from the ophthalmic imaging apparatus 11A by, for example, at least one of wired communication, wireless communication, a detachable storage medium (for example, a USB memory), and the like.

The ophthalmic image processing apparatus 21 will be described. The ophthalmologic image processing device 21 is arranged, for example, in a facility (for example, a hospital or a health examination facility) for diagnosing or examining a subject. The ophthalmic image processing device 21 includes a control unit 22 that performs various control processes and a communication I / F 25. The control unit 22 includes a CPU 23, which is a controller that controls control, and a storage device 24 that can store programs, data, and the like. The storage device 24 stores an ophthalmic image processing program for executing ophthalmic image processing (see FIG. 5) described later. The ophthalmic image processing program includes a program that realizes a mathematical model constructed by the mathematical model building apparatus 1. The communication I / F 25 connects the ophthalmic image processing device 21 to other devices (for example, the ophthalmic imaging device 11B and the mathematical model building device 1).

The ophthalmic image processing device 21 is connected to the operation unit 27 and the display device 28. As the operation unit 27 and the display device 28, various devices can be used in the same manner as the operation unit 7 and the display device 8 described above.

The ophthalmic image processing device 21 can acquire an ophthalmic image from the ophthalmic image capturing device 11B. The ophthalmic image processing device 21 may acquire an ophthalmic image from the ophthalmic image capturing device 11B by, for example, at least one of wired communication, wireless communication, a detachable storage medium (for example, a USB memory), and the like. Further, the ophthalmic image processing device 21 may acquire a program or the like for realizing the mathematical model constructed by the mathematical model building device 1 via communication or the like.

The ophthalmic imaging devices 11A and 11B will be described. As an example, in the present embodiment, a case where an ophthalmic image capturing device 11A for providing an ophthalmic image to the mathematical model building apparatus 1 and an ophthalmologic imaging device 11B for providing an ophthalmic image to the ophthalmic image processing device 21 will be described. .. However, the number of ophthalmic imaging devices used is not limited to two. For example, the mathematical model construction device 1 and the ophthalmic image processing device 21 may acquire ophthalmic images from a plurality of ophthalmic imaging devices. Further, the mathematical model construction device 1 and the ophthalmology image processing device 21 may acquire an ophthalmology image from one common ophthalmology image capturing device.

Further, in the present embodiment, the OCT apparatus is exemplified as the ophthalmologic imaging apparatus 11 (11A, 11B). However, an ophthalmologic imaging device other than the OCT device (for example, a laser scanning optometry device (SLO), a fundus camera, a Scheimpflug camera, a corneal endothelial cell imaging device (CEM), etc.) may be used.

The ophthalmic imaging apparatus 11 (11A, 11B) includes a control unit 12 (12A, 12B) that performs various control processes, and an ophthalmic imaging unit 16 (16A, 16B). The control unit 12 includes a CPU 13 (13A, 13B) which is a controller that controls control, and a storage device 14 (14A, 14B) capable of storing programs, data, and the like. When the ophthalmic image capturing device 11 executes at least a part of the ophthalmic image processing (see FIG. 5) described later, at least a part of the ophthalmic image processing program for executing the ophthalmic image processing is stored in the storage device 14. Needless to say.

The ophthalmic imaging unit 16 includes various configurations necessary for capturing an ophthalmic image of the eye to be inspected. The ophthalmic imaging unit 16 of the present embodiment is provided with an OCT light source, a branched optical element that branches OCT light emitted from the OCT light source into measurement light and reference light, a scanning unit for scanning the measurement light, and measurement light. It includes an optical system for irradiating the eye examination, a light receiving element that receives the combined light of the light reflected by the tissue of the eye to be inspected and the reference light, and the like.

The ophthalmologic imaging apparatus 11 can capture a two-dimensional tomographic image and a three-dimensional tomographic image of the fundus of the eye to be inspected. Specifically, the CPU 13 scans the OCT light (measurement light) on the scan line to take a two-dimensional tomographic image (see FIG. 2 and the like) of the cross section intersecting the scan line. Further, the CPU 13 can capture a three-dimensional tomographic image (see FIG. 3 and the like) of the tissue by scanning the OCT light two-dimensionally. For example, the CPU 13 acquires a plurality of two-dimensional tomographic images by scanning measurement light on each of a plurality of scan lines having different positions in a two-dimensional region when the tissue is viewed from the front. Next, the CPU 13 acquires a three-dimensional tomographic image by combining a plurality of captured two-dimensional tomographic images.

Further, the CPU 13 can capture a plurality of ophthalmic images of the same site by scanning the measurement light a plurality of times on the same site on the tissue (in the present embodiment, on the same scan line). The CPU 13 can acquire an averaging image in which the influence of speckle noise is suppressed by performing an averaging process on a plurality of ophthalmic images of the same portion. The image quality of the two-dimensional tomographic image can be improved by performing the addition averaging processing on a plurality of two-dimensional tomographic images of the same part. It is also possible to improve the image quality of the three-dimensional tomographic image by using each of the plurality of two-dimensional tomographic images constituting the three-dimensional tomographic image as an averaging image. The addition averaging process may be performed, for example, by averaging the pixel values of the pixels at the same position in a plurality of ophthalmic images. The larger the number of images to be subjected to the averaging process, the easier it is to suppress the influence of speckle noise, but the longer the shooting time. The ophthalmologic image capturing device 11 executes a tracking process for tracking the scanning position of the OCT light with the movement of the eye to be inspected while taking a plurality of ophthalmic images of the same site.

(Mathematical model construction process)
The mathematical model construction process executed by the mathematical model construction apparatus 1 will be described with reference to FIGS. 2 to 4. The mathematical model construction process is executed by the CPU 3 according to the mathematical model construction program stored in the storage device 4.

In the mathematical model construction process, the mathematical model is trained by the training data set, so that the mathematical model that outputs medical information based on the ophthalmic image is constructed. The training data set includes data on the input side (training data for input) and data on the output side (training data for output). It is possible to output various medical information to the mathematical model. The type of training data set used for training the mathematical model is determined according to the type of medical information output to the mathematical model. Hereinafter, an example of the relationship between the types of medical information output to the mathematical model and the training data set used for training the mathematical model will be described.

First, a case where a two-dimensional tomographic image (high-quality image) in which the image quality of the basic image is improved by inputting the two-dimensional tomographic image as a basic image into the mathematical model is output to the mathematical model as medical information will be described. In this case, in the present embodiment, the two-dimensional tomographic image of the tissue of the eye to be inspected is used as input training data, and the two-dimensional tomographic image of the same site having higher image quality than the input training data is used as output training data. Mathematical models are trained. The high-quality image indicates, for example, at least one of an image in which noise of the input base image is reduced, an image in which the resolution of the original image is increased, an image in which the visibility of the original image is improved, and the like.

FIG. 2 shows an example of input training data and output training data when a high-quality two-dimensional tomographic image is output to a mathematical model. In the example shown in FIG. 2, the CPU 3 acquires a set 40 of a plurality of two-dimensional tomographic images 400A to 400X in which the same part of the tissue is photographed. The CPU 3 uses a part of the plurality of two-dimensional tomographic images 400A to 400X in the set 40 (the number of images smaller than the number of images used for the addition averaging of the output training data described later) as the input training data. Further, the CPU 3 acquires the additional average images 41 of the plurality of two-dimensional tomographic images 400A to 400X in the set 40 as output training data. When a mathematical model is trained using the input training data and output training data illustrated in FIG. 2, the influence of speckle noise is suppressed by inputting a two-dimensional tomographic image as a base image in the trained mathematical model. The high-quality image is output.

Next, a case where an Enface image (high-quality image) with improved image quality of the base image is output to the mathematical model by inputting the Enface image as the base image into the mathematical model will be described. In this case, in the present embodiment, the mathematical model is trained by using the Enface image of the tissue of the eye to be inspected as the input training data and the Enface image of the same portion having a higher image quality than the input training data as the output training data. Will be done.

An example of the Enface image 55 will be described with reference to FIG. The Enface image 55 is a direction (front surface) along the optical axis of the measurement light of the ophthalmic imaging apparatus (OCT apparatus) 11 for a part of the three-dimensional tomographic image (partial image) included in the entire image region of the three-dimensional tomographic image 50. It is a two-dimensional image when viewed from the direction). The Enface image 55 may be, for example, an integrated image obtained by integrating the pixel values of the partial images in the depth direction (Z direction) at each position in the XY direction intersecting the optical axis. Further, the Enface image 55 may be generated by the integrated value of the spectral data of the partial image at each position in the XY direction. In the present embodiment, of the image region of the three-dimensional tomographic image 50, the three-dimensional tomographic image sandwiched between the two boundary surfaces (slabs) 52A and 52B is regarded as a partial image, and the Enface image 55 of the partial image is generated. .. In the example shown in FIG. 3, two boundary surfaces 52A and 52B are formed on one two-dimensional tomographic image 51 (for example, a two-dimensional tomographic image passing through the center of the three-dimensional tomographic image 50) constituting the three-dimensional tomographic image 50. The location is illustrated. When the positions of the boundary surfaces 52A and 52B are changed, the partial image and the Enface image 55 generated from the partial image are also changed.

An example of input training data and output training data when a high-quality Enface image 55 is output to a mathematical model will be described. The CPU 3 acquires a plurality of three-dimensional tomographic images taken of the same tissue (in the present embodiment, the fundus of the same eye to be inspected) by the fundus imaging apparatus 11A. The plurality of three-dimensional tomographic images include a low-quality three-dimensional tomographic image and a high-quality three-dimensional tomographic image. As described above, the high-quality three-dimensional tomographic image may be acquired by using each of the plurality of two-dimensional tomographic images constituting the image as an averaging image. Next, the CPU 3 extracts a part of the three-dimensional tomographic image as a partial image from the low-quality three-dimensional tomographic image, and uses the Enface image generated from the extracted partial image as input training data. The CPU 3 extracts a three-dimensional tomographic image in the same range as the input training data as a partial image from the high-quality three-dimensional tomographic image, and uses the Enface image generated from the extracted partial image as the output training data. When a mathematical model is trained by input training data and output training data, an Enface image is input to the trained mathematical model as a base image, so that a high-quality image with improved image quality of the base image is output. To.

It is also possible to change the method of generating a high-quality two-dimensional tomographic image or three-dimensional tomographic image. For example, the image quality may be improved by a process other than the addition averaging process. Moreover, the training data for input is not limited to the tomographic image of the tissue.

The mathematical model may also be trained to output medical information other than high quality images. For example, by inputting an ophthalmic image into a mathematical model, it is possible to output the analysis result of the structure of the tissue of the eye to be examined to the mathematical model as medical information. In this case, the mathematical model is trained using the ophthalmic image of the tissue of the eye to be inspected as input training data and the medical data showing the structure of the tissue shown in the input training data (ophthalmic image) as output training data. To. As an example, a two-dimensional tomographic image of the fundus tissue is used as input training data, and data indicating the position of the structure of the tissue (for example, the position of at least one of the layers and boundaries of the tissue) shown in the input training data is output. A mathematical model may be trained as training data. In this case, when a two-dimensional tomographic image is input to the trained mathematical model, the analysis result of the structure of the tissue (for example, at least one position of the layer and the boundary) is output. It is also possible to output the analysis results of at least one of the fundus blood vessels, the fovea centralis, and the optic nerve head of the eye to be inspected to a mathematical model. The training data for input does not have to be a two-dimensional tomographic image, and may be a two-dimensional front image or the like.

In addition, by inputting an ophthalmic image into the mathematical model, it is possible to output the automatic analysis result indicating whether or not there is any disease in the eye to be examined to the mathematical model as medical information. In this case, an ophthalmic image of the tissue of the eye to be inspected (for example, at least one of a two-dimensional tomographic image, a three-dimensional tomographic image, an Enface image, etc.) is used as input training data, and an imaging target reflected in the input training data. A mathematical model may be trained using data indicating the presence or absence of a disease in the above and at least one of the types of diseases as training data for output.

The mathematical model construction process will be described with reference to FIG. The CPU 3 acquires at least a part of the ophthalmic image taken by the ophthalmic image capturing device 11A as input training data (S1). In the present embodiment, the data of the ophthalmic image is generated by the ophthalmic imaging apparatus 11A and then acquired by the mathematical model construction apparatus 1. However, the CPU 3 acquires the data of the ophthalmic image by acquiring the signal (for example, OCT signal) that is the basis for generating the ophthalmic image from the ophthalmic imaging apparatus 11A and generating the ophthalmic image based on the acquired signal. You may.

Next, the CPU 3 acquires output training data corresponding to the input training data acquired in S1 (S3). An example of the correspondence between the input training data and the output training data is as described above.

Next, the CPU 3 uses a machine learning algorithm to train a mathematical model using the training data set (S3). As machine learning algorithms, for example, neural networks, random forests, boosting, support vector machines (SVMs), and the like are generally known.

Neural networks are a technique that mimics the behavior of biological nerve cell networks. Neural networks include, for example, feed-forward (forward propagation) neural networks, RBF networks (radial basis functions), spiking neural networks, convolutional neural networks, recursive neural networks (recurrent neural networks, feedback neural networks, etc.), and probabilities. Neural networks (Boltzmann machines, Basian networks, etc.).

Random forest is a method of learning based on randomly sampled training data to generate a large number of decision trees. When using a random forest, the branches of a plurality of decision trees learned in advance as a discriminator are traced, and the average (or majority vote) of the results obtained from each decision tree is taken.

Boosting is a method of generating a strong classifier by combining a plurality of weak classifiers. A strong classifier is constructed by sequentially learning simple and weak classifiers.

SVM is a method of constructing a two-class pattern classifier using a linear input element. The SVM learns the parameters of the linear input element based on, for example, the criterion (hyperplane separation theorem) of obtaining the margin maximizing hyperplane that maximizes the distance from each data point from the training data.

A mathematical model refers, for example, to a data structure for predicting the relationship between input data and output data. Mathematical models are constructed by training with training datasets. As described above, the training data set is a set of training data for input and training data for output. For example, training updates the correlation data (eg, weights) for each input and output.

In this embodiment, a multi-layer neural network is used as a machine learning algorithm. A neural network includes an input layer for inputting data, an output layer for generating the data to be predicted, and one or more hidden layers between the input layer and the output layer. A plurality of nodes (also called units) are arranged in each layer. Specifically, in this embodiment, a convolutional neural network (CNN), which is a kind of multi-layer neural network, is used. However, other machine learning algorithms may be used. For example, a hostile generative network (GAN) that utilizes two competing neural networks may be adopted as a machine learning algorithm.

The processes of S1 to S3 are repeated until the construction of the mathematical model is completed (S5: NO). When the construction of the mathematical model is completed (S5: YES), the mathematical model construction process is completed. The program and data for realizing the constructed mathematical model are incorporated in the ophthalmologic image processing apparatus 21.

(Ophthalmic image processing)
An example of ophthalmic image processing executed by the ophthalmic image processing apparatus 21 will be described with reference to FIGS. 5 to 11. 5 to 11 show an example in which a plurality of two-dimensional tomographic images constituting the three-dimensional tomographic image of the fundus are used as partial images, and a high-quality image in which the image quality of each partial image is improved is acquired and displayed. .. The ophthalmic image processing illustrated in FIG. 5 is executed by the CPU 23 according to the ophthalmic image processing program stored in the storage device 24.

As shown in FIG. 5, the CPU 23 acquires an ophthalmologic image of the tissue of the eye to be inspected taken by the ophthalmologic imaging apparatus (OCT apparatus in this embodiment) 11B (S11). In S11 of the present embodiment, a three-dimensional tomographic image of the fundus tissue of the eye to be inspected (see, for example, the three-dimensional tomographic image 50 shown in FIG. 3) is acquired. As described above, a three-dimensional tomographic image is generated by combining (arranging) a plurality of two-dimensional tomographic images.

Next, the CPU 23 causes the display device 28 to display the partial image selection image 61 (S12). FIG. 6 is an example of a display screen 60 displayed on the display device 28 during execution of ophthalmic image processing. A partial image selection image 61 and a selection image display field 65 are displayed on the display screen 60.

The partial image selection image 61 is displayed for the user to select at least one of a plurality of partial images 66 (two-dimensional tomographic images in the present embodiment) included in the three-dimensional tomographic image. In the example shown in FIG. 6, the partial image selection image 61 includes a two-dimensional front image when the three-dimensional tomographic image is viewed from the optical axis direction of the OCT measurement light. A selection range 62 in which the partial image 66 can be selected is displayed in the image area of the two-dimensional front image. Within the selection range 62, a selection line 63 through which the selected partial image 66 passes is displayed. The user operates the operation unit 27 (for example, a mouse or the like) to set (for example, move) the position of the selection line 63 to a desired position, thereby inputting an instruction to select the partial image 66 to the ophthalmic image processing device 21. can do. The CPU 23 sets the partial image 66 at the position where the selection line 63 passes as the partial image 66 selected by the user.

In the selected image display field 65, at least one of the partial image 66 selected by the user and the high-quality image 67 (see FIG. 7) relating to the selected partial image 66 is displayed. In the example shown in FIG. 6, since the inference process for acquiring the high-quality image 67 related to the partial image 66 is in progress, the partial image 66 itself is displayed. When the acquisition of the high-quality image 67 is completed, the high-quality image 67 is displayed instead of the partial image 66. Further, when the high-quality image 67 related to the selected partial image 66 has already been acquired by the pre-acquisition process (see FIG. 11) described later, the partial image 66 is selected in the selected image display field 65. At this point, the high quality image 67 for the selected partial image 66 is displayed. Therefore, the work time of the user is appropriately shortened.

Further, the display screen 60 (in the present embodiment, the selected image display field 65 in the display screen 60) includes partial image switching buttons 68F and 68B. The partial image switching button 68F adjacents the selected partial image 66 in the forward direction (for example, in the example shown in FIG. 6, the downward direction of the screen in the selection range 62) from the partially image 66 selected at that time. It is operated to change (switch) to the existing partial image 66. Further, the partial image switching button 68B adjacents the selected partial image 66 in the opposite direction (for example, in the example shown in FIG. 6, the upward direction on the screen in the selection range 62) from the partial image 66 selected at that time. It is operated to change to the partial image 66. The user can sequentially switch the partial image 66 in a predetermined direction by continuously operating the partial image switching button 68F or the partial image switching button 68B.

Further, the image type display unit 69 is displayed on the display screen 60. The image type display unit 69 displays whether the image displayed in the selected image display field 65 at that time is a partial image 66 selected by the user or a high-quality image 67 related to the partial image 66. Will be done. Therefore, the user can easily grasp whether or not the image displayed in the selected image display field 65 is the high-quality image 67.

In this embodiment, the thickness distribution of a specific layer (the layer between ILM and RPE / BM in the example shown in FIG. 6) in the fundus tissue is distributed in the selection range 62 in the partial image selection image 61. A three-dimensional thickness map is superimposed and displayed. Therefore, the user can select the partial image 66 after considering the distribution of the thickness of a specific layer in the fundus.

Further, when the CPU 23 starts the display of the display screen 60 in S12, the CPU 23 displays a high-quality image 67 relating to the partial image 66 of the default pattern (for example, the partial image 66 passing through the center of the image area of the two-dimensional front image). First, it is automatically displayed in the selected image display field 65. Here, before displaying the display screen 60, the CPU 23 inputs the partial image 66 of the default pattern into the mathematical model to acquire the high-quality image 67 related to the partial image 66 of the default pattern in advance. deep. The process of pre-acquiring the high-quality image 67 corresponding to the default pattern is also an example of the pre-acquisition process. When displaying the display screen 60, the CPU 23 displays the high-quality image 67 corresponding to the default pattern in the selected image display field 65 from the beginning. Therefore, the high-quality image 67 corresponding to the default pattern is smoothly displayed.

Returning to the description of FIG. The CPU 23 determines whether or not the pre-acquisition pattern is set by the user (S13). FIG. 8 shows an example of a setting screen displayed on the display device 28 when the user sets the pre-acquisition pattern. The pre-acquisition pattern is a pattern of the partial image 66 in which medical information (high-quality image 67 in the present embodiment) is acquired in advance among a plurality of partial images 66 in an ophthalmic image (three-dimensional tomographic image in the present embodiment). Is.

As shown in FIG. 8, when the user inputs an instruction to set the pre-acquisition pattern, the CPU 23 displays the pattern selection field 70, the position designation field 71, the angle designation field 72, and the size designation field 73 on the display device 28. To display. In the pattern selection field 70, a plurality of types of pre-acquired patterns in which at least one of the arrangement, the number, and the shape of the lines through which the partial image 66 passes is different from each other are displayed. The user can arbitrarily select one or more of a plurality of types of pre-acquisition patterns.

In the position designation field 71, a position on the partial image selection image 61 (see FIG. 6) for setting the selected pre-acquisition pattern is designated. For example, if "macula center" is specified, the position of the pre-acquired pattern is set so that the center of the selected pre-acquired pattern coincides with the macula (eg, the fovea centralis, which is the center of the macula). The positions of the macula, papilla, abnormal site, etc. in the ophthalmic image may be detected by performing image processing on the ophthalmic image. In addition, the position of the macula or the like may be acquired by using a mathematical model trained by a machine learning algorithm. When "manual setting" is specified, the selected pre-acquisition pattern is set at the position specified by the instruction from the user. In the angle designation field 72, the angle of the selected pre-acquired pattern on the ophthalmologic image is designated. In the size designation field 90, the size of the selected pre-acquired pattern on the ophthalmologic image is set.

Returning to the description of FIG. When the pre-acquisition pattern is set (S13: YES), the CPU 23 inputs the partial image 66 corresponding to the set pre-acquisition pattern into the mathematical model to obtain the image quality of the input partial image 66. The improved high-quality image 67 is acquired as medical information. The process of acquiring the high-quality image 67 corresponding to the pre-acquisition pattern is an example of the pre-acquisition process. The CPU 23 stores the acquired high-quality image 67 in the storage device 24 and displays it on the display device 28 (S14). Therefore, the user can appropriately confirm the high-quality image 67 corresponding to the desired pattern.

The high-quality image 67 corresponding to the pre-acquisition pattern may be executed in advance before the process (S12) of starting the display of the display screen 60. In this case, in S12, the high-quality image 67 corresponding to the pre-acquisition pattern may be displayed in the selected image display field 65 from the beginning. As a result, the high-quality image 67 corresponding to the pre-acquisition pattern is displayed on the display device 28 more smoothly.

Next, the CPU 23 determines whether or not the follow-up position information is stored in the storage device 24 (S16). The follow-up position information refers to the position of the partial image 66 selected by the user (that is, the position of the reference image described later) when the ophthalmic image processing is performed on the ophthalmic image having the same imaging target in the past. This is the information to be shown. When the follow-up position information is stored (S16: YES), the CPU 23 sets the partial image 66 of the follow-up position as the reference image. The reference image will be described later. When a plurality of follow-up positions are stored, one of the follow-up positions (for example, the position of the partial image 66 first selected by the user in the past processing) may be adopted. When the follow-up position information is stored, the partial image 66 at the follow-up position may be set as the default partial image 66 for first displaying the high-quality image 67 on the display screen 60.

When the follow-up position information is not stored (S16: NO), the CPU 23 acquires a specific position of the tissue shown in the ophthalmologic image (S18). The specific position may be, for example, at least one of the macula on the fundus, the optic nerve head, the lesion site, the fundus blood vessel, and the like. The CPU 23 may detect a specific position by performing image processing on an ophthalmic image, for example. Further, the CPU 23 may acquire a specific position by using a mathematical model trained by a machine learning algorithm. The CPU 23 sets the partial image 66 at the specific position as a reference image described later (S19). However, the specific position may be predetermined. For example, the position of the partial image 66 of the default pattern described above may be set as a specific position.

Next, the CPU 23 sets the priority when executing the pre-acquisition process (see FIG. 9) described later for each of the plurality of partial images 66 included in the ophthalmic image based on the reference image (S21). The priority order indicates the order in which the pre-acquisition process is executed for each of the plurality of partial images 66. That is, the process of acquiring medical information (high-quality image 66 in the present embodiment) for each partial image 66 is executed in order of priority. The reference image is a partial image 66 that serves as a reference for setting a priority for a plurality of partial images 66.

In S21 of the present embodiment, the CPU 23 sets the priority of the plurality of partial images 66 so that the closer to the reference image, the higher the priority. Specifically, in S21 of the present embodiment, the CPU 23 sets the priority of the plurality of partial images 66 so that the closer the distance to the reference image, the higher the priority.

FIG. 9 is an explanatory diagram for explaining an example of a method of setting priorities for a plurality of partial images 66 with reference to the reference image BL. FIG. 9 shows a state in which a line through which each of the plurality of partial images 66 passes is viewed from a direction along the optical axis of the OCT measurement light. As shown in FIG. 9, in S21 of the present embodiment, the CPU 23 gives priority to two partial images 66 adjacent to each of the forward direction (lower direction in the figure) and the reverse direction (upper direction in the figure) of the reference image BL. The ranking is the highest P1. Further, the CPU 23 sets the priority order so that the farther the distance from the reference image BL is, the lower the priority order is in the order of P2 and P3 in both the forward direction and the reverse direction. As a result, medical information regarding the partial image 66 close to the reference image BL is preferentially acquired.

Returning to the description of FIG. The CPU 23 starts the pre-acquisition process (S22). In the pre-acquisition process, medical information regarding the partial image 66 that has not yet been selected by the user is acquired and stored in advance. Details of the pre-acquisition process will be described later with reference to FIG.

Next, the CPU 23 determines whether or not the instruction to select the partial image 66 has been input by the user (S24). If no instruction is input (S24: NO), the process proceeds to S29 as it is. When the CPU 23 receives the input of the instruction to select the partial image 66 (S24: YES), the CPU 23 sets the selected partial image 66 as a new reference image (S25). The CPU 23 stores the newly set position information of the reference image in the storage device 24 as follow-up position information to be used when performing ophthalmic image processing on the same ophthalmologic image of the eye to be inspected in the future. S26). The CPU 23 resets the priority order for the plurality of partial images 66 in the order of proximity to the newly set reference image (S27).

In S27, when the partial image 66 selected by the user is changed, the CPU 23 performs a plurality of partial images 66 along the direction of the changed partial image 66 with respect to the changed partial image 66. Priority can be set in order. For example, when the selected partial image 66 is switched by operating the partial image switching button 66F or the partial image switching button 66B (see FIG. 6), the user can switch in the same direction (forward or reverse direction). In many cases, the partial image 66 to be selected is continuously switched along with the above. Therefore, when the partial image 66 is switched by the partial image switching button 66F or the partial image switching button 66B, the CPU 23 of the present embodiment resets the priority order along the direction in which the partial image 66 is switched.

FIG. 10 is an explanatory diagram for explaining an example of a method of setting priorities for a plurality of partial images 66 along the direction in which the partial images 66 selected by the user are switched. In the example shown in FIG. 10, the partial image 66 selected by the user is switched from BL1 to BL2. In this case, the CPU 23 changes the reference image from BL1 to BL2 (S25), and the direction of the changed partial image 66 (BL2) with respect to the changed partial image 66 (BL1) (in FIG. 10). In the example shown, the priority is reset along the forward direction (downward of the drawing) (S27). That is, with respect to the changed direction, the priority is set so that the priority is lowered in the order of P1, P2, P3 as the distance from the reference image BL2 increases.

Next, the CPU 23 causes the medical information (high-quality image 67 in the present embodiment) regarding the partial image 66 selected in S24 to be displayed on the display device 28 (in the present embodiment, the selected image display field 65 shown in FIG. 6). (S28). Specifically, when the medical information regarding the selected partial image 66 has not yet been acquired, the CPU 23 inputs the selected partial image 66 into the mathematical model to obtain the medical information regarding the input partial image 66. Get and display. As described above, the selected partial image 66 is displayed in the selected image display field 65 while the process of acquiring medical information is being executed. When the medical information regarding the selected partial image 66 has already been acquired and stored in the storage device 24, the CPU 23 causes the display device 24 to display the stored medical information.

The processes of S24 to S29 are repeated until an instruction to end the process is input (S29: NO). When the end instruction is input (S29: YES), the ophthalmologic image processing ends.

An example of the pre-acquisition process executed by the ophthalmologic image processing apparatus 21 will be described with reference to FIG. As described above, in the pre-acquisition process, medical information regarding the partial image 66 that has not yet been selected by the user is acquired and stored in advance. The medical information acquisition process for each partial image 66 is executed in order according to the set priority. The pre-acquisition process illustrated in FIG. 11 is executed by the CPU 23 according to the ophthalmologic image processing program stored in the storage device 24. The pre-acquisition process is executed in the background in parallel with the ophthalmologic image processing illustrated in FIG.

First, the CPU 23 sets the value of the counter N for counting the priority to "1" (S31). The CPU 23 determines whether or not the medical information regarding the partial image 66 having the Nth priority has already been acquired (S32). When the medical information regarding the Nth partial image 66 is already stored in the storage device 24 (S32: YES), the value of the counter N is set to "1" in order to avoid duplication of processing and shorten the processing time. "Is added (S33). The process returns to S32. That is, when the medical information regarding the Nth partial image 66 has already been acquired, the process of acquiring the medical information is omitted.

If medical information about the Nth partial image 66 has not yet been obtained (S32: NO), the CPU 23 uses the Nth partial image 66 as the base image and is based on a mathematical model trained by a machine learning algorithm. Enter the image. As a result, an inference process using a mathematical model for outputting a high-quality image 67 (see FIG. 7) with improved image quality of the base image as medical information is started (S34).

The CPU 23 determines whether or not a new reference image has been set in the ophthalmic image processing (see FIG. 5) (S35). If the reference image is not newly set (S35: NO), it is determined whether or not the acquisition process of medical information regarding the Nth partial image 66 (that is, the inference process started in S34) is completed (that is, whether or not the inference process started in S34) is completed ( S36). If the medical information acquisition process is not completed (S36: NO), the process returns to S35, and the processes of S35 and S36 are repeated.

When the process of acquiring the medical information regarding the Nth partial image 66 is completed (S36: YES), the CPU 23 stores the acquired medical information in the storage device 24 (S38). The CPU 23 adds "1" to the value of the counter N for counting the priority (S39). The CPU 24 performs a series of medical information acquisition processes (that is, a group of portions executed using the current counter N) for a plurality of partial images 66 for which priorities have been set in ophthalmic image processing (see FIG. 5). It is determined whether or not the medical information acquisition process for the image 66) has been completed (S40). If it is not completed (S40: NO), the process returns to S32, and the process for the partial image 40 in which the next priority is set is performed (S32 to S38).

Before the acquisition process of medical information regarding the Nth partial image 66 is completed (S36: NO), if a new reference image is set in the ophthalmologic image processing (see FIG. 5) (S35: YES), medical treatment It is desirable to change the priority for acquiring information to the priority based on the newly set reference image. Therefore, the CPU 23 interrupts the pre-acquisition process (inference process) being executed (S46). The priority set reset in S27 based on the newly set reference image is set again (S47). The process returns to S31, and the pre-acquisition process (S31 to S40) is executed according to the priority set again. That is, the CPU 23 interrupts the processing for the group of partial images 66 that has been executed, and resumes the processing for the new group of partial images 66.

When the medical information acquisition process for the group of partial images 66, which was executed using the current counter N, is completed (S40: YES), the CPU 23 processes the other group of partial images 66 that was interrupted in S46. It is determined whether or not there is (S41). If there is an interrupted process (S41: YES), the CPU 23 resets the priority of the interrupted process and the counter N (S42), and the process returns to S32. As a result, the suspended processing is properly resumed.

If there is no interrupted process (S41: NO), it is determined whether or not the process of acquiring medical information regarding all the partial images 66 included in the ophthalmologic image is completed (S44). If it is not completed (S44: NO), a new priority is set to execute the process for the partial image 66 for which medical information has not yet been acquired (S49), and the process returns to S31. When the medical information regarding all the partial images 66 is acquired (S44: YES), the pre-acquisition process ends.

(First transformation example)
A first transformation example of the above embodiment will be described with reference to FIG. FIG. 12 is an example of the display screen 80 displayed on the display device 28 during the execution of the ophthalmic image processing in the first transformation example. The ophthalmic image processing apparatus 1 according to the first transformation example uses a part of the three-dimensional tomographic image included in the entire image area of the three-dimensional tomographic image as a partial image, and generates an Enface image 55 from the partial image. The ophthalmic image processing apparatus 1 acquires high-quality images 85 (85A, 85B, 85C) with improved image quality of the generated Enface image 55 as medical information. At least a part of the ophthalmologic image processing (see FIG. 5) and the pre-acquisition processing (see FIG. 11) exemplified in the above embodiment can be similarly adopted in the following first transformation example. Therefore, the description of the process for which the same process as that of the above embodiment can be adopted will be omitted or simplified.

In the ophthalmologic image processing (see FIG. 5) according to the first transformation example, first, a three-dimensional tomographic image of the fundus tissue of the eye to be inspected (see, for example, the three-dimensional tomographic image 50 shown in FIG. 3) is acquired (S11). Next, the CPU 23 causes the display device 28 to display the partial image selection image 81 (see FIG. 12) (S12). As shown in FIG. 12, during the execution of the ophthalmic image processing, the partial image selection image 81 and the high-quality images 85A, 85B, 85C are displayed on the display screen 80. For the partial image selection image 81 of the first transformation example, one two-dimensional tomographic image constituting the three-dimensional tomographic image acquired in S11 is used. The partial image selection image 81 displays the positions of the two boundary surfaces (slabs) 82A and 82B for defining the range of the partial image. The three-dimensional tomographic image in the region sandwiched between the two boundary surfaces 82A and 82B is the partial image in the first transformation example.

Internally, for each of the plurality of two-dimensional tomographic images constituting the three-dimensional tomographic image acquired in S11, at least one of the layer of the tissue and the boundary between the layers reflected in the image (hereinafter, simply "layer / boundary"). The position of) has been acquired. The position of the layer / boundary may be acquired by image processing on the two-dimensional tomographic image, or may be acquired by using a machine learning algorithm. Further, the position may be acquired by the user specifying the position of the layer / boundary.

The user operates the operation unit 27 to set (for example, move) at least one of the displayed two boundary surfaces 82A and 82B to a desired position. As a result, the positions of the boundary surfaces 82A and 82B that define the range of the partial image are set. That is, the user can select a partial image in the ophthalmic image acquired in S11 by setting the positions of the boundary surfaces 82A and 82B. As an example, the CPU 23 has a plurality of two-dimensional tomographic images that constitute a three-dimensional tomographic image based on the distance and direction of the boundary surfaces 82A and 82B set by the user with respect to the positions of the layers and boundaries of the tissue that have already been acquired. Two interface planes are set for each of the images. The CPU 23 uses a three-dimensional tomographic image of a region sandwiched between the two set boundary surfaces as a partial image.

Returning to the description of FIG. The CPU 23 determines whether or not the pre-acquisition pattern is set by the user (S13). In the pre-acquisition pattern in the first transformation example, a pattern in which a specific layer that is likely to be noticed by the user is used as a partial image is provided in advance. In the example shown in FIG. 12, a pattern in which the vitreous interface is a partial image and a pattern in which the inner layer of the retina is a partial image are set. The CPU 23 generates an Enface image 55 (see FIG. 3) based on the partial image corresponding to the set pre-acquisition pattern. The CPU 23 acquires high-quality images 85A and 85B by inputting the generated Enface image 55 into a mathematical model as a base image. The process of acquiring the high-quality images 85A and 85B corresponding to the pre-acquisition pattern is an example of the pre-acquisition process. The CPU 23 stores the acquired high-quality images 85A and 85B in the storage device 24 and displays them on the display device 28 (S14).

As described above, the high-quality images 85A and 85B corresponding to the pre-acquisition pattern may be executed in advance before the process (S12) of starting the display of the display screen 80. In this case, in S12, the high-quality images 85A and 85B corresponding to the pre-acquisition pattern may be displayed on the display screen 80 from the beginning. As a result, the high-quality images 85A and 85B corresponding to the pre-acquisition pattern are displayed on the display device 28 more smoothly.

Next, the CPU 23 determines whether or not the follow-up position information is stored in the storage device 24 (S16). The follow-up position may be set based on the positions of the two boundary surfaces 82A and 82B selected by the user in the past processing. Further, the specific position used in the processing of S18 and S19 may be the position of a specific layer / boundary.

In S21 in the first transformation example, the priority of the partial image is set based on at least one of the closeness of the distance, the closeness of the range, the closeness of the overlap ratio, etc. to the three-dimensional tomographic image used as the reference image. To. For example, the CPU 23 moves the positions of the two boundary surfaces 82A and 82B repeatedly by a unit distance while keeping the distance between the two boundary surfaces 82A and 82B constant, and the CPU 23 moves to the boundary surfaces 82A and 82B at the respective positions. Priority may be set for a plurality of sandwiched partial images in order of proximity to the reference image. Further, the CPU 23 gives priority to a plurality of partial images sandwiched between the boundary surfaces 82A and 82B at each position in order of proximity to the reference image when one position of the two boundary surfaces 81A and 82B is repeatedly moved by a unit distance. You may set the order.

Further, in S27 in the first transformation example, the priority may be set according to the direction in which the partial image is changed (that is, the direction in which the boundary surfaces 82A and 82B are changed). Further, in S28 in the first transformation example, as shown in FIG. 12, the high-quality image 85C of the Enface image 55 based on the selected partial image is displayed on the display screen 28.

Further, in S34 (see FIG. 11) in the first transformation example, the CPU 23 generates an Enface image 55 based on the Nth priority partial image. The CPU 23 acquires the high-quality image 85C by inputting the generated Enface image 55 into the mathematical model.

(Second transformation example)
A second transformation example of the above embodiment will be described. The ophthalmic image processing apparatus 1 of the above embodiment and the first modified example acquires a high-quality image with improved image quality of the base image as medical information. However, the medical information acquired by the ophthalmic image processing apparatus 1 is not limited to high-quality images. The ophthalmic image processing device 1 of the second transformation example acquires the analysis result of the structure of the tissue of the eye to be inspected as medical information by inputting the partial image or the image generated based on the partial image into the mathematical model. Specifically, the medical information may be the analysis result (segmentation result) of the tissue layer / boundary. Further, the medical information may be the analysis result of the lesion site of the tissue, the blood vessel of the fundus, or the like. In this case, the above-mentioned training method or the like can be adopted as the training method of the mathematical model. In addition, the medical information may be the analysis result regarding the disease of the eye to be examined.

The techniques disclosed in the above embodiments are merely examples. Therefore, it is possible to modify the techniques exemplified in the above embodiments. First, it is also possible to implement only some of the plurality of techniques exemplified in the above embodiments. For example, it is possible to omit at least one of S13 to S14, S16 to S17, and S18 to S19 in the ophthalmic image processing (see FIG. 5). Further, instead of executing the processes of S18 and S19, the CPU 23 sets a predetermined partial image (for example, a partial image including the center of the image area of the ophthalmic image acquired in S11) as a reference image. You may. Further, the process (S22) for starting the pre-acquisition process may be executed after the partial image selection instruction is received.

The process of acquiring an ophthalmic image in S11 of FIG. 5 is an example of the “image acquisition step”. The process of acquiring medical information in S14, S28 of FIG. 5 and S34 to S38 of FIG. 11 is an example of the “medical information acquisition step”. The process of receiving the partial image selection instruction in S24 of FIG. 5 is an example of the “instruction receiving step”. The process of displaying the medical information on the display device 28 in S14 and S28 of FIG. 5 is an example of the “medical information display step”.

11A, 11B Ophthalmic Imaging Device 21 Ophthalmic Image Processing Device 23 CPU
24 Storage device 27 Operation unit 28 Display device 50 Three-dimensional tomographic image 51 Two-dimensional tomographic image 55 Enface image 66 Partial image 67 High-quality image 82A, 82B Boundary surface 85A, 85B, 85C High-quality image

Claims (10)

An ophthalmic image processing program executed by an ophthalmic image processing device that processes an ophthalmic image that is an image of the tissue of the eye to be inspected.
When the ophthalmic image processing program is executed by the control unit of the ophthalmic image processing device,
An image acquisition step of acquiring an ophthalmic image of the tissue of the eye to be inspected taken by the OCT device, and
By using a partial image of the acquired ophthalmic image or an image generated based on the partial image as a base image and inputting the base image into a mathematical model trained by a machine learning algorithm. , A medical information acquisition step of acquiring a high-quality image with improved image quality of the basic image as medical information and storing it in a storage device.
An instruction receiving step that accepts an input of an instruction for selecting the partial image in the acquired ophthalmic image, and
When the instruction is accepted, a medical information display step of displaying the medical information acquired for the partial image selected by the instruction on the display unit, and a medical information display step.
Is executed by the ophthalmic image processing apparatus.
The medical information acquisition step is characterized in that a pre-acquisition process of pre-acquiring and storing the medical information regarding the partial image of the ophthalmic images that has not yet been selected by the instruction is executed. Ophthalmic image processing program to do. The ophthalmic image processing program according to claim 1.
In the pre-acquisition process, the control unit sequentially executes the acquisition process of the medical information regarding each of the partial images in descending order of priority set for the plurality of the partial images in the ophthalmic image. An ophthalmic image processing program characterized by. The ophthalmic image processing program according to claim 2.
In the pre-acquisition process, the control unit uses one or more of the plurality of partial images in the ophthalmic image as a reference image, and the higher the priority of each of the plurality of the partial images, the closer to the reference image. An ophthalmic image processing program characterized by setting. The ophthalmic image processing program according to claim 3.
In the pre-acquisition process, the control unit acquires a specific position of the tissue to be captured in the ophthalmic image, and uses the partial image at the acquired specific position as the reference image. The ophthalmic image processing program according to claim 3 or 4.
In the pre-acquisition process, when the instruction is received, the control unit sets the partial image selected by the instruction as the reference image, which is an ophthalmologic image processing program. The ophthalmic image processing program according to claim 5.
In the pre-acquisition process, when the partial image selected by the instruction is changed, the control unit follows the direction of the partial image after the change with respect to the partial image before the change. An ophthalmic image processing program, wherein the priority order for a plurality of the partial images is set in order. The ophthalmic image processing program according to claim 5 or 6.
In the pre-acquisition process, the control unit
When the partial image selected by the instruction is set as the reference image, the position of the reference image in the ophthalmic image is stored in the storage device, and the reference image is stored.
When processing the ophthalmic image whose imaging target is the same as that of the ophthalmic image in which the position of the reference image is stored in the past, the partial image of the position stored in the past among the ophthalmic images is used as the reference. An ophthalmic image processing program characterized by setting an image. The ophthalmic image processing program according to any one of claims 3 to 7.
When the reference image is newly set, the control unit interrupts the pre-acquisition process that has been executed, and restarts the pre-acquisition process according to the priority based on the newly set reference image. An ophthalmic image processing program characterized by this. The ophthalmic image processing program according to any one of claims 1 to 8.
Of the plurality of partial images in the ophthalmic image, a pattern of the partial image from which the medical information is acquired in advance is provided in advance.
An ophthalmic image processing program characterized in that, in the pre-acquisition process, the control unit executes an acquisition process of medical information regarding the partial image corresponding to the pattern. An ophthalmic image processing device that processes an ophthalmic image, which is an image of the tissue of the eye to be inspected.
The control unit of the ophthalmic image processing device
An image acquisition step of acquiring an ophthalmic image of the tissue of the eye to be inspected taken by the OCT device, and
By using a partial image of the acquired ophthalmic image or an image generated based on the partial image as a base image and inputting the base image into a mathematical model trained by a machine learning algorithm. , A medical information acquisition step of acquiring a high-quality image with improved image quality of the basic image as medical information and storing it in a storage device.
An instruction receiving step that accepts an input of an instruction for selecting the partial image in the acquired ophthalmic image, and
When the instruction is accepted, a medical information display step of displaying the medical information acquired for the partial image selected by the instruction on the display unit, and a medical information display step.
And
In the medical information acquisition step, the control unit executes a pre-acquisition process of pre-acquiring and storing the medical information regarding the partial image of the ophthalmic images that has not yet been selected by the instruction. An ophthalmic image processing device characterized by this.