WO2022157838A1

WO2022157838A1 - Image processing method, program, image processing device and ophthalmic system

Info

Publication number: WO2022157838A1
Application number: PCT/JP2021/001731
Authority: WO
Inventors: 泰士田邉; 真梨子向井; 媛テイ吉; 仁志田淵
Original assignee: 株式会社ニコン; 株式会社シンクアウト
Priority date: 2021-01-19
Filing date: 2021-01-19
Publication date: 2022-07-28
Also published as: JPWO2022157838A1

Abstract

This image processing method comprises: a process of acquiring a wide-angle fundus image obtained by imaging with an ophthalmic apparatus; a process of extracting a plurality of partial images from the wide-angle fundus image to which an abnormal finding is added; a selection process of selecting a first partial image showing an abnormal region that provides a reason of the abnormal finding from among the plurality of partial images; and a display process of displaying the first partial image.

Description

Image processing method, program, image processing apparatus, ophthalmic system

The present invention relates to an image processing method, program, image processing apparatus, and ophthalmic system.

Conventionally, there has been known a method of displaying an enlarged image by a user's operation in order to observe details in a fundus image (see Patent Document 1, for example).

U.S. Patent Application Publication No. 2020/0069175

The technology of the present disclosure provides a novel image processing method.

One embodiment of the present invention includes processing for acquiring a wide-angle fundus image obtained by photographing with an ophthalmologic apparatus, processing for extracting a plurality of partial images from the wide-angle fundus image to which an abnormal finding has been added, and the abnormal finding. and a display process for displaying the first partial image.

1 is an overall configuration diagram of an information processing system according to a first embodiment; FIG. It is a figure which shows the hardware constitutions of the information processing apparatus which concerns on 1st Embodiment. 1 is a diagram showing the configuration of an ophthalmologic apparatus according to a first embodiment; FIG. It is a figure which shows the functional structure of the server which concerns on 1st Embodiment. It is a figure which shows the structure of the detection part which concerns on 1st Embodiment. 4 is a flowchart showing an overview of processing according to the first embodiment; 6 is a flowchart showing estimation processing according to the first embodiment; 4 is a flowchart showing output processing according to the first embodiment; 6 is a flowchart showing addition processing according to the first embodiment; FIG. 10 is a diagram showing an example of partial area setting of a wide-angle fundus image and an example of a GUI using the generated partial image; FIG. 10 is a diagram showing an example of partial area setting of a wide-angle fundus image and an example of a GUI using the generated partial image; It is a figure which shows an example of GUI. It is a figure which shows an example of GUI. It is a figure which shows an example of GUI. It is a figure which shows the functional structure of the server which concerns on 2nd Embodiment. 9 is a flowchart showing estimation processing according to the second embodiment;

<First Embodiment>
BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, the present invention will be described in accordance with a first embodiment, which is one embodiment thereof, with reference to the drawings.

〔Constitution〕
FIG. 1 shows the configuration of an information processing system 1 according to one embodiment of the present invention. The information processing system 1 includes a server 10 , a terminal 20 and an ophthalmologic apparatus 30 . The server 10, the terminal 20, and the ophthalmologic apparatus 30 are connected via the network 5 so as to be able to transmit and receive data to each other.

The network 5 is a wireless or wired communication means, such as the Internet, WAN (Wide Area Network), LAN (Local Area Network), public communication network, dedicated line, and the like. Although the information processing system 1 according to this embodiment is composed of a plurality of information management devices, the present invention does not limit the number of these devices. Therefore, the information processing system 1 can be configured with one or more devices as long as they have the following functions.

The server 10 and the terminal 20 are information processing devices installed in medical institutions such as hospitals and clinics, and are mainly operated by staff members of the medical institutions to acquire images captured by the ophthalmic device 30 and to obtain images. edit and analyze the data.

The ophthalmic device 30 is a device that performs SLO (Scanning Laser Ophthalmoscope) and OCT (Optical Coherence Tomography) (Fig. 3). The ophthalmologic apparatus 30 has a control device 31 and an imaging device 32 .

FIG. 2 shows an example of hardware (hereinafter referred to as "information processing apparatus 100") used for realizing the server 10, the terminal 20, and the control device 31 of the ophthalmologic apparatus 30. FIG. As shown in the figure, the information processing apparatus 100 includes a processor 101 , a main memory device 102 , an auxiliary memory device 103 , an input device 104 , an output device 105 and a communication device 106 . These are communicably connected to each other via communication means such as a bus (not shown).

It should be noted that the information processing apparatus 100 does not necessarily have to be implemented entirely by hardware. may be realized by virtual resources such as

The processor 101 is configured using a CPU (Central Processing Unit), an MPU (Micro Processing Unit), and the like. The functions of the server 10, the terminal 20, and the control device 31 are implemented by the processor 101 reading out and executing the programs stored in the main storage device 102. FIG.

The main memory device 102 is a device that stores programs and data, and includes ROM (Read Only Memory), RAM (Random Access Memory), nonvolatile semiconductor memory (NVRAM (Non Volatile RAM)), and the like.

The auxiliary storage device 103 is, for example, SSD (Solid State Drive), various non-volatile memories (NVRAM: Non-volatile memory) such as SD memory cards, hard disk drives, optical storage devices (CD (Compact Disc), DVD (Digital Versatile Disc), etc.), cloud server storage area, etc. Programs and data stored in the auxiliary storage device 103 are read into the main storage device 102 at any time.

The input device 104 is an interface that accepts input of information, and includes, for example, a keyboard, mouse, touch panel, card reader, voice input device (microphone, etc.), voice recognition device, and the like. The information processing device 100 may be configured to receive input of information from another device via the communication device 106 .

The output device 105 is an interface that outputs various types of information. synthesizer and the like. The information processing device 100 may be configured to output information to another device via the communication device 106 . The output device 105 corresponds to the display section in the present invention.

The communication device 106 is a wired or wireless communication interface that realizes communication with other devices via the network 5, and includes, for example, a NIC (Network Interface Card), a wireless communication module, a USB (Universal Serial Interface ) module, serial communication module, and the like.

The configuration of the ophthalmologic apparatus 30 is shown in FIG. The ophthalmologic apparatus 30 includes an imaging device 32 and a control device 31 . Note that the control device 31 may be provided in the same housing as the imaging device 32 or may be provided separately from the imaging device 32 .

The imaging device 32 operates under the control of the control device 31. The imaging device 32 includes an SLO unit 33 , an imaging optical system 34 and an OCT unit 35 . The imaging optical system 34 includes an optical scanner 341 and a wide-angle optical system 342 . The photographing device 32 photographs an image of the subject's eye. The imaging device 32 captures, for example, the fundus of the subject's eye, and obtains a fundus image and a tomographic image (OCT image), which will be described later.

The SLO unit 18 acquires an image of the fundus 12A of the eye 12 to be examined. The OCT unit 20 acquires a tomographic image of the eye 12 to be examined. Hereinafter, a front view image of the retina created based on the SLO data acquired by the SLO unit 18 is referred to as an SLO image, and a tomographic image or a front view image of the retina created based on the OCT data acquired by the OCT unit 20 is referred to as an SLO image. An image (en-face image) or the like is called an OCT image. Note that the SLO image is sometimes referred to as a two-dimensional fundus image. An OCT image may also be referred to as a fundus tomographic image, a posterior segment tomographic image, or an anterior segment tomographic image, depending on the imaging region of the subject's eye 12 .

Hereinafter, the image captured by the ophthalmologic apparatus 30 may also be referred to as the image of the eye to be examined. Further, when the image of the subject's eye is an image of the fundus captured using a wide-angle optical system as described later, it may be referred to as a wide-angle fundus image P.

The optical scanner 341 scans the light emitted from the SLO unit 33 in the X direction and the Y direction. The optical scanner 341 may be an optical element capable of deflecting a light beam, and for example, a polygon mirror, a galvanomirror, or the like can be used.

The wide-angle optical system 342 includes an objective optical system. Wide-angle optics 342 provide a wide-angle field of view at the fundus. The SLO system is realized by the control device 31, the SLO unit 33, and the imaging optical system 34 shown in FIG. Since the SLO system includes the wide-angle optical system 342, observation in a wide field of view (FOV) is realized at the fundus. FOV indicates a range that can be photographed by the photographing device 32 . FOV can be expressed as a viewing angle. A viewing angle can be defined by an internal illumination angle and an external illumination angle in this embodiment. The external irradiation angle is an irradiation angle defined by using the pupil as a reference for the irradiation angle of the light flux irradiated from the UWF ophthalmic apparatus 110 to the eye to be examined. Further, the internal illumination angle is an illumination angle defined with the center O of the eyeball as a reference for the illumination angle of the luminous flux with which the fundus 12A is illuminated. The external illumination angle and the internal illumination angle are in correspondence. For example, an external illumination angle of 120 degrees corresponds to an internal illumination angle of approximately 160 degrees. In this embodiment, the internal illumination angle is 200 degrees. Here, an SLO fundus image obtained by photographing at an internal illumination angle of 160 degrees or more is referred to as a UWF-SLO fundus image.

The wide-angle optical system 342 may be a reflective optical system using a concave mirror such as an elliptical mirror, a refractive optical system using a wide-angle lens, or a catadioptric system combining concave mirrors and lenses. By using a wide-angle optical system using an elliptical mirror, a wide-angle lens, etc., it is possible to photograph not only the central part of the fundus but also the peripheral part of the retina. When using a system including an elliptical mirror, a configuration using a system using an elliptical mirror described in International Publication WO2016/103484 or International Publication WO2016/103489 may be used. The disclosure of International Publication WO2016/103484 and the disclosure of International Publication WO2016/103489 are each incorporated herein by reference in its entirety.

The SLO unit 33 includes a blue light (B light) light source 331B, a green light (G light) light source 331G, a red light (R light) light source 331R, and an infrared light (IR light) light source such as near infrared light. 331IR and an optical system 335 that reflects or transmits light from these light sources and guides them to one optical path. The SLO unit 33 also includes a beam splitter 332 and

detection elements

333B, 333G, 333R, and 333IR that detect B light, G light, R light, and IR light, respectively.

The SLO unit 33 can switch between a light source that emits light or a combination of light sources that emit light, such as a mode that emits B light, R light, and G light, and a mode that emits IR light.

The beam splitter 332 has the function of splitting the reflected light from the fundus into B light, R light, G light, and IR light, and reflecting each light toward the

detection elements

333B, 333G, 333R, and 333IR. .

The

detection elements

333B, 333G, 333R, and 333IR can detect B light, R light, G light, and IR light, respectively.

The light incident on the imaging optical system 34 from the SLO unit 33 is scanned in the X direction and the Y direction by the optical scanner 341 . The scanning light passes through the wide-angle optical system 342 and irradiates the fundus. Reflected light reflected by the fundus enters the SLO unit 33 via the wide-angle optical system 342 and the optical scanner 341 .

The reflected light reflected by the fundus enters the SLO unit 33 via the wide-angle optical system 342 and the optical scanner 341, and is decomposed into B light, R light, G light, and IR light by the beam splitter 332. These lights are detected by

detection elements

333B, 333G, 333R, and 333IR, respectively.

By collecting information of B light, R light, G light, and IR light detected by the

detection elements

333B, 333G, 333R, and 333IR, the processor 101 of the control device 31 can generate an SLO fundus image.

The OCT system is composed of the control device 31, the OCT unit 35, and the imaging optical system 34 shown in FIG. OCT unit 35 comprises light source 351 , sensor 352 , first optical coupler 353 , reference optics 354 , collimating lens 355 and second optical coupler 356 .

The light emitted from the light source 351 is split by the first optical coupler 353 . One of the split lights is collimated by the collimating lens 355 and enters the imaging optical system 34 as measurement light. Light incident on the imaging optical system 34 is scanned in the X and Y directions by an optical scanner 341 . The scanning light passes through the wide-angle optical system 342 and irradiates the fundus. The measurement light reflected by the fundus enters the OCT unit 35 via the wide-angle optical system 342 and enters the second optical coupler 356 via the collimating lens 355 and the first optical coupler 353 .

The other light emitted by the light source 351 and branched by the first optical coupler 353 enters the second optical coupler 356 via the reference optical system 354 as reference light.

The reference light and the measurement light reflected by the fundus are interfered by the second optical coupler 356 to generate interference light. The interfering light is received by sensor 352 . The control device 31 receives signals from the sensor 352 and generates a tomographic image. Imaging using an OCT system and an image obtained by the imaging may hereinafter be simply referred to as OCT imaging and OCT image, respectively.

[Software configuration]
FIG. 4 shows main functions (functional configuration) of the server 10 . As shown in the figure, the server 10 has functions of a database 114 and a management unit 120 . The management unit 120 particularly has the functions of the image processing unit 116 and the detection unit 118 . The database 114 is stored in the main storage device 102 of the server 10 . Each function of the management unit 120 is implemented by the processor 101 of the server 10 reading and executing a program stored in the main storage device 102 of the server 10 .

In addition to the functions described above, the server 10 also has functions such as an operating system, a file system, a device driver, and a DBMS (DataBase Management System).

The management unit 120 performs processing executed by the server 10, such as acquisition and management of images. Images acquired and managed by the management unit include images captured by the ophthalmologic apparatus 30 . The image processing unit 116 mainly performs GUI generation and processing of images captured by the ophthalmologic apparatus 30 .

The detection unit 118 detects the presence or absence of abnormal findings such as bleeding or retinal detachment (hereinafter referred to as “abnormal findings”) in the fundus including the retina or choroid, and the details of the findings, captured by the ophthalmologic apparatus 30 . It has a function of estimating from the captured image. In the detection unit 118, the presence or absence of an abnormal finding and its details are estimated based on the abnormal region in the image of the subject's eye. In this embodiment, the detection unit 118 is a trained model generated by machine learning.

Specifically, the detection unit 118 is a model that performs deep learning to learn the image feature amount of the abnormal region in the image of the eye to be inspected. The detection unit 118 constructs a neural network that outputs information indicating the result of estimating the presence or absence of an abnormal finding for the input image of the eye to be inspected. For example, the neural network is a deep neural network (DNN).

The detection unit 118 has an input layer that receives the input of the image of the subject's eye, an output layer that outputs the estimation result of the presence or absence of an abnormal finding, and an intermediate layer that extracts the image feature amount of the image of the subject's eye (Fig. 5). Each of the input layer, the output layer, and the intermediate layer has nodes (indicated by white circles in the figure), and the nodes of these layers are connected by edges (indicated by arrows in the figure). Note that the configuration of the detection unit 118 shown in FIG. 5 is an example, and the number of nodes and edges, the number of intermediate layers, and the like can be changed as appropriate.

When the detection unit 118 is a CNN (Convolutional Neural Network), the intermediate layer has a convolution layer for convolving the pixel value of each pixel in the image of the eye to be inspected input from the input layer, and a pooling layer for mapping the pixel value. , and these layers are used to extract the feature amount of the image of the subject's eye. The output layer has one or more neurons that output results of estimating abnormal findings of the input image of the subject's eye.

The detection unit 118 can also output the likelihood of the estimation result together with the estimation result. The likelihood is, for example, a probability value output from the output layer of the detection unit 118, and for example, the degree of reliability of the estimated abnormal finding is indicated by a value from "0" to "1". . By notifying the user of the likelihood, the user can know how accurate the estimation result is.

The detection unit 118 also outputs the severity of the abnormal finding. The severity can be explained as the seriousness of the symptoms, the grade of the symptoms, the speed of progression, the magnitude of the effects of the symptoms on the human body, and the like. For example, if the symptom or the like is bleeding, the detection unit 118 estimates the severity of the bleeding from the size, amount, and the like of the bleeding. Similarly, the detection unit 118 outputs the degree of severity for other abnormal findings such as retinal detachment and neovascularization.

In particular, in the present embodiment, the detection unit 118 is described as being a CNN, but the detection unit 118 is not limited to a CNN, and may be a neural network other than a CNN, or a trained model constructed by another learning algorithm. you can

The database 114 stores images such as wide-angle fundus images P and OCT images obtained by photographing fundus tissues having retinas and choroids. Annotations can also be added to the image of the eye to be examined by a medical professional or the like. In the database 114, annotations are added in the image in association with the image of the eye to be examined and an image showing a part of the image of the eye to be examined. It is possible to save the marked locations and the contents of the annotations together. The saved data is used for learning and re-learning of the detection unit 118 . The annotation includes information indicating the abnormal region of the fundus and details of abnormal findings such as macular degeneration attached to the abnormal region. Therefore, when used for learning of the detection unit 118, the annotation has a function as data indicating the correct answer of the abnormal findings and abnormal regions of the input image. Database 114 also stores patient medical records, such as electronic medical records, patient IDs, and other data, including historical records of images of the eye being examined. The database 118 also stores information about findings obtained by the detection unit 118, which will be described later, in association with the image.

The deep learning of the detection unit 118 uses a dataset containing a large number of images containing abnormal areas such as bleeding and neovascularization, and annotations attached to these images. Deep learning of the detection unit 118 is performed by having the detection unit 118 learn or re-learn this data set.

〔process〕
An example of processing executed by the information processing system 1 will be described below with reference to flowcharts of FIGS. 6 to 9. FIG. A program stored in the main storage device 102 of the server 10 is started, and the processing of the information processing system 1 is executed by the management unit 120 of the server 10 as follows. In addition, below, the process performed by the management part 120 of the server 10 may be described simply as what "the server 10" performs.

The outline of the processing performed by the server 10 is roughly composed of four steps, as shown in FIG. First, the server 10 acquires an image of the subject's eye obtained by photographing the retina with the ophthalmologic apparatus 30 (S1). In this example, the wide-angle fundus image P is used as the image captured by the ophthalmologic apparatus 30 . The wide-angle fundus image P is captured by the SLO system of the ophthalmologic apparatus 30 as described above. The management unit 120 acquires the wide-angle fundus image P stored in the ophthalmologic apparatus 30 via the network 5 .

Next, the management unit 120 estimates whether there is an abnormal finding in the acquired wide-angle fundus image P (S3). After that, the management unit 120 performs output processing of the image of the subject's eye in which the presence or absence of an abnormal finding is estimated (S5).

Next, the management unit 120 performs additional processing (S7). The additional processing includes filling in an electronic medical record, re-learning, etc., and is mainly executed according to instructions from the user.

(estimation process)
Details of the estimation process (S3) are shown in FIG. 7 and described below. In step S<b>31 , the management unit 120 inputs the wide-angle fundus image P to the detection unit 118 .

Next, the management unit 120 acquires an estimation result regarding the presence or absence of an abnormality in the detection unit 118 (S33). When it is estimated that there is an abnormality, the output of the detection unit 118 includes the type of abnormal finding and information specifying the abnormal region. An abnormal region is a region in which there is a difference from a normal eye. Examples of abnormal regions captured in images of the subject's eye include bleeding points, neovascular regions, retinal detachment regions, non-perfused regions, etc. in the retina or choroid. A finding based on an abnormal region is an abnormal finding, and examples of types of abnormal findings include the presence and degree of hemorrhage, neovascularization, retinal detachment, and non-perfused region in the retina or choroid. FIG. 10 shows a wide-angle fundus image P including a macula M as an example of an abnormal finding. As the estimation result, an image area estimated to include an abnormal area and an identification result identifying the type of abnormal finding are obtained.

The detection unit 118 also outputs the likelihood of the estimation result and the severity of the abnormal findings. Then, the management unit 120 generates abnormal finding information including an image area estimated to include an abnormal area, the identification result of identifying the type of abnormal finding, the likelihood of the estimation result, the severity of the abnormal finding, and the like. do.

As a result of the above, when an abnormal finding with sufficient certainty is output, the wide-angle fundus image P is stored in association with the abnormal finding information. The estimation process generates a wide-angle fundus image P associated with abnormal finding information.

(output processing)
Details of the output processing (S5) will be described below with reference to FIGS. 8, 10 and 11. FIG. FIG. 10 shows an example of a wide-angle fundus image P to which an abnormal finding is added by the estimation process (S3). In the following explanation, bleeding is used as an example of an abnormal finding. A bleeding point B is displayed in the wide-angle fundus image P as an abnormal area.

The management unit 120 uses the image processing unit 116 to extract a partial image indicating a partial area in the wide-angle fundus image P (S51). The management unit 120 sets a partial area of the wide-angle fundus image P and generates a partial image. A plurality of partial images are extracted, and the plurality of partial images may or may not overlap with each other. If an abnormal finding is given, the area of the partial image is determined such that the abnormal area is included in any one of the plurality of partial images. When a plurality of abnormal regions are located apart from each other, the regions of the partial images may be determined so that one partial image includes all the abnormal regions, and the partial images are determined so that the plurality of partial images include the abnormal regions. A region of the image may be determined.

10 and 11, frames F1, F2, F3, and F4 superimposed on the wide-angle fundus image P indicate four partial regions extracted from the wide-angle fundus image P. FIG. In addition, four partial images D1, D2, D3, and D4 generated as a result of extracting each partial area of the wide-angle fundus image P are shown on the right side of each figure. Partial images D1, D2, D3, and D4 correspond to the areas indicated by frames F1, F2, F3, and F4, respectively.

The image processing unit 116 extracts partial images from the wide-angle fundus image P such that one of the partial areas includes the bleeding point B, which is an abnormal area. In the examples of FIGS. 10 and 11, the region of the frame F2 includes the bleeding point B, and the other regions (F1, F3, F4) do not include the bleeding point B. FIG. As shown in FIG. 10 as an example, the image processing unit 116 determines the size and arrangement of the regions so that the partial images D1, D2, D3, and D4 all include the entire macula M. FIG. At the same time, the image processing unit 116 determines the size and arrangement of the region so that the bleeding point B is included in one of the regions, in other words, the bleeding point B is displayed in one of the partial images. . As a result, as shown on the right side of FIG. 10, the macula M is displayed in each of the partial images D1, D2, D3, and D4. A bleeding point B is also displayed in the partial image D2.

When characteristic structures of the fundus such as the macula and the optic papilla are displayed in each partial image as in the present embodiment, it is easy for the user to understand the position of each partial image and the abnormal region in the wide-angle fundus image P.

In the partial image extraction example of FIG. 11, the image processing unit 116 determines the division positions so that the areas corresponding to the frames F1 to F4, that is, the partial images D1 to D4 do not overlap with the macula M at the center. Also, the image processing unit 116 adjusts the arrangement and size of the partial images so that the bleeding point B is displayed in one of the partial images. As a result, the bleeding point B is displayed in the partial image D1.

It should be noted that when there is no abnormal area, the method of setting partial images is determined according to a predetermined method. For example, the placement and size of each image may be determined so that the frames F1 to F4, that is, the partial images D1 to D4 all include the entire macula M (FIG. 10). Alternatively, the division positions may be determined so that the frames F1 to F4, that is, the regions of the partial images D1 to D4 do not overlap with the macula M as the center (FIG. 11).

The image processing unit 116 extracts partial images from the wide-angle fundus image P as described above, and creates partial images D1, D2, D3, and D4.

Next, the image processing unit 116 performs image processing for sharpening the abnormal area (S53). In the example of FIGS. 10 and 11, in the partial image D2 displaying the bleeding point B, processing is performed to change the pixel values of the bleeding point B and other areas so that the bleeding point B can be easily recognized. Further, the image processing unit 116 removes reflection of eyelashes and other artifacts from the partial images D1, D2, D3, D4 and the wide-angle fundus image P to make the images easy to view.

The image processing unit 116 further performs processing for setting the method of emphasizing the abnormal region (S55). The emphasis method is appropriately selected according to the type, severity, and certainty of the abnormal findings. For example, when abnormal findings are recognized at multiple locations, a partial image including an abnormal region with a high degree of severity is set to be preferentially displayed in the subsequent display processing (S59). Further, the image processing unit 116 performs processing for setting a display method for displaying a partial image in which an abnormal region is displayed among the plurality of partial images in a manner to distinguish it from a partial image in which an abnormal region is not displayed. Specifically, the enlargement ratio and display order of the partial images in which the abnormal region is displayed are selected according to the type of abnormal finding or likelihood. Further, the image processing unit 116 selects an image such as a frame or an icon to be displayed superimposed on the portion to be emphasized according to the type or probability of the abnormal finding, and is displayed as an emphasized display in the subsequent display processing (S59). set to Note that these emphasis methods may be set by accepting user input.

The image processing unit 116 creates a GUI (Graphical User Interface) according to the partial image extraction method, abnormal region enhancement method, and highlighting selected up to step S55 (S57). This GUI is transmitted to the terminal 20 via the network 5 and displayed on the output device 105 of the terminal 20 (S59). The GUI receives instructions from the user who operates the terminal 20, changes the display according to the instructions, and further displays images such as icons and frames.

Examples of GUIs are shown in Figures 10 to 14. As shown in each figure, in the GUI, a wide-angle fundus image P is arranged on the left side, and four partial images D1 to D4 are arranged on the right side thereof. Frames F1, F2, F3, and F4 corresponding to the partial images D1, D2, D3, and D4 are superimposed on the wide-angle fundus image P and displayed. By displaying the frames F1 to F4, it is easy to understand which region of the wide-angle fundus image P each of the partial images D1 to D4 indicates.

In FIG. 10, the partial image D2 displaying the bleeding point B, which is the abnormal area, is preferentially displayed in the largest size.

FIG. 12 shows the GUI when the enhancement method (see S55) for further enlarging the partial image including the abnormal region is selected. Moreover, all of the partial images D1 to D4 are set so as to display the macula M (see S51). As shown, the partial image D2 in which the bleeding point B is displayed has a larger magnification than the other partial images D1, D3, and D4. Therefore, the user can easily confirm the bleeding point B.

In the GUI of FIG. 13, the bleeding point B displayed in the partial image D2 is highlighted by a small frame S (see S55). According to the user's instruction, the image processing unit 116 further enlarges the image within the small frame S and displays it as an enlarged image L. FIG. Thereby, the user can easily confirm the bleeding point B. The user's instruction is performed by, for example, clicking the image of the small frame S with a mouse or tapping on the display screen.

It should be noted that it is possible to display not only one enlarged image L but also a plurality of enlarged images at the same time. Further, when there are a plurality of abnormal regions, a method may be adopted in which the enlarged images L are sequentially displayed according to the user's instruction. Alternatively, all the enlarged images L may be sequentially arranged and displayed as thumbnails. At this time, it is desirable to arrange the enlarged images L based on the type, severity, or likelihood of the abnormal findings.

The magnification or angle of view of the enlarged image L is arbitrarily set. For example, the magnification is set to 5 times and the angle of view is set to 30 degrees, which is easy for the doctor to visually recognize. It is also possible to change the enlargement ratio or the angle of view according to the user's instruction. Alternatively, the enlargement ratio or the angle of view preset by the doctor may be used. Also, the user can change the positions of the partial images D1 to D4 and change the range of enlargement by operating the positions and sizes of the frames F1 to F4.

In the GUI of FIG. 14, icons IC1 to IC4 are highlighted. These highlights differ depending on the type of abnormal finding. In FIG. 14, the area judged to be abnormal by the doctor and added to the display, the abnormal area output by AI, the area judged to be abnormal by the doctor and AI, and the area judged to be not abnormal by the doctor Icons IC1 to IC4 with different displays are used for the four types of areas instructed to change the display on the GUI. In this way, the method of highlighting is differentiated according to the type of abnormal findings. The icons IC1 to IC4 are displayed differently from each other by, for example, changing line types, shapes, colors, and the like.

Also, if there are multiple abnormal areas, it is possible to display enlarged images in order of severity of abnormal findings. Also, the display order of the partial images D1 to D4 or the display mode of the enlarged image L is changed according to the severity.

The mode of highlighting will also change according to the severity. For example, brighter or more saturated color highlighting is used for more severe anomalous regions.

In addition, any type of emphasis can be selected, regardless of the image division method or GUI display mode, as shown in FIGS. It is also possible to combine multiple enhancement methods. For example, in the GUI display in which the partial image D2 is enlarged as shown in FIG. 11, the small frame S and the enlarged image L may be superimposed and displayed on the partial image D2.

The image processing unit 116 can cooperate with the ophthalmologic apparatus 30 to perform OCT imaging using the OCT system and acquire a tomographic image (S61). An instruction for OCT imaging can also be accepted via the GUI in the display processing (S59). For example, the user can specify an abnormal region to be subjected to OCT imaging by operating a mouse click or the like.

When there is an instruction from the user, the image processing unit 116 identifies the position of the fundus where the abnormal area is located, and instructs the ophthalmologic apparatus 30 via the network 5 to take an image. Based on this instruction, the ophthalmologic apparatus 30 uses the OCT system to perform OCT imaging on the abnormal region. A tomographic image of the retina obtained by OCT imaging is obtained by the server 10 and displayed on the output device 105 of the terminal 20 . At that time, it may be displayed together with the wide-angle fundus image P or the partial images D1 to D4 on the GUI.

OCT imaging may be performed without user instructions. For example, when there is an abnormal finding, the image processing unit 116 identifies the position of the fundus where the abnormal region exists, and instructs the ophthalmologic apparatus 30 to take an image via the network 5 . Based on this instruction, the ophthalmologic apparatus 30 uses the OCT system to perform OCT imaging on the abnormal region.

During OCT imaging, imaging may be performed according to the patient's information. For example, if a diagnosis of macular degeneration or the like is obtained in the electronic medical record stored in the database 114, the server 10 instructs the ophthalmologic apparatus 30 to perform OCT imaging of the vicinity of the macula M. FIG. Based on this instruction, the ophthalmologic apparatus 30 performs OCT imaging on the abnormal region.

(additional processing)
The addition process will be explained using the flow of FIG. 9 . The user can check the abnormal findings and their regions in the display process (S59), and then add annotations to the estimation results and save them (S71). The saved annotations are saved in the database 114 and saved in association with the wide-angle fundus image P and the estimation result.

The saved annotations and wide-angle fundus image P are used for re-learning of the detection unit 118 (S73). For example, the detection unit 118 can improve the ability to estimate an abnormal finding by learning the wide-angle fundus image P together with annotations indicating the types of abnormal findings and correct abnormal regions.

<Second embodiment>
In the first embodiment described above, the detection unit 118 uses a trained model with CNN. As an embodiment different from the first embodiment, an image processing algorithm may be used for estimating an abnormal finding instead of a trained model that has undergone machine learning. A second embodiment will be described below.

FIG. 15 shows the software configuration of the server 10 according to the second embodiment. The second embodiment includes a detection unit 119 configured by an image processing algorithm.

In the second embodiment, the configuration of the server 10, the terminal 20, and the ophthalmologic apparatus 30 is the same as in the first embodiment, except for the detection unit 119. The same reference numerals as used in the first embodiment are assigned to the same configurations as in the first embodiment, and the description thereof is omitted.

The estimation process (S3) by the detection unit 119 in the second embodiment will be described below. In this example, an algorithm for estimating the presence or absence of abnormal findings such as bleeding in the wide-angle fundus image P will be described.

In step S35, the detection unit 119 extracts a blood region. Here, the blood region is a region in the wide-angle fundus image P that includes a blood vessel region showing blood vessels and a bleeding region formed by bleeding from blood vessels. As a result of the processing in step S35, an image obtained by extracting the blood region from the wide-angle fundus image P is obtained.

At step S36, the detection unit 119 extracts only the blood vessel region.

A general image analysis method is applied in the region extraction in steps S35 and S36. For example, the wide-angle fundus image P is displayed in 256 gradations or the like, and is converted into a binarized image by comparing the gradation value of each pixel with the threshold value.

When binarizing, in step S35, a threshold value is set so as to distinguish between the blood region and other regions. At step S36, a threshold value is set so as to distinguish between the blood vessel region and other regions. Various methods such as the mode method can be adopted as the method of setting the threshold.

In step S37, the detection unit 119 removes noise in the binarized image, particularly around the blood vessel region. For noise removal processing, for example, dilation processing (Dilation) and erosion processing (Erosion) are used. Through step S37, an image obtained by extracting the blood vessel region from the wide-angle fundus image P is obtained.

In step S38, the detection unit 119 can obtain an image showing the bleeding area by taking the difference between the image showing the blood area and the image showing the blood vessel area.

The detection unit 119 estimates the presence or absence of bleeding based on the image acquired in this way (S39). When determining that there is bleeding, the detection unit 119 outputs an abnormal finding of bleeding and an image showing the bleeding region.

In the above, an example of estimation processing related to bleeding was shown, but in addition, the same processing is performed in the case of abnormal findings such as neovascularization. In this case, the parameters used for region extraction, such as threshold settings for binarization, are appropriately set according to the type of abnormal finding.

<effect>
The image processing in the above embodiment includes a process (S1) of acquiring a wide-angle fundus image (P) obtained by photographing with an ophthalmologic apparatus, and a plurality of partial images (D1 -D4), a selection process (S51) for selecting a first partial image showing the abnormal region (B) that is the basis of the abnormal finding from the plurality of partial images, and the first partial image. An image processing method including display processing (S59) for displaying.
including.

With the above configuration, it is possible to obtain a display that allows easy visual recognition of the abnormal region while covering a wide range of abnormal findings of the fundus.

In the display process (S59), the partial areas where abnormal findings are observed are displayed on the wide-angle fundus image P so that they can be distinguished. Further, a partial image having an abnormal finding among the partial images D1 to D4 can be enlarged and displayed.

With such a configuration, the user can easily visually recognize the details of the abnormal finding and the abnormal area.

Also, in the display process (S59), it is possible to highlight the abnormal area. This highlighting is changed according to the type of abnormal finding.

With the above configuration, details of abnormal findings and abnormal areas can be easily visually recognized. In addition, even when there are multiple types of abnormal findings, the user can organize and check the abnormal findings for each type.

The above embodiment includes an estimation process (S3) for estimating an abnormal finding in the wide-angle fundus image P. The estimation processing includes processing (S31) of inputting the wide-angle fundus image P to the

detection units

118 and 119 for estimating abnormal findings, and processing for causing the

detection units

118 and 119 to estimate abnormal findings.

Such estimation processing makes it possible to assist the diagnosis and findings of doctors and medical staff. It is also possible to prevent an abnormality from being overlooked.

The above embodiment includes a process of accepting addition of annotations related to abnormal findings for the wide-angle fundus image P (S71), and a process of re-learning the detection unit 118 with the annotations and the wide-angle fundus image P (S73).

Since the detection unit 118 re-learns using the annotated data, the accuracy of the estimation processing of the detection unit 118 can be improved. That is, it is possible to reduce the probability of outputting false positives or false negatives.

In the estimation process (S3), the severity of the abnormal finding is estimated. Also, the display method of the first area is changed according to the estimated severity.

The above-described embodiment includes a process of specifying the position of the abnormal region on the fundus (S61), and a process of acquiring a tomographic image at the specified position by performing OCT imaging with an OCT system on the specified position (S61). further includes

With the above configuration, it is possible not only to confirm the wide-angle fundus image P, but also to accurately or easily acquire a tomographic image when further confirmation of abnormal findings is desired.

<Modification>
Although the partial area setting of the wide-angle fundus image P is performed using a rectangular area in the above embodiment, the present invention is not limited to this. For example, the division may be performed by extending the division lines radially, or by dividing into slits.

Annotations can also include the shape of partial area settings. For example, the partial area setting shape determined by the user to be optimal may be included in the annotation, and the detection unit 119 may learn and re-learn. In this case, the optimum partial area shape and partial area position are output in accordance with the abnormal findings estimated by the detection unit 119 . In the partial image extraction process ( S<b>51 ), partial image extraction of the wide-angle fundus image P may be performed according to the partial image extraction method output by the detection unit 119 .

The display mode in the display processing (S59) and the like was a mode in which images were used to distinguish and display abnormal areas by color and shape, but the present invention is not limited to such a viewing mode. For example, a mode of setting display variations by voice may be used.

Although the ophthalmologic apparatus 30 having an OCT system is used in the above embodiment, the present invention is not limited to this, and ophthalmologic apparatuses that do not have an OCT system or that use other systems may be used.

In the above embodiment, a plurality of terminals are connected to one server 10, and the above functions are exhibited. The present invention does not limit the number of servers or the number of terminals, and for example, the functions described above may be realized by only one device. Also, the number of terminals and the number of servers may be further increased. Also, each function does not necessarily have to be implemented by the server 10 or the like, and may be shared by a plurality of devices to implement the function. That is, the present invention does not limit the number of controllers or devices, or the sharing of functions among devices.

1 Information Processing System 10 Server 20 Terminal 30 Ophthalmic Apparatus

Claims

a process of acquiring a wide-angle fundus image obtained by photographing with an ophthalmologic apparatus;
a process of extracting a plurality of partial images from the wide-angle fundus image to which an abnormal finding is added;
a selection process of selecting a first partial image showing an abnormal region serving as a basis for the abnormal finding from the plurality of partial images;
display processing for displaying the first partial image;
An image processing method including
The image processing method according to claim 1, wherein the display process is a process of displaying a second partial image in which the abnormal region is not shown among the plurality of partial images, together with the first partial image.
The image processing method according to claim 2, wherein the display processing is processing for displaying the first partial image and the second partial image in different display modes.
The image processing method according to claim 3, wherein the display processing displays the first partial image in a size larger than the second partial image.
The image processing method according to claim 3 or 4, wherein the display processing extracts the abnormal region from the first partial image and displays the abnormal region at a display magnification higher than that of the second partial image.
6. The display process according to any one of claims 1 to 5, wherein the display process is a process of displaying a position display image displaying the position of the first partial image in the wide-angle fundus image at the same time as the first partial image. Image processing method.
The image processing method according to any one of claims 1 to 6, further comprising processing for highlighting the abnormal region in the first partial image.
The image processing method according to claim 7, wherein the highlighting is changed according to the type of the abnormal finding.
The image processing method according to any one of claims 1 to 8, further comprising estimation processing for estimating the abnormal finding in the wide-angle fundus image.
The estimation process includes
A process of inputting the wide-angle fundus image into a trained model that estimates an abnormal finding;
and causing the trained model to estimate the abnormal finding.
a process of receiving addition of annotations regarding abnormal findings for the wide-angle fundus image;
11. The image processing method according to claim 10, further comprising re-learning the trained model using the annotation and the wide-angle fundus image.
The estimation process further includes a process of estimating the severity of the abnormal finding,
The display process includes
13. The image processing method according to any one of claims 9 to 12, further comprising a process of changing a display method of said first partial image according to said severity.
a process of identifying the position of the abnormal region in the wide-angle fundus image;
A process of performing imaging with an OCT system at the specified position and acquiring a tomographic image at the position;
13. The image processing method according to any one of claims 1 to 12, further comprising:
A program that causes a computer to execute the image processing method according to any one of claims 1 to 13.
An image processing apparatus comprising a processing unit that executes the image processing method according to any one of claims 1 to 13.
An ophthalmic system comprising a processing unit that executes the image processing method according to any one of claims 1 to 13.