KR102301058B1 - Diagnosis assistance system and control method thereof - Google Patents

Abstract

The present invention provides a diagnostic assistance system for assisting diagnosis of a plurality of diseases based on a fundus image, wherein a fundus image acquisition unit for acquiring a fundus image, and a first neural network model for the fundus image A first processing unit for obtaining a first result related to an observation, a second processing unit for obtaining a second result related to a second observation on the subject by using a second neural network model for the fundus image, the first result and the second result The present invention relates to a diagnostic assistance system comprising: a third processing unit that determines diagnostic information on a subject based on

Description

DIAGNOSIS ASSISTANCE SYSTEM AND CONTROL METHOD THEREOF

The present invention relates to a diagnosis assistance system and a control method thereof, and to a diagnosis assistance system for providing diagnosis assistance information based on an image and a control method thereof.

Fundus examination can observe abnormalities in the retina, optic nerve, and macula, and is a diagnostic aid frequently used in ophthalmology because the results can be checked relatively simply through imaging. Recently, the use of the fundus examination is increasing in that it is possible to observe the degree of vascular damage caused by chronic diseases such as high blood pressure and diabetes in a non-invasive way as well as eye diseases.

On the other hand, due to the rapid development of deep learning technology in recent years, the development of artificial intelligence for diagnosis is being actively carried out in the medical diagnosis field, particularly in the image-based diagnosis field. Global companies such as Google and IBM are also sparingly investing in the development of artificial intelligence for various image and medical data analysis, such as inputting large-scale data in collaboration with the medical community. Tool development was also successful.

However, when trying to predict a plurality of values from one test data through a deep learning trained model, there is a problem in that prediction accuracy is reduced and processing speed is lowered. Accordingly, there is a need for a learning and diagnosis system capable of accurately predicting a plurality of diagnostic characteristics at a high data processing speed.

It is an object of the present invention to provide a method for learning a neural network model for acquiring diagnostic assistance information from a fundus image.

Another object of the present invention is to provide a method for training a plurality of neural network models in parallel to acquire a plurality of diagnostic auxiliary information from a fundus image.

Another object of the present invention is to provide a method for rapidly acquiring a plurality of diagnostic auxiliary information from a fundus image using a machine-learned neural network model.

The problem to be solved by the present invention is not limited to the above-mentioned problems, and the problems not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the present specification and the accompanying drawings. .

According to one aspect of the present invention, there is provided a diagnosis assistance system for assisting in diagnosis of a plurality of diseases based on a fundus image, comprising: a fundus image acquisition unit configured to acquire a target fundus image that is a basis for acquiring diagnostic assistance information for a subject; a first for obtaining a first result related to a first observation on the subject by using a first neural network model with respect to the target fundus image, wherein the first neural network model is machine learned based on the first set of fundus images. 1 processing unit, a second neural network model for the target fundus image, wherein the second neural network model is machine-learned based on a second fundus image set that is at least partially different from the first fundus image set. A second processing unit for obtaining a second result related to a second opinion on A diagnostic assistance system including a diagnostic information output unit for providing may be provided. In this case, the first findings and the second findings may be used for diagnosis of different diseases.

According to another aspect of the present invention, obtaining a first training data set including a plurality of fundus images, processing the fundus image included in the first training data set, and using the first training data set , a learning apparatus for learning the first neural network model may be provided.

a diagnostic apparatus for acquiring a target fundus image for acquiring diagnostic assistance information, and acquiring the diagnostic assistance information based on the target fundus image using the learned first neural network model; As a control method of the learning apparatus included in a system comprising performing, serializing the preprocessed first fundus image, using the serialized first fundus image, training the first neural network model to classify the target fundus image into a first label or a second label A method of controlling a learning device including the steps may be provided.

Solutions of the present invention are not limited to the above-described solutions, and solutions not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the present specification and the accompanying drawings. will be able

According to the present invention, rapid acquisition of diagnostic auxiliary information can be achieved based on the fundus image.

Further, according to the present invention, it is possible to more accurately predict diagnostic auxiliary information related to a plurality of diseases based on the fundus image.

Also, according to the present invention, it is possible to easily obtain an optimized neural network model for predicting diagnostic auxiliary information related to a plurality of diseases.

The effects of the present invention are not limited to the above-mentioned effects, and the effects not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the present specification and the accompanying drawings.

1 illustrates a diagnosis assistance system according to an embodiment of the present invention.
2 is a block diagram illustrating a learning apparatus according to an embodiment of the present invention.
3 is a block diagram for explaining in more detail a learning apparatus according to another embodiment of the present invention.
4 is a block diagram illustrating a diagnosis apparatus according to an embodiment of the present invention.
5 is a diagram for explaining a diagnosis apparatus according to another embodiment of the present invention.
6 illustrates a diagnosis assistance system according to an embodiment of the present invention.
7 is a block diagram illustrating a client device according to an embodiment of the present invention.
8 is a diagram for explaining a diagnosis assistance process according to an embodiment of the present invention.
9 is a diagram for explaining the configuration of a learning unit according to an embodiment of the present invention.
10 is a conceptual diagram illustrating an image data set according to an embodiment of the present invention.
11 is a diagram for explaining image resizing according to an embodiment of the present invention.
12 is a diagram for explaining the expansion of an image data set according to an embodiment of the present invention.
13 is a block diagram illustrating a learning process of a neural network model according to an embodiment of the present invention.
14 is a block diagram illustrating a learning process of a neural network model according to an embodiment of the present invention.
15 is a diagram for explaining a method of controlling a learning apparatus according to an embodiment of the present invention.
16 is a diagram for explaining a method of controlling a learning apparatus according to an embodiment of the present invention.
17 is a diagram for explaining a method of controlling a learning apparatus according to an embodiment of the present invention.
18 is a diagram for explaining the configuration of a diagnosis unit according to an embodiment of the present invention.
19 is a diagram for explaining diagnosis target data according to an embodiment of the present invention.
20 is a diagram for explaining a diagnosis process according to an embodiment of the present invention.
21 is a diagram for explaining a parallel diagnosis assistance system according to some embodiments of the present invention.
22 is a diagram for explaining a parallel diagnosis assistance system according to some embodiments of the present invention.
23 is a diagram for explaining a configuration of a learning apparatus including a plurality of learning units according to an embodiment of the present invention.
24 is a diagram for explaining a parallel learning process according to an embodiment of the present invention.
25 is a diagram for explaining a parallel learning process according to another embodiment of the present invention.
26 is a block diagram illustrating a diagnosis unit according to an embodiment of the present invention.
27 is a diagram for explaining a diagnosis assistance process according to an embodiment of the present invention.
28 is a diagram for explaining a diagnosis assistance system according to an embodiment of the present invention.
29 is a diagram for explaining a graphical user interface according to an embodiment of the present invention.
30 is a diagram for explaining a graphical user interface according to an embodiment of the present invention;

The above-described objects, features and advantages of the present invention will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. However, since the present invention may have various changes and may have various embodiments, specific embodiments will be exemplified in the drawings and described in detail below.

In the drawings, the thicknesses of layers and regions are exaggerated for clarity, and an element or layer is also referred to as “on” or “on” another component or layer. It includes all cases where another layer or other component is interposed in the middle as well as directly on top of another component or layer. Throughout the specification, like reference numerals refer to like elements in principle. In addition, components having the same function within the scope of the same idea shown in the drawings of each embodiment will be described using the same reference numerals.

If it is determined that a detailed description of a known function or configuration related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, numbers (eg, first, second, etc.) used in the description process of the present specification are only identification symbols for distinguishing one component from other components.

In addition, the suffixes "module" and "part" for the components used in the following description are given or mixed in consideration of only the ease of writing the specification, and do not have a meaning or role distinct from each other by themselves.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

1. Diagnostic auxiliary systems and processes

1.1 Purpose and Definition

Hereinafter, a diagnosis assistance system and method for assisting in the determination of the presence or absence of a disease or disease or an abnormality as a basis for the determination based on the fundus image will be described. In particular, a diagnosis assistance system or method for constructing a neural network model for diagnosing a disease using a deep learning technique and assisting in the detection of the presence or absence of a disease or abnormal findings using the constructed model will be described.

According to an embodiment of the present invention, there is provided a diagnostic assistance system or method for acquiring diagnostic information related to the presence or absence of a disease or observation information used for diagnosing the presence or absence of a disease based on a fundus image and assisting the diagnosis by using the acquired information can be

According to an embodiment of the present invention, a diagnosis assistance system or method for assisting in diagnosing an eye disease based on a fundus image may be provided. For example, a diagnostic assistance system or method may be provided to assist a diagnosis by acquiring diagnostic information related to the presence or absence of glaucoma, cataract, macular degeneration, and retinopathy of prematurity of a test subject.

According to another embodiment of the present invention, a diagnostic assistance system or method for assisting in the diagnosis of other diseases other than ocular diseases (eg, systemic diseases or chronic diseases) may be provided. For example, a diagnostic assistance system or method may be provided to assist in diagnosis by acquiring diagnostic information of systemic diseases such as hypertension, diabetes, Alzheimer's, cytomegalovirus, stroke, heart disease, arteriosclerosis, and the like.

According to another embodiment of the present invention, there may be provided a diagnostic auxiliary system or method for detecting abnormal fundus findings that can be used for diagnosing eye diseases or other diseases. For example, overall fundus color abnormality, lens opacity, optic disc ratio (C/D ratio; cup to disc ratio) abnormality, macular abnormality (eg, macular foramen), blood vessel diameter, travel A diagnostic assistance system or method for acquiring information on findings such as abnormality of the back, diameter abnormality of retinal artery, retinal hemorrhage, occurrence of exudate, drusen, and the like may be provided.

In the present specification, the diagnosis auxiliary information may be understood to include diagnosis information according to the determination of the presence or absence of a disease, or observation information that is the basis thereof.

1.2 Configuring the Diagnostic Auxiliary System

According to an embodiment of the present invention, a diagnostic assistance system may be provided.

1 illustrates a diagnostic assistance system 10 according to an embodiment of the present invention. Referring to FIG. 1 , the diagnosis assistance system 10 includes a learning device 1000 for training a diagnosis model, a diagnosis device 2000 for performing diagnosis using the diagnosis model, and a client device 3000 for obtaining a diagnosis request. may include The diagnosis assistance system 10 may include a plurality of learning devices, a plurality of diagnostic devices, or a plurality of client devices.

The learning apparatus 1000 may include a learning unit 100 . The learning unit 100 may train the neural network model. As an example, the learning unit 100 may acquire a fundus image data set and train a neural network model for detecting a disease or abnormality from the fundus image.

The diagnosis apparatus 2000 may include a diagnosis unit 200 . The diagnosis unit 200 may diagnose a disease or acquire auxiliary information used for diagnosis by using the neural network model. For example, the diagnosis unit 200 may obtain the diagnosis auxiliary information by using a diagnosis model trained by the learning unit.

The client device 3000 may include an imaging unit 300 . The imaging unit 300 may capture a fundus image. The client device may be an ophthalmic fundus imaging device. Alternatively, the client device 3000 may be a handheld device such as a smart phone or a tablet PC.

In the diagnosis assistance system 10 according to the present embodiment, the learning apparatus 1000 determines a neural network model to be used for diagnosis assistance by acquiring a data set and learning the neural network model, and the diagnosis apparatus requests information from a client When this is obtained, the diagnosis auxiliary information according to the diagnosis target image is obtained using the determined neural network model, and the client device may request the information from the diagnosis device and obtain the transmitted diagnosis auxiliary information in response thereto.

A diagnosis assistance system according to another exemplary embodiment may include a diagnosis device and a client device for learning a diagnosis model and performing diagnosis using the training model. A diagnosis assistance system according to another exemplary embodiment may include a diagnosis apparatus that learns a diagnosis model, obtains a diagnosis request, and performs diagnosis. A diagnosis assistance system according to another exemplary embodiment may include a learning apparatus for learning a diagnostic model, and a diagnostic apparatus for obtaining a diagnosis request and performing a diagnosis.

The diagnosis assistance system disclosed herein is not limited to the above-described embodiments, and a learning unit for learning a model, a diagnosis unit for acquiring diagnosis assistance information according to the learned model, and an imaging unit for acquiring an image to be diagnosed It may be implemented in any form including.

Hereinafter, some embodiments of each device constituting the system will be described.

1.2.1 Learning device

The learning apparatus according to an embodiment of the present invention may perform training of a neural network model that assists in diagnosis.

2 is a block diagram illustrating a learning apparatus 1000 according to an embodiment of the present invention. Referring to FIG. 2 , the learning apparatus 1000 may include a control unit 1200 and a memory unit 1100 .

The learning apparatus 1000 may include a controller 1200 . The controller 1200 may control the operation of the learning apparatus 1000 .

The control unit 1200 includes at least one of a central processing unit (CPU), a random access memory (RAM), a graphic processing unit (GPU), one or more microprocessors, and other electronic components capable of processing input data according to a predetermined logic. may include.

The controller 1200 may read a system program and various processing programs stored in the memory unit 1100 . For example, the controller 1200 may develop a data processing process, a diagnostic process, etc. for performing diagnosis assistance, which will be described later, on the RAM, and may perform various processes according to the developed program. The controller 1200 may perform learning of a neural network model, which will be described later.

The learning apparatus 1000 may include a memory unit 1100 . The memory unit 1100 may store data and a learning model required for learning.

The memory unit 1100 may include a nonvolatile semiconductor memory, a hard disk, a flash memory, a RAM, a read only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), or other tangible nonvolatile recording media, etc. can be implemented as

The memory unit 1100 may store various processing programs, parameters for performing program processing, or data as a result of such processing. For example, the memory unit 1100 may include a data processing process program, a diagnostic process program, parameters for performing each program, and data (eg, processed data or diagnostics) obtained according to the execution of the program to be described later for performing the diagnosis assistance. result) and so on.

The learning apparatus 1000 may include a separate learning unit (or learning module). The learning unit may perform learning of the neural network model. In relation to the performance of learning, it will be described in more detail in Table of Contents 2. Learning process below.

The learning unit may be included in the above-described control unit 1200 . The learning unit may be stored in the aforementioned memory unit 1100 . The learning unit may be implemented by some configurations of the control unit 1200 and the memory unit 1100 described above. For example, the learning unit may be stored in the memory unit 1100 and driven by the control unit 1200 .

The learning apparatus 1000 may further include a communication unit 1300 . The communication unit 1300 may communicate with an external device. For example, the communication unit 1300 may communicate with a diagnostic device, a server device, or a client device, which will be described later. The communication unit 1300 may perform wired or wireless communication. The communication unit 1300 may perform bi-directional or unidirectional communication.

3 is a block diagram for explaining in more detail the learning apparatus 1000 according to another embodiment of the present invention. Referring to FIG. 3 , the learning apparatus 1000 may include a processor 1050 , a volatile memory 1010 , a nonvolatile memory 1030 , a mass storage device 1070 , and a communication interface 1090 .

The processor 1050 of the learning apparatus 1000 may include a data processing module 1051 and a learning module 1053 . The processor 1050 may process a data set stored in a mass storage device or a non-volatile memory through the data processing module 1051 . The processor 1050 may train the diagnostic auxiliary neural network model through the learning module 1053 . The processor 1050 may include local memory. The communication interface 1090 may be connected to the network 1110 .

However, the learning apparatus 1000 illustrated in FIG. 3 is merely an example, and the configuration of the learning apparatus 1000 according to the present invention is not limited thereto. In particular, the data processing module or the learning module may be provided at a location different from that shown in FIG. 3 .

1.2.2 Diagnostic device

The diagnostic apparatus may obtain diagnostic auxiliary information by using the neural network model.

4 is a block diagram illustrating a diagnosis apparatus 2000 according to an embodiment of the present invention. Referring to FIG. 4 , the diagnosis apparatus 2000 may include a control unit 2200 and a memory unit 2100 .

The controller 2200 may generate the diagnosis auxiliary information by using the diagnosis auxiliary neural network model. The controller 2200 may acquire diagnostic data for diagnosis (eg, fundus data of a subject) and acquire auxiliary diagnostic information predicted by the diagnostic data using the learned diagnostic auxiliary neural network model.

The memory unit 2100 may store the learned diagnostic auxiliary neural network model. The memory unit 2100 may store parameters and variables of the auxiliary diagnostic neural network model.

The diagnosis apparatus 2000 may further include a communication unit 2300 . The communication unit 2300 may communicate with the learning device and/or the client device. For example, the diagnostic apparatus 2000 may be provided in the form of a server communicating with a client device. In this regard, it will be described in more detail below.

5 is a diagram for explaining a diagnosis apparatus 2000 according to another embodiment of the present invention. Referring to FIG. 5 , a diagnostic apparatus 2000 according to an embodiment of the present invention includes a processor 2050 , a volatile memory 2030 , a non-volatile memory 2010 , a mass storage device 2070 , and a communication interface 2090 . may include.

The processor 2050 of the diagnostic apparatus may include a data processing module 2051 and a diagnostic module 2053 . The processor 2050 may process diagnostic data through the data processing module 2051 and obtain diagnostic auxiliary information according to the diagnostic data through the diagnostic module 2053 .

1.2.3 Server device

According to an embodiment of the present invention, the diagnosis assistance system may include a server device. A diagnosis assistance system according to an embodiment of the present invention may include a plurality of server devices.

The server device may store and/or run the neural network model. The server device may store weight values constituting the learned neural network model. The server device may collect or store data used for diagnosis assistance.

The server device may output the result of the diagnostic assistance process using the neural network model to the client device. The server device may obtain feedback from the client device. The server device may operate similarly to the aforementioned diagnostic device.

6 illustrates a diagnostic assistance system 20 according to an embodiment of the present invention. Referring to FIG. 6 , the diagnosis assistance system 20 according to an embodiment of the present invention may include a diagnosis server 4000 , a learning device, and a client device.

The diagnosis server 4000 , that is, the server device may communicate with a plurality of learning devices or a plurality of diagnostic devices. Referring to FIG. 6 , the diagnosis server 4000 may communicate with the first learning apparatus 1000a and the second learning apparatus 1000b. Referring to FIG. 6 , the diagnosis server 4000 may communicate with a first client device 3000a and a second client device 3000b.

For example, the diagnosis server 4000 may be configured to train the first learning apparatus 1000a for learning the first diagnosis auxiliary neural network model to obtain the first diagnosis auxiliary information and the second diagnosis auxiliary neural network model to obtain the second diagnosis auxiliary information. It may communicate with the second learning apparatus 1000b.

The diagnostic server 4000 stores a first diagnostic auxiliary neural network model for obtaining the first diagnostic auxiliary information and a second diagnostic auxiliary neural network model for obtaining the second diagnostic auxiliary information, the first client device 3000a or the second client device The diagnosis assistance information may be acquired in response to a request to obtain the diagnosis assistance information from 3000b , and the acquired diagnosis assistance information may be transmitted to the first client device 3000a or the second client device 3000b .

Alternatively, the diagnosis server 4000 may communicate with the first client device 3000a requesting the first diagnosis assistance information and the second client device 3000b requesting the second diagnosis assistance information.

1.2.4 Client Device

The client device may request the diagnostic assistance information from the diagnostic device or the server device. The client device may acquire data required for diagnosis and transmit the acquired data to the diagnosis device.

The client device may include a data acquisition unit. The data acquisition unit may acquire data necessary for diagnosis assistance. The data acquisition unit may be an imaging unit that acquires an image used for the diagnosis assistance model.

7 is a block diagram illustrating a client device 3000 according to an embodiment of the present invention. Referring to FIG. 7 , a client device 3000 according to an embodiment of the present invention may include an imaging unit 3100 , a control unit 3200 , and a communication unit 3300 .

The imaging unit 3100 may acquire an image or image data. The imaging unit 3100 may acquire a fundus image. However, the client device 3000 may be replaced with a data acquisition unit other than the imaging unit 3100 .

The communication unit 3300 may communicate with an external device, for example, a diagnostic device or a server device. The communication unit 3300 may perform wired or wireless communication.

The controller 3200 may control the imaging unit 3100 to acquire an image or data. The controller 3200 may control the imaging unit 3100 to acquire a fundus image. The controller 3200 may transmit the acquired fundus image to the diagnosis apparatus. The controller may transmit the image acquired through the imaging unit 3100 to the server device through the communication unit 3300, and may acquire the generated diagnostic auxiliary information based thereon.

Although not shown, the client device may further include an output unit. The output unit may include a display that outputs an image or image, or a speaker that outputs an audio. The output unit may output an image or image data acquired by the acquired imaging unit. The output unit may output the diagnosis auxiliary information obtained from the diagnosis apparatus.

Although not shown, the client device may further include an input unit. The input unit may obtain a user input. For example, the input unit may obtain a user input for requesting diagnosis auxiliary information. The input unit may obtain user information for evaluating the diagnosis auxiliary information obtained from the diagnosis apparatus.

Also, although not shown, the client device may further include a memory unit. The memory unit may store the image acquired by the imaging unit.

1.3 Diagnostic Auxiliary Process Overview

A diagnostic assistance process may be performed by a diagnostic assistance system or a diagnostic assistance apparatus disclosed herein. The diagnosis assistance process may be largely divided into a training process for learning a diagnosis assistance model used for diagnosis assistance, and a diagnosis process using the diagnosis assistance model.

8 is a diagram for explaining a diagnosis assistance process according to an embodiment of the present invention. Referring to FIG. 8 , the diagnostic assistance process according to an embodiment of the present invention acquires and processes data (S110) to learn a neural network model (S130), and obtains variables of the learned neural network model (S150) It may include a learning process and a diagnosis assistance process of acquiring the diagnosis target data ( S210 ) and acquiring the diagnosis auxiliary information using the neural network model S230 learned based on the diagnosis target data ( S250 ).

More specifically, the training process may include a data processing process of processing the input training image data to a state that can be used for training of the model, and a learning process of training the model using the processed data. The training process may be performed by the above-described learning apparatus.

The diagnosis process may include a data processing process of processing input image data to be examined to a state capable of performing diagnosis using a neural network model, and a diagnosis process of performing diagnosis using the processed data. The diagnostic process may be performed by the aforementioned diagnostic device or a server device.

Below, each process is demonstrated.

2. Training process

According to an embodiment of the present invention, a process for training a neural network model may be provided. As a specific example, a process of training a neural network model for performing or assisting a diagnosis based on a fundus image may be disclosed.

The training process described below may be performed by the above-described learning apparatus.

2.1 Study Department

According to an embodiment of the present invention, the training process may be performed by the learning unit. The learning unit may be provided in the above-described learning apparatus.

9 is a diagram for explaining the configuration of the learning unit 100 according to an embodiment of the present invention. Referring to FIG. 9 , the learning unit 100 may include a data processing module 110 , a queue module 130 , a learning module 150 , and a learning result acquisition module 170 . Each module may perform separate steps of the data processing process and the learning process as described below. However, not all of the components and functions performed by the components described in FIG. 9 are essential, and some elements may be added or some elements may be omitted depending on the learning type.

2.2 Data processing process

2.2.1 Image data acquisition

According to an embodiment of the present invention, a data set may be obtained. According to an embodiment of the present invention, the data processing module may acquire a data set.

The data set may be an image data set. Specifically, it may be a fundus image data set. The fundus image data set may be acquired using a general annovulatory fundus camera or the like. The fundus image may be a panoramic image. The fundus image may be a red-free image. The fundus image may be an infrared photographed image. The fundus image may be an autofluorescence photographed image. The image data may be acquired in any one of JPG, PNG, DCM (DICOM), BMP, GIF, and TIFF format.

The data set may include a training data set. The data set may include a test data set. The data set may include a validation data set. In other words, the data set may be assigned as at least one of a training data set, a test data set, and a validation data set.

The data set may be determined in consideration of the diagnostic auxiliary information to be obtained using a neural network model learned through the data set. For example, when training a neural network model for acquiring cataract-related diagnostic assistance information, the acquired data set may be determined as an infrared fundus image data set. Alternatively, when training a neural network model for acquiring diagnostic auxiliary information related to macular degeneration, the acquired data set may be an autofluorescence-captured fundus image data set.

Individual data included in a data set may include a label. There may be a plurality of labels. In other words, individual data included in the data set may be labeled with respect to at least one feature. For example, the data set is a fundus image data set including a plurality of fundus image data, and each fundus image data is a diagnostic information label (eg, presence or absence of a specific disease) and/or observation information (eg, specific Whether there is an abnormality in the site) may include a label.

As another example, the data set is a fundus image data set, and each fundus image data may include a peripheral information label for the corresponding image. For example, each fundus image data includes left and right eye information on whether the fundus image is an image of the left eye or an image of the right eye, gender information on whether the fundus image is a female fundus image or a male fundus image, and a subject who has taken the fundus image It may include a peripheral information label including age information and the like for the age of the .

10 is a conceptual diagram illustrating an image data set DS according to an embodiment of the present invention. Referring to FIG. 10 , an image data set DS according to an exemplary embodiment may include a plurality of image data IDs. Each image data ID may include an image I and a label L assigned to the image. Referring to FIG. 10 , the image data set DS may include first image data ID1 and second image data ID2 . The first image data ID1 may include a first image I1 and a first label L1 corresponding to the first image.

In FIG. 10 , the description is based on the case where one image data includes one label, but as described above, one image data may include a plurality of labels.

2.2.2 Image resizing

According to an embodiment of the present invention, the size of the acquired image data may be adjusted. That is, images may be resized. According to an embodiment of the present invention, image resizing may be performed by the data processing module of the above-described learning unit.

The size or aspect ratio of the image can be adjusted. The size of the plurality of acquired images may be adjusted to have a constant size. Alternatively, the images may be resized to have a constant aspect ratio. Resizing the image may be applying an image transformation filter to the image.

When the size or capacity of the acquired individual images is excessively large or small, the size or capacity of the image may be adjusted to convert the image to an appropriate size. Alternatively, when the sizes or capacities of individual images vary, the sizes or capacities may be unified through resizing.

According to an embodiment, the capacity of the image may be adjusted. For example, when the capacity of the image exceeds an appropriate range, the image may be reduced through down sampling. Alternatively, when the capacity of the image is less than an appropriate range, the image may be enlarged through upsampling or interpolating.

According to another embodiment, the size or aspect ratio of the image may be adjusted by cropping the image or adding pixels to the acquired image. For example, when an image includes a part unnecessary for learning, a part of the image may be cropped to remove it. Alternatively, when a part of the image is cropped and the aspect ratio does not match, a column or row may be added to adjust the image aspect ratio. In other words, you can adjust the aspect ratio by adding margin or padding to the image.

According to another embodiment, the capacity and size or aspect ratio of the image may be adjusted together. For example, when the size of the image is large, the image capacity may be reduced by downsampling the image, and unnecessary parts included in the reduced image may be cropped to convert the image data into appropriate image data.

Also, according to another embodiment of the present invention, the orientation of image data may be changed.

As a specific example, when a fundus image data set is used as the data set, the capacity or size of each fundus image may be adjusted. Cropping for removing a blank portion excluding the fundus portion of the fundus image or padding for adjusting the aspect ratio by supplementing the cropped portion of the fundus image may be performed.

11 is a diagram for explaining image resizing according to an embodiment of the present invention. Referring to FIG. 11 , the acquired fundus image may be resized by the image resizing process according to an embodiment of the present invention.

Specifically, in the original fundus image (a), a blank area unnecessary for obtaining diagnostic information may be cropped (b) and its size may be reduced (c) to improve learning efficiency.

2.2.3 Image preprocessing

According to an embodiment of the present invention, pre-processing of an image may be performed. When an image is used as it is for learning, an overfitting phenomenon may occur as a result of learning about unnecessary characteristics, and learning efficiency may also decrease.

In order to prevent this, learning efficiency and performance can be improved by appropriately pre-processing and using image data to meet the purpose of learning. For example, pre-processing for easy detection of abnormal signs of eye disease or pre-processing to emphasize changes in retinal blood vessels or blood flow may be performed on the fundus image.

Pre-processing of the image may be performed by the data processing module of the above-described learning unit. The data processing module may acquire a resized image and perform pre-processing requested for learning.

The image pre-processing may be performed on the image on which the above-described resizing process has been completed. However, the content of the invention disclosed in the present specification is not limited thereto, and pre-processing of the image may be performed while omitting the resizing process. Applying pre-processing of the image may be applying a pre-processing filter to the image.

According to an embodiment, a blur filter may be applied to the image. A Gaussian filter may be applied to the image. A Gaussian blur filter may be applied to the image. Alternatively, a deblur filter for sharpening the image may be applied to the image.

According to another embodiment, a filter for adjusting or modulating the color of the image may be applied. For example, values of some components of RGB values constituting the image may be changed, or a filter for binarizing the image may be applied.

According to another exemplary embodiment, a filter to emphasize a specific element in an image may be applied. For example, on fundus image data, pre-processing may be performed such that vascular elements are emphasized from each image. In this case, the pre-processing to emphasize the vascular element may be to apply one or more filters sequentially or in combination.

According to an embodiment of the present invention, pre-processing of an image may be performed in consideration of characteristics of diagnostic auxiliary information to be acquired. For example, when it is desired to acquire diagnostic auxiliary information related to findings such as retinal hemorrhage, drusen, microvascular flow, and exudate, preprocessing of converting the acquired fundus image into a red-free fundus image form may be performed.

2.2.4 Image augmentation

According to an embodiment of the present invention, the image may be augmented or expanded. Augmentation of the image may be performed by the data processing module of the above-described learning unit.

The augmented images may be used to improve training performance of the neural network model. For example, when the amount of data for training a neural network model is insufficient, by modulating the existing training image data to expand the number of data for training, and using the modulated (or changed) image together with the original image. The number of training image data can be increased. Accordingly, overfitting can be suppressed, the layers of the model can be formed more deeply, and the prediction accuracy can be improved.

For example, the expansion of the image data may be performed by inverting the left and right sides of the image, cutting out a part of the image, correcting a color value of the image, or adding artificial noise. As a specific example, cutting out a part of the image may be performed by cutting out some areas of elements constituting the image or randomly cutting out some areas. As more examples, image data can be expanded by inverting left and right, inverting up and down, rotating, resizing at a certain rate, cropping, padding, adjusting color, or adjusting brightness.

As an example, the above-described augmentation or expansion of image data may be generally applied to a training data set. However, it may be applied to other data sets, for example, a test data set, that is, a data set for testing a model that has been trained using training data and verified using verification data.

As a specific example, when a fundus image data set is used as a data set, by randomly applying one or more processing of inverting the image, cropping, adding noise, or changing the color, in order to increase the number of data, An augmented fundus image data set may be acquired.

12 is a diagram for explaining the expansion of an image data set according to an embodiment of the present invention. 12 , an image according to embodiments of the present invention may be modified to improve prediction accuracy of a neural network model.

Specifically, referring to FIG. 12 , in an image according to an embodiment of the present invention, a partial region is dropped out (a), left and right inverted (b), a center point is moved (c, d), or a partial region is The color may be modulated (e).

2.2.5 Image serialization

According to an embodiment of the present invention, image data may be linearized. The image may be serialized by the data processing module of the above-described learning unit. The serialization module can serialize the preprocessed image data and pass it to the queue module.

When image data is used for learning as it is, decoding is required because image data has the form of image files such as JPG, PNB, DCM, etc. If learning is performed after decoding each time, the performance of model learning may deteriorate. . Accordingly, it is possible to perform learning by serializing the image file without using it for learning as it is. Accordingly, image data can be serialized to improve learning performance and speed. The image data to be serialized may be image data to which one or more of the above-described image resizing and image pre-processing has been applied, or image data to which both are not processed.

Each image data included in the image data set may be converted into a string form. The image data may be converted into a binarized data form. In particular, image data may be converted into a data form suitable for use in training a neural network model. As an example, image data may be converted into a TFRecord form for use in training a neural network model using tensorflow.

As a specific example, when a fundus image set is used as a data set, the acquired fundus image set may be converted into a TFRecord form and used for training a neural network model.

2.2.6 Queue

A queue may be used to solve a data bottleneck. The above-described queue module of the learning unit may store image data in a queue and transmit it to the learning model module.

In particular, when the learning process is performed using both the CPU (Central Processing Unit) and the GPU (Graphic Processing Unit), by using a queue, the bottleneck between the CPU and GPU is minimized, access to the database is smooth, and memory Use efficiency can be improved.

The queue may store data used for training the neural network model. A queue can store image data. The image data stored in the queue may be image data processed by at least one of the above-described data processing processes (ie, resizing, pre-processing, and augmentation) or an image as it is acquired.

The queue may store image data, preferably serialized image data as described above. The queue stores image data and can supply image data to the neural network model. The queue can deliver image data in units of batch size to the neural network model.

A queue may provide image data. The queue may provide data to a learning module, which will be described later. As data is extracted from the learning module, the number of data accumulated in the queue may be reduced.

If the number of data stored in the queue decreases below the standard as the neural network model is trained, the queue may request data replenishment. A queue may request replenishment of certain types of data. In the queue, the learning unit may replenish data to the queue when replenishment of data is requested.

The queue may be provided in the system memory of the learning device. For example, the queue may be formed in a random access memory (RAM) of a central processing unit (CPU). In this case, the size of the queue, that is, the capacity may be determined according to the RAM capacity of the CPU. As the queue, a First In First Out (FIFO) queue, a primary queue, or a random queue may be used.

2.3 Learning process

According to an embodiment of the present invention, a learning process of a neural network model may be initiated.

According to an embodiment of the present invention, learning of the neural network model may be performed by the above-described learning apparatus. The learning process may be performed by a control unit of the learning apparatus. The learning process may be performed by the learning module of the learning unit described above.

13 is a block diagram illustrating a learning process of a neural network model according to an embodiment of the present invention. Referring to FIG. 13 , the learning process of a neural network model according to an embodiment of the present invention acquires data (S1010), trains the neural network model (S1030), validates the trained model (S1050), and the trained model It can be performed by obtaining the variable of ( S1070 ).

Hereinafter, with reference to FIG. 13 , some embodiments will be described for a learning process of a neural network model.

2.3.1 Data entry

A data set for training of a diagnostic auxiliary neural network model may be obtained.

The data obtained may be an image data set processed by the data processing process described above. As an example, the data set may include serialized fundus image data after it has been resized, a pre-processing filter has been applied, and the data has been augmented.

In the training phase of the neural network model, a training data set may be acquired and used. In the verification phase of the neural network model, a verification data set may be obtained and used. In the testing phase of the neural network model, a test data set may be acquired and used. Each data set may include fundus images and labels.

A data set may be obtained from a queue. The data set may be obtained from the queue in batch size units. For example, when 60 is designated as the batch size, the data set may be extracted from the queue in units of 60. The size of the batch size may be limited by the RAM capacity of the GPU.

The data set may be obtained randomly from the queue to the learning module. The data sets may be acquired in the order in which they are accumulated in the queue.

The learning module may extract by specifying the configuration of the data set obtained from the queue. For example, the learning module may extract fundus image data having a left eye label and fundus image data having a right eye label of a specific subject to be used for learning together.

The learning module may obtain a data set of a specific label from the queue. As an example, the learning module may acquire the fundus image data set in which the diagnostic information label is abnormal from the queue. The learning module may acquire the data set by specifying the ratio of the number of data according to the label from the queue. As an example, the learning module may acquire the fundus image data set such that the number of fundus image data having an abnormal diagnostic information label and the number of fundus image data having a normal diagnostic information label are 1:1 from the queue.

2.3.2 Model design

The neural network model may be a diagnostic auxiliary model that outputs diagnostic auxiliary information based on image data. A structure of a diagnostic auxiliary neural network model for obtaining diagnostic auxiliary information may have a predetermined shape. A neural network model may include a plurality of layers or layers.

The neural network model may be implemented in the form of a classifier that generates diagnostic auxiliary information. The classifier can perform double classification or multiple classification. For example, the neural network model may be a binary classification model that classifies input data into a normal or abnormal class with respect to target diagnosis auxiliary information such as a specific disease or abnormal symptom. Alternatively, the neural network model may be a multi-classification model that classifies input data into a plurality of class classes for a specific characteristic (eg, a degree of disease progression). Alternatively, the neural network model may be implemented as a regression type model that outputs a specific value related to a specific disease.

The neural network model may include a convolutional neural network (CNN). can As the CNN structure, at least one of AlexNet, LENET, NIN, VGGNet, ResNet, WideResnet, GoogleNet, FractaNet, DenseNet, FitNet, RitResNet, HighwayNet, MobileNet, and DeeplySupervisedNet may be used. The neural network model may be implemented using a plurality of CNN structures.

As an example, the neural network model may be implemented to include a plurality of VGGNet blocks. As a more specific example, the neural network model has a first structure in which a CNN layer having 64 filters of 3x3 size, a batch normalization (BN) layer, and a ReLu layer are sequentially combined, and a CNN layer having 128 filters of a 3x3 size, a ReLu layer and The second block in which the BN layers are sequentially coupled may be coupled to each other.

The neural network model includes a max pooling layer following each CNN block, and a global average pooling (GAP) layer, a fully connected (FC) layer, and an activation layer (eg, sigmoid, softmax, etc.) at the end. have.

2.3.3 Training the model

A neural network model can be trained using a training data set.

Neural network models can be trained using labeled data sets. However, the learning process of the diagnostic auxiliary neural network model described in this specification is not limited thereto, and the neural network model may be trained in an unsupervised form using unlabeled data.

Learning of a neural network model is based on training image data, obtains a result value using a neural network model to which arbitrary weight values are given, compares the obtained result value with a label value of training data, and back propagates according to the error by optimizing the weight values. In addition, learning of the neural network model may be influenced by a verification result of a model, a test result, and/or feedback from a diagnosis step, which will be described later.

Learning of the above-described neural network model may be performed using TensorFlow. However, the present invention is not limited thereto, and frameworks such as Theano, Keras, Caffe, Torch, and CNTK (Microsoft Cognitive Toolkit) are used for learning the neural network model. could be

2.3.4 Model validation

A neural network model can be validated using a validation data set. Validation of the neural network model may be performed by obtaining a result value for the validation data set from the neural network model in which training has been performed, and comparing the result value with the label of the validation data set. Verification may be performed by measuring the accuracy of the result value. According to the verification result, parameters (eg, weights and/or biases) or hyperparameters (eg, learning rate) of the neural network model may be adjusted.

As an example, the learning apparatus according to an embodiment of the present invention trains a neural network model for predicting diagnostic auxiliary information based on a fundus image, and corresponds to the verified fundus image with diagnostic auxiliary information on the verified fundus image of the learned model. The validation of the diagnostic auxiliary neural network model can be performed by comparing it with the validation label.

For the validation of the neural network model, a separate external data set, that is, a data set having distinct factors not included in the training data set, may be used. For example, the separate validation set may be a data set in which factors such as race, environment, age, and gender are distinguished from a training data set.

2.3.5 Testing the model

The neural network model can be tested using a test data set.

Although not shown in FIG. 13 , according to the learning process according to an embodiment of the present invention, a neural network model can be tested using a test data set that is distinguished from a training data set and a validation data set. Depending on the test results, parameters (eg, weights and/or biases) or hyperparameters (eg, learning rate) of the neural network model may be adjusted.

For example, the learning apparatus according to an embodiment of the present invention obtains, from a neural network model trained to predict diagnostic auxiliary information based on a fundus image, a result value of inputting test fundus image data that is not used for training and verification as an input. Thus, it is possible to test the trained and verified diagnostic auxiliary neural network model.

For testing the neural network model, a separately prepared external data set, that is, a data set having factors distinct from training and/or verification data, may be used.

2.3.6 Output of results

As a result of learning the neural network model, parameter values of the optimized model may be obtained. As described above, more appropriate parameter (or variable) values can be obtained by iteratively training the model using the test data set. When the learning proceeds sufficiently, an optimized value of a weight and/or a bias may be obtained.

According to an embodiment of the present invention, the learned neural network model and/or parameters or variables of the learned neural network model may be stored in a learning device and/or a diagnosis device (or server). The learned neural network model may be used for prediction of diagnostic auxiliary information by a diagnostic device and/or a client device. In addition, parameters or variables of the learned neural network model may be updated by feedback obtained from a diagnostic device or a client device.

2.3.7 Model ensemble

According to an embodiment of the present invention, in the process of learning one diagnostic auxiliary neural network model, a plurality of sub-models may be simultaneously learned. The plurality of sub-models may have different hierarchical structures.

In this case, the diagnostic auxiliary neural network model according to an embodiment of the present invention may be implemented by combining a plurality of sub-neural network models. In other words, learning of the neural network model can be performed using an ensemble technique that combines a plurality of sub-neural networks.

When an ensemble is formed to construct a diagnostic auxiliary neural network model, the prediction can be performed by synthesizing the results predicted from various types of sub-neural network models, and thus the accuracy of the prediction of the results can be further improved.

14 is a block diagram illustrating a learning process of a neural network model according to an embodiment of the present invention. Referring to FIG. 14 , in the learning process of a neural network model according to an embodiment of the present invention, a data set is acquired ( S1011 ), and a first model (ie, a first neural network model) and a second model using the acquired data Train the model (that is, the second neural network model) (S1031, S1033), verify the trained first neural network model and the second neural network model (S1051), determine the final neural network model, and obtain its parameters or variables (S10721) )can do.

Hereinafter, with reference to FIG. 14 , some embodiments will be described for a learning process of a neural network model.

According to an embodiment of the present invention, a plurality of sub-neural network models may acquire the same training data set and individually generate output values. In this case, an ensemble of a plurality of sub neural network models may be determined as a final neural network model, and parameter values for each of the plurality of sub neural network models may be obtained as a learning result. The output value of the final neural network model may be determined as an average value of output values by each sub-neural network model. Alternatively, the output value of the final neural network model may be determined as a weighted average value of the output values of each sub neural network model in consideration of the accuracy obtained as a result of verification for each sub neural network model.

As a more specific example, when the neural network model includes the first sub neural network model and the second sub neural network model, the parameter values optimized for the first sub neural network model and the optimized parameter values for the second sub neural network model by machine learning can be obtained. In this case, an average value of output values (eg, probability values for specific diagnostic auxiliary information) respectively obtained from the first sub-neural network model and the second sub-neural network model may be determined as the output value of the final neural network model.

According to another embodiment of the present invention, the accuracy of each sub-neural network model may be evaluated based on output values from each of the plurality of sub-neural network models. In this case, any one of a plurality of sub-neural network models may be selected based on the accuracy and determined as the final sub-neural network model. The structure of the determined sub-neural network model and parameter values of the determined sub-neural network model obtained as a result of learning may be stored.

As a more specific example, when the neural network model includes the first sub neural network model and the second sub neural network model, accuracy according to each of the first sub neural network model and the second sub neural network model is obtained, and a more accurate sub neural network model is finalized It can be determined with a neural network model.

According to another embodiment of the present invention, at least one sub-neural network among a plurality of neural network models is combined, an ensemble of the combined at least one sub-neural network model is formed, and each ensemble is evaluated, but the plurality of ensembles A combination of sub-neural network models that form an ensemble with high accuracy can be determined as the final neural network model. In this case, the ensemble may be performed on all possible cases of selecting at least one of the plurality of sub neural network models, and the sub neural network combination evaluated as having the highest accuracy may be determined as the final neural network model.

As a more specific example, when the neural network model includes the first sub neural network model and the second sub neural network model, the accuracy of the first sub neural network model, the accuracy of the second sub neural network model, and the ensemble of the first and second sub neural network models By comparing the accuracies by , it is possible to determine the sub-neural network model configuration in the most accurate case as the final neural network model.

2.4 Example 1 - Control method of a learning device.

15 is a diagram for explaining a method of controlling a learning apparatus according to an embodiment of the present invention.

15 , the control method of the learning apparatus according to an embodiment of the present invention includes performing pre-processing of a first fundus image ( S110 ), serializing the pre-processed first fundus image ( S130 ), first It may include training the neural network model (S150).

A method of controlling a learning apparatus according to an embodiment of the present invention obtains a first training data set including a plurality of fundus images, processes the fundus image included in the first training data set, and performs the first training A learning apparatus for learning a first neural network model using a data set, acquires a target fundus image for acquiring diagnostic assistance information, and uses the learned first neural network model to assist the diagnosis based on the target fundus image It may be a method of controlling a learning device included in a system including a diagnostic device for acquiring information.

In the step of performing pre-processing of the first fundus image ( S110 ), pre-processing of the first fundus image is performed so that the first fundus image included in the first training data set is converted into a form suitable for learning the first neural network model. may further include

The method of controlling a learning apparatus according to an embodiment of the present invention may include serializing the preprocessed first fundus image (S130). The first fundus image may be serialized in a form that is easy for training of the neural network model.

In this case, the step of training the first neural network model ( S150 ) further includes training the first neural network model for classifying the target fundus image into a first label or a second label by using the serialized first fundus image. can do.

The learning apparatus may acquire a second training data set that includes a plurality of fundus images and is at least partially different from the first training data set, and trains a second neural network model by using the second training data set.

According to an embodiment of the present invention, the method for controlling the learning apparatus includes performing pre-processing of the second fundus image so that the second fundus image included in the second training set is suitable for learning the second neural network model; The method may further include the steps of serializing the second fundus images and training a second neural network model for classifying the target fundus image into a third label or a fourth label by using the serialized second fundus image.

16 is a diagram for explaining a method of controlling a learning apparatus according to an embodiment of the present invention. Referring to FIG. 16 , the method for controlling a learning apparatus according to an embodiment of the present invention includes performing pre-processing of a second fundus image (S210), serializing the pre-processed second fundus image (S230), a second It may include training the neural network model (S250).

In FIG. 16 , for convenience of explanation, preprocessing of the first fundus image, serialization of the first fundus image, and learning using the first fundus image, followed by preprocessing of the first fundus image, serialization of the second fundus image, and the first fundus image Although it has been described as being performed following learning using , the content of the present invention is not limited thereto.

The preprocessing of the second fundus image included in the second training set, and the serialization of the second fundus image and the learning using the second fundus image are the above-described preprocessing of the first fundus image, serialization of the first fundus image, and the first fundus. It can be performed independently of learning using images. The preprocessing of the second fundus image included in the second training set, and the serialization of the second fundus image and the learning using the second fundus image are the above-described preprocessing of the first fundus image, serialization of the first fundus image, and the first fundus. It can be performed in parallel with learning using images. In other words, the pre-processing of the second fundus image included in the second training set, and the serialization of the second fundus image and learning using the second fundus image must necessarily include the above-described preprocessing of the first fundus image and serialization of the first fundus image. And it does not have to be done after or before learning using the first fundus image, and the processing of the first fundus image and the processing of the second fundus image may be performed without mutual dependence.

The first pre-processing performed on the fundus image included in the first training data set may be distinguished from the second pre-processing performed on the fundus image included in the second training data set. For example, the first preprocessing may be a blood vessel enhancement preprocessing, and the second preprocessing may be a color modulation preprocessing. Each preprocessing may be determined in consideration of the diagnostic auxiliary information to be acquired through each neural network model.

The method for controlling a learning apparatus according to an embodiment of the present invention evaluates the accuracy of the learned first neural network model using a first verification data set that is at least partially different from a first training data set, Validating the first neural network model and evaluating the accuracy of the learned first neural network model by using a second validation data set that is at least partially distinct from a second training data set, and the second neural network model It may further include the step of verifying. In this case, the verification of the first neural network model and the second neural network model may be independently performed.

The serialized first fundus image is sequentially stored in a first queue, and the serialized fundus image stored in the first queue is used for training the first neural network model in a predetermined capacity unit from the first queue, A second fundus image is sequentially stored in a second queue distinct from the first queue, and the serialized fundus image stored in the second queue is used in training of the first neural network model from the second queue in a predetermined capacity unit. can be used

The first neural network model may include a first sub neural network model and a second sub neural network model. In this case, the classification of the target fundus image into the first label or the second label is performed in consideration of the first predicted value predicted by the first sub-neural network model and the second predicted value predicted by the second sub-neural network model. can be performed.

The second neural network model may include a third sub neural network model and a fourth sub neural network model. In this case, the classification of the target fundus image into the third label or the fourth label is performed in consideration of the third predicted value predicted by the third sub-neural network model and the fourth predicted value predicted by the fourth sub-neural network model. can be performed.

The first training data set may include at least part of the fundus image labeled with the first label, and the second training data set may include at least part of the fundus image labeled with the third label. In this case, the fundus image labeled with the first label and at least a portion of the fundus image labeled with the third label may be partially in common.

The first label is a normal label indicating that the subject corresponding to the target fundus image is normal with respect to the first observation, and the second label is an abnormal label indicating that the subject is abnormal with respect to the second observation. can

The pre-processing of the first fundus image may include cropping the first fundus image to satisfy a reference aspect ratio and changing the size of the first fundus image.

The pre-processing of the first fundus image may further include applying, by the processing unit, a blood vessel enhancement filter to the fundus image so that the blood vessels included in the first fundus image are emphasized.

The serialized first fundus image may be sequentially stored in a queue, and the serialized first fundus image stored in the queue may be used to train the first neural network model in a unit of a predetermined capacity from the queue. The queue may request replenishment of the serialized first fundus image when the serialized first fundus image not used for the first learning decreases below a reference capacity.

The first finding may be any one of retinal hemorrhage, retinal exudate, lens opacity, and diabetic retinopathy.

17 is a diagram for explaining a method of controlling a learning apparatus according to an embodiment of the present invention.

Referring to FIG. 17 , the method of controlling a learning apparatus according to an embodiment of the present invention may further include verifying the first neural network model ( S170 ) and updating the first neural network model ( S190 ).

The step of verifying the first neural network model ( S170 ) is to evaluate the accuracy of the learned first neural network model by using a first validation data set that is at least partially different from the first training data set to obtain one neural network model. It may further include verifying.

The method may further include updating the first neural network model by reflecting the verification result obtained from the step of updating the first neural network model ( S190 ) and the step of verifying the first neural network model ( S170 ).

Meanwhile, the first neural network model may include a first sub neural network model and a second sub neural network model. In this case, the step of training the first neural network model may include verifying the first sub-neural network model using a first validation data set to obtain accuracy of the first sub-neural network model, and using the first validation data set. Acquire the accuracy of the second sub neural network model by verifying the second sub neural network model, and compare the accuracy of the first sub neural network model and the accuracy of the second sub neural network model to convert the more accurate sub neural network model into the final neural network model This may include deciding

3. Diagnostic Auxiliary Process

According to an embodiment of the present invention, a diagnosis assistance process (or diagnosis process) for obtaining diagnosis assistance information using a neural network model may be provided. As a specific example, the diagnosis assistance information (eg, diagnosis information or observation information) may be predicted by using the fundus image and through the learned diagnosis assistance neural network model by the diagnosis assistance process.

The diagnostic assistance process described below may be performed by the diagnostic apparatus.

3.1 Diagnostics

According to an embodiment of the present invention, the diagnosis process may be performed by the diagnosis unit 200 . The diagnosis unit 200 may be provided in the above-described diagnosis apparatus.

18 is a diagram for explaining the configuration of the diagnosis unit 200 according to an embodiment of the present invention. Referring to FIG. 18 , the diagnosis unit 200 may include a diagnosis request obtaining module 210 , a data processing module 230 , a diagnosis module 250 , and an output module 270 .

Each module may perform separate steps of the data processing process and the learning process as described below. However, not all of the components and functions performed by the components described in FIG. 16 are essential, and some elements may be added or some elements may be omitted depending on the aspect of diagnosis.

3.2 Data Acquisition and Diagnosis Requests

The diagnosis apparatus according to an embodiment of the present invention may obtain diagnosis target data and may acquire diagnosis auxiliary information based thereon. The diagnosis target data may be image data. Data acquisition and diagnosis request acquisition may be performed by the diagnosis request acquiring module of the diagnosis unit.

19 is a diagram for explaining diagnosis target data TD according to an embodiment of the present invention. Referring to FIG. 19 , the diagnosis target data TD may include a diagnosis target image TI and patient information (PI).

The diagnosis target image TI may be an image for acquiring diagnosis auxiliary information on the diagnosis target object. For example, the diagnosis target image may be a fundus image. Diagnosis target (TI) It may have any one format of JPG, PNG, DCM (DICOM), BMP, GIF, and TIFF.

The diagnosis object information PI may be information for identifying a diagnosis target object. Alternatively, the diagnosis object information PI may be characteristic information of a diagnosis target object or image. For example, the diagnosis object information PI may include information such as an imaging date and time of a diagnosis target image, imaging equipment, an identification number, ID, name, gender, age, or weight of the subject to be diagnosed. When the diagnosis target image is a fundus image, the diagnosis object information PI may further include eye-related information, such as binocular information indicating whether the left eye or the right eye.

The diagnostic device may obtain a diagnostic request. The diagnosis apparatus may acquire diagnosis target data together with the diagnosis request. When a diagnosis request is obtained, the diagnosis apparatus may acquire auxiliary diagnosis information by using the learned diagnosis auxiliary neural network model. The diagnostic device may obtain a diagnostic request from the client device. Alternatively, the diagnosis apparatus may obtain a diagnosis request from the user through a separately provided input means.

3.3 Data Processing Process

The acquired data can be processed. The data processing may be performed by the data processing module of the above-described diagnostic unit.

The data processing process may be generally performed similarly to the data processing process in the learning process described above. Hereinafter, the data processing process in the diagnosis process will be described with a focus on the difference from the data processing process in the learning process.

In the diagnostic process, the diagnostic device may acquire data as in the learning process. In this case, the obtained data may be in the same format as the data obtained in the learning process. For example, in the learning process, when the learning apparatus trains a diagnostic auxiliary neural network model using DCM format image data, the diagnostic apparatus may acquire a DCM image and obtain diagnostic auxiliary information using the learned neural network model. .

In the diagnosis process, the acquired diagnosis object image may be resized similarly to image data used in the learning process. The shape of the diagnosis target image may be adjusted to have an appropriate capacity, size, and/or aspect ratio in order to efficiently perform diagnosis assistance information prediction through the learned diagnosis assistance neural network model.

For example, when the diagnosis target image is a fundus image, in order to predict diagnosis information based on the fundus image, resizing, such as cropping an unnecessary part of the image or reducing its size, may be performed.

In the diagnosis process, a preprocessing filter may be applied to the acquired diagnosis target image, similarly to the image data used in the learning process. An appropriate filter may be applied to the diagnosis target image so that the accuracy of prediction of the diagnosis auxiliary information through the learned diagnosis auxiliary neural network model is improved.

For example, when the diagnosis target image is a fundus image, preprocessing for facilitating prediction of positive diagnostic information, such as image preprocessing for emphasizing blood vessels or image preprocessing for emphasizing or weakening a specific color, may be applied to the diagnosis target image. have.

In the diagnosis process, the acquired diagnosis target image may be serialized, similar to the image data used in the learning process. The image to be diagnosed can be converted or serialized into a form that makes it easy to drive a diagnostic model in a specific framework.

Serialization of the diagnostic target image may be omitted. This may be because, unlike in the learning stage, in the diagnosis stage, the number of data processed by the processor at one time is not large, and the burden on the data processing speed is relatively small.

In the diagnosis process, the acquired diagnosis target image may be stored in a queue, similar to image data used in the learning process. However, since the number of processed data in the diagnosis process is smaller than that in the learning process, the step of storing data in the queue may be omitted.

Meanwhile, since an increase in the number of data is not required in the diagnosis process, it is preferable not to use data augmentation or image augmentation procedures, unlike the learning process, in order to obtain accurate diagnosis auxiliary information.

3.4 Diagnostic process

According to an embodiment of the present invention, a diagnosis process using a learned neural network model may be initiated. The diagnostic process may be performed in the aforementioned diagnostic apparatus. The diagnostic process may be performed in the aforementioned diagnostic server. The diagnosis process may be performed by the control unit of the above-described diagnosis apparatus. The diagnosis process may be performed by the diagnosis module of the above-described diagnosis unit.

20 is a diagram for explaining a diagnosis process according to an embodiment of the present invention. Referring to FIG. 20 , the diagnosis process may be performed by acquiring the diagnosis target data ( S2010 ), using the learned neural network model ( S2030 ), and obtaining a result corresponding to the acquired diagnosis target data ( S2050 ). However, data processing may be selectively performed.

Hereinafter, each step of the diagnosis process will be described with reference to FIG. 20 .

3.4.1 Data entry

According to an embodiment of the present invention, the diagnosis module may acquire diagnosis target data. The acquired data may be data processed as described above. As an example, the acquired data may be fundus image data of a subject to which a size is adjusted and pre-processing for emphasizing blood vessels is applied. According to an embodiment of the present invention, a left eye image and a right eye image of a single subject may be input as diagnostic target data together.

3.4.2 Data classification

The diagnostic auxiliary neural network model prepared in the form of a classifier may classify an input diagnostic target image into a positive or negative class with respect to a predetermined label.

The trained diagnostic auxiliary neural network model may receive diagnostic target data and output predicted labels. The trained diagnostic auxiliary neural network model may output a predicted value of the diagnostic auxiliary information. Diagnosis auxiliary information may be acquired using the learned diagnostic auxiliary neural network model. The diagnostic assistance information may be determined based on the predicted label.

For example, the diagnostic auxiliary neural network model may predict diagnostic information (ie, information on the presence or absence of a disease) or observation information (ie, information on the presence or absence of abnormal findings) on an eye disease or systemic disease of the subject. In this case, diagnostic information or observation information may be output in the form of probability. For example, the probability that the subject has a specific disease or the probability that the subject has a specific abnormality in the fundus image of the subject may be output. In the case of using a diagnostic auxiliary neural network model prepared in the form of a classifier, a predicted label may be determined in consideration of whether an output probability value (or prediction score) exceeds a threshold value.

As a specific example, the diagnostic auxiliary neural network model may output the presence or absence of diabetic retinopathy of the subject as a probability value by using the fundus photograph of the subject as a diagnosis target image. In the case of using a classifier-type diagnostic auxiliary neural network model in which 1 is normal, the fundus photograph of the subject is input to the diagnostic auxiliary neural network model, and the probability value of normal: abnormal is 0.74:0.26 for whether the subject has diabetic retinopathy or not. It can be obtained in the form of

Herein, the description has been made based on the case of classifying data using a classifier-type diagnostic auxiliary neural network model, but the present invention is not limited thereto, and specific diagnostic auxiliary numerical values ( For example, blood pressure, etc.) may be predicted.

According to another embodiment of the present invention, suitability information of an image may be obtained. The suitability information may indicate whether the diagnosis target image is suitable for acquiring the diagnosis auxiliary information using the diagnosis auxiliary neural network model.

The suitability information of the image may be quality information of the image. The quality information or the suitability information may indicate whether the diagnosis target image reaches the reference level.

For example, when the diagnosis target image has a defect due to a defect in imaging equipment or the influence of lighting during imaging, a non-conformity result may be output as suitability information for the diagnosis target image. When noise is included in the diagnosis target image by a certain level or more, the diagnosis target image may be determined to be inappropriate.

The fitness information may be a value predicted using a neural network model. Alternatively, the suitability information may be information obtained through a separate image analysis process.

According to an embodiment, even when the image is classified as inappropriate, the diagnosis auxiliary information obtained based on the inappropriate image may be obtained.

According to an embodiment, the image classified as inappropriate may be reviewed by a diagnostic auxiliary neural network model.

In this case, the diagnostic auxiliary neural network model for performing the review may be different from the diagnostic auxiliary neural network model for performing the initial review. For example, the diagnostic apparatus may store the first diagnostic auxiliary neural network model and the second diagnostic auxiliary neural network model, and images classified as inappropriate through the first diagnostic auxiliary neural network model may be reviewed through the second diagnostic auxiliary neural network model. .

According to another embodiment of the present invention, a CAM (Class Activation Map) may be obtained from a learned neural network model. The diagnostic auxiliary information may include a CAM. The CAM may be acquired along with other diagnostic assistance information.

In the case of CAM, it can be acquired selectively. For example, in the case of the CAM, the CAM may be extracted and/or output when diagnostic information or observation information obtained by the diagnostic assistance model is classified as an abnormal class.

3.5 Output of diagnostic auxiliary information

The diagnostic assistance information may be determined based on a label predicted from the diagnostic assistance neural network model.

The output of the diagnostic auxiliary information may be performed by the above-described output module of the diagnostic unit. The diagnostic auxiliary information may be output from the diagnostic device to the client device. The diagnostic auxiliary information may be output from the diagnostic device to the server device. The diagnostic auxiliary information may be stored in a diagnostic device or a diagnostic server. The diagnosis auxiliary information may be stored in a separately provided server device or the like.

Diagnostic auxiliary information may be managed in a database. For example, the acquired diagnosis auxiliary information may be stored and managed together with a diagnosis target image of the subject according to the identification number of the subject. In this case, the diagnosis target image and diagnosis auxiliary information of the subject may be managed according to a time sequence. By managing the diagnosis auxiliary information and the diagnosis target image in time-series, tracking and history management of individual diagnosis information may be facilitated.

Diagnostic auxiliary information may be provided to the user. The diagnostic auxiliary information may be provided to the user through an output means of the diagnostic device or the client device. The diagnosis auxiliary information may be output so that the user can recognize it through a visual or audio output means provided in the diagnosis device or the client device.

According to an embodiment of the present invention, an interface for effectively providing diagnostic assistance information to a user may be provided. This user interface will be described in more detail in 5. User Interface to be described later.

When the CAM is acquired by the neural network model, an image of the CAM may be provided together. In the case of a CAM image, it may be optionally provided. For example, when the diagnostic information obtained through the auxiliary diagnostic neural network model is normal observation information or normal diagnostic information, a CAM image is not provided, and when the acquired diagnostic information is abnormal observation information or abnormal diagnostic information, for more accurate clinical diagnosis A CAM image may be provided.

When the image is classified as inappropriate, suitability information of the image may be provided together. As an example, when an image is classified as inappropriate, the diagnosis auxiliary information and non-conformity determination information obtained according to the image may be provided together.

The diagnosis target image determined to be inappropriate may be classified as a re-photography target image. In this case, a rephotographing guide for a target object of an image classified as a rephotographing target may be provided together with suitability information.

Meanwhile, in response to providing the diagnostic assistance information obtained through the neural network model, feedback related to learning of the neural network model may be obtained. For example, feedback may be obtained to adjust parameters or hyperparameters related to training of the neural network model. The feedback may be obtained through a user input unit provided in the diagnosis device or the client device.

According to an embodiment of the present invention, the diagnosis auxiliary information corresponding to the diagnosis target image may include grade information. The rating information may be selected from among a plurality of ratings. The grade information may be determined based on diagnosis information and/or observation information obtained through the neural network model. The grade information may be determined in consideration of suitability information or quality information of an image to be diagnosed. When the neural network model is a classifier model that performs multiple classification, grade information may be determined in consideration of a class into which a diagnosis target image is classified by the neural network model. When the neural network model is a regression model that outputs a numerical value related to a specific disease, grade information may be determined in consideration of the output numerical value.

For example, the diagnosis auxiliary information obtained in response to the diagnosis target image may include any one selected from the first grade information and the second grade information. The grade information may be selected as the first grade information when abnormal observation information or abnormal diagnosis information is obtained through the neural network model. The grade information may be selected as second grade information when abnormal finding information or abnormal diagnosis information is not obtained through the neural network model. Alternatively, the grade information may be selected as the first grade information when the numerical value obtained through the neural network model exceeds the reference value, and may be selected as the second grade information when the acquired value does not meet the reference value. The first grade information may indicate that abnormal information stronger than the second grade information exists in the diagnosis target image.

Meanwhile, the grade information may be selected as the third grade information when it is determined that the quality of the image to be diagnosed is less than or equal to the standard using image analysis or a neural network model. Alternatively, the diagnosis auxiliary information may include the third level information together with the first or second level information.

When the diagnosis auxiliary information includes the first level information, the first user guide may be output through the output means. The first user guide may indicate that a more precise examination is required for the subject (ie, the patient) corresponding to the diagnosis auxiliary information. For example, the first user guidance may indicate that a secondary diagnosis (eg, diagnosis in a separate medical institution or power supply procedure) is required for the subject. Alternatively, the first user guidance may indicate a required treatment for the subject. As a specific example, when abnormal information on the macular degeneration of the subject is acquired by the diagnostic auxiliary information, the first user guide provides information on the injection prescription and power supply procedure for the subject (eg, at a hospital where power is available). list) can be included.

When the diagnosis auxiliary information includes the second level information, the second user guide may be output through the output means. The second user guide may include a post-management plan for the subject corresponding to the diagnosis auxiliary information. For example, the second user guide may instruct the subject's next treatment time, next treatment subject, and the like.

When the diagnosis target information includes the third level information, the third user guide may be output through the output means. The third user guide may indicate that recapture is required for the diagnosis target image. The third user guide may include information on the quality of the image to be diagnosed. For example, the third user guide may include information on defects present in the diagnosis target image (eg, whether bright artifacts or dark artifacts, or a degree thereof).

4. Diagnostic assistance system for multiple labels

According to an embodiment of the present invention, a diagnosis assistance system for predicting with respect to a plurality of labels (eg, a plurality of diagnostic assistance information) may be provided. To this end, the diagnostic assistance neural network of the aforementioned diagnostic assistance system may be designed to predict a plurality of labels.

Alternatively, in the above-described diagnostic assistance system, a plurality of diagnostic assistance neural networks that predict different labels may be used in parallel. Hereinafter, such a parallel diagnosis assistance system will be described.

4.1 Parallel Diagnostic Auxiliary System Configuration

According to an embodiment of the present invention, a parallel diagnosis assistance system for acquiring a plurality of diagnostic assistance information may be provided. The parallel diagnosis assistance system may train a plurality of neural network models for acquiring a plurality of diagnostic assistance information, and acquire a plurality of diagnostic assistance information using the learned plurality of neural network models.

For example, the parallel diagnosis assistance system includes a first neural network model that acquires first diagnostic auxiliary information related to the presence or absence of an eye disease in the subject and second diagnostic auxiliary information related to the presence or absence of a systemic disease in the subject, based on the fundus image. The second neural network model may be trained, and diagnostic auxiliary information regarding the presence or absence of an eye disease and a systemic disease of the subject may be output by using the learned first neural network model and the second neural network model.

21 and 22 are diagrams for explaining a parallel diagnosis assistance system according to some embodiments of the present invention. 21 and 22 , the parallel diagnosis assistance system may include a plurality of learning units.

Referring to FIG. 21 , the parallel diagnosis assistance system 30 according to an embodiment of the present invention may include a learning device 1000 , a diagnosis device 2000 , and a client device 3000 . In this case, the learning apparatus 1000 may include a plurality of learning units. For example, the learning apparatus 1000 may include a first learning unit 100a and a second learning unit 100b.

Referring to FIG. 22 , a parallel diagnosis assistance system 40 according to an embodiment of the present invention includes a first learning device 1000a, a second learning device 1000b, a diagnosis device 2000, and a client device 3000. may include The first learning apparatus 1000a may include a first learning unit 100a. The second learning apparatus 1000b may include a second learning unit 100b.

21 and 22 , the first learning unit 100a may obtain a first data set and may output a first parameter set of the first neural network model obtained as a result of learning the first neural network model. The second learning unit 100b may obtain a second data set and output a second parameter set of the second neural network model obtained as a result of learning the second neural network model.

The diagnosis apparatus 2000 may include a diagnosis unit 200 . The contents described with reference to FIG. 1 may be similarly applied to the diagnosis apparatus 2000 and the diagnosis unit 200 . The diagnosis unit 200 obtains the first diagnosis auxiliary information and the second diagnosis auxiliary information by using the first neural network model and the second neural network model learned from the first learning unit 100a and the second learning unit 100b, respectively. can do. The diagnosis unit 2000 may store the learned parameters of the first neural network model and the learned parameters of the second neural network model obtained from the first learner 100a and the second learner 100b.

The client device 3000 may include a data acquisition unit, for example, an imaging unit 300 . However, the imaging unit 300 may be replaced with a data acquisition means used to acquire other diagnostic auxiliary information. The client device may transmit a diagnosis request and diagnosis target data (eg, a fundus image obtained from an imaging unit) to the diagnosis device. In response to transmitting the diagnosis request, the client device 3000 may acquire a plurality of diagnosis auxiliary information according to the transmitted diagnosis target data from the diagnosis device.

Meanwhile, in FIGS. 21 and 22 , the diagnosis assistance system 40 has been described with reference to the case in which the first learning unit 100a and the second learning unit 100b are included, but the present invention is not limited thereto. According to another embodiment of the present invention, the learning apparatus may include a learning unit that acquires three or more pieces of different diagnosis auxiliary information. Alternatively, the diagnosis assistance system may include a plurality of learning devices for acquiring different diagnosis assistance information.

More specific operations of the learning device, the diagnostic device, and the client device will be described in more detail below.

4.2 Parallel training process

According to an embodiment of the present invention, a plurality of neural network models may be trained. The training process for learning each neural network model may be performed in parallel.

4.2.1 Parallel Learning Unit

The training process may be performed by a plurality of learning units. Each training process may be performed independently of each other. The plurality of learning units may be provided in one learning apparatus or may be provided in each of the plurality of learning apparatuses.

23 is a diagram for explaining a configuration of a learning apparatus including a plurality of learning units according to an embodiment of the present invention. The configuration and operation of each of the first learning unit 100a and the second learning unit 100b may be implemented similarly to those described above with reference to FIG. 9 .

Referring to FIG. 23 , in the process of a neural network model according to an embodiment of the present invention, a first data processing module 110a, a first queue module 130a, a first learning module 150a, and a first learning result are obtained The first learning unit 100a and the second data processing module 110b, the second queue module 130b, the second learning module 150b and the second learning result obtaining module 170b including the module 170a This may be performed by the learning apparatus 1000 including the second learning unit 100b.

Referring to FIG. 23 , the training process of the neural network model according to an embodiment of the present invention may be performed by the first learner 100a and the second learner 100b, respectively. The first learning unit 100a and the second learning unit 100b may independently learn the first neural network model and the second neural network model. Referring to FIG. 23 , the first learning unit 100a and the second learning unit 100b may be provided in the above-described learning apparatus. Alternatively, the first learning unit and the second learning unit may be provided in different learning devices.

4.2.2 Parallel Data Acquisition

According to an embodiment of the present invention, a plurality of learning units may acquire data. The plurality of learning units may acquire different data sets. Alternatively, a plurality of learning units may acquire the same data set. In some cases, a plurality of learning units may acquire a data set in which some are common. The data set may be a fundus image data set.

The first learning unit may obtain a first data set, and the second learning unit may obtain a second data set. The first data set and the second data set may be distinguished. The first data set and the second data set may have some in common. The first data set and the second data set may be labeled fundus image data sets.

The first data set may include data labeled as normal for the first characteristic and data labeled as abnormal for the first characteristic. For example, the first data set may include fundus images labeled as normal and fundus images labeled as abnormal with respect to lens opacity.

The second data set may include data labeled as normal for the second characteristic (distinct from the first characteristic) and data labeled as abnormal for the second characteristic. For example, with respect to diabetic retinopathy, the second data set may include fundus images labeled as normal and fundus images labeled as abnormal.

The data set labeled as normal for the first feature and data labeled as normal for the second feature included in each of the first data set and the second data set may be common. For example, the first data set includes fundus images labeled normal and abnormal with respect to lens opacity, and the second data set includes fundus images labeled normal and abnormal with respect to diabetic retinopathy. the fundus image labeled as being normal with respect to lens opacity included in the first data set and the fundus image labeled as normal with respect to diabetic retinopathy included in the second data set may be common. can

Alternatively, data labeled as abnormal for the first characteristic and data labeled as abnormal for the second characteristic included in each of the first data set and the second data set may be common. That is, data labeled with respect to a plurality of features may be used for training a neural network model for a plurality of features.

Meanwhile, the first data set may be a fundus image data set photographed by the first method, and the second data set may be a fundus image data set photographed by the second method. The first method and the second method may be any one method selected from red-free photographing, panoramic photographing, autofluorescence photographing, infrared photographing, and the like.

The data set used in each learning unit may be determined in consideration of the diagnosis auxiliary information obtained by the learned neural network. For example, when the first learning unit trains the first neural network model to acquire diagnostic auxiliary information related to retinal abnormalities (eg, microvascular flow, exudate, etc.), red-free photographed first fundus image data set can be obtained. Alternatively, when the second learning unit trains the second neural network model to acquire diagnostic assistance information related to macular degeneration, the second learning unit may acquire the autofluorescence-captured second fundus image data set.

4.2.3 Parallel data processing

The plurality of learning units may process the acquired data, respectively. As described in 2.2 Data Processing Process above, each learning unit may process data by applying one or more of image resizing, pre-processing filter application, image augmentation, and image serialization processes to the acquired data. The first data processing module of the first learning unit may process the first data set, and the second data processing module of the second learning unit may process the second data set.

The first learning unit and the second learning unit included in the plurality of learning units may differently process the obtained data set in consideration of the diagnosis auxiliary information obtained from the neural network model trained by each. For example, in order to train the first neural network model for acquiring the first diagnostic auxiliary information related to hypertension, the first learning unit performs preprocessing for emphasizing blood vessels with respect to fundus images included in the first fundus image data set. can be done Alternatively, in order to train the second neural network model for acquiring the second diagnostic auxiliary information related to abnormal findings such as retinal exudates, microvessels, and the like, the second learning unit relates to the fundus images included in the second fundus image data set. You can also perform pre-processing to convert it to a red-free image.

4.2.4 Parallel queues

The plurality of learning units may store data in a queue. As described in 2.2.6 Queue above, each learning unit can store processed data in a queue and deliver it to the learning module. For example, the first learning unit may store the first data set in the first queue module and provide it to the first learning module sequentially or randomly. The second learning module may store the second data set in the second queue module and provide it to the second learning module sequentially or randomly.

4.2.5 Parallel Learning Process

The plurality of learning units may train the neural network model. Each learning module can independently train a diagnostic auxiliary neural network model that predicts different labels using a training data set. The first learning module of the first learning unit may train the first neural network model, and the second learning module of the second learning unit may train the second neural network module.

A plurality of diagnostic auxiliary neural network models may be trained in parallel and/or independently. As described above, by training the model to predict different labels through a plurality of neural network models, the prediction accuracy for each label can be improved, and the efficiency of the prediction operation can be increased.

Each diagnostic auxiliary neural network model can be prepared similarly to that described in 2.3.2 Model design. Each sub-learning process may be performed similarly to that described above in 2.3.1 to 2.3.5.

The parallel learning process according to an embodiment of the present invention may include training a diagnostic auxiliary neural network model that predicts different labels. The first learning unit may train the first diagnostic auxiliary neural network model for predicting the first label. The second learning unit may train a second diagnostic auxiliary neural network model for predicting the second label.

The first learning unit may acquire the first data set and train the first diagnostic auxiliary neural network model to predict the first label. For example, the first learning unit may train a first diagnostic auxiliary neural network model for predicting whether or not the subject is macular degeneration from the fundus image by using the fundus image training data set labeled with respect to macular degeneration.

The second learning unit may acquire the second data set and train the first diagnostic auxiliary neural network model to predict the second label. For example, the second learning unit may train a second diagnostic auxiliary neural network model for predicting whether the subject has diabetic retinopathy from the fundus image by using the fundus image training data set labeled for diabetic retinopathy.

The learning process of the neural network model will be described in more detail below with reference to FIGS. 24 and 25 .

24 is a diagram for explaining a parallel learning process according to an embodiment of the present invention. The parallel learning process may be applied to both the case where the parallel diagnosis assistance system is implemented as shown in FIG. 21 , the case where the parallel diagnosis assistance system is implemented as shown in FIG. 22 , and other cases. However, for convenience of explanation, the following description will be based on the parallel diagnosis auxiliary system implemented as shown in FIG. 21 .

Referring to FIG. 24 , the parallel learning process may include a plurality of sub-learning processes for respectively learning a plurality of diagnostic auxiliary neural network models that predict different labels. The parallel learning process may include a first sub-learning process for training the first neural network model and a second sub-learning process for training the second neural network model.

For example, the first sub-learning process obtains the first data (S1010a), uses the first neural network model (S1030a), validates the first model (ie, the first diagnostic auxiliary neural network model), and (S1050a) the first This may be performed by acquiring the parameters of the neural network model (S1070a). The second sub-learning process acquires the second data (S1010b), uses the second neural network model (S1030b), validates the second neural network model (ie, the second diagnostic auxiliary neural network model), and (S1050b) the second neural network This may be performed by acquiring the parameters of the model (S1070b).

The sub-learning process may include training the neural network model by inputting training data into the sub-neural network model, verifying the model by comparing the label value obtained as an output with the input training data, and reflecting the validation result back to the sub-neural network model. have.

Each sub-learning process obtains a result value using a neural network model to which arbitrary weight values are given, compares the obtained result value with a label value of training data, and performs backpropagation according to the error, may include optimizing them.

In each sub-learning process, the diagnostic auxiliary neural network model may be validated through a validation data set distinct from the training data set. A validation data set for validating the first neural network model and the second neural network model may be distinguished.

The plurality of learning units may acquire a learning result. Each of the learning result acquisition modules may acquire information about the neural network module learned from the learning module. Each of the learning result acquisition modules may acquire parameter values of the neural network module learned from the learning unit. The first learning result acquisition module of the first learning unit may acquire a first parameter set of the first neural network model learned from the first learning module. The second learning result acquisition module of the second learning unit may acquire a second parameter set of the second neural network model learned from the second learning module.

By each sub-learning process, optimized parameter values of the trained neural network model, that is, a parameter set may be obtained. As learning proceeds using more training data sets, more appropriate parameter values may be obtained.

A first parameter set of the first diagnostic auxiliary neural network model learned by the first sub-learning process may be obtained. A second parameter set of the second diagnostic auxiliary neural network model learned by the second sub-learning process may be obtained. As the learning progresses sufficiently, optimized values of weights and/or biases of the first diagnostic auxiliary neural network model and of the second diagnostic auxiliary neural network model may be obtained.

The acquired parameter set of each neural network model may be stored in a learning device and/or a diagnostic device (or server). The first parameter set of the first diagnostic auxiliary neural network model and the second parameter set of the second diagnostic auxiliary neural network model may be stored together or separately. A parameter set of each learned neural network model may be updated by feedback obtained from a diagnostic device or a client device.

4.2.6 Parallel Ensemble Learning Process

Even when a plurality of neural network models are trained in parallel, the aforementioned ensemble type model learning may be used. Each sub-learning process may include training a plurality of sub-neural network models. The plurality of sub-models may have different hierarchical structures. Hereinafter, the contents described in 2.3.7 may be similarly applied unless otherwise noted.

When a plurality of diagnostic auxiliary neural network models are trained in parallel, some of the sub-learning processes for training each diagnostic auxiliary neural network model train a single model, and some sub-learning processes train a plurality of sub-models together. It may be implemented in the form of learning.

As the model is trained using the ensemble in each sub-learning process, a more optimized form of a neural network model can be obtained in each sub-process, and prediction errors can be reduced.

25 is a diagram for explaining a parallel learning process according to another embodiment of the present invention. Referring to FIG. 25 , each learning process may include training a plurality of sub-neural network models.

Referring to FIG. 25 , the first sub-learning process acquires first data (S1011a), uses a 1-1 neural network model and a 1-2 neural network model (S1031a, S1033a), and a 1-1 neural network model and This may be performed by verifying the 1-2 neural network model (S1051a), and determining the final shape of the first neural network model and its parameters (S1071a). The second sub-learning process acquires the second data (S1011b), uses the 2-1 neural network model and the 2-2 neural network model (S1031b, S1033b), and the 2-1 neural network model and the 2-2 neural network model can be performed by verifying (S1051b) to determine the final shape of the first model (ie, the first diagnostic auxiliary neural network model) and its parameters (S1071b).

The first neural network learned in the first sub-learning process may include a 1-1 neural network model and a 1-2 neural network model. The 1-1 neural network model and the 1-2 neural network model may be provided in different hierarchical structures. The 1-1 neural network model and the 1-2 neural network model may obtain a first data set and output predicted labels, respectively. Alternatively, a label predicted by the ensemble of the 1-1 neural network model and the 1-2 neural network model may be determined as the final predicted label.

In this case, the 1-1 neural network model and the 1-2 neural network model may be verified using the verification data set, and a neural network model with high accuracy may be determined as the final neural network model. Alternatively, the ensemble of the 1-1 neural network model, the 1-2 neural network model, and the 1-1 neural network model and the 1-2 neural network model is verified, and the neural network model form when the accuracy is high among them is determined as the final first neural network. It can also be determined by the model.

Similarly for the second sub-learning process, a neural network of a high-accuracy form among the ensemble of the 2-1 neural network model, the 2-2 neural network model, the 2-1 neural network model, and the 2-2 neural network model is finally obtained as the second sub-learning process. It may be determined as a model (ie, the second diagnostic auxiliary neural network model).

Meanwhile, in FIG. 25 , for convenience, each sub-learning process has been described based on a case in which two sub-models are included, but this is only an example, and the present invention is not limited thereto. The neural network model trained in each sub-learning process may include only one neural network model, or may include three or more sub-models.

4.3 Parallel diagnostic process

According to an embodiment of the present invention, a diagnostic process for obtaining a plurality of diagnostic auxiliary information may be provided. The diagnostic process for acquiring a plurality of diagnostic auxiliary information may be implemented in the form of a parallel diagnostic auxiliary process including a plurality of independent diagnostic processes.

4.3.1 Parallel Diagnostics

According to an embodiment of the present invention, the diagnostic assistance process may be performed by a plurality of diagnostic modules. Each diagnostic aid process can be performed independently.

26 is a block diagram illustrating the diagnosis unit 200 according to an embodiment of the present invention.

Referring to FIG. 26 , the diagnosis unit 200 according to an embodiment of the present invention includes a diagnosis request obtaining module 211 , a data processing module 231 , a first diagnosis module 251 , and a second diagnosis module 253 . and an output module 271 . Each module of the diagnosis unit 200 may operate similarly to the diagnosis module of the diagnosis unit illustrated in FIG. 18 , unless otherwise specified.

In FIG. 26, even when the diagnostic unit 200 includes a plurality of diagnostic modules, the diagnostic request acquisition module 211, the data processing module 231, and the output module 271 are shown as being common, but the present invention It is not limited to this configuration, and a plurality of diagnostic request acquisition modules, data processing modules, and/or output modules may also be provided. A plurality of diagnostic request obtaining modules, data processing modules and/or output modules may also operate in parallel.

For example, the diagnosis unit 200 includes a first data processing module that performs first processing on the input diagnostic target image and a second processing module that performs second data processing on the diagnostic target image, The first diagnosis module may obtain first diagnosis auxiliary information based on the first processed diagnosis subject image, and the second diagnosis module may obtain second diagnosis auxiliary information based on the second processed diagnosis subject image. The first processing and/or the second processing may be any one selected from image resizing, image color modulation, blur filter application, blood vessel enhancement processing, red-free transformation, partial region cropping, and partial element extraction.

The plurality of diagnostic modules may acquire different diagnostic auxiliary information. The plurality of diagnosis modules may acquire the diagnosis auxiliary information by using different diagnosis auxiliary neural network models. For example, the first diagnosis module acquires first diagnostic auxiliary information related to whether the subject has an eye disease by using a first neural network model that predicts whether the subject has an eye disease, and the second diagnosis module Second diagnostic auxiliary information related to whether the subject corresponds to a systemic disease may be obtained by using the second neural network model for predicting whether the subject corresponds to a systemic disease.

As a more specific example, the first diagnosis module obtains first diagnostic auxiliary information regarding whether the subject has diabetic retinopathy by using a first diagnosis auxiliary neural network model that predicts whether the subject has diabetic retinopathy based on the fundus image, , the second diagnosis module may obtain second diagnosis auxiliary information related to whether the subject has hypertension by using a second diagnosis auxiliary neural network model that predicts whether the subject has hypertension based on the fundus image.

4.3.2 Parallel diagnostic process

A diagnosis assistance process according to an embodiment of the present invention may include a plurality of sub-diagnosis processes. Each sub-diagnosis process may be performed using a different diagnostic auxiliary neural network model. Each sub-diagnosis process may be performed in different diagnostic modules. For example, the first diagnosis module may perform a first sub-diagnosis process of acquiring the first diagnosis assistance information through the first diagnosis assistance neural network model. Alternatively, the second diagnosis module may perform a second sub-diagnosis process of acquiring the second diagnosis assistance information through the second diagnosis assistance neural network model.

The plurality of learned neural network models may output predicted labels or probabilities by receiving diagnosis target data as input. Each neural network model may be provided in the form of a classifier, and input diagnostic target data may be classified with respect to a predetermined label. In this case, the plurality of neural network models may be provided in the form of classifiers learned for different characteristics. Each neural network model can classify diagnostic target data as described above in 3.4.2.

Meanwhile, from each diagnostic auxiliary neural network model, a CAM may be obtained, and the CAM may be selectively obtained. The CAM may be extracted when a predetermined condition is satisfied. For example, when the first diagnostic assistance information indicates that the subject is abnormal with respect to the first characteristic, the first CAM may be obtained from the first diagnostic assistance neural network model.

27 is a diagram for explaining a diagnosis assistance process according to an embodiment of the present invention.

Referring to FIG. 27 , the diagnosis assistance process according to an embodiment of the present invention acquires diagnosis target data (S2011), and diagnoses using a first diagnosis auxiliary neural network model and a second diagnosis auxiliary neural network model (S2031a, S2031b) It may include obtaining (S2051) diagnosis auxiliary information according to the target data. The diagnostic target data may be processed data.

The diagnosis assistance process according to an embodiment of the present invention includes acquiring first diagnosis auxiliary information through the learned first diagnosis auxiliary neural network model and acquiring second diagnosis auxiliary information through the learned second diagnosis auxiliary neural network model. may include The first diagnosis auxiliary neural network model and the second diagnosis auxiliary neural network model may acquire first diagnosis auxiliary information and second diagnosis auxiliary information, respectively, based on the same diagnosis target data.

For example, the first diagnostic auxiliary neural network model and the second diagnostic auxiliary neural network model may include, based on the fundus image to be diagnosed, the first diagnostic auxiliary information regarding whether the subject has macular degeneration and the second diagnosis auxiliary information regarding whether the subject corresponds to diabetic retinopathy. Each of the 2 diagnostic auxiliary information may be acquired.

In addition, unless otherwise specified, the diagnosis assistance process described with reference to FIG. 27 may be implemented similarly to the diagnosis assistance process described with reference to FIG. 20 .

4.3.3 Output of diagnostic auxiliary information

According to an embodiment of the present invention, the diagnosis auxiliary information obtained by the parallel diagnosis auxiliary process may be obtained. The obtained diagnostic assistance information may be stored in a diagnostic device, a server device, and/or a client device. The acquired diagnostic auxiliary information may be transmitted to an external device.

The plurality of diagnostic auxiliary information may indicate a plurality of labels predicted by the plurality of diagnostic auxiliary neural network models, respectively. The plurality of diagnostic auxiliary information may respectively correspond to a plurality of labels predicted by the plurality of diagnostic auxiliary neural network models. Alternatively, the diagnostic auxiliary information may be information determined based on a plurality of labels predicted by a plurality of diagnostic auxiliary neural network models. The diagnostic auxiliary information may correspond to a plurality of labels predicted by the plurality of diagnostic auxiliary neural network models.

In other words, the first diagnostic auxiliary information may be diagnostic auxiliary information corresponding to the first label predicted through the first diagnostic auxiliary neural network model. Alternatively, the first diagnosis auxiliary information may be diagnosis auxiliary information determined by considering the first label predicted through the first diagnosis auxiliary neural network model and the second label predicted through the second diagnosis auxiliary neural network model together.

Meanwhile, CAM images obtained from a plurality of diagnostic auxiliary neural network models may be output. The CAM image may be output when a predetermined condition is satisfied. For example, when the first diagnostic auxiliary information indicates that the subject is abnormal with respect to the first characteristic or the second diagnostic auxiliary information indicates that the subject is abnormal with respect to the second characteristic, A CAM image obtained from a diagnostic auxiliary neural network model to which diagnostic auxiliary information indicated to be abnormal is output may be output.

A plurality of diagnostic auxiliary information and/or CAM images may be provided to the user. A plurality of diagnostic auxiliary information and the like may be provided to a user through an output means of a diagnostic device or a client device. The diagnostic auxiliary information may be visually output. In this regard, 5. User interface will be described in more detail.

According to an embodiment of the present invention, the diagnosis auxiliary information corresponding to the diagnosis target image may include grade information. The rating information may be selected from among a plurality of ratings. The grade information may be determined based on a plurality of diagnostic information and/or observation information obtained through the neural network model. The grade information may be determined in consideration of suitability information or quality information of an image to be diagnosed. The grade information may be determined in consideration of the class in which the diagnosis target image is classified by a plurality of neural network models. The grade information may be determined in consideration of numerical values output from a plurality of neural network models.

For example, the diagnosis auxiliary information obtained in response to the diagnosis target image may include any one selected from the first grade information and the second grade information. The grade information may be selected as the first grade information when at least one abnormal finding information or abnormal diagnosis information is acquired from among diagnostic information acquired through a plurality of neural network models. The grade information may be selected as the second grade information when abnormal findings information or abnormal diagnostic information is not obtained from among the diagnostic information acquired through the neural network model.

The grade information is selected as the first grade information when at least one of the values obtained through the neural network model exceeds the reference value, and is selected as the second grade information when all of the obtained values do not meet the reference value could be The first grade information may indicate that abnormal information stronger than the second grade information exists in the diagnosis target image.

The rating information may be selected as the third rating information when it is determined that the quality of the image to be diagnosed is below a standard using image analysis or a neural network model. Alternatively, the diagnosis auxiliary information may include the third level information together with the first or second level information.

When the diagnosis auxiliary information includes the first level information, the first user guide may be output through the output means. The first user guide may include at least one piece of abnormal observation information included in the diagnosis auxiliary information or information corresponding to the abnormal diagnosis information. For example, the first user guide may indicate that a more precise examination is required for a subject (ie, a patient) corresponding to abnormal information included in the diagnosis auxiliary information. For example, the first user guidance may indicate that a secondary diagnosis (eg, diagnosis in a separate medical institution or power supply procedure) is required for the subject. Alternatively, the first user guidance may indicate a required treatment for the subject. As a specific example, when abnormal information on the macular degeneration of the subject is acquired by the diagnostic auxiliary information, the first user guide provides information on the injection prescription and power supply procedure for the subject (eg, at a hospital where power is available). list) can be included.

When the diagnosis auxiliary information includes the second level information, the second user guide may be output through the output means. The second user guide may include a post-management plan for the subject corresponding to the diagnosis auxiliary information. For example, the second user guide may instruct the subject's next treatment time, next treatment subject, and the like.

When the diagnosis target information includes the third level information, the third user guide may be output through the output means. The third user guide may indicate that recapture is required for the diagnosis target image. The third user guide may include information on the quality of the image to be diagnosed. For example, the third user guide may include information on defects present in the diagnosis target image (eg, whether bright artifacts or dark artifacts, or a degree thereof).

The first to third grade information may be output by an output unit of the client device or the diagnostic device. Specifically, it may be output through a user interface to be described later.

4.4 Example 2 - Diagnostic Aid System

A diagnosis assistance system according to an embodiment of the present invention may include a fundus image acquisition unit, a first processing unit, a second processing unit, a third processing unit, and a diagnostic information output unit.

According to an embodiment of the present invention, the diagnosis assistance system may include a diagnosis apparatus. The diagnosis apparatus may include a fundus image acquisition unit, a first processing unit, a second processing unit, a third processing unit, and/or a diagnostic information output unit. However, the present invention is not limited thereto, and each unit included in the diagnosis assistance system may be located at an appropriate location on the learning device, the diagnosis device, the learning diagnosis server, and/or the client device. Hereinafter, for convenience, a case in which the diagnosis apparatus of the diagnosis assistance system includes the fundus image acquisition unit, the first processing unit, the second processing unit, the third processing unit, and the diagnostic information output unit will be described.

28 is a diagram for explaining a diagnosis assistance system according to an embodiment of the present invention. Referring to FIG. 28 , the diagnosis assistance system may include a diagnosis apparatus, and the diagnosis apparatus may include an fundus image acquisition unit, a first processing unit, a second processing unit, a third processing unit, and a diagnostic information output unit.

According to an embodiment of the present invention, a diagnosis assistance system supporting diagnosis of a plurality of diseases based on a fundus image includes a fundus image acquisition unit configured to acquire a target fundus image that is a basis for acquiring diagnostic assistance information for a subject. , a first neural network model with respect to the target fundus image - the first neural network model is machine learned based on the first fundus image set - to obtain a first result related to the first observation on the subject The subject using a first processing unit, a second neural network model for the target fundus image, wherein the second neural network model is machine-learned based on a second fundus image set that is at least partially different from the first fundus image set A second processing unit for obtaining a second result related to a second opinion on It may include a diagnostic information output unit that provides In this case, the first findings and the second findings may be used for diagnosis of different diseases.

The first neural network model is trained to classify the input fundus image into any one of a normal label and an abnormal label in relation to the first observation, and the first processing unit converts the target fundus image to the normal label using the first neural network model. Alternatively, the first result may be obtained by classifying one of the abnormal labels.

The third processing unit may determine whether the diagnosis information according to the target fundus image is normal information or abnormal information by considering the first result and the second result together.

The third processing unit may determine diagnostic information on the subject by giving priority to the abnormal label so as to improve diagnostic accuracy.

If the first label is a normal label for the first observation and the second label is a normal label for the second observation, the third processing unit determines the diagnostic information as normal, and the first label is the first When it is not a normal label for the finding or the second label is not a normal label for the second finding, the diagnostic information may be determined to be abnormal.

The first finding may be related to an eye disease, and the first result may indicate whether the subject is normal to the eye disease. The second finding may be related to a systemic disease, and the second result may indicate whether the subject is normal to the systemic disease.

A first finding is related to a first ocular disease, the first result indicates whether the subject is normal for the first ocular disease, and the second finding is a second ocular disease distinct from the first ocular disease. related to a disease, and the second result may indicate whether the subject is normal to the second eye disease.

The first observation is an observation for diagnosing a first ocular disease, the first result indicates whether the subject is normal for the first ocular disease, and the second observation is for diagnosing the first ocular disease. It is a finding distinct from the first finding, and the second result may indicate whether the subject is normal to the second eye disease.

The first neural network model includes a first sub neural network model and a second sub neural network model, and a first result is a first predicted value predicted by the first sub neural network model and a second predicted value predicted by the second sub neural network model It may be determined by considering the predicted values together.

The first processing unit may obtain a CAM (Class Activation Map) related to the first label through the first neural network model, and the diagnostic information output unit may output an image of the CAM.

The diagnosis information output unit may output the CAM image when the diagnosis information acquired by the third processing unit is abnormal diagnosis information.

The diagnosis assistance system may further include a fourth processing unit configured to acquire quality information of the target fundus image, and the diagnostic information output unit may output quality information of the target fundus image acquired by the fourth processing unit.

When the fourth processing unit determines that the quality information of the target fundus image is equal to or less than a predetermined quality level, the diagnostic information output unit provides the user with the quality information of the target fundus image together with the determined diagnostic information to the user at the predetermined quality level. Information indicating the following may be provided together.

5. User Interface

According to an embodiment of the present invention, the above-described client device or diagnosis device may have a display unit for providing diagnosis auxiliary information to a user. In this case, the display unit may be provided to clearly transmit the diagnosis auxiliary information to the user and to easily obtain feedback from the user.

As an example of the display unit, a display providing visual information to a user may be provided. In this case, a graphic user interface for visually delivering the diagnosis auxiliary information to the user may be used. For example, in a fundus diagnosis assisting system for acquiring diagnosis assisting information based on a fundus image, a graphical user interface for effectively displaying the acquired diagnosis assisting information and helping a user to understand may be provided.

29 and 30 are diagrams for explaining a graphical user interface for providing diagnostic information to a user according to some embodiments of the present invention. Hereinafter, with reference to FIGS. 29 and 30 , a user interface that can be used in the fundus diagnosis assistance system will be described with reference to some embodiments.

Referring to FIG. 29 , the user interface according to an embodiment of the present invention may display identification information of a subject corresponding to a fundus image to be diagnosed. The user interface may include a target image identification information display unit 401 that displays identification information of a subject (ie, a patient) and/or imaging information (eg, imaging date) of a fundus image to be diagnosed.

The user interface according to an embodiment of the present invention may include a fundus image display unit 405 that displays the fundus image of the left eye and the fundus image of the right eye of the same subject. The fundus image display unit 405 may display a CAM image.

The user interface according to an embodiment of the present invention displays the image of the left eye or the right eye for each of the fundus image of the left eye and the fundus image of the right eye, and diagnostic information of each image and a diagnostic information indicator indicating whether to confirm the user It may include a diagnostic information indicating unit 403 .

The color of the diagnostic information indicator may be determined in consideration of the diagnostic auxiliary information obtained based on the target fundus image. The diagnostic information indicator may be displayed in a first color or a second color according to the diagnostic auxiliary information. For example, when the first to third diagnostic auxiliary information is acquired from one target fundus image, when even one diagnostic auxiliary information includes abnormal (ie, abnormal findings) information, the diagnostic information indicator is displayed in red, , when all of the diagnostic auxiliary information includes normal (ie, no abnormal findings) information, the diagnostic information indicator may be displayed in green.

The form of the diagnostic information indicator may be determined according to whether the user confirms it. The diagnostic information indicator may be displayed in the first form or in the second form according to whether the user confirms it. For example, referring to FIG. 25 , the diagnostic information indicator corresponding to the target fundus image reviewed by the user is displayed as a filled circle, and the diagnostic information indicator corresponding to the target fundus image not reviewed by the user is displayed as a filled semicircle. can be displayed.

The user interface according to an embodiment of the present invention may include a diagnosis information indicating unit 407 indicating auxiliary diagnosis information. The diagnostic auxiliary information indicating unit may be located in the left eye and right eye images, respectively. The diagnostic auxiliary information indicating unit may indicate a plurality of observation information or diagnostic information.

The diagnostic auxiliary information indicator may include at least one diagnostic auxiliary information indicator. The diagnostic auxiliary information indicator may indicate corresponding diagnostic auxiliary information through color change.

For example, with respect to the fundus image to be diagnosed, first diagnosis auxiliary information indicating the presence or absence of lens opacity through the first diagnosis auxiliary neural network model, and second diagnosis auxiliary information indicating the presence or absence of glaucoma findings through the second diagnosis auxiliary neural network model , when the third diagnosis auxiliary information indicating the presence or absence of retinal abnormality is obtained through the third diagnosis auxiliary neural network model, the diagnosis information indicating unit instructs the first diagnosis auxiliary information, the second diagnosis auxiliary information, and the third diagnosis auxiliary information, respectively and first to third diagnostic auxiliary information indicators.

As a more specific example, referring to FIG. 29 , in relation to the left eye fundus image, the diagnostic information indicating unit 407 provides first diagnostic auxiliary information indicating that the diagnostic auxiliary information obtained based on the left eye fundus image of the subject is lens opacity abnormal. , when the second diagnostic auxiliary information indicating normal glaucoma (no glaucoma) and the third diagnostic auxiliary information indicating retinal abnormality (with abnormal findings) are obtained, the first diagnostic auxiliary information indicator having a first color; A first diagnostic auxiliary information indicator having a second color and a third diagnostic auxiliary information indicator having a first color may be displayed.

The user interface according to an embodiment of the present invention may obtain a user comment on the fundus image to be diagnosed from the user. The user interface may include a user comment object 409 and, in response to a user selection for the user comment object, may display a user input window. The comments obtained from the user may be used to update the diagnostic auxiliary neural network model. For example, a user input window displayed in response to selection of a user comment object may acquire a user evaluation of the diagnostic assistance information through a neural network, and the obtained user evaluation may be used to update a neural network model.

The user interface according to an embodiment of the present invention may include a review instruction object 411 for indicating whether the user reviews each diagnostic target fundus image. The review instruction object may receive a user input indicating that the user review for each diagnosis target image is completed, and the display may be changed from the first state to the second state. For example, referring to FIGS. 29 and 30 , the review instruction object may be changed from a first state of displaying a confirmation request text to a second state indicating that it has been confirmed when a user input is obtained.

A list 413 of fundus images to be diagnosed may be displayed. In the list, identification information of the subject, an image capture date, and an indicator 403 for reviewing both eyes may be displayed together.

A review completion indicator 415 indicating a review target fundus image may be displayed in the list 413 of the diagnostic target fundus image. The review completion indicator 415 may be displayed when the user selects all of the review instruction objects 411 for both eyes of the corresponding image.

Referring to FIG. 30 , the user graphic interface may include a low-quality warning object 417 instructing the user that there is an abnormality in the quality of the target fundus image when it is determined that there is a quality abnormality in the fundus image to be diagnosed. have. The low-quality warning object 417 is displayed when it is determined by the diagnostic unit that the fundus image to be diagnosed does not reach a level at which appropriate diagnostic auxiliary information can be predicted from the diagnostic auxiliary neural network model (ie, the reference quality level). can be

Also, referring to FIG. 28 , a low-quality warning object 419 may be displayed in the list 413 of the fundus image to be diagnosed.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (31)

A diagnosis assistance system for assisting in diagnosis of a plurality of diseases based on a fundus image, comprising:
a fundus image acquisition unit configured to acquire a target fundus image, which is a basis for acquiring diagnostic auxiliary information on the subject;
a first for obtaining a first result related to a first observation on the subject by using a first neural network model with respect to the target fundus image, wherein the first neural network model is machine learned based on the first set of fundus images. 1 processing unit;
Using a second neural network model for the target fundus image, wherein the second neural network model is machine-learned based on a second fundus image set that is at least partially different from the first fundus image set, a second neural network model for the subject is used. a second processing unit for obtaining a second result related to the observation;
a third processing unit configured to determine auxiliary diagnostic information for the subject based on the first result and the second result; and
an information output unit for providing the determined diagnostic auxiliary information to a user; containing
The first findings and the second findings are used for diagnosis of different diseases,
Diagnostic assistance system.
According to claim 1,
wherein the third processing unit determines whether the diagnosis auxiliary information is abnormal information when at least one of the first result and the second result indicates whether the subject is abnormal;
Diagnostic assistance system.
According to claim 1,
The third processing unit includes first diagnostic auxiliary information related to the first finding of the subject obtained based on the first result and a second diagnosis related to the second finding of the subject obtained based on the second result 2 determining the diagnostic auxiliary information including the diagnostic auxiliary information;
Diagnostic assistance system.
According to claim 1,
The third processing unit may include: first diagnostic auxiliary information related to the first finding of the subject acquired based on the first result and related to the second finding of the subject acquired based on the second result determining the diagnostic auxiliary information to include at least one of the second diagnostic auxiliary information;
Diagnostic assistance system.
According to claim 1,
The first neural network model is trained to classify the input fundus image into any one of a normal label and an abnormal label in relation to the first observation,
The first processing unit obtains the first result by classifying the target fundus image as either the normal label or the abnormal label using the first neural network model.
Diagnostic assistance system.
According to claim 1,
wherein the third processing unit determines whether the diagnosis auxiliary information according to the target fundus image is normal information or abnormal information by considering the first result and the second result together;
Diagnostic assistance system.
7. The method of claim 6,
The third processing unit is configured to give priority to the abnormal information to determine the diagnosis auxiliary information for the subject so as to improve diagnosis accuracy;
Diagnostic assistance system.
7. The method of claim 6,
When the first result is normal information for the first observation and the second result is normal information for the second observation, the third processing unit determines the diagnosis auxiliary information as normal information;
When the first result is not normal information for the first observation or the second result is not normal information for the second observation, determining the diagnostic auxiliary information as abnormal information,
Diagnostic assistance system.
According to claim 1,
The first finding is related to an eye disease, and the first result indicates whether the subject is normal to the eye disease,
wherein the second finding is related to a systemic disease, and the second result indicates whether the subject is normal to the systemic disease,
Diagnostic assistance system.
According to claim 1,
The first finding is related to a first eye disease, and the first result indicates whether the subject is normal to the first eye disease,
The second finding is related to a second eye disease distinct from the first eye disease, and the second result indicates whether the subject is normal for the second eye disease.
Diagnostic assistance system.
According to claim 1,
The first neural network model includes a first sub-neural network model and a second sub-neural network model,
The first result is determined by considering a first prediction value predicted by the first sub-neural network model and a second prediction value predicted by the second sub-neural network model together,
Diagnostic assistance system.
According to claim 1,
The first processing unit obtains a CAM (Class Activation Map) related to the first result through the first neural network model,
The information output unit outputs the image of the CAM,
Diagnostic assistance system.
13. The method of claim 12,
The information output unit outputs the image of the CAM when the diagnosis auxiliary information obtained by the third processing unit is abnormal information;
Diagnostic assistance system.
According to claim 1,
The diagnosis assistance system further includes a fourth processing unit for acquiring quality information of the target fundus image,
The information output unit outputs quality information of the target fundus image obtained by the fourth processing unit,
Diagnostic assistance system.
15. The method of claim 14,
When the fourth processing unit determines that the quality information of the target fundus image is less than or equal to a predetermined quality level,
The information output unit,
Providing the user with information indicating that the quality information of the target fundus image is equal to or less than the predetermined quality level together with the determined auxiliary information for diagnosis,
Diagnostic assistance system.
Learning to acquire a first training data set including a plurality of fundus images, process the fundus image included in the first training data set, and train a first neural network model using the first training data set Device;
a diagnostic apparatus for acquiring a target fundus image for acquiring diagnostic assistance information, and acquiring the diagnostic assistance information based on the target fundus image using the learned first neural network model; In the control method of the learning device included in a system comprising a,
performing pre-processing of the first fundus image so that the first fundus image included in the first training data set is converted into a form suitable for training of the first neural network model;
serializing the preprocessed first fundus image; and
training the first neural network model for classifying the target fundus image into a first label or a second label using the serialized first fundus image; containing
How to control the learning device.
17. The method of claim 16,
The learning apparatus acquires a second training data set that includes a plurality of fundus images and is at least partially different from the first training data set, and uses the second training data set to train a second neural network model;
The control method of the learning device is
performing pre-processing of the second fundus image so that the second fundus image included in the second training set is suitable for learning of the second neural network model;
serializing the preprocessed second fundus image;
training the second neural network model for classifying the target fundus image into a third label or a fourth label by using the serialized second fundus image; further comprising
How to control the learning device.
18. The method of claim 17,
The first preprocessing performed on the fundus image included in the first training data set is distinct from the second preprocessing performed on the fundus image included in the second training data set,
How to control the learning device.
18. The method of claim 17,
verifying the first neural network model by evaluating an accuracy of the learned first neural network model using a first validation data set that is at least partially different from the first training data set; and
verifying the second neural network model by evaluating an accuracy of the learned first neural network model using a second validation data set that is at least partially different from the second training data set; further comprising,
The verification of the first neural network model and the second neural network model is performed independently,
How to control the learning device.
18. The method of claim 17,
The serialized first fundus image is sequentially stored in a first queue, and the serialized fundus image stored in the first queue is used for learning the first neural network model in a predetermined capacity unit from the first queue,
The serialized second fundus image is sequentially stored in a second queue distinct from the first queue, and the serialized fundus image stored in the second queue is the first neural network model from the second queue in a predetermined capacity unit. used for learning
How to control the learning device.
18. The method of claim 17,
The first neural network model includes a first sub-neural network model and a second sub-neural network model,
Classifying the target fundus image into the first label or the second label is performed in consideration of the first predicted value predicted by the first sub-neural network model and the second predicted value predicted by the second sub-neural network model become,
The second neural network model includes a third sub neural network model and a fourth sub neural network model,
Classifying the target fundus image into the third label or the fourth label is performed in consideration of a third predicted value predicted by the third sub-neural network model and a fourth predicted value predicted by the fourth sub-neural network model felled,
How to control the learning device.
18. The method of claim 17,
wherein the first training data set includes at least a portion of the fundus image labeled with a first label, and the second training data set includes at least a portion of the fundus image labeled with a third label,
The fundus image labeled with the first label and at least a portion of the fundus image labeled with the third label have some common,
How to control the learning device.
17. The method of claim 16,
The first label is a normal label indicating that the subject corresponding to the target fundus image is normal with respect to the first observation, and the second label is an abnormal label indicating that the subject is abnormal with respect to the first observation. ,
How to control the learning device.
17. The method of claim 16,
The pre-processing of the first fundus image comprises removing at least a portion of the first fundus image and changing the size of the first fundus image so that the first fundus image satisfies a reference aspect ratio
How to control the learning device.
17. The method of claim 16,
The pre-processing of the first fundus image further comprises applying a blood vessel enhancement filter to the fundus image so that the blood vessels included in the first fundus image are emphasized.
How to control the learning device.
17. The method of claim 16,
The serialized first fundus image is sequentially stored in a queue, and the serialized first fundus image stored in the queue is used for learning the first neural network model from the queue in a predetermined capacity unit,
The queue requests replenishment of the serialized first fundus image when the serialized first fundus image that is not used for training of the first neural network model decreases below a reference capacity,
How to control the learning device.
24. The method of claim 23,
The first finding is any one of retinal hemorrhage, retinal exudate, lens opacity, and diabetic retinopathy.
How to control the learning device.
17. The method of claim 16,
verifying the first neural network model by evaluating an accuracy of the learned first neural network model using a first validation data set that is at least partially different from the first training data set; further comprising
How to control the learning device.
29. The method of claim 28,
and updating the first neural network model by reflecting the verification result obtained by verifying the first neural network model.
How to control the learning device.
17. The method of claim 16,
The first neural network model includes a first sub-neural network model and a second sub-neural network model,
The step of training the first neural network model is
Validate the first sub-neural network model using a first validation data set to obtain accuracy of the first sub-neural network model;
Validate the second sub-neural network model using the first verification data set to obtain accuracy of the second sub-neural network model;
Comparing the accuracy of the first sub neural network model and the accuracy of the second sub neural network model, determining a more accurate sub neural network model as the final neural network model
How to control the learning device.
A computer-readable recording medium in which a program for performing any one of the methods for controlling the learning apparatus of claim 16 to 30 is recorded.

Images