KR102333670B1 - Diagnostic auxiliary image providing device based on eye image - Google Patents

Abstract

According to an embodiment, in the apparatus for providing a diagnosis auxiliary image for diagnosing a disease based on a fundus image, a neural network model for diagnosing a disease with respect to an input fundus image - the neural network model includes the Learned to diagnose disease-; a CAM model for outputting a CAM image based on at least one of the input fundus image and information obtained from the neural network model so as to easily identify a region affecting the diagnosis performance of the neural network model; and a lesion emphasizing model for emphasizing a region affecting the diagnosis performance for each characteristic of the disease based on the characteristic of the disease with respect to at least one of the input fundus image and the CAM image, wherein the lesion emphasis model A diagnosis auxiliary image providing apparatus may be provided in which an auxiliary diagnosis image is provided in which an onset location and an onset region of the disease are emphasized through the .

Description

Device for providing diagnostic auxiliary image based on eye image {DIAGNOSTIC AUXILIARY IMAGE PROVIDING DEVICE BASED ON EYE IMAGE}

The following embodiments relate to an apparatus for providing an auxiliary diagnostic image based on a fundus image.

The following embodiments relate to a system for providing an auxiliary image for diagnosis for each lesion based on a fundus image.

Fundus examination can observe abnormalities in the retina, optic nerve, and macula, and is a diagnostic aid frequently used in ophthalmology because the results can be confirmed through relatively simple imaging. Recently, the use of the fundus examination is increasing in that it is possible to observe the degree of damage to blood vessels caused by chronic diseases such as high blood pressure and diabetes in a non-invasive way as well as eye diseases.

On the other hand, due to the rapid development of deep learning technology in recent years, the development of artificial intelligence for diagnosis is being actively carried out in the medical diagnosis field, particularly in the image-based diagnosis field. Global companies such as Google and IBM are also sparingly investing in the development of artificial intelligence for various image medical data analysis, such as inputting large-scale data in collaboration with the medical community. Tool development was also successful.

However, in providing the diagnosis result based on the fundus image, it may be difficult for the user to understand the diagnosis result through the fundus image depending on the degree of background knowledge. In order to consider the degree of background knowledge of various users, there is a need to provide a diagnosis auxiliary image that can easily understand the diagnosis result obtained through the fundus image.

One task is to provide a diagnosis auxiliary image corresponding to a diagnosis result of the fundus image.

One task is to provide a diagnostic auxiliary image that is emphasized according to the degree of influence on the diagnostic result for the fundus image.

One task is to provide a diagnostic auxiliary image that is emphasized according to the degree of influence on the diagnostic result for the fundus image, but considers the type or location of the lesion.

The problem to be solved is not limited to the above-mentioned problems, and the problems not mentioned will be clearly understood by those of ordinary skill in the art to which an embodiment belongs from the present specification and the accompanying drawings.

According to an embodiment, in the apparatus for providing a diagnosis auxiliary image for diagnosing a disease based on a fundus image, a neural network model for diagnosing a disease with respect to an input fundus image - the neural network model includes the Learned to diagnose disease-; a CAM model for outputting a CAM image based on at least one of the input fundus image and information obtained from the neural network model so as to easily identify a region affecting the diagnosis performance of the neural network model; and a lesion emphasizing model for emphasizing an area affecting the diagnosis performance by characteristics of the disease based on characteristics of the disease with respect to at least one of the input fundus image and the CAM image, wherein the lesion emphasis model A diagnosis auxiliary image providing apparatus may be provided in which an auxiliary diagnosis image is provided in which an onset location and an onset region of the disease are emphasized through the .

According to one embodiment, there is provided an apparatus for providing an auxiliary diagnosis image for diagnosing a plurality of diseases based on a fundus image, comprising: a data acquisition unit configured to acquire an input fundus image; a first model providing a first diagnosis auxiliary image of a first disease with respect to the input fundus image; and a second model providing a second diagnosis auxiliary image of a second disease with respect to the input fundus image, wherein the first model and the second model convert the input fundus image to the first disease or the second disease. 2 In relation to the disease, the disease is classified as either normal or abnormal, and only when the diagnosis result of the neural network model is abnormal, the first diagnosis auxiliary image or the second diagnosis auxiliary image is output, and the first disease and the second disease is a disease in which at least a portion of an onset region is different, and the first auxiliary diagnosis image and the second auxiliary diagnosis image are respectively output based on characteristics of the first disease and the second disease, wherein the first disease and the second disease 2 An apparatus for providing an auxiliary diagnosis image may be provided, in which the disease onset region is highlighted and output.

According to an exemplary embodiment, a diagnosis result of the diagnosis auxiliary apparatus and/or system for the fundus image may be easily understood through the diagnosis auxiliary image.

According to an embodiment, by providing the diagnosis result of the diagnosis assistance device and/or system for the fundus image through the diagnosis assistance image, the user's level of background knowledge may be taken into consideration, so that the diagnosis result may be easily understood.

According to an embodiment, as the diagnosis auxiliary image is provided in consideration of the specificity of each lesion, the diagnosis result of the diagnosis auxiliary device and/or system for the fundus image may be provided specifically for the lesion, so that the user can more easily and clearly The diagnosis result can be understood and used.

Effects of an embodiment are not limited to the above-described effects, and effects not mentioned will be clearly understood by those of ordinary skill in the art to which an embodiment belongs from the present specification and the accompanying drawings.

1 illustrates a diagnosis assistance system according to an exemplary embodiment.
2 is a block diagram illustrating a learning apparatus according to an exemplary embodiment.
3 is a block diagram illustrating in more detail a learning apparatus according to another exemplary embodiment.
4 is a block diagram illustrating a diagnosis apparatus according to an exemplary embodiment.
5 is a diagram for describing a diagnosis apparatus according to another exemplary embodiment.
6 illustrates a diagnosis assistance system according to an exemplary embodiment.
7 is a block diagram illustrating a client device according to an embodiment.
8 is a diagram for describing a diagnosis assistance process according to an exemplary embodiment.
9 is a diagram for explaining a configuration of a learning unit according to an embodiment.
10 is a conceptual diagram illustrating an image data set according to an exemplary embodiment.
11 is a diagram for describing image resizing according to an exemplary embodiment.
12 is a diagram for explaining an extension of an image data set according to an exemplary embodiment.
13 is a block diagram illustrating a learning process of a neural network model according to an embodiment.
14 is a block diagram illustrating a learning process of a neural network model according to an embodiment.
15 is a diagram for describing a method of controlling a learning apparatus according to an exemplary embodiment.
16 is a diagram for describing a method of controlling a learning apparatus according to an exemplary embodiment.
17 is a diagram for describing a method of controlling a learning apparatus according to an exemplary embodiment.
18 is a diagram for describing a configuration of a diagnosis unit according to an exemplary embodiment.
19 is a diagram for describing diagnosis target data according to an example.
20 is a diagram for describing a diagnosis process according to an exemplary embodiment.
21 is a diagram for describing a parallel diagnosis assistance system according to some embodiments.
22 is a diagram for describing a parallel diagnosis assistance system according to some embodiments.
23 is a diagram for describing a configuration of a learning apparatus including a plurality of learning units according to an exemplary embodiment.
24 is a diagram for describing a parallel learning process according to an embodiment.
25 is a diagram for describing a parallel learning process according to an embodiment.
26 is a block diagram illustrating a diagnosis unit according to an exemplary embodiment.
27 is a diagram for describing a diagnosis assistance process according to an exemplary embodiment.
28 is a diagram for describing a diagnosis assistance system according to an exemplary embodiment.
29 is a diagram for describing a graphic user interface according to an embodiment.
30 is a diagram for describing a graphic user interface according to an exemplary embodiment.
31 is a diagram for describing an auxiliary diagnosis image according to an exemplary embodiment.
32 is a diagram for explaining a configuration of a diagnosis auxiliary information providing unit according to an exemplary embodiment.
33 is a diagram for describing a diagnostic process and a CAM providing process performed by a diagnostic apparatus according to an exemplary embodiment.
34 is a diagram for explaining a diagnosis process by a learned diagnosis auxiliary neural network model.
35 is a diagram for describing a process of determining whether classification data of a diagnostic apparatus corresponds to an abnormality, according to an exemplary embodiment.
36 is a diagram for describing a process of determining whether classification data of a diagnostic apparatus corresponds to normal, according to an exemplary embodiment.
Fig. 37 is a diagram for explaining a process of outputting a CAM image from input data and information obtained from a diagnosis process.
38 is a view for explaining a CAM image providing process according to a diagnosis process in consideration of the degree of progression of a diagnosis disease.
39 is a view for explaining division of a region of a fundus image according to another embodiment of the present invention.
40 is a diagram illustrating a fundus image in which a plurality of diseases diagnosed by the diagnosis assistance system according to an exemplary embodiment;
41 is a diagram for explaining the configuration of a diagnosis auxiliary information providing unit according to an exemplary embodiment.
42 is a diagram for describing a diagnostic process and a diagnostic auxiliary image providing process performed by a diagnostic apparatus according to an exemplary embodiment.
43 to 46 are diagrams illustrating auxiliary diagnostic images when the diagnostic disease of the diagnostic process is glaucoma.
47 to 50 are diagrams illustrating auxiliary diagnostic images when a diagnostic disease of the diagnostic process occurs in a non-specific region.
51 and 52 are diagrams illustrating auxiliary images for diagnosis when the input data of the diagnosis process has a cataract.
53 is a flowchart illustrating a method of providing a diagnostic auxiliary image.
54 is a diagram illustrating a configuration of a parallel diagnosis auxiliary information providing unit for a fundus image according to an exemplary embodiment.
FIG. 55 is a diagram for explaining a diagnosis process performed by the diagnosis auxiliary information providing unit of FIG. 54 and a diagnosis auxiliary image providing process.
56 is a diagram illustrating a configuration of a parallel diagnosis auxiliary information providing unit for a fundus image according to an exemplary embodiment.
FIG. 57 is a diagram for explaining a diagnosis process performed by the diagnosis auxiliary information providing unit of FIG. 56 and a diagnosis auxiliary image providing process.

Hereinafter, a specific embodiment of an embodiment will be described in detail with reference to the drawings. However, the spirit of one embodiment is not limited to the presented embodiment, and those skilled in the art who understand the spirit of one embodiment may add, change, delete, etc. other components within the scope of the same idea, and may use other degenerative inventions or work. Other embodiments included within the scope of the spirit of the embodiment may be easily proposed, but this will also be included within the scope of the spirit of the present invention.

In addition, components having the same function within the scope of the same idea shown in the drawings of each embodiment will be described using the same reference numerals.

According to an embodiment, in the apparatus for providing a diagnosis auxiliary image for diagnosing a disease based on a fundus image, a neural network model for diagnosing a disease with respect to an input fundus image - the neural network model includes the Learned to diagnose disease-; a CAM model for outputting a CAM image based on at least one of the input fundus image and information obtained from the neural network model so as to easily identify a region affecting the diagnosis performance of the neural network model; and a lesion emphasizing model for emphasizing an area affecting the diagnosis performance by characteristics of the disease based on characteristics of the disease with respect to at least one of the input fundus image and the CAM image, wherein the lesion emphasis model A diagnosis auxiliary image providing apparatus may be provided in which an auxiliary diagnosis image is provided in which an onset location and an onset region of the disease are emphasized through the .

In addition, the lesion enhancement model may provide an apparatus for providing a diagnostic auxiliary image in which lesion enhancement is performed in which pixels of a predetermined threshold value or more are selectively output among pixels of the CAM image based on the pixel value of the CAM image. have.

In addition, when the disease occurs specifically in a location or region corresponding to an anatomical region included in the fundus image, the lesion emphasis model is a lesion in which a boundary line corresponding to the anatomical region overlaps on the CAM image. An apparatus for providing a diagnostic auxiliary image in which emphasis is performed may be provided.

In addition, when the disease occurs non-specifically with respect to an anatomical region included in the fundus image, the lesion emphasis model is based on the CAM image. An apparatus for providing a diagnostic auxiliary image in which lesion highlighting is performed may be provided.

In addition, when the diseased lesion region covers at least a portion of anatomical regions included in the fundus image, the lesion highlighting model includes a boundary line corresponding to the anatomical region covered by the lesion region on the fundus image. An apparatus for providing a diagnostic auxiliary image in which overlapping lesion highlighting is performed may be provided.

In addition, the neural network model classifies the input fundus image as either normal or abnormal in relation to the diagnosis disease, and the diagnosis auxiliary image providing device provides a diagnosis auxiliary image only when the diagnosis result of the neural network model is abnormal. An outputted, diagnostic auxiliary image providing apparatus may be provided.

In addition, the disease may be any one of glaucoma, retinal hemorrhage, retinal exudate, lens opacity, and diabetic retinopathy, a diagnostic auxiliary image providing apparatus may be provided.

In addition, when the disease is divided into first and second stages according to the onset period and the onset degree and diagnosed, the neural network model includes a first sub-neural network model and a second sub-neural network model, and the first sub-neural network model classifies one of normal and abnormal in relation to the first stage of the disease, and the second sub-neural network model classifies it as either normal or abnormal in relation to the first stage, wherein the diagnostic auxiliary image providing apparatus includes the first stage A diagnostic auxiliary image providing apparatus may be provided, in which a diagnostic auxiliary image is output only when a diagnosis result of at least one of the sub neural network model and the second sub neural network model is abnormal.

In addition, when a diagnosis result of at least one of the first sub-neural network model and the second sub-neural network model is abnormal, in order to provide a diagnosis auxiliary image in consideration of the degree of disease progression, the diagnosis auxiliary image is the first Output based on first diagnostic information of a model determined to be normal among the sub-neural network model and the second sub-neural network model and second diagnostic information of a model determined to be abnormal among the first sub-neural network model and the second sub-neural network model, A diagnostic auxiliary image providing device may be provided.

According to one embodiment, there is provided an apparatus for providing an auxiliary diagnosis image for diagnosing a plurality of diseases based on a fundus image, comprising: a data acquisition unit configured to acquire an input fundus image; a first model providing a first diagnosis auxiliary image of a first disease with respect to the input fundus image; and a second model providing a second diagnosis auxiliary image of a second disease with respect to the input fundus image, wherein the first model and the second model convert the input fundus image to the first disease or the second disease. 2 In relation to the disease, the disease is classified as either normal or abnormal, and only when the diagnosis result of the neural network model is abnormal, the first diagnosis auxiliary image or the second diagnosis auxiliary image is output, and the first disease and the second disease is a disease in which at least a portion of an onset region is different, and the first auxiliary diagnosis image and the second auxiliary diagnosis image are respectively output based on characteristics of the first disease and the second disease, wherein the first disease and the second disease 2 An apparatus for providing an auxiliary diagnosis image may be provided, in which the disease onset region is highlighted and output.

In addition, the first model includes a first neural network model, a first CAM model, and a first lesion enhancement model, and the second model includes a second neural network model, a second CAM model, and a second lesion enhancement model. A diagnostic auxiliary image providing apparatus may be provided.

In addition, the first neural network model is trained to diagnose a first disease based on a plurality of fundus image sets, and the second neural network model is trained to diagnose a second disease based on a plurality of fundus image sets, and the second neural network model is trained to diagnose a second disease based on a plurality of fundus image sets. 1 Based on the diagnostic information obtained from the neural network model, the first CAM model outputs a first CAM image, and the first lesion emphasis model with respect to the first CAM image performs lesion emphasis in consideration of the characteristics of the first diagnostic disease. performed, and based on the diagnostic information obtained from the second neural network model, the second CAM model outputs a second CAM image, and the second lesion emphasis model determines the characteristics of the second diagnostic disease with respect to the second CAM image. An apparatus for providing a diagnostic auxiliary image in which lesion highlighting is performed in consideration of this may be provided.

In addition, the first diagnostic auxiliary image is an image in which a lesion enhancement is performed in which a first area above a predetermined threshold is selectively output based on the pixel value of the first CAM image based on the first CAM image. , the second diagnostic auxiliary image is an image in which lesion enhancement is performed in which a second area above a predetermined threshold is selectively output based on the pixel value of the second CAM image based on the second CAM image, An apparatus for providing an auxiliary diagnosis image may be provided, wherein the first area and the second area are at least partially different.

In addition, the first diagnosis auxiliary image is an image on which a lesion enhancement is performed in which a first boundary line formed with a predetermined threshold value among pixel values of the first CAM image is overlapped based on the first CAM image, and the second diagnosis The auxiliary image is an image on which lesion enhancement is performed in which a second boundary line formed with a predetermined threshold value among pixel values of the second CAM image is overlapped based on the second CAM image, and the first boundary line and the second boundary line are An apparatus for providing a diagnostic auxiliary image, which is at least partially different, may be provided.

In addition, when the onset region of the first disease corresponds to at least any one region among a plurality of anatomical regions included in the fundus image, and the disease onset region is non-specifically located in the fundus image, the first The auxiliary diagnosis image is an image in which lesion enhancement is performed in which a boundary line corresponding to the anatomical region overlaps the first CAM image, and the second diagnosis auxiliary image is the second CAM image based on the second CAM image. An apparatus for providing an auxiliary diagnosis image may be provided, which is an image on which lesion enhancement is performed in which a boundary line formed by a predetermined threshold value among pixel values of is overlapped.

In addition, the first disease onset region corresponds to the first anatomical region included in the fundus image, and the second disease onset region is in a second anatomical region that is at least partially different from the first disease onset region. In a corresponding case, the first auxiliary diagnosis image is an image on which lesion enhancement is performed in which a boundary line corresponding to the first anatomical region overlaps the first CAM image, and the second auxiliary diagnosis image is the second CAM image. An apparatus for providing an auxiliary diagnosis image may be provided, which is an image on which lesion enhancement is performed in which a boundary line corresponding to the second anatomical region overlaps with respect to the image.

In addition, when the first diseased region of the first disease is nonspecifically located in the fundus image, and the second diseased region is nonspecifically located in the fundus image, but is at least partially different from the first diseased region, the The first auxiliary diagnosis image is an image on which lesion enhancement is performed in which a boundary line formed by a predetermined threshold among pixel values of the first CAM image is overlapped based on the first CAM image, and the second auxiliary diagnosis image is An apparatus for providing an auxiliary diagnosis image may be provided, which is an image on which lesion enhancement is performed in which a boundary line formed with a predetermined threshold value among pixel values of the second CAM image is overlapped based on the 2 CAM image.

In addition, a haze region is generated in the fundus image by the first disease, and the diseased region of the second disease corresponds to an anatomical region included in the fundus image, wherein the anatomical region is in the turbid region of the first disease In the case of being obscured by , the second diagnosis auxiliary image may be an image on which lesion emphasis is performed in which the boundary line of the anatomical region overlaps in the input fundus image, the diagnosis auxiliary image providing apparatus may be provided.

In addition, a haze region is generated in the fundus image due to the first disease, the first diagnostic auxiliary image is output identically to the input fundus image, and the diseased region of the second disease is non-specifically located in the fundus image , when the diseased region is obscured by the turbid region of the first disease, the second diagnosis secondary image is an image on which the lesion emphasis is performed in which the boundary line of the diseased region overlaps in the input fundus image. A providing device may be provided.

In addition, the first model includes a first neural network model and a first CAM model, the second model includes a second neural network model and a second CAM model, and the diagnostic auxiliary image providing apparatus further includes a lesion highlighting model. A diagnostic auxiliary image providing apparatus including, may be provided.

In addition, the first neural network model is trained to diagnose a first disease based on a plurality of fundus image sets, and the second neural network model is trained to diagnose a second disease based on a plurality of fundus image sets, and the second neural network model is trained to diagnose a second disease based on a plurality of fundus image sets. 1 Based on the diagnostic information obtained from the neural network model, the first CAM model outputs a first CAM image, and the second CAM model outputs a second CAM image based on the diagnostic information obtained from the second neural network model, A diagnostic auxiliary image providing apparatus may be provided in which lesion enhancement is performed on the first CAM image and the second CAM image in consideration of characteristics of the first disease and the second disease by the lesion enhancement model, respectively. .

In addition, the first disease is any one of glaucoma, retinal hemorrhage, retinal exudate generation, lens opacity, and diabetic retinopathy, and the second disease is glaucoma, retinal hemorrhage, retinal exudate generation, lens opacity, and diabetic retinopathy. An apparatus for providing an auxiliary diagnosis image, which is any one except for 1 disease, may be provided.

Hereinafter, a diagnosis assistance apparatus and/or system for providing a diagnosis assistance result and a diagnosis assistance image using a fundus image according to an exemplary embodiment will be described.

1. Diagnosis assistance using fundus images

1.1 Diagnostic Auxiliary Systems and Processes

1.1.1 Purpose and Definition

Hereinafter, a diagnosis assistance system and method for assisting in the determination of the presence or absence of a disease or disease or an abnormality as a basis for the determination based on the fundus image will be described. In particular, a diagnosis assistance system or method for constructing a neural network model for diagnosing a disease using a deep learning technique and assisting in detecting the presence or absence of a disease or abnormal findings using the constructed model will be described.

According to an embodiment, a diagnosis assistance system or method may be provided for acquiring diagnostic information related to the presence or absence of a disease or observation information used for diagnosing the presence or absence of a disease based on a fundus image and assisting the diagnosis by using the acquired information. .

According to an embodiment, a diagnosis assistance system or method for assisting in diagnosing an eye disease based on a fundus image may be provided. For example, a diagnostic assistance system or method may be provided to assist a diagnosis by acquiring diagnostic information related to the presence or absence of glaucoma, cataract, macular degeneration, and retinopathy of prematurity of a test subject.

According to an embodiment, a diagnostic assistance system or method for assisting in the diagnosis of other diseases (eg, systemic diseases or chronic diseases) other than eye diseases may be provided. For example, a diagnostic assistance system or method may be provided to assist in diagnosis by acquiring diagnostic information of systemic diseases such as hypertension, diabetes, Alzheimer's, cytomegalovirus, stroke, heart disease, arteriosclerosis, and the like.

According to an embodiment, a diagnostic auxiliary system or method for detecting abnormal fundus findings that can be used for diagnosing an eye disease or other disease may be provided. For example, overall fundus color abnormality, lens opacity, optic disc ratio (C/D ratio: cup to disc ratio) abnormality, macular abnormality (eg, macular foramen), blood vessel diameter, travel A diagnostic assistance system or method for acquiring information on findings such as abnormality of the back, abnormal diameter of retinal artery, retinal hemorrhage, occurrence of exudate, drusen, and the like may be provided.

In the present specification, the diagnosis auxiliary information may be understood to include diagnosis information according to the determination of the presence or absence of a disease, or observation information that is the basis thereof.

1.1.2 Diagnostic Auxiliary System Configuration

According to an embodiment, a diagnosis assistance system may be provided.

1 illustrates a diagnosis assistance system 10 according to an exemplary embodiment. Referring to FIG. 1 , the diagnosis assistance system 10 includes a learning device 1000 for training a diagnosis model, a diagnosis device 2000 for performing diagnosis using the diagnosis model, and a client device 3000 for obtaining a diagnosis request. may include The diagnosis assistance system 10 may include a plurality of learning devices, a plurality of diagnostic devices, or a plurality of client devices.

The learning apparatus 1000 may include the learning unit 100 . The learning unit 100 may train the neural network model. As an example, the learner 100 may acquire a fundus image data set and train a neural network model for detecting a disease or abnormality from the fundus image.

The diagnosis apparatus 2000 may include a diagnosis unit 200 . The diagnosis unit 200 may diagnose a disease or acquire auxiliary information used for diagnosis by using the neural network model. For example, the diagnosis unit 200 may obtain the diagnosis auxiliary information by using a diagnosis model trained by the learning unit 100 .

The client device 3000 may include an imaging unit 300 . The imaging unit 300 may capture a fundus image. The client device may be an ophthalmic fundus imaging device. Alternatively, the client device 3000 may be a handheld device such as a smart phone or a tablet PC.

In the diagnosis assistance system 10 according to the present embodiment, the learning apparatus 1000 determines a neural network model to be used for diagnosis assistance by acquiring a data set and learning the neural network model, and the diagnosis apparatus 2000 is a client When the information request is obtained from can be obtained '

A diagnosis assistance system according to another exemplary embodiment may include a diagnosis apparatus and a client apparatus for learning a diagnosis model and performing diagnosis using the same. A diagnosis assistance system according to another exemplary embodiment may include a diagnosis apparatus that learns a diagnosis model, obtains a diagnosis request, and performs diagnosis. A diagnosis assistance system according to another embodiment may include a learning apparatus for learning a diagnostic model, and a diagnostic apparatus for obtaining a diagnosis request and performing a diagnosis.

The diagnosis assistance system disclosed herein is not limited to the above-described embodiments, and a learning unit for learning a model, a diagnosis unit for acquiring diagnosis assistance information according to the learned model, and an imaging unit for acquiring an image to be diagnosed It may be implemented in any form including.

Hereinafter, some embodiments of each device constituting the system will be described.

1.1.2.1 Learning device

The learning apparatus according to an embodiment may train a neural network model that assists in diagnosis.

2 is a block diagram illustrating the learning apparatus 1000 according to an exemplary embodiment. Referring to FIG. 2 , the learning apparatus 1000 may include a control unit 1200 and a memory unit 1100 .

The learning apparatus 1000 may include a controller 1200 . The controller 1200 may control the operation of the learning apparatus 1000 .

The control unit 1200 includes at least one of a central processing unit (CPU), a random access memory (RAM), a graphic processing unit (GPU), one or more microprocessors, and other electronic components capable of processing input data according to a predetermined logic. may include

The controller 1200 may read a system program and various processing programs stored in the memory unit 1100 . For example, the controller 1200 may develop a data processing process, a diagnostic process, etc. for performing diagnosis assistance, which will be described later, on the RAM, and may perform various processes according to the developed program. The controller 1200 may perform learning of a neural network model, which will be described later.

The learning apparatus 1000 may include a memory unit 1100 . The memory unit 1100 may store data required for learning and a learning model.

The memory unit 1100 may include a nonvolatile semiconductor memory, a hard disk, a flash memory, a RAM, a read only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), or other tangible nonvolatile recording media, etc. can be implemented as

The memory unit 1100 may store various processing programs, parameters for performing processing of the programs, or data as a result of such processing. For example, the memory unit 1100 may include a data processing process program, a diagnostic process program, parameters for execution of each program, and data (eg, processed data or diagnostics) obtained according to the execution of the program to be described later for performing the diagnosis assistance. result) and so on.

The learning apparatus 1000 may include a separate learning unit (or learning module). The learning unit may perform learning of the neural network model. In relation to the performance of learning, it will be described in more detail in Table of Contents 2. Learning process below.

The learning unit may be included in the above-described control unit 1200 . The learning unit may be stored in the aforementioned memory unit 1100 . The learning unit may be implemented by some configurations of the control unit 1200 and the memory unit 1100 described above. For example, the learning unit may be stored in the memory unit 1100 and driven by the control unit 1200 .

The learning apparatus 1000 may further include a communication unit 1300 . The communication unit 1300 may communicate with an external device. For example, the communication unit 1300 may communicate with a diagnostic device, a server device, or a client device, which will be described later. The communication unit 1300 may perform wired or wireless communication. The communication unit 1300 may perform bi-directional or unidirectional communication.

3 is a block diagram illustrating the learning apparatus 1000 according to an exemplary embodiment in more detail. Referring to FIG. 3 , the learning apparatus 1000 may include a processor 1050 , a volatile memory 1010 , a nonvolatile memory 1030 , a mass storage device 1070 , and a communication interface 1090 .

The processor 1050 of the learning apparatus 1000 may include a data processing module 1051 and a learning module 1053 . The processor 1050 may process a data set stored in a mass storage device or a non-volatile memory through the data processing module 1051 . The processor 1050 may train the diagnostic auxiliary neural network model through the learning module 1053 . The processor 1050 may include local memory. The communication interface 1090 may be connected to the network 1110 .

However, the learning apparatus 1000 illustrated in FIG. 3 is merely an example, and the configuration of the learning apparatus 1000 according to an exemplary embodiment is not limited thereto. In particular, the data processing module 1051 or the learning module 1053 may be provided at a location different from that shown in FIG. 3 .

1.1.2.2 Diagnostic device

The diagnostic apparatus may obtain diagnostic auxiliary information by using the neural network model.

4 is a block diagram illustrating a diagnosis apparatus 2000 according to an exemplary embodiment. Referring to FIG. 4 , the diagnosis apparatus 2000 may include a control unit 2200 and a memory unit 2100 .

The controller 2200 may generate the diagnosis auxiliary information by using the diagnosis auxiliary neural network model. The controller 2200 may acquire diagnostic data for diagnosis (eg, fundus data of a subject) and acquire auxiliary diagnostic information predicted by the diagnostic data using the learned diagnostic auxiliary neural network model.

The memory unit 2100 may store the learned diagnostic auxiliary neural network model. The memory unit 2100 may store parameters and variables of the auxiliary diagnostic neural network model.

The diagnosis apparatus 2000 may further include a communication unit 2300 . The communication unit 2300 may communicate with the learning device and/or the client device. For example, the diagnosis apparatus 2000 may be provided in the form of a server communicating with a client device. In this regard, it will be described in more detail below.

5 is a diagram for describing a diagnosis apparatus 2000 according to an exemplary embodiment. Referring to FIG. 5 , the diagnostic apparatus 2000 according to an embodiment includes a processor 2050 , a volatile memory 2030 , a non-volatile memory 2010 , a mass storage device 2070 , and a communication interface 2090 . can do.

The processor 2050 of the diagnostic apparatus may include a data processing module 2051 and a diagnostic module 2053 . The processor 2050 may process diagnostic data through the data processing module 2051 and obtain diagnostic auxiliary information according to the diagnostic data through the diagnostic module 2053 .

1.1.2.3 Server device

According to an embodiment, the diagnosis assistance system may include a server device. A diagnosis assistance system according to an embodiment may include a plurality of server devices.

The server device may store and/or run the neural network model. The server device may store weight values constituting the learned neural network model. The server device may collect or store data used for diagnosis assistance.

The server device may output the result of the diagnostic auxiliary process using the neural network model to the client device. The server device may obtain feedback from the client device. The server device may operate similarly to the diagnostic device described above.

6 illustrates a diagnosis assistance system 20 according to an exemplary embodiment. Referring to FIG. 6 , the diagnosis assistance system 20 according to an embodiment may include a diagnosis server 4000 , a learning device, and a client device.

The diagnosis server 4000 , that is, a server device may communicate with a plurality of learning devices or a plurality of diagnostic devices. Referring to FIG. 6 , the diagnosis server 4000 may communicate with the first learning apparatus 1000a and the second learning apparatus 1000b. Referring to FIG. 6 , the diagnosis server 4000 may communicate with a first client device 3000a and a second client device 3000b .

For example, the diagnosis server 4000 may be configured to train the first learning apparatus 1000a for learning the first auxiliary diagnosis neural network model to obtain the first diagnosis auxiliary information and the second diagnosis auxiliary neural network model to obtain the second diagnosis auxiliary information. It may communicate with the second learning apparatus 1000b.

The diagnostic server 4000 stores a first diagnostic auxiliary neural network model for obtaining the first diagnostic auxiliary information and a second diagnostic auxiliary neural network model for obtaining the second diagnostic auxiliary information, the first client device 3000a or the second client Diagnosis auxiliary information may be obtained in response to a request for obtaining zidan auxiliary information from the device 3000b , and the obtained diagnosis auxiliary information may be transmitted to the first client device 3000a or the second client device 3000b .

Alternatively, the diagnosis server 4000 may communicate with the first client device 3000a requesting the first diagnosis assistance information and the second client device 3000b requesting the second diagnosis assistance information.

1.1.2.4 Client Device

The client device may request the diagnostic assistance information from the diagnostic device or the server device. The client device may acquire data required for diagnosis and transmit the acquired data to the diagnosis device.

The client device may include a data acquisition unit. The data acquisition unit may acquire data necessary for diagnosis assistance. The data acquisition unit may be an imaging unit that acquires an image used for the diagnosis assistance model.

7 is a block diagram illustrating a client device 3000 according to an exemplary embodiment. Referring to FIG. 7 , the client device 3000 according to an embodiment may include an imaging unit 3100 , a control unit 3200 , and a communication unit 3300 .

The imaging unit 3100 may acquire an image or image data. The imaging unit 3100 may acquire a fundus image. However, the client device 3000 may be replaced with a data acquisition unit other than the imaging unit 3100 .

The communication unit 3300 may communicate with an external device, for example, a diagnostic device or a server device. The communication unit 3300 may perform wired or wireless communication.

The controller 3200 may control the imaging unit 3100 to acquire an image or data. The controller 3200 may control the imaging unit 3100 to acquire a fundus image. The controller 3200 may transmit the acquired fundus image to the diagnosis apparatus. The controller may transmit the image acquired through the imaging unit 3100 to the server device through the communication unit 3300, and may acquire the generated diagnostic auxiliary information based thereon.

Although not shown, the client device may further include an output unit. The output unit may include a display that outputs an image or image, or a speaker that outputs an audio. The output unit may output an image or image data acquired by the acquired imaging unit. The output unit may output the diagnosis auxiliary information obtained from the diagnosis apparatus.

Although not shown, the client device may further include an input unit. The input unit may obtain a user input. For example, the input unit may obtain a user input for requesting diagnosis auxiliary information. The input unit may obtain user information for evaluating the diagnosis auxiliary information obtained from the diagnosis apparatus.

Also, although not shown, the client device may further include a memory unit. The memory unit may store an image acquired by the imaging unit.

1.1.3 Diagnostic Auxiliary Process Overview

A diagnostic assistance process may be performed by a diagnostic assistance system or a diagnostic assistance apparatus disclosed herein. The diagnosis assistance process may be divided into a training process for learning a diagnosis assistance model used for diagnosis assistance and a diagnosis process using the diagnosis assistance model.

8 is a diagram for describing a diagnosis assistance process according to an exemplary embodiment. Referring to FIG. 8 , the diagnosis assistance process according to an embodiment includes a learning process of acquiring and processing data (S110) to learn a neural network model (S130), and acquiring variables of the learned neural network model (S150); It may include a diagnosis auxiliary process of acquiring the diagnosis target data (S210) and acquiring the diagnosis auxiliary information using the neural network model (S230) learned based on the diagnosis target data (S250).

More specifically, the training process may include a data processing process of processing the input training image data to a state that can be used for training of the model, and a learning process of training the model using the processed data. The training process may be performed by the above-described learning device.

The diagnosis process may include a data processing process of processing input test target image data to a state capable of performing diagnosis using a neural network model, and a diagnosis process of performing diagnosis using the processed data. The diagnostic process may be performed by the aforementioned diagnostic device or the server device.

Below, each process is demonstrated.

1.2 Training process

According to an embodiment, a process for training a neural network model may be provided. As a specific example, a process of training a neural network model for performing or assisting a diagnosis based on a fundus image may be disclosed.

The training process to be described below may be performed by the above-described learning apparatus.

1.2.1 Study Department

According to an embodiment, the training process may be performed by the learner. The learning unit may be provided in the above-described learning apparatus.

9 is a diagram for explaining the configuration of the learning unit 100 according to an embodiment. Referring to FIG. 9 , the learning unit 100 may include a data processing module 110 , a queue module 130 , a learning module 150 , and a learning result module 170 . Each module may perform separate steps of the data processing process and the learning process as described below. However, not all of the components and functions performed by the components described in FIG. 9 are essential, and some elements may be added or some elements may be omitted depending on the learning form.

1.2.2 Data Processing Process

1.2.2.1 Image data acquisition

According to an embodiment, a data set may be obtained. According to an embodiment, the data processing module may acquire a data set.

The data set may be an image data set. Specifically, it may be a fundus image data set. The fundus image data set may be acquired using a general annovulatory fundus camera or the like. The fundus image may be a panoramic image. The fundus image may be a red-free image. The fundus image may be an infrared photographed image. The fundus image may be an autofluorescence photographed image. The image data may be acquired in any one of JPG, PNG, DCM (DICOM), EMP, GIF, and TIFF format.

The data set may include a training data set. The data set may include a test data set. The data set may include a validation data set. In other words, the data set may be assigned as at least one of a training data set, a test data set, and a validation data set.

The data set may be determined in consideration of auxiliary diagnostic information to be obtained using a neural network model learned through the data set. For example, when training a neural network model for acquiring cataract-related diagnostic assistance information, the acquired data set may be determined as an infrared fundus image data set. Alternatively, when training a neural network model for acquiring diagnostic auxiliary information related to macular degeneration, the acquired data set may be an autofluorescence-captured fundus image data set.

Individual data included in a data set may include a label. There may be a plurality of labels. In other words, individual data included in the data set may be labeled with respect to at least one characteristic. For example, the data set is a fundus image data set including a plurality of fundus image data, and each fundus image data is a diagnostic information label (eg, presence or absence of a specific disease) and/or observation information (eg, specific Whether there is an abnormality in the site) label may be included.

As another example, the data set is a fundus image data set, and each fundus image data may include a peripheral information label for the corresponding image. For example, each fundus image data includes left and right eye information on whether the corresponding fundus image is an image of the left eye or the image of the right eye, gender information on whether the fundus image is a female fundus image or a male fundus image, and a subject who has taken the corresponding fundus image It may include a peripheral information label including age information on the age of the .

10 is a conceptual diagram illustrating an image data set DS according to an exemplary embodiment. Referring to FIG. 10 , an image data set DS according to an embodiment may include a plurality of image data IDs. Each image data ID may include an image I and a label L assigned to the image. Referring to FIG. 10 , in the image data set DS, the first image data ID1 may include a first image I1 and a first label L1 corresponding to the first image.

In FIG. 10 , the description is based on a case in which one image data includes one label, but as described above, one image data may include a plurality of labels.

1.2.2.2 Image resizing

According to an embodiment, the size of the acquired image data may be adjusted. That is, images may be resized. According to an embodiment, the image resizing may be performed by the data processing module of the above-described learning unit.

The size or aspect ratio of the image can be adjusted. The size of the plurality of acquired images may be adjusted to have a constant size. Alternatively, the images may be resized to have a constant aspect ratio. Resizing the image may be applying an image transformation filter to the image.

When the size or capacity of the obtained individual images is excessively large or small, the size or capacity of the image may be adjusted to convert the image to an appropriate size. Alternatively, when the sizes or capacities of individual images vary, the sizes or capacities may be unified through resizing.

According to an embodiment, the capacity of the image may be adjusted. For example, when the capacity of the image exceeds an appropriate range, the image may be reduced through down sampling. Alternatively, when the capacity of the image is less than an appropriate range, the image may be enlarged through upsampling or interpolating.

According to another embodiment, the size or aspect ratio of the image may be adjusted by cropping the image or adding pixels to the acquired image. For example, when an image includes a part unnecessary for learning, a part of the image may be cropped to remove it. Alternatively, if a part of the image is cropped and the aspect ratio does not match, a column or row may be added to adjust the image aspect ratio. In other words, you can adjust the aspect ratio by adding margin or padding to the image.

According to another embodiment, the capacity and size or aspect ratio of the image may be adjusted together. For example, when the size of the image is large, the image capacity may be reduced by downsampling the image, and unnecessary parts included in the reduced image may be cropped to convert the image data into appropriate image data.

Also, according to an embodiment, the orientation of the image data may be changed.

As a specific example, when a fundus image data set is used as the data set, the capacity or size of each fundus image may be adjusted. Cropping for removing a blank portion excluding the fundus portion of the fundus image or padding for adjusting the aspect ratio by supplementing the cropped portion of the fundus image may be performed.

11 is a diagram for describing image resizing according to an exemplary embodiment. Referring to FIG. 11 , the acquired fundus image may be resized by the image resizing process according to an embodiment.

Specifically, in the original fundus image (a), a blank portion unnecessary for acquiring diagnostic information may be cropped (b), and its size may be reduced (c) to improve learning efficiency.

1.2.2.3 Image preprocessing

According to an embodiment, image pre-processing may be performed. When an image is used as input for learning, overfitting may occur as a result of learning about unnecessary characteristics, and learning efficiency may also decrease.

In order to prevent this, learning efficiency and performance can be improved by appropriately pre-processing and using image data to meet the purpose of learning. For example, a preprocessing for easy detection of abnormal signs of an eye disease or a preprocessing for emphasizing changes in retinal blood vessels or blood flow may be performed on the fundus image.

Pre-processing of the image may be performed by the data processing module of the above-described learning unit. The data processing module may acquire a resized image and perform pre-processing requested for learning.

The image pre-processing may be performed on the image on which the above-described resizing process has been completed. However, the content of the invention disclosed in the present specification is not limited thereto, and pre-processing of the image may be performed while omitting the resizing process. Applying pre-processing of the image may be applying a pre-processing filter to the image.

According to an embodiment, a blur filter may be applied to an image. A Gaussian filter may be applied to the image. A Gaussian blur filter may be applied to the image. Alternatively, a deblur filter for sharpening the image may be applied to the image.

According to another embodiment, a filter for adjusting or modulating the color of the image may be applied. For example, a filter for changing the values of some components among RGB values constituting the image or for binarizing the image may be applied.

According to another embodiment, a filter for emphasizing a specific element in an image may be applied. For example, pre-processing may be performed on fundus image data so that vascular elements are emphasized from each image. In this case, the pre-processing for emphasizing the vascular element may be applying one or more filters sequentially in combination.

According to an embodiment, the image pre-processing may be performed in consideration of the characteristics of the diagnostic auxiliary information to be acquired. For example, when it is desired to acquire diagnostic auxiliary information related to findings such as retinal hemorrhage, drusen, microvascular flow, and exudate, preprocessing of converting the acquired fundus image into a red-free fundus image form may be performed.

1.2.2.4 Image augmentation

According to an embodiment, the image may be augmented or expanded. Augmentation of the image may be performed by the data processing module of the above-described learning unit.

The augmented images may be used to improve training performance of the neural network model. For example, when the amount of data for training the neural network model is insufficient, by performing modulation of the existing training image data to expand the number of data for training, and using the modulated (or changed) image together with the original image The number of training image data can be increased. Accordingly, overfitting can be suppressed, a model layer can be formed more deeply, and prediction accuracy can be improved.

For example, the image data expansion may be performed by inverting the left and right sides of the image, cutting out a part of the image, correcting a color value of the image, or adding artificial noise. As a specific example, cutting out a part of the image may be performed by cutting out some areas of elements constituting the image or randomly cutting out some areas. As more examples, image data may be expanded by inverting left and right, inverting up/down, rotating, resizing, cropping, padding, adjusting color, or adjusting brightness of image data.

As an example, the above-described augmentation or expansion of image data may be generally applied to a training data set. However, it may be applied to other data sets, for example, a test data set, that is, a data set for testing a model that has been trained using training data and verified using verification data.

As a specific example, when a fundus image data set is used as a data set, by randomly applying one or more processing of inverting the image, cropping, adding noise, or changing the color, in order to increase the number of data, An augmented fundus image data set may be acquired.

12 is a diagram for describing an extension of an image data set according to an exemplary embodiment. Referring to FIG. 12 , an image according to embodiments may be modified to improve prediction accuracy of a neural network model.

Specifically, referring to FIG. 12 , in an image according to an embodiment, a partial region is dropped out (a), left and right inverted (b), a center point is moved (c, d), or a color of a partial region is changed It can be modulated (e).

1.2.2.5 Image serialization

According to an embodiment, the image data may be linearized. The image may be serialized by the data processing module of the above-described learning unit. The serialization module can serialize the preprocessed image data and pass it to the queue module.

If the image data is used for learning as it is, decoding is required because the image data has the form of an image file such as JPG, PNB, DCM. . Accordingly, it is possible to perform learning by serializing the image file without using it for learning as it is. Accordingly, image data can be serialized to improve learning performance and speed. Accordingly, image data can be serialized to improve learning performance and speed. The image data to be serialized may be image data to which one or more of the above-described image resizing and image pre-processing has been applied, or image data to which both are not processed.

Each image data included in the image data set may be converted into a string form. The image data may be converted into a binary data form. In particular, image data may be converted into a data format suitable for training of a neural network model. As an example, image data may be converted into a TFRecord form for use in training a neural network model using tensorflow.

As a specific example, when a fundus image set is used as a data set, the acquired fundus image set may be converted into a TFReocrd form and used for training a neural network model.

1.2.2.6 Queue

A queue may be used to solve a data bottleneck. The image data of the queue module of the above-described learning unit may be stored in a queue and transmitted to the learning model module.

In particular, when the learning process is performed using both the CPU (Central Processing Unit) and the GPU (Graphic Processing Unit), by using the queue, the bottleneck between the CPU and GPU is minimized, access to the database is smooth, and the memory Use efficiency can be improved.

The queue may store data used for training the neural network model. A queue can store image data. The image data stored in the queue may be image data processed by at least one of the aforementioned data processing processes (ie, resizing, pre-processing, and augmentation) or an image as it is acquired.

The queue may store image data, preferably serialized image data as described above. The queue stores image data and can supply image data to the neural network model. The queue can deliver image data in units of batch size to the neural network model.

A queue may provide image data. The queue may provide data to a learning module, which will be described later. As data is extracted from the learning module, the number of data accumulated in the queue may be reduced.

If the number of data stored in the queue decreases below the standard as the neural network model is trained, the queue may request data replenishment. A queue may request replenishment of certain types of data. The queue may supplement data to the queue when the learning unit requests replenishment of data.

The queue may be provided in the system memory of the learning device. For example, the queue may be formed in a random access memory (RAM) of a central processing unit (CPU). In this case, the size of the queue, that is, the capacity may be determined according to the RAM capacity of the CPU. As the queue, a First In First Out (FIFO) queue, a primary queue, or a random queue may be used.

1.2.3 Learning process

According to an embodiment, a learning process of a neural network model may be started.

According to an embodiment, learning of the neural network model may be performed by the above-described learning apparatus. The learning process may be performed by a control unit of the learning apparatus. The learning process may be performed by the learning module of the learning unit described above.

13 is a block diagram illustrating a learning process of a neural network model according to an embodiment. Referring to FIG. 13 , the learning process of a neural network model according to an embodiment acquires data (S1010), trains the neural network model (S1030), validates the learned model (S1050), and determines the parameters of the learned model Acquisition (S1070) may be performed.

Hereinafter, with reference to FIG. 13 , some embodiments will be described for a learning process of a neural network model.

1.2.3.1 Data entry

A data set for training of a diagnostic auxiliary neural network model may be obtained.

The data obtained may be an image data set processed by the data processing process described above. As an example, the data set may include serialized fundus image data after it has been resized, a pre-processing filter has been applied, and the data has been augmented.

In the learning phase of the neural network model, a training data set may be acquired and used. In the verification phase of the neural network model, a verification data set may be obtained and used. In the testing phase of the neural network model, a test data set may be acquired and used. Each data set may include fundus images and labels.

A data set may be obtained from a queue. The data set may be obtained from the queue in units of batch sizes. For example, when 60 is designated as the batch size, the data set may be extracted from the queue in units of 60. The size of the batch size may be limited by the RAM capacity of the GPU.

The data set may be obtained randomly from the queue to the learning module. Data sets may be acquired in the order in which they are accumulated in the queue.

The learning module may extract by specifying the configuration of the data set obtained from the queue. As an example, the learning module may extract fundus image data having a left eye label and fundus image data having a right eye label of a specific subject to be used for learning together.

The learning module may obtain a data set of a specific label from the queue. As an example, the learning module may acquire the fundus image data set in which the diagnostic information label is abnormal from the queue. The learning module may acquire the data set by specifying the ratio of the number of data according to the label from the queue. As an example, the learning module may acquire the fundus image data set from the queue so that the number of fundus image data having an abnormal diagnostic information label and the number of fundus image data having a normal diagnostic information label are 1:1.

1.2.3.2 Model design

The neural network model may be a diagnostic assistance model that outputs diagnostic assistance information based on image data. A structure of a diagnostic auxiliary neural network model for obtaining diagnostic auxiliary information may have a predetermined shape. A neural network model may include a plurality of layers or layers.

The neural network model may be implemented in the form of a classifier that generates diagnostic auxiliary information. The classifier can perform double classification or multiple classification. For example, the neural network model may be a binary classification model that classifies input data into a normal or abnormal class with respect to target diagnosis auxiliary information such as a specific disease or abnormal symptom. Alternatively, the neural network model may be a multi-classification model that classifies input data into a plurality of class classes for a specific characteristic (eg, a degree of disease progression). Alternatively, the neural network model may be implemented as a regression type model that outputs a specific value related to a specific disease.

The neural network model may include a convolutional neural network (CNN). As the CNN structure, at least one of Alexnet, LENET, NIN, VGGNet, ResNEt, WideResnet, GoogleNet, FractaNet, DenseNet, FitNet, RitResNet, HighwayNet, MobileNet, and DeeplySuperviseNet may be used. The neural network model may be implemented using a plurality of CNN structures.

As an example, the neural network model may be implemented to include a plurality of VGGNet blocks. As a more specific example, the neural network model consists of a first structure in which a CNN layer having 64 filters of 3x3 size, a batch normalization (BN) layer, and a ReLu layer are sequentially combined, and a CNN layer and ReLu layer having 128 filters of 3x3 size. and a second block in which BN layers are sequentially combined may be combined.

The neural network model includes a max pooling layer following each CNN block, and a Global Average Pooling (GAP) layer, a Fully Connected (FC) layer, and an activation layer (eg, sigmoid, soft max, etc.) at the end. can

1.2.3.3 Training the model

A neural network model can be trained using a training data set.

A neural network model can be trained using labeled data sets. However, the learning process of the diagnostic auxiliary neural network model described in this specification is not limited thereto, and the neural network model may be trained in an unsupervised form using unlabeled data.

Learning of a neural network model is based on training image data, obtains a result value using a neural network model to which arbitrary weight values are given, compares the obtained result value with a label value of training data, and back propagates according to the error by optimizing the weight values. In addition, learning of the neural network model may be influenced by a verification result of a model, a test result, and/or a feedback from a diagnosis step, which will be described later.

Learning of the above-described neural network model may be performed using TensorFlow. However, one embodiment is not limited thereto, and frameworks such as Theano, Keras, Caffe, Torch, and CNTK (Microsoft Cognitive Toolkit) are used to train the neural network model. it might be

1.2.3.4 Model validation

The neural network model can be validated using a validation data set. The validation of the neural network model may be performed by obtaining a result value for the validation data set from the neural network model in which training has been performed, and comparing the result value with the label of the validation data set. Verification may be performed by measuring the accuracy of the result value. According to the verification result, parameters (eg, weight and/or bias) or hyperparameters (eg, learning rate) of the neural network model may be adjusted.

For example, the learning apparatus according to an embodiment trains a neural network model for predicting diagnostic auxiliary information based on a fundus image, and verifies the diagnostic auxiliary information on the verified fundus image of the learned model. A verification label corresponding to the fundus image By comparing with , validation of the diagnostic auxiliary neural network model can be performed.

For the validation of the neural network model, a separate external data set, that is, a data set having distinct factors not included in the training data set, may be used. For example, the separate validation set may be a data set in which factors such as race, environment, age, and gender are distinguished from a training data set.

1.2.3.5 Testing the model

The neural network model can be tested using a test data set.

Although not shown in FIG. 13 , according to the learning process according to an embodiment, the neural network model may be tested using a test data set that is distinguished from a training data set and a verification data set. Depending on the test results, parameters (eg, weights and/or biases) or hyperparameters (eg, learning rate) of the neural network model may be adjusted.

For example, the learning apparatus according to an embodiment acquires, from a neural network model trained to predict diagnostic auxiliary information based on a fundus image, a result value of which test fundus image data that is not used for training and verification is input, and learns and testing of the validated diagnostic auxiliary neural network model.

For testing the neural network model, a separately prepared external data set, that is, a data set having factors distinct from training and/or verification data, may be used.

1.2.3.6 Output of results

As a result of learning the neural network model, parameter values of the optimized model may be obtained. As described above, more appropriate parameter (or variable) values can be obtained by iteratively training the model using the test data set. When the learning proceeds sufficiently, an optimized value of a weight and/or a bias may be obtained.

According to an embodiment, the learned neural network model and/or parameters or variables of the learned neural network model may be stored in a learning device and/or a diagnosis device (or a server). The learned neural network model may be used for prediction of diagnostic auxiliary information by a diagnostic device and/or a client device. In addition, parameters or variables of the learned neural network model may be updated by feedback obtained from a diagnostic device or a client device.

1.2.3.7 Model ensemble

According to an embodiment, in the process of learning one diagnostic auxiliary neural network model, a plurality of sub-models may be simultaneously learned. The plurality of sub-models may have different hierarchical structures.

In this case, the auxiliary diagnostic neural network model according to an embodiment may be implemented by combining a plurality of sub-neural network models. In other words, learning of the neural network model may be performed using an ensemble technique that combines a plurality of sub-neural networks.

When an ensemble is formed to configure a diagnostic auxiliary neural network model, prediction can be performed by synthesizing the results predicted from various types of sub-neural network models, so that the accuracy of result prediction can be further improved.

14 is a block diagram illustrating a learning process of a neural network model according to an embodiment. Referring to FIG. 14 , in the learning process of a neural network model according to an embodiment, a data set is acquired ( S1011 ), and a first model (ie, a first neural network model) and a second model (ie, a first neural network model) and a second model (ie, a first neural network model) using the acquired data , the second neural network model) is trained (S1031, S1033), the learned first neural network model and the second neural network model are verified (S1051), the final neural network model is determined, and its parameters or variables can be obtained (S1072). have.

Hereinafter, with reference to FIG. 14 , a learning process of a neural network model will be described with reference to some embodiments.

According to an embodiment, the plurality of sub-neural network models may acquire the same training data set and individually generate output values. In this case, an ensemble of a plurality of sub neural network models may be determined as a final neural network model, and parameter values for each of the plurality of sub neural network models may be obtained as a learning result. The output value of the final neural network model may be determined as an average value of output values by each sub-neural network model. Alternatively, the output value of the final neural network model may be determined as a weighted average value of the output values of each sub-neural network model.

As a more specific example, when the neural network model includes the first sub neural network model and the second sub neural network model, the parameter values optimized for the first sub neural network model and the optimized parameter values of the second sub neural network model are obtained by machine learning. can be obtained In this case, an average value of output values (eg, probability values for specific diagnostic auxiliary information) respectively obtained from the first sub-neural network model and the second sub-neural network model may be determined as the output value of the final neural network model.

According to another embodiment of the present embodiment, the accuracy of each sub-neural network model may be evaluated based on output values from each of the plurality of sub-neural network models. In this case, one of a plurality of sub-neural network models may be selected based on the accuracy and determined as the final sub-neural network model. The structure of the determined sub-neural network model and parameter values of the determined sub-neural network model obtained as a result of learning may be stored.

As a more specific example, when the neural network model includes the first sub-neural network model and the second sub-neural network model, accuracy according to each of the first sub-neural network model and the second sub-neural network model is obtained, and the more accurate sub-neural network model is converted into a final neural network can be determined by the model.

According to another embodiment of the present embodiment, at least one sub-neural network among a plurality of neural network models is combined, an ensemble of the combined at least one sub-neural network model is formed, and each ensemble is evaluated, but the plurality of ensembles A combination of sub-neural network models that form an ensemble with high accuracy can be determined as the final neural network model. In this case, the ensemble may be performed on all possible cases of selecting at least one of the plurality of sub neural network models, and the sub neural network combination evaluated as having the highest accuracy may be determined as the final neural network model.

As a more specific example, when the neural network model includes the first sub-neural network model and the second sub-neural network model, the accuracy of the first sub-neural network model, the accuracy of the second sub-neural network model, and the ensemble of the first and second sub-neural network models By comparing the accuracies by , it is possible to determine the sub-neural network model configuration in the most accurate case as the final neural network model.

1.2.4 Example 1 - Control method of learning device

15 is a diagram for describing a method of controlling a learning apparatus according to an exemplary embodiment.

15 , the control method of the learning apparatus according to an embodiment includes performing pre-processing of a first fundus image ( S110 ), serializing the pre-processed first fundus image ( S130 ), and a first neural network model. It may include the step of learning (S150).

The control method of the learning apparatus according to an embodiment obtains a first training data set including a plurality of fundus images, processes the fundus image included in the first training data set, and uses the first training data set a learning apparatus for learning the first neural network model, acquires a target fundus image for acquiring diagnosis auxiliary information, and uses the learned first neural network model to acquire the diagnostic auxiliary information based on the target fundus image It may be a control method of a learning device included in a system including a diagnostic device for

In the step of performing pre-processing of the first fundus image ( S110 ), pre-processing of the first fundus image is performed so that the first fundus image included in the first training data set is converted into a form suitable for learning the first neural network model. may further include

The method of controlling the learning apparatus according to an embodiment may include serializing the preprocessed first fundus image (S130). The first fundus image may be serialized in a form that is easy for training of the neural network model.

In this case, the step of training the first neural network model (S150) further includes training the first neural network model for classifying the target fundus image into a first label or a second label using the serialized first fundus image. can do.

The learning apparatus may acquire a second training data set that includes a plurality of fundus images and is at least partially different from the first training data set, and trains a second neural network model by using the second training data set.

According to an embodiment, the method of controlling the learning apparatus includes preprocessing the second fundus image so that the second fundus image included in the second training set is suitable for learning the second neural network model, the preprocessed second fundus image The method may further include training a second neural network model for classifying the target fundus image into a third label or a fourth label by using the serializing step and the serialized second fundus image.

16 is a diagram for describing a method of controlling a learning apparatus according to an exemplary embodiment. Referring to FIG. 16 , the method for controlling the learning apparatus according to an embodiment includes performing pre-processing of a second fundus image ( S210 ), serializing the pre-processed second fundus image ( S230 ), and a second neural network model. It may include a learning step (S250).

In FIG. 16, for convenience of explanation, preprocessing of the first fundus image, serialization of the first fundus image, and learning using the first fundus image, followed by preprocessing of the first fundus image, serialization of the second fundus image, and the first fundus image Although it has been described as being performed following learning using

The preprocessing of the second fundus image included in the second training set, and the serialization of the second fundus image and the learning using the second fundus image are the above-described preprocessing of the first fundus image, serialization of the first fundus image, and the first fundus. It can be performed independently of learning using images. The preprocessing of the second fundus image included in the second training set, and the serialization of the second fundus image and the learning using the second fundus image are the above-described preprocessing of the first fundus image, serialization of the first fundus image, and the first fundus. It can be performed in parallel with learning using images. In other words, the pre-processing of the second fundus image included in the second training set, and the serialization of the second fundus image and learning using the second fundus image must necessarily include the above-described preprocessing of the first fundus image and serialization of the first fundus image. And it does not have to be done after or before learning using the first fundus image, and the processing of the first fundus image and the processing of the second fundus image may be performed without mutual dependence.

The first pre-processing performed on the fundus image included in the first training data set may be distinguished from the second pre-processing performed on the fundus image included in the second training data set. For example, the first preprocessing may be a blood vessel enhancement preprocessing, and the second preprocessing may be a color modulation preprocessing. Each preprocessing may be determined in consideration of the diagnostic auxiliary information to be acquired through each neural network model.

The method of controlling a learning apparatus according to an embodiment includes evaluating the accuracy of the learned first neural network model using a first verification data set that is at least partially different from a first training data set, and the first Validating the neural network model and using a second validation data set that is at least partially distinct from the second training data set, evaluating the accuracy of the learned first neural network model, and validating the second neural network model It may include further steps. In this case, the verification of the first neural network model and the second neural network model may be independently performed.

The serialized first fundus image is sequentially stored in a first queue, and the serialized fundus image stored in the first queue is used for training of the first neural network model in a predetermined capacity unit from the first queue, A second fundus image is sequentially stored in a second queue distinct from the first queue, and the serialized fundus image stored in the second queue is used in training of the first neural network model from the second queue in a predetermined capacity unit. can be used

The first neural network model may include a first sub neural network model and a second sub neural network model. In this case, classifying the target fundus image into the first label or the second label includes the first predicted value predicted by the first sub-neural network model and the second predicted value predicted by the second sub-neural network model together. It can be done taking into account.

The second neural network model may include a third sub neural network model and a fourth sub neural network model. In this case, classifying the target fundus image into the third label or the fourth label includes a third predicted value predicted by the third sub-neural network model and a fourth predicted value predicted by the fourth sub-neural network model. It can be done taking into account.

The first training data set may include at least part of the fundus image labeled with the first label, and the second training data set may include at least part of the fundus image labeled with the third label. In this case, the fundus image labeled with the first label and at least a portion of the fundus image labeled with the third label may have some common features.

The first label may be normal indicating that the subject corresponding to the target fundus image is normal with respect to the first finding, and the second label may be abnormal indicating that the subject corresponding to the target fundus image is abnormal with respect to the second finding. .

The pre-processing of the first fundus image may include cropping the first fundus image to satisfy a reference aspect ratio and changing the size of the first fundus image.

The pre-processing of the first fundus image may further include applying, by the processing unit, a blood vessel enhancement filter to the fundus image so that the blood vessels included in the first fundus image are emphasized.

The serialized first fundus image may be sequentially stored in a queue, and the serialized first fundus image stored in the queue may be used to train the first neural network model in a unit of a predetermined capacity from the queue. The queue may request replenishment of the serialized first fundus image when the serialized first fundus image not used for the first learning decreases below a reference capacity.

The first finding may be any one of retinal hemorrhage, retinal exudate, lens opacity, and diabetic retinopathy.

17 is a diagram for describing a method of controlling a learning apparatus according to an exemplary embodiment.

Referring to FIG. 17 , the method of controlling a learning apparatus according to an embodiment may further include verifying the first neural network model ( S170 ) and updating the first neural network model ( S190 ).

The step of verifying the first neural network model ( S170 ) is to evaluate the accuracy of the learned first neural network model by using a first validation data set that is at least partially different from the first training data set to obtain one neural network model. It may further include verifying.

The method may further include updating the first neural network model by reflecting the verification result obtained from the step of updating the first neural network model ( S190 ) and the step of verifying the first neural network model ( S170 ).

Meanwhile, the first neural network model may include a first sub neural network model and a second sub neural network model. In this case, the step of training the first neural network model includes verifying the first sub-neural network model using a first validation data set to obtain accuracy of the first sub-neural network model, and using the first validation data set. Acquire the accuracy of the second sub-neural network model by verifying the second sub-neural network model, and compare the accuracy of the first sub-neural network model with the accuracy of the second sub-neural network model to convert the more accurate sub-neural network model into the final neural network model This may include deciding

1.3 Diagnostic Auxiliary Process

According to an embodiment, a diagnosis assistance process (or diagnosis process) for acquiring diagnosis assistance information using a neural network model may be provided. As a specific example, the diagnosis assistance information (eg, diagnosis information or observation information) may be predicted by using the fundus image and through the learned diagnosis assistance neural network model by the diagnosis assistance process.

The diagnostic assistance process described below may be performed by the diagnostic apparatus.

1.3.1 Diagnostics

According to an embodiment, the diagnosis process may be performed by the diagnosis unit 200 . The diagnosis unit 200 may be provided in the above-described diagnosis apparatus.

18 is a diagram for explaining the configuration of the diagnosis unit 200 according to an exemplary embodiment. Referring to FIG. 18 , the diagnosis unit 200 may include a diagnosis request obtaining module 210 , a data processing module 230 , a diagnosis module 250 , and an output module 270 .

Each module may perform separate steps of the data processing process and the learning process as described below. However, not all of the components and functions performed by the components described in FIG. 16 are essential, and some elements may be added or some elements may be omitted depending on the aspect of diagnosis.

1.3.2 Data Acquisition and Diagnosis Request

The diagnostic apparatus according to an embodiment may acquire diagnostic target data and, based on the acquired diagnostic target data, acquire diagnostic auxiliary information. The diagnosis target data may be image data. Data acquisition and diagnosis request acquisition may be performed by the diagnostic request acquisition module of the diagnostic unit.

19 is a diagram for describing diagnosis target data TD according to an example. Referring to FIG. 19 , the diagnosis target data TD may include a diagnosis target image T1 and patient information (PI).

The diagnosis target image TI may be an image for acquiring diagnosis auxiliary information on the diagnosis target object. For example, the diagnosis target image may be a fundus image. The diagnosis target image TI may have any one format among JPG, PNG, DCM (DICOM), BMP, GIF, and TIFF.

The diagnosis object information PI may be information for identifying a diagnosis target object. Alternatively, the diagnosis object information PI may be characteristic information of a diagnosis target object or image. For example, the diagnosis object information PI may include information such as an imaging date and time of a diagnosis target image, imaging equipment, an identification number, ID, name, gender, age, or weight of a subject to be diagnosed. When the diagnosis target image is the fundus image, the diagnosis object information PI may further include eye-related information such as binocular information indicating whether the left eye is the right eye.

The diagnostic device may obtain a diagnostic request. The diagnosis apparatus may acquire diagnosis target data together with the diagnosis request. When a diagnosis request is obtained, the diagnosis apparatus may acquire auxiliary diagnosis information by using the learned diagnosis auxiliary neural network model. The diagnostic device may obtain a diagnostic request from the client device. Alternatively, the diagnosis apparatus may obtain a diagnosis request from the user through a separately provided input means.

1.3.3 Data Processing Process

The obtained data may be processed. The data processing may be performed by the data processing module of the above-described diagnostic unit.

The data processing process may be generally performed similarly to the data processing process in the learning process described above. Hereinafter, the data processing process in the diagnostic process will be described with a focus on the difference from the data processing process in the learning process.

In the diagnostic process, the diagnostic device may acquire data as in the learning process. In this case, the obtained data may be in the same format as the data obtained in the learning process. For example, in the learning process, when the learning apparatus trains a diagnostic auxiliary neural network model using DCM format image data, the diagnostic apparatus acquires a DCM image and obtains diagnostic auxiliary information using the learned neural network model. .

In the diagnosis process, the acquired diagnosis target image may be resized similarly to image data used in the learning process. The shape of the diagnosis target image may be adjusted to have an appropriate capacity, size, and/or aspect ratio in order to efficiently perform diagnosis assistance information prediction through the learned diagnosis assistance neural network model.

For example, when the diagnosis target image is a fundus image, in order to predict diagnosis information based on the fundus image, resizing such as cropping an unnecessary part of the image or reducing its size may be performed.

In the diagnosis process, a preprocessing filter may be applied to the acquired diagnosis target image, similarly to image data used in the learning process. An appropriate filter may be applied to the diagnosis target image so that the accuracy of prediction of the diagnosis auxiliary information through the learned diagnosis auxiliary neural network model is improved.

For example, when the diagnosis target image is a fundus image, preprocessing for facilitating prediction of positive diagnostic information, such as image preprocessing for emphasizing blood vessels or image preprocessing for emphasizing or weakening a specific color, may be applied to the diagnosis target image. have.

In the diagnosis process, the acquired diagnosis target image may be serialized, similar to image data used in the learning process. The image to be diagnosed can be converted or serialized into a form that makes it easy to drive a diagnostic model in a specific framework.

Serialization of the diagnosis target image may be omitted. This may be because, unlike in the learning stage, in the diagnosis stage, the number of data processed by the processor at one time is not large, and the burden on the data processing speed is relatively small.

In the diagnosis process, the acquired diagnosis target image may be stored in a queue, similar to image data used in the learning process. However, since the number of processed data in the diagnosis process is smaller than that in the learning process, the step of storing data in the queue may be omitted.

Meanwhile, since an increase in the number of data is not required in the diagnosis process, it is preferable not to use data augmentation or image augmentation procedures, unlike the learning process, in order to obtain accurate diagnosis auxiliary information.

1.3.4 Diagnostic process

According to an embodiment, a diagnosis process using a learned neural network model may be initiated. The diagnostic process may be performed in the aforementioned diagnostic apparatus. The diagnostic process may be performed in the aforementioned diagnostic server. The diagnosis process may be performed by the control unit of the above-described diagnosis apparatus. The diagnosis process may be performed by the diagnosis module of the above-described diagnosis unit.

20 is a diagram for describing a diagnosis process according to an exemplary embodiment. Referring to FIG. 20 , the diagnosis process may be performed by acquiring the diagnosis target data ( S2010 ), using the learned neural network model ( S2030 ), and obtaining a result corresponding to the acquired diagnosis target data ( S2050 ). However, data processing may be selectively performed.

Hereinafter, each step of the diagnosis process will be described with reference to FIG. 20 .

1.3.4.1 Data entry

According to an embodiment, the diagnosis module may acquire diagnosis target data. The acquired data may be data processed as described above. As an example, the acquired data may be fundus image data of a subject to which a size is adjusted and pre-processing for emphasizing blood vessels is applied. According to an embodiment, the left eye image and the right eye image of one subject may be input together as diagnosis target data.

1.3.4.2 Data classification

The diagnostic auxiliary neural network model prepared in the form of a classifier may classify an input diagnostic target image into a positive or negative class with respect to a predetermined label.

The trained diagnostic auxiliary neural network model may receive diagnostic target data and output predicted labels. The trained diagnostic auxiliary neural network model may output a predicted value of the diagnostic auxiliary information. Diagnosis auxiliary information may be obtained by using the learned diagnostic auxiliary neural network model. The diagnostic assistance information may be determined based on the predicted label.

For example, the diagnostic auxiliary neural network model may predict diagnostic information (ie, information on the presence or absence of a disease) or observation information (ie, information on the presence or absence of abnormal findings) on an eye disease or systemic disease of the subject. In this case, diagnostic information or observation information may be output in the form of probability. For example, a probability that the subject has a specific disease or a probability that the subject has a specific abnormality in the fundus image of the subject may be output. In the case of using a diagnostic auxiliary neural network model prepared in the form of a classifier, a predicted label may be determined in consideration of whether an output probability value (or prediction score) exceeds a threshold value.

As a specific example, the diagnostic auxiliary neural network model may output the presence or absence of diabetic retinopathy of the subject as a probability value by using the fundus photograph of the subject as a diagnosis target image. In the case of using a classifier type diagnostic auxiliary neural network model in which 1 is normal, the fundus photograph of the subject is input to the diagnostic auxiliary neural network model, and the probability value of normal: abnormal is 0.74:0.26 for whether the subject has diabetic retinopathy or not. It can be obtained in the form of

Herein, the case where data is classified using a classifier-type diagnostic auxiliary neural network model has been described, but one embodiment is not limited thereto, and a specific diagnostic auxiliary numerical value ( For example, blood pressure, etc.) may be predicted.

According to another exemplary embodiment, suitability information of an image may be obtained. The suitability information may indicate whether the diagnosis target image is suitable for acquiring the diagnosis assistance information using the diagnosis assistance neural network model.

The suitability information of the image may be quality information of the image. The quality information or the suitability information may indicate whether the diagnosis target image reaches the reference level.

For example, when the diagnosis target image has a defect due to a defect in imaging equipment or the influence of lighting during imaging, a non-conformity result may be output as suitability information for the diagnosis target image. When noise is included in the diagnosis target image at a certain level or more, the diagnosis target image may be determined to be inappropriate.

The fitness information may be a value predicted using a neural network model. Alternatively, the suitability information may be information obtained through a separate image analysis process.

According to an embodiment, even when the image is classified as inappropriate, the diagnosis auxiliary information obtained based on the inappropriate image may be obtained.

According to an embodiment, the image classified as inappropriate may be reviewed by a diagnostic auxiliary neural network model.

In this case, the diagnostic auxiliary neural network model for performing the review may be different from the diagnostic auxiliary neural network model for performing the initial review. For example, the diagnostic apparatus may store the first diagnostic auxiliary neural network model and the second diagnostic auxiliary neural network model, and images classified as inappropriate through the first diagnostic auxiliary neural network model may be reviewed through the second diagnostic auxiliary neural network model. .

According to another embodiment of an embodiment, a CAM (Class Activation Map) may be obtained from a learned neural network model. The diagnostic auxiliary information may include a CAM. The CAM may be acquired along with other diagnostic assistance information.

In the case of CAM, it can be acquired selectively. For example, in the case of CAM, when diagnostic information or observation information acquired by the diagnostic assistance model is classified as an abnormal class, the CAM may be extracted and/or output.

1.3.5 Output of diagnostic auxiliary information

The diagnostic assistance information may be determined based on a label predicted from the diagnostic assistance neural network model.

The output of the diagnostic auxiliary information may be performed by the above-described output module of the diagnostic unit. The diagnostic auxiliary information may be output from the diagnostic device to the client device. The diagnostic auxiliary information may be output from the diagnostic device to the server device. The diagnostic auxiliary information may be stored in a diagnostic device or a diagnostic server. The diagnosis auxiliary information may be stored in a separately provided server device or the like.

Diagnostic auxiliary information may be managed in a database. For example, the acquired diagnosis auxiliary information may be stored and managed together with a diagnosis target image of the subject according to the identification number of the subject. In this case, the diagnosis target image of the subject and the diagnosis auxiliary information may be managed according to a time sequence. By managing the diagnosis auxiliary information and the diagnosis target image in time series, it is possible to easily track and manage individual diagnosis information.

Diagnostic auxiliary information may be provided to the user. The diagnostic auxiliary information may be provided to the user through an output means of the diagnostic device or the client device. The diagnosis auxiliary information may be output so that the user can recognize it through a visual or auditory output means provided in the diagnosis apparatus or the client apparatus.

According to an embodiment, an interface for effectively providing diagnostic assistance information to a user may be provided. This user interface will be described in more detail in 5. User Interface to be described later.

When the CAM is acquired by the neural network model, an image of the CAM may be provided together. In the case of a CAM image, it may be optionally provided. For example, when the diagnostic information obtained through the auxiliary diagnostic neural network model is normal observation information or normal diagnostic information, a CAM image is not provided, and when the acquired diagnostic information is abnormal observation information or abnormal diagnostic information, for more accurate clinical diagnosis A CAM image may be provided.

When the image is classified as inappropriate, suitability information of the image may be provided together. As an example, when an image is classified as inappropriate, the diagnosis auxiliary information and non-conformity determination information obtained according to the image may be provided together.

The diagnosis target image determined to be inappropriate may be classified as a re-photography target image. In this case, a rephotographing guide for a target object of an image classified as a rephotographing target may be provided together with suitability information.

Meanwhile, in response to providing the diagnostic assistance information obtained through the neural network model, feedback related to learning of the neural network model may be obtained. For example, feedback for adjusting parameters or hyperparameters related to training of the neural network model may be obtained. The feedback may be obtained through a user input unit provided in the diagnosis device or the client device.

According to an embodiment, the diagnosis auxiliary information corresponding to the diagnosis target image may include grade information. The rating information may be selected from among a plurality of ratings. The grade information may be determined based on diagnosis information and/or observation information obtained through the neural network model. The grade information may be determined in consideration of suitability information or quality information of an image to be diagnosed. When the neural network model is a classifier model that performs multiple classification, grade information may be determined in consideration of a class into which a diagnosis target image is classified by the neural network model. When the neural network model is a regression model that outputs a numerical value related to a specific disease, grade information may be determined in consideration of the output numerical value.

For example, the diagnosis auxiliary information obtained in response to the diagnosis target image may include any one selected from the first grade information and the second grade information. The grade information may be selected as the first grade information when abnormal observation information or abnormal diagnosis information is acquired through the neural network model. The grade information may be selected as second grade information when abnormal findings information or abnormal diagnosis information is not obtained through the neural network model. Alternatively, the grade information may be selected as the first grade information when the numerical value obtained through the neural network model exceeds the reference value, and may be selected as the second grade information when the obtained value does not meet the reference value. The first grade information may indicate that abnormal information stronger than the second grade information exists in the diagnosis target image.

Meanwhile, the grade information may be selected as the third grade information when it is determined that the quality of the image to be diagnosed is less than or equal to the standard using image analysis or a neural network model. Alternatively, the diagnosis auxiliary information may include the third level information together with the first or second level information.

When the diagnosis auxiliary information includes the first level information, the first user guide may be output through the output means. The first user guide may indicate that a more precise examination is required for the subject (ie, the patient) corresponding to the diagnosis auxiliary information. For example, the first user guidance may indicate that a secondary diagnosis (eg, diagnosis at a separate medical institution or power supply procedure) is required for the subject. Alternatively, the first user guidance may indicate a required treatment for the subject. As a specific example, when abnormal information on the macular degeneration of the subject is acquired by the diagnostic auxiliary information, the first user guide provides information on the injection prescription and power supply procedure for the subject (eg, at a hospital where power is available). list) can be included.

When the diagnosis auxiliary information includes the second level information, the second user guide may be output through the output means. The second user guide may include a post-management plan for the subject corresponding to the diagnosis auxiliary information. For example, the second user guide may instruct the subject's next treatment time, next treatment subject, and the like.

When the diagnosis target information includes the third level information, the third user guide may be output through the output means. The third user guide may indicate that recapture is required for the diagnosis target image. The third user guide may include information on the quality of the image to be diagnosed. For example, the third user guide may include information on defects present in the diagnosis target image (eg, whether bright artifacts or dark artifacts, or a degree thereof).

1.4 Diagnostic assistance system for multiple labels

According to an embodiment, a diagnosis assistance system for predicting with respect to a plurality of labels (eg, a plurality of diagnostic assistance information) may be provided. To this end, the diagnostic assistance neural network of the aforementioned diagnostic assistance system may be designed to predict a plurality of labels.

Alternatively, in the above-described diagnostic assistance system, a plurality of diagnostic assistance neural networks that predict different labels may be used in parallel.

Hereinafter, such a parallel diagnosis assistance system will be described.

1.4.1 Parallel Diagnostic Auxiliary System Configuration

According to an embodiment, a parallel diagnosis assistance system for acquiring a plurality of diagnostic assistance information may be provided. The parallel diagnosis assistance system may train a plurality of neural network models for acquiring a plurality of diagnostic assistance information, and acquire a plurality of diagnostic assistance information using the learned plurality of neural network models.

For example, the parallel diagnosis assistance system includes a first neural network model that acquires first diagnostic auxiliary information related to the presence or absence of an eye disease in the subject and second diagnostic auxiliary information related to the presence or absence of a systemic disease in the subject, based on the fundus image. The second neural network model may be trained, and diagnostic auxiliary information regarding the presence or absence of an eye disease and a systemic disease of the subject may be output by using the learned first neural network model and the second neural network model.

21 and 22 are diagrams for explaining a parallel diagnosis assistance system according to some example embodiments. 21 and 22 , the parallel diagnosis assistance system may include a plurality of learning units.

Referring to FIG. 21 , the parallel diagnosis assistance system 30 according to an embodiment may include a learning device 1000 , a diagnosis device 2000 , and a client device 3000 . In this case, the learning apparatus 1000 may include a plurality of learning units. For example, the learning apparatus 1000 may include a first learning unit 100a and a second learning unit 100b.

Referring to FIG. 22 , the parallel diagnosis assistance system 40 according to an embodiment may include a first learning device 1000a, a second learning device 1000b, a diagnosis device 2000, and a client device 3000 . have. The first learning apparatus 1000a may include a first learning unit 100a. The second learning apparatus 1000b may include a second learning unit 100b.

21 and 22 , the first learning unit 100a may obtain a first data set and may output a first parameter set of the first neural network model obtained as a result of learning the first neural network model. The second learning unit 100b may obtain a second data set and output a second parameter set of the second neural network model obtained as a result of learning the second neural network model.

The diagnosis apparatus 2000 may include a diagnosis unit 200 . The contents described with reference to FIG. 1 may be similarly applied to the diagnosis apparatus 2000 and the diagnosis unit 200 . The diagnosis unit 200 obtains the first diagnosis auxiliary information and the second diagnosis auxiliary information by using the first neural network model and the second neural network model learned from the first learning unit 100a and the second learning unit 100b, respectively. can do. The diagnosis unit 2000 may store the learned parameters of the first neural network model and the learned parameters of the second neural network model obtained from the first learner 100a and the second learner 100b.

The client device 3000 may include a data acquisition unit, for example, an imaging unit 300 . However, the imaging unit 300 may be replaced with a data acquisition means used to acquire other diagnostic auxiliary information. The client device may transmit a diagnosis request and diagnosis target data (eg, a fundus image obtained from an imaging unit) to the diagnosis device. In response to transmitting the diagnosis request, the client device 3000 may acquire a plurality of diagnosis auxiliary information according to the transmitted diagnosis target data from the diagnosis device.

Meanwhile, in FIGS. 21 and 22 , a case in which the diagnosis assistance system 40 includes the first learning unit 100a and the second learning unit 100b has been described as a reference, but the content of the exemplary embodiment is not limited thereto. According to another embodiment of an embodiment, the learning apparatus may include a learning unit that acquires three or more pieces of different diagnosis auxiliary information. Alternatively, the diagnosis assistance system may include a plurality of learning devices for acquiring different diagnosis assistance information.

More specific operations of the learning device, the diagnostic device, and the client device will be described in more detail below.

1.4.2 Parallel training process

According to an embodiment, a plurality of neural network models may be trained. The training process for learning each neural network model may be performed in parallel.

1.4.2.1 Parallel Learning Unit

The training process may be performed by a plurality of learning units. Each training process may be performed independently of each other. The plurality of learning units may be provided in one learning apparatus or may be provided in each of the plurality of learning apparatuses.

23 is a diagram for describing a configuration of a learning apparatus including a plurality of learning units according to an exemplary embodiment. The configuration and operation of each of the first learning unit 100a and the second learning unit 100b may be implemented similarly to those described above with reference to FIG. 9 .

Referring to FIG. 23 , the neural network model process according to an embodiment includes a first data processing module 110a, a first queue module 130a, a first learning module 150a, and a first learning result obtaining module 170a. ) including a first learning unit 100a and a second data processing module 110b, a second queue module 130b, a second learning module 150b and a second learning result obtaining module 170b including 2 It may be performed by the learning apparatus 1000 including the learning unit 100b.

Referring to FIG. 23 , the training process of the neural network model according to an embodiment may be performed by the first learner 100a and the second learner 100b, respectively. The first learning unit 100a and the second learning unit 100b may independently learn the first neural network model and the second neural network model. Referring to FIG. 23 , the first learning unit 100a and the second learning unit 100b may be provided in the above-described learning apparatus. Alternatively, the first learning unit and the second learning unit may be provided in different learning devices.

1.4.2.2 Parallel Data Acquisition

According to an embodiment, the plurality of learning units may acquire data. The plurality of learning units may acquire different data sets. Alternatively, a plurality of learning units may acquire the same data set. In some cases, a plurality of learning units may acquire a data set in which some are common. The data set may be a fundus image data set.

The first learning unit may obtain a first data set, and the second learning unit may obtain a second data set. The first data set and the second data set may be distinguished. The first data set and the second data set may have some common features. The first data set and the second data set may be labeled fundus image data sets.

The first data set may include data labeled as normal for the first characteristic and data labeled as abnormal for the first characteristic. For example, the first data set may include fundus images labeled as normal and fundus images labeled as abnormal with respect to lens opacity.

The second data set may include data labeled as normal for the second characteristic (distinct from the first characteristic) and data labeled as abnormal for the second characteristic. For example, with respect to diabetic retinopathy, the second data set may include fundus images labeled as normal and fundus images labeled as abnormal.

The data set labeled as normal for the first feature and data labeled as normal for the second feature included in each of the first data set and the second data set may be common. For example, the first data set includes fundus images labeled normal and abnormal with respect to lens opacity, and the second data set includes fundus images labeled normal and abnormal with respect to diabetic retinopathy. a fundus image labeled as being normal with respect to lens opacity included in the first data set and fundus images labeled normal with respect to diabetic retinopathy included in the second data set may be common. can

Alternatively, data labeled as abnormal for the first characteristic and data labeled as abnormal for the second characteristic included in each of the first data set and the second data set may be common. That is, data labeled with respect to a plurality of features may be used for training a neural network model for a plurality of features.

Meanwhile, the first data set may be a fundus image data set photographed by the first method, and the second data set may be a fundus image data set photographed by the second method. The first method and the second method may be any one method selected from red-free photographing, panoramic photographing, autofluorescence photographing, infrared photographing, and the like.

The data set used in each learning unit may be determined in consideration of the diagnosis auxiliary information obtained by the learned neural network. For example, when the first learning unit trains the first neural network model to acquire diagnostic auxiliary information related to retinal abnormalities (eg, microvascular flow, exudates, etc.), red-free photographed first fundus image data set can be obtained. Alternatively, when the second learning unit trains the second neural network model to acquire diagnostic auxiliary information related to macular degeneration, the second learning unit may acquire the autofluorescence-captured second fundus image data set.

1.4.2.3 Parallel data processing

The plurality of learning units may process the obtained data, respectively. As described above in 2.2 Data Processing Process, each learning unit may process data by applying one or more of image resizing, pre-processing filter application, image augmentation, and image serialization processes to the acquired data. The first data processing module of the first learning unit may process the first data set, and the second data processing module of the second learning unit may process the second data set.

The first learning unit and the second learning unit included in the plurality of learning units may differently process the obtained data set in consideration of the diagnosis auxiliary information obtained from the neural network model trained by each. For example, in order to train the first neural network model for acquiring the first diagnostic auxiliary information related to hypertension, the first learning unit performs preprocessing for emphasizing blood vessels with respect to fundus images included in the first fundus image data set. can be done Alternatively, in order to train the second neural network model for acquiring second diagnostic auxiliary information related to abnormal findings such as retinal exudate, microvascular, and the like, the second learning unit relates to the fundus images included in the second fundus image data set. It is also possible to perform pre-processing to convert it to a red-free image.

1.4.2.4 Parallel queues

The plurality of learning units may store data in a queue. As described in 2.2.6 Queue above, each learning unit can store processed data in a queue and deliver it to the learning module. For example, the first learning unit may store the first data set in the first queue module and provide it to the first learning module sequentially or randomly. The second learning module may store the second data set in the second queue module and provide it to the second learning module sequentially or randomly.

1.4.2.5 Parallel Learning Process

The plurality of learning units may train the neural network model. Each learning module can independently train a diagnostic auxiliary neural network model that predicts different labels using a training data set. The first learning module of the first learning unit may train the first neural network model, and the second learning module of the second learning unit may train the second neural network module.

A plurality of diagnostic auxiliary neural network models may be trained in parallel and/or independently. As described above, by training the model to predict different labels through a plurality of neural network models, the prediction accuracy for each label can be improved, and the efficiency of the prediction operation can be increased.

Each diagnostic auxiliary neural network model can be prepared similarly to that described in 2.3.2 Model design. Each sub-learning process may be performed similarly to that described above in 2.3.1 to 2.3.5.

The parallel learning process according to an embodiment may include training a diagnostic auxiliary neural network model that predicts different labels. The first learning unit may train the first diagnostic auxiliary neural network model for predicting the first label. The second learning unit may train the second diagnostic auxiliary neural network model for predicting the second label.

The first learning unit may acquire the first data set and train the first diagnostic auxiliary neural network model to predict the first label. For example, the first learning unit may train a first diagnostic auxiliary neural network model for predicting whether the subject is macular degeneration from the fundus image by using the fundus image training data set labeled for macular degeneration.

The second learning unit may acquire the second data set and train the first diagnostic auxiliary neural network model to predict the second label. For example, the second learning unit may train a second diagnostic auxiliary neural network model for predicting whether the subject has diabetic retinopathy from the fundus image by using the fundus image training data set labeled with respect to diabetic retinopathy.

The learning process of the neural network model will be described in more detail below with reference to FIGS. 24 and 25 .

24 is a diagram for describing a parallel learning process according to an embodiment. The parallel learning process may be applied to both the case where the parallel diagnosis assistance system is implemented as shown in FIG. 21 , the case where the parallel diagnosis assistance system is implemented as shown in FIG. 22 , and other cases. However, for convenience of explanation, the following description will be based on the parallel diagnosis auxiliary system implemented as shown in FIG. 21 .

Referring to FIG. 24 , the parallel learning process may include a plurality of sub-learning processes for respectively learning a plurality of diagnostic auxiliary neural network models that predict different labels. The parallel learning process may include a first sub-learning process for training the first neural network model and a second sub-learning process for training the second neural network model.

For example, the first sub-learning process obtains the first data (S1010a), uses the first neural network model (S1030a), validates the first model (ie, the first diagnostic auxiliary neural network model), and (S1050a) the first This may be performed by acquiring the parameters of the neural network model (S1070a). The second sub-learning process acquires the second data (S1010b), uses the second neural network model (S1030b), validates the second neural network model (ie, the second diagnostic auxiliary neural network model), and (S1050b) the second neural network This may be performed by acquiring the parameters of the model (S1070b).

The sub-learning process may include training the neural network model by inputting training data into the sub-neural network model, verifying the model by comparing the label value obtained as an output with the input training data, and reflecting the validation result back to the sub-neural network model. have.

Each sub-learning process obtains a result value using a neural network model to which arbitrary weight values are given, compares the obtained result value with a label value of training data, and performs backpropagation according to the error, may include optimizing them.

In each sub-learning process, the diagnostic auxiliary neural network model may be validated through a validation data set distinct from the training data set. A validation data set for validating the first neural network model and the second neural network model may be distinguished.

The plurality of learning units may acquire a learning result. Each of the learning result acquisition modules may acquire information about the neural network module learned from the learning module. Each learning result acquisition module may acquire parameter values of the neural network module learned from the learning unit. The first learning result obtaining module of the first learning unit may obtain a first parameter set of the first neural network model learned from the first learning module. The second learning result obtaining module of the second learning unit may obtain a second parameter set of the second neural network model learned from the second learning module.

By each sub-learning process, optimized parameter values of the learned neural network model, that is, a parameter set may be obtained. As learning proceeds using more training data sets, more appropriate parameter values may be obtained.

A first parameter set of the first diagnostic auxiliary neural network model learned by the first sub-learning process may be obtained. A second parameter set of the second diagnostic auxiliary neural network model learned by the second sub-learning process may be obtained. As the learning progresses sufficiently, optimized values of weights and/or biases of the first diagnostic auxiliary neural network model and of the second diagnostic auxiliary neural network model may be obtained.

The acquired parameter set of each neural network model may be stored in a learning device and/or a diagnostic device (or server). The first parameter set of the first diagnostic auxiliary neural network model and the second parameter set of the second diagnostic auxiliary neural network model may be stored together or separately. The parameter set of each learned neural network model may be updated by feedback obtained from a diagnostic device or a client device.

1.4.2.6 Parallel Ensemble Learning Process

Even when a plurality of neural network models are trained in parallel, the aforementioned ensemble type model learning may be used. Each sub-learning process may include training a plurality of sub-neural network models. The plurality of sub-models may have different hierarchical structures. Hereinafter, the contents described in 2.3.7 may be similarly applied unless otherwise noted.

When a plurality of diagnostic auxiliary neural network models are trained in parallel, some of the sub-learning processes for training each diagnostic auxiliary neural network model train a single model, and some sub-learning processes train a plurality of sub-models together. It may be implemented in the form of learning.

As the model is trained using an ensemble in each sub-learning process, a more optimized form of a neural network model can be obtained in each sub-process, and prediction errors can be reduced.

25 is a diagram for describing a parallel learning process according to another embodiment of an embodiment. Referring to FIG. 25 , each learning process may include training a plurality of sub-neural network models.

Referring to FIG. 25 , the first sub-learning process acquires first data (S1011a), uses a 1-1 neural network model and a 1-2 neural network model (S1031a, S1033a), and a 1-1 neural network model and This may be performed by verifying the 1-2 neural network model (S1051a), and determining the final shape of the first neural network model and its parameters (S1071a). The second sub-learning process acquires the second data (S1011b), uses the 2-1 neural network model and the 2-2 neural network model (S1031b, S1033b), and the 2-1 neural network model and the 2-2 neural network model can be performed by verifying (S1051b) to determine the final shape of the first model (ie, the first diagnostic auxiliary neural network model) and its parameters (S1071b).

The first neural network learned in the first sub-learning process may include a 1-1 neural network model and a 1-2 neural network model. The 1-1 neural network model and the 1-2 neural network model may be provided in different hierarchical structures. The 1-1 neural network model and the 1-2 neural network model may obtain a first data set and output predicted labels, respectively. Alternatively, a label predicted by the ensemble of the 1-1 neural network model and the 1-2 neural network model may be determined as the final predicted label.

In this case, the 1-1 neural network model and the 1-2 neural network model may be verified using the validation data set, and a neural network model with high accuracy may be determined as the final neural network model. Alternatively, the ensemble of the 1-1 neural network model, the 1-2 neural network model, the 1-1 neural network model, and the 1-2 neural network model is verified, and the neural network model form when the accuracy is high among them is determined as the final first neural network. It can also be determined by the model.

Similarly for the second sub-learning process, a neural network of a high-accuracy form among the ensembles of the 2-1 neural network model, the 2-2 neural network model, the 2-1 neural network model, and the 2-2 neural network model is finally obtained as the second sub-learning process. It may be determined as a model (ie, the second diagnostic auxiliary neural network model).

Meanwhile, in FIG. 25 , for convenience, each sub-learning process has been described based on a case in which two sub-models are included, but this is only an example, and an embodiment is not limited thereto. The neural network model trained in each sub-learning process may include only one neural network model, or may include three or more sub-models.

1.4.3 Parallel diagnostic process

According to an embodiment, a diagnostic process for acquiring a plurality of diagnostic auxiliary information may be provided. The diagnostic process for acquiring a plurality of diagnostic auxiliary information may be implemented in the form of a parallel diagnostic auxiliary process including a plurality of independent diagnostic processes.

1.4.3.1 Parallel Diagnostics

According to an embodiment, the diagnostic auxiliary process may be performed by a plurality of diagnostic modules. Each diagnostic auxiliary process may be performed independently.

26 is a block diagram illustrating the diagnosis unit 200 according to an exemplary embodiment.

Referring to FIG. 26 , the diagnosis unit 200 according to an exemplary embodiment includes a diagnosis request obtaining module 211 , a data processing module 231 , a first diagnosis module 251 , a second diagnosis module 253 , and an output module. (271) may be included. Each module of the diagnosis unit 200 may operate similarly to the diagnosis module of the diagnosis unit illustrated in FIG. 18 , unless otherwise specified.

In FIG. 26 , even when the diagnostic unit 200 includes a plurality of diagnostic modules, the diagnostic request obtaining module 211 , the data processing module 231 , and the output module 271 are illustrated as being common, but in one embodiment It is not limited to this configuration, and a plurality of diagnostic request acquisition modules, data processing modules, and/or output modules may also be provided. A plurality of diagnostic request obtaining modules, data processing modules and/or output modules may also operate in parallel.

For example, the diagnosis unit 200 includes a first data processing module that performs first processing on the input diagnosis target image and a second processing module that performs second data processing on the diagnosis target image, The first diagnosis module may obtain first diagnosis auxiliary information based on the first processed diagnosis subject image, and the second diagnosis module may obtain second diagnosis auxiliary information based on the second processed diagnosis subject image. The first processing and/or the second processing may be any one selected from image resizing, image color modulation, blur filter application, blood vessel enhancement processing, red-free transformation, partial region cropping, and partial element extraction.

The plurality of diagnostic modules may acquire different diagnostic auxiliary information. The plurality of diagnosis modules may acquire the diagnosis auxiliary information by using different diagnosis auxiliary neural network models. For example, the first diagnosis module obtains first diagnostic auxiliary information related to whether the subject has an eye disease by using a first neural network model that predicts whether the subject has an eye disease, and the second diagnosis module Second diagnostic auxiliary information related to whether the subject corresponds to a systemic disease may be obtained by using the second neural network model for predicting whether the subject corresponds to a systemic disease.

As a more specific example, the first diagnosis module obtains first diagnostic auxiliary information regarding whether the subject has diabetic retinopathy by using a first diagnosis auxiliary neural network model that predicts whether the subject has diabetic retinopathy based on the fundus image, , the second diagnosis module may obtain second diagnosis auxiliary information related to whether the subject has hypertension by using a second diagnosis auxiliary neural network model that predicts whether the subject has hypertension based on the fundus image.

1.4.3.2 Parallel diagnostic process

A diagnosis auxiliary process according to an embodiment may include a plurality of sub-diagnosis processes. Each sub-diagnosis process may be performed using a different diagnostic auxiliary neural network model. Each sub-diagnosis process may be performed in different diagnostic modules. For example, the first diagnosis module may perform a first sub-diagnosis process of acquiring the first diagnosis assistance information through the first diagnosis assistance neural network model. Alternatively, the second diagnosis module may perform a second sub-diagnosis process of acquiring second diagnosis auxiliary information through the second diagnosis auxiliary neural network model.

The plurality of learned neural network models may output predicted labels or probabilities by receiving diagnosis target data as input. Each neural network model may be provided in the form of a classifier, and input diagnostic target data may be classified with respect to a predetermined label. In this case, the plurality of neural network models may be provided in the form of classifiers learned for different characteristics. Each neural network model can classify diagnostic target data as described above in 3.4.2.

Meanwhile, from each diagnostic auxiliary neural network model, a CAM may be obtained, and the CAM may be selectively obtained. The CAM may be extracted when a predetermined condition is satisfied. For example, when the first diagnostic assistance information indicates that the subject is abnormal with respect to the first characteristic, the first CAM may be obtained from the first diagnostic assistance neural network model.

27 is a diagram for describing a diagnosis assistance process according to an exemplary embodiment.

Referring to FIG. 27 , the diagnosis assistance process according to an exemplary embodiment acquires diagnosis target data (S2011), and uses the first diagnosis auxiliary neural network model and the second diagnosis auxiliary neural network model (S2031a, S2031b) to obtain the diagnosis target data. It may include acquiring ( S2051 ) diagnostic auxiliary information. The diagnosis target data may be processed data.

The diagnosis assistance process according to an embodiment may include acquiring first diagnosis assistance information through the learned first diagnosis assistance neural network model and acquiring second diagnosis assistance information through the learned second diagnosis assistance neural network model. have. The first auxiliary diagnosis neural network model and the second auxiliary diagnosis neural network model may acquire first diagnosis auxiliary information and second diagnosis auxiliary information, respectively, based on the same diagnosis target data.

For example, the first diagnostic auxiliary neural network model and the second diagnostic auxiliary neural network model may include, based on the fundus image to be diagnosed, the first diagnostic auxiliary information regarding whether the subject has macular degeneration and the second diagnosis auxiliary information regarding whether the subject has diabetic retinopathy. Each of the 2 diagnostic auxiliary information can be obtained.

In addition, unless otherwise specified, the diagnosis assistance process described with reference to FIG. 27 may be implemented similarly to the diagnosis assistance process described with reference to FIG. 20 .

1.4.3.3 Output of diagnostic auxiliary information

According to an embodiment, the diagnosis auxiliary information obtained by the parallel diagnosis auxiliary process may be obtained. The obtained diagnostic assistance information may be stored in a diagnostic device, a server device, and/or a client device. The acquired diagnostic auxiliary information may be transmitted to an external device.

The plurality of diagnostic auxiliary information may indicate a plurality of labels predicted by the plurality of diagnostic auxiliary neural network models, respectively. The plurality of diagnostic auxiliary information may respectively correspond to a plurality of labels predicted by the plurality of diagnostic auxiliary neural network models. Alternatively, the diagnostic auxiliary information may be information determined based on a plurality of labels predicted by a plurality of diagnostic auxiliary neural network models. The diagnostic auxiliary information may correspond to a plurality of labels predicted by the plurality of diagnostic auxiliary neural network models.

In other words, the first diagnostic auxiliary information may be diagnostic auxiliary information corresponding to the first label predicted through the first diagnostic auxiliary neural network model. Alternatively, the first diagnosis auxiliary information may be diagnosis auxiliary information determined by considering the first label predicted through the first diagnosis auxiliary neural network model and the second label predicted through the second diagnosis auxiliary neural network model together.

Meanwhile, CAM images obtained from a plurality of diagnostic auxiliary neural network models may be output. The CAM image may be output when a predetermined condition is satisfied. For example, in either case where the first diagnostic auxiliary information indicates that the subject is abnormal with respect to the first characteristic or the second diagnostic auxiliary information indicates that the subject is abnormal with respect to the second characteristic, A CAM image obtained from a diagnostic auxiliary neural network model to which diagnostic auxiliary information indicated to be abnormal may be output.

A plurality of diagnostic auxiliary information and/or CAM images may be provided to the user. A plurality of diagnostic auxiliary information and the like may be provided to a user through an output means of a diagnostic device or a client device. The diagnostic auxiliary information may be visually output. In this regard, 5. User interface will be described in more detail.

According to an embodiment, the diagnosis auxiliary information corresponding to the diagnosis target image may include grade information. The rating information may be selected from among a plurality of ratings. The grade information may be determined based on a plurality of diagnostic information and/or observation information obtained through the neural network model. The grade information may be determined in consideration of suitability information or quality information of an image to be diagnosed. The grade information may be determined in consideration of the class into which the diagnosis target image is classified by a plurality of neural network models. The grade information may be determined in consideration of numerical values output from a plurality of neural network models.

For example, the diagnosis auxiliary information obtained in response to the diagnosis target image may include any one selected from the first grade information and the second grade information. The grade information may be selected as the first grade information when at least one abnormal finding information or abnormal diagnosis information is acquired from among diagnostic information acquired through a plurality of neural network models. The grade information may be selected as the second grade information when abnormal findings information or abnormal diagnostic information is not acquired among the diagnostic information acquired through the neural network model.

The grade information is selected as the first grade information when at least one of the values obtained through the neural network model exceeds the reference value, and is selected as the second grade information when all of the obtained values do not meet the reference value it might be The first grade information may indicate that abnormal information stronger than the second grade information exists in the diagnosis target image.

The rating information may be selected as the third rating information when it is determined that the quality of the image to be diagnosed is less than or equal to a standard using image analysis or a neural network model. Alternatively, the diagnosis auxiliary information may include the third level information together with the first or second level information.

When the diagnosis auxiliary information includes the first level information, the first user guide may be output through the output means. The first user guide may include at least one piece of abnormal observation information included in the diagnosis auxiliary information or items corresponding to the abnormal diagnosis information. For example, the first user guide may indicate that a more precise examination is required for the subject (ie, the patient) corresponding to the abnormal information included in the diagnosis auxiliary information. For example, the first user guidance may indicate that a secondary diagnosis (eg, diagnosis at a separate medical institution or power supply procedure) is required for the subject. Alternatively, the first user guidance may indicate a required treatment for the subject. As a specific example, when abnormal information on the macular degeneration of the subject is acquired by the diagnostic auxiliary information, the first user guide provides information on the injection prescription and power supply procedure for the subject (eg, at a hospital where power is available). list) can be included.

When the diagnosis auxiliary information includes the second level information, the second user guide may be output through the output means. The second user guide may include a post-management plan for the subject corresponding to the diagnosis auxiliary information. For example, the second user guide may instruct the subject's next treatment time, next treatment subject, and the like.

When the diagnosis target information includes the third level information, the third user guide may be output through the output means. The third user guide may indicate that recapture is required for the diagnosis target image. The third user guide may include information on the quality of the image to be diagnosed. For example, the third user guide may include information on defects present in the diagnosis target image (eg, whether bright artifacts or dark artifacts, or a degree thereof).

The first to third grade information may be output by an output unit of the client device or the diagnostic device. Specifically, it may be output through a user interface to be described later.

1.4.4 Example 2 - Diagnostic Aid System

A diagnosis assistance system according to an embodiment may include a fundus image acquisition unit, a first processing unit, a second processing unit, a third processing unit, and a diagnostic information output unit.

According to an embodiment, the diagnosis assistance system may include a diagnosis apparatus. The diagnosis apparatus may include a fundus image acquisition unit, a first processing unit, a second processing unit, a third processing unit, and/or a diagnostic information output unit. However, the exemplary embodiment is not limited thereto, and each unit included in the diagnosis assistance system may be located at an appropriate location on the learning device, the diagnosis device, the learning diagnosis server, and/or the client device. Hereinafter, for convenience, a case in which the diagnosis apparatus of the diagnosis assistance system includes the fundus image acquisition unit, the first processing unit, the second processing unit, the third processing unit, and the diagnostic information output unit will be described.

28 is a diagram for describing a diagnosis assistance system according to an exemplary embodiment. Referring to FIG. 28 , the diagnosis assistance system may include a diagnosis apparatus, and the diagnosis apparatus may include an fundus image acquisition unit, a first processing unit, a second processing unit, a third processing unit, and a diagnostic information output unit.

According to an embodiment of the present disclosure, a diagnosis assistance system supporting diagnosis of a plurality of diseases based on a fundus image includes: a fundus image acquisition unit configured to acquire a target fundus image that is a basis for acquiring diagnostic assistance information on a subject; A first processing unit for obtaining a first result related to a first observation on the subject by using a first neural network model with respect to the fundus image, wherein the first neural network model is machine learned based on the first set of fundus images , a second neural network model for the target fundus image, wherein the second neural network model is machine learned based on a second fundus image set that is at least partially different from the first fundus image set. 2 A second processing unit for obtaining a second result related to the observation, a third processing unit for determining diagnostic information about the subject based on the first result and the second result, and providing the determined diagnostic information to a user It may include a diagnostic information output unit. In this case, the first observation and the second observation may be used for diagnosis of different diseases.

The first neural network model is trained to classify the input fundus image as either normal or abnormal in relation to the first observation, and the first processing unit converts the target fundus image to the normal or abnormal using the first neural network model. The first result may be obtained by classifying it as any one of them.

The third processing unit may determine whether the diagnosis information according to the target fundus image is normal information or abnormal information by considering the first result and the second result together.

The third processing unit may determine the diagnosis information on the subject by giving priority to the abnormality so as to improve diagnosis accuracy.

When the first label is normal for the first observation and the second label is normal for the second observation, the third processing unit determines the diagnostic information as normal, and the first label is normal for the first observation When it is not normal or the second label is not normal for the second observation, the diagnostic information may be determined to be abnormal.

The first finding may be related to an eye disease, and the first result may indicate whether the subject is normal to the eye disease. The second finding may be related to a systemic disease, and the second result may indicate whether the subject is normal to the systemic disease.

A first finding is related to a first ocular disease, the first result indicates whether the subject is normal for the first ocular disease, and the second finding is a second ocular disease distinct from the first ocular disease. related to a disease, and the second result may indicate whether the subject is normal to the second eye disease.

The first observation is an observation for diagnosing a first ocular disease, the first result indicates whether the subject is normal for the first ocular disease, and the second observation is for diagnosing the first ocular disease. It is a finding distinct from the first finding, and the second result may indicate whether the subject is normal to the second eye disease.

The first neural network model includes a first sub neural network model and a second sub neural network model, and a first result is a first predicted value predicted by the first sub neural network model and a second predicted value predicted by the second sub neural network model It may be determined by considering the predicted values together.

The first processing unit may obtain a CAM (Class Activation Map) related to the first label through the first neural network model, and the diagnostic information output unit may output an image of the CAM.

The diagnosis information output unit may output the CAM image when the diagnosis information acquired by the third processing unit is abnormal diagnosis information.

The diagnosis assistance system may further include a fourth processing unit configured to acquire quality information of the target fundus image, and the diagnostic information output unit may output quality information of the target fundus image acquired by the fourth processing unit.

When the fourth processing unit determines that the quality information of the target fundus image is equal to or less than a predetermined quality level, the diagnostic information output unit provides the user with the quality information of the target fundus image together with the determined diagnostic information to the user at the predetermined quality level. Information indicating the following may be provided together.

1.5 User Interface

According to an embodiment, the above-described client device or diagnosis device may have a display unit for providing diagnosis auxiliary information to a user. In this case, the display unit may be provided to clearly transmit the diagnosis auxiliary information to the user and to easily obtain feedback from the user.

As an example of the display unit, a display providing visual information to a user may be provided. In this case, a graphic user interface for visually delivering the diagnosis auxiliary information to the user may be used. For example, in a fundus diagnosis assistance system that acquires diagnosis assistance information based on a fundus image, a graphical user interface for effectively displaying the acquired diagnosis assistance information and helping a user to understand may be provided.

29 and 30 are diagrams for explaining a graphical user interface for providing diagnostic information to a user according to some embodiments of the present disclosure; Hereinafter, with reference to FIGS. 29 and 30 , a user interface that can be used in the fundus diagnosis assistance system will be described with reference to some embodiments.

Referring to FIG. 29 , the user interface according to an embodiment may display identification information of a subject corresponding to a fundus image to be diagnosed. The user interface may include a target image identification information display unit 401 that displays identification information of a subject (ie, a patient) and/or imaging information (eg, imaging date) of a fundus image to be diagnosed.

The user interface according to an embodiment may include a fundus image display unit 405 that displays the fundus image of the left eye and the fundus image of the right eye of the same subject. The fundus image display unit 405 may display a CAM image.

The user interface according to an embodiment displays a left eye or right eye image for each of the fundus image of the left eye and the fundus image of the right eye, and the diagnostic information indicating unit for displaying the diagnostic information of each image and a diagnostic information indicator indicating whether to confirm the user (403) may be included.

The color of the diagnostic information indicator may be determined in consideration of the diagnostic auxiliary information obtained based on the target fundus image. The diagnostic information indicator may be displayed in a first color or a second color according to the diagnostic auxiliary information. For example, when the first to third diagnostic auxiliary information is acquired from one target fundus image, when even one diagnostic auxiliary information includes abnormal (that is, abnormal findings) information, the diagnostic information indicator is displayed in red, , when all of the diagnostic auxiliary information includes normal (ie, no abnormal findings) information, the diagnostic information indicator may be displayed in green.

The type of the diagnostic information indicator may be determined according to whether the user confirms it. The diagnostic information indicator may be displayed in a first form or a second form depending on whether the user confirms it. For example, referring to FIG. 25 , the diagnostic information indicator corresponding to the target fundus image reviewed by the user is displayed as a filled circle, and the diagnostic information indicator corresponding to the target fundus image not reviewed by the user is represented by a filled semicircle. can be displayed.

The user interface according to an embodiment may include a diagnosis information indicating unit 407 indicating auxiliary diagnosis information. The diagnostic auxiliary information indicating unit may be located in the left eye and right eye images, respectively. The diagnosis auxiliary information indicating unit may indicate a plurality of observation information or diagnostic information.

The diagnostic auxiliary information indicator may include at least one diagnostic auxiliary information indicator. The diagnostic auxiliary information indicator may indicate corresponding diagnostic auxiliary information through color change.

For example, with respect to the fundus image to be diagnosed, first diagnostic auxiliary information indicating the presence or absence of lens opacity through the first diagnostic auxiliary neural network model, and second diagnostic auxiliary information indicating the presence or absence of diabetic retinopathy through the second diagnostic auxiliary neural network model When the third diagnosis auxiliary information indicating the presence or absence of retinal abnormality is acquired through the information and the third diagnosis auxiliary neural network model, the diagnosis information indicating unit displays the first diagnosis auxiliary information, the second diagnosis auxiliary information, and the third diagnosis auxiliary information, respectively. It may include first to third diagnostic auxiliary information indicators indicating.

As a more specific example, referring to FIG. 29 , in relation to the left eye fundus image, the diagnostic information indicating unit 407 provides first diagnostic auxiliary information indicating that the diagnostic auxiliary information obtained based on the left eye fundus image of the subject is lens opacity abnormal. , first diagnostic auxiliary information having a first color when second diagnostic auxiliary information indicating normal diabetic retinopathy (no abnormal findings) and third diagnostic auxiliary information indicating retinal abnormalities (abnormal findings) are obtained An indicator, a first diagnostic auxiliary information indicator having a second color, and a third diagnostic auxiliary information indicator having a first color may be displayed.

The user interface according to an embodiment may obtain a user comment on the fundus image to be diagnosed from the user. The user interface may include a user comment object 409 and, in response to a user selection for the user comment object, may display a user input window. The comments obtained from the user may be used to update the diagnostic auxiliary neural network model. For example, a user input window displayed in response to a selection of a user comment object may acquire a user evaluation of diagnostic assistance information through a neural network, and the obtained user evaluation may be used to update a neural network model.

The user interface according to an embodiment may include a review instruction object 411 indicating whether the user reviews each diagnosis target fundus image. The review instruction object may receive a user input indicating that the user review for each diagnosis target image is completed, and the display may be changed from the first state to the second state. For example, referring to FIGS. 29 and 30 , the review instruction object may be changed from a first state of displaying a confirmation request phrase to a second state indicating that it has been confirmed when a user input is obtained.

A list 413 of fundus images to be diagnosed may be displayed. In the list, identification information of the subject, an image capture date, and an indicator 403 for reviewing both eyes may be displayed together.

A review completion indicator 415 indicating a review target fundus image may be displayed in the list 413 of the diagnostic target fundus image. The review completion indicator 415 may be displayed when the user selects all of the review instruction objects 411 for both eyes of the corresponding image.

Referring to FIG. 30 , the user graphic interface may include a low-quality warning object 417 instructing the user that there is an abnormality in the quality of the target fundus image when it is determined that there is a quality abnormality in the fundus image to be diagnosed. have. The low-quality warning object 417 is displayed when it is determined by the diagnostic unit that the fundus image to be diagnosed does not reach a level at which appropriate diagnostic auxiliary information can be predicted from the diagnostic auxiliary neural network model (that is, the reference quality level). can be

Also, referring to FIG. 28 , a low-quality warning object 419 may be displayed in the list 413 of the fundus image to be diagnosed.

2. Diagnosis auxiliary image using fundus image

2.1 Background and purpose

In the present specification, an apparatus and method for providing a diagnosis auxiliary image in performing diagnosis of a plurality of lesions using an fundus image and providing a diagnosis result of the plurality of lesions will be described.

The diagnostic result provided by the diagnostic auxiliary device and/or system according to an embodiment of the present specification may be determined by a plurality of labels for the fundus image, and preferably provided as suspicious (or abnormal) or normal. can The suspicion may mean a case in which possession of a disease is suspected, and the normal may mean a case in which there is no possession of a disease.

However, the user or the examinee may have a need to understand the basis for deriving a diagnosis result provided by the diagnosis auxiliary device and/or system from the fundus image. In addition, the user or the examinee may have a need to identify a region on the fundus image that affects the diagnosis result provided by the diagnosis assistance device and/or system.

The present specification provides a diagnostic auxiliary imaging device and/or system capable of outputting a diagnostic auxiliary image together with a diagnostic auxiliary device and/or system for performing diagnosis for a plurality of diseases through a machine-learned neural network model on the fundus image, In addition, the diagnosis auxiliary image may be provided to determine which region of the fundus image affects the diagnosis result obtained through the diagnosis auxiliary device and/or system.

Through this, the user can easily identify the region that has affected the diagnosis result through the diagnosis auxiliary image, and thus the user may not be limited to an expert on the diagnosis result. In addition, as a non-expert, the user may explain the diagnosis result to the subject through the diagnosis auxiliary image. In addition, the user may easily understand the diagnosis result through the diagnosis auxiliary image as the subject himself/herself.

2.2 CAM (Class Activation Map) image providing device and system

2.2.1 CAM image

31 is a diagram for describing an auxiliary diagnosis image according to an exemplary embodiment. In FIG. 31 , (a) is an image input to the diagnostic auxiliary apparatus and/or system, and (b) is a CAM image as a diagnostic auxiliary image provided by the diagnostic auxiliary apparatus and/or system.

When a neural network according to machine learning is used in a learning model and/or a diagnostic model, the apparatus for providing an auxiliary diagnostic image may provide an image capable of providing a basis for a result obtained through the learning model and/or the diagnostic model. have. When a convolutional neural network is used in a learning model and/or a diagnostic model, the CAM image may be an image that can provide a basis for results obtained through the synthetic neural network. According to an embodiment, when a convolutional neural network is used as a diagnostic assistance system using a fundus image, the CAM image may be provided as an image that may provide a basis for a diagnosis result with respect to the fundus image. In addition, when a convolutional neural network is used as a diagnostic assistance system using a fundus image, the CAM image may be provided as an image that may indicate a region that has affected a diagnosis result for the fundus image. In addition, the CAM image may be an image representing an area that the diagnostic assistance device and/or system focused on to derive a diagnostic result. The CAM image may be an image in which a target area for determining a diagnosis result of the diagnostic auxiliary device and/or system is detected and the corresponding target area is emphasized.

Referring to FIG. 31 ( a ) , when the area L indicated by a dotted line in the input image is a lesion area or an area that has affected the diagnosis result of the diagnostic auxiliary device and/or system, the degree of influence of the diagnostic auxiliary image may be expressed in different colors depending on the In addition, the diagnosis auxiliary images may have the same color but different brightness or intensity depending on the degree of influence on the diagnosis result.

Referring to FIG. 31 ( b ) , the CAM image shows the degree of influence on the diagnosis result in the first level 15010 , the second level 15020 , the third level 15030 , the fourth level 15040 , and the fifth level. It can be divided into levels 15050 , and the first class 15010 to the fifth class 15050 can be expressed with different colors having a predetermined difference from each other. In addition, the CAM image may be an image displayed through heat-map processing of an input fundus image based on a diagnosis result derived by a diagnosis model included in the diagnosis auxiliary device and/or system.

The CAM image is provided based on RGB color in the form of a heat-map that is processed as a separate color, but the area that has the most influence on the diagnosis result is treated as red, and the area with the least influence is treated as blue can do. The CAM image is provided in the form of a heat-map that is processed as a separate color and is based on black and white, but the area that has the most influence on the diagnosis result is treated as black and the area with the least influence is treated as white can do. Since the CAM image is an image in the form of a heat map that is superimposed on the image, the user can easily recognize the area that has affected the diagnosis result, and can visually identify it more clearly.

In addition, the CAM image may be provided concurrently with or separately from the diagnostic results of the diagnostic assistance device and/or system. The CAM image may be provided overlaid on the fundus image input to the diagnostic assistance device and/or system. The CAM image may be provided separately from the fundus image input to the diagnostic assistance device and/or system. The CAM image may be provided through a separate processing for the image input to the diagnostic assistance device and/or the system. As an example, the CAM image may be provided when a user clicks a fundus image input to the diagnostic auxiliary device and/or system with a mouse or places a mouse cursor on the fundus image.

The CAM image is one of visual explanation images, and the CAM image may be replaced with a visual explanation image of various techniques. The CAM image may be replaced with a visual explanatory image using a Local Interpretable Model-agnostic Explainations (LIME ) technique, a Gradient-Class Activation Map (Grad-CAM) technique, a Randomized Input Sampling (RISE) technique, or the like. An embodiment is intended to describe in more detail a diagnostic auxiliary imaging apparatus and/or system using a CAM image.

2.2.2 CAM image delivery process

According to an embodiment, the CAM image providing apparatus may provide a CAM image providing process (or diagnostic auxiliary image providing process) for acquiring a CAM image as diagnostic auxiliary information (or diagnostic auxiliary image) using a neural network model. As a specific example, the CAM image providing apparatus may output a CAM image through a CAM image providing process through the diagnostic auxiliary information providing unit, and the user can easily grasp the diagnosis result determined by the diagnostic auxiliary apparatus based on the fundus image through the CAM image. can

32 is a diagram for explaining the configuration of the diagnostic auxiliary information providing unit 400 according to an exemplary embodiment. Referring to FIG. 32 , the diagnostic auxiliary information providing unit 400 provides a diagnostic request obtaining module 410 , a data processing module 430 , a diagnostic module 450 , a CAM providing module 470 , and an output module 490 . may include

33 is a diagram for describing a diagnostic process and a CAM providing process performed by a diagnostic apparatus according to an exemplary embodiment.

The CAM image providing process described below may be provided through a diagnostic auxiliary device or a separate device.

2.2.2.1 Diagnostic auxiliary information provider

According to an embodiment, the CAM image providing process may be performed by the diagnostic auxiliary information providing unit 400 . The diagnosis auxiliary information providing unit 400 may be provided in the above-described diagnosis apparatus. The diagnosis auxiliary information providing unit 400 may output auxiliary diagnosis information by the diagnosis apparatus, and the auxiliary diagnosis information may output a CAM image together with or separately from the diagnosis result.

Each module may perform individual steps, such as a diagnostic assistance process and a CAM provisioning process, as described below. However, not all components and functions performed by the components described in FIG. 32 are essential, and some components and functions may be added or omitted.

2.2.2.2 Data Acquisition and Processing Process

The data acquisition and processing process may be the same as described above in 1.3.2 Data acquisition and diagnosis request and 1.3.3 data processing process. However, each process is not essential, and may be omitted or a separate process may be added depending on a diagnosis target, a diagnosis disease, and the like.

2.2.2.3 Diagnostic process

Referring to FIG. 32 , the diagnosis process may be performed by the diagnosis module 450 . Referring to FIG. 33 , the diagnosis process may include data acquisition ( S4010 ), a learned diagnosis auxiliary neural network model ( S4020 ), and outputting a diagnosis result ( S4030 ). Referring to FIG. 33 , in the diagnosis process, the diagnosis auxiliary information may be output ( S4060 ) through data acquisition ( S4010 ), the learned diagnosis auxiliary neural network model ( S4020 ), and the diagnosis result output ( S4030 ). The diagnosis process may output a diagnosis result as the diagnosis auxiliary information. A diagnosis result obtained in the diagnosis process may be used in a CAM image providing process to be described later.

The contents regarding the diagnosis process may be the same as those described above in 1.3.4 diagnosis process. However, each process is not essential, and may be omitted or a separate process may be added according to a diagnosis target, a diagnosis disease, provided information, and the like.

2.2.2.4 CAM image delivery process

Referring to FIG. 32 , in the diagnosis apparatus according to an embodiment, a CAM image providing process may be performed by the CAM providing module 470 . The CAM providing module 470 may perform a CAM image providing process based on at least one of inputted diagnostic target data and information obtained by the diagnostic module 450 . The CAM providing module 470 may perform a CAM image providing process based on the inputted diagnostic target data and information obtained from the diagnostic module 450 .

Referring to FIG. 33 , the CAM providing process may include data acquisition (S4010), a CAM image providing module (S4040), and a CAM image output (S4050). In the CAM providing process, diagnostic auxiliary information may be output ( S4060 ) through data acquisition ( S4010 ), the CAM image providing module ( S4040 ), and the CAM image output ( S4050 ). In the CAM provision process, a CAM image may be output as the diagnosis auxiliary information. In the CAM providing process, a CAM image may be output as the diagnosis auxiliary information based on at least one of the acquired data and the information acquired in the diagnosis process. The CAM providing process is based on at least one of the data obtained in the data acquisition (S4010), the information obtained in the learned diagnostic auxiliary neural network model (S4020), and the result obtained through the diagnosis result output (S4030), the diagnosis assistance A CAM image may be output as information.

In the following description, the CAM providing process will be described in detail.

2.2.2.4.1 Information obtained from the diagnostic process

34 is a diagram for explaining a diagnosis process by the learned diagnosis auxiliary neural network model S4020.

32 to 34 , the CAM image providing apparatus may provide a CAM image based on information obtained by a diagnosis process using the learned diagnosis auxiliary neural network model S4020.

According to an embodiment, the CAM image providing apparatus may provide a CAM image based on information obtained by a diagnosis process using a convolutional neural network model including a plurality of layers 500 . Also, according to an embodiment, the CAM image providing apparatus may provide a CAM image based on information obtained by a diagnosis process using a convolutional neural network model. However, the learned diagnostic auxiliary neural network model S4020 is not limited thereto, and the CAM image may be provided by other neural network models as well.

The CAM image providing apparatus may provide a CAM image based on at least one of first information and second information provided from the learned diagnostic auxiliary neural network model S4020. The first information may be a feature value of the feature map 510 of the last layer obtained from the learned diagnostic auxiliary neural network model S4020, and the second information may be a diagnosis disease based on the feature map 510 of the last layer. It may be a probability value (W) corresponding to a normal label or an abnormal label for .

Hereinafter, among the information obtained by the diagnosis process, information that can be used in the CAM image providing apparatus will be described in more detail. To this end, first of all, the learned diagnosis auxiliary neural network model S4020 used in the diagnosis process will be described.

The learned diagnostic auxiliary neural network model S4020 may be a neural network having a plurality of layers. The learned diagnostic auxiliary neural network model S4020 may be a convolutional neural network model. When the learned diagnosis auxiliary neural network model S4020 is a convolutional neural network model, the learned diagnosis auxiliary neural network model S4020 may have a neural network having a plurality of layers 500 .

The plurality of layers 500 may be a plurality of convolution layers. The convolutional layer may perform a convolution operation on input data. The convolution layer may extract a plurality of feature maps through a convolution operation on input data. The number of the feature maps may vary according to the number of filters used in the convolutional layer. The convolutional layer may be subjected to a convolution operation by the same filter on the input data set, and a plurality of feature maps obtained through this may be accumulated and input to a next layer or an output layer. Also, in the convolution layer, convolution operation by different filters may be performed on an input data set, respectively, and a plurality of feature maps obtained through this may be respectively accumulated and input to a next layer or an output layer.

As input data passes through the plurality of convolutional layers, features of the input data may be extracted. The feature of the input data may be extracted based on a plurality of filters used in the plurality of convolutional layers. As for the features of the input data, abstract features obtained based on the plurality of filters may be extracted. A characteristic obtained from the input data based on the plurality of filters may be a core characteristic of the input data related to the diagnosis result.

In addition, a pooling layer may be included between the plurality of convolutional layers. The pooling layer may be used for the feature map extracted by the convolutional layer. The pooling layer may perform a pooling operation on input data. The pooling layer may perform a pooling operation on the feature map to reduce the size of the feature map and remove noise from the feature map. The pooling layer may be performed by a method such as max pooling or average pooling. When the pooling layer is performed through max pooling, noise having a small feature value is removed from the feature map, and accordingly, a key feature displayed on the feature map may be better extracted. In addition, when the pooling layer is performed through average pooling, a feature value is extracted from the feature map as an average value, and thus the overall feature displayed on the feature map can be better extracted. The pooling operation may be used by selecting various calculation methods based on an object to be determined by the model, a result, a classification object, and the like.

According to an embodiment, the diagnosis result may be extracted based on the feature map 510 obtained from the last layer among the plurality of layers 500 included in the learned auxiliary neural network model S4020. For the plurality of feature maps 510 obtained from the last layer, a diagnosis result may be extracted based on the feature map 510 obtained from the last convolutional layer among the plurality of layers 500 . In the plurality of feature maps 510 obtained from the last layer among the plurality of layers 500 , the most abstract feature (or key feature) with respect to a diagnosis disease may be extracted from the input data. In the plurality of feature maps 510 obtained from the last layer, the most abstract features (or core features) with respect to the diagnosis disease are extracted, so that the diagnosis result of the diagnosis disease is derived through the plurality of feature maps 510 . It can be easy.

Feature values obtained from the plurality of feature maps 510 may be first information. The feature values obtained from the plurality of feature maps 510 may be provided to the CAM providing apparatus as first information. In order to consider the abstracted features extracted from the plurality of feature maps 510 , the feature values obtained from the plurality of feature maps 510 may be provided as first information to the CAM providing apparatus. The CAM providing apparatus may provide, as a CAM image, which characteristics of a plurality of pixels constituting input data are characteristics related to the diagnosis disease in relation to the diagnosis disease through the first information.

Also, according to an embodiment, the plurality of feature maps 510 of the last layer may serve as a basis for determining whether a diagnosis result related to a diagnosis disease is abnormal (or suspicious) or normal. The plurality of feature maps 510 of the last layer may go through a classification step of determining abnormality or normality with respect to a diagnosed disease. In order to go through the classification step, a separate process 520 may be performed on the plurality of feature maps 510 . The plurality of feature maps 510 on which the process 520 has been performed may become a plurality of classification data 530 . As for the plurality of classification data 530, the classification data 530 may correspond to the corresponding label based on the weight value (W) of the corresponding label as the classification of the abnormal or normal related to the diagnosis disease is performed. You can get a probability value.

The separate processing 520 may mean passing through a Global Average Pooling (GAP) or a Fully-Connected (FC) layer. When GAP is performed as the separate process 520 , spatial information of features of the plurality of feature maps 510 may be maintained. The plurality of feature maps 510 may obtain GAP data 530 as GAP is performed. The GAP data 530 may determine the diagnosis result, and the GAP data 530 may determine the diagnosis result based on the weight value W for the corresponding label.

As the classification data 530 is classified, the weight value W, which is a basis for determining the corresponding label of the classification data 530 , may be the second information. The weight value W of the classification data 530 may be provided to the CAM providing apparatus as second information. Each weight value W of the plurality of feature maps 510 for a diagnosis disease may be provided to the CAM providing apparatus as second information. The weight value W may represent an important part in determining the label related to the diagnostic disease among the feature values constituting the plurality of feature maps 510. noise can be removed.

FIG. 35 is a diagram for explaining a process of determining whether the classification data 530 corresponds to abnormal, and FIG. 36 is a diagram for explaining a process for determining whether the classification data 530 corresponds to normal am.

35 and 36 , the plurality of feature maps 510 according to an embodiment may include a first feature map 511 , a second feature map 513 , and an n-th feature map 515 . The GAP data 530 may include first GAP data 531 , second GAP data 533 , to n-th GAP data 535 . The first GAP data 531 is obtained from the first feature map 511 as GAP is performed, and the first GAP data 531 is a first abnormal weight value ( W1) can be considered and judged. The second GAP data 533 to the n-th GAP data 535 are also obtained by performing GAP from the second feature map 513 to the n-th feature map 515, respectively, and the second GAP data ( 533) to the nth GAP data 535 may be determined by considering the second abnormal weight value W2 to the nth abnormal weight value Wn, respectively.

In addition, the first GAP data 531 may be determined in consideration of the first normal weight value W1 ′ in order to determine the normality of the diagnosed disease. The second GAP data 533 to the n-th GAP data 535 are also obtained by performing GAP from the second feature map 513 to the n-th feature map 515, respectively, and the second GAP data ( 533) to the n-th GAP data 535 may be determined by considering the second normal weight value W2' to the n-th normal weight value Wn' in order to determine the normality of the diagnosed disease.

Accordingly, as the weight value W is provided to the CAM providing apparatus as second information, the CAM providing apparatus may consider the second information in the first information, so that the CAM providing apparatus determines the diagnosis disease from the input data. It is possible to provide the CAM image 540 from which the key features of the extracted. In addition, the weight value may vary depending on a label to be determined for a diagnosed disease, and the abnormal weight values (W1 to Wn) and the normal weight values (W1' to Wn') for the abnormal are the second information, respectively. can be provided as That is, the CAM providing apparatus may provide the CAM image 540 in the case of abnormality through second information that is abnormal weight values (W1 to Wn) for abnormality together with first information on input data, and , through the second information that is the normal weight values (W1' to Wn') for the normal together with the first information, the CAM image 540 ′ in the case of the normal may be provided.

However, according to an embodiment, the purpose of providing the CAM image is to more easily determine a location where a diagnostic disease occurs, so that the abnormal weight values (W1 to Wn) for the abnormality together with the first information are provided. It may be a main purpose to provide the CAM image 540 for the abnormality through the second information including.

2.2.2.4.2 CAM image output

Fig. 37 is a diagram for explaining a process of outputting a CAM image from input data and information obtained from a diagnosis process;

31 to 37 , the CAM image providing apparatus may provide a CAM image based on at least one of input data, the first information, and the second information. The CAM image may be provided based on at least one of input data, the first information, and the second information. The CAM image may be provided based on a value in consideration of the second information with respect to the feature values included in the first information.

35 to 37 , the CAM image may be displayed in consideration of second information for a plurality of feature maps 510 provided as the first information. The plurality of feature maps 510 may pass through a plurality of layers and may have a smaller size than the input data. According to an embodiment, the CAM image providing apparatus is a second CAM having the same size as input data based on a first CAM image 541 considering the second information in a plurality of feature maps 510 as the first information. An image 549 may be provided.

The CAM image providing apparatus is a first feature in which a first feature map 531 and a first abnormal weight value W1 are considered in order to determine an abnormality related to a diagnosis disease based on the first information and the second information. value, the second feature value considering the second feature map 533 and the second abnormal weight value W2, to the nth feature value considering the nth feature map 535 and the nth abnormal weight value Wn All of these can be considered. The CAM image providing apparatus includes a first feature map 531 and a first abnormal weight value W1 in consideration, a second feature map 533 and a second abnormal weight value W2 in consideration of the first feature value W2. It is possible to determine abnormality regarding a diagnosed disease based on the information on the diagnostic feature map to which all of the 2nd feature values, to the nth feature map 535 and the nth feature value in which the nth abnormal weight value Wn is considered are added. . Even if this is the first information and the second information for judging normal, the contents may correspond.

The first CAM image may include a CAM image 540 in consideration of the abnormal diagnosis result and a CAM image 540' in consideration of the normal diagnosis result. As the first CAM image 541 is provided in consideration of the weight value W according to the corresponding label based on the plurality of feature maps 510, the size may be smaller than that of the input data and the sharpness may be lowered. . Accordingly, the first CAM image 541 may be provided as the second CAM image 549 as additional processing is performed.

According to an embodiment, the first CAM image 541 may be provided based on the feature map 510 , and the diagnostic feature values 543 constituting the feature map may be separated from the first CAM image. can In this case, the separated diagnostic feature values 543 may be displayed in the form of a color spectrum or a heat map based on the value, or may be displayed based on a numerical value. The separated diagnostic feature values 543 may be enlarged to correspond to the size of the input data. The separated diagnostic feature values 543 are enlarged to correspond to the size of the input data, and the enlarged image 545 may be displayed in the form of a color spectrum or a heat map based on the values. The enlarged image 545 may be superimposed on the input data 547 . The image 549 in which the enlarged image 545 is superimposed on the input data 547 may be provided as a second CAM image. The second CAM image may be an image superimposed on the input data 547 after the enlarged image 545 undergoes a process to increase transparency.

Unlike the first CAM image, the second CAM image may display input data clearly, so that a location related to a diagnosis result of a diagnosis disease may be more easily identified on the input data. As the second CAM image is displayed by emphasizing features in consideration of the first information and the second information on the input data, even an unskilled person can easily identify a location related to a diagnosis result of a diagnosed disease.

2.2.2.5 Diagnostic aids

2.2.2.5.1 Definition of Diagnostic Auxiliary Information

As described in 1.3.5, the diagnosis auxiliary information may be information to assist the diagnosis result obtained through the diagnosis auxiliary device. Also, the diagnosis auxiliary information may be information on factors affecting the diagnosis result. Also, the diagnosis auxiliary information may be information on elements related to the diagnosis result. For example, the diagnosis auxiliary information may be information expressed or composed of images, text, numbers, etc., and the information may be provided simultaneously or separately.

According to an embodiment, the auxiliary diagnosis information may include a diagnosis result obtained through the diagnosis auxiliary device and an auxiliary diagnosis image. In addition, the diagnosis auxiliary information may include other information other than the diagnosis result and the diagnosis auxiliary image, and the other information may be factors related to or influencing the diagnosis disease. For example, the diagnosis auxiliary information may include gender, age, smoking status, blood pressure, body information, etc. about a patient, which is an object of input data, in addition to the diagnosis result and the diagnosis auxiliary image. In addition, the diagnosis auxiliary information may be provided with a previously performed diagnosis result and a diagnosis auxiliary image in addition to the diagnosis result and the diagnosis auxiliary image.

2.2.2.5.2 Output of diagnostic auxiliary information

31 to 37 , the diagnosis auxiliary device may output the CAM image S4050 output together or separately with the diagnosis result S4030 output as the diagnosis auxiliary information by the output module 490 (S4060). . The CAM image may be selectively provided only when a diagnosis result is abnormal according to a diagnosis process performed in the diagnosis module. When a diagnosis result is abnormal according to a diagnosis process performed in the diagnosis module, the CAM image may be provided overlapping the diagnosis target data or may be provided separately. In addition, the output of the diagnostic auxiliary information may be the same as the contents described in 1.3.5 or include the contents described in 1.3.5. Also, as in the 1.5 user interface, the diagnostic assistance information may be provided on the user interface.

2.2.3 How to improve the accuracy of CAM images

2.2.3.1 Background and purpose

As described above, the CAM image may be provided based on a diagnosis result for a diagnostic disease, which may be provided together with or separately from the diagnosis result. According to an embodiment, when the diagnosis result is provided for the presence or absence of a lesion, the CAM image additionally considers the progress of a diagnosis disease, so that the accuracy of the CAM image providing apparatus may be further improved. In addition, according to an embodiment, a separate processing for noise removal is additionally performed on the CAM image, so that the CAM image providing apparatus can be more accurate.

2.2.3.2 Multi-binary system

38 is a view for explaining a CAM image providing process according to a diagnosis process in consideration of the degree of progression of a diagnosis disease.

Referring to FIG. 32 , the diagnosis process may be performed by the diagnosis module 450 . Also, referring to FIG. 38 , the diagnosis process includes data acquisition (S4011), a learned first diagnostic auxiliary neural network model to a learned third diagnostic auxiliary neural network model (S4021, S4023, S4025), and outputting a first diagnosis result to a second 3 diagnostic result outputs ( S4031 , S4033 , and S4035 ) and diagnostic auxiliary information output ( S4061 ). The diagnosis process may output a diagnosis result regarding whether or not the diagnosis disease is possessed and the degree of progression as the diagnosis auxiliary information through labeling according to the degree of progression of the diagnosed disease. A diagnosis result obtained in the diagnosis process may be used in a CAM image providing process to be described later.

The learned first diagnostic auxiliary neural network model or the learned third diagnostic auxiliary neural network model S4021 , S4023 , and S4025 may be a model capable of labeling according to the degree of disease progression. According to an embodiment, the learned first diagnosis auxiliary neural network model S4021 may be a model for classifying whether or not it corresponds to the initial stage of the diagnosis disease, and the learned second diagnosis auxiliary neural network model S4023 is It may be a model for classifying whether or not it corresponds to the intermediate stage of the diagnosed disease, and the learned third diagnostic auxiliary neural network model S4025 may be a model for classifying whether or not it corresponds to the terminal stage of the diagnosed disease. As an example, when the same input data is input to the learned first diagnostic auxiliary neural network model to the learned third diagnostic auxiliary neural network model S4021, S4023, and S4025, respectively, in the learned first diagnostic auxiliary neural network model S4021 When a diagnosis is classified as normal for a disease, classified as abnormal in the learned second diagnosis auxiliary neural network model (S4023), and classified as normal in the learned third diagnosis auxiliary neural network model (S4025), the diagnosis process is Diagnosis results corresponding to the middle stage of a diagnostic disease may be provided.

The contents of the diagnosis process other than the above may be the same as those described in 1.3.4 diagnosis process and 2.2.2.3 diagnosis process. However, each process is not essential, and may be omitted or a separate process may be added according to a diagnosis target, a diagnosis disease, provided information, and the like.

Hereinafter, a CAM image that may be provided according to a diagnosis result according to a diagnosis process in consideration of the degree of progression of a diagnosis disease will be described in detail.

Referring to FIG. 34 , the first information and weight values W2, which are the last feature maps 510 by the learned first diagnostic auxiliary neural network model to the learned third diagnostic auxiliary neural network models S4021, S4023, and S4025, respectively. The second information may be provided to the CAM provision process.

According to an embodiment, the input data is classified as abnormal by the learned first auxiliary diagnosis neural network model S4021, and the learned second diagnosis auxiliary neural network model S4023 and the learned third auxiliary diagnosis neural network model S4021 When classified as normal by (S4025), the last feature map 510 determined by the learned first diagnostic auxiliary neural network model (S4021) is an initial stage associated with an abnormality in which the input data corresponds to an initial stage of a diagnosis disease. Features may be present. The learned second diagnostic auxiliary neural network model (S4023) and the learned third diagnostic auxiliary neural network model (S4025) are characteristics of intermediate and terminal stages associated with normality in which the input data does not correspond to the intermediate and terminal stages of a diagnosis disease each may exist. That is, early, intermediate, and late features are compared together with respect to features included in the input data, and the CAM providing process may provide a CAM image only for features meaning an initial stage in consideration of this. The same may be true for other steps.

Diagnosis auxiliary images considering the degree of disease progression can increase the use of CAM images by non-specialists. In addition, the CAM image in consideration of the progress of the diagnosed disease may have improved accuracy compared to the CAM image based on the diagnosis result in which only the presence or absence of the diagnosed disease is determined.

2.2.3.3 Normalization

According to an embodiment, referring to FIGS. 31 to 37 , the CAM image may be provided in consideration of the feature value and the weight value W shown in the last feature map 510 . As an example, in the CAM image providing process, the CAM image may be provided based on the product of the feature value and the weight value W shown in the last feature map 510 . In this case, the CAM image providing apparatus may perform a normalization process based on the maximum and minimum values indicated in the product of the feature value and the weight value W indicated in the last feature map 510 . In the normalization process, the maximum value shown in the product of the feature value and the weight value (W) shown in the last feature map 510 is 1, and the maximum value shown in the product of the feature value and the weight value (W) shown in the last feature map 510 is Normalization may be performed based on the minimum value of 0.

According to an embodiment, in the normalization process, a degree of progression of a diagnostic disease may be considered. For example, in the case of a diagnostic process for classifying normal or abnormal in the terminal stage of a diagnostic disease, the maximum value shown in the product of the feature value and the weight value W shown in the last feature map 510 may be normalized to 1. have. On the other hand, in the case of a diagnostic process for classifying normal or abnormal in the intermediate stage of a diagnosis disease, the maximum value shown in the product of the feature value and the weight value W shown in the last feature map 510 is normalized to a value less than 1. can be performed. Also, the diagnostic process related to the initial stage of the diagnostic disease may be normalized to a value smaller than the maximum value of the diagnostic process in the intermediate stage.

Diagnosis auxiliary images considering the degree of disease progression can increase the use of CAM images by non-specialists. In addition, by visualizing the degree of progression of a diagnosed disease, a non-specialist may increase an understanding of the diagnosed disease.

2.2.3.4 Deconvolution layer

The deconvolution layer may include an unpooling layer. The deconvolution layer may increase the size of the feature map while maintaining the pattern obtained from the feature map to more accurately represent a region in which a feature related to a diagnosis disease is detected on an input image. In addition, as the deconvolution layer, the pooling layer included in the last convolution layer, which is the previous step, and the unpooling layer included in the deconvolution layer are used together, so that the accuracy of the CAM image can be further improved.

According to an embodiment, referring to FIGS. 31 to 38 , the learned diagnosis auxiliary neural network model included in the diagnosis process may further include a deconvolution layer. The deconvolution layer may be disposed after the last convolution layer. The plurality of feature maps 510 of the last convolutional layer perform a convolution operation by the deconvolution layer, but the accuracy can be further improved while the patterns of the plurality of feature maps 510 of the last convolutional layer are maintained.

Through this, the diagnostic assistance device and/or system may improve the accuracy of specifying the lesion site in the CAM image.

2.3 Diagnostic auxiliary image providing device and system

2.3.1 Diagnostic auxiliary image

The diagnosis auxiliary image may be an image suitable for assisting in diagnosis of a disease diagnosable through the fundus image. Also, the diagnosis auxiliary image may be an image suitable for supporting a diagnosis result obtained in a diagnosis process. In addition, the diagnosis auxiliary image may be an image in which at least one of an input fundus image and a CAM image is subjected to a process in which the lesion region of the diagnosis disease is emphasized. In addition, the auxiliary diagnosis image may be an image in which at least one of an input fundus image and a CAM image is processed in consideration of a specific property of the diagnosis disease in relation to a diagnosis disease diagnosed in the diagnosis process. In addition, the diagnosis auxiliary image is a process in which a lesion region of a diagnosis disease is emphasized based on a specific property of the diagnosis disease with respect to at least one of an input fundus image and a CAM image with respect to a diagnosis disease diagnosed in the diagnosis process may be an image on which is performed. In addition, as in the 1.5 user interface, the diagnosis auxiliary image may be provided as diagnosis auxiliary information together with or separately from the diagnosis result on the user interface.

According to an embodiment, the apparatus and system for providing an auxiliary diagnosis image may provide an auxiliary diagnosis image suitable for assisting in the diagnosis of a diagnosable disease through the fundus image in consideration of characteristics of the fundus image used as input data. A disease diagnosable through the fundus image may have disease-specific properties according to the disease. For example, the disease-specific property may include an onset location, an onset form, a lesion shape, and a lesion size. According to an embodiment, the apparatus and system for providing an auxiliary diagnosis image may provide a diagnosis auxiliary image in consideration of specific properties of a diagnosis disease. Also, the apparatus and system for providing an auxiliary diagnosis image may provide an auxiliary diagnosis image in consideration of a region of occurrence of a diagnosis disease. As an example, the apparatus and system for providing a diagnostic auxiliary image may provide a diagnostic auxiliary image on which lesion highlighting is performed in consideration of disease-specific properties of the CAM image provided in the above-described CAM provision process.

2.3.2 Disease-specific properties

2.3.2.1 Area of fundus image

39 is a view for explaining division of a region of a fundus image according to another embodiment of the present invention. According to an embodiment, the fundus image may include a plurality of regions determined in consideration of positions of anatomical elements included in the fundus image.

Referring to FIG. 39 , the fundus image may include a fundus area FA and a non-fundus area NFA. The fundus area FA may include a macula MA, an optic nerve head ODA, a blood vessel BVA, and a peripheral area SA. The fundus image may be divided to include a macula, an optic nerve head, a blood vessel, and a periphery based on the location of the macula and/or the optic nerve head detected from the fundus image.

The macula MA may be determined to include a region in which the macula of the fundus region is distributed. The optic nerve head (ODA) may be determined to include a region in which the optic nerve head of the fundus region is distributed. The blood vessel part BVA may be determined to include a region in which blood vessels of the fundus region are distributed. The peripheral area SA may be determined as an area excluding the macular area MA, the optic nerve head (ODA), and the vascular area (BVA) from the fundus area.

Boundaries of the macula MA, the optic nerve head (ODA), the blood vessel area BVA, and the peripheral area SA may be determined based on positions of the macula, the optic nerve head, blood vessels, and the like. The boundary of the macula MA may be determined to surround an area in which the macula is distributed. A boundary of the optic nerve head (ODA) may be determined to surround a region in which the optic nerve head is distributed. A boundary of the blood vessel portion BVA may be determined to surround a region in which blood vessels are distributed.

2.3.2.2 Diagnosed disease and lesion areas

40 is a diagram illustrating a fundus image in which a plurality of diseases diagnosed by a diagnosis assistance system according to an exemplary embodiment. 40 (a) is a fundus image in which glaucoma has occurred, (b) is a fundus image in which retinopathy has occurred, and (c) is a fundus image in which cataract or opacity has occurred. it has been shown

Referring to FIGS. 39 and 40 ( a ), it can be confirmed that glaucoma occurs centered on the optic nerve head (ODA). In particular, in the optic nerve head (OD, Optic Disk) and the optic nerve cup (OC, Optic Cup) included in the optic nerve head (ODA), glaucoma refers to a case in which the ratio of the optic nerve cup in the optic nerve head is greater than or equal to a predetermined ratio. can Accordingly, in the case of glaucoma, the optic nerve head (ODA) may be specifically affected, and in particular, the ratio of the optic nerve cup (OC) to the optic nerve head (OD) may be an important criterion for determining. That is, the diagnosis assistance system for diagnosing glaucoma may provide a diagnosis assistance image centered on the optic nerve head (ODA).

Referring to FIGS. 39 and 40 (b), it can be seen that retinopathy (retina) is sporadically occurring in the fundus area (FA). It can be seen that it is difficult to see that the disease occurs in a specific area, and the fundus area FA should be considered as a whole. However, considering that the size and location of the retinopathy-infected area are random, it may be necessary to emphasize the area and process it. That is, the diagnosis assistance system for diagnosing retinopathy may provide a diagnosis assistance image for judging the fundus area FA as a whole.

Referring to FIGS. 39 and 40 (c) , it can be confirmed that cataract or opacity occurs in the entire area of the fundus area FA. In the case of a cataract, a blurred area generated on the fundus area FA may be an important criterion for determining, and may be more clearly identified compared to other diagnostic diseases. However, in the case of a fundus image in which a cataract has occurred, a blurred area exists on the fundus area FA, so it may not be easy to determine another diagnostic disease. Therefore, in the fundus image in which the cataract has occurred, other diagnostic diseases included in the blurred area on the fundus area FA may need to be processed by emphasizing the lesion area. That is, the diagnosis assistance system for diagnosing a cataract may provide a diagnosis assistance image in which an area in which another disease occurs in the fundus area FA is emphasized and processed.

The type of lesion and the lesion area for the plurality of diagnostic diseases may not be limited to the above description. For example, a disease in which a lesion region is located in a specific region may have a need similar to glaucoma, and a disease in which a lesion region is located in a non-specific region may have a need similar to retinopathy, and when the fundus image is blurred It may have a need similar to that of a cataract. For this, a diagnosis auxiliary image in consideration of each need may be provided.

2.3.3 Diagnostic auxiliary image provision process

41 is a diagram for explaining the configuration of the diagnostic auxiliary information providing unit 600 according to an exemplary embodiment. Referring to FIG. 41 , the diagnosis auxiliary information providing unit 600 includes a diagnosis request obtaining module 610 , a data processing module 630 , a diagnosis module 650 , a CAM providing module 670 , a lesion highlighting module 680 , and an output module 690 .

42 is a diagram for describing a diagnostic process and a diagnostic auxiliary image providing process performed by a diagnostic apparatus according to an exemplary embodiment.

Each module may perform individual steps, such as a diagnostic auxiliary process and a diagnostic auxiliary image providing process, as described below. However, not all components and functions performed by the components described in FIGS. 41 and 42 are essential, and some components and functions may be added or omitted.

2.3.3.1 Data Acquisition and Processing Process

The data acquisition and processing process may be the same as described above in 1.3.2 Data Acquisition and Diagnosis Request, 1.3.3 Data Processing Process, and 2.2.2.2 Data Acquisition and Processing Process. However, each process is not essential, and may be omitted or a separate process may be added depending on a diagnosis target, a diagnosis disease, and the like.

2.3.3.2 Diagnostic process

41 and 42 , the diagnosis process may be performed by the diagnosis module 650 . The diagnosis process may include data acquisition ( S6010 ), a learned diagnosis auxiliary neural network model ( S6020 ), and outputting a diagnosis result ( S6030 ). In addition, the diagnosis process may output ( S6060 ) the diagnosis auxiliary information through data acquisition ( S6010 ), the learned diagnosis auxiliary neural network model ( S6020 ), and the output of the diagnosis result ( S6030 ). The diagnosis process may output a diagnosis result as the diagnosis auxiliary information. A diagnosis result obtained in the diagnosis process may be used in a diagnosis auxiliary image providing process to be described later.

The contents of the diagnosis process may be the same as those described above in 1.3.4 diagnosis process and 2.2.2.3 diagnosis process. However, each process is not essential, and may be omitted or a separate process may be added according to a diagnosis target, a diagnosis disease, provided information, and the like.

2.3.3.3 CAM image presentation process

The contents of the CAM image providing process may be the same as those described above in 2.2.2.4 CAM image providing process. However, each process is not essential, and may be omitted or a separate process may be added depending on a diagnosis target, a diagnosis disease, and the like.

The CAM image provided through the CAM image providing process may undergo a diagnostic auxiliary information output (S6060) by the output module 690, and the CAM image is output as a diagnostic auxiliary image through the lesion highlighting module 680 additionally. can be The lesion highlighting module 680 and the diagnosis auxiliary image will be described in detail in a process for providing a diagnosis auxiliary image to be described later.

2.3.3.4 Diagnostic Auxiliary Image Delivery Process

Referring to FIG. 41 , the diagnostic auxiliary image providing process may be performed by the lesion highlighting module 680 . The lesion highlighting module 680 is configured to perform a diagnosis auxiliary image providing process based on at least one of the inputted diagnosis target data, the information obtained from the diagnosis module 650, and the CAM image obtained from the CAM providing module 670. can The lesion highlighting module 680 may perform a diagnosis auxiliary image providing process based on the inputted diagnosis target data, the information obtained by the diagnosis module 650 , and the CAM image obtained by the CAM providing module 670 .

Referring to FIG. 42 , the process of providing a diagnostic auxiliary image may include data acquisition ( S6010 ), a CAM image providing model ( S6040 ), a lesion highlighting model ( S6080 ), and outputting a diagnostic auxiliary image ( S6050 ).

The diagnostic auxiliary image providing process may output diagnostic auxiliary information (S6060) through data acquisition (S6010), a CAM image providing model (S6040), a lesion highlighting model (S6080), and an auxiliary diagnostic image output (S6050). In the diagnosis auxiliary image providing process, at least one of a CAM image and a diagnosis auxiliary image may be output as the diagnosis auxiliary information. In the diagnosis auxiliary image providing process, a CAM image and a diagnosis auxiliary image may be output as the diagnosis auxiliary information. The diagnostic auxiliary image providing process is based on at least one of the data obtained in data acquisition (S6010), the result obtained in the learned diagnostic auxiliary neural network model (S6020), and the CAM image obtained in the CAM image providing model (S6040). , a diagnosis auxiliary image may be output as the diagnosis auxiliary information.

In the following description, the process for providing the diagnostic auxiliary image will be described in detail.

2.3.3.4.1 Lesion Highlighting Model

39 to 42 , in the diagnosis auxiliary image providing process according to an exemplary embodiment, lesion emphasis may be performed on the CAM image in consideration of a diagnosis result in the diagnosis process. In addition, the diagnostic auxiliary image providing process includes the diagnosis result of the diagnostic auxiliary neural network model (S6020) learned for a specific diagnostic disease through the lesion highlighting model (S6080), the disease-specific characteristics of the corresponding diagnostic disease, and the CAM image providing model ( A diagnosis auxiliary image in which lesion enhancement is performed in consideration of at least one of the CAM images of S6040) may be provided. In addition, the diagnostic auxiliary image providing process corresponds to the diagnostic auxiliary neural network model (S6020) learned for a specific disease, through the lesion highlighting model (S6080) considering the disease-specific characteristics of the specific disease, the CAM image providing model ( A diagnosis auxiliary image in which lesion enhancement is performed on the CAM image of S6040) may be provided.

According to an embodiment, the lesion enhancement model S6080 may perform lesion enhancement on the CAM image based on pixel values of the CAM image. The lesion enhancement model S6080 may perform lesion enhancement on the CAM image based on a predetermined threshold based on pixel values of the CAM image. As an example, the lesion enhancement model S6080 may perform lesion enhancement that is output only to pixels greater than or equal to a predetermined threshold based on the pixel values of the CAM image. In addition, the lesion enhancement model S6080 may perform lesion enhancement by converting the pixel value of the CAM image to a value of 0 for pixels greater than or equal to a predetermined threshold value based on the pixel values of the CAM image. In addition, the lesion enhancement model S6080 may perform lesion enhancement by superimposing the acquired data on only pixels having a threshold value or higher based on the pixel values of the CAM image. For example, in the lesion enhancement model S6080, lesion enhancement by inserting a boundary line based on a predetermined threshold value based on the pixel value of the CAM image may be performed.

According to an embodiment, in the lesion highlighting model S6080, lesion emphasis may be performed on the CAM image based on a disease-specific property of a diagnosed disease. As an example, in the lesion highlighting model S0680, lesion emphasis may be performed on the CAM image based on the location of the lesion occurrence of the diagnostic disease. In the lesion highlighting model S6080, lesion emphasis may be performed on the CAM image based on an anatomical region corresponding to a lesion occurrence location of the diagnosis disease. In the lesion highlighting model S6080, lesion emphasis may be performed by superimposing an anatomical region corresponding to a lesion occurrence location of the diagnosis disease on the CAM image. In addition, in the lesion highlighting model ( S6080 ), lesion emphasis may be performed by inserting a boundary line of an anatomical region corresponding to a lesion occurrence location of the diagnosis disease into the CAM image. In the lesion highlighting model S6080, lesion highlighting may be performed in which a boundary line of an anatomical region corresponding to a lesion occurrence location of the diagnosis disease is superimposed on the CAM image.

Referring to FIG. 39 , the anatomical region may be one of the regions included in the fundus region FA. For example, the anatomical region may be at least one of the macula MA, the optic nerve head ODA, the vascular region BVA, and the peripheral region SA. In particular, the anatomical region may be an optic nerve head (ODA), a blood vessel (BVA), or a macula (MA).

The lesion enhancement method of the lesion enhancement model S6080 is not limited to the above description, and various methods for outputting the diagnosis auxiliary image may be performed. The lesion highlighting model S6080 may perform lesion highlighting methods according to the diagnosis disease of the diagnosis process, which will be described in detail later.

2.3.3.4.2 Diseases that occur in specific anatomical areas

According to an embodiment, when the diagnosis disease of the diagnosis process has a disease-specific property occurring in a specific anatomical region, the diagnosis auxiliary image providing apparatus may provide a diagnosis auxiliary image in consideration of this.

For example, according to an embodiment, when the diagnosis disease of the diagnosis process is glaucoma, the diagnosis auxiliary image providing apparatus may provide a diagnosis auxiliary image in consideration of disease-specific properties occurring in the optic nerve head (ODA). The lesion-emphasizing model (S060) to be described later will be described as an example when the diagnostic disease is glaucoma, but the diagnostic disease is not limited to glaucoma, and the disease occurring in a specific anatomical region is a lesion-emphasizing model (S6080) to be described later. A diagnostic auxiliary image in which lesion highlighting is performed in the same or similar manner as described above may be provided.

43 to 46 are diagrams illustrating auxiliary diagnostic images when the diagnostic disease of the diagnostic process is glaucoma. 43 to 46 , when the diagnosis disease of the diagnosis process corresponds to glaucoma, the lesion emphasis performed on the CAM image will be described in detail.

Fig. 43 is a view showing a diagnosis auxiliary image when the diagnosis disease of the diagnosis process is glaucoma, Fig. 43 (a) is input data corresponding to glaucoma, Fig. 43 (b) is a CAM image of (a), Fig. 43 (c) is a diagnosis auxiliary image in which (b) is highlighted lesion. According to an embodiment, referring to FIG. 43 , the diagnosis auxiliary image may be an image in which lesion enhancement is performed on the CAM image based on pixel values of the CAM image. The diagnosis auxiliary image may be an image on which lesion enhancement is performed so that only pixels greater than or equal to a threshold value are output based on pixel values of the CAM image. The lesion highlighting based on the pixel value of the CAM image has the advantage that it can be performed simply and quickly. In addition, when the diagnosis disease of the diagnosis process is glaucoma, the lesion emphasis based on the pixel value of the CAM image can be processed by emphasizing the optic nerve head (ODA), so that the user can easily determine whether or not glaucoma is present. have.

44 and 45 are diagrams illustrating an enlarged diagnosis auxiliary image of the optic nerve head (ODA) in the fundus area (FA) when the diagnostic disease of the diagnostic process is glaucoma. 44 shows a diagnosis auxiliary image corresponding to normal when the diagnosis disease of the diagnosis process is glaucoma, and FIG. 45 shows a diagnosis auxiliary image corresponding to suspicious or abnormality. 44 (a) and 45 (a) are input data, FIGS. 44 (b) and 45 (b) are CAM images of (a), and FIGS. 44 (c) and 45 (c) are (b) ) is the lesion-highlighted diagnostic auxiliary image.

44 and 45 , according to an exemplary embodiment, the diagnosis auxiliary image is an image in which the CAM image of the optic nerve head (ODA) of the input data is enlarged and the lesion is emphasized. When the diagnosis disease of the diagnosis process is glaucoma, the diagnosis auxiliary image may be an image in consideration of the fact that the onset location of the glaucoma corresponds to the optic nerve head (ODA). In addition, when the diagnosis disease of the diagnosis process is glaucoma, the diagnosis auxiliary image may be an image in which the optic disc OD and the optic disc OC located in the optic disc ODA are emphasized. For example, the diagnosis auxiliary image may be an image in which lesion enhancement is performed so that only pixels greater than or equal to a threshold are output based on pixel values of a CAM image with respect to the enlarged image of the optic nerve head (ODA). As described above, the auxiliary diagnosis image is a diagnosis auxiliary image of a CAM image with respect to an enlarged image of the optic nerve head, and the user can easily determine the ratio of the optic nerve head (OD) and the optic nerve cup (OC) through the diagnosis auxiliary image. can

46 is a view showing a diagnosis auxiliary image when the diagnosis disease of the diagnosis process is glaucoma, FIG. 46 (a) is input data corresponding to glaucoma, FIG. 46 (b) is a CAM image of (a), 46 (c) is a diagnosis auxiliary image in which (b) is highlighted lesion. When the diagnosis disease of the diagnosis process is glaucoma, the diagnosis auxiliary image may be an image processed by emphasizing the optic nerve head (ODA). The diagnosis auxiliary image may be an image in which the boundary lines of the optic nerve head (OD) and the optic nerve cup (OC) located in the optic nerve head (ODA) are processed in the CAM image. The diagnosis auxiliary image may be an image in which the boundary lines of the optic nerve head (OD) and the optic nerve cup (OC) located in the optic nerve head (ODA) overlap with respect to the CAM image. A boundary line between the optic nerve head OD and the optic nerve cup OC may be output from input data. In the diagnosis auxiliary image, the boundary lines of the optic nerve head (OD) and the optic nerve cup (OC) overlap in the CAM image, so that the user can easily determine glaucoma through the diagnosis auxiliary image.

When the diagnosis disease of the diagnosis process is glaucoma, the lesion highlighting method performed on the input data or the CAM image is not limited to the above description, and various methods may be performed in consideration of the disease-specific nature of the glaucoma.

When the diagnosis disease of the diagnosis process is glaucoma, the auxiliary diagnosis image may be an image on which lesion highlighting is performed in consideration of disease-specific properties of glaucoma. The auxiliary diagnosis image can be processed by emphasizing the optic nerve head (ODA) through the above-described lesion emphasis, so that the auxiliary diagnosis image can assist in diagnosing glaucoma more easily and clearly compared to the CAM image. In addition, since the optic nerve head (ODA) corresponds to some of the input data, it is somewhat difficult for a non-specialist to diagnose glaucoma by looking only at the input data or the CAM image. Accordingly, the diagnosis auxiliary image emphasizing the optic nerve head (ODA) in which the lesion is emphasized may assist a non-specialist in diagnosing glaucoma through the diagnosis auxiliary image.

2.3.3.4.3 Diseases occurring in non-specific areas

According to an embodiment, when the diagnosis disease of the diagnosis process has a disease-specific property that occurs in a non-specific region in the fundus region (FA), the diagnosis auxiliary image providing apparatus may provide a diagnosis auxiliary image in consideration of this. .

47 to 50 are diagrams illustrating auxiliary diagnostic images when a diagnostic disease of the diagnostic process occurs in a non-specific region. With reference to FIGS. 47 to 50 , when the diagnostic disease of the diagnostic process occurs in a non-specific region, lesion highlighting in progress with respect to the CAM image will be described in detail.

47 is a diagram illustrating input data when a diagnostic disease of the diagnostic process occurs in a non-specific region.

Referring to (a) of FIG. 47 , hemorrhage may be diagnosed based on the fundus image. As an example, as shown in (a), bleeding L1 may be detected from any location in the entire fundus region.

Referring to FIG. 47B , drusen may be diagnosed based on the fundus image. As an example, as shown in (b), the drusen L2 may be found from any position in the entire fundus region.

Referring to FIG. 47C , a macular hole may be diagnosed based on the fundus image. As an example, as shown in (c), macular perforation findings (L3) may be found from the macula of the fundus image.

Referring to (d) of FIG. 47 , the macular epiretinal membrane may be diagnosed from the fundus image. As an example, as shown in (d), an anterior macular finding (L4) may be found near the macula of the fundus image. Anterior macular findings can be found from the macula of the fundus image.

Referring to (e) of FIG. 47 , myelinated nerve fiber symptoms can be diagnosed from the fundus image. As an example, as shown in FIG. (e), findings of myelinated nerve fibers (L5) can be found from any region of the fundus image.

Referring to FIG. 47 , when a diagnostic disease of the diagnostic process occurs in a non-specific region in the fundus region (FA), it is necessary to provide a diagnostic auxiliary image highlighting the disease occurrence region. In addition, in the case of a disease occurring in the macular region (MA), since the region of the macular region (MA) is relatively wide and the boundary is vague, the lesion highlighting is performed in the same or similar way as the disease occurring in the non-specific region. can

48 is a diagram illustrating a CAM image of FIG. 47 . 49 and 50 are diagrams illustrating images in which lesion enhancement is performed on the CAM image of FIG. 48 .

According to an embodiment, with reference to FIGS. 48 and 49 , when the diagnosis disease of the diagnosis process occurs in a non-specific region, the diagnosis auxiliary image highlights the lesion on the CAM image based on the pixel value of the CAM image. may be an image on which is performed. Also, the diagnosis auxiliary image may be an image on which lesion enhancement is performed so that only pixels greater than or equal to a threshold value are output based on the pixel values of the CAM image. The lesion highlighting based on the pixel value of the CAM image has the advantage that it can be performed simply and quickly.

According to an embodiment, referring to FIGS. 48 and 50 , when a diagnostic disease of the diagnostic process occurs in a non-specific region, the diagnosis auxiliary image emphasizes a lesion that inserts a boundary line of the disease occurrence region in the CAM image can be performed. In addition, in the diagnosis auxiliary image, lesion emphasis may be performed on the CAM image to overlap the boundary line of the disease occurrence region. The boundary line of the disease occurrence region may be output from a predetermined pixel value of the CAM image or input data.

When the diagnostic disease of the diagnostic process occurs in a non-specific region, it may be difficult for a non-specialist to identify and diagnose the occurrence region of the diagnostic disease from input data. In addition, when the diagnostic disease of the diagnostic process occurs in a non-specific region, it may be difficult for a non-specialist to identify and diagnose the occurrence region of the diagnostic disease in the CAM image. Accordingly, the diagnosis auxiliary image on which the lesion emphasis is performed emphasizing the occurrence region of the diagnosis disease may assist a non-specialist in diagnosing the diagnosis disease.

2.3.3.4.4 Diagnostic auxiliary image for multiple diseases

According to an embodiment, when the input data of the diagnosis process includes a plurality of diseases, the diagnosis auxiliary image providing apparatus may provide a diagnosis auxiliary image in consideration of the plurality of diseases. For example, when the input data of the diagnosis process includes a plurality of diseases, the apparatus for providing an auxiliary diagnosis image may provide each of the auxiliary diagnosis images for a region of occurrence of each disease.

For example, according to an embodiment, when the diagnosis disease of the diagnosis process is cataract, the apparatus for providing an auxiliary diagnosis image may provide input data as an output image without additional processing. Also, when the diagnosis disease of the diagnosis process is cataract, the diagnosis auxiliary image providing apparatus may output an image in which lesion emphasis is performed on input data without a CAM image. Also, when the diagnosis disease of the diagnosis process is a disease other than a cataract, the diagnosis auxiliary image providing apparatus may output an image in which lesion emphasis is performed on input data having a cataract. The apparatus for providing an auxiliary diagnosis image may provide a diagnosis auxiliary image processed in consideration of a disease-specific property in which a cataract is generated in a blurred area on the fundus area FA.

The lesion highlighting model S060 to be described later will be described as an example when the input data has a cataract. A diagnosis auxiliary image in which lesion enhancement is performed in the same or similar method as the lesion enhancement model to be used ( S6080 ) may be provided.

51 and 52 are diagrams illustrating auxiliary images for diagnosis when the input data of the diagnosis process has a cataract. With reference to FIGS. 51 and 52 , when the input data of the diagnosis process has a cataract, lesion highlighting performed with respect to the input data will be described in detail.

Fig. 51 (a) shows input data having a cataract, and Fig. 51 (b) is a view showing the vascular part (BVA) in (a) with emphasis. Referring to FIG. 51 ( a ) , the fundus area FA of the input data having a cataract may be partially covered by the cataract generation area L6 , so it may not be easy to determine other diagnostic diseases. Referring to FIG. 51 ( b ) , the diagnosis auxiliary image may be an image in which the vascular part BVA is separated from the input data and the lesion is emphasized on the cataract generation region L6 . Through this, the diagnosis auxiliary image provides information on the vascular part BVA, which was not easy to determine due to the cataract generation region L6, so that the diagnosis of other diagnostic diseases related to the vascular part BVA can be aided. can In addition, considering that it is not easy for a non-specialist to judge other diagnostic diseases due to the cataract generation region L6, the diagnostic auxiliary image emphasizing the vascular area (BVA) shows that the non-specialist can diagnose diseases other than cataract There may be an effect that can be easily identified and diagnosed.

Fig. 52 (a) shows input data having all of glaucoma, retinopathy, and glaucoma, Fig. 52 (b) is a view showing the glaucoma occurrence region (L7) in (a) with emphasis, and Fig. 52 (c) is In (a), the retinopathy occurrence region (L8) is emphasized.

Referring to FIG. 52 (a), the fundus area FA of the input data having a cataract is at least one of the glaucoma occurrence area L7 and the retinopathy occurrence area L8 by the cataract generation area L6. may not be easily understood. Accordingly, an expert or non-specialist can grasp the diagnosis of cataract by the cataract generation region L6, but it may be difficult to grasp the diagnosis of glaucoma or retinopathy. In addition, an expert or non-specialist can easily grasp the diagnosis result for cataract by the cataract generation region L6, but it may be difficult to easily grasp the diagnosis result for glaucoma or retinopathy.

Referring to FIG. 52 ( b ) , in the diagnosis auxiliary image, the boundary line of the glaucoma occurrence region L7 or the optic nerve head ODA is inserted on the cataract occurrence region L6 of the fundus region FA of the input data or Overlapping lesion highlighting may be performed. The boundary line may be separated or derived from the input data. The diagnosis auxiliary image may assist in diagnosing not only cataract but also glaucoma through the boundary line on the cataract generation region L6.

Referring to FIG. 52 ( c ), the diagnosis auxiliary image has a lesion emphasis in which the boundary line of the retinopathy occurrence region L8 is inserted or overlapped on the retinopathy occurrence region L8 of the fundus region FA of the input data. can be performed. The boundary line may be separated or derived from the input data. The diagnosis auxiliary image may assist in diagnosing not only a cataract but also retinopathy through the boundary line on the cataract generation region L6.

The diagnosis auxiliary image may assist in diagnosis even for diseases other than the aforementioned glaucoma and retinopathy by highlighting a separate lesion on the cataract generation region L6 according to the corresponding diagnosis disease. In addition, in the diagnosis auxiliary image, lesion emphasis may be performed on the CAM image as well as the input data according to the degree of progression of the cataract. In addition, the diagnosis auxiliary image is not limited to the above-described glaucoma, retinopathy, and cataract, and lesion emphasis may be performed on input data having a plurality of diagnostic diseases by the same or similar method as the above-described method. In addition, the diagnosis auxiliary image is not limited to the above-described method, and lesion highlighting may be performed by a separate method in consideration of the disease-specific properties of the diagnosis disease.

2.3.3.5 Diagnostic Auxiliary Image Output

The diagnostic auxiliary image providing apparatus may output a CAM image or a diagnostic auxiliary image as diagnostic auxiliary information. The diagnosis auxiliary image providing apparatus may output at least one of a CAM image and a diagnosis auxiliary image as diagnosis auxiliary information. The diagnostic auxiliary image providing apparatus may output a CAM image and a diagnostic auxiliary image as diagnostic auxiliary information. For the same input data, the CAM image and the diagnosis auxiliary image may have different regions at least in part according to a diagnosis disease of the diagnosis process.

The CAM image and the diagnosis auxiliary image may be output as diagnosis auxiliary information for recognizing a diagnosis result of a diagnosis disease of the diagnosis process. The CAM image and the diagnosis auxiliary image may be output as diagnosis auxiliary information in consideration of a disease-specific property of a diagnosis disease of the diagnosis process with respect to the same input data. The diagnosis auxiliary image may assist in recognizing a diagnosis disease more easily by highlighting the lesion on the input data or the CAM image.

2.3.3.6 How to provide diagnostic auxiliary images

Hereinafter, a method for providing a diagnostic auxiliary image according to an exemplary embodiment will be described. 53 is a flowchart illustrating a method of providing a diagnostic auxiliary image. Referring to FIG. 53 , the method for providing an auxiliary diagnosis image includes a data acquisition step (S610), a diagnosis execution step of the acquired data (S620), a diagnosis result output step (S630), a CAM image generation step (S640), and a diagnostic auxiliary image It may include a generating step S650 and a diagnostic auxiliary image outputting step S660 . The method for providing the diagnostic auxiliary image is not limited to the order shown in FIG. 53 , and steps other than the above steps may be included.

In the data acquisition step ( S610 ), data may be acquired as the fundus image of the object is input. In addition, in the data acquisition step ( S610 ), data may be acquired as a pre-processed fundus image is input in order to perform a diagnosis. In addition, in the data acquisition step ( S610 ), data is acquired as the fundus image of the object is input, and additional pre-processing may be performed accordingly.

In the step of performing the diagnosis of the acquired data ( S620 ), the diagnosis of the data acquired in the step of acquiring the data ( S610 ) may be performed. The diagnosis of the acquired data step ( S620 ) may include a learned neural network model, and the diagnosis of the data acquired in the data acquisition step ( S610 ) may be performed by the learned neural network model. In addition, the learned neural network model may be a neural network model in which learning for a diagnosis disease of the method for providing a diagnostic auxiliary image is performed, and the learned neural network model may be one or a plurality of trained neural network models. In the step of performing the diagnosis of the acquired data ( S620 ), a diagnosis result according to the diagnosis performed by the learned neural network model may be provided, and a plurality of pieces of information used in the diagnosis result may also be provided.

In the step of outputting the diagnosis result ( S630 ), the diagnosis result provided in the step of performing the diagnosis of the acquired data ( S620 ) may be output. In the step of outputting the diagnosis result ( S630 ), whether the diagnosis disease is normal or abnormal may be output through the probability value that is the diagnosis result provided in the diagnosis execution step ( S620 ) of the acquired data. In addition, in the step of outputting the diagnosis result ( S630 ), the degree of progression of the diagnosis disease may be output through the probability value that is the diagnosis result provided in the step ( S620 ) of performing the diagnosis of the acquired data.

In the generating the CAM image ( S640 ), a CAM image may be generated based on at least one of a diagnosis result provided in the diagnosis execution step ( S620 ) of the acquired data and a plurality of pieces of information used in the diagnosis result. In the CAM image generating step (S640), CAM images processed with different colors may be generated according to the degree of being associated with the diagnosis based on the information obtained in the diagnosis performing step (S620) of the acquired data.

The generating of the auxiliary diagnosis image ( S650 ) may include at least one of a diagnosis result provided in the diagnosis performing step ( S620 ) of the acquired data, a plurality of pieces of information used in the diagnosis result, and a CAM image obtained in the CAM image generating step ( S620 ). A diagnosis auxiliary image may be generated based on the . In the generating of the auxiliary diagnosis image ( S650 ), the auxiliary diagnosis image in which lesion emphasis processing is performed on the CAM image or the input fundus image of the object may be generated. In the generating the diagnosis auxiliary image ( S650 ), a diagnosis auxiliary image in which a disease-specific lesion highlighting process is performed on the CAM image or an input fundus image of an object may be generated based on the characteristics of the diagnosis disease.

In the step of outputting the auxiliary diagnosis image ( S660 ), the auxiliary diagnosis image may be output. In the step of outputting the auxiliary diagnosis image ( S660 ), the auxiliary diagnosis image may be processed and output in a size corresponding to or the same as the input fundus image of the object. In the step of outputting the auxiliary diagnosis image ( S660 ), in addition to the auxiliary diagnosis image, at least one of the inputted fundus image of the object and the CAM image may be output together. In the step of outputting the auxiliary diagnosis image ( S660 ), the auxiliary diagnosis image is output and provided to be superimposed on the inputted fundus image of the subject, and the outputted auxiliary diagnosis image is provided separately from the inputted fundus image of the subject. It can be provided as work is performed. For example, the outputted diagnosis auxiliary image may be provided as a user moves a mouse cursor over or clicks on the input fundus image of the object.

Also, in the method for providing an auxiliary diagnosis image, the auxiliary diagnosis information obtained in the step of outputting the diagnosis result ( S630 ) and the step of outputting the auxiliary diagnosis image ( S660 ) may be provided. The auxiliary diagnosis information may include the diagnosis result obtained in the step of outputting the diagnosis result ( S630 ) and the auxiliary diagnosis image obtained at the output of the auxiliary diagnosis image ( S660 ). In the method for providing an auxiliary diagnosis image, a diagnosis auxiliary image for the input fundus image is provided, and the diagnosis result obtained in the step of outputting the diagnosis result ( S630 ) may be provided together.

The description of the method for providing the auxiliary diagnosis image is not limited thereto, and the method for providing the auxiliary diagnosis image is described in FIGS. 1 to 53 and 2.3.3. It may include the contents described in the process of providing a diagnostic auxiliary image.

2.4 Diagnosis auxiliary image providing system for multiple diseases

The diagnosis assistance system for a plurality of diseases may include an apparatus for providing an auxiliary diagnosis image using the above-described fundus image. The system for providing an auxiliary diagnosis image for a plurality of diseases may include an apparatus for providing an auxiliary diagnosis image using the above-described fundus image. A system for providing an auxiliary diagnosis image for a plurality of diseases using an fundus image may include a diagnosis auxiliary image providing apparatus in consideration of disease-specific characteristics of a diagnosis disease in parallel. In diagnosing a plurality of diseases with respect to the same fundus image, the diagnostic auxiliary image providing system for the plurality of diseases may include a diagnosis auxiliary image providing apparatus in consideration of disease-specific characteristics of the diagnostic diseases in parallel.

The system for providing a parallel diagnosis auxiliary image for multiple diseases is not limited to the contents to be described later, and a configuration in which a lesion highlighting module and a lesion highlighting process are added is added to the contents described in 1.4 A diagnosis assistance system for multiple labels described above may be the same as or similar to In the following description, the lesion highlighting module and the lesion highlighting process will be mainly described with respect to a system for providing a parallel diagnosis auxiliary image for a plurality of diseases.

2.4.1 Example 1 - Parallel diagnostic auxiliary image providing system

54 is a diagram illustrating a configuration of a parallel diagnosis auxiliary information providing unit 700 for a fundus image according to an exemplary embodiment. Referring to FIG. 54 , the diagnosis auxiliary information providing unit 700 includes a diagnosis acquisition module 710 , a data processing module 720 , a first diagnosis module 730 , a second diagnosis module 740 , and a first CAM providing module. 750 , a second CAM providing module 760 , a first lesion highlighting module 770 , a second lesion highlighting module 780 , and an output module 790 .

FIG. 55 is a diagram for explaining a diagnosis process performed by the diagnosis auxiliary information providing unit 700 of FIG. 54 and a diagnosis auxiliary image providing process. Referring to FIG. 55 , the diagnostic process and the diagnostic auxiliary image providing process include data acquisition (S7010), a learned first diagnostic auxiliary neural network model (S7020), a learned second diagnostic auxiliary neural network model (S7030), and providing a first CAM It may include a model S7040, a second CAM providing model S7050, a first lesion emphasis model S7060, a second lesion emphasis model S7070, and an output of diagnostic auxiliary information (S7080).

Each module may perform individual steps, such as a diagnostic auxiliary process and a diagnostic auxiliary image providing process, as described below. However, not all components and functions performed by the components described in FIGS. 54 and 55 are essential, and some components and functions may be added or omitted.

54 and 55 , the data acquisition and processing process may be the same as described above in 1.3.2 Data Acquisition and Diagnosis Request, 1.3.3 Data Processing Process, and 2.2.2.2 Data Acquisition and Processing Process can In addition, each process is not essential, and may be omitted or a separate process may be added according to a diagnosis target, a diagnosis disease, and the like. However, in the processing process, different processing may be appropriately performed on the same input data in consideration of the disease-specific properties of each disease diagnosed by the parallel diagnosis process. For example, when a disease diagnosed by at least some of the parallel diagnosis processes is diagnosed based on a blood vessel in the fundus image, the processing process may be performed through a filter or a separate processing in which the blood vessel of the input data is emphasized. can

54 and 55 , the diagnosis process may be performed by a first diagnosis module 730 and a second diagnosis module 740 . The diagnosis process may include data acquisition ( S7010 ), a learned first diagnosis auxiliary neural network model ( S7020 ), a learned second diagnosis auxiliary neural network model ( S7030 ), and outputting diagnosis auxiliary information ( S7080 ). The diagnosis process may output a diagnosis result as the diagnosis auxiliary information. A diagnosis result obtained in the diagnosis process may be used in a diagnosis auxiliary image providing process to be described later. The contents regarding the diagnosis process may be the same as those described above in 1.3.4 diagnosis process and 2.2.2.3 diagnosis process. In addition, each process is not essential, and may be omitted or a separate process may be added according to a diagnosis target, a diagnosis disease, provided information, and the like.

54 and 55 , the diagnostic auxiliary image providing process includes a first CAM providing module 750 , a second CAM providing module 760 , a first lesion highlighting module 770 , and a second lesion highlighting module ( 780) can be performed. In the diagnosis auxiliary image providing process, each diagnosis auxiliary image corresponding to diagnosis auxiliary information on a plurality of diseases may be provided. In the process of providing the auxiliary diagnosis image, each auxiliary diagnosis image may be provided together with the diagnosis result corresponding to the diagnosis auxiliary information for a plurality of diseases.

For example, in the process of providing a diagnostic auxiliary image, a first diagnostic auxiliary neural network model (S7020), a first CAM provision model (S7040), and a first lesion emphasis are learned for a first diagnostic disease based on the acquired data (S7010) A second diagnostic auxiliary neural network model ( S7030 ), a second CAM, which may include a model ( S7060 ) and an output of diagnostic auxiliary information ( S7080 ), and learned for a second diagnostic disease based on the acquired data ( S7010 ) It may include a providing model ( S7050 ), a second lesion highlighting model ( S7070 ), and an output of auxiliary diagnosis information ( S7080 ). Accordingly, in the process of providing an auxiliary diagnosis image, lesion highlighting in consideration of the disease-specific properties of each diagnosis disease is performed on the CAM image formed based on the diagnosis result performed in the parallel diagnosis process supporting diagnosis of a plurality of diseases. can

For example, when the first diagnostic disease occurs in a specific anatomical region and the second diagnostic disease occurs in a non-specific anatomical region, the diagnostic auxiliary image providing process is performed in the first CAM providing model S7040 Lesion enhancement associated with a specific anatomical region may be performed on the provided CAM image through the first lesion enhancement model S7060. Simultaneously or sequentially, the diagnostic auxiliary image providing process is performed on the CAM image provided by the second CAM providing model (S7050) through the second lesion highlighting model (S7070). can

The diagnosis auxiliary image providing process provides a diagnosis auxiliary image to assist the diagnosis result of each diagnosis disease performed in the diagnosis process, wherein the diagnosis auxiliary image is capable of highlighting lesions in consideration of the disease-specific properties of each diagnosis disease. Thus, the diagnosis auxiliary image has an effect of easily matching the input data and the diagnosis result. In addition, the process of providing an auxiliary diagnosis image includes a first lesion emphasizing model corresponding to a first diagnosis disease and a second lesion emphasizing model corresponding to a second diagnosis disease, and thus emphasizing the first lesion emphasizing model and the second lesion emphasizing model The model can be performed disease-specific, saving time for performing each lesion highlighting.

2.4.2 Example 2 - Parallel diagnostic auxiliary image providing system

56 is a diagram illustrating the configuration of a parallel diagnosis auxiliary information providing unit 800 for a fundus image according to an exemplary embodiment. Referring to FIG. 56 , the diagnosis auxiliary information providing unit 800 includes a diagnosis acquisition module 810 , a data processing module 820 , a first diagnosis module 830 , a second diagnosis module 840 , and a first CAM providing module. 850 , a second CAM providing module 860 , a lesion highlighting module 870 , and an output module 890 .

57 is a diagram for explaining a diagnosis process and a diagnosis auxiliary image providing process performed by the diagnosis auxiliary information providing unit 800 of FIG. 56 . Referring to FIG. 57 , the diagnostic process and the diagnostic auxiliary image providing process include data acquisition (S8010), a learned first diagnostic auxiliary neural network model (S8030), a learned second diagnostic auxiliary neural network model (S8040), and providing a first CAM It may include a model (S8050), a second CAM provision model (S8060), a lesion highlighting model (S8070), and an output of diagnostic auxiliary information (S8090).

Each module may perform individual steps, such as a diagnostic auxiliary process and a diagnostic auxiliary image providing process, as described below. However, not all components and functions performed by the components described in FIGS. 55 and 56 are essential, and some components and functions may be added or omitted.

The system for providing a diagnosis auxiliary image according to an embodiment may be the same as or similar to the contents described in Embodiment 1 described above, and there may be some differences in the contents regarding the lesion highlighting model in the diagnosis auxiliary image providing process, so that the lesion I would like to focus on the differences between the emphasis models.

56 and 57 , the diagnostic auxiliary image providing process may be performed by the first CAM providing module 850 , the second CAM providing module 860 , and the lesion highlighting module 870 .

The diagnostic auxiliary image providing process includes a first diagnostic auxiliary neural network model (S8030), a first CAM provision model (S8050), a lesion highlighting model (S8070), learned for a first diagnostic disease based on the acquired data (S8010); and a diagnostic auxiliary information output (S8090), a second diagnostic auxiliary neural network model (S8040), a second CAM providing model (S8060), learned for a second diagnostic disease based on the acquired data (S8010), It may include a lesion highlighting model (S8070) and an output of diagnostic auxiliary information (S8090). Accordingly, in the diagnosis auxiliary image providing process, lesion emphasis may be performed on each CAM image formed based on a diagnosis result performed in a parallel diagnosis process supporting diagnosis of a plurality of diseases.

In the diagnostic auxiliary image providing process, lesion enhancement is performed by the lesion enhancement model (S8070) in which the CAM image output from the first CAM image providing model and the CAM image output from the second CAM image providing model are common. , the procedure and execution time by the lesion highlighting model (S8070) can be shortened. In the diagnosis auxiliary image providing process, the same lesion emphasis is performed on the CAM image output from the first CAM image providing model and the CAM image output from the second CAM image providing model through the lesion enhancement model (S8070), or Disease-specific lesion highlighting according to the diagnosis disease may be performed, respectively.

For example, the lesion enhancement model (S8070) is output only for pixels each having a predetermined threshold value with respect to the CAM image output from the first CAM image providing model and the CAM image output from the second CAM image providing model. Lesion enhancement may be performed, lesion enhancement may be performed to insert a boundary line based on pixels each having a predetermined threshold value, or lesion enhancement may be performed to insert a boundary line of a specific anatomical region, respectively. In this case, the procedure for providing the diagnosis auxiliary image and the execution time by the lesion highlighting model S8070 may be shortened.

As an example, the lesion enhancement model (S8070) may perform lesion enhancement by a plurality of methods, so that the CAM image output from the first CAM image providing model and the CAM image output from the second CAM image providing model are applied. Disease-specific lesion highlighting can be performed for In this case, the auxiliary diagnosis image providing process may provide a diagnosis auxiliary image suitable for a diagnosis disease while simplifying the overall model configuration.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (22)

A diagnostic auxiliary image providing apparatus for providing a diagnostic auxiliary image related to a target disease based on an eyeball image, the apparatus comprising:
A neural network model unit for obtaining diagnostic assistance information related to the target disease based on the eye image - The neural network model unit includes a neural network model learned based on a plurality of eye images, wherein the target disease includes a first disease obtaining first auxiliary diagnosis information related to the first disease, and obtaining second diagnosis auxiliary information related to the second disease when the target disease includes a second disease;
a characteristic map acquisition unit configured to acquire a characteristic map related to the diagnosis auxiliary information from the neural network model unit;
a lesion highlighting unit configured to visually emphasize a lesion-related region based on the characteristic map to obtain a lesion-emphasized image; and
A diagnosis auxiliary image providing unit for providing a diagnosis auxiliary image based on at least one of the eyeball image, the characteristic map, and the lesion emphasis image;
The neural network model unit includes a first neural network model for obtaining the first diagnostic auxiliary information or a second neural network model for obtaining the second diagnostic auxiliary information,
The characteristic map acquisition unit is configured to acquire a first characteristic map related to the first disease when the target disease includes a first disease, and a second characteristic map related to the second disease when the target disease includes a second disease Acquire a characteristic map,
The lesion highlighting unit may be configured to, when the target disease includes the first disease, perform a first lesion highlighting on the first characteristic map to obtain a first lesion highlighting image, and if the target disease includes the second disease If included, performing a second lesion enhancement on the second characteristic map to obtain a second lesion highlighting image,
wherein the first lesion highlighting comprises modifying the first characteristic map based on a first anatomical region associated with the first disease;
The second lesion highlighting includes extracting pixels having a pixel value equal to or greater than a reference value from the second characteristic map,
Diagnostic auxiliary image providing device.
According to claim 1,
The first disease is a disease having a pathological property occurring in a specific anatomical region,
The second disease is a disease having pathological properties occurring in a non-specific area,
Diagnostic auxiliary image providing device.
According to claim 1,
The first disease is glaucoma, and the second disease is any one of hemorrhage, drusen, macular perforation, anterior macular membrane, lens opacity, diabetic retinopathy, and myelinated neurofibrosis,
Diagnostic auxiliary image providing device.
According to claim 1,
The neural network model unit further comprises a segmented neural network model that divides the eyeball region of the eyeball image into one or more anatomical regions, wherein the one or more anatomical regions include the first anatomical region;
The emphasizing the first lesion includes emphasizing one or more pixels included in a first anatomical region related to the first disease among pixels included in the first characteristic map.
Diagnostic auxiliary image providing device.
According to claim 1,
The second lesion emphasizing includes emphasizing a pixel greater than or equal to the reference value among pixels included in the second characteristic map.
Diagnostic auxiliary image providing device.
According to claim 1,
The neural network model unit further comprises a segmented neural network model that divides the eyeball region of the eyeball image into one or more anatomical regions, wherein the one or more anatomical regions include the first anatomical region;
The first lesion highlighting includes superimposing a boundary of the first anatomical region on the eye image and a heat map included in the first characteristic map and related to the first diagnostic auxiliary information.
Diagnostic auxiliary image providing device.
According to claim 1,
The second lesion emphasis includes superimposing a boundary composed of pixels having the reference value among the pixel values of the characteristic map on the eye image.
Diagnostic auxiliary image providing device.
According to claim 1,
The neural network model unit further comprises a segmented neural network model that divides the eyeball region of the eyeball image into one or more anatomical regions, wherein the one or more anatomical regions include the first anatomical region;
Acquiring the second lesion-emphasized image includes extracting a lesion pixel whose pixel value is equal to or greater than the reference value, and obtaining an eye image in which a boundary of the anatomical region in which the lesion pixel is located is displayed and the lesion pixel is emphasized including that,
Diagnostic auxiliary image providing device.
According to claim 1,
The diagnosis auxiliary information obtained by the neural network model unit includes normal information indicating that the subject is normal for the target disease or abnormal information indicating that the subject is abnormal with respect to the target disease,
The characteristic map acquisition unit acquires the characteristic map indicating a region related to the abnormal information of the eyeball image,
wherein the lesion highlighting unit acquires the lesion highlighting image based on the characteristic map indicating a region related to the abnormality information;
Diagnostic auxiliary image providing device.
According to claim 1,
When the target disease includes the first disease and the second disease, the characteristic map acquiring unit acquires the first characteristic map and the second characteristic map, and the lesion highlighting unit emphasizes the first lesion and the second characteristic map. 2 acquiring a third lesion highlighting image in which the first disease and the second disease-related characteristic are displayed based on the lesion emphasis;
Diagnostic auxiliary image providing device.
A diagnostic auxiliary image providing apparatus for providing a diagnostic auxiliary image related to a target disease based on an eyeball image, the apparatus comprising:
a data acquisition unit for acquiring an eye image;
A neural network model unit for obtaining diagnostic assistance information related to the target disease based on the eye image - The neural network model unit includes a neural network model learned based on a plurality of eye images, wherein the target disease includes a first disease obtaining first auxiliary diagnosis information related to the first disease, and obtaining second diagnosis auxiliary information related to the second disease when the target disease includes a second disease;
a characteristic map acquisition unit configured to acquire a characteristic map related to the diagnosis auxiliary information;
a lesion highlighting unit configured to visually emphasize a lesion-related region based on the characteristic map to obtain a lesion-emphasized image; and
A diagnosis auxiliary image providing unit for providing a diagnosis auxiliary image based on at least one of the eyeball image, the characteristic map, and the lesion emphasis image;
The neural network model unit further includes a segmented neural network model that divides the eyeball region of the eyeball image into one or more anatomical regions, wherein the one or more anatomical regions include a first anatomical region and a second anatomical region, and ,
The characteristic map acquisition unit acquires a first characteristic map related to the first disease when the target disease includes the first disease, and obtains a first characteristic map associated with the target disease when the target disease includes the second disease. acquiring a related second characteristic map,
The lesion highlighting unit may be configured to, when the target disease includes the first disease, perform a first lesion highlighting on a first characteristic map related to the first diagnosis auxiliary information to obtain a first lesion highlighting image, When the disease is the second disease, a second lesion highlighting image is obtained by performing a second lesion emphasis on the first characteristic map related to the second diagnosis auxiliary information,
The second disease is different from the first disease in at least a part of an onset region,
The first lesion emphasizing includes emphasizing a region corresponding to the first anatomical region from the first characteristic map, and the second lesion emphasizing corresponds to the second anatomical region from the second characteristic map. including emphasizing the area to be
Diagnostic auxiliary image providing device.
12. The method of claim 11,
The first lesion emphasizing includes emphasizing pixels having a pixel value greater than or equal to a first reference value among pixels included in a region corresponding to the first anatomical region of the first characteristic map,
The second lesion emphasizing includes emphasizing pixels having a pixel value greater than or equal to a second reference value among pixels included in a region corresponding to the second anatomical region of the second characteristic map.
Diagnostic auxiliary image providing device.
12. The method of claim 11,
the first lesion highlighting image includes a pixel value related to the first disease obtained from the first characteristic map and a boundary of the first anatomical region;
wherein the second lesion highlighting image includes a pixel value related to the second disease obtained from the second characteristic map and a boundary of the second anatomical region;
Diagnostic auxiliary image providing device.
12. The method of claim 11,
wherein the second anatomical region comprises the first anatomical region;
the first lesion highlighting image includes a set of pixels obtained from the first characteristic map and corresponding to the first anatomical region and having a pixel value equal to or greater than a first reference value and a boundary of the first anatomical region;
wherein the second lesion highlighting image includes a boundary comprising a set of pixels obtained from the second characteristic map and having a pixel value associated with the second disease as a second reference value;
Diagnostic auxiliary image providing device.
12. The method of claim 11,
The first lesion-emphasized image includes a boundary comprising a set of pixels obtained from the first characteristic map and having a pixel value associated with the first disease as a first reference value;
wherein the second lesion highlighting image includes a boundary comprising a set of pixels obtained from the second characteristic map and having a pixel value associated with the second disease as a second reference value;
Diagnostic auxiliary image providing device.
12. The method of claim 11,
The first disease is any one of glaucoma, retinal hemorrhage, retinal exudate, lens opacity, and diabetic retinopathy,
The second disease is any one of glaucoma, retinal hemorrhage, retinal exudate occurrence, lens opacity, and diabetic retinopathy except for the first disease,
Diagnostic auxiliary image providing device. delete delete delete delete delete delete

Images