KR20210106096A - Disease prediction system and method based on deep learning using fundus image - Google Patents

Abstract

The present invention discloses a disease prediction system based on deep learning using a fundus image. The disease prediction system according to the present invention includes: a data amplifying unit applying p filtering elements to one fundus image to amplify p images; an inference model generator in which the p training images generated by the data amplifying unit are divided into q training image sets, and the q training image sets for each r different model architectures based on deep learning are inputted, thereby generating q*r different inference models; an inference model execution unit in which the q * r inference models are loaded into a memory, the p determination images generated by the data amplification unit are divided into q determination image sets, and one image set for determination for each inference model is inputted, thereby calculating a disease inferred value; and a determination unit that calculates disease index using the disease inference value of each inference model calculated by the inference model execution unit. According to the present invention, the fundus image reading time is greatly reduced, so that the inconvenience between doctors and patients is greatly reduced. In addition, a web-based fundus image reading service is possible to easily obtain accurate reading results even in small ophthalmic hospitals or in regions or countries with poor medical access, thereby contributing greatly to the early detection and treatment of diseases, and accordingly, contributing to the improvement of the quality of medical services while preventing the waste of medical expenses.

Description

Disease prediction system and method based on deep learning using fundus image}

The present invention relates to a deep-learning-based disease prediction system using fundus images, and specifically, a deep-learning-based disease prediction that can improve the accuracy of determination while dramatically shortening the determination time to the extent that a web-based real-time service is possible. It relates to systems and methods.

Recently, interest in fundus imaging has increased significantly in the medical field. Fundus images can be obtained non-invasively, simply and inexpensively by using a fundus camera, as well as glaucoma, diabetic retinopathy, macular degeneration, etc. by analyzing blood vessels, retinal nerve fiber layer, macula, optic nerve papilla, etc. This is because it can diagnose eye diseases at an early stage or predict their risk.

Recently, research results have been published that it is possible to diagnose heart disease, dementia, etc. using fundus imaging, and the importance of fundus imaging is increasing.

However, in order to accurately read microscopic lesions shown on fundus images with the naked eye, high-level training is required, so there is a significant shortage of specialists in reading compared to the demand for examination in the actual medical field. In addition, there is a problem in that fundus image analysis is not easy to the extent that the reading results are often inconsistent even among ophthalmologists with a lot of experience.

Accordingly, recently, research to improve the accuracy of reading by analyzing fundus images using deep learning-based artificial intelligence is being actively conducted around the world. It is known that it has reached a level where it can be read accurately.

However, despite these development results, there are few cases where deep learning-based diagnostic assistance software has been commercialized so far, and even if commercialized, the price is too high, so it is expected that it can only be used in large hospitals. In addition, there is a problem in that it is difficult for a doctor in treatment to check the patient's condition in real time because the time required for the determination is too long.

Therefore, it is necessary to prepare a way to more easily utilize deep learning-based fundus image reading technology even in small hospitals or areas with poor medical access.

Korean Patent Registration No. 10-1880678 (2018.07.20 Announcement)

The present invention was devised against this background, and an object of the present invention is to provide a disease prediction system with high ease of use for doctors and patients by significantly reducing the time for determining whether a disease is present compared to a conventional fundus image analysis model.

In addition, the present invention can easily use deep-learning-based fundus image reading technology even in small hospitals or areas with poor medical access, and through this, the quality of medical services can be improved by diagnosing related diseases more quickly and accurately, while avoiding overlapping or over-testing. The purpose is to prevent patient inconvenience and waste of medical expenses.

In order to achieve this object, an aspect of the present invention, a data amplification unit for amplifying the p number of images by applying p filtering elements to one fundus image; It divides the p training images generated by the data amplification unit into q training image sets, and generates q * r different inference models by inputting q training image sets for each r different deep learning-based model architectures. inference model generation unit; Load q * r inference models into the memory, divide the p judgment images generated by the data amplification unit into q judgment image sets, and input one judgment image set for each inference model to obtain a disease inference value. an inference model execution unit that calculates ; It provides a disease prediction system including a determination unit that calculates a disease index by using the disease inference value of each inference model calculated by the inference model execution unit.

In the disease prediction system according to an aspect of the present invention, the data amplification unit provides an image code that means a filtering element applied to each generated image, and the inference model generation unit is an image set having the same image code in the same inference model for every learning step. , and the inference model execution unit may input a different set of images for determination for each inference model, but a set of images for determination having the same image code as the image set for training used in the learning process may be input to each inference model.

In addition, in the disease prediction system according to an aspect of the present invention, the data amplification unit generates the image for determination only by memory operation without storing the related file in the storage when generating p images for determination, and then transmits it to each inference model. have.

In addition, in the disease prediction system according to an aspect of the present invention, the inference model generator, after generating the inference model, stores each model architecture in a json file format, and the weight file of each inference model generated as a result of learning is in the hdf5 file format. can be saved

Also, in the disease prediction system according to an aspect of the present invention, the inference model execution unit may generate q*r threads for determination and execute one inference model for each thread.

Also, in the disease prediction system according to an aspect of the present invention, the inference model execution unit executes q * r inference models to calculate p disease inference values corresponding to the input p judgment images, respectively, and the determination unit p A disease index can be calculated by averaging *q*r disease inferred values.

Also, in the disease prediction system according to an aspect of the present invention, different r model architectures include at least one of InceptionV3, VGGNet, VGG, ResNet, AlexNet, GoogleNet, and InceptionResNetV2, or transform at least one hidden layer among them. may include one.

Another aspect of the present invention, a learning data amplification step of applying p filtering elements to one fundus image to amplify the p images; Inference that generates q * r different inference models by dividing the p training images generated in the data amplification step into q training image sets, and inputting q training image sets for each r different deep learning-based model architectures. model generation step; a decision data amplification step of applying p filtering elements to one fundus image and amplifying them into p images; Loading q * r inference models into memory, dividing p judgment images generated in the decision data amplification step into q judgment image sets, and inputting one judgment image set for each inference model an inference model execution step of calculating an inference value; It provides a disease prediction method including a determination step of calculating a disease index using the disease inference value of each inference model calculated in the inference model execution step.

In the disease prediction method according to the present invention, in the learning data amplification step and the judgment data amplification step, an image code indicating a filtering element applied to each generated image is respectively given, and in the inference model generation step, the same reasoning model for every learning step An image set having the same image code is input to the inference model execution stage, and a different image set for judgment is input for each inference model in the inference model execution step, but each inference model has the same image code as the image set for training used in the learning process. You can enter an image set.

According to the present invention, since the fundus image reading time is greatly reduced, the discomfort of the doctor and the patient can be greatly reduced.

In addition, as a web-based fundus image reading service is available, accurate reading results can be easily obtained even in small ophthalmic hospitals or in regions or countries with poor medical access, which can greatly contribute to early detection and treatment of diseases, thereby reducing medical costs. It can contribute to improving the quality of medical services while preventing

1 is a schematic configuration diagram of a disease prediction system according to an embodiment of the present invention;
2 is a block diagram illustrating a hardware configuration of a reading device;
3 is a functional block diagram of a reading device;
4 is a diagram illustrating a pre-processing method of a fundus image;
5 is a diagram illustrating a deep learning model architecture
6 is a diagram illustrating various image sets extracted from the original image by the data amplification unit;
7 is a conceptual diagram illustrating a method of generating a plurality of inference models using a model architecture and a plurality of image sets.
8 is a conceptual diagram showing a state in which an inference model is generated by linking a model architecture and a weight file;
9 illustrates an embodiment of multithreading;
10 is a conceptual diagram illustrating a method of inferring a disease using a fundus image in a disease prediction system according to an embodiment of the present invention;
11 is a diagram illustrating another embodiment of multithreading;

Hereinafter, a preferred embodiment of the present invention will be described in detail with reference to the drawings.

For reference, in the present specification, if a part includes or includes any component, it means that other components may be further included or provided without excluding other components unless otherwise stated. In addition, when one element is connected, combined, or communicated with another element, it is not only directly connected, combined, or communicated with another element, but also indirectly connected with another element in between. , combining, or communicating. In addition, when one component is directly connected or directly coupled to another component, it means that another element is not interposed therebetween. In addition, since the accompanying drawings in the present specification are merely illustrative for easy understanding of the gist of the present invention, it is clarified in advance that the scope of the present invention is not limited thereto.

As shown in the schematic configuration diagram of FIG. 1 , the disease prediction system according to an embodiment of the present invention is a reading device that communicates with the image providing terminal 100 through a plurality of image providing terminals 100 and a communication network 200 . (400).

The image providing terminal 100 is a terminal capable of transmitting the fundus image to the reading device 400, and the specific type is not limited. For example, the image providing terminal 100 may be at least one of a desktop computer, a laptop computer, and a hospital information system server installed in a hospital, and a smartphone, tablet, or handheld used by a doctor or nurse. It may be a portable computing device such as a PC, PDA, or PMP. Also, the image providing terminal 100 may be a computing device used by a patient or a third party.

Although not shown in the drawings, the image providing terminal 100 may receive and store the fundus image from the fundus camera by communicating with the fundus camera by wire or wirelessly.

The communication network 200 relays the communication between the image providing terminal 100 and the reader 400, and may be composed of a wired network, a wireless network, or an appropriate combination thereof, which is already known or developed later.

The reading device 400 analyzes the fundus image received from the image providing terminal 100 to determine the presence or risk of a specific disease, and transmits the determination result to the image providing terminal 100 .

As illustrated in the block diagram of FIG. 2 , the reading device 400 includes a central processing unit (CPU) 410 , a first graphics processing unit (GPU) 420 , a second GPU 430 , and a memory 440 . ), a storage 450 , an input unit 460 , a display 470 , a communication unit 480 , a bus 490 , and the like.

CPU 410 . The first GPU 420 and the second GPU 430 load a computer program and data stored in the storage 450 into the memory 440 and perform a predetermined operation or data processing according to the program code.

In an embodiment of the present invention, the first and second GPUs 420 and 430 are used to perform graphic processing tasks such as preprocessing of fundus images, data amplification, learning of deep learning inference models using fundus images, and execution of inference models. , the CPU 410 may be used to perform the overall operation of the reading device 400 excluding the graphic processing.

In an embodiment of the present invention, a plurality of threads are allocated to the first and second GPUs 420 and 430, respectively, in order to improve the loading and determination speed of the inference model, so that a plurality of predictive models can be executed simultaneously. However, the number of prediction models for reading fundus images and the number of threads for parallel processing may be variously selected according to the type of disease to be read, required accuracy, system environment, and the like.

The memory 440 temporarily stores instructions, data, etc. for the operation of the reading device 400 , and is divided into a system memory (RAM) for the CPU 410 and a video memory (VRAM) for the GPUs 420 and 430 . can If necessary, the CPU 410 and the GPUs 420 and 430 may utilize at least one of the system memory (RAM) and the video memory (VRAM) as a shared memory.

The storage 450 stores various computer programs and data necessary for the operation of the reading device 400 , and the type thereof is not particularly limited. For example, it may include internal or external HDD (Hard Disc Drive), ODD (Optical Disc Drive), SDD (Solid State Drive), etc., and includes Network Attached Storage (NAS), Storage Area Network (SAN), etc. You may.

The storage 450 may store an OS program for the operation of the reading device 400 , a program for analyzing the fundus image based on deep learning, and various data.

Referring to the functional block diagram of FIG. 3 , the programs and data stored in the storage 450 include a web server 510 , an image preprocessor 520 , a data amplifier 530 , an inference model generator 540 , and reasoning. It may include a model execution unit 550 , a determination unit 560 , a thread control unit 570 , a model architecture 580 , a weight file 590 , and the like.

The input unit 460 is an input device used by the manager of the reading device 400 and may include at least one of a keyboard, a button, a touch pad, a touch screen, and a mouse.

The display 470 may output an operation state of the reading device 400 or a model execution result.

The communication unit 480 includes a wired or wireless communication interface for communication with the communication network 200 , and may further include a short-range wireless communication (WiFi, Bluetooth, etc.) interface.

The bus 490 is a transmission path of data or electrical signals between each component of the reading device 400 , and may be provided in the form of a circuit pattern or a cable.

Hereinafter, the operation of the reading apparatus 400 will be described with reference to the functional block diagram of FIG. 3 .

First, the reading device 400 may transmit/receive data to and from the image providing terminal 100 through the web server 510 . The web server 510 may provide a predetermined web page for member registration, patient registration, fundus image upload, reading request, history search, etc. to the image providing terminal 100 connected through the communication network 200 .

The image preprocessor 520 may serve to extract a region of interest (ROI) from the fundus image received from the image providing terminal 100 and to scale the image size to a set size (eg, 299*299).

For example, the image preprocessor 520 may perform an operation of extracting a region of interest (ROI) by removing unnecessary regions from the original image based on the left and right tangents and/or the upper and lower tangents of the fundus image, as shown in FIG. 4 . have.

The data amplifier 530 transforms the fundus image preprocessed by the image preprocessor 520 in various ways to generate a plurality of images to be used for model learning and reading.

[Table 1]

As illustrated in Table 1, the data amplifier 530 may generate various transformed images from the original image by applying a Bilateral filter, a Gaussian filter, a Histogram equalization filter, a Median filter, a Sharpening filter, and the like.

Also, the data amplifier 530 may generate a modified image cut out to a predetermined size (eg, 299*299) after magnifying the original image by 120%.

Also, the data amplifier 530 may generate various modified images by rotating the original image by 90 degrees, 180 degrees, and 270 degrees, respectively.

Also, the data amplifier 530 may transform the original color image into a black-and-white image.

When all of these methods are applied, the data amplification unit 530 receives a total of 96 (6*2*4) images including the original image according to whether filtering, enlargement, rotation, and color from one original image that has undergone image preprocessing. *2) images can be created.

However, since the deformable elements shown in Table 1 are merely examples, the data amplifier 530 may apply only some of them, or may generate more deformed images by adding more filtering elements.

6 shows the image codes of Table 1 after dividing 96 images into 12 image sets. Since the method of dividing the image set into 12 is not limited to the method of FIG. 6 , 12 image sets may be set by bundling 8 images in various ways.

In an embodiment of the present invention, four deep learning model architectures 580 are learned using the image expanded in this way, and a total of 48 inference models are generated for use in reading fundus images.

In addition, even when the learned inference model is executed to determine whether or not a disease is present, the same data amplification method is applied to one fundus image received from the image providing terminal 100 to generate a total of 96 images, and select 96 images. In this way, it is divided into 48 inference models and executed to determine whether or not there is a disease.

The inference model generator 540 inputs the fundus image labeled with the disease to the deep learning-based model architecture 580 to which an arbitrary weight is set, calculates an output value (eg, disease probability), and uses a backpropagation algorithm An inference model is created by repeating the task of updating the weights based on the output value.

The deep learning-based model architecture 580 has one or more hidden layers between an input layer and an output layer as illustrated in FIG. 5, and a node (neuron) of an arbitrary first layer through repeated learning. By updating the weights ([W]1,,,,,[W]n) given between and the node (neuron) of the second layer to an optimal value, the accuracy of the output value can be improved.

On the other hand, in an embodiment of the present invention, in order to increase the accuracy of determination, an ensemble technique of averaging the output of each inference model is applied after generating a plurality of inference models without using only one deep learning inference model.

In an embodiment of the present invention to generate a plurality of inference models, in a state in which four basic model architectures 580 are stored in the storage 450, 96 images generated through data amplification from one fundus image are 12 After dividing into 4 image sets, a total of 48 inference models are generated by inputting and learning 12 image sets for each of the 4 model architectures 580 .

The four basic model architectures 580 may be represented by different first to fourth model architectures 581 , 582 , 583 , and 584 , as illustrated in FIG. 7 .

At this time, the first to fourth model architectures 581,582,583,584 different from each other may be selected from known deep learning architectures such as InceptionV3, VGGNet, VGG, ResNet, AlexNet, GoogleNet, InceptionResNetV2, and the like. Alternatively, after selecting one from a known deep learning model architecture, it is also possible to change some of the hidden layers (eg, convolutional layer, pooling layer, FC layer, etc.) of the architecture to use it as a new deep learning model architecture. For example, the first model architecture 581 is InceptionV3, the second model architecture 582 is a modified part of the hidden layer of InceptionV3, the third model architecture 583 is InceptionResNetV2, and the fourth model architecture 584 is ) may be a modified part of the hidden layer of InceptionResNetV2.

In any way, if four different model architectures (581,582,583,584) are prepared, a total of 48 models can be generated by inputting 12 different image sets for each model architecture (581,582,583,584) and performing deep learning training.

Meanwhile, the four different model architectures (581,582,583,584) are stored as json files, respectively, and the weights generated and updated by deep learning learning of each model architecture (581,582,583,584) are preferably stored as hdf5 files.

When the number of inference models to be executed is large, it takes a considerable amount of time to load the compiled inference model into separate files including weights, respectively, but if the model architecture and weights are saved as separate files in this way, the loading time into memory Since it can greatly shorten the time required for judgment, it is possible to greatly shorten the

Referring to FIG. 6 , when the code of Table 1 is assigned to 96 data-amplified images in one fundus image, the first image is [O,O,O,C], and the second image is [O,O,O ,G], the third image may be [O,O,R90,C],,,,, and the 96th image may be displayed as [S,U,R270,G]. That is, each of the 96 data-amplified images may be given different image codes indicating the applied filtering factors.

And if 96 images are sequentially or randomly divided into 8 images, a total of 12 image sets including 8 different image codes can be generated.

When the 12 image sets are generated in this way, as illustrated in FIG. 7 , 8 images of the first image set are sequentially input to the first model architecture 581 and the weights are updated through learning. Inference models can be created.

In addition, the second inference model may be generated by sequentially inputting eight images of the second image set to the first model architecture 581 and updating the weights through learning.

By continuing to apply this method, the twelfth inference model can be generated by sequentially inputting 8 images of the twelfth image set to the first model architecture 581 and updating the weights through learning.

The same method may be applied to the second model architecture 582 to input the first to twelfth image sets and to generate thirteenth to twenty-fourth inference models through learning.

In addition, by applying the same method to the third model architecture 583 , the first to twelfth image sets may be input and 25th to 36th inference models may be generated through learning.

In addition, by applying the same method to the fourth model architecture 584, the first to twelfth image sets may be input and 37th to 48th inference models may be generated through learning.

In the above process, model learning was performed using one fundus image, so in order to increase the accuracy of determination, model learning using more fundus images should be performed. The present applicant was able to significantly improve the accuracy of glaucoma determination by performing supervised learning using 2,000 normal fundus images and 2,000 glaucoma fundus images. However, it goes without saying that the amount of learning is not particularly limited.

Meanwhile, even when model learning is performed using other fundus images, it is preferable to input 8 images for each inference model after amplifying each fundus image to 96.

In particular, in order to increase the accuracy of determination, it is preferable that the image codes of the image sets input to each inference model are always the same.

As an example, if a first image set (refer to FIG. 6 ) among 96 images amplified from the first training image is input to the first model architecture 581 to generate the first inference model, thereafter, the second training image Even when the data is amplified to 96 and input to the first inference model, it is preferable to input the first image set of the second training image (see FIG. 6 ) having the same code as the image code used in the previous learning. In addition, even when the first inference model is executed after amplifying the data of 96 judgment fundus images for actual disease determination, the first inference model has the same code as the image code used for learning. It is preferable to input a set (see Fig. 6).

As another example, if a second image set (refer to FIG. 6 ) from among 96 images amplified from the first training image is input to the second model architecture 582 to generate the 14th inference model, thereafter, the second training image It is preferable to input the second image set (see FIG. 6 ) of the second training image having the same code as the image code used in the previous training even when the data is amplified to 96 and then input to the 14th inference model. In addition, even when the 14th inference model is executed after amplifying the data of 96 judgment fundus images for actual disease determination, the 14th reasoning model has the same code as the image code used for learning the second image of the judgment fundus image. It is preferable to enter a set.

As another example, if the twelfth image set (refer to FIG. 6 ) among 96 images amplified from the first training image is input to the third model architecture 583 to generate the 36th inference model, thereafter, for the second training Even when the data is amplified to 96 images and then input to the 36th inference model, it is preferable to input the twelfth image set (refer to FIG. 6 ) of the second training image having the same code as the image code used for the previous learning. In addition, even when the 36th inference model is executed after amplifying the data of 96 judgment fundus images for actual disease determination, the 36th inference model has the same code as the image code used for learning. It is preferable to enter a set.

Similarly, if the first image set (see FIG. 6 ) among 96 images amplified from the first training image is input to the fourth model architecture 584 to generate the 37th inference model, thereafter, the second training image is reduced to 96. Even when the 37th inference model is executed after data amplification, it is preferable to input the same first image set (see FIG. 6 ) of the second training image as the image code used for the previous training. In addition, even when the 37th inference model is executed after amplifying the data of 96 judgment fundus images for actual disease determination, the 37th inference model has the same code as the image code used for learning the first image of the judgment fundus image. It is preferable to enter a set.

As described above, according to an embodiment of the present invention, 48 inference models can be generated using four model architectures 581,582,583,584 and 12 image sets amplified from one fundus image.

When 48 inference models are generated, as illustrated in FIG. 8 , 48 weight files 591-1, 591-2,,,,,594-12 corresponding to each inference model are generated.

At this time, 48 inference models may be compiled into separate files using 48 weight files 591-1,591-2,,,,,594-12 and corresponding model architectures 581,582,583,584.

However, in the embodiment of the present invention, as described above, in order to shorten the model loading time, the model architecture (581,582,583,584) is stored as a json file and 48 weight files (591-1,591-2,,,,,594-12) is saved as an hdf5 file. When the model architecture and the weight file are stored separately as described above, the loading time of the model architecture (581,582,583,584) and the weight file (591-1,591-2,,,,,594-12) into the memory 440 in the determination step is greatly increased. can be shortened

The inference model execution unit 550 reads the model architecture (581,582,583,584) and weight files (591-1,591-2,,,,,594-12) stored in the storage 450 when a read command is input together with the fundus image, and reads the memory Load 440 and run 48 inference models.

The inference model execution unit 550 requests preprocessing and data amplification of the fundus image to be read by the image preprocessing unit 520 and the data amplification unit 530, and after dividing 96 data-amplified images into 12 image sets, each input into the inference model.

On the other hand, in the embodiment of the present invention, when image pre-processing and data amplification are performed on the fundus image to be determined in order to shorten the time taken for determination, the file generated in the image processing process is not stored in the storage 450 but the fundus In a state in which the image is loaded into the memory 440 , image preprocessing and data amplification may be performed only by memory operation, and the processing result may be input to the inference model.

Even in this case, as described above, 96 images generated through data amplification are divided into 12 image sets, and an image set having the same image code as the image set used to train the inference model is input to the corresponding inference model. It is preferable to do

In addition, in the embodiment of the present invention, 48 threads are allocated in the decision step in order to shorten the decision time, and 48 inference models are concurrently processed in parallel.

That is, as shown in FIG. 9 , the first to twenty-fourth threads 601,,,,,624 are allocated to the first GPU 420 through the thread control unit 570, and the twenty-fifth threads are allocated to the second GPU 430. After allocating the to 48th threads 625,,,,,648, it may be set to execute one inference model for each thread.

In this way, as shown in FIG. 10, an image set including 8 images is input to each of the 48 inference models, and 8 inference values are output for each inference model through a deep learning operation.

In the embodiment of the present invention, since this process is concurrently processed in parallel by 48 threads, the speed of determination can be greatly reduced.

The determination unit 560 may output a disease index indicating the presence and/or degree of a disease in various ways using a total of 384 inference values outputted from each of the 48 inference models, 8 each. For example, only the average value of multiple inference values may be output.

The disease index output from the determination unit 560 may be transmitted to the image providing terminal 100 through the web server 510 or may be transmitted to a preset terminal.

According to an embodiment of the present invention, since 48 inference models are distributed and loaded on two GPUs 420 and 430 and processed in parallel through 48 threads in the determination step, the determination time of the inference model can be greatly reduced. According to the experiment, when the inference model was loaded, the time required to calculate the average probability from the fundus image was less than 1.5 seconds. can

In addition, according to the embodiment of the present invention, by storing the model architecture and the weight file separately, the model loading time can be greatly shortened, and thus, the convenience of the device operation and the administrator can be greatly improved.

On the other hand, the thread control unit 570 performs the roles of creating a thread pool, allocating a thread, and a thread state (execution waiting, pause, termination, etc.) for multi-threading.

In the embodiment of the present invention, the same number of threads as the number of inference models are created in the decision step to improve the decision speed, but threads may be allocated in a different way in the learning step.

For example, as illustrated in FIG. 11 , a first thread 710 and a second thread 720 are allocated to the first GPU 420 , and a third thread 730 and a fourth thread 720 are allocated to the second GPU 430 . After allocating the thread 740 , the first thread 710 executes the first to twelfth reasoning models, the second thread 720 executes the thirteenth to 24th reasoning models, and the third thread 730 . In , the 25th to 36th inference models are executed, and the fourth thread 740 may be controlled to execute the 37th to 48th inference models.

In this case, the thread control unit 570 may randomly allocate a reasoning model to be executed to each thread, or may allocate a reasoning model of a similar size to match the workload of each thread to a similar level. That is, for example, one inference model of a similar size is selected from among the first to twelfth models, the 13th to the 24th models, the 25th to the 36th models, and the 37th to the 48th models, and simultaneously in the four threads 710, 720, 730 and 740. You can also control it to run.

Also, the thread control unit 570 may allocate an inference model having a common characteristic among 48 inference models to the same GPU or the same thread. For example, among the inference models, a model learned based on a color image may be executed on the first GPU 420 , and a model learned based on a black-and-white image may be set to be executed on the second GPU 430 .

On the other hand, the web server 510, the image preprocessor 520, the data amplification unit 530, the inference model generation unit 540, the inference model execution unit 550, the determination unit ( 560), and at least one or a part of each component such as the thread control unit 570 may be implemented as hardware or a combination of software and hardware. In this case, the hardware may be an application specific integrated circuit (ASIC).

In addition, the disease prediction method according to an embodiment of the present invention may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable recording medium.

In this case, the computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the recording medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art related to computer software.

The computer-readable recording medium includes magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magnetic media such as floppy disks. It may include at least one of optical media (Magneto-Optical Media), ROM, RAM, flash memory, and the like.

In addition, the program instructions may include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

In addition, although preferred embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments and may be implemented with various modifications or modifications in a specific application process.

As an example, since the present specification does not specifically limit the disease to be predicted using the fundus image, the disease prediction method according to the present invention is glaucoma, diabetic retinopathy, macular degeneration, heart through deep learning learning using the fundus image. It can be used to predict various diseases such as disease and dementia.

As another example, in the above description, the case in which the reading device 400 receives the fundus image from the remote image providing terminal 100 through the communication network 200 has been described, but the present invention is not limited thereto. For example, the reader 400 may be installed in a hospital, etc. and the fundus image may be directly received from the fundus camera to infer whether there is a disease.

As another example, in the above, image filter, image enlargement, image rotation, and color are applied as data amplification methods using the original image. You can also change the number of

As another example, in the above embodiment, it has been described that 48 inference models are generated using 4 model architectures and 12 image sets, but the number of model architectures, image sets, and inference models is not necessarily limited thereto. am.

As another example, in the above, it has been described that the reading device 400 includes two GPUs 420 and 430, but the number of GPUs is not limited thereto. In some cases, only one GPU may be used, or three or more GPUs. can also be used.

As another example, since the number of threads allocated by the thread controller 570 is not necessarily limited to the above-described embodiment, the number of threads may be appropriately adjusted according to the type of inference model or input data.

As described above, the present invention can be implemented with various modifications or modifications in the specific application process, and the modified or modified embodiments are also within the scope of the present invention if the technical ideas of the present invention disclosed in the claims to be described later are included. belonging will be taken for granted.

100: image providing terminal 200: communication network 400: reading device
410: CPU 420: first GPU 430: second GPU
440: memory 450: storage 460: input
470: display 480: communication unit 510: web server
520: image preprocessor 530: data amplification unit 540: inference model generation unit
550: inference model execution unit 560: determination unit 570: thread control unit
580: model architecture 590: weight file

Claims (9)

a data amplifying unit applying p filtering elements to one fundus image to amplify p images;
It divides the p training images generated by the data amplification unit into q training image sets, and generates q * r different inference models by inputting q training image sets for each r different deep learning-based model architectures. inference model generation unit;
Load q * r inference models into the memory, divide the p judgment images generated by the data amplification unit into q judgment image sets, and input one judgment image set for each inference model to obtain a disease inference value. an inference model execution unit that calculates ;
A decision unit that calculates a disease index using the disease inference value of each inference model calculated by the inference model execution unit
Disease prediction system comprising According to claim 1,
The data amplification unit gives an image code that means a filtering factor applied to each generated image,
The inference model generator inputs an image set having the same image code to the same inference model at every learning step,
A disease prediction system, characterized in that the inference model execution unit inputs a different set of images for judgment for each inference model, but inputs a set of images for judgment having the same image code as the set of images for learning used in the learning process to each inference model. According to claim 1,
The data amplification unit generates the image for determination only by memory operation without storing the related file in storage when generating p images for determination, and then transmits the image to each inference model. According to claim 1,
Inference model generation unit, after generating the inference model, each model architecture is stored in a json file format, and the weight file of each inference model generated as a result of learning is stored in an hdf5 file format. According to claim 1,
The inference model execution unit generates q * r threads for determination and executes one inference model for each thread. According to claim 1,
The inference model execution unit executes q * r inference models, respectively, and calculates p disease inference values corresponding to the input p judgment images, respectively,
A disease prediction system, characterized in that the determination unit calculates a disease index by averaging p * q * r disease inferred values According to claim 1,
The different r model architectures include at least one of InceptionV3, VGGNet, VGG, ResNet, AlexNet, GoogleNet, and InceptionResNetV2, or at least one hidden layer from among them. Disease prediction system, characterized in that it comprises a modified one. A learning data amplification step of applying p filtering elements to one fundus image and amplifying the p images;
It divides the p training images generated in the data amplification step into q training image sets, and generates q * r different inference models by inputting q training image sets for each r different deep learning-based model architectures. inference model generation step;
a decision data amplification step of applying p filtering elements to one fundus image and amplifying them into p images;
q * r inference models are loaded into the memory, p judgment images generated in the decision data amplification step are divided into q judgment image sets, and one judgment image set is input for each inference model. an inference model execution step of calculating an inference value;
A decision step of calculating a disease index using the disease inference value of each inference model calculated in the inference model execution step
A disease prediction method comprising 9. The method of claim 8,
In the data amplification step for learning and the data amplification step for judgment, an image code, which means a filtering element applied to each generated image, is given, respectively,
In the inference model generation step, an image set with the same image code is input to the same inference model at every learning step,
In the inference model execution step, a different judgment image set is input for each inference model, but a judgment image set having the same image code as the training image set used in the learning process is input to each inference model.

Images