KR101887194B1 - Method for facilitating dignosis of subject based on medical imagery thereof, and apparatus using the same - Google Patents

Abstract

The present invention relates to a method for supporting the reading of a medical image of a subject and a computing apparatus using the same. Specifically, the computing apparatus according to the present invention acquires the medical image of the subject, generates reading assistance information on the medical image based on a machine learning model from the input of the medical image, and provides the generated reading assistance information to an external entity. Accordingly, the present invention enables medical staff to conveniently write a reading result.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for supporting reading of a medical image of a subject,

The present invention relates to a method for supporting reading of a medical image of a subject and a computing device using the same. Specifically, the computing device according to the present invention acquires a medical image of the subject, generates read assistance information on the medical image based on a machine learning model from the input of the medical image, And provides the generated read assistance information to an external entity.

Medical images, especially chest PA (posteroanterior) X-ray images, are frequently used for reading because they can observe the inside of the body, for example, lungs, bronchi and heart.

Some of the findings that can be read through chest PA x-ray imaging are that the surgeon can easily distinguish the corresponding features and forms only through years of training. . In particular, the higher the degree of difficulty of reading, such as a nodule, the more likely it is that the physician may be overlooked even if a high degree of attention is paid.

The need for computer aided diagnosis (CAD) has aroused the need to assist in the reading of images that humans can easily overlook, and conventional CAD technology aids physician judgment in very limited areas .

For example, Korean Patent Laid-Open Publication No. 10-2014-0091176 discloses an apparatus and a method for assisting diagnosis of a lesion. However, this prior art document does not specifically disclose a process in which a judgment model for judging a lesion peripheral region (or a 'judgment model' as a mixed word in the prior art document) is performed, or statistical or computational steps, There is a problem in that a person having a general knowledge in the technical field (hereinafter referred to as " ordinary technician ") can not read the contents of the disclosure and can not carry out the technique.

Korean Patent Laid-Open Publication No. 10-2017-0047423 discloses a technical idea concerning an automatic tuberculosis diagnostic predicting system of a CAD-based digital X-ray. In this case, in order to screen suspicious lesions of tuberculosis infection in a target lung image, (k-nearest neighbors algorithm) or SVM (support vector machine) to find the optimal hyper-parameter to detect the lesion itself, This is because there is a disadvantage that supervised learning repeated by an expert doctor is required. Therefore, there is a primary question about the reliability of the detection of the lesion itself, and thus, , It is necessary to determine whether information on various types of findings can be provided Not only when, there is a disadvantage that individual findings and also a separate written findings and decision pandokmun cumbersome through careful observation of the end decision-pandokmun need not be provided for them.

Accordingly, the present invention solves these problems, and provides a method for supporting reading of chest PA images, which enables medical personnel to read chest PA X-ray images more efficiently and accurately, and to easily generate or create the findings and the readings, and a device using the same I want to.

KR 10-2014-0091176 A KR 10-2017-0047423 A

The present invention aims to provide convenience of chest image reading by providing information to the medical staff on which the chest image reading is based using AI (artificial intelligence).

In addition, the present invention aims to provide a user interface suitable for chest image reading so that the medical staff can quickly and accurately generate and arrange the read result data.

Specifically, the present invention aims to automatically identify features such as the location and shape of major findings and structural characteristics such as lung, bronchus, and heart, thereby facilitating the generation or creation of diagnostic information and a report.

The characteristic configuration of the present invention for achieving the object of the present invention as described above and realizing the characteristic effects of the present invention described below is as follows.

According to an aspect of the present invention, there is provided a method of supporting reading of a chest view of a subject, the method comprising: (a) the computing device acquiring a chest image of the subject, Supporting other devices associated with the device to acquire; (b) supporting the computing device to generate or cause the other device to generate read assistance information regarding the chest image based on a deep learning model from input of the chest image; And (c) when the read assistance information is generated, the computing device provides the read assistance information to an external entity or to provide the other device with assistance.

Preferably, the read assistance information comprises (i) at least one finding information for the chest image, or (ii) a mask image corresponding to the at least one finding information and the finding information, respectively The mask image may be provided as an indication to support individual observation information corresponding to the mask image by being superposed on at least a part of the chest image.

The method may further include the step of (d) supporting the computing device to obtain evaluation information on the read assistance information or to acquire the evaluation information on the read assistance information, wherein the evaluation information includes the finding information and the finding (I) information input directly by the external entity, (ii) a basic reading provided to correspond to the finding information, (iii) correspondence to the finding information, and Or (iv) any one of (i) to (iii) above may be the result of being modified by the external entity.

According to another aspect of the present invention, there is also provided a computer program stored in a machine readable non-transitory medium, comprising instructions embodied to perform the method according to the invention.

According to still another aspect of the present invention, there is provided a computing device for supporting reading of a chest view of a subject, the computing device comprising: a communication unit for acquiring a chest image of the subject; And a processor for generating read assistance information about the chest image based on a deep learning model from an input of the chest image or supporting other devices to be interlocked through the communication unit to generate, Provides the read assistance information to an external entity or to provide the other device with the read assistance information when the read assistance information is generated.

Preferably, the processor is capable of acquiring or acquiring evaluation information on the read assistance information, and when the evaluation information is acquired, updating the deep learning model based on the evaluation information, It may also help other devices update.

According to the present invention, a user interface suitable for reading the chest image can be provided, so that the medical staff can more easily grasp the characteristics of the chest image, and the reading result can be conveniently generated based on the reading assistance information.

Particularly, it is possible to prevent a main feature such as a nodule from being overlooked in a conventional chest image reading method, and to provide a user interface suitable for reading a chest image, so that a medical staff can quickly and accurately read the data So that it can be generated and arranged.

Further, according to the present invention, it is possible to organize data efficiently, and it is easy to acquire additional data for AI learning such as the deep learning model of the present invention, and to use the expert knowledge, Initial feature information, such as information on a main feature, which is easily used for reading a chest image can be provided with high accuracy by using the learned information based on learning data including a disease characteristic judged by an expert There is an effect.

Accordingly, the present invention includes features such as location and shape of major findings, lung, and bronchus. Heart, and the like, thereby facilitating the generation or creation of diagnostic information and readings. Also, it is possible to prevent unreasonable readings by generating highly probable read assistance information in the system, And it is advantageous that the accuracy of the deep learning model is continuously updated as the user's experience of use is accumulated in the future.

In short, by enabling efficient and accurate reading, it is possible to save the time of the medical staff, improve the quality of the medical care, and innovate the workflow in the medical field.

According to the present invention, it is possible to utilize the images obtained and secured in the chest images conventionally used in hospitals, for example, X-ray imaging equipment and medical image storage transmission system (PACS) it is of course not dependent on the operating system.

BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiments of the invention and, Other drawings can be obtained.
1 is a diagram showing a main concept of a CNN (convolutional neural network) as an example of a deep learning model that can be used in the present invention.
2 is a conceptual diagram schematically showing an exemplary configuration of a computing device that performs a method of supporting reading of a chest view of a subject (hereinafter referred to as "medical image reading support method") according to the present invention .
3 is an exemplary block diagram illustrating the hardware or software components of a computing device that performs a method of supporting medical image reading in accordance with the present invention.
FIG. 4 is another exemplary block diagram illustrating interaction among a plurality of computing devices when a plurality of computing devices performing the medical image reading support method according to the present invention is configured.
5 is a flowchart exemplarily showing an embodiment of a medical image reading support method according to the present invention.
FIG. 6 is a diagram conceptually illustrating a multi-feature segmentation and classification process for generating read assistance information through a deep learning model in an embodiment of a medical image read support method according to the present invention.
7A to 7D are diagrams illustrating actual data used as learning data by a deep learning model of a medical image reading support method according to the present invention.
8 is a diagram illustrating a configuration of an exemplary user interface (UI) provided in the medical image reading support method according to the present invention.
9 is a diagram illustrating a configuration of an exemplary user interface in which diagnostic information is provided as read assistance information in a medical image reading support method according to the present invention.
FIG. 10 and FIG. 11 are diagrams illustrating the configuration of an arbitrary user interface provided to allow a user to input or modify a readout corresponding to feature information in the medical image reading support method according to the present invention.
FIG. 12 is a diagram illustrating an ROC curve (ROC curve) as a tool for evaluating the performance of the deep learning model used in the medical image reading support method according to the present invention.
13 is a diagram illustrating an ROC curve of the deep learning model used in the medical image reading support method according to the present invention.

The following detailed description of the invention refers to the accompanying drawings, which illustrate, by way of example, specific embodiments in which the invention may be practiced in order to clarify the objects, technical solutions and advantages of the invention. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention.

Throughout the detailed description and claims of the present invention, the term 'DICOM' (Digital Imaging and Communications in Medicine) is a generic term for various standards used in digital image representation and communication in medical devices, The standard is presented at the Joint Committee composed of the American Radiation Medical Association (ACR) and the American Electrical Manufacturers Association (NEMA).

In addition, throughout the specification and claims of the present invention, the term "Picture Archiving and Communication System (PACS)" refers to a system for storing, processing and transmitting in accordance with the DICOM standard, Can be stored in the DICOM format and transmitted to the terminals inside and outside the hospital through the network, and the result of reading and the medical record can be added to the medical image.

Throughout the detailed description and claims of the present invention, 'learning' or 'learning' refers to performing machine learning through computing according to a procedure, It will be understood by those of ordinary skill in the art that the present invention is not intended to be so-called.

And throughout the description and claims of this invention, the word 'comprise' and variations thereof are not intended to exclude other technical features, additions, elements or steps. Other objects, advantages and features of the present invention will become apparent to those skilled in the art from this description, and in part from the practice of the invention. The following examples and figures are provided by way of illustration and are not intended to limit the invention.

Moreover, the present invention encompasses all possible combinations of embodiments shown herein. It should be understood that the various embodiments of the present invention are different, but need not be mutually exclusive. For example, certain features, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the invention in connection with one embodiment. It should also be understood that the position or arrangement of individual components within each disclosed embodiment may be varied without departing from the spirit and scope of the present invention. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present invention is to be limited only by the appended claims, along with the full scope of equivalents to which such claims are entitled, if properly explained. In the drawings, like reference numerals refer to the same or similar functions throughout the several views.

Unless otherwise indicated herein or clearly contradicted by context, items referred to in the singular are intended to encompass a plurality unless otherwise specified in the context. In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings, so that those skilled in the art can easily carry out the present invention.

Deep learning model.

1 is a diagram showing a main concept of a CNN (convolutional neural network) as an example of a deep learning model that can be used in the present invention.

Referring to FIG. 1, the deep learning model used in the present invention can be briefly described as a form in which artificial neural networks are stacked in multiple layers. That is, it is expressed as a deep neural network (deep neural network) in the sense of a network of deep structure. As shown in FIG. 1, by learning a large amount of data in a structure of a multi-layer network, And learns the network by a method that minimizes the error of the objective function, that is, the prediction accuracy of the fatal symptom. This is compared to the connections between neurons in the human brain, and such a neural network is becoming a next generation model of AI.

Among the deep learning models that can be utilized in the present invention, CNN (Convolutional Neural Network) is a model suitable for classification of two-dimensional images such as images. Each region of an image is divided into feature maps map, and a sub-sampling layer that extracts features that are invariant to changes in position or rotation by reducing the size of the feature map. By repeating the sub-sampling layer, low-level features such as points, lines, It is possible to extract various levels of features from complex to meaningful high-level features. Finally, if the extracted features are used as input values of the existing classification model, it is possible to construct a classification model with higher accuracy .

Next, Fig. 2 is a conceptual diagram schematically showing an exemplary configuration of a computing device that performs a medical image reading support method according to the present invention.

2, a computing device 200 according to an embodiment of the present invention includes a communication unit 210 and a processor 220. The communication unit 210 communicates with an external computing device (not shown) Communication is possible.

In particular, the computing device 200 may be a computing device, such as a computer, a processor, a memory, a storage, an input and output device, or any other device capable of including components of a conventional computing device; Electronic communication devices, electronic information storage systems such as network-attached storage (NAS) and storage area networks (SAN), and computer software (i.e., computing devices that enable a computing device to function in a particular manner) Commands) to achieve the desired system performance.

The communication unit 210 of the computing device can send and receive requests and responses to and from other interworking computing devices. As an example, such requests and responses can be made by the same transmission control protocol (TCP) session But not limited to, a user datagram protocol (UDP) datagram, for example. In addition, in a broad sense, the communication unit 210 may include a keyboard, a mouse, and other external input devices for receiving commands or instructions.

The processor 220 of the computing device may also be a micro processing unit (MPU), a central processing unit (CPU), a graphics processing unit (GPU) or a tensor processing unit (TPU), a cache memory, a data bus ). ≪ / RTI > It may further include a software configuration of an operating system and an application that performs a specific purpose.

Medical image reading support device.

3 is an exemplary block diagram illustrating a hardware or software component of a computing device (" medical image reading support device ") that performs a medical image reading support method in accordance with the present invention, FIG. 8 is yet another exemplary block diagram illustrating interaction between a plurality of computing devices in the case where a plurality of computing devices performing the read support method is configured.

3, the computing device 200 may include an image acquisition module 310 as a component of the method and apparatus according to the present invention. It is to be understood that the image acquisition module 310 may be implemented by the communication unit 210 included in the computing device 200 or by the interlocking of the communication unit 210 and the processor 220, will be.

The image acquisition module 310 may acquire a chest image (e.g., chest PA X-ray image) of the subject. This chest image may be obtained from, but is not limited to, the X-ray imaging device 410 illustrated in FIG. 4 or the medical image storage transmission system (PACS) 420 associated therewith. For example, the chest image of the subject is captured through the X-ray imaging device 410, transmitted to the PACS 420 according to the DICOM standard, and then acquired by the image acquisition module 310 of the computing device 200 .

Next, the acquired chest image can be transmitted to the discrimination module 320. In the discrimination module 320, the characteristic of the chest image is extracted based on the deep learning model, and the read assistance information on the chest shape is generated The bar and the read assistance information will be described later in detail.

For example, this deep-running model may include segmentation networks to extract features relating to the shape of part of an image, etc. An embodiment of a fraction network that can be used here is as follows.

conv-conv-maxpool (2) -conv-conv-maxpool (2) -conv-conv-maxpool (2) conv-dense (256) -dense (1)

The data of the chest image input to the fraction network of this embodiment was the RGB image format converted to the z-score of 512x512, and the stride of conv and maxpool was represented by (x). Here, the activation function is selu, the last dense layer is sigmoid, and the loss function is a binary cross entropy. The other data format, conv, maxpool stride, activation function, loss Functions and the like can be used.

As another example, a z-score may be obtained for each channel for an RGB image of 640x640 in another network, and a segmented image may be used as an input. For example, batch-normalization may be used. Cross entropy may be used.

In another embodiment, upsampling may be performed as a preprocessing for an input image. For example, the length and the width may be increased through bilinear interpolation.

In another embodiment, a fractional network model in which a convolution block in the form of a visual geometry group (VGG) network is stacked and a fully-connected layer is connected to the aggregate block may be used.

Learning of the deep learning model used in the above-described discrimination module 320 may be performed by the discrimination module 320 or may be learned by a separate learning module 330. [

If read assistance information is extracted for the chest image based on the deep learning model, the read assistance information may be stored via the storage and transmission module 340 together with the evaluation information evaluated for the read assistance information, or may be stored in an external entity, Such as a PACS 420, which may be provided to the computing device 200 or interfaced to the computing device 200.

Here, the external entity includes a user, an administrator of the computing apparatus, a medical professional in charge of the subject, and the like, but it should be understood that any subject can read the chest image something to do. For example, the external entity may be an external artificial intelligence device.

3, a result input module 350 capable of obtaining the evaluation information from a user, and an update module 360 capable of updating the deep learning model based on the evaluation information, The modules 310 to 360 may be implemented by the communication unit 210 and the processor 220.

Although the components shown in FIG. 3 are illustrated as being realized in one computing device (e.g., a single standalone workstation) for convenience of explanation, the computing device 200 performing the method of the present invention may be configured as a plurality, Referring to FIG. 4, the image acquisition module 310 and the result input module 350 illustrated in FIG. 3 are included in a workstation 430, which is a computing device for providing an interface to a user, The learning module 330 and the update module 360 may be included in the AI engine 440 as a computing device for providing the function of the deep learning model and the storage and transmission module 340 The workstation 430 and the AI engine 440. In this case, As shown in FIG. 4, the interworking of the work station 430 and the AI engine 440 may be performed by a web service, but is not limited thereto.

A method of supporting medical image reading.

An embodiment of a medical image reading support method according to the present invention will now be described in detail with reference to FIG. 5 is a flowchart exemplarily showing an embodiment of a medical image reading support method according to the present invention.

(Not shown) before the deep learning model is learned before the medical image read support method according to this embodiment is performed, and a learning module 330 for the deep learning model is provided to the processor 220 Lt; / RTI >

For example, the learning module 330 may include data on individual chest images and data labeled thereon for a plurality of existing subjects, for example, data on findings such as a disease determined by a doctor, and a mask The deep learning model can be learned using mask data as learning data. As an example of the deep learning model for this purpose, some types of fraction networks are effective as described above.

Referring to FIG. 5, a medical image reading support method according to this embodiment includes an image acquisition module 310 implemented by the communication unit 210 of the computing device 200, as described above with reference to FIG. 3 , Acquiring a chest image of the subject, or assisting another device (e.g., PACS 420) associated with the computing device 200 to acquire (S510). Here, the chest image may be a chest PA (posteroanterior) image, for example, a chest PA x-ray image. Of course, the CT image and the MRI image are also available, and reliable diagnosis results can be obtained by using them. However, according to the medical image reading support method of the present invention as described in detail below, highly accurate diagnostic information can be obtained at low cost.

A method of supporting a medical image reading according to an embodiment of the present invention may include a step of determining whether a discrimination module 320 implemented by the processor 220 of the computing device 200 detects a deep learning model (S520) of generating or suggesting read assistance information from the chest image or creating or suggesting the other device based on a deep learning model (S520) of the computing device (200) (S530) of providing or providing to the external entity the read assistance information generated or proposed by the storage and transmission module 340 implemented by the storage and transmission module 220 implemented by the storage and transmission module 340.

Wherein the read assistance information refers to information provided to a user to assist a user, e.g., a physician, in reading the chest image, the read assistance information comprising (i) at least one finding information for the chest image, Or (ii) a mask image corresponding to each of the at least one feature information and the feature information, wherein the mask image is superimposed on at least a part of the chest image, Is provided as an indication to support information, preferably in the form of a density map.

Although it is described herein that the read assistance information is provided to the user to assist the user's judgment, this means that the present invention is limited to the extent that the read aid information is limited to the 'aid' of the readout It is not clear. As an alternative embodiment of the present invention, a modification in which the read assistance information is provided to an external entity including the user as the end result of the readout itself, a modification in which the read assistance information is processed by natural language processing and provided as a final result is also possible It will be appreciated by those of ordinary skill in the art.

The at least one finding information may be selected from, for example, a normal finding and an abnormal finding of the subject's chest. As another example, the at least one finding information includes at least one of nodule finding, consolidation finding, pleural effusion finding, pneumothorax finding, interstitial opacity finding and normal normal < / RTI > findings.

FIG. 6 is a diagram conceptually illustrating a multi-feature segmentation and classification process for generating read assistance information through a deep learning model in an embodiment of a medical image read support method according to the present invention.

Referring to FIG. 6, the learning process of the deep learning model according to the present invention will be described in more detail. As learning data of the deep learning model, individual chest images 610a and 610b and data 620a and 620b , The labeling data may include mask data indicating a region corresponding to a finding such as a disease judged by a doctor. 6, mask data corresponding to the chest image 610a of abnormal findings is displayed at 620a, and mask data corresponding to the chest image 610b of normal findings is displayed at 620b.

In step S520, the computing device 200 generates the read assistance information by performing multi-feature segmentation and classification based on the deep learning model or supports to cause the other device to generate Referring to FIG. 6, ROI fractions (indicated by 'segmentation') and image classification ('classification') are generated for each feature of each of the at least one feature information by the deep learning model The output pixel classification information and the output image classification information are generated as the basic information 630 of the read auxiliary information and the basic information 630 of the read auxiliary information is generated Accordingly, the feature information and the mask image are generated as the read assistance information.

7A to 7D are diagrams showing actual data (left side of each drawing) used as learning data by the deep learning model of the medical image reading support method according to the present invention, and mask images (right side FIG.

On the left side of FIG. 7A, a chest image judged to be a nodule by a radiologist is displayed. On the chest image, mask data for learning indicated by a portion corresponding to the nodule by a specialist are superimposed on the chest image, On the right side of FIG. 7A, a mask image generated by the deep learning model for the same chest image is superimposed on the same chest image. Similarly, Figs. 7B, 7C, and 7D show that mask data (left side) and mask image (right side) generated by a deep training model are superimposed on the same chest image, respectively, for hardening, pleural effusion, and pneumothorax Is displayed.

8 is a diagram illustrating a configuration of an exemplary user interface (UI) provided in the medical image reading support method according to the present invention.

8, test list information 810, chest image information 820, AI proposal information 830, and final report information 840 may be provided through the user interface.

According to this example, in step S510, a list of chest images of individual subjects held, stored, or stored by the computing device 200 or the other device, e.g., PACS 420, 810 via a user interface on a predetermined output device, e.g., a display device, the output device may be included in or associated with the computing device.

At this time, if it is detected that the user selects the specific test subject shown in the check list information, the computing device 200 performs the above-described step S520 on the chest image of the specific test subject, (Identification name, age, sex, etc.) of the subject, photographing indicating the region of the chest image to be photographed, photographing of the chest image of the subject, A region identifier, and the like may be provided as the chest image information 820 through the user interface, as illustrated in FIG.

Thereafter, in step S530, the read assistance information may be provided through the output device without additional user manipulation or in response to a user request, and the mask image may be provided to the thoracic image information 820 And the at least one feature information may be provided through the user interface as AI suggestion information 830, as illustrated in FIG.

Preferably, in step S530, if a confidence value is obtained for each of the at least one finding information by the deep learning model, the computing device 200 may calculate the confidence value, The display device displays the at least one feature information displayed through the predetermined output device in a descending or ascending order based on the reliability value or displays each of the feature information based on the reliability value in the AI suggestion information 830 , At least one of the hue, saturation, and brightness displayed on the display screen.

Advantageously, in step S530, the feature information generated or suggested by the deep learning model can be modified by the user's selection, for example, in the list displayed in the AI suggestion information 830 It can be done by an operation of selecting one item.

9 is a diagram illustrating a configuration of an exemplary user interface in which diagnostic information is provided as read assistance information in a medical image reading support method according to the present invention.

In the exemplary user interface of FIG. 9, a selection indication 832 for an interstitial opacity item, which is one of the items displayed in the AI proposal information 830, It can be said that it corresponds to the feature information or the modified feature information as described above.

Not only one but also several such information can be selected. However, there may be an exclusive relationship between these findings, for example, normal findings are exclusive of all other findings. This is intended to prevent logical and pathological readings and unreasonable readings.

In step S530, preferably, the mask image 822 corresponding to the finding information is displayed (820) over at least a part of the chest image, together with generated or selected feature information. Or may be displayed in the form of a density map as illustrated by way of example.

Referring to FIG. 5 again, the medical image reading support method according to the embodiment may be configured such that, after performing the step S530, the computing device 200 acquires evaluation information on the read assistance information, (Step S540). Here, the evaluation information may include feature information modified by a user's selection after being created, proposed, created or suggested by the deep learning model, and information of a corresponding report corresponding to the feature information. (Ii) a basic reading provided to correspond to the finding information; and (iii) a reading option provided to correspond to the finding information. Or (iv) any of (i) to (iii) above may be the result of being modified by the external entity.

In operation S540, the computing device 200 may provide a user interface (S532) (not shown) through the predetermined output device so that the user can select a report corresponding to the finding information, and In response to the request of the user, the computing device 200 may include the step S534 (not shown) of acquiring the findings information and the information of the report corresponding to the findings information as evaluation information.

Here, the report selection provided according to the findings information may be as shown in Table 1 below, for example.

category Report (size) (Area, characteristic) (location) normal FINDINGS:
Normal size and shape of heart.
Normal pulmonary vascularity.
No abnormality in both lung fields and pleurae.
Normal bony thorax.
CONCLUSION: Normal chest. nodule There is an about 1cm-sized, 2cm-sized, 3cm-sized nodule in right upper lung zone, in right middle lung zone, in right lower lung zone, etc ... consolidation There is focal, extensive, multifocal consolidation 〃 pleural effusion There is scanty, small amount of, large amount of, loculated plueral effusion in right hemithorax, in left hemithorax, in both hemithoraces pneumothorax There is 〃 pneumothorax 〃 interstitial opacity There is subtle, mild, extensive interstitial opacity in both basal lungs, in both lungs, etc ...

For example, if the finding information is normal, the report shown in Table 1 may be provided as a default report, or simply a blank may be provided as a base report, and if normal findings are selected after other findings are selected, The information in the readings or final report information 840 for the information may be cleared.

As another example, if the finding information is a nodule finding, referring to Table 1, the report selection sheet for that determination finding may be a size (e.g., 1 cm-sized, 2 cm-sized, 3 cm- (E.g., an in right upper lung zone, an in-right middle lung, an in-right lower lung zone, etc.), and the rest are also similar to those of ordinary skill in the art.

The process of inputting such a read-right option to the computing device 200 through a user interface will be conceptually described below.

FIG. 10 and FIG. 11 are diagrams illustrating the configuration of an assistive user interface provided to allow a user to input or modify a readout corresponding to feature information in the medical image reading support method according to the present invention.

Referring to FIG. 10, an interstitial opacity feature, which is an item in the list displayed in the AI proposal information 830, is generated or suggested by the deep learning model. Initial report information (denoted as 'There is extensive interstitial opacity in both lungs.') Is selected by the deep learning model. The portion of the report information that can be selected by the user may be displayed differently from other portions, for example, at least one of color, saturation, and brightness may be displayed differently. In the example of FIG. 10, 'extensive' and 'in both lungs' portions are marked differently. At this time, a user, e.g., a doctor, can select another read-ahead option by manipulating (e.g., clicking with the mouse) the 'extensive' text displayed on the user interface. For example, a read-only selection may be provided as a drop-down menu for convenience, and the user may select the item of sub-selection ('subtle', 'mild', 'extensive' ), The information of the read-out statement can be modified.

Referring to the user interface of FIG. 11, information of a report is selected for an item in the list displayed in the AI proposal information 830, i.e., an epileptic turbid appearance, and a UI element (e.g., (E.g., an operation of pressing the button) is detected, the computing device 200 causes the information of the selected report to be written to the final report information 840 in the form of a text . This final report information 840 may be directly modified or edited by the user.

Preferably, in step S540, the user can further write a conclusion into the final report information 840, which is a portion labeled 'CONCLUSION' at the bottom of the final report information 840 illustrated in FIG. 11 As illustrated.

Advantageously, in step S540, the user may manipulate UI elements, such as the 'save' button provided on the user interface, to cause the user to pass and store the evaluation information or final report to an external entity, such as the PACS 420 It will be possible.

In this embodiment, it is possible to support the reading of the chest image based on the previously learned deep learning model, and evaluation information including information of the created, proposed or modified view information and information corresponding to the generated view information is again sent to the deep learning model The method of supporting reading of a medical image according to this embodiment of the present invention is characterized in that the depth learning model is more accurate than the deep learning model, , The processor 220 acquires the evaluation information through the result input module 350 and updates the deep learning model through the update module 360 based on the evaluation information, (Step S550). At this time, since the chest image, the finding information, and the information of the report which are not taken into consideration in the previous learning are further considered, the error in the previous learning can be corrected, so that the accuracy of the deep learning model is improved, There is an advantage that the effect of the medical image reading support method according to the present invention is continuously and cumulatively improved. The evaluation information for this need to be provided by a trusted external entity, such as a medical professional.

In another embodiment of the method for supporting medical image reading according to the present invention, the reading of the chest image may be performed by a plurality of users, whereby at least one user who is supported to read the chest image, A plurality of readings can be performed for one of the chest images by selecting among Kn users included in the n-th user group, and thereby, for at least one of the read assistance information and the evaluation information, Cross-validation may be achieved.

ROC curve.

Since the deep learning model used in the medical image reading support method of the present invention serves as a classifier, the performance of the deep learning model can be confirmed through an ROC curve (receiver operating characteristic curve) representing the performance of the classifier .

12 is a diagram illustrating an ROC curve as a tool for evaluating the performance of the deep running model. 12, the values on the horizontal axis of the ROC curve graph are 1-specificity = false positive rate, and the specificity is the specificity = true negative / ( False positive + true negative. &Lt; / RTI &gt; In other words, the peculiarity means the ratio of judging 'wrong (voice)' to be wrong. Therefore, it can be said that the percentage of false positives judged to be 'wrong' (positive) In addition, the values on the vertical axis of the ROC curve are the sensitivity = true positive rate, sensitivity = true positive / true positive + false negative )). In other words, the sensitivity means the ratio of judging the 'right' to be right, so the tendency of the graph to be 'up' is that the percentage of mistaken 'wrong' is wrong. Therefore, as the classifier correctly determines, the area under curve (AUC) of the graph curve increases. The AUC is 0.5 when the classifier does not have any classification performance that it judges correctly. Usually, the AUC values are noninformative (AUC = 0.5), less accurate (0.5 <AUC ≤ 0.7), moderate (0.7 <AUC ≤ 0.9), very accurate (0.9 <AUC < AUC = 1).

Testing of an exemplary deep-running model of the present invention using chest PA image data.

An example of the ROC curve of the exemplary deep-run model used in the chest view read support method according to the present invention is shown in FIG.

Referring to FIG. 13, the ROC curve of the result obtained by evaluating the ability of the present inventor to classify normal and abnormal abnormalities using the above deep learning model with respect to a subject obtained from Asan Medical Center (A hospital) Color, and the ROC curve of the result of the evaluation on the subject obtained from the hospital B is shown in blue.

The AUC, sensitivity, and specificity of these ROC curves are summarized in Table 2 below.

AUC (area under curve) Sensitivity (T = 0.5) Specificity (T = 0.5) Hospital A 0.99 97.6% 96.7% Hospital B 0.98 96.5% 95.5%

Hypothesis verification for an exemplary deep-running model of the present invention. Hypothesis testing was conducted with the null hypothesis that AUC = 0.5, which means that no model has classification performance at all, and the hypothetical hypothesis that the exemplary deep learning model has classification performance. If the p-value is lower than the predetermined probability, the null hypothesis is rejected and the alternative hypothesis is adopted. The predetermined probability is referred to as the significance level, , A value of 0.05 is usually taken at a significant level. The significance of the deep learning model, which was found using the chest PA image data obtained by the present inventors, was 0.001 for AUC = 0.98. As can be seen from the point of view. The p-values of this deep learning model are significantly lower than the significance level of 0.05, demonstrating that the null hypothesis is rejected and has an extremely significant classification performance.

Table 3 shows the results of evaluating the individual findings classified by the deep learning model according to the corresponding diseases. The validation dataset source of the data set used for verification is B hospital.

Pneumothath Hardening Pleural effusion nodule Epilepsy turbidity Test 1 96% 94% 95% 95% 89%

The numerical values shown in the above Tables 2 and 3 show that the deep learning model shows very significant results, that is, the deep learning model is a very significant model capable of generating highly accurate information. .

As described above, according to the present invention, the chest image can be screened for a plurality of diseases, so that not only a plurality of diseases can be predicted, but also lesions corresponding to the plurality of diseases And it is also possible to automate generation and creation of a report so that conventional medical professionals can efficiently and precisely read out the chest PA image based on their experience or knowledge Thereby saving time for medical staff, improving the quality of medical care, and innovating the workflow in the medical field.

Based on the description of the above embodiments, one of ordinary skill in the art can clearly understand that the present invention can be achieved through a combination of software and hardware, or can be accomplished by hardware alone. Objects of the technical solution of the present invention or portions contributing to the prior art can be recorded in a machine-readable recording medium implemented in the form of program instructions that can be executed through various computer components. The machine-readable recording medium may include program commands, data files, data structures, and the like, alone or in combination. The program instructions recorded on the machine-readable recording medium may be those specially designed and constructed for the present invention or may be those known to those of ordinary skill in the computer software arts. Examples of the machine-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CD-ROM, DVD, Blu-ray, magneto-optical media such as floptical disks magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those generated by the compiler, as well as bytecode, as well as high-level language code that can be executed by a computer using an interpreter, etc.

The hardware device may be configured to operate as one or more software modules for performing the processing according to the present invention, and vice versa. The hardware device may include a processor, such as a CPU, a GPU, a TPU, coupled to a memory, such as ROM / RAM, for storing program instructions, and configured to execute instructions stored in the memory, And the like. In addition, the hardware device may include a keyboard, a mouse, and other external input devices for receiving commands generated by the developers.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, And various modifications and changes may be made thereto by those skilled in the art to which the present invention pertains.

Therefore, the spirit of the present invention should not be construed as being limited to the above-described embodiments, and all of the equivalents or equivalents of the claims, as well as the following claims, I will say.

Equally or equivalently modified such methods will include logically equivalent methods which can yield, for example, the same results as those of the method according to the invention.

Claims (6)

A method of supporting reading of a medical image of a subject,
(a) supporting a computing device to acquire a medical image of the subject or acquire another device associated with the computing device;
(b) supporting the computing device to generate or cause the other device to generate read assistance information about the medical image based on a machine learning model from an input of the medical image; And
(c) when the read assistance information is generated, causing the computing device to provide the read assistance information to an external entity or to provide the other device with the read assistance information
Lt; / RTI &gt;
The read-
(i) at least one finding information for the medical image, or
(ii) a mask image corresponding to each of the at least one finding information and the finding information,
Wherein the mask image is provided as an indication to support individual disposition information corresponding to the mask image by being superposed on at least a portion of the medical image,
The method comprises:
(d) the computing device acquiring evaluation information about the read assistance information or supporting the other device to acquire
Further comprising:
Wherein the evaluation information includes information of the inspection information corresponding to the finding information and the finding information,
The report includes, as an initial value,
(i) a basic report provided so as to correspond to the normal findings when the findings information is normal findings, or (ii) at least one set provided to correspond to the findings information when the findings information is not normal findings A plurality of As a selectable readout which is a result selected by the computing device based on the machine learning model,
The step (d)
Wherein when the modification input of the external entity is obtained, the computing device supports modifying the response or modifying the other device in response to a modification input of the external entity. The method according to claim 1,
The step (b)
Wherein the computing device is adapted to generate or generate the read assistance information by performing multi-feature segmentation and classification based on the machine learning model, Wherein the region of interest segmentation and the image classification for the feature of each of the at least one feature information are simultaneously or sequentially performed. The method according to claim 1,
(e) when the evaluation information is obtained, the computing device updates the machine learning model or supports the other device to update based on the evaluation information
Wherein the medical image reading support method further comprises: A computer program stored in a machine readable non-transitory medium, comprising instructions embodied in a computer-readable medium for causing a computing device to perform the method of any one of claims 1 to 3. A computing device for supporting reading of a medical image of a subject,
A communication unit for acquiring a medical image of the subject; And
A processor for generating read assistance information on the medical image based on a machine learning model from the input of the medical image or for generating another assistant information through the communication unit,
, &Lt; / RTI &
The processor comprising:
And when the read assistance information is generated, providing the read assistance information to an external entity or providing the other device with assistance,
The read-
(i) at least one finding information for the medical image, or
(ii) a mask image corresponding to each of the at least one finding information and the finding information,
Wherein the mask image is provided as an indication to support individual disposition information corresponding to the mask image by being superposed on at least a portion of the medical image,
The processor comprising:
Performing the process of obtaining evaluation information on the read assistance information or assisting the other device to acquire the evaluation information,
Wherein the evaluation information includes information of the inspection information corresponding to the finding information and the finding information,
The report includes, as an initial value,
(i) a basic report provided so as to correspond to the normal findings when the findings information is normal findings, or (ii) at least one set provided to correspond to the findings information when the findings information is not normal findings ) As a selectable readout that is a result selected by the computing device based on the machine learning model,
The processor comprising:
And when the correction input of the external entity is obtained, corrects the readout in response to the correction input of the external entity or supports the other device linked through the communication unit to correct it. 6. The method of claim 5,
The processor,
And when the evaluation information is acquired, updates the machine learning model or updates the other device based on the evaluation information.

Images