KR20240053280A - Deep learning-based augmented reality system for medical diagnosis - Google Patents

Abstract

A deep learning-based augmented reality system for disease diagnosis according to an embodiment of the present invention generates, transmits and receives images and video medical information in digital format through at least one medical equipment, and medical information of the medical terminal. It includes a client terminal that receives and visualizes images and videos, and a server that is connected to the client terminal and analyzes, processes, and stores the medical information written in the PACS viewer, wherein the server stores the medical information written in the PACS viewer. ROI module that captures and generates region of interest information at the expected location of the lesion through ROI extraction based on the captured medical information, information on the region of interest through learning a deep learning model, and information in a database where the characteristics of each lesion are stored. Deep learning for disease diagnosis including an extraction module that compares and extracts lesion information, a visualization module that converts the lesion information into augmented virtual visual information, and an overlay module that overlays the virtual visual information on the medical information of the PACS viewer. Provides a base augmented reality system.

Description

Deep learning-based augmented reality system for medical diagnosis}

The present invention is a deep learning method for disease diagnosis that is initiated by overlaying the presence and location of lesions in the form of symptom reality on a PACS viewer based on digital image and video medical information generated through medical equipment through deep learning. This is about a learning-based augmented reality system.

In medicine, the importance of optimal diagnosis or prescription through data analysis is increasing. However, with the development of various medical devices and hospital information systems, the amount of medical data generated and stored is explosively increasing.

Despite this surge in data, the shortage of medical staff is expected to accelerate.

In order to solve this shortage of medical personnel, there is a need for technology that speeds up the judgment of medical staff and increases reliability by increasing the accuracy of judgment.

Most medical industries use PACS (Picture Archiving Communication System) as a storage and transmission system for medical images to process large amounts of medical data.

PACS acquires and stores video and image information in a digital state, and provides integrated functions to transmit and search the readings and medical records.

In particular, PACS converts all radiological test results taken by CT, MRI, PET, and SPECT into digital images and stores them in a large-capacity memory device at the same time as they are taken, so that radiologists can read them through a monitor to monitor the patient's condition. The doctor can upload the captured film into a computer image within seconds and use it for medical treatment.

However, this method of medical data using PACS determines whether there is a lesion and the type of lesion by looking at the disclosed image.

At this time, if the lesions disclosed in the PACS are not identified by mistake due to the carelessness of the medical staff, or if a situation occurs where the medical staff does not understand due to lack of medical knowledge, a serious medical accident may occur.

To solve this problem, by using artificial intelligence to detect and display lesion information and adding medical knowledge of the medical staff, a quick judgment on the lesion information disclosed in PACS is possible and at the same time, it is possible to diagnose diseases that can reduce medical staff's mistakes. The development of a deep learning-based augmented reality system was urgently needed.

As a prior art, Korean Patent No. 10-2108401 “Artificial intelligence reading server, PACS-based image processing system and method including same” is disclosed.

The present invention was invented to solve the problems described above. Medical information acquired through medical equipment is disclosed in a digital image format in PACS, and the lesion information in the existing database is compared with the images disclosed in PACS through a deep learning method. A deep learning-based augmented reality system for disease diagnosis that generates lesion information, converts the lesion information into virtual visual information, and overlays the virtual visual information on the image of the previously launched PACS to increase the reliability of lesion detection. The purpose is to provide

A deep learning-based augmented reality system for disease diagnosis according to an embodiment of the present invention to solve the problems presented above is a medical terminal that generates, transmits and receives images and video medical information in digital format through at least one medical device. , a client terminal that receives and visualizes the medical information of the medical terminal as images and videos, and a server that is connected to the client terminal and analyzes, processes, and stores the medical information written in the PACS viewer, the server comprising: ROI module that captures the display screen of the client terminal and generates region of interest information from the expected location of medical information on the captured screen, information on the region of interest through learning a deep learning model, and information in a database where the characteristics of each lesion are stored It may include an extraction module that compares and extracts lesion information, a visualization module that converts the lesion information into an enhanced virtual visual information form, and an overlay module that overlays the virtual visual information on the medical information of the PACS viewer.

In addition, the client terminal further includes a UI module, and the UI module includes a UI function for displaying or hiding the lesion information visualized through the visualization module according to selection and adjusting transparency of the lesion information. can do.

Additionally, the ROI module may be capable of manually and automatically setting the location of the medical information.

In addition, the ROI module may be characterized in that it minimizes the difference in resolution between the region of interest information after capture and the medical information before capture by adjusting the resolution size when extracting region of interest information after specifying the location of the medical information. You can.

Additionally, the extraction module may be characterized in that, when extracting the lesion information, it is segmented according to the type of lesion and the symptoms of the lesion.

In addition, the extraction module improves visibility when the lesion information is overlaid on the medical information of the PACS viewer by enabling setting color differences according to the type and characteristics of each lesion when segmenting the lesion information. It can be characterized as being ordered.

Additionally, the visualization module may improve lesion detection accuracy by adjusting the depth, width, and resolution of the virtual visualization information at a certain ratio.

In addition, the overlay module may be characterized in that when overlaying the virtual visualization information on the PACS viewer, the name of the lesion and information about the lesion are overlaid together.

Additionally, the overlay module may further include a learning module that adds the lesion information to the database by learning the lesion information after the overlay is completed.

The deep learning-based augmented reality system for disease diagnosis according to an embodiment of the present invention can increase the reliability of detecting lesions by converting lesion information into virtual visual information and overlaying the virtual visual information on the image of the previously launched PACS. there is.

Additionally, by capturing the image of the PACS viewer and generating virtual visual information accordingly, it can be used without changing the system configuration of the PACS viewer.

Additionally, images and videos from the PACS viewer can be transmitted to the client terminal without a removable disk.

1 is a block diagram of a deep learning-based augmented reality system for disease diagnosis according to an embodiment of the present invention.
Figure 2 is an example diagram showing the medical information transmission process of a medical terminal.
Figure 3 is an enlarged view showing the UI module of Figure 2.
Figure 4 is a block diagram of how a server operates according to an embodiment of the present invention.
Figure 5 is an example diagram showing the process of generating region of interest information through the ROI module.
Figure 6 is an example diagram showing the lesion information generation process through the extraction module.
Figure 7 is an example diagram showing lesion information generated through Figure 6.
Figure 8 is an example diagram showing augmented virtual visualization information.
Figure 9 is an example diagram showing virtual visual information overlaid on medical information.

Hereinafter, the description of the present invention with reference to the drawings is not limited to specific embodiments, and various changes may be made and various embodiments may be possible. In addition, the content described below should be understood to include all conversions, equivalents, and substitutes included in the spirit and technical scope of the present invention.

In the following description, terms such as first, second, etc. are terms used to describe various components, and their meaning is not limited, and is used only for the purpose of distinguishing one component from other components.

Like reference numerals used throughout this specification refer to like elements.

As used herein, singular expressions include plural expressions, unless the context clearly dictates otherwise. In addition, terms such as “comprise,” “provide,” or “have” used below are intended to designate the presence of features, numbers, steps, operations, components, parts, or a combination thereof described in the specification. It should be construed and understood as not precluding the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the technical field to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having meanings consistent with the meanings they have in the context of the related technology, and should not be interpreted in an idealized or overly formal sense unless explicitly defined in the present application. No.

In addition, when describing with reference to the accompanying drawings, identical components will be assigned the same reference numerals regardless of the reference numerals, and overlapping descriptions thereof will be omitted. In describing the present invention, if it is determined that a detailed description of related known technologies may unnecessarily obscure the gist of the present invention, the detailed description will be omitted.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying FIGS. 1 to 9.

Figure 1 is a block diagram of a deep learning-based augmented reality system for disease diagnosis according to an embodiment of the present invention, Figure 2 is an example diagram showing a UI module formed in the form of a user interface on a display, and Figure 3 is a diagram of Figure 2 This is an enlarged view of the UI module.

First, referring to FIG. 1, the deep learning-based augmented reality system 1 for disease diagnosis may include a medical terminal 100, a client terminal 200, and a server 300.

The medical terminal 100 can digitize information captured as images or images of the inside and outside of the patient's body through medical equipment such as computed tomography (CT), X-ray, and endoscopy equipment.

At this time, the medical terminal 100 may generate images and video medical information 110 in digital format through at least one medical equipment.

The medical information 110 is preferably formed in a DCM format file format, but is not limited to this.

At this time, DCM (Digital Imaging and Communications in Medicine, DICOM) is an abbreviation for medical digital imaging and communication standard and will be used as a general term for various standards used for digital image expression and communication in medical devices.

The client terminal 200 may receive medical information 110 generated from the medical terminal 100 in the form of images and videos.

At this time, the client terminal 200 is preferably formed to include a display 210.

The display 210 may visualize the medical information 110 in the form of images and videos received by the client terminal 200 and provide them to the user.

Referring to FIG. 2, the display 210 can simultaneously display the UI module 220 and the medical information 110 written in the PACS viewer on one screen.

At this time, the UI module 220 may be formed in the form of a user interface on the screen of the display 210.

According to the dictionary definition, the user interface form refers to the interaction between a person and a computer system program, and as it is easy for a person skilled in the art to understand, detailed explanations will be omitted.

Meanwhile, the UI module 220 can display or hide lesion information 321, which will be described in detail below, depending on selection, and adjust the transparency of the lesion information 321.

Referring to FIG. 3 in detail, the UI module 220 may include an extraction range control UI 221, a lesion selection UI 222, and a transparency control UI 223.

At this time, the UI module 220 can be freely adjusted in size on the display 210.

The server 300 may be formed by being connected to the client terminal 200.

At this time, the server 300 may analyze, process, and store the medical information 110 recorded in the PACS viewer.

The PACS viewer is a medical image storage and transmission system that stores and reads medical information (110) in DCM format, adds diagnostic records, etc., and transmits it wirelessly to other terminals connected to the network according to the DICOM protocol standard. It can be equipped with a communication function.

Figure 4 is a block diagram of the operation method of the server according to an embodiment of the present invention, Figure 5 is an example diagram showing the process of generating region of interest information through the ROI module, and Figure 6 is a diagram showing the process of generating lesion information through the extraction module. Figure 7 is an example diagram showing lesion information generated through Figure 6, Figure 8 is an example diagram showing augmented virtual visualization information, and Figure 9 shows virtual visual information overlaid on medical information. This is also an example.

The server 300 may include an ROI module 310, an extraction module 320, a visualization module 330, and an overlay module 340.

Specifically, referring to FIG. 5 , the ROI module 310 may capture the screen of the display 210 and generate region of interest information 311 by estimating the location of the medical information 110 in the captured screen.

At this time, the region of interest information 311 can be extracted by adjusting the resolution size at the time of extraction, thereby minimizing the difference in resolution from the medical information 110 before capture.

In order to overcome the resolution difference described above, the resolution can be maintained by using a method of maintaining the ratio of the number of pixels per unit area of the image and video constant.

Additionally, the ROI module 310 can manually and automatically set the extraction range 312 of the medical information 110 through the extraction range adjustment UI 221.

When automatically set, the extraction range 312 can be extracted from an image captured of the display 210 screen based on region of interest information 311 previously learned through YOLO, one of the deep learning models.

At this time, the deep learning model YOLO is one of the object detection models, and processes the image captured through the above process through a convolutional neural network. Accordingly, the region of interest information 311 can be estimated and displayed on the display 210 screen. .

Here, the convolutional neural network is a deep neural network technique that can effectively process images by applying filtering techniques to artificial neural networks. It classifies images through a process in which each element of the filter expressed in a matrix is automatically learned to be suitable for data processing. I'm talking about technique. Since the content is easy for a person skilled in the art to understand, detailed explanation will be omitted.

At this time, YOLO can set the extraction range 312 on the display 210 screen by comparing the captured image in detail with the region of interest information 311 data stored in the database.

Additionally, when setting the extraction range 312 manually, the region of interest information 311 can be set directly to prepare for error situations that may occur during automatic setting.

Next, referring to FIGS. 6 and 7, the extraction module 320 compares the region of interest information 311 with the information through deep learning model learning and the database (D) information in which the characteristics of each lesion are stored to obtain the lesion information 321. can be extracted.

The lesion information 321 can be subdivided and extracted according to the type of lesion and the symptoms of the lesion.

For example, in the case of fundus disease, the lesion information 321 can be subdivided into fundus diseases such as EX (Hard Exudate), SE (Soft Exudate), HE (Hemorrhage), and MA (Microaneurysm).

Through this, the segmentation process of the lesion information 321 can identify which lesion occurred at which location in a short time, thereby increasing the detection speed and reliability of the lesion.

Additionally, the lesion information 321 can be set to show color differences according to the type and characteristics of each lesion when segmented.

Accordingly, the lesion information 321 can increase the visibility of the user who determines and determines the presence and location of the lesion.

In the case of segmentation of lesion information 321, it is not limited to the fundus disease described above and can be applied to all images and images that can be acquired through the PACS viewer.

After the lesion information 321 is subdivided and extracted, the lesion selection UI 222 can be used to select whether to display it or not according to the type of lesion and the symptoms of the lesion.

At this time, the lesion information 321 can control whether or not to display segmented lesions through the checkbox (C) displayed in the lesion selection UI 222.

Referring to FIG. 8, the visualization module 330 may convert lesion information 321 into enhanced virtual visual information 331.

At this time, the visualization module 330 can adjust the lesion information 321 to a desired transparency through the transparency adjustment UI 223.

Here, the fact that the virtual visual information 331 is augmented refers to the existing AR (Augmented Reality) technology, which refers to a technology that overlaps a 3D virtual image on a real image or video to express it as a single video or image. It can be interpreted as doing so.

At this time, the virtual visual information 331 can improve lesion detection accuracy by adjusting the depth, width, and resolution at a certain ratio.

The virtual visual information 331 can find the optimal ratio by introducing a Compound Scaling equation that appropriately combines the ratios of depth, width, and resolution through EfficientNet and adjust the depth, width, and resolution to a certain ratio.

The overlay module 340 can overlay the virtual visual information 331 on the medical information 110 of the PACS viewer.

Referring to FIG. 9, when overlaying the virtual visualization information 331 on the medical information 110 of the PACS viewer, the overlay module 340 can overlay the name of the lesion and information about the lesion together.

At this time, the overlay module 340 can display or hide the lesion information 321 according to selection and adjust the transparency of the lesion information 321 through the UI module 220 even after it is overlaid on the medical information 110. .

Additionally, the overlay module 340 may further include a learning module 341 that adds the lesion information 321 to the database D by learning the lesion information 321 after the overlay is completed.

The learning module 341 can increase the reliability of lesion detection by transmitting the lesion information 321 to the database D through a feedback function and relearning it.

In addition, the learning module 341 learns through a feedback action each time the virtual visual information 331 is overlaid on the medical information 110 through the above series of processes, which will gradually increase the reliability of lesion detection and overlay accuracy. You can.

Although embodiments of the present invention have been described above with reference to the attached drawings, those skilled in the art can realize that the present invention can be implemented in other specific forms without changing the technical idea or essential features of the present invention. You will be able to understand it. Therefore, the embodiments described above are illustrative in all respects and are not restrictive.

1: Deep learning-based augmented reality system for disease diagnosis
100: Medical terminal
110: Medical information
200: client terminal
210: display
220: UI module
221: Extraction range adjustment UI
222: Lesion selection UI
223: Transparency control UI
300: server
310: ROI module
311: Area of interest information
312: Extraction range
320: Extraction module
321: Lesion information
330: Visualization module
331: Virtual visual information
340: Overlay module
341: Learning module
D: database
C: checkbox 300

Claims (9)

A medical terminal that generates, transmits and receives images and video medical information in digital format through at least one medical device;
A client terminal that receives and visualizes medical information from the medical terminal as images and videos, and
It is connected to the client terminal and includes a server that analyzes, processes, and stores the medical information recorded in the PACS viewer,
The server is,
an ROI module that captures a display screen of the client terminal and generates region of interest information at an expected location of medical information in the captured screen; An extraction module that extracts lesion information by comparing the region of interest information with information through deep learning model learning and information in a database storing the characteristics of each lesion;
A visualization module that converts the lesion information into enhanced virtual visual information and
A deep learning-based augmented reality system for disease diagnosis including an overlay module that overlays the virtual visual information on the medical information of the PACS viewer.

According to clause 1,
The client terminal is,
Includes more UI modules,
The UI module is,
A deep learning-based augmented reality system for disease diagnosis, comprising a UI function that displays or hides the lesion information visualized through the visualization module according to selection and adjusts transparency of the lesion information.
According to clause 1,
The ROI module is,
A deep learning-based augmented reality system for disease diagnosis, characterized in that the capture range of the medical information can be manually and automatically set.
According to clause 3,
The ROI module is,
A deep learning-based augmented reality system for disease diagnosis, characterized in that the resolution difference between the region of interest information after capture and the medical information before capture is minimized by extracting the region of interest information after capture by adjusting the resolution size.
According to clause 1,
The extraction module is,
A deep learning-based augmented reality system for disease diagnosis, characterized in that when extracting the lesion information, it is segmented according to the type of lesion and the symptoms of the lesion.
According to clause 6,
The extraction module is,
When segmenting the lesion information, it is possible to set color differences according to the type and characteristics of each lesion, thereby improving visibility when the lesion information is overlaid on the medical information of the PACS viewer. A deep learning-based augmented reality system for.
According to clause 1,
The visualization module is,
A deep learning-based augmented reality system for disease diagnosis, characterized in that it improves lesion detection accuracy by adjusting the depth, width, and resolution of the virtual visualization information at a certain ratio.
According to clause 1,
The overlay module is,
A deep learning-based augmented reality system for disease diagnosis, characterized in that when the virtual visualization information is overlaid on the PACS viewer, the name of the lesion and information about the lesion are overlaid together.
According to clause 1,
The overlay module is,
A deep learning-based augmented reality system for disease diagnosis, further comprising a learning module that adds to the database by learning the lesion information after completing the overlay.

Images