KR20220143187A - Method and apparatus for automatically extracting emphysema using deep learning - Google Patents

Abstract

The present invention relates to a method for automatically extracting images of emphysema using deep learning and an apparatus therefor, which can obtain accurate information on improvement or aggravation of a condition. According to one embodiment of the present invention, the method for automatically extracting images of emphysema using deep learning may include: a step of acquiring an image group including a plurality of images generated in response to consecutive thoracic volumes; a step of generating first image information, which has extracted images of a lung area from each image, by inputting the image group to a trained first network function; a step of generating second image information, which has detected a predetermined lesion area from each image, by inputting the image group to a trained second network function; and a step of generating a second image including the lesion area based on the first image information and the second image information.

Description

Method and apparatus for automatic emphysema extraction using deep learning

The present invention relates to a method and apparatus for automatically extracting an emphysema site using deep learning.

Unlike X-ray imaging, computed tomography (CT) acquires a transverse cross-sectional image of the human body. Compared to simple X-ray imaging, CT has the advantage of being able to see structures and lesions more clearly because there is less overlap of structures. In addition, CT scan is a basic examination method when a lesion is suspected in most organs and diseases and a detailed examination is necessary because the examination cost is cheaper and examination time is shorter than that of MRI.

Recently, with the spread of multi-channel CT (MDCT, multi-detector CT), it is possible to freely obtain desired cross-sectional and three-dimensional (3D) images like MRI by reconstructing the image after imaging. have.

However, the number of specialists who read CT images is limited, and there is no system capable of requesting medical image reading. Because of this, even if an individual owns a CT image, reliable reading cannot be requested online.

In particular, since pneumonia caused by a viral infection such as coronavirus can progress to a severe state in a short period of time, a prompt and accurate diagnosis is requested, but there is also a problem that symptoms can rapidly worsen while waiting for a doctor's reading.

Accordingly, Korea Patent Publication No. 1887194 has introduced a technique for reading a chest image based on a deep learning model when a chest image of a subject is input.

The present invention is to solve the above problems, and an object of the present invention is to provide a method and apparatus for automatic extraction of emphysema sites using deep learning.

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

According to an embodiment of the present invention, there is provided a method for automatic extraction of emphysema sites using deep learning. The method may include: acquiring an image group including a plurality of images generated corresponding to successive volumes of the chest; generating first image information obtained by extracting a lung region from each image by inputting the image group into a learned first network function; generating second image information obtained by detecting a predetermined lesion region from each image by inputting the image group into a learned second network function; and generating a second image including the lesion region based on the first image information and the second image information.

According to the present invention, the accuracy of automatic lung segmentation can be increased by automatically extracting the site where the pleural effusion exists and recognizing it as a part of the lung rather than other soft tissues.

In addition, according to the present invention, if the extracted lung region is accurate, it is possible to more accurately calculate the total volume of the lung. Not only can it be analyzed and diagnosed, but also information about improvement or deterioration of the condition can be accurately acquired.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the following description.

In order to more fully understand the drawings cited herein, a brief description of each drawing is provided.
1 is a flowchart for explaining a method for automatically extracting an emphysema site using deep learning according to an embodiment of the present invention.
2 is a flowchart for explaining a method for automatically extracting an emphysema site using deep learning according to an embodiment of the present invention.
3 exemplarily shows the operation of a network function in the method for automatic extraction of emphysema sites using deep learning according to an embodiment of the present invention.
4 exemplarily shows an image generated through the automatic extraction method for emphysema site using deep learning according to an embodiment of the present invention.
5 and 6 exemplarily show images generated through the automatic extraction method for emphysema sites using deep learning according to an embodiment of the present invention.
7 is a block diagram schematically illustrating the configuration of an apparatus according to an embodiment of the present invention.

Since the technical spirit of the present invention may have various changes and may have various embodiments, specific embodiments will be illustrated in the drawings and described in detail. However, this is not intended to limit the technical spirit of the present invention to specific embodiments, and it should be understood to include all changes, equivalents or substitutes included in the scope of the technical spirit of the present invention.

In describing the technical idea of the present invention, if it is determined that a detailed description of a related known technology may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted. In addition, numbers (eg, first, second, etc.) used in the description of the present invention are merely identification symbols for distinguishing one component from another.

In addition, in the present invention, when a component is referred to as “connected” or “connected” with another component, the component may be directly connected or directly connected to the other component, but in particular It should be understood that, unless there is a description to the contrary, it may be connected or connected through another element in the middle.

In addition, terms such as "~ unit", "~ group", "~ character", and "~ module" described in the present invention mean a unit that processes at least one function or operation, which is a processor, a micro Processor (Micro Processor), Micro Controller (Micro Controller), CPU (Central Processing Unit), GPU (Graphics Processing Unit), APU (Accelerate Processor Unit), DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA It may be implemented by hardware or software such as (Field Programmable Gate Array), or a combination of hardware and software.

And it is intended to clarify that the classification of the constituent parts in the present invention is merely a division for each main function that each constituent unit is responsible for. That is, two or more components to be described below may be combined into one component, or one component may be divided into two or more for each more subdivided function. In addition, each of the constituent units to be described below may additionally perform some or all of the functions of other constituent units in addition to the main function it is responsible for. Of course, it can also be performed by being dedicated to it.

Hereinafter, embodiments of the present invention will be described in detail in turn.

Throughout this specification, a network function may be used synonymously with a neural network and/or a neural network. Here, a neural network (neural network) may be composed of a set of interconnected computational units that may be generally referred to as nodes, and these nodes may be referred to as neurons. A neural network is generally configured to include a plurality of nodes. Nodes constituting the neural network may be interconnected by one or more links.

Some of the nodes constituting the neural network may configure one layer based on distances from the initial input node. For example, a set of nodes having a distance of n from the initial input node may constitute n layers.

The neural network described herein may include a deep neural network (DNN) including a plurality of hidden layers in addition to an input layer and an output layer.

1 is a flowchart for explaining a method for automatically extracting an emphysema site using deep learning according to an embodiment of the present invention.

The method 100 for automatic extraction of emphysema site using deep learning according to an embodiment of the present invention is performed on a personal computer, a workstation, a computer device for a server, etc. with computing power, or a separate device for this can be performed in

In addition, the emphysema site automatic extraction method 100 using deep learning may be performed in one or more computing devices. For example, at least one or more steps of the automatic emphysema site extraction method 100 using deep learning according to an embodiment of the present invention may be performed in a client device, and other steps may be performed in a server device. In this case, the client device and the server device may be connected through a network to transmit/receive calculation results. Alternatively, the emphysema site automatic extraction method 100 using deep learning may be performed by distributed computing technology.

In operation S110 , the device may acquire an image group including a plurality of 2D images generated corresponding to the continuous volume of the chest. For example, the image group may be a computed tomography (CT) image of the subject's chest. That is, the plurality of 2D images constituting the image group may be a plurality of slices obtained by continuously photographing a cross section of the chest including the lungs in one direction through the computed tomography method. By stacking these two-dimensional images, it is possible to acquire three-dimensional image information on the chest.

In step S120, the device inputs the image group to the learned first network function, extracts a lung region from each 2D image, and generates first image information obtained by dividing the lung region into a plurality of lobe regions can do. The first network function may be one in which learning of lung region extraction and lobe region segmentation has been performed in advance through learning data (eg, chest CT image on which lung region extraction and lobe region segmentation have been performed by an expert, etc.) .

In an embodiment, the first image information may be another 2D image obtained by masking the lung region and/or lobe region in a plurality of 2D images, or may be data including coordinate information for the lung region and/or lobe region.

In operation S130 , the device may generate second image information obtained by detecting a predetermined lesion region from each 2D image by inputting the image group into the learned second network function. The second network function may be one in which learning for lesion region detection has been performed in advance through learning data (eg, a chest CT image on which lesion region detection is performed by an expert, etc.).

In an embodiment, the second image information may be another two-dimensional image obtained by masking a lesion region on a plurality of two-dimensional images, or data including coordinate information on the lesion region.

In embodiments, the lesion region may include a region of pneumonia caused by a viral infection.

In operation S140 , the device may generate and/or display a 3D image including the lesion region based on the first image information and the second image information. For example, the 3D image may be displayed by superimposing a 3D lesion area generated based on the second image on a 3D lung image generated based on the first image information.

In an embodiment, the 3D image may include information on a plurality of divided lobe regions. For example, a plurality of 3D lobe regions corresponding to the lobe regions divided in step S120 may be displayed in a 3D image through different colors.

Meanwhile, although not shown, the method 100 may further include calculating a ratio of the lesion area to each of the three-dimensional lobe areas. This ratio is based on which part of the lung lesions such as pneumonia based on a specific virus are concentrated, where the lesions are concentrated according to the severity, where the lesions are concentrated according to the patient's physical characteristics, etc. It can be used as statistical data for judging or estimating

2 is a flowchart for explaining a method for automatically extracting an emphysema site using deep learning according to an embodiment of the present invention.

The method 200 of FIG. 2 may further include steps S210 to S240 in addition to the method 100 of FIG. 1 .

In operation S210 , the device may generate a plurality of second images corresponding to the plurality of 2D images based on at least one of the first image information and the second image information.

For example, the second image may be an image in which a lung region and/or a lobe region is displayed on each of the 2D images, or an image in which a lesion region is displayed. Also, for example, the second image may be an image in which both a lung region (and/or a lobe region) and a lesion region are displayed with respect to each of the 2D images.

In step S220, the device may receive a user input for setting the reference coordinates. The reference coordinate serves as a reference for selecting a two-dimensional second image to be displayed by matching the three-dimensional image, and may be a vertical coordinate value in a three-dimensional coordinate space to which the generated three-dimensional image is mapped.

For example, the user input may be an input such as clicking or scrolling an area of a 3D image displayed on the display device.

In an embodiment, the device may be set to display the second image corresponding to the reference coordinate set as a default value by matching the 3D image before the user input for the reference coordinate is received.

In step S230, the device may search for a second image corresponding to the reference coordinates.

For example, the 3D image may be generated in a predetermined 3D coordinate space by vertically stacking the first image information and the second image information obtained from the 2D image included in the input image group, 3 When the reference coordinate corresponding to one of the vertical coordinates of the dimensional image is input, the device may search for a second image corresponding to the corresponding position from among the plurality of second images.

In operation S240 , the device may display the searched second image by matching it with the 3D image based on the reference coordinates.

For example, the device may arrange the found second image to be perpendicular to the vertical coordinate axis of the 3D space, and display the lung region of the second image by matching the horizontal section of the 3D image (lung image).

Meanwhile, when the reference coordinates are changed by the user input, the device may update and display the second image. That is, the device may search for a new second image corresponding to the changed reference coordinates, match it with the 3D image, and display it again.

3 exemplarily shows the operation of a network function in the method for automatic extraction of emphysema sites using deep learning according to an embodiment of the present invention.

Referring to FIG. 3 , a chest CT image including a plurality of two-dimensional tomography images may be input to a learned first network function and a second network function, respectively.

Accordingly, the first network function extracts a lung region from each of a plurality of input two-dimensional tomography images, or in addition, divides the lung region into a plurality of lobe regions, and divides the extracted and/or separated regions into a two-dimensional tomography image. The first image information may be generated by masking each of the .

Meanwhile, the second network function may generate second image information by detecting a lesion area corresponding to pneumonia from each of the plurality of input 2D tomography images, and masking the detected lesion area to the 2D tomography image, respectively.

4 exemplarily shows an image generated through an automatic extraction method for emphysema sites using deep learning according to an embodiment of the present invention.

Referring to FIG. 4 , the apparatus may generate a 3D lung image based on first image information and second image information generated by the first network function and the second network function, respectively.

In an embodiment, the three-dimensional lung image may include five three-dimensional lobe regions 410 and a three-dimensional lesion region 420 distributed in each of the corresponding regions, as shown in (a) of FIG. 4 . have.

Also, in an embodiment, the 3D lung image may include the 3D lesion area 420 distributed over the entire lung area without distinction of the lobe area.

5 and 6 exemplarily show images generated through the automatic extraction method for emphysema sites using deep learning according to an embodiment of the present invention.

First, referring to FIG. 5 , the device generates a plurality of second images corresponding to each 2D tomography image based on the first image information and/or the second image information, and converts one of them into a 3D lung image It can be displayed by matching with . Here, each second image may include information on a lung region, a lobe region, and/or a lesion region.

As shown, a second image corresponding to a reference coordinate input by the user or set as a default value is selected, and the second image is arranged in a horizontal direction so that the three-dimensional lung image is perpendicular to the vertical coordinate axis, but The lung region may be displayed so that the 3D lung image is matched with the horizontal section.

Meanwhile, when the user performs a predetermined user input for changing the reference coordinates, as shown in FIG. 6 , the second image may be updated and displayed.

For example, if the user scrolls upwardly in the area on which the 3D lung image is displayed or the area around it, the reference coordinates increase, second images corresponding thereto are sequentially detected, and the detected second image is displayed in the vertical direction. It can be displayed in alignment with the 3D lung image while moving upward.

7 is a block diagram schematically illustrating the configuration of an apparatus according to an embodiment of the present invention.

The communication unit 710 may receive input data (such as a chest CT image) for reading the lesion. The communication unit 710 may include a wired/wireless communication unit. When the communication unit 710 includes a wired communication unit, the communication unit 710 is a local area network (LAN), a wide area network (WAN), a value added network (VAN), a mobile communication network ( mobile radio communication network), a satellite communication network, and one or more components that allow communication through a combination thereof. In addition, when the communication unit 710 includes a wireless communication unit, the communication unit 710 wirelessly transmits and receives data or signals using cellular communication, a wireless LAN (eg, Wi-Fi). can In an embodiment, the communication unit may transmit/receive data or signals to and from an external device or an external server under the control of the processor 740 .

The input unit 720 may receive various user commands through external manipulation. To this end, the input unit 720 may include or connect one or more input devices. For example, the input unit 720 may be connected to an interface for various inputs, such as a keypad and a mouse, to receive a user command. To this end, the input unit 720 may include an interface such as a Thunderbolt as well as a USB port. Also, the input unit 720 may include or combine various input devices such as a touch screen and a button to receive an external user command.

The memory 730 may store a program for the operation of the processor 740 and may temporarily or permanently store input/output data. The memory 730 may include a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg, SD or XD memory, etc.), and a RAM (RAM). , SRAM, ROM, EEPROM, PROM, magnetic memory, a magnetic disk, and an optical disk may include at least one type of storage medium.

In addition, the memory 730 may store various network functions and algorithms, and may store various data, programs (one or more instructions), applications, software, commands, codes, etc. for driving and controlling the device 700 . have.

The processor 740 may control the overall operation of the device 700 . The processor 740 may execute one or more programs stored in the memory 730 . The processor 740 may mean a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which methods according to the inventive concept are performed.

In one embodiment, the processor 740 obtains an image group including a plurality of two-dimensional images generated corresponding to the continuous volume of the chest, and inputs the image group to the learned first network function to each A first image information obtained by extracting a lung region from a two-dimensional image is generated, and the image group is input to a learned second network function to generate second image information obtained by detecting a predetermined lesion region from each of the two-dimensional images, , a 3D image including the lesion region may be generated based on the first image information and the second image information.

The emphysema site automatic extraction method using deep learning according to an embodiment may be implemented in the form of a program command that can be performed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

In addition, the automatic extraction method for emphysema site using deep learning according to the disclosed embodiments may be provided included in a computer program product (computer program product). Computer program products may be traded between sellers and buyers as commodities.

The computer program product may include a S/W program and a computer-readable storage medium in which the S/W program is stored. For example, computer program products may include products (eg, downloadable apps) in the form of S/W programs distributed electronically through manufacturers of electronic devices or electronic markets (eg, Google Play Store, App Store). have. For electronic distribution, at least a portion of the S/W program may be stored in a storage medium or may be temporarily generated. In this case, the storage medium may be a server of a manufacturer, a server of an electronic market, or a storage medium of a relay server temporarily storing a SW program.

The computer program product, in a system consisting of a server and a client device, may include a storage medium of the server or a storage medium of the client device. Alternatively, if there is a third device (eg, a smart phone) that is communicatively connected to the server or the client device, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include the S/W program itself transmitted from the server to the client device or the third device, or transmitted from the third device to the client device.

In this case, one of the server, the client device and the third device may execute the computer program product to perform the method according to the disclosed embodiments. Alternatively, two or more of a server, a client device, and a third device may execute a computer program product to distribute the method according to the disclosed embodiments.

For example, a server (eg, a cloud server or an artificial intelligence server) may execute a computer program product stored in the server to control a client device communicatively connected with the server to perform the method according to the disclosed embodiments.

Although the embodiments have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention as defined in the following claims are also included in the scope of the present invention. belongs to

Claims (1)

In the automatic extraction method of emphysema site using deep learning,
acquiring an image group including a plurality of images generated corresponding to a continuous volume of the chest;
generating first image information obtained by extracting a lung region from each image by inputting the image group into a learned first network function;
generating second image information obtained by detecting a predetermined lesion region from each image by inputting the image group into a learned second network function; and
and generating a second image including the lesion region based on the first image information and the second image information.

Images