KR20240109873A - 3 dimensional heatmap generating method using 2 dimensional medical image and image processing apparatus - Google Patents

Abstract

A method of generating a 3D heat map using 2D medical images includes the steps of an image processing device receiving 2D medical images of a subject in different directions, and the image processing device performing deep learning on each of the 2D medical images. Generating two-dimensional heat maps by inputting them into a model, generating a three-dimensional heat map by the image processing device combining the two-dimensional heat maps for different directions, and generating the three-dimensional heat map by the image processing device. It includes removing areas with intensity values below a certain threshold.

Description

Method and image processing device for generating 3D heatmap using 2D medical image {3 DIMENSIONAL HEATMAP GENERATING METHOD USING 2 DIMENSIONAL MEDICAL IMAGE AND IMAGE PROCESSING APPARATUS}

The technology described below relates to a technique for reconstructing 2D heatmaps generated by an artificial intelligence model that interprets 2D medical images into 3D heatmaps.

Medical imaging is a representative tool used to diagnose a patient's disease. For example, chest CT (Computer Tomography) can be used to diagnose a subject's lung disease.

Recently, various artificial intelligence models that interpret medical images are being studied. Artificial intelligence models can interpret medical images and calculate information about the subject's phenotype. Additionally, the artificial intelligence model can provide visualization of areas related to the disease. Typically, CAM (Class Activation Map) is a technology that visualizes areas important for inference in an input frame in the form of a heatmap.

US Published Patent US2022-0157048

Heatmaps for 2D medical images lose location information for missing dimensions. Meanwhile, although 3D artificial intelligence models can provide heatmaps for 3D medical images, it is difficult to visualize finer areas than heatmaps for 2D images.

The technology described below seeks to provide a technique for reconstructing a 3D heatmap using 2D heatmaps.

A method of generating a 3D heat map using 2D medical images includes the steps of an image processing device receiving 2D medical images of a subject in different directions, and the image processing device performing deep learning on each of the 2D medical images. Generating two-dimensional heat maps by inputting them into a model, generating a three-dimensional heat map by the image processing device combining the two-dimensional heat maps for different directions, and generating the three-dimensional heat map by the image processing device. It includes removing areas with intensity values below a certain threshold.

The image processing device that generates a 3D heat map is an interface device that receives 2D medical images of a subject in different directions, and a storage device that stores a deep learning model that receives 2D medical images and generates a 2D heatmap. and inputting the input 2-dimensional medical images into a deep learning model to generate 2-dimensional heat maps, combining the 2-dimensional heat maps for different directions to generate a 3-dimensional heat map, and generating the 3-dimensional heat map. It includes an arithmetic device that removes areas with intensity values below a certain threshold.

The technology described below uses a 2D heatmap for medical images to visualize fine areas while providing a 3D heatmap without loss of location information. The technology described below can be used for diagnosis or treatment by accurately visualizing the location and information of abnormal areas in three dimensions.

Figure 1 is an example of a system that provides a 3D heatmap for medical images.
Figure 2 is an example of a process for generating a 3D heatmap for a medical image.
Figure 3 is an example of a process for generating a 3D heatmap using 2D heatmaps of medical images.
Figure 4 is an example of an image processing device that generates a 3D heat map for a medical image.

The technology described below may be subject to various changes and may have various embodiments, and specific embodiments will be illustrated in the drawings and described in detail. However, this is not intended to limit the technology described below to specific embodiments, and should be understood to include all changes, equivalents, and substitutes included in the spirit and scope of the technology described below.

Terms such as first, second, A, B, etc. may be used to describe various components, but the components are not limited by the terms, and are only used for the purpose of distinguishing one component from other components. It is used only as For example, a first component may be named a second component without departing from the scope of the technology described below, and similarly, the second component may also be named a first component. The term and/or includes any of a plurality of related stated items or a combination of a plurality of related stated items.

In terms used in this specification, singular expressions should be understood to include plural expressions, unless clearly interpreted differently from the context, and terms such as “including” refer to the described features, numbers, steps, operations, and components. , it means the existence of parts or a combination thereof, but should be understood as not excluding the possibility of the presence or addition of one or more other features, numbers, step operation components, parts, or combinations thereof.

Before providing a detailed description of the drawings, it would be clarified that the division of components in this specification is merely a division according to the main function each component is responsible for. That is, two or more components, which will be described below, may be combined into one component, or one component may be divided into two or more components for more detailed functions. In addition to the main functions it is responsible for, each of the components described below may additionally perform some or all of the functions handled by other components, and some of the main functions handled by each component may be performed by other components. Of course, it can also be carried out exclusively by .

In addition, when performing a method or operation method, each process that makes up the method may occur in a different order from the specified order unless a specific order is clearly stated in the context. That is, each process may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the opposite order.

The technology described below is a technique that provides heat maps for medical images. Heatmaps can express areas important for diagnosis and areas where lesions are located using distinct colors. Medical images can be diverse, including X-ray, CT, and MRI (Magnetic Resonance Imaging).

Hereinafter, it will be explained that the image processing device uses a learning model to reconstruct 2D heat maps of medical images into 3D heat maps. An image processing device is a computer device capable of processing images. An image processing device can be implemented as a PC, a server on a network, a smart device, or a chipset with an embedded dedicated program.

The learning model receives two-dimensional images as input and creates a two-dimensional heat map. The learning model may be a CNN (Convolutional Neural Network)-based model. For convenience of explanation, it is assumed that the learning model below is a model that generates CAM. For example, a deep learning model may have a layer that performs global average pooling in the last layer, rather than using a typical fully connected layer (FC). Hereinafter, the model that generates a heatmap for the input image is called a visualization model.

Figure 1 is an example of a system that provides a 3D heatmap for medical images. FIG. 1 shows an example in which the image processing device is a computer terminal 130 and a server 140.

The medical imaging equipment 110 generates medical images for the subject (patient). Medical imaging equipment 110 may be CT equipment, MRI equipment, etc. The medical image may be an image composed of multiple slices. The medical imaging equipment 110 may store the generated medical images in an Electronic Medical Record (EMR) 120 or a separate database (DB).

The computer terminal 130 may receive medical images from the medical imaging equipment 110 or the EMR 120 through a wired or wireless network. The computer terminal 130 generates two-dimensional heat maps for each two-dimensional images (slices) using a CNN-based visualization model. The computer terminal 130 reconstructs the 2D heat maps into 3D heat maps. User A can check the 3D heat map on the computer terminal 130.

The server 140 may receive medical images from the medical imaging equipment 110 or the EMR 120. The server 140 generates two-dimensional heat maps for each two-dimensional images (slices) using a CNN-based visualization model. The server 140 reconstructs the 2D heatmaps into 3D heatmaps. The server 140 may transmit the 3D heatmap to user A's terminal.

The computer terminal 130 and/or the server 140 may store the 3D heat map in the EMR 120.

Figure 2 is an example of a process 200 for generating a 3D heatmap for a medical image. The image processing device receives the subject's medical image (210). Figure 2 is an example of a chest medical image. The image processing device can receive 3D CT data. In this case, the image processing device can extract a 2D medical image by projecting the 3D CT in a certain direction (220). For example, an image processing device can project a 3D CT into the axial, sagittal, and coronal planes to extract two-dimensional medical images for each direction. In some cases, the image processing device may receive two-dimensional medical images from different directions.

The image processing device can input each of the two-dimensional medical images into a visualization model and generate a two-dimensional heat map for each image (230). The visualization model may be a CNN-based model (2D AI) that processes two-dimensional medical images.

The image processing device can generate a 3D heatmap by reconstructing multiple 2D heatmaps (240).

The process of generating a 3D heatmap using 2D heatmaps is explained in more detail. Figure 3 is an example of a process 300 of generating a 3D heatmap using 2D heatmaps of medical images. Figure 3 is an example of a chest medical image. Figure 3 is an example of the process of generating heatmaps for the left lung and right lung, respectively.

The image processing device generates a two-dimensional heat map for each of the left lung and right lung (310). The image processing device can acquire two-dimensional images in three different directions by projecting a three-dimensional CT image in three different directions (eg, axial plane, sagittal plane, and coronal plane). The image processing device can acquire two-dimensional images in three directions for each of the left lung and right lung. The image processing device can generate a two-dimensional heat map by inputting each two-dimensional image into a visualization model.

An image processing device generates a heat map for each continuous two-dimensional image (slice) in one direction. An image processing device can generate a type of 3D heatmap by stacking continuous 2D heatmaps in one direction. A 3D heatmap generated using a one-directional 2D heatmap is called an initial 3D heatmap. However, because the initial 3D heatmap is created using a 2D image, information in other directions (dimensions) may be missing.

The image processing device can generate an initial 3D heatmap by aligning consecutive 2D heatmaps to a certain 3D area (320). At this time, a certain three-dimensional area may be defined as a template or mask. The image processing device sorts the initial 3D heatmaps for three directions in a certain 3D area and can determine whether information at a specific location is missing from one initial 3D heatmap. If information is missing in a specific dimension, the image processing device can preserve or restore the missing information by replicating a heat map in another dimension (direction).

The image processing device can combine the initial 3D heat maps for the three directions to ultimately generate one 3D heat map (330).

At this time, as the final 3D heatmap combines multiple heatmaps, the intensity value of the highly influential location for each 3D location increases, and the loss of 3D location information is reduced. However, the final 3D heatmap is a sum of all replicated 3D data, but there is a possibility that unnecessary areas contain high values. In this case, the image processing device can perform post-processing on the final 3D heatmap. The image processing device can remove points or areas with a value (color value) below a certain threshold from the entire 3D heatmap (340). For example, the image processing device may determine an intensity value corresponding to 50% of the entire 3D heatmap and remove areas below the intensity value.

Figure 4 is an example of an image processing device 400 that generates a 3D heat map for a medical image. The image processing device 400 corresponds to the image processing device described above (130 and 140 in FIG. 1). The image processing device 400 may be physically implemented in various forms. For example, the image processing device 400 may take the form of a computer device such as a PC, a network server, or a chipset dedicated to data processing.

The image processing device 400 may include a storage device 410, a memory 420, an arithmetic device 430, an interface device 440, a communication device 450, and an output device 460.

The storage device 410 can store medical images of a subject. At this time, the medical image may be a 3D medical image or 2D slices.

The storage device 410 may store a visualization model that generates a 2D heat map from a 2D medical image. The visualization model is a model learned in advance according to the disease that is the target of analysis.

The storage device 410 can store a 3D heat map.

The memory 420 may store data and information generated while the image processing device processes medical images.

The interface device 440 is a device that receives certain commands and data from the outside. The interface device 440 may receive a medical image of a subject from a physically connected input device or an external storage device. The interface device 440 may transmit a 3D heatmap generated by processing a medical image to an external object.

Meanwhile, the interface device 440 may receive data or information transmitted through the communication device 450 below.

The communication device 450 refers to a configuration that receives and transmits certain information through a wired or wireless network. The communication device 450 may receive a medical image of a subject from an external object. The communication device 450 may transmit a 3D heat map generated by processing a medical image to an external object such as a user terminal.

The output device 460 is a device that outputs certain information. The output device 460 can output an interface required for the data processing process, an input medical image, a 3D heat map, etc.

The computing device 430 may generate two-dimensional images by projecting the input three-dimensional medical image of the subject in different directions. The computing device 430 can generate consecutive two-dimensional slices in three different directions. The three directions may include a first direction, a second direction, and a third direction.

The computing unit 430 generates a 2D heat map by inputting 2D images in different directions into a visualization model. The computing device 430 generates a 2D heat map for each of the consecutive 2D images in one direction. The computing device 430 may generate first 2D heatmaps for the first direction, second 2D heatmaps for the second direction, and third 2D heatmaps for the third direction.

The computing device 430 arranges a two-dimensional heat map for one direction based on a certain three-dimensional area. At this time, the computing device 430 can check the location or area where information is missing in one direction (dimension). The computing device 430 replicates (restores) the location of the missing information using a two-dimensional heat map in a different direction (dimension). A 3D heatmap created as a 2D heatmap for one direction was called an initial 3D heatmap.

For example, the computing device 430 copies the information of the specific location in the second two-dimensional heat maps or the third two-dimensional heat map for a specific location where information is missing among the first two-dimensional heat maps and generates the first A 2D heatmap can be restored.

The computing device 430 may generate a final 3D heatmap by combining the initial 3D heatmaps for the three directions.

As described above, the final 3D heatmap may include intensity values in specific areas according to duplication of missing information or combining multiple 3D heatmaps. The computing device 430 may post-process the 3D heatmap by removing areas below the intensity value based on a certain intensity value. At this time, the constant intensity value can be an experimentally determined value.

The computing device 430 may be a device such as a processor that processes data and performs certain operations, an AP, or a chip with an embedded program.

Additionally, the medical image processing method or the 3D heat map generation method described above may be implemented as a program (or application) including an executable algorithm that can be executed on a computer. The program may be stored and provided in a temporary or non-transitory computer readable medium.

A non-transitory readable medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as registers, caches, and memories. Specifically, the various applications or programs described above include CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM (read-only memory), PROM (programmable read only memory), and EPROM (Erasable PROM, EPROM). Alternatively, it may be stored and provided in a non-transitory readable medium such as EEPROM (Electrically EPROM) or flash memory.

Temporarily readable media include Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), and Enhanced SDRAM (Enhanced RAM). It refers to various types of RAM, such as SDRAM (ESDRAM), synchronous DRAM (Synclink DRAM, SLDRAM), and Direct Rambus RAM (DRRAM).

This embodiment and the drawings attached to this specification only clearly show some of the technical ideas included in the above-described technology, and those skilled in the art can easily understand them within the scope of the technical ideas included in the specification and drawings of the above-described technology. It is self-evident that all inferable modifications and specific embodiments are included in the scope of rights of the above-described technology.

Claims (4)

A step in which an image processing device receives two-dimensional medical images of a subject in different directions;
generating, by the image processing device, two-dimensional heat maps by inputting each of the two-dimensional medical images into a deep learning model;
generating, by the image processing device, a 3D heatmap by combining the 2D heatmaps for different directions; and
A method of generating a 3D heatmap using a 2D medical image, including the step of removing, by the image processing device, an area with an intensity value below a certain threshold from the 3D heatmap. According to paragraph 1,
The different directions include a first direction, a second direction, and a third direction,
In the step of generating the 2D heatmaps, the image processing apparatus may generate first 2D heatmaps for the first direction, second 2D heatmaps for the second direction, and third heatmaps for the third direction. 3 Create two-dimensional heat maps,
The image processing device replicates the information of the specific location in the second 2D heat maps or the third 2D heat map for a specific location where information is missing among the first 2D heat maps, and copies the information of the specific location in the first 2D heatmap. A method of generating a 3D heatmap using a 2D medical image further comprising the step of restoring the 2D heatmap. An interface device that receives two-dimensional medical images of a subject in different directions;
A storage device that stores a deep learning model that receives two-dimensional medical images and generates a two-dimensional heat map; and
The input 2D medical images are each input into a deep learning model to generate 2D heat maps, and the 2D heat maps for different directions are combined to generate a 3D heat map. In the 3D heat map, An image processing device that generates a 3D heat map including a processing unit that removes areas with intensity values below a certain threshold. According to paragraph 3,
The different directions include a first direction, a second direction, and a third direction,
The processing unit generates first two-dimensional heat maps for the first direction, second two-dimensional heat maps for the second direction, and third two-dimensional heat maps for the third direction,
For a specific location where information is missing among the first two-dimensional heat maps, the information of the specific location is copied from the second two-dimensional heat maps or the third two-dimensional heat map to create the first two-dimensional heat map. An image processing device that generates a 3D heat map to be restored.

Images