KR20240063376A - Transformation model building apparatus and method and image transformation apparatus using the same - Google Patents

Abstract

A conversion model building device for constructing a conversion module for converting the type of an image according to the present invention includes: a memory storing a conversion model building program; and a processor that executes the transformation model building program, wherein the transformation model building program converts a first training image of a first type into a first transformation image of a second type through a first generation module, and The first conversion image and the second learning image of the second type are applied to the determination module to update the first generation module so that the first conversion image is determined to be real, and the first conversion image is generated through the second generation module. Converting the first type to a second converted image, comparing the second converted image and the first learning image to update the first generation module to learn the conversion model, wherein the conversion model includes the first training image. It includes a first generation module, the second generation module, and the first discrimination module, wherein the first generation module processes the image of the first type into the second type, and the second generation module includes the Processing a second type of image into the first type, the first determination module judges at least one input image as real or fake according to set conditions, and the first type is a positron emission fault This is an image type taken by PET (Positron Emission Tomography) or CT (Computerized Tomography), and the second type is an image type taken by Magnetic Resonance Imaging (MRI). .

Description

Transformation model building device and image conversion device using the same {TRANSFORMATION MODEL BUILDING APPARATUS AND METHOD AND IMAGE TRANSFORMATION APPARATUS USING THE SAME}

The present invention relates to a technology for converting the type of image, and to an image conversion device for converting a first type of image into a second type.

There is growing interest in nuclear medicine imaging devices that can capture functional images of the human body for early detection of dementia and cancer. However, there is a problem that both radiology and nuclear medicine tests must be performed to make an accurate diagnosis. This may cause discomfort and financial burden to the patient and may result in additional radiation exposure.

Recently, technologies such as creating high-dose medical images from low-dose medical images using deep learning technology have been developed to reduce the imaging time of medical imaging devices using radiation, but patients are still required to undergo radiological examinations and nuclear medicine examinations. It did not solve the problem of having to do it all.

Therefore, technology that can overcome these problems is required.

In order to solve the above-described problem, the present invention builds a conversion model learned to convert the first type of image into the second type by using the first type of first training data and the second type of second training data. The technical task is to provide a device that does this.

In addition, in order to solve the above-mentioned problems, the present invention aims to provide an image conversion device that converts a first image of a first type into a second image of a second type using a conversion model.

However, the technical challenges that this embodiment aims to achieve are not limited to the technical challenges described above, and other technical challenges may exist.

As a technical means for solving the above-described technical problem, a conversion model building device according to an embodiment of the present invention includes a memory storing a conversion model building program; and a processor that executes the transformation model building program, wherein the transformation model building program converts a first training image of a first type into a first transformation image of a second type through a first generation module, and The first conversion image and the second learning image of the second type are applied to the determination module to update the first generation module so that the first conversion image is determined to be real, and the first conversion image is generated through the second generation module. Converting the first type to a second converted image, comparing the second converted image and the first learning image to update the first generation module to learn the conversion model, wherein the conversion model includes the first training image. It includes a first generation module, the second generation module, and the first discrimination module, wherein the first generation module processes the image of the first type into the second type, and the second generation module includes the Processing a second type of image into the first type, the first determination module judges at least one input image as real or fake according to set conditions, and the first type is a positron emission fault This is an image type taken by PET (Positron Emission Tomography) or CT (Computerized Tomography), and the second type is an image type taken by Magnetic Resonance Imaging (MRI). .

Additionally, an image conversion device according to another embodiment of the present invention includes a memory storing an image conversion program; and a processor executing the image conversion program, wherein the image conversion program generates a second image of a second type by applying a first image of a first type to a conversion model, and the conversion model includes: Machine learning is performed through first and second generation modules and first and second determination modules to convert the first type of image into the second type, and the first generation module converts the first type of image into the second type. processing the image into a second type, and the second generation module processes the image of the second type into the first type, and the first and second determination modules are set for at least one input image. Real and fake are judged according to conditions, and the first type is an image type taken with Positron Emission Tomography (PET) or Computerized Tomography (CT), and the second type is is a type of image taken with magnetic resonance imaging (MRI).

According to the problem-solving means of the present invention described above, anatomical image information can be obtained by converting the PET or CT images of patients who require MRI imaging during nuclear medicine examination, such as dementia patients or cancer patients, into MRI images. It can reduce the financial burden on patients.

1 is a conceptual diagram of a transformation model building device according to an embodiment of the present invention.
Figures 2A to 2C are exemplary diagrams for explaining the operation of a conversion model building program.
Figure 3 is a conceptual diagram schematically showing an image conversion device according to an embodiment of the present invention.
FIG. 4 is a block diagram schematically showing the configuration of the image conversion device shown in FIG. 3.

Hereinafter, the present invention will be described in detail with reference to the attached drawings. However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein. In addition, the attached drawings are only intended to facilitate understanding of the embodiments disclosed in this specification, and the technical idea disclosed in this specification is not limited by the attached drawings. In order to clearly explain the present invention in the drawings, parts not related to the description are omitted, and the size, shape, and shape of each component shown in the drawings may be modified in various ways. Throughout the specification, identical/similar parts are given identical/similar reference numerals.

The suffixes "module" and "part" for components used in the following description are given or used interchangeably only for the ease of preparing the specification, and do not have distinct meanings or roles in themselves. Additionally, in describing the embodiments disclosed in this specification, if it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed in this specification, the detailed descriptions are omitted.

Throughout the specification, when a part is said to be “connected (connected, contacted, or combined)” with another part, this means not only when it is “directly connected (connected, contacted, or combined),” but also when it has other members in between. It also includes cases where they are “indirectly connected (connected, contacted, or combined).” Additionally, when a part is said to "include (equip or provide)" a certain component, this does not exclude other components, unless specifically stated to the contrary, but rather "includes (provides or provides)" other components. It means that you can.

Terms representing ordinal numbers, such as first, second, etc., used in this specification are used only for the purpose of distinguishing one component from another component and do not limit the order or relationship of the components. For example, a first component of the present invention may be named a second component, and similarly, the second component may also be named a first component.

Figure 1 is a block diagram schematically showing a transformation model building device according to an embodiment of the present invention.

Referring to FIG. 1, a transformation model building device 100 according to an embodiment of the present invention will be described. The conversion model building device 100 builds a conversion model that converts the first type of image into the second type. Here, the first type is an image type taken with Positron Emission Tomography (PET) or Computerized Tomography (CT), and the second type is a type of image taken with Magnetic Resonance Imaging (MRI). This is an image type taken with Imaging. To perform this, the conversion model building device 100 includes a memory 110 and a processor 120.

The memory 110 stores a conversion model building program. The memory 110 refers to a non-volatile storage device that continues to maintain stored information even when power is not supplied and a volatile storage device that requires power to maintain the stored information. It should be interpreted as doing so. The memory 110 may perform a function of temporarily or permanently storing data processed by the processor 120. The memory 110 may include magnetic storage media or flash storage media in addition to volatile storage devices that require power to maintain stored information, but the scope of the present invention is not limited thereto. no.

Then, the processor 120 builds a conversion model by executing the conversion model building program stored in the memory 110. In this embodiment, the processor 120 is a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), or an FPGA. (field programmable gate array), etc., but the scope of the present invention is not limited thereto.

Figure 2 is an example diagram for explaining the operation of a conversion model building program. Referring to FIG. 2, the process by which the conversion module building program builds the conversion module will be described in detail.

First, referring to FIG. 2A, the transformation model is machine learned through deep learning using a first type of first learning image (1) and a second type of second learning image (2). It includes a first generation module 10, a first discrimination module 20, a second generation module 30, and a second discrimination module 40. The first generation module 10 processes the first type of image into the second type, the second generation module 30 processes the second type of image into the first type, and the first discrimination module 20 ) and the second determination module 40 determines at least one input image as real or fake according to set conditions.

Here, the first learning image (1) and the second learning image (2) are images of the same object, and the first learning image (1) is an image of the object using Positron Emission Tomography (PET). , It is an image taken by computerized tomography (CT), and the second learning image (2) is an image of the object taken by magnetic resonance imaging (MRI).

Construction of the transformation model can be divided into a process of updating the first generation module 10 and a process of updating the second generation module 30.

Next, the process of updating the first generation module 10 and the second generation module 30 will be described with reference to FIGS. 2B and 2C.

First, when explaining the process of updating the first generation module 10 with reference to FIG. 2b, the transformation model building program applies the first learning image 1 of the first type to the first generation module 10 Convert to a first converted image (3) of the second type, and apply the first converted image (3) and the second learning image (2) of the second type to the first discrimination module 20 to create a first converted image ( 3) Updates the first generation module 10 so that it is determined to be genuine. Here, the first determination module 20 is set to determine the second learning image 2 as real and the first converted image 3 as fake.

Then, the conversion model building program applies the first conversion image 3 to the second generation module 20 to convert it into a second conversion image 4 of the first type, and the second conversion image 4 and the first The first generation module 10 is updated by comparing the learning image 1. Here, the conversion model building program updates the first generation module 10 by comparing the difference value between the first learning image 1 and the second conversion image 4. The difference value is a value calculated by calculating the degree of loss between the actual image and the predicted image.

Additionally, the conversion model building program additionally updates the first generation module 10 by comparing the difference value between the second learning image 2 and the first conversion image 3.

The conversion model building program is a process of comparing the first learning image (1) and the second conversion image (4) until the first conversion image (3) is recognized as real by the first determination module (20) and the second learning image The first generation module 10 is updated by repeating the process of comparing (2) with the first converted image (3).

When explaining the process of updating the second generation module 20 with reference to FIG. 2C, the transformation model building program applies the second learning image 2 of the second type to the second generation module 30 to create the first Convert it into a tangible third transformed image (5), and apply the third transformed image (5) and the first learning image (1) to the second determination module 40 so that the third transformed image (5) is determined to be real. The second generation module 30 is updated. Here, the second determination module 40 is set to determine the first learning image 1 as real and the third learning image 5 as fake.

Then, the conversion model building program applies the third conversion image 5 to the first generation module 10 to convert it into a fourth conversion image 6 of the second type, and the fourth conversion image 6 and the second The second generation module 30 is updated by comparing the learning image 2. Here, the conversion model building program updates the second generation module 30 by comparing the difference value between the second learning image 2 and the fourth conversion image 6.

Additionally, the conversion model building program additionally updates the second generation module 30 by comparing the difference value between the first learning image 1 and the third conversion image 5.

The conversion model building program is a process of comparing the second learning image (2) and the fourth conversion image (6) until the third conversion image (5) is recognized as real by the second determination module (40) and the first learning image The second generation module 10 is updated by repeating the process of comparing (1) with the third converted image (5).

In this way, the conversion model is built through the process of updating the first generation module 10 and the process of updating the second generation module 30 described above. The transformation model building program updates the first generation module 10. The process and the process of updating the second generation module 30 can be performed simultaneously or sequentially to build a conversion model.

In addition, the constructed conversion model may convert the first type of image into a second type through the first generation module 10 and the second generation module 30, and convert the second type of image into a first type of image. It can also be converted to . That is, an image taken with PET or CT can be converted to an image taken with MRI, and an image taken with MRI can be converted into an image taken with PET or CT.

The communication module 130 may include a device that includes hardware and software necessary to perform data communication with an external device and transmit and receive signals such as control signals or data signals through wired or wireless connections with other network devices.

The database 140 may store various data for learning or operating a transformation model. For example, training images for learning a transformation model may be stored.

Meanwhile, the conversion model building device 100 according to an embodiment of the present invention is a server that receives the first learning image (1) and the second learning image (2) from an external computing device and builds a conversion model based on them. It can also operate in the form of .

FIG. 4 is a conceptual diagram schematically showing an image conversion device according to an embodiment of the present invention, and FIG. 5 is a block diagram schematically showing the configuration of the image conversion device shown in FIG. 4.

An image conversion device 200 according to an embodiment of the present invention will be described with reference to FIGS. 4 and 5 . The image conversion device 200 converts a first type of image into a second type of image. To perform this, the image conversion device 200 includes a memory 210 and a processor 220. Here, the first type is an image type taken with Positron Emission Tomography (PET) or Computerized Tomography (CT), and the second type is a type of image taken with Magnetic Resonance Imaging (MRI). This is an image type taken with Imaging.

The memory 210 stores an image conversion program. Memory 210 should be interpreted as a general term for non-volatile storage devices that continue to retain stored information even when power is not supplied and volatile storage devices that require power to maintain stored information. The memory 210 may perform the function of temporarily or permanently storing data processed by the processor 220. The memory 210 may include magnetic storage media or flash storage media in addition to volatile storage devices that require power to maintain stored information, but the scope of the present invention is not limited thereto. no.

Then, the processor 220 executes the image conversion program stored in the memory 110 to convert the first type of image into a second type of image. When explaining the operation of the image matching program with reference to FIG. 5, the image conversion program converts a first image 7 of a first type into a second image 8 of a second type by applying a conversion model. Here, the first type is an image type taken with Positron Emission Tomography (PET) or Computerized Tomography (CT), and the second type is a type of image taken with Magnetic Resonance Imaging (MRI). This is an image type taken with Imaging.

Here, the conversion model is machine-learned through the first generation module, first determination module, second generation module, and second determination module to convert the first type of image into the second type. The first generation module processes the first type of image into the second type, and the second generation module processes the second type of image into the first type. And, the first determination module and the second determination module determine whether at least one input image is real or fake according to set conditions. Here, the conversion model may convert the first type to the second type or the second type to the first type by the first generation module and the second generation module.

In this embodiment, the processor 220 is a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), or an FPGA. (field programmable gate array), etc., but the scope of the present invention is not limited thereto.

The communication module 230 may include a device including hardware and software required to transmit and receive signals such as control signals or data signals through a wired or wireless connection with other network devices in order to perform data communication for signal data with an external device. You can.

The database 240 may store various data for operating an image conversion program.

Meanwhile, the image conversion device 200 is a server that receives image data from an external device and inputs it into a conversion model to convert the first image 7 of the first type into the second image 8 of the second type. It can also operate in any form.

The present invention may also be implemented in the form of a recording medium containing instructions executable by a computer, such as program modules executed by a computer. Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and non-volatile media, removable and non-removable media. Additionally, computer-readable media may include computer storage media. Computer storage media includes both volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data.

Additionally, although the methods and systems of the present invention have been described with respect to specific embodiments, some or all of their components or operations may be implemented using a computer system having a general-purpose hardware architecture.

Those of ordinary skill in the technical field to which the present invention pertains will be able to understand that the present invention can be easily modified into other specific forms without changing the technical idea or essential features of the present invention based on the above description. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive. The scope of the present invention is indicated by the patent claims described below, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention.

The scope of the present application is indicated by the claims described below rather than the detailed description above, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present application.

100: Transformation model building device
110: memory
120: processor
200: Image conversion device
210: memory
220: processor

Claims (15)

In a conversion model building device that builds a conversion module that converts the type of image,
A memory where a conversion model building program is stored; and
Comprising a processor that executes the conversion model building program,
The conversion model building program is,
Convert the first learning image of the first type into a first converted image of the second type through the first generation module, and apply the first converted image and the second learning image of the second type to the first determination module. Update the first generation module so that the first transformation image is determined to be genuine, convert the first transformation image into a second transformation image of the first type through a second generation module, and convert the second transformation image and the first transformation image into a second transformation image of the first type. 1 By comparing learning images, the first generation module is updated to learn the transformation model,
The transformation model includes the first generation module, the second generation module, and the first discrimination module,
The first generation module is for processing the first type of image into the second type, and the second generation module is for processing the second type of image into the first type,
The first determination module determines at least one input image as real or fake according to set conditions,
The first type is an image type taken with Positron Emission Tomography (PET) or Computerized Tomography (CT), and the second type is a type of image taken with Magnetic Resonance Imaging (MRI). A conversion model building device, which is an image type taken with Imaging. According to paragraph 1,
The conversion model building program is,
Compare the difference value between the first learning image and the second converted image to update the first generation module,
The difference value is a value that numerically represents the degree of loss of the converted image with respect to the learning image. According to paragraph 1,
The conversion model building program is,
Compare the difference value between the second learning image and the first converted image to update the first generation module,
The difference value is a value that numerically represents the degree of loss of the converted image with respect to the learning image. According to paragraph 1,
The transformation model further includes a second discrimination module,
The conversion model building program is,
Convert the second learning image into a third transformed image of a first type through the second generation module, and apply the first learning image and the third transformed image to the second determination module to create the third transformed image. A transformation model building device that updates the second generation module so that is determined to be real. According to clause 3,
The conversion model building program is,
The third conversion image is converted into a fourth conversion image of the second type through the first generation module, and the difference value between the second learning image and the fourth conversion image is compared to generate the second generation module. update,
The difference value is a value that numerically represents the degree of loss of the converted image with respect to the learning image. According to paragraph 1,
The first determination module is,
It is set to determine that the first converted image is fake and the second learning image is real,
The conversion model building program is,
A transformation model building device that updates the generation module by repeatedly applying the first transformation image to the first generation module until the first transformation image is determined to be genuine in the first determination module. According to clause 3,
The second determination module is,
It is set to determine that the third converted image is fake and the first learning image is real,
The conversion model building program is,
A transformation model building device that updates the second generation module by repeatedly applying the third transformation image to the second generation module until the third transformation image is determined to be genuine in the second determination module. According to paragraph 1,
The first generation module and the second generation module are transformation model building devices configured with a Resnet (Residual neural network)-based architecture. In the image conversion device for converting a first type of image into a second type,
Memory where an image conversion program is stored; and
Including a processor that executes the image conversion program,
The image conversion program is,
Applying a first image of a first type to a transformation model to generate a second image of a second type,
The conversion model is,
Machine learning is performed through first and second generation modules and first and second determination modules to convert the first type of image into the second type,
The first generation module is for processing the first type of image into the second type, and the second generation module is for processing the second type of image into the first type,
The first and second determination modules determine whether at least one input image is real or fake according to conditions set,
The first type is a type of image taken with Positron Emission Tomography (PET) or Computerized Tomography (CT), and the second type is a type of image taken with Magnetic Resonance Imaging (MRI). ), an image conversion device, which is the type of image taken. According to clause 9,
The conversion model is,
Convert the first learning image of the first type into a first converted image of the second type through the first generation module, and apply the first converted image and the second learning image of the second type to the first determination module. Update the first generation module so that the first converted image is determined to be real,
Converting the first converted image into a second converted image of the first type through the second generation module, and comparing the second converted image with the first learning image to update the first generation module,
An image conversion device that is learned by updating the first generation module by comparing the second learning image and the first conversion image. According to clause 10,
The conversion model is,
Convert the second learning image into a third transformed image of a first type through the second generation module, and apply the first learning image and the third transformed image to the second determination module to create the third transformed image. An image conversion device that is learned by updating the second generation module so that it is determined to be real. According to clause 11,
The conversion model is,
Learning is performed by converting the third converted image into a fourth converted image of the second type through the first generation module, and updating the second generating module by comparing the second learning image and the fourth transformed image. In,image conversion device. According to clause 11,
The conversion model is,
The first determination module is set to determine the first converted image as fake and the second learning image as real,
The image conversion device is learned by updating the first generation module by repeatedly applying the first transformation image to the first generation module until the first transformation image is determined to be genuine in the first determination module. According to clause 11,
The conversion model is,
The second determination module is set to determine the third converted image as fake and the first learning image as real,
The image conversion device is learned by updating the second generation module by repeatedly applying the third transformation image to the second generation module until the third transformation image is determined to be genuine in the second determination module. According to clause 11,
The conversion model is,
It is learned by updating the first generation module or the second generation module by comparing the difference value between the learning image and the converted image,
The difference value is a numerical value representing the degree of loss of the converted image with respect to the learning image.

Images