WO2023204342A1

WO2023204342A1 - Deep learning-based image conversion method and system therefor

Info

Publication number: WO2023204342A1
Application number: PCT/KR2022/006685
Authority: WO
Inventors: 유승훈; 나창주; 이태석; 박정훈
Original assignee: 탈로스 메디컬 테크널러지 컴퍼니 리미티드
Priority date: 2022-04-22
Filing date: 2022-05-10
Publication date: 2023-10-26
Also published as: KR20230151146A

Abstract

Provided are a deep learning-based image conversion method and a system therefor. An image conversion method, according to some embodiments of the present disclosure, may comprise: acquiring an unpaired image set including a low-quality image and a high-quality image; generating a paired image set from the acquired unpaired image set; and constructing a deep learning model that performs quality conversion (enhancement) on an input image by using the generated paired image set. Accordingly, a high-performance deep learning model that accurately performs quality conversion on an image may be easily constructed.

Description

Deep learning-based image conversion method and system

The present disclosure relates to a deep learning-based image conversion method and system. More specifically, a method of converting an image of a source domain into an image of a target domain using a deep learning model. and a system for performing the method.

Deep-learning technology is attracting attention by showing results that exceed the performance of existing technologies in analyzing various types of data such as images, voices, and texts. In addition, deep learning technology is being introduced and utilized in various fields due to the scalability and flexibility inherent in the technology itself, and is also being used for purposes such as data transformation (or creation) beyond data analysis.

Among various fields, the medical field is the field that is most actively introducing deep learning technology to improve existing diagnostic processes. For example, in the medical field, deep learning models that detect lesions or diagnose diseases in medical images of patients are being introduced and utilized in the actual diagnosis process.

Recently, attempts have been made to convert medical images into a more useful form using a GAN (Generative Adversarial Networks) model that performs domain translation or domain transform. However, in the medical field, it is very difficult to secure paired image sets for multiple domains, so the above attempts have not yet shown effective results.

The technical problem to be solved through some embodiments of the present disclosure is a method for accurately converting an image of a source domain into an image of a target domain using a deep learning model, and how to perform the method. providing a system.

Another technical problem to be solved through some embodiments of the present disclosure is to provide a method for accurately converting a low-quality image (e.g. MR image) into a high-quality image using a deep learning model and a system for performing the method. It is done.

Another technical problem to be solved through some embodiments of the present disclosure is a method for accurately converting an MR (Magnetic Resonance) image into a CT (Computed Tomography) image using a deep learning model, and a system for performing the method. is to provide.

Another technical problem to be solved through some embodiments of the present disclosure is to provide a method for building a high-performance deep learning model that accurately performs domain conversion and a system for performing the method.

Another technical problem to be solved through some embodiments of the present disclosure is to provide a method for generating a paired image set from an unpaired image set for multiple domains and a system for performing the method. It is provided.

Another technical problem to be solved through some embodiments of the present disclosure is to provide a method for accurately evaluating the quality of a synthetic image generated through a deep learning model and a system for performing the method.

The technical problems of the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by those skilled in the art from the description below.

In order to solve the above-described technical problem, an image conversion method according to some embodiments of the present disclosure is a method performed by at least one computing device, and includes an unpaired image conversion method including an image of a first domain and an image of a second domain. Obtaining an (unpaired) image set - the image of the second domain is a higher quality image than the image of the first domain -, generating a paired image set from the obtained unpaired image set, and It may include building a deep learning model that performs quality conversion on the input image using the generated paired image set.

In one embodiment, the unpaired image set is a magnetic resonance (MR) image set, and the image of the second domain may be an MR image with a higher field than the image of the first domain.

In one embodiment, generating the paired image set may include generating the paired image set by matching the image of the first domain and the image of the second domain.

In one embodiment, the step of generating the paired image set includes synthesizing the image of the first domain into the second domain using a GAN (Generative Adversarial Networks) model with domain translation capability. It may include converting the image into a synthetic image and configuring the image of the first domain and the composite image of the second domain into a pair.

In one embodiment, the step of generating the paired image set includes matching an image of the first domain and an image of the second domain to generate a matched image, at least a portion of the unpaired image set and the It may include learning the generated matched image to build a GAN (Generative Adversarial Networks) model with domain transformation capability, and generating at least a portion of the paired image set using the built GAN model. there is.

In one embodiment, the generated registered image is an image belonging to the second domain, the GAN model is constructed by learning the image of the first domain and the generated registered image, and at least one of the paired image set The part generating step includes converting the image of the first domain into a synthetic image of the second domain through the constructed GAN model and synthesizing the image of the first domain and the second domain. It may include configuring images into pairs.

In one embodiment, the step of converting the first image into a second image of higher quality than the first image using the constructed deep learning model may be further included.

In one embodiment, further comprising evaluating the quality of the second image using a designated evaluation metric, wherein the designated evaluation metric includes a correlation coefficient between the first image and the second image, and the first image. It may include a difference in signal-to-noise ratio of the second image and a difference in pixel intensity between the first image and the second image.

In one embodiment, the first image is a MR (Magnetic Resonance) image of the patient being diagnosed, and the second image is converted to a CT image using another deep learning model learned to convert the MR image into a CT (Computer Tomography) image. A conversion step may be further included.

In one embodiment, the deep learning model includes an encoder and a decoder, the encoder includes a plurality of encoding blocks for extracting feature data of different abstraction levels from an input image, and the decoder includes the plurality of encoding blocks. and a plurality of decoding blocks corresponding to the plurality of encoding blocks, and a skip-connection that transfers the extracted feature data may be formed between the plurality of encoding blocks and the plurality of decoding blocks.

In order to solve the above-mentioned technical problem, the image conversion method according to some other embodiments of the present disclosure is a method performed by at least one computing device, and is a method of acquiring a first MR (Magnetic Resonance) image for the subject being diagnosed. Step, converting the first MR image into a second MR image through a first deep learning model - the second MR image is a higher quality image than the first MR image - and converting the first MR image into a second MR image through a second deep learning model. It may include converting the second MR image into a CT (Computed Tomography) image.

In order to solve the above-described technical problem, an image conversion method according to another embodiment of the present disclosure is a method performed by at least one computing device, including an image of a first domain and an image of a second domain. Obtaining an unpaired image set, generating a paired image set from the obtained unpaired image set, and performing domain transformation on the input image using the generated paired image set. It may include the step of building a deep learning model.

An image conversion system according to some embodiments of the present disclosure for solving the above-described technical problems, and an image conversion system according to some other embodiments of the present disclosure for solving the above-mentioned technical problems. Comprising one or more processors and a memory that stores one or more instructions, wherein the one or more processors execute the one or more stored instructions, thereby generating an unpaired image including an image of a first domain and an image of a second domain. An operation of acquiring a set - the image of the second domain is a higher quality image than the image of the first domain -, an operation of generating a paired image set from the obtained unpaired image set, and the generated pair Using the image set, you can build a deep learning model that performs quality conversion on the input image.

A computer program according to some embodiments of the present disclosure for solving the above-described technical problem includes the steps of acquiring an unpaired image set including an image of a first domain and an image of a second domain - the second domain The image of the domain is a higher quality image than the image of the first domain. - Generating a paired image set from the obtained unpaired image set and using the generated paired image set to input the image. It can be stored in a computer-readable recording medium to execute the step of building a deep learning model that performs quality conversion.

According to some embodiments of the present disclosure, a paired image set is generated from an unpaired image set for multiple domains, and a deep learning model that performs domain transformation using the generated paired image set. This can be built. Accordingly, a deep learning model that accurately performs domain conversion can be constructed, and the image of the first domain (i.e., source domain) can be accurately converted to an image of the second domain (i.e., target domain) through the constructed deep learning model. can be converted. For example, a low-quality image (e.g. MR image) can be accurately converted into a high-quality image, and a first type of image (e.g. MR image) can be accurately converted into a second type of image (e.g. CT image).

In addition, the MR (Magnetic Resonance) image of the patient being diagnosed can be converted into a higher quality MR image through a deep learning model. In this case, high-quality MR images can be easily secured even in hospitals (or environments) equipped only with outdated MRI equipment (e.g. low-field MRI imaging equipment), and the diagnostic process and overall medical environment can be greatly improved.

Additionally, the MR image of the patient being diagnosed can be converted into a CT (Computer Tomography) image through a deep learning model. In this case, the patient being diagnosed can be free from the risk of radiation exposure because there is no need for a CT scan. In addition, the overall diagnostic process can be simplified because the patient does not need to undergo multiple imaging tests to receive a comprehensive diagnosis of various diseases. Furthermore, the time and financial costs required for imaging tests can be reduced.

In addition, the MR image of the patient being diagnosed can be converted into a higher quality MR image through a deep learning model, and the MR image converted through another deep learning model can be converted into a CT image. In this case, because CT images of fairly high quality can be generated, the patient being diagnosed can be completely free from the risk of radiation exposure. In addition, high-quality MR and CT images can be easily secured even in hospitals equipped only with outdated MRI equipment (e.g. low-field MRI imaging equipment), which can greatly improve the diagnostic process and overall medical environment.

Additionally, the quality of synthetic images (i.e., images converted through deep learning models) generated based on correlation coefficients, signal-to-noise ratio differences, and/or pixel intensity (or signal intensity) differences can be quantitatively determined. can be accurately evaluated.

Additionally, a paired image set can be created from an unpaired image set using an image matching technique. Accordingly, paired image sets for multiple domains can be easily secured, and the time and human costs required to secure a paired image set can be greatly reduced. In addition, by using the generated paired image set, a high-performance deep learning model that accurately performs domain transformation can be easily built.

Additionally, a paired image set can be created from an unpaired image set using a GAN (Generative Adversarial Networks) model with domain transformation capability. Accordingly, paired image sets for multiple domains can be easily secured, and the time and human costs required to secure a paired image set can be greatly reduced. In addition, by using the generated paired image set, a high-performance deep learning model that accurately performs domain transformation can be easily built.

Additionally, a high-quality paired image set can be generated from an unpaired image set using a GAN model with image matching techniques and domain transformation capabilities. Accordingly, high-quality paired image sets for multiple domains can be secured at a low cost, and as a result, high-performance deep learning models that accurately perform domain transformation can be more easily built.

The effects according to the technical idea of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below.

1 to 3 are exemplary diagrams for explaining an image conversion system according to some embodiments of the present disclosure.

4 illustrates an example diagnostic process that has changed with the introduction of an image conversion system according to some embodiments of the present disclosure.

FIG. 5 is an exemplary diagram schematically illustrating a method of generating a paired image set of an image conversion system according to some embodiments of the present disclosure.

6 is an example flowchart illustrating an image conversion method according to some embodiments of the present disclosure.

FIG. 7 is an exemplary diagram illustrating the structure and operation of a deep learning model with image quality conversion capability according to an embodiment of the present disclosure.

FIG. 8 is an exemplary diagram for explaining the structure and operation of a deep learning model with image type conversion capability according to an embodiment of the present disclosure.

Figure 9 is an exemplary flowchart showing a method of generating a paired image set according to an embodiment of the present disclosure.

10 and 11 are exemplary diagrams to further explain a method of generating a paired image set according to an embodiment of the present disclosure.

FIG. 12 is an exemplary flowchart showing a method of generating a paired image set according to another embodiment of the present disclosure.

13 and 14 are exemplary diagrams for explaining the structure and learning method of a GAN (Generative Adversarial Networks) model with domain transformation capability according to some embodiments of the present disclosure.

15 and 16 are exemplary diagrams to further explain a method of generating a paired image set according to another embodiment of the present disclosure.

Figure 17 is an example flowchart showing a method of generating a paired image set according to another embodiment of the present disclosure.

18 and 19 are exemplary diagrams to further explain a method of generating a paired image set according to another embodiment of the present disclosure.

20 illustrates an example computing device that can implement an image conversion system according to some embodiments of the present disclosure.

Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the attached drawings. The advantages and features of the present disclosure and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the technical idea of the present disclosure is not limited to the following embodiments and may be implemented in various different forms. The following examples are merely intended to complete the technical idea of the present disclosure and to be used in the technical field to which the present disclosure belongs. It is provided to fully inform those skilled in the art of the scope of the present disclosure, and the technical idea of the present disclosure is only defined by the scope of the claims.

When adding reference numerals to components in each drawing, it should be noted that identical components are given the same reference numerals as much as possible even if they are shown in different drawings. Additionally, in describing the present disclosure, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present disclosure, the detailed description will be omitted.

Unless otherwise defined, all terms (including technical and scientific terms) used in this disclosure may be used with meanings that can be commonly understood by those skilled in the art to which this disclosure pertains. Additionally, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless clearly specifically defined. The terminology used in this disclosure is for describing embodiments and is not intended to limit the disclosure. In this disclosure, the singular includes the plural, unless specifically stated otherwise in the context.

Additionally, in describing the components of the present disclosure, terms such as first, second, A, B, (a), and (b) may be used. These terms are only used to distinguish the component from other components, and the nature, order, or order of the component is not limited by the term. When a component is described as being “connected,” “coupled,” or “connected” to another component, that component may be directly connected or connected to that other component, but there is another component between each component. It will be understood that elements may be “connected,” “combined,” or “connected.”

As used in this disclosure, “comprises” and/or “comprising” refers to a referenced component, step, operation and/or element that includes one or more other components, steps, operations and/or elements. Does not exclude presence or addition.

Hereinafter, various embodiments of the present disclosure will be described in detail according to the attached drawings.

FIG. 1 is an exemplary diagram for explaining an image conversion system 10 according to some embodiments of the present disclosure.

As shown in Figure 1, the image conversion system 10 uses a deep learning model 13 to convert the image 11 of the first domain (i.e., source domain) into an image of the second domain (i.e., target domain). It may be a system that can be converted to (12). For example, the image conversion system 10 may convert (improve) the quality of the image 11 or convert the type of the image 11 using the deep learning model 13. Hereinafter, for convenience of explanation, the image conversion system 10 will be abbreviated as 'conversion system 10'.

The

images

11 and 12 of the two domains may be, for example, images from the medical field, but the scope of the present disclosure is not limited thereto. For example, the

images

11 and 12 of the two domains may be images from a non-medical field. Additionally, the first domain image 11 and the second domain image 12 may be images from different fields.

Images in the medical field can encompass all various types of images handled in the medical field, such as MR (Magnetic Resonance) images, CT (Computed Tomography) images, etc. Additionally, MR images and CT images may be, for example, two-dimensional slice (cross-section) images, three-dimensional images, etc., but the scope of the present disclosure is not limited thereto.

More specifically, the transformation system 10 builds at least one deep learning model 13 with domain translation (domain translation or domain transform) capability, and uses the built deep learning model 13 to transform the system in various ways. Domain conversion can be performed.

For example, as shown in FIG. 2, the conversion system 10 can convert a low-quality image 21 into a high-quality image 22 through the deep learning model 23 (i.e., in this example , the first domain consists of low-quality images, and the second domain consists of high-quality images). In this example, the deep learning model 23 may be a model equipped with quality conversion (improvement) capabilities by learning an image set (i.e., training image set) composed of low-quality training images and high-quality training images. Here, a high-quality training image refers to an image with relatively higher quality than a low-quality training image, such as pixel intensity, resolution, signal-to-noise ratio, and sharpness (e.g. pixel value). contrast) may be an excellent image, but the scope of the present disclosure is not limited thereto.

As a more specific example, the conversion system 10 can convert a low-quality MR image (e.g. 21) into a high-quality MR image (e.g. 22) through a deep learning model (e.g. 23). This deep learning model (e.g. 23) can be built by learning an image set composed of low-quality training MR images and high-quality training MR images. Here, high-quality training MR images may be, for example, MR images with relatively excellent signal-to-noise ratio and/or signal intensity, high-field MR images, etc. However, the scope of the present disclosure is not limited thereto. For example, a deep learning model (e.g. 23) that learned an image set consisting of a low-field training MR image and a high-field training MR image converts the input low-field MR image (e.g. 21) into a high-field MR image (e.g. To be precise, it can be converted into an image with characteristics/quality similar to that of a high-magnetic field MR image. According to this example, high-quality MR images can be easily secured even in hospitals (or environments) equipped only with outdated MRI equipment (e.g. low-field MRI imaging equipment), greatly improving the diagnostic process and overall medical environment. there is. In some cases, the conversion system 10 may convert a low-quality CT image (e.g. 21) into a high-quality CT image (e.g. 22) through a deep learning model (e.g. 23).

As another example, as shown in FIG. 3, the conversion system 10 may convert the first type of image 31 into the second type of image 32 through the deep learning model 33 (i.e. , in this example, the first domain consists of images of the first type, and the second domain consists of images of the second type). This deep learning model 33 can be constructed by learning an image set consisting of a first type of training image and a second type of training image.

As a more specific example, as shown in FIG. 4, the conversion system 10 may convert the MR image 41 into a CT image 42 through the deep learning model 43. This deep learning model 43 can be built by learning an image set composed of training MR images and training CT images. According to this example, the patient being diagnosed can be free from the risk of radiation exposure because there is no need for a CT scan. In addition, the overall diagnostic process can be simplified because the patient does not need to undergo multiple imaging tests to receive a comprehensive diagnosis of various diseases. Furthermore, the time and financial costs required for imaging tests can be reduced. In some cases, the conversion system 10 may convert a CT image (e.g. 31) into an MR image (e.g. 32) through a deep learning model (e.g. 33).

As another example, the conversion system 10 may convert an image with a high difficulty in abnormality detection into an image with a low difficulty in abnormality detection through a deep learning model. This deep learning model can be built by learning an image set composed of training images with high difficulty in detecting anomalies and training images with low difficulty in detecting anomalies. Additionally, the difficulty of anomaly detection may be determined based on, for example, the confidence score of a deep learning model that performs anomaly detection.

As a more specific example, the conversion system 10 converts a medical image with a high difficulty in lesion detection (e.g. an image for which a deep learning model performing lesion detection outputs an ambiguous confidence score) into a medical image with a low difficulty in lesion detection through a deep learning model. It can be converted to . This deep learning model can be built by learning an image set consisting of training images with high lesion detection difficulty and training images with low lesion detection difficulty. According to this example, by lowering the difficulty of detecting lesions in medical images, the accuracy of deep learning models that perform lesion detection can be improved, and the psychological costs, time costs, and financial costs of the diagnosed person caused by false detections can also be reduced. It can be.

In another example, conversion system 10 may perform image conversion based on a combination of the various examples described above. For example, the conversion system 10 converts the first image into a higher-quality second image using a first deep learning model (e.g. 23), and converts the second image to a higher quality second image using a second deep learning model (e.g. 33). It can be converted to other types of images. For a more specific example, the conversion system 10 converts the first MR image into a higher quality second MR image through a first deep learning model (e.g. 23), and converts the first MR image into a higher quality second MR image through a second deep learning model (e.g. 33). The second MR image can be converted to a CT image. In this case, because CT images of fairly high quality can be generated, the patient being diagnosed can be completely free from the risk of radiation exposure. In addition, high-quality MR and CT images can be easily secured even in hospitals equipped only with outdated MRI equipment (e.g. low-field MRI imaging equipment), which can greatly improve the diagnostic process and overall medical environment.

Description will be made again with reference to FIG. 1 .

As described above, the conversion system 10 performs image conversion using a deep learning model 13 with domain conversion capability, so it may be important to build a high-performance deep learning model 13. In addition, in order to build a high-performance deep learning model 13, paired image sets for multiple domains (e.g. first domain and second domain) are typically required.

However, in most fields, it is very difficult to secure paired image sets for multiple domains. For example, let's assume that we are building a deep learning model that converts low-quality MR images into high-quality MR images in the medical field. In this case, securing a paired image set for MR images of different quality (e.g. low-field MR images and high-field MR images) may be nearly impossible or consume significant time and cost. This is because it is very rare for a single patient to undergo imaging using different MRI equipment at the same hospital.

According to some embodiments of the present disclosure to solve this problem, the conversion system 10 generates a paired image set from an unpaired image set for multiple domains and uses the generated paired image set. Thus, a high-performance deep learning model (13) can be built. For example, as shown in FIG. 5, the transformation system 10 pairs the unpaired image set 51 using a GAN (Generative Adversarial Networks) model 53 with an image matching technique and/or domain transformation capability. It can be converted to image set (52). By doing so, a high-performance deep learning model 13 can be built at a relatively low cost, which will be described in detail later with reference to FIGS. 9 to 19.

Detailed operations of the conversion system 10 will be described in more detail later with reference to the drawings of FIG. 6 and below.

Meanwhile, FIG. 1 shows the transformation system 10 as if performing a one-way transformation from one domain (ie, the first domain) to another domain (ie, the second domain). However, transformation system 10 may also operate to perform bidirectional transformation between multiple domains.

Transformation system 10 may be implemented with at least one computing device. For example, all of the functionality of conversion system 10 may be implemented in a single computing device, with a first function of conversion system 10 being implemented in a first computing device and a second functionality in a second computing device. It may be implemented. Alternatively, certain functions of conversion system 10 may be implemented on multiple computing devices.

Computing devices may encompass various types of devices equipped with computing functions, and for an example of such devices, refer to FIG. 20.

So far, the conversion system 10 according to some embodiments of the present disclosure has been described with reference to FIGS. 1 to 5. Below, various methods that can be performed in the conversion system 10 will be described.

Hereinafter, in order to provide convenience of understanding, the description will be continued assuming that all steps/operations of the methods to be described later are performed in the conversion system 10 described above. Accordingly, if the subject of a specific step/action is omitted, it may be understood as being performed in the conversion system 10. However, in a real environment, some steps of the methods to be described later may be performed on other computing devices.

First, an image conversion method according to some embodiments of the present disclosure will be described with reference to FIGS. 6 to 8.

6 is an example flowchart schematically illustrating an image conversion method according to some embodiments of the present disclosure. However, this is only a preferred embodiment for achieving the purpose of the present disclosure, and of course, some steps may be added or deleted as needed.

As shown in FIG. 6, the image conversion method according to embodiments may begin at step S61 of acquiring an unpaired image set. As described above, an unpaired image set may mean an image set composed of a low-quality image and a high-quality image, but the two images are not paired with each other. For example, low-quality images and high-quality images may be obtained from different patients (or hospitals, etc.).

In step S62, a paired image set may be created from the obtained unpaired image set. However, the specific method of generating a paired image set may vary depending on the embodiment.

In one embodiment, a paired image set may be created using an image matching technique. This embodiment will be described in detail later with reference to FIGS. 9 to 11.

In another embodiment, a paired image set can be created using a GAN model with domain transformation capabilities. This embodiment will be described in detail later with reference to FIGS. 12 to 16.

In another embodiment, a paired image set can be created using an image matching technique and a GAN model together. In this case, a better quality paired image set can be generated, and this embodiment will be described in detail later with reference to FIGS. 17 to 19.

Meanwhile, according to an embodiment of the present disclosure, the similarity between two images may be evaluated during the process of generating a paired image set. Additionally, based on the evaluation result that the similarity is greater than or equal to the standard value, the two images may be configured as a pair. For example, the conversion system 10 may calculate the correlation coefficient of the two images and configure the two images as a pair in response to a determination that the correlation coefficient is greater than a reference value. By doing so, a higher quality paired image set can be generated. Anyone working in the relevant technical field will already be familiar with the formula (or method) for calculating the correlation coefficient between two images, so description of the correlation coefficient itself will be omitted.

In step S63, a deep learning model that performs quality conversion (improvement) on the input image can be built using the generated paired image set. For example, the conversion system 10 can train a deep learning model using a pair of low-quality images and high-quality images, thereby enabling the deep learning model to have the ability to convert (improve) the quality of the input image. However, the detailed structure of the deep learning model can be designed and implemented in various ways.

In one embodiment, as shown in FIG. 7, a deep learning model may be composed of an encoder that encodes the input image 71 and a decoder that decodes the encoded data and outputs a higher quality image 76. Figure 7 illustrates the case where the input image 71 is an MR image. More specifically, the encoder of the deep learning model may be composed of a plurality of encoding blocks that extract feature data (e.g. 72, 73; i.e., feature map) of different abstraction levels from the input image 71. Each encoding block may be composed of, for example, a convolutional layer, a down-sampling layer (e.g. pooling layer), etc., but the scope of the present disclosure is not limited thereto. The decoder of a deep learning model may be composed of a plurality of decoding blocks, and each decoding block may be divided into, for example, a convolution layer, a de-convolutional layer, an upsampling layer, etc. However, the scope of the present disclosure is not limited thereto. In addition, a plurality of decoding blocks may correspond to a plurality of encoding blocks, and there is a skip-connection (e.g. 74, 75; skip-connection) that transfers feature data (e.g. 72, 73) between the decoding block and the encoding block. can be formed. Since each decoding block performs decoding by collecting (e.g. concatenation) the feature data (e.g. 72, 73) received through skip-connection (e.g. 74, 75) from the feature data decoded at the lower level, the quality conversion function is performed. It can be performed more accurately. Figure 7 illustrates that the deep learning model is composed of a U-shaped structure, but the deep learning model may be composed of a W-shaped structure (i.e., a structure in which a U-shaped network is repeated). For reference, the deep learning model illustrated in FIG. 7 is a model specialized for performing pixel-wise prediction (pixel-wise prediction or dense prediction) tasks, and quality conversion (improvement) for the input image 71 also ultimately involves the performance of each pixel. Because it is a type of task that predicts values, quality conversion can be performed with fairly high accuracy using the deep learning model illustrated in FIG. 7.

In another embodiment, the deep learning model may be implemented as a GAN model with domain transformation capabilities. Examples of such GAN models may include UNIT (Unsupervised Image-to-Image Translation), MUNIT (Multimodal Unsupervised Image-to-Image Translation), Star-GAN, etc., but the scope of the present disclosure is not limited thereto.

This will be described again with reference to FIG. 6 .

In step S64, quality conversion on the image may be performed using the constructed deep learning model. For example, the conversion system 10 can convert a medical image of a diagnosed patient into a higher quality medical image through the built deep learning model. For example, the conversion system 10 can convert an MR image of a patient being diagnosed into a higher quality MR image or convert a CT image into a higher quality CT image.

Meanwhile, although not shown in FIG. 6, according to an embodiment of the present disclosure, in order to verify the quality of an image (i.e., synthetic image) converted (generated) through a deep learning model, the quality of the converted image is evaluated. Additional steps may be performed. For example, the conversion system 10 may convert a first image into a higher quality second image through a deep learning model and evaluate the quality of the second image using a designated evaluation metric. At this time, the designated evaluation metric may be, for example, a correlation coefficient between the first image and the second image, a difference in signal-to-noise ratio, a difference in pixel intensity (or signal intensity) (e.g. difference in average pixel intensity value), etc. The scope of the present disclosure is not limited thereto. As a more specific example, the conversion system 10 determines that the larger the value of the correlation coefficient and the larger the difference in signal-to-noise ratio (i.e., the more the signal-to-noise ratio value of the second image increases compared to the first image), the difference in the average pixel intensity value. The larger the (i.e., the more the average pixel intensity value of the second image increases compared to the first image), the more the quality of the second image can be evaluated to be improved.

In addition, although not shown in FIG. 6, according to an embodiment of the present disclosure, a step of converting the type of the converted image through another deep learning model learned to convert the type of the image may be further performed. For example, the conversion system 10 converts the first MR image of the diagnosed patient into a higher-quality second MR image through the constructed deep learning model, and converts the second MR image into a CT image through another deep learning model. You can. By doing so, the diagnosed patient can be free from the risk of radiation exposure, and high-quality MR images and CT images can be easily secured even in hospitals equipped only with outdated MRI equipment (e.g. low-field MRI imaging equipment), thereby facilitating diagnosis. Processes and the overall healthcare environment can be significantly improved. In some cases, the conversion system 10 may evaluate the quality of the second MR image and convert the second MR image into a CT image in response to a determination that the evaluated quality satisfies a predetermined condition. At this time, the quality of the MR image can be evaluated based on evaluation metrics such as correlation coefficient, signal-to-noise ratio difference, signal intensity difference, etc., as described above.

In the previous embodiment, a deep learning model that converts the type of image (e.g. a deep learning model that converts an MR image to a CT image) has a similar structure to the deep learning model that performs image quality conversion and can be built in a similar manner. there is. For example, as shown in FIG. 8, the deep learning model that converts the MR image 81 into the CT image 82 may be implemented as a neural network with a U-shaped similar structure (e.g. U-shaped, W-shaped), and FIG. 6 It may have been constructed according to the method illustrated in . In some cases, the deep learning model may be implemented as a GAN model with domain transformation capabilities. For a description of the deep learning model, please refer to the description in FIG. 7.

So far, an image conversion method according to some embodiments of the present disclosure has been described with reference to FIGS. 6 to 8 . According to the above, a paired image set can be created from an unpaired image set for multiple domains, and a deep learning model that performs image quality conversion using the generated paired image set can be built. Accordingly, a deep learning model that accurately performs image quality conversion can be built, and low-quality images (e.g. MR images) can be accurately converted into high-quality images.

Meanwhile, the above-described image conversion method can be applied to other examples of domain conversion without changing the actual technical idea. For example, the conversion system 10 acquires an unpaired image set consisting of an image of a first domain (e.g. an image of a first type) and an image of a second domain (e.g. an image of a second type), and the obtained unpaired image A paired image set is created from the set, and various deep learning models that perform domain transformation on the input image can be built using the created paired image set. For example, the conversion system 10 is a deep learning model that generates a paired image set from an unpaired image set composed of MR images and CT images, and converts the input MR image into a CT image using the generated paired image set. You can also build .

Hereinafter, a method for generating a paired image set according to some embodiments of the present disclosure will be described with reference to FIGS. 9 to 19. In the following description, the images constituting the given unpaired image set will be referred to as 'original images' to distinguish them from other images (e.g. registered images and composite images, which will be described later). Additionally, to provide convenience of understanding, the explanation will be continued assuming that the unpaired image set consists of low-quality images and high-quality images. However, the scope of the present disclosure is not limited thereto, and even when the unpaired image set consists of a first type of image (e.g. MR image) and a second type of image (e.g. CT image), the following description does not apply to practical technical purposes. It can be applied without changing the ideology.

First, a method for generating a paired image set according to an embodiment of the present disclosure will be described with reference to FIGS. 9 to 11.

Figure 9 is an exemplary flowchart showing a method of generating a paired image set according to an embodiment of the present disclosure. However, this is only a preferred embodiment for achieving the purpose of the present disclosure, and of course, some steps may be added or deleted as needed.

As shown in FIG. 9, this embodiment may begin at step S91, where a low-quality original image and a high-quality original image are matched to generate a registered image. For example, the conversion system 10 may generate a low-quality registered image by matching (or transforming) a low-quality original image to a high-quality image. Alternatively, the conversion system 10 may generate a high-quality registered image by matching (or transforming) a high-quality original image to a low-quality image. At this time, registration of the two images may be performed through, for example, a deformable registration technique, but the scope of the present disclosure is not limited thereto, and other registration techniques may be applied.

In step S92, the original image and the registered image may be configured as a pair. For example, the conversion system 10 may configure a low-quality original image and a high-quality registered image as a pair. Alternatively, the conversion system 10 may configure a high-quality original image and a low-quality matched image as a pair.

By repeatedly performing steps S91 and S92 described above on a plurality of images constituting the unpaired image set, the unpaired image set can be converted to a paired image set.

In order to provide easier understanding, further explanation will be made with reference to FIGS. 10 and 11.

Figures 10 and 11 illustrate that a paired image set (112 in Figure 11) is generated from an unpaired image set (111 in Figure 11) through a transformation matching technique. In addition, the low-quality image 101 constituting the unpaired image set (111 in FIG. 11) is a low-field MR image (e.g. 0.06T), and the high-quality image 102 is a high-field MR image (e.g. 1.5T). It illustrates this. Additionally, in the drawings below, 'deformed 1.5T' refers to a registered MR image generated from the original 1.5T MR image (see 'Original 1.5T').

As shown in FIG. 10, a high-quality registered image 103 can be generated from a high-quality image 102 through a transformation registration technique. That is, by transforming the high-quality image 102 to fit the low-quality image 101, a high-quality registered image 103 can be generated. However, in some cases, a low-quality registered image (not shown) may be generated by transforming the low-quality image 101 to fit the high-quality image 102.

Next, the generated registered image 103 and the original image 101 of different quality can be composed of a pair, and as this process is repeated, as shown in FIG. 11, from the unpaired image set 111 A paired image set 112 may be created.

So far, a method for generating a paired image set according to an embodiment of the present disclosure has been described with reference to FIGS. 9 to 11 . According to the above, a paired image set (e.g. 112) can be created from an unpaired image set (e.g. 111) using an image matching technique. Accordingly, a paired image set (e.g. 112) for multiple domains can be easily secured, and the time and human costs required to secure a paired image set (e.g. 112) can be greatly reduced. In addition, by using the generated paired image set (e.g. 112), a high-performance deep learning model that accurately performs domain transformation can be easily built.

Hereinafter, a method for generating a paired image set according to another embodiment of the present disclosure will be described with reference to FIGS. 12 to 16.

Figure 12 is an example flowchart showing a paired image set according to another embodiment of the present disclosure. However, this is only a preferred embodiment for achieving the purpose of the present disclosure, and of course, some steps may be added or deleted as needed.

As shown in FIG. 12, this embodiment may begin at step S121 of training a GAN model using an unpaired image set. At this time, the GAN model refers to a model that can have domain transformation capabilities by learning an unpaired image set. Examples of such models include Cycle-GAN, UNIT (Unsupervised Image-to-Image Translation), and MUNIT (Multimodal Unsupervised Image-Translation). to-Image Translation), etc. However, the scope of the present disclosure is not limited thereto.

At this stage, the conversion system 10 can learn the GAN model using the unpaired image set as is, and as a result, the GAN model has the ability to convert low-quality images into high-quality images. In order to provide easier understanding, the structure and learning method of the GAN model will be further explained with reference to FIGS. 13 and 14.

Figure 13 is an example diagram for explaining the structure and learning method of a GAN model according to an embodiment of the present disclosure. In particular, Figure 13 shows that the GAN model is a Cycle-GAN model, and the unpaired image set consists of low-quality MR images (e.g. low-field MR images of 0.06T) and high-quality MR images (e.g. high-field MR images of 1.5T). It illustrates this. Additionally, in the drawings below, 'Synthetic 0.06T' and 'Synthetic 1.5T' are synthesis generated from original (actual) MR images of 0.06T and 1.5T, respectively (see 'Original 0.06T' and 'Original 1.5T'). It means MR image.

As shown in Figure 13, the GAN model according to this embodiment will be configured to include a first generator 131, a second generator 132, a first discriminator 133, and a second discriminator 134. You can.

The first generator 131 may convert high quality images 135-1 and 136-2 into low quality images 135-2 and 136-3. Additionally, the first discriminator 133 can determine whether the input low-quality images 136-1 and 135-2 are real images (see real) or synthetic images (see fake). Adversarial learning may be performed between the first generator 131 and the first discriminator 133.

Additionally, the first generator 131 may be further trained using consistency loss. At this time, the consistency error may be calculated based on the difference between the actual image 135-1 input to the first generator 131 and the composite image 135-3 converted through the second generator 132. By learning the consistency error, the first generator 131 can accurately perform image quality conversion even if the image set is not composed of pairs.

Next, the second generator 132 may convert the low-quality images 136-1 and 135-2 into high-quality images 136-2 and 135-3. Additionally, the second discriminator 134 can determine whether the input high-quality images 135-1 and 136-2 are real images (see real) or synthetic images (see fake). Adversarial learning may be performed between the second generator 132 and the second discriminator 134.

Additionally, the second generator 132 may also be further learned using the consistency error. Since the learning method of the second generator 132 is similar to that of the first generator 131, further description will be omitted.

When learning of the GAN model is completed, a high-quality image (e.g. 135-1) can be converted into a low-quality synthetic image (e.g. 135-2) through the first generator 131, and through the second generator 132 A low-quality image (e.g. 136-1) can be converted into a high-quality composite image (e.g. 136-2).

Figure 14 is an example diagram for explaining the structure and learning method of a GAN model according to another embodiment of the present disclosure. In particular, Figure 14 illustrates that the GAN model is a UNIT model, and the unpaired image set is composed of low-quality MR images (e.g. low-field MR images of 0.06T) and high-quality MR images (e.g. high-field MR images of 1.5T). I'm doing it.

As shown in Figure 14, the GAN model according to this embodiment includes a first encoder 141, a first generator 143, and a first discriminator 145 associated with high quality, and a second encoder 142 associated with low quality. ), a second generator 144, and a second discriminator 146.

The first encoder 141 may encode the high-quality image 148-1 into encoding data 147, and the second encoder 142 may encode the low-quality image 148-1 into encoding data 147. You can. Encoding data 147 may be understood as data on a latent shared space shared between high-quality and low-quality images (i.e., between different domains).

The first generator 143 may generate a high-quality composite image 148-2 based on the encoding data 147, and the second generator 144 may generate a low-quality composite image 148-2 based on the encoding data 147. 149-2) can be generated.

The first discriminator 145 can perform a discrimination operation on high-quality images (e.g. 148-2), and the second discriminator 146 can perform a discrimination operation on low-quality images (e.g. 149-2). there is. Adversarial learning may be performed between the first discriminator 145 and the first generator 143, and adversarial learning may also be performed between the second discriminator 146 and the second generator 144.

Additionally, learning may be performed according to the flow shown in FIG. 14 to convert the quality of the image. For example, when encoding data 147 for a low-quality image 149-1 is input to the first generator 143, the low-quality image 149-1 is converted into a high-quality composite image 148-2. Learning can be carried out as much as possible. Similarly, when the encoding data 147 for the high-quality image 148-1 is input to the second generator 144, the high-quality image 148-1 is converted into a low-quality composite image 149-2. Learning can be performed. Regarding specific learning methods, please refer to the literature titled “UNIT; Unsupervised Image-to-Image Translation”. Actual image conversion can also be performed according to the flow shown in FIG. 14.

This will be described again with reference to FIG. 12.

In step S122, the original image of the unpaired image set can be converted into a synthetic image of different quality through the constructed GAN model. For example, referring again to FIG. 13 , the conversion system 10 may convert a high-quality original image (e.g. 135-1) into a low-quality composite image (e.g. 135-2) through the first generator 131. Alternatively, the conversion system 10 may convert a low-quality original image (e.g. 136-1) into a high-quality composite image (e.g. 136-2) through the second generator 132.

In step S123, a composite image of different quality from the original image may be configured as a pair. For example, the conversion system 10 can convert an unpaired image set into a paired image set by pairing a low-quality original image and a high-quality composite image and repeating this process. Alternatively, the conversion system 10 may convert an unpaired image set into a paired image set by pairing a high-quality original image and a low-quality composite image and repeating this process.

In order to provide easier understanding, further explanation will be made with reference to FIGS. 15 and 16.

Figures 15 and 16 illustrate that a paired image set (162, 163 in Figure 16) is generated from an unpaired image set (161 in Figure 16) through the GAN model 153. In addition, as in the previous example, the low-quality image 151 constituting the unpaired image set (161 in FIG. 16) is a low-field MR image (e.g. 0.06T), and the high-quality image 152 is a high-field MR image. (e.g. 1.5T).

As shown in FIG. 15, a low-quality image 151 can be converted into a high-quality MR image 154 through the learned GAN model 153. Alternatively, the high-quality image 152 may be converted into a low-quality composite image 155.

Next, the high-quality composite image 154 and the low-quality original image 151 can be composed of a pair, and as this process is repeated, as shown in FIG. 16, the unpaired image set 161 is created. 1 Paired image set 162 may be created. Alternatively, the low-quality composite image 155 and the high-quality original image 152 may be composed of a pair, and as this process is repeated, a second image set from the unpaired image set 161 is obtained, as shown in FIG. 16. A paired image set 163 may be created.

So far, a method for generating a paired image set according to another embodiment of the present disclosure has been described with reference to FIGS. 12 to 16. According to the above, a paired image set (e.g. 162, 163) can be created from an unpaired image set (e.g. 161) using a GAN model with domain transformation capability. Accordingly, paired image sets (e.g. 162, 163) for multiple domains can be easily secured, and the time and human costs required to secure paired image sets (e.g. 162, 163) can be greatly reduced. . In addition, by using the generated paired image sets (e.g. 162, 163), a high-performance deep learning model that accurately performs domain transformation can be easily built.

Hereinafter, a method of generating a paired image set according to another embodiment of the present disclosure will be described with reference to FIGS. 17 to 19. However, in order to exclude redundant descriptions, descriptions of content that overlaps with the previous embodiments will be omitted.

Figure 17 is an example flowchart showing a method of generating a paired image set according to another embodiment of the present disclosure. However, this is only a preferred embodiment for achieving the purpose of the present disclosure, and of course, some steps may be added or deleted as needed.

As shown in FIG. 17, this embodiment relates to a method of generating a paired image set using an image matching technique and a GAN model together.

Specifically, this embodiment may begin at step S171, where a low-quality original image and a high-quality original image are matched to generate a matched image. For a description of this step, refer to the description of step S91 described above.

In step S172, a GAN model with domain transformation capability can be built using a paired image set consisting of an original image and a matched image. For example, the conversion system 10 can build a GAN model using a paired image set composed of a low-quality original image and a high-quality registered image, and a paired image set composed of a high-quality original image and a low-quality registered image. You can also build a GAN model using it. By using paired image sets, a more high-performance GAN model can be built. At this stage, the GAN model refers to a model that can acquire domain transformation capabilities by learning unpaired image sets and/or paired image sets. Examples of such models include Cycle-GAN, UNIT, MUNIT, Star-GAN, etc. can be mentioned. However, the scope of the present disclosure is not limited thereto. For a description of this step, refer to the description of step S121 described above.

In step S173, the original image (or registered image) may be converted into a synthetic image of different quality through the constructed GAN model. For example, the conversion system 10 may convert a low-quality original image (or registered image) into a high-quality composite image. Alternatively, the conversion system 10 may convert a high-quality original image (or registered image) into a low-quality composite image.

In step S174, a composite image of different quality from the original image (or registered image) may be configured as a pair. For example, the conversion system 10 may configure a low-quality original image (or registered image) and a high-quality composite image as a pair (e.g. a low-quality original image and a high-quality composite image or a low-quality registered image and a high-quality composite image). there is. Alternatively, the conversion system 10 may configure a low-quality composite image and a high-quality original image (or registered image) as a pair (e.g. a low-quality composite image and a high-quality original image or a low-quality composite image and a high-quality registered image). there is. Additionally, the conversion system 10 may also configure a registered image of a different quality from the original image into a pair (e.g. a low-quality original image and a high-quality registered image, or a low-quality registered image and a high-quality original image) (steps S171 and S172). reference).

In order to provide easier understanding, further explanation will be made with reference to FIGS. 18 and 19.

Figures 18 and 19 show the second and third paired image sets (Figure 19) from the first paired image set (191 in Figure 19) consisting of a low-quality original image and a high-quality matched image through the GAN model 183. 192 and 193) are examples of being created. In addition, as in the previous example, the low-quality image 181 constituting the paired image set (191 in FIG. 19) is a low-field MR image (e.g. 0.06T), and the high-quality image 184 is a high-magnetic field matched MR image. This is an example of an image (e.g. 1.5T).

As shown in FIG. 18, a low-quality original image 181 can be converted into a high-quality synthetic image 182 through the learned GAN model 183. Alternatively, the high-quality registered image 184 may be converted into a low-quality composite image 185.

Next, the low-quality original image 181 and the high-quality composite image 182 may be composed of a pair, and as this process is repeated, as shown in FIG. 19, the first paired image set 191 The second paired image set 192 can be generated from. Alternatively, the low-quality composite image 185 and the high-quality matched image 184 may be composed of a pair, and as this process is repeated, as shown in FIG. 19, from the first paired image set 191 A third paired image set 193 may be created.

So far, a method for generating a paired image set according to another embodiment of the present disclosure has been described with reference to FIGS. 17 to 19. According to the above, a high-quality paired image set (e.g. 191 to 193) can be generated from an unpaired image set using a GAN model equipped with an image matching technique and domain transformation capability. Accordingly, high-quality paired image sets (e.g. 191 to 193) for multiple domains can be secured at a low cost, and as a result, a high-performance deep learning model that accurately performs domain conversion can be more easily built.

Hereinafter, an exemplary computing device 200 capable of implementing the conversion system 10 according to some embodiments of the present disclosure will be described with reference to FIG. 20.

FIG. 20 is an exemplary hardware configuration diagram showing the computing device 200.

As shown in FIG. 20, the computing device 200 includes one or more processors 201, a bus 203, a communication interface 204, and a memory (loading) a computer program executed by the processor 201. 202) and a storage 205 that stores a computer program 206. However, only components related to the embodiment of the present disclosure are shown in FIG. 20. Accordingly, a person skilled in the art to which this disclosure pertains can recognize that other general-purpose components may be included in addition to the components shown in FIG. 20 . That is, the computing device 200 may further include various components in addition to those shown in FIG. 20 . In some cases, the computing device 200 may be implemented with some of the components shown in FIG. 20 omitted.

The processor 201 may control the overall operation of each component of the computing device 200. The processor 201 includes at least one of a Central Processing Unit (CPU), Micro Processor Unit (MPU), Micro Controller Unit (MCU), Graphic Processing Unit (GPU), or any type of processor well known in the art of the present disclosure. It can be configured to include. Additionally, the processor 201 may perform operations on at least one application or program to execute methods/operations according to various embodiments of the present disclosure. Computing device 200 may include one or more processors.

Next, memory 202 may store various data, instructions and/or information. Memory 202 may load one or more programs 206 from storage 205 to execute methods/operations according to various embodiments of the present disclosure. The memory 202 may be implemented as a volatile memory such as RAM, but the technical scope of the present disclosure is not limited thereto.

Next, the bus 203 may provide communication functionality between components of the computing device 200. The bus 203 may be implemented as various types of buses, such as an address bus, a data bus, and a control bus.

Next, the communication interface 204 may support wired and wireless Internet communication of the computing device 200. Additionally, the communication interface 204 may support various communication methods other than Internet communication. To this end, the communication interface 204 may be configured to include a communication module well known in the technical field of the present disclosure.

Next, storage 205 may non-transitory store one or more computer programs 206. The storage 205 may be a non-volatile memory such as Read Only Memory (ROM), Erasable Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), flash memory, a hard disk, a removable disk, or a device well known in the art to which this disclosure pertains. It may be configured to include any known type of computer-readable recording medium.

The computer program 206 may include one or more instructions that, when loaded into the memory 202, cause the processor 201 to perform methods/operations according to various embodiments of the present disclosure. That is, the processor 201 may perform methods/operations according to various embodiments of the present disclosure by executing one or more instructions. Here, instructions are a series of computer-readable instructions grouped based on function and are a component of a computer program and are executed by a processor.

For example, the computer program 206 may perform an operation of acquiring an unpaired image set including an image of a first domain (e.g. a low-quality image) and an image of a second domain (e.g. a high-quality image), the obtained unpaired image It may include instructions for creating a paired image set from a set and constructing a deep learning model that performs domain conversion (e.g. quality conversion) on the input image using the generated paired image set. there is. In this case, the conversion system 10 according to some embodiments of the present disclosure may be implemented through the computing device 200.

So far, an exemplary computing device 200 capable of implementing the conversion system 10 according to some embodiments of the present disclosure has been described with reference to FIG. 20.

So far, various embodiments of the present disclosure and effects according to the embodiments have been mentioned with reference to FIGS. 1 to 20. The effects according to the technical idea of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below.

The technical idea of the present disclosure described so far with reference to FIGS. 1 to 20 may be implemented as computer-readable code on a computer-readable medium. The computer-readable recording medium may be, for example, a removable recording medium (CD, DVD, Blu-ray disk, USB storage device, removable hard disk) or a fixed recording medium (ROM, RAM, computer-equipped hard disk). You can. The computer program recorded on the computer-readable recording medium can be transmitted to another computing device through a network such as the Internet, installed on the other computing device, and thus used on the other computing device.

In the above, even though all the components constituting the embodiments of the present disclosure have been described as being combined or operated in combination, the technical idea of the present disclosure is not necessarily limited to these embodiments. That is, within the scope of the purpose of the present disclosure, all of the components may be operated by selectively combining one or more of them.

Although operations are shown in the drawings in a specific order, it should not be understood that the operations must be performed in the specific order shown or sequential order or that all illustrated operations must be performed to obtain the desired results. In certain situations, multitasking and parallel processing may be advantageous. Moreover, the separation of the various components in the embodiments described above should not be construed as necessarily requiring such separation, and the program components and systems described may generally be integrated together into a single software product or packaged into multiple software products. You must understand that it exists.

Although embodiments of the present disclosure have been described above with reference to the attached drawings, those skilled in the art will understand that the present disclosure can be implemented in other specific forms without changing the technical idea or essential features. I can understand that there is. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive. The scope of protection of this disclosure should be interpreted in accordance with the claims below, and all technical ideas within the equivalent scope should be interpreted as being included in the scope of rights of the technical ideas defined by this disclosure.

Claims

A method performed by at least one computing device, comprising:

Obtaining an unpaired image set including an image of a first domain and an image of a second domain, wherein the image of the second domain is a higher quality image than the image of the first domain;

generating a paired image set from the obtained unpaired image set; and

Comprising the step of building a deep learning model that performs quality conversion on the input image using the generated paired image set,

How to convert images.
According to paragraph 1,

The unpaired image set is an MR (Magnetic Resonance) image set,

The image of the second domain is a higher-field MR image than the image of the first domain,

How to convert images.
According to paragraph 1,

The step of generating the paired image set is,

Comprising the step of generating the paired image set by matching the image of the first domain and the image of the second domain,

How to convert images.
According to paragraph 3,

Registration between the image of the first domain and the image of the second domain is performed through a deformable registration technique,

How to convert images.
According to paragraph 1,

The step of generating the paired image set is,

Converting the image of the first domain into a synthetic image of the second domain using a Generative Adversarial Networks (GAN) model with domain translation capabilities; and

Comprising the step of configuring the image of the first domain and the composite image of the second domain as a pair,

How to convert images.
According to clause 5,

The GAN model is a Cycle-GAN model,

How to convert images.
According to paragraph 1,

The step of generating the paired image set is,

Converting the image of the second domain into a synthetic image of the first domain using a Generative Adversarial Networks (GAN) model with domain translation capabilities; and

Comprising the step of configuring the image of the second domain and the composite image of the first domain as a pair,

How to convert images.
According to paragraph 1,

The step of generating the paired image set is,

generating a matched image by matching the image of the first domain and the image of the second domain;

Constructing a Generative Adversarial Networks (GAN) model with domain transformation capability by learning at least a portion of the unpaired image set and the generated matched image; and

Including generating at least a portion of the paired image set using the constructed GAN model,

How to convert images.
According to clause 8,

The generated matched image is an image belonging to the second domain,

The step of generating the paired image set is,

Further comprising configuring the image of the first domain and the generated matched image as a pair,

How to convert images.
According to clause 8,

The generated matched image is an image belonging to the second domain,

The GAN model is constructed by learning the image of the first domain and the generated matched image,

The step of generating at least a portion of the paired image set includes:

Converting the image of the first domain into a synthetic image of the second domain through the constructed GAN model; and

Comprising the step of configuring the image of the first domain and the composite image of the second domain as a pair,

How to convert images.
According to clause 10,

Further comprising the step of configuring the pair,

In response to determining that the correlation coefficient between the image of the first domain and the composite image of the second domain is greater than or equal to a reference value, comprising configuring the two images as a pair,

How to convert images.
According to clause 8,

The generated matched image is an image belonging to the second domain,

The GAN model is constructed by learning the image of the first domain and the generated matched image,

The step of generating at least a portion of the paired image set includes:

Converting the registered image or the image of the second domain into a synthetic image of the first domain through the constructed GAN model; and

Comprising the step of configuring the composite image of the first domain and the image of the second domain or the matched image as a pair,

How to convert images.
According to clause 8,

The generated matched image is an image belonging to the first domain,

The GAN model is constructed by learning the image of the second domain and the generated matched image,

The step of generating at least a portion of the paired image set includes:

Converting the image of the first domain or the registered image into a synthetic image of the second domain through the constructed GAN model; and

Comprising the step of configuring the image or the matched image of the first domain and the composite image of the second domain as a pair,

How to convert images.
According to paragraph 1,

Further comprising converting the first image into a second image of higher quality than the first image using the constructed deep learning model,

How to convert images.
According to clause 14,

Further comprising evaluating the quality of the second image using a designated evaluation metric,

The evaluation metrics specified above are:

Correlation coefficient between the first image and the second image,

Signal-to-noise ratio difference between the first image and the second image, and

Including the difference in pixel intensity between the first image and the second image,

How to convert images.
According to clause 14,

The first image is an MR (Magnetic Resonance) image of the diagnosed terminal,

Converting the second image into a CT image using another deep learning model learned to convert an MR image into a CT (Computer Tomography) image,

How to convert images.
According to paragraph 1,

The deep learning model includes an encoder and a decoder,

The encoder includes a plurality of encoding blocks that extract feature data of different abstraction levels from the input image,

The decoder includes a plurality of decoding blocks corresponding to the plurality of encoding blocks,

A skip-connection is formed between the plurality of encoding blocks and the plurality of decoding blocks to transmit the extracted feature data,

How to convert images.
A method performed by at least one computing device, comprising:

Obtaining a first MR (Magnetic Resonance) image for the terminal being diagnosed;

Converting the first MR image into a second MR image through a first deep learning model, where the second MR image is a higher quality image than the first MR image; and

Including converting the second MR image into a CT (Computed Tomography) image through a second deep learning model,

How to convert images.
According to clause 18,

The step of converting to the CT image is,

evaluating the quality of the second MR image using a designated evaluation metric; and

In response to determining that the evaluated quality satisfies a predetermined condition, converting the second MR image into the CT image,

The evaluation metrics specified above are:

Correlation coefficient between the first MR image and the second MR image,

Signal-to-noise ratio difference between the first MR image and the second MR image, and

Including a difference in signal intensity between the first MR image and the second MR image,

How to convert images.
According to clause 18,

The first deep learning model was built using a paired image set,

The paired image set is generated from an unpaired image set using a GAN (Generative Adversarial Networks) model with domain translation capability,

The unpaired image set includes an MR image of a first domain and an MR image of a second domain,

The MR image of the second domain is a higher quality image than the MR image of the first domain,

How to convert images.
One or more processors; and

Includes memory for storing one or more instructions,

The one or more processors:

By executing one or more instructions stored above,

An operation of acquiring an unpaired image set including an image of a first domain and an image of a second domain, wherein the image of the second domain is a higher quality image than the image of the first domain.

An operation of generating a paired image set from the obtained unpaired image set, and

Performing an operation to build a deep learning model that performs quality conversion on the input image using the generated paired image set,

Image conversion system.
A method performed by at least one computing device, comprising:

Obtaining an unpaired image set including an image of a first domain and an image of a second domain;

generating a paired image set from the obtained unpaired image set; and

Comprising the step of building a deep learning model that performs domain transformation on the input image using the generated paired image set,

How to convert images.