KR102628166B1 - Method, program, and apparatus for super-resolution based on artificial neural network - Google Patents

Abstract

An artificial neural network-based resolution enhancement method, program, and device according to an embodiment of the present disclosure are disclosed. The method includes acquiring a plurality of learning data having a plurality of acquisition matrix sizes; And based on the plurality of learning data, a neural network model that outputs data having an image size greater than or equal to the acquisition matrix size of the first label whose acquisition matrix size is the maximum among the labels included in the plurality of learning data. It may include a learning step.

Description

Artificial neural network-based resolution enhancement method, program, and device {METHOD, PROGRAM, AND APPARATUS FOR SUPER-RESOLUTION BASED ON ARTIFICIAL NEURAL NETWORK}

This disclosure relates to image processing technology, and specifically to a method and device for training an artificial neural network using data composed of various resolutions and improving the resolution of the data using the learned artificial neural network.

In order to train an artificial neural network that performs a task called super-resolution, pairs of low-resolution images and high-resolution images are typically used as learning data. When an artificial neural network is trained using a low-resolution image as input and a high-resolution image as a label, the artificial neural network is trained to increase the resolution of the input closer to the label.

Meanwhile, in reality, it is often difficult to obtain high-resolution learning data with a single resolution. In addition, it is more common that the resolution of the low-resolution image that will be input to the trained artificial neural network is not actually single. For example, in the case of magnetic resonance imaging, the resolution of the image varies depending on the type of equipment, imaging area, pulse sequence, or preferences of the hospital and medical staff.

In a realistic situation where learning data composed of a single resolution cannot be secured, if an artificial neural network is trained using learning data composed of various resolutions. There may be a disadvantage in that the artificial neural network is not trained to output data with a resolution corresponding to the maximum value among the training data, and the artificial neural network is affected by low-resolution training data and outputs data with a resolution smaller than the maximum value.

For example, assume that there are two training images that are similar but not of the same resolution. At this time, if among the two images, image A has a resolution of 512x512 and image B has a resolution of 256x256, when training the artificial neural network through a typical learning method, the artificial neural network will use an image with a resolution corresponding to the maximum value of 512x512. Printing is not guaranteed. Since A data with a resolution of 512x512 and B data with a resolution of 256x256 will be used as labels, an image with a resolution between the two images may be output through an artificial neural network, or an abnormal image with artifacts may be output.

Republic of Korea Patent Publication No. 10-2018-0114488 (2018.10.18)

The present disclosure was made in response to the above-described background technology, and provides a method and device for training an artificial neural network to output data close to the maximum resolution among data with various resolutions and improving the resolution of the data through the learned artificial neural network. We would like to provide

However, the problems to be solved by this disclosure are not limited to the problems mentioned above, and other problems not mentioned can be clearly understood based on the description below.

According to an embodiment of the present disclosure for realizing the above-described problem, an artificial neural network-based resolution enhancement method performed by a computing device is disclosed. The method includes acquiring a plurality of learning data having a plurality of acquisition matrix sizes; And based on the plurality of learning data, a neural network model that outputs data having an image size greater than or equal to the acquisition matrix size of the first label whose acquisition matrix size is the maximum among the labels included in the plurality of learning data. It may include a learning step.

Alternatively, based on the plurality of learning data, a neural network model outputs data having an image size greater than or equal to the acquisition matrix size of the first label whose acquisition matrix size is the maximum among the labels included in the plurality of learning data. The training step includes using the neural network model to generate output data having an image size larger than or equal to the size of the acquisition matrix of the first label, based on inputs included in the plurality of training data; adjusting the image size of the output data to correspond to the image size of a second label matching the input used to generate the output data; And it may include calculating an error between the image size-adjusted output data and the second label through a first loss function.

Alternatively, adjusting the image size of the output data to correspond to the image size of the second label matching the input used to generate the output data includes performing interpolation on the output data. By doing so, it may include reducing the image size of the output data to correspond to the image size of the second label.

Alternatively, the first loss function is based on the acquisition matrix size of the image size-reduced output data and the acquisition matrix size of the second label, between the image size-reduced output data and the second label. It may be a loss function for calculating similarity.

Alternatively, based on the plurality of learning data, a neural network model outputs data having an image size greater than or equal to the acquisition matrix size of the first label whose acquisition matrix size is the maximum among the labels included in the plurality of learning data. The learning step includes adjusting the image size of the second label to correspond to the image size of the generated output data; And it may further include calculating an error between the generated output data and the second label whose image size is adjusted through a second loss function.

Alternatively, adjusting the image size of the second label to correspond to the image size of the generated output data may include performing interpolation on the second label to correspond to the image size of the generated output data. Preferably, the step of enlarging the image size of the second label may be included.

Alternatively, the second loss function is based on the acquisition matrix size of the generated output data and the acquisition matrix size of the second label with the image size enlarged, the generated output data and the image size are enlarged. It may be a loss function for calculating similarity between second labels.

Alternatively, based on the plurality of learning data, a neural network model outputs data having an image size greater than or equal to the acquisition matrix size of the first label whose acquisition matrix size is the maximum among the labels included in the plurality of learning data. The learning step may further include training the neural network model based on a weighted sum of the calculation results through the first loss function and the calculation results through the second loss function.

According to an embodiment of the present disclosure for realizing the above-described problem, an artificial neural network-based resolution enhancement method performed by a computing device is disclosed. The method includes acquiring a low-resolution medical image; and generating a high-resolution medical image with improved resolution compared to the low-resolution medical image using a pre-trained neural network model. At this time, the neural network model includes a neural network that outputs data having an image size larger than or equal to the acquisition matrix size of the first label whose acquisition matrix size is the maximum among labels included in a plurality of learning data having a plurality of acquisition matrix sizes. can do.

According to an embodiment of the present disclosure for realizing the above-described object, a computer program stored in a computer-readable storage medium is disclosed. When the computer program is executed on one or more processors, it performs operations to improve the resolution of data based on an artificial neural network. At this time, the operations include acquiring a plurality of learning data having a plurality of acquisition matrix sizes; And, based on the plurality of learning data, an operation of training a neural network model that outputs data having an image size greater than or equal to the acquisition matrix size of the first label whose acquisition matrix size is the maximum among the labels included in the plurality of learning data. may include.

According to an embodiment of the present disclosure for realizing the above-described problem, a computing device that improves the resolution of data based on an artificial neural network is disclosed. The device includes a processor including at least one core; a memory containing program codes executable on the processor; And it may include a network unit that acquires a plurality of learning data having a plurality of acquisition matrix sizes. At this time, the processor is a neural network that, based on the plurality of learning data, outputs data having an image size greater than or equal to the acquisition matrix size of the first label whose acquisition matrix size is the maximum among the labels included in the plurality of learning data. You can train a model.

In the present disclosure, even if training data with a single resolution cannot be secured in reality, the resolution improvement performance of the artificial neural network for arbitrary data can be guaranteed by effectively training the artificial neural network using training data composed of various resolutions.

1 is a block diagram of a computing device according to an embodiment of the present disclosure.
Figure 2 is a block diagram showing the learning process of a neural network model according to an embodiment of the present disclosure.
Figure 3 is a flow chart showing a resolution improvement method according to an embodiment of the present disclosure.
Figure 4 is a flowchart showing the learning process of a neural network model according to an embodiment of the present disclosure.
Figure 5 is a flow chart showing a resolution improvement method according to an alternative embodiment of the present disclosure.

Below, with reference to the attached drawings, embodiments of the present disclosure are described in detail so that those skilled in the art (hereinafter referred to as skilled in the art) can easily implement the present disclosure. The embodiments presented in this disclosure are provided to enable any person skilled in the art to use or practice the subject matter of this disclosure. Accordingly, various modifications to the embodiments of the present disclosure will be apparent to those skilled in the art. That is, the present disclosure can be implemented in various different forms and is not limited to the following embodiments.

The same or similar reference numerals refer to the same or similar elements throughout the specification of this disclosure. Additionally, in order to clearly describe the present disclosure, reference numerals in the drawings may be omitted for parts that are not related to the description of the present disclosure.

As used in this disclosure, the term “or” is intended to mean an inclusive “or” and not an exclusive “or.” That is, unless otherwise specified in the present disclosure or the meaning is not clear from the context, “X uses A or B” should be understood to mean one of natural implicit substitutions. For example, unless otherwise specified in the present disclosure or the meaning is not clear from the context, “X uses A or B” means that It can be interpreted as one of the cases where all B is used.

The term “and/or” as used in this disclosure should be understood to refer to and include all possible combinations of one or more of the listed related concepts.

The terms “comprise” and/or “comprising” as used in this disclosure should be understood to mean that certain features and/or elements are present. However, the terms "comprise" and/or "including" should be understood as not excluding the presence or addition of one or more other features, other components, and/or combinations thereof.

Unless otherwise specified in this disclosure or the context is clear to indicate a singular form, the singular should generally be construed to include “one or more.”

The term “Nth (N is a natural number)” used in the present disclosure can be understood as an expression used to distinguish the components of the present disclosure according to a predetermined standard such as a functional perspective, a structural perspective, or explanatory convenience. there is. For example, in the present disclosure, components performing different functional roles may be distinguished as first components or second components. However, components that are substantially the same within the technical spirit of the present disclosure but must be distinguished for convenience of explanation may also be distinguished as first components or second components.

Meanwhile, the term "module" or "unit" used in this disclosure refers to a computer-related entity, firmware, software or part thereof, hardware or part thereof. , can be understood as a term referring to an independent functional unit that processes computing resources, such as a combination of software and hardware. At this time, the “module” or “unit” may be a unit composed of a single element, or may be a unit expressed as a combination or set of multiple elements. For example, a "module" or "part" in the narrow sense is a hardware element of a computing device, or set of pieces thereof, an application program that performs a specific function of software, a process implemented through software execution, or a program execution. It may refer to a set of instructions for . Additionally, as a broad concept, “module” or “unit” may refer to the computing device itself constituting the system, or an application running on the computing device. However, since the above-described concept is only an example, the concept of “module” or “unit” may be defined in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.

As used in this disclosure, the term "model" refers to a system implemented using mathematical concepts and language to solve a specific problem, a set of software units to solve a specific problem, or a process to solve a specific problem. It can be understood as an abstract model of a process. For example, a neural network “model” may refer to an overall system implemented as a neural network that has problem-solving capabilities through learning. At this time, the neural network can have problem-solving capabilities by optimizing parameters connecting nodes or neurons through learning. A neural network “model” may include a single neural network or a neural network set in which multiple neural networks are combined.

The term “image” used in this disclosure may refer to multidimensional data composed of discrete image elements. In other words, “image” can be understood as a term referring to a digital representation of an object that can be seen by the human eye. For example, “image” may refer to multidimensional data consisting of elements corresponding to pixels in a two-dimensional image. “Image” may refer to multidimensional data consisting of elements corresponding to voxels in a three-dimensional image.

As used in this disclosure, the term “image size” may mean the absolute number of pixels that make up the height axis, width axis, and/or depth axis of an image. Additionally, the term “acquisition matrix size” used in the present disclosure refers to the size in pixels obtained when taking an image, and can be understood as the minimum size that can include all information of the image. In other words, the term “acquisition matrix size” is a term related to the resolution of the original image, and may mean the minimum number of pixels for decomposing the original image without loss of information.

For example, an image with an acquisition matrix size of 800x600 may mean an image obtained with a height of 800 pixels and a width of 600 pixels. At this time, the acquisition matrix size 800x600 may be equal to or smaller than the image size. Assume that an image with an acquisition matrix size of 800x600 is enlarged by 2 times so that the image size is 1600x1200. At this time, the image size of the enlarged image is 1600x1200, but the acquisition matrix size is 800x600. According to the description of the size of the acquisition matrix described above, the height axis signal values of the enlarged image can be decomposed or expressed by at least 800 pixels, and cannot be decomposed into pixels smaller than that. Conversely, assume that an image with an acquisition matrix size and image size of 800x600 is reduced to an image size of 400x300. At this time, the reduced image can be decomposed into at least 400x300 pixels, so the acquisition matrix size is reduced to 400x300. Also, the image size is 400x300.

In the present disclosure, “accelerated imaging” can be understood as an imaging technique that shortens imaging time by reducing the number of excitations (NEX) for magnetic resonance signals compared to general imaging. The number of excitations can be understood as the number of repetitions when repeatedly acquiring lines of a magnetic resonance signal in the K-space domain. Therefore, as the number of excitations increases, the capturing time of the magnetic resonance image may increase proportionally. That is, when the number of excitations is reduced when taking a magnetic resonance image, accelerated imaging with a shortened magnetic resonance image taking time can be implemented.

In the present disclosure, “accelerated imaging” can be understood as an imaging technique that obtains an image with relatively low resolution by obtaining a narrower range of signals in the phase encoding direction in the K-space domain. In other words, accelerated imaging of the present disclosure can be understood as a imaging technique that reduces phase resolution compared to general imaging. Phase resolution can be understood as the number of lines sampled in the phase encoding direction in the K-space domain divided by a preset reference value. Therefore, as the phase resolution increases, the capturing time of the magnetic resonance image may increase proportionally. That is, when the phase resolution is reduced when capturing a magnetic resonance image, accelerated imaging with a shortened magnetic resonance image capturing time can be implemented.

In the present disclosure, “accelerated shooting” can be understood as a shooting technique that shortens the shooting time by increasing the acceleration factor compared to regular shooting. The acceleration index is a term used in parallel imaging techniques, and can be understood as the number of signal lines fully sampled in K-space divided by the number of signal lines sampled through imaging. For example, an acceleration index of 2 can be understood as acquiring half the number of signal lines compared to the number of fully sampled signal lines when acquiring a line by sampling a magnetic resonance signal in the phase encoding direction. Therefore, as the acceleration index increases, the capturing time of the magnetic resonance image may decrease proportionally. That is, if the acceleration index is increased when taking a magnetic resonance image, accelerated imaging with a shortened magnetic resonance image taking time can be implemented.

In the present disclosure, “accelerated imaging” can be understood as an imaging technique that generates a magnetic resonance image by acquiring a sub-sampled magnetic resonance signal. At this time, subsampling can be understood as an operation of sampling a magnetic resonance signal at a sampling rate lower than the Nyquist sampling rate. Accordingly, the medical data of the present disclosure may be images obtained by sampling magnetic resonance signals at a sampling rate lower than the Nyquist sampling rate.

However, since the above description of “accelerated photography” is only an example, the concept of accelerated photography may be defined in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.

The explanation of the foregoing terms is intended to aid understanding of the present disclosure. Therefore, if the above-mentioned terms are not explicitly described as limiting the content of the present disclosure, it should be noted that the content of the present disclosure is not used in the sense of limiting the technical idea.

1 is a block diagram of a computing device according to an embodiment of the present disclosure.

The computing device 100 according to an embodiment of the present disclosure may be a hardware device or part of a hardware device that performs comprehensive processing and calculation of data, or may be a software-based computing environment connected to a communication network. For example, the computing device 100 may be a server that performs intensive data processing functions and shares resources, or it may be a client that shares resources through interaction with the server. Additionally, the computing device 100 may be a cloud system in which a plurality of servers and clients interact to comprehensively process data. Since the above description is only an example related to the type of computing device 100, the type of computing device 100 may be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.

Referring to FIG. 1, a computing device 100 according to an embodiment of the present disclosure may include a processor 110, a memory 120, and a network unit 130. there is. However, since FIG. 1 is only an example, the computing device 100 may include other components for implementing a computing environment. Additionally, only some of the configurations disclosed above may be included in computing device 100.

The processor 110 according to an embodiment of the present disclosure may be understood as a structural unit including hardware and/or software for performing computing operations. For example, the processor 110 may read a computer program and perform data processing for machine learning. The processor 110 may process computational processes such as processing input data for machine learning, extracting features for machine learning, and calculating errors based on backpropagation. The processor 110 for performing such data processing includes a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), and a custom processing unit (TPU). It may include a semiconductor (ASIC: application specific integrated circuit), or a field programmable gate array (FPGA: field programmable gate array). Since the type of processor 110 described above is only an example, the type of processor 110 may be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.

The processor 110 can learn a neural network model to improve the resolution of data. The processor 110 may train the neural network model to improve the size of the acquisition matrix of the input based on a plurality of learning data having various acquisition matrix sizes. At this time, the neural network model may include a neural network that generates an output whose acquisition matrix size is increased compared to the input and has a constant image size. Additionally, the image size of the output generated by the neural network may be larger than or equal to the acquisition matrix size of a specific label whose acquisition matrix size is the maximum among labels included in a plurality of learning data having various acquisition matrix sizes. That is, the processor 110 may train the neural network model to output an image close to data whose acquisition matrix size is the maximum among a plurality of training data having various acquisition matrix sizes. For example, the processor 110 inputs data corresponding to the input from the training data obtained through the network unit 130, which will be described later, into a neural network model to create a constant image size determined based on the size of the acquisition matrix of the label of the training data. Can produce output. The processor 110 may compare the size of the acquisition matrix between the label corresponding to the input of the neural network model and the output generated based on the input of the neural network model. The processor 110 may train the neural network model to output data with an increased acquisition matrix size compared to the input by updating the parameters of the neural network included in the neural network model based on the comparison result.

The processor 110 may adjust the image size of the output of the neural network model during the learning process of the neural network model. For example, the processor 110 may adjust the image size of the output of the neural network model to match the size of the acquisition matrix of the label corresponding to the input used to generate the output. While the image size of the output of the neural network model is constant, the image size of the label can be configured in various ways to match the size of the acquisition matrix. Therefore, in order to easily and accurately compare the acquisition matrix size between the output of the neural network model and the label, the processor 110 sets the image size of the output determined based on the maximum value of the acquisition matrix size of the label to the label to be compared with the output. It can be reduced to fit the size of the acquisition matrix.

The processor 110 may adjust the image size of the learning data during the learning process of the neural network model. For example, the processor 110 may adjust the image size of a label whose image size is larger than the acquisition matrix size among labels included in a plurality of learning data. The processor 110 may reduce the image size of a label whose image size is larger than the acquisition matrix size to correspond to the acquisition matrix size. If the image size is larger than the acquisition matrix size, the amount of information representing the image is bound to be the same even if the image size is reduced. Therefore, in order to easily and accurately compare the size of the acquisition matrix between the output of the neural network model and the label, the processor 110 may adjust the image size of the label to match the size of the acquisition matrix of the label.

The processor 110 can convert low-resolution data into high-resolution data at a level desired by the user using a pre-trained neural network model as described above. The processor 110 can generate data with improved resolution compared to low-resolution data by improving the size of the acquisition matrix of low-resolution data through a pre-trained neural network model. For example, the processor 110 may input medical images into a neural network model to improve the resolution of the medical images to meet the quality level required by the hospital. At this time, the neural network model is trained to output data close to the maximum value among the various acquisition matrix sizes present in the training data, so the processor 110 can generate an image with improved resolution according to the maximum value learned by the neural network model. .

The memory 120 according to an embodiment of the present disclosure may be understood as a structural unit including hardware and/or software for storing and managing data processed in the computing device 100. That is, the memory 120 can store any type of data generated or determined by the processor 110 and any type of data received by the network unit 130. For example, the memory 120 may be a flash memory type, hard disk type, multimedia card micro type, card type memory, or random access memory (RAM). ), SRAM (static random access memory), ROM (read-only memory), EEPROM (electrically erasable programmable read-only memory), PROM (programmable read-only memory), magnetic memory , a magnetic disk, or an optical disk may include at least one type of storage medium. Additionally, the memory 120 may include a database system that controls and manages data in a predetermined system. Since the type of memory 120 described above is only an example, the type of memory 120 may be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.

The memory 120 can manage data necessary for the processor 110 to perform operations, a combination of data, and program code executable on the processor 110 by structuring and organizing them. For example, the memory 120 may store learning data obtained through the network unit 130, which will be described later. The memory 120 includes program code that operates the neural network model to learn based on learning data, program code that operates the neural network model to perform feature analysis (or inference) on the data, and various data calculated as the program code is executed. etc. can be saved.

The network unit 130 according to an embodiment of the present disclosure may be understood as a structural unit that transmits and receives data through any type of known wired or wireless communication system. For example, the network unit 130 is a local area network (LAN), wideband code division multiple access (WCDMA), long term evolution (LTE), and WiBro (wireless). broadband internet, 5th generation mobile communication (5G), ultra wide-band wireless communication, ZigBee, radio frequency (RF) communication, wireless LAN, wireless fidelity ), data transmission and reception can be performed using a wired or wireless communication system such as near field communication (NFC), or Bluetooth. Since the above-described communication systems are only examples, the wired and wireless communication systems for data transmission and reception of the network unit 130 may be applied in various ways other than the above-described examples.

The network unit 130 may receive data necessary for the processor 110 to perform calculations through wired or wireless communication with any system or client. Additionally, the network unit 130 may transmit data generated through the calculation of the processor 110 through wired or wireless communication with any system or any client. For example, the network unit 130 may receive medical images of body parts through communication with a medical image storage and transmission system. The network unit 130 may transmit various data generated through calculations of the processor 110 through communication with the above-described system, etc.

Figure 2 is a block diagram showing the learning process of a neural network model according to an embodiment of the present disclosure.

The processor 110 of the computing device 100 according to an embodiment of the present disclosure may use images having various acquisition matrix sizes as training data for a neural network model to improve resolution. At this time, images used as learning data can be distinguished as inputs or labels of a neural network model. Additionally, the size of the acquisition matrix of the image used as learning data may be the same as or smaller than the image size. When the acquisition matrix size is smaller than the image size, for smooth learning of the neural network model, the processor 110 may reduce the image size of the image used as training data to match the acquisition matrix size.

For example, referring to FIG. 2 , the processor 110 may distinguish and use images having different acquisition matrix sizes as the input set 10 or the label set 20 of the neural network model 200. To learn the neural network model 200, the processor 110 may input input A 11 to the neural network model 200. Additionally, the processor 110 may use the label A (21) matching the input A (11) to verify the output of the neural network model 200 that receives the input A (11).

Meanwhile, the neural network model 200 may include a neural network that receives an image to be used as an input to the neural network model and generates an output with a constant image size. At this time, the constant image size may be larger than or equal to the acquisition matrix size of the image with the maximum acquisition matrix size among the images to be used as labels. That is, the neural network model 200 may include a neural network that generates an output with an image size that is greater than or equal to the maximum acquisition matrix size based on images to be used as labels. The neural network model 200 can be trained so that the output can express the maximum value of the acquisition matrix size no matter what size the acquisition matrix is input through this neural network.

For example, in FIG. 2, let us assume that among the label set 20, the label with the largest acquisition matrix size is label B(22), and the acquisition matrix size of label B(22) is R. At this time, R may be the maximum value in the width (x) axis direction, the maximum value in the height (y) axis direction, and/or the maximum value in the depth (z) axis direction of the image. The neural network included in the neural network model 200 can generate an output whose image size has a constant value of C no matter what input it obtains. At this time, C may be a value greater than or equal to R. In other words, the neural network model 200 generates an output with an image size of a constant value of C, which is greater than or equal to the acquisition matrix size R of label B (22), regardless of whether input A (11) or input B (12) is input. can do.

The processor 110 may input an image to be used as an input to the neural network model among the training data into the neural network model and generate an output with an increased acquisition matrix size compared to the input. At this time, the output generated from the neural network model may have an image size that is greater than or equal to the maximum value among the acquisition matrix sizes of labels. To learn a neural network model, the processor 110 may adjust the image size of the output to match the size of the acquisition matrix of the label to be compared with the output. Additionally, the processor 110 may calculate the error between the output with the image size adjusted and the label through a loss function, and train a neural network model based on the calculated error.

For example, as shown in FIG. 2, assume that the neural network model 200 is trained using input A(11) and label A(21) with an acquisition matrix size r1 among the training data. The processor 110 may input input A 11 into the neural network model 200 and generate output A 31 with an image size C. The processor 110 performs interpolation on output A (31) to reduce the image size of output A (31) generated using the neural network model 200 to r1, which is the acquisition matrix size of label A (21). can be performed. The processor 110 may use the first loss function to calculate an error between the output A' (32) and the label A (21) generated through reduced interpolation for the output A (31). At this time, the first loss function is based on the acquisition matrix size of output A' (32) with an image size of r1 and the acquisition matrix size of label A (21) with an acquisition matrix size of r1, output A' (32) and label It may be a loss function that calculates the similarity between A(21). As the first loss function, a structural similarity (SSIM) loss function, (MSE mean squared error) loss function, etc. may be used, but the present disclosure is not limited to these examples. The processor 110 configures the neural network model 200 so that the neural network model 200 generates an image with a maximum matrix acquisition size R while having a constant image size C based on the similarity calculation result derived through the first loss function. It can be learned.

By repeatedly performing the above-described process for input B 12, etc., the neural network model 200 can be stably trained even with images having various acquisition matrix sizes. In more detail, let us assume that the image to be used for learning the neural network model 200 has the following acquisition matrix size.

- Input A(11): 320x320 / Label A(21): 400X400

- Input B(12): 320x320 / Label B(22): 512x512

At this time, it is assumed that the acquisition matrix size and image size of each image are the same. And, since the acquisition matrix size of label B (22) has the maximum value of 512x512, it is assumed that the neural network model 200 includes a neural network that receives an image of 640x640 and outputs an image of 640x640, which is larger than 512x512. .

According to this assumption, the processor 110 can input input A 11 into the neural network model 200 and generate output A 31 having an image size of 640x640. At this time, in order to input input A (11) into the neural network model 200, the processor 110 may enlarge the image size of input A (11) from 320x320 to 640x640 through interpolation. The processor 110 may generate output A' (32) by reducing the image size of output A (31) to 400x400, which is the size of the acquisition matrix of label A (21), through interpolation. The processor 110 can learn a neural network model by calculating a loss value based on output A' (32) with an image size of 400x400 and label A (21) with an acquisition matrix size of 400x400.

After learning about input A (11), the processor 110 may input input B (12) into the neural network model 200 to generate output B with an image size of 640x640. At this time, in order to input input B (12) into the neural network model 200, the processor 110 may enlarge the image size of input B (12) from 320x320 to 640x640 through interpolation. The processor 110 may generate output B' by reducing the image size of output B to 512x512, which is the size of the acquisition matrix of label B 22, through interpolation. The processor 110 may calculate a loss value based on output B' with an image size of 512x512 and label B(22) with an acquisition matrix size of 512x512, and train a neural network model to minimize the loss value.

As described above, when an output with a constant image size is generated according to the maximum acquisition matrix size of the label and learning is performed to adjust the image size of the output to compare the label corresponding to the output and the acquisition matrix size, the neural network model 200 Learning can be performed so that both label A (21) and label B (22) can be appropriately predicted. That is, when learning is performed as described above, the neural network model 200 is trained so that the acquisition matrix size of the output with an image size of 640x640 is 512x512 or more, so that the neural network model 200 does not use any labels included in the training data. It is predictable. If learning is not performed as described above, the neural network model 200 generates an output with an acquisition matrix size between 400x400, which is the acquisition matrix size of label A (21), and 512x512, which is the acquisition matrix size of label B (22). Otherwise, it has no choice but to produce output containing artifacts. In other words, if learning is not performed as described above, the neural network model 200 can predict label A (21) with an acquisition matrix size of 400x400, but cannot predict label B (22) with an acquisition matrix size of 512x512. does not exist.

Meanwhile, if the image size of label A (21) or label B (22) is larger than the acquisition matrix size, the processor 110 reduces the image size of label A (21) or label B (22) by the acquisition matrix size. You can. Since it is the minimum size that can contain all the information of the image, the amount of information included in the label remains the same even if the image size is reduced by the size of the acquisition matrix. Therefore, so that the neural network model can exclude unnecessary redundant information and perform learning efficiently, the processor 110 can reduce the image size of label A (21) or label B (22) by the size of the acquisition matrix. .

Alternatively, for training of a neural network model, the processor 110 may adjust the image size of the output to match the size of the acquisition matrix of the labels to be compared with the output. Additionally, the processor 110 may adjust the image size of the label to be compared with the output according to the image size of the output source. The processor 110 can calculate the error between the image resized output and the label as well as the error between the image resized label and the output source through a loss function, and learn a neural network model based on the calculated errors.

For example, as shown in FIG. 2, assume that the neural network model 200 is trained using input A(11) and label A(21) with an acquisition matrix size r1 among the training data. The processor 110 may input input A 11 into the neural network model 200 and generate output A 31 with an image size C. The processor 110 performs interpolation on output A (31) to reduce the image size of output A (31) generated using the neural network model 200 to r1, which is the acquisition matrix size of label A (21). can be performed. Additionally, the processor 110 may perform interpolation on label A (21) to enlarge the image size of label A (21) to C, which is the image size of the output (31). The processor 110 may use the first loss function to calculate an error between the output A' (32) and the label A (21) generated through reduced interpolation for the output A (31). Additionally, the processor 110 may use the second loss function to calculate the error between label A' (40) generated through expanded interpolation for label A (21) and output A (31). At this time, the second loss function is output A ( It may be a loss function that calculates the similarity between 31) and label A'(40). As the second loss function, a structural similarity (SSIM) loss function, (MSE mean squared error) loss function, etc. may be used, but the present disclosure is not limited to these examples. The processor 110 determines that the neural network model 200 has a constant image size C and a maximum matrix acquisition size R based on the similarity calculation result derived through the first loss function and the similarity calculation result derived through the second loss function. The neural network model 200 can be trained to generate an image with

In more detail, the processor 110 may train the neural network model 200 using a loss function expressed as a weighted sum of the first loss function and the second loss function. At this time, the first loss function is the acquisition between output A'(32) and label A(21), which is generated by reducing and interpolating output A(31), which has an image size of 640x640, to 400X400, which is the acquisition matrix size of label A(21). Based on the matrix size, it may be a loss function for calculating the similarity between output A' (32) and label A (21). And, the second loss function is output A (31), which has an image size of 640x640, and label A' (40), which is generated by enlarging and interpolating the image size of label A (21) to 640x640, which is the image size of output A (31). It may be a loss function for calculating the similarity between output A (31) and label A' (40) based on the size of the acquisition matrix between them. Therefore, the loss function expressed as the weighted sum of the first loss function and the second loss function can be expressed as follows [Equation 1].

[Equation 1]

L _total = L _One (output A'(32), label A(21)) + α*L ₂ (output A(31), label A'(40))

At this time, L ₁ represents the first loss function and L ₂ represents the second loss function. And, the weight α can be applied to the second loss function L ₂ to adjust the ratio of the second loss function L ₂ to the overall loss function. In this way, by additionally considering the second loss function L ₂ , the proportion of which can be adjusted with the weight α, in addition to the first loss function _{L 1} , the processor 110 configures the neural network model 200 to minimize artifacts that may occur in the output. (200) can be learned. That is, the processor 110 can learn the neural network model 200 more effectively by additionally reflecting the second loss function in the loss value calculation.

Figure 3 is a flow chart showing a resolution improvement method according to an embodiment of the present disclosure. And, Figure 4 is a flowchart showing the learning process of a neural network model according to an embodiment of the present disclosure.

Referring to FIG. 3, the computing device 100 according to an embodiment of the present disclosure may acquire a plurality of learning data having a plurality of acquisition matrix sizes (S110). Here, acquisition of learning data refers not only to receiving learning data through a wireless communication network with an external terminal, device, or system, but also to generating or receiving learning data in an on-device form. It can be understood. For example, the computing device 100 may receive an image generated by an image capture device as learning data through cloud communication with the image capture device. The computing device 100 may be mounted on an image capture device and directly collect captured images as learning data. In reality, it is difficult to acquire images with uniformly distributed acquisition matrix sizes as learning data depending on the shooting conditions of the video capture device, the shooting environment, the user's preference, etc., so the computing device 100 collects images with various acquisition matrix sizes. It can be obtained as learning data.

The computing device 100 may train a neural network model that outputs data having an image size greater than or equal to the acquisition matrix size of the first label with the maximum acquisition matrix size among the labels included in the plurality of training data (S120). That is, assuming that N labels (N is a natural number) are secured, the image size of the output of the neural network model may be larger than the acquisition matrix size of the first label with the largest acquisition matrix size among the N labels. For example, if the size of the acquisition matrix of the first label is 128x128, the output of the neural network model may have an image size of 256x256, 512x512, etc., which is greater than 128x128. The computing device 100 may calculate the label and error corresponding to the output of the neural network model and train the neural network model so that the calculated error is minimized.

More specifically, referring to FIG. 4, the computing device 100 uses a neural network model to generate output data having an image size that is greater than or equal to the size of the acquisition matrix of the first label, based on the input included in the plurality of training data. You can do it (S121). At this time, the neural network model may generate output data having an image size larger than the acquisition matrix size of the input data and an image size larger than the acquisition matrix size of the first label. For example, assuming that the acquisition matrix size of the input data is 64x64, the acquisition matrix size of the first label is 128x128, and the image size of the output of the neural network model is 256x256, the neural network model has an acquisition matrix size of 64x64 or more and produces an image of 256x256. Output data of any size can be generated. At this time, the computing device 100 may adjust the image size of the input data to match the input size of the neural network model.

The computing device 100 may adjust the image size of the output data to correspond to the image size of the second label matching the input used to generate the output data (S122). The computing device 100 may reduce the image size of the output data to correspond to the image size of the second label by performing interpolation on the output data. For example, assuming that the acquisition matrix size of the second label is 100x100, the neural network model can reduce the image size of the output data from 256x256 to 100x100 through interpolation.

The computing device 100 may use the first loss function to calculate an error between the output data whose image size has been adjusted through step S122 and the second label (S123). At this time, the first loss function may be a loss function for calculating the difference between the acquisition matrix size of the output data with the image size reduced and the acquisition matrix size of the second label. For example, in step S122, the image size of the output data and the image size of the second label may be the same as 100x100, but the acquisition matrix size of the output data and the acquisition matrix size of the second label may be different. Accordingly, the computing device 100 may use the first loss function to calculate the similarity between the output data and the second label as a loss value, based on the size of the acquisition matrix of the output data and the size of the acquisition matrix of the second label.

The computing device 100 may adjust the image size of the second label to correspond to the image size of the output data generated through the neural network model (S124). By performing interpolation on the second label, the image size of the second label can be enlarged to correspond to the image size of the output data. For example, assuming that the image size of the second label is 100x100, the neural network model can enlarge the image size of the second label from 100x100 to 256x256 through interpolation.

Meanwhile, interpolation techniques for reducing the image size performed in step S122 or enlarging the image size performed in step S124 include bilinear interpolation, bicubic interpolation, and nearest interpolation. The same techniques may be used, but the disclosure is not limited to this example.

The computing device 100 may calculate an error between the generated output data and the second label with the image size adjusted through the second loss function (S125). The second loss function may be a loss function for calculating the difference between the acquisition matrix size of the output data and the acquisition matrix size of the second label with the enlarged image size. The error calculation results according to the second loss function are used for learning in addition to the error calculation results according to the first loss function, so that the neural network model can help remove artifacts that may appear in the output data.

The computing device 100 may update the parameters of the neural network model based on the error calculation result through the first loss function (S126). The computing device 100 may update the parameters of the neural network model so that the error derived through the first loss function is minimized. The computing device 100 may update the parameters of the neural network model based on the weighted sum of the calculation results through the first loss function and the calculation results through the second loss function (S126). The computing device 100 may update the parameters of the neural network model so that the error derived through the weighted sum of the first loss function and the second loss function is minimized. In this way, when the second loss function is used together, feature interpretation (or inference) of the neural network model can be effectively derived so that artifacts are not generated in the output of the neural network model.

Figure 5 is a flow chart showing a resolution improvement method according to an alternative embodiment of the present disclosure.

Referring to FIG. 5, the computing device 100 according to an alternative embodiment of the present disclosure can acquire a low-resolution medical image (S210). For example, the computing device 100 may receive a magnetic resonance image acquired through accelerated imaging through a medical image storage and transmission system or cloud communication with a magnetic resonance imaging device. The computing device 100 may receive a magnetic resonance image restored to a normal shooting level but with low resolution through communication with an artificial intelligence-based device that restores an accelerated magnetic resonance image. Additionally, the computing device 100 may be mounted on an artificial intelligence-based device that restores accelerated magnetic resonance images and directly obtain low-resolution magnetic resonance images.

The computing device 100 may use a pre-trained neural network model to generate a high-resolution medical image with improved resolution compared to a low-resolution medical image (S220). At this time, the neural network model has an image size that is greater than or equal to the acquisition matrix size of the first label whose acquisition matrix size is the maximum among the labels included in the plurality of learning data, based on a plurality of learning data having different acquisition matrix sizes. It may include a neural network that outputs data. The computing device 100 may generate a high-resolution medical image by inputting a low-resolution medical image into a neural network model learned through the learning process described with reference to FIG. 4 above.

For example, the computing device 100 may improve resolution by inputting a magnetic resonance image obtained through an accelerated imaging and restoration process into a neural network model. At this time, the degree of resolution improvement through the neural network model may vary depending on the size of the acquisition matrix of the learning data used to learn the neural network model. Since the neural network model is trained to generate output close to the maximum value among the acquisition matrix sizes of labels included in a plurality of learning data, the resolution of the magnetic resonance image can be improved based on the maximum acquisition matrix size learned by the neural network model.

The various embodiments of the present disclosure described above may be combined with additional embodiments and may be changed within the scope understandable to those skilled in the art in light of the above detailed description. The embodiments of the present disclosure should be understood in all respects as illustrative and not restrictive. For example, each component described as unitary may be implemented in a distributed manner, and similarly, components described as distributed may also be implemented in a combined form. Accordingly, all changes or modified forms derived from the meaning and scope of the claims of the present disclosure and their equivalent concepts should be construed as being included in the scope of the present disclosure.

Claims (11)

An artificial neural network-based resolution enhancement method performed by a computing device including at least one processor, comprising:
Obtaining a plurality of learning data having a plurality of acquisition matrix sizes; and
Based on the plurality of learning data, learning a neural network model that outputs data having an image size greater than or equal to the acquisition matrix size of the first label whose acquisition matrix size is the maximum among the labels included in the plurality of learning data. ordering step;
Including,
The step of training a neural network model that outputs data having an image size greater than or equal to the acquisition matrix size of the first label whose acquisition matrix size is the maximum among the labels included in the plurality of learning data, based on the plurality of learning data. ,
Using the neural network model, based on input included in the plurality of learning data, generating output data having an image size greater than or equal to the size of the acquisition matrix of the first label;
adjusting the image size of the output data to correspond to the image size of a second label matching the input used to generate the output data; and
calculating an error between the image size-adjusted output data and the second label through a first loss function;
Including,
method.
delete According to claim 1,
Adjusting the image size of the output data to correspond to the image size of the second label matching the input used to generate the output data,
Reducing the image size of the output data to correspond to the image size of the second label by performing interpolation on the output data;
Including,
method.
According to claim 3,
The first loss function is,
A loss function for calculating similarity between the output data with the image size reduced and the second label, based on the acquisition matrix size of the output data with the image size reduced and the acquisition matrix size of the second label,
method.
According to claim 4,
The step of training a neural network model that outputs data having an image size greater than or equal to the acquisition matrix size of the first label whose acquisition matrix size is the maximum among the labels included in the plurality of learning data, based on the plurality of learning data. ,
adjusting the image size of the second label to correspond to the image size of the generated output data; and
calculating an error between the generated output data and a second label whose image size has been adjusted through a second loss function;
Containing more,
method.
According to claim 5,
The step of adjusting the image size of the second label to correspond to the image size of the generated output data,
enlarging the image size of the second label to correspond to the image size of the generated output data by performing interpolation on the second label;
Including,
method.
According to claim 6,
The second loss function is,
A loss function for calculating similarity between the generated output data and the second label with an enlarged image size, based on the acquisition matrix size of the generated output data and the acquisition matrix size of the second label with the enlarged image size. person,
method.
According to claim 7,
The step of training a neural network model that outputs data having an image size greater than or equal to the acquisition matrix size of the first label whose acquisition matrix size is the maximum among the labels included in the plurality of learning data, based on the plurality of learning data. ,
Learning the neural network model based on a weighted sum of the calculation results through the first loss function and the calculation results through the second loss function;
Containing more,
method.
An artificial neural network-based resolution enhancement method performed by a computing device including at least one processor, comprising:
Obtaining a low-resolution medical image; and
Using a pre-trained neural network model, generating a high-resolution medical image with improved resolution compared to the low-resolution medical image;
Including,
The neural network model is,
Based on the input included in a plurality of learning data having a plurality of acquisition matrix sizes, an image size that is greater than or equal to the acquisition matrix size of the first label whose acquisition matrix size is the maximum among the labels included in the plurality of learning data Generate output data with
Learned by calculating an error between the output data and the second label, with the image size adjusted to correspond to the image size of the second label matching the input used to generate the output data, through a first loss function.
method.
A computer program stored in a computer-readable storage medium, wherein the computer program, when executed on one or more processors, performs operations to improve the resolution of data based on an artificial neural network,
The above operations are,
Obtaining a plurality of learning data having a plurality of acquisition matrix sizes; and
Based on the plurality of learning data, learning a neural network model that outputs data having an image size greater than or equal to the acquisition matrix size of the first label whose acquisition matrix size is the maximum among the labels included in the plurality of learning data. The action of telling;
Including,
Based on the plurality of learning data, learning a neural network model that outputs data having an image size greater than or equal to the acquisition matrix size of the first label whose acquisition matrix size is the maximum among the labels included in the plurality of learning data. The action to do is,
Using the neural network model, based on input included in the plurality of learning data, generating output data having an image size larger than or equal to the size of the acquisition matrix of the first label;
adjusting the image size of the output data to correspond to the image size of a second label matching the input used to generate the output data; and
calculating an error between the image size-adjusted output data and the second label through a first loss function;
Including,
computer program.
A computing device for improving the resolution of data based on an artificial neural network,
A processor including at least one core;
a memory containing program codes executable on the processor; and
A network unit that acquires a plurality of learning data having a plurality of acquisition matrix sizes;
Including,
The processor,
Based on the plurality of learning data, learning a neural network model that outputs data having an image size greater than or equal to the acquisition matrix size of the first label whose acquisition matrix size is the maximum among the labels included in the plurality of learning data. To do it,
Using the neural network model, based on input included in the plurality of learning data, generate output data having an image size that is larger than or equal to the size of the acquisition matrix of the first label,
Adjusting the image size of the output data to correspond to the image size of the second label matching the input used to generate the output data,
Calculating an error between the image size-adjusted output data and the second label through a first loss function,
Device.