KR102442591B1 - Method, program, and apparatus for generating label - Google Patents

Abstract

Disclosed is a method for generating a label performed by a computing device according to one embodiment of the present disclosure. The method may comprise: a step of detecting a lesion present in a medical image using a neural network model; and a step of generating a label indicating a lesion area in the medical image, based on an output of a predetermined layer included in the neural network model.

Description

Label generation methods, programs and devices {METHOD, PROGRAM, AND APPARATUS FOR GENERATING LABEL}

The present disclosure relates to a label generating method, and more particularly, to an artificial intelligence-based technology for generating a label capable of displaying a lesion in a medical image.

Medical images are data that enable us to understand the physical state of various organs in the human body for diagnosis and treatment of diseases. As medical imaging devices have been continuously developed, medical images are X-ray images, computed tomography (CT) images, positron emission tomography (PET) images, or magnetic resonance imaging. (MRI: magnetic resonance imaging), etc. are diversifying. With the development of image analysis technology and artificial intelligence technology, medical images are being used in various ways as analysis data to assist in the diagnosis of diseases. Typically, medical images are actively used in the field of lesion detection for detecting lesions captured on images for effective diagnosis of diseases.

Artificial intelligence technologies in the field of computer vision that have been used in various ways in the field of lesion detection so far are traditionally based on supervised learning. Since a label is absolutely necessary for supervised learning, existing technologies use data labeled after an expert directly identifies a lesion in a medical image for learning. However, since labels can only be created one by one through the manual work of experts, it is practically difficult to secure a sufficient amount of labels to guarantee the learning performance of artificial intelligence. That is, for supervised learning, the overall cost of generating a label is inevitably required considerably.

Republic of Korea Patent Publication No. 10-2013-0012297 (2013.02.04.)

The present disclosure has been devised in response to the above background art, and an object of the present disclosure is to provide a method for efficiently generating a label based on inference of artificial intelligence for lesion detection.

However, the problems to be solved in the present disclosure are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood based on the description below.

A label generation method performed by a computing device according to an embodiment of the present disclosure for realizing the above-described problems is disclosed. The method includes the steps of using a neural network model to detect a lesion present in a medical image; and generating a label indicating a lesion region in the medical image based on an output of a predetermined layer included in the neural network model.

Alternatively, the medical image may be a magnetic resonance image generated through a Fourier transform based on k-space data obtained from a magnetic resonance signal.

Alternatively, the neural network model may receive the medical image and a scan parameter that can be adjusted when the medical image is captured.

Alternatively, the detecting of the lesion present in the medical image may include, based on the first feature calculated in the i-th (i is a natural number) layer of the neural network model, the second feature through a Fourier transform. generating; and inputting the first feature calculated from the i-th layer and the second feature generated through the Fourier transform into the i+1th layer of the neural network model.

Alternatively, the generating of the label indicating the lesion region in the medical image may include, based on the output of the predetermined layer, generating a heatmap corresponding to the feature analysis of the neural network model for the medical image. to do; and generating a label indicating the presence or absence of a lesion in a specific region in the medical image based on the heat map.

Alternatively, the generating the heat map corresponding to the feature analysis of the neural network model for the medical image includes extracting a gradient from the output of any one of the feature extraction layers included in the neural network model, It may include generating the heat map based on the gradient.

Alternatively, generating a label indicating the presence or absence of a lesion in a specific region in the medical image based on the heat map may include correcting a blank region existing in the lesion region expressed in the heat map. .

Alternatively, generating a label indicating the presence or absence of a lesion in a specific region in the medical image based on the heat map may include correcting noise present in the heat map.

Alternatively, the neural network model may be trained to classify the presence or absence of a lesion based on at least one of a first learning image in which a lesion exists and a second learning image in which a lesion does not exist.

A computer program stored in a computer-readable storage medium according to an embodiment of the present disclosure for realizing the above-described problems is disclosed. When the computer program is executed on one or more processors, the computer program performs operations for generating a label. In this case, the operations may include: detecting a lesion present in a medical image using a neural network model; and generating a label indicating a lesion region in the medical image based on an output of a predetermined layer included in the neural network model.

A computing device for generating a label is disclosed according to an embodiment of the present disclosure for realizing the above-described problem. The apparatus includes: a processor including at least one core; a memory including program codes executable by the processor; and a network unit for receiving a medical image. In this case, the processor may detect a lesion present in the medical image using the neural network model, and generate a label indicating the lesion region in the medical image based on the output of a predetermined layer included in the neural network model. have.

The present disclosure may provide a method for efficiently generating a label based on feature analysis of an artificial intelligence medical image for lesion detection.

1 is a block diagram of a computing device according to an embodiment of the present disclosure.
2 is a block diagram illustrating a label generation process of a computing device according to an embodiment of the present disclosure.
3 is a conceptual diagram illustrating a calculation process using a neural network model according to an embodiment of the present disclosure.
4 is a flowchart illustrating a label generation method according to an embodiment of the present disclosure.

Hereinafter, with reference to the accompanying drawings, embodiments of the present disclosure will be described in detail so that those skilled in the art (hereinafter, those skilled in the art) can easily implement. The embodiments presented in the present disclosure are provided to enable any person skilled in the art to make or use the subject matter of the present disclosure. Accordingly, various modifications to the embodiments of the present disclosure will be apparent to those skilled in the art. That is, the present disclosure may be implemented in several different forms, and is not limited to the following embodiments.

The same or similar reference numbers refer to the same or similar elements throughout the specification of this disclosure. In addition, in order to clearly describe the present disclosure, reference numerals of parts not related to the description of the present disclosure in the drawings may be omitted.

The term “or” as used in this disclosure is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless otherwise specified in the present disclosure or the meaning is not clear from context, "X employs A or B" should be understood to mean one of the natural implicit substitutions. For example, "X employs A or B" means that X employs A, X employs B, or X is A and It can be interpreted as any one of the cases in which all B are used.

It should be understood that the term “and/or” as used in this disclosure refers to and includes all possible combinations of one or more of the listed related concepts.

As used herein, the terms “comprises” and/or “comprising” should be understood to mean that certain features and/or elements are present. However, it should be understood that the terms "comprises" and/or "comprising" do not exclude the presence or addition of one or more other features, other elements, and/or combinations thereof.

Unless otherwise specified in the present disclosure or unless it is clear from context to refer to a singular form, the singular is generally to be construed as being able to include "one or more".

The term “Nth (N is a natural number)” used in the present disclosure may be understood as an expression used to distinguish the components of the present disclosure from a functional point of view, a structural point of view, or a predetermined standard such as convenience of explanation. have. For example, in the present disclosure, components performing different functional roles may be distinguished as a first component or a second component. However, components that are substantially the same within the technical spirit of the present disclosure but should be divided for convenience of description may also be distinguished as a first component or a second component.

On the other hand, the term "module" or "unit" used in the present disclosure refers to a computer-related entity, firmware, software or a part thereof, hardware or a part thereof. , may be understood as a term referring to an independent functional unit that processes computing resources, such as a combination of software and hardware. In this case, a “module” or “unit” may be a unit composed of a single element, or a unit expressed as a combination or set of a plurality of elements. For example, as a narrow concept, "module" or "unit" refers to a hardware element or a set of a computing device, an application program that performs a specific function of software, a process implemented through software execution, or a program. It may refer to a set of instructions for execution, and the like. Also, as a broad concept, “module” or “unit” may refer to a computing device itself constituting a system, an application executed in the computing device, or the like. However, since the above-described concept is only an example, the concept of “module” or “part” may be defined in various ways within a range that can be understood by those skilled in the art based on the contents of the present disclosure.

As used herein, the term "model" refers to a system implemented using mathematical concepts and language to solve a specific problem, a set of software units to solve a specific problem, or a process to solve a specific problem. It can be understood as an abstract model about the process. For example, a neural network “model” may refer to an overall system implemented as a neural network having problem-solving ability through learning. In this case, the neural network may have problem solving ability by optimizing parameters connecting nodes or neurons through learning. A neural network “model” may include a single neural network, or a neural network set in which a plurality of neural networks are combined.

As used herein, the term “image” may refer to multidimensional data composed of discrete image elements. In other words, "image" may be understood as a term referring to a digital representation of an object that can be seen by the human eye. For example, "image" may refer to multidimensional data composed of elements corresponding to pixels in a two-dimensional image. An “image” may refer to multidimensional data composed of elements corresponding to voxels in a 3D image.

As used herein, the term "picture archiving and communication system (PACS)" refers to storing, processing, and It may refer to a system that transmits. For example, the "medical image storage and transmission system" is linked with digital medical imaging equipment to transmit medical images such as magnetic resonance imaging (MRI) and computed tomography (CT) images to medical digital images and It can be saved according to the communication standard. The "medical image storage and transmission system" can transmit medical images to terminals inside and outside the hospital through a communication network. In this case, meta information such as a reading result and a medical record may be added to the medical image.

The term “k-space” used in the present disclosure may be understood as an arrangement of numbers representing spatial frequencies of a magnetic resonance image. In other words, “K-space” may be understood as a frequency space corresponding to a three-dimensional space corresponding to magnetic resonance space coordinates.

The term "Fourier transform" used in the present disclosure may be understood as a computational medium capable of describing the correlation between the time domain and the frequency domain. In other words, the "Fourier transform" used in the present disclosure may be understood as a broad concept representing an operation process for mutual transformation of a time domain and a frequency domain. Therefore, the "Fourier transform" used in the present disclosure is a concept that encompasses both a narrow Fourier transform that decomposes a signal in a time domain into a frequency domain and an inverse Fourier transform that transforms a signal in the frequency domain into a time domain. can be understood as

Descriptions of the foregoing terms are provided to aid understanding of the present disclosure. Therefore, it should be noted that, if the above-mentioned terms are not explicitly described as matters limiting the contents of the present disclosure, the contents of the present disclosure are not used in the meaning of limiting the technical idea.

1 is a block diagram of a computing device according to an embodiment of the present disclosure.

The computing device 100 according to an embodiment of the present disclosure may be a hardware device or a part of a hardware device that performs comprehensive data processing and calculation, or may be a software-based computing environment connected through a communication network. For example, the computing device 100 may be a server that performs an intensive data processing function and shares resources, or may be a client that shares resources through interaction with the server. Also, the computing device 100 may be a cloud system in which a plurality of servers and clients interact to comprehensively process data. Since the above description is only one example related to the type of the computing device 100 , the type of the computing device 100 may be variously configured within a range that can be understood by those skilled in the art based on the contents of the present disclosure.

Referring to FIG. 1 , a computing device 100 according to an embodiment of the present disclosure may include a processor 110 , a memory 120 , and a network unit 130 . have. However, since FIG. 1 is only an example, the computing device 100 may include other components for implementing a computing environment. In addition, only some of the disclosed components may be included in the computing device 100 .

The processor 110 according to an embodiment of the present disclosure may be understood as a structural unit including hardware and/or software for performing a computing operation. For example, the processor 110 may read a computer program and perform data processing for machine learning. The processor 110 may process an arithmetic process such as processing input data for machine learning, feature extraction for machine learning, and error calculation based on backpropagation. The processor 110 for performing such data processing is a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), an on-demand It may include an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). Since the above-described type of the processor 110 is only one example, the type of the processor 110 may be variously configured within a range that can be understood by those skilled in the art based on the contents of the present disclosure.

The processor 110 may analyze a medical image to detect a lesion of a subject to be photographed in the medical image. The processor 110 may analyze the medical image to estimate a lesion present in the medical image. In order to estimate the lesion, the processor 110 may use a neural network model including at least one neural network. In this case, the term detection or estimation used in the present disclosure may be understood as a broad concept for a vision task performed by a neural network model. That is, the term detection or estimation used in the present disclosure refers to all of the classification, localization, detection, and segmentation performed by the neural network model to identify a specific target present in the image. It can be understood as including meaning. For example, to estimate the lesion, the processor 110 may use a classification model. The classification model may be understood as a neural network model that identifies a relationship between classes included in data and determines a class included in newly input data. Specifically, the processor 110 may use a classification model based on a convolutional neural network as one of the classification models as a neural network model. In this case, the convolutional neural network can be understood as a multi-layer neural network capable of effectively recognizing and classifying features of adjacent regions while maintaining spatial information of input data. However, since the above-described classification model is only an example, a classification model and a technical concept are common in a category that can be understood by those skilled in the art, and a predictive model or a regression model capable of performing lesion detection can all be applied to the neural network model of the present disclosure. have.

The processor 110 may train the neural network model to classify the presence or absence of a lesion based on the medical image. For example, the processor 110 may input either a learning image in which a lesion exists or a learning image in which a lesion does not exist into the neural network model to train the neural network model so that the neural network model classifies whether a lesion exists in the image. can In this case, the learning image may be an image in which the presence or absence of a lesion is labeled. The processor 110 may train the neural network model to minimize an error value by comparing the classification result of the training image of the neural network model with the label of the training image.

The processor 110 may generate a label indicating the lesion region in the medical image by using the pre-trained neural network model. The processor 110 may generate a label indicating the presence or absence of a lesion in a specific region in the image by identifying which features of the image were recognized by the learned neural network model in the process of classifying the lesion. In other words, the processor 110 may generate a label that can be used for supervised learning for lesion detection by estimating which region of the image the neural network model identifies as a main feature and classifies the lesion. The processor 110 improves the speed of the manual labeling operation by using the information on which region of the image has affected the feature interpretation (or inference) of the neural network model trained to detect the lesion to generate the label. and manual labeling costs can be greatly reduced. In addition, the processor 110 may use the feature analysis (or inference) of the neural network model trained to detect the lesion to generate a label, so that the labeling task compared to the expert's manual work can be performed more precisely and accurately. In this case, the label generation of the processor 110 may be performed in parallel with lesion detection using a neural network model.

If necessary, the processor 110 may generate a user interface that provides an environment for interaction with a user of the computing device 100 or a user of any client. For example, the processor 110 may generate a user interface that visualizes a classification result derived using a neural network for lesion detection, a label, or the like. The processor 110 may generate a user interface for implementing functions such as output, correction, change, or addition of data based on an external input signal applied from a user. Since the role of the user interface described above is only one example, the role of the user interface may be defined in various ways within a range that can be understood by those skilled in the art based on the contents of the present disclosure.

The memory 120 according to an embodiment of the present disclosure may be understood as a structural unit including hardware and/or software for storing and managing data processed by the computing device 100 . That is, the memory 120 may store any type of data generated or determined by the processor 110 and any type of data received by the network unit 130 . For example, the memory 120 may include a flash memory type, a hard disk type, a multimedia card micro type, a card type memory, and a random access memory (RAM). ), SRAM (static random access memory), ROM (read-only memory), EEPROM (electrically erasable programmable read-only memory), PROM (programmable read-only memory), magnetic memory , a magnetic disk, and an optical disk may include at least one type of storage medium. In addition, the memory 120 may include a database system for controlling and managing data in a predetermined system. Since the above-described type of the memory 120 is only one example, the type of the memory 120 may be variously configured within a range that can be understood by those skilled in the art based on the contents of the present disclosure.

The memory 120 may structure and organize and manage data necessary for the processor 110 to perform an operation, a combination of data, and program codes executable by the processor 110 . For example, the memory 120 may store a medical image received through the network unit 130 to be described later. The memory 120 executes a program code that operates the neural network model to detect a lesion in a medical image, a program code that operates the processor 110 to generate a label based on analysis (or inference) of features of the neural network model, and the program code is executed. The generated data can be saved. The memory 120 may store a lesion estimation result of a medical image generated through operation of the processor 110 and a label indicating the lesion.

The network unit 130 according to an embodiment of the present disclosure may be understood as a structural unit for transmitting and receiving data through a known wired/wireless communication system of any type. For example, the network unit 130 may include a local area network (LAN), wideband code division multiple access (WCDMA), long term evolution (LTE), and WiBro (WiBro). broadband internet), 5th generation mobile communication (5G), ultra wide-band, ZigBee, radio frequency (RF) communication, wireless LAN, wireless fidelity ), near field communication (NFC), or a wired/wireless communication system such as Bluetooth may be used to transmit/receive data. Since the above-described communication systems are only examples, the wired/wireless communication system for data transmission/reception of the network unit 130 may be variously applied other than the above-described examples.

The network unit 130 may receive data necessary for the processor 110 to perform an operation through wired/wireless communication with any system or any client. In addition, the network unit 130 may transmit data generated through the operation of the processor 110 through wired/wireless communication with any system or any client. For example, the network unit 130 may receive a medical image through communication with a medical image storage and transmission system, a cloud server performing tasks such as resolution improvement or standardization of a medical image, or a computing device. The network unit 130 may transmit the lesion detection result corresponding to the output of the neural network model and the label generated through the operation of the processor 110 through communication with the aforementioned system, server, or computing device.

If necessary, the network unit 130 may provide the user interface generated by the processor 110 to any system or any client through wired/wireless communication with any system or any client. For example, the network unit 130 provides a user interface for visualizing data processed by the processor 110 through communication with a medical image storage and transmission system or a computing device including a display as described above for the system or can be provided to the device. The network unit 130 may receive an external input signal applied from the user through the user interface from the aforementioned system or device and transmit it to the processor 110 . In this case, the processor 110 may operate to implement functions such as output, correction, change, or addition of data based on the external input signal transmitted from the network unit 130 . On the other hand, without communication through the network unit 130, the system or device provided with the user interface may operate to implement functions such as output, correction, change or addition of data according to an external input signal applied from the user. have.

2 is a block diagram illustrating a label generation process of a computing device according to an embodiment of the present disclosure.

Referring to FIG. 2 , the processor 110 of the computing device 100 according to an embodiment of the present disclosure may input a medical image 10 for detecting a lesion into the neural network model 200 . In this case, the medical image 10 may be an image captured by a medical imaging apparatus for diagnosing and treating a disease. For example, the medical image 10 may include a magnetic resonance image generated through a Fourier transform based on K-space data obtained from the magnetic resonance signal. In this case, the magnetic resonance image included in the medical image 10 may be data generated through accelerated imaging.

In the present disclosure, accelerated imaging may be understood as an imaging technique for reducing imaging time by reducing the number of excitations (NEX) for a magnetic resonance signal compared to general imaging. The number of excitations may be understood as the number of repetitions when lines of the magnetic resonance signal are repeatedly acquired in the K-space region. Accordingly, as the number of excitations increases, the imaging time of the magnetic resonance image may increase proportionally. That is, when the number of excitations is reduced during the imaging of the magnetic resonance image, accelerated imaging in which the imaging time of the magnetic resonance image is shortened may be implemented.

In the present disclosure, accelerated imaging may be understood as a imaging technique for acquiring an image having a relatively low resolution by acquiring a signal of a narrower range in the phase encoding direction in the K-space region. In other words, the accelerated imaging of the present disclosure may be understood as an imaging technique in which phase resolution is reduced compared to general imaging. The phase resolution may be understood as a value obtained by dividing the number of lines sampled in the phase encoding direction in the K-space region by a preset reference value. Accordingly, as the phase resolution increases, the imaging time of the magnetic resonance image may increase proportionally. That is, when the phase resolution is reduced when capturing the magnetic resonance image, accelerated photographing with a shortened magnetic resonance image photographing time may be implemented.

In the present disclosure, accelerated imaging may be understood as an imaging technique for reducing imaging time by increasing an acceleration factor compared to general imaging. The acceleration index is a term used in a parallel imaging technique, and may be understood as a value obtained by dividing the number of signal lines sampled fully in the K-space region by the number of signal lines sampled through imaging. For example, an acceleration index of 2 may be understood as obtaining half the number of signal lines compared to the number of fully sampled signal lines when obtaining lines by sampling the magnetic resonance signal in the phase encoding direction. Accordingly, as the acceleration index increases, the imaging time of the magnetic resonance image may decrease proportionally. That is, when the acceleration index is increased during the imaging of the magnetic resonance image, the accelerated imaging in which the imaging time of the magnetic resonance image is shortened may be implemented.

In the present disclosure, accelerated imaging may be understood as an imaging technique for generating a MR image by acquiring a sub-sampled MR signal. In this case, the sub-sampling may be understood as an operation of sampling the magnetic resonance signal at a sampling rate lower than a nyquist sampling rate. Accordingly, the magnetic resonance image of the present disclosure may be an image obtained by sampling the magnetic resonance signal at a sampling rate lower than the Nyquist sampling rate.

However, since the above-described description of accelerated imaging is only an example, the concept of accelerated imaging may be defined in various ways within a range that can be understood by those skilled in the art based on the contents of the present disclosure.

Referring to FIG. 2 , the processor 110 may generate inference data 20 by estimating a lesion existing in the medical image 10 using a neural network model receiving the medical image 10 . In this case, the inference data 20 may include at least one of a result indicating whether a lesion exists in the medical image 10 or a result of detecting an area in which a lesion exists in the medical image 10 when the lesion exists. can For example, the processor 110 may extract a feature for lesion detection from the medical image 10 using the neural network model 200 . A feature extracted through the neural network model 200 may be understood as an operation value extracted based on a pixel value or a voxel value of the medical image 10 or a combination of operation values. The processor 110 may derive a classification result indicating whether a lesion exists in the medical image 10 based on the features extracted for lesion detection by using the neural network model 200 . In this case, the classification result may be a probability expression for the presence or absence of a lesion. If a lesion is present, the classification result may be derived as 1. If there is no lesion, the classification result may be derived as 0. However, since the expression of the classification result is only an example, the classification result may be expressed in various ways within a range that can be understood by those skilled in the art based on the contents of the present disclosure.

The processor 110 may generate a label indicating a lesion region in an image based on an output of a predetermined layer included in the neural network model. The processor 110 may generate a label indicating a lesion region present in a medical image to which the neural network model is input, based on a feature corresponding to an output of at least one layer included in the neural network model. In this case, the label is information indicating the presence or absence of a lesion in a medical image, and may be understood as a ground truth generated based on inference of a neural network model. As such, the label generated based on the feature analysis of the neural network model does not require manual work by a specialist, and has no choice but to include more precise and accurate information than the label generated by the subjective judgment of a person. Therefore, the label generating method of the present disclosure has an efficient advantage in terms of time and cost compared to the conventional method in which labeling was performed manually.

For example, the processor 110 may generate a heatmap corresponding to the feature analysis of the neural network model for the medical image, based on the output of a predetermined layer included in the neural network model. The processor 110 may generate a label indicating the presence or absence of a lesion in a specific region in the image based on the heat map. At this time, since there is a possibility that an error exists in the heat map itself generated based on the output of a predetermined layer included in the neural network model, the processor 110 may perform additional image processing in the process of generating the label from the heat map. have. Specifically, the processor 110 may correct a blank area existing in the lesion area expressed in the heat map. On the assumption that the lesion region is a continuous region without holes, the processor 110 performs correction to fill in the blank region even though the lesion region is expressed in the heat map based on the expression value of the region adjacent to the blank region. can Also, the processor 110 may correct noise existing in the heat map. In order to correct an error that is not a lesion region but is displayed as a lesion region due to an error in the heat map, the processor 110 may remove values corresponding to noise by selecting the largest connection element among the lesion regions displayed on the heat map. Since the above-mentioned two corrections are only examples that can be applied to the present disclosure, various correction methods may be applied to the present disclosure in a range that can be understood by those skilled in the art based on the contents described in the present disclosure in addition to the above-described two corrections.

Meanwhile, the processor 110 may input a scan parameter that can be adjusted when the medical image 10 is captured together with the medical image 10 to the neural network model 200 . In this case, the scan parameter may be understood as a variable adjusted in the imaging environment of the medical image to determine the imaging state or condition of the medical image. A signal obtained when a medical image is captured may vary according to a value of a scan parameter, and characteristics of a medical image generated according to the signal may vary. For example, when the medical image 10 is a magnetic resonance image, the scan parameters include repetition time (TR), echo time (TE), inversion time (TI), and field of view (FOV). : field of view), the number of excitations, or an acceleration index. The processor 110 may further improve the inference performance of the neural network model 200 by including the scan parameters as in the above-described example together with the medical image 10 in the input of the neural network model 200 .

3 is a conceptual diagram illustrating a calculation process using a neural network model according to an embodiment of the present disclosure. The description of FIG. 3 assumes a process of classifying a lesion in a magnetic resonance image and generating a label using a neural network model including a convolutional neural network.

Referring to FIG. 3 , the processor 110 of the computing device 100 according to an embodiment of the present disclosure may input an accelerated magnetic resonance image 40 to the first neural network 210 . The processor 110 may extract a feature for estimating a lesion region of the magnetic resonance image 40 using the first neural network 210 including the feature extraction layer. In this case, the first neural network 210 may include at least one convolutional layer and at least one pooling layer. For example, the first neural network 210 may include a filter, which is a common parameter for finding features of an image, a stride indicating an interval at which the filter traverses an image, and/or a method for maintaining the size of the feature. A feature may be extracted from the magnetic resonance image 40 by adjusting padding. The convolution layer included in the first neural network 210 may output features of the MR image 40 while maintaining spatial information of the MR image 40 by performing a convolution operation using a filter. The pooling layer included in the first neural network 210 may receive the output of the convolutional layer and reduce the size of the output of the convolutional layer or play a role of emphasizing specific information. That is, the first neural network 210 may extract features of the MR image 40 through a combination of at least one convolutional layer and at least one pooling layer.

Meanwhile, in the feature extraction process of the first neural network 210 , the processor 110 performs the second function through Fourier transform based on the first feature calculated from the i-th layer (i is a natural number) of the first neural network 210 . You can create features. In addition, the processor 110 may input the first feature calculated from the i-th layer and the second feature generated through the Fourier transform to the i+1-th layer of the first neural network 210 . In this case, the first feature based on the magnetic resonance image 40 corresponds to an image feature of a general image region. Accordingly, it may be understood that the second feature generated through the Fourier transform of the first feature corresponds to the image feature of the k-space domain. For example, the processor 110 may generate a second feature of the K-space domain through a Fourier transform based on a first feature corresponding to an output of any one of the convolutional layers of the first neural network 210 . have. The processor 110 may input the first feature and the second feature together as a next layer of the convolutional layer from which the first feature is extracted. As described above, the processor 110 improves the inference performance of the first neural network 210 by using the characteristics of the K-space region generated through the Fourier transform in the calculation of the layers included in the first neural network 210 together. can be further improved.

The processor 110 may input the feature map generated through the first neural network 210 to the second neural network 220 . The processor 110 may generate a vector indicating the class of the lesion based on the characteristics of the magnetic resonance image 40 using the second neural network 220 . Then, the processor 110 uses the third neural network 230 to derive a probability value corresponding to the lesion class corresponding to the lesion detection result 50 based on the vector generated through the second neural network 200 . can In this case, the second neural network 220 may include a fully connected layer (FCL). Also, the third neural network 230 may include a softmax layer. For example, the second neural network 220 may generate a vector having a size equal to the number of lesion classes to be classified in the magnetic resonance image 40 based on the characteristics of the magnetic resonance image 40 . The third neural network 230 may receive the output vector of the second neural network 220 and output a probability value indicating the presence or absence of a lesion. The probability value output through the third neural network 230 may be a lesion detection result 50 indicating the presence or absence of a lesion for each lesion class. When a lesion exists, a probability value corresponding to the lesion detection result 50 may be derived as 1. When a lesion does not exist, a probability value corresponding to the lesion detection result 50 may be derived as 0. However, since the expression of such a probability value is only one example, the probability value may be variously expressed in a range that can be understood by those skilled in the art based on the contents of the present disclosure.

Referring to FIG. 3 , the processor 110 may generate a heat map 60 for generating a lesion label based on an output of any one of the layers included in the first neural network 10 . The processor 110 may extract a gradient indicating a change in a characteristic of the MR image 40 based on an output of any one of the layers included in the first neural network 10 . The processor 110 may generate a label indicating the presence or absence of a lesion in a specific region in the magnetic resonance image 40 based on the extracted gradient. For example, the processor 110 may extract a gradient representing a feature change of the MR image 40 from a feature corresponding to the output of the last layer among the convolutional layers included in the first neural network 10 . The processor 110 may generate a heat map capable of visualizing which region of the magnetic resonance image 40 has affected the lesion classification based on the gradient generated from the output of the last layer of the convolutional layer. In addition, the processor 110 may perform labeling on the lesion 65 existing in the magnetic resonance image 40 using the heat map.

On the other hand, the neural network model 200 including the first neural network 210, the second neural network 220, and the third neural network 230 of the present disclosure is a first learning image in which a lesion exists, or a first learning image in which a lesion does not exist. It may be learned to classify the presence or absence of a lesion based on at least one of the 2 learning images. For example, the neural network model 200 including the convolutional neural network may be trained to classify the presence or absence of a lesion based on a learning image in which the first learning image and the second learning image are combined. The neural network model 200 may be trained to binary classify the presence or absence of a lesion. When the feature analysis of the medical image of the neural network model 200 trained to binary classify the presence or absence of a lesion is used to generate the label of the present disclosure, compared to the learning to multi-classify or detect the lesion, the learning of the neural network model 200 is cost can be reduced. Therefore, when labeling is required for a lesion having a large size, in the present disclosure, the neural network model 200 trained to binary classify the presence or absence of a lesion may be used for label generation.

4 is a flowchart illustrating a label generation method according to an embodiment of the present disclosure.

The computing device 100 according to an embodiment of the present disclosure may receive a medical image through communication with a medical image storage and transmission system, a server, or a client. In this case, the computing device 100 is a cloud system and may be a computing environment capable of storing, managing, and transmitting/receiving data on a communication network. For example, the computing device 100 may receive, store, and manage the magnetic resonance image through communication with the magnetic resonance imaging system. The computing device 100 may detect a lesion present in the magnetic resonance image by analyzing the magnetic resonance image received through communication with the magnetic resonance imaging system using the neural network model. In addition, the computing device 100 may label the lesion present in the magnetic resonance image by using information about which feature the neural network model uses to detect the lesion in the magnetic resonance image.

Referring to FIG. 4 , in step S100 , the computing device 100 may detect a lesion present in a medical image using a pre-trained neural network model. In other words, the computing device 100 may input a medical image to the neural network model and classify whether a lesion exists in the medical image. In this case, the neural network model may include a neural network for extracting features of a medical image and a neural network for classifying lesions based on features of the medical image. For example, the neural network model may include, but is not limited to, a convolutional neural network.

In operation S200 , the computing device 100 may generate a label indicating a lesion region in a medical image based on an output of a predetermined layer included in the neural network model. In other words, the computing device 100 may generate a label indicating the presence or absence of a lesion in the medical image based on an output derived from a specific layer while the neural network model extracts features of the medical image. In this case, the predetermined layer may be a layer in a predetermined order according to the structure of the neural network model. For example, in the case of a neural network model including a convolutional neural network, the predetermined layer may be the last convolutional layer.

On the other hand, since steps S100 and S200 are only codes given to distinguish the steps from each other, steps S100 and S200 may be performed sequentially or in parallel.

Various embodiments of the present disclosure described above may be combined with additional embodiments and may be modified within the scope understood by those skilled in the art in light of the above detailed description. It should be understood that the embodiments of the present disclosure are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and likewise components described as distributed may be implemented in a combined form. Accordingly, all changes or modifications derived from the meaning, scope, and equivalent concept of the claims of the present disclosure should be construed as being included in the scope of the present disclosure.

Claims (11)

A method for generating a label, performed by a computing device comprising at least one processor, the method comprising:
detecting a lesion present in a medical image using a neural network model;
A heatmap corresponding to extracting a gradient from the output of any one of the feature extraction layers included in the neural network model, and analyzing the features of the neural network model for the medical image based on the gradient creating a; and
generating a label indicating the presence or absence of a lesion in a specific region in the medical image based on the heat map;
containing,
Way.
The method of claim 1,
The medical image is
Based on the k-space data obtained from the magnetic resonance signal, the magnetic resonance image generated through the Fourier transform (Fourier transform),
Way.
The method of claim 1,
The neural network model is
receiving the medical image and a scan parameter that can be adjusted when the medical image is taken,
Way.
The method of claim 1,
The step of detecting the lesion present in the medical image,
generating a second feature through a Fourier transform based on a first feature calculated from an i-th (i is a natural number) layer of the neural network model; and
inputting the first feature calculated from the i-th layer and the second feature generated through the Fourier transform into the i+1-th layer of the neural network model;
containing,
Way.
delete delete The method of claim 1,
Generating a label indicating the presence or absence of a lesion in a specific region in the medical image based on the heat map comprises:
correcting a blank area existing in the lesion area expressed in the heat map;
containing,
Way.
The method of claim 1,
Generating a label indicating the presence or absence of a lesion in a specific region in the medical image based on the heat map comprises:
correcting noise present in the heat map;
containing,
Way.
The method of claim 1,
The neural network model is
Based on at least one of the first learning image in which the lesion exists and the second learning image in which the lesion does not exist, it is learned to classify the presence or absence of the lesion,
Way.
A computer program stored in a computer readable storage medium, wherein the computer program, when executed on one or more processors, performs operations for generating a label,
The actions are
detecting a lesion present in a medical image using a neural network model;
A heatmap corresponding to extracting a gradient from the output of any one of the feature extraction layers included in the neural network model, and analyzing the features of the neural network model for the medical image based on the gradient creating an action; and
generating a label indicating the presence or absence of a lesion in a specific region in the medical image based on the heat map;
containing,
computer program.
A computing device for generating a label, comprising:
a processor including at least one core;
a memory including program codes executable by the processor; and
a network unit for receiving a medical image;
including,
The processor is
Using a neural network model to detect lesions present in medical images,
A heatmap corresponding to extracting a gradient from the output of any one of the feature extraction layers included in the neural network model, and analyzing the features of the neural network model for the medical image based on the gradient creates,
generating a label indicating the presence or absence of a lesion in a specific region in the medical image based on the heat map,
Device.

Images