KR20230114170A - Method, program, and apparatus for generating label - Google Patents

Abstract

A label generation method performed by a computing device according to an embodiment of the present disclosure is disclosed. The method may include detecting a lesion present in a medical image using a neural network model; and generating a label indicating a lesion area in the medical image based on an output of a predetermined layer included in the neural network model.

Description

Label generation method, program and device {METHOD, PROGRAM, AND APPARATUS FOR GENERATING LABEL}

The present disclosure relates to a method for generating a label, and more specifically, to an artificial intelligence-based technology for generating a label capable of displaying a lesion in a medical image.

Medical images are data that enable us to understand the physical state of various organs of the human body for diagnosis and treatment of diseases. As medical imaging devices have been continuously developed, medical images include X-ray images, computed tomography (CT) images, positron emission tomography (PET) images, or magnetic resonance images. (MRI: magnetic resonance imaging). With the development of image analysis technology and artificial intelligence technology, medical images are used in various ways as analysis data to assist in the diagnosis of diseases. Typically, medical imaging is actively used in the field of lesion detection for detecting lesions photographed on images for effective diagnosis of diseases.

Until now, artificial intelligence technologies in the field of computer vision, which have been widely used in the field of lesion detection, are traditionally based on supervised learning. Since labels are essential for supervised learning, existing technologies use the labeled data for learning after experts directly identify lesions in medical images. However, since labels can only be created manually by experts, it is practically difficult to secure enough labels to guarantee the learning performance of artificial intelligence. That is, for supervised learning, the overall cost of generating a label is inevitably required.

Republic of Korea Patent Publication No. 10-2013-0012297 (2013.02.04.)

The present disclosure has been made in response to the above background art, and an object of the present disclosure is to provide a method for efficiently generating a label based on artificial intelligence inference for lesion detection.

However, the problems to be solved in the present disclosure are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood based on the description below.

A label generation method performed by a computing device according to an embodiment of the present disclosure for realizing the above object is disclosed. The method may include detecting a lesion present in a medical image using a neural network model; and generating a label indicating a lesion area in the medical image based on an output of a predetermined layer included in the neural network model.

Alternatively, the medical image may be a magnetic resonance image generated through Fourier transform based on k-space data obtained from a magnetic resonance signal.

Alternatively, the neural network model may receive the medical image and a scan parameter that can be adjusted when capturing the medical image.

Alternatively, the step of detecting the lesion present in the medical image may include obtaining a second feature through Fourier transformation based on a first feature calculated from the ith (i is a natural number) layer of the neural network model. generating; and inputting a first feature calculated in the i-th layer and a second feature generated through the Fourier transform into an i+1-th layer of the neural network model.

Alternatively, generating a label indicating a lesion area in the medical image may include generating a heatmap corresponding to feature analysis of the neural network model for the medical image based on an output of the predetermined layer. doing; and generating a label indicating whether or not there is a lesion in a specific region in the medical image based on the heat map.

Alternatively, the generating of the heat map corresponding to the feature analysis of the neural network model for the medical image may include extracting a gradient from an output of any one of feature extraction layers included in the neural network model; Generating the heat map based on the gradient may be included.

Alternatively, generating a label indicating the presence or absence of a lesion in a specific region in the medical image based on the heat map may include correcting a blank region existing in the lesion region represented in the heat map. .

Alternatively, generating a label representing the presence or absence of a lesion in a specific region in the medical image based on the heat map may include correcting noise present in the heat map.

Alternatively, the neural network model may be trained to classify the presence or absence of a lesion based on at least one of the first training image in which a lesion is present or the second training image in which a lesion is not present.

A computer program stored in a computer readable storage medium is disclosed according to an embodiment of the present disclosure for realizing the above object. When the computer program is executed on one or more processors, operations for generating labels are performed. In this case, the operations may include an operation of detecting a lesion present in a medical image using a neural network model; and generating a label indicating a lesion area in the medical image based on an output of a predetermined layer included in the neural network model.

A computing device for generating a label is disclosed according to an embodiment of the present disclosure for realizing the above object. The apparatus includes a processor including at least one core; a memory containing program codes executable by the processor; and a network unit for receiving medical images. In this case, the processor may detect a lesion present in the medical image using a neural network model, and generate a label indicating a lesion area in the medical image based on an output of a predetermined layer included in the neural network model. there is.

The present disclosure may provide a method for efficiently generating a label based on feature interpretation of a medical image by artificial intelligence for lesion detection.

1 is a block diagram of a computing device according to an embodiment of the present disclosure.
2 is a block diagram illustrating a label creation process of a computing device according to an embodiment of the present disclosure.
3 is a conceptual diagram illustrating a calculation process using a neural network model according to an embodiment of the present disclosure.
4 is a flowchart illustrating a label generation method according to an embodiment of the present disclosure.

Hereinafter, embodiments of the present disclosure will be described in detail so that those skilled in the art (hereinafter, those skilled in the art) can easily practice with reference to the accompanying drawings. The embodiments presented in this disclosure are provided so that those skilled in the art can use or practice the contents of this disclosure. Accordingly, various modifications to the embodiments of the present disclosure will be apparent to those skilled in the art. That is, the present disclosure may be implemented in many different forms, and is not limited to the following embodiments.

Same or similar reference numerals designate the same or similar elements throughout the specification of this disclosure. In addition, in order to clearly describe the present disclosure, reference numerals of parts not related to the description of the present disclosure may be omitted in the drawings.

The term “or” as used in this disclosure is intended to mean an inclusive “or” rather than an exclusive “or”. That is, unless otherwise specified in this disclosure or the meaning is not clear from the context, "X employs A or B" should be understood to mean one of the natural inclusive substitutions. For example, unless otherwise specified in this disclosure or where the meaning is not clear from the context, “X employs A or B” means that X employs A, X employs B, or X employs A and It can be interpreted as any one of the cases of using all of B.

The term “and/or” as used in this disclosure should be understood to refer to and include all possible combinations of one or more of the listed related concepts.

The terms "comprises" and/or "comprising" as used in this disclosure should be understood to mean that certain features and/or elements are present. However, it should be understood that the terms "comprising" and/or "comprising" do not exclude the presence or addition of one or more other features, other elements, and/or combinations thereof.

Unless otherwise specified in this disclosure, or where the context clearly indicates that a singular form is indicated, the singular shall generally be construed as possibly including “one or more”.

The term "Nth (N is a natural number)" used in the present disclosure can be understood as an expression used to distinguish the components of the present disclosure from each other according to a predetermined criterion such as a functional point of view, a structural point of view, or explanatory convenience. there is. For example, components performing different functional roles in the present disclosure may be classified as first components or second components. However, components that are substantially the same within the technical spirit of the present disclosure but should be distinguished for convenience of description may also be classified as first components or second components.

On the other hand, the term "module" or "unit" used in this disclosure refers to a computer-related entity, firmware, software or part thereof, hardware or part thereof , It can be understood as a term referring to an independent functional unit that processes computing resources, such as a combination of software and hardware. In this case, a “module” or “unit” may be a unit composed of a single element or a unit expressed as a combination or set of a plurality of elements. For example, as a narrow concept, a "module" or "unit" is a hardware element or set thereof of a computing device, an application program that performs a specific function of software, a process implemented through software execution, or a program. It may refer to a set of instructions for execution. Also, as a concept in a broad sense, a “module” or “unit” may refer to a computing device constituting a system or an application executed in the computing device. However, since the above concept is only an example, the concept of “module” or “unit” may be defined in various ways within a range understandable by those skilled in the art based on the contents of the present disclosure.

The term "model" used in this disclosure refers to a system implemented using mathematical concepts and language to solve a specific problem, a set of software units to solve a specific problem, or a process to solve a specific problem. It can be understood as an abstract model for a process. For example, a neural network “model” may refer to an overall system implemented as a neural network having problem-solving capabilities through learning. At this time, the neural network may have problem solving ability by optimizing parameters connecting nodes or neurons through learning. A neural network "model" may include a single neural network or may include a neural network set in which a plurality of neural networks are combined.

The term "image" used in this disclosure may refer to multidimensional data composed of discrete image elements. In other words, "image" can be understood as a term referring to a digital representation of an object that is visible to the human eye. For example, “image” may refer to multidimensional data composed of elements corresponding to pixels in a 2D image. “Image” may refer to multidimensional data composed of elements corresponding to voxels in a 3D image.

As used in this disclosure, the term "picture archiving and communication system (PACS)" is used to store, process, and store medical images in accordance with digital imaging and communications in medicine (DICOM) standards. It can refer to the system that transmits. For example, the "medical image storage and transmission system" is interlocked with digital medical imaging equipment to transmit medical images such as magnetic resonance imaging (MRI) and computed tomography (CT) images to medical digital images and It can be stored according to the communication standard. The "medical image storage and transmission system" can transmit medical images to terminals inside and outside the hospital through a communication network. In this case, meta information such as reading results and medical records may be added to the medical image.

The term "k-space" used in the present disclosure may be understood as an array of numbers representing spatial frequencies of a magnetic resonance image. In other words, "K-space" can be understood as a frequency space corresponding to a three-dimensional space corresponding to magnetic resonance space coordinates.

The term "Fourier transform" used in the present disclosure may be understood as an operation medium capable of explaining a correlation between a time domain and a frequency domain. In other words, "Fourier transform" used in the present disclosure can be understood as a broad concept representing an operation process for mutual transformation between time domain and frequency domain. Therefore, "Fourier transform" used in the present disclosure is a concept encompassing both a narrow Fourier transform that decomposes a signal in the time domain into a frequency domain and an inverse Fourier transform that transforms a signal in the frequency domain into a time domain. can be understood as

Explanations of the foregoing terms are intended to facilitate understanding of the present disclosure. Therefore, it should be noted that, when the above terms are not explicitly described as matters limiting the content of the present disclosure, the content of the present disclosure is not used in the sense of limiting the technical idea.

1 is a block diagram of a computing device according to an embodiment of the present disclosure.

The computing device 100 according to an embodiment of the present disclosure may be a hardware device or part of a hardware device that performs comprehensive processing and calculation of data, or may be a software-based computing environment connected through a communication network. For example, the computing device 100 may be a server that performs intensive data processing functions and shares resources, or may be a client that shares resources through interaction with the server. Also, the computing device 100 may be a cloud system in which a plurality of servers and clients interact to comprehensively process data. Since the above description is only one example related to the type of the computing device 100, the type of the computing device 100 may be configured in various ways within a range understandable by those skilled in the art based on the contents of the present disclosure.

Referring to FIG. 1 , a computing device 100 according to an embodiment of the present disclosure may include a processor 110, a memory 120, and a network unit 130. there is. However, since FIG. 1 is only an example, the computing device 100 may include other configurations for implementing a computing environment. Also, only some of the components disclosed above may be included in the computing device 100 .

The processor 110 according to an embodiment of the present disclosure may be understood as a structural unit including hardware and/or software for performing computing operations. For example, the processor 110 may read a computer program and perform data processing for machine learning. The processor 110 may process input data processing for machine learning, feature extraction for machine learning, calculation of an error based on backpropagation, and the like. The processor 110 for performing such data processing includes a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), and on-demand It may include a semiconductor (application specific integrated circuit (ASIC)) or a field programmable gate array (FPGA). Since the above-described type of processor 110 is just one example, the type of processor 110 may be variously configured within a range understandable by those skilled in the art based on the content of the present disclosure.

The processor 110 may analyze the medical image to detect a lesion of a subject to be captured of the medical image. The processor 110 may estimate a lesion present in the medical image by analyzing the medical image. To estimate the lesion, the processor 110 may use a neural network model including at least one neural network. At this time, the term detection or estimation used in the present disclosure may be understood as a broad concept of a vision task performed by a neural network model. That is, the term detection or estimation used in the present disclosure refers to all classification, localization, detection, and segmentation performed by a neural network model to identify a specific target present in an image. It can be understood in the meaning of including. For example, to estimate a lesion, processor 110 may use a classification model. A classification model may be understood as a neural network model that identifies a relationship between classes included in data and determines a class included in newly input data. Specifically, the processor 110 may use a classification model based on a convolutional neural network as one of the classification models as a neural network model. In this case, the convolutional neural network can be understood as a multilayer neural network capable of effectively recognizing and classifying features of adjacent regions while maintaining spatial information of input data. However, since the above-described classification model is only an example, a classification model and a technical idea common to those skilled in the art can be applied to the neural network model of the present disclosure, such as a prediction model or a regression model capable of detecting lesions. there is.

The processor 110 may train the neural network model to classify the presence or absence of a lesion based on the medical image. For example, the processor 110 inputs either a training image with a lesion or a training image without a lesion to a neural network model to train the neural network model to classify whether or not a lesion exists in the image. can In this case, the learning image may be an image in which the presence or absence of a lesion is labeled. The processor 110 may train the neural network model so that an error value is minimized by comparing a classification result of the training image of the neural network model with a label of the training image.

The processor 110 may generate a label representing a lesion area in a medical image using a pretrained neural network model. The processor 110 may determine which feature of the image is recognized by the trained neural network model in the process of classifying the lesion, and generate a label indicating the presence or absence of a lesion in a specific region in the image. In other words, the processor 110 may generate a label that can be used for supervised learning for detecting a lesion by estimating which region of the image is identified as a main feature by the neural network model and classifies the lesion. The processor 110 improves the speed of a manually performed labeling task by using information about which region of the image has an effect on the feature interpretation (or inference) of the neural network model learned to detect the lesion to generate a label. and can significantly reduce manual labeling costs. In addition, the processor 110 may perform a labeling task more precisely and accurately compared to the manual work of an expert by using feature analysis (or inference) of a neural network model learned to detect a lesion to generate a label. At this time, label generation by the processor 110 may be performed in parallel with lesion detection using a neural network model.

If necessary, the processor 110 may create a user interface that provides an environment for interaction with a user of the computing device 100 or a user of an arbitrary client. For example, the processor 110 may create a user interface for visualizing a classification result derived using a neural network for detecting a lesion or a label. The processor 110 may generate a user interface that implements functions such as outputting, modifying, changing, or adding data based on an external input signal applied from a user. Since the above-described role of the user interface is only one example, the role of the user interface may be defined in various ways within a range understandable by those skilled in the art based on the contents of the present disclosure.

The memory 120 according to an embodiment of the present disclosure may be understood as a unit including hardware and/or software for storing and managing data processed by the computing device 100 . That is, the memory 120 may store any type of data generated or determined by the processor 110 and any type of data received by the network unit 130 . For example, the memory 120 may include a flash memory type, a hard disk type, a multimedia card micro type, a card type memory, and random access memory (RAM). ), SRAM (static random access memory), ROM (read-only memory), EEPROM (electrically erasable programmable read-only memory), PROM (programmable read-only memory), magnetic memory , a magnetic disk, and an optical disk may include at least one type of storage medium. In addition, the memory 120 may include a database system that controls and manages data in a predetermined system. Since the above-described type of memory 120 is just one example, the type of memory 120 may be configured in various ways within a range understandable by those skilled in the art based on the contents of the present disclosure.

The memory 120 may organize and manage data necessary for the processor 110 to perform calculations, data combinations, program codes executable by the processor 110, and the like. For example, the memory 120 may store medical images received through the network unit 130 to be described later. The memory 120 executes program codes that operate the neural network model to detect lesions in medical images, program codes that operate the processor 110 to generate labels based on feature analysis (or inference) of the neural network model, and program codes. As a result, the generated data can be stored. The memory 120 may store a lesion estimation result of the medical image generated through the operation of the processor 110 and a label indicating the lesion.

The network unit 130 according to an embodiment of the present disclosure may be understood as a unit that transmits and receives data through any type of known wired/wireless communication system. For example, the network unit 130 may include a local area network (LAN), wideband code division multiple access (WCDMA), long term evolution (LTE), and WiBro (wireless). broadband internet), 5th generation mobile communication (5G), ultra wide-band wireless communication, ZigBee, radio frequency (RF) communication, wireless LAN, wireless fidelity ), near field communication (NFC), or data transmission/reception may be performed using a wired/wireless communication system such as Bluetooth. Since the above-described communication systems are only examples, a wired/wireless communication system for data transmission and reception of the network unit 130 may be applied in various ways other than the above-described examples.

The network unit 130 may receive data necessary for the processor 110 to perform an operation through wired/wireless communication with an arbitrary system or an arbitrary client. In addition, the network unit 130 may transmit data generated through the operation of the processor 110 through wired/wireless communication with an arbitrary system or an arbitrary client. For example, the network unit 130 may receive medical images through communication with a medical image storage and transmission system, a cloud server that performs tasks such as resolution improvement or standardization of medical images, or a computing device. The network unit 130 may transmit the lesion detection result corresponding to the output of the neural network model and the label generated through the operation of the processor 110 through communication with the aforementioned system, server, or computing device.

As needed, the network unit 130 may provide the user interface generated by the processor 110 to any system or client through wired/wireless communication with any system or client. For example, the network unit 130 provides a user interface for visualizing data processed by the processor 110 through communication with a medical image storage and transmission system or a computing device including a display. device can be provided. The network unit 130 may receive an external input signal applied from a user through a user interface from the above-described system or device and transmit it to the processor 110 . At this time, the processor 110 may operate to implement functions such as outputting, modifying, changing, or adding data based on the external input signal transmitted from the network unit 130 . On the other hand, even without communication through the network unit 130, a system or device provided with a user interface may itself operate to implement functions such as output, modification, change, or addition of data according to an external input signal applied from a user. there is.

2 is a block diagram illustrating a label creation process of a computing device according to an embodiment of the present disclosure.

Referring to FIG. 2 , the processor 110 of the computing device 100 according to an embodiment of the present disclosure may input a medical image 10 to detect a lesion into the neural network model 200 . In this case, the medical image 10 may be an image captured by a medical imaging device for diagnosis and treatment of a disease. For example, the medical image 10 may include a magnetic resonance image generated through a Fourier transform based on K-space data obtained from a magnetic resonance signal. In this case, the magnetic resonance image included in the medical image 10 may be data generated through accelerated imaging.

In the present disclosure, accelerated imaging may be understood as a imaging technique that shortens imaging time by reducing the number of excitations (NEX) for magnetic resonance signals compared to general imaging. The number of times of excitation may be understood as the number of repetitions when lines of a magnetic resonance signal are repeatedly obtained in the K-space region. Accordingly, as the number of excitations increases, the imaging time of the magnetic resonance image may increase proportionally. That is, when the number of excitations is reduced when capturing an MR image, accelerated imaging with a shortened MR image capturing time may be realized.

In the present disclosure, accelerated photographing may be understood as a photographic technique for obtaining an image having a relatively low resolution by obtaining a signal of a narrower range in a phase encoding direction in the K-space domain. In other words, the accelerated imaging of the present disclosure may be understood as a imaging technique in which phase resolution is reduced compared to general imaging. The phase resolution may be understood as a value obtained by dividing the number of lines sampled in the phase encoding direction in the K-space domain by a preset reference value. Accordingly, as the phase resolution increases, the imaging time of the magnetic resonance image may increase proportionally. That is, when the phase resolution is reduced when capturing the magnetic resonance image, accelerated imaging in which the capturing time of the magnetic resonance image is shortened may be implemented.

In the present disclosure, accelerated imaging may be understood as a imaging technique that shortens imaging time by increasing an acceleration factor compared to general imaging. An acceleration index is a term used in a parallel imaging technique, and can be understood as a value obtained by dividing the number of signal lines sampled in the K-space domain by the number of signal lines sampled through imaging. For example, an acceleration index of 2 may be understood as obtaining half the number of signal lines compared to the number of full-sampled signal lines when lines are obtained by sampling the magnetic resonance signal in the phase encoding direction. Accordingly, as the acceleration index increases, the imaging time of the magnetic resonance image may decrease proportionally. That is, when an acceleration index is increased when capturing an MR image, accelerated imaging in which a capturing time of an MR image is shortened may be implemented.

In the present disclosure, accelerated imaging may be understood as an imaging technique for generating a magnetic resonance image by acquiring a sub-sampled magnetic resonance signal. In this case, subsampling may be understood as an operation of sampling a magnetic resonance signal at a sampling rate lower than a Nyquist sampling rate. Accordingly, the magnetic resonance image of the present disclosure may be an image obtained by sampling the magnetic resonance signal at a sampling rate lower than the Nyquist sampling rate.

However, since the description of accelerated imaging described above is only an example, the concept of accelerated imaging may be defined in various ways within a range understandable by those skilled in the art based on the contents of the present disclosure.

Referring to FIG. 2 , the processor 110 may generate inference data 20 by estimating a lesion present in the medical image 10 using a neural network model receiving the medical image 10 . In this case, the reasoning data 20 may include at least one of a result indicating whether a lesion exists in the medical image 10 or a result of detecting a region in which a lesion exists when a lesion exists in the medical image 10 . can For example, the processor 110 may extract features for lesion detection from the medical image 10 using the neural network model 200 . A feature extracted through the neural network model 200 may be understood as an operation value extracted based on a pixel value or voxel value of the medical image 10 or a combination of operation values. The processor 110 may use the neural network model 200 to derive a classification result indicating whether a lesion exists in the medical image 10 based on features extracted for lesion detection. In this case, the classification result may be a probability expression for the presence or absence of a lesion. If a lesion is present, the classification result can be drawn as 1. If no lesion is present, the classification result may be zero. However, since the expression of the classification result is only one example, the classification result may be expressed in various ways within a range understandable by those skilled in the art based on the contents of the present disclosure.

The processor 110 may generate a label indicating a lesion area in an image based on an output of a predetermined layer included in the neural network model. The processor 110 may generate a label indicating a lesion area present in a medical image received by the neural network model, based on a feature corresponding to an output of at least one layer included in the neural network model. In this case, the label is information indicating the presence or absence of a lesion in the medical image, and may be understood as ground truth generated based on inference of a neural network model. As such, a label generated based on the feature analysis of a neural network model does not require a manual operation by a specialist, and inevitably contains more precise and accurate information than a label generated by subjective judgment of a person. Therefore, the label generation method of the present disclosure has an advantage of being efficient in terms of time and cost compared to the conventional method in which labeling is performed manually.

For example, the processor 110 may generate a heatmap corresponding to feature analysis of a neural network model for a medical image, based on an output of a predetermined layer included in the neural network model. The processor 110 may generate a label indicating whether or not there is a lesion in a specific region in the image based on the heat map. At this time, since an error may exist in the heat map itself generated based on the output of a predetermined layer included in the neural network model, the processor 110 may perform an additional image processing operation in the process of generating a label from the heat map. there is. Specifically, the processor 110 may correct a blank area present in the lesion area represented on the heat map. Under the assumption that the lesion area is a continuous area without holes, the processor 110 performs correction to fill in the area expressed as blank even though the lesion area is expressed on the heat map based on the expression value of the area adjacent to the blank area. can Also, the processor 110 may correct noise present in the heat map. In order to correct an error that is not a lesion area but is marked as a lesion area due to an error of the heat map, the processor 110 may remove values corresponding to noise by selecting the largest connection element among the lesion areas displayed on the heat map. Since the above two corrections are just one example that can be applied to the present disclosure, various correction methods may be applied to the present disclosure within a scope understandable by those skilled in the art based on the contents described in the present disclosure, in addition to the above two corrections.

Meanwhile, the processor 110 may input, along with the medical image 10 , scan parameters that can be adjusted when capturing the medical image 10 to the neural network model 200 . In this case, the scan parameter may be understood as a variable adjusted in a medical image capturing environment in order to determine a medical image capturing state or condition. Depending on the value of the scan parameter, a signal obtained when capturing a medical image may vary, and characteristics of a generated medical image may vary according to the signal. For example, when the medical image 10 is a magnetic resonance image, scan parameters include repetition time (TR), echo time (TE), inversion time (TI), and field of view (FOV). : field of view), excitation number, or acceleration index. The processor 110 may further improve inference performance of the neural network model 200 by including the medical image 10 and scan parameters as in the above-described example as an input of the neural network model 200 .

3 is a conceptual diagram illustrating a calculation process using a neural network model according to an embodiment of the present disclosure. The description of FIG. 3 assumes a process of classifying lesions in an MRI and generating labels using a neural network model including a convolutional neural network.

Referring to FIG. 3 , the processor 110 of the computing device 100 according to an embodiment of the present disclosure may input the accelerated magnetic resonance image 40 to the first neural network 210 . The processor 110 may extract features for estimating the lesion area of the magnetic resonance image 40 using the first neural network 210 including a feature extraction layer. In this case, the first neural network 210 may include at least one convolutional layer and at least one pooling layer. For example, the first neural network 210 uses a filter, which is a common parameter for finding features of an image, a stride indicating an interval at which the filter traverses the image, and/or a feature size for maintaining Features may be extracted from the magnetic resonance image 40 by adjusting padding. The convolution layer included in the first neural network 210 may output features of the MRI 40 while maintaining spatial information of the MRI 40 by performing a convolution operation using a filter. The pooling layer included in the first neural network 210 may serve to reduce the size of the output of the convolutional layer or emphasize specific information by receiving the output of the convolutional layer. That is, the first neural network 210 may extract features of the magnetic resonance image 40 through a combination of at least one convolutional layer and at least one pooling layer.

Meanwhile, in the feature extraction process of the first neural network 210, the processor 110, based on the first feature calculated in the i-th (i is a natural number) layer of the first neural network 210, through Fourier transform, the processor 110 generates a second features can be created. Then, the processor 110 may input the first feature calculated in the i-th layer and the second feature generated through Fourier transform to the i+1th layer of the first neural network 210 . In this case, the first feature based on the magnetic resonance image 40 corresponds to an image feature of a general image area. Accordingly, the second feature generated through the Fourier transform of the first feature may be understood to correspond to an image feature in the K-space region. For example, the processor 110 may generate a second feature of the K-space domain through a Fourier transform based on a first feature corresponding to an output of any one of the convolutional layers of the first neural network 210. there is. The processor 110 may input the first feature and the second feature together to a layer following the convolutional layer from which the first feature is extracted. As described above, the processor 110 improves inference performance of the first neural network 210 by utilizing the characteristics of the K-space domain generated through Fourier transform in the operation of the layer included in the first neural network 210. can be further improved.

The processor 110 may input the feature map generated through the first neural network 210 to the second neural network 220 . The processor 110 may use the second neural network 220 to generate a vector representing a class of a lesion based on the characteristics of the magnetic resonance image 40 . Then, the processor 110 uses the third neural network 230 to derive a probability value corresponding to the lesion class corresponding to the lesion detection result 50 based on the vector generated through the second neural network 200. can In this case, the second neural network 220 may include a fully connected layer (FCL). Also, the third neural network 230 may include a softmax layer. For example, the second neural network 220 may generate vectors equal in size to the number of lesion classes to be classified in the magnetic resonance image 40 based on the characteristics of the magnetic resonance image 40 . The third neural network 230 may receive the output vector of the second neural network 220 and output a probability value indicating the presence or absence of a lesion. The probability value output through the third neural network 230 may be a lesion detection result 50 indicating the presence or absence of a lesion for each lesion class. When a lesion exists, a probability value corresponding to the lesion detection result 50 may be derived as 1. When no lesion exists, a probability value corresponding to the lesion detection result 50 may be derived as 0. However, since the expression of such a probability value is only one example, the probability value may be expressed in various ways within a range understandable by those skilled in the art based on the contents of the present disclosure.

Referring to FIG. 3 , the processor 110 may generate a heat map 60 for generating a lesion label based on an output of any one of the layers included in the first neural network 10 . The processor 110 may extract a gradient indicating a feature change of the magnetic resonance image 40 based on an output of any one of the layers included in the first neural network 10 . The processor 110 may generate a label indicating the presence or absence of a lesion in a specific region in the magnetic resonance image 40 based on the extracted gradient. For example, the processor 110 may extract a gradient representing a feature change of the magnetic resonance image 40 from a feature corresponding to an output of a last layer among convolutional layers included in the first neural network 10 . The processor 110 may generate a heat map capable of visualizing which region of the MRI 40 has an effect on the classification of the lesion, based on the gradient generated from the output of the last layer of the convolutional layer. In addition, the processor 110 may label the lesion 65 present in the magnetic resonance image 40 using the heat map.

Meanwhile, the neural network model 200 including the first neural network 210, the second neural network 220, and the third neural network 230 according to the present disclosure is a first training image with a lesion or a second training image without a lesion. Based on at least one of the 2 learning images, it may be learned to classify the presence or absence of a lesion. For example, the neural network model 200 including a convolutional neural network may be trained to classify the presence or absence of a lesion based on a training image in which a first training image and a second training image labeled with the presence or absence of a lesion are combined. The neural network model 200 may be trained to binary classify the presence or absence of a lesion. When the feature analysis of medical images of the neural network model 200 trained to binary classify the presence or absence of lesions is used for label generation according to the present disclosure, the learning of the neural network model 200 is more effective than learning to multi-classify or detect lesions. can reduce costs. Therefore, when labeling for large-sized lesions is required, in the present disclosure, the neural network model 200 trained to binaryly classify the presence or absence of lesions can be used for label generation.

4 is a flowchart illustrating a label generation method according to an embodiment of the present disclosure.

The computing device 100 according to an embodiment of the present disclosure may receive a medical image through communication with a medical image storage and transmission system, a server, or a client. In this case, the computing device 100 is a cloud system and may be a computing environment capable of storing, managing, and transmitting/receiving data over a communication network. For example, the computing device 100 may receive, store, and manage a magnetic resonance image through communication with a magnetic resonance imaging system. The computing device 100 may detect lesions present in the MRI by analyzing the MRI received through communication with the MRI system using a neural network model. In addition, the computing device 100 may label lesions present in the MRI by using information about which features are used by the neural network model to detect lesions in the MRI.

Referring to FIG. 4 , in step S100, the computing device 100 may detect a lesion present in a medical image by using a pretrained neural network model. In other words, the computing device 100 may classify whether or not a lesion exists in the medical image by inputting the medical image to the neural network model. In this case, the neural network model may include a neural network for extracting features of the medical image and a neural network for classifying lesions based on the features of the medical image. For example, the neural network model may include, but is not limited to, a convolutional neural network.

In step S200, the computing device 100 may generate a label indicating a lesion area in the medical image based on an output of a predetermined layer included in the neural network model. In other words, the computing device 100 may generate a label indicating the presence or absence of a lesion in the medical image based on an output derived from a specific layer in the process of extracting features of the medical image by the neural network model. In this case, the predetermined layer may be a layer in a predetermined order according to the structure of the neural network model. For example, in the case of a neural network model including a convolutional neural network, a predetermined layer may be a last convolutional layer.

On the other hand, since steps S100 and S200 are only codes assigned to distinguish the steps from each other, steps S100 and S200 may be performed sequentially or in parallel.

Various embodiments of the present disclosure described above may be combined with additional embodiments, and may be modified within the scope understandable by those skilled in the art in light of the above detailed description. It should be understood that the embodiments of the present disclosure are illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form. Therefore, all changes or modified forms derived from the meaning, scope and equivalent concepts of the claims of the present disclosure should be construed as being included in the scope of the present disclosure.

Claims (10)

A method of generating a label, performed by a computing device comprising at least one processor, comprising:
detecting a lesion present in a medical image using a neural network model; and
generating a label indicating a lesion area in the medical image based on an output of a predetermined layer included in the neural network model;
Including,
The step of detecting a lesion present in the medical image,
generating a second feature through a Fourier transform based on a first feature calculated in an i-th (i is a natural number) layer of the neural network model; and
inputting a first feature calculated in the i-th layer and a second feature generated through the Fourier transform into an i+1-th layer of the neural network model;
including,
method.
According to claim 1,
The medical image,
A magnetic resonance image generated through Fourier transform based on k-space data obtained from a magnetic resonance signal,
method.
According to claim 1,
The neural network model,
Receiving the medical image and a scan parameter that can be adjusted when capturing the medical image,
method.
According to claim 1,
Generating a label indicating a lesion area in the medical image,
generating a heatmap corresponding to feature analysis of the neural network model for the medical image, based on an output of the predetermined layer; and
generating a label indicating presence/absence of a lesion in a specific region in the medical image based on the heat map;
including,
method.
According to claim 4,
Generating the heat map corresponding to the feature analysis of the neural network model for the medical image,
extracting a gradient from an output of any one of feature extraction layers included in the neural network model, and generating the heat map based on the gradient;
including,
method.
According to claim 4,
Generating a label indicating the presence or absence of a lesion in a specific region in the medical image based on the heat map,
correcting a blank area existing in the lesion area expressed in the heat map;
including,
method.
According to claim 4,
Generating a label indicating the presence or absence of a lesion in a specific region in the medical image based on the heat map,
correcting noise present in the heat map;
including,
method.
According to claim 1,
The neural network model,
Learning to classify the presence or absence of a lesion based on at least one of the first learning image in which a lesion exists or the second learning image in which a lesion does not exist,
method.
A computer program stored on a computer readable storage medium, wherein the computer program, when executed on one or more processors, performs operations for generating a label,
These actions are
detecting a lesion present in a medical image using a neural network model; and
generating a label indicating a lesion area in the medical image, based on an output of a predetermined layer included in the neural network model;
Including,
The operation of detecting a lesion existing in the medical image,
generating a second feature through a Fourier transform based on a first feature calculated in an i-th (i is a natural number) layer of the neural network model; and
inputting a first feature calculated in the ith layer and a second feature generated through the Fourier transform into the i+1th layer of the neural network model;
including,
computer program.
A computing device for generating a label, comprising:
a processor including at least one core;
a memory including program codes executable by the processor; and
a network unit for receiving medical images;
including,
the processor,
Using a neural network model, detecting lesions present in medical images,
Based on an output of a predetermined layer included in the neural network model, a label indicating a lesion area in the medical image is generated,
The processor, in order to detect a lesion present in the medical image,
Based on the first feature calculated in the ith (i is a natural number) layer of the neural network model, a second feature is generated through a Fourier transform;
Inputting a first feature calculated in the ith layer and a second feature generated through the Fourier transform into the i+1th layer of the neural network model,
Device.