KR20220060746A - Method and apparatus for generating medical image - Google Patents

Abstract

Disclosed is a medical image generation method performed by a computing device. A medical image generation method according to an embodiment of the present invention may include the steps of: acquiring a first feature vector and a second feature vector from a first candidate medical image and a second candidate medical image, respectively; synthesizing the acquired first and second feature vectors to generate a third feature vector corresponding to a target rating which is positioned between a first rating corresponding to the first candidate medical image and a second rating corresponding to the second candidate medical image; and generating a target medical image corresponding to the target rating on the basis of the third feature vector. Therefore, it is possible to improve the reading accuracy of a reading model which extracts a feature vector from the medical image.

Description

METHOD AND APPARATUS FOR GENERATING MEDICAL IMAGE

The present invention relates to a method and apparatus for generating a medical image.

An artificial neural network that extracts a feature vector of an input medical image and calculates a result (eg, malignancy of a disease, etc.) corresponding to the feature vector based thereon may be learned based on prepared learning data. In this case, there is a high possibility that the performance will decrease for the age at which some training data is omitted.

In particular, when using time-series continuous medical images as learning data, a section that does not exist in the representative medical image, that is, a section corresponding to an intermediate stage (eg, a natural number as a representative image according to the malignancy of diabetic retinopathy) When only fundus images of grades (grade 1, grade 2??, etc.) exist, the reading accuracy of the artificial neural network may decrease because training data is missing for grades 1.2, 2.3, etc.).

An object of the present invention is to provide a means for assisting in reading a medical image by generating a medical image corresponding to a non-databaseized grade using a pre-stored graded medical image.

Another object of the present invention is to provide a reading model learning method capable of improving the reading accuracy of a reading model for extracting feature vectors from a medical image.

Also, the image synthesis method according to the present invention may be applied to any medical image to which a predetermined grade (eg, an arbitrary numerical factor such as age or disease progression) is assigned to each medical image. Through this, the present invention can provide a means for generating a medical image corresponding to a non-existent grade from existing medical images in various fields of using medical images.

The characteristic configuration of the present invention for achieving the object of the present invention as described above and for realizing the characteristic effects of the present invention to be described later is as follows.

According to an exemplary embodiment, a method of generating a medical image includes: obtaining a first feature vector and a second feature vector from each of a first candidate medical image and a second candidate medical image; A third feature vector corresponding to a target class is generated by synthesizing the obtained first feature vector and the second feature vector, wherein the target class is a first class corresponding to the first candidate medical image and the second candidate locating between second grades corresponding to medical images; and generating a target medical image corresponding to the target class based on the third feature vector.

In the generating of the third feature vector, the computing device generates based on a weighted sum of a first label corresponding to the grade of the first candidate medical image and a second label corresponding to the grade of the second candidate medical image. A first weight corresponding to the first label and a second weight corresponding to the second label are determined so that a third label to be used corresponds to the target class, and the first feature vector to which the first weight is given and the second weight corresponding to the second label are determined. The third feature vector may be generated based on the sum of the second feature vectors to which the second weight is assigned.

In the step of generating the third feature vector, an interval between the first grade and the second grade may be selected to be less than or equal to a predetermined interval.

In the generating of the target medical image, the computing device may generate the target medical image based on either the first candidate medical image or the second candidate image along with the third feature vector.

The first feature vector and the second feature vector are extracted by inputting a first candidate medical image and the second candidate medical image to a pre-trained first reading model, and the first reading model has a corresponding grade. It may be learned to extract a feature vector for the input medical image based on the labeled training medical image, and to read a bone age corresponding to the input medical image based on the extracted feature vector.

The first reading model may be additionally learned based on a synthetic training medical image in which two or more existing medical training images are synthesized, and the composite training medical image may correspond to a grade that does not exist in the training medical image. there is.

The target medical image may be an image output by inputting the third feature vector to a pre-trained second reading model.

A computing device for generating a medical image may include: a communication unit; and a processor, wherein the processor includes a first feature vector and a second feature vector obtained from each of the first candidate medical image and the second candidate medical image obtained to generate a medical image corresponding to the input target class. to generate a third feature vector corresponding to the target class by combining It may be located between a first grade that is a given medical image and a second grade corresponding to the second candidate medical image.

According to an embodiment of the present invention, by additionally providing a medical image corresponding to a grade that is not in a database, it is possible to improve the accuracy of reading a grade (such as a malignancy of a disease) of the medical image.

In addition, based on the improved learning method using the processed learning data, it is possible to provide a remarkable effect of improving the performance of a reading model that is trained to extract a feature vector from a medical image.

According to the present invention, there is a potential effect of ultimately improving the diagnosis accuracy of medical personnel and providing a reference medical image corresponding to an individual grade, thereby innovating a workflow in a medical field.

In addition, in the present invention, since the conventional medical image used in hospitals can be used as it is, it goes without saying that the method of the present invention is not dependent on a specific type of image or platform.

The accompanying drawings for use in the description of the embodiments of the present invention are only a part of the embodiments of the present invention, and for those of ordinary skill in the art to which the present invention belongs (hereinafter referred to as "those skilled in the art") Other drawings may be obtained based on these drawings without an effort to arrive at the invention.
1 is a conceptual diagram schematically illustrating an exemplary configuration of a computing device performing a method for generating a medical image according to the present invention.
2 is an exemplary block diagram illustrating hardware or software components of a computing device performing a method for generating a medical image according to the present invention.
3 is a flowchart illustrating a method of generating a medical image according to an exemplary embodiment.
4 is an exemplary block diagram illustrating hardware or software components of a computing device performing a method for generating a bone image according to the present invention.
5 is a flowchart illustrating a method for generating a bone image according to an embodiment.
6 is a diagram illustrating an example in which a method for generating a bone image according to an embodiment is performed.
7 is a diagram illustrating a distribution of feature vectors according to age.
8 is a diagram illustrating an example of an interface to which a method for generating a bone image according to an embodiment is applied.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The following detailed description of the present invention refers to the accompanying drawings, which show by way of illustration a specific embodiment in which the present invention may be practiced, in order to clarify the objects, technical solutions and advantages of the present invention. These embodiments are described in sufficient detail to enable those skilled in the art to practice the present invention.

The term "image" or "image data" as used throughout the present description and claims refers to multidimensional data composed of discrete image elements (e.g., pixels in a two-dimensional image, voxels in a three-dimensional image). refers to For example, “imaging” means X-ray imaging, (cone-beam) computed tomography, magnetic resonance imaging (MRI), ultrasound or any other medical imaging known in the art. It may be a medical image of a subject, that is, a subject collected by the system. Also, the image may be provided in a non-medical context, for example, a remote sensing system, an electron microscopy, or the like.

Throughout the description and claims of the present invention, 'image' refers to an image (eg, displayed on a video screen) or an image (eg, a file corresponding to a pixel output of a CT, MRI detector, etc.) of an image. A term referring to digital representations.

Throughout the detailed description and claims of the present invention, the 'DICOM (Digital Imaging and Communications in Medicine)' standard is a generic term for various standards used for digital image representation and communication in medical devices, DICOM Standards are published by a joint committee formed by the American Society of Radiological Medicine (ACR) and the National Electrical Engineers Association (NEMA).

In addition, throughout the detailed description and claims of the present invention, 'picture archiving and communication system (PACS)' is a term that refers to a system that stores, processes, and transmits in accordance with the DICOM standard, X-ray, CT , medical imaging images acquired using digital medical imaging equipment such as MRI are stored in DICOM format and transmitted to terminals inside and outside the hospital through a network, to which reading results and medical records can be added.

And throughout the detailed description and claims of the present invention, 'learning' or 'learning' is a term referring to performing machine learning through computing according to a procedure, and a mental action such as human educational activity. It will be understood by those of ordinary skill in the art that this is not intended to be a reference.

And throughout this description and claims, the word 'comprise' and variations thereof are not intended to exclude other technical features, additions, components or steps. In addition, 'one' or 'an' is used to mean more than one, and 'another' is limited to at least a second or more.

Other objects, advantages and characteristics of the present invention will become apparent to a person skilled in the art, in part from this description, and in part from practice of the present invention. The following illustrations and drawings are provided by way of illustration and are not intended to limit the invention. Therefore, the details disclosed herein with respect to a specific structure or function are not to be construed in a limiting sense, but merely representative It should be interpreted as basic data.

Moreover, the invention encompasses all possible combinations of the embodiments indicated herein. It should be understood that various embodiments of the present invention are different but need not be mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the present invention in relation to one embodiment. In addition, it should be understood that the position or arrangement of individual components in each disclosed embodiment may be changed without departing from the spirit and scope of the present invention. Accordingly, the detailed description set forth below is not intended to be taken in a limiting sense, and the scope of the present invention, if properly described, is limited only by the appended claims, along with all scope equivalents as those claimed. Like reference numerals in the drawings refer to the same or similar functions throughout the various aspects.

In this specification, unless indicated otherwise or clearly contradicted by context, items referred to in the singular encompass the plural unless the context requires otherwise. In addition, in describing the present invention, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present invention, the detailed description thereof will be omitted.

Hereinafter, in order to enable those skilled in the art to easily practice the present invention, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram schematically illustrating an exemplary configuration of a computing device performing a method for generating a medical image according to the present invention.

Referring to FIG. 1 , a computing device 100 according to an embodiment of the present invention includes a communication unit 110 and a processor 120 , and is directly or indirectly connected to an external computing device (not shown) through the communication unit 110 . can communicate effectively.

Specifically, the computing device 100 includes typical computer hardware (eg, a device that may include a computer processor, memory, storage, input and output devices, and other components of conventional computing devices; routers, switches, etc.). Electronic communication devices; electronic information storage systems such as network-attached storage (NAS) and storage area networks (SANs)) and computer software (i.e., that enable the computing device to function in a particular way) instructions) to achieve the desired system performance.

The communication unit 110 of such a computing device may transmit/receive a request and a response to/from another computing device that is interlocked. As an example, such a request and a response may be made by the same transmission control protocol (TCP) session. , but is not limited thereto, and may be transmitted and received as, for example, user datagram protocol (UDP) datagrams. In addition, in a broad sense, the communication unit 110 may include a keyboard, a mouse, other external input devices, printers, displays, and other external output devices for receiving commands or instructions.

In addition, the processor 120 of the computing device includes a micro processing unit (MPU), a central processing unit (CPU), a graphics processing unit (GPU) or a tensor processing unit (TPU), a cache memory, and a data bus. ) may include a hardware configuration such as In addition, the operating system may further include a software configuration of an application for performing a specific purpose.

Also, the present invention may be applied to medical images that have been graded according to any criteria. An example of the graded medical image may include a fundus image in which the degree of progression of diabetic retinopathy for the eye included in the image is graded and labeled, but the scope of the present invention is not limited thereto. Those of ordinary skill in the art will understand that it is applicable to medical images graded according to any condition, such as graded bone images.

In the case of an embodiment in which the present invention is applied to a bone image, an X-ray image of a bone is targeted as an example of a bone image, but the scope of the present invention is not limited thereto, and it can be applied to all types of bone images in general, Those skilled in the art will readily understand that it can be applied in the process of reading the bone age of the target image.

2 is an exemplary block diagram illustrating hardware or software components of a computing device performing a method for generating a medical image according to the present invention.

The individual modules shown in FIG. 2 may be implemented by, for example, the communication unit 110 or the processor 120 included in the computing device 100 or the communication unit 110 and the processor 120 interworking. technicians will be able to understand.

A brief overview of the configuration of the method and apparatus according to the present invention with reference to FIG. 2 , the computing device 100 may include the target rating information receiving module 210 as a component thereof. The target rating information receiving module 210 may receive target rating information from an input device included in the computing device and an interlocked user terminal. The grade referred to herein may mean a grade quantified for a medical image according to any condition. For example, the fundus image may be classified into a grade 0 corresponding to a normal state to a grade 4 having a maximum degree of progression according to the degree of progression of diabetic retinopathy. In addition, the bone image may be classified according to bone age as described above, and additionally, the grade of the individual bone image may be determined according to the malignancy of the lesion present in the bone image. As such, the grade referred to in this specification may refer to quantified information determined for an individual medical image based on a predetermined condition (eg, age, malignancy of a lesion, degree of progression of a predetermined disease, etc.) Those of ordinary skill in the art will understand. Also, the target grade information may refer to information on a grade of a medical image to be newly created through an existing medical image.

Based on the target grade information acquired through the target grade information receiving module 210 , the feature vector acquisition module 220 may acquire a feature vector for generating a medical image corresponding to the target grade information.

According to an embodiment, the feature vector acquisition module 220 may include a pre-trained first reading model to extract a feature vector from an input medical image and read a grade of the medical image based thereon. The first reading model is, for example, a Deep Neural Network (DNN), a Convolutional Neural Network (CNN), a Deep Convolution Neural Network (DCNN), a Recurrent Neural Network (RNN), a Restricted Boltzmann Machine (RBM), or a Deep Belief Network (DBN). ), a Single Shot Detector (SSD), You Only Look Once (YOLO), etc. may be used as will be understood by those skilled in the art. The type of artificial neural network that can be used in the present invention is not limited to the presented example, and it can be learned to extract a feature vector from a medical image based on the labeled training data, and to read the grade of the medical image based on the extracted feature vector. It will be understood by those of ordinary skill in the art that it may include any artificial neural network capable of being used.

According to an embodiment, the first reading model included in the feature vector acquisition module 220 extracts a feature vector for the input medical image based on the training medical image labeled with the corresponding grade, and adds the extracted feature vector to the first reading model. It may be learned in advance to read a grade corresponding to the input medical image based on it.

Also, the first reading model may be additionally learned based on the synthesized training medical image generated based on the synthesis between the existing training medical images. For example, if the existing learning medical image includes fundus images corresponding to grades 1 and 3 of the progression of diabetic retinopathy, the first reading model is In addition to the fundus image, learning may be performed based on the fundus image corresponding to the second grade generated based on the synthesis of the fundus images corresponding to the first and third grades. The generation of the synthetic learning medical image using the existing learning medical image may be performed based on the operations of the synthesizing module 230 and the medical image generating and transmitting module 240 to be described below. In addition, the method for generating the synthetic learning medical image is not limited to the example shown, and more various grades of the synthetic learning medical image may be generated based on the existing medical learning image. For example, synthetic learning fundus images corresponding to grades 1.6, 2.2, 2.5, and the like may be generated through the learning fundus images corresponding to grades 1 and 3.

The feature vector acquisition module 220 may acquire a feature vector for an input medical image through an operation of acquiring an output of the intermediate layer of the first read model as a feature vector.

The feature vector acquisition module 220 acquires a first candidate medical image and a second candidate medical image for generating a medical image corresponding to the target class, and uses the first read model to obtain the first medical candidate image and the second medical candidate image A first feature vector and a second feature vector corresponding to each image may be extracted. For example, when target grade information is input as grade 2, which is the degree of progression of diabetic retinopathy with respect to the fundus image, the feature vector acquisition module 220 configures the first candidate medical image of grade 0 corresponding to the normal state ( 1 candidate fundus image) and a grade 3 second medical image (second candidate fundus image) in which diabetic retinopathy has progressed to a certain degree, and a first feature vector and a second feature vector corresponding to each candidate medical image are obtained. can be extracted. The method of acquiring the candidate fundus image is not limited to the presented exemplary method, and in order to generate a fundus image corresponding to the target grade of grade 2, the first candidate fundus image of grade 1 and the second candidate fundus image of grade 4 are generated. It will be understood by those skilled in the art that it can be obtained.

According to another embodiment, the feature vector acquisition module 220 may acquire a first feature vector and a second feature vector corresponding to target class information from among the databased feature vectors. According to an embodiment, a feature vector corresponding to each grade may be databased, and when the target grade is grade 2, the feature vector obtaining module 220 provides a feature vector corresponding to grade 0 and a feature corresponding to grade 3 The vector can be obtained from the database.

The synthesis module 230 may generate a third feature vector corresponding to the target class information by synthesizing the first feature vector and the second feature vector obtained through the feature vector obtaining module 220 . Specifically, the synthesis module 230 may preferentially determine whether to synthesize the first feature vector and the second feature vector based on a predetermined condition. For example, the synthesis module 230 may determine whether to synthesize the first feature vector and the second feature vector based on Equation (1).

는 제1 특징 벡터에 대응되는 제1 레이블(등급),

는 제2 특징 벡터에 대응되는 제2 레이블(등급),

은 미리 설정된 임계값을 나타낼 수 있다.

is a first label (grade) corresponding to the first feature vector,

is a second label (grade) corresponding to the second feature vector,

may indicate a preset threshold value.

은 합성 결과의 정확성을 위해 미리 설정되는 임계값일 수 있다. 구체적으로, 당뇨성망막병증 2등급에 대응되는 특징 벡터를 생성하기 위하여 당뇨성망막병증이 발생하지 않는 안저 영상에 해당하는 0등급에 대응되는 특징 벡터 및 당뇨성망막병증 4등급에 대응되는 특징 벡터를 사용하는 경우보다, 당뇨성망막병증 1등급에 대응되는 특징 벡터 및 당뇨성망막병증 3등급에 대응되는 특징 벡터를 이용하는 것이 보다 정확도가 높은 결과가 산출될 수 있다. 따라서, 합성 모듈(230)은 소정의 임계값에 따라 특징 벡터의 합성 여부를 결정할 수 있다.

may be a threshold value set in advance for the accuracy of the synthesis result. Specifically, in order to generate a feature vector corresponding to diabetic retinopathy grade 2, a feature vector corresponding to grade 0 corresponding to a fundus image in which diabetic retinopathy does not occur and a feature vector corresponding to grade 4 diabetic retinopathy A result with higher accuracy may be obtained by using a feature vector corresponding to grade 1 diabetic retinopathy and a feature vector corresponding to grade 3 diabetic retinopathy than in the case of using . Accordingly, the synthesis module 230 may determine whether to synthesize the feature vector according to a predetermined threshold value.

According to another embodiment, when the interval between the labels of the feature vectors is equal to or less than a threshold, the synthesis module 230 may determine whether to proceed with the synthesis according to a predetermined probability.

When it is determined to synthesize the feature vector based on Equation 1, the synthesis module 230 may synthesize the feature vector and a label corresponding to the feature vector based on Equations 2 and 3 based on Equation 2 and Equation 3 .

,

,

는 각각 제1 특징 벡터, 제2 특징 벡터 및 제3 특징 벡터이고,

,

,

는 각각 제1 특징 벡터에 대응되는 제1 레이블, 제2 특징 벡터에 대응되는 제2 레이블 및 제3 특징 벡터에 대응되는 제3 레이블을 의미하고,

는 합성 정도를 나타내는 임의의 변수일 수 있으며, 예를 들어,

는 beta 분포를 따르는 변수일 수 있다.

,

,

are a first feature vector, a second feature vector, and a third feature vector, respectively,

,

,

denotes a first label corresponding to the first feature vector, a second label corresponding to the second feature vector, and a third label corresponding to the third feature vector, respectively;

may be any variable indicating the degree of synthesis, for example,

may be a variable that follows a beta distribution.

Based on Equations 2 and 3, a third feature vector and a third label (grade) corresponding thereto may be determined.

A person skilled in the art will understand that the method of performing the synthesis by the synthesis module 230 may include any data synthesis method including a method using a linear interpolation or a fully connected layer.

The medical image generating and transmitting module 240 may generate a medical image corresponding to the target class based on the third feature vector generated by the synthesizing module 230 . Specifically, the medical image generation and transmission module 240 uses a pre-trained second reading model to generate a target medical image corresponding to the above characteristic information by inputting characteristic information corresponding to the target medical image to be generated. A target medical image can be generated. For example, the second reading model may include an artificial neural network model such as an autoencoder or a GAN (eg, CycleGAN, BiBigGAN), and based on this, a medical image corresponding to a target grade may be generated. The medical image generating and transmitting module 240 inputs a third feature vector (for example, a feature vector generated based on the synthesis of feature vectors obtained from a candidate medical image) to the second read model, and adds it to the third feature vector. A corresponding medical image (a medical image corresponding to the target class) may be generated.

Additionally, the artificial neural network included in the second reading model may be pre-trained to generate a target medical image corresponding to the feature vector by inputting an initial medical image together with a feature vector corresponding to the target medical image to be generated. As the initial medical image, a target medical image may be provided, but the present invention is not limited thereto, and an arbitrary medical image may be provided.

Using the second reading model learned by using the initial medical image as an additional input, the medical image generation and transmission module 240 may generate a third feature vector and a first candidate medical image or A target medical image corresponding to the third feature vector may be generated from the second candidate medical image. The second reading model trained to generate the target medical image by inputting the initial medical image together with the third feature vector may generate the target medical image with improved quality compared to the case where only the feature vector is used.

A person skilled in the art can easily realize that the medical image generating and transmitting module 240 is not limited to the presented example, and can be implemented through any method of generating a target medical image based on the synthesis of feature vectors extracted from a candidate medical image. can be understood

The medical image generating and transmitting module 240 may store the generated medical image in a database or provide it to an external entity. When a medical image is provided to an external entity, the medical image generating and transmitting module 240 may provide the generated medical image to an external entity using a predetermined display device or the like, or through a communication unit provided therein. Here, the meaning of the external entity includes a user, an administrator, and a medical professional in charge of the computing device 100, but it should be understood that any subject that requires a bone image for the target age is included in addition to this. will be. For example, the external entity may be an external AI device including a separate artificial intelligence (AI) hardware and/or software module utilizing the bone image. In addition, 'external' in the external entity is not intended to exclude an embodiment in which the AI hardware and/or software module using the bone image is integrated into the computing device 100, but the method of the present invention It should be noted that the bone image, which is the result of the performing hardware and/or software module, is used to suggest that it can be utilized as input data of other methods. That is, the external entity may be the computing device 100 itself. Although the components shown in FIG. 2 are exemplified as being realized in one computing device for convenience of description, the computing device 100 performing the method of the present invention is It will be understood that a plurality may be configured to interwork with each other.

3 is a flowchart illustrating a method of generating a medical image according to an exemplary embodiment.

Referring to FIG. 3 , the computing device may receive target rating information in operation 310 . A person skilled in the art will understand that the target grade information corresponds to information on a target grade of a medical image to be newly generated, and may be received from an external entity through a communication unit or may be inputted through a provided input device.

In operation 320 , the computing device may acquire a first candidate medical image and a second candidate medical image for generating a medical image corresponding to the target class information. When the target grade is included between the grade corresponding to the first candidate medical image and the grade corresponding to the second candidate medical image, the computing device is configured to provide an interval between the grade of the first candidate medical image and the grade of the second candidate medical image. It can be determined whether the interval is smaller than a predetermined interval. When the interval between the grade of the first candidate medical image and the grade of the second candidate medical image is less than or equal to a predetermined interval, the computing device may perform a subsequent step 330 . If the interval between the grade of the first candidate medical image and the grade of the second candidate medical image is greater than a predetermined interval, the computing device may be configured to generate the first candidate medical image and A process of re-acquiring the second candidate medical image may be performed. This is to improve the accuracy of a calculation result by using a medical image corresponding to a grade close to the target grade in the process of generating a medical image corresponding to the target grade.

The computing device may obtain a first feature vector and a second feature vector for each of the first candidate medical image and the second candidate medical image, based on the first reading model learned in step 330 . As described above, the first reading model may be an artificial neural network trained to extract a feature vector from an input medical image and read a grade of the input medical image based thereon.

The first reading model is an input medical image based on the training medical image processed as described above (including learning based on the existing learning image and a synthetic learning image generated based on the synthesis of the existing learning image) It may be additionally learned to extract a feature vector of , and calculate a grade of an input medical image based on the extracted feature vector. A person skilled in the art can easily understand how the first reading model is trained to read the grade of the medical image by using the medical image labeled with the corresponding grade as training data. The first reading model used in the present invention may be trained using a processed training medical image to improve reading accuracy. Specifically, the processed learning medical image may further include a third learning medical image generated by synthesizing the first medical learning image and the second learning medical image obtained in the learning medical image together with the existing medical learning image. can The third medical training image may correspond to a third grade corresponding to a grade between a first grade corresponding to the first training medical image and a second grade corresponding to the second training medical image. The processed learning data may include all elements corresponding to individual grades compared to the existing learning data. The present invention may provide a reading result with higher accuracy through the first reading model learned using the processed training data.

The computing device may generate a third feature vector corresponding to the target class by synthesizing the first feature vector and the second feature vector in operation 340 . The computing device may generate the third feature vector by interpolating the first feature vector and the second feature vector, and the method of generating the third feature vector is described in Equations 1 to 3 described above. same as it has been

In operation 350 , the computing device may generate a target medical image corresponding to the target class based on the third feature vector.

4 is an exemplary block diagram illustrating hardware or software components of a computing device performing a method for generating a bone image according to the present invention.

The individual modules shown in FIG. 4 may be implemented by, for example, the communication unit 110 or the processor 120 included in the computing device 100 or the communication unit 110 and the processor 120 interworking. technicians will be able to understand.

A brief overview of the configuration of the method and apparatus according to the present invention with reference to FIG. 4 , the computing device 100 may include a target bone age information receiving module 410 as a component thereof. The target bone age information receiving module 410 may receive target bone age information from an input device included in the computing device and an interlocked user terminal.

Based on the target bone age information acquired through the target bone age information receiving module 410 , the feature vector acquisition module 420 may acquire a feature vector for generating a bone image corresponding to the target bone age information.

According to an embodiment, the feature vector acquisition module 420 may include a pre-trained first 'reading model to extract a feature vector from an input bone image and read a bone age based on the extracted feature vector. The first 'reading model is, for example, a Deep Neural Network (DNN), a Convolutional Neural Network (CNN), a Deep Convolution Neural Network (DCNN), a Recurrent Neural Network (RNN), a Restricted Boltzmann Machine (RBM), or a Deep Belief (DBN). Network), SSD (Single Shot Detector), YOLO (You Only Look Once), etc. may be used as will be understood by those skilled in the art. The type of artificial neural network that can be used in the present invention is not limited to the presented example, and it can be learned to extract a feature vector from a bone image based on the labeled learning data, and to read the bone age based on the extracted feature vector. It will be understood by those skilled in the art that it may include any artificial neural network.

According to an embodiment, the first 'reading model included in the feature vector acquisition module 420 extracts a feature vector for the input bone image based on the training bone image labeled with the corresponding age, and the extracted feature vector It may be learned in advance to read the bone age corresponding to the input bone image based on the .

In addition, in the first 'reading model, additional learning may be performed based on a synthesized learning bone image generated by synthesizing two or more existing learning bone images. For example, if the pre-existing learning bone image includes bone images corresponding to 1 year old and 3 years old, in addition to the existing bone images corresponding to 1 year old and 3 years old, corresponding to 1 year old and 3 years old Learning of the first 'reading model may be performed based on the bone image corresponding to the age of 2 generated based on the synthesis of the bone image. A person skilled in the art knows that the method of generating a synthetic learning goal image based on an existing learning goal image may be performed based on the operations of the synthesizing module 430 and the bone image generation and transmission module 440 to be described below. will understand In addition, the method of generating the synthetic learning bone image is not limited to the example shown, and synthetic learning bone images of various ages may be generated based on the existing learning bone image. For example, through the learning bone images corresponding to 1 year old and 3 years old, synthetic learning bone images corresponding to 1.6 years old, 2.2 years old, 2.5 years old, etc. It may be performed based on an operation method of the image generation and transmission module 440 .

The feature vector acquisition module 420 may acquire a feature vector for the input valley image through an operation of acquiring the output of the intermediate layer in the first 'reading model as a feature vector.

The feature vector acquisition module 420 may acquire at least two or more candidate bone images corresponding to the target bone age, and extract a feature vector corresponding to the bone image through a pre-learned first 'reading model. For example, when the target bone age is 4 years old, the feature vector acquisition module acquires a first candidate bone image corresponding to 3 years old and a second candidate bone image corresponding to 5 years old, and the first candidate bone image corresponding to each image A 'feature vector and a second' feature vector may be extracted.

According to another embodiment, the feature vector acquisition module 420 may acquire a first' feature vector and a second' feature vector corresponding to the target bone age from among the databased feature vectors. According to an embodiment, feature vectors corresponding to each age may be databased, and when the target age is 4 years old, the feature vector acquisition module 420 may generate a feature vector corresponding to 3 years old and a feature vector corresponding to 5 years old. can be obtained from the database.

The synthesis module 430 may generate a third feature vector corresponding to the target age by synthesizing the first 'feature vector and the second' feature vector obtained through the feature vector acquisition module 420 . Specifically, the synthesis module 430 may preferentially determine whether to synthesize the first' feature vector and the second' feature vector based on a predetermined condition. For example, the synthesis module 430 may determine whether to synthesize the first' feature vector and the second' feature vector based on Equation (4).

는 제1' 특징 벡터에 대응되는 제1' 레이블(나이),

는 제2' 특징 벡터에 대응되는 제2' 레이블(나이),

은 미리 설정된 임계값을 나타낼 수 있다.

is the first label (age) corresponding to the first 'feature vector,

is the second' label (age) corresponding to the second' feature vector,

may indicate a preset threshold value.

은 합성 결과의 정확성을 위해 미리 설정되는 임계값일 수 있다. 구체적으로, 10세에 대응되는 특징 벡터를 생성하기 위하여 1세에 대응되는 특징 벡터 및 19세에 대응되는 특징 벡터를 사용하는 경우보다, 9세에 대응되는 특징 벡터 및 11세에 대응되는 특징 벡터를 이용하는 것이 보다 정확도가 높은 결과가 산출될 수 있다. 따라서, 합성 모듈(430)은 소정의 임계값에 따라 특징 벡터의 합성 여부를 결정할 수 있다.

may be a threshold value set in advance for the accuracy of the synthesis result. Specifically, rather than using a feature vector corresponding to 1 year old and a feature vector corresponding to 19 years old to generate a feature vector corresponding to 10 years old, a feature vector corresponding to 9 years old and a feature vector corresponding to 11 years old A more accurate result can be obtained by using . Accordingly, the synthesis module 430 may determine whether to synthesize the feature vector according to a predetermined threshold value.

According to another embodiment, when the interval between labels of feature vectors is equal to or less than a threshold, the synthesis module 430 may determine whether to proceed with synthesis according to a predetermined probability.

When it is determined to synthesize the feature vector based on Equation 4, the synthesis module 430 may synthesize the feature vector and a label corresponding to the feature vector based on Equations 5 and 6 based on Equation (4).

,

,

는 제1 후보 골 영상, 제2 후보 골 영상, 및 타깃 연령에 대응되는 골 영상 각각에 대응되는 제1' 특징 벡터, 제2' 특징 벡터, 제3' 특징 벡터이고,

,

,

는 각각 제1' 특징 벡터에 대응되는 제1' 레이블(나이), 제2' 특징 벡터에 대응되는 제2' 레이블(나이), 제3' 특징 벡터에 대응되는 제3' 레이블(나이)을 의미하고,

는 합성 정도를 나타내는 임의의 변수일 수 있으며, 예를 들어,

는 beta 분포를 따르는 변수일 수 있다.

,

,

is a first 'feature vector, a second' feature vector, and a third' feature vector corresponding to each of the first candidate bone image, the second candidate bone image, and the bone image corresponding to the target age,

,

,

1' label (age) corresponding to the 1' feature vector, 2' label (age) corresponding to the 2' feature vector, and 3' label (age) corresponding to the 3' feature vector, respectively means,

may be any variable indicating the degree of synthesis, for example,

may be a variable that follows a beta distribution.

Based on Equations 5 and 6, a third' feature vector and a third' label (age) corresponding thereto may be determined.

A person skilled in the art will understand that the method of performing the synthesis by the synthesis module 430 may include any data synthesis method including a method using a linear interpolation or a fully connected layer.

The bone image generation and transmission module 440 may generate a bone image corresponding to the target age based on the 3' feature vector generated by the synthesis module 430 . More specifically, the bone image generation and transmission module 240 receives the feature information corresponding to the target bone image to be generated as an input, and uses a pre-trained second 'reading model to generate the target bone image corresponding to the above feature information. A target bone image may be generated using the . For example, the second 'reading model may include an artificial neural network model such as an Autoencoder or a Generative Adversarial Network (GAN) (eg, CycleGAN, BiBigGAN), based on which a bone image corresponding to the target age is generated can be The bone image generation and transmission module 440 inputs a 3' feature vector (for example, a feature vector generated based on the synthesis of a feature vector extracted from a candidate bone image) into the 2' read model, and the 3' feature vector It is possible to generate a target bone image (a bone image corresponding to the target age) corresponding to .

Additionally, the artificial neural network included in the second 'reading model may be pre-trained to generate a target bone image corresponding to the feature vector by inputting an initial bone image along with a feature vector corresponding to the target bone image to be generated. The initial bone image may be input as a target bone image, but is not limited thereto, and any bone image may be provided.

The bone image generation and transmission module 440 inputs the 3' feature vector and the first candidate bone image or the second candidate bone image, which is the basis for the generation of the 3' feature vector, into the second' readout model to obtain a third ' It is possible to generate a target bone image corresponding to the feature vector. A person skilled in the art knows that the second reading model trained to generate a target bone image by inputting an initial bone image along with the third feature vector can generate a target bone image of improved quality compared to the case where only the feature vector is used. will understand

A person skilled in the art can easily understand that the bone image generation and transmission module 440 is not limited to the presented example, and may be implemented through any method for generating a bone image based on a feature vector.

The bone image generation and transmission module 440 may store the generated bone image in a database or provide it to an external entity. When the bone image is provided to an external entity, the bone image generation and transmission module 440 may provide the generated bone image to an external entity using a predetermined display device or the like, or through a communication unit provided. Here, the external entity includes a user, an administrator, and a medical professional in charge of the computing device 100, but it should be understood that any subject that requires a bone image for the target age is included in addition to this. something to do. For example, the external entity may be an external AI device including a separate artificial intelligence (AI) hardware and/or software module utilizing the bone image. In addition, 'external' in the external entity is not intended to exclude an embodiment in which the AI hardware and/or software module using the bone image is integrated into the computing device 100, but the method of the present invention It should be noted that the bone image, which is the result of the performing hardware and/or software module, is used to suggest that it can be utilized as input data of other methods. That is, the external entity may be the computing device 100 itself.

Although the components shown in FIG. 4 are exemplified as being implemented in one computing device for convenience of description, it will be understood that a plurality of computing devices 100 performing the method of the present invention may be configured to interwork with each other.

5 is a flowchart illustrating a method for generating a bone image according to an embodiment.

Referring to FIG. 5 , the computing device may receive a target bone age through the communication unit in operation 510 .

The computing device may acquire a first candidate bone image and a second candidate bone image for generating a bone image corresponding to the target bone age in operation 520 . If the target bone age is included between the bone age corresponding to the first candidate bone image and the bone age corresponding to the second candidate bone image, the computing device is configured to include the bone age of the first candidate bone image and the bone age of the second candidate bone image. It may be determined whether the interval between the ages is smaller than a preset predetermined interval. When the interval between the bone age of the first candidate bone image and the bone age of the second candidate bone image is less than or equal to a predetermined interval, the computing device may perform a subsequent step 530 . If the interval between the bone age of the first candidate goal image and the bone age of the second candidate goal image is greater than a predetermined interval, the computing device is configured to configure the first candidate bone to be less than or equal to the predetermined interval. A process of re-acquiring at least one of the image and the second candidate bone image may be performed. This is to improve the accuracy of the calculation result by using a bone image corresponding to an age close to the target bone age in the process of generating an image corresponding to the target bone age, as described above.

The computing device may extract a first' feature vector and a second' feature vector for each of the first candidate bone image and the second candidate bone image, based on the first 'reading model learned in advance in step 530 . there is. As described above, the pre-trained first 'reading model may be an artificial neural network trained to extract a feature vector from the input bone image and read the age of the input bone image based thereon.

The first 'reading model may be learned to output a feature vector of an input bone image based on the processed learning bone image. A person skilled in the art can easily understand how the first 'reading model is trained to read the age of the bone image by using the corresponding age-labeled bone image as training data. The first 'reading model used in the present invention may be learned using a processed learning bone image to improve the accuracy of reading. Specifically, the processed learning goal image may further include a third learning goal image generated by synthesizing the first learning goal image and the second learning goal image obtained from the learning goal image together with the existing learning goal image. can The third learning bone image may correspond to a third bone age corresponding to an age between the first bone age corresponding to the first learning bone image and the second bone age corresponding to the second learning bone image. The processed learning bone image may include all elements corresponding to individual ages compared to existing learning data. The present invention has a remarkable effect in that it is possible to provide a reading result with higher accuracy through a reading model learned using a processed learning bone image.

The computing device may generate a third' feature vector corresponding to the target bone age by synthesizing the first' feature vector and the second' feature vector in operation 540 . The computing device may generate a third' feature vector by interpolating the first feature vector and the second feature vector, and the method of generating the third' feature vector is not limited to the presented example as described above. does not

The computing device may generate a target bone image corresponding to the age of the target bone based on the 3' feature vector in operation 550 .

6 is a diagram illustrating an example in which a method for generating a bone image according to an embodiment is performed.

The system for reading bone age through the image may provide a reference bone image corresponding to an individual age to the reader, and the reader may read the age of the input bone image through comparison with the reference bone image. If the reference bone image 610 corresponding to the age of 3 is absent among the reference bone images stored in the database, as shown on the left, it may be difficult for the reader to read the bone age of the corresponding age group. In addition, in the case of an infant with a very rapid development rate, since the difference in bone growth can be large even after a few months, even if a reference bone image corresponding to 1 year old and a reference bone image corresponding to 2 years are used, 1 year to 2 years old Predicting the bone age of infants between the ages of three and three can be very difficult.

The computing device 620 receives the first candidate bone image 631 and the second candidate bone image 632 corresponding to the two-year-old in order to generate the missing three-year-old reference bone image, and based on this, the three A bone image 640 corresponding to the age may be generated.

The computing device 620 may generate a bone image corresponding to the target age based on a feature vector previously stored through age-based labeling in a database as well as the suggested method. In this case, in the process of generating the bone image corresponding to the target age, the computing device 620 may generate the bone image only through information on the target bone age without receiving an input of the bone image.

7 is a diagram illustrating a distribution of feature vectors according to age.

Referring to FIG. 7 , a graph 710 may be a distribution of a feature vector for a male bone image, and a graph 720 may be a distribution of a feature vector for a female bone image. Each of the points 711 and 721 included in the graphs 710 and 720 may correspond to a feature vector of an individual bone image, and the color of the points 711 and 721 may indicate a corresponding age. Referring to graphs 710 and 720 , it can be seen that points 711 and 721 corresponding to similar ages are distributed in adjacent areas. Considering this distribution, it can be understood that the feature vector calculated through the synthesis of feature vectors of adjacent ages can reveal the characteristics of the target age.

8 is a diagram illustrating an example of an interface to which a method for generating a bone image according to an embodiment is applied.

Referring to FIG. 8 , reference bone images 821 , 822 , and 823 of a similar age to the current bone image 820 may be provided through the interface 810 . The reference bone images 821 , 822 , and 823 corresponding to each age may be pre-stored in a database. In the reading process, when a reference bone image between the ages of 10 and 11 is requested, the reader may request to generate a reference bone image corresponding to the corresponding age based on a user input to the graphic object 830 . When a user input is made to the graphic object 830 , the graphic object 840 for inputting the target age may be displayed on the screen. The computing device may generate a reference bone image 850 corresponding to the target age and display it on the screen.

Based on the description of the above embodiments, those skilled in the art will appreciate that the method and/or processes of the present invention, and the steps thereof, may be implemented in hardware, software, or any combination of hardware and software suitable for a particular application. point can be clearly understood. The hardware may include general purpose computers and/or dedicated computing devices or specific computing devices or special features or components of specific computing devices. The processes may be realized by one or more microprocessors, microcontrollers, embedded microcontrollers, programmable digital signal processors or other programmable devices, having internal and/or external memory. Additionally, or alternatively, the processes may be configured to process an application specific integrated circuit (ASIC), programmable gate array, programmable array logic (PAL) or electronic signals. It may be implemented with any other device or combination of devices. Moreover, the objects of the technical solution of the present invention or parts contributing to the prior arts may be implemented in the form of program instructions that can be executed through various computer components and recorded in a machine-readable recording medium. The machine-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded in the machine-readable recording medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. Examples of the machine-readable recording medium include a hard disk, a magnetic medium such as a floppy disk and a magnetic tape, an optical recording medium such as a CD-ROM, DVD, Blu-ray, and a magneto-optical medium such as a floppy disk (magneto-optical media), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include any one of the devices described above, as well as heterogeneous combinations of processors, processor architectures, or combinations of different hardware and software, or stored and compiled or interpreted for execution on a machine capable of executing any other program instructions. can be created using a structured programming language such as C, an object-oriented programming language such as C++, or This includes not only bytecode, but also high-level language code that can be executed by a computer using an interpreter or the like.

Accordingly, in one aspect according to the present invention, when the above-described method and combinations thereof are performed by one or more computing devices, the methods and combinations of methods may be implemented as executable code for performing respective steps. In another aspect, the method may be implemented as systems performing the steps, the methods may be distributed in various ways across devices or all functions may be integrated into one dedicated, standalone device or other hardware. In another aspect, the means for performing the steps associated with the processes described above may include any of the hardware and/or software described above. All such sequential combinations and combinations are intended to fall within the scope of this disclosure.

For example, the hardware device may be configured to operate as one or more software modules to perform processing in accordance with the present invention, and vice versa. The hardware device may include a processor, such as an MPU, CPU, GPU, TPU, coupled with a memory such as ROM/RAM for storing program instructions and configured to execute instructions stored in the memory, an external device and a signal It may include a communication unit that can send and receive. In addition, the hardware device may include a keyboard, a mouse, and other external input devices for receiving commands written by developers.

In the above, the present invention has been described with specific matters such as specific components and limited embodiments and drawings, but these are provided to help a more general understanding of the present invention, and the present invention is not limited to the above embodiments, Those of ordinary skill in the art to which the present invention pertains can devise various modifications and variations from these descriptions.

Accordingly, the spirit of the present invention should not be limited to the above-described embodiments, and not only the claims described below, but also all modifications equivalently or equivalently to the claims described below belong to the scope of the spirit of the present invention. will do it

Such equivalent or equivalent modifications shall include, for example, logically equivalent methods capable of producing the same results as practiced by the methods according to the present invention, the spirit and scope of the present invention. should not be limited by the above examples, and should be understood in the broadest sense permitted by law.

Claims (9)

A method for generating a medical image performed by a computing device, the method comprising:
obtaining a first feature vector and a second feature vector from each of the first candidate medical image and the second candidate medical image;
A third feature vector corresponding to a target class is generated by synthesizing the obtained first feature vector and the second feature vector, wherein the target class is a first class corresponding to the first candidate medical image and the second candidate locating between second grades corresponding to medical images; and
generating a target medical image corresponding to the target class based on the third feature vector
Including, a method of generating a medical image.
According to claim 1,
The step of generating the third feature vector comprises:
The third label generated based on a weighted sum of a first label corresponding to the grade of the first candidate medical image and a second label corresponding to the grade of the second candidate medical image corresponds to the target grade. determining a first weight corresponding to the first label and a second weight corresponding to the second label;
and generating the third feature vector based on a sum of the first feature vector to which the first weight is assigned and the second feature vector to which the second weight is assigned.
According to claim 1,
The step of generating the third feature vector comprises:
and an interval between the first grade and the second grade is selected to be less than or equal to a predetermined interval.
According to claim 1,
The generating of the target medical image includes:
and generating the target medical image based on either the first candidate medical image or the second candidate image in addition to the third feature vector.
According to claim 1,
The first feature vector and the second feature vector are,
extracted by inputting the first candidate medical image and the second candidate medical image to the pre-trained first reading model,
The first reading model is
A method for generating a medical image, wherein a feature vector for the input medical image is extracted based on the training medical image labeled with the corresponding grade, and the bone age corresponding to the input medical image is learned based on the extracted feature vector.
6. The method of claim 5,
The first reading model is
It is additionally learned based on the synthesized learning medical image in which two or more of the pre-existing learning medical images are synthesized,
The synthetic learning medical image is,
A method of generating a medical image corresponding to a grade that does not exist in the training medical image.
According to claim 1,
The target medical image is
and an image output by inputting the third feature vector to a pre-trained second reading model.
A computer program, stored in a machine-readable non-transitory recording medium, comprising instructions implemented to cause a computing device to perform the method of claim 1 .
A computing device for generating a medical image, comprising:
communication department; and
processor
including,
The processor is
In order to generate a medical image corresponding to the input target class, first and second feature vectors obtained from the obtained first candidate medical image and second candidate medical image are synthesized, respectively, to generate a first medical image corresponding to the target class. 3 Create a feature vector,
generating a target medical image corresponding to the target class based on the third feature vector;
and the target grade is located between a first grade corresponding to the first candidate medical image and a second grade corresponding to the second candidate medical image.

Images