WO2021118068A1

WO2021118068A1 - Medical image generation method and device using same

Info

Publication number: WO2021118068A1
Application number: PCT/KR2020/015487
Authority: WO
Inventors: 배병욱; 정규환
Original assignee: 주식회사 뷰노
Priority date: 2019-12-09
Filing date: 2020-11-06
Publication date: 2021-06-17

Abstract

Disclosed is a medical image generation method performed by a computing device. A medical image generation method according to an embodiment may comprise the steps of: acquiring a first feature vector and a second feature vector from a first candidate medical image and a second candidate medical image, respectively; synthesizing the acquired first and second feature vectors to generate a third feature vector corresponding to a target rating which is positioned between a first rating corresponding to the first candidate medical image and a second rating corresponding to the second candidate medical image; and generating a target medical image corresponding to the target rating on the basis of the third feature vector.

Description

Medical image generation method and device using same

The present invention relates to a method for generating a medical image including a bone image and an apparatus using the same.

An artificial neural network that extracts a feature vector of an input medical image and calculates a result (eg, age) corresponding to the feature vector based thereon may be trained based on prepared learning data. In this case, there is a high possibility that the performance will decrease for the age at which some training data is omitted.

In particular, when using time-series continuous medical images as training data, in the section that does not exist in the representative medical image, that is, the section corresponding to the intermediate stage (eg, in the case of bone images, 10.4 years old, 21.3 years old, etc.) For example, the reading accuracy of the artificial neural network may decrease because the training data is missing.

An object of the present invention is to provide a means for assisting in reading a bone image by generating a bone image corresponding to an age that is not stored in a database using a previously stored bone image.

Another object of the present invention is to provide a reading model learning method capable of improving the reading accuracy of a reading model that extracts a feature vector from a bone image.

In addition, the synthesis method of the method for generating a bone image according to the present invention is not limitedly applied to a bone image, and each medical image has a predetermined grade (eg, an arbitrary numerical factor such as age, disease progression, etc.). It can be extended to any given medical image. Through this, the present invention can be applied to a method of generating a medical image corresponding to a non-existent grade from an existing medical image.

The characteristic configuration of the present invention for achieving the object of the present invention as described above and for realizing the characteristic effects of the present invention to be described later is as follows.

According to an aspect of the present invention, a method for generating a bone image performed by a computing device includes the steps of: (a) receiving a target bone age; (b) acquiring a first candidate bone image and a second candidate bone image for generating a bone image corresponding to the target bone age; (c) extracting a first feature vector and a second feature vector for each of the first candidate bone image and the second candidate bone image based on the pre-trained read model; (d) generating a third feature vector corresponding to the target bone age by synthesizing the first feature vector and the second feature vector; and (e) generating a target bone image corresponding to the target bone age based on the third feature vector.

According to an embodiment, the pre-trained reading model is pre-trained to output a feature vector of an input goal image based on a processed learning goal image, and the processed learning goal image is a first obtained learning goal image. By synthesizing the learning goal image and the second learning goal image, the third bone age corresponding to the age between the first bone age corresponding to the first learning goal image and the second bone age corresponding to the second learning goal image It may be generated by generating a third learning goal image corresponding to .

A computing device for generating a bone image according to an aspect includes: a communication unit for receiving an input for a target bone age; and a processor generating a target bone image corresponding to the target bone age, wherein the processor is configured to generate a first feature vector for each of the first candidate bone image and the second candidate bone image based on a pre-trained reading model. and obtaining a second feature vector, synthesizing the first feature vector and the second feature vector to generate a third feature vector corresponding to the target bone age, and based on the third feature vector, the target bone age It is possible to generate a target bone image corresponding to .

According to an embodiment of the present invention, it is possible to improve the accuracy of bone age reading by additionally providing a bone image corresponding to the age that is not in the database.

In addition, based on the improved learning method using the processed learning data, it is possible to provide a remarkable effect that can improve the performance of the reading model trained to extract the feature vector from the bone image.

According to the present invention, there is a potential effect of ultimately improving the diagnosis accuracy of medical personnel and providing a reference bone image corresponding to an individual age, thereby innovating a workflow in a medical field.

And, in the present invention, the conventional medical images used in hospitals can be used as they are, so it goes without saying that the method of the present invention is not dependent on a specific type of image or platform.

Also, according to the present invention, a medical image corresponding to a predetermined grade (a medical image of a non-existing grade) may be generated by synthesizing a feature vector extracted from an existing medical image.

The accompanying drawings for use in the description of the embodiments of the present invention are only a part of the embodiments of the present invention, and for those of ordinary skill in the art to which the present invention pertains (hereinafter referred to as "a person skilled in the art"), Other drawings may be obtained based on these drawings without an effort leading to the invention.

1 is a conceptual diagram schematically illustrating an exemplary configuration of a computing device for performing a method for generating a medical image including a bone image according to the present invention.

2 is an exemplary block diagram illustrating hardware or software components of a computing device performing a method for generating a bone image according to the present invention.

3 is a flowchart illustrating a method for generating a bone image according to an embodiment.

4 is a diagram illustrating an example in which a method for generating a bone image according to an embodiment is performed.

5 is a diagram illustrating a distribution of feature vectors according to age.

6 is a diagram illustrating an example of an interface to which a method for generating a bone image according to an embodiment is applied.

7 is an exemplary block diagram illustrating hardware or software components of a computing device performing a method for generating a medical image according to the present invention.

8 is a flowchart illustrating a method of generating a medical image according to an exemplary embodiment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The following detailed description of the present invention refers to the accompanying drawings, which show by way of illustration a specific embodiment in which the present invention may be practiced in order to clarify the objects, technical solutions and advantages of the present invention. These embodiments are described in sufficient detail to enable those skilled in the art to practice the present invention.

The term "image" or "image data" as used throughout the present description and claims refers to multidimensional data composed of discrete image elements (e.g., pixels in a two-dimensional image, voxels in a three-dimensional image). refers to For example, "imaging" means X-ray imaging, (cone-beam) computed tomography, magnetic resonance imaging (MRI), ultrasound or any other medical imaging known in the art. It may be a medical image of a subject, ie, a subject, collected by the system. Also, the image may be provided in a non-medical context, for example, a remote sensing system, an electron microscopy, and the like.

Throughout the description and claims of the present invention, 'image' refers to a viewable image (eg, displayed on a video screen) or an image (eg, a file corresponding to a pixel output of a CT, MRI detector, etc.) A term that refers to digital representations.

Throughout the detailed description and claims of the present invention, the 'DICOM (Digital Imaging and Communications in Medicine)' standard is a generic term for various standards used for digital image representation and communication in medical devices, DICOM Standards are published by a joint committee formed by the American Society of Radiological Medicine (ACR) and the National Electrical Engineers Association (NEMA).

In addition, throughout the detailed description and claims of the present invention, 'picture archiving and communication system (PACS)' is a term that refers to a system that stores, processes, and transmits in accordance with the DICOM standard, X-ray, CT , medical imaging images acquired using digital medical imaging equipment such as MRI are stored in DICOM format and can be transmitted to terminals inside and outside the hospital through a network, to which reading results and medical records can be added.

And throughout the detailed description and claims of the present invention, 'learning' or 'learning' is a term referring to performing machine learning through computing according to a procedure, and a mental action such as human educational activity. It will be understood by those of ordinary skill in the art that this is not intended to be a reference.

And throughout this description and claims, the word 'comprise' and variations thereof are not intended to exclude other technical features, additions, components or steps. In addition, 'one' or 'an' is used to mean more than one, and 'another' is limited to at least a second or more.

Other objects, advantages and characteristics of the present invention will appear to a person skilled in the art, in part from this description, and in part from practice of the present invention. The following illustrations and drawings are provided by way of illustration and are not intended to limit the invention. Accordingly, the details disclosed herein with respect to a specific structure or function should not be construed in a limiting sense, but merely represent a representative providing guidance for those skilled in the art to variously practice the present invention with virtually any suitable detailed structure. should be interpreted as basic data.

Moreover, the invention encompasses all possible combinations of the embodiments indicated herein. It should be understood that various embodiments of the present invention are different but need not be mutually exclusive. For example, certain shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the invention in relation to one embodiment. In addition, it should be understood that the position or arrangement of individual components in each disclosed embodiment may be changed without departing from the spirit and scope of the present invention. Accordingly, the detailed description set forth below is not intended to be taken in a limiting sense, and the scope of the invention, if properly described, is limited only by the appended claims, along with all scope equivalents to those claimed. Like reference numerals in the drawings refer to the same or similar functions throughout the various aspects.

Unless otherwise indicated herein or otherwise clearly contradicted by context, items referred to in the singular encompass the plural unless the context requires otherwise. In addition, in describing the present invention, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present invention, the detailed description thereof will be omitted.

Hereinafter, in order to enable those skilled in the art to easily practice the present invention, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Referring to FIG. 1 , a computing device 100 according to an embodiment of the present invention includes a communication unit 110 and a processor 120 , and is directly or indirectly connected to an external computing device (not shown) through the communication unit 110 . can communicate effectively.

Specifically, the computing device 100 includes typical computer hardware (eg, a computer processor, memory, storage, input device and output device, device that may include other components of a conventional computing device; router, switch, etc.) Electronic communication devices; electronic information storage systems, such as network-attached storage (NAS) and storage area networks (SANs)) and computer software (i.e., that enable the computing device to function in a particular way). instructions) to achieve the desired system performance.

The communication unit 110 of such a computing device may transmit and receive a request and a response to and from another computing device that is interlocked. As an example, such a request and a response may be made by the same transmission control protocol (TCP) session. , but is not limited thereto, and may be transmitted and received as, for example, a user datagram protocol (UDP) datagram. In addition, in a broad sense, the communication unit 110 may include a keyboard, a mouse, other external input devices, printers, displays, and other external output devices for receiving commands or instructions.

In addition, the processor 120 of the computing device may include a micro processing unit (MPU), a central processing unit (CPU), a graphics processing unit (GPU) or a tensor processing unit (TPU), a cache memory, and a data bus. ) may include a hardware configuration such as In addition, it may further include an operating system, a software configuration of an application for performing a specific purpose.

In the following description, the image of the present invention is a bone image, and the bone image is a bone X-ray image as an example. However, the scope of the present invention is not limited thereto, and it is applied to all general types of bone images. A person skilled in the art will readily understand that this is possible and can be applied in the process of reading the bone age of a target image.

In addition, the present invention can be extended to medical images that have been graded in addition to bone images, and in an example of a graded medical image, the degree of progression of diabetic retinopathy for the eye included in the image is graded and labeled A person skilled in the art will understand that the present invention may include a fundus image that has been developed, but the scope of the present invention is not limited thereto, and is applicable to a medical image graded according to any condition.

The individual modules shown in FIG. 2 may be implemented by, for example, the communication unit 110 or the processor 120 included in the computing device 100 or the communication unit 110 and the processor 120 interworking. technicians will be able to understand.

A brief overview of the configuration of the method and apparatus according to the present invention with reference to FIG. 2 , the computing device 100 may include the target bone age information receiving module 210 as a component thereof. The target bone age information receiving module 210 may receive target bone age information from an input device included in the computing device and an interlocked user terminal.

Based on the target bone age information acquired through the target bone age information receiving module 210 , the feature vector acquisition module 220 may acquire a feature vector for generating a bone image corresponding to the target bone age information.

According to an embodiment, the feature vector acquisition module 220 may include a pre-trained first reading model to extract a feature vector from an input bone image and read a bone age based on the extracted feature vector. The first read model is, for example, a Deep Neural Network (DNN), a Convolutional Neural Network (CNN), a Deep Convolution Neural Network (DCNN), a Recurrent Neural Network (RNN), a Restricted Boltzmann Machine (RBM), or a Deep Belief Network (DBN). ), a Single Shot Detector (SSD), You Only Look Once (YOLO), etc. may be used, as will be understood by those skilled in the art. The type of artificial neural network that can be used in the present invention is not limited to the presented example, and it can be learned to extract a feature vector from a bone image based on the labeled learning data, and to read the bone age based on the extracted feature vector. It will be understood by those skilled in the art that it may include any artificial neural network.

According to an embodiment, the first reading model included in the feature vector acquisition module 220 extracts a feature vector for the input bone image based on the training bone image labeled with the corresponding age, and adds the extracted feature vector to the image. It may be learned in advance to read the bone age corresponding to the input bone image based on the input bone image.

In addition, additional learning may be performed on the first reading model based on a synthesized learning bone image generated by synthesizing two or more existing learning bone images. For example, if the pre-existing learning bone image includes bone images corresponding to 1 year old and 3 years old, in addition to the existing bone images corresponding to 1 year old and 3 years old, the bone images corresponding to 1 year and 3 years old are included. Learning of the first reading model may be performed based on the bone image corresponding to the age of 2 generated based on the synthesis of the bone image. A person of ordinary skill in the art knows that a method of generating a synthetic learning goal image based on an existing learning goal image may be performed based on the operations of the synthesizing module 230 and the bone image generation and transmission module 240 to be described below. will understand In addition, the method of generating the synthetic learning bone image is not limited to the example shown, and synthetic learning bone images of various ages may be generated based on the existing learning bone image. For example, through the learning bone images corresponding to 1 year old and 3 years old, synthetic learning bone images corresponding to 1.6 years old, 2.2 years old, 2.5 years old, etc. may be generated, which are the synthesis module 230 and the bone to be described below. It may be performed based on an operation method of the image generation and transmission module 240 .

The feature vector acquisition module 220 may acquire a feature vector for the input valley image through an operation of acquiring an output of an intermediate layer in the first read model as a feature vector.

The feature vector acquisition module 220 may acquire at least two or more candidate bone images corresponding to the target bone age, and extract a feature vector corresponding to the bone image through a pre-learned first reading model. For example, when the target bone age is 4 years old, the feature vector acquisition module acquires a first candidate bone image corresponding to 3 years of age and a second candidate bone image corresponding to 5 years of age, and the first candidate bone image corresponding to each image A feature vector and a second feature vector may be extracted.

According to another embodiment, the feature vector acquisition module 220 may acquire a first feature vector and a second feature vector corresponding to the target bone age from among the databased feature vectors. According to an embodiment, feature vectors corresponding to each age may be databased, and when the target age is 4 years old, the feature vector acquisition module 220 may provide a feature vector corresponding to 3 years old and a feature vector corresponding to 5 years old. can be obtained from the database.

The synthesis module 230 may generate a third feature vector corresponding to the target age by synthesizing the first feature vector and the second feature vector obtained through the feature vector acquisition module 220 . Specifically, the synthesis module 230 may preferentially determine whether to synthesize the first feature vector and the second feature vector based on a predetermined condition. For example, the synthesis module 230 may determine whether to synthesize the first feature vector and the second feature vector based on Equation (1).

L ₁ may represent a first label (age) corresponding to the first feature vector, L ₂ may represent a second label (age) corresponding to the second feature vector, and m may represent a preset threshold value.

m may be a threshold value set in advance for the accuracy of the synthesis result. Specifically, compared to the case of using a feature vector corresponding to 1 year old and a feature vector corresponding to 19 years old to generate a feature vector corresponding to 10 years old, a feature vector corresponding to 9 years old and a feature vector corresponding to 11 years old Using , more accurate results can be obtained. Accordingly, the synthesis module 230 may determine whether to synthesize the feature vector according to a predetermined threshold value.

According to another embodiment, when the interval between the labels of the feature vectors is equal to or less than a threshold value, the synthesis module 230 may determine whether to proceed with the synthesis according to a predetermined probability.

When it is determined to synthesize the feature vector based on Equation 1, the synthesis module 230 may synthesize the feature vector and a label corresponding to the feature vector based on Equation 2 based on Equation 2 .

F ₁ ,F ₂ , F ₃ are a first candidate bone image, a second candidate bone image, and a first feature vector, a second feature vector, and a third feature vector respectively corresponding to the bone image corresponding to the target age, L ₁ , L ₂ and L ₃ denote a first label (age) corresponding to the first feature vector, a second label (age) corresponding to the second feature vector, and a third label (age) corresponding to the third feature vector, respectively , β may be any variable indicating the degree of synthesis, for example, β may be a variable following a beta distribution.

Based on

Equations

2 and 3, a third feature vector and a third label (age) corresponding thereto may be determined.

A person skilled in the art will understand that the method of performing the synthesis by the synthesis module 230 may include any data synthesis method including a method using a linear interpolation or a fully connected layer.

The bone image generation and transmission module 240 may generate a bone image corresponding to the target age based on the third feature vector generated by the synthesis module 230 . More specifically, the bone image generation and transmission module 240 uses the pre-trained second reading model to generate the target bone image corresponding to the above characteristic information by inputting feature information corresponding to the target bone image to be generated. Thus, a target bone image may be generated. For example, the second reading model may include an artificial neural network model such as an Autoencoder or a Generative Adversarial Network (GAN) (eg, CycleGAN, BiBigGAN), based on which a bone image corresponding to the target age will be generated. can The bone image generation and transmission module 240 inputs a third feature vector (for example, a feature vector generated based on the synthesis of feature vectors extracted from a candidate bone image) into the second read model, and the third feature vector is A target bone image (a bone image corresponding to the target age) may be generated.

Additionally, the artificial neural network included in the second reading model may be pre-trained to generate a target bone image corresponding to the feature vector by inputting an initial bone image together with a feature vector corresponding to the target bone image to be generated. The initial bone image may be input as a target bone image, but is not limited thereto, and any bone image may be provided.

The bone image generation and transmission module 240 inputs the third feature vector and the first candidate bone image or the second candidate bone image, which is the basis for the generation of the third feature vector, into the second read model, and adds it to the third feature vector. A corresponding target bone image may be generated. Those skilled in the art will understand that the second reading model trained to generate a target bone image by inputting an initial bone image together with a third feature vector can generate a target bone image with improved quality compared to the case where only the feature vector is used. .

A person skilled in the art can easily understand that the bone image generation and transmission module 240 is not limited to the presented example, and may be implemented through any method for generating a bone image based on a feature vector.

The bone image generation and transmission module 240 may store the generated bone image in a database or provide it to an external entity. When the bone image is provided to an external entity, the bone image generation and transmission module 240 may provide the generated bone image to an external entity using a predetermined display device or the like, or through a communication unit provided. Here, the external entity includes a user, an administrator, and a medical professional in charge of the computing device 100, but it should be understood that any subject that requires a bone image for the target age is included in addition to this. something to do. For example, the external entity may be an external AI device including a separate artificial intelligence (AI) hardware and/or software module utilizing the bone image. In addition, 'external' in the external entity is not intended to exclude an embodiment in which the AI hardware and/or software module using the bone image is integrated into the computing device 100, but the method of the present invention It should be noted that the bone image, which is the result of the performing hardware and/or software module, is used to suggest that it can be utilized as input data of other methods. That is, the external entity may be the computing device 100 itself.

Although the components shown in FIG. 2 are exemplified as being realized in one computing device for convenience of description, it will be understood that a plurality of computing devices 100 performing the method of the present invention may be configured to interwork with each other.

Referring to FIG. 3 , the computing device may receive a target bone age through the communication unit in step 310 .

The computing device may acquire a first candidate bone image and a second candidate bone image for generating a bone image corresponding to the target bone age in operation 320 . If the target bone age is included between the bone age corresponding to the first candidate bone image and the bone age corresponding to the second candidate bone image, the computing device is configured to include the bone age of the first candidate bone image and the bone age of the second candidate bone image. It may be determined whether the interval between ages is smaller than a preset predetermined interval. When the interval between the bone age of the first candidate bone image and the bone age of the second candidate bone image is less than or equal to a predetermined interval, the computing device may perform the subsequent step 330 . If the interval between the bone age of the first candidate goal image and the bone age of the second candidate goal image is greater than a predetermined interval, the computing device is configured to configure the first candidate bone to be equal to or smaller than the predetermined interval. A process of re-acquiring at least one of the image and the second candidate bone image may be performed. This is to improve the accuracy of the calculation result by using the bone image corresponding to the age close to the target bone age in the process of generating the image corresponding to the target bone age, as described above.

The computing device may extract a first feature vector and a second feature vector for each of the first candidate bone image and the second candidate bone image based on the first read model learned in step 330 . As described above, the pre-trained first reading model may be an artificial neural network trained to extract a feature vector from the input bone image and read the age of the input bone image based thereon.

The first reading model may be trained to output a feature vector of an input bone image based on the processed training bone image. A person skilled in the art can easily understand how the first reading model is trained to read the age of the bone image by using the corresponding age-labeled bone image as training data. The first reading model used in the present invention may be learned using a processed learning bone image to improve the accuracy of reading. Specifically, the processed learning goal image may further include a third learning goal image generated by synthesizing the first learning goal image and the second learning goal image obtained from the learning goal image together with the existing learning goal image. can The third learning bone image may correspond to a third bone age corresponding to an age between the first bone age corresponding to the first learning bone image and the second bone age corresponding to the second learning bone image. The processed learning bone image may include all elements corresponding to individual ages compared to existing learning data. The present invention has a remarkable effect in that it is possible to provide a reading result with higher accuracy through a reading model learned using a processed learning bone image.

The computing device may generate a third feature vector corresponding to the target bone age by synthesizing the first feature vector and the second feature vector in operation 340 . The computing device may generate the third feature vector by interpolating the first feature vector and the second feature vector, and the method of generating the third feature vector is not limited to the presented example as described above.

The computing device may generate a target bone image corresponding to the age of the target bone based on the third feature vector in operation 350 .

The system for reading bone age through the image may provide a reference bone image corresponding to an individual age to the reader, and the reader may read the age of the input bone image through comparison with the reference bone image. If the reference bone image 410 corresponding to the age of 3 is absent among the reference bone images stored in the database as shown on the left, it may be difficult for the reader to read the bone age of the corresponding age group. In addition, in the case of an infant with a very rapid development rate, since the difference in bone growth can be large even after a few months, even if the reference bone image corresponding to the age of 1 and the reference bone image corresponding to the age of 2 are used, 1 to 2 years of age Predicting the bone age of infants between the ages of three can be very difficult.

The computing device 420 receives the first candidate bone image 431 and the second candidate bone image 432 corresponding to the two-year-old in order to generate the missing three-year-old reference bone image, and based on this, A bone image 440 corresponding to the age may be generated.

The computing device 420 may generate a bone image corresponding to the target age based on a feature vector previously stored through age-based labeling in a database as well as the suggested method. In this case, in the process of generating the bone image corresponding to the target age, the computing device 420 may generate the bone image only through information on the target bone age without receiving the bone image.

5 is a diagram illustrating a distribution of feature vectors according to age.

Referring to FIG. 5 , a graph 510 may be a distribution of a feature vector for a male bone image, and a graph 520 may be a distribution of a feature vector for a female bone image. Each of the points 511 and 512 included in the

graphs

510 and 520 may correspond to a feature vector of an individual bone image, and the color of the points 511 and 512 may indicate a corresponding age. Referring to

graphs

510 and 520 , it can be seen that points 511 and 512 corresponding to similar ages are distributed in adjacent areas. Considering this distribution, it can be understood that the feature vector calculated through the synthesis of feature vectors of adjacent ages can reveal the characteristics of the target age.

Referring to FIG. 6 ,

reference bone images

621 , 622 , and 623 of a similar age to the current bone image 620 may be provided through the interface 610 . The

reference bone images

621 , 622 , and 623 corresponding to each age may be pre-stored in a database. In the reading process, when a reference bone image between the ages of 10 and 11 is requested, the reader may request to generate a reference bone image corresponding to the corresponding age based on a user input to the graphic object 630 . When a user input is made to the graphic object 630 , the graphic object 640 for inputting the target age may be displayed on the screen. The computing device may generate a reference bone image 650 corresponding to the target age and display it on the screen.

The individual modules shown in FIG. 7 may be implemented by, for example, the communication unit 110 or the processor 120 included in the computing device 100 or the communication unit 110 and the processor 120 interworking. technicians will be able to understand.

A brief overview of the configuration of the method and apparatus according to the present invention with reference to FIG. 7 , the computing device 100 may include a target rating information receiving module 710 as a component thereof. The target rating information receiving module 710 may receive target rating information from an input device included in the computing device and an interlocked user terminal. The grade referred to in this specification may mean a numerical grade for a medical image according to any condition. For example, the fundus image may be classified into a grade 0 corresponding to a normal state to a grade 4 having a maximum degree of progression according to the degree of progression of diabetic retinopathy. In addition, the bone image may be classified according to bone age as described above, and additionally, the grade of the individual bone image may be determined according to the malignancy of the lesion present in the bone image. As such, the grade referred to in this specification may refer to quantified information determined for an individual medical image based on a predetermined condition (eg, age, malignancy of a lesion, degree of progression of a predetermined disease, etc.) Those of ordinary skill in the art will understand. Also, the target grade information may refer to information on a grade of a medical image to be newly created through an existing medical image.

Based on the target grade information acquired through the target grade information receiving module 710 , the feature vector acquisition module 720 may acquire a feature vector for generating a medical image corresponding to the target grade information.

According to an embodiment, the feature vector acquisition module 720 may include a pre-trained first 'reading model to extract a feature vector from an input medical image and read a grade of the medical image based thereon. Those skilled in the art will understand that the first 'reading model may be implemented based on the same kind of artificial neural network as the feature vector acquisition module 720 of FIG. 2 above.

According to an embodiment, the first 'reading model included in the feature vector acquisition module 720 extracts a feature vector for the input medical image based on the training medical image labeled with the corresponding class, and the extracted feature vector It may be learned in advance to read a grade corresponding to the input medical image based on the .

In addition, the first 'reading model may be additionally learned based on the synthesized training medical image generated based on the synthesis between the existing training medical images. For example, if the existing learning medical image includes fundus images corresponding to

grades

1 and 3 of the progression of diabetic retinopathy, the first reading model corresponds to the existing

grades

1 and 3 In addition to the fundus image to be used, learning may be performed based on the fundus image corresponding to the second grade generated based on the synthesis of the fundus images corresponding to the first and third grades. The generation of the synthetic learning medical image using the existing learning medical image may be performed based on the operations of the synthesizing module 730 and the medical image generating and transmitting module 740 , which will be described below. In addition, the method of generating the synthetic learning medical image is not limited to the example shown, and more various grades of the synthetic learning medical image may be generated based on the existing medical learning image. For example, synthetic learning fundus images corresponding to grades 1.6, 2.2, 2.5, and the like may be generated through learning fundus images corresponding to

grades

1 and 3.

The feature vector acquisition module 720 may acquire a feature vector for the input medical image through an operation of acquiring the output of the intermediate layer of the first 'reading model as a feature vector.

The feature vector acquisition module 720 acquires a first candidate medical image and a second candidate medical image for generating a medical image corresponding to the target class, and uses the first 'reading model to obtain the first candidate medical image and the second candidate. A first' feature vector and a second' feature vector corresponding to each of the medical images may be extracted. For example, when target grade information is input as grade 2, which is the degree of progression of diabetic retinopathy with respect to the fundus image, the feature vector acquisition module 720 generates a first candidate medical image of grade 0 corresponding to the normal state ( 1 candidate fundus image) and a grade 3 second medical image (second candidate fundus image) in which diabetic retinopathy has progressed to a certain extent, and a first 'feature vector and a second' feature corresponding to each candidate medical image vector can be extracted. The method of acquiring the candidate fundus image is not limited to the presented exemplary method, and the first candidate fundus image of the first grade and the second candidate fundus image of the fourth grade are used to generate a fundus image corresponding to the target grade of the second grade. It will be understood by those skilled in the art that it can be obtained.

According to another embodiment, the feature vector acquisition module 720 may acquire a first' feature vector and a second' feature vector corresponding to target class information from among the databased feature vectors. According to an embodiment, a feature vector corresponding to each grade may be databased, and when the target grade is grade 2, the feature vector obtaining module 720 may provide a feature vector corresponding to grade 0 and a feature corresponding to grade 3 The vector can be obtained from the database.

The synthesis module 730 may synthesize the first' feature vector and the second' feature vector obtained through the feature vector acquisition module 720 to generate a third' feature vector corresponding to the target class information. Specifically, the synthesis module 730 may preferentially determine whether to synthesize the first' feature vector and the second' feature vector based on a predetermined condition. For example, the synthesis module 730 may determine whether to synthesize the first' feature vector and the second' feature vector based on Equation (4).

L _1' may represent a first label (grade) corresponding to the first' feature vector, L _2' may represent a second' label (grade) corresponding to the second' feature vector, and m may represent a preset threshold. .

m may be a threshold value set in advance for the accuracy of the synthesis result. Specifically, in order to generate a feature vector corresponding to diabetic retinopathy grade 2, a feature vector corresponding to grade 0 corresponding to a fundus image in which diabetic retinopathy does not occur and a feature vector corresponding to grade 4 diabetic retinopathy A result with higher accuracy may be obtained by using a feature vector corresponding to grade 1 diabetic retinopathy and a feature vector corresponding to grade 3 diabetic retinopathy than when using . Accordingly, the synthesis module 730 may determine whether to synthesize the feature vector according to a predetermined threshold value.

According to another embodiment, when the interval between the labels of the feature vectors is less than or equal to a threshold value, the synthesis module 730 may determine whether to proceed with synthesis according to a predetermined probability.

When it is determined to synthesize the feature vector based on Equation 4, the synthesis module 730 may perform synthesis of the feature vector and a label corresponding to the feature vector based on Equations 5 and 6 based on Equation (4).

F _1' ,F _2' and F _3' are the first, second, and third feature vectors, respectively, L _1' ,L _2' and L _3' denote a first label corresponding to the first feature vector, a second label corresponding to the second feature vector, and a third label corresponding to the third feature vector, respectively. , β may be any variable indicating the degree of synthesis, for example, β may be a variable following a beta distribution.

Based on Equations 5 and 6, a 3' feature vector and a 3' label (grade) corresponding thereto may be determined.

A person skilled in the art will understand that the method for performing the synthesis by the synthesis module 730 may include any data synthesis method including a method using a linear interpolation or a fully connected layer.

The medical image generating and transmitting module 740 may generate a medical image corresponding to the target class based on the 3' feature vector generated by the synthesizing module 730 . Specifically, the medical image generation and transmission module 740 uses the pre-trained second 'reading model to generate the target medical image corresponding to the characteristic information by inputting characteristic information corresponding to the target medical image to be generated. Thus, a target medical image may be generated. For example, the second 'reading model may include an artificial neural network model such as an autoencoder or a GAN (eg, CycleGAN, BiBigGAN), and based on this, a medical image corresponding to a target grade may be generated. The medical image generation and transmission module 740 inputs a 3' feature vector (eg, a feature vector generated based on the synthesis of a feature vector obtained from a candidate medical image) into the second' read model to receive a third ' A medical image corresponding to the feature vector (a medical image corresponding to the target class) may be generated.

Additionally, the artificial neural network included in the second 'reading model may be pre-trained to generate a target medical image corresponding to the feature vector by inputting an initial medical image along with a feature vector corresponding to the target medical image to be generated. As the initial medical image, a target medical image may be provided, but the present invention is not limited thereto, and an arbitrary medical image may be provided.

Using the second 'reading model learned by using the initial medical image as an additional input, the medical image generation and transmission module 740 generates a third' feature vector and a first candidate that is a basis for generating the third' feature vector. A target medical image corresponding to the 3' feature vector may be generated from the medical image or the second candidate medical image. The second 'reading model trained to generate a target medical image by inputting an initial medical image along with the 3' feature vector may generate a target medical image of improved quality compared to the case where only the feature vector is used.

A person skilled in the art can easily realize that the medical image generation and transmission module 740 is not limited to the presented example, and can be implemented through any method of generating a target medical image based on the synthesis of feature vectors extracted from a candidate medical image. can be understood

The medical image generation and transmission module 740 may store the generated medical image in a database or provide it to an external entity. When a medical image is provided to an external entity, the medical image generation and transmission module 740 may provide the generated medical image to an external entity using a predetermined display device or the like or through a communication unit provided therein. Here, the meaning of the external entity may be the same as described above with reference to FIG. 2 .

Although the components shown in FIG. 7 are exemplified as being realized in one computing device for convenience of description, it will be understood that a plurality of computing devices 100 performing the method of the present invention may be configured to interwork with each other.

Referring to FIG. 8 , the computing device may receive target rating information in operation 810 . A person skilled in the art will understand that the target grade information corresponds to information on a target grade of a medical image to be newly generated, and may be received from an external entity through a communication unit or may be input through an provided input device.

In operation 820 , the computing device may acquire a first candidate medical image and a second candidate medical image for generating a medical image corresponding to the target class information. When the target grade is included between the grade corresponding to the first candidate medical image and the grade corresponding to the second candidate medical image, the computing device is configured to provide an interval between the grade of the first candidate medical image and the grade of the second candidate medical image. It can be determined whether the interval is smaller than a predetermined interval. When the interval between the grade of the first candidate medical image and the grade of the second candidate medical image is less than or equal to a predetermined interval, the computing device may perform a subsequent operation 830 . If the interval between the grade of the first candidate medical image and the grade of the second candidate medical image is greater than a predetermined interval, the computing device may generate the first candidate medical image and the second medical image so that the interval is less than or equal to the predetermined interval. A process of re-acquiring the second candidate medical image may be performed. This is to improve accuracy of a calculation result by using a medical image corresponding to a grade close to the target grade in the process of generating a medical image corresponding to the target grade.

The computing device may obtain a first' feature vector and a second' feature vector for each of the first candidate medical image and the second candidate medical image, based on the pre-trained first 'reading model in step 830 . have. As described above, the first reading model may be an artificial neural network trained to extract a feature vector from an input medical image and read a grade of the input medical image based thereon.

The first 'reading model is based on the training medical image processed as described above (including the synthesized training image generated based on the synthesis of the existing training image along with the existing training image), the characteristics of the input medical image It may be learned to extract a vector and calculate a grade of an input medical image based on the extracted feature vector. A person skilled in the art can easily understand how the first 'reading model is trained to read the grade of the medical image by using the medical image labeled with the corresponding grade as training data. The first 'reading model used in the present invention may be trained using a processed learning medical image to improve the accuracy of reading. Specifically, the processed learning medical image may further include a third learning medical image generated by synthesizing the first medical learning image and the second learning medical image obtained in the learning medical image together with the existing medical learning image. can The third medical training image may correspond to a third grade corresponding to a grade between a first grade corresponding to the first training medical image and a second grade corresponding to the second training medical image. The processed learning data may include all elements corresponding to individual grades compared to the existing learning data. The present invention may provide a more accurate reading result through the third reading model learned using the processed training data.

The computing device may generate a third' feature vector corresponding to the target class by synthesizing the first' feature vector and the second' feature vector in operation 840 . The computing device may generate a third' feature vector by interpolating the first' feature vector and the second' feature vector, and the method of generating the third' feature vector is based on Equations 4 through Equations 4 to Equation 4 described above. As described in Equation 6.

In operation 850 , the computing device may generate a target medical image corresponding to the target class based on the 3' feature vector.

Based on the description of the above embodiments, those skilled in the art will recognize that the method and/or processes of the present invention, and the steps thereof, may be implemented in hardware, software, or any combination of hardware and software suitable for a particular application. point can be clearly understood. The hardware may include general purpose computers and/or dedicated computing devices or specific computing devices or special features or components of specific computing devices. The processes may be realized by one or more microprocessors, microcontrollers, embedded microcontrollers, programmable digital signal processors or other programmable devices, having internal and/or external memory. Additionally, or alternatively, the processes may be configured to process an application specific integrated circuit (ASIC), programmable gate array, programmable array logic (PAL) or electronic signals. It may be implemented with any other device or combination of devices. Moreover, the objects of the technical solution of the present invention or parts contributing to the prior arts may be implemented in the form of program instructions that can be executed through various computer components and recorded in a machine-readable recording medium. The machine-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the machine-readable recording medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. Examples of the machine-readable recording medium include a hard disk, a magnetic medium such as a floppy disk and a magnetic tape, an optical recording medium such as CD-ROM, DVD, Blu-ray, and a magneto-optical medium such as a floppy disk (magneto-optical media), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include any one of the devices described above, as well as heterogeneous combinations of processors, processor architectures or combinations of different hardware and software, or stored and compiled or interpreted for execution on a machine capable of executing any other program instructions. can be created using a structured programming language such as C, an object-oriented programming language such as C++, or a high-level or low-level programming language This includes not only bytecode, but also high-level language code that can be executed by a computer using an interpreter or the like.

Accordingly, in one aspect according to the present invention, when the method and combinations thereof described above are performed by one or more computing devices, the methods and combinations of methods may be implemented as executable code for performing respective steps. In another aspect, the method may be implemented as systems that perform the steps, the methods may be distributed in various ways across devices or all functions may be integrated into one dedicated, standalone device or other hardware. In another aspect, the means for performing the steps associated with the processes described above may include any of the hardware and/or software described above. All such sequential combinations and combinations are intended to fall within the scope of this disclosure.

For example, the hardware device may be configured to operate as one or more software modules to perform processing in accordance with the present invention, and vice versa. The hardware device may include a processor such as an MPU, CPU, GPU, TPU coupled with a memory such as ROM/RAM for storing program instructions and configured to execute instructions stored in the memory, and an external device and signal It may include a communication unit that can send and receive. In addition, the hardware device may include a keyboard, a mouse, and other external input devices for receiving commands written by developers.

In the above, the present invention has been described with specific matters such as specific components and limited embodiments and drawings, but these are only provided to help a more general understanding of the present invention, and the present invention is not limited to the above embodiments, Those of ordinary skill in the art to which the present invention pertains can make various modifications and variations from these descriptions.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and not only the claims described below, but also all modifications equivalently or equivalently to the claims described below belong to the scope of the spirit of the present invention. will do it

Such equivalent or equivalent modifications shall include, for example, logically equivalent methods capable of producing the same results as practiced by the methods according to the present invention, the spirit and scope of the present invention. should not be limited by the above examples, and should be understood in the broadest sense permitted by law.

Claims

In the method for generating a bone image corresponding to a target age performed by a computing device,

acquiring a first candidate bone image and a second candidate bone image for generating a bone image corresponding to the input target bone age;

extracting a first feature vector and a second feature vector for each of the first candidate bone image and the second candidate bone image based on the pre-trained first read model;

generating a third feature vector corresponding to the target bone age by synthesizing the first feature vector and the second feature vector; and

generating a target bone image corresponding to the target bone age by inputting the third feature vector into a pre-trained second reading model

Containing, bone image generation method.
According to claim 1,

The first reading model is

Based on the training bone image labeled with the corresponding age, extracting a feature vector for the input bone image, and learning to read the bone age corresponding to the input bone image based on the extracted feature vector,

The first reading model is

It is additionally learned based on the synthesized learning goal image in which two or more of the existing learning goal images are synthesized,

The synthetic learning goal image is,

Corresponding to the age that does not exist in the learning bone image, bone image generation method.
According to claim 1,

The step of generating the third feature vector comprises:

The method of generating a bone image, generating the third feature vector by interpolating the first feature vector and the second feature vector.
4. The method of claim 3,

The step of generating the third feature vector comprises:

Through the sum of a first label corresponding to the bone age of the first candidate bone image to which a first weight has been given and a second label corresponding to the bone age of the second candidate bone image to which a second weight has been given determining the first weight and the second weight so that the generated third label corresponds to the target bone age;

and generating the third feature vector based on a sum of the first feature vector to which the first weight is assigned and the second feature vector to which the second weight is assigned.
According to claim 1,

The step of generating the third feature vector comprises:

and an interval between the first bone age of the first candidate bone image and the second bone age of the second candidate bone image is selected to be less than or equal to a predetermined interval.
According to claim 1,

The step of generating the target bone image comprises:

In addition to the third feature vector, the bone image generating method of generating the target bone image based on any one of the first candidate bone image and the second candidate bone image.
A computer program, stored in a machine-readable non-transitory recording medium, comprising instructions implemented to cause a computing device to perform the method of claim 1 .
A method for generating a medical image performed by a computing device, the method comprising:

obtaining a first feature vector and a second feature vector from each of the first candidate medical image and the second candidate medical image;

A third feature vector corresponding to a target class is generated by synthesizing the obtained first feature vector and the second feature vector, wherein the target class is a first class corresponding to the first candidate medical image and the second candidate locating between second grades corresponding to medical images; and

generating a target medical image corresponding to the target class based on the third feature vector;

Including, a method of generating a medical image.
9. The method of claim 8,

The step of generating the third feature vector comprises:

The third label generated based on a weighted sum of a first label corresponding to the grade of the first candidate medical image and a second label corresponding to the grade of the second candidate medical image corresponds to the target grade. determining a first weight corresponding to the first label and a second weight corresponding to the second label;

and generating the third feature vector based on a sum of the first feature vector to which the first weight is assigned and the second feature vector to which the second weight is assigned.
9. The method of claim 8,

The step of generating the third feature vector comprises:

and an interval between the first grade and the second grade is selected to be less than or equal to a predetermined interval.
9. The method of claim 8,

The step of generating the target medical image includes:

and generating the target medical image based on any one of the first candidate medical image and the second candidate image in addition to the third feature vector.
A computer program, stored in a machine-readable non-transitory recording medium, comprising instructions implemented to cause a computing device to perform the method of claim 8 .
A computing device for generating a medical image, comprising:

communication department; and

processor

including,

The processor is

In order to generate a medical image corresponding to the input target class, first and second feature vectors obtained from the obtained first candidate medical image and the second candidate medical image are synthesized, respectively, to generate a first medical image corresponding to the target class. 3 Create a feature vector,

and generating a target medical image corresponding to the target class based on the third feature vector.