KR102063294B1 - Method and apparatus for converting ultrasound image into real image - Google Patents

Abstract

The present invention discloses a method for a computing device to convert an image and a computing device. The method includes obtaining a fetal ultrasound image; Detecting at least one body region included in the fetal ultrasound image using a plurality of layers included in a first network; And synthesizing the characteristic information of the photo image corresponding to each attribute information based on the plurality of attribute information extracted from the body region by using the plurality of layers included in the second network.

Description

METHOD AND APPARATUS FOR CONVERTING ULTRASOUND IMAGE INTO REAL IMAGE}

The present invention relates to a method and an apparatus for converting an ultrasound image into a real image.

Most mothers perform ultrasound tests for medical purposes to check the development of the fetus, deformity, the amount of amniotic fluid, and uterine health. Recently, however, in addition to such medical purposes, ultrasound examinations are increasing to confirm and preserve the shape of the fetus. Accordingly, three-dimensional ultrasound examination that provides a three-dimensional ultrasound image that can more specifically observe the fetus's face, neck, fetal movement has been in the spotlight.

However, such a 3D ultrasound image has only fetal morphological information, so it is difficult to recognize the natural face shape of the fetus, and it is likely that a face shape different from the actual one is provided due to the distortion of the ultrasound image.

In this regard, recently, a technology of scanning a fetal face from a stereoscopic ultrasound image and implementing the same with a 3D printer has been developed. This technology enables the user to have a feeling of seeing a real child by three-dimensionally shaping the face of the fetus. However, this technology does not express the skin color or texture of the fetus, and merely provides the 3D shape of the fetal morphological information obtained from the ultrasound image. Therefore, there is still a need for research to provide a more natural appearance of the fetus.

SUMMARY OF THE INVENTION The present invention has been made in view of the above-mentioned problems of the prior art, and an object thereof is to provide a method for converting an ultrasound image of a fetus into a live image using a neural network. However, the technical problem to be achieved by the present embodiment is not limited to the technical problem as described above, and other technical problems may exist.

As a technical means for achieving the above-described technical problem, the image conversion method according to the first aspect of the present invention, obtaining a fetal ultrasound image; Detecting at least one body region included in the fetal ultrasound image using a plurality of layers included in a first network; And synthesizing the characteristic information of the photo image corresponding to each attribute information based on the plurality of attribute information extracted from the body region by using the plurality of layers included in the second network.

In addition, the computing device according to the second aspect of the present invention, a memory for storing a program for converting the fetal ultrasound image into a live-action image; And a processor for executing the program. In this case, as the program is executed, the processor acquires a fetal ultrasound image, detects at least one body region included in the fetal ultrasound image, using a plurality of layers included in the first network, 2, using the plurality of layers included in the network, the feature information of the photo image corresponding to each attribute information is synthesized based on the plurality of attribute information extracted from the body region.

A third aspect of the invention also provides a computer readable recording medium having recorded thereon a program for executing the method of the first aspect on a computer.

According to the above-described problem solving means of the present invention, the present invention synthesizes the characteristic information of the photo image corresponding to each attribute information on the basis of the morphological attribute information of the ultrasound image of the fetus using a neural network, the actual skin color of the fetus And skin textures can be restored like live action. In addition, by learning the neural network with photo images of newborns, the image images are synthesized to correspond to the probability distribution of the actual newborn face, thereby making it possible to compensate for the distortion of the ultrasound image.

1 illustrates an ultrasound image conversion system according to an embodiment of the present invention.
2 is an example of a neural network according to an embodiment of the present invention.
3 illustrates a second network according to another embodiment of the present invention.
4 shows a second network according to another embodiment of the present invention.
5 shows a configuration of a server according to an embodiment of the present invention.
FIG. 6 is a flowchart illustrating an operation of converting a fetal ultrasound image into a photorealistic image by a processor according to an exemplary embodiment.
7 is a diagram illustrating a processor according to an embodiment of the present invention.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and like reference numerals designate like parts throughout the specification.

Throughout the specification, when a part is "connected" to another part, this includes not only "directly connected" but also "electrically connected" with another element in between. . In addition, when a part is said to "include" a certain component, this means that it may further include other components, except to exclude other components unless otherwise stated.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 shows an ultrasound image conversion system 1 according to an embodiment of the present invention.

Referring to FIG. 1, an ultrasound image conversion system 1 according to an embodiment of the present invention includes a client terminal 20 that provides an ultrasound image of a fetus, and a server 10 that converts an ultrasound image of a fetus into a real image. It includes.

First, the client terminal 20 may be a medical device such as an ultrasound imaging device for capturing a fetus, or may be an electronic device that receives a fetus ultrasound image from an ultrasound imaging device. The client terminal 20 may access the server 10 by using a pre-installed program or the address of the server 10, and provide the fetal ultrasound image captured or stored by the client terminal 20 to the server 10. have. Meanwhile, fetal ultrasound images may be two-dimensional and three-dimensional images.

The server 10 may be a computing device having a processing function of converting a fetal ultrasound image provided from the client terminal 20 into a photorealistic image through the communication network 30. Here, the live-action image may mean an image of restoring the skin color, skin texture, and the like of the fetus.

The server 10 may convert the fetal ultrasound image into a live image using a neural network 11. The neural network 11 may be a set of algorithms that extract various attribute information of the fetal ultrasound image and convert the fetal ultrasound image into a real image based on the extracted attribute information by using the result of statistical machine learning. In addition, the neural network 11 may be implemented by software or an engine or the like for executing the above-described algorithm set. The neural network implemented in software or an engine may be executed by a processor in the server 10.

2 is an example of a neural network 11 according to an embodiment of the present invention.

Referring to FIG. 2, the neural network 11 according to an embodiment of the present invention may include a first network 201 and a second network 202 including a plurality of layers.

First, the first network 201 detects and / or detects a body region of the fetus included in the fetal ultrasound image 12 by abstracting various attributes included in the fetal ultrasound image 12 input to the neural network 11. You can judge. Here, to abstract the attributes in the image may be to detect attribute information from the fetal ultrasound image 12 and to determine a core attribute that can represent the body region of the fetus among the detected attribute information.

In this case, the first network 201 may include an abstraction network 211 including a plurality of layers and at least one region classification layer 212.

The server 10 may extract attribute information of the fetal ultrasound image 12 based on a plurality of layers included in the abstraction network 211 of the first network 201. Here, the attribute information may include, by way of non-limiting example, edge, polygon, sharpness, depth, brightness, contrast, blur, and the like. Can be. The server 10 may obtain a feature map based on attribute information of the fetal ultrasound image 12 extracted from the last layer among the plurality of layers. Here, the feature map may include at least one attribute vector representing the attribute of the fetal ultrasound image 12 as a combination of extracted attribute information.

According to an embodiment, the abstraction network 211 may generate a feature map by combining attribute information of not only the last layer but also another layer (eg, an intermediate layer) in the plurality of layers. Through this, it is possible to more accurately identify the body region in the fetal ultrasound image 12.

The feature map may be applied as input data of at least one area classification layer 212. The server 10 distinguishes the body region from the background region based on a bounding box obtained as a result of inputting the feature map into the at least one region classification layer 212, and / or information about the body region and / or the body. Information about the boundary of the area can be obtained.

The server 10 may apply information (eg, boundary information, area information, etc.) about the body area acquired in the first network 201 as input data of the second network 202.

The second network 202 abstracts various attributes included in the body region of the fetal ultrasound image 12 based on the fetal ultrasound image 12 input to the second network 202, information about the body region, The fetal ultrasound image 12 may be converted into the photo-synthesized image 13 by synthesizing the characteristic information of the photo-image corresponding to each attribute information.

In this case, the second network 202 may include an abstraction network 213 and a composite network 214 including a plurality of layers. Similar to the abstraction network 211 of the first network 201, the server 10 extracts various attribute information from the body region using a plurality of layers included in the abstraction network 213, and extracts the extracted attributes. The feature map can be obtained by combining the information.

The server 10 may apply the feature map extracted from the abstraction network 213 of the second network 202 as input data of the synthesis network 214. The server 10 may convert the body region into a photorealistic image by extracting and composing feature information of a photo image corresponding to each attribute information by using a plurality of layers included in the synthesis network 214. Here, the photographic image is an image of a newborn baby, which may be an image previously stored in a database (not shown) of the server 10, but is not limited thereto. The server 10 may be obtained from an external server or an external database. It may be an image. In addition, the characteristic information of the photo image may include brightness, color, reflection color, texture, depth, blending, and the like.

The server 10 may obtain a final photorealistic image based on the photo composite image 13 extracted from the last layer of the plurality of layers of the composite network 214. For example, the server 10 may perform a post-processing process such as resolution adjustment, size conversion, background region processing, etc. on the photo composite image 13 extracted from the synthesis network 214, thereby obtaining a final photorealistic image. have. In addition, when the photo-composite images of the plurality of body regions are extracted, the server 10 may integrate the photo-composite images of the plurality of body regions into one image.

In addition, the server 10 may include a synthetic network when one body region corresponds to the face of the fetus, and when some of the body elements (eg, eyes, nose, mouth, and ears) in the face are covered by other body regions. The plurality of layers of 214 may be used to restore the corresponding body element based on symmetry of the face.

Meanwhile, according to an exemplary embodiment, the server 10 obtains a photo composite image 13 in which a composite image obtained by combining feature information of a photo image extracted from at least one layer among a plurality of layers of the composite network 214 is combined. can do. In this case, the composite image of each layer may be combined with different weights according to the characteristics of each layer.

In addition, according to an embodiment, the first network 201 may further include a classifier (not shown) capable of determining the category of the body region. In addition, the category of the body region may include, for example, a face, a hand, a foot, a neck, and the like. In addition, categories of body regions may include eyes, nose, mouth, ears, and the like. The server 10 may obtain a category of the body region as a result of inputting the feature map region corresponding to the body region into the classifier (not shown). In addition, the server 10 may adjust parameters of at least one layer among the plurality of layers included in the second network 202 for each category of the body region, or omit and / or add at least one layer in an operation. . Alternatively, the server 10 may process the body region as the background region by not inputting a specific body region (eg, hand) to the second network 202 based on the category of the body region.

Meanwhile, in the above description, for convenience of description, the first network 201 and the second network 202 are described as separate networks, but both networks may be implemented as one integrated network, and the plurality of networks may be further divided. It may be implemented in networks of. Similarly, the abstraction network 213 and the synthetic network 214 of the second network 202 may be implemented in one network, or may be implemented in a plurality of more granular networks.

3 shows a second network 301 according to another embodiment of the present invention.

Referring to FIG. 3, the second network 301 may further include a discriminator network 215 including a plurality of layers, in addition to the abstraction network 213 and the synthesis network 214 of FIG. 2.

In this case, the photo-synthesized image extracted by the synthesis network 214 of FIG. 2 may be applied to the discriminator network 215 as input data. The server 10 may obtain determination information that determines a probability that the corresponding photo composite image corresponds to the actually photographed image based on the photo composite image input to the discriminator network 215. Herein, the actually photographed image may mean a single image obtained by photographing a single subject at one point in time.

The server 10 may feed back the discrimination information (ie, probability value) of the discriminator network 215 to the abstraction network 213 and the synthesis network 214. The server 10 repeatedly feeds back the determination information to at least one of the abstraction network 213 and the synthesis network 214 until the determination information of the discriminator network 215 exceeds a threshold, and based on the determination information, A photographic composite image of the body region may be reacquired. However, the present invention is not limited thereto, and the server 10 may reacquire the photo composite image by repeatedly performing the abstraction network 213 and the synthesis network 214 based on a preset number.

At this time, the server 10 adjusts parameters (eg, weight, etc.) of each layer of the abstraction network 213 and the synthesis network 214 to increase the probability that the photographic composite image is actually determined to be taken. Alternatively, at least one layer of the plurality of layers may be omitted and / or added in an operation, but is not limited thereto.

Through the above-described process, the server 10 may convert the fetal ultrasound image into a live image that is close to the actual captured image.

4 shows a second network 401 according to another embodiment of the present invention.

Referring to FIG. 4, the second network 401 includes a first abstraction network 411, a first synthesis network 412, and a first discriminator network 413 for converting the body region of the fetal ultrasound image into a live-action image. In addition, the apparatus may further include a second abstraction network 414, a second synthesis network 415, and a second discriminator network 416 for converting the photo-realistic image into an ultrasound image. In this case, the first abstraction network 411 and the first synthesis network 412 correspond to the abstraction network 213 and the synthesis network 214 of FIG. 2, and the first discriminator network 413 is the discriminator of FIG. 3. Since it corresponds to the network 215, detailed description thereof will be omitted.

The server 10 may apply the photo composite image extracted from the first composite network 412 as input data of the second abstraction network 414. The server 10 may extract various attribute information from the photo composite image by using the plurality of layers included in the second abstraction network 414 and obtain a feature map from the extracted attribute information. In this case, the server 10 may obtain the extracted feature map based on the attribute information extracted from at least one layer of the plurality of layers of the second abstraction network 414.

In addition, the server 10 may apply the feature map extracted from the second abstraction network 414 as input data of the second synthesis network 415. The server 10 may obtain an ultrasound synthesized image obtained by synthesizing the feature information of the ultrasound image corresponding to each attribute information based on the feature map by using the plurality of layers included in the second synthesis network 415. Here, the ultrasound image may be an actual captured ultrasound image and may be an image previously stored in a database (not shown) of the server 10, and the feature information of the ultrasound image may include edge, polygon, sharpness, brightness, brightness ratio, and the like. Can be.

In this case, the ultrasound composite image extracted by the second composite network 415 may be applied as input data of the second discriminator network 416. The server 10 may obtain second discrimination information that determines a probability corresponding to the actual ultrasound image based on the ultrasound synthesized image input to the second discriminator network 416. The server 10 transmits the second discrimination information to at least one of the first abstraction network 411, the second abstraction network 414, the first synthesis network 412, and the second synthesis network 415. You can feedback. Accordingly, parameters of at least one layer included in the second network 401 may be adjusted, or at least one layer may be omitted and / or added to the operation. In this way, the server 10 may reacquire the photo composite image and the ultrasonic composite image based on the second determination information by using the plurality of layers included in the second network 401.

Thereafter, the server 10 may repeat the second network 401 based on the proximity between the final ultrasound synthesized image extracted from the second synthesized network 415 and the first fetal ultrasound image. Here, the proximity may be obtained as a sum of absolute values for the difference between the two image images. Alternatively, the proximity may be a distance between feature vectors obtained from each image, but is not limited thereto.

As such, the second network 401 according to an embodiment causes the real image generated by the first abstraction network 411 and the first synthesis network 412 to be converted to include the unique information of the initial fetal ultrasound image. As a result, it is possible to prevent a decrease in consistency that may occur during the image conversion process.

5 shows a configuration of a server 10 according to an embodiment of the present invention. The server 10 includes a memory 110 and a processor 120.

The memory 110 may store programs (one or more instructions) for processing and controlling the processor 120. Programs stored in the memory 110 may be divided into a plurality of modules according to functions. According to an embodiment, the memory 110 may store a program for converting a fetal ultrasound image into a live image. The program may include a neural network module.

The neural network module may include a plurality of layers included in the first network and a plurality of layers included in the second network. In addition, the plurality of layers included in the first network may include a plurality of layers included in the first abstraction network extracting attribute information from the fetal ultrasound image, and at least one region classification layer for identifying and / or detecting a body region. It can be divided into. In addition, the plurality of layers included in the second network may include a plurality of layers included in the second abstraction network extracting attribute information from the body region and a plurality of layers included in the first synthesis network synthesizing the characteristic information of the photographic image. It can be divided into layers.

In addition, according to an embodiment, the second network may further include a plurality of layers included in the first discriminator network that determines a probability that the photo-synthesized image extracted from the synthesis network corresponds to the actually photographed image. The plurality of layers included in the second network may include the plurality of layers included in the third abstraction network extracting the feature map based on the attribute information of the photo composite image extracted from the first composite network, and the feature map. It may be divided into a plurality of layers included in the second synthesis network that extracts the ultrasound composite image on the basis.

In this case, each layer included in the first to third abstraction networks may include a convolutional layer including one or more instructions for generating a feature map by extracting attribute information of the image from the input image, and / or the extracted layer. It may include a pooling layer including one or more instructions for determining a representative value from the attribute information. In addition, the region classification layer of the first network may include a convolutional layer and / or a pooling layer including one or more instructions that distinguish the body region from the background region based on the feature map obtained from the abstraction network of the first network. Can be.

In addition, the plurality of layers included in the first and second composite networks may include a deconvolutional layer including one or more instructions for synthesizing the characteristic information of at least one photo image and an ultrasound image corresponding to the attribute information. It may include.

In addition, the plurality of layers included in the first and second discriminator networks may include one or more instruments for determining a probability that the photo composite image and the ultrasonic composite image correspond to the actually captured image and the actually captured ultrasonic image. It may include a convolutional layer and / or a pooling layer.

The processor 120 may include a connection passage (for example, a bus) for transmitting and receiving signals with one or more cores (not shown) and a graphics processor (not shown) and / or other components. .

According to an embodiment, the processor 120 may process one or more instructions included in the first network and the second network in the neural network module in parallel.

Hereinafter, referring to FIG. 6, an operation of the processor 120 converting a fetal ultrasound image into a live image will be described in detail.

First, the processor 120 obtains a fetal ultrasound image (S110).

Thereafter, the processor 120 detects at least one body region included in the fetal ultrasound image by using the plurality of layers included in the first network (S120).

In exemplary embodiments, the processor 120 may extract edge information from the fetal ultrasound image by using a first layer among a plurality of layers included in the first abstraction network. In addition, the processor 120 may extract the polymer from the second layer by applying the extracted edge information as input data to the second layer connected to the first layer. As described above, the processor 120 may extract various attribute information of the image by inputting the fetal ultrasound image to each of the plurality of layers or by applying the attribute information extracted from the previous layer as input data. Subsequently, the processor 120 may acquire the feature map representing the feature of the fetal ultrasound image by combining attribute information extracted using at least one layer. In this case, the processor 120 may further combine the attribute information extracted from another layer in addition to the attribute information extracted from the last layer of the first abstraction network to obtain a feature map. At this time, the attribute information of different layers may be combined with different weights, but is not limited thereto.

In addition, the processor 120 may distinguish the body region from the background region by using at least one region classification layer to identify the body region and / or the boundary of the body region. In this case, the processor 120 may distinguish the body region and the background region based on the boundary box obtained as a result of inputting the feature map into the corresponding layer.

In addition, the processor 120 may obtain a category of the body region as a result of inputting the feature map region corresponding to the body region identified in the classifier included in the first network.

Meanwhile, after step S120, the processor 120 may rotate the body region to correspond to the preset slope based on the identified slope of the body region. The preset slope may be, for example, 0 degrees, 90 degrees, 180 degrees, or the like, but is not limited thereto. Meanwhile, the rotation of the body region may be performed by adding at least one layer for identifying an inclination of the body region and at least one layer for rotating the body region to the first network. However, the present invention is not limited thereto, and the processor 120 may rotate the body region using an additional network, or may rotate the body region using an image rotation function.

Thereafter, the processor 120 extracts one or more attribute information of the body region by using the plurality of layers included in the second network, and generates a photorealistic image based on the extracted attribute information (S130).

In exemplary embodiments, the processor 120 may extract edge information from a body region using a first layer among a plurality of layers included in the second abstraction network. In addition, the processor 120 may extract the polymer from the second layer by applying the extracted color feature to the second layer connected to the first layer as input data. As described above, the processor 120 may obtain a feature map by inputting a body region to each of the plurality of layers or by applying attribute information extracted from a previous layer as input data. In this case, the processor 120 may obtain a feature map by further combining the attribute information extracted from another layer in addition to the attribute information extracted from the last layer of the second abstraction network.

Thereafter, the processor 120 obtains a photo-synthesized image obtained by synthesizing the feature information of the photo-image using the plurality of layers included in the first composite network. For example, the processor 120 may synthesize color information corresponding to the polymer based on the feature map using the first layer of the first synthesis network. In addition, the processor 120 may synthesize texture information corresponding to the polymer from the second layer by applying the composite image extracted from the first layer as input data to the second layer connected to the first layer. As described above, the processor 120 may acquire the composite image by synthesizing the feature information of the photo image corresponding to each attribute information in each of the plurality of layers. In this case, the processor 120 may obtain a final composite image by further combining a composite image extracted from another layer in addition to the composite image extracted from the last layer of the first composite network. In this case, the composite images of the different layers may be combined with different weights, but are not limited thereto.

In addition, the characteristic information of the photographic image may be extracted based on at least one of skin color, nationality, race, sex, and conversion target age (or conversion target months) of the fetus. In exemplary embodiments, the second network may receive, as a parameter, environmental information of at least one fetus such as skin color, nationality, race, gender, etc. of the fetus. In this case, each layer included in the second network may convert the ultrasound image of the fetus into a live image in consideration of the corresponding parameter. In addition, the second network may enable the ultrasound image of the fetus to be converted into a live image based on feature information of photographic images of a target age (ie, a transform target age). To this end, the second network may further receive a conversion target age as a parameter. In this case, the processor 120 may receive at least one parameter input to the second network from the client terminal 20.

Subsequently, the processor 120 may perform post-processing operations such as resolution correction, size conversion, background region processing, and the like, on the photo composite image extracted from the second network. In exemplary embodiments, the processor 120 may apply an interpolation filter, a convolution filter, or the like to the photo composite image. In addition, when the photo composite image of the plurality of body regions is extracted, the processor 120 may integrate the photo composite image of the plurality of body regions into a single image. In this case, the processor 120 may further use an additional network including a plurality of layers for synthesizing an image, but is not limited thereto.

Thereafter, the processor 120 may provide the client terminal 20 with the final photographed composite image having completed the post-processing as a live image. In addition, the processor 120 may convert the newborn face included in the photo image into the final photo composite image based on the photo image including various backgrounds and the newborn face. In this way, the processor 120 may provide a more realistic photorealistic image to the client terminal 20.

Meanwhile, according to an embodiment, when the second network further includes the first discriminator network, the processor 120 may determine the first discrimination information extracted from the first discriminator network (that is, the image in which the photo composite image is actually photographed). In order to increase the probability value corresponding to, the parameters of at least one layer included in the second network may be adjusted, or at least one layer may be omitted and / or added to the calculation.

In addition, when the second network further includes a second abstraction network, a second synthesis network, and a second discriminator network, the processor 120 may determine the second discrimination information extracted from the second discriminator network (that is, the ultrasound synthesized image). In order to increase the probability value corresponding to the actually captured ultrasound image), parameters of at least one layer included in the second network may be adjusted, or at least one layer may be omitted and / or added to the calculation. In addition, the processor 120 may repeat the second network based on the proximity between the ultrasound synthesized image extracted from the second network and the first fetal ultrasound image.

In addition, the second network may include a plurality of layers divided by fetal poses. Here, the pose of the fetus may include, by way of non-limiting example, the face angle, orientation, depth, etc. of the fetus. In this manner, the second network may allow the feature information of the photo image matching the corresponding pose to be synthesized for each fetal pose. In this case, after step S120, the processor 120 identifies a pose of the fetus from the identified body region and uses some layers corresponding to the identified poses among the plurality of layers included in the second network. Feature information of the photographic image corresponding to the attribute information and the pose may be synthesized.

Meanwhile, in steps S120 and S130, the reference for detecting the body region, the reference for synthesizing the characteristic information of the photographic image and / or the ultrasound image, the photographic composite image and / or the ultrasound composite image are actually captured images and / or the actual captured images. A criterion for determining a probability corresponding to the ultrasound image may be determined through learning. For example, the processor 120 may increase the accuracy of body region identification results, photographic image synthesis results, and / or ultrasound image synthesis results, and determination results for each of a plurality of fetal ultrasound images performed before S120. You can learn a plurality of layers. This will be described later in more detail with reference to FIG. 7.

7 is a diagram for describing a processor 120 according to an embodiment of the present invention.

Referring to FIG. 7, the processor 120 may include a data learner 710 and a data recognizer 720.

The data learning unit 710 may include a first criterion for detecting at least one body region included in the fetal ultrasound image, a second criterion for synthesizing the characteristic information of the photographic image, a third criterion for synthesizing the characteristic information of the ultrasound image, A fourth criterion for determining a probability that the photo-synthesized image corresponds to the actually photographed image and a fifth criterion for determining a probability that the ultrasound composite image corresponds to the actually photographed ultrasound image may be learned.

In this case, the data learner 710 may learn the first to fifth criteria independently and / or dependently. For example, the data learner 710 acquires an arbitrary number of ultrasound images and / or photographic images of newborns corresponding to each ultrasound image from a database (not shown) in which a plurality of ultrasound images are stored. It can be learned by inputting to the second network. In this case, the learning may be adjusting parameters (eg, weights, etc.) of each layer included in the first and second networks, or omitting and / or adding at least one layer in an operation.

For example, the data learner 710 may learn the first criterion by inputting an ultrasound image and body region information and / or boundary information of the body region into the first network. In addition, the data learning unit 710 converts an ultrasound image into a live image using a second network, and then inputs a photographic image of newborns corresponding to each ultrasound image to the first discriminator network to learn the fourth criterion. Can be. In addition, the data learner 710 feeds the discrimination information extracted from the first discriminator network to the remaining networks of the second network (that is, the second abstraction network and the first synthetic network) to learn the second criterion. Can be. Since the third and fifth criteria are similar to the process of learning the second and fourth criteria described above, detailed description thereof will be omitted.

As described above, the above-described learning process learns the photographic images of the newborns, so that the data learner 710 synthesizes the characteristic information of the photographic images so as to correspond to the probability distribution between the body elements of the newborns in the data recognition unit 720. In this way, distortion of the ultrasound image may be compensated for.

In addition, when the second network includes a plurality of layers divided by the poses of the fetus, the plurality of layers may be learned by the poses. In addition, when the second network is configured to receive the fetus's environmental information such as skin color, nationality, race, and / or conversion target age (or months) information as parameters, the body elements of newborns corresponding to each environmental information. The second to fifth criteria may be learned to learn a probability distribution between the two.

The data recognizer 720 may perform an operation of converting the fetal ultrasound image into a live image based on the criteria learned by the data learner 710. This has been described above with reference to FIGS. 1 to 6, and thus a detailed description thereof will be omitted.

At least one of the data learner 710 and the data recognizer 720 may be manufactured in the form of at least one hardware chip and mounted on the server 10. For example, at least one of the data learner 710 and the data recognizer 720 may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or an existing general purpose processor (eg, a CPU). Alternatively, the processor may be manufactured as a part of an application processor or a graphics dedicated processor (eg, a GPU) and mounted on the aforementioned various computing devices. In this case, the data learner 710 and the data recognizer 720 may be mounted in one device or may be mounted in separate devices, respectively.

Meanwhile, at least one of the data learner 710 and the data recognizer 720 may be implemented as a software module. When at least one of the data learner 710 and the data recognizer 720 is implemented as a software module (or a program module including instructions), the software module may be computer readable non-transitory readable. It may be stored in a non-transitory computer readable media. In this case, at least one software module may be provided by an operating system (OS) or by a predetermined application. Alternatively, some of the at least one software module may be provided by an operating system (OS), and others may be provided by a predetermined application.

Meanwhile, in the above description, the client terminal and the server have been described separately, but the operations of the server may be performed in the client terminal. In other words, the client terminal may take an ultrasound image of the fetus, and then convert the ultrasound image into a photorealistic image according to the above-described operation, and may include a neural network for this.

One embodiment of the present invention can also be implemented in the form of a recording medium containing instructions executable by a computer, such as a program module executed by the computer. Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. In addition, the computer readable medium may include a computer storage medium. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

Although the systems and methods of the present invention have been described in connection with specific embodiments, some or all of their components or operations may be implemented using a computer system having a general hardware architecture.

The foregoing description of the present invention is intended for illustration, and it will be understood by those skilled in the art that the present invention may be easily modified in other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is shown by the following claims rather than the above description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present invention. do.

1: video conversion system
10: Server 11: Neural Network
12: Fetal Ultrasound Image 13: Photo Composite Image
20: client terminal
30: network
110: memory 120: processor
201: first network 202: second network
110: memory
120: processor
710: data learning unit
720: data recognition unit

Claims (22)

In the image conversion method of the computing device,
Obtaining a fetal ultrasound image;
Detecting at least one body region included in the fetal ultrasound image using a plurality of layers included in a first network; And
Using the plurality of layers included in the second network, the feature information of the photo image corresponding to each attribute information is synthesized based on the plurality of attribute information extracted from the body region, and the body region is converted into a photorealistic image. The second network comprises a second abstraction network and a first synthetic network;
The first network is
A first abstraction network for obtaining a feature map based on attribute information extracted from the fetal ultrasound image, and
Area classification for outputting a feature of the corresponding body region without omitting features including body regions that do not correspond when the image is converted by identifying the boundary of the body region and the category of the body region by applying the feature map as input data. That includes the network,
Wherein the second abstraction network obtains a feature map based on the plurality of attribute information extracted from the body region,
The first synthesis network synthesizes feature information of a photo image corresponding to each attribute information based on the feature map.
Image conversion method. delete The method of claim 1,
The first network is
And a classifier for determining category information of the body region. delete The method of claim 1,
The second network is
Further comprising a first discriminator network for determining the probability that the photo-synthesized image extracted from the first synthesis network corresponds to the actual photographed image,
The image conversion method,
Feeding back the first discrimination information extracted from the first discriminator network to at least one of the second abstraction network and the first synthesized network; And
And reacquiring a photographic composite image using the plurality of layers included in the second network. The method of claim 5,
The image conversion method,
Adjusting a parameter of at least one layer of a plurality of layers included in at least one of the second abstraction network and the first synthesis network, or omitting or adding at least one layer of the plurality of layers in an operation. An image conversion method that includes. The method of claim 1,
The second network is
A third abstraction network extracting a plurality of attribute information from the photo-synthesized image extracted by the first composite network, and obtaining a feature map from the plurality of attribute information;
A second synthesis network for synthesizing an ultrasound image corresponding to each attribute information based on the feature map, and
And a second discriminator network for determining a probability that the ultrasound synthesized image extracted from the second synthesized network corresponds to an actual ultrasound image. The method of claim 7, wherein
The image conversion method,
Feedbacking the second discrimination information extracted from the second discriminator network to at least one of the second abstraction network, the third abstraction network, the first synthesis network, and the second synthesis network. ; And
And reacquiring a photographic composite image and the ultrasound composite image based on the second discrimination information using the plurality of layers included in the second network. The method of claim 8,
The image conversion method
And repeating the second network based on the proximity between the ultrasound composite image and the fetal ultrasound image. The method of claim 1,
The image conversion method,
And after detecting the body region, rotating the body region to correspond to a preset slope based on the slope of the body region. The method of claim 1,
The second network is
Image transformation method comprising a plurality of layers divided by the pose (fetus) of the fetus. The method of claim 11,
The image conversion method
After detecting the body region, further comprising identifying a pose of the fetus from the body region,
Synthesizing the feature information of the photographic image,
And synthesizing the attribute information and feature information of the photo image corresponding to the pose by using some layers corresponding to the pose among the plurality of layers included in the second network. The method of claim 1,
The image conversion method,
Based on a photo image comprising a background and a newborn face, converting the newborn face of the photo image into an image synthesized via the second network. The method of claim 1,
The feature information of the photographic image is extracted based on at least one of the skin color, nationality, race, sex and conversion target age of the fetus. A memory storing a program for converting a fetal ultrasound image into a live-action image; And
A processor for executing the program,
The processor, as the program is executed,
Acquire a fetal ultrasound image,
Detecting at least one body region included in the fetal ultrasound image by using a plurality of layers included in a first network,
Using the plurality of layers included in the second network, the feature information of the photo image corresponding to each attribute information is synthesized based on the plurality of attribute information extracted from the body region, and the body region is converted into a photorealistic image. The second network comprises a second abstraction network and a first synthetic network;
The first network is
A first abstraction network for obtaining a feature map based on attribute information extracted from the fetal ultrasound image, and
Area classification for outputting a feature of the corresponding body region without omitting features including body regions that do not correspond when the image is converted by identifying the boundary of the body region and the category of the body region by applying the feature map as input data. That includes the network,
Wherein the second abstraction network obtains a feature map based on the plurality of attribute information extracted from the body region,
And the first synthesizing network synthesizes feature information of a photo image corresponding to the respective attribute information based on the feature map. delete delete The method of claim 15,
The second network is
Further comprising a first discriminator network for determining the probability that the photo-synthesized image extracted from the first synthesis network corresponds to the actual photographed image,
The processor,
Feeding back the first discrimination information extracted from the first discriminator network to at least one of the second abstraction network and the first synthesized network;
And reacquiring a photographic composite image using the plurality of layers included in the second network. The method of claim 18,
The processor is
Computing parameters of at least one layer among a plurality of layers included in at least one of the second abstraction network and the first synthesis network, or omitting or adding at least one layer of the plurality of layers in an operation. Device. The method of claim 18,
The second network is
A third abstraction network extracting a plurality of attribute information from the photo-synthesized image extracted by the first composite network, and obtaining a feature map from the plurality of attribute information;
A second synthesis network for synthesizing an ultrasound image corresponding to each attribute information based on the feature map, and
And a second discriminator network configured to determine a probability that the ultrasound synthesized image extracted from the second synthesized network corresponds to an actual ultrasound image. The method of claim 15,
The processor,
And after detecting the body region, rotating the body region to correspond to a predetermined slope based on the inclination of the body region. A computer-readable recording medium having recorded thereon a program for executing the method of any one of claims 1, 3, and 5-14.

Images