KR20210032945A - Signal translation system and signal translation method - Google Patents

Abstract

A signal conversion method according to embodiments of the present application may comprise the steps of: receiving a source signal of a first domain; identifying the error features and valid features in the source signal; generating a first virtual signal of a second domain from which at least one first error feature included in the source signal is removed; and outputting the first virtual signal. According to embodiments of the present application, the virtual signal of the second domain from which the error features in the source signal of the first domain are removed may be outputted.

Description

Signal conversion system and signal conversion method {SIGNAL TRANSLATION SYSTEM AND SIGNAL TRANSLATION METHOD}

The present application relates to a signal conversion system and a signal conversion method, and the following embodiments are a signal conversion system and a signal conversion method for generating a virtual signal of a second domain from which error features included in the input signal of the first domain are removed It is about the field.

Machine learning is a field of artificial intelligence that develops algorithms and technologies that enable computers to learn the characteristics of data using multiple data. Machine learning is being applied in various fields, and character recognition is the most representative example.

Machine learning can be classified into supervised learning, unsupervised learning, and reinforcement learning according to the learning method. In supervised learning, a model is trained using a pair of pre-built training input and output data, and in unsupervised learning, the model analyzes or clusters the data itself using only input data. Reinforcement learning learns through feedback, giving appropriate rewards for the outcome of learning performance.

Recently, as a field of machine learning, deep learning technology that attempts a high level of abstraction through a combination of several nonlinear techniques that can process data in a computer is attracting attention. It is being utilized.

The following embodiments aim to generate and provide a virtual signal in consideration of error characteristics included in a source signal through conversion between signals in different domains.

A signal conversion method according to an embodiment of the present application includes the steps of receiving a source signal of a first domain; Identifying error characteristics and valid characteristics in the source signal; Generating a first virtual signal of a second domain from which at least one first error characteristic included in the source signal is removed; And outputting the first virtual signal.

The first virtual signal may be generated using a parameter learned in advance to generate a virtual signal from which error features included in the source signal are removed.

The pre-learned parameter may be continuously updated to improve the quality of the virtual signal.

In addition, the signal conversion method according to another embodiment of the present application further includes the step of determining whether the quality of the first virtual signal corresponds to a predetermined level, wherein the quality of the first virtual signal is predetermined. When it is determined that it meets the level, the first virtual signal may be output.

Inversely transforming the first virtual signal into a first reconstructed signal in a first domain when it is determined that the quality of the first virtual signal does not meet a predetermined level; And generating a second virtual signal by using the first reconstructed signal as an input signal. Further comprising, the steps may be repeatedly performed until the quality of the second virtual signal meets a predetermined level.

The second virtual signal may be obtained by further removing at least one second error feature included in the first restoration signal.

The first virtual signal may be generated by synthesizing features of the destination signal of the second domain with the source signal of the first domain.

The effective feature may be related to a region of interest (ROI) in the source signal preset by a user.

A learning method according to another embodiment of the present application is a learning method using at least one neural network, the method comprising: identifying at least one first error feature from a source signal of a first domain; And calculating a first virtual signal of a second domain from which the first error feature has been removed. Determining whether the quality of the first virtual signal corresponds to a predetermined level, but when it is determined that the quality of the first virtual signal does not meet the predetermined level, the first virtual signal is Inverse transforming into a first reconstructed signal in a first domain; And calculating a first weight using a difference between the first reconstructed signal and the source signal; further comprising, wherein the at least one first error feature included in the source signal is learned through the first weight. Can be.

When it is determined that the quality of the first virtual signal does not meet a predetermined level, generating a second virtual signal of a second domain including the first error characteristic by using the first weight; Inversely transforming the second virtual signal into a second reconstructed signal in a first domain; And adjusting a parameter by using a difference between the second reconstructed signal and the source signal.

Identifying the first error characteristic may include obtaining one or more characteristics from the source signal; And classifying a first feature related to valid features and a second feature related to error features among the one or more features.

The generating of the second virtual signal may include applying the first weight to a first error signal calculated based on the at least one first error characteristic, and applying a second weight to the first virtual signal. Can be.

The second weight may be calculated based on the first weight.

In addition, the method may further include calculating a degree of similarity between the first virtual signal and the destination signal of the second domain, and the neural network may be trained to increase the degree of similarity between the first virtual signal and the destination signal. have.

A learning method according to another embodiment of the present application is a learning method using at least one neural network, the method comprising: acquiring a plurality of features from an input signal of a first domain; Generating a first virtual signal using only a first feature among the plurality of features, and generating a second virtual signal using all of the plurality of features; And calculating a similarity between the first virtual signal and the destination signal of the second domain. And adjusting a parameter applied to the network by using a difference between a first reconstructed signal obtained by inversely transforming the second virtual signal into a signal of a first domain and the source signal.

The steps may be repeatedly performed until the quality of the first virtual signal reaches a predetermined level.

In addition, the neural network may be trained to increase the similarity between the first virtual signal and the destination signal of the second domain.

In this case, the similarity may be a similarity between qualitative features between the first virtual signal and the destination signal.

In addition, the computer-readable medium according to another embodiment of the present application may be a recording program for executing the signal processing method according to the embodiments of the present application on a computer.

According to embodiments of the present application, a virtual signal of the second domain from which error features in the input signal of the first domain have been removed may be output.

1 is a diagram schematically illustrating an entire environment of a system for providing signal conversion according to an exemplary embodiment of the present application.
2 is a block diagram schematically illustrating a signal conversion process provided by a signal conversion providing system according to an embodiment of the present application.
3 is a block diagram illustrating an exemplary configuration of a signal conversion module according to an embodiment of the present application.
4 is a flowchart illustrating a signal conversion method according to the first embodiment of the present application by way of example.
5 is a flowchart illustrating a signal conversion method according to a second embodiment of the present application by way of example.
6 is a diagram illustrating a signal conversion method according to a second embodiment of the present application by way of example.
7 and 8 are diagrams for illustratively explaining a signal conversion learning process performed by one or more neural networks (NN) according to an embodiment of the present application.
9 is a diagram schematically illustrating a signal conversion learning process according to an embodiment of the present application.
10 is a flowchart illustrating a learning method according to a fourth embodiment of the present application by way of example.
11 is a diagram for illustratively explaining a learning method according to a fifth embodiment of the present application.
12 is a flowchart illustrating a learning method according to a fifth embodiment of the present application by way of example.

The above objects, features, and advantages of the present invention will become more apparent through the following detailed description in conjunction with the accompanying drawings. However, in the present invention, various modifications may be made and various embodiments may be provided. Hereinafter, specific embodiments will be illustrated in the drawings and described in detail.

In the drawings, the thicknesses of layers and regions are exaggerated for clarity, and elements or layers are "on" or "on" of other elements or layers. What is referred to includes not only directly above another component or layer, but also a case in which another layer or other component is interposed in the middle. The same reference numerals throughout the specification refer to the same elements in principle. In addition, components having the same function within the scope of the same idea shown in the drawings of each embodiment will be described using the same reference numerals.

If it is determined that a detailed description of known functions or configurations related to the present invention may unnecessarily obscure the subject matter of the present invention, a detailed description thereof will be omitted. In addition, numbers (eg, first, second, etc.) used in the description of the present specification are merely identification symbols for distinguishing one component from other components.

In addition, the suffixes "module" and "unit" for constituent elements used in the following description are given or used interchangeably in consideration of only the ease of preparation of the specification, and do not have meanings or roles that are distinguished from each other by themselves.

Hereinafter, exemplary embodiments related to signal conversion between various signals in different domains will be described.

The signal conversion according to the embodiments of the present application may be related to signal conversion between an input signal of the first domain and a virtual signal of the second domain.

In the present specification, a signal may include, for example, an image, an audio, and 3D information. In addition, in the present specification, the error feature may be any feature that the user does not want included in the input signal.

As an example of voice signal conversion, a male voice in a first domain may be converted into a female voice in a second domain. In this case, noise and other voices may be mixed with the male voice of the first domain, and the converted virtual signal may be obtained by removing noise and other voices included in the signal of the first domain.

As another example, in the case of natural language signal conversion, a Korean sentence in the first domain may be converted into an English sentence in the second domain. In this case, the Korean sentence of the first domain may contain typos and incorrect words, and the converted virtual signal may be obtained by removing the typos and incorrect words included in the signal of the first domain.

As an example of sound signal conversion, classical music in the first domain may be converted into jazz music in the second domain. In this case, noise and human voice (not vocals) may be mixed with classical music of the first domain, and the converted virtual signal may be obtained by removing noise and human voice included in the signal of the first domain.

As another example, in the case of 3D signal conversion, a 3D voxel in the first domain may be converted into a 2D image in the second domain. In this case, an error characteristic may be included in the 3D voxel of the first domain, and the converted virtual signal may be obtained by removing the error characteristic included in the signal of the first domain.

Also, as an example of image conversion, an ultrasound image in the first domain may be converted into a photographic image in the second domain. In this case, the ultrasound image of the first domain may include errors and features that the user does not want, and the converted virtual image may be obtained by removing errors and unwanted features included in the signal of the first domain.

The above-described embodiments of signal conversion are described for illustrative purposes, and the present invention is not limited to the above-described embodiments, and in signal conversion between signals in different domains, the source signal of the first domain including the error signal It can be used in all fields for converting to a virtual signal of the second domain from which the error signal has been removed.

Hereinafter, for convenience of explanation, an image conversion for outputting a virtual image of a second domain by using an image of the first domain as an input signal will be described.

In this case, the input image may be an image acquired through various imaging apparatuses.

For example, the input image may be a medical image acquired through an X-ray, computed tomography (CT), magnetic resonance imaging (MRI), an ultrasound imaging device, or the like.

Alternatively, for example, the input image may be a photographic image captured through various types of camera devices.

In addition, the virtual image may be an image of the second domain including at least one ROI of the user in the input image of the first domain.

For example, the virtual image may be an image of the second domain from which at least one artifact included in the input image of the first domain has been removed.

The artifact may be, for example, various types of imaging errors included in the input image. Alternatively, the artifact may be features that the user does not want. The user can define unwanted features included in the input image as artifacts.

1. Signal Conversion Provision System

Hereinafter, the entire environment of the signal conversion providing system 1 for providing a virtual signal converted through various signal conversions described above with reference to FIGS. 1 and 2 will be schematically described.

The signal conversion providing system 1 according to an embodiment of the present application converts the source signal of the first domain input from the user device 30 into a virtual signal of the second domain and provides it through the corresponding user device 30 It may be a system for doing.

In addition, the signal conversion providing system 1 may be a system that has been learned in advance to provide a virtual signal from which error features in the source signal have been removed.

1 is a diagram schematically illustrating an entire environment of a system for providing signal conversion according to an exemplary embodiment of the present application. The signal conversion providing system 1 according to the exemplary embodiment of the present application may include a server 10, a database 20, a user device 30, and the like, and the server 10, the database 20 ), the user device 30 and the like may be connected through the communication network 2. In addition, the signal conversion providing system 1 may include one or more servers 10, a database 20, and a user device 30.

The server 10 may be a device for converting and providing a source signal of the first domain received from the user device 30 into a virtual signal of the second domain.

For example, the server 10 is based on a learning module (M) that performs learning to generate a virtual signal from which error features in the source signal are removed, and a predetermined criterion obtained through the learning module (M). Thus, it may include a signal conversion module (T) or the like that performs a signal conversion operation.

The learning module (M) and the signal conversion module (T) are exemplarily described for convenience of description, and the server 10 may provide all functions by integrating through one module, and provide other virtual signals. All functions necessary to create and provide can be provided. Detailed operations performed by the learning module M and the signal conversion module T will be described in a related section below.

In addition, for example, the server 10 may include a memory, one processor that performs signal processing, or a plurality of processors.

A program for signal processing may be stored in the memory (not shown), and the program may include one or more modules.

One or more machine learning algorithms for performing machine learning may be provided to the processor (not shown). Specifically, various machine learning models may be used for signal processing according to an exemplary embodiment of the present application, and for example, a deep learning model may be used.

Deep learning is a set of algorithms that attempt high-level abstractions through a combination of several nonlinear transformation methods. As a core model of deep learning, a deep neural network (DNN) can be used. A deep neural network (DNN) includes several hidden layers between an input layer and an output layer. Deep Auto encoder, Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Generative Adversarial Network (GAN), etc. may be used. Here, learning is to understand the characteristics of the data for a given purpose, and in deep learning, the connection weight is adjusted. For example, in the case of a convolutional neural network (CNN) that can be applied to learning two-dimensional data such as images, one or several convolutional layers and pooling layers, and fully connected layers It can be composed of (fully connected layers) and can be trained through a backpropagation algorithm.

For example, an input signal of the first domain received from the user device 30 through the processor may be converted into a virtual signal of the second domain using one or more machine learning algorithms described above. Meanwhile, the server 10 may perform only a signal conversion operation based on a predetermined reference transmitted through the learning module M located outside. Alternatively, the server 10 may perform only the signal conversion learning operation and provide a predetermined reference calculated as a result of the learning to the external signal conversion module T. Alternatively, the server 10 may provide only the virtual signals converted from the externally located learning module M and the signal conversion module T to the user.

In the following, for convenience of explanation, both the learning module (M) and the signal conversion module (T) are located in the server 10, and the neural network described above in the server 10, extensions thereof, or other deep learning approaches are used. Thus, a case of performing signal conversion will be described as an example. In this case, the converted virtual signal may be a signal from which at least one error feature included in the input signal has been removed.

The database 20 is configured to store various types of data and may include one or more memories.

* For example, the database 20 may store one or more signals received from the user device 30 and may store various types of data generated during a signal conversion process.

In addition, for example, the database 20 may store one or more signals obtained from an external server or an external database, for example, image, voice, characteristic information of 3D information, and the like.

Meanwhile, the database 20 may be located in the server 10 described above, or may be located separately from the server 10.

The user device 30 may be an electronic device for acquiring an image, a voice, and 3D information of a specific object. For example, the user device 30 may be a camera, X-ray, CT, MRI, ultrasound imaging device, computer, portable terminal, or the like. Alternatively, the user device 30 may be an electronic device such as a computer or a portable terminal that receives a signal generated from an external device. For example, the computer may include a desktop, a notebook, and the like. Or, for example, the portable terminal may include a tablet, a smart phone, or the like.

For example, when the external device is an ultrasound imaging device, the user device 30 may be a terminal in a hospital receiving an image captured from the ultrasound imaging device.

The user device 30 may access the server 10 through a program installed on the user device 30, a website provided on the user device 30, an application, or the like.

For example, the user device 30 may provide at least one of voice, natural language, music, and image data previously stored in the user device 30 as an input signal to the server 10. In addition, for example, the user device 30 may provide at least one of voice, natural language, music, image data, etc. acquired in real time through the user device 30 to the server 10 as an input signal. have.

The communication network 2 may be a wired or wireless communication network of various types for connecting the server 10 and the user device 30. Alternatively, the communication network 2 may be a local area network or a wide area network. For example, it may be wireless communication such as wired communication through USB, Wi-Fi, Wibro, Bluetooth, RF, infrared communication (IrDA), and the like. The communication network 2 is not limited to the above-described example.

2. Signal conversion process

2 is a block diagram schematically illustrating a signal conversion process provided by the signal conversion providing system 1 according to an embodiment of the present application.

For example, the signal conversion process according to the exemplary embodiment of the present application may be performed through the signal conversion module T located inside the server 10 described above.

At this time, one or more machine learning algorithms may be provided to the signal conversion module T. For example, as illustrated in FIG. 2, the signal conversion module T may perform signal conversion using at least one neural network (NN). Detailed operations performed by the signal conversion module T will be described with reference to related embodiments below.

Referring to FIG. 2, the source signal X of the first domain received from the user device 30 described above may be applied to the neural network NN as input data. That is, when the ultrasound image X received from the user device 30 is input to the neural network NN, the actual photo image Yv of the second domain converted using the neural network NN is It may be output on the user device 30.

For example, as illustrated in FIG. 2, the actual photographic image Yv may be a virtual image of the second domain from which error features such as a fetus's hand in the ultrasound image X have been removed. In this case, the signal conversion module T may generate a virtual image of the second domain from which error features included in the source image X of the first domain have been removed according to a preset criterion. For example, the preset criterion may be a parameter obtained as a result of machine learning, and the parameter may be continuously updated.

Hereinafter, exemplary embodiments of a signal conversion method performed in the signal conversion module T will be described in detail with reference to FIGS. 3 to 6.

3 is a block diagram illustrating an exemplary configuration of a signal conversion module T according to an exemplary embodiment of the present application. For example, the signal conversion module T may include a signal transmission/reception unit 11, a feature analysis unit 12, a virtual signal generation unit 13, an inverse signal conversion unit 14, and a control unit 15. . Each component is described as an example, and the signal conversion module T may include one or more of the above-described components, and operations performed by each component may be integrated into one component and provided. In addition, operations performed by each component may be performed sequentially or in parallel as necessary.

Hereinafter, it is assumed that the source signal of the first domain is an ultrasound image of the fetus, and the destination signal of the second domain is a real photo image of a newborn baby. Let's explain with an example.

In this case, the virtual signal of the second domain may be an image obtained by synthesizing features of the actual photographic image of the newborn with the ultrasound image of the fetus, and at least one artifact included in the ultrasound image of the fetus has been removed. It may be a photographic image of.

Here, the artifact may be, for example, various types of imaging errors included in the ultrasound image of the fetus, or any feature other than a face region of the fetus.

Hereinafter, a face region of the fetus is defined as a region of interest (ROI) of a user, and a region including at least one artificial object in the ultrasound image of the fetus is defined as an error region.

In addition, the ultrasound image of the fetus may be a two-dimensional or three-dimensional image of the fetus captured through an ultrasound imaging device, and the actual photographic image of the newborn may be a two-dimensional image of the newborn. Meanwhile, in the signal processing according to the exemplary embodiment of the present application, the fetus in the ultrasound image and the newborn baby in the actual photograph image may not coincide with each other.

The signal transmission/reception unit 11 receives the source signal X of the first domain from the user and transmits the virtual signal of the second domain generated by the virtual signal generation unit 13 to be described later on the user device 30 It can be a configuration for.

For example, when the source signal X of the first domain is an ultrasound image of a fetus, the source signal X may be an image received in real time from an ultrasound imaging apparatus. Alternatively, the source signal X may be an image selected by the user from among images stored in the user's portable terminal.

The feature analysis unit 12 may be a component for extracting a plurality of features in the source signal X of the first domain obtained through the signal transmission/reception unit 11.

For example, the feature analysis unit 12 may extract a plurality of features in the source signal X using at least one neural network. For example, the network may be a convolutional network and may include a plurality of layers. Accordingly, the feature analysis unit 12 may extract a plurality of features included in the source signal X by using the plurality of layers.

In addition, the feature analysis unit 12 may classify the plurality of features into valid features related to the ROI and/or error features related to an error area according to a preset criterion. The preset criterion may be, for example, a criterion previously input from a user, and may be updated according to a result previously learned through the learning module M.

For example, the feature analysis unit 12 may identify a region of interest (ROI) and an error region in the source signal through one or more feature maps (FM). In this case, the feature map may be a combination of the plurality of features.

For example, when the source signal X is an ultrasound image of a fetus, the region of interest may be a face region of the fetus, and the error region may be a body region other than a face region of the fetus. For example, the face area may include the eyes, nose, mouth, and ears of the fetus. For example, the error region may include a fetus's hand, foot, placenta, amniotic fluid, and the like.

Additionally, the feature analysis unit 12 may extract features of the destination signal of the second domain. In this case, the method of extracting features of the destination signal may correspond to a method of extracting features of the source signal.

In addition, optionally, the signal conversion module T may provide a list of valid features and/or error features analyzed by the feature analysis unit 12 through the user device 30. For example, the user may change attribute information by selecting one of the features of the list. For example, the attribute information may include skin color, hair color, and the like.

The virtual signal generation unit 13 may be a component for converting a source signal of a first domain into a first virtual signal of a second domain by using one or more features obtained from the above-described feature analysis unit 12.

For example, the virtual signal generator 13 may convert the source signal into a first virtual signal of a second domain using at least one neural network. For example, the network may include a plurality of layers, and the virtual signal generator 13 may generate the first virtual signal by applying a pre-learned parameter to the plurality of layers.

The pre-learned parameter may be pre-learned by the learning module M to generate a first virtual signal having a quality of a certain level or higher. The process of calculating the parameter will be described later in the section on the learning method performed in the learning module M below.

In addition, for example, the virtual signal generation unit 13 may recombine characteristics of a destination signal necessary for signal conversion based on valid characteristics of the source signal, and a destination recombined to the valid characteristics of the source signal. The characteristics of the signal may be synthesized and converted into a first virtual signal of the second domain. In this case, the first virtual signal converted by the virtual signal generation unit 13 may be obtained by removing at least one error characteristic in the source signal.

For example, the characteristics of the destination signal may include brightness, color, reflection color, texture, depth, blending, shape or combination of shapes of the image.

Meanwhile, the destination signal and/or the characteristics of the destination signal may be obtained from the above-described feature analysis unit 12, the database 20, a memory, or an external server. Alternatively, the virtual signal generation unit 13 may extract features of the destination signal using the neural network, and the process of extracting the features of the destination is performed by the above-described feature analysis unit 12 It may correspond to a method of extracting features in the signal.

For example, when the source signal is an ultrasound image of a fetus, error features included in the ultrasound image of the fetus may be identified according to a preset criterion, and the error features are removed through the virtual signal generator 13. Virtual images can be created. For example, the virtual image generated by the virtual signal generator 13 may be an actual photographic image of the second domain from which error features such as hands, feet, placenta, etc. of the fetus in the ultrasound image have been removed.

As another example, when a user requests to change attribute information by selecting one of the valid features and/or features of the valid feature list provided on the user device 30, the virtual signal generator 13 A virtual signal can be generated by reflecting the change request.

Additionally, the virtual signal generation unit 13 may generate a second virtual signal by using the reconstructed signal converted through the signal inverse conversion unit 14 to be described later as an input signal. The method of generating the second virtual signal by using the restored signal as an input signal may correspond to the method of generating the first virtual signal described above.

The signal inverse transform unit 14 may be a component for inversely transforming the first virtual signal of the second domain generated by the above-described virtual signal generator 13 into a reconstructed signal of the first domain.

For example, the signal inverse transform unit 14 may inversely transform the first virtual signal into a reconstructed signal of a first domain using at least one neural network.

In this case, the process of converting the first virtual signal into a reconstructed signal of the first domain may correspond to a signal conversion operation performed by the above-described virtual signal generator 13.

For example, the neural network may include two sub-networks, and the signal inverse transform unit 14 extracts a plurality of features in the first virtual signal using a plurality of layers included in the first sub-network. can do. In this case, the process of extracting a plurality of features by the signal inverse transform unit 14 using the first sub-network may correspond to an operation performed by the feature analysis unit 12 described above. In this case, the process of generating the reconstructed signal by the signal inverse transform unit 14 may correspond to the operation performed by the above-described virtual signal generation unit 13.

That is, the signal inverse transform unit 14 may generate a reconstructed signal by mapping features of the source signal of the first domain to the plurality of features using a plurality of layers included in the second sub-network.

Meanwhile, since the first virtual signal may have at least one error feature removed from the source signal, the reconstructed signal generated through the signal inverse transform unit 14 may also have the at least one error feature removed. .

For example, the control unit 15 to be described later repeatedly performs signal conversion and inverse conversion through the above-described virtual signal generation unit 13 and signal inverse conversion unit 14 until the quality of the virtual signal meets a predetermined level. By controlling to be performed, error features included in the source signal can be gradually/stepwise removed. A detailed description of the signal conversion and inverse conversion process will be described in detail through the following related embodiments.

The control unit 15 may be a configuration for controlling all operations performed by the signal conversion module T to generate a virtual signal of a second domain with improved quality by identifying error signals included in a source signal input from a user. have.

For example, the control unit 15 may obtain the optimum parameter calculated from the learning module M, and the parameter may be applied to the above-described feature analysis unit 12, a virtual signal generation unit 13, and/or a signal. It can be transmitted to the inverse transform unit 14.

In addition, for example, the control unit 15 outputs the virtual image of the second domain finally converted through the virtual signal generation unit 13 on the user device 30 through the signal transmission/reception unit 11. Can be controlled.

Also, for example, the control unit 15 may further determine whether the quality of the virtual signal generated through the virtual signal generation unit 13 meets a predetermined level.

Here, the predetermined level may be a reference previously input by the user, and for example, the user may set the operation by the virtual signal generator 13 and the signal inverse transform unit 14 to be repeated more than a predetermined number of times. The quality of the virtual signal may be determined according to various criteria according to the type of signal, and a detailed description thereof will be described in detail through the following embodiments.

For example, when it is determined that the quality of the virtual signal corresponds to a predetermined level, the control unit 15 controls to output the virtual image on the user device 30 through the signal transmission/reception unit 11. I can.

Alternatively, when it is determined that the quality of the virtual signal does not meet a predetermined level, the virtual signal may be transmitted to the signal inverse transform unit 14 described above.

That is, the controller 15 repeats the signal conversion process and the signal inverse conversion process until the quality of the virtual signal generated by the virtual signal generator 13 meets a predetermined level. 13) And by controlling the operation of the signal inverse transform unit 14, it is possible to remove the error features included in the source signal.

Hereinafter, various embodiments of the signal conversion method performed by the signal conversion module T described above will be described with reference to FIGS. 4 and 5.

4 is a flowchart illustrating a signal conversion method according to the first embodiment of the present application by way of example. Referring to FIG. 4, the signal conversion method according to the first embodiment of the present application includes receiving a source signal of a first domain (S41), identifying error characteristics and valid characteristics in the source signal (S42), Generating a first virtual signal of a second domain from which at least one first error characteristic included in the source signal has been removed (S43), and outputting the first virtual signal (S45).

Hereinafter, it is assumed that the source signal of the first domain is an ultrasound image (X) of the fetus, and the virtual signal of the second domain is an image (Yv) in which features of the actual photographic image of the newborn are combined with the ultrasound image of the fetus It will be described in detail.

The signal conversion module T may receive a source signal of the first domain (S41).

As described with reference to FIG. 3, the signal conversion module T may receive a source signal X of the first domain from a user through the signal transmission/reception unit 11. For example, the signal conversion module T may receive an ultrasound image of a fetus from an ultrasound imaging apparatus located in a hospital.

The signal conversion module T may identify error characteristics and valid characteristics in the source signal (S42).

For example, the signal conversion module T may extract a plurality of features included in the source signal through the feature analysis unit 12 as described above. In addition, for example, the feature analysis unit 12 may classify the plurality of features into valid features and/or error features according to a preset criterion.

The signal conversion module T may generate a first virtual signal of a second domain from which at least one first error characteristic included in the source signal has been removed (S43).

For example, the signal conversion module T may convert the source signal of the first domain into a first virtual signal of the second domain, excluding at least one first error characteristic identified in step S42. For example, the at least one first error feature may be related to a hand, foot, placenta, amniotic fluid, etc. included in an ultrasound image of a fetus.

Also, for example, the signal conversion module T may generate the first virtual signal using a parameter learned in advance. In this case, the pre-learned parameter may be continuously updated to improve the quality of the first virtual signal.

For example, the virtual signal generator 13 excludes the first error features identified in step S42 among a plurality of features included in the ultrasound image of the fetus, and features related to the fetal face area (eg, face, Based on the eyes, nose, mouth, ears, etc.), the characteristics of the newborn's actual photographic image (e.g., brightness, color, reflection color, texture, depth, blending, shape or combination of shapes, etc.) required for signal conversion Can be recombined. In this case, the virtual signal generator 13 may generate a first virtual signal by synthesizing features of the real photographic image recombined with the features of the face area of the womb.

The signal conversion module T may output the first virtual signal on the user device (S45).

For example, the signal conversion module T may transmit the first virtual signal generated by the virtual signal generation unit 13 through the signal transmission/reception unit 11 described above. In this case, the first virtual image output on the user device may be an image with improved quality from which error features in the ultrasound image of the fetus are removed.

Meanwhile, the signal conversion module T may provide a virtual signal with improved quality to a user by repeatedly performing the above-described signal conversion operation at least two or more times. In this case, the criteria for repeatedly performing the signal conversion operation in the signal conversion module T may be various.

According to the signal conversion method according to the second embodiment of the present application, the quality is improved by repeatedly performing the signal conversion process until the quality of the first virtual signal generated through the above-described steps S41 to S43 meets a predetermined level. It is possible to output an improved virtual signal.

That is, the signal conversion method according to the second embodiment of the present application determines whether the quality of the virtual signal meets a predetermined level, and only when it is determined that the quality of the first virtual signal meets the predetermined level. The first virtual signal may be output through the user device 30.

In other words, according to the signal conversion method according to the second embodiment of the present application, the step of checking the quality of the first virtual signal before outputting the first virtual signal generated through the step S43 on the user device 30 is performed. By further coarse, it is possible to provide a virtual signal of a predetermined quality or higher to the user.

5 is a flowchart illustrating a signal conversion method according to a second embodiment of the present application by way of example. Referring to FIG. 5, in the signal conversion method according to the second embodiment of the present application, the step of determining whether the quality of a first virtual signal corresponds to a predetermined level (S44), and the quality of the first virtual signal is If it is determined that it does not meet a predetermined level, inversely converting the first virtual signal into a first reconstructed signal in a first domain (S46), and generating a second virtual signal by using the first reconstructed signal as an input signal. It may further include a step (S48).

The signal conversion module T may determine whether the quality of the first virtual signal meets a predetermined level (S44).

That is, the above-described control unit 15 may determine whether the quality of the first virtual signal generated in step S43 corresponds to a predetermined level. In this case, whether the quality of the first virtual signal meets a predetermined level may be determined based on a similarity between the first virtual signal and a destination signal of the second domain.

Here, the similarity may mean similarity between the qualitative features of the virtual signal and the destination signal. For example, when the source signal is an ultrasound image of a fetus, the similarity may be expressed as a probability that the actual photographic image of the fetus generated through the virtual signal generator 13 corresponds to the actual photographic image of the newborn. have.

For example, the controller 15 may calculate a probability that the first virtual image corresponds to a destination image, and when the probability that the first virtual image corresponds to a destination image is greater than or equal to a predetermined value, the first virtual image It can be determined that the quality of the signal corresponds to a predetermined level. Accordingly, the controller 15 may control to output the first virtual image on the user device 30.

As another example, the controller 15 may calculate a probability that the first virtual image corresponds to a destination image, and when the probability that the first virtual image corresponds to a destination image is less than a predetermined value, the first virtual image It may be determined that the quality of the signal does not meet a predetermined level.

In this case, the signal conversion module T may inversely convert the first virtual signal into a first reconstructed signal in the first domain (S46).

As described above, the signal inverse transform unit 14 may generate a first reconstructed signal of the first domain by mapping features of the source signal of the first domain to a plurality of features included in the first virtual signal. . In this case, the first recovery signal may be obtained by removing the aforementioned first error feature.

As an example, the source signal of the first domain is an ultrasound image of a fetus, and the ultrasound image of the fetus includes effective features related to facial areas such as eyes, nose, mouth, and ears of the fetus, and error areas such as hands, feet, amniotic fluid, and placenta. Error features related to may be included. In this case, when the first error characteristic is a characteristic related to the placenta in the ultrasound image, the first virtual signal may be an actual photographic image of the second domain from which the characteristic related to the placenta in the ultrasound image has been removed. Accordingly, in step S46, the first reconstructed signal inversely transformed through the signal inverse transform unit 14 may have a placenta-related feature in the ultrasound image removed.

A second virtual signal may be generated by using the first restoration signal as an input signal (S48).

As described above, the virtual signal generator 13 generates a second virtual signal of the second domain by mapping features of the destination signal of the second domain to a plurality of features included in the first restoration signal. can do.

For example, the process of generating the second virtual signal may correspond to the operation performed in steps S42 and S43 described above.

As an example, the signal conversion module T may further identify at least one second error characteristic present in the first recovery signal similar to the operation performed in step S42, and the at least one second error characteristic In consideration of these, the first reconstructed signal may be converted into a second virtual signal of a second domain.

As another example, the signal conversion module T may convert the first reconstructed signal into a second virtual signal using a parameter learned in advance. In this case, the second virtual signal generated through the virtual signal generation unit 13 may have the second error characteristic further removed.

Therefore, according to the signal conversion method according to the second embodiment of the present application, a virtual signal having a quality of a certain level or higher can be provided to a user by repeatedly performing the above steps until the quality of the virtual signal meets a predetermined level. have.

For example, referring to FIG. 6, a first virtual image (y0) obtained by converting an ultrasound image (x0) of a first domain into an actual photographic image of a second domain includes a fetus among error features in the source image (x0). It may contain features related to your hand. In this case, the controller 15 may determine that the quality of the first virtual image y0 does not meet a predetermined level, and the first virtual image y0 is inversely transformed into the first domain. The first virtual image y0 may be transmitted to the signal inverse transform unit 13 to generate the reconstructed image x1. In addition, the control unit 15 re-inputs the first reconstructed image x1 generated through the signal inverse transform unit 14 to the virtual signal generation unit 13 to obtain the first reconstructed image x1 It is possible to re-convert to a second virtual image of the 2 domain.

That is, as shown in Fig. 6, the control unit 15 can cause the signal conversion process by the virtual signal generation unit 13 and the signal inverse conversion process by the signal inverse conversion unit 14 to be sequentially/repetitively performed. have. As a result, the control unit 15 operates the signal conversion unit T to output a virtual image (y*) having a quality corresponding to a predetermined level from which unwanted features included in the source image (x0) are removed. Can be controlled.

In addition, the signal conversion method according to the third embodiment of the present application may output a virtual signal with improved quality by repeatedly performing the above-described signal conversion operation a predetermined number of times.

For example, the control unit 15 generates a first virtual signal of the second domain from which at least one first error feature included in the source signal is removed in step S43, and then whether the signal conversion operation has been performed a predetermined number of times. It may further include the step of confirming.

In this case, when it is determined that the signal conversion operation has been performed a predetermined number of times, the controller 15 may output the first virtual signal on the user device in step S45.

Conversely, when it is determined that the signal conversion operation has not been performed a predetermined number of times, the control unit 15 may return to step S43 and control the operation of the virtual signal generation unit 13 to generate a second virtual signal. For example, the virtual signal generator 13 may generate a second virtual signal from which at least one second error characteristic in the source signal is further removed.

Alternatively, when it is determined that the signal conversion operation has not been performed a predetermined number of times, the control unit 15 performs the operation corresponding to steps S42 and S43. You can control the operation.

Accordingly, the control unit 15 may simply repeat the signal conversion operation by the virtual signal generation unit 13 a predetermined number of times, thereby outputting a virtual signal having a quality of a predetermined level or higher on the user device.

3. Signal Transformation Learning Process

Hereinafter, a machine learning process for optimizing the above-described signal conversion operation will be described in detail with reference to FIGS. 7 and 8.

The machine learning process according to an embodiment of the present application may be performed by a learning module M located inside or outside the server 10 described above, and the learning module M includes one or more neural networks (NN). It may include.

7 and 8 are diagrams for illustratively explaining a signal conversion learning process performed by one or more neural networks (NN) according to an embodiment of the present application.

The signal conversion learning according to the exemplary embodiment of the present application may be performed using a plurality of data acquired from a memory or a database. Hereinafter, an example of image conversion between images in different domains is assumed and described. do.

7 is a diagram illustrating a configuration of a neural network (NN) according to an embodiment of the present application. For example, the neural network NN may include one or more networks, and the network may include a plurality of layers. Hereinafter, for convenience of description, it is assumed that signal conversion learning is performed using one or more networks. Referring to FIG. 7, the neural network NN includes, for example, a first network 101, a second network 102, a third network 103, a fourth network 104, and a fifth network ( 105). In the first to fifth networks, each step for signal conversion learning may be performed sequentially or in parallel.

Hereinafter, operations that can be performed in each network will be described in detail.

First, the first network 101 may be a network for converting the source signal X of the first domain into a feature map (FM).

For example, the first network 101 may be a convolutional network and may include a plurality of layers. In this case, in the first network 101, a plurality of features included in the source signal X may be extracted using the plurality of layers.

When the source signal X is an image, the plurality of features are, for example, edge, sharpness, depth, brightness, contrast, and blur. , Shapes or combinations of shapes, and the like, and the plurality of features are not limited to the above-described examples.

The feature map may be a combination of the plurality of features, and regions in the source signal may be identified through one or more feature maps. Each of the regions may include a region of interest (ROI) of the user and an error region including at least one artifact in the source signal. For example, when the source signal X is an ultrasound image of a fetus, the region of interest may be a face region of the fetus, and the error region may be a body region other than the face region of the fetus. . For example, the face area may include the eyes, nose, mouth, and ears of the fetus. In addition, for example, the error region may include hands, feet, placenta, and amniotic fluid of the fetus.

Accordingly, a feature map may be obtained from at least one of the plurality of layers using the first network 101. Alternatively, a feature map may be obtained from the last layer among the plurality of layers. One or more feature maps generated in the first network 101 may be used as input data of the second network 102 and/or the third network 103.

Meanwhile, parameters for the plurality of layers may be adjusted according to a result of signal processing performed by the neural network according to the exemplary embodiment of the present application, and at least one of the plurality of layers may be removed or added in an operation. A detailed description of the parameter adjustment will be given in the related section.

The second network 102 may be a network for generating a virtual signal Yv of the second domain by using one or more feature maps input from the first network 101 described above as input data. The second network 102 may be trained to convert the source signal X of the first domain into a virtual signal Yv of the second domain.

For example, the second network 102 may include a plurality of layers. In the second network 102, a virtual signal Yv in which features of a destination signal of a second domain are mapped to corresponding features of the feature map may be generated using the plurality of layers. .

The characteristics of the destination signal, when the source signal X is an image, may include, for example, brightness, color, reflection color, texture, depth, blending, shape or combination of shapes of the image.

One or more feature maps input to the second network 102 from the first network 101 described above include features of a region of interest (eg, eyes, nose, mouth, ears, etc.) in the source signal X. I can.

Alternatively, the second network 102 may be trained to identify a feature related to an error region (eg, hand, foot, placenta, etc.) among one or more feature maps input from the first network 101 described above.

* The virtual signal Yv is generated using the one or more feature maps and may be a signal including a region of interest in the source signal X.

For example, when the source signal (X) is an ultrasound image of a fetus, the virtual signal (Yv) converted by the second network (102) is a first including fewer features of an error region in the ultrasound image (X). It may be an image of 2 domains. Alternatively, for example, the virtual signal Yv may be an image of a second domain from which features of at least one error region in the ultrasound image X have been removed. Alternatively, for example, the virtual signal Yv may be an image of the second domain in which all of the features of the error region in the ultrasound image X have been removed.

The destination signal and/or the characteristics of the destination signal may be obtained from the aforementioned database 20, a memory, or an external server. Alternatively, the second network 102 may further include a sub-network (not shown) for extracting features of the destination signal. The sub-network may be, for example, a convolutional network, and may include a plurality of layers. The operation of extracting features of the destination signal through the sub-network may correspond to the operation performed in the first network 101 described above.

Accordingly, the server 10 may obtain a virtual signal of a second domain in which features of the destination signal are synthesized in the at least one feature map through the second network 102. When a plurality of synthesized signals for a plurality of feature maps are extracted, the server 10 may obtain one virtual signal by integrating the synthesized signals.

Meanwhile, parameters for the plurality of layers may be adjusted according to a result of signal processing performed by the neural network according to the exemplary embodiment of the present application, and at least one of the plurality of layers may be removed or added in an operation. A detailed description of the parameter adjustment will be given in the related section.

The third network 103 may be a network for generating an error signal Ye by using one or more feature maps input from the first network 101 described above as input data. For example, the error signal Ye may be a signal of a first domain, a second domain, or an intermediate domain between the first domain and the second domain.

The one or more feature maps input to the third network 103 may include at least one of features of an error region (eg, hand, foot, placenta, amniotic fluid, etc.) other than a region of interest in the source signal X. have.

The error signal Ye may be generated using the one or more feature maps, and may be a signal including at least one of features related to an error region in the source signal X.

Accordingly, the server 10 may obtain the error signal Ye through the third network 103. For example, when a plurality of synthesized signals for a plurality of feature maps are extracted, the server 10 may obtain one error signal by integrating the synthesized signals.

Alternatively, the third network 103 may be trained to extract or separate an error signal Ye including at least one of features related to an error region included in the source signal X of the first domain.

For example, the third network 103 may be a convolutional or deconvolutional network, and may include a plurality of layers. The third network 103 may generate an error signal Ye using at least one of the plurality of layers. For example, a first error signal may be extracted through a first layer among the plurality of layers, and a second error signal may be extracted through a second layer. In other words, a plurality of error signals may be acquired or extracted through a plurality of layers.

_{In addition, the server 10 may obtain a target signal (Y T} ) in which the virtual signal (Yv) obtained from the second network (102) and the error signal (Ye) obtained from the third network (103) are integrated. The target signal Y _T may be used as input data of the fourth network 104. Hereinafter, a process of obtaining the target signal Y _T will be described in detail.

The target signal Y _T may have a first weight applied to the error signal Ye and a second weight applied to the virtual signal Yv, and may be calculated as [Equation 1]. . Referring to [Equation 1], the target signal Y _T is an element-wise product of the virtual signal Yv and the second weight, and the error signal Ye and the first weight It can be generated by the sum of the product of each element.

Here, We may be a first weight applied to the error signal Ye, and (1-We) may be a second weight applied to the virtual signal Yv. The first weight applied to the error signal Ye may be a value adjusted according to a result of image processing performed by the neural network NN according to an embodiment of the present application.

Also, a second weight applied to the virtual signal Yv may be calculated based on the first weight. In this case, the first weight and the second weight may be the same or different values.

For example, the first weight may be a value calculated based on the difference between the reconstructed signal X'' and the source signal X restored using only the virtual signal Yv in the fourth network 104 to be described later. have. The first weight calculation method will be described in detail in the following related embodiments.

_{The fourth network 104 inversely transforms the target signal Y T} of the second domain generated through the above-described second network 102 and the third network 103 into a reconstructed signal X′ of the first domain. It can be a network for

For example, the fourth network 104 may improve the consistency of content between the restored signal and the source signal by accurately reconstructing the source signal X. Here, the consistency of content may be a similarity between the fetus in the ultrasound image and the converted real photograph image when the ultrasound image of the fetus is converted into an actual photographic image.

Or, for example, the neural network (NN) transmits the source signal (X) of the first domain to the target signal (Y _T ) of the second domain through the second network 102 and the third network 103 Learn to convert the domain while maintaining consistency of the entire learning contents by simultaneously learning the process of converting to and converting the converted target signal (Y _{T) of the second domain into a reconstructed signal (X') of the first domain.} can do.

In this case, a parameter applied to at least one of the neural networks NN may be adjusted based on a difference value between the reconstructed signal X′ generated by inverse transforming _{the target signal Y T and the source signal X. That is, the parameter may be adjusted by inversely transferring a loss function L rec} occurring from a difference between the reconstructed signal X'and the source signal X to at least one of the aforementioned networks. The loss function (L _rec ) can be expressed as [Equation 2] below.

Alternatively, the fourth network 104 is a network for inversely transforming the virtual signal Yv generated from the second network 102 into a reconstructed signal X'' of the first domain in order to calculate the above-described first weight. I can.

At this time, since the fourth network 104 generates a reconstructed signal (X'') of the first domain using only the virtual signal (Yv), the virtual signal ( Yv) alone, the neural network NN may be trained to identify a portion that cannot be reconstructed (eg, an error region). A method of guiding information on the error area so that the neural network NN identifies the error area will be described in detail in the related part below.

The fourth network 104 may include one or more networks, and may include a plurality of layers. For example, the fourth network 104 first _{inputs a first sub-network (not shown) for converting the target signal (Y T} ) or the virtual signal (Yv) into a feature map and the converted feature map. It may include a second sub-network (not shown) for generating reconstructed signals X′ and X″ by applying them as data. For example, the first sub-network may be a convolutional network, and the second sub-network may be a deconvolutional network.

Hereinafter, for convenience of description, it is assumed that the fourth network 104 includes at least a first sub-network and a second sub-network. In addition, signal conversion may be performed in the first sub-network in the same or similar manner as the above-described first network, and in the second sub-network as the second and third networks described above.

The first sub-network may include a plurality of layers, and the target signal (Y _T ) or the virtual signal (Yv) may be converted into one or more feature maps using the plurality of layers. For example, in the first sub-network, _{a plurality of feature information included in the target signal Y T} or the virtual signal Yv may be extracted using the plurality of layers, and the plurality of One or more feature maps may be generated based on the feature information. One or more feature maps generated in the first sub-network may be applied as input data of the second sub-network.

The second subnetwork may include a plurality of layers, and the target signal (Y _T ) or the virtual signal (Yv) may be converted into a reconstructed signal (X′, X″) using the plurality of layers. . That is, in the second sub-network, features of the source signal X are mapped to each feature of one or more feature maps input from the first sub-network to generate the reconstructed signals X′ and X″. . The characteristic information of the source signal X may be information extracted through the first network 101 described above, and may be obtained from a storage unit (not shown) of the server 10.

Accordingly, the server 10 obtains the reconstructed signals (X', X'') of the first domain in which the features of the source signal (X) are synthesized in the at least one feature map through the fourth network 104. can do. When a plurality of synthesized signals for a plurality of feature maps are extracted, the server 10 may obtain one reconstructed signal (X′, X″) by integrating the synthesized signals.

_{Meanwhile, the virtual signal Y V} of the second domain generated using the above-described second network 102 may be used as input data of the fifth network 105.

The fifth network 105 may be a network for distinguishing the _{virtual signal Y V} generated from the above-described second network 102 from the destination signal Y _{R of the second domain.}

For example, the fifth network 105 may be trained to identify between _{the virtual signal Y V} and the destination signal Y _R. For example, the fifth network 105 may be a convolutional network and may include a plurality of layers. In this case, the fifth network 105 may output information related to a similarity between the virtual signal and a destination signal through a last layer among the plurality of layers. Here, the degree of similarity may mean similarity between qualitative features of the virtual signal and the destination signal.

The destination signal may be obtained from the above-described database 20, and for example, the destination signal may be an image actually photographed of a newborn's face.

The information related to the similarity may be information indicating a probability that _{the virtual signal Y V corresponds to a destination signal.} For example, when the virtual image Y _V corresponds to an image actually captured, the similarity value may be '1' or a value close to 1. In addition, for example, when the virtual signal Y _V is not an actual captured image, the similarity value may be '0' or a value close to 0.

Accordingly, the server 10 may feed back information related to the similarity to at least one of the first network 101 to the third network 103. The server 10 may re-acquire _{the virtual signal Y V} by repeatedly performing the operations of the second network 102 and the third network 103 based on the information related to the similarity.

Alternatively, for example, the fifth network 105 may feed back a loss function generated from the difference between the virtual signal and the destination signal to the first network 101 and the second network 102 described above. Accordingly, a parameter of at least one layer among a plurality of layers included in the first network and the second network may be adjusted. Alternatively, at least one of a plurality of layers included in the first network 101 and the second network 102 may be removed or added in an operation. In other words, the first network 101 and the second network 102 may be learned using a difference between the virtual signal and a destination signal fed back from the fifth network 105.

8 is a block diagram illustrating a configuration of a neural network (NN) according to another embodiment of the present application. As shown in FIG. 8, the neural network NN according to another embodiment of the present application may further include a feature classification network 106.

The feature classification network 106 may be a network for classifying features related to an error area and features related to an ROI among respective areas in the source signal X. In this case, the feature classification network 106 may be a part of the first network 101. Alternatively, as shown in FIG. 8, the feature classification network 106 may be configured separately from the first network 101, and the first network 101, the second network 102, and the third network Can be placed between 103.

For example, the feature classification network 106 may generate a first feature map and a second feature map using one or more feature maps generated in the first network 101. For example, the first feature map may be related to valid features and may include fewer features related to an error area. For example, the second feature map may be related to error features and may include fewer features related to an ROI. For example, when the source signal is an ultrasound image of a fetus, features related to the ROI may include eyes, nose, mouth, and ears of the fetus. Also, for example, features related to the error region may include, for example, a fetus's hand, foot, placenta, amniotic fluid, and the like.

In addition, for example, the feature classification network 106 may classify a first feature map related to an ROI in the source signal X and a second feature map related to the error region based on a preset criterion. have. The preset reference may be learned in advance based on information related to the region of interest and the error region in the source signal X.

Accordingly, the feature classification network 106 according to another embodiment of the present application may transmit the first feature map related to the ROI as input data of the second network 102, and the second feature map related to the error region The feature map may be transmitted as input data of the third network 103. Features related to the error area may be more accurately classified through the feature classification network 106. For example, the second network 102 removes features related to at least one error region in the source signal by using a first feature map related to the ROI input through the feature classification network 106. It is possible to generate a virtual signal (Y _{v ).}

Meanwhile, the feature classification network 106 may be updated by reflecting a parameter adjusted using a difference between the reconstructed signal X'and the source signal X. Alternatively, the feature classification network 106 may be updated to reflect the parameter adjusted using the difference between the virtual signal and the destination signal. Detailed description of the above parameters will be described in the related section.

Hereinafter, exemplary embodiments of the method of learning signal transformation performed through the aforementioned neural network (NN) will be described in detail.

9 is a diagram schematically illustrating a signal conversion process according to a fourth embodiment of the present application. In addition, FIG. 10 is a flowchart illustratively illustrating a learning method for converting and providing a source signal X of a first domain into a virtual signal Yv of a second domain according to the fourth embodiment of the present application. .

For example, the learning method according to the fourth embodiment of the present application may be performed by the learning module M located in the server 10 described above, and the learning module M includes the first network or At least one or more of the fifth networks may be provided.

Referring to FIG. 9, in the learning method according to the fourth embodiment of the present application, a source signal of a first domain is converted into a virtual signal of a second domain, and the converted virtual signal is inversely transformed into a reconstructed signal of the first domain. It can be configured to repeatedly perform the process of doing. For example, the source signal of the first domain may be an ultrasound image (X) of the fetus, and the virtual signal of the second domain may be an image (Yv) in which features of the actual photographic image of the newborn are combined with the ultrasound image of the fetus. . In this case, in the process of converting the source signal X into the virtual signal Y, one or more error signals e _o , e ₁ , …, e _n may be identified.

Hereinafter, steps in which signal conversion learning is performed according to the fourth embodiment of the present application will be described in detail with reference to FIG. 10.

Referring to FIG. 10, in the learning method according to the fourth embodiment of the present application, in a method of converting a source signal of a first domain into a virtual signal of a second domain using at least one network, the source of the first domain Identifying at least one first error characteristic from the signal (S61), calculating a first virtual signal of a second domain from which the first error characteristic has been removed (S62), and the quality of the first virtual signal is It may include determining whether the quality of the first virtual signal does not meet the predetermined level (S63), and when it is determined that the quality of the first virtual signal does not meet the predetermined level, the first virtual signal is converted to the first domain of the first virtual signal. 1 Inverse transforming into a reconstructed signal (S64), calculating a first weight using a difference between the first reconstructed signal and the source signal, and sharing the first weight with the network (S65), the first Generating a second virtual signal of a second domain including the first error characteristic using a weight (S66), inverse transforming the second virtual signal into a second reconstructed signal of the first domain (S67), It may further include a step (S68) of adjusting a parameter of the network by using a difference between the second reconstructed signal and the source signal.

Hereinafter, as described with reference to FIG. 9, the source signal of the first domain is the ultrasound image (X) of the fetus, and the virtual signal of the second domain combines the features of the actual photographic image of the newborn to the ultrasound image of the fetus It will be described in detail by taking the case of the image Y as an example.

First, as described above, at least one ultrasound image X may be input through the first network 101, and the learning module M may perform learning using the ultrasound image X as input data. have.

The learning module M may identify at least one first error characteristic from the source signal of the first domain (S61).

For example, the learning module M uses one or more features provided through the first network 101 as described above, and at least one error feature (eg, hand , Feet, placenta, etc.) can be identified. For example, the error feature may be extracted through an arbitrary layer among a plurality of layers included in the first network 101, and the step of extracting the error feature may be repeatedly performed.

For example, the learning module M may extract the first error feature e1 through a first layer among a plurality of layers included in the first network 101. In addition, the learning module M may extract the second error feature e2 through an arbitrary second layer among a plurality of layers included in the first network 101. For example, the first error feature e1 may include a feature related to the fetus's hand included in the ultrasound image X of the fetus. In addition, the second error feature e2 may include features related to the fetus's foot included in the ultrasound image X of the fetus.

Meanwhile, the process of extracting the error feature in the learning module M is performed in the first network 101, the second network 102, the third network 103, and/or the feature classification network 106 described above. May correspond to the operation.

The learning module M may calculate a first virtual signal of the second domain from which the first error feature has been removed (S62).

For example, the learning module M may generate the first virtual signal Y of the second domain by reflecting the first error characteristic identified in step S61. Referring to FIG. 9, the learning module M may synthesize a virtual image Y from which features related to a fetus's hand have been removed. The process of calculating a virtual signal performed by the learning module M may correspond to an operation performed by the second network 102 described above.

Meanwhile, in the learning method according to the fourth embodiment of the present application, the learning module M may further generate a first error signal by using the error characteristic identified in step S61. In this case, the process of generating the error signal may correspond to an operation performed in the third network 103 described above.

The learning module M may determine whether the quality of the first virtual signal meets a predetermined level (S63).

In other words, the learning module M repeats the following learning process until the quality of the first virtual signal calculated in step S62 reaches a predetermined level, and determines that the quality of the virtual signal has reached a predetermined level. If so, the learning process can be terminated.

In this case, whether or not the quality of the virtual signal has reached a predetermined level may be determined in various ways.

Here, the preset termination condition may be a condition previously input by the user.

For example, the user may set the learning module M to repeat the learning process a predetermined number of times.

Alternatively, for example, the user may set to repeat the learning process until the loss function of the neural network NN is not decreased.

Alternatively, for example, the learning module M may acquire information related to the similarity between the virtual signal and the destination signal of the second domain through the above-described fifth network 105. The destination signal may be obtained from the above-described database 20, and for example, the destination signal may be an image actually photographed of a newborn's face. The information related to the similarity indicates a probability that the virtual signal corresponds to an image actually photographed, and a detailed description thereof is omitted below since the description has been described above.

When it is determined that the quality of the first virtual signal does not meet a predetermined level, the learning module M may perform the following learning processes.

The learning module M may inversely transform the first virtual signal into a first reconstructed signal in the first domain (S64).

For example, the learning module M may inversely transform the first virtual signal generated in step S62 into a first reconstructed signal in the first domain. The process of converting the first virtual signal into a first reconstructed signal may correspond to an operation performed in the fourth network 104 described above.

Further, the learning module M may calculate a first weight by using a difference between the first reconstructed signal and the source signal, and share the first weight with the network (S65).

For example, the first weight may be a difference between the first reconstructed signal and the source signal.

Alternatively, for example, the first weight may be a gradient calculated by differentiating a difference between the first reconstructed signal and the source signal X with respect to a virtual image or an arbitrary feature map. For example, the slope may be calculated as in [Equation 3].

Accordingly, since the first weight reflects information on an area that cannot be restored (eg, an error area) only with the first virtual signal Yv, information on an error area included in the source signal can be guided. have.

In this case, the source signal X may be 3D data in which three RGB channels are merged, and a slope g for each channel may be adjusted through normalization. The normalization process may be performed by various methods, and for example, as shown in Equation 4 below, it may be calculated using a standard normal distribution using an average and a standard deviation for each channel.

Here, c means each channel of R, G, and B.

In addition, by applying a sigmoid function to the normalized slope (g) value using [Equation 4], the weight (W _e ) value is adjusted to have a range of 0 to 1, for example, 0.5 to 1. It can be, and it can be calculated as in Equation [5] below.

In other words, the information on the error region in the source signal X may be shared with at least one of the first to fifth networks.

The learning module M may generate a second virtual signal of the second domain including the first error characteristic using the first weight (S66), and convert the second virtual signal to the second virtual signal of the first domain. It is possible to inverse transform into a restored signal (S67).

For example, in the learning process using the neural network (NN), in order to maintain the consistency of the contents in the source signal, a second virtual signal may be generated by integrating a first virtual signal and a first error signal. 2 The process of generating the virtual signal may correspond to the method of generating the target signal T described above with reference to FIG. 7.

In addition, the learning module M may further inverse transform the second virtual signal into a second reconstructed signal of the first domain using the above-described fourth network 104. That is, the inverse transformed signal and the consistency of the content in the source signal may be compared using the difference between the inverse transformed signal and the source signal based on the second virtual signal.

The learning module M may adjust the parameters of the network by using the difference between the second reconstructed signal and the source signal (S68).

Here, the parameter may be adjusted according to a loss function calculated as a difference between the second restoration signal and the source signal.

The learning module (M) is a first neural network 101, a second neural network 102, a third neural network 103, a fourth neural network 104, and a fifth neural network 105 described above. ) And/or back to the classification network 106. Accordingly, a parameter applied to a plurality of layers included in each network may be adjusted, and any one of the plurality of layers may be removed.

That is, according to the learning method according to the fourth embodiment of the present application, by repeatedly performing the process of converting the source signal of the first domain into the virtual signal of the second domain, a plurality of Optimal parameters applied to the layer can be derived.

Therefore, the signal conversion module T according to the embodiment of the present application performs a signal conversion operation using the parameter transmitted from the learning module M to provide a virtual signal from which error features in the source signal are removed. I can.

At this time, the server 10 may update an optimal parameter applied to the neural network NN by continuously performing the machine learning process performed by the learning module M again.

Accordingly, according to the signal conversion providing system 1 according to the exemplary embodiment of the present application, the quality of a virtual signal may be improved.

11 is a diagram schematically illustrating a signal conversion learning process according to a fifth embodiment of the present application. 12 is a flowchart illustrating a learning method for converting and providing a source signal X of a first domain into a virtual signal Yv of a second domain according to the fifth embodiment of the present application. .

For example, the learning method according to the second embodiment of the present application may be performed by the learning module M located in the server 10 described above, and the learning module M includes the first network or At least one or more of the fifth networks may be provided.

Referring to FIG. 11, in the learning method according to the fifth embodiment of the present application, in a process of converting a source signal of a first domain into a virtual signal of a second domain, the source signal while maintaining the consistency of the content in the source signal It may be configured to generate _{a first virtual signal I A} and a second virtual signal I _B , respectively, in order to learn features related to an error region in the signal.

For example, the first virtual signal I _A may be converted by selecting only features of the user's ROI in the source signal. Also, for example, the second virtual signal I _B may be transformed using all features in the source signal.

As shown in FIG. 11, the source signal of the first domain is an ultrasound image (I _S ) of the fetus, and the first virtual signal (I _A ) of the second domain is features related to the face area of the fetus in the ultrasound image. On the other hand, the characteristics of the actual photographic image of the newborn may be a composite image. In addition, the second virtual signal I _B of the second domain may be an image in which features of an actual photograph image of a newborn baby are combined with all features including error features in the ultrasound image.

Hereinafter, steps in which signal conversion learning is performed according to a fifth embodiment of the present application will be described in detail with reference to FIG. 12.

Referring to FIG. 12, in the learning method according to the fifth embodiment of the present application, the step of acquiring a plurality of features from a source signal of a first domain (S111), a first virtual signal and a plurality of features using the first feature Generating a second virtual signal using (S112), inversely transforming the second virtual signal into a reconstructed signal of the first domain, and comparing the consistency using the difference between the inversely transformed reconstructed signal and the source signal (S113), the first A step (S115) of determining whether the quality of the first virtual signal corresponds to a predetermined level by calculating a similarity between the first virtual signal and the destination signal of the second domain (S114). The steps described below are illustratively described for convenience of description, and each step may be performed sequentially or in parallel.

Hereinafter, as described with reference to FIG. 11, the source signal of the first domain is the ultrasound image (I _S ) of the fetus, and the first virtual signal (I _A ) and the second virtual signal (I _B ) of the second domain Will be described in detail by taking the case where the features of the actual photographic image of the newborn are synthesized with the ultrasound image of the fetus as an example.

First, as described above, at least one ultrasound image I _S may be input through the first network 101, and the learning module M performs learning by using the ultrasound image I _S as input data. You can do it.

The learning module M may acquire a plurality of features from the source signal of the first domain (S111).

For example, the learning module M may extract a plurality of features from the ultrasound image of the first domain. For example, the plurality of features may include a first feature (effective feature) related to a user's region of interest (ROI) and a second feature (error feature) related to at least one error region.

In addition, for example, the plurality of features may be extracted through an arbitrary layer among a plurality of layers included in the first network 101, and the process of extracting an error feature in the learning module M is described above. It may correspond to an operation performed in a first network 101.

The learning module M may generate a first virtual signal using the first feature and a second virtual signal using a plurality of features (S112).

The learning module M may distinguish between a first feature and/or a second feature among a plurality of features acquired in step S111. For example, the learning module M may select features related to an error region among the plurality of features using a preset reference or a parameter transmitted through a previous learning process.

For example, the first feature may include features related to the face area of the fetus among the plurality of features, and at least one second feature may be removed. For example, the first feature may include features related to facial areas such as eyes, nose, mouth, and ears of the fetus, and may not include features other than facial areas such as hands and feet of the fetus.

Accordingly, the learning module (M) uses the above-described second network 102 and/or third network 103 to remove the first virtual signal from which features related to the hands and feet of the womb are removed, and the hands and hands of the fetus. Each of the second virtual signals including features related to the foot may be generated. The learning module M may inversely transform the second virtual signal into a reconstructed signal of the first domain, and compare the consistency using a difference between the reconstructed signal and the source signal (S113).

That is, the learning module M may inversely transform the second virtual signal generated in step S112 into a reconstructed signal of the first domain. Since the process of converting the second virtual signal into a reconstructed signal may correspond to the operation performed in the above-described fourth network 104, a detailed description will be omitted hereinafter.

For example, referring to FIG. 11, the learning module M includes a reconstructed image I's and the source signal Is reconstructed from _{a second virtual signal I B including features related to the fetus's hand.} You can compare the differences.

Therefore, the learning module (M) inversely transfers the difference between the reconstructed signal (I's) and the source signal (Is), inversely transformed using only _{the second virtual signal (I B), to the neural network (NN),} Content consistency can be maintained.

In addition, the learning module M may distinguish between the first virtual signal and the destination signal of the second domain (S114).

That is, the learning module (M) uses the above-described fifth network 105 to remove the at least one error feature included in the source signal, the first virtual signal (I _A ) and the destination (I) of the second domain. _R ) It is possible to determine the similarity of the signal.

Here, the similarity may mean similarity between the qualitative features of the virtual signal and the destination signal.

For example, referring to FIG. 11, in the learning module M, a first virtual image (I _B ) from which a hand-related feature in the ultrasound image of the fetus is removed corresponds to an actual photo image (I _R ) of a newborn baby. You can calculate the probability of becoming. Accordingly, the neural network NN may _{perform training to generate a virtual image close to the actual photographic image I R} , so that the quality of the virtual image may be further improved.

In addition, the learning module M may determine whether the quality of the first virtual signal meets a predetermined level (S115).

For example, the learning module M may determine whether the quality of the first virtual signal IA meets a predetermined level using the similarity determined in step S114, and The above-described steps can be repeatedly performed until the quality reaches a predetermined level.

Alternatively, for example, the learning module M may update a parameter for generating a virtual signal of a desired quality by repeatedly performing steps S111 to S114 until a preset termination condition is satisfied.

Here, the preset termination condition may be a condition previously input by the user.

Or, for example, the user may set the learning module M to repeat steps S111 to S114 a preset number of times.

Alternatively, for example, the user may set to repeat steps S111 to S114 until a loss function of the neural network NN is not decreased.

At this time, when the learning module M determines that the quality of the first virtual signal has not reached a predetermined level as a result of the determination in step S115, the plurality of neural networks NN is based on the virtual signals and the loss function. The parameters for the layer of can be adjusted.

Accordingly, the learning module M may transfer parameters for a plurality of layers of the neural network NN derived through steps S113 and S114 of a corresponding learning process to the signal conversion module T.

Conversely, when it is determined in step S115 that the quality of the first virtual signal has reached a predetermined level, the learning module M may end the learning process.

Therefore, according to the learning method according to the fifth embodiment of the present application, the source signal is converted back to the signal of the first domain by inversely transforming the second virtual signal converted by using all the features included in the source signal of the first domain. You can learn the consistency of the contents within. In addition, error features included in the source signal are removed by identifying at least one error feature and distinguishing the first virtual signal I _{A from which the error features are removed from the destination signal I R} of the second domain. May be learned to generate the first virtual signal.

That is, according to the learning method according to the fifth embodiment of the present application, the above-described neural network is repeatedly performed by converting the source signal of the first domain into a virtual signal of the second domain and measuring the loss function to adjust the parameter. Optimal parameters applied to a plurality of layers included in the network NN may be derived.

Accordingly, the signal conversion module T according to the exemplary embodiment of the present application may provide a virtual signal having a predetermined quality or higher by performing a signal conversion operation using a parameter transmitted from the learning module M. In this case, the server 10 may update an optimal parameter applied to the neural network NN by re-performing the machine learning process performed by the learning module M every predetermined period.

Meanwhile, in the above-described embodiments of the present application _{, not only the process of converting the source signal I S} of the first domain to the virtual signals I _{A and} I _{B of} the second domain, but also the virtual signal of the second domain ( in the process of the inverse transform I _B) to restore the signal (I _'s) of the first domain may be a learning carried out in the same manner.

For example, as shown in FIG. 11, _{an image (I's) obtained by converting an actual photo image (I R} ) of a newborn baby in a second domain into an ultrasound image of a fetus in the first domain (I's) and an ultrasound image ( Is) can be learned to distinguish. In addition, "the inverse image (I, by comparison of the actual photographic images of _(R and newborn (I _{R) R} image is converted into an ultrasonic image (I's) to re-image I) 'inverse to the actual images of the first domain ) And the content in the actual photographic image I _R.

As a result, the signal conversion providing system 1 according to the exemplary embodiment of the present application may provide a virtual signal of a predetermined quality on the user device 30 by repeatedly performing the above-described learning process.

For example, when the input signal of the first domain is an ultrasound image of a fetus, the user may receive the error even when the fetal face in the source image of the first domain is covered by at least one error feature of hand, foot, placenta, and amniotic fluid. A virtual image of the second domain from which features have been removed may be obtained.

Until now, for convenience of explanation, the description has been made on the assumption that each function of signal conversion is performed in the server 10 described above, but each function of the signal conversion may be recorded and provided in a medium readable on a computer.

That is, the signal conversion method according to the exemplary embodiment of the present application may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

As described above, although the embodiments have been described with reference to limited embodiments and drawings, various modifications and variations are possible from the above description to those of ordinary skill in the art. For example, the described techniques are performed in a different order from the described method, and/or components such as systems, structures, devices, circuits, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and those equivalent to the claims also fall within the scope of the claims to be described later.

1: signal conversion provision system
2: communication network
10: server
11: signal transmission/reception unit
12: feature analysis unit
13: Virtual signal generator
14: signal inverse transform unit
15: control unit

Claims (1)

Receiving a source signal of a first domain;
Identifying error characteristics and valid characteristics in the source signal;
Generating a first virtual signal of a second domain from which at least one first error characteristic included in the source signal is removed; And
Including; outputting the first virtual signal
Signal conversion method.

Images