KR102334666B1 - A method for creating a face image - Google Patents

Abstract

In one embodiment of the present disclosure for solving the above problems, a method of creating a face image performed by one or more processors of a computing device is disclosed. The method may include the steps of: receiving, by the processor, first image data and second image data; extracting, by the processor, common characteristic information corresponding to a plurality of first reference images constituting the first image data; deriving, by the processor, one or more similar images based on the extracted common characteristic information; and generating, by the processor, the one or more similar images based on a plurality of second reference images constituting the second image data, thereby creating a target prediction image.

Description

{A METHOD FOR CREATING A FACE IMAGE}

The present disclosure relates to an image generating method, and more particularly, to a method for generating a face image similar to a target predicted face by using an artificial neural network.

In the recent paradigm shift of the 4th industrial revolution, artificial intelligence is attracting attention as a key driver leading a new era. Artificial intelligence is a software technology that implements the overall human thinking process such as cognition, learning, reasoning, and judgment through algorithm design, and has the characteristics of a general-purpose technology that dramatically improves productivity across all industrial areas, not limited to specific industries.

On the other hand, since the proposal of deep learning, image generation technology based on artificial intelligence technology has been greatly developed. In particular, the proposal of a generative adversarial neural network (GAN) made it possible to generate data that looked like it was created by a human by providing a learning method that can reproduce detailed information as well as general structural characteristics of the data. The generative adversarial neural network aims to find the Nash equilibrium between the generator and the discriminator, and greatly improves the performance of existing models in estimating the distribution of data.

The image generation technology using such artificial intelligence aims to create an image that meets the user's intention. However, as the image expression method for expressing the image for each user or the target image (ie, the target image) to be output based on a specific image (eg, a reference image) are different, the user's intention is not met. It may not be easy to learn an artificial neural network using a target image or a base learning image. That is, there is a limit to generating and providing an image or image suitable for an arbitrary intention.

For a specific example, if an image or image related to the future face of a fetus is to be generated based on information about an ultrasound image of the fetus, the target image is based on the non-uniform shooting angle and noise of the ultrasound image that is based on the image generation. It can be difficult to create images related to future faces. That is, in the case of an ultrasound image used as a reference image, it is difficult to recognize the natural face shape of the fetus because it contains only the morphological information of the fetus. There is a risk that this will be learned.

Accordingly, there may be a demand in the industry for a computing device that generates and provides a target image with high prediction accuracy even for various reference images.

Korean Patent Laid-Open Patent No. 10-2021-0047920

SUMMARY OF THE INVENTION An object of the present disclosure is to solve the above problems, and to provide a computing device that generates a face image similar to a target predicted face by using an artificial neural network.

The problems to be solved by the present disclosure are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

Disclosed is a method for generating a face image performed by one or more processors of a computing device according to various embodiments of the present disclosure for solving the above-described problems. The method may include: obtaining, by the processor, first image data and second image data; extracting, by the processor, common characteristic information corresponding to a plurality of first reference images constituting the first image data; deriving one or more similar images based on the extracted common characteristic information, and adjusting the one or more similar images by the processor based on a plurality of second reference images constituting the second image data. It may include generating a prediction image.

According to an alternative embodiment, the first image data and the second image data are configured to include different domain information, and the common characteristic information may include information about a face landmark. .

According to an alternative embodiment, the deriving the one or more similar images may include: processing, by the processor, each of the plurality of first reference images as an input of a feature extraction model to provide one or more feature information corresponding to each first reference image It may include obtaining, by the processor, the common characteristic information based on the one or more characteristic information, and the processor deriving the one or more similar images based on the common characteristic information.

According to an alternative embodiment, the feature extraction model is configured through at least a part of a learned autoencoder, and the one or more common feature information and one or more domain information corresponding to each of the plurality of first reference images It may be characterized by outputting

According to an alternative embodiment, in the step of the processor deriving the one or more similar images based on the common characteristic information, the processor searches a target image database based on the common characteristic information to perform a predetermined similarity degree and deriving the one or more similar images, wherein the target image database is configured to correspond to each of a plurality of domains.

According to an alternative embodiment, the processor processing each of a plurality of first images constituting the first image data as an input of a feature extraction model to obtain one or more feature information corresponding to each image, the processor calculating, by the processor, a probability value corresponding to each of a plurality of items related to a component included in a face image based on the one or more characteristic information, and selecting, by the processor, a part of the plurality of first images based on the respective probability values It may include selecting the plurality of first reference images.

According to an alternative embodiment, generating, by the processor, the target prediction image by adjusting the one or more similar images based on a plurality of second reference images constituting the second image data may include: processing the second reference image as an input of a feature extraction model to obtain one or more additional characteristic information; obtaining, by the processor, the additional common characteristic information based on the one or more additional characteristic information; The method may include generating the target prediction image based on additional common characteristic information and the common characteristic information.

According to an alternative embodiment, the generating of the target prediction image may include: obtaining, by the processor, final characteristic information based on the additional common characteristic information and the common characteristic information; and generating, by the processor, the final characteristic information into an image and generating the target prediction image by processing it as an input of a model, wherein the image generation model may be configured through at least a part of a learned autoencoder.

According to another embodiment of the present disclosure, a computing device for performing a method of generating a face image is disclosed. The computing device includes a memory for storing one or more instructions and a processor for executing one or more instructions stored in the memory, wherein the processor executes the one or more instructions to perform the face image method described above. can

According to another embodiment of the present disclosure, a computer program stored in a computer-readable recording medium is disclosed. The computer program may be combined with a computer, which is hardware, to perform the above-described method for generating a face image.

According to various embodiments of the present disclosure, it is possible to provide a computing device that generates a face image similar to a target predicted face by using an artificial neural network.

Effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

Various aspects are now described with reference to the drawings, wherein like reference numbers are used to refer to like elements collectively. In the following example, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more aspects. It will be evident, however, that such aspect(s) may be practiced without these specific details.
1 is a conceptual diagram illustrating a system in which various aspects of a computing device for providing a method for generating a face image related to an embodiment of the present disclosure may be implemented.
2 is a block diagram of a computing device for providing a method for generating a face image according to an embodiment of the present disclosure.
3 is a diagram exemplarily showing an overall schematic diagram of a method for generating a face image according to an embodiment of the present disclosure.
4 is an exemplary diagram illustrating a process of outputting common characteristic information corresponding to a plurality of first reference images related to an embodiment of the present disclosure.
5 is an exemplary diagram illustrating a process of generating a target prediction image based on a second reference image related to an embodiment of the present disclosure.
6 is a flowchart exemplarily illustrating a method for generating a face image related to an embodiment of the present disclosure.
7 is a schematic diagram illustrating one or more network functions related to an embodiment of the present disclosure.

Various embodiments are now described with reference to the drawings. In this specification, various descriptions are presented to provide an understanding of the present disclosure. However, it is apparent that these embodiments may be practiced without these specific descriptions.

The terms “component,” “module,” “system,” and the like, as used herein, refer to a computer-related entity, hardware, firmware, software, a combination of software and hardware, or execution of software. For example, a component can be, but is not limited to being, a process running on a processor, a processor, an object, a thread of execution, a program, and/or a computer. For example, both an application running on a computing device and the computing device may be a component. One or more components may reside within a processor and/or thread of execution. A component may be localized within one computer. A component may be distributed between two or more computers. In addition, these components can execute from various computer readable media having various data structures stored therein. Components may communicate via a network such as the Internet with another system, for example, via a signal having one or more data packets (eg, data and/or signals from one component interacting with another component in a local system, distributed system, etc.) may communicate via local and/or remote processes depending on the data being transmitted).

In addition, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless otherwise specified or clear from context, "X employs A or B" is intended to mean one of the natural implicit substitutions. That is, X employs A; X employs B; or when X employs both A and B, "X employs A or B" may apply to either of these cases. It should also be understood that the term “and/or” as used herein refers to and includes all possible combinations of one or more of the listed related items.

Also, the terms "comprises" and/or "comprising" should be understood to mean that the feature and/or element in question is present. However, it should be understood that the terms "comprises" and/or "comprising" do not exclude the presence or addition of one or more other features, elements and/or groups thereof. Also, unless otherwise specified or unless it is clear from context to refer to a singular form, the singular in the specification and claims should generally be construed to mean “one or more”.

Those skilled in the art will further appreciate that the various illustrative logical blocks, configurations, modules, circuits, means, logics, and algorithm steps described in connection with the embodiments disclosed herein may be implemented in electronic hardware, computer software, or combinations of both. It should be recognized that they can be implemented with To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, configurations, means, logics, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application. However, such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

Descriptions of the presented embodiments are provided to enable those of ordinary skill in the art to use or practice the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art of the present disclosure. The generic principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present disclosure is not limited to the embodiments presented herein. This disclosure is to be interpreted in the widest scope consistent with the principles and novel features presented herein.

In this specification, a computer refers to all types of hardware devices including at least one processor, and may be understood as encompassing software configurations operating in the corresponding hardware device according to embodiments. For example, a computer may be understood to include, but is not limited to, smart phones, tablet PCs, desktops, notebooks, and user clients and applications running on each device.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

Each step described in this specification is described as being performed by a computer, but the subject of each step is not limited thereto, and at least a portion of each step may be performed in different devices according to embodiments.

1 is a conceptual diagram illustrating a system in which various aspects of a computing device for providing a method for generating a face image related to an embodiment of the present disclosure may be implemented.

A system according to embodiments of the present disclosure may include a computing device 100 , a user terminal 10 , an external server 20 , and a network. The components illustrated in FIG. 1 are exemplary, and additional components may be present or some of the components illustrated in FIG. 1 may be omitted. The computing device 100 , the user terminal 10 and the external server 20 according to embodiments of the present disclosure may mutually transmit/receive data for the system according to embodiments of the present disclosure through a network.

Networks according to embodiments of the present disclosure include Public Switched Telephone Network (PSTN), x Digital Subscriber Line (xDSL), Rate Adaptive DSL (RADSL), Multi Rate DSL (MDSL), Very High Speed DSL (VDSL). ), a variety of wired communication systems such as Universal Asymmetric DSL (UADSL), High Bit Rate DSL (HDSL), and Local Area Network (LAN) can be used.

In addition, the networks presented herein are Code Division Multi Access (CDMA), Time Division Multi Access (TDMA), Frequency Division Multi Access (FDMA), Orthogonal Frequency Division Multi Access (OFDMA), Single Carrier-FDMA (SC-FDMA) and Various wireless communication systems may be used, such as other systems.

The network according to the embodiments of the present disclosure may be configured regardless of its communication mode, such as wired and wireless, and is composed of various communication networks such as a personal area network (PAN) and a wide area network (WAN). can be In addition, the network may be a well-known World Wide Web (WWW), and may use a wireless transmission technology used for short-range communication, such as infrared (IrDA) or Bluetooth (Bluetooth). The techniques described herein may be used in the networks mentioned above, as well as in other networks.

According to an embodiment of the present disclosure, the user terminal 10 may be a terminal related to a user who accesses the computing device 100 to obtain a target prediction image based on the reference image. The user terminal 10 may transmit a reference image photographed or stored in the user terminal 10 to the computing device 100 . For example, the reference image may be an image related to an ultrasound image of a fetus, and the target prediction image may be a live-action image corresponding to an ultrasound image of a fetus. For example, the live-action image may mean a live-action image by restoring the skin color, skin texture, eye, nose, and mouth sizes of the fetus. For another example, the reference image may be an actual photographed image of the user, and the target prediction image may be an illustration image obtained by illustrating the user's actual photographed image. The detailed description of the above-described reference image and target prediction image is only an example, and the present disclosure is not limited thereto.

The user terminal 10 may be, for example, a terminal related to an examiner (eg, a specialist) that provides a live-action image of a fetus based on a reference image related to an ultrasound image. When the user terminal 10 is a terminal related to an examiner providing an actual image of a fetus, the actual image received from the computing device 100 may be used as medical assistance information for reading a fetal health condition. The user terminal 10 has a display, so it can receive a user's input and provide an output of any type to the user.

A user of the user terminal 10 is a medical professional, which may mean a doctor, a nurse, a clinical pathologist, a medical imaging specialist, or the like, and may be a technician repairing a medical device, but is not limited thereto. Also, for example, the user may mean the mother herself who wants to acquire a live-action image of the fetus using the system according to the disclosed embodiment.

The user terminal 10 may refer to any type of entity(s) in a system having a mechanism for communication with the computing device 100 . For example, the user terminal 10 is a personal computer (PC), a notebook (note book), a mobile terminal (mobile terminal), a smart phone (smart phone), a tablet PC (tablet pc), and a wearable device (wearable device) and the like, and may include all types of terminals capable of accessing a wired/wireless network. In addition, the user terminal 10 may include an arbitrary server implemented by at least one of an agent, an application programming interface (API), and a plug-in. In addition, the user terminal 10 may include an application source and/or a client application.

According to an embodiment of the present disclosure, the external server 20 may be a server that stores a reference image related to an ultrasound image of a fetus and an actual image related to each reference image. For example, the external server 20 may be at least one of a hospital server and a government server, and may be a server that stores information about reference images related to a plurality of fetal ultrasound images and actual images related to each reference image. have. Information stored in the external server 20 may be utilized as training data, verification data, and test data for learning the neural network in the present disclosure. That is, the external server 20 may be a server that stores information about a data set for training the neural network model of the present disclosure.

The computing device 100 of the present disclosure may build a training data set based on a plurality of reference images from the external server 20 and actual images corresponding to each reference image, and one or more network functions through the training data set By training a neural network model including

The external server 20 is a digital device, and may be a digital device equipped with a processor, such as a laptop computer, a notebook computer, a desktop computer, a web pad, and a mobile phone, and having a computing capability with a memory. The external server 20 may be a web server that processes a service. The above-described types of servers are merely examples, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the computing device 100 may generate a target prediction image based on the reference image received from the user terminal 10 . Here, the reference image is an image based on generation of the target prediction image, and may include, for example, an image related to an ultrasound image of a fetus and an actual photographed image of a user. In addition, the target prediction image is a target image to be generated in relation to the user's intention, and may include, for example, a live-action image corresponding to an ultrasound image of a fetus or an illustrated image corresponding to an actual photographed image. . That is, the computing device 100 may generate a target prediction image corresponding to the reference image based on the reference image. The generation of the target prediction image may be to generate an image having a specific expression method based on the reference image. For example, the fetal face may be predicted based on an image related to an ultrasound image of the fetus, which is difficult to visually recognize. In an embodiment, the reference image may include a first reference image based on the similar image derivation and a second reference image based on the similar image adjustment. That is, the computing device 100 may generate a target prediction image related to the final output of the present disclosure through first image data including a plurality of first reference images and second image data including a plurality of second reference images. can

More specifically, the computing device 100 may receive the first image data and the second image data from the user terminal 10 . The first image data may be image data composed of a plurality of first reference images, and the second image data may be image data composed of a plurality of second reference images. In other words, the first image data may be an image including a plurality of first reference images as frames, and the second image data may be an image including a plurality of second reference images as frames. As a specific example, the plurality of first reference images may be images related to an ultrasound image of a fetus, and the plurality of second reference images may be images related to an actual image of a parent. The detailed description of each reference image described above is only an example, and the present disclosure is not limited thereto.

In the present disclosure, each of the first reference image and the second reference image may be utilized as an image for deriving a similarity image and an image for adjusting the derived similarity image, respectively, as described above. That is, the computing device 100 of the present disclosure provides a target prediction image by adjusting a process of deriving one or more similar images based on a plurality of first reference images and adjusting the derived one or more similar images based on a plurality of second reference images. The process of creating . The computing device 100 derives one or more similar images having a degree of similarity or higher through the first reference image, and adjusts the derived one or more similar images based on the second reference image, thereby meeting the user's intention. A predictive image can be generated.

As described above, the computing device 100 of the present disclosure performs two processes, such as deriving a similar image and adjusting the derived similar image through various reference images (ie, the first reference image and the second reference image). , it is possible to improve the accuracy of the output target prediction image. This may have an effect of generating a target prediction image with high accuracy in response to a reference image related to an unclear reference image, an image having a non-uniform shooting angle, or an image containing a lot of noise.

In an embodiment, the computing device 100 may be a terminal or a server, and may include any type of device. The computing device 100 is a digital device, and may be a digital device equipped with a processor, such as a laptop computer, a notebook computer, a desktop computer, a web pad, and a mobile phone, and having a computing power having a memory. The computing device 100 may be a web server that processes a service. The types of computing devices described above are merely examples, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the computing device 100 may be a server that provides a cloud computing service. More specifically, the computing device 100 is a type of Internet-based computing, and may be a server that provides a cloud computing service that processes information not with a user's computer but with another computer connected to the Internet. The cloud computing service may be a service that stores data on the Internet and allows the user to use it anytime and anywhere through Internet access without installing necessary data or programs on his/her computer. Easy to share and deliver with a click. In addition, cloud computing service not only stores data on a server on the Internet, but also allows users to perform desired tasks using the functions of applications provided on the web without installing a separate program, and multiple people can simultaneously view documents. It may be a service that allows you to work while sharing. In addition, the cloud computing service may be implemented in the form of at least one of Infrastructure as a Service (IaaS), Platform as a Service (PaaS), Software as a Service (SaaS), a virtual machine-based cloud server, and a container-based cloud server. . That is, the computing device 100 of the present disclosure may be implemented in the form of at least one of the above-described cloud computing services. The detailed description of the above-described cloud computing service is merely an example, and may include any platform for building the cloud computing environment of the present disclosure.

A learning method, a learning process, a method of deriving a similar image based on a reference image, a method of adjusting a similar image based on the reference image, and a method of generating a target prediction image corresponding to the reference image for the neural network in the present disclosure A detailed description will be provided later with reference to FIG. 2 below.

2 is a block diagram of a computing device for providing a method for generating a face image according to an embodiment of the present disclosure.

As shown in FIG. 2 , the computing device 100 may include a network unit 110 , a memory 120 , and a processor 130 . Components included in the aforementioned computing device 100 are exemplary and the scope of the present disclosure is not limited to the aforementioned components. That is, additional components may be included or some of the above-described components may be omitted according to implementation aspects for the embodiments of the present disclosure.

According to an embodiment of the present disclosure, the computing device 100 may include the user terminal 10 and the network unit 110 for transmitting and receiving data to and from the external server 20 . The network unit 110 may transmit/receive data for performing the method for generating a face image according to an embodiment of the present disclosure and a training data set for learning a neural network model to/from other computing devices, servers, and the like. That is, the network unit 110 may provide a communication function between the computing device 100 , the user terminal 10 , and the external server 20 . For example, the network unit 110 may receive reference image data from the user terminal 10 . As another example, the network unit 110 may receive a training data set for learning the feature extraction model or the image generation model of the present disclosure from the external server 20 . Additionally, the network unit 110 may allow information transfer between the computing device 100 and the user terminal 10 and the external server 20 by calling a procedure to the computing device 100 .

The network unit 110 according to an embodiment of the present disclosure includes a Public Switched Telephone Network (PSTN), x Digital Subscriber Line (xDSL), Rate Adaptive DSL (RADSL), Multi Rate DSL (MDSL), VDSL ( A variety of wired communication systems such as Very High Speed DSL), Universal Asymmetric DSL (UADSL), High Bit Rate DSL (HDSL), and Local Area Network (LAN) can be used.

In addition, the network unit 110 presented herein is CDMA (Code Division Multi Access), TDMA (Time Division Multi Access), FDMA (Frequency Division Multi Access), OFDMA (Orthogonal Frequency Division Multi Access), SC-FDMA ( A variety of wireless communication systems can be used, such as Single Carrier-FDMA) and other systems.

In the present disclosure, the network unit 110 may be configured regardless of its communication mode, such as wired and wireless, and may be composed of various communication networks such as a short-range network (PAN: Personal Area Network) and a local area network (WAN: Wide Area Network). can In addition, the network may be a well-known World Wide Web (WWW), and may use a wireless transmission technology used for short-range communication, such as infrared (IrDA) or Bluetooth (Bluetooth). The techniques described herein may be used in the networks mentioned above, as well as in other networks.

According to an embodiment of the present disclosure, the memory 120 may store a computer program for performing the method for generating a face image according to an embodiment of the present disclosure, and the stored computer program is read and driven by the processor 130 . can be In addition, the memory 120 may store any type of information generated or determined by the processor 130 and any type of information received by the network unit 110 . Also, the memory 120 may store information about similar images corresponding to the reference images or actual images corresponding to the reference images. For example, the memory 120 may temporarily or permanently store input/output data (eg, first image data, second image data, one or more similar images, target prediction images, actual images, etc.) .

According to an embodiment of the present disclosure, the memory 120 is a flash memory type, a hard disk type, a multimedia card micro type, and a card type memory (eg, a SD or XD memory, etc.), Random Access Memory (RAM), Static Random Access Memory (SRAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Programmable Read (PROM) -Only Memory), a magnetic memory, a magnetic disk, and an optical disk may include at least one type of storage medium. The computing device 100 may operate in relation to a web storage that performs a storage function of the memory 120 on the Internet. The description of the above-described memory is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the processor 130 may be configured with one or more cores, and may include a central processing unit (CPU) and a general purpose graphics processing unit (GPGPU) of a computing device. , data analysis such as a tensor processing unit (TPU), and a processor for deep learning.

The processor 130 may read a computer program stored in the memory 120 to perform data processing for deep learning according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, the processor 130 may perform an operation for learning the neural network. The processor 130 for learning of the neural network, such as processing input data for learning in deep learning (DL), extracting features from input data, calculating an error, updating the weight of the neural network using backpropagation calculations can be performed.

Also, in the processor 130, at least one of a CPU, a GPGPU, and a TPU may process learning of a network function. For example, the CPU and the GPGPU can process learning of a network function and data classification using the network function. Also, in an embodiment of the present disclosure, learning of a network function and data classification using the network function may be processed by using the processors of a plurality of computing devices together. In addition, the computer program executed in the computing device according to an embodiment of the present disclosure may be a CPU, GPGPU or TPU executable program.

In the present specification, a network function may be used interchangeably with an artificial neural network and a neural network. In the present specification, the network function may include one or more neural networks, and in this case, the output of the network function may be an ensemble of the outputs of the one or more neural networks.

The processor 130 may read a computer program stored in the memory 120 to provide a feature extraction model and an image generation model according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, the processor 130 may generate a target prediction image based on a reference image. According to an embodiment of the present disclosure, the processor 130 may perform calculation for training the classification model.

According to an embodiment of the present disclosure, the processor 130 may typically process the overall operation of the computing device 100 . The processor 130 processes signals, data, information, etc. input or output through the above-described components or runs an application program stored in the memory 120, thereby providing or processing appropriate information or functions to the user or user terminal. can do.

According to an embodiment of the present disclosure, the processor 130 may acquire image data including a plurality of reference images. Acquisition of image data according to an embodiment of the present disclosure may include receiving or loading image data stored in the memory 120 . Also, the image data acquisition may be receiving or loading data from another storage medium, another computing device, or a separate processing module in the same computing device based on a wired/wireless communication means.

According to an embodiment of the present disclosure, the processor 130 may acquire first image data and second image data. Here, the image data may mean a combination of image data. Each image data may be composed of a plurality of images as a frame.

In the present disclosure, the first image data may be image data including a plurality of first reference images based on derivation of one or more similar images, and the second image data may include a plurality of images based on adjustment of the derived similar images. may be image data including a second reference image of

That is, as shown in FIG. 3 , the processor 130 derives one or more similar images based on a plurality of first reference images constituting the first image data, and a plurality of second images constituting the second image data. By adjusting the similarity image derived based on the reference image, a target prediction image matching the user's intention may be generated.

According to an embodiment of the present disclosure, image data may include domain information. Here, the domain information may be information subdivided according to various methods of configuring an image. That is, the domain information of the image data may include information serving as a reference for distinguishing one group of image data from another group. As a specific example, the first image data may include first domain information indicating that first images constituting the first image data are related to an ultrasound image of a fetus. As another example, the second image data may include second domain information indicating that the second images constituting the second image data relate to the parent of the fetus. As another example, the third image data may include third domain information indicating that third images constituting the third image data are related to the side view of the face. As an additional example, the fourth image data may include fourth domain information indicating that fourth images constituting the fourth image data relate to the frontal view of the face. The domain information related to each of the above-described image data is merely an example to help the understanding of the present disclosure, and the present disclosure is not limited thereto.

According to an embodiment, the first image data and the second image data may be configured to include different domain information. As a specific example, when the first image data includes first domain information relating to an ultrasound image of a fetus, the second image data may include second domain information relating to an actual image of a parent of a fetus. That is, the processor 130 obtains two image data having different domain information, uses images included in the first image data in one or more similar image search processes, and uses images included in the second image data. It can be used in the process of adjusting the derived similar image. In other words, the output accuracy of the target prediction image may be further improved through a plurality of image data configured in different ways.

According to an embodiment of the present disclosure, when acquiring a plurality of image data, the processor 130 may divide each image data into first image data and second image data. In the present disclosure, the first image data may include first reference images serving as a reference for deriving a similar image, and the second image data may include second reference images serving as a reference for adjusting the derived similar image. . That is, when acquiring two pieces of image data, the processor 130 determines one of the image data as the first image data that is a reference for deriving the similar image, and adjusts the other image data to the derived similar image. It may be determined as the second image data as a reference for .

In more detail, the processor 130 may divide each image data into first image data and second image data based on domain information of each of the plurality of image data. Specifically, when the processor 130 acquires a plurality of image data, domain information of each image data may be identified. The domain information of each image data may be information serving as a reference for distinguishing one group of image data from another group. The processor 130 may determine the first image data and the second image data through comparison between domain information of each image data. The processor 130 identifies domain information of each image data, determines image data including domain information expected to include less feature information on the face image as the first image data, and compares the image data with respect to the face image relatively. Image data including domain information expected to include more feature information may be determined as the second image data. For example, since the second reference images used for adjusting similar images are images that serve as a reference for finely adjusting or correcting the derived similar images, face than the first reference images used for deriving (or searching for) similar images. It may be necessary to further include information on the detailed details of Accordingly, the processor 130 may determine, as the second image data, image data including domain information, which is expected to include a relatively large amount of feature information on the face image.

As a specific example, when one image data includes first domain information that relates to an ultrasound image of a fetus and the other image data includes second domain information that relates to an actual image of a parent of a fetus, the processor ( 130) determines one image data including the first domain information as the first image data as a reference for deriving a similar image, and uses the other image data including the second domain information as a reference for adjusting the similarity image. This may be determined as the second image data. That is, the processor 130 determines that, based on the domain information of each image data, the actual image of the parent contains relatively more specific information about the face than the ultrasound image of the fetus (that is, it relates to detailed details that can be used for detailed adjustments). (including more information), one image data may be determined as the first image data, and the other image data may be determined as the second image data. The detailed description of domain information related to each of the above-described image data is only an example, and the present disclosure is not limited thereto.

That is, when acquiring a plurality of image data, the processor 130 converts each image data to first image data for deriving a similar image and second image data for adjusting similar images based on domain information of each of the acquired image data. can be distinguished as In other words, when receiving a plurality of image data from the user terminal 10, the processor 130 automatically determines how to utilize each of the plurality of image data in the process of generating the final output image, thereby providing convenience to the user. can provide Additionally, the output accuracy of the target prediction image, which will be described below, may be further improved through the above-described operation of dividing the plurality of image data by the processor 130 .

According to an embodiment of the present disclosure, the processor 130 may derive one or more similar images based on the first image data. The processor 130 may extract common characteristic information corresponding to a plurality of first reference images constituting the first image data. The common characteristic information is information that is a basis for deriving a similar image, and may include information about a face landmark. The common characteristic information may be information regarding characteristic information commonly included in the plurality of first reference images. The characteristic information is information related to the feature values included in each image, and for example, information about composition, sharpness, brightness, contrast, obstacles (hands, feet, placenta), eyes, nose, mouth, face outline, etc. can The common characteristic information may be information on image characteristics commonly included in the plurality of first reference images and similar images. The processor 130 may derive one or more similar images based on the common characteristic information. That is, the processor 130 may identify common characteristics on the first reference images through common characteristic information, and derive one or more similar images based on the identified common characteristics. In other words, the processor 130 may derive one or more similar images based on a composition, a contrast, an obstacle, an eye, a nose, a mouth, a face outline, etc. commonly observed in the first reference images.

In more detail, the processor 130 may obtain one or more characteristic information corresponding to the first reference image by processing each of the plurality of first reference images constituting the first image data as an input of the feature extraction model.

In an embodiment, the feature extraction model may be configured through a convolutional neural network (CNN) trained by a heat map regression method. Specifically, the feature extraction model may be a neural network model trained to output a heatmap indicating the existence probability of each feature point for each feature point. For example, this feature extraction model uses a machine learning algorithm to identify specific points called 68 landmarks on the face (e.g., the top of the chin, the outer edge of the eye, the inner eyebrow? can be learned to do. The feature extraction model may acquire one or more feature information corresponding to each of the plurality of first reference images, and may acquire common feature information based thereon. That is, the common characteristic information may include information on facial feature points commonly included in each first reference image.

In another embodiment, the feature extraction model may be constructed via at least a portion of a learned autoencoder. Specifically, the feature extraction model may be implemented through a dimensionality reduction network function constituting a learning autoencoder.

The autoencoder learned in the present disclosure may be configured with a dimensionality reduction network function and a dimensionality restoration network function learned to output output data similar to input data by the processor 130 . The autoencoder may be a kind of artificial neural network for outputting output data similar to input data. More specifically, the autoencoder may include at least one hidden layer, and at least one hidden layer may be disposed between the input/output layers. The number of nodes in each layer may be reduced from the number of nodes of the input layer to an intermediate layer called the bottleneck layer (encoding), and then expanded symmetrically with reduction from the bottleneck layer to the output layer (symmetrically with the input layer). Autoencoders can perform non-linear dimensionality reduction. The number of input layers and output layers may correspond to the number of items of input data remaining after preprocessing of input data. In the autoencoder structure, the number of nodes of the hidden layer included in the dimensionality reduction network function (ie, the encoder) may have a structure that decreases as the distance from the input layer increases. If the number of nodes in the bottleneck layer (the layer with the fewest nodes located between the encoder and decoder) is too small, a sufficient amount of information may not be conveyed, so a certain number or more (e.g., more than half of the input layer, etc.) ) may be maintained.

According to an embodiment, the processor 130 may learn the autoencoder through an unsupervised learning method. Specifically, the processor 130 may train a dimensionality reduction network function (eg, encoder) and a dimensionality restoration network function (eg, decoder) that configure the autoencoder to output output data similar to input data. Only the core feature data (or features) of the input image data in the encoding process through the dimensionality reduction network function can be learned through the hidden layer and the rest of the information can be lost. In this case, the output data of the hidden layer may be an approximation of the input data rather than a perfect copy value in the decoding process through the dimensional reconstruction network function. That is, the processor 130 may learn the autoencoder by adjusting the weight so that the output data and the input data are the same as possible. The input data used for learning of the autoencoder in the present disclosure may be, for example, a live-action image of a fetus.

In an embodiment, the autoencoder learned through the above-described process may output an output image (ie, a live-action image) corresponding to the real image of the fetus by receiving the actual image of the fetus as an input. In this case, the output image may be an image related to an approximation rather than a value that is a perfect copy of the actual image of the fetus.

The dimensionality reduction network function (ie, encoder) included in the learned autoencoder may take an image (eg, ultrasound image of a fetus) as input and extract features (ie, embeddings). That is, the dimension reduction network function receives learning input data related to an ultrasound image of a fetus from the processor 130 and designates a feature vector column of the learning input data as an output to learn an intermediate process in which the input data is converted into features. . In addition, the processor 130 may pass the embeddings (ie, features) related to the output of the dimension reduction network function to the dimension reconstruction network function, and the dimension reconstruction network function receives the feature as an input and an image (ie, a fetus) related to the feature. of ultrasound image and approximation) can be output.

In other words, the dimensionality reduction network function included in the learned autoencoder may be trained to extract features for input data, and the dimensionality restoration network function may be trained to output an image corresponding to the features.

According to an embodiment of the present disclosure, the processor 130 outputs one or more characteristic information by processing each of the plurality of first reference images constituting the first image data as an input of a characteristic extraction model composed of a dimensionality reduction network function. can Also, the processor 130 may acquire common characteristic information based on one or more characteristic information output in response to each first reference image.

4 , the processor 130 generates a first reference first image 211 , a first reference second image 212 , and a first reference n-th image 213 constituting the first image data. ) can be treated as inputs to each of one or more dimensionality reduction network functions. In this case, each dimension reduction network function is implemented through the encoder part in the learned autoencoder, and when a specific image is input, core features corresponding to the specific image, that is, characteristic information can be output. Here, the characteristic information is information related to the feature value included in each image, and includes, for example, information about composition, sharpness, brightness, contrast, obstacles (hands, feet, placenta), eyes, nose, mouth, face outline, etc. can do.

In other words, the first-dimensional reconstruction network function 221 receives the first reference first image 211 as an input and outputs the first characteristic information 231, and the second-dimensional reconstruction network function 222 receives the first reference The second image 212 is input to output the second characteristic information 232 , and the n-th dimensional reconstruction network function 223 takes the first reference n-th image 213 as an input and outputs the n-th characteristic information ( 233) can be printed.

The processor 130 may acquire the common characteristic information 240 based on the characteristic information output through each dimension restoration network function. The processor 130 may generate common characteristic information by merging the first characteristic information 231 , the second characteristic information 232 , and the n-th characteristic information 233 . In an embodiment, the processor 130 may generate common characteristic information through an average value or a weighted sum of each characteristic information. In an additional embodiment, a separate additional neural network model for summing each characteristic information may be configured, and common characteristic information may be generated through the additional neural network model. That is, when a plurality of pieces of characteristic information are input, the additional neural network model may output common characteristic information by merging each characteristic information. The detailed description of the above-described method for generating common characteristic information is only an example, and the present disclosure is not limited thereto.

That is, the processor 130 may generate common characteristic information by merging a plurality of characteristic information corresponding to each of the plurality of first reference images by using a weighted sum of each characteristic information, an average value, or an additional neural network model. . Accordingly, the generated common characteristic information may include information on characteristics commonly included in the first reference images.

In a further embodiment, the feature extraction model may output one or more feature information and one or more domain information corresponding to each of the first reference images. The feature extraction model may be implemented through a part of the learned autoencoder, that is, the encoder. In the pre-learning process of the processor 130 autoencoder, domain information may be tagged to input data. For example, in the process of inputting a live-action image of a fetus into the autoencoder as first input data, the processor 130 may tag domain information of "an ultrasound picture of a fetus" to the first input data. Accordingly, the autoencoder may be learned to output domain information such as output data similar to the first input data and an ultrasound picture of a fetus in the learning process.

Also, the processor 130 may derive one or more similar images based on the common characteristic information. More specifically, the processor 130 may derive one or more similarity images with a degree of similarity or higher by performing a search on the target image database based on the common characteristic information. In this case, the target image database may be configured to correspond to each of a plurality of domains. For example, the target image database may include a first target image database classified into a first domain related to an ultrasound image of a fetus and a second target image database classified into a second domain related to a user's actual photographed image. can The detailed description of the above-described first domain and second domain is only an example, and the present disclosure is not limited thereto.

In an embodiment, the processor 130 determines or identifies a target image database for deriving or searching for a similar image among target image databases constructed corresponding to each of a plurality of domains based on domain information corresponding to the first reference image. can do. For example, the processor 130 may acquire domain information about the first reference images through the feature extraction model, and determine a target database divided into domains matching the domain information as a target image database for deriving similar images. can That is, the processor 130 may improve data retrieval efficiency by classifying and constructing target image databases based on similar image derivation or retrieval corresponding to various domains. In other words, since a similar image is searched in the target image database that matches the domain information of the first reference image, the search speed is increased, and thus the efficiency in the similar image derivation process can be maximized.

In an embodiment, the processor 130 may derive one or more similar images with a predetermined similarity or higher by performing a search on a target image database based on common characteristic information. In this case, the similarity determination for determining whether the similarity is greater than or equal to the predetermined similarity may be based on the Euclidean distance or the cosine similarity. Specifically, the common characteristic information may be generated through merging between characteristic information of each image data. In this case, each of the feature information is output through the feature extraction model, and may be a feature vector including information about the feature of each image (ie, the first reference image). For example, each characteristic information may be displayed in one region on the vector space. Here, similar characteristic information may be located close to each other on a similar region, and may be located relatively farther apart as they are different.

That is, the processor 130 may derive, as one or more similar images, images including feature vectors disposed to be adjacent in distance to feature vectors corresponding to the corresponding common feature information in the target image database. In other words, the processor 130 may evaluate the similarity by comparing the common characteristic information and the distance on the vector space of images included in the target image database.

In an additional embodiment, the processor 130 may determine at least some of the plurality of first images constituting the first image data as the plurality of first reference images. In the present disclosure, the plurality of first reference images may be images based on a similar image search. Since the plurality of first reference images serves as a reference for similar image search, when the corresponding first reference image includes clear characteristic information, the accuracy of the searched similar image may be increased. In other words, the more clearly the characteristics are reflected in the image, the more similar images close to the user's intention can be retrieved from the target image database. For example, when images that are not clear through composition, noise, contrast, obstacles (eg, hand, foot, placenta, etc.) are determined as the first reference image, one or more similar images retrieved from the target image database are may be different from the intention of , and, as a result, may cause generation of target prediction data that does not conform to the user's intention. Accordingly, the processor 130 may select images having relatively many characteristics (ie, relatively clear) from among the plurality of first images constituting the first image data and determine the plurality of first images.

Specifically, the processor 130 may obtain one or more characteristic information corresponding to each image by processing each of the plurality of first images constituting the first image data as an input of the feature extraction model. Here, the characteristic information may be related to embedding in a vector space corresponding to each image.

Also, the processor 130 may calculate a probability value corresponding to each of the plurality of items based on one or more characteristic information. In this case, the plurality of items relate to components included in the face image, and may be, for example, information about eyes, nose, mouth, ears, and the like. The probability value may mean an output value related to the degree of prediction of whether the components are included in each image. Specifically, the processor 130 processes each of one or more characteristic information related to the output of the characteristic extraction model as an input of a probability estimation model including one or more network functions, and a probability value for each item corresponding to each characteristic information can be printed out. Here, the probability estimation model may be a neural network model trained to convert the output of the feature extraction model (ie, feature information related to embedding in a vector space) into probability values of components. The probability estimation model may include one or more logistic regression models trained to output each of one or more probability values corresponding to each of a plurality of items (ie, components of a face) constituting the face image. For example, the first logistic regression model may output a probability value related to 'nose' as 0.6 by inputting the first characteristic information as an input, and the second logistic regression model may receive the first characteristic information as an input and output the 'eye' as an input. The probability value related to ' can be output as 0.85. The detailed description of the output value of the logistic regression model described above is only an example, and the present disclosure is not limited thereto. In one embodiment, at least some of the parameters of each of the one or more logistic regression models may be shared with each other.

Also, the processor 130 may select some of the plurality of first images as the plurality of first reference images based on probability values corresponding to each of the plurality of items. As a specific example, when the first images are input to the feature extraction model, one or more feature information corresponding to each image may be output. The processor 130 may process one or more characteristic information as inputs of the probability estimation model to output a probability value corresponding to each of the plurality of items. For example, the probability estimation model may output a probability value of 0.1 for an eye, an output value of a nose as 0.8, and an output value of a mouth as 0.9 based on the characteristic information corresponding to one first image. That is, probability values related to each of the eyes, nose, and mouth may be obtained in correspondence to the corresponding image. In this way, the probability estimation model may output respective probability values in relation to items included in the corresponding image. In this case, the higher the probability value related to a specific item, the higher the probability that the characteristic of the corresponding item exists in the image, and the lower the probability value, the lower the probability that the characteristic of the corresponding item exists in the image. The processor 130 may determine images having a maximum number of items equal to or greater than a predetermined probability value (eg, 0.8) as the plurality of first reference images. That is, the processor 130 may identify an image including more item values from among the first images constituting the first image data and determine the plurality of first reference images. In other words, the processor 130 may select frames having a plurality of characteristic information as the first reference image based on the generation of the common characteristic information. This improves the accuracy of common characteristic information obtained by configuring a plurality of first reference images through relatively clear data (or image data including a plurality of characteristic information) within the image data, and as a result, Generation of target prediction data more conforming to the user's intention can be achieved.

According to an embodiment of the present disclosure, the processor 130 may generate a target prediction image by adjusting one or more similar images based on a plurality of second reference images constituting the second image data. For example, the plurality of second reference images may be images related to an actual image of a parent. A configuration for generating a target prediction image by adjusting one or more similar images will be described in detail later with reference to FIG. 5 .

Specifically, the processor 130 may obtain one or more additional characteristic information by processing one or more second reference images (ie, second reference first image to second reference n-th image) as an input of the feature extraction model. . Here, the additional characteristic information may mean each characteristic information output in response to each second reference image. That is, the additional characteristic information is information related to a feature value included in each of the plurality of second reference images, for example, composition, sharpness, brightness, contrast, obstacles (hands, feet, placenta), eyes, nose, and mouth. , and may include information about a face outline, etc. Also, the processor 130 may acquire the additional common characteristic information 312 based on one or more pieces of additional characteristic information. The processor 130 may generate the additional common characteristic information 312 by merging the first additional characteristic information to the nth additional characteristic information. That is, the generated additional common characteristic information may be information on image characteristics commonly included in the plurality of second reference images.

In an embodiment, the processor 130 may generate the additional common characteristic information 312 through an average value or a weighted sum of each additional characteristic information. In an additional embodiment, a separate additional neural network model for merging each additional characteristic information may be configured, and additional common characteristic information 312 may be generated through the additional neural network model. That is, when a plurality of additional characteristic information is input, the additional neural network model may output additional common characteristic information 312 by merging each additional characteristic information. The detailed description of the above-described method for generating the additional common characteristic information 312 is only an example, and the present disclosure is not limited thereto.

That is, the processor 130 generates additional common characteristic information by merging a plurality of additional characteristic information corresponding to each of the plurality of second reference images by using a weighted sum of each additional characteristic information, an average value, or an additional neural network model. can do. Accordingly, the generated additional common characteristic information may include information on characteristics commonly included in the second reference images.

The processor 130 may generate a target prediction image based on the additional common characteristic information 312 and the common characteristic information 2410 . Specifically, the processor 130 may acquire the final characteristic information 320 based on the additional common characteristic information 312 and the common characteristic information 240 . The final characteristic information 320 may be generated by reflecting the additional common characteristic information 312 in the common characteristic information 240 . That is, the final characteristic information 320 may include information in which at least a portion of the common characteristic information 240 is adjusted based on the additional common characteristic information 312 .

In addition, the processor 130 may generate the target prediction image by processing the final characteristic information 320 as an input of the image generation model 330 . Here, the image generation model 330 may be configured through at least a part of the learned autoencoder. The learned autoencoder may include a dimensionality reduction network function and a dimensionality restoration network function. In this case, the dimensionality reduction network function may be pre-trained to extract features for the input data, and the dimension restoration network function may be pre-trained to output an image corresponding to the features.

The image generation model 330 may be implemented through a dimensional reconstruction network function (ie, decoder) part of the learned autoencoder. The image generation model 330 composed of a dimensional reconstruction network function can be pre-trained to output an image related to the feature (ie, an image that is approximate to the actual photographed image) by taking a feature (ie, embedding in vector space) as an input. have.

That is, the processor 130 processes the final characteristic information 320 , which is information adjusted based on the additional common characteristic information 312 in which at least a portion of the common characteristic information 240 to the image generation model 330 , as an input to target the target A predictive image can be generated. In this case, since the final characteristic information 320 is information in which the common characteristic information 240 corresponding to the first reference images is adjusted based on the additional common characteristic information 312 corresponding to the second reference images, the image is generated The target prediction image output through the model 330 may be related to an image in which an image similar to the first reference images is adjusted based on the second reference image.

For example, as the image expression method for each user and the target image to be output based on a specific image (eg, a reference image) are different, it is difficult to generate a target image that meets the intention of each user. there may be For a specific example, if an image related to the future face of the fetus is to be generated based on information about the ultrasound image of the fetus, the target image may be changed according to the non-uniform shooting angle and noise of the ultrasound image that is the basis for generating the target image. It can be difficult to create. In other words, in the case of an ultrasound image used as a reference image, it contains only the morphological information of the fetus, so it is difficult to learn the neural network by recognizing the natural face shape of the fetus. There is a fear that the neural network is trained to output the face shape. That is, there is a limit to generating and providing an image or image suitable for an arbitrary intention.

The processor 130 of the present disclosure generates a target prediction image by deriving one or more similar images based on a plurality of first reference images and adjusting the derived one or more similar images based on a plurality of second reference images. steps can be performed. The processor 130 derives one or more similar images having a degree of similarity or more preset through the first reference image, and adjusts the derived one or more similar images based on the second reference image, thereby predicting a target matching the user's intention. You can create an image. For example, the processor 130 derives a similar image by searching for a similar image from a target database based on an ultrasound image of a specific fetus, and adjusts the derived similar image based on the actual image of the parent, so that prediction accuracy is improved A target prediction image (ie, a photorealistic image of the future face of the fetus) may be generated.

As described above, the processor 130 of the present disclosure performs two processes, such as deriving a similar image and adjusting the derived similar image through various reference images (that is, the first reference image and the second reference image), It is possible to improve the accuracy of the output target prediction image. This may have an effect of generating a target prediction image with high accuracy in response to a reference image related to an unclear reference image, an image having a non-uniform shooting angle, or an image containing a lot of noise.

6 is a flowchart exemplarily illustrating a method for generating a face image related to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, the method may include obtaining ( 410 ) first image data and second image data.

According to an embodiment of the present disclosure, the method may include extracting common characteristic information corresponding to a plurality of first reference images constituting the first image data ( 420 ).

According to an embodiment of the present disclosure, the method may include deriving one or more similar images based on common characteristic information ( 430 ).

According to an embodiment of the present disclosure, the method may include generating (440) a target prediction image by adjusting one or more similar images based on a plurality of second reference images constituting the second image data. can

The order of the steps illustrated in FIG. 6 described above may be changed if necessary, and at least one or more steps may be omitted or added. That is, the above-described steps are merely an embodiment of the present disclosure, and the scope of the present disclosure is not limited thereto.

7 is a schematic diagram illustrating one or more network functions related to an embodiment of the present disclosure.

Throughout this specification, computational model, neural network, network function, and neural network may be used interchangeably. A neural network may be composed of a set of interconnected computational units, which may generally be referred to as “nodes”. These “nodes” may also be referred to as “neurons”. A neural network is configured to include at least one or more nodes. Nodes (or neurons) constituting neural networks may be interconnected by one or more “links”.

In the neural network, one or more nodes connected through a link may relatively form a relationship between an input node and an output node. The concepts of an input node and an output node are relative, and any node in an output node relationship with respect to one node may be in an input node relationship in a relationship with another node, and vice versa. As described above, an input node-to-output node relationship may be created around a link. One or more output nodes may be connected to one input node through a link, and vice versa.

In the relationship between the input node and the output node connected through one link, the value of the output node may be determined based on data input to the input node. Here, a node interconnecting the input node and the output node may have a weight. The weight may be variable, and may be changed by a user or an algorithm in order for the neural network to perform a desired function. For example, when one or more input nodes are interconnected to one output node by respective links, the output node sets values input to input nodes connected to the output node and links corresponding to the respective input nodes. An output node value may be determined based on the weight.

As described above, in a neural network, one or more nodes are interconnected through one or more links to form an input node and an output node relationship in the neural network. The characteristics of the neural network may be determined according to the number of nodes and links in the neural network, the correlation between the nodes and the links, and the value of a weight assigned to each of the links. For example, when the same number of nodes and links exist and there are two neural networks having different weight values between the links, the two neural networks may be recognized as different from each other.

A neural network may include one or more nodes. Some of the nodes constituting the neural network may configure one layer based on distances from the initial input node. For example, a set of nodes having a distance of n from the initial input node is You can configure n layers. The distance from the initial input node may be defined by the minimum number of links that must be passed to reach the corresponding node from the initial input node. However, the definition of such a layer is arbitrary for description, and the order of the layer in the neural network may be defined in a different way from the above. For example, a layer of nodes may be defined by a distance from the final output node.

The initial input node may mean one or more nodes to which data is directly input without going through a link in a relationship with other nodes among nodes in the neural network. Alternatively, in a relationship between nodes based on a link in a neural network, it may mean nodes that do not have other input nodes connected by a link. Similarly, the final output node may refer to one or more nodes that do not have an output node in relation to other nodes among nodes in the neural network. In addition, the hidden node may mean nodes constituting the neural network other than the first input node and the last output node. The neural network according to an embodiment of the present disclosure may be a neural network in which the number of nodes in the input layer may be the same as the number of nodes in the output layer, and the number of nodes decreases and then increases again as progresses from the input layer to the hidden layer. can Also, in the neural network according to another embodiment of the present disclosure, the number of nodes in the input layer may be less than the number of nodes in the output layer, and the number of nodes may be reduced as the number of nodes progresses from the input layer to the hidden layer. have. In addition, the neural network according to another embodiment of the present disclosure may be a neural network in which the number of nodes in the input layer may be greater than the number of nodes in the output layer, and the number of nodes increases as the number of nodes progresses from the input layer to the hidden layer. can The neural network according to another embodiment of the present disclosure may be a neural network in a combined form of the aforementioned neural networks.

A deep neural network (DNN) may refer to a neural network including a plurality of hidden layers in addition to an input layer and an output layer. Deep neural networks can be used to identify the latent structures of data. In other words, it can identify the potential structure of photos, texts, videos, voices, and music (e.g., what objects are in the photos, what the text and emotions are, what the texts and emotions are, etc.) . Deep neural networks include convolutional neural networks (CNNs), recurrent neural networks (RNNs), auto encoders, generative adversarial networks (GANs), and restricted boltzmann machines (RBMs). machine), a deep trust network (DBN), a Q network, a U network, a Siamese network, and the like. The description of the deep neural network described above is only an example, and the present disclosure is not limited thereto.

The neural network may be learned by at least one of teacher learning (supervised learning), unsupervised learning, and semi-supervised learning. The training of the neural network is to minimize the error in the output. In the training of a neural network, iteratively input the training data into the neural network, calculate the output of the neural network and the target error for the training data, and calculate the error of the neural network from the output layer of the neural network to the input layer in the direction to reduce the error. It is a process of updating the weight of each node in the neural network by backpropagation in the direction. In the case of teacher learning, learning data in which the correct answer is labeled in each learning data is used (ie, labeled learning data), and in the case of comparative learning, the correct answer may not be labeled in each learning data. That is, for example, learning data in the case of teacher learning related to data classification may be data in which categories are labeled in each of the learning data. The labeled training data is input to the neural network, and an error can be calculated by comparing the output (category) of the neural network with the label of the training data. As another example, in the case of comparison learning related to data classification, an error may be calculated by comparing the input training data with the neural network output. The calculated error is back propagated in the reverse direction (ie, from the output layer to the input layer) in the neural network, and the connection weight of each node of each layer of the neural network may be updated according to the back propagation. The change amount of the connection weight of each node to be updated may be determined according to a learning rate. The computation of the neural network on the input data and the backpropagation of errors can constitute a learning cycle (epoch). The learning rate may be applied differently according to the number of repetitions of the learning cycle of the neural network. For example, in the early stage of learning of a neural network, a high learning rate can be used to enable the neural network to quickly obtain a certain level of performance, thereby increasing efficiency, and using a low learning rate at a later stage of learning can increase accuracy.

In the training of neural networks, in general, the training data may be a subset of real data (that is, data to be processed using the trained neural network), and thus, the error on the training data is reduced, but the error on the real data is reduced. There may be increasing learning cycles. Overfitting is a phenomenon in which errors on actual data increase by over-learning on training data as described above. For example, a phenomenon in which a neural network that has learned a cat by seeing a yellow cat does not recognize that it is a cat when it sees a cat other than yellow may be a type of overfitting. Overfitting can act as a cause of increasing errors in machine learning algorithms. In order to prevent such overfitting, various optimization methods can be used. In order to prevent overfitting, methods such as increasing training data, regularization, or dropout in which a part of nodes in the network are omitted in the process of learning, may be applied.

Throughout this specification, computational model, neural network, network function, and neural network may be used interchangeably. (Hereinafter, the neural network is unified and described.) The data structure may include a neural network. And the data structure including the neural network may be stored in a computer-readable medium. Data structures, including neural networks, may also include data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation functions associated with each node or layer of the neural network, and loss functions for learning the neural network. have. A data structure comprising a neural network may include any of the components disclosed above. That is, the data structure including the neural network includes all or all of the data input to the neural network, the weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, the activation function associated with each node or layer of the neural network, and the loss function for training the neural network. may be configured including any combination of In addition to the above-described configurations, a data structure including a neural network may include any other information that determines a characteristic of a neural network. In addition, the data structure may include all types of data used or generated in the operation process of the neural network, and is not limited to the above. Computer-readable media may include computer-readable recording media and/or computer-readable transmission media. A neural network may be composed of a set of interconnected computational units, which may generally be referred to as nodes. These nodes may also be referred to as neurons. A neural network is configured to include at least one or more nodes.

The data structure may include data input to the neural network. A data structure including data input to the neural network may be stored in a computer-readable medium. The data input to the neural network may include learning data input in a neural network learning process and/or input data input to the neural network in which learning is completed. Data input to the neural network may include pre-processing data and/or pre-processing target data. The preprocessing may include a data processing process for inputting data into the neural network. Accordingly, the data structure may include data to be pre-processed and data generated by pre-processing. The above-described data structure is merely an example, and the present disclosure is not limited thereto.

The data structure may include the weights of the neural network. (In this specification, weight and parameter may be used interchangeably.) And the data structure including the weight of the neural network may be stored in a computer-readable medium. The neural network may include a plurality of weights. The weight may be variable, and may be changed by a user or an algorithm in order for the neural network to perform a desired function. For example, when one or more input nodes are interconnected to one output node by respective links, the output node sets values input to input nodes connected to the output node and links corresponding to the respective input nodes. An output node value may be determined based on the parameter. The above-described data structure is merely an example, and the present disclosure is not limited thereto.

By way of example and not limitation, the weight may include a weight variable in a neural network learning process and/or a weight in which neural network learning is completed. The variable weight in the neural network learning process may include a weight at a time point at which a learning cycle starts and/or a weight variable during the learning cycle. The weight for which neural network learning is completed may include a weight for which a learning cycle is completed. Accordingly, the data structure including the weights of the neural network may include a data structure including the weights that vary in the process of learning the neural network and/or the weights on which the learning of the neural network is completed. Therefore, it is assumed that the above-described weights and/or combinations of weights are included in the data structure including the weights of the neural network. The above-described data structure is merely an example, and the present disclosure is not limited thereto.

The data structure including the weights of the neural network may be stored in a computer-readable storage medium (eg, memory, hard disk) after being serialized. Serialization can be the process of converting a data structure into a form that can be reconstructed and used later by storing it on the same or a different computing device. The computing device may serialize the data structure to send and receive data over the network. A data structure including weights of the serialized neural network may be reconstructed in the same computing device or in another computing device through deserialization. The data structure including the weight of the neural network is not limited to serialization. Furthermore, the data structure including the weights of the neural network is a data structure to increase the efficiency of computation while using the resources of the computing device to a minimum (e.g., B-Tree, Trie, m-way search tree, AVL tree, Red-Black Tree). The foregoing is merely an example, and the present disclosure is not limited thereto.

The data structure may include hyper-parameters of the neural network. In addition, the data structure including the hyperparameters of the neural network may be stored in a computer-readable medium. The hyper parameter may be a variable variable by a user. Hyperparameters are, for example, learning rate, cost function, number of iterations of the learning cycle, weight initialization (e.g., setting the range of weight values to be initialized for weights), Hidden Unit It may include the number (eg, the number of hidden layers, the number of nodes of the hidden layer). The above-described data structure is merely an example, and the present disclosure is not limited thereto.

The steps of a method or algorithm described in relation to an embodiment of the present disclosure may be implemented directly in hardware, implemented as a software module executed by hardware, or implemented by a combination thereof. A software module may include random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside in any type of computer-readable recording medium well known in the art to which the present disclosure pertains.

Components of the present disclosure may be implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium. Components of the present disclosure may be implemented as software programming or software components, and similarly, embodiments may include various algorithms implemented as data structures, processes, routines, or combinations of other programming constructs, including C, C++ , Java, assembler, etc. may be implemented in a programming or scripting language. Functional aspects may be implemented in an algorithm running on one or more processors.

Those of ordinary skill in the art of the present disclosure will recognize that the various illustrative logical blocks, modules, processors, means, circuits, and algorithm steps described in connection with the embodiments disclosed herein include electronic hardware, (convenience For this purpose, it will be understood that it may be implemented by various forms of program or design code (referred to herein as "software") or a combination of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. A person skilled in the art of the present disclosure may implement the described functionality in various ways for each specific application, but such implementation decisions should not be interpreted as a departure from the scope of the present disclosure.

The various embodiments presented herein may be implemented as methods, apparatus, or articles of manufacture using standard programming and/or engineering techniques. The term “article of manufacture” includes a computer program, carrier, or media accessible from any computer-readable device. For example, computer-readable media include magnetic storage devices (eg, hard disks, floppy disks, magnetic strips, etc.), optical disks (eg, CDs, DVDs, etc.), smart cards, and flash memory. devices (eg, EEPROMs, cards, sticks, key drives, etc.). Also, various storage media presented herein include one or more devices and/or other machine-readable media for storing information. The term “machine-readable medium” includes, but is not limited to, wireless channels and various other media that can store, hold, and/or convey instruction(s) and/or data.

It is to be understood that the specific order or hierarchy of steps in the presented processes is an example of exemplary approaches. Based on design priorities, it is to be understood that the specific order or hierarchy of steps in the processes may be rearranged within the scope of the present disclosure. The appended method claims present elements of the various steps in a sample order, but are not meant to be limited to the specific order or hierarchy presented.

The description of the presented embodiments is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present disclosure is not intended to be limited to the embodiments presented herein, but is to be construed in the widest scope consistent with the principles and novel features presented herein.

Claims (10)

A method performed on one or more processors of a computing device, comprising:
acquiring, by the processor, a plurality of image data having different domain information;
dividing, by the processor, the plurality of image data into first image data and second image data based on domain information of each image data;
extracting, by the processor, common characteristic information corresponding to a plurality of first reference images constituting the first image data;
deriving, by the processor, one or more similar images based on the extracted common characteristic information; and
generating, by the processor, a target prediction image by adjusting the one or more similar images based on a plurality of second reference images constituting the second image data;
includes,
The first image data includes the plurality of first reference images serving as a reference for deriving the one or more similar images, and the second image data includes the plurality of first reference images serving as a reference for adjusting the derived one or more similar images. characterized by including second reference images of
A method of generating a facial image performed on one or more processors of a computing device.
According to claim 1,
The common characteristic information is
Information as a basis for deriving the similar image, including information on face landmarks,
A method of generating an image of a face performed on one or more processors of a computing device.
According to claim 1,
The step of deriving the one or more similar images comprises:
obtaining, by the processor, one or more characteristic information corresponding to each of the first reference images by processing each of the plurality of first reference images as inputs of a feature extraction model;
obtaining, by the processor, the common characteristic information based on the one or more characteristic information; and
deriving, by the processor, the one or more similar images based on the common characteristic information;
containing,
A method of generating an image of a face performed on one or more processors of a computing device.
4. The method of claim 3,
The feature extraction model is
Is configured through at least a part of the learned autoencoder (AutoEncoder), characterized in that outputting the one or more common characteristic information and one or more domain information corresponding to each of the plurality of first reference images,
A method of generating an image of a face performed on one or more processors of a computing device.
4. The method of claim 3,
The step of the processor deriving the one or more similar images based on the common characteristic information comprises:
deriving, by the processor, the one or more similar images having a degree of similarity greater than or equal to a predetermined similarity by performing a search on a target image database based on the common characteristic information;
includes,
The target image database,
characterized in that it is configured corresponding to each of a plurality of domains,
A method of generating a facial image performed on one or more processors of a computing device.
According to claim 1,
obtaining, by the processor, one or more characteristic information corresponding to each image by processing, by the processor, each of a plurality of first images constituting the first image data as an input of a feature extraction model;
calculating, by the processor, a probability value corresponding to each of a plurality of items related to a component included in a face image based on the one or more characteristic information; and
selecting, by the processor, some of the plurality of first images as the plurality of first reference images based on the respective probability values;
further comprising,
A method of generating an image of a face performed on one or more processors of a computing device.
According to claim 1,
generating, by the processor, the one or more similar images based on a plurality of second reference images constituting the second image data, to generate a target prediction image;
processing, by the processor, the one or more second reference images as an input of a feature extraction model to obtain one or more additional feature information;
obtaining, by the processor, the additional common characteristic information based on the one or more additional characteristic information; and
generating, by the processor, the target prediction image based on the additional common characteristic information and the common characteristic information;
containing,
A method of generating an image of a face performed on one or more processors of a computing device.
8. The method of claim 7,
The step of generating the target prediction image includes:
obtaining, by the processor, final characteristic information based on the additional common characteristic information and the common characteristic information; and
generating, by the processor, the target prediction image by processing the final characteristic information as an input of an image generation model;
including,
The image generation model is
configured through at least part of the learned autoencoder,
A method of generating a facial image performed on one or more processors of a computing device.
a memory storing one or more instructions; and
a processor executing one or more instructions stored in the memory; including,
The processor by executing the one or more instructions,
A computing device for performing the method of claim 1 .
A computer program stored in a computer-readable recording medium in combination with a computer, which is hardware, to perform the method of claim 1 .

Images