KR20220009676A - Baby face prediction system - Google Patents

Abstract

The present invention relates to a baby face prediction system and, more specifically, to a baby face prediction system for finding a feature point from the faces of a baby's parent and a fetus and predicting and providing a baby face based on the found feature point. According to one aspect of the present invention, the baby face prediction system for compositing a baby face using a parent picture and an ultrasound fetus picture includes: a user terminal; a server providing a baby face compositing service; and a database storing a baby face image including a feature point. The server includes: an image receiving unit receiving a parent image and an ultrasound fetus picture from the user terminal; a parent image face detection unit detecting father and mother face areas from the received parent image; a feature point detection unit detecting a feature point from each of the father's and mother's faces from the detected parent face areas; a baby face search unit searching for the baby face image most similar to the detected feature points of the father and mother from the database storing the baby face image; and a baby face compositing unit for compositing the father's and mother's feature points with the baby face image searched from the baby face search unit.

Description

Baby face prediction system {BABY FACE PREDICTION SYSTEM}

The present invention relates to a baby face prediction system, and more particularly, to a baby face prediction system that finds a feature point from a baby's parent's face and predicts and provides a face of a baby to be born based on the found feature point.

In general, facial images can best represent individual characteristics and have been used as a medium to make communication easy and natural. In addition, in recent years, due to the spread of digital cameras or portable devices equipped with digital cameras, techniques for synthesizing face images of people by photographing and synthesizing them as if they were taken together are widely used. .

Image processing technology related to face synthesis is generally to create a virtual face from a face image obtained from a camera, etc. For example, an image mosaic technique or image warping technique that relies on simple computer graphic technology is used. is known

The image mosaic technique is a technique to obtain a deformed image by combining two or more images in the entire image, and the image warping technique applies an equation to a distortion filter, etc. It is a technique that transforms them so that they have a new position value.

However, in the virtual face obtained through the image mosaic technique or the image warping technique, since the synthesized face parts are not in harmony overall, a separate manual operation is required to correct it.

Recently, it can be seen that various attempts are being made to utilize these technologies for entertainment or entertainment. One example of this is a service that finds the face of a spouse or ideal type through face recognition. However, there is a problem in that a natural baby face is not provided because the service that predicts the expected face shape of a two-year-old who will be born is also provided to the user by simply synthesizing the parents' faces.

Korean Patent No. 2063294 (2019.12.31) Republic of Korea Patent Publication No. 10-2002-0007735 (Jan. 29, 2002)

An object of the present invention is to provide a baby face prediction system capable of providing a natural and relatively accurate baby face by predicting the face of a baby to be born based on the feature points of the parents' faces. do it with

In order to solve the above problems, according to an aspect of the present invention, there is provided a baby face prediction system for synthesizing a baby face using a photograph of a parent and an ultrasound photograph of a fetus, the system comprising: a user-side terminal; a server that provides baby face synthesis services; and a database in which baby face images including feature points are stored,

server,

an image receiver configured to receive a parent image and an ultrasound picture of a fetus from a user-side terminal;

a parent image face detection unit for detecting a parent and a mother face region from the received parent image;

a feature point detector for detecting feature points of each face of the mother and father from the detected face region of the parent;

a baby face search unit that searches for a baby face image most similar to parental and maternal feature points detected from a database in which baby face images are stored;

and a baby face synthesizing unit for synthesizing paternal and maternal feature points in the baby face image retrieved from the baby face search unit is characteristically configured.

In the above aspect, the server further includes a baby information receiving unit for receiving the similarity ratio of the baby's face to the parent's face to be synthesized from the user terminal, the baby's gender, and the baby's age.

In addition, in the above aspect, the baby face image stored in the baby face database is generated through a generative adversarial network (GAN) algorithm, and the feature point for the baby face image generated through the feature point detection algorithm is detected and generated It is stored in the database along with the old baby face images.

In addition, in the above-described aspect, the characteristic points of the face of the mother and father are the shape and ratio of the face, the distance between the eyes, the shape of the jaw line, the ratio of the face to the eyes, the length and ratio of the nose, the thickness of the lips, the width of the forehead, the eyebrows It includes feature points related to the length of the eyebrows, the shape of the eyebrows, and the size of the pupils.

In addition, in the above aspect, the feature points of the baby's face and the feature points of the mother's and father's faces stored in the database are extracted through a deep learning network using the ArcFace function.

In addition, in the above aspect, the feature points of the baby's face stored in the database may include: collecting baby face pictures through a network; classifying the collected baby face photo data by gender, country, race, and age; setting a feature point area including the feature points of the classified baby face photos; generating a baby face image through a GAN algorithm; analyzing baby facial feature points from the generated baby face image; and storing the generated baby face image and baby facial feature points in a database.

In addition, in the above aspect, the baby face synthesizing unit may be configured to synthesize the baby face by reflecting the similarity ratio with the parent's face received from the baby information receiving unit, the baby's gender, and the baby's age.

According to the present invention, the synthesized baby's face is related to the parent's facial feature points, and the mixing ratio of each parent's facial feature points can be adjusted so that the effect of predicting the child's face relatively accurately in various aspects can be obtained. can

1 is a view showing the overall configuration of a baby face prediction system according to the present invention;
2 is a functional block diagram schematically showing the configuration of a server that provides a service for predicting and synthesizing a baby face in the baby face prediction system according to the present invention;
3 is a view showing 96 facial feature points used in the baby face prediction system according to the present invention;
4 is a flowchart illustrating an operation of generating a baby face image stored in a database in the baby face prediction system according to the present invention;
5 is an explanatory diagram for explaining the flow of baby face prediction (or synthesis) of the baby face prediction system according to the present invention.

Advantages and features of the present invention, and methods for achieving them, will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in various different forms.

In the present specification, the present embodiment is provided to complete the disclosure of the present invention, and to fully inform those of ordinary skill in the art to which the present invention pertains to the scope of the invention. And the invention is only defined by the scope of the claims. Accordingly, in some embodiments, well-known components, well-known operations, and well-known techniques have not been specifically described in order to avoid obscuring the present invention.

Like reference numerals refer to like elements throughout. And, the terms used (mentioned) in this specification are for the purpose of describing the embodiments and are not intended to limit the present invention. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase. In addition, elements and operations referred to as 'include (or include)' do not exclude the presence or addition of one or more other elements and operations.

Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used with the meaning commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not ideally or excessively interpreted unless they are defined.

1 is an explanatory diagram schematically showing an example of a baby face synthesis system (1) according to the present invention. As shown in FIG. 1 , the baby face synthesis system 1 is connected through a network and is connected to a plurality of user terminals 10 possessed by users who desire baby face service, and to each of the user terminals through a wired or wireless network. a server 20 providing a baby face synthesis service; and a baby face database 30 including an image of a baby's face and corresponding feature points. The database 30 may be connected to the web server 20 through a network or may be included in the server 20 .

In addition, the baby faces stored in the database 30 may include the faces of real babies, but rather may be virtual baby faces generated using a Generative Adversarial Network (GAN), and each of the baby faces is a facial feature point. contains information.

In the case of the user terminal 10, various information processing and communication devices, including not only a smartphone or tablet PC, but also a PC or a notebook computer may be used. Such a user terminal 10 is not particularly limited as long as it can take an image and transmit it to the server 20 , or transmit an image stored in the user terminal 10 to the server 20 . An application providing an interface for requesting a service or receiving and confirming a service result requested from the server 20 may be pre-installed.

The server 20 is configured to receive a service request from the user terminal 10 and synthesize a baby face picture in response thereto. 2 is a diagram schematically illustrating an internal configuration of a server 20 that provides a baby face synthesis service.

As shown in FIG. 2 , the server 20 receives the image receiving unit 210 for receiving the parent face image and the ultrasound picture image of the fetus from the user terminal, and information for receiving the information selected by the parent about the baby face to be synthesized from the user terminal. The receiving unit 215, the face detection unit 220 for respectively detecting the faces of the parent and the fetus from the received image, the feature point detection unit 230 for detecting the characteristic points of the parents and the fetus from the detected faces of the parents and the fetus, respectively, detection A baby face search unit 240 that searches the database 30 for a baby face having the most similar feature points based on each feature point of the parent and fetus, and a parental resemblance ratio for calculating a parental resemblance ratio for a baby face to be synthesized calculation unit 250; a baby face synthesizing unit 260 for synthesizing the feature points of the parent's face based on the similarity ratio; and a synthesized image output unit 217 for outputting the synthesized face to the user's terminal 10 .

The image receiving unit 210 receives a photo image of a parent and an ultrasound photo image of a fetus from the user terminal 10 . The parent's photo image may be an individual photo image or a photo image in which the parents are all displayed in one image.

The information receiving unit 215 receives information selected by the parent for baby face synthesis from the user terminal 10 . For example, the user transmits the parent's picture and the fetal ultrasound picture to the image receiving unit 210 through the user terminal 10, and at the same time inputs the parental resemblance ratio of the baby to be synthesized, the baby's age, and the baby's gender. , the input information is received by the information receiving unit 215 as baby face synthesis information, and a baby face is synthesized based on the received information.

The face detection unit 220 includes a sub-face detection unit 230a for detecting a sub-image face from a photograph, a mother face detection unit 230b for detecting a mother image face from the photograph, and a fetal face from an ultrasound photograph of the fetus. is composed of a fetal face detection unit 203c for detecting

As shown in FIG. 2 , the minor face detection unit 220a detects a negative face from the photo image, determines it as an area of interest, and transmits it to the minor feature point detection unit 230a, and the mother face detection unit 220b receives the mock face from the photo image. is detected, determined as a region of interest, and transmitted to the mother feature detection unit 230b, and the fetal face detection unit 220c detects a fetal face from an ultrasound photograph image of the fetus, determines it as a region of interest, and determines the fetal feature point detector 230c send to

The feature detection unit 230 includes a sub-feature point detector 230a, a maternal feature point detector 230b, and a fetal feature point detector 230c as described above. The sub-feature point detector 230a analyzes the negative region of interest (face region) to detect the feature point of the sub-face, and the mother feature point detector 230b analyzes the simulated region of interest (face region) to detect the feature point of the mother face, The fetal feature point detection unit 230c analyzes the fetal interest region (face region) to detect the fetal feature point, and operates to digitize the shape/ratio.

Examples of the feature points in the embodiment of the present invention, as shown in FIG. 3 , may include 96 facial feature points that are mainly used as references in current face recognition, for example, face shape/ratio, eyes and eyes. The distance between them, the shape of the jaw line, the ratio of the eyes to the face, the length and ratio of the nose, the thickness of the lips, the width of the forehead, the length of the eyebrows, the shape of the eyebrows, and the size of the pupils may be included, but are not necessarily limited thereto.

The feature point detection of the face may divide a region of interest into a plurality of regions, and calculate a feature vector of the feature point by applying a feature point detection algorithm set to each of the divided regions. Here, the feature point detection algorithm includes a deep learning network, and specifically, the facial feature points are extracted through the ArcFace (Additive Angular Margin Loss) function of Jiankang Deng et al.'s "ArcFace: Additive Angular Margin Loss for Deep Face Recognition" paper ( https://arxiv.org/abs/1801.07698 ).

Negative facial feature points, simulated facial feature points, and fetal facial feature points detected and digitized through the feature point detection unit 230 are stored in a storage device such as a memory, and then the baby face search unit 240 stores the stored parent and mother and fetal features. Based on the facial feature points, a baby face having the most similar feature points is searched and found from the baby face database 30 .

The baby face stored in the baby face database 30 is generated through a generative adversarial network (GAN) algorithm, and the generated baby face is detected and digitized through the feature point detection algorithm for the face as described above. It is stored in the database 30 together with the feature points.

The "Generative Adversarial Network (GAN)" algorithm is a model that includes a generator and a discriminator. Based on the learning data, the generator generates data close to the learning data, and the discriminator It is determined whether the data is the same data as the training data or the data generated by the generator. Therefore, the GAN algorithm means an algorithm that creates an image close to the learning data by learning to deceive the discriminator by the generator in this process.

The generator learns the real data to generate similar data that can fool the discriminator that it is identical to the training data. Then, the discriminator determines whether the input data is the same as the training data (True) or whether the data is randomly generated by the generator (False) and distinguishes them. The generator repeats the learning process by receiving information that has not been deceived by the discriminator, and the discriminator receives the information deceived by the generator, so that the generator generates data that is close to the real thing.

4 is a flowchart for explaining generation of a baby picture using the GAN algorithm as described above. As shown in FIG. 4 , in step S10 , the server 20 or the server's processor collects baby pictures. The collected photos may be photos provided by the user through the user terminal 10 or photos collected through access to or search for the Internet or other databases.

After the baby photos are collected in the subsequent step S20, the server 20 classifies and stores the baby photo data by male/female, country, race, and age. Such a classification task can improve processing efficiency in the data learning process.

After the baby photos are classified in step S20, the server 20 processes the data by giving a predetermined weight to each of the classified baby face photos. The processing of baby photo data includes specific areas of baby photo, such as eyes, nose, mouth, skin, face shape/ratio, eye-to-eye distance, jawline shape, face-to-eye ratio, nose length, and It includes facial features including proportions, thickness of lips, width of forehead, length of eyebrows, shape of eyebrows, and size of pupils.

For example, the server 20 recognizes eye, nose, and mouth regions for baby photo data, estimates (x, y) coordinate values of the corresponding regions, and focuses on the (x, y) coordinate values of the region. and by estimating (x, y) coordinate values of the mouth region as boundary values, the feature point region may be set.

After the keypoint area to be used in the image to be generated is set in this way, the server 20 generates an image through the GAN algorithm based on the keypoint area. Through this process, only the baby's face region can be characteristically recognized and learned, so that feature detection and data mapping by the GAN algorithm can be performed smoothly.

As described above, the GAN algorithm generates a baby face image close to the real thing through competition between a generator and a discriminator based on the baby's feature points.

For the baby face picture obtained through the GAN algorithm, using the deep learning network, which is the feature point detection algorithm as described above in the subsequent step S50, for example, the shape/ratio of the face, the distance between the eyes, the shape of the jaw line, The feature points including the ratio of the eyes to the face, the length and ratio of the nose, the thickness of the lips, the width of the forehead, the length of the eyebrows, the shape of the eyebrows, and the size of the pupils are analyzed again. It is saved in the database together with the generated baby face picture.

In this way, the baby face photo data generated through the GAN algorithm is stored close to infinity in the database, and in the present invention, the face of a baby to be born is predicted based on the baby face photo generated by the GAN algorithm.

Referring back to FIG. 2 , the baby face search unit 240 compares the feature points detected by the parent and fetal feature point detection unit 230 with the baby face feature points stored in the baby face database 30 , A baby face with a feature point most similar to the feature point is found as a basic template for compositing.

After finding the baby face of the basic template for baby face synthesis, the server 20 calculates a similarity ratio with the parent face. The ratio of resemblance to the parent's face may be input from the user. For example, when the server 20 receives the parent image from the image receiving unit 210 , the ratio of resemblance to the parent's face to the baby's face to be synthesized (eg, 70% minor, 30% parent).

The ratio of resemblance to the parent's face represents the rate at which parental or parental feature points are synthesized on the baby's face to be synthesized. That is, when negative feature points are set to 70% and mock feature points to be 30%, the synthesized baby face has stronger negative facial feature points than the simulated facial feature points, and the baby's face is synthesized to resemble the negative face more than the mock face. Conversely, when the simulated feature points are 70% and the negative feature points are set to 30%, the simulated facial feature points appear stronger than the negative facial feature points in the synthesized baby face, and the baby's face appears more like the mock face than the negative face features.

For example, in the case of a baby's nose that is synthesized when the negative feature points are set to 70% and the mother feature points are set to 30%, 70% of the negative feature points are reflected and 30% of the mother feature points are reflected, so that the baby's nose is synthesized.

In addition, the basic baby face template to be synthesized is generated for each birth age, such as 30 days, 50 days, and 100 days after birth. ) will synthesize the baby's face based on the prepared age template.

5 is a diagram schematically illustrating a process of synthesizing a baby photo face using the system as described above. As shown in FIG. 5 , the image receiving unit 210 receives the parent's picture from the user terminal 10 , and the information receiving unit 215 receives the information selected by the parent for synthesis from the user terminal 10 .

The face detection unit 220 of the server 20 detects the negative face, the mother's face, and the fetal face from the received photos of the parent and the fetus, and the feature point detection unit 230 detects the detected father and mother and the fetal face. A feature point is detected in the region, and the detected facial feature points of the parent and fetus are stored.

Then, the server compares the feature points of the parent and fetus with the feature points included in the similar baby face data (generated by the GAN algorithm) stored in the database using the detected feature points of the parent and fetus, and selects the baby face with the most similar feature points. detect

The server generates a plurality of age templates for the baby face by adding age-specific feature points to the detected baby face, for example, 30-day, 50-day, and 100-day templates.

After that, the server reflects the baby face by reflecting the parental feature points to the baby face template using the baby face template corresponding to the age input by the user. At this time, the baby face is synthesized and synthesized in response to the similarity ratio input by the user A face is provided to the user.

On the other hand, the server is described as synthesizing baby face photos for the age input by the user, but the present invention is not limited thereto. It is also possible to be

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA). , a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The software may include a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or apparatus, to be interpreted by or to provide instructions or data to the processing device. may be embodied in The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims should be construed as falling within the scope of the following claims.

10: user terminal
20: Baby Face Synthesis Server
30: Baby Face Database
210: video receiver
215: information receiving unit
220: face detection unit
230: feature point detection unit
240: baby face search unit
250: parent similarity ratio calculator
260: baby face synthesizing unit

Claims (7)

A baby face prediction system for synthesizing a baby face using a parent's picture and an ultrasound picture of a fetus,
user-side terminal;
a server that provides baby face synthesis services; and
A database in which a baby face image including feature points is stored,
The server is
an image receiver configured to receive a parent image and an ultrasound picture of a fetus from a user-side terminal;
a face detection unit for detecting each facial region of a parent, a mother, and a fetus from the received parent image and an ultrasound picture of the fetus;
a feature point detection unit for detecting feature points of each of the faces of the parents and the fetus from the detected face regions of the parents and the fetus;
a baby face search unit that searches for a baby face image most similar to the feature points of parents, mothers, and fetuses detected from a database in which baby face images are stored;
and a baby face synthesizing unit for synthesizing parental and maternal feature points in the baby face image retrieved from the baby face search unit.
Baby face prediction system.
According to claim 1,
The server is
Characterized in that it further comprises an information receiving unit for receiving the similarity ratio of the baby's face to the parent's face to be synthesized from the user terminal, the gender of the baby, and the age of the baby
Baby face prediction system.
According to claim 1,
The baby face image stored in the baby face database is generated through a generative adversarial network (GAN) algorithm,
A feature point for the baby face image generated through the feature point detection algorithm is detected and stored in a database together with the generated baby face image
Baby face prediction system.
According to claim 1,
The characteristic points of the face of the mother and the fetus are the shape and proportion of the face, the distance between the eyes, the shape of the jaw line, the ratio of the face to the eyes, the length and ratio of the nose, the thickness of the lips, the width of the forehead, the length of the eyebrows, Characterized in that it includes a feature point related to the shape of the eyebrow and the size of the pupil.
Baby face prediction system.
5. The method of claim 4,
The feature points of the baby's face stored in the database, and the feature points of the mother's and fetus' faces are detected using a face detection algorithm including a deep learning network using an ArcFace function.
Baby face prediction system.
6. The method of claim 5,
The feature points of the baby's face stored in the database are,
collecting pictures of the baby's face over the network;
classifying the collected baby face photo data by gender, country, race, and age;
setting a feature point area including the feature points of the classified baby face photograph;
generating a baby face image through a GAN algorithm;
analyzing baby facial feature points from the generated baby face image; and
Characterized in that it is formed by the operation of storing the generated baby face image and baby facial feature points in a database
Baby face prediction system.
According to claim 1,
The baby face synthesizing unit is configured to synthesize the baby face by reflecting the similarity ratio with the parent's face received from the baby information receiving unit, the baby's gender, and the baby's age.
Baby face prediction system.

Images