KR102480960B1 - Device and method for transforming human face into animal face - Google Patents

Abstract

The present invention relates to a face conversion technology, and more particularly, to a face conversion apparatus and method for converting a human face into an animal face using Delaunay triangulation and Voronoi diagram. According to an embodiment of the present invention, it is possible to assist in automating various transformation tasks between various human and animal faces by providing a transformation vector for transformation from a human face to an animal face to be transformed.

Description

Face conversion device and method {DEVICE AND METHOD FOR TRANSFORMING HUMAN FACE INTO ANIMAL FACE}

The present invention relates to a face conversion technology, and more particularly, to a face conversion apparatus and method for converting a human face into an animal face using Delaunay triangulation and Voronoi diagram.

The development and generalization of various technologies, including AI, is rapidly increasing the production of media contents. Against this backdrop, various media contents are being developed. In particular, in many media contents, unique characters are developed and used to differentiate the contents, and the importance of the characters used is gradually increasing. In addition, unlike existing media, the form of smart media that reacts according to the situation is spreading, and for this reason, system support including automation is becoming important in the development of media contents.

The character industry is an industry with many potentials, and various analysis studies are being conducted on the attributes of characters. In particular, characters developed by personifying animals, such as Ryan from Kakao Friends and Pengsoo from EBS Giant Peng TV, show the high profitability and potential of the character industry. According to the standard Korean dictionary, a character is defined as a character appearing in a novel or a play, or a character's personality and image revealed in the contents of a work, or a unique character or animal appearance in a novel, cartoon, or play. It emphasizes the unique personality and characteristics of each individual. In order to express such diverse individual characteristics and personalities, the development of characters often involves anthropomorphizing various entities centering on animals. Personification of a character is to analyze the personality and characteristics of a person to be expressed and express the image like a person based on the generalized characteristics of animals or non-human objects. Therefore, if regularity can be found in selecting the type of animal to be applied to the development of a character for a specific person, more efficient work will be possible by automating many parts of character development or receiving help from the system.

The background art of the present invention is published in Korean Registered Patent No. 2131109.

The present invention automatically calculates and provides a conversion vector capable of converting the coordinates of each feature point of a human face according to the characteristics of each animal species based on various reference information about the face of an animal and a human face. An apparatus and method are provided.

The object of the present invention is not limited to those mentioned above, and other objects having extension and versatility are determined to have sufficient utilization potential in the technical field to which the present invention belongs from the following description.

According to one aspect of the present invention, a face conversion device is provided.

A face conversion device according to an embodiment of the present invention includes an animal data management unit that receives animal face data, calculates and stores animal data for face conversion, receives human face data, and calculates human data for face conversion. A human face is converted into an animal face by using a conversion vector for extracting matched animal face data by comparing the human data processing unit and the human data and stored animal data, and converting the input human face data into the matched animal face data. It may include a human-animal conversion unit that converts.

According to another aspect of the present invention, a face conversion method and a computer readable recording medium on which a computer program executing the same is recorded are provided.

According to an embodiment of the present invention, a face conversion method and a recording medium storing a computer program for executing the same include the steps of receiving animal face data and extracting animal face feature points, generating an animal face pattern using Delaunay triangulation, and The method may include calculating an area of each animal face region using a Voronoi diagram, calculating a distance between animal face components based on a face center point, and storing the calculated animal face data.

According to an embodiment of the present invention, it is possible to assist in automating various transformation tasks between various human and animal faces by providing a transformation vector for transformation from a human face to an animal face to be transformed.

In addition, according to an embodiment of the present invention, since the provision of a conversion vector does not require much data processing, it can be derived within a short time, and can contribute to the automation of related industries or systems.

1 to 4 are views for explaining a face conversion device according to an embodiment of the present invention.
5 to 11 are diagrams illustrating a face conversion method according to an embodiment of the present invention.

Since the present invention can make various changes and have various embodiments, specific embodiments are illustrated in the drawings and will be described in detail through detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. In describing the present invention, if it is determined that a detailed description of related known technologies may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted. Also, as used in this specification and claims, the terms "a" and "an" are generally to be construed to mean "one or more" unless stated otherwise.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. do it with

1 to 4 are diagrams for explaining a face conversion device according to an embodiment of the present invention.

Referring to FIG. 1 , the face conversion device includes an animal data management unit 100, a human data input unit 200, a human data processing unit 300, and a human-animal conversion unit 400.

The animal data management unit 100 receives animal face data, calculates area information for each face region for face conversion and distance information between components, and stores them. The animal data management unit 100 will be described in more detail with reference to FIG. 2 below.

Referring to FIG. 2 , the animal data management unit 100 includes an animal data input unit 110, an animal feature point extraction unit 120, an animal pattern generator 130, an area calculation unit 140 for each animal region, and animal components. A distance calculator 150 and an animal data storage unit 160 are included.

The animal data input unit 110 inputs animal face data. Here, the animal face data may be, for example, an animal face image. The animal data input unit 110 can use, for example, 60 types of animal face images from Pixabay.com, which have no copyright problem, as face images of animals necessary for experiments, and 60 types of animal faces used in comparative experiments. The animal selection criteria in can be based on Order, Family, and Species according to Linnaeus' biological classification system in 1753. When the animal data input unit 110 classifies through deep learning using real photos rather than characters, species that are not clearly distinguished are integrated into a higher category, and 20 kinds of animals are selected again. Actual face photos of 20 kinds of animals can be obtained and used in experiments.

The animal feature point extraction unit 120 extracts feature points from input animal face data. The animal feature point extraction unit 120 may extract coordinates of feature points common to all animal images, for example. Here, as common feature points, seven coordinates of the center point of the face, the center point of the left eye, the center point of the right eye, the center point of the nose, the center point of the mouth, the center point of the left ear, and the center point of the right ear can be extracted.

The animal pattern generator 130 generates an animal face pattern using Delaunay triangulation. The animal pattern generation unit 130 may generate an animal face pattern by, for example, applying Delaunay triangulation to the coordinates of other feature points other than the coordinates of the center point of the face as the center point. Here, the reason for excluding the coordinates of the center point of the face is that it is a reference point for calculating the distance to each coordinate and at the same time may cause pattern deformation by causing interference with coordinates located near the center of the face, such as the nose and mouth.

The area calculation unit 140 for each animal region calculates the area for each animal face region using the Voronoi diagram. The area calculation unit 140 for each animal area may generate a Voronoi diagram using, for example, the 1.5.4 version of the Scipy library of Python, and calculate the area of each Voronoi polygon.

The distance calculating unit 150 between the animal components calculates the distance between the animal face components based on the center point of the face. The distance calculating unit 150 between animal components may calculate a distance between at least one of the coordinates of the left eye, right eye, nose, mouth, left ear, and right ear based on the coordinates of the center point of the face.

The animal data storage unit 160 may store at least one of calculated area information of each Voronoi polygon region of the animal face and distance information between animal face components.

Referring back to FIG. 1 , the human data processing unit 300 receives human face data and calculates area information for each face region and distance information between components for face conversion. The human data processing unit 300 will be described in more detail with reference to FIG. 3 below.

Referring to FIG. 3 , the human data processor 300 includes a human feature point extractor 320, a human pattern generator 330, an area calculator 340 for each human region, and a distance calculator 350 between human elements. include

The human feature point extraction unit 320 receives human face data and extracts human face feature points. Here, the human facial feature points may include, for example, at least one of a face, a left eye, a right eye, a nose, a mouth, a left ear, and a right ear, like the aforementioned animal facial feature points.

The human pattern generation unit 330 generates a human face pattern using Delaunay triangulation. The face conversion apparatus 100 may generate a human face pattern by applying Delaunay triangulation to the human face with all coordinates except for the coordinates of the center point of the face as the center point, as in the method of generating the animal face pattern described above.

The area calculation unit 340 for each human region calculates an area for each human face region using a Voronoi diagram. As applied to the face of the animal described above, the face conversion device 100 may generate a Voronoi diagram using, for example, the 1.5.4 version of the Python Scipy library, and calculate the area of each Voronoi polygon region. .

The distance calculation unit 350 between human face elements calculates the distance between human face elements based on the center point of the face. The face conversion device 100 may calculate a distance between at least one of the left eye, right eye, nose, mouth, left ear, and right ear coordinates based on the face center point coordinates.

Referring back to FIG. 1 , the human-animal conversion unit 400 compares the calculated human face data with the stored animal face data, extracts matched animal face data, and converts the calculated human face into a matched animal face. A human face is converted into an animal face using a conversion vector for

Referring to FIG. 4 , the human-animal conversion unit 400 includes a human-animal comparison unit 410 and a human-animal transformation vector calculation unit 420 .

The human-animal comparator 410 compares human face data with stored animal face data and matches the most similar animal face data. The human-animal comparator 410 compares the areas of the regions of the Voronoi polygons calculated at each of the 34 center points of the human face and the animal face, and compares the areas of each region of the human face and each region of the animal face. The difference in area can be calculated. The human animal comparator 410 selects an animal with the smallest error when calculating the total by calculating the area difference for each region as a reference animal for generating a conversion vector. Here, it can be seen that the animal having the closest area for each region has the closest shape to the input person's face.

The human-animal conversion vector calculation unit 420 calculates a conversion vector for converting human face data into matched animal face data, and converts the human face into an animal face using the calculated conversion vector. The human-animal conversion vector calculation unit 420 calculates the intersection point of the extension line for each point determined based on the shape of each region of the face and the size of the original face of the animal, and uses the ratio of the distance from the corresponding position to the original point. Thus, the size of the transform vector for each region can be calculated.

5 to 11 are diagrams for explaining a face conversion method according to an embodiment of the present invention. The process performed by each functional unit constituting the face conversion device 100 described below or the subject of each step will be collectively referred to as the face conversion device 100 for a concise and clear description of the present invention.

Referring to FIG. 5 , in step S505, the face conversion device 100 receives animal face data and extracts animal face feature points. Here, as the animal face feature points, various feature points may be extracted. The animal facial features may include, for example, at least one of a face, a left eye, a right eye, a nose, a mouth, a left ear, and a right ear.

The table of FIG. 6 shows the types of coordinates to be extracted in deep learning-based animal facial feature point extraction. Any deep learning model for extracting coordinates may be used. The extraction of the coordinates of the center point of the face, FACE1, is related to the face, and can be derived by calculating the center point after extracting FACE2, the coordinates of the left end of the face to FACE5, the coordinates of the right end of the face. In addition, LEYE1, the coordinates of the center point of the left eye, is extracted by extracting LEYE2, the coordinates of the left end of the left eye and LEYE5, the coordinates of the right end of the left eye, and then calculating the center point. The extraction of REYE1, the coordinates of the center point of the right eye, is derived by calculating the center point after extracting REYE2, the coordinates of the left end of the right eye to REYE5, the coordinates of the right end of the right eye. NOSE1, the coordinates of the center point of the nose, is derived by calculating the center point after extracting NOSE2, the coordinates of the left end of the nose to NOSE5, the coordinates of the right end of the nose. The extraction of MOUT1, the coordinates of the center point of the mouth, is derived by calculating the center point after extracting MOUT2, the coordinates of the left end of the mouth and MOUT5, the coordinates of the right end of the mouth. Extraction of LEAR1, the coordinates of the center point of the left ear, is derived by calculating the center point after extracting LEAR2, the coordinates of the left end of the left ear and LEAR5, the coordinates of the right side of the left ear. The extraction of REAR1, the coordinates of the center point of the right ear, is derived by calculating the center point after extracting REAR2, the coordinates of the left end of the right ear and REAR5, the coordinates of the right end of the right ear.

In step S510, the face conversion apparatus 100 generates an animal face pattern using Delaunay triangulation.

Referring to FIG. 7 , the face conversion apparatus 100 may generate an animal face pattern by applying Delaunay triangulation to all coordinates except FACE1, which is a coordinate of the center point of the face, as a center point. Here, the reason for excluding FACE1, which is the center point coordinate of the face, is that it is a reference point for calculating the distance to each coordinate, and at the same time, it can cause pattern deformation by causing interference with coordinates located near the center of the face, such as the nose and mouth. . The face conversion device 100 may use, for example, 34 feature points excluding the coordinates of the center point of the face among 35 feature points composed of 5 coordinates for each of 7 components. The reason why coordinates for the face itself are used in addition to the components of the face, such as the eyes, nose, ears, and mouth, is because the length and width of the face are different for each animal, so the location, arrangement ratio, and area occupied between each component are different. .

In step S515, the face conversion device 100 calculates the area of each animal face region using the Voronoi diagram.

Referring to FIG. 8 , the face conversion apparatus 100 shows a Voronoi diagram of an animal's face created by using each coordinate obtained in FIG. 6 as a central point. By comparing the area ratio of each area polygon of the Voronoi diagram with each other, using the extension line of the center points of each face, the coordinates of the center point of the face for 2D (a point that is the same as FACE1 or with a slight error is calculated) and the face for 3D The coordinates of the center point (different from FACE1) can be calculated. Here, the animal's face image has a different face for each animal, and the angle and size of the face are also different. Also, the number of images used in the experiment is different for each animal. Therefore, in the present invention, an average value can be calculated and used for each component for all images of the same species of animal. The face conversion device 100 may generate a Voronoi diagram using version 1.5.4 of the Python Scipy library.

은 다각형의 정점을 나타낸다. After generating the Voronoi diagram, the face conversion apparatus 100 calculates the area of each Voronoi polygon. The face conversion device 100 may apply, for example, a shoestring formula to calculate the area of a polygon. Here, the shoelace formula is an algorithm for determining the area of a polygon composed of a plurality of points existing on a plane to which the Cartesian coordinate system is applied, that is, the area. The face conversion device 100 calculates the area using a matrix by making each triangle based on a straight line connecting each vertex of the Voronoi polygon whose area is to be obtained and the center point inside the Voronoi polygon, and calculates the area using a matrix. Calculation can be performed by intersecting in both directions, and the area of the desired polygon can be obtained by subtracting the intersecting area from the value in both directions and dividing by 2. The equation below of the shoelace formula can be expressed as (1). Here, in Equation (1), A represents the area of the polygon and n represents the number of sides of the polygon.

represents the vertex of the polygon.

수학식 (1)

Equation (1)

FIG. 9 shows an integrated pattern showing the correlation between the animal face pattern figure obtained in FIG. 7 and the Voronoi diagram obtained in FIG. Using the integrated pattern, it is possible to check the relationship between the area of each corresponding region and the position of the pattern.

In step S520, the face conversion device 100 calculates the distance between animal face components based on the center point of the face. The face conversion device 100 may calculate a distance between at least one of the left eye, right eye, nose, mouth, left ear, and right ear coordinates based on the face center point coordinates.

In step S525, the face conversion device 100 stores the calculated animal face data in the animal data storage unit 160. The face conversion apparatus 100 may store in the animal data storage unit 160 at least one of area information of each Voronoi polygon region of the animal face and distance information between animal face components.

In step S530, the face conversion device 100 receives human face data and extracts human face feature points. Here, the human facial feature points may include, for example, at least one of a face, a left eye, a right eye, a nose, a mouth, a left ear, and a right ear, like the aforementioned animal facial feature points.

In step S535, the face conversion apparatus 100 generates a human face pattern using Delaunay triangulation. The face conversion apparatus 100 may generate a human face pattern by applying Delaunay triangulation to the human face with all coordinates except for the coordinates of the center point of the face as the center point, as in the method of generating the animal face pattern described above.

In step S540, the face conversion device 100 calculates the area of each human face region using the Voronoi diagram. As applied to the face of the animal described above, the face conversion device 100 may generate a Voronoi diagram using, for example, the 1.5.4 version of the Python Scipy library, and calculate the area of each Voronoi polygon region. .

In step S545, the face conversion device 100 calculates the distance between human face components based on the center point of the face. The face conversion device 100 may calculate a distance between at least one of the left eye, right eye, nose, mouth, left ear, and right ear coordinates based on the face center point coordinates.

In step S550, the face conversion device 100 compares the human face data with the animal face data stored in the animal data storage unit 160 and matches the most similar animal face data. The face conversion device 100 includes area information for each human face region and distance information between components included in the calculated human face data, and area information for each animal face region and distance information between components stored in the animal data storage unit 160. Compare The face conversion apparatus 100 compares the areas of the Voronoi polygon regions calculated at each of the 34 center points of the human face and the animal face, and compares the areas of each region of the human face and each region of the animal face. can calculate the difference of The face conversion device 100 selects an animal with the smallest error as a reference animal for generating a conversion vector when the sum is calculated by calculating the difference in area for each region. Here, it can be seen that the animal having the closest area for each region has the closest shape to the input person's face. Since the Delaunay triangulation lies in a dual relationship with the Voronoi polygon, the area difference can be obtained in the same way. The face conversion apparatus 100 may improve accuracy by comparing the ratio between the areas of the Voronoi polygon and the area ratio of the Delaunay triangulation.

In step S555, the face conversion device 100 calculates a conversion vector for converting the human face data into matched animal face data, and converts the human face into an animal face using the calculated conversion vector.

The face conversion device 100 calculates the intersection of the extension line for each point determined based on the shape of each region of the face and the size of the original face of the animal, and uses the ratio of the distance from the corresponding position to the original point for each region. Calculate the size of the transformation vector for Since the distance between the angles is not constant, the face conversion device 100 may derive a distance ratio for a difference in height of each point between the animal face and the human face. In almost all cases of different polygonal shapes for each region, the extension line connecting each vertex does not intersect at a single point. The face conversion device 100 calculates the intersection of the extension line connecting each vertex in the polygon for each region of the human face and animal face, and then uses a point located in the center based on the coordinates of each intersection as a reference point for distance ratio calculation. Specify and perform calculations.

The face conversion device 100 converts a human face into a matching animal face using the derived conversion vector.

10 shows the concept of finding a reference point using the intersection of extension lines connecting each vertex. The size of the transformation vector for each region can be calculated using the ratio of the distance from the calculated reference point to the original vertex. Since the distance from the reference point to the vertex is not constant for each area, the distance ratio for the difference in height between the animal face and the human face for each area can be derived. When the face conversion device 100 increases or decreases each vertex by a given ratio based on the ratio value derived in this way, the area for each region is converted into a form close to the face of an animal similar to the input human face, and the calculated value It can be stored as a conversion vector.

11 shows the results of generating pattern figures and Voronoi diagrams by Delaunay triangulation for 20 types of animals frequently used in character design according to an embodiment of the present invention. Depending on the classification of each animal, the difference in the shape and arrangement of the Delaunay triangulation pattern model and the Voronoi diagram polygon can be confirmed.

The above-described face conversion method may be implemented as computer readable code on a computer readable medium. The computer-readable recording medium may be, for example, a removable recording medium (CD, DVD, Blu-ray disc, USB storage device, removable hard disk) or a fixed recording medium (ROM, RAM, computer-equipped hard disk). can The computer program recorded on the computer-readable recording medium may be transmitted to another computing device through a network such as the Internet, installed in the other computing device, and thus used in the other computing device.

In the above, even though all the components constituting the embodiment of the present invention have been described as being combined or operated as one, the present invention is not necessarily limited to these embodiments. That is, within the scope of the object of the present invention, all of the components may be selectively combined with one or more to operate.

Although actions are shown in a particular order in the drawings, it should not be understood that the actions must be performed in the specific order shown or in a sequential order, or that all shown actions must be performed to obtain a desired result. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of the various components in the embodiments described above should not be understood as requiring such separation, and the described program components and systems may generally be integrated together into a single software product or packaged into multiple software products. It should be understood that there is

So far, the present invention has been looked at mainly by its embodiments. Those skilled in the art to which the present invention pertains will be able to understand that the present invention can be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from a descriptive point of view rather than a limiting point of view. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the equivalent scope will be construed as being included in the present invention.

100: animal data management department
200: human data input unit
300: human data processing unit
400: human animal conversion unit

Claims (10)

In the face conversion device,
an animal data management unit that receives animal face data, calculates and stores animal data for face conversion;
a human data processing unit that receives human face data and calculates human data for face transformation;
Human or animal that compares the human data with stored animal data to extract matching animal face data, and converts the human face into an animal face using a conversion vector for converting the input human face data into matched animal face data. conversion unit;
The human animal conversion unit
Comparing the areas of Voronoi polygons calculated from each of the 34 center points of the human and animal faces, calculating the area difference between each area on the human face and each area on the animal's face, and calculating the sum of the area differences a human-animal comparison unit that selects an animal with the smallest error as a reference animal for generating a conversion vector and matches human face data with animal face data most similar to human face data; and
Calculate the intersection of the extension line for each point based on the shape of each area of the face and the size of the original face of the animal, and calculate the conversion vector for each area using the ratio of the distance from the location to the original point A face conversion device including a human-animal conversion vector calculation unit that converts a human face into an animal face using the calculated conversion vector.


According to claim 1,
The Animal Data Management Department
an animal data input unit inputting animal face data;
an animal feature point extraction unit extracting feature points from input animal face data;
an animal pattern generation unit generating an animal face pattern using Delaunay triangulation;
an area calculation unit for each animal face region using a Voronoi diagram;
a distance calculation unit for calculating a distance between animal face components based on a face center point; and
A face conversion device comprising an animal data storage unit configured to store at least one of calculated area information of each Voronoi polygon area of the animal face and distance information between animal face components.
According to claim 1,
The human data processor
a human feature point extraction unit extracting feature points from input human face data;
a human pattern generating unit generating a human face pattern using Delaunay triangulation;
an area calculation unit for each human face region using a Voronoi diagram; and
A face conversion device comprising a distance calculation unit for calculating a distance between human face components based on a face center point.
delete According to claim 2 or 3,
The feature point is
A face conversion device comprising at least one of a face, a left eye, a right eye, a nose, a mouth, a left ear, and a right ear.
In the face conversion method,
receiving animal face data, calculating and storing animal data for face transformation;
receiving human face data and calculating human data for face transformation;
Human or animal that compares the human data with stored animal data to extract matching animal face data, and converts the human face into an animal face using a conversion vector for converting the input human face data into matched animal face data. conversion step;
The human-animal transformation step
Comparing the areas of Voronoi polygons calculated from each of the 34 center points of the human and animal faces, calculating the area difference between each area on the human face and each area on the animal's face, and calculating the sum of the area differences a human-animal comparison step of matching human face data with animal face data most similar to human face data by selecting an animal with the smallest error as a reference animal for generating a conversion vector when the result is the same; and
Calculate the intersection of the extension line for each point based on the shape of each area of the face and the size of the original face of the animal, and calculate the conversion vector for each area using the ratio of the distance from the location to the original point , converting a human face into an animal face using the calculated conversion vector.


According to claim 6,
receiving animal face data and extracting animal face feature points;
generating an animal face pattern using Delaunay triangulation; and
calculating an area for each animal face region using a Voronoi diagram;
calculating a distance between animal face components based on a face center point; and
A face conversion method comprising the step of storing calculated animal face data.


According to claim 6,
receiving human face data and extracting human facial feature points;
generating a human face pattern using Delaunay triangulation; and
calculating an area for each human face region using a Voronoi diagram;
and calculating a distance between human face components based on a face center point.
According to claim 7 or 8,
The feature point is
A face conversion method comprising at least one of a face, a left eye, a right eye, a nose, a mouth, a left ear, and a right ear.
A computer program recorded on a computer-readable recording medium that executes the method of claim 6.

Images