KR20240003944A - Method and apparatus for facial symmetry analysis through registration of 3d facial landmark - Google Patents

Abstract

The present invention relates to a facial symmetry analysis method performed in a facial symmetry analysis device, comprising: acquiring first and second facial image data taken at different viewpoints or with different expressions; Detecting a first landmark and detecting a second landmark for the second facial image data, scale matching, global registration, and point for the first and second landmarks Applying a point-to-plane ICP algorithm to match the first and second landmarks, and analyzing facial symmetry based on the registered first and second landmarks. .

Description

Facial symmetry analysis method and device through registration of 3D facial landmarks {METHOD AND APPARATUS FOR FACIAL SYMMETRY ANALYSIS THROUGH REGISTRATION OF 3D FACIAL LANDMARK}

The present invention relates to facial symmetry analysis technology, and more specifically, to a facial symmetry analysis method and device for analyzing facial symmetry by matching 3D facial landmarks detected from different facial image data.

Facial paralysis is a disease that causes paralysis of the face due to problems with the function of the facial nerve, which moves facial muscles. It is a disease that affects the muscles on one side of the mouth and eyes, causing the face to become distorted. If the initial treatment for facial paralysis is not done properly, it can have serious aftereffects and cause external discomfort, psychological anxiety or depression, so an accurate diagnosis of facial paralysis is necessary and it is important to accurately know the progress of facial paralysis. .

One of the quantitative evaluation methods used to accurately diagnose facial paralysis is a method using an optical marker or sensor. The degree of facial paralysis can be measured by attaching a marker to the face or using a laser. This method can accurately identify facial features, but has the disadvantage of requiring additional equipment, being applicable only in limited environments, and patients feeling uncomfortable when using it. Another method is to use image information and depth information to photograph a face using a depth camera and extract landmarks on the face. This method allows for relatively accurate evaluation, but it has the disadvantage of having limitations on equipment and environment because it must be equipped with a depth camera capable of acquiring depth information and the distance between the camera and the user must be a certain distance or more.

Korean Patent Publication No. 10-2021-0156796

The purpose of the present invention is to provide a facial symmetry analysis method and device for detecting three-dimensional facial landmarks from two-dimensional RGB facial image data and matching different facial image data using the three-dimensional facial landmarks.

The object of the present invention is to provide a facial symmetry analysis method and device for analyzing changes in facial symmetry according to facial expressions or changes in facial symmetry over time by calculating the moving distance, distance symmetry, or angular symmetry of landmarks in registered facial image data. is to provide.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned can be clearly understood by those skilled in the art from the description below.

The first aspect of the present invention for achieving the above object is a facial symmetry analysis method performed in a facial symmetry analysis device, comprising: (a) first and second facial image data taken at different times or with different expressions; Obtaining, (b) detecting a first landmark for the first face image data and detecting a second landmark for the second face image data, (c) the first and second lands registering the first and second landmarks by applying scale matching, global registration, and point-to-plane ICP algorithms to the marks, and (d) and analyzing facial symmetry based on the matched first and second landmarks.

Preferably, step (b) applies a deep learning algorithm corresponding to HR-net, CNN-CRF, or Mediapipe to the first and second face image data to determine the first land for the first face image data. Detecting a mark and a second landmark for the second facial image data, wherein the first and second landmarks each include a plurality of facial landmarks and eye landmarks, and between the landmarks A series of numbers are assigned and managed, and landmarks that correspond to each other in the first and second landmarks may be assigned the same number.

Preferably, the step (c) includes (c-1) calculating a scale factor based on the coordinates of a pair of landmarks and the origin coordinates corresponding to each other in the first and second landmarks. , and (c-2) may include the step of converting the scale of the second landmark using the scale factor.

Preferably, step (c) includes: (c-3) globally registering the first landmark and the scale-transformed second landmark; (c-4) a first-order transformation matrix obtained through the global registration; performing a point-to-plane ICP algorithm on the first landmark and a scale-converted second landmark using , and (c-5) obtaining a secondary transformation matrix through performing the point-to-plane ICP algorithm Additional steps may be included.

Preferably, step (c) is performed by performing steps (c-1) to (c-5) for each pair of landmarks corresponding to each other in the first and second landmarks, so that all pairs of lands It may include obtaining a secondary transformation matrix for each mark.

Preferably, the step (c) includes determining, as the final transformation matrix, a secondary transformation matrix with the smallest root mean square deviation among the secondary transformation matrices obtained for each of all pairs of landmarks, and the final transformation The method may further include matching the first and second landmarks using a matrix.

Preferably, the step (d) includes setting the midplane of the first and second facial image data based on a landmark representing the iris among the registered first and second landmarks, the first and second facial image data 2 A step of inverting one landmark to the other side based on the midplane for each landmark, and for each of the first and second landmarks based on the distance between the inverted landmark and the landmark on the other side. It may include calculating distance symmetry.

Preferably, step (d) uses vectors representing corresponding landmarks on both sides based on the midplane and cosine similarity for the normal vector of the midplane to connect the first and second lands. A step of calculating angular symmetry for each mark may be further included.

Preferably, the step (d) is performed based on the distance symmetry or angle symmetry calculated for each of the first and second landmarks. The time point at which the first face image data is acquired and the second face image data are acquired A step of analyzing the degree of change in facial symmetry during the time point may be further included.

Preferably, the step (d) includes calculating a right facial landmark movement amount and a left facial landmark movement amount corresponding to the right facial landmark with respect to the first and second landmarks, and Calculating left and right facial symmetry according to expression changes in the first face image data and the second face image data, based on the landmark movement amount calculated for each pair of landmarks on the right face and left face. Including, the movement amount of the landmark may correspond to the movement amount between the landmark of the first face image data and the landmark of the second face image data corresponding to the landmark of the first face image data.

A second aspect of the present invention for achieving the above object is a facial symmetry analysis device, comprising: an image data acquisition unit for acquiring first and second facial image data taken at different viewpoints or with different expressions; A landmark detection unit that detects a first landmark for face image data and a second landmark for the second face image data, scale matching, and global registration for the first and second landmarks (global registration), and a landmark registration unit that applies the point-to-plane ICP algorithm to register the first and second landmarks, and based on the registered first and second landmarks It includes a facial symmetry analysis unit that analyzes facial symmetry.

A third aspect of the present invention for achieving the above object is characterized in that, in a computer program stored in a computer-readable medium, when an instruction of the computer program is executed, a data matching method is performed.

As described above, according to the present invention, facial symmetry can be analyzed using only facial image data without a separate device such as a sensor or depth camera or environmental constraints, and the analysis results can be obtained numerically.

In addition, the symmetry of the left and right sides can be obtained numerically according to the amount of landmark movement in different facial expressions, and the degree of change in facial symmetry over time can be obtained numerically according to the distance symmetry or angle symmetry at different viewpoints. There is a possible effect.

In addition, this numerical analysis of facial symmetry can be helpful in the evaluation of facial paralysis or aging diagnosis.

1 is a block diagram showing a facial symmetry analysis device according to a preferred embodiment of the present invention.
Figure 2 is a flowchart showing a facial symmetry analysis method according to one embodiment.
Figure 3 is an example diagram for explaining a landmark according to an embodiment.
Figure 4 is an example diagram for explaining a landmark matching process according to an embodiment.
Figure 5 is an example diagram for explaining grouping of landmarks according to an embodiment.
Figure 6 is an exemplary diagram for explaining the median plane according to one embodiment.
Figures 7 to 9 are exemplary diagrams for explaining numerical analysis of facial symmetry according to one embodiment.

Hereinafter, the advantages and features of the present invention and methods for achieving them will become clear with reference to the embodiments described in detail below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and will be implemented in various different forms. The present embodiments only serve to ensure that the disclosure of the present invention is complete and that common knowledge in the technical field to which the present invention pertains is not limited. It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout the specification. “And/or” includes each and every combination of one or more of the mentioned items.

Although first, second, etc. are used to describe various elements, elements and/or sections, it is understood that these elements, elements and/or sections are not limited by these terms. These terms are merely used to distinguish one element, element, or section from other elements, elements, or sections. Therefore, it goes without saying that the first element, first element, or first section mentioned below may also be a second element, second element, or second section within the technical spirit of the present invention.

In addition, identification codes (e.g., a, b, c, etc.) for each step are used for convenience of explanation. The identification codes do not explain the order of each step, and each step is clearly specified in the context. Unless the order is specified, it may occur differently from the specified order. That is, each step may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the opposite order.

The terminology used herein is for describing embodiments and is not intended to limit the invention. As used herein, singular forms also include plural forms, unless specifically stated otherwise in the context. As used in the specification, “comprises” and/or “comprising” refers to the presence of one or more other components, steps, operations and/or elements. or does not rule out addition.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings that can be commonly understood by those skilled in the art to which the present invention pertains. Additionally, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless clearly specifically defined.

Additionally, in describing the embodiments of the present invention, if it is determined that a detailed description of a known function or configuration may unnecessarily obscure the gist of the present invention, the detailed description will be omitted. The terms described below are terms defined in consideration of functions in the embodiments of the present invention, and may vary depending on the intention or custom of the user or operator. Therefore, the definition should be made based on the contents throughout this specification.

1 is a block diagram showing a facial symmetry analysis device according to a preferred embodiment of the present invention.

Referring to FIG. 1, the facial symmetry analysis device 100 includes an image data acquisition unit 110, a landmark detection unit 120, a landmark matching unit 130, a facial symmetry analysis unit 140, and a control unit 150. Includes. Here, the control unit 150 controls the operations and data flow of the image data acquisition unit 110, the landmark detection unit 120, the landmark matching unit 130, and the facial symmetry analysis unit 140.

The facial symmetry analysis device 100 is a device that acquires a plurality of facial image data and analyzes facial symmetry. Preferably, the facial symmetry analysis device 100 is a computer that can install and execute an application or program for performing a facial symmetry analysis method, and is provided with a user interface to control input and output of data. Here, a computer refers to all types of hardware devices including at least one processor, and depending on the embodiment, it may be understood as encompassing software configurations that operate on the hardware device. For example, a computer can be understood to include, but is not limited to, a smartphone, tablet PC, desktop, laptop, and user clients and applications running on each device.

The image data acquisition unit 110 acquires a plurality of facial image data taken at different viewpoints or with different facial expressions. Here, the face image data is an RGB image and can be acquired through a general photographing device. Preferably, the acquired facial image data can be managed separately according to the subject, shooting time, or expression, and when facial symmetry analysis is performed according to the elapse of the shooting time, the previously stored facial image data and the newly captured face Image data may be used.

The landmark detection unit 120 detects landmarks for each face image data through a deep learning algorithm that detects keypoints, that is, landmarks, that can be features of the face in the face image data. Here, the landmark may correspond to a 3D facial landmark.

The landmark registration unit 130 performs registration by aligning a plurality of face image data obtained from different coordinate systems into one coordinate system. The plurality of facial image data acquired through the image data acquisition unit 110 are captured at different viewpoints, and the landmarks detected from the facial image data from different viewpoints have different scales, movements, or rotations, so facial symmetry is achieved through comparison. Since analysis is not possible, the landmark registration unit 130 aligns the coordinate system through landmark registration of a plurality of face image data.

The facial symmetry analysis unit 140 compares the landmarks of the plurality of facial image data registered through the landmark matching unit 130 to analyze the symmetry of the left and right sides of the face when the facial expression changes or to analyze the symmetry of the left and right sides of the face over time. Analyze the degree of change.

Operations performed through each component of the facial symmetry analysis device 100 shown in FIG. 1 will be described in detail below with reference to FIG. 2. Each step to be described with reference to FIG. 2 is described as being performed by a different configuration, but is not limited thereto, and depending on the embodiment, at least some of the steps may be performed by the same or different configuration.

Figure 2 is a flowchart showing a facial symmetry analysis method according to one embodiment.

Referring to FIG. 2, the image data acquisition unit 110 acquires first and second facial image data (step S210). Preferably, the image data acquisition unit 110 may acquire first and second facial image data taken at different viewpoints or with different facial expressions. The image data acquisition unit 110 can acquire facial image data pre-stored in the facial symmetry analysis device 100 or pre-stored and managed in a separate storage device depending on the purpose of analysis, and facial images captured in real time. Data can also be obtained. For example, in order to analyze the degree of change in facial symmetry over time, the image data acquisition unit 110 may acquire pre-stored first facial image data from a previous point in time and second facial image data captured in real time. In order to analyze facial symmetry according to changes in facial expression, the image data acquisition unit 110 acquires first and second facial image data taken with different facial expressions at a specific time and stored in advance, or photographed with different facial expressions in real time. First and second facial image data may be acquired.

The landmark detection unit 120 detects first and second landmarks for the first and second face image data (step S220). Preferably, the landmark detection unit 120 applies a deep learning algorithm corresponding to HR-net, CNN-CRF, or Mediapipe to the first and second face image data to detect the first landmark and the first face image data. A second landmark for the second face image data may be detected. Both the first and second landmarks may correspond to 3D facial landmarks, and any network capable of real-time operation may be applied to the algorithm for detecting the landmark.

Preferably, the first and second landmarks each include a plurality of facial landmarks and eye landmarks, the landmarks are managed by assigning a series of numbers, and the first and second landmarks correspond to each other. Landmarks may be assigned the same number. For example, the landmark detection unit 120 may extract 478 landmarks for each of the first and second face image data and may be composed of 468 face landmarks and 10 eye landmarks, and the face image data A landmark for can be detected as a light green dot shown in FIG. 3.

In one embodiment, the first landmark (A) detected for the first face image data and the second landmark (B) detected for the second face image data may be configured as in [Equation 1] below: there is. Here, n is the number of landmarks.

[Equation 1]

The landmark matching unit 130 matches the first and second landmarks (step S230). Preferably, the landmark matching unit 130 applies scale matching, global registration, and point-to-plane ICP algorithms to the first and second landmarks. The first and second landmarks can be registered.

Referring to FIG. 4, (a) shows the first and second landmarks obtained through step S220, where light green corresponds to the first landmark and red corresponds to the second landmark. Referring to (a), it can be seen that the scales of the first and second landmarks obtained from the first and second face image data captured at different viewpoints are not aligned. In the global matching described below, scale matching is not performed, so scale matching must be performed first.

In order to perform scale matching, the landmark matching unit 130 calculates a scale factor based on the coordinates and origin coordinates of a pair of landmarks corresponding to each other in the first and second landmarks, and calculates the scale factor The scale of the second landmark is converted using . Referring to (b) of FIG. 4, it can be seen that the scales of the first and second landmarks are matched by scale conversion from (a). In one embodiment, the i-th scale factor for scale conversion is calculated as [Equation 2]. Here, O is the origin coordinate, A _i is the i-th landmark of the first landmark, and B _i is the i-th landmark of the second landmark.

[Equation 2]

Next, the landmark registration unit 130 globally registers the first landmark and the scale-converted second landmark. Since global registration does not require an initial transformation matrix, an approximate transformation matrix is found through global registration. The landmark registration unit 130 performs a point-to-plane ICP algorithm on the first landmark and the scale-transformed second landmark using the first-order transformation matrix obtained through global registration, and performs the point-to-plane ICP algorithm. Obtain the second-order transformation matrix through execution. The point-to-plane ICP algorithm is a method of finding a transformation matrix that minimizes the distance from the plane of the normal vectors of the first landmark and the second landmark. Here, global registration and point-to-plane ICP algorithms can be easily understood and applied by those skilled in the art to which the present invention pertains, so they will not be described in detail.

Preferably, the landmark registration unit 130 calculates a scale factor for each pair of landmarks that correspond to each other in the first and second landmarks, converts the scale of the second landmark, and performs global registration and point-to-plane ICP. By repeating the process of performing the algorithm, a secondary transformation matrix for each pair of landmarks can be obtained. As shown in Figure 4, the scale transformation, global registration, and point-to-plane ICP algorithm are repeatedly performed for all pairs of the first and second landmarks to obtain a secondary transformation matrix ( T) can be obtained. Here, T _i is a transformation matrix obtained by converting the scale of the second landmark with the scale factor obtained as the ith landmark and then performing global registration and point-to-plane ICP algorithms.

[Equation 3]

That is, the first scale factor is calculated for the first landmark of the first and second landmarks using [Equation 2], scale conversion is performed on the second landmark using the first scale factor, and then the global By performing registration and point-to-plane ICP algorithm, the second transformation matrix corresponding to T ₁ is obtained, and the second scale factor is calculated using [Equation 2] for the second landmark of the first and second landmarks. After performing scale transformation on the second landmark using the second scale factor, global registration and point-to-plane ICP algorithm are performed to obtain a secondary transformation matrix corresponding to T2, and in the same manner, the first and Scale factor calculation, scale transformation, global registration, and point-to-plane ICP algorithms are performed for each of the n landmarks corresponding to the second landmark, and a secondary transformation matrix as shown in [Equation 3] is obtained.

When secondary transformation matrices for each pair of first and second landmarks are obtained, the landmark registration unit 130 determines the smallest root mean square error (RMSE) among all acquired secondary transformation matrices. A secondary transformation matrix having is determined as the final transformation matrix, and the first and second landmarks are matched using the final transformation matrix. That is, the first and second landmarks are unified into one coordinate system by moving and rotating the first and second landmarks using the final transformation matrix. For example, the final transformation matrix can be applied to the second landmark to align it with the coordinate system of the first landmark. Preferably, the equation for calculating RMSE may be composed of [Equation 4] and [Equation 5].

[Equation 4]

[Equation 5]

Here, n is the number of landmarks, i means the i-th landmark, and e _i is the i-th land of the first and second landmarks after conversion to the second-order transformation matrix (T). It is defined as the Euclidean distance of the mark. For example, after transforming the first and second landmarks into T ₁ , each corresponding landmark (A ₁ and B ₁ , A ₂ and B ₂ , … A _n and B _n ) From [Equation 5], e ₁ , e ₂ , … The value of e _n can be obtained, and based on this, the RMSE for T ₁ can be calculated through [Equation 4]. In this way, RMSE is calculated for each of T ₂ to T _n , and the landmark matching unit 130 determines the secondary transformation matrix with the smallest RMSE as the final transformation matrix.

As a result, when the final transformation matrix is determined as above and the first and second landmarks are matched, the coordinate system can be aligned to have the smallest error. Referring to FIG. 4, the scale for each landmark from (a) It can be seen that the first and second landmarks are registered as shown in (c) through the final transformation matrix determined by repeatedly performing transformation, global registration, and point-to-plane ICP. After the first and second landmarks are aligned, it becomes possible to quantitatively measure the degree of rehabilitation of facial paralysis, degree of aging, etc. through facial symmetry analysis.

The facial symmetry analysis unit 140 analyzes facial symmetry based on the first and second landmarks matched in step S230 (step S240).

Preferably, when the first and second landmarks are matched, the facial symmetry analysis unit 140 may group the landmarks according to facial muscles. For example, referring to Figure 5, the face can be divided into 17 regions and the number of landmarks included in each region can be classified as shown in the table. The method of grouping landmarks can be modified in various ways. Distance symmetry, angle symmetry, and landmark movement amount, which will be explained below, can be calculated as the average value of the values for landmarks grouped in each region, and through this, facial symmetry can be comprehensively analyzed.

In one embodiment, the facial symmetry analysis unit 140 may calculate distance symmetry and angle symmetry to analyze left-right symmetry in the same facial expression.

First, in order to calculate distance symmetry and angle symmetry, a regular plane must be set. Preferably, the facial symmetry analysis unit 140 may set the midplane of the first and second facial image data based on a landmark representing the iris among the first and second landmarks. Referring to Figure 6, the mid-sagittal plane can be defined as the perpendicular bisector of the iris and the line connecting the iris, and when expanded in three dimensions, the vector connecting the landmark representing the iris is the normal vector. The plane passing through the midpoint of the landmark representing the iris may be defined as the median plane.

Preferably, the facial symmetry analysis unit 140 inverts the landmark on one side to the other side based on the midplane for each of the first and second landmarks, and based on the distance between the inverted landmark and the landmark on the other side. Distance symmetry for each of the first and second face image data can be calculated. For example, referring to Figure 7, the landmark on the left is reversed to the right based on the midplane, and the distance symmetry between the landmark reversed to the right (reversed keypoint) and the landmark on the right (original keypoint) is as follows. It can be calculated using [Equation 6], and the smaller the value, the more complete symmetry it means.

[Equation 6]

Here, p _i ^R is the ith landmark of the right face, and p _i ^rL is the ith landmark of the inverted left face in the midplane, for all corresponding landmarks of the right face and the inverted left face, respectively. Facial symmetry can be analyzed based on the calculated distance symmetry.

Preferably, the facial symmetry analysis unit 140 uses vectors representing corresponding landmarks on both sides based on the midplane and cosine similarity for the normal vector of the midplane to analyze the first and second facial image data. The angle symmetry for each can be calculated. For example, referring to Figure 8, angular symmetry is calculated using [Equation 7] below, and the more the vectors of a pair of corresponding landmarks on both sides are the same, it means complete symmetry, and when the value of angular symmetry is 0, means complete asymmetry, and 1 means complete symmetry.

[Equation 7]

Here, A is the normal vector of the midplane, B is the vector of the i-th landmark pair, and the vector of the i-th landmark pair is calculated as [Equation 8] below.

[Equation 8]

Here, p _i ^R _,x is the x coordinate of the ith landmark of the right face, and p _i ^L _,x is the x coordinate of the ith landmark of the left face.

Preferably, the facial symmetry analysis unit 140 is configured to obtain the first facial image data and the second facial image data based on the distance symmetry or angle symmetry calculated for each of the first and second facial image data. The degree of change in facial symmetry over a period of time can be analyzed. Here, it is assumed that the facial expressions in the first and second facial image data are the same.

In another embodiment, the amount of movement of the landmark may be calculated to analyze left-right symmetry in different facial expressions at the same time, for example, a blank expression and a smiling expression, and the landmarks of the left and right faces at the same location may be calculated for changes in expression. The left-right symmetry of the face can be analyzed by checking how uniformly it can move.

Preferably, the facial symmetry analysis unit 140 calculates the movement amount of the landmark of the right face and the movement amount of the landmark of the left face corresponding to the landmark of the right face with respect to the landmarks of the first and second facial image data. You can. The movement amount of the landmark corresponds to the movement amount between the landmark of the first face image data and the landmark of the second face image data corresponding to the landmark of the first face image data. For example, referring to FIG. 9, when the face image with a neutral expression is the first face image data and the face image with a smiling expression is the second face image, the first and second lands The left and right landmark movement amounts corresponding to the ith landmark in the mark can be calculated using [Equation 9] below.

[Equation 9]

Here, s _i ^L is the ith landmark position of the left face in the second landmark with a smiling expression (indicated as the smile i ^th L keypoint in FIG. 9), and n _i ^L is the left face in the first landmark with an expressionless expression. is the location of the ith landmark (indicated as neutral i ^th L keypoint in Figure 9).

Based on the left and right landmark movement amounts calculated in [Equation 9], the movement amount of the ith landmark can be calculated using [Equation 10] below, and the landmark movement amount means complete symmetry at 0.

[Equation 10]

Based on the landmark movement amount calculated for each pair of landmarks on the right face and left face, left and right facial symmetry according to expression changes in the first and second facial image data can be calculated.

In one embodiment, the distance symmetry, angular symmetry, and landmark movement amount calculated through the facial symmetry analysis unit 140 may be calculated each time facial symmetry is analyzed, or the distance symmetry, angular symmetry, and landmark movement amount may be calculated each time facial symmetry is analyzed. Depending on the object or purpose to be analyzed, facial symmetry may be analyzed by selectively calculating a portion.

In the above description, the explanation was made based on the analysis of symmetry of the face, but the facial symmetry analysis method according to the present invention can also be applied to other objects whose left and right sides are symmetrical with respect to a specific face, and this is in accordance with the general technical field of the present invention. It can be easily understood by technicians.

Meanwhile, the steps of the method or algorithm described in relation to the embodiment of the present invention may be implemented directly in hardware, implemented as a software module executed by hardware, or a combination thereof. The software module may be RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), Flash Memory, hard disk, removable disk, CD-ROM, or It may reside on any type of computer-readable recording medium well known in the art to which the present invention pertains.

The components of the present invention may be implemented as a program (or application) and stored in a medium in order to be executed in conjunction with a hardware computer. Components of the invention may be implemented as software programming or software elements, and similarly, embodiments may include various algorithms implemented as combinations of data structures, processes, routines or other programming constructs, such as C, C++, , may be implemented in a programming or scripting language such as Java, assembler, etc. Functional aspects may be implemented as algorithms running on one or more processors.

Although preferred embodiments of the facial symmetry analysis device and method according to the present invention have been described above, the present invention is not limited thereto and may be implemented with various modifications within the scope of the claims, the detailed description of the invention, and the accompanying drawings. It is possible and this also belongs to the present invention.

100: Facial symmetry analysis device
110: Image data acquisition unit
120: Landmark detection unit
130: Landmark matching unit
140: Facial symmetry analysis unit
150: control unit

Claims (12)

In the facial symmetry analysis method performed in a facial symmetry analysis device,
(a) acquiring first and second facial image data taken at different times or with different expressions;
(b) detecting a first landmark for the first facial image data and detecting a second landmark for the second facial image data;
(c) Applying scale matching, global registration, and point-to-plane ICP algorithms to the first and second landmarks to determine the first and second lands. registering marks; and
(d) Facial symmetry analysis method comprising the step of analyzing facial symmetry based on the matched first and second landmarks.
The method of claim 1, wherein step (b) is:
Apply a deep learning algorithm corresponding to HR-net, CNN-CRF, or Mediapipe to the first and second face image data to determine the first landmark for the first face image data and the second face image data. Including detecting a second landmark,
The first and second landmarks each include a plurality of face landmarks and eye landmarks, the landmarks are managed by assigning a series of numbers, and the first and second landmarks correspond to each other. A facial symmetry analysis method characterized in that marks are given the same number.
The method of claim 1, wherein step (c) is:
(c-1) calculating a scale factor based on the coordinates of a pair of landmarks and the origin coordinates corresponding to each other in the first and second landmarks; and
(c-2) A facial symmetry analysis method comprising the step of converting the scale of the second landmark using the scale factor.
The method of claim 3, wherein step (c) is,
(c-3) globally matching the first landmark and the scale-converted second landmark;
(c-4) performing a point-to-plane ICP algorithm on the first landmark and the scale-transformed second landmark using the first-order transformation matrix obtained through the global registration; and
(c-5) A facial symmetry analysis method further comprising the step of obtaining a second-order transformation matrix by performing the point-to-plane ICP algorithm.
The method of claim 4, wherein step (c) is,
Steps (c-1) to (c-5) are repeatedly performed for each pair of landmarks corresponding to each other in the first and second landmarks to obtain a secondary transformation matrix for each pair of landmarks. A facial symmetry analysis method comprising the step of obtaining.
The method of claim 5, wherein step (c) is,
determining a secondary transformation matrix with the smallest root mean square deviation among secondary transformation matrices obtained for each of the pairs of landmarks as a final transformation matrix; and
Facial symmetry analysis method further comprising the step of matching the first and second landmarks using the final transformation matrix.
The method of claim 1, wherein step (d) is,
setting the median plane of the first and second facial image data based on a landmark representing an iris among the matched first and second landmarks;
For each of the first and second landmarks, inverting one landmark to the other with respect to the midplane; and
A facial symmetry analysis method comprising calculating distance symmetry for each of the first and second landmarks based on the distance between the inverted landmark and the landmark on the other side.
The method of claim 7, wherein step (d) is,
Calculating angular symmetry for each of the first and second landmarks using vectors representing corresponding landmarks on both sides of the median plane and cosine similarity for the normal vector of the median plane. A facial symmetry analysis method further comprising:
The method of claim 8, wherein step (d) is,
Analyzing the degree of change in facial symmetry between the time of acquiring the first facial image data and the time of acquiring the second facial image data based on the distance symmetry or angular symmetry calculated for each of the first and second landmarks. A facial symmetry analysis method characterized by further comprising steps.
The method of claim 1, wherein step (d) is,
Calculating a right facial landmark movement amount and a left facial landmark movement amount corresponding to the right facial landmark with respect to the first and second landmarks; and
Calculating left and right facial symmetry according to expression changes in the first face image data and the second face image data, based on the landmark movement amount calculated for each pair of landmarks of the right face and left face. Including,
Facial symmetry analysis method, characterized in that the movement amount of the landmark corresponds to the movement amount between the landmark of the first face image data and the landmark of the second face image data corresponding to the landmark of the first face image data.
an image data acquisition unit that acquires first and second facial image data taken at different times or with different expressions;
a landmark detection unit that detects a first landmark for the first face image data and a second landmark for the second face image data;
Registering the first and second landmarks by applying scale matching, global registration, and point-to-plane ICP algorithms to the first and second landmarks. a landmark matching unit; and
A facial symmetry analysis device comprising a facial symmetry analysis unit that analyzes facial symmetry based on the matched first and second landmarks.
In a computer program stored on a computer-readable medium,
A computer program stored in a computer-readable medium, wherein when the command of the computer program is executed, the method according to any one of claims 1 to 10 is performed.