KR20170141018A - System and Method for Detecting Face Landmark considering Occlusion Landmark - Google Patents

Abstract

According to the present invention, a system and a method for detecting a face landmark considering a blind landmark are disclosed. According to an aspect of the present invention, the face landmark detection system comprises: a landmark detector for detecting at least one landmark in a face detection area of an image; a first rotation degree calculator for calculating a rotation angle of a face by using two-dimensional coordinates of the at least one landmark; and a non-face determiner for determining a face state of the at least one landmark, and checking whether a blind situation occurs, that is the situation that any one side of left and right sides of the face is normally detected by using a determination result and the rotation angle of the face, and the other side of the face is not normally detected.

Description

Technical Field [0001] The present invention relates to a system and method for detecting a face landmark in consideration of a blind landmark,

The present invention relates to a face detection technology, and more particularly, to a face landmark detection system and method considering a blind landmark according to a face rotation.

In recent years, there has been a growing interest in various application systems for image processing of human faces by development of human machine interface technology.

 Application techniques for processing human face images include automatic sorting of portrait photographs, finding faces of people with similar directions, and recognizing head gestures.

Algorithms for detecting face landmarks include Active Contour Model (ACM), Active Shape Model (ASM), and Active Appearance Model (AAM).

Here, the landmark may be a feature point representing each feature of the object in the face detection area. For example, the landmark may be a point corresponding to the eyes, nose, mouth, eyebrows, etc., which are characteristics representative of a person's face. Also, at least one landmark may be set for each feature of the object, and a plurality of features may be set for one feature. As such, the number of features set for the object in the face detection area and the number of landmarks for each feature can be specified as needed.

However, when the rotation of the face is large, occlusion occurs in the landmark of the face, so that the noise value is inputted to the input value for tracking the landmark, which may interfere with the correct fitting.

Korean public patent 2011-0067480 (June 22, 2011)

SUMMARY OF THE INVENTION It is an object of the present invention to provide a system and method for detecting a face landmark in consideration of a blind landmark which can exclude a landmark that has been occluded by face rotation from face tracking.

The objects of the present invention are not limited to the above-mentioned objects, and other objects not mentioned can be clearly understood by those skilled in the art from the following description.

A face landmark detection system according to an embodiment of the present invention includes: a landmark detector that detects at least one landmark in a face detection area of an image; A first rotation degree calculator for calculating a rotation angle of the face using the two-dimensional coordinates of the at least one landmark; And determining whether or not the face of the at least one landmark is normal, using a result of the determination and the face rotation angle to determine whether the landmark of one of the left and right faces is normally detected, And a non-face determiner for confirming whether or not an occurrence situation is detected.

A method for detecting a face landmark by at least one processor according to another aspect of the present invention includes: detecting at least one landmark in a face detection area of an image; Calculating a face rotation angle using the two-dimensional coordinates of the at least one landmark; Determining whether the at least one landmark is a face; Determining whether a landmark on one of the left and right faces is normally detected and the landmark on the other face is not normally detected using the determination result of the determining step and the face rotation angle; And tracking only the normal landmark which is the landmark of the one face other than the blind landmark which is the landmark of the other face, if the occlusion occurs.

According to the present invention, the face landmark detection range can be extended.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a block diagram showing a face landmark detection system according to an embodiment of the present invention; FIG.
2 is a diagram illustrating a learning process of a face landmark detection system according to an embodiment of the present invention.
3 is a diagram for explaining a rotation angle calculation process according to an embodiment of the present invention.
4 is a diagram showing a model of a fisher linear discriminant according to an embodiment of the present invention.
5 is a view showing input / output of a non-face determiner according to an embodiment of the present invention.
6 is a view for explaining a rotation angle calculation process according to an embodiment of the present invention;
7 is a flowchart illustrating a method of detecting a face landmark according to an embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The above and other objects, advantages and features of the present invention and methods of achieving them will be apparent from the following detailed description of embodiments thereof taken in conjunction with the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Is provided to fully convey the scope of the invention to those skilled in the art, and the invention is only defined by the scope of the claims. It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. In the present specification, the singular form includes plural forms unless otherwise specified in the specification. As used herein, the terms " comprises, " and / or "comprising" refer to the presence or absence of one or more other components, steps, operations, and / Or additions.

Hereinafter, a face landmark detection system according to an embodiment of the present invention will be described with reference to FIGS. 1 to 6. FIG.

FIG. 1 is a block diagram showing a face landmark detection system according to an embodiment of the present invention, and FIG. 2 is a diagram illustrating a learning process of a face landmark detection system according to an embodiment of the present invention. FIG. 3 is a view for explaining a rotation angle calculation process according to an embodiment of the present invention, and FIG. 4 is a diagram illustrating a model of a fisher linear discriminant function according to an embodiment of the present invention. FIG. 5 is a view illustrating input / output of a non-face determiner according to an embodiment of the present invention, and FIG. 6 is a view for explaining a rotation angle calculation process according to an embodiment of the present invention.

1, the face landmark detection system 10 according to the embodiment of the present invention includes a face detector 110, a landmark detector 120, a first rotation degree calculator 130, A periodic table 140 and a second rotation degree calculator 150.

The face detector 110 extracts a face detection area, which is an area of interest used for landmark detection, from the input image.

The landmark detector 120 extracts the feature value? _J from the face detection area, and detects the face landmark from the extracted feature value? J. At this time, the landmark detector 120 can detect a landmark using an ACM (Active Contour Model), an ASM (Active Shape Model), an AAM (Active Appearance Model), or the like.

Further, the landmark detector 120 uses the previously learned conversion function R _n to fit the detected landmarks into the two-dimensional frontal face shape, respectively. To this end, the landmark detector 120 is a using a landmark (Land mark) the combination of facial images facial features (x ₀₎ as shown in Fig. 2 the front position; transform function (R changing to (x _GT Ground Truth) _n ).

As a result, the landmark detector 120 can output the landmark values (index, characteristic value, coordinates) of each detected landmark. At this time, the landmark detector 120 judges that the landmark is successfully detected only when the landmark of a predetermined number or more is successfully fitted, and can output the landmark value only in that case.

Further, the landmark detector 120 may store the index of the landmark to be tracked among the detected landmarks, and may continuously track the landmark of the index. Specifically, when the next frame of the input image is input and the face detection area is updated, the landmark detector 120 can update the current landmark position in the front face form according to the current pixel value of the updated face detection area have.

The landmark detector 120 may not update the tracking-landmark position for the area determined as non-face by the non-face determiner 140. [

The first rotation degree calculator 130 detects the rotation angle of the face by estimating the three-dimensional position based on the two-dimensional coordinates of the landmark according to each landmark value.

4, the first rotation degree calculator 130 calculates a first rotation angle Pitch based on the X axis by performing the PosIt algorithm, a second angle Yaw based on the Y axis, The inclination of the third angle (Roll) rotated about the Z axis can calculate the rotation parameter. In addition, the first rotation degree calculator 130 performs the PosIt algorithm to calculate an X distance parallel to the X axis, a Y distance parallel to the Y axis, and a Z distance including a Z distance parallel to the Z axis Scale (e.g., width and height, etc.) parameters. Then, the first rotation degree calculator 130 can estimate the three-dimensional position of each landmark by fitting each landmark to the three-dimensional face shape using six three-dimensional parameters. At this time, the three-dimensional position estimation process using the PosIt algorithm can be derived from the prior art by an ordinary technician, so a detailed description thereof will be omitted.

The non-face determiner 140 determines whether each landmark is a face or a non-face by at least one predetermined Fisher Linear Discriminant (FLD).

As shown in FIG. 4, at least one fisher linear discriminant includes a frontal model 410 capable of distinguishing landmarks of the entire face, a left-turn model 420 capable of distinguishing landmarks of the left face with respect to the face center point, A right-handed rotation model 430 can be distinguished by a fisher-linear discriminant. In FIG. 4, the fisher linear discriminant for determining the face area and the non-face area using five landmarks has been described as an example, but the present invention is not limited thereto.

The non-face determiner 140 confirms whether or not the occurrence of a blind occurred in the landmark hidden by the face rotation using the determination result for each landmark and the face direction angle Yaw. Referring to FIG. 5, the non-face determiner 140 determines that the facial Yaw angle is -40 while the occluded state of the left face, as a result of the determination using the Fischer's linear discriminant of the front face model 410. Here, the occlusion situation of the left face differs from that of the general fitting (tracking) failure in that the landmark of the right face is a fitting success (red mark of FIG. 5) or a landmark of the left face (see blue mark of FIG. 5) It may be the case that only the fitting fails. At this time, the non-face determiner 140 can select a landmark located on the left face as a blind landmark.

In the case of the occlusion occurrence, the non-face determiner 140 determines the face and the non-face region using the fisher linear discriminant of the left-handed rotation model 420 at the time of left rotation, Face and non-face regions can be determined using a linear discriminant.

Also, the non-face determiner 140 returns information about the blind landmark to at least one of the face detector 110 and the landmark detector 120. In detail, the non-face determiner 140 can transmit normal uncovered landmark information to the landmark detector 120 and transmit the blind landmark information to the face detector 110. [

Then, the landmark detector 120 can set the critical number, which is a criterion for determining success of the landmark detection, to be lower by a preset value in correspondence with the blind landmark information. For example, when the total number of landmarks to be actually detected in the face detection area is 30, the landmark detector 120 sets the threshold number to 25 when the face is not rotating, and sets the threshold number to the blind land It can be set to be lower than 25 corresponding to the number of marks.

At this time, the landmark detector 120 may not transmit the landmark value of the blind landmark to at least one of the first rotation degree calculator 130 and the non-face determiner 140.

In other words, the landmark detector 120 may no longer track the blurry landmark until it is rotated forward as the index of the blurry landmark is excluded from the index of the landmark to track.

On the other hand, after confirming the rotation of the face in one direction, it is necessary to check whether or not the rotated face is returned to the front position. This can be performed by one of a method using a normal landmark and a method using a blind landmark. Hereinafter, each method will be described separately.

▷ How to use normal landmarks

The second rotation degree calculator 150 may store the face rotation angle at the time when the face rotation is determined by the non-face determiner 140 and track the subsequent face rotation angle.

)를 산출하고, 법선 벡터의 각도를 얼굴 회전각도로 확인할 수 있다. 또한, 제2 회전도 산출기(150)는 법선 벡터의 각도가 얼굴이 정면 위치의 각도에 대응하면, 얼굴이 정면 위치로 회전됨을 확인하고, 비얼굴 판정기(140)에 알릴 수 있다.At this time, the second rotation degree calculator 150 calculates the normal vector of the rotated face using the normal landmark (three points) not covered as shown in FIG. 6

), And the angle of the normal vector can be confirmed by the face rotation angle. In addition, the second rotation degree calculator 150 can confirm that the face is rotated to the front position and notify the non-face determiner 140 if the angle of the normal vector corresponds to the angle of the front face position.

Alternatively, the second rotation degree calculator 150 may perform the posit algorithm by itself at four or more matching points to check the rotation angle of the face at the time when the rotation of the face is determined. At this time, the second rotation degree calculator 150 may receive the face rotation angle (Yaw angle) from the first rotation degree calculator 130.

▷ How to use Garim Landmark

When the first rotation degree calculator 130 confirms that the face is rotated, it detects the three-dimensional coordinates of the normal landmark, calculates the virtual position of the blind landmark so as to correspond to the normal landmark, (140). For example, the virtual position may be a symmetric value of the normal landmark relative to the face center.

The non-face determiner 140 determines whether the feature point of the virtual position is a face or a non-face, and when it is confirmed that the face is a face, it can be determined that the face has returned to the front position. When the face returns to the front position, the non-face determiner 140 may change the phisher linear discriminant to the face model 410 and notify it to at least one of the landmark detector 120 and the face detector 110 .

On the other hand, when the non-face determiner 140 determines that the area of the landmark's failure of fitting is uneven as in the occlusion occurrence situation, the non-face determiner 140 can determine that the face tracking has failed. Then, the non-face determiner 140 informs the landmark detector 120 of the face tracking failure, the landmark detector 120 initializes the previously stored tracking landmark index, and then, using the received face detection area, Mark tracking can be restarted.

As described above, according to the embodiment of the present invention, since the landmark to be tracked is covered with a lot of faces rotated, the normal (visible) landmark is not fitted with the hidden (invisible) landmark even though the fitting is good It is possible to improve the problem that the model error value is accumulated and the face tracking fails.

Further, the embodiment of the present invention can extend the range of the landmark detection, thereby improving the application range and reliability of the face tracking system.

Hereinafter, a face landmark detection method according to an embodiment of the present invention will be described with reference to FIG. 7 is a flowchart illustrating a method of detecting a face landmark according to an embodiment of the present invention.

Referring to FIG. 7, the face detector 110 extracts a face detection area from an input image (S700).

The landmark detector 120 extracts feature points from the face detection area, selects the landmarks among them, and fits the two-dimensional face of the front position (S710).

The landmark detector 120 determines a landmark value (index, feature value, and coordinates) of the landmark to be tracked using the fitting result, and stores an index of the landmark (S720). Thereafter, the landmark detector 120 may track the landmark of the index if traceable.

The first rotation degree calculator 130 converts the two-dimensional coordinates of the landmark into a three-dimensional position using the PosIt algorithm, and calculates yaw, roll, pitch angle and face scale (S730).

The non-face determiner 140 determines whether or not the detected landmark is a face by applying a fisher linear discriminant of the front model to the detected landmark value (S740). At this time, the non-face determiner 140 can further calculate the fitting degree of each landmark.

The non-face determiner 140 checks whether the landmark in the left direction or the right direction of the detected landmark is a non-face detected state, that is, an occluded state in which the Yaw angle is rotated in one direction ( S750). At this time, the non-face determiner 140 can further confirm the degree of fit of each landmark. For example, the non-face determiner 140 may determine whether to use an intermediate landmark according to the degree of fit of the landmark located in the middle, or the non-face determiner 140 may determine whether to use the landmark It is possible to judge whether or not the fitting is successful by using the fitting degree value at the time of occurrence of the occlusion occurrence status for the first time.

As a result, if the occlusion occurs, the non-face determiner 140 removes the index of the occluded landmark from the tracked landmark index (S760).

Thereafter, the non-face determiner 140 can determine the non-face landmark using the left-turn model or the right-turn model until the face returns to the front.

At this time, the second rotation degree calculator 150 stores the first face angle at which the occlusion starts using the normal vector of the normal landmark or the detected Yaw angle (S770).

The second rotation degree calculator 150 compares the face angle calculated by the normal vector of the normal landmark detected in the frame with the first face angle to check whether the face returns to the front (S780).

When the face returns to the front, the non-face determiner 140 adds the index of the blind landmark to the landmark index to be tracked (S790).

On the other hand, if the area of the landmark's fitting failure is uneven, unlike the occlusion occurrence situation, the non-face determiner 140 determines that the face tracking has failed (S800). At this time, the non-face determiner 140 informs the landmark detector 120 of the tracking failure.

Then, the landmark detector 120 initializes the tracking landmark index currently being tracked (S810). Then, the landmark detector 120 may detect the landmark again in the next frame of the input image and perform tracking or the like.

In the embodiment described above, the case where the second rotation degree calculator 150 confirms whether the face rotated in one direction returns to the front face has been described as an example. Alternatively, the first rotation angle calculator 130 may calculate the virtual position of the blind landmark using the normal landmark, and if the non-face determiner 140 determines that the landmark at the virtual position is the face area It is possible to confirm whether or not the face is returned to the front position.

As described above, according to the embodiment of the present invention, since the landmark to be tracked is covered with a lot of faces rotated, the normal (visible) landmark is not fitted with the hidden (invisible) landmark even though the fitting is good It is possible to improve the problem that the model error value is accumulated and the face tracking fails.

Further, the embodiment of the present invention can extend the range of the landmark detection, thereby improving the application range and reliability of the face tracking system.

While the present invention has been described in detail with reference to the accompanying drawings, it is to be understood that the invention is not limited to the above-described embodiments. Those skilled in the art will appreciate that various modifications, Of course, this is possible. Accordingly, the scope of protection of the present invention should not be limited to the above-described embodiments, but should be determined by the description of the following claims.

10: face landmark detection system 110: face detector
120: Landmark detector 130: First rotation degree calculator
140: Non-face plate periodizer 150: Second rotation degree calculator

Claims (15)

A landmark detector for detecting at least one landmark in a face detection area of an image;
A first rotation degree calculator for calculating a rotation angle of the face using the two-dimensional coordinates of the at least one landmark; And
A face detection unit for detecting whether or not the face of the at least one landmark is normal, detecting a landmark on one of the left and right faces of the face by using the determination result and the face rotation angle, Non-face-to-face routine to check for situational awareness
And the face landmark detection system. 2. The landmark detector of claim 1,
Wherein the face landmark detection system tracks only the normal landmark, which is the landmark of the one face except for the blind landmark which is the landmark of the other face. 3. The landmark detector of claim 2,
If the number of the at least one landmarks is equal to or greater than the threshold number, it is determined that landmark detection is successful,
And after the occurrence of the occlusion, the number of thresholds of the at least one landmark is reduced corresponding to the number of the occluded landmarks, thereby determining whether or not the detection of the landmark is successful. 3. The method of claim 2,
Calculates and stores a first rotation angle based on a normal vector of the normal landmark at a point of time when the occlusion occurrence status is detected,
A second rotation degree calculator for checking whether the face returns to the front position using the change in the first rotation angle confirmed from the normal landmark after the occurrence of the blindness,
Further comprising: a face landmark detection system for detecting a face landmark; The non-face judging device according to claim 1,
Determining whether or not the face of the at least one landmark is a face of the face model in a situation where the face is in the front position,
Determining whether or not the face of the at least one landmark is faced with a fisher linear discriminant of a left-turn model in a situation where the face is rotated to the left,
Wherein the face landmark detection system determines whether or not the face of the at least one landmark is faced with a fisher linear discriminant of a right-turn model in a situation where the face is rotated to the right. 3. The method of claim 2,
The first rotation degree calculator calculates the virtual position of the blind landmark by the normal landmark if the blind occurrence occurs,
Wherein the non-face determiner determines whether the face has returned to the front position by determining whether the feature point existing at the virtual position is a face. 7. The method of claim 6,
Dimensional position of the normal landmark is set to a symmetric value of the three-dimensional position of the normal landmark. The method of claim 1,
The non-face judging unit judges that the face tracking has failed if at least a part of the landmarks of the left and right faces are not normally detected as a result of the predetermined discrimination formula,
Wherein the landmark detector stops tracing a landmark being tracked and detects a new landmark when confirming that the face tracking fails. A method for detecting a face landmark by at least one processor,
Detecting at least one landmark in a face detection area of an image;
Calculating a face rotation angle using the two-dimensional coordinates of the at least one landmark;
Determining whether the at least one landmark is a face;
Determining whether a landmark on one of the left and right faces is normally detected and the landmark on the other face is not normally detected using the determination result of the determining step and the face rotation angle; And
Tracking only the normal landmark which is the landmark of the one face other than the blind landmark which is the landmark of the other face
And detecting the face landmark. 10. The method of claim 9, wherein detecting the landmark comprises:
Determining that landmark detection is successful if the number of the at least one landmarks is equal to or greater than a threshold number; And
Determining whether the landmark detection is successful by decreasing the critical number of the at least one landmark in correspondence with the number of the blind landmarks after the occurrence of the blind occurrence
Wherein the face landmark detection method comprises the steps of: The method of claim 9,
Calculating and storing a first rotation angle by a normal vector of the normal landmark at a point of time when the occlusion occurrence situation is detected; And
Confirming whether or not the face returns to the front position using the change in the first rotation angle confirmed from the normal landmark after the occlusion occurrence situation
Further comprising the steps of: 10. The method of claim 9,
A fisher linear discriminant of a frontal model for determining whether or not the face of the at least one landmark is in a face-to-face position;
A fisher linear discriminant of a left-turn model for determining whether or not the face of the at least one landmark faces when the face is rotated to the left; And
Wherein one of the fisher linear discriminant expressions of the right-turn model for determining whether or not the face of the at least one landmark faces is used in a situation where the face is rotated to the right. The method of claim 9,
Calculating a virtual position of the blind landmark by the normal landmark if the blind occurrence occurs; And
Determining whether the face is returned to the front position by determining whether the feature point existing at the virtual position is a face or not
Further comprising the steps of: 14. The method of claim 13,
Dimensional position of the normal landmark is set to a symmetric value of the three-dimensional position of the normal landmark. The method of claim 9,
Determining that the face tracking has failed if at least a part of the landmarks of the left and right faces are not normally detected as a result of the determination of the predetermined discrimination formula;
If it is confirmed that the face tracking fails, the step of stopping the tracking of the landmark being tracked and detecting the new landmark
Further comprising the steps of:

Images