CN114495221A - Method for positioning key points of face with mask - Google Patents
Method for positioning key points of face with mask Download PDFInfo
- Publication number
- CN114495221A CN114495221A CN202210061922.8A CN202210061922A CN114495221A CN 114495221 A CN114495221 A CN 114495221A CN 202210061922 A CN202210061922 A CN 202210061922A CN 114495221 A CN114495221 A CN 114495221A
- Authority
- CN
- China
- Prior art keywords
- point
- mask
- face
- points
- key
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Image Analysis (AREA)
Abstract
The invention relates to the technical field of face recognition, in particular to a method for positioning key points of a face of a mask. The method comprises the following steps: prepositioning mask wearing face key points; constructing a face data set of the mask; mask face key point is worn in accurate location. The invention provides a novel scheme of a mask wearing face key point 'prepositioning model-data set labeling-accurate positioning model'. According to the invention, the prepositioning of 68-point face key points of the mask wearing face is established through the characteristic of the fit between the face and the mask, so that the mask wearing face key points of a public data set are standardized, a corresponding face data set containing key point rough positioning is established, and then the accurate positioning of the mask wearing face is realized by adopting a regression tree integration algorithm. Compared with other methods, the method provided by the invention has lower key point positioning error under the condition of wearing the mask, and can provide technical support for rapid identification of the face identity wearing the mask.
Description
Technical Field
The invention relates to the technical field of face recognition, in particular to a method for positioning key points of a face of a mask.
Background
In order to prevent haze and prevent the spread of respiratory diseases, the mask wearing trip has become a normal state. Meanwhile, the large-area human face shielding caused by wearing the mask brings new challenges to the face recognition technology. The traditional face recognition application scenes such as airport security inspection, station ticket checking, entrance and exit access control, card punching attendance, shopping payment and the like have higher requirements on face recognition accuracy; meanwhile, most image acquisition terminals are embedded systems, the number of people to be identified is large, and the time requirement of the scenes on face identification is strict.
After the mask is worn, the feature expression of the face is changed and cannot be matched with the face feature record stored by the system, so that the traditional rapid face recognition technology is not effective any more. At present, most people walk to a camera to take off the mask for face recognition in a mask wearing face processing mode of the scene. The treatment mode is inconvenient to operate and low in efficiency, and brings certain infection risks to the examinees and security personnel. Therefore, it has become an urgent need to solve face recognition under the wearing mask condition in the above-described scenario.
The positioning of key points of the human face, also called human face alignment, is an important link in human face recognition. The pre-positioned coordinates mainly used for characterizing the facial image features are usually marked on the outline or the center of the nose, eyes, mouth, chin and other parts of the face, and are also called as facial feature points. Meanwhile, a plurality of information such as the position, the posture, the contour and the like of facial features can be roughly confirmed through the coordinates of the key points of the face, so that the method is often used as an important link in the process of face visual recognition.
In addition, in the field of face tracking, the motion following and tracking of the face model can be realized by utilizing the position information of the key points. In the field of facial expression recognition, local features provided by key points and relative position relations can be used as a basis for judging facial expressions. In the field of three-dimensional face reconstruction, face information determined through key points is combined with a prior model, and three-dimensional reconstruction of a face model can be realized.
The method solves the problem of face shielding of the mask wearing the mask passively from the angle of information loss caused by reducing mask shielding to face identification, but does not actively solve the problem of face identification of the mask wearing the mask from the angle of characteristic description of the face of the mask wearing the mask, and also fails to solve the problem of face key point loss caused by mask shielding, so that the face identification performance cannot be further optimized from the angle of face key point positioning.
Therefore, the key point positioning method for the face wearing the mask is provided, a face shape model of the face wearing the mask is designed by utilizing the shape characteristic of the face wearing the mask, and 68 key point distributions containing the face and the mask are prepositioned. On the basis, a corresponding face data set of the mask wearing face is constructed, the accurate positioning of the preset position of the face wearing face is realized by adopting a cascade regression tree algorithm, and the effectiveness of the method is verified through tests.
Disclosure of Invention
The invention aims to provide a method for positioning key points of a face with a mask, which solves the following technical problems: the problem of face shielding of a wearing mask is solved from an information missing angle caused by passively reducing mask shielding for face identification, but the problem of face identification of the wearing mask is actively solved from an angle of feature depiction of the face of the wearing mask, the problem of missing key points of the face caused by mask shielding is not solved, and further optimization of face identification performance can not be realized from a face key point positioning angle.
In order to achieve the purpose, the invention adopts the following technical scheme:
a method for locating key points of a face with a mask, comprising the following steps:
prepositioning mask wearing face key points;
constructing a face data set of the mask;
mask face key point is worn in accurate location.
Further preferably, the prepositioning of the key points of the face of the respirator mainly comprises:
divide the face after the gauze mask laminating face into two parts: the human face area and the mask-human face attaching area are not shielded;
for the non-occluded face region: prepositioning an eyebrow part, an eye part and a nose bridge, and recording the eyebrow part, the eye part and the nose bridge as a left eyebrow and right eyebrow key point, a left eye and right eye key point and a nose bridge key point respectively;
for mask-face fit areas: replacing the face with the lower edge part of the mask for pre-positioning, and recording as the key point of the lower edge of the mask; the prepositioning of the nose part is replaced by the upper edge of the mask, and is marked as the key point of the upper edge of the mask; and (4) replacing the mouth features with the textures inside the mask to perform pre-positioning, and recording as mask texture key points.
Further preferably, the total number of the face key points is 68, and the method includes: 17 gauze mask lower edge key points, 10 left eyebrow right eyebrow key points, 4 bridge of the nose key points, 12 left eye right eye key points, 5 gauze mask upper edge key points, 20 gauze mask texture key points.
Further preferably, the mask lower edge key points comprise: the characteristic points of the lower edge of the mask are easy to identify and the characteristic points of the lower edge of the mask are not easy to identify;
easily discern gauze mask lower limb characteristic point and include:
point 9, lowest point of the mask;
point 1 and point 17, the highest points of the left and right masks;
point 3 and point 15, intersection point of the first texture of the mask and the left and right boundaries;
the non-easy-to-identify mask lower edge feature points include:
points 5-8, 4 points are taken at equal intervals between the point 4 and the point 9;
points 10-13, 4 points equally spaced between point 14 and point 9;
the key point of the left eyebrow and the right eyebrow comprises: identifying the eyebrow feature points easily and identifying the eyebrow feature points not easily;
the feature points of the eyebrow part easy to identify comprise;
the feature points of the eyebrow part which are not easy to identify comprise;
points 19-21, 3 points equally spaced along the curve generated by the upper edge of the eyebrow between point 18 and point 22;
points 24-25, 3 points equidistant from point 23 and point 27 along the curve generated by the upper edge of the eyebrow;
the nose bridge key points comprise easy-to-identify nose bridge feature points and non-easy-to-identify nose bridge feature points;
the easy-to-identify nose bridge feature point includes:
the non-easily identifiable nose bridge feature points include:
point 29 and point 30, the two bisectors between point 31 and point 28;
the left-eye and right-eye key points comprise eye feature points which are easy to identify and eye feature points which are not easy to identify;
the easily recognizable eye feature points include:
point 37 and point 40, point 43 and point 46, left and right intraocular outer canthus;
the non-easily recognizable eye feature points include:
2 points of the upper eyelid curve between point 38 and point 39, point 37 and point 40, taken equidistant from the upper eyelid curve;
2 points of the lower eyelid curve between point 41 and point 42, point 37 and point 40, taken equidistant;
2 points of the upper eyelid curve between point 44 and point 45, point 43 and point 46, taken equidistant from each other;
2 points of the lower eyelid curve between point 47 and point 48, and between point 43 and point 46, taken equidistant;
the mask upper edge key points comprise: the upper edge feature points of the mask are easy to identify and the upper edge feature points of the mask are not easy to identify;
easily discern gauze mask upper edge characteristic point includes:
point 34, intersection of the lower edge of the mask beam and the extension line of the nose bridge;
point 32, point 33, point 35, point 36: making an intersection point of a parallel line and the upper edge of the mask from key points of the outer canthus in the left eye and the right eye along the extension line of the nose bridge;
the mask texture key points include: the mask texture feature points are easy to identify and the mask texture feature points are not easy to identify;
the mask texture feature points easy to identify comprise:
the mask texture feature points which are not easy to identify comprise:
2 points taken equidistant between point 63 and point 67, point 52 and point 58;
points 49-51, 3 points taken equidistant between point 52 and point 3;
points 53-55, 3 points taken equidistant between point 52 and point 15;
point 64 and point 65, 2 points taken equidistant between point 63 and point 14, and point 65 being the farther point from point 63;
Further preferably, wherein the constructing of the mask-worn face data set mainly comprises:
selecting MaskedFace-Net as an experimental data set source;
randomly selecting 400 pieces of face images of the mask according to CMFD of MaskedFace-Net, accurately marking key points of each piece of face of the mask according to key point distribution predefined by a morphological model, wherein the marked key points and a face image set of the mask form a face data set of the mask in a two-dimensional coordinate mode;
dividing the mask wearing face data set into two parts: wherein, a mask wearing face data set consisting of 300 images and corresponding key points is used as training set data for learning of a positioning model; and taking a wear-mask face data set consisting of the remaining 100 images and the corresponding key points as test set data for detecting the performance of the accurate positioning model.
Further preferred, wherein, accurate location dress gauze mask people face key point mainly includes:
selecting a regression tree integration algorithm as an accurate positioning algorithm for realizing key points of the face of the mask;
establishing a cascade regressor to predict the update vector of the face shape step by step;
selecting an initial shape and entering a regressorLine initialization, using the face image and the corresponding face shape as training data (I)1,S1),(I2,S2),...,(In,Sn) Predicting the next-level regression update difference according to the training data
And obtaining an initial regression function r0；
Iterating and updating the initial regressor to obtain K-level cascade regression function
Face training data (I) to be worn on a mask1,S1),(I2,S2),...,(In,Sn) As input, setting training parameters to generate a regressor;
and introducing IPN and/or ION as the basis for optimizing the positioning and judging standard of the key points of the human face, and analyzing the influence of the training parameters on the performance of the regressor so as to adjust the parameters to achieve the optimal performance of the regressor.
Further preferably, a cascade regressor is established to predict the update vector of the face shape step by step, and the expression is as follows:
wherein, I represents a face image, and t represents a cascade serial number; r istA regressor representing a current level; s represents a set of coordinates of all face key points in the face image, namely the face shape; while
Then the current estimate of the face shape S is represented, namely:
wherein the content of the first and second substances,Xi∈R2(ii) a 1,2, p, p is the number of key points in each face, XiThe real coordinates of the ith key point in the face are obtained.
It is further preferred that, wherein an initial shape is selected and the regressor is initialized, the face image and the corresponding face shape are taken as training data (I)1,S1),(I2,S2),...,(In,Sn) Predicting the next-level regression update difference according to the training data
And obtaining an initial regression function r0The definition is as follows:
wherein, pi is belonged to {1,. eta., n },
further preferably, the initial regressor is iterated and updated to obtain K-level cascaded regression functions
The expression is as follows:
rikrepresents the kth cascade regressor in the ith human face,
is a weak regression function where i 1. N-nR, R is the number of initialized faces in each picture, i.e. the number of oversampled faces.
The invention has at least the following beneficial effects:
the invention provides a novel scheme of a mask wearing face key point pre-positioning model, a data set marking and an accurate positioning model. Firstly, the prepositioning of 68-point face key points of the mask wearing face is established through the characteristic of the fit of the face and the mask, so that the mask wearing face key points of a public data set are standardized, a corresponding face data set containing key point rough positioning is established, then the accurate positioning of the mask wearing face is realized by adopting a regression model, and the effectiveness of the mask wearing face is verified through experiments. Compared with other methods, the method provided by the invention has lower key point positioning error under the condition of wearing the mask, and can provide technical support for rapid identification of the face identity wearing the mask.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a schematic diagram of a mask wearing face key point mark;
FIG. 2 is a schematic diagram of a key point mark of a human face at the eyebrow part of a wearing mask;
FIG. 3 is a schematic view of a face key point mark on the face of a respirator mask;
FIG. 4 is a schematic diagram illustrating the effect of cascade depth on model ION error;
FIG. 5 is a graph illustrating the effect of the number of trees in each cascade on model ION error;
FIG. 6 is a diagram illustrating the effect of oversampling number on model ION error;
fig. 7 is an effect diagram of key point positioning of a face wearing a mask.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The invention relates to a method for positioning key points of a face with a mask, which specifically comprises the following steps:
s1, mask wearing face key point pre-positioning
The face key point shape model is composed of a plurality of face key point sets. According to the characteristics of the identified object, the researcher performs corresponding prepositioning on the distribution of the key points. At present, the key point distribution of the common human face shape model comprises eyes, a nose and a mouth, but the wearing of the mask causes the loss of the key points of the lower face part. Therefore, the existing human face shape model is no longer applicable in the environment shielded by the mask, and a human face key point shape model worn by the mask needs to be designed.
1.1. Wearing mask human face shape model analysis
In order to completely characterize the face, the distribution of the key points needs to include the non-occluded part (eyebrow, eye and nose bridge) of the real face, and also needs to be able to indirectly characterize the mask part (mask outline and internal texture) of the face.
Thus, the wearer's face is divided into two parts: the human face area and the human face-mask joint area are not shielded, and the following design principle is determined:
(1) non-shielding face area
The non-shielded face area comprises the eyebrow part, the eye part and the nose bridge. According to the data of [9] [10] and the like, the upper part of the face has rich identification information, particularly the eye part [11] can represent the information of the face to a great extent, and therefore the key point which does not shield the face part can keep the prepositioning of the eyebrow part and the nose bridge part of the face.
(2) Face-mask attaching area
Due to the wearing fit characteristic of the face-mask, the contour of the mask can fit with the contour of the face. In addition, the disposable medical mask main body is formed by three layers of folds, deformation is generated under the support of the nose bridge and the lower face part, and the formed internal texture can represent part of face information. Therefore, the key points at the mask replace the face with the lower edge part of the mask, the upper edge of the mask replaces the nose part to be shielded, and the internal texture of the mask replaces the completely shielded lower face features.
1.2. Wearing mask face key point pre-positioning
The key points of the human face are usually pre-positioned in a two-dimensional or three-dimensional coordinate mode on contour points, corner points and bisector points of important parts of the human face.
Wear gauze mask face form model and carry out prepositioning to 68 key points altogether, including under the gauze mask along key point 17, left eyebrow, right eyebrow key point 10, the bridge of the nose key point 4, left eye, right eye key point 12, along 5 key points on the gauze mask, gauze mask texture key point 20. Each mask wearing face key point is numbered and marked according to the number sequence. Wherein the key points of the face of the wearer are selected as shown in figure 1.
(1) Key point of lower edge of mask 1-17
Firstly, determining characteristic points which are easy to identify on the lower edge of the mask, wherein the characteristic points comprise the lowest point (point 9) of the mask, the highest points (point 1 and point 17) of the left mask and the right mask, intersection points (point 2 and point 16) of the lower edge of a cross beam of the mask and left and right boundaries, intersection points (point 3 and point 15) of a first texture of the mask and the left and right boundaries, and intersection points (point 4 and point 14) of a second texture of the mask and the left and right boundaries;
and then determining key points which are not easy to identify, wherein the key points comprise 4 points (points 5-8 and 10-13) which are equidistant from the intersection point of the second texture of the mask and the left and right boundaries to the lowest point of the mask.
The total number of the key points is 17, and the highest point of the lower edge of the left mask is used as the starting point, and the highest point of the lower edge of the right mask passes through the lowest point of the lower edge of the left mask from left to right.
(2) Key points of the left eyebrow and the right eyebrow 18-22/23-27
Firstly, determining characteristic points which are easy to identify on the eyebrow part, including the upper eyebrow heads and the tail eyebrows (points 18, 22, 23 and 27) along the left eyebrow and the right eyebrow; then, key points which are not easy to identify are determined, and 3 points (points 19-21 and points 24-25) are respectively taken from the left and the right of the eyebrow to the left and the right of the eyebrow along the curve generated by the upper edge of the eyebrow from the head of the left and the right eyebrow to the tail of the left and the right eyebrow.
The total number of the marked key points is 10, starting from the tail of the left eyebrow and ending from the tail of the right eyebrow.
(3) Key points of nose bridge 28-31
Firstly, determining characteristic points which are easy to identify, including an intersection point (point 31) between the bridge of the nose and the upper edge of the mask, and an intersection point (point 28) between a connecting line of the inner canthus of the left eye and the right eye and the bridge of the nose;
less readily identifiable key points are then determined, including two bisectors (point 29 and point 30) between the intersection of the bridge of the nose with the upper edge of the mask and the intersection between the line connecting the corners of the left and right eyes with the bridge of the nose.
The above total number of key points is 4, and the marking is performed in the order from top to bottom.
(4) Key points of left eye and right eye 37-42/43-48
The marking information includes: the corresponding position information of the equant eyelid points and the marked eye corner points of the eyes in the face image of the mask, wherein the equant eyelid points comprise: marking an upper eyelid curve and a lower eyelid curve on the basis of the mask-face image, and determining upper eyelid trisection points of upper eyelids and lower eyelid trisection points of lower eyelids of eyes in the face image;
specifically, the characteristic points which are easily identified by the eyes are firstly determined, including the left and right intraocular external canthi (point 37 and point 40, point 43 and point 46); less readily identifiable key points are then determined, including taking 2 points (points 38 and 39, points 41 and 42, points 44 and 45, points 47 and 48) equally spaced left and right on the upper and lower eyelid curves determined from the left and right intraocular canthi to the left and right external canthus key points.
The total number of the marked key points is 12, starting from the external canthus of the left eye, the curve of the upper eyelid passing through the left eye and then the curve of the lower eyelid passing through the left eye is marked clockwise along the curve of the upper eyelid and the lower eyelid. The right eye is treated in the same way. The keypoint markers of the eyebrow portion are shown in fig. 2.
(5) Key points of the upper edge of the mask are 32-36
Firstly, determining characteristic points which are easy to identify on the upper edge of the mask, and the intersection of the lower edge of the cross beam of the mask and the extension line of the nose bridge (point 34); then, key points which are not easy to identify are determined, and intersection points (points 32, 33, 35 and 36) of parallel lines and the upper edge of the mask are determined from key points of the inner canthus of the left eye and the outer canthus of the right eye along the extension line of the nose bridge.
Above mark key point altogether 5, from a left side to the right side, make on parallel lines and the gauze mask along the bridge of the nose extension line with the outer canthus of the left eye along the bridge of the nose extension line and follow intersect (point 32) for the beginning, make on parallel lines and the gauze mask along the bridge of the nose extension line along the inner canthus of the left eye along the bridge of the nose extension line intersect (point 33), gauze mask crossbeam lower limb and bridge of the nose extension line intersect department (point 34), the inner canthus of the right eye makes on parallel lines and the gauze mask along the bridge of the nose extension line along the bridge of the nose intersection point (point 35), make on parallel lines and the gauze mask along the bridge of the nose extension line with the outer canthus of the right eye along the bridge of the nose extension line and follow intersect (point 36) until.
(6) Mask texture key points of 49-68
Firstly, determining feature points which are easy to identify in the textures of the mask, namely an intersection point (point 52) of a first texture of the mask from top to bottom and a nose bridge extension line, and an intersection point (point 58) of the first texture of the mask from bottom to top and the nose bridge extension line.
Then, determining the characteristic points which are not easy to identify in the texture of the mask, and 2 equally dividing points (point 63 and point 67) between the two points;
3 points (points 49-51 and 53-55) are respectively and equidistantly taken from the left side and the right side between the intersection point (point 52) of the first texture of the mask and the extension line of the nose bridge and the intersection point (point 3 and point 15) of the first texture of the mask and the left and the right boundaries.
2 points (point 59, point 60, point 56 and point 57) are respectively and equidistantly taken from left to right between the intersection point (point 58) of the first texture and the extension line of the nose bridge and the intersection point (point 4 and point 14) of the second texture and the left and right boundaries of the mask from bottom to top.
2 points (point 61 and point 62, point 64 and point 65, and point 61 and point 65 are farther from point 63) are respectively taken at equal distances from the left and right between the point 63 and the intersection point of the second texture of the mask and the left and right boundaries (point 4 and point 14).
The midpoints (points 68 and 66) are taken between the point 61 and the point 67, and between the point 65 and the point 67, respectively.
The total number of the key points is 20, and the key points are marked from the outer circle to the inner circle clockwise from the L point. The key point marks of the mask portion are shown in fig. 3.
S2, constructing a face data set of the mask
The pre-positioning of the distribution of key points of the face of the mask wearing the mask is realized by utilizing the mask wearing face shape model, and in order to realize the accurate positioning of the key points, a mask wearing face data set marked by utilizing the shape model needs to be constructed. Therefore, a certain number of face images of the wearer mask need to be acquired and key points need to be accurately marked.
2.1. Wearing mask face dataset composition
The experimental data set mainly comes from MaskedFace-Net and comprises 137,016 large data sets of high-quality mask face images. Whether the mask is worn or not and whether the mask is worn or not accurately can be regarded as a baseline data set of a machine learning model related to mask face analysis, and the baseline data set consists of a mask face data set (CMFD) worn correctly, a mask face data set (IMFD) worn incorrectly and a combined data set (MaskedFace-Net).
2.2. Wearing mask face dataset labeling
And randomly selecting 400 mask-wearing face images in the CMFD, and accurately marking the key points of each mask-wearing face according to the key point distribution predefined by the morphological model. The marked key points and the mask wearing face image set form a mask wearing face data set in a two-dimensional coordinate mode.
The accurate positioning model realizes the function of accurate positioning by learning the distribution of key points corresponding to the mask wearing face image in the data set, but simultaneously needs to rely on another part of data which is not learned to detect the positioning performance. The data set of the face of the wearer mask consisting of the image of the face of 400 pieces of the wearer mask and its key points is thus divided into two parts. Wherein, 300 images and the corresponding key point distribution are used as training set data for learning of the positioning model, and the remaining 100 images and the corresponding key point distribution are used as test set data for detecting the performance of the accurate positioning model.
S3, accurate positioning of key points of face of mask
The Regression tree integration (ERT) algorithm is a face key point localization algorithm based on a Regression tree, and has the same loss function when training can be maximally reduced while performing shape-invariant feature selection compared with the traditional algorithm.
The regression tree integration algorithm has the characteristics of easy understanding, easy construction and high speed, and therefore, the regression tree integration algorithm can be effectively used for realizing accurate positioning of key points of the face of the mask.
3.1. Regression tree integration algorithm principle
The regression tree integration algorithm firstly establishes a cascade regressor to predict the update vector of the human face shape step by step, and the expression is as follows:
wherein, I represents a face image, and t represents a cascade serial number; r istA regressor representing a current level; s represents a set of coordinates of all face key points in the face image, namely the face shape; while
Then the current estimate of the face shape S is represented, namely:
wherein, Xi∈R2(ii) a i is 1,2, and p is the number of key points in each face, and is set as 68 in the experiment. XiThe real coordinates of the ith key point in the face are obtained.
Next, an initial shape is selected and the regressor is initialized, for which purpose the face image and the corresponding face shape are required as training data (I)1,S1),(I2,S2),...,(In,Sn) Predicting the next-level regression update difference according to the training data
And obtaining an initial regression function r0It is defined as follows:
wherein, pi is belonged to {1,. eta., n },
then, iteration and updating are carried out on the initial regressor to obtain a K-level cascade regression function
The expression is as follows:
rikrepresents the kth cascade regressor in the ith human face,
is a weak regression function where i 1. N-nR, R is the number of initialized faces in each picture, i.e. the number of oversampled faces.
The algorithm obtains a finished regressor through a learning tree, and finally obtains the coordinate position of the key point of the face image.
3.2. Training parameter optimization
Training and generating regressor mask face training data (I)1,S1),(I2,S2),...,(In,Sn) As input, and sets the training parameters to generate the regressor. ERT algorithm is a common human face key point positioning algorithm, but because of wearing the mouthThe particularity of the mask face is that the distribution of the prepositioned key points is different from that of the face, so that the original parameters of the algorithm need to be optimized.
The oversampling number R, the cascade depth K, and the number of trees in each cascade are important parameters in the regression tree integration algorithm. The influence of the parameters on the performance of the regressor is analyzed to adjust the parameters to achieve the optimal performance of the regressor. Therefore, it is necessary to introduce one or more criteria for positioning and judging key points of the human face as a basis for optimization.
3.3. Key point positioning error evaluation standard
In recent research on keypoint localization, a generally accepted method for evaluating keypoint localization performance is to calculate a localization error, and common measures of face keypoint detection errors for a single facial image are: inter-pupil distance normalized error (IPNE) and Inter-ocular distance normalized error (IONE). The 300-W data set is the most commonly used public data set at present, and two face alignment error calculation methods, namely IPN and ION, are used as standards for judging the performance of the face alignment method. Therefore, the two methods are common judgment methods for the face alignment error at present.
(1) Interpupillary distance normalization error
Inter-pupil distance normalized error (IPNE) is the keypoint placement error calculated by dividing the average distance between the predicted face keypoints and the real face keypoints by the Inter-pupil distance (distance between eye centers). The calculation method is shown in formula (7):
xpreito predict face keypoints, xgtiKey points of real faces, dIPDRepresenting the distance between the pupils of the eye. x is the number ofqCoordinate x value, y representing the qth keypointqRepresents the coordinate y value of the qth keypoint. The 36 th to 41 th key points are key points of the left eye part, and the 42 th to 47 th key points are key points of the right eye partAnd (4) key points.
left_x＝(x36+x37+x38+x39+x40+x41)/6
left_y＝(y36+y37+y38+y39+y40+y41)/6
right_x＝(x42+x43+x44+x45+x46+x47)/6
right_y＝(y42+y43+y44+y45+y46+y47)/6
dIPD＝D((left_x,left_y),(right_x,right_y))
(2) Normalized error of interocular distance
Inter-ocular distance normalized error (ion) is the key point location error calculated by calculating the average distance between the predicted face key points and the real face key points divided by the distance between the outer corners of the eyes. The calculation method is shown in formula (8):
wherein d isIOD dIODRepresenting the distance between the outer corners of the eye.
The difference between the two error calculation methods is that the denominator calculation as error normalization is different. And calculating the average number of the normalized errors of each face with the mask in the test data set, taking the average number as the judgment standard of the performance of the regressor, and optimizing the training parameters according to the judgment standard.
S4, experiment and test
In order to verify the effectiveness of the mask-wearing face key point positioning method, it needs to be proved that the regressor can accurately position the key points. Therefore, the tuning parameters explore the cascade depth, the number of trees in each cascade level and the influence of the oversampling number on the regressor and obtain the optimal performance.
4.1. Experiment platform
During testing, the CPU used is Intel (R) core (TM) i5-9500 CPU, the CPU frequency is 3.0GHZ, the RAM is 8.00GB, and the type of the used display card is NVIDIA GeForce GTX 1650. The whole experimental process is completed on python, and the key point marking and the model training are carried out through a key point marking tool imglab.
4.2. Experimental testing
Firstly, the influence of Cascade depth (Cascade depth) on the ION positioning error of key points of the face of the mask is researched, and partial experimental results are shown in fig. 4.
Next, the influence of the number of trees in each cascade (Num trees per cascade level) on the performance of the mask-worn face key point model is explored, and some experimental results are shown in fig. 5:
the effect of the oversampling number on the model ION error was then explored and the result is shown in fig. 6.
According to experimental results, the ION error has a minimum peak under the influence of the cascade depth and the number of trees in each cascade. Meanwhile, the ION error shows a stable descending trend after the oversampling number exceeds 20. Finally, it was found that the model had better results when the cascade depth was 7, the number of trees in each cascade was set to 1200 and the number of initialization sample points was set to 400.
For this reason, the experiment also compared the effect of cascade depth and the number of trees in each cascade on performance with the number of oversampled and total number of trees in all cascades unchanged, and some results are shown in table 1:
TABLE 1 Cascade depth and number of trees in each cascade level impact on model performance
The final effect of the model on locating key points of the face wearing the mask is shown in fig. 7.
Because a mask wearing face key point morphological model is adopted in the regressor, the morphological model designed aiming at normal faces in most public data sets cannot be used for verifying the effectiveness of the model. The test data set can not compare the IPN and ION error result tested by the model with the result tested by other methods in the public data set, and only can use the result of the 300-Wchallinging data set of other methods under the similar mask shielding condition for reference, and the results are shown in tables 2 and 3:
TABLE 2 error in test data set of face key point localization method for face wearing mask
| IPN error | ION error | |
| Ours | 5.68 | 3.65 |
TABLE 3 error in 300-W challinging dataset for other methods
| IPN error | ION error | |
| ERT[13] | 6.40 | - |
| PFLD1X+[15] | 6.33 | - |
| LAB[16] | - | 4.85 |
The average errors of IPN and ION tested by other methods on the 300-W challinging data set were referenced by taking the IPN and ION errors obtained on the test set as results. According to the result, the average error of IPN obtained by the experimental model is 5.68, and the average error of ION is 3.65. The average error of IPN tested by ERT in 300-W data set is 6.40, the average error of IPN tested by PFLD 1X + in challinging data set is 6.33, and the average error of ION tested by LAB algorithm is 4.85.
The experimental results preliminarily prove that the method has lower key point positioning error compared with other methods under the condition of wearing the mask, and can provide certain technical support for the rapid identification of the face identity wearing the mask. Positioning is of reference in comparison to other methods under similar conditions.
In conclusion:
the invention provides a novel scheme of a mask wearing face key point pre-positioning model, a data set marking and an accurate positioning model. Firstly, the prepositioning of 68-point face key points of the mask wearing face is established through the characteristic of the fit of the face and the mask, so that the mask wearing face key points of a public data set are standardized, a corresponding face data set containing key point rough positioning is established, then the accurate positioning of the mask wearing face is realized by adopting a regression model, and the effectiveness of the mask wearing face is verified through experiments.
The foregoing shows and describes the general principles, principal features, and advantages of the invention. It will be understood by those skilled in the art that the present invention is not limited to the embodiments described above, which are merely illustrative of the principles of the invention, but that various changes and modifications may be made without departing from the spirit and scope of the invention, which fall within the scope of the invention as claimed. The scope of the invention is defined by the appended claims and equivalents thereof.
Claims (9)
1. A method for positioning key points of a face with a mask, which is characterized by comprising the following steps:
prepositioning mask wearing face key points;
constructing a face data set of the mask;
mask face key point is worn in accurate location.
2. The method for locating key points of a face of a mask as claimed in claim 1, wherein the pre-locating of key points of the face of the mask mainly comprises:
divide the face after the gauze mask laminating face into two parts: the face area and the mask-face fit area are not shielded;
for an unobstructed face region: prepositioning an eyebrow part, an eye part and a nose bridge, and recording the eyebrow part, the eye part and the nose bridge as a left eyebrow and right eyebrow key point, a left eye and right eye key point and a nose bridge key point respectively;
for mask-face fit areas: replacing the face with the lower edge part of the mask for pre-positioning, and recording as the key point of the lower edge of the mask; the prepositioning of the nose part is replaced by the upper edge of the mask, and is marked as the key point of the upper edge of the mask; and (4) replacing the mouth features with the textures inside the mask to perform pre-positioning, and recording as mask texture key points.
3. The method for locating key points of a face with a mask according to claim 2, wherein the key points of the face are 68 in number, comprising: 17 gauze mask lower edge key points, 10 left eyebrow right eyebrow key points, 4 bridge of the nose key points, 12 left eye right eye key points, 5 gauze mask upper edge key points, 20 gauze mask texture key points.
4. The method for locating key points of a face with a mask as claimed in claim 3, wherein the key points of the lower edge of the mask comprise: the characteristic points of the lower edge of the mask are easy to identify and the characteristic points of the lower edge of the mask are not easy to identify;
easily discern gauze mask lower limb characteristic point and include:
point 9, lowest point of the mask;
point 1 and point 17, the highest points of the left and right masks;
point 2 and point 16, intersection point of lower edge of the mask beam and left and right boundaries;
point 3 and point 15, intersection point of the first texture of the mask and the left and right boundaries;
point 4 and point 14, intersection of the second texture of the mask and the left and right boundaries;
the non-easy-to-identify mask lower edge feature points include:
points 5-8, 4 points are taken at equal intervals between the point 4 and the point 9;
points 10-13, 4 points equally spaced between point 14 and point 9;
the key point of the left eyebrow and the right eyebrow comprises: identifying the eyebrow feature points easily and identifying the eyebrow feature points not easily;
the feature points of the eyebrow part easy to identify comprise;
point 18, point 22, point 23 and point 27, which are the upper edges of the left and right eyebrows along the brow heads and brow tails;
the feature points of the eyebrow part which are not easy to identify comprise;
points 19-21, 3 points equally spaced along the curve generated by the upper edge of the eyebrow between point 18 and point 22;
points 24-25, 3 points equidistant from point 23 and point 27 along the curve generated by the upper edge of the eyebrow;
the nose bridge key points comprise easy-to-identify nose bridge feature points and non-easy-to-identify nose bridge feature points;
the easy-to-identify nose bridge feature point includes:
point 31, the intersection of the bridge of the nose and the upper edge of the mask;
point 28, the intersection between the line connecting the canthus of the left and right eyes and the bridge of the nose;
the non-easily identifiable nose bridge feature points include:
point 29 and point 30, the two bisectors between said point 31 and point 28;
the left-eye and right-eye key points comprise eye feature points which are easy to identify and eye feature points which are not easy to identify;
the easily recognizable eye feature points include:
point 37 and point 40, point 43 and point 46, left and right intraocular outer canthus;
the non-easily recognizable eye feature points include:
2 points of the upper eyelid curve between point 38 and point 39, point 37 and point 40, taken equidistant from the upper eyelid curve;
2 points of the lower eyelid curve between point 41 and point 42, point 37 and point 40, taken equidistant;
2 points of the upper eyelid curve between point 44 and point 45, point 43 and point 46, taken equidistant from each other;
2 points of the lower eyelid curve between point 47 and point 48, and between point 43 and point 46, taken equidistant;
the mask upper edge key points comprise: the upper edge feature points of the mask are easy to identify and the upper edge feature points of the mask are not easy to identify;
easily discern gauze mask upper edge characteristic point includes:
point 34, intersection of the lower edge of the mask beam and the extension line of the nose bridge;
point 32, point 33, point 35, point 36: making an intersection point of a parallel line and the upper edge of the mask from key points of the outer canthus in the left eye and the right eye along the extension line of the nose bridge;
the mask texture key points include: the mask texture feature points are easy to identify and the mask texture feature points are not easy to identify;
the mask texture feature points easy to identify comprise:
point 52, which is the intersection point of the first texture of the mask from top to bottom and the extension line of the nose bridge;
point 58, which is the intersection point of the first texture of the mask from bottom to top and the extension line of the nose bridge;
the mask texture feature points which are not easy to identify comprise:
2 points taken equidistant between point 63 and point 67, point 52 and point 58;
points 49-51, 3 points taken at equal distances between point 52 and point 3;
points 53-55, 3 points taken equidistant between point 52 and point 15;
point 59 and point 60, 2 points taken equidistant between point 58 and point 4;
point 56 and point 57, 2 points taken equidistant between point 58 and point 14;
point 61 and point 62, 2 points taken equidistant between point 63 and point 4, with point 61 being the farther point from point 63;
point 64 and point 65, 2 points taken equidistant between point 63 and point 14, and point 65 being the farther point from point 63;
points 68 and 66, which are the midpoints between points 61 and 67, and between points 65 and 67, respectively.
5. The method for locating key points on a face of a wearer's mask as claimed in claim 1, wherein constructing the data set of the face of the wearer's mask mainly comprises:
selecting MaskedFace-Net as an experimental data set source;
randomly selecting 400 pieces of face images of the mask according to CMFD of MaskedFace-Net, accurately marking key points of each piece of face of the mask according to key point distribution predefined by a morphological model, wherein the marked key points and a face image set of the mask form a face data set of the mask in a two-dimensional coordinate mode;
dividing the mask wearing face data set into two parts: wherein, a mask wearing face data set consisting of 300 images and corresponding key points is used as training set data for learning of a positioning model; and taking a wear-mask face data set consisting of the remaining 100 images and the corresponding key points as test set data for detecting the performance of the accurate positioning model.
6. The method for locating key points of a face with a mask according to claim 1, wherein the accurate locating of key points of the face with the mask mainly comprises:
selecting a regression tree integration algorithm as an accurate positioning algorithm for realizing key points of the face of the mask;
establishing a cascade regressor to predict the update vector of the face shape step by step;
selecting an initial shape and initializing a regressor, using the face image and the corresponding face shape as training data (I)1,S1),(I2,S2),...,(In,Sn) Predicting the next-level regression update difference according to the training data
And obtaining an initial regression function r0；
Iterating and updating the initial regressor to obtain K-level cascade regression function
Face training data (I) to be worn on a mask1,S1),(I2,S2),...,(In,Sn) As input, setting training parameters to generate a regressor;
and introducing IPN and/or ION as the basis for optimizing the positioning and judging standard of the key points of the human face, and analyzing the influence of the training parameters on the performance of the regressor so as to adjust the parameters to achieve the optimal performance of the regressor.
7. The method as claimed in claim 6, wherein a cascade regressor is established to predict the updated vector of the face shape step by step, and the expression is as follows:
wherein, I represents a face image, and t represents a cascade serial number; r istA regressor representing a current level; s represents a set of coordinates of all face key points in the face image, namely the face shape; while
Then the current estimate of the face shape S is represented, namely:
wherein, Xi∈R2(ii) a 1,2, p, p is the number of key points in each face, XiThe real coordinates of the ith key point in the face are obtained.
8. The method according to claim 6, wherein the initial shape is selected and the regressor is initialized, and the face image and the corresponding face shape are used as training data (I)1,S1),(I2,S2),...,(In,Sn) Predicting the next-level regression update difference according to the training data
And obtaining an initial regression function r0The definition is as follows:
wherein, pi is belonged to {1,. eta., n },
9. the method of claim 6, wherein the initial regressor is iterated and updated to obtain K-level cascaded regression function
The expression is as follows:
rikrepresents the kth cascade regressor in the ith human face,
is a weak regression function where i 1. N-nR, R is the number of initialized faces in each picture, i.e. the number of oversampled faces.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210061922.8A CN114495221A (en) | 2022-01-19 | 2022-01-19 | Method for positioning key points of face with mask |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210061922.8A CN114495221A (en) | 2022-01-19 | 2022-01-19 | Method for positioning key points of face with mask |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114495221A true CN114495221A (en) | 2022-05-13 |
Family
ID=81471714
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210061922.8A Pending CN114495221A (en) | 2022-01-19 | 2022-01-19 | Method for positioning key points of face with mask |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114495221A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115205951A (en) * | 2022-09-16 | 2022-10-18 | 深圳天海宸光科技有限公司 | Wearing mask face key point data generation method |
| CN115460502A (en) * | 2022-11-11 | 2022-12-09 | 成都智元汇信息技术股份有限公司 | Headset identification method and system based on reduced target area |
-
2022
- 2022-01-19 CN CN202210061922.8A patent/CN114495221A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115205951A (en) * | 2022-09-16 | 2022-10-18 | 深圳天海宸光科技有限公司 | Wearing mask face key point data generation method |
| CN115460502A (en) * | 2022-11-11 | 2022-12-09 | 成都智元汇信息技术股份有限公司 | Headset identification method and system based on reduced target area |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114495221A (en) | Method for positioning key points of face with mask | |
| Chow et al. | Towards a system for automatic facial feature detection | |
| CN102592260B (en) | Certificate image cutting method and system | |
| WO2021174819A1 (en) | Face occlusion detection method and system | |
| CN111344703B (en) | User authentication device and method based on iris recognition | |
| US8369625B2 (en) | Method for grouping 3D models to classify constitution | |
| CN112818901B (en) | Wearing mask face recognition method based on eye attention mechanism | |
| CN112507963B (en) | Automatic generation of batch mask face samples and mask face recognition method | |
| CN105138965A (en) | Near-to-eye sight tracking method and system thereof | |
| CN109886080A (en) | Human face in-vivo detection method, device, electronic equipment and readable storage medium storing program for executing | |
| CN111967296B (en) | Iris living body detection method, access control method and device | |
| CN111898569B (en) | Face identification method based on living body detection | |
| CN106570447A (en) | Face photo sunglass automatic removing method based on gray histogram matching | |
| CN112069986A (en) | Machine vision tracking method and device for eye movements of old people | |
| CN113657195A (en) | Face image recognition method, face image recognition equipment, electronic device and storage medium | |
| CN113705466B (en) | Face five sense organ shielding detection method for shielding scene, especially under high imitation shielding | |
| CN105701486B (en) | A method of it realizing face information analysis in video camera and extracts | |
| Feng et al. | Using eye aspect ratio to enhance fast and objective assessment of facial paralysis | |
| CN114894337A (en) | Temperature measurement method and device for outdoor face recognition | |
| US20230020160A1 (en) | Method for determining a value of at least one geometrico-morphological parameter of a subject wearing an eyewear | |
| CN114120426A (en) | Mask face recognition method based on local blocking attention double-branch optimization | |
| CN110211679A (en) | A kind of self-service examination machine eyesight detection intelligent processing method and device | |
| CN111241505A (en) | Terminal device, login verification method thereof and computer storage medium | |
| CN109344791B (en) | Identification and identification method based on intelligent face scanning | |
| CN111241870A (en) | Terminal device and face image recognition method and system thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |