WO2021051611A1

WO2021051611A1 - Face visibility-based face recognition method, system, device, and storage medium

Info

Publication number: WO2021051611A1
Application number: PCT/CN2019/118428
Authority: WO
Inventors: 孙太武; 张艳; 周超勇; 刘玉宇
Original assignee: 平安科技（深圳）有限公司
Priority date: 2019-09-19
Filing date: 2019-11-14
Publication date: 2021-03-25
Also published as: CN110751043B; CN110751043A

Abstract

The present application relates to the technical field of face recognition, and discloses a face visibility-based face recognition method, applicable to an electronic device. The method comprises: performing detection on an image to be processed, and acquiring a position of a face region in the image; determining, on the basis of a key point alignment technique, key points of a face in the face region, and performing orientation correction processing on the face; performing quality assessment on the face region that has undergone the orientation correction processing, and acquiring a quality assessment score; performing visibility assessment on a face region having a quality assessment score falling within a pre-determined value range, and acquiring a corresponding visibility assessment score; and performing face recognition and feature extraction on a face region having a visibility assessment score falling within a pre-determined value range. The present application solves, by means of face visibility, problems caused by occlusion during image quality assessment, and improves the accuracy and efficiency of face recognition.

Description

Face recognition method, system, device and storage medium based on face visibility

This application requires the priority of the patent application whose application number is 201910885914.3, the application date is September 19, 2019, and the invention and creation title is "Face Recognition Method, Device and Storage Medium Based on Facial Visibility".

Technical field

This application relates to the field of face recognition technology, and in particular to a face recognition method, system, device and storage medium based on face visibility.

Background technique

The face recognition technology has already been demonstrated in academia, and it has also been applied in a small range in industry. However, with the increase in the level of the database, for example, from thousands of people in the same building, to tens of thousands of people in a community, to tens of millions of people in a city, the difficulty of identification gradually increases. In this process, in order to maintain the user experience, most of the identification work is based on the user's non-cooperative operation, for example, the user does not need to be required to identify at a specific angle, specific light, and stationary, etc. At the same time, this also brings great difficulty to face recognition.

In order to overcome the above-mentioned problems, existing face recognition solutions will have a quality evaluation module, that is, when the quality of the photographed photo does not meet the requirements, no recognition is performed, and only when the quality meets the requirements, recognition is performed. This process is equivalent to a preliminary filtering before the picture is recognized. However, the applicant realizes that the existing quality evaluation algorithms mostly filter blurs (including motion blur, low picture pixels and other blurs), insufficient lighting (too bright and dim), and can’t be occluded. Perfect solution (for example, the user wears sunglasses, face mask, etc.). However, this partially occluded picture also greatly affects the accuracy of recognition, resulting in face recognition failure or poor results.

For this reason, there is an urgent need for a technology that can analyze the situation of face occlusion, so as to improve the accuracy of face recognition.

Summary of the invention

This application provides a face recognition method, system, electronic device, and storage medium based on face visibility. Its main purpose is to solve the problem of occlusion that cannot be solved in image quality evaluation through face visibility, and it can also greatly improve the face. Accuracy and time of recognition.

In order to achieve the above objective, the present application provides a face recognition method based on face visibility, the method includes:

Detecting the picture to be processed, and obtaining the position of the face area in the picture to be processed;

Based on the key point alignment technology, determine the key points of the face in the face area, and perform a straightening process on the face;

Perform quality evaluation on the face area after the straightening process, and obtain a quality evaluation score;

Perform visibility evaluation on the face area whose quality evaluation score meets the preset value range, and obtain a corresponding visibility evaluation score;

Face recognition and feature extraction are performed on the face area whose visibility evaluation score meets the preset value range.

Correspondingly, this application provides a face recognition system based on face visibility, including:

A position determining unit, configured to detect the picture to be processed, and obtain the position of the face area in the picture to be processed;

The face straightening unit is configured to determine the key points of the face in the face area based on the key point alignment technology, and perform straightening processing on the face;

A quality evaluation unit, configured to evaluate the quality of the face area after the straightening process, and obtain a quality evaluation score;

The visibility evaluation unit is configured to evaluate the visibility of the face area whose quality evaluation score meets the preset value range, and obtain the corresponding visibility evaluation score;

The face recognition unit is used to perform face recognition and feature extraction on the face area whose visibility evaluation score meets the preset value range.

In addition, in order to achieve the above object, the present application also provides an electronic device, which includes a memory and a processor, the memory includes a face recognition program based on face visibility, and the When the face recognition program is executed by the processor, the following steps are implemented:

Detect the picture to be processed, and obtain the position of the face in the picture to be processed;

Based on the key point alignment technology, determine the key points of the face in the picture to be processed, and perform a straightening process on the face;

In addition, in order to achieve the above object, the present application also provides a storage medium, the storage medium includes a face recognition program based on the visibility of the face, when the face recognition program based on the visibility of the face is executed by the processor , To achieve any step in the face recognition method based on the visibility of the face as described above.

The face recognition method, system, electronic device, and computer-readable storage medium proposed in this application are based on the visibility of the face, which acquires and aligns the key points of the face in the picture to be processed, and then combines quality evaluation and visibility The evaluation gradually filters the face area and performs face recognition operations on pictures that meet the filtering conditions. This can not only solve the problem of face occlusion in the picture, but also improve the accuracy and speed of face recognition.

Description of the drawings

FIG. 1 is a schematic diagram of an application environment of a specific embodiment of face recognition based on face visibility in this application;

2 is a schematic diagram of modules of a specific embodiment of the face recognition program based on face visibility in FIG. 1;

3 is a flowchart of a specific embodiment of a face recognition method based on face visibility according to this application;

Figure 4 is a schematic diagram of the multi-task model structure of the application assistance system.

The realization, functional characteristics, and advantages of the purpose of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings.

detailed description

It should be understood that the specific embodiments described here are only used to explain the application, and not used to limit the application.

Example one

This application provides a face recognition method based on face visibility, which is applied to an electronic device 1. Referring to FIG. 1, this is a schematic diagram of an application environment of a specific embodiment of a face recognition method based on face visibility of this application.

In this embodiment, the electronic device 1 may be a terminal device with arithmetic function, such as a server, a smart phone, a tablet computer, a portable computer, a desktop computer, and the like.

The electronic device 1 includes a processor 12, a memory 11, a network interface 14 and a communication bus 15.

The memory 11 includes at least one type of readable storage medium. The at least one type of readable storage medium may be a non-volatile storage medium such as flash memory, hard disk, multimedia card, card-type memory 11, and the like. In some embodiments, the readable storage medium may be an internal storage unit of the electronic device 1, such as a hard disk of the electronic device 1. In other embodiments, the readable storage medium may also be the external memory 11 of the electronic device 1, such as a plug-in hard disk or a smart memory card (Smart Media Card, SMC) equipped on the electronic device 1. , Secure Digital (SD) card, Flash Card (Flash Card), etc.

In this embodiment, the readable storage medium of the memory 11 is generally used to store the face recognition program 10 based on the visibility of the face installed in the electronic device 1 and the like. The memory 11 can also be used to temporarily store data that has been output or will be output.

In some embodiments, the processor 12 may be a central processing unit (CPU), a microprocessor or other data processing chip, which is used to run the program code or process data stored in the memory 11, for example, to execute based on human face. Visibility of the face recognition program 10 etc.

The network interface 14 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface), and is generally used to establish a communication connection between the electronic device 1 and other electronic devices.

The communication bus 15 is used to realize the connection and communication between these components.

FIG. 1 only shows the electronic device 1 with the components 11-15, but it should be understood that it is not required to implement all the illustrated components, and more or fewer components may be implemented instead.

Optionally, the electronic device 1 may also include a user interface, a display, and a touch sensor.

In addition, the area of the display of the electronic device 1 may be the same as or different from the area of the touch sensor. Optionally, the display and the touch sensor are stacked to form a touch display screen. The device detects the touch operation triggered by the user based on the touch screen.

Optionally, the electronic device 1 may also include a radio frequency (RF) circuit, a sensor, an audio circuit, etc., which will not be repeated here.

In the device embodiment shown in FIG. 1, the memory 11 as a computer storage medium may include an operating system and a face recognition program 10 based on face visibility; the processor 12 executes the human-based face recognition program stored in the memory 11 The face recognition program 10 for face visibility implements the following steps:

Based on the key point alignment technology, determine the key points of the face region in the picture to be processed, and perform a straightening process on the face;

Preferably, the step of determining the key points of the human face in the human face area and performing the correction processing on the human face includes:

Acquiring annotated image data, and training an alignment model based on the annotated image data;

Inputting the picture to be processed into the alignment model, and outputting key point coordinate information of the face corresponding to the picture to be processed;

Based on the key point coordinate information, obtain the correction angle of the picture to be processed, and rotate the picture to be processed according to the correction angle to obtain the face area after correction.

Preferably, the step of acquiring annotated image data and training an alignment model based on the annotated image data includes:

Obtain an image set with pre-marked key points as annotated image data;

The face image corresponding to the labeled image is used as the network input of the training model of the alignment model, and the pre-labeled key point coordinate positions of the face image are used as the label of the training model for training;

Obtain the sum of Euclidean distances between the output of the training model and the label, and perform normalization processing to obtain a loss function;

Perform parameter iteration based on the loss function until a trained alignment model is obtained.

Preferably, the step of evaluating the quality of the face area after the straightening and obtaining the quality evaluation score includes:

Training quality evaluation model;

Performing quality evaluation on the face area after the straightening based on the quality evaluation model, and obtaining a quality evaluation score;

The training steps of the quality assessment model include:

Train a multi-task neural network, the input of the multi-task neural network is the face area after the square, the output of the multi-task neural network is the face feature of the face area and the face feature corresponding to the face Point value

Multiplying the face feature and the corresponding score value to obtain the final face recognition feature;

Perform network training based on the final face recognition feature and loss function to obtain the quality evaluation model.

Preferably, the step of performing visibility evaluation on the face area whose quality evaluation score meets a preset value range, and obtaining the corresponding visibility evaluation score includes:

Training visibility evaluation model;

Performing visibility evaluation on the face area that meets the quality evaluation based on the visibility evaluation model;

The training steps of the visibility evaluation model include:

Based on the multi-task neural network, input the visibility of the face key points of the face area after the correction, and output the visibility probability of the face key points of the face area;

Perform network training based on the visibility probability and loss function of the key points of the face to obtain the visibility evaluation model. The loss function is the number of errors in the visibility judgment of the key points.

The electronic device 1 proposed in the above embodiment integrates the three modules of the alignment model, the quality evaluation model, and the visibility evaluation model, which can not only solve the problem that the quality evaluation model cannot solve the problem of reduced recognition accuracy caused by face occlusion, but also improve the recognition accuracy, It can also solve the recognition problem through a multi-task neural network, which greatly reduces the reasoning time of the entire process.

In other embodiments, the face recognition program 10 based on face visibility can also be divided into one or more modules, and the one or more modules are stored in the memory 11 and executed by the processor 12 to complete this Application. The module referred to in this application refers to a series of computer program instruction segments that can complete specific functions. Referring to FIG. 2, it is a program module diagram of a preferred embodiment of the face recognition program 10 based on the visibility of the face in FIG. 1. The face recognition program 10 based on face visibility can be divided into: a position determination unit 11, a face straightening unit 12, a quality evaluation unit 13, a visibility evaluation unit 14, and a face recognition unit 15. The functions or operation steps implemented by the modules 11-15 are all similar to the above, and will not be described in detail here. Illustratively, for example:

The position determining unit 11 is configured to detect the picture to be processed, and obtain the position of the face area in the picture to be processed.

The face straightening unit 12 is configured to determine the key points of the face in the face area based on the key point alignment technology, and perform straightening processing on the face.

The quality evaluation unit 13 is configured to evaluate the quality of the face area after the straightening process, and obtain a quality evaluation score.

The visibility evaluation unit 14 is configured to evaluate the visibility of the face area whose quality evaluation score meets the preset value range, and obtain the corresponding visibility evaluation score.

The face recognition unit 15 is configured to perform face recognition and feature extraction on a face region whose visibility evaluation score meets a preset value range.

Example two

In addition, this application also provides a face recognition method based on face visibility. Referring to FIG. 3, this is a flowchart of a specific embodiment of a face recognition method based on face visibility of this application. The method can be executed by a device, and the device can be implemented by software and/or hardware.

In this embodiment, the face recognition method based on face visibility provided in the present application includes: step S110-step S150.

S110: Detect the picture to be processed, and obtain the position of the face area in the picture to be processed.

S120: Based on the key point alignment technology, determine the key points of the face in the face area, and perform a straightening process on the face.

Among them, the face point, that is, the landmarks obtained by Face Alignment, is a step that must be done in face recognition. First, we perform face detection from the image to be processed to get the position of the face. Secondly, we use key point alignment technology to get the key points of the face (such as eye points, nose points, mouth points, etc.), and perform a straightening operation on the face. In a preferred embodiment of the present application, the key points of the human face are determined to be 68.

Further, the step of determining the key points of the face in the face area in the picture to be processed, and correcting the face includes:

1. Obtain annotated image data, and train an alignment model based on the annotated image data;

2. Input the picture to be processed into the alignment model, and output the key point coordinate information of the face corresponding to the picture to be processed;

3. Based on the key point coordinate information, obtain the squared angle of the picture to be processed, and rotate the picture to be processed to obtain the face area after squared.

In addition, the step of acquiring annotated image data as described above, and training an alignment model based on the annotated image data includes:

1. Obtain an image set with pre-marked key points as annotated image data;

2. The face image corresponding to the annotation image is used as the network input of the training model of the alignment model, and the pre-annotated key point coordinate positions of the face image are used as the label of the training model for training;

3. Obtain the sum of Euclidean distances between the output of the training model and the label, and perform normalization processing to obtain a loss function;

4. Perform parameter iteration based on the loss function until the trained alignment model is obtained.

Among them, the annotated image data is a picture set or image set with pre-marked key points. The coordinates of the key points can be obtained by the regression method, and the x and y coordinates of the 68 key points in the picture or image are directly returned, that is, the output It is a one-dimensional vector of 68*2=136. When the x and y coordinates of the left eye point and the right eye point of the face are determined respectively, a rotation angle can be obtained, and the face can be straightened by rotating the picture based on the rotation angle.

In order to obtain the alignment model, a large amount of face data marked with key points is required. The network input of the training model of the alignment model is an image of a face obtained after face detection, and the label of the training model is the coordinate position of 68 points on the face. It should be noted that for these 68 points, we need to pay attention to the order, that is, we need to label the key point coordinate positions of the face image as the label of the training model according to the preset labeling sequence. For example, point 1 to point 5 on the left eyebrow, and point 6 to point 10 on the right eyebrow, and mark the key areas of the face such as the nose, mouth, and chin in turn. Our label input is as follows, x1, y1, x2, y2...x68, y68 total 136 values. In the network training process, calculate the Euclidean distance sum of the 68 points of the network output and the label, and normalize it to obtain our loss function. After multiple iterations of parameters, the final alignment model or key point alignment model is obtained.

S130: Perform quality evaluation on the face area after the straightening process, and obtain a quality evaluation score.

Wherein, the step of performing quality evaluation on the face area after the straightening process and obtaining the quality evaluation score includes:

Training quality evaluation model;

Performing quality evaluation on the face area after the straightening process based on the quality evaluation model, and obtaining a quality evaluation score;

The training steps of the quality assessment model include:

1. Train a multi-task neural network, the input of the multi-task neural network is the face area after the square, the output of the multi-task neural network is the face feature of the face area and the face feature Corresponding score value;

2. Multiply the face feature and the corresponding score value to obtain the final face recognition feature;

3. Perform network training based on the final face recognition feature and loss function to obtain the quality evaluation model.

Further, the quality evaluation of the face region is carried out through the quality evaluation model. The quality evaluation model can be done by a multi-task network based on a simple recognition model. One of the branches of the multi-task network is extracted after face recognition. The face feature of, the other branch, after passing the sigmoid function, a score value between 0-1 is obtained. Multiply the score value with the previously extracted face features to obtain the final face recognition feature; then, after the triplet loss loss function is trained, the final quality evaluation model can be obtained. When inferring the model, input a picture to get a quality evaluation score. The closer the quality evaluation score is to 1, the higher the quality of the face region is considered.

Among them, Triplet Loss is a loss function in deep learning, which is used to train samples with small differences, such as human faces. Feed data includes Anchor examples, Positive examples, and Negative examples. By optimizing the distance between the anchor example and the negative example, and subtracting the distance between the anchor example and the positive example, the larger the difference, the better.

S140: Perform visibility evaluation on the face area whose quality evaluation score meets a preset value range, and obtain a corresponding visibility evaluation score.

Wherein, the step of performing visibility evaluation on the face area whose quality evaluation score meets the preset value range, and obtaining the corresponding visibility evaluation score includes:

Training visibility evaluation model;

The training steps of the visibility evaluation model include:

S150: Perform face recognition and feature extraction on a face region whose visibility evaluation score meets a preset value range.

It should be noted that whether it is an alignment model, a quality evaluation model, or a visibility evaluation model, they are all implemented based on the characteristics of the human face. In particular, face alignment coordinates and visibility are different attributes of the face, and the quality of face points is also part of the quality evaluation model. Therefore, this application designs a face recognition assistance system that integrates alignment, quality evaluation, and visibility evaluation into a multi-task model.

Specifically, the multi-task model of the auxiliary system includes a weight-sharing bottom layer, and multiple branches connected to the bottom layer and independent of each other, as shown in FIG. 4.

The multi-task model branch of the auxiliary system further includes: an alignment model branch for obtaining key points of a face, a quality evaluation model branch, and a visibility evaluation model branch for judging occlusion. In the multi-task model training process, the above three branches are trained separately, and finally the model is fine-tuned with pictures with three tags (with key points, quality evaluation scores, and visibility evaluation scores).

Among them, the alignment model is first trained, that is, shared weights and landmarks. At this time, the parameters of the IQA (image quality assessment) model and the Visibility (visibility) model branch are fixed, and only return during the training process. Pass the loss of the landmarks branch. After the alignment model branch training is completed, the fixed weights share the parameters of the underlying shared weights and the parameters of the landmarks. Then, open the IQA and Visibility modules respectively to train these two branches. Finally, turn on the weights of all modules, and use pictures with three tabs for short-term fine-tuning. So far, the multi-task model training is completed.

The following is a detailed description of the training process of multi-task model training:

When training landmarks, it is consistent with a separate keypoint alignment network. The input of the training model is the picture and the 68 point coordinates of the picture, the output is the coordinates of 68 points, and the loss function is the normalized L2Loss (that is, normalized Euclidean distance difference).

When training the visibility branch, the input of the training model is the visibility of 68 points (visible as 1, and invisible as 0), and finally a 2*68 vector is output (2 represents the probability of visible and invisible, visible and invisible The sum of the probability is 1, we take the result greater than 0.5 as the final output result), the loss function is the number of visibility judgment errors of 68 points, and it is multiplied by a set coefficient to prevent overfitting .

When training the IQA branch, our input is three images (anchor, positive, negative), where anchor and positive are the same person, negative is a different person, and the final loss is the triplet multiplied by the coefficient (that is, IQA score) Loss, makes the features of the first two anchor and positive closer, and the distance between the first and the third (anchor and negative) is the farthest.

In the end, when all modules are trained together, we need to input three pictures, as well as the coordinates of the landmarks of these three pictures, and the visibility of the landmarks. The final loss loss is the algebraic sum of the loss loss of the three modules.

Specifically, after running the above-mentioned multi-task model to obtain the result, we obtain a quality evaluation result through the IQA branch. When the quality evaluation score is judged to be less than 0.5, it indicates that the image quality cannot meet the recognition requirements, and the subsequent operations are not continued. When the judgment quality evaluation score is greater than 0.5, the visibility branch continues to be used for visibility judgment. For example, when it is judged that more than 20% of the points of the face in the picture are invisible (68*20%=14), it indicates that the face is more occluded. The face recognition operation is not continued, on the contrary, the face recognition operation is performed.

Using the above-mentioned face recognition method based on the visibility of face points, the three modules of face point model, visibility model and quality evaluation are integrated together, which not only solves the problem of occlusion that cannot be solved in the quality evaluation module, but also greatly Improve the accuracy of recognition; and use multi-tasking to solve multiple modules, which can also greatly reduce the reasoning time of the entire process.

In addition, the above-mentioned multi-task model can be used for face recognition as well as other face attributes. For example, when we classify eyelids, when we judge that the points on the eyelids are blocked by the glasses frame, we can Eyelid classification is not performed. When performing beard classification, if the beard is hidden by objects such as hands, microphones, etc., the beard classification can also not be performed. This can significantly improve the inference speed of face attributes and avoid recognition errors caused by occlusion. The resulting accuracy is reduced.

Example three

Corresponding to the aforementioned face recognition method and electronic device based on face visibility, this application also provides a face recognition system based on face visibility. Its logical structure is similar to that of the aforementioned electronic device based on face visibility. The module composition of the face recognition program 10 (shown in Figure 2) is similar, and the functions implemented by the position determining unit 11, the face straightening unit 12, the quality evaluation unit 13, the visibility evaluation unit 14, and the face recognition unit 15 are or The operation steps are similar to the logical composition of the face recognition system based on face visibility in this embodiment. For example:

The position determining unit is used to detect the picture to be processed and obtain the position of the face area in the picture to be processed;

The face straightening unit is used to determine the key points of the face in the face area obtained by the position determining unit based on the key point alignment technology, and perform the straightening process on the face;

The quality evaluation unit is used to evaluate the quality of the face region after the face correction unit has been straightened, and obtain a quality evaluation score;

The visibility evaluation unit is used to evaluate the visibility of the face area whose quality evaluation score meets the preset value range, and obtain the corresponding visibility evaluation score;

Correspondingly, in the face recognition system based on face visibility of this embodiment, the face straightening unit may include an alignment model training unit, a coordinate information determining unit, and a straightening unit (not shown in the figure). Wherein, the alignment model training unit is used to obtain annotated image data and train an alignment model based on the annotated image data; the coordinate information determination unit is used to input the image to be processed into the alignment model, and output the same as the image to be processed. Corresponding key point coordinate information of the human face; a rotating correction unit for obtaining the correction angle of the picture to be processed based on the key point coordinate information, and rotating the picture to be processed according to the correction angle, Get the face area after straightening.

The annotated image data is a picture set or image set with pre-marked key points. The coordinates of the key points can be obtained by the regression method. The x and y coordinates of the 68 key points in the image are directly returned, that is, the output is 68 *2 = a one-dimensional vector of 136. When the x and y coordinates of the left eye point and the right eye point of the face are determined respectively, a rotation angle can be obtained, and the face can be straightened by rotating the picture based on the rotation angle.

In order to obtain the alignment model, a large amount of face data labeled with key points is required. In this embodiment, the number of face key points in the face area is set to 68. The network input of the training model of the alignment model is an image of a face obtained after face detection, and the label of the training model is the coordinate position of 68 points on the face.

In another preferred implementation of this embodiment, the alignment model unit further includes an annotation image data acquisition unit, a training unit, a normalization unit, and an iteration unit (not shown in the figure).

Wherein, the annotated image data acquiring unit is used to acquire an image set with pre-annotated key points as annotated image data; the training unit is used to use the face image corresponding to the annotated image as the network input of the training model of the alignment model, The pre-labeled key point coordinate positions of the face image are used as the label of the training model for training; the normalization unit is used to obtain the sum of the Euclidean distance between the output of the training model and the label, and Perform normalization processing to obtain a loss function; an iterative unit is used to iterate parameters based on the loss function until a trained alignment model is obtained.

In another preferred implementation of this embodiment, the quality evaluation unit further includes a first evaluation model training unit and a first evaluation unit (not shown in the figure).

Among them, the first evaluation model training unit is used to train the quality evaluation model; the first evaluation unit is used to evaluate the quality of the corrected face region based on the quality evaluation model trained by the first evaluation model training unit, and obtain the quality Evaluation score.

In another preferred implementation of this embodiment, the first evaluation model training unit further includes a first network training unit, a face recognition feature determination unit, and an evaluation model acquisition unit (not shown in the figure). Wherein, the network training unit is used to train a multi-task neural network, the input of the multi-task neural network is the face area after the square, the output of the multi-task neural network is the face features of the face area and the The score value corresponding to the face feature; the face recognition feature determination unit is used to multiply the face feature and the corresponding score value to obtain the final face recognition feature; the evaluation model acquisition unit is used to based on the final face recognition feature and The loss function is used for network training to obtain a quality evaluation model.

The quality evaluation unit can be done with a multi-task network based on a simple recognition model. One of the branches is the facial features extracted after face recognition, and the other branch, after passing through the sigmoid function, gets a value between 0-1 The number of points in between. Multiply the score value with the previously extracted face features to obtain the final face recognition feature; then, after the triplet loss loss function is trained, the final quality evaluation model can be obtained. When inferring the model, input a picture to get a quality evaluation score. The closer the quality evaluation score is to 1, the higher the quality of the face region is considered.

In another preferred implementation of this embodiment, the visibility evaluation unit further includes a second evaluation model training unit and a second evaluation unit (not shown in the figure).

Among them, the second evaluation model training unit is used to train the visibility evaluation model; the second evaluation unit is used to evaluate the visibility of the face area that meets the quality evaluation based on the visibility evaluation model trained by the second evaluation model training unit .

Further, the above-mentioned second evaluation model training unit further includes a second network training unit and a second evaluation model acquisition unit (not shown in the figure). Wherein, the second network training unit is used for inputting the visibility of face key points in the face area after correcting based on a multi-task neural network, and outputting the visibility probability of face key points in the face area;

The second evaluation model acquisition unit is used to perform network training based on the visibility probability and loss function of key points of the face to obtain the visibility evaluation model, where the loss function is the number of errors in the visibility judgment of the key points.

It should be understood that the foregoing implementation manners are not all implementation manners of the third embodiment, and the specific implementation manners of the third embodiment are substantially the same as the foregoing specific implementation manners of the face recognition method and electronic device based on the visibility of the face, and will not be omitted here. Go into details.

Example four

In addition, an embodiment of the present application also proposes a storage medium that includes a face recognition program based on face visibility. When the face recognition program based on face visibility is executed by a processor, the implementation is as described above. The steps of the face recognition method based on face visibility and the operation of the face recognition system based on face visibility as described above.

The specific implementation of the computer-readable storage medium of the present application is substantially the same as the specific implementation of the above-mentioned face recognition method, system, and electronic device based on face visibility, and will not be repeated here.

It should be noted that in this article, the terms "include", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements not only includes those elements, It also includes other elements not explicitly listed, or elements inherent to the process, device, article, or method. If there are no more restrictions, the element defined by the sentence "including a..." does not exclude the existence of other identical elements in the process, device, article, or method that includes the element.

The serial numbers of the foregoing embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments. Through the description of the above implementation manners, those skilled in the art can clearly understand that the above-mentioned embodiment method can be implemented by means of software plus the necessary general hardware platform, of course, it can also be implemented by hardware, but in many cases the former is better.的实施方式。 Based on this understanding, the technical solution of this application essentially or the part that contributes to the existing technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as ROM/RAM) as described above. , Magnetic disks, optical disks), including several instructions to make a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) execute the method described in each embodiment of the present application.

The above are only the preferred embodiments of the application, and do not limit the scope of the patent for this application. Any equivalent structure or equivalent process transformation made using the content of the description and drawings of the application, or directly or indirectly applied to other related technical fields , The same reason is included in the scope of patent protection of this application.

Claims

A face recognition method based on face visibility, applied to an electronic device, characterized in that the method includes:

Detecting the picture to be processed, and obtaining the position of the face area in the picture to be processed;

Based on the key point alignment technology, determine the key points of the face in the face area, and perform a straightening process on the face;

Perform quality evaluation on the face area after the straightening process, and obtain a quality evaluation score;

Perform visibility evaluation on the face area whose quality evaluation score meets the preset value range, and obtain a corresponding visibility evaluation score;

Face recognition and feature extraction are performed on the face area whose visibility evaluation score meets the preset value range.
The face recognition method based on the visibility of the face according to claim 1, wherein the step of determining the key points of the face in the face area, and correcting the face include:

Acquiring annotated image data, and training an alignment model based on the annotated image data;

Inputting the picture to be processed into the alignment model, and outputting key point coordinate information of the face corresponding to the picture to be processed;

Based on the key point coordinate information, obtain the correction angle of the picture to be processed, and rotate the picture to be processed according to the correction angle to obtain the face area after correction.
The face recognition method based on face visibility according to claim 2, wherein the step of acquiring annotated image data, and training an alignment model based on the annotated image data comprises:

Obtain an image set with pre-marked key points as annotated image data;

Using the face image corresponding to the labeled image as the network input of the training model of the alignment model, and using the pre-annotated key point coordinate positions of the face image as the label of the training model for training;

Obtain the sum of Euclidean distances between the output of the training model and the label, and perform normalization processing to obtain a loss function;

Perform parameter iteration based on the loss function until a trained alignment model is obtained.
The face recognition method based on face visibility according to claim 3, wherein the labeled image data is a set of pictures or image sets marked with key points in advance, and the coordinates of the key points adopt a regression method Obtain, directly return the x-coordinate and y-coordinate of the key point in the picture or image.
The face recognition method based on the visibility of the face according to claim 4, wherein the number of key points of the face in the face area is 68.
The face recognition method based on face visibility according to claim 3, characterized in that,

Label the key point coordinate positions of the face image according to a preset labeling sequence as the label of the training model.
The face recognition method based on face visibility according to claim 1, wherein the step of evaluating the quality of the face area after the straightening process and obtaining the quality evaluation score comprises:

Training quality evaluation model;

Based on the quality evaluation model, the quality evaluation of the face area after the correction is performed, and the quality evaluation score is obtained.
The face recognition method based on face visibility according to claim 7, wherein the training step of the quality evaluation model comprises:

Train a multi-task neural network, the input of the multi-task neural network is the face area after the square, the output of the multi-task neural network is the face feature of the face area and the face feature corresponding to the face Point value

Multiplying the face feature and the corresponding score value to obtain the final face recognition feature;

Perform network training based on the final face recognition feature and loss function to obtain the quality evaluation model.
The face recognition method based on the visibility of the face according to claim 8, wherein the step of performing quality assessment on the face area after the correction based on the quality assessment model and obtaining the quality assessment score comprises:

A multi-task network based on a simple recognition model is used as the quality evaluation model; wherein, one branch of the multi-task network is the facial features extracted after face recognition, and the other branch of the multi-task network passes through After the sigmoid function, a score value between 0-1 is obtained;

Multiplying the score value with the extracted facial features to obtain the final facial recognition feature;

The final feature of face recognition is trained through a triplet loss loss function to obtain a final quality evaluation model.
The face recognition method based on face visibility according to claim 8, wherein the visibility evaluation is performed on the face area whose quality evaluation score meets a preset value range, and the corresponding visibility is obtained The steps to assess scores include:

Training visibility evaluation model;

Based on the visibility evaluation model, the visibility evaluation of the face area that meets the quality evaluation is performed.
The face recognition method based on face visibility according to claim 10, wherein the training step of the visibility evaluation model comprises:

Based on the multi-task neural network, the visibility of the key points of the face in the face area after the correction is input, and the output is the visibility probability of the key points of the face in the face area;

Perform network training based on the visibility probability and loss function of the key points of the face to obtain the visibility evaluation model. The loss function is the number of errors in the visibility judgment of the key points.
A face recognition system based on face visibility, which is characterized in that it includes:

A position determining unit, configured to detect the picture to be processed, and obtain the position of the face area in the picture to be processed;

The face straightening unit is configured to determine the key points of the face in the face area based on the key point alignment technology, and perform straightening processing on the face;

A quality evaluation unit, configured to evaluate the quality of the face area after the straightening process, and obtain a quality evaluation score;

The visibility evaluation unit is configured to evaluate the visibility of the face area whose quality evaluation score meets the preset value range, and obtain the corresponding visibility evaluation score;

The face recognition unit is used to perform face recognition and feature extraction on the face area whose visibility evaluation score meets the preset value range.
The face recognition system based on face visibility according to claim 12, wherein the face straightening unit comprises:

An alignment model training unit, configured to obtain labeled image data, and train an alignment model based on the labeled image data;

The coordinate information determining unit is configured to output the key point coordinate information of the face corresponding to the picture to be processed by inputting the picture to be processed into the alignment model;

The rotating straightening unit is configured to obtain the straightening angle of the picture to be processed based on the key point coordinate information, and to rotate the picture to be processed according to the straightening angle to obtain the face area after straightening.
The face recognition system based on face visibility according to claim 13, wherein the alignment model unit comprises:

Annotated image data acquisition unit for acquiring an image set with pre-annotated key points as annotated image data;

The training unit is configured to use the face image corresponding to the labeled image as the network input of the training model of the alignment model, and the pre-labeled key point coordinate positions of the face image are used as the label of the training model for training ；

A normalization unit, configured to obtain the sum of Euclidean distances between the output of the training model and the label, and perform normalization processing to obtain a loss function;

The iterative unit is used to iterate parameters based on the loss function until a trained alignment model is obtained.
The face recognition system based on face visibility according to claim 12, wherein the quality evaluation unit comprises:

The first evaluation model training unit is used to train the quality evaluation model;

The evaluation score obtaining unit is configured to evaluate the quality of the face area after the correction based on the quality evaluation model, and obtain the quality evaluation score.
An electronic device, characterized in that the electronic device includes a memory and a processor, the memory includes a face recognition program based on the visibility of the face, and the face recognition program based on the visibility of the face is used by the When the processor executes, the following steps are implemented:

Detecting the picture to be processed, and obtaining the position of the face area in the picture to be processed;

Based on the key point alignment technology, determine the key points of the face in the face area, and perform a straightening process on the face;

Perform quality evaluation on the face area after the straightening process, and obtain a quality evaluation score;

Perform visibility evaluation on the face area whose quality evaluation score meets the preset value range, and obtain a corresponding visibility evaluation score;

Face recognition and feature extraction are performed on the face area whose visibility evaluation score meets the preset value range.
The electronic device according to claim 16, wherein the step of determining the key points of the human face in the human face area and performing the correction processing on the human face comprises:

Acquiring annotated image data, and training an alignment model based on the annotated image data;

Inputting the picture to be processed into the alignment model, and outputting key point coordinate information of the face corresponding to the picture to be processed;

Based on the key point coordinate information, obtain the correction angle of the picture to be processed, and rotate the picture to be processed according to the correction angle to obtain the face area after correction.
The electronic device according to claim 16, wherein the step of acquiring annotated image data, and training an alignment model based on the annotated image data comprises:

Obtain an image set with pre-marked key points as annotated image data;

The face image corresponding to the labeled image is used as the network input of the training model of the alignment model, and the pre-labeled key point coordinate positions of the face image are used as the label of the training model for training;

Obtain the sum of Euclidean distances between the output of the training model and the label, and perform normalization processing to obtain a loss function;

Perform parameter iteration based on the loss function until a trained alignment model is obtained.
The electronic device according to claim 16, wherein the step of evaluating the quality of the face area after the straightening and obtaining the quality evaluation score comprises:

Training quality evaluation model;

Performing quality evaluation on the face area after the straightening based on the quality evaluation model, and obtaining a quality evaluation score;

The training steps of the quality assessment model include:

Train a multi-task neural network, the input of the multi-task neural network is the face area after the square, the output of the multi-task neural network is the face feature of the face area and the face feature corresponding to the face Point value

Multiplying the face feature and the corresponding score value to obtain the final face recognition feature;

Perform network training based on the final face recognition feature and loss function to obtain the quality evaluation model.
A computer-readable storage medium, wherein the computer-readable storage medium includes a face recognition program based on face visibility, and when the face recognition program based on face visibility is executed by a processor, The steps of realizing the face recognition method based on face visibility according to any one of claims 1 to 11.