CN111783748A - Face recognition method and device, electronic equipment and storage medium - Google Patents

Face recognition method and device, electronic equipment and storage medium Download PDF

Info

Publication number
CN111783748A
CN111783748A CN202010808939.6A CN202010808939A CN111783748A CN 111783748 A CN111783748 A CN 111783748A CN 202010808939 A CN202010808939 A CN 202010808939A CN 111783748 A CN111783748 A CN 111783748A
Authority
CN
China
Prior art keywords
face
preset
matrix
image
dimensional
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202010808939.6A
Other languages
Chinese (zh)
Other versions
CN111783748B (en
Inventor
王朋远
彭菲
黄磊
张建
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hanwang Technology Co Ltd
Original Assignee
Hanwang Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hanwang Technology Co Ltd filed Critical Hanwang Technology Co Ltd
Priority to CN202010808939.6A priority Critical patent/CN111783748B/en
Publication of CN111783748A publication Critical patent/CN111783748A/en
Application granted granted Critical
Publication of CN111783748B publication Critical patent/CN111783748B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/161—Detection; Localisation; Normalisation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/168—Feature extraction; Face representation
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Abstract

The application discloses a face recognition method, belongs to the technical field of face recognition, and is beneficial to improving the accuracy and reliability of face recognition. The method comprises the following steps: acquiring two-dimensional face images and three-dimensional face images which are synchronously acquired by different image acquisition devices aiming at a target face; determining pixel positions and two-dimensional image content information of all preset face key points in the two-dimensional face image, and determining three-dimensional space coordinates of all preset face key points in the three-dimensional face image; constructing corresponding nodes according to the preset face key points, and constructing an undirected graph by expressing the adjacency relation of the nodes corresponding to the adjacent preset face key points in the preset face key points through undirected edges; the method comprises the steps of extracting face features through a pre-trained graph convolution neural network based on a node feature matrix, a preset adjacency matrix and an identity matrix of a preset adjacency matrix of an undirected graph, and carrying out face recognition on a target face according to the extracted face features.

Description

Face recognition method and device, electronic equipment and storage medium
Technical Field
The present application relates to the field of face recognition technologies, and in particular, to a face recognition method, an apparatus, an electronic device, and a computer-readable storage medium.
Background
The face recognition is widely applied to the fields of attendance checking, entrance guard, security protection and the like. In order to improve the recognition degree of the face recognition on the living body face and improve the reliability of the face recognition, in the prior art, the face depth image collected by the depth image collecting device is combined with the face plane image collected by the visible light image collecting device to comprehensively perform the face recognition. For example, the face key points in the face depth image are determined by combining the face key point information in the face plane image, then, the face living body detection is performed according to the depth information of the face key points in the face depth image, and when the acquired face plane image is determined to be a living body face image, the face comparison and identification are further performed according to the face features extracted from the face plane image. Because the three-dimensional face features in the face depth image are difficult to extract, the scheme of carrying out face recognition by using the three-dimensional face features is rarely adopted in the prior art.
Disclosure of Invention
The application provides a face recognition method which is beneficial to improving the accuracy and reliability of face recognition.
In order to solve the above problem, in a first aspect, an embodiment of the present application provides a face recognition method, including:
acquiring two-dimensional face images and three-dimensional face images which are synchronously acquired by different image acquisition devices aiming at a target face;
determining pixel positions and two-dimensional image content information of all preset face key points in the two-dimensional face image, and determining three-dimensional space coordinates of all preset face key points in the three-dimensional face image;
constructing corresponding nodes according to the preset face key points, and constructing an undirected graph by expressing the adjacency relation of the nodes corresponding to the adjacent preset face key points in the preset face key points through undirected edges; the undirected graph comprises: a preset adjacency matrix representing the adjacency relation between the nodes and a node characteristic matrix formed by key point information stored by each node in the undirected graph;
extracting graph features based on the node feature matrix, the preset adjacent matrix and the unit matrix of the preset adjacent matrix through a pre-trained graph convolution neural network;
and performing face recognition on the target face by using the features obtained by extracting the image features by the image convolution neural network as face features.
In a second aspect, an embodiment of the present application provides a face recognition apparatus, including:
the face image acquisition module is used for acquiring a two-dimensional face image and a three-dimensional face image which are synchronously acquired by different image acquisition devices aiming at a target face;
the face key point information acquisition module is used for determining the pixel position and the two-dimensional image content information of each preset face key point in the two-dimensional face image and determining the three-dimensional space coordinate of each preset face key point in the three-dimensional face image;
the face key point undirected graph construction module is used for constructing corresponding nodes according to the preset face key points, expressing the adjacency relation of the nodes corresponding to the adjacent preset face key points in the preset face key points through undirected edges and constructing an undirected graph; the undirected graph comprises: a preset adjacency matrix representing the adjacency relation between the nodes and a node characteristic matrix formed by key point information stored by each node in the undirected graph;
the graph feature extraction module is used for extracting the graph features based on the node feature matrix, the preset adjacent matrix and the unit matrix of the preset adjacent matrix through a pre-trained graph convolution neural network;
and the face recognition module is used for performing face recognition on the target face by taking the feature obtained by extracting the image feature by the image convolution neural network as the face feature.
In a third aspect, an embodiment of the present application further discloses an electronic device, which includes a memory, a processor, and a computer program that is stored in the memory and can be run on the processor, and when the processor executes the computer program, the face recognition method according to the embodiment of the present application is implemented.
In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, where the computer program is executed by a processor, and the computer program performs the steps of the face recognition method disclosed in the embodiments of the present application.
The embodiment of the application discloses a face recognition method, which comprises the steps of obtaining two-dimensional face images and three-dimensional face images which are synchronously collected by different image collection devices aiming at a target face; determining pixel positions and two-dimensional image content information of all preset face key points in the two-dimensional face image, and determining three-dimensional space coordinates of all preset face key points in the three-dimensional face image; constructing corresponding nodes according to the preset face key points, and constructing an undirected graph by expressing the adjacency relation of the nodes corresponding to the adjacent preset face key points in the preset face key points through undirected edges; the undirected graph comprises: a preset adjacency matrix representing the adjacency relation between the nodes and a node characteristic matrix formed by key point information stored by each node in the undirected graph; extracting graph features based on the node feature matrix, the preset adjacent matrix and the unit matrix of the preset adjacent matrix through a pre-trained graph convolution neural network; and performing face recognition on the target face by using the features obtained by extracting the image features by the image convolution neural network as the face features, which is favorable for improving the accuracy and reliability of face recognition.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.
Fig. 1 is a flowchart of a face recognition method according to a first embodiment of the present application;
fig. 2 is a schematic diagram illustrating distribution of key points of a face in the face recognition method according to the first embodiment of the present application;
fig. 3 is a schematic diagram of an adjacency relation of key points of a face in the face recognition method according to the first embodiment of the present application;
fig. 4 is a schematic diagram illustrating a principle of establishing a three-dimensional coordinate system in the face recognition method according to the first embodiment of the present application;
fig. 5 is a flowchart of a face recognition method according to a second embodiment of the present application;
fig. 6 is a schematic structural diagram of a face recognition apparatus according to a second embodiment of the present application;
fig. 7 is a second schematic structural diagram of a face recognition apparatus according to a second embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some, but not all, embodiments of the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Example one
As shown in fig. 1, the face recognition method disclosed in the embodiment of the present application includes steps 110 to 150.
And step 110, acquiring two-dimensional face images and three-dimensional face images which are synchronously acquired by different image acquisition devices aiming at the target face.
In some embodiments of the present application, the different image capturing devices may be a binocular camera or a structured light camera on an electronic device, or may be two independent cameras. One of the different image acquisition devices is an image acquisition device for acquiring a three-dimensional face image (such as a face depth image), and the other one can be an image acquisition device for acquiring a two-dimensional face image (such as a visible light face image or an infrared light face image). The different image acquisition devices synchronously and respectively acquire a two-dimensional face image and a three-dimensional face image of the target face aiming at the target face. For example, the different image capturing devices may be structured light image capturing devices, by which a visible light image (i.e., a two-dimensional face image) and a depth image (i.e., a three-dimensional face image) of a face to be recognized are captured simultaneously.
Step 120, determining the pixel position and the two-dimensional image content information of each preset face key point in the two-dimensional face image, and determining the three-dimensional space coordinate of each preset face key point in the three-dimensional face image.
In some embodiments of the present application, the determining a pixel position and two-dimensional image content information of each preset face key point in the two-dimensional face image, and determining a three-dimensional space coordinate of each preset face key point in the three-dimensional face image includes: carrying out image calibration on the two-dimensional face image and the three-dimensional face image; detecting preset face key points corresponding to the target face according to the two-dimensional face image obtained after image calibration, and determining the pixel position and the two-dimensional image content information of each preset face key point; and determining the three-dimensional space coordinates of each preset human face key point in the three-dimensional human face image obtained after image calibration according to the pixel positions of the preset human face key points.
Because different image acquisition devices have physical position difference relative to an acquired object, image contents in a two-dimensional face image and a three-dimensional face image of a target face acquired by different image acquisition devices have certain visual difference, and in specific implementation, firstly, image calibration is carried out on an acquired original two-dimensional face image and an acquired three-dimensional face image based on relevant parameters of a camera of the image acquisition devices according to an imaging principle and installation positions of the different image acquisition devices. In specific implementation, an image calibration method in the prior art can be adopted to perform image calibration on the two-dimensional face image and the three-dimensional face image, and details are not repeated in the embodiment of the application.
In the embodiment of the application, the image calibration method of the two-dimensional face image and the three-dimensional face image is not limited.
After the calibrated two-dimensional face image is acquired, face key points in the two-dimensional face image are detected by adopting a method in the prior art. In some embodiments of the present application, the two-dimensional face image may be a visible light face image or an infrared light face image, and the two-dimensional face image collected under different spectral conditions will determine the face key points in the two-dimensional face image by using a corresponding face key point detection method in the prior art.
Taking the two-dimensional face image as a visible light face image as an example, a Dlib library (an open source library for machine learning) can be adopted to perform face detection and face key point alignment on the two-dimensional face image, so as to determine the pixel position of the face key point in the two-dimensional face image. Different face detection algorithms can acquire different numbers of face key points, and in a specific application process, preset face key points can be selected for subsequent face recognition according to specific face recognition requirements. In some embodiments of the present application, 104 of the detected face key points as shown in fig. 2 may be selected as preset face key points for subsequent face recognition.
In some embodiments of the application, when face key points are detected on a two-dimensional face image, pixel positions of preset face key points in the two-dimensional face image can be acquired, and meanwhile, content information of the two-dimensional image of each preset face key point can also be acquired. For example, for a visible light face image, a color value (such as a value of an RGB color space) and a transparency of each preset face key point may be obtained. For another example, for an infrared light face image, an infrared brightness value of each preset face key point may be acquired.
Because the pixel positions of the two-dimensional face image and the three-dimensional face image which are subjected to the image calibration processing are in one-to-one correspondence, the pixel position of each face key point in the two-dimensional face image obtained by detection is also the pixel position of the corresponding face key point in the three-dimensional face image obtained by the image calibration processing. That is, the pixel position of each face key point in the two-dimensional face image can be used as the pixel position of the corresponding face key point in the three-dimensional face image.
Furthermore, according to the pixel position of the corresponding face key point in the three-dimensional face image, a three-dimensional space coordinate can be obtained. For example, three-dimensional space coordinates of face key points are determined by reconstructing a face point cloud, each face key point in the point cloud represents the position of each pixel position in a two-dimensional face image in a real scene, and the position is in an established right-hand coordinate system by taking the mirror surface center of the image acquisition device as an origin. As shown in fig. 3, in constructing a coordinate system based on the imaging principle shown in fig. 3, the three-dimensional space coordinates of the corresponding face key points can be obtained by converting the pixel positions of the preset face key points in the three-dimensional image according to the following formula 1:
Figure BDA0002630205020000061
wherein (col, row) represents the pixel position of a certain point in the three-dimensional face image, depth represents the pixel value at the pixel position of (col, row) in the three-dimensional face image, opt and FoculLength are the parameter matrix of the image acquisition device of the three-dimensional face image, and (x, y, z) are the three-dimensional space coordinates of the key points of the face at the pixel position of (col, row) in the three-dimensional face image.
In some embodiments of the present application, other methods may also be used to perform conversion from pixel positions of face key points in a three-dimensional face image to three-dimensional space coordinates, which are not listed in this embodiment. The specific method adopted for converting the pixel position of the face key point in the three-dimensional face image into the three-dimensional space coordinate is not limited.
Step 130, constructing corresponding nodes according to the preset face key points, and constructing an undirected graph by representing the adjacency relation of the nodes corresponding to the adjacent preset face key points in the preset face key points through undirected edges.
Wherein the undirected graph comprises: a preset adjacency matrix representing the adjacency relation between the nodes and a node characteristic matrix formed by key point information stored by each node in the undirected graph. And the elements in the preset adjacency matrix represent whether an adjacency relation exists between any two nodes, namely whether the face key points corresponding to the two nodes are adjacent. The preset adjacency matrix is used for storing data of the relationship (such as whether edges are connected) of each node in the undirected graph (for example, the value of an edge element between nodes corresponding to two adjacent face key points is 1, and the value of an edge element between nodes corresponding to two non-adjacent face key points is 0).
In some embodiments of the application, in order to extract the face features in the two-dimensional face image and the three-dimensional face image, the data features and the relationships of the face key points in the two-dimensional face image and the three-dimensional face image are expressed by constructing an undirected graph. For example, a node data is respectively constructed according to the data of each preset face key point, each preset face key point corresponds to a node in an undirected graph one by one, and then the nodes corresponding to the face key points with an adjacent relation (such as adjacent) in the preset face key points are obtained through the undirected edge detection, so that the undirected graph consisting of the nodes and the edges can be obtained. For example, for face keypoints 0 and 1, two nodes d0 and d1 can be created from face keypoints 0 and 1, and assuming that face keypoints 0 and 1, a facieless L01 is created connecting nodes d0 and d 1. That is, in the undirected graph, there is an edge with a value of 1 between nodes d0 and d 1.
In some embodiments of the application, the three-dimensional face key points of the face are connected in the facial region, and then the facial regions with the adjacent relation are connected, so that all the detected face key points are connected into a complete image, and the undirected graph of the face is obtained. And the preset adjacency matrix is used for indicating whether an adjacency relation exists between any two nodes of the undirected graph. For example, for 104 preset face key points, whether an adjacency relationship exists between any two face key points in the 104 preset face key points can be represented by a matrix of 104 × 104. Taking the preset adjacency matrix as an example A, the matrix element A in the preset adjacency matrix AijIndicating whether or not there is an adjacency between the ith personal face key point and the jth personal face key point, e.g., Aij1 denotes the ith individual face key point and the jth individualAdjacent relationships exist between key points of the face, Aij0 means that there is no adjacency between the ith personal face key point and the jth personal face key point (i.e., the ith personal face key point is not related to the jth personal face key point), where i and j are integers.
The generation process of the preset adjacency matrix a is exemplified below.
In some embodiments of the present application, all of the detected preset face key points are divided into a cheek (e.g., 33 face key points with numbers 0 to 32 in fig. 2), a left eyebrow (e.g., 9 face key points with numbers 33 to 41 in the upper left corner in fig. 2), a right eyebrow (e.g., 9 face key points with numbers 42 to 50 in the upper right corner in fig. 2), a left eye (e.g., 9 face key points with numbers 66 to 74 below the left eyebrow in fig. 2), a right eye (e.g., 9 face key points with numbers 75 to 83 below the right eyebrow in fig. 2), a nose (e.g., 15 face key points with middle positions in fig. 2, numbers 51 to 65), a mouth (e.g., 20 face key points with numbers 84 to 103 in the lower middle region in fig. 2), and 7 facial region in total according to the five sense organs and the distribution of the preset face key points.
In some embodiments of the application, the face key points in the above-mentioned five sense organ regions are connected respectively, so that the adjacency relation between some face key points can be obtained, and the values of the corresponding matrix elements in the preset adjacency matrix are set according to the obtained adjacency relation.
For example, 33 face key points of the cheek are sequentially connected from 0 to 32, that is, 0 and 1, 1 and 2, … …, and 31 and 32 have an adjacent relationship respectively among the 33 face key points of the cheek, and in the preset adjacent relationship matrix, an edge connected with the face key point corresponding node having the adjacent relationship is set as an edge having a relationship value of 1. For example, the relationship value of the face key points 0 and 1 corresponding to the edge formed by connecting two nodes is set to 1. Correspondingly, the matrix element A of the preset adjacency matrix A0,1Is 1. For another example, the relationship value of the edge formed by connecting two nodes corresponding to the face key points 0 and 3 is set to 0, and correspondingly, the matrix element a of the preset adjacency matrix a0,3Is 0.
Also for example, the left and right eyebrow regions are eachAdjacent human face key points in the 9 human face key points are connected in sequence and are connected in a closed mode to form a ring. If the face key points 33 and 34, 34 and 35, and … … 41 and 33 in fig. 3 are respectively adjacent, the nodes corresponding to each group of adjacent face key points are respectively connected by an edge with a relation value of 1, such as the matrix element a of the preset adjacent matrix a33,34 Is 1. For another example, if the face key points 33 and 35 in fig. 3 are not adjacent, the nodes corresponding to the face key points 33 and 35 are connected by an edge with a relation value of 0, that is, the matrix element a of the preset adjacent matrix a33,35Is 0.
For another example, the neighboring face key points in the 8 edge face key points (as labeled 66 to 73 in fig. 3) in the left eye region are sequentially connected, the neighboring face key points in the 8 edge face key points (as labeled 75 to 82 in fig. 3) in the right eye region are sequentially connected, and are closed and connected to form a ring, and then the pupil key points (as labeled 73 and 84) are respectively connected to the 8 edge face key points in the left eye region and the 8 edge face key points in the right eye region.
The face key points on the outer edge of the nose are connected in sequence, closed into a ring, and then connected in sequence (51, 52, 53, 54 and 60 in fig. 3) in the nose tip region and the nose bridge region.
And for 20 human face key points in the mouth region, sequentially connecting the human face key points at the outer edge and the human face key points at the inner edge respectively, and closing and connecting to form a ring to form a double-ring shape.
In some embodiments of the application, for the face key points which are not in the same facial region, the face key points which are closer to each other may also be considered to have an adjacent relationship, and therefore, the face key points which are not in the same facial region and are closer to each other are sequentially connected into a line, so that all the preset face key points are connected into a complete graph, and the distribution information of the face key points is more comprehensively embodied.
For example, the external eyebrow corner, the external eye corner, and the face key point on the uppermost portion of the cheek are considered to have an adjacent relationship, and the nodes corresponding to the external eyebrow corner and the external eye corner, and the nodes corresponding to the face key point on the uppermost portion of the cheek are connected by the edges having a relationship value of 1, respectively. For another example, the face key points at the inner eyebrow angle, the inner canthus and the uppermost part of the nose bridge are considered to have an adjacent relationship, and the nodes corresponding to the inner eyebrow angle and the inner canthus and the nodes corresponding to the face key points at the inner canthus and the uppermost part of the nose bridge are respectively connected by the edges with the relationship value of 1. For example, it is considered that key points of the face on the central axis of the face, such as the eyebrow point, the nose apex, the nose bridge point, the nose tip point, the upper lip midpoint, the mouth midpoint, the lower lip midpoint, and the lower jaw midpoint, have an adjacent relationship in this order, and the nodes corresponding to the key points of the face having the adjacent relationship on the central axis of the face are connected by the edges having a relationship value of 1.
In other embodiments of the present application, adjacent face key points may also be determined in other manners. For example, regarding the above-mentioned face key points on the central axis of the face, it is considered that the face key points on the central axis of the face, such as the nasal apex, the nasal bridge point, the nasal cusp point, the upper lip midpoint, the mouth midpoint, the lower lip midpoint, and the lower jaw midpoint, have an adjacent relationship in order. Because the process of extracting the features by the graph neural network has strong robustness, the prediction result of the neural network cannot be influenced by only changing the relation value of one edge (namely, only changing the identification of the adjacent relation of a few face key points).
For two face key points with an adjacent relation (namely the two face key points which are connected), the nodes corresponding to the two face key points are connected through an edge with a relation value of 1; for two face key points without adjacency relation (i.e. the two face key points which are not connected), the nodes corresponding to the two face key points are connected by an edge with a relation value of 0. Accordingly, the relationship value of the edge connecting two nodes may be represented by the value of the corresponding matrix element of the preset adjacency matrix.
According to the method, the preset adjacency matrix of the undirected graph of the face can be obtained.
In some embodiments of the present application, redefining the face key points having the adjacency relationship will result in different adjacency matrices. In a face recognition application, it is preferable that the adjacency matrix of the undirected graph corresponding to each face is the same. That is, in a face recognition application, the adjacency matrix may be generated in advance according to the definition and connection rule of the face key points having the adjacency relationship.
On the other hand, for the graph structure, the data comprises two parts of nodes and edges, and the node storage data of the undirected graph generated in the previous step is represented by a node characteristic matrix. The key point information stored by each node includes: the three-dimensional space coordinates and the two-dimensional image content information of the preset human face key points corresponding to the nodes; the two-dimensional image content information comprises any one or more of the following items of information: color value, infrared brightness, transparency. Taking the node feature matrix X as an example of a dimension matrix of N × d, where N and d are natural numbers greater than 1, where N is the number of the preset face key points, and d is the dimension number of the key point information of each preset face key point, where the key point information includes: three-dimensional space coordinates and two-dimensional image content information. Specifically, for the present embodiment, N is equal to 104, d is equal to 6, that is, the keypoint information may be represented as (r, g, b, x, y, z), where r, g, b are color values of the face keypoint in the two-dimensional face image, and x, y, z are three-dimensional spatial coordinates of the face keypoint in the three-dimensional face image.
In other embodiments of the present application, the key point information may be represented as (r, g, b, a, x, y, z), where r, g, b are color values of the face key points in the two-dimensional face image, a is a transparency value of the face key points in the two-dimensional face image, and x, y, z are three-dimensional spatial coordinates of the face key points in the three-dimensional face image.
In other embodiments of the present application, if the two-dimensional face image is an infrared image, the key point information may be represented as (i, x, y, z), where i is an infrared brightness value of the face key point in the two-dimensional face image, and x, y, z is a three-dimensional spatial coordinate of the face key point in the three-dimensional face image.
And 140, extracting the graph characteristics based on the node characteristic matrix, the preset adjacent matrix and the unit matrix of the preset adjacent matrix through a pre-trained graph convolution neural network.
In some embodiments of the present application, the graph feature extraction is performed through a pre-trained graph convolution neural network based on the node feature matrix, the preset adjacency matrix, and the unit matrix of the preset adjacency matrix, and includes: and carrying out spatial information aggregation on the key point information in the node feature matrix of the undirected graph based on a preset adjacent matrix of the undirected graph and a unit matrix of the preset adjacent matrix through a pre-trained graph convolution neural network.
In an image recognition application, an input object is a picture, and the input object is a standard two-dimensional structure, wherein features are extracted by translating a convolution kernel on the picture and performing convolution. Since the picture has translation without deformation, the internal structure of a small window is the same no matter where the small window is moved to the picture. CNN can only process data in euclidean space. The structure of the graph is an irregular topological structure, can be regarded as data of an infinite dimension, has no translational invariance, the surrounding structure of each node is unique, and the data of the structure cannot be processed by both the CNN and the RNN (recurrent neural network). The graph convolution neural network (GCN) can extract features from graph data through a delicate design, and can use the features to perform node classification, graph classification, edge prediction, and the like on the data. When the method is specifically implemented, the face features in the undirected graph obtained based on the face image are extracted by adopting the graph convolution neural network.
The input data of the graph convolution neural network in the embodiment of the application comprises two parts: the node feature matrix and the adjacency matrix of the undirected graph (i.e., the aforementioned preset adjacency matrix). For example, the graph convolution neural network extracts features by forward propagation through equation 2 below:
Figure BDA0002630205020000111
in the above-mentioned formula 2, the first,
Figure BDA0002630205020000112
is the adjacency matrix A of the undirected graph G plusA sum matrix obtained from the unit matrix I of the adjacent matrix A;
Figure BDA0002630205020000113
is a degree matrix (degree matrix) with the calculation formula of
Figure BDA0002630205020000114
Figure BDA0002630205020000115
For diagonal matrices, the elements on the diagonal are corresponding nodes (e.g. matrix element A in adjacency matrix A)ijCorresponding node) degree + 1;
Figure BDA0002630205020000116
has the effect of
Figure BDA0002630205020000117
Normalization is performed due to the matrix
Figure BDA0002630205020000118
The method is not normalized, some unpredictable problems are easily generated in the calculation process, and the normalization is carried out
Figure BDA0002630205020000119
The values of the elements are all between (0 and 1); h(l)Input feature vector for hidden layer l, for input layer, H(l)Equal to the original node feature matrix X,
Figure BDA00026302050200001110
the role is to realize the aggregation of spatial information; w(l)The weight matrix of the first hidden layer is determined by pre-training the value of the weight matrix; σ is a nonlinear activation function.
In the embodiment of the application, the preset adjacent matrix is added with the identity matrix, so that the connection of one node to each node in the undirected graph is added. When forward propagation is performed according to formula 2, if the identity matrix is not added, when features are extracted through one layer of graph convolution, such as the face key points in fig. 2, the pupil corresponding node will only extract the data features of 8 nodes around the pupil, but ignore the data features of its own node. After the unit matrix is added, the graph convolution extracts the data characteristics of the neighborhood nodes and the pupil nodes.
The input of the 1 st hidden layer of the graph convolution neural network is an original node feature matrix X, the graph convolution neural network can aggregate information on a first-order neighborhood after each hidden layer processing, and l-order neighborhood features are fused for each node after l hidden layers. Weight matrix W(l)Has a dimension of Fl*Fl+1Namely, the second dimension of the weight matrix determines the feature number of the next layer, and the core network has several hidden layers, and the weights W of several layers are trained. The activation function may be, for example, Sigmoid, ReLu, etc.
In some embodiments of the present application, a 5-tier graph volume may be employed
The network structure of the cumulative neural network is GCN 5+ FC1+ FC2, wherein GCN is a graph convolution network, and FC1 and FC2 are full connection layers. After 5 layers of GCN and + FC1, an output vector of a specified dimension (for example 1024) can be acquired, and the output vector can be used as a face feature of a face corresponding to an input undirected graph.
In some embodiments of the present application, before the step of extracting the graph feature based on the node feature matrix, the preset adjacency matrix, and the unit matrix of the preset adjacency matrix through the pre-trained graph convolution neural network, as shown in fig. 5, the method further includes:
and step 100, training the graph convolution neural network.
Wherein training the graph convolutional neural network comprises: for a plurality of training faces, constructing an undirected graph corresponding to the training faces, wherein the undirected graph comprises a preset adjacency matrix and a node feature matrix of the corresponding training faces; for each training face, constructing a training sample corresponding to the training face by taking the sum matrix of the preset adjacency matrix and the unit matrix and the node feature matrix corresponding to the training face as sample data and the real value of the classification result of the training face as a sample label; and for each training sample, respectively extracting graph features based on the sum matrix and the node feature matrix through the graph convolutional neural network, calculating a classification result predicted value according to the features obtained by extracting the graph features, then calculating errors of the classification result predicted value and the classification result true value of all the training samples through a cross entropy loss function, and optimizing network parameters of the graph convolutional neural network by taking the error minimum as a target until the errors converge to meet preset conditions, thereby completing the training process of the graph convolutional neural network.
In the training phase of the graph-convolution neural network, a large number of training samples are constructed first. Each training sample corresponds to a two-dimensional face image and a three-dimensional face image which are synchronously acquired aiming at the same training face. The sample data of each training sample comprises: presetting an adjacency matrix, generating a node characteristic matrix according to the two-dimensional face image and the three-dimensional face image, and setting a sample label as a real value of a classification result of the training face. Taking a training sample generated by a face image including 5 ten thousand persons in a training sample set as an example, the sample label is used to represent the probability that the current training sample belongs to 5 ten thousand categories, and may be represented in a form such as "0001000 …".
The specific implementation of detecting the preset face key point according to the two-dimensional face image and the three-dimensional face image of the training face, determining the two-dimensional image content information in the two-dimensional face image, and determining the three-dimensional space coordinate of the preset face key point in the three-dimensional face image is described in the foregoing steps, and is not repeated here. For a specific implementation of constructing an undirected graph according to the three-dimensional space coordinates and the two-dimensional image content information, reference is made to the foregoing step description, which is not repeated herein. The adjacency matrix in the undirected graph constructed in the training stage is the same as the preset adjacency matrix in the recognition process, and the unit matrix of the adjacency matrix can be generated according to the adjacency matrix by adopting the prior art.
Taking the example that the graph convolution neural network comprises 5 graph convolution layers and 2 full-connection layers, in the training process, for each input training sample (namely, labeled undirected graph), the graph convolution neural network performs spatial information aggregation on the sum matrix of the preset adjacent matrix and the unit matrix and the node feature matrix in the input undirected graph through each graph convolution layer, and the 5 th graph convolution layer outputs an output vector with 1024 dimensions; then, sequentially performing feature mapping on the two full-connection layers to obtain a face classification result predicted value corresponding to the input undirected graph; then, calculating errors of the classification result predicted values and the classification result true values of all training samples through cross entropy loss to obtain model errors of the graph convolution neural network; and then, aiming at the minimum model error, adjusting network parameters by adopting a gradient descent method, and recalculating the training sample until the model error is converged to finish the training process of the graph convolution neural network.
In some embodiments of the present application, when the 1024-dimensional output vector output by the previous fully-connected layer is mapped to C (e.g., 5 ten thousand) classes through the last fully-connected layer, the classification probability may be calculated by softmax. The softmax function may be as shown in equation 3 below.
Figure BDA0002630205020000131
In the above formula 3, giP (i) is the probability that the output vector is mapped to the ith category, 1 ≦ k ≦ C.
For example, for 100 ten thousand training samples generated according to 100 ten thousand human face images of 5 ten thousand individuals (20 pairs of images per person, each pair of human face images including one two-dimensional human face image and one three-dimensional human face image), feature extraction is performed on 100 ten thousand training samples through a graph convolution layer and a first full connection layer of the graph convolution neural network respectively, feature vectors obtained after feature extraction are mapped to corresponding 5 ten thousand categories (one category corresponding to each person) through a last full connection layer respectively, and if the mapping accuracy exceeds a certain threshold (for example, 99%), it can be considered that the features extracted by the graph convolution layer and the first full connection layer of the graph convolution neural network are unique features of each person, and the network training is completed.
And 150, performing face recognition on the target face by using the features obtained by extracting the image features by the image convolution neural network as face features.
In some embodiments of the present application, a face feature library needs to be established in advance before face recognition is performed. The preset face feature library comprises a plurality of face feature libraries for registering faces; wherein each group of the face features in the face feature library is obtained by the following method: acquiring a two-dimensional face image and a three-dimensional face image which are synchronously acquired by different image acquisition devices aiming at a registered face; determining pixel positions and two-dimensional image content information of all preset face key points in the two-dimensional face image acquired aiming at the registered face, and determining three-dimensional space coordinates of all preset face key points in the three-dimensional face image acquired aiming at the registered face; constructing the undirected graph corresponding to the registered face according to the three-dimensional space coordinates and the two-dimensional image content information of each preset face key point of the registered face; extracting the image features based on a node feature matrix and a preset adjacency matrix of the undirected graph corresponding to the registered face and a unit matrix of the preset adjacency matrix through a pre-trained image convolution neural network; taking the feature obtained by extracting the graph feature of the graph convolution neural network as the face feature of the registered face; and storing the face characteristics of the registered face in the face characteristic database.
For a specific implementation of obtaining the two-dimensional face image and the three-dimensional face image of each registered face, refer to the specific implementation of obtaining the two-dimensional face image and the three-dimensional face image of the target face in the foregoing steps, which is not described herein again.
Determining pixel positions and two-dimensional image content information of preset face key points in the two-dimensional face image acquired aiming at the registered face, and determining a specific implementation mode of three-dimensional space coordinates of the preset face key points in the three-dimensional face image acquired aiming at the registered face, referring to a specific implementation mode of determining the two-dimensional image content information and the three-dimensional space coordinates in a recognition stage or a training stage, which is not repeated herein.
The specific implementation of constructing the undirected graph according to the two-dimensional image content information and the three-dimensional space coordinates is referred to in the specific implementation of constructing the undirected graph in the recognition stage or the training stage, which is not described herein again.
Then, for each registered face, inputting an undirected graph generated according to the image of the registered face into a graph convolution neural network trained in advance, extracting graph features of the input undirected graph by the graph convolution neural network, then, taking the features obtained by the graph convolution neural network after the graph feature extraction step as a group of face features of the registered face, and storing the group of face features of the registered face in the face feature library.
In the recognition stage, the preset hidden vector of the graph convolution neural network is used as a face feature to perform face recognition on the target face, and the face recognition method comprises the following steps: comparing the similarity of the features obtained by extracting the image features by the image convolution neural network with the face features in a preset face feature library; and determining a recognition result of face recognition of the target face according to the result of the similarity comparison.
In some embodiments of the present application, when similarity comparison is performed between the features extracted by the graph convolution neural network and the face features in the preset face feature library, a method for calculating cosine similarity (for example, the following formula 4) may be adopted to calculate a comparison score:
Figure BDA0002630205020000151
in the above formula 4, sim (x, y) represents the similarity between the face feature x and the face feature y,
Figure BDA0002630205020000152
in order to calculate the dot product, x is a modulus, the calculated similarity sim (x, y) is a number between (0-1), the closer the value of sim (x, y) is to 1, the greater the similarity between the human face feature x and the human face feature y is, the closer the value of sim (x, y) is to 0, and the human face isThe smaller the similarity of the feature x and the face feature y.
In some embodiments of the present application, the face features in the face feature library with the largest similarity to the face features extracted by the graph convolution neural network may be taken as successfully matched face features, and the related information of the registered face to which the face features in the face feature library belong, which are successfully matched, is taken as a face recognition result.
In other embodiments of the present application, other methods may also be used to calculate the similarity between the facial features extracted by the graph convolutional neural network and each group of facial features in the facial feature library, and a specific implementation manner of calculating the similarity between two groups of facial features is not limited in this embodiment of the present application.
In other embodiments of the present application, other methods may also be used to determine a face recognition result according to the similarity between the face features extracted by the graph convolution neural network and each group of face features in the face feature library, and the specific implementation manner of determining the face recognition result according to the similarity between the face features is not limited in the present application.
The embodiment of the application discloses a face recognition method, which comprises the steps of obtaining two-dimensional face images and three-dimensional face images which are synchronously collected by different image collection devices aiming at a target face; determining pixel positions and two-dimensional image content information of all preset face key points in the two-dimensional face image, and determining three-dimensional space coordinates of all preset face key points in the three-dimensional face image; constructing corresponding nodes according to the preset face key points, and constructing an undirected graph by expressing the adjacency relation of the nodes corresponding to the adjacent preset face key points in the preset face key points through undirected edges; the undirected graph comprises: a preset adjacency matrix representing the adjacency relation between the nodes and a node characteristic matrix formed by key point information stored by each node in the undirected graph; extracting graph features based on the node feature matrix, the preset adjacent matrix and the unit matrix of the preset adjacent matrix through a pre-trained graph convolution neural network; and performing face recognition on the target face by using the features obtained by extracting the image features by the image convolution neural network as the face features, which is favorable for improving the accuracy and reliability of face recognition.
Compared with the face recognition method in the prior art, the face recognition method disclosed by the embodiment of the application uses two-dimensional face features and three-dimensional face features at the same time in the comparison stage, namely, the texture information of the face is utilized, and the three-dimensional information of the face is utilized, so that the recognition precision can be improved, the face recognition method has strong advantages in the aspect of resisting the attack of printing photos and videos, and the safety and the reliability are higher.
The face recognition method disclosed by the embodiment of the application is more suitable for extracting the data features in the undirected graph constructed by the face key points, namely more suitable for extracting the position relation features among the face key points, by adopting the graph convolution neural network to replace a common convolution neural network.
Example two
Corresponding to the method embodiment, another embodiment of the present application discloses a face recognition apparatus, as shown in fig. 6, the apparatus includes:
a face image acquisition module 610, configured to acquire a two-dimensional face image and a three-dimensional face image that are synchronously acquired by different image acquisition devices for a target face;
a face key point information obtaining module 620, configured to determine pixel positions and two-dimensional image content information of preset face key points in the two-dimensional face image, and determine three-dimensional space coordinates of the preset face key points in the three-dimensional face image;
a face key point undirected graph constructing module 630, configured to construct corresponding nodes according to each of the preset face key points, and construct an undirected graph by using undirected edges to represent an adjacency relationship between the nodes in the preset face key points, which are adjacent to the preset face key points; the undirected graph comprises: a preset adjacency matrix representing the adjacency relation between the nodes and a node characteristic matrix formed by key point information stored by each node in the undirected graph;
a graph feature extraction module 640, configured to perform graph feature extraction based on the node feature matrix, the preset adjacency matrix, and a unit matrix of the preset adjacency matrix through a pre-trained graph convolution neural network;
and the face recognition module 650 is configured to perform face recognition on the target face by using features obtained by extracting the image features by the image convolution neural network as face features.
In some embodiments of the present application, the graph feature extraction module 640 is further configured to:
performing spatial information aggregation on the key point information in the node feature matrix of the undirected graph based on a preset adjacency matrix of the undirected graph and a unit matrix of the preset adjacency matrix through a pre-trained graph convolution neural network
In some embodiments of the present application, the key point information stored by each node includes: the three-dimensional space coordinates and the two-dimensional image content information of the preset human face key points corresponding to the nodes; the two-dimensional image content information comprises any one or more of the following items of information: color value, infrared brightness, transparency.
In some embodiments of the present application, as shown in fig. 7, the apparatus further comprises: a graph convolution neural network training module 600, the graph convolution neural network training module 600 to:
for a plurality of training faces, constructing an undirected graph corresponding to the training faces, wherein the undirected graph comprises a preset adjacency matrix and a node feature matrix of the corresponding training faces;
for each training face, constructing a training sample corresponding to the training face by taking the sum matrix of the preset adjacency matrix and the unit matrix and the node feature matrix corresponding to the training face as sample data and the real value of the classification result of the training face as a sample label;
and for each training sample, respectively extracting graph features based on the sum matrix and the node feature matrix through the graph convolutional neural network, calculating a classification result predicted value according to the features obtained by extracting the graph features, then calculating errors of the classification result predicted value and the classification result true value of all the training samples through a cross entropy loss function, and optimizing network parameters of the graph convolutional neural network by taking the error minimum as a target until the errors converge to meet preset conditions, thereby completing the training process of the graph convolutional neural network.
In some embodiments of the present application, the face recognition module 650 is further configured to:
comparing the similarity of the features obtained by extracting the image features by the image convolution neural network with the face features in a preset face feature library;
and determining a recognition result of face recognition of the target face according to the result of the similarity comparison.
In some embodiments of the present application, the preset face feature library includes a plurality of face feature libraries for registering faces; wherein each group of the face features in the face feature library is obtained by the following method:
acquiring a two-dimensional face image and a three-dimensional face image which are synchronously acquired by different image acquisition devices aiming at a registered face;
determining pixel positions and two-dimensional image content information of all preset face key points in the two-dimensional face image acquired aiming at the registered face, and determining three-dimensional space coordinates of all preset face key points in the three-dimensional face image acquired aiming at the registered face;
constructing the undirected graph corresponding to the registered face according to the three-dimensional space coordinates and the two-dimensional image content information of each preset face key point of the registered face;
extracting the image features based on a node feature matrix and a preset adjacency matrix of the undirected graph corresponding to the registered face and a unit matrix of the preset adjacency matrix through a pre-trained image convolution neural network;
taking the feature obtained by extracting the graph feature of the graph convolution neural network as the face feature of the registered face;
and storing the face characteristics of the registered face in the face characteristic database.
The face recognition device disclosed in this embodiment is used to implement the face recognition method described in the foregoing embodiment, and the specific implementation of each module of the device refers to corresponding steps in the method, which is not described in detail in this embodiment.
The face recognition device disclosed by the embodiment of the application acquires two-dimensional face images and three-dimensional face images which are synchronously acquired by different image acquisition devices aiming at a target face; determining pixel positions and two-dimensional image content information of all preset face key points in the two-dimensional face image, and determining three-dimensional space coordinates of all preset face key points in the three-dimensional face image; constructing corresponding nodes according to the preset face key points, and constructing an undirected graph by expressing the adjacency relation of the nodes corresponding to the adjacent preset face key points in the preset face key points through undirected edges; the undirected graph comprises: a preset adjacency matrix representing the adjacency relation between the nodes and a node characteristic matrix formed by key point information stored by each node in the undirected graph; extracting graph features based on the node feature matrix, the preset adjacent matrix and the unit matrix of the preset adjacent matrix through a pre-trained graph convolution neural network; and performing face recognition on the target face by using the features obtained by extracting the image features by the image convolution neural network as the face features, which is favorable for improving the accuracy and reliability of face recognition.
Compared with the face recognition method in the prior art, the face recognition device disclosed by the embodiment of the application uses two-dimensional face features and three-dimensional face features at the same time in the comparison stage, namely, the texture information of the face is utilized, and the three-dimensional information of the face is utilized, so that the recognition precision can be improved, the face recognition device also has strong advantages in the aspect of resisting the attack of printing photos and videos, and the safety and the reliability are higher.
The face recognition method disclosed by the embodiment of the application is more suitable for extracting the data features in the undirected graph constructed by the face key points, namely more suitable for extracting the position relation features among the face key points, by adopting the graph convolution neural network to replace a common convolution neural network.
Correspondingly, the application also discloses an electronic device, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor executes the computer program to realize the face recognition method according to the first embodiment of the application. The electronic device can be a PC, a mobile terminal, a personal digital assistant, a tablet computer and the like.
The present application also discloses a computer-readable storage medium, on which a computer program is stored, which when executed by a processor, implements the steps of the face recognition method according to the first embodiment of the present application.
The embodiments in the present specification are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is simple, and for the relevant points, refer to the partial description of the method embodiment.
The face recognition method and device provided by the present application are introduced in detail, and a specific example is applied in the text to explain the principle and the implementation of the present application, and the description of the above embodiment is only used to help understanding the method and the core idea of the present application; meanwhile, for a person skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.

Claims (10)

1. A face recognition method, comprising:
acquiring two-dimensional face images and three-dimensional face images which are synchronously acquired by different image acquisition devices aiming at a target face;
determining pixel positions and two-dimensional image content information of all preset face key points in the two-dimensional face image, and determining three-dimensional space coordinates of all preset face key points in the three-dimensional face image;
constructing corresponding nodes according to the preset face key points, and constructing an undirected graph by expressing the adjacency relation of the nodes corresponding to the adjacent preset face key points in the preset face key points through undirected edges; the undirected graph comprises: a preset adjacency matrix representing the adjacency relation between the nodes and a node characteristic matrix formed by key point information stored by each node in the undirected graph;
extracting graph features based on the node feature matrix, the preset adjacent matrix and the unit matrix of the preset adjacent matrix through a pre-trained graph convolution neural network;
and performing face recognition on the target face by using the features obtained by extracting the image features by the image convolution neural network as face features.
2. The method according to claim 1, wherein the step of performing graph feature extraction based on the node feature matrix, the preset adjacency matrix, and the identity matrix of the preset adjacency matrix through a pre-trained graph convolution neural network comprises:
and carrying out spatial information aggregation on the key point information in the node feature matrix of the undirected graph based on a preset adjacent matrix of the undirected graph and a unit matrix of the preset adjacent matrix through a pre-trained graph convolution neural network.
3. The method of claim 1, wherein the keypoint information stored by each of the nodes comprises: the three-dimensional space coordinates and the two-dimensional image content information of the preset human face key points corresponding to the nodes; the two-dimensional image content information comprises any one or more of the following items of information: color value, infrared brightness, transparency.
4. The method according to any one of claims 1 to 3, wherein before the step of performing graph feature extraction based on the node feature matrix, the preset adjacency matrix, and the identity matrix of the preset adjacency matrix through a pre-trained graph convolution neural network, the method comprises:
for a plurality of training faces, constructing an undirected graph corresponding to the training faces, wherein the undirected graph comprises a preset adjacency matrix and a node feature matrix of the corresponding training faces;
for each training face, constructing a training sample corresponding to the training face by taking the sum matrix of the preset adjacency matrix and the unit matrix and the node feature matrix corresponding to the training face as sample data and the real value of the classification result of the training face as a sample label;
and for each training sample, respectively extracting graph features based on the sum matrix and the node feature matrix through the graph convolutional neural network, calculating a classification result predicted value according to the features obtained by extracting the graph features, then calculating errors of the classification result predicted value and the classification result true value of all the training samples through a cross entropy loss function, and optimizing network parameters of the graph convolutional neural network by taking the error minimum as a target until the errors converge to meet preset conditions, thereby completing the training process of the graph convolutional neural network.
5. The method according to any one of claims 1 to 3, wherein the step of performing face recognition on the target face by using the feature obtained by performing the image feature extraction on the image convolutional neural network as a face feature comprises:
comparing the similarity of the features obtained by extracting the image features by the image convolution neural network with the face features in a preset face feature library;
and determining a recognition result of face recognition of the target face according to the result of the similarity comparison.
6. The method of claim 5, wherein the predetermined facial feature library comprises a facial feature library of a plurality of registered faces; wherein each group of the face features in the face feature library is obtained by the following method:
acquiring a two-dimensional face image and a three-dimensional face image which are synchronously acquired by different image acquisition devices aiming at a registered face;
determining pixel positions and two-dimensional image content information of all preset face key points in the two-dimensional face image acquired aiming at the registered face, and determining three-dimensional space coordinates of all preset face key points in the three-dimensional face image acquired aiming at the registered face;
constructing the undirected graph corresponding to the registered face according to the three-dimensional space coordinates and the two-dimensional image content information of each preset face key point of the registered face;
extracting the image features based on a node feature matrix and a preset adjacency matrix of the undirected graph corresponding to the registered face and a unit matrix of the preset adjacency matrix through a pre-trained image convolution neural network;
taking the feature obtained by extracting the graph feature of the graph convolution neural network as the face feature of the registered face;
and storing the face characteristics of the registered face in the face characteristic database.
7. A face recognition apparatus, comprising:
the face image acquisition module is used for acquiring a two-dimensional face image and a three-dimensional face image which are synchronously acquired by different image acquisition devices aiming at a target face;
the face key point information acquisition module is used for determining the pixel position and the two-dimensional image content information of each preset face key point in the two-dimensional face image and determining the three-dimensional space coordinate of each preset face key point in the three-dimensional face image;
the face key point undirected graph construction module is used for constructing corresponding nodes according to the preset face key points, expressing the adjacency relation of the nodes corresponding to the adjacent preset face key points in the preset face key points through undirected edges and constructing an undirected graph; the undirected graph comprises: a preset adjacency matrix representing the adjacency relation between the nodes and a node characteristic matrix formed by key point information stored by each node in the undirected graph;
the graph feature extraction module is used for extracting the graph features based on the node feature matrix, the preset adjacent matrix and the unit matrix of the preset adjacent matrix through a pre-trained graph convolution neural network;
and the face recognition module is used for performing face recognition on the target face by taking the feature obtained by extracting the image feature by the image convolution neural network as the face feature.
8. The apparatus of claim 7, wherein the graph feature extraction module is further configured to:
and carrying out spatial information aggregation on the key point information in the node feature matrix of the undirected graph based on a preset adjacent matrix of the undirected graph and a unit matrix of the preset adjacent matrix through a pre-trained graph convolution neural network.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the face recognition method of any one of claims 1 to 6 when executing the computer program.
10. A computer-readable storage medium, on which a computer program is stored, which program, when being executed by a processor, is adapted to carry out the steps of the face recognition method of any one of claims 1 to 6.
CN202010808939.6A 2020-08-12 2020-08-12 Face recognition method and device, electronic equipment and storage medium Active CN111783748B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010808939.6A CN111783748B (en) 2020-08-12 2020-08-12 Face recognition method and device, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010808939.6A CN111783748B (en) 2020-08-12 2020-08-12 Face recognition method and device, electronic equipment and storage medium

Publications (2)

Publication Number Publication Date
CN111783748A true CN111783748A (en) 2020-10-16
CN111783748B CN111783748B (en) 2023-07-14

Family

ID=72761928

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010808939.6A Active CN111783748B (en) 2020-08-12 2020-08-12 Face recognition method and device, electronic equipment and storage medium

Country Status (1)

Country Link
CN (1) CN111783748B (en)

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112329736A (en) * 2020-11-30 2021-02-05 姜召英 Face recognition method and financial system
CN112541575A (en) * 2020-12-06 2021-03-23 支付宝(杭州)信息技术有限公司 Method and device for training graph neural network
CN112733807A (en) * 2021-02-22 2021-04-30 佳都新太科技股份有限公司 Face comparison graph convolution neural network training method and device
CN112784709A (en) * 2021-01-06 2021-05-11 华南理工大学 Remote multi-target efficient detection and identification method
CN113158908A (en) * 2021-04-25 2021-07-23 北京华捷艾米科技有限公司 Face recognition method and device, storage medium and electronic equipment
CN113537194A (en) * 2021-07-15 2021-10-22 Oppo广东移动通信有限公司 Illumination estimation method, illumination estimation device, storage medium, and electronic apparatus
CN113673610A (en) * 2021-08-25 2021-11-19 上海鹏冠生物医药科技有限公司 Image preprocessing method for tissue cell pathological image diagnosis system
CN116524581A (en) * 2023-07-05 2023-08-01 南昌虚拟现实研究院股份有限公司 Human eye image facula classification method, system, equipment and storage medium

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110659723A (en) * 2019-09-03 2020-01-07 腾讯科技（深圳）有限公司 Data processing method, device, medium and electronic equipment based on artificial intelligence
WO2020037937A1 (en) * 2018-08-20 2020-02-27 深圳壹账通智能科技有限公司 Facial recognition method and apparatus, terminal, and computer readable storage medium
CN111079587A (en) * 2019-12-03 2020-04-28 北京迈格威科技有限公司 Face recognition method and device, computer equipment and readable storage medium
CN111243093A (en) * 2020-01-07 2020-06-05 腾讯科技（深圳）有限公司 Three-dimensional face grid generation method, device, equipment and storage medium
CN111259875A (en) * 2020-05-06 2020-06-09 中国人民解放军国防科技大学 Lip reading method based on self-adaptive magnetic space-time diagramm volumetric network

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020037937A1 (en) * 2018-08-20 2020-02-27 深圳壹账通智能科技有限公司 Facial recognition method and apparatus, terminal, and computer readable storage medium
CN110659723A (en) * 2019-09-03 2020-01-07 腾讯科技（深圳）有限公司 Data processing method, device, medium and electronic equipment based on artificial intelligence
CN111079587A (en) * 2019-12-03 2020-04-28 北京迈格威科技有限公司 Face recognition method and device, computer equipment and readable storage medium
CN111243093A (en) * 2020-01-07 2020-06-05 腾讯科技（深圳）有限公司 Three-dimensional face grid generation method, device, equipment and storage medium
CN111259875A (en) * 2020-05-06 2020-06-09 中国人民解放军国防科技大学 Lip reading method based on self-adaptive magnetic space-time diagramm volumetric network

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
ZUHENG MING等: "Dynamic Deep Multi-task Learning for Caricature-Visual Face Recognition", 2019 INTERNATIONAL CONFERENCE ON DOCUMENT ANALYSIS AND RECOGNITION WORKSHOPS (ICDARW) *
李睿;李科;孙家炜;: "基于二维纹理重建三维人脸深度图像后的人脸识别", 现代计算机(专业版), no. 10 *
蝈蝈: "何时能懂你的心——图卷积神经网络（GCN）", 《知乎》 *
蝈蝈: "何时能懂你的心——图卷积神经网络（GCN）", 《知乎》, 28 June 2019 (2019-06-28) *

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112329736A (en) * 2020-11-30 2021-02-05 姜召英 Face recognition method and financial system
CN112329736B (en) * 2020-11-30 2022-04-12 上海华瑞银行股份有限公司 Face recognition method and financial system
CN112541575A (en) * 2020-12-06 2021-03-23 支付宝(杭州)信息技术有限公司 Method and device for training graph neural network
CN112541575B (en) * 2020-12-06 2023-03-10 支付宝(杭州)信息技术有限公司 Method and device for training graph neural network
CN112784709A (en) * 2021-01-06 2021-05-11 华南理工大学 Remote multi-target efficient detection and identification method
CN112733807A (en) * 2021-02-22 2021-04-30 佳都新太科技股份有限公司 Face comparison graph convolution neural network training method and device
CN113158908A (en) * 2021-04-25 2021-07-23 北京华捷艾米科技有限公司 Face recognition method and device, storage medium and electronic equipment
CN113537194A (en) * 2021-07-15 2021-10-22 Oppo广东移动通信有限公司 Illumination estimation method, illumination estimation device, storage medium, and electronic apparatus
CN113673610A (en) * 2021-08-25 2021-11-19 上海鹏冠生物医药科技有限公司 Image preprocessing method for tissue cell pathological image diagnosis system
CN116524581A (en) * 2023-07-05 2023-08-01 南昌虚拟现实研究院股份有限公司 Human eye image facula classification method, system, equipment and storage medium
CN116524581B (en) * 2023-07-05 2023-09-12 南昌虚拟现实研究院股份有限公司 Human eye image facula classification method, system, equipment and storage medium

Also Published As

Publication number Publication date
CN111783748B (en) 2023-07-14

Similar Documents

Publication Publication Date Title
CN111783748B (en) Face recognition method and device, electronic equipment and storage medium
CN111709409B (en) Face living body detection method, device, equipment and medium
EP4075324A1 (en) Face recognition method and face recognition device
WO2019227479A1 (en) Method and apparatus for generating face rotation image
WO2020103700A1 (en) Image recognition method based on micro facial expressions, apparatus and related device
Han et al. Face recognition with contrastive convolution
KR101901591B1 (en) Face recognition apparatus and control method for the same
CN112801015B (en) Multi-mode face recognition method based on attention mechanism
JP4743823B2 (en) Image processing apparatus, imaging apparatus, and image processing method
CN111444881A (en) Fake face video detection method and device
CN111310731A (en) Video recommendation method, device and equipment based on artificial intelligence and storage medium
CN111476806B (en) Image processing method, image processing device, computer equipment and storage medium
KR20210025020A (en) Face image recognition using pseudo images
Haque et al. Two-handed bangla sign language recognition using principal component analysis (PCA) and KNN algorithm
WO2021218238A1 (en) Image processing method and image processing apparatus
CN110222718A (en) The method and device of image procossing
CN113610046B (en) Behavior recognition method based on depth video linkage characteristics
CN110796100A (en) Gait recognition method and device, terminal and storage device
CN112036260A (en) Expression recognition method and system for multi-scale sub-block aggregation in natural environment
CN111639580A (en) Gait recognition method combining feature separation model and visual angle conversion model
CN115205933A (en) Facial expression recognition method, device, equipment and readable storage medium
Satybaldina et al. Deep learning based static hand gesture recognition
CN114170690A (en) Method and device for living body identification and construction of living body identification model
CN112906520A (en) Gesture coding-based action recognition method and device
CN112418250A (en) Optimized matching method for complex 3D point cloud

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant