CN115995121A - Multi-mode biological identification method based on attention module - Google Patents
Multi-mode biological identification method based on attention module Download PDFInfo
- Publication number
- CN115995121A CN115995121A CN202211367427.6A CN202211367427A CN115995121A CN 115995121 A CN115995121 A CN 115995121A CN 202211367427 A CN202211367427 A CN 202211367427A CN 115995121 A CN115995121 A CN 115995121A
- Authority
- CN
- China
- Prior art keywords
- training
- fingerprint
- mode
- finger vein
- biological
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Collating Specific Patterns (AREA)
Abstract
The invention relates to a multi-mode biological identification method based on an attention module, belonging to the field of computer biological identification. The method comprises a data acquisition and preprocessing stage, a neural network model training and verification stage, an attribute module feature fusion stage and a deployment application stage. The method has the advantages that the multi-mode biological characteristics are adopted for identity recognition, the problems of insufficient biological characteristics and unsafe single-mode biological characteristics in a single mode are solved, the multi-mode biological characteristics are mutually influenced, the extraction of the more comprehensive and effective biological characteristics is facilitated, an arclos function is adopted as a loss function in the extraction method, compared with a softmax loss function, the model recognition rate is improved, the method has better characteristic fusion performance based on the idea of an attention mechanism in the characteristic layer fusion process, meanwhile, compared with the traditional method for learning a weight for the characteristics of each mode, the characteristics are weighted and cascaded, and therefore the characteristics of each mode can play a role better.
Description
Technical Field
The invention belongs to the field of computer biological recognition, and particularly relates to a multi-mode biological recognition method based on an attention module.
Background
At present, the biological recognition technology mainly uses a single mode as a system recognition basis, but is subject to external factors and individual differences, and the single mode biological characteristics are used as the recognition basis, so that certain limitations exist. Such as: because of factors such as light, the face recognition accuracy is affected; the abrasion of fingerprints and perspiration of fingers during long-term working can affect the fingerprint identification effect. Meanwhile, if the single-mode biological characteristics are revealed, the privacy of the user is permanently damaged, and the multi-mode biological identification technology can greatly strengthen the identification safety. Thus, multi-modal biometric technology is a new research direction and has been rapidly developed. The multi-mode identification scheme performs information fusion according to the characteristics among different features, so as to achieve the purpose of complementary advantages. The multi-mode system has more excellent robustness and safety, and provides a better selection scheme for identity authentication. In 1995, brunelli et al proposed using face and voice fusion feature information for identity recognition, and feature fusion was performed at the decision layer to demonstrate the feasibility and effectiveness of a multi-modal recognition system. In 2013, shekhar et al combined sparse linear combination method represents the characteristic information of human face, iris and fingerprint, and discusses the correlation and coupling between different characteristics in the fusion process. In 2016, wild et al studied the problem of dynamic detection and suppression of outliers by active recognition of spoofed samples under a multi-modal face and fingerprint fusion structure, and studied its inherent normalization.
The existing multi-mode biological recognition is mainly focused on aspects of faces, fingerprints, voiceprints and the like, and researches on the multi-mode biological recognition of fingers are few. The finger has rich characteristic information, such as biological information of fingerprints, knuckle veins, finger veins and the like, is convenient to collect, and is suitable for being used as a multi-mode characteristic fusion research object. Fingerprint identification technology has the earliest development, the most mature algorithm and the most widely applied market. The fingerprint identification technology utilizes the combination of unique ridge lines and valley lines on the skin of the front surface of the tail end of the finger to identify. In the traditional fingerprint identification method, fingerprint image registration and splicing can be realized based on a deep iteration nearest point algorithm (Iterative Closest Point, ICP), and the method based on sequential image splicing can be used for calculating the offset of the sliding direction under the condition of non-uniform speed sliding of images. In the learning-based method, the variability among the global direction modes is learned through a training set, and the most appropriate mode in the training set is used for replacing the fingerprint area, so that the noise influence can be effectively exerted.
The finger vein belongs to subcutaneous tissue, and has the advantages of high anti-counterfeiting performance, high recognition rate, high user friendliness and the like compared with other biological feature recognition. At present, a finger vein enhancement algorithm based on a Gabor filter, a scattering model and self-adaptive curve transformation is used for solving the problem of uneven illumination of finger vein images by combining the Gabor filter with the Weber theorem. And extracting features of the finger vein section by using a feature descriptor extraction palm print and a finger vein recognition algorithm, and classifying feature vectors formed by the extracted features by using a SIFT algorithm and a SURF algorithm.
The knuckle lines refer to a curved muscle line area or texture area on the finger knuckle, and have rich texture features and line features. The knuckle pattern image can be obtained through the low-resolution camera, and the acquisition is convenient and the cost is low. These advantages make knuckle lines ideal biometric features, and knuckle feature-based identification techniques have evolved rapidly in recent years. The knuckle pattern recognition algorithm based on the multichannel Gabor filter achieves a good recognition effect by reducing the dimension of the extracted features by using a Principal Component Analysis (PCA) and a Linear Discriminant Analysis (LDA).
Biometric acquisition is also one of the important steps in multi-modal biometric identification. Biometric acquisition is a pattern recognition system. First, the system obtains biometric data of an individual and then extracts a feature set from the data. By comparing two different sets of biometric characteristics, the system can obtain a determination of whether the two are from the same person. Thus, in general, an automated biometric identification system can be thought of as being made up of 1) an acquisition module; 2) And 3) a characteristic extraction module and a matching module. In addition, the system includes a database for accessing and managing feature sets.
How to collect multi-modal biological characteristics has also been studied in recent years by the academia. Yang Jinfeng et al have completed the design of a multi-modal biometric acquisition system of 3 modes of finger veins, knuckle prints and fingerprints in 2016, and further designed a new finger bimodal image acquisition system in 2019, and integrated finger veins and knuckle prints ROI positioning algorithm. The currently researched multi-mode biological characteristic acquisition system can be applied to an intelligent coded lock.
The existing multi-mode biological recognition mode mainly uses a decision layer fusion method, and the feature layer fusion method has the problems that the data volume is large, the training volume is large, the difference among multi-mode features is large, the research is less, the finger vein is an emerging finger feature, the feature extraction mode is less, and how to extract effective features and perform multi-mode feature fusion is a difficult point.
In the existing multi-mode fusion method, the methods such as direct feature splicing identification, correlation analysis and the like are available, the limitation is large when the methods face more complex feature correlation, and how to find a more effective feature fusion method is a difficulty of multi-mode biological identification.
Disclosure of Invention
The invention provides a multi-mode biological identification method based on an identification module, which aims to solve the problems of insufficient biological characteristics on a single mode and unsafe single-mode biological characteristics in identification.
The technical scheme adopted by the invention is that the method comprises the following steps:
(1) Collecting data, namely collecting biological characteristics of three modes including fingerprint, finger vein and knuckle vein by using a biological characteristic collector;
(2) Preprocessing the acquired data of the biological characteristics of the three modes, so that the data accords with the input requirements of the models corresponding to the three modes;
(3) In the training and verification stage of the neural network model, feature extraction is carried out on the data of the multiple modes preprocessed in the step (2) through ResNet and DenseNet, and parameters of the ResNet and the DenseNet are obtained through training and verification;
(4) The characteristic fusion of the attribute module is carried out, the extracted characteristic vectors of the three modes are cascaded to form a total characteristic vector, then the total characteristic vector is sent to a classifier for characteristic classification, an attribute mechanism is adopted, firstly, a Q, K, V matrix is initialized according to the characteristic vector formed by the three mode cascade, wherein K is a key value key in the calculation of the attribute mechanism, Q represents a query, K is a key value key, then, according to the Q, K and V obtained by initialization, the optimal weight is obtained by training and learning, the weighting summation is calculated, the combined characteristic vector is obtained, and the combined characteristic vector is sent to the classifier for classification, so that a better recognition effect is achieved;
(5) And the deployment application is used for deploying and applying the trained algorithm to the intelligent coded lock, so that the visitor provides the required biological characteristics, and the intelligent coded lock identifies whether the user is a valid user or not according to the data in the database.
The data acquisition in the step (1) comprises the following steps: collecting a fingerprint image data set A, a finger vein image data set B and a knuckle image data set C of a user by using a biological characteristic collector;
the biological characteristic collector is a biological characteristic collector module of the intelligent coded lock, and the intelligent coded lock comprises a main board, a clutch, the biological characteristic collector, a memory and a microprocessor.
The data preprocessing in the step (2) is to preprocess the collected biological image data sets of three modes, and the preprocessing steps are as follows:
1) Automatically labeling the acquired biological images to obtain the numbers of volunteers and the numbers of fingers corresponding to each biological image, and dividing the biological images into a training set and a testing set according to the proportion;
2) Performing image preprocessing on the acquired biological image, subtracting the average value of pixels of each image from the biological image, enhancing by using standard colors, and then normalizing in batches;
3) Enhancing the image; respectively enhancing the input fingerprint image, finger vein image and knuckle pattern image by using Sobel operator, and then positioning and intercepting by using ROI to divide effective fingerprint, finger vein and knuckle pattern data set and training set, wherein the effective fingerprint training set is A 1 The effective fingerprint test set is A 2 The effective finger vein training set is B 1 The effective finger vein test set is B 1 The effective knuckle line training set is C 1 The effective knuckle test set is C 2 。
The specific steps in the step (3) are as follows:
1) And (3) a model training stage, wherein feature extraction is carried out on the data of the multiple modes preprocessed in the step (2) through ResNet and DenseNet, and the fingerprint training set A is respectively carried out 1 Training set B of finger vein 1 The two networks of input ResNet34, resNet50 train independently:
(1) fingerprint training set A is loaded by using dataloaders respectively 1 And finger vein training set B 1 Storing the fingerprint and the finger vein training database in a tensor mode;
(2) establishing ResNet34 and ResNet50 networks, wherein ResNet34 and ResNet50 respectively comprise 4 residual blocks, initializing weights of the networks respectively, selecting an arclos function as a loss function, defining an optimizer as an Adam optimizer, and setting the learning rate of the optimizer;
(3) network training is carried out on the ResNet34 and the ResNet50 from tensors of training databases of fingerprints and finger veins loaded in the dataload, gradient descent is continuously carried out, iterative training is carried out for multiple times, error and loss between positive examples and negative examples are calculated by using a loss function, back propagation of gradients is carried out on parameters in the two networks respectively, and the flow is repeated and the loss is dynamically updated;
(4) mapping the fingerprints to be matched and the finger vein images into the probability of each category through an arclose function, and completing final single-mode feature matching;
2) Inputting the images of the knuckle training set into the DenseNet121 network for training independently:
(1) respectively loading a fingerprint training set and a finger vein training set by using a datalink, and storing the fingerprint training set and the finger vein training set in a tensor mode into a fingerprint training database and a finger vein training database;
(2) constructing a DenseNet121 network, wherein the DenseNet121 network comprises three DenseBlock and a Transition Block, initializing the weight of the network, selecting an arclos function as a loss function, defining an optimizer as an Adam optimizer, and setting the learning rate of the optimizer;
(3) starting network training on DenseNet121 by tensors of training databases of fingerprints and finger veins loaded in datalink, continuously performing gradient descent, performing repeated iterative training, calculating errors and losses between positive examples and negative examples by using a loss function, respectively performing gradient counter-propagation on parameters in the two networks, repeating the process, and dynamically updating the loss;
(4) mapping the fingerprints to be matched and the finger vein images into the probability of each category through an arclose function, and completing final single-mode feature matching;
3) Model verification: and respectively verifying the trained ResNet34, resNet50 and DenseNet121 networks:
(1) fingerprint test set A is loaded by using dataloaders respectively 2 Finger vein test set B 2 Knuckle test set C 2 Storing the fingerprint, finger vein and knuckle test data set in tensor mode;
(2) and (3) starting to perform network verification on the ResNet34, the ResNet50 and the DenseNet121 respectively by using tensors of verification databases of fingerprints, finger veins and knuckle veins loaded in the dataload, and finally outputting a label with the highest probability in a matching result, namely identity information corresponding to the image, and continuously adjusting parameters through previous training until the model converges.
The invention in step (4) carries out the attribute feature fusion, which comprises the following steps:
1) The training steps of the attention module are as follows:
firstly, respectively extracting fingerprint training sets A according to the single-mode feature extraction model adjusted in the step (3) 1 Fingerprint training set B 1 Fingerprint test set C 1 Respectively initializing and endowing the extracted data features of a plurality of modes with a weight to form an initial mode feature weight matrix;
(1) the extracted feature vectors are subjected to a dot product self-attention calculation to obtain dynamically weighted vectors, which are available here according to the calculation formula of self-attention, in particular, here the activation function selects relu 2 As an activation function;
(2) the concept of multiple heads is introduced in the intent model to allow the model to learn H different representation subspaces at each location while maintaining the same computational efficiency, typically expressed as inputs with parameterization;
(3) training the model by using a batch with the size of 32, and filling all the features into the longest sequence in the batch for the consistency of the same batch in the data processing process; because the dimensions of the modal features are different, they need to be mapped into the internal space of the model, the dimension of the internal space is 1024 dimensions, the model is based on a transducer, N is 2 layers, and H is 4 heads; the output dimension of the final generator is consistent with the vector size of the descriptive word and is 10172 dimension; gamma=0.7, loss probability p=0.1, and learning rate 10 were used in final loss calculation tag smoothing -5 Training the description generator by Adam optimizer;
obtaining the optimal weight of the data characteristic of each mode through model training and learning; and carrying out weighted fusion on the data characteristics of all modes: multiplying the weight of each mode by the data feature of the corresponding mode, cascading feature vectors obtained by all modes, wherein the feature vectors obtained by all modes are fingerprint feature vectors obtained by a ResNet model, finger vein feature vectors and finger knuckle feature vectors obtained by a DenseNet model, so as to form a joint feature vector, and finally, sending the joint feature vector to a classifier for classification;
2) The authentication module verifies the following steps:
(1) respectively loading a fingerprint training set and a finger vein verification set by using a datalink, and storing the fingerprint training set and the finger vein verification set in a tensor mode into a fingerprint and finger vein verification database;
(2) the tensor of the verification database of the fingerprint and the finger vein loaded in the dataload is firstly subjected to feature extraction through the ResNet34 and the ResNet50, then subjected to weight configuration through the attention module, then subjected to feature fusion, finally output the label with the highest probability in the matching result, namely the identity information corresponding to the image, and parameters are continuously adjusted through previous training until the model converges.
In the step (5), the fusion algorithm based on the attention module obtained through training and verification is stored in a memory of the intelligent coded lock, and when a visitor verifies, the intelligent coded lock can obtain a recognition result according to the biological characteristic information of the visitor and the trained algorithm.
The invention has the following advantages:
1. the invention adopts the multi-mode biological characteristics to carry out the identity recognition, overcomes the problems of insufficient biological characteristics and unsafe single-mode biological characteristics in a single mode, and is beneficial to extracting more comprehensive and effective biological characteristics by mutual influence among the multi-modes.
2. In the method for extracting the single-mode characteristics based on the fingerprint, the finger vein and the knuckle vein, the arclos function is adopted as the loss function, so that the model recognition rate is improved compared with the softmax loss function.
3. In the process based on feature layer fusion, the concept of an attribute mechanism is adopted, so that the feature fusion performance is better than that of simple feature splicing, and meanwhile, the traditional softmax activation function is replaced by relu according to the traditional attribute mechanism 2 Activating the function, simultaneously as compared to the transferThe system learns a weight for the features of each mode, and then performs weighted cascade connection on the features, so that the features of each mode can better play a role.
Drawings
FIG. 1 is a flow chart of the present invention;
FIG. 2 is a schematic block diagram of an intelligent coded lock of the present invention;
FIG. 3 is a flow chart of step (3) of the present invention;
FIG. 4 is a flow chart of step (4) of the present invention.
Detailed Description
Referring to fig. 1, the steps are as follows:
1. collecting data, namely collecting biological characteristics of three modes including fingerprint, finger vein and knuckle vein by using a biological characteristic collector;
2. preprocessing data, namely preprocessing the acquired images of biological characteristics of three modes to enable the images to meet the input requirements of models corresponding to the three modes;
3. training and verifying the neural network model, extracting characteristics of the data of the multiple modes preprocessed in the step (2) through ResNet and DenseNet, and obtaining parameters of the ResNet and the DenseNet through training and verifying;
and 4. Feature fusion of the attribute module, namely cascading the extracted feature vectors of the three modes to form a total feature vector, then sending the total feature vector to a classifier for feature classification, and initializing a Q, K, V matrix by adopting an attribute mechanism according to the feature vector formed by cascading the three modes, wherein K is a key value key in the calculation of the attribute mechanism, Q represents a query, and K is the key value key. Then according to Q, K and V obtained by initialization, calculating weighted summation to obtain optimal weight through training and learning to obtain a combined feature vector, and sending the combined feature vector to a classifier for classification, so that a better recognition effect is achieved;
5. the deployment application is used for deploying and applying the trained algorithm to the intelligent coded lock, the visitor provides the required biological characteristics, and the intelligent coded lock identifies whether the user is a valid user or not according to the data in the database.
The data acquisition stage of the fingerprint, finger vein and knuckle data in the step 1 comprises the following steps:
the fingerprint image data set A, the finger vein image data set B and the knuckle vein image data set C of a user are acquired by using a biological characteristic acquisition device, wherein the biological characteristic acquisition device is a biological characteristic acquisition device module of an intelligent coded lock, and the intelligent coded lock is shown in a figure 2 and comprises: the device comprises a main board, a clutch, a biological characteristic collector, a memory and a microprocessor.
In the step 2, in the data preprocessing stage, preprocessing is performed on the collected data of the biological characteristics of the three modes, so that the data accords with the input requirements of the models corresponding to the three modes, and the preprocessing steps are as follows:
(1) Automatically labeling the acquired biological images to obtain a volunteer number and a finger number corresponding to each biological image, and dividing the biological images into a training set and a testing set according to the ratio of 4:1;
(2) Performing image preprocessing on the acquired biological image, subtracting the average value of pixels of each image from the biological image, enhancing by using standard colors, and normalizing in batches;
(3) Enhancing the image; and respectively enhancing the input fingerprint image, finger vein image and finger joint vein image by using a Sobel operator, and then positioning and intercepting by using the ROI to divide the effective fingerprint, finger vein and finger joint vein image. The formula of the Sobel operator is as follows:
wherein A represents an original image before processing, gx and Gy represent images obtained through transverse and longitudinal edge detection respectively; the lateral and longitudinal gradient approximation G for each pixel in the original image can be expressed as:
the gradient direction θ is:
wherein the effective fingerprint training set is A 1 The effective fingerprint test set is A 2 The effective finger vein training set is B 1 The effective finger vein test set is B 1 The effective knuckle line training set is C 1 The effective knuckle test set is C 2 。
In the step 3, feature extraction is performed on the data of the plurality of modes preprocessed in the step 2 through ResNet and DenseNet, and the feature extraction step is shown in FIG. 3. Specifically, the training and verification phase of the neural network model includes:
3.1ResNet network feature extraction and training, namely respectively training the fingerprint training set A 1 Finger vein training set B 1 Inputting ResNet34 and ResNet50 networks to perform feature extraction and training, and determining network model parameters;
(1) Respectively loading a fingerprint training set and a finger vein training set by using a datalink, and storing the fingerprint training set and the finger vein training set in a tensor mode into a fingerprint training database and a finger vein training database;
(2) The method comprises the steps of constructing ResNet34 and ResNet50 networks, wherein ResNet34 and ResNet50 each comprise 4 residual blocks, initializing weights of the networks respectively, and selecting an arclose function as a loss function, wherein an arclose function formula is as follows:
wherein N is the number of samples, i represents the ith sample, s is the radius, m is the angular interval between two classes, and θ is the included angle between the parameter w and the feature vector x;
defining the optimizer as an Adam optimizer, and setting the learning rate of the optimizer, wherein the learning rate is set to be 10 -4 ；
(3) Network training is carried out on the ResNet34 and the ResNet50 respectively from tensors of training databases of fingerprints and finger veins loaded in the dataload, gradient descent is carried out according to an Adam optimizer, 5 times of iterative training are carried out in total, after all data in a training set are subjected to one time of network training, one time of iterative training is shown to be completed, error and loss between a positive example and a negative example are calculated by using a loss function, gradient back propagation is carried out on parameters in the two networks respectively, and the flow is repeated and the loss quantity is dynamically updated;
(4) Mapping the fingerprints to be matched and the finger vein images into the probability of each category through an arclose function, and completing final single-mode feature matching;
3.2DenseNet network feature extraction and training: inputting the knuckle training database into the DenseNet121 network for training independently:
(1) Loading a knuckle print training set C with a dataluader 1 Storing the fingerprint and the finger vein training database in a tensor mode;
(2) Constructing a DenseNet121 network, wherein the DenseNet121 network comprises three DenseBlock and a Transition Block, respectively initializing the weights of the network, selecting an arclos function as a loss function, defining an optimizer as an Adam optimizer, and setting the learning rate of the optimizer;
(3) Starting to perform network training on DenseNet121 by tensors of training databases of fingerprints and finger veins loaded in a dataload, performing gradient descent according to an Adam optimizer, performing 5 times of iterative training altogether, after all data in a training set are subjected to network training once, indicating that one time of iterative training is completed, calculating errors and losses between positive examples and negative examples by using a loss function, performing gradient back propagation on parameters in the two networks respectively, repeating the flow, and dynamically updating the loss;
(4) Mapping the fingerprints to be matched and the finger vein images into the probability of each category through an arclose function, and completing final single-mode feature matching;
3.3, resNet network and DenseNet network authentication: and respectively verifying the trained ResNet34, resNet50 and DenseNet121 networks:
(1) Fingerprint test set A is loaded by using dataloaders respectively 2 Finger vein test set B 2 KnuckleGrain test set C 2 Storing the fingerprint and finger vein verification database in a tensor mode;
(2) Network verification is carried out on ResNet34 and ResNet50 respectively from tensors of a verification database of fingerprints and finger veins loaded in a dataload, and finally, a label with the highest probability in a matching result is output, namely identity information corresponding to the image, parameters are continuously adjusted through previous training until the model converges, and the convergence condition is that the loss is less than a target value 10 -5 ；
The method for fusing the multi-modal feature vectors by the attribute module in the step 4 is shown in fig. 4, and the attribute module feature fusion stage includes training and verification processes:
4.1, performing feature fusion on the data of the multiple modes extracted in the step 3, wherein as shown in fig. 4, the training steps are as follows:
(1) Respectively aiming at the fingerprint training set A according to the trained network model in the step 3 1 Training set B of finger vein 1 Knuckle line training set C 1 Extracting biological characteristics, respectively initializing and endowing the extracted data characteristics of a plurality of modes with a weight to form an initial mode characteristic weight matrix, wherein the weights are equal and are 1.
(2) The multi-head attention mechanism is adopted for calculation, the extracted feature vector is calculated to carry out dot product self-attention calculation to obtain a dynamically weighted vector, and a self-attention vector formula is as follows:
wherein, each feature vector is taken as K in the formula, K is a key value key in self-attention calculation, V=Q=K, Q represents a query, V represents a value term value, and QK T Represents a dot product operation for obtaining the similarity between all vectors,
representing scale, N represents length in terms of vector, head (Q, K, V) is a multi-head attention mechanismCalculating a middle single head;
then will
By relu 2 The function is activated, wherein the relu function can be written as:
f represents the output result of the activation function;
(3) The multi-headed attention is then defined as being mapped back to the query subspace D q Where H is a different number of representing subspaces,
Attention(q,k,v)＝[head 1 (q,k,v),head 2 (q,k,v),…,head H (q,k,v)]W out
(4) Training the model by using a batch with the size of 32, and filling all the features into the longest sequence in the batch for the consistency of the same batch in the data processing process; because the dimensions of the modal features are different, they need to be mapped into the internal space of the model, the dimension of the internal space is 1024 dimensions, the model is based on a transducer, N is 2 layers, and H is 4 heads; the output dimension of the final generator is consistent with the vector size of the descriptive word and is 10172 dimension; gamma=0.7, loss probability p=0.1, and learning rate 10 were used in final loss calculation tag smoothing -5 Training the description generator by Adam optimizer;
obtaining the optimal weight of the data characteristic of each mode through model training and learning; and carrying out weighted fusion on the data characteristics of all modes: multiplying the weight of each mode by the data characteristics of the corresponding mode, cascading the characteristic vectors obtained by all modes, wherein the characteristic vectors obtained by all modes are the fingerprint characteristic vector obtained by the ResNet34 model, the finger vein characteristic vector obtained by the ResNet50 model and the finger vein characteristic vector obtained by the DenseNet121 model, so as to form a joint characteristic vector, and finally, sending the joint characteristic vector to a classifier for classification;
4.2, verifying the attribute fusion module:
(1) Fingerprint test set A is loaded by using dataloaders respectively 2 Finger vein test set B 2 Knuckle test set C 2 Storing the fingerprint and finger vein verification database in a tensor mode;
(2) The tensor of the verification database of the fingerprint and the finger vein loaded in the dataload is firstly subjected to feature extraction through the ResNet34 and the ResNet50, then subjected to weight configuration through the attention module, then subjected to feature fusion, and finally output the label with the highest probability in the matching result, namely the identity information corresponding to the image. The parameters are continuously adjusted through the previous training until the attribute module converges, wherein the convergence condition is that the error value is smaller than the preset expected value 10 -5 。
The step 5 is used for deploying an application stage, and the fusion algorithm based on the attention module obtained in the step 1-4 through training and verification is stored in a memory of the intelligent coded lock;
specifically, when a visitor verifies, the intelligent coded lock firstly collects biological characteristic information of the visitor, performs characteristic extraction according to a multi-mode biological characteristic recognition method trained in advance, compares the obtained fusion characteristic with the trained fusion characteristic in the database, selects a group of characteristics with highest similarity, and if the characteristics are matched, the characteristics are identified to be matched, otherwise, the visitor is illegal and does not pass through the database.
Claims (6)
1. A multi-modal biometric method based on an attention module, comprising the steps of:
(1) Collecting data, namely collecting biological characteristics of three modes including fingerprint, finger vein and knuckle vein by using a biological characteristic collector;
(2) Preprocessing the acquired data of the biological characteristics of the three modes, so that the data accords with the input requirements of the models corresponding to the three modes;
(3) In the training and verification stage of the neural network model, feature extraction is carried out on the data of the multiple modes preprocessed in the step (2) through ResNet and DenseNet, and parameters of the ResNet and the DenseNet are obtained through training and verification;
(4) The characteristic fusion of the attribute module is carried out, the extracted characteristic vectors of the three modes are cascaded to form a total characteristic vector, then the total characteristic vector is sent to a classifier for characteristic classification, an attribute mechanism is adopted, firstly, a Q, K, V matrix is initialized according to the characteristic vector formed by the three mode cascade, wherein K is a key value key in the calculation of the attribute mechanism, Q represents a query, K is a key value key, then, according to the Q, K and V obtained by initialization, the optimal weight is obtained by training and learning, the weighting summation is calculated, the combined characteristic vector is obtained, and the combined characteristic vector is sent to the classifier for classification, so that a better recognition effect is achieved;
(5) And the deployment application is used for deploying and applying the trained algorithm to the intelligent coded lock, so that the visitor provides the required biological characteristics, and the intelligent coded lock identifies whether the user is a valid user or not according to the data in the database.
2. The multi-modal biometric identification method based on the attention module as recited in claim 1 wherein the data collection in step (1) comprises: collecting a fingerprint image data set A, a finger vein image data set B and a knuckle image data set C of a user by using a biological characteristic collector;
the biological characteristic collector is a biological characteristic collector module of the intelligent coded lock, and the intelligent coded lock comprises a main board, a clutch, the biological characteristic collector, a memory and a microprocessor.
3. The multi-modal biological recognition method based on the attention module according to claim 1, wherein the data preprocessing in the step (2) is to preprocess the collected three-modal biological image data sets, and the preprocessing steps are as follows:
1) Automatically labeling the acquired biological images to obtain the numbers of volunteers and the numbers of fingers corresponding to each biological image, and dividing the biological images into a training set and a testing set according to the proportion;
2) Performing image preprocessing on the acquired biological image, subtracting the average value of pixels of each image from the biological image, enhancing by using standard colors, and then normalizing in batches;
3) Enhancing the image; respectively enhancing the input fingerprint image, finger vein image and knuckle pattern image by using Sobel operator, and then positioning and intercepting by using ROI to divide effective fingerprint, finger vein and knuckle pattern data set and training set, wherein the effective fingerprint training set is A 1 The effective fingerprint test set is A 2 The effective finger vein training set is B 1 The effective finger vein test set is B 1 The effective knuckle line training set is C 1 The effective knuckle test set is C 2 。
4. The multi-modal biological recognition method based on the attention module of claim 1, wherein the specific steps in the step (3) are as follows:
1) And (3) a model training stage, wherein feature extraction is carried out on the data of the multiple modes preprocessed in the step (2) through ResNet and DenseNet, and the fingerprint training set A is respectively carried out 1 Training set B of finger vein 1 The two networks of input ResNet34, resNet50 train independently:
(1) fingerprint training set A is loaded by using dataloaders respectively 1 And finger vein training set B 1 Storing the fingerprint and the finger vein training database in a tensor mode;
(2) establishing ResNet34 and ResNet50 networks, wherein ResNet34 and ResNet50 respectively comprise 4 residual blocks, initializing weights of the networks respectively, selecting an arclos function as a loss function, defining an optimizer as an Adam optimizer, and setting the learning rate of the optimizer;
(3) network training is carried out on the ResNet34 and the ResNet50 from tensors of training databases of fingerprints and finger veins loaded in the dataload, gradient descent is continuously carried out, iterative training is carried out for multiple times, error and loss between positive examples and negative examples are calculated by using a loss function, back propagation of gradients is carried out on parameters in the two networks respectively, and the flow is repeated and the loss is dynamically updated;
(4) mapping the fingerprints to be matched and the finger vein images into the probability of each category through an arclose function, and completing final single-mode feature matching;
2) Inputting the images of the knuckle training set into the DenseNet121 network for training independently:
(1) respectively loading a fingerprint training set and a finger vein training set by using a datalink, and storing the fingerprint training set and the finger vein training set in a tensor mode into a fingerprint training database and a finger vein training database;
(2) constructing a DenseNet121 network, wherein the DenseNet121 network comprises three DenseBlock and a Transition Block, initializing the weight of the network, selecting an arclos function as a loss function, defining an optimizer as an Adam optimizer, and setting the learning rate of the optimizer;
(3) starting network training on DenseNet121 by tensors of training databases of fingerprints and finger veins loaded in datalink, continuously performing gradient descent, performing repeated iterative training, calculating errors and losses between positive examples and negative examples by using a loss function, respectively performing gradient counter-propagation on parameters in the two networks, repeating the process, and dynamically updating the loss;
(4) mapping the fingerprints to be matched and the finger vein images into the probability of each category through an arclose function, and completing final single-mode feature matching;
3) Model verification: and respectively verifying the trained ResNet34, resNet50 and DenseNet121 networks:
(1) fingerprint test set A is loaded by using dataloaders respectively 2 Finger vein test set B 2 Knuckle test set C 2 Storing the fingerprint, finger vein and knuckle test data set in tensor mode;
(2) and (3) starting to perform network verification on the ResNet34, the ResNet50 and the DenseNet121 respectively by using tensors of verification databases of fingerprints, finger veins and knuckle veins loaded in the dataload, and finally outputting a label with the highest probability in a matching result, namely identity information corresponding to the image, and continuously adjusting parameters through previous training until the model converges.
5. The multi-modal biometric method based on the attention module as recited in claim 1 wherein performing attention feature fusion in step (4) comprises:
1) The training steps of the attention module are as follows:
firstly, respectively extracting fingerprint training sets A according to the single-mode feature extraction model adjusted in the step (3) 1 Fingerprint training set B 1 Fingerprint test set C 1 Respectively initializing and endowing the extracted data features of a plurality of modes with a weight to form an initial mode feature weight matrix;
(1) the extracted feature vectors are subjected to a dot product self-attention calculation to obtain dynamically weighted vectors, which are available here according to the calculation formula of self-attention, in particular, here the activation function selects relu 2 As an activation function;
(2) the concept of multiple heads is introduced in the intent model to allow the model to learn H different representation subspaces at each location while maintaining the same computational efficiency, typically expressed as inputs with parameterization;
(3) training the model by using a batch with the size of 32, and filling all the features into the longest sequence in the batch for the consistency of the same batch in the data processing process; because the dimensions of the modal features are different, they need to be mapped into the internal space of the model, the dimension of the internal space is 1024 dimensions, the model is based on a transducer, N is 2 layers, and H is 4 heads; the output dimension of the final generator is consistent with the vector size of the descriptive word and is 10172 dimension; gamma=0.7, loss probability p=0.1, and learning rate 10 were used in final loss calculation tag smoothing -5 Training the description generator by Adam optimizer;
obtaining the optimal weight of the data characteristic of each mode through model training and learning; and carrying out weighted fusion on the data characteristics of all modes: multiplying the weight of each mode by the data feature of the corresponding mode, cascading feature vectors obtained by all modes, wherein the feature vectors obtained by all modes are fingerprint feature vectors obtained by a ResNet model, finger vein feature vectors and finger knuckle feature vectors obtained by a DenseNet model, so as to form a joint feature vector, and finally, sending the joint feature vector to a classifier for classification;
2) The authentication module verifies the following steps:
(1) respectively loading a fingerprint training set and a finger vein verification set by using a datalink, and storing the fingerprint training set and the finger vein verification set in a tensor mode into a fingerprint and finger vein verification database;
(2) the tensor of the verification database of the fingerprint and the finger vein loaded in the dataload is firstly subjected to feature extraction through the ResNet34 and the ResNet50, then subjected to weight configuration through the attention module, then subjected to feature fusion, finally output the label with the highest probability in the matching result, namely the identity information corresponding to the image, and parameters are continuously adjusted through previous training until the model converges.
6. The multi-mode biological recognition method based on the attention module according to claim 1, wherein in the step (5), a fusion algorithm based on the attention module obtained through training and verification is stored in a memory of the intelligent coded lock, and when a visitor verifies, the intelligent coded lock can obtain a recognition result according to biological characteristic information of the visitor and combined with the trained algorithm.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211367427.6A CN115995121A (en) | 2022-11-02 | 2022-11-02 | Multi-mode biological identification method based on attention module |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211367427.6A CN115995121A (en) | 2022-11-02 | 2022-11-02 | Multi-mode biological identification method based on attention module |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115995121A true CN115995121A (en) | 2023-04-21 |
Family
ID=85994404
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211367427.6A Pending CN115995121A (en) | 2022-11-02 | 2022-11-02 | Multi-mode biological identification method based on attention module |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115995121A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117292466A (en) * | 2023-10-17 | 2023-12-26 | 江苏新巢天诚智能技术有限公司 | Multi-mode computer vision and biological recognition based Internet of things unlocking method |
| CN117690178A (en) * | 2024-01-31 | 2024-03-12 | 江西科技学院 | Face image recognition method and system based on computer vision |
| CN117831138A (en) * | 2024-03-05 | 2024-04-05 | 天津科技大学 | Multi-mode biological feature recognition method based on third-order knowledge distillation |
-
2022
- 2022-11-02 CN CN202211367427.6A patent/CN115995121A/en active Pending
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117292466A (en) * | 2023-10-17 | 2023-12-26 | 江苏新巢天诚智能技术有限公司 | Multi-mode computer vision and biological recognition based Internet of things unlocking method |
| CN117292466B (en) * | 2023-10-17 | 2024-05-17 | 江苏新巢天诚智能技术有限公司 | Multi-mode computer vision and biological recognition based Internet of things unlocking method |
| CN117690178A (en) * | 2024-01-31 | 2024-03-12 | 江西科技学院 | Face image recognition method and system based on computer vision |
| CN117690178B (en) * | 2024-01-31 | 2024-04-05 | 江西科技学院 | Face image recognition method and system based on computer vision |
| CN117831138A (en) * | 2024-03-05 | 2024-04-05 | 天津科技大学 | Multi-mode biological feature recognition method based on third-order knowledge distillation |
| CN117831138B (en) * | 2024-03-05 | 2024-05-24 | 天津科技大学 | Multi-mode biological feature recognition method based on third-order knowledge distillation |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111814706B (en) | Face recognition and attribute classification method based on multitask convolutional neural network | |
| CN115995121A (en) | Multi-mode biological identification method based on attention module | |
| US20130266195A1 (en) | Hash-Based Face Recognition System | |
| Bharadi et al. | Off-line signature recognition systems | |
| US20200302041A1 (en) | Authentication verification using soft biometric traits | |
| CN107944356B (en) | The identity identifying method of the hierarchical subject model palmprint image identification of comprehensive polymorphic type feature | |
| CN104951940A (en) | Mobile payment verification method based on palmprint recognition | |
| Kumar et al. | An improved biometric fusion system of fingerprint and face using whale optimization | |
| AU2021101613A4 (en) | Multi-modal feature fusion–based fingerprint liveness detection method | |
| CN111178130A (en) | Face recognition method, system and readable storage medium based on deep learning | |
| KR20120026232A (en) | Method and apparatus for generating an objected descriptor using extended curvature gabor filter | |
| Rajasekar et al. | Efficient multimodal biometric recognition for secure authentication based on deep learning approach | |
| Charfi | Biometric recognition based on hand schape and palmprint modalities | |
| CN103942545A (en) | Method and device for identifying faces based on bidirectional compressed data space dimension reduction | |
| CN114511901A (en) | Age classification assisted cross-age face recognition algorithm | |
| CN103942572A (en) | Method and device for extracting facial expression features based on bidirectional compressed data space dimension reduction | |
| CN114913610A (en) | Multi-mode identification method based on fingerprints and finger veins | |
| Prasanth et al. | Fusion of iris and periocular biometrics authentication using CNN | |
| George | Development of efficient biometric recognition algorithms based on fingerprint and face | |
| Tang et al. | A review of recent advances in identity identification technology based on biological features | |
| Goranin et al. | Evolutionary Algorithms Application Analysis in Biometric Systems. | |
| Yang et al. | CGAN-DA: A Cross-Modality Domain Adaptation Model for Hand-Vein Biometric-based Authentication | |
| CN112241680A (en) | Multi-mode identity authentication method based on vein similar image knowledge migration network | |
| Ibrahem et al. | Age face invariant recognition model based on VGG face based DNN and support vector classifier | |
| Zhang | Children’s face recognition based on convolutional neural network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |