CN111539247B - Hyper-spectrum face recognition method and device, electronic equipment and storage medium thereof - Google Patents

Hyper-spectrum face recognition method and device, electronic equipment and storage medium thereof Download PDF

Info

Publication number
CN111539247B
CN111539247B CN202010163505.5A CN202010163505A CN111539247B CN 111539247 B CN111539247 B CN 111539247B CN 202010163505 A CN202010163505 A CN 202010163505A CN 111539247 B CN111539247 B CN 111539247B
Authority
CN
China
Prior art keywords
face
image set
training
visible light
infrared
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202010163505.5A
Other languages
Chinese (zh)
Other versions
CN111539247A (en
Inventor
曹志诚
庞辽军
岑熙
赵恒�
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Jingying Information Technology Co ltd
Original Assignee
Xi'an Xd Xin'an Intelligent Technology Co ltd
Xidian University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Xi'an Xd Xin'an Intelligent Technology Co ltd, Xidian University filed Critical Xi'an Xd Xin'an Intelligent Technology Co ltd
Priority to CN202010163505.5A priority Critical patent/CN111539247B/en
Publication of CN111539247A publication Critical patent/CN111539247A/en
Application granted granted Critical
Publication of CN111539247B publication Critical patent/CN111539247B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/168—Feature extraction; Face representation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/172—Classification, e.g. identification

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • Computing Systems (AREA)
  • Biomedical Technology (AREA)
  • Computational Linguistics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Biophysics (AREA)
  • Multimedia (AREA)
  • Human Computer Interaction (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Molecular Biology (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

The invention discloses a hyper-spectral face recognition method, a hyper-spectral face recognition device, electronic equipment and a storage medium thereof, wherein the method comprises the steps of acquiring a visible light face image set and an infrared face image set; respectively preprocessing the visible light face image set and the infrared face image set; constructing a hyperspectral image fusion network model, and training the hyperspectral image fusion network model; obtaining a hyper-spectrum face image set through a hyper-spectrum image fusion network model; constructing a convolutional neural network face recognition model, and training the convolutional neural network face recognition model; and obtaining a face feature set through a convolutional neural network face recognition model, and classifying the face feature set by using a support vector machine classifier so as to realize hyperspectral face recognition. The invention provides a set of complete hyper-spectral face recognition technology, which can overcome the defects of narrow application range, low recognition performance, poor feature extraction robustness and the like of the traditional face recognition technology.

Description

Hyper-spectrum face recognition method and device, electronic equipment and storage medium thereof
Technical Field
The invention belongs to the technical field of pattern recognition and digital image processing, and particularly relates to a hyper-spectral face recognition method and device, electronic equipment and a storage medium thereof.
Background
Face recognition is a technique that uses a computer to automatically recognize or verify the identity of an object from an image or video. Face recognition has become one of the most interesting biometric modalities due to its advantages of low acquisition cost and non-invasiveness. With the application of theories and technologies such as deep learning, face recognition has been developed greatly.
At present, a visible light imaging and collecting means is mostly adopted in a face recognition technology, the technology is mostly limited to a good background condition with sufficient light in the daytime, and the technology usually does not perform well in environments such as insufficient light, severe climate and the like. With the emergence of applications in various complex environments in the real world, the face recognition technology based on visible light is increasingly difficult to meet the requirements. The infrared imaging technology has the advantages of low background light requirement, imaging in rainy days, foggy days and other climates, and the like, and makes up for the defects of visible light imaging environment. The face recognition technology only uses visible light wave band, so that the utilization of useful face information is limited when face recognition features are extracted, the improvement of the final face recognition performance is limited, and according to the imaging principle, a visible light image can provide texture details with high spatial resolution and definition in a mode consistent with a human visual system; in contrast, an infrared image can distinguish the target from the background according to the radiation difference, providing information such as thermal radiation or skin reflection properties that can be different types of light. Therefore, the image fusion technology can be utilized to combine the thermal radiation information or skin reflection attribute in the infrared image with the detailed texture information in the visible light image, and the hyper-spectrum face image capable of meeting the requirement of higher face recognition performance is obtained. Up to now, most methods for face image fusion are conventional methods, such as Discrete Wavelet Transform (DWT), principal Component Analysis (PCA), non-subsampled contour transform (NSCT), and Cross Bilinear Filter (CBF); with the growing maturity of deep learning, a few image fusion methods based on convolutional neural networks, such as the DenseFuse method, have appeared.
However, the traditional face image fusion algorithms are all designed manually, and are generally complex and complicated in design, good in performance but poor in robustness for specific situations, and poor in recognition effect under illumination change; the DenseeFuse method based on deep learning is not designed for hyper-spectral face recognition task, and the effect is not ideal in face recognition occasion.
Disclosure of Invention
In order to solve the above problems in the prior art, the present invention provides a hyper-spectral face recognition method, a hyper-spectral face recognition device, an electronic device, and a storage medium thereof.
One embodiment of the invention provides a hyper-spectral face recognition method, which comprises the following steps:
acquiring a visible light face image set and an infrared face image set, and dividing the visible light face image set into a visible light face training image set and a visible light face testing image set, wherein the infrared face image set is divided into an infrared face training image set and an infrared face testing image set;
respectively preprocessing the visible face training image set, the visible face testing image set, the infrared face training image set and the infrared face testing image set to obtain a visible face preprocessing training image set, a visible face preprocessing testing image set, an infrared face preprocessing training image set and an infrared face preprocessing testing image set;
constructing a hyperspectral image fusion network model, and training the hyperspectral image fusion network model according to the visible light face preprocessing training image set and the infrared face preprocessing training image set to obtain a trained hyperspectral image fusion network model;
inputting the visible light face test image set and the infrared face test image set into the trained hyperspectral image fusion network model to obtain a hyperspectral face image set, and dividing the hyperspectral face image set into a hyperspectral face training image set and a hyperspectral face test image set;
constructing a convolutional neural network face recognition model, and training the convolutional neural network face recognition model according to the hyper-spectral face training image set to obtain a trained convolutional neural network face recognition model;
and inputting the hyper-spectral face test image set into the trained convolutional neural network face recognition model to obtain a face feature set, and classifying the face feature set by using a support vector machine classifier to realize hyper-spectral face recognition.
In an embodiment of the present invention, the pre-processing the visible light face training image set, the visible light face testing image set, the infrared face training image set, and the infrared face testing image set to obtain a visible light face preprocessing training image set, a visible light face preprocessing testing image set, an infrared face preprocessing training image set, and an infrared face preprocessing testing image set respectively includes:
respectively carrying out gray level conversion and normalization processing on the visible light face training image set and the visible light face testing image set to obtain a visible light face preprocessing training image set and a visible light face preprocessing testing image set;
and respectively carrying out image enhancement and normalization processing on the infrared face training image set and the infrared face testing image set to obtain an infrared face preprocessing training image set and an infrared face preprocessing testing image set.
In one embodiment of the invention, the constructed hyperspectral image fusion network model comprises a pre-fusion layer, an encoder module, a fusion layer and a decoder module, wherein,
the input of the pre-fusion layer is connected with an input image, the output of the pre-fusion layer is connected with the input of the encoder module, the encoder module comprises a first convolution layer and a dense residual module which are sequentially connected, the output of the dense residual module and the output of the pre-fusion layer are subjected to global residual connection output, and the global residual connection output is connected with the input of the fusion layer;
the output of the fusion layer is connected with the input of the decoder module, the decoder module comprises second to fifth convolution layers and a feedback layer which are sequentially connected, and the output of the feedback layer and the output of the fusion layer are input to the second convolution layer again to form feedback connection.
In an embodiment of the present invention, the dense residual module includes a first dense residual connection layer, a third dense residual connection layer, a multi-scale splicing layer, and a fourth dense residual connection layer, which are sequentially connected, an input of the first dense residual connection layer is further connected to an output of the first dense residual connection layer, an output of the second dense residual connection layer, and an output of the third dense residual connection layer, an input of the second dense residual connection layer is further connected to an output of the second dense residual connection layer and an output of the third dense residual connection layer, an input of the third dense residual connection layer is further connected to an output of the third dense residual connection layer, an output of the fourth dense residual connection layer is further locally connected to and laminated with an input of the first volume residual connection layer, and a local residual connection output and an output of the pre-fusion layer are globally connected and output.
In an embodiment of the present invention, training the hyperspectral image fusion network model according to the visible light face preprocessing training image set and the infrared face preprocessing training image set to obtain a trained hyperspectral image fusion network model, includes:
constructing a composite loss function based on structural loss, pixel loss and average gradient loss;
and training the hyperspectral image fusion network model according to the visible face preprocessing training image set and the infrared face preprocessing training image set by using the composite loss function based on structural loss, pixel loss and average gradient loss to obtain the trained hyperspectral image fusion network model.
In one embodiment of the invention, the constructed convolutional neural network face recognition model comprises a ResNet feature extraction module, a feature normalization module and a feature space mapping module which are sequentially connected.
In an embodiment of the present invention, training the convolutional neural network face recognition model according to the hyper-spectral face training image set to obtain a trained convolutional neural network face recognition model, including:
constructing a triple loss function Triplet loss;
and training the convolutional neural network face recognition model according to a hyper-spectral face training image set by utilizing the triple loss function triple loss to obtain a trained convolutional neural network face recognition model.
Another embodiment of the present invention provides a hyper-spectral face recognition apparatus, including:
the data acquisition module is used for acquiring a visible light face image set and an infrared face image set, dividing the visible light face image set into a visible light face training image set and a visible light face testing image set, and dividing the infrared face image set into an infrared face training image set and an infrared face testing image set;
the data preprocessing module is used for respectively preprocessing the visible light face training image set, the visible light face testing image set, the infrared face training image set and the infrared face testing image set to obtain a visible light face preprocessing training image set, a visible light face preprocessing testing image set, an infrared face preprocessing training image set and an infrared face preprocessing testing image set;
the first model construction training module is used for constructing a hyperspectral image fusion network model, and training the hyperspectral image fusion network model according to the visible light face preprocessing training image set and the infrared face preprocessing training image set to obtain a trained hyperspectral image fusion network model;
the data generation module is used for inputting the visible light face test image set and the infrared face test image set into the trained hyper-spectral image fusion network model to obtain a hyper-spectral face image set, and dividing the hyper-spectral face image set into a hyper-spectral face training image set and a hyper-spectral face test image set;
the second model construction training module is used for constructing a convolutional neural network face recognition model, and training the convolutional neural network face recognition model according to the hyper-spectral face training image set to obtain a trained convolutional neural network face recognition model;
and the data identification module is used for inputting the hyper-spectral face test image set into the trained convolutional neural network face identification model to obtain a face feature set, and classifying the face feature set by using a support vector machine classifier to realize hyper-spectral face identification.
The invention further provides hyper-spectrum face recognition electronic equipment, which comprises an image collector, a display, a processor, a communication interface, a memory and a communication bus, wherein the image collector, the display, the processor, the communication interface and the memory finish mutual communication through the communication bus;
the image collector is used for collecting image data;
the display is used for displaying the image identification data;
the memory is used for storing a computer program;
the processor is configured to implement any one of the above hyper-spectral face recognition methods when executing the computer program stored in the memory.
Yet another embodiment of the present invention provides a computer-readable storage medium having stored therein a computer program which, when executed by a processor, implements a hyper-spectral face recognition method as described in any of the above.
Compared with the prior art, the invention has the beneficial effects that:
the invention provides a set of complete hyper-spectral face recognition technology, which can overcome the defects of narrow application range, low recognition performance, poor feature extraction robustness and the like of the traditional face recognition technology; the embodiment provides a new theory and a new method for the practicability of the face recognition technology, so that the face recognition technology becomes more practical, reliable and popularized.
The present invention will be described in further detail with reference to the accompanying drawings and examples.
Drawings
Fig. 1 is a schematic flow chart of a hyper-spectral face recognition method according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of a hyper-spectral image fusion network model in the hyper-spectral face recognition method according to the embodiment of the present invention;
fig. 3 is a schematic structural diagram of a dense residual module in a hyper-spectral face recognition method according to an embodiment of the present invention;
FIG. 4 is a schematic structural diagram of a convolutional neural network face recognition model in a hyper-spectral face recognition method according to an embodiment of the present invention;
fig. 5a to 5d are schematic diagrams illustrating examples of a visible light face image and an infrared face image in a hyper-spectral face recognition method according to an embodiment of the present invention;
fig. 6a to 6b are schematic diagrams illustrating examples of face images after hyper-spectral fusion in a hyper-spectral face recognition method according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of a hyper-spectral face recognition apparatus according to an embodiment of the present invention;
FIG. 8 is a schematic structural diagram of an electronic device for hyper-spectral face recognition according to an embodiment of the present invention;
fig. 9 is a schematic structural diagram of a computer-readable storage medium according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to specific examples, but the embodiments of the present invention are not limited thereto.
Example one
At present, the face recognition technology mostly adopts visible light imaging and acquisition means, so the technology is mostly limited to the good background condition with sufficient light in the daytime and generally has poor performance in the environments of insufficient light, severe climate and the like. With the emergence of applications in various complex environments in the real world, the face recognition technology based on visible light is more and more difficult to meet the requirements, and the infrared imaging technology has the advantages of low background light requirements, imaging in rainy days, foggy days and other climates, and the like, and makes up for the defects of the visible light imaging environment. Therefore, referring to fig. 1, fig. 1 is a schematic flow chart of a hyper-spectral face recognition method according to an embodiment of the present invention, where the embodiment provides a hyper-spectral face recognition method, and the method includes the following steps:
step 1, a visible light face image set and an infrared face image set are obtained, the visible light face image set is divided into a visible light face training image set and a visible light face testing image set, and the infrared face image set is divided into an infrared face training image set and an infrared face testing image set.
Specifically, in the embodiment, the visible light and the infrared cameras are used to simultaneously acquire the face images of the individuals to obtain the visible light face image set and the infrared face image set. And the visible light face image set and the infrared face image set are used for subsequent face recognition.
And 2, respectively preprocessing the visible light face training image set, the visible light face testing image set, the infrared face training image set and the infrared face testing image set to obtain a visible light face preprocessing training image set, a visible light face preprocessing testing image set, an infrared face preprocessing training image set and an infrared face preprocessing testing image set.
Specifically, in order to realize better image fusion, in this embodiment, before image fusion, normalization and contrast adjustment are performed on face images in the visible light face training image set, the visible light face testing image set, the infrared face training image set, and the infrared face testing image set, specifically, step 2 in this embodiment includes step 2.1 and step 2.2:
and 2.1, respectively carrying out gray level conversion and normalization processing on the visible light face training image set and the visible light face testing image set to obtain a visible light face preprocessing training image set and a visible light face preprocessing testing image set.
Specifically, in this embodiment, the face images in the visible light face training image set and the visible light face testing image set are firstly subjected to gray scale conversion into gray scale images, and the formula design of the gray scale conversion is specifically as follows:
I gray ＝0.2989×R+0.5870×G+0.1140×B (1)
wherein, I gray For the gray image output after gray conversion, R, G, B is the RGB values corresponding to the image before gray conversion, which is specifically the RGB values corresponding to the face images in the visible light face training image set and the visible light face testing image set in this embodiment.
Then, the gray image I is processed gray A normalization process is performed, which normalizes to [0, 255]]The normalized formula design is as follows:
Figure BDA0002406617480000091
wherein, I n As a grey scale image I gray Normalized image output of (1), I max And I min Respectively a gray scale image I gray Maximum and minimum gray values.
In this embodiment, each of the visible light face training image and the visible light face testing image in the visible light face training image set and the visible light face testing image set is processed by the above formula (1) and formula (2), so as to obtain a visible light face preprocessing training image set and a visible light face preprocessing testing image set.
And 2.2, respectively carrying out image enhancement and normalization processing on the infrared face training image set and the infrared face testing image set to obtain an infrared face preprocessing training image set and an infrared face preprocessing testing image set.
Specifically, in this embodiment, the face images in the infrared face training image set and the infrared face testing image set are first subjected to image enhancement by using a log operator, and the formula design for image enhancement is specifically as follows:
I＝log(1+X) (3)
the embodiment specifically includes an infrared face training image set and an infrared face testing image set, where I is an image after image enhancement, and X is an image before image enhancement.
Then, the enhanced image I is normalized to [0, 255] by the above formula (2).
In this embodiment, each of the infrared face training image set and the infrared face testing image set is processed by the above formula (3) and formula (2), so as to obtain an infrared face preprocessing training image set and an infrared face preprocessing testing image set.
And 3, constructing a hyperspectral image fusion network model, and training the hyperspectral image fusion network model according to the visible light face preprocessing training image set and the infrared face preprocessing training image set to obtain the trained hyperspectral image fusion network model.
Specifically, in order to obtain a high-quality hyperspectral image with rich texture information and thermal information, the present embodiment proposes a novel feedback type depth fusion framework to fuse an infrared image and a visible light image, and specifically, step 3 of the present embodiment includes step 3.1 and step 3.2:
and 3.1, constructing a hyperspectral image fusion network model.
Specifically, referring to fig. 2, fig. 2 is a schematic structural diagram of a hyper-spectral image fusion network model in a hyper-spectral face recognition method according to an embodiment of the present invention, where the embodiment provides a hyper-spectral image fusion network model RFFuseNet with high robustness and specially for hyper-spectral face image fusion, specifically: the constructed hyperspectral image fusion network model comprises a pre-fusion layer PF, an encoder module, a fusion layer F and a decoder module, wherein the input of the pre-fusion layer PF is connected with an input image, the output of the pre-fusion layer PF is connected with the input of the encoder module, the encoder module comprises a first convolution layer C1 and an intensive residual module RDB which are sequentially connected, the output of the intensive residual module RDB and the output of the pre-fusion layer PF are subjected to global residual connection output, and the global residual connection output is connected with the input of the fusion layer F; the output of the fusion layer F is connected with the input of the decoder module, the decoder module comprises second to fifth convolution layers C2 to C5 and a feedback layer FB which are connected in sequence, and the output of the feedback layer FB and the output of the fusion layer F are input to the second convolution layer C2 again to form feedback connection. Referring to fig. 3, fig. 3 is a schematic structural diagram of a dense residual module in a hyper-spectral face recognition method according to an embodiment of the present invention, where the dense residual module RDB includes a first dense residual connection layer C _ R _1 to a third dense residual connection layer C _ R _3, a multi-scale splicing layer CC, and a fourth dense residual connection layer C _ R _4, which are sequentially connected, an input of the first dense residual connection layer is further connected to an output of the first dense residual connection layer, an output of the second dense residual connection layer, and an output of the third dense residual connection layer, an input of the second dense residual connection layer is further connected to an output of the second dense residual connection layer and an output of the third dense residual connection layer, an output of the fourth dense residual connection layer C _ R _4 is further connected to an input of the first volume of the packed layer C1 for local residual connection output, and an output of the local residual connection layer is connected to an output of the pre-spectral layer for global residual output PF.
And 3.2, training the hyperspectral image fusion network model according to the visible face preprocessing training image set and the infrared face preprocessing training image set to obtain the trained hyperspectral image fusion network model.
Specifically, in the training process of this embodiment, a composite loss function based on the structural loss, the pixel loss, and the average gradient loss is constructed, specifically, the sum of the structural loss, the pixel loss, and the average gradient loss is used as a loss function, and specifically, the composite loss function is designed as follows:
Loss＝λL ssim +L p +0.05L ag (4)
wherein λ is a parameter, and may be 1, 10, 100 and 1000 ssim 、L p And L ag Respectively, structural loss, pixel loss and average gradient loss, specifically L ssim 、L p And L ag The design is as follows:
L ssim ＝1-SSIM(O,I) (5)
L p ＝||O-I|| 2 (6)
Figure BDA0002406617480000121
wherein, O and I respectively represent an output image and an input image corresponding to the hyper-spectral image fusion network model, M multiplied by N represents the size of the output image,
Figure BDA0002406617480000122
and
Figure BDA0002406617480000123
respectively representing the gradients of the output image in the horizontal direction and the vertical direction, and the SSIM (O, I) representing the structural similarity degree between the output image and the input image, the specific calculation mode of the SSIM (O, I) is as follows:
SSIM(O,I)＝[l(O,I) α ·c(O,I) β ·s(O,I) γ ] (8)
wherein l (O, I) represents the brightness mean value of the output image and the input image, c (O, I) represents the contrast variance of the output image and the input image, s (O, I) represents the structural similarity value of the output image and the input image, and alpha, beta and gamma are parameters for adjusting the proportion of three components in the hyperspectral image fusion network model.
And further, training the hyperspectral image fusion network model according to the visible light face preprocessing training image set and the infrared face preprocessing training image set by using a composite loss function based on structural loss, pixel loss and average gradient loss to obtain the trained hyperspectral image fusion network model.
Specifically, in this embodiment, based on the composite loss function of the hyperspectral image fusion network model constructed by the above formula (4), the visible face preprocessing training image set and the face training image set in the infrared face preprocessing training image set are input into the hyperspectral image fusion network model for training, and the hyperspectral image fusion network model can be trained by specifically adopting a back propagation algorithm in the training process, so that the hyperspectral image fusion network model trained finally in this embodiment is obtained and used for generating a subsequent hyperspectral face image set.
And 4, inputting the visible light face test image set and the infrared face test image set into the trained hyperspectral image fusion network model to obtain a hyperspectral face image set, and dividing the hyperspectral face image set into a hyperspectral face training image set and a hyperspectral face test image set.
Specifically, in this embodiment, a trained hyper-spectral image fusion network model is obtained in step 3, and the hyper-spectral image fusion network model is used to perform face image fusion processing on the face images in the visible light face test image set and the infrared face test image set, so as to obtain a high-quality hyper-spectral face image set having rich texture information and thermal information. The hyper-spectral face image set is divided into a hyper-spectral face training image set and a hyper-spectral face testing image set for subsequent hyper-spectral face recognition, and preferably, the division ratio of the hyper-spectral face training image set and the hyper-spectral face testing image set is 3:1.
And 5, constructing a convolutional neural network face recognition model, and training the convolutional neural network face recognition model according to the hyper-spectral face training image set to obtain the trained convolutional neural network face recognition model.
Specifically, in this embodiment, a deep learning face recognition technology and a deep hyper-spectral fusion technology are combined to improve the final recognition rate of a face, after a hyper-spectral face image set is generated in step 4, a hyper-spectral face recognition framework based on deep learning is proposed, that is, a convolutional neural network face recognition model is provided, specifically, step 5 in this embodiment includes step 5.1 and step 5.2:
and 5.1, constructing a convolutional neural network face recognition model.
Specifically, referring to fig. 4, fig. 4 is a schematic structural diagram of a convolutional neural network face recognition model in a hyper-spectral face recognition method provided in an embodiment of the present invention, where the convolutional neural network face recognition model constructed in this embodiment includes a ResNet feature extraction module, a feature normalization module L2, and a feature space mapping module ED, which are connected in sequence. Firstly, cutting an input image Batch of the convolutional neural network face recognition model to a fixed size; the ResNet feature extraction module, the feature normalization module L2, and the feature space mapping module ED may be implemented by the existing common methods, respectively, and the specific implementation manner is not limited.
And 5.2, training the convolutional neural network face recognition model according to the hyper-spectral face training image set to obtain the trained convolutional neural network face recognition model.
Specifically, in the embodiment, a triple loss function triplets loss is constructed in the face recognition training process, and the triple loss function triplets loss is used to train a convolutional neural network face recognition model, and the triple loss function triplets loss is designed as follows:
Figure BDA0002406617480000141
wherein,
Figure BDA0002406617480000142
and
Figure BDA0002406617480000143
three input images representing a convolutional neural network face recognition model,
Figure BDA0002406617480000144
and
Figure BDA0002406617480000145
are two images from the same category and,
Figure BDA0002406617480000146
and
Figure BDA0002406617480000147
are two images from a heterogeneous set, alpha is the interval parameter, and accordingly,
Figure BDA0002406617480000148
and
Figure BDA0002406617480000149
and forming a Triplet group of the output characteristics of the convolutional neural network face recognition model as a loss function Triplet loss.
It can be seen that in this embodiment, the triple loss function Triplet loss needs to receive features of three images and corresponding labels as input, and aims to make the intra-class distance smaller than the inter-class distance through a large amount of triple training, the optimizer selected in the training process is Adam, the learning rate is 0.001, the batch size is 128, and a suitable threshold is set by looking up the curve of the loss function value and the iteration times through a sensor Board plug-in to stop the model training, so that the trained convolutional neural network face recognition model is obtained. The image received by the triple loss function Triplet loss is from a hyper-spectral face training image set.
And 6, inputting the hyper-spectral face test image set into a trained convolutional neural network face recognition model to obtain a face feature set, and classifying the face feature set by using a support vector machine classifier to realize hyper-spectral face recognition.
Specifically, in the embodiment, the euclidean distance is used as a measurement function, a hyper-spectral face test image set is input to a trained convolutional neural network face recognition model for feature extraction to obtain a face feature set, the face feature set and a label thereof are input to a Support Vector Machine (SVM) classifier, and the face recognition accuracy is obtained through the SVM classifier to realize hyper-spectral face recognition. The method comprises the following steps of obtaining the accuracy of face recognition through a Support Vector Machine (SVM) classifier, wherein the steps are as follows:
(1) In-class maximum Euclidean distance D max Setting a threshold value, namely setting a minimum Euclidean distance D between the prediction label and the current image bw_min And D max Comparing, if less than D max And considering the prediction classification to be correct, otherwise, considering the prediction classification to be wrong, and counting the number of the classes with correct classification and wrong classification.
(2) And (2) calculating to obtain the classification Accuracy (Accuracy) according to the category number of the classification Accuracy and the category number of the classification errors counted in the step (1), wherein a specific formula is designed as follows:
Figure BDA0002406617480000151
where TN denotes a true positive case, TP denotes a true negative case, FP denotes a false positive case, and FN denotes a false negative case.
In summary, the introduction of the hyper-spectral face recognition technology for depth image fusion of the present embodiment gradually improves the final recognition accuracy rate through the deep learning and image fusion technologies, specifically: firstly, before image fusion, normalization and contrast adjustment are carried out on a visible light image and an infrared image, then the infrared image and the visible light image are fused by utilizing a depth fusion framework RFFUSENet (hyper-spectral image fusion network model) provided by the application so as to obtain a high-quality hyper-spectral image with rich texture information and thermal information, and finally, a deep learning face recognition method (convolutional neural network face recognition model) is utilized to recognize the hyper-spectral face image, so that the traditional face recognition performance based on the visible light image is improved.
In order to verify the superiority of the hyper-spectral face recognition method provided by the present application, the face data set used by the present application is a CASIA face image set and a QFIRE face image set, please refer to fig. 5a to 5d, and fig. 5a to 5d are schematic diagrams illustrating examples of visible light face images and infrared face images in the hyper-spectral face recognition method provided by the embodiment of the present invention, specifically: fig. 5 (a) and 5 (c) show face images obtained by photographing visible light at 1.5m, fig. 5 (b) and 5 (d) show face images obtained by photographing visible light at 1.5m in the near infrared, fig. 5 (a) shows a CASIA visible light image set, fig. 5 (b) shows a CASIA near infrared image set, fig. 5 (c) shows a QFIRE visible light image set, and fig. 5 (d) shows a QFIRE near infrared image set.
In the verification process of the embodiment, the parameter design of each layer in the hyperspectral image fusion network model is specifically shown in table 1, and the padding mode in the convolution process is 0 filling. The parameters of each layer in the hyperspectral image fusion network model are designed according to actual conditions, and the embodiment is designed according to specific parameters in table 1 for identification and verification.
TABLE 1 design of parameters for each layer in a hyperspectral image fusion network model
Figure BDA0002406617480000161
The experiments designed in this example were demonstrated in two ways:
(1) To show that the fusion effect of the hyperspectral image fusion network model RFFUSENet is superior to that of the traditional face image fusion and other mainstream deep learning methods, the hyperspectral image fusion network model RFFUSENet is compared with image fusion based on CBF (traditional method), image fusion based on DenseeFuses (mainstream deep learning method) and the like, and an entropy value (EN), structural Similarity (SSIM), image edge fidelity (Q) are calculated abf ) And artificial noise (N) abf ) And the like. Referring to fig. 6a to 6b, fig. 6a to 6b are schematic diagrams illustrating face images after performing hyper-spectral fusion in a hyper-spectral face recognition method according to an embodiment of the present invention, where fig. 6 (a) is a result of QFIRE image fusion, and fig. 6 (b) is a result of CASIA image fusion. Referring to table 2, table 2 shows Entropy (EN) and image edge fidelity (Q) of different image fusion methods provided in this embodiment abf ) Structural Similarity (SSIM) and artificial noise (N) abf ) The comparison result of (1). Wherein, entropy value (EN), image edge fidelity (Q) abf ) The larger the Structural Similarity (SSIM) value is, the better the fusion effect is, and the artificial noise (N) abf ) Smaller values indicate better fusion.
Table 2 comparison results of different image fusion methods provided in this embodiment
Figure BDA0002406617480000171
The comparison result in table 2 shows that the hyper-spectral face image fused by the rffuset network of the present application has the maximum entropy value, the structural similarity, the image edge fidelity and the minimum artificial noise, so that the fusion performance of the rffuset method designed by the present application is superior to that of the traditional image fusion method and the mainstream deep learning method.
(2) In order to prove that the face recognition performance of the hyper-spectral image fusion network model RFFuseNet is higher than the recognition performance without using the fusion technology, face recognition experiments are respectively carried out on the situations without using fusion single-spectral images (namely visible light images and infrared images), using CBF-based image fusion, using DenseeFuse-based image fusion, using RFFuseNet fusion and the like. Referring to table 3, table 3 shows the face recognition accuracy comparison results of different methods provided in this embodiment, including results before and after the depth fusion technique is used.
Table 3 comparison results of face recognition accuracy rates of different methods provided in this embodiment
Figure BDA0002406617480000181
As can be seen from the comparison results in table 3, the hyperspectral image recognition rate using the RFFuseNet fusion of the present application is significantly higher than that of two single-spectrum images, which proves that the RFFuseNet fusion method of the present application can effectively improve the face recognition performance, and meanwhile, the hyperspectral image recognition rate using the RFFuseNet fusion of the present application is also higher than that using CBF and DenseFuse, which proves that the hyperspectral face recognition performance of the RFFuseNet model of the present application is better than that of other face fusion methods.
Therefore, the embodiment introduces an image fusion idea aiming at the limitation that the traditional face recognition method only adopts visible light, and fuses the visible light face image and the infrared ray face image to obtain a hyperspectral face image, so that the face image has complementary information (namely abundant texture and thermal information) of visible light and infrared ray at the same time, and the effect of improving the face recognition performance is achieved; the embodiment designs a novel residual feedback type hyper-spectral image fusion network model RFFUSENet aiming at the problem of face image fusion, and experiments show that compared with the traditional fusion method, the hyper-spectral image fusion network model RFFUSENet provided by the application does not need to manually design fusion rules and select fusion parameters, can fuse hyper-spectral face images with higher quality and richer contained information, and has better fusion indexes compared with other fusion methods based on deep learning; in the embodiment, a deep learning face recognition technology and a deep hyper-spectrum fusion technology are combined, a hyper-spectrum face recognition framework based on deep learning is provided, the problems that the traditional face recognition is limited to visible light and the recognition performance is insufficient can be successfully solved, and experiments show that the final face recognition rate can be obviously improved by using the method.
The embodiment provides a set of complete hyper-spectral face recognition technology, which can overcome the defects of narrow application range, low recognition performance, poor feature extraction robustness and the like of the traditional face recognition technology; the embodiment provides a new theory and a new method support for the practicability of the face recognition technology, so that the face recognition technology becomes more practical, reliable and popularized; the embodiment can be widely applied to outdoor, night, rain and snow and other application occasions such as attendance checking, civil monitoring, public security law enforcement, access control, cell entrance and the like in complex environments.
Example two
On the basis of the first embodiment, please refer to fig. 7, and fig. 7 is a schematic structural diagram of a hyper-spectral face recognition apparatus according to an embodiment of the present invention. This embodiment provides a hyperspectral human face recognition device, and the device includes:
and the data acquisition module is used for acquiring a visible light face image set and an infrared face image set, dividing the visible light face image set into a visible light face training image set and a visible light face test image set, and dividing the infrared face image set into an infrared face training image set and an infrared face test image set.
And the data preprocessing module is used for respectively preprocessing the visible light face training image set, the visible light face testing image set, the infrared face training image set and the infrared face testing image set to obtain a visible light face preprocessing training image set, a visible light face preprocessing testing image set, an infrared face preprocessing training image set and an infrared face preprocessing testing image set.
Specifically, the data preprocessing module of this embodiment respectively preprocesses a visible light face training image set, a visible light face test image set, an infrared face training image set, and an infrared face test image set to obtain a visible light face preprocessing training image set, a visible light face preprocessing test image set, an infrared face preprocessing training image set, and an infrared face preprocessing test image set, and the data preprocessing module includes:
respectively carrying out gray level conversion and normalization processing on the visible light face training image set and the visible light face testing image set to obtain a visible light face preprocessing training image set and a visible light face preprocessing testing image set;
and respectively carrying out image enhancement and normalization processing on the infrared face training image set and the infrared face testing image set to obtain an infrared face preprocessing training image set and an infrared face preprocessing testing image set.
And the first model construction training module is used for constructing a hyperspectral image fusion network model, and training the hyperspectral image fusion network model according to the visible light face preprocessing training image set and the infrared face preprocessing training image set to obtain the trained hyperspectral image fusion network model.
Specifically, the hyperspectral image fusion network model constructed in the first model construction training module of the embodiment includes a pre-fusion layer, an encoder module, a fusion layer and a decoder module, wherein,
the input of the pre-fusion layer is connected with the input image, the output of the pre-fusion layer is connected with the input of the encoder module, the encoder module comprises a first convolution layer and a dense residual module which are sequentially connected, the output of the dense residual module and the output of the pre-fusion layer are subjected to global residual connection output, and the global residual connection output is connected with the input of the fusion layer;
the output of the fusion layer is connected with the input of the decoder module, the decoder module comprises second to fifth convolution layers and a feedback layer which are sequentially connected, and the output of the feedback layer and the output of the fusion layer are input into the second convolution layer again to form feedback connection.
The dense residual module comprises a first dense residual connecting layer, a third dense residual connecting layer, a multi-scale splicing layer and a fourth dense residual connecting layer which are sequentially connected, wherein the input of the first dense residual connecting layer is further connected with the output of the first dense residual connecting layer, the output of the second dense residual connecting layer and the output of the third dense residual connecting layer, the input of the second dense residual connecting layer is further connected with the output of the second dense residual connecting layer and the output of the third dense residual connecting layer, the input of the third dense residual connecting layer is further connected with the output of the third dense residual connecting layer, the output of the fourth dense residual connecting layer is further connected with the input of the first volume layer in a local residual mode and the output of the first volume layer in a local residual connecting mode, and the output of the local residual connecting layer and the output of the pre-fusion layer in a global residual connecting mode.
Further, training the hyperspectral image fusion network model according to the visible face preprocessing training image set and the infrared face preprocessing training image set to obtain a trained hyperspectral image fusion network model, comprising:
constructing a composite loss function based on structural loss, pixel loss and average gradient loss;
and training the hyperspectral image fusion network model according to the visible light face preprocessing training image set and the infrared face preprocessing training image set by utilizing a composite loss function based on structural loss, pixel loss and average gradient loss to obtain the trained hyperspectral image fusion network model.
And the data generation module is used for inputting the visible light face test image set and the infrared face test image set into the trained hyperspectral image fusion network model to obtain a hyperspectral face image set, and dividing the hyperspectral face image set into a hyperspectral face training image set and a hyperspectral face test image set.
And the second model construction training module is used for constructing a convolutional neural network face recognition model, and training the convolutional neural network face recognition model according to the hyper-spectral face training image set to obtain the trained convolutional neural network face recognition model.
Specifically, the convolutional neural network face recognition model constructed in the second model construction training module in this embodiment includes a ResNet feature extraction module, a feature normalization module, and a feature space mapping module, which are connected in sequence.
Further, training the convolutional neural network face recognition model according to the hyper-spectral face training image set to obtain a trained convolutional neural network face recognition model, comprising:
constructing a triple loss function Triplet loss;
and training the convolutional neural network face recognition model by utilizing a triple loss function triple loss according to the hyper-spectral face training image set to obtain the trained convolutional neural network face recognition model.
And the data identification module is used for inputting the hyper-spectral face test image set into a trained convolutional neural network face identification model to obtain a face feature set, and classifying the face feature set by using a support vector machine classifier to realize hyper-spectral face identification.
The hyper-spectral face recognition device provided by this embodiment can execute the above hyper-spectral face recognition method embodiment, and its implementation principle and technical effect are similar, and are not described herein again.
EXAMPLE III
On the basis of the second embodiment, please refer to fig. 8, and fig. 8 is a schematic structural diagram of an electronic device for hyper-spectral face recognition according to an embodiment of the present invention. The embodiment provides hyper-spectrum face recognition electronic equipment, which comprises an image collector, a display, a processor, a communication interface, a memory and a communication bus, wherein the image collector, the display, the processor, the communication interface and the memory finish mutual communication through the communication bus;
the image collector is used for collecting image data;
the display is used for displaying the image identification data;
a memory for storing a computer program;
a processor for executing the computer program stored in the memory, the computer program when executed by the processor performing the steps of:
step 1, controlling an image collector to collect face images, acquiring a visible light face image set and an infrared face image set, dividing the visible light face image set into a visible light face training image set and a visible light face testing image set, and dividing the infrared face image set into an infrared face training image set and an infrared face testing image set.
And 2, respectively preprocessing the visible light face training image set, the visible light face testing image set, the infrared face training image set and the infrared face testing image set to obtain a visible light face preprocessing training image set, a visible light face preprocessing testing image set, an infrared face preprocessing training image set and an infrared face preprocessing testing image set.
Specifically, in step 2 of this embodiment, the pre-processing is performed on the visible light face training image set, the visible light face testing image set, the infrared face training image set, and the infrared face testing image set respectively to obtain a visible light face preprocessing training image set, a visible light face preprocessing testing image set, an infrared face preprocessing training image set, and an infrared face preprocessing testing image set, and the method includes:
respectively carrying out gray level conversion and normalization processing on the visible light face training image set and the visible light face testing image set to obtain a visible light face preprocessing training image set and a visible light face preprocessing testing image set;
and respectively carrying out image enhancement and normalization processing on the infrared face training image set and the infrared face testing image set to obtain an infrared face preprocessing training image set and an infrared face preprocessing testing image set.
And 3, constructing a hyperspectral image fusion network model, and training the hyperspectral image fusion network model according to the visible light face preprocessing training image set and the infrared face preprocessing training image set to obtain the trained hyperspectral image fusion network model.
Specifically, the hyperspectral image fusion network model constructed in step 3 of this embodiment includes a pre-fusion layer, an encoder module, a fusion layer, and a decoder module, wherein,
the input of the pre-fusion layer is connected with the input image, the output of the pre-fusion layer is connected with the input of the encoder module, the encoder module comprises a first convolution layer and a dense residual module which are sequentially connected, the output of the dense residual module and the output of the pre-fusion layer are subjected to global residual connection output, and the global residual connection output is connected with the input of the fusion layer;
the output of the fusion layer is connected with the input of the decoder module, the decoder module comprises second to fifth convolution layers and a feedback layer which are sequentially connected, and the output of the feedback layer and the output of the fusion layer are input into the second convolution layer again to form feedback connection.
Furthermore, the dense residual module comprises a first dense residual connecting layer, a third dense residual connecting layer, a multi-scale splicing layer and a fourth dense residual connecting layer which are sequentially connected, wherein the input of the first dense residual connecting layer is further connected with the output of the first dense residual connecting layer, the output of the second dense residual connecting layer and the output of the third dense residual connecting layer, the input of the second dense residual connecting layer is further connected with the output of the second dense residual connecting layer and the output of the third dense residual connecting layer, the input of the third dense residual connecting layer is further connected with the output of the third dense residual connecting layer, the output of the fourth dense residual connecting layer is further connected with the input of the first volume layer in a local residual manner and the output of the local residual connecting layer is connected with the output of the pre-fusion layer in a global residual manner and output.
Further, training the hyperspectral image fusion network model according to the visible face preprocessing training image set and the infrared face preprocessing training image set to obtain a trained hyperspectral image fusion network model, comprising:
constructing a composite loss function based on structural loss, pixel loss and average gradient loss;
and training the hyperspectral image fusion network model according to the visible light face preprocessing training image set and the infrared face preprocessing training image set by utilizing a composite loss function based on structural loss, pixel loss and average gradient loss to obtain the trained hyperspectral image fusion network model.
And 4, inputting the visible light face test image set and the infrared face test image set into the trained hyper-spectral image fusion network model to obtain a hyper-spectral face image set, and dividing the hyper-spectral face image set into a hyper-spectral face training image set and a hyper-spectral face test image set.
And 5, constructing a convolutional neural network face recognition model, and training the convolutional neural network face recognition model according to the hyper-spectral face training image set to obtain the trained convolutional neural network face recognition model.
Specifically, the convolutional neural network face recognition model constructed in step 5 of this embodiment includes a ResNet feature extraction module, a feature normalization module, and a feature space mapping module, which are connected in sequence.
Further, training the convolutional neural network face recognition model according to the hyper-spectral face training image set to obtain a trained convolutional neural network face recognition model, comprising:
constructing triple loss function Triplet loss;
and training the convolutional neural network face recognition model by utilizing a triple loss function triple loss according to the hyper-spectral face training image set to obtain the trained convolutional neural network face recognition model.
And 6, inputting the hyper-spectral face test image set into a trained convolutional neural network face recognition model to obtain a face feature set, and classifying the face feature set by using a support vector machine classifier to realize hyper-spectral face recognition. And finally, outputting the hyper-spectrum face recognition result to a display.
The hyper-spectral face recognition electronic device provided by this embodiment may implement the hyper-spectral face recognition method embodiment and the hyper-spectral face recognition apparatus embodiment, and the implementation principle and the technical effect are similar, which are not described herein again.
Example four
On the basis of the third embodiment, please refer to fig. 9, and fig. 9 is a schematic structural diagram of a computer-readable storage medium according to an embodiment of the present invention. The present embodiment provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the following steps:
step 1, a visible light face image set and an infrared face image set are obtained, the visible light face image set is divided into a visible light face training image set and a visible light face testing image set, and the infrared face image set is divided into an infrared face training image set and an infrared face testing image set.
And 2, respectively preprocessing the visible light face training image set, the visible light face testing image set, the infrared face training image set and the infrared face testing image set to obtain a visible light face preprocessing training image set, a visible light face preprocessing testing image set, an infrared face preprocessing training image set and an infrared face preprocessing testing image set.
Specifically, in step 2 of this embodiment, the pre-processing is performed on the visible light face training image set, the visible light face testing image set, the infrared face training image set, and the infrared face testing image set respectively to obtain a visible light face preprocessing training image set, a visible light face preprocessing testing image set, an infrared face preprocessing training image set, and an infrared face preprocessing testing image set, and the method includes:
respectively carrying out gray level conversion and normalization processing on the visible light face training image set and the visible light face testing image set to obtain a visible light face preprocessing training image set and a visible light face preprocessing testing image set;
and respectively carrying out image enhancement and normalization processing on the infrared face training image set and the infrared face testing image set to obtain an infrared face preprocessing training image set and an infrared face preprocessing testing image set.
And 3, constructing a hyperspectral image fusion network model, and training the hyperspectral image fusion network model according to the visible light face preprocessing training image set and the infrared face preprocessing training image set to obtain the trained hyperspectral image fusion network model.
Specifically, the hyperspectral image fusion network model constructed in step 3 of this embodiment includes a pre-fusion layer, an encoder module, a fusion layer, and a decoder module, wherein,
the input of the pre-fusion layer is connected with the input image, the output of the pre-fusion layer is connected with the input of the encoder module, the encoder module comprises a first convolution layer and a dense residual module which are sequentially connected, the output of the dense residual module and the output of the pre-fusion layer are subjected to global residual connection output, and the global residual connection output is connected with the input of the fusion layer;
the output of the fusion layer is connected with the input of the decoder module, the decoder module comprises second to fifth convolution layers and a feedback layer which are sequentially connected, and the output of the feedback layer and the output of the fusion layer are input into the second convolution layer again to form feedback connection.
The dense residual module comprises a first dense residual connecting layer, a third dense residual connecting layer, a multi-scale splicing layer and a fourth dense residual connecting layer which are sequentially connected, wherein the input of the first dense residual connecting layer is further connected with the output of the first dense residual connecting layer, the output of the second dense residual connecting layer and the output of the third dense residual connecting layer, the input of the second dense residual connecting layer is further connected with the output of the second dense residual connecting layer and the output of the third dense residual connecting layer, the input of the third dense residual connecting layer is further connected with the output of the third dense residual connecting layer, the output of the fourth dense residual connecting layer is further connected with the input of the first volume layer in a local residual mode and the output of the first volume layer in a local residual connecting mode, and the output of the local residual connecting layer and the output of the pre-fusion layer in a global residual connecting mode.
Further, training the hyperspectral image fusion network model according to the visible face preprocessing training image set and the infrared face preprocessing training image set to obtain a trained hyperspectral image fusion network model, comprising:
constructing a composite loss function based on structural loss, pixel loss and average gradient loss;
and training the hyperspectral image fusion network model according to the visible light face preprocessing training image set and the infrared face preprocessing training image set by utilizing a composite loss function based on structural loss, pixel loss and average gradient loss to obtain the trained hyperspectral image fusion network model.
And 4, inputting the visible light face test image set and the infrared face test image set into the trained hyperspectral image fusion network model to obtain a hyperspectral face image set, and dividing the hyperspectral face image set into a hyperspectral face training image set and a hyperspectral face test image set.
And 5, constructing a convolutional neural network face recognition model, and training the convolutional neural network face recognition model according to the hyper-spectral face training image set to obtain the trained convolutional neural network face recognition model.
Specifically, the convolutional neural network face recognition model constructed in step 5 of this embodiment includes a ResNet feature extraction module, a feature normalization module, and a feature space mapping module, which are connected in sequence.
Further, training the convolutional neural network face recognition model according to the hyper-spectral face training image set to obtain a trained convolutional neural network face recognition model, comprising:
constructing a triple loss function Triplet loss;
and training the convolutional neural network face recognition model by utilizing a triple loss function triple loss according to the hyper-spectral face training image set to obtain the trained convolutional neural network face recognition model.
And 6, inputting the hyper-spectral face test image set into a trained convolutional neural network face recognition model to obtain a face feature set, and classifying the face feature set by using a support vector machine classifier to realize hyper-spectral face recognition.
The computer-readable storage medium provided in this embodiment may implement the above-mentioned hyper-spectrum face recognition method embodiment, the above-mentioned hyper-spectrum face recognition apparatus embodiment, and the above-mentioned hyper-spectrum face recognition electronic device embodiment, and the implementation principle and technical effect thereof are similar, and details are not repeated herein.
The foregoing is a more detailed description of the invention in connection with specific preferred embodiments and it is not intended that the invention be limited to these specific details. For those skilled in the art to which the invention pertains, several simple deductions or substitutions can be made without departing from the spirit of the invention, and all shall be considered as belonging to the protection scope of the invention.

Claims (7)

1. A hyper-spectral face recognition method is characterized by comprising the following steps:
acquiring a visible light face image set and an infrared face image set, and dividing the visible light face image set into a visible light face training image set and a visible light face testing image set, wherein the infrared face image set is divided into an infrared face training image set and an infrared face testing image set;
respectively preprocessing the visible face training image set, the visible face testing image set, the infrared face training image set and the infrared face testing image set to obtain a visible face preprocessing training image set, a visible face preprocessing testing image set, an infrared face preprocessing training image set and an infrared face preprocessing testing image set;
constructing a hyperspectral image fusion network model, and training the hyperspectral image fusion network model according to the visible light face preprocessing training image set and the infrared face preprocessing training image set to obtain a trained hyperspectral image fusion network model;
inputting the visible light face test image set and the infrared face test image set into the trained hyperspectral image fusion network model to obtain a hyperspectral face image set, and dividing the hyperspectral face image set into a hyperspectral face training image set and a hyperspectral face test image set;
constructing a convolutional neural network face recognition model, and training the convolutional neural network face recognition model according to the hyper-spectral face training image set to obtain a trained convolutional neural network face recognition model;
inputting the hyper-spectral face test image set into the trained convolutional neural network face recognition model to obtain a face feature set, and classifying the face feature set by using a support vector machine classifier to realize hyper-spectral face recognition;
the constructed hyper-spectral image fusion network model comprises a pre-fusion layer, an encoder module, a fusion layer and a decoder module, wherein,
the input of the pre-fusion layer is connected with an input image, the output of the pre-fusion layer is connected with the input of the encoder module, the encoder module comprises a first convolution layer and a dense residual module which are sequentially connected, the output of the dense residual module and the output of the pre-fusion layer are subjected to global residual connection output, and the global residual connection output is connected with the input of the fusion layer;
the output of the fusion layer is connected with the input of the decoder module, the decoder module comprises second to fifth convolution layers and a feedback layer which are sequentially connected, and the output of the feedback layer and the output of the fusion layer are input into the second convolution layer again to form feedback connection;
the dense residual module comprises a first dense residual connecting layer, a third dense residual connecting layer, a multi-scale splicing layer and a fourth dense residual connecting layer which are sequentially connected, wherein the input of the first dense residual connecting layer is further connected with the output of the first dense residual connecting layer, the output of the second dense residual connecting layer and the output of the third dense residual connecting layer, the input of the second dense residual connecting layer is further connected with the output of the second dense residual connecting layer and the output of the third dense residual connecting layer, the input of the third dense residual connecting layer is further connected with the output of the third dense residual connecting layer, the output of the fourth dense residual connecting layer is further subjected to local residual connecting output with the input of the first convolution layer, and the local residual connecting output is subjected to global residual connecting output with the output of the pre-fusion layer;
training the hyperspectral image fusion network model according to the visible face preprocessing training image set and the infrared face preprocessing training image set to obtain a trained hyperspectral image fusion network model, and the method comprises the following steps:
constructing a composite loss function based on structural loss, pixel loss and average gradient loss;
and training the hyperspectral image fusion network model according to the visible face preprocessing training image set and the infrared face preprocessing training image set by using the composite loss function based on structural loss, pixel loss and average gradient loss to obtain the trained hyperspectral image fusion network model.
2. The hyperspectral face recognition method according to claim 1, wherein the visible light face training image set, the visible light face testing image set, the infrared face training image set and the infrared face testing image set are respectively preprocessed to obtain a visible light face preprocessing training image set, a visible light face preprocessing testing image set, an infrared face preprocessing training image set and an infrared face preprocessing testing image set, and the hyperspectral face recognition method comprises the following steps:
respectively carrying out gray level conversion and normalization processing on the visible light face training image set and the visible light face testing image set to obtain a visible light face preprocessing training image set and a visible light face preprocessing testing image set;
and respectively carrying out image enhancement and normalization processing on the infrared face training image set and the infrared face testing image set to obtain an infrared face preprocessing training image set and an infrared face preprocessing testing image set.
3. The hyper-spectral face recognition method according to claim 1, wherein the constructed convolutional neural network face recognition model comprises a ResNet feature extraction module, a feature normalization module and a feature space mapping module which are connected in sequence.
4. The hyperspectral face recognition method according to claim 1, wherein training the convolutional neural network face recognition model according to the hyperspectral face training image set to obtain a trained convolutional neural network face recognition model, comprises:
constructing triple loss functions Tripletloss;
and training the convolutional neural network face recognition model according to a hyper-spectral face training image set by utilizing the triple loss function triple loss to obtain a trained convolutional neural network face recognition model.
5. A hyper-spectral face recognition apparatus, the apparatus comprising:
the data acquisition module is used for acquiring a visible light face image set and an infrared face image set, dividing the visible light face image set into a visible light face training image set and a visible light face testing image set, and dividing the infrared face image set into an infrared face training image set and an infrared face testing image set;
the data preprocessing module is used for respectively preprocessing the visible light face training image set, the visible light face testing image set, the infrared face training image set and the infrared face testing image set to obtain a visible light face preprocessing training image set, a visible light face preprocessing testing image set, an infrared face preprocessing training image set and an infrared face preprocessing testing image set;
the first model construction training module is used for constructing a hyperspectral image fusion network model, and training the hyperspectral image fusion network model according to the visible light face preprocessing training image set and the infrared face preprocessing training image set to obtain a trained hyperspectral image fusion network model;
the data generation module is used for inputting the visible light face test image set and the infrared face test image set into the trained hyperspectral image fusion network model to obtain a hyperspectral face image set, and dividing the hyperspectral face image set into a hyperspectral face training image set and a hyperspectral face test image set;
the second model construction training module is used for constructing a convolutional neural network face recognition model, and training the convolutional neural network face recognition model according to the hyper-spectral face training image set to obtain a trained convolutional neural network face recognition model;
and the data identification module is used for inputting the hyper-spectral face test image set into the trained convolutional neural network face identification model to obtain a face feature set, and classifying the face feature set by using a support vector machine classifier to realize hyper-spectral face identification.
6. The hyper-spectrum face recognition electronic equipment is characterized by comprising an image collector, a display, a processor, a communication interface, a memory and a communication bus, wherein the image collector, the display, the processor, the communication interface and the memory finish mutual communication through the communication bus;
the image collector is used for collecting image data;
the display is used for displaying the image identification data;
the memory is used for storing a computer program;
the processor is configured to implement the hyper-spectral face recognition method according to any one of claims 1 to 4 when executing the computer program stored in the memory.
7. A computer-readable storage medium, in which a computer program is stored, which, when being executed by a processor, implements the hyper-spectral face recognition method according to any one of claims 1 to 4.
CN202010163505.5A 2020-03-10 2020-03-10 Hyper-spectrum face recognition method and device, electronic equipment and storage medium thereof Active CN111539247B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010163505.5A CN111539247B (en) 2020-03-10 2020-03-10 Hyper-spectrum face recognition method and device, electronic equipment and storage medium thereof

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010163505.5A CN111539247B (en) 2020-03-10 2020-03-10 Hyper-spectrum face recognition method and device, electronic equipment and storage medium thereof

Publications (2)

Publication Number Publication Date
CN111539247A CN111539247A (en) 2020-08-14
CN111539247B true CN111539247B (en) 2023-02-10

Family

ID=71974802

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010163505.5A Active CN111539247B (en) 2020-03-10 2020-03-10 Hyper-spectrum face recognition method and device, electronic equipment and storage medium thereof

Country Status (1)

Country Link
CN (1) CN111539247B (en)

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112016478B (en) * 2020-08-31 2024-04-16 中国电子科技集团公司第三研究所 Complex scene recognition method and system based on multispectral image fusion
CN111986240A (en) * 2020-09-01 2020-11-24 交通运输部水运科学研究所 Drowning person detection method and system based on visible light and thermal imaging data fusion
CN112800921A (en) * 2021-01-22 2021-05-14 广东电网有限责任公司广州供电局 Multi-band light source face recognition method based on convolutional neural network
CN113723246B (en) * 2021-08-20 2022-09-02 南京华图信息技术有限公司 Attention mechanism-based infrared wide-spectrum face recognition method and system
CN113673487A (en) * 2021-10-21 2021-11-19 杭州觅睿科技股份有限公司 Pet identification method, device, equipment and storage medium
CN113688806A (en) * 2021-10-26 2021-11-23 南京智谱科技有限公司 Infrared and visible light image fused multispectral target detection method and system
CN114694214B (en) * 2022-03-16 2024-08-27 西北工业大学 Cross-modal infrared polarization face recognition method based on codec
CN115240042B (en) * 2022-07-05 2023-05-16 抖音视界有限公司 Multi-mode image recognition method and device, readable medium and electronic equipment

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105608450A (en) * 2016-03-01 2016-05-25 天津中科智能识别产业技术研究院有限公司 Heterogeneous face identification method based on deep convolutional neural network
CN108427921A (en) * 2018-02-28 2018-08-21 辽宁科技大学 A kind of face identification method based on convolutional neural networks
WO2019128367A1 (en) * 2017-12-26 2019-07-04 广州广电运通金融电子股份有限公司 Face verification method and apparatus based on triplet loss, and computer device and storage medium
CN110110606A (en) * 2019-04-12 2019-08-09 天津大学 The fusion method of visible light neural network based and infrared face image

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7602942B2 (en) * 2004-11-12 2009-10-13 Honeywell International Inc. Infrared and visible fusion face recognition system

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105608450A (en) * 2016-03-01 2016-05-25 天津中科智能识别产业技术研究院有限公司 Heterogeneous face identification method based on deep convolutional neural network
WO2019128367A1 (en) * 2017-12-26 2019-07-04 广州广电运通金融电子股份有限公司 Face verification method and apparatus based on triplet loss, and computer device and storage medium
CN108427921A (en) * 2018-02-28 2018-08-21 辽宁科技大学 A kind of face identification method based on convolutional neural networks
CN110110606A (en) * 2019-04-12 2019-08-09 天津大学 The fusion method of visible light neural network based and infrared face image

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
基于决策融合的红外与可见光图像人脸识别研究;赵云丰等;《激光与红外》;20080620(第06期);全文 *
基于多特征融合CNN的人脸识别算法研究;罗金梅等;《航空计算技术》;20190525(第03期);全文 *

Also Published As

Publication number Publication date
CN111539247A (en) 2020-08-14

Similar Documents

Publication Publication Date Title
CN111539247B (en) Hyper-spectrum face recognition method and device, electronic equipment and storage medium thereof
CN112766160A (en) Face replacement method based on multi-stage attribute encoder and attention mechanism
Bhalla et al. Fusion of Infrared and Visible Images Using Fuzzy Based Siamese Convolutional Network.
CN109242888B (en) Infrared and visible light image fusion method combining image significance and non-subsampled contourlet transformation
CN108573276A (en) A kind of change detecting method based on high-resolution remote sensing image
CN109871845B (en) Certificate image extraction method and terminal equipment
CN111539246B (en) Cross-spectrum face recognition method and device, electronic equipment and storage medium thereof
CN112115979B (en) Fusion method and device of infrared image and visible image
CN108197669B (en) Feature training method and device of convolutional neural network
CN110276831B (en) Method and device for constructing three-dimensional model, equipment and computer-readable storage medium
CN110472495B (en) Deep learning face recognition method based on graphic reasoning global features
CN110825900A (en) Training method of feature reconstruction layer, reconstruction method of image features and related device
CN114782298B (en) Infrared and visible light image fusion method with regional attention
CN111242868B (en) Image enhancement method based on convolutional neural network in scotopic vision environment
CN116757986A (en) Infrared and visible light image fusion method and device
CN114332942A (en) Night infrared pedestrian detection method and system based on improved YOLOv3
CN110969164A (en) Low-illumination imaging license plate recognition method and device based on deep learning end-to-end
CN116883303A (en) Infrared and visible light image fusion method based on characteristic difference compensation and fusion
CN113378672A (en) Multi-target detection method for defects of power transmission line based on improved YOLOv3
Babu et al. An efficient image dahazing using Googlenet based convolution neural networks
Chen et al. SFCFusion: Spatial-Frequency Collaborative Infrared and Visible Image Fusion
CN114882537A (en) Finger new visual angle image generation method based on nerve radiation field
CN114155165A (en) Image defogging method based on semi-supervision
CN117496019B (en) Image animation processing method and system for driving static image
CN110490053B (en) Human face attribute identification method based on trinocular camera depth estimation

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
CP03 Change of name, title or address
CP03 Change of name, title or address

Address after: No.2, Taibai South Road, Yanta District, Xi'an City, Shaanxi Province

Patentee after: XIDIAN University

Country or region after: China

Patentee after: Xi'an Xiyue Xin'an Intelligent Technology Co.,Ltd.

Address before: No.2, Taibai South Road, Yanta District, Xi'an City, Shaanxi Province

Patentee before: XIDIAN University

Country or region before: China

Patentee before: Xi'an XD Xin'an Intelligent Technology Co.,Ltd.

TR01 Transfer of patent right
TR01 Transfer of patent right

Effective date of registration: 20240122

Address after: No.2, Taibai South Road, Yanta District, Xi'an City, Shaanxi Province

Patentee after: XIDIAN University

Country or region after: China

Address before: No.2, Taibai South Road, Yanta District, Xi'an City, Shaanxi Province

Patentee before: XIDIAN University

Country or region before: China

Patentee before: Xi'an Xiyue Xin'an Intelligent Technology Co.,Ltd.

TR01 Transfer of patent right
TR01 Transfer of patent right

Effective date of registration: 20240412

Address after: Room 303-9, 3rd Floor, Building 1, No. 88 Huilong Road, Qingpu District, Shanghai, 201702

Patentee after: Shanghai Jingying Information Technology Co.,Ltd.

Country or region after: China

Address before: No.2, Taibai South Road, Yanta District, Xi'an City, Shaanxi Province

Patentee before: XIDIAN University

Country or region before: China