CN117152397B - Three-dimensional face imaging method and system based on thermal imaging projection - Google Patents
Three-dimensional face imaging method and system based on thermal imaging projection Download PDFInfo
- Publication number
- CN117152397B CN117152397B CN202311393671.4A CN202311393671A CN117152397B CN 117152397 B CN117152397 B CN 117152397B CN 202311393671 A CN202311393671 A CN 202311393671A CN 117152397 B CN117152397 B CN 117152397B
- Authority
- CN
- China
- Prior art keywords
- thermal imaging
- rgb
- data set
- depth
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000001931 thermography Methods 0.000 title claims abstract description 139
- 238000003384 imaging method Methods 0.000 title claims abstract description 34
- 238000013135 deep learning Methods 0.000 claims abstract description 24
- 238000000034 method Methods 0.000 claims abstract description 17
- 238000007781 pre-processing Methods 0.000 claims abstract description 17
- 238000007689 inspection Methods 0.000 claims description 29
- 238000012549 training Methods 0.000 claims description 19
- 238000012545 processing Methods 0.000 claims description 11
- 230000001502 supplementing effect Effects 0.000 claims description 11
- 238000004590 computer program Methods 0.000 claims description 9
- 239000002184 metal Substances 0.000 claims description 6
- 230000005284 excitation Effects 0.000 claims description 5
- 238000013507 mapping Methods 0.000 claims description 5
- 238000000605 extraction Methods 0.000 claims description 3
- 238000001914 filtration Methods 0.000 claims description 3
- 238000005457 optimization Methods 0.000 claims description 3
- 238000005070 sampling Methods 0.000 claims description 3
- 230000001815 facial effect Effects 0.000 abstract description 5
- 238000003745 diagnosis Methods 0.000 abstract description 4
- 230000006870 function Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 2
- 239000003814 drug Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 239000003086 colorant Substances 0.000 description 1
- 238000003331 infrared imaging Methods 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/20—Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/10—Image acquisition
- G06V10/12—Details of acquisition arrangements; Constructional details thereof
- G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/778—Active pattern-learning, e.g. online learning of image or video features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/161—Detection; Localisation; Normalisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/172—Classification, e.g. identification
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Multimedia (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Human Computer Interaction (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Computer Graphics (AREA)
- Computer Hardware Design (AREA)
- Architecture (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computing Systems (AREA)
- Evolutionary Computation (AREA)
- Medical Informatics (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a three-dimensional face imaging method and system based on thermal imaging projection. The method comprises the following steps: acquiring a data set, marking the data set, preprocessing the data set, identifying by using a deep learning network model, judging whether the data set is credible, if so, performing texture filling, and if not, re-acquiring the image and repeating the process. The acquired facial images are comprehensively identified in a mode that the depth camera, the RGB camera and the thermal imaging camera are respectively combined with the corresponding deep learning network, so that the actual facial imaging result of the thermal imaging image reflected by the mirror surface is avoided, the accuracy of facial imaging is improved, and the success rate of facial diagnosis is improved.
Description
Technical Field
The invention relates to the technical field of image processing, in particular to a three-dimensional face imaging method and system based on thermal imaging projection.
Background
When the inspection is performed in the traditional Chinese medicine diagnosis and treatment, different facial colors can be changed when different lamplight irradiates, so that a standard light source is required to be arranged in the multidimensional inspection acquisition box, the quality of pictures shot by an internal camera is ensured not to be influenced by external environment, no dead angle exists in the inspection acquisition box when the light source irradiates, and the phenomena of shadow and unbalanced reflection cannot occur.
The existing traditional Chinese medicine inspection acquisition equipment is characterized in that an RGB camera is arranged in an acquisition box to shoot a high-definition color two-dimensional plane image of a target tongue surface and is used for clinically analyzing the texture characteristics of the target, and the three-dimensional characteristics of the target tongue surface and the temperature information thereof cannot be intuitively reflected due to the fact that the texture image shot by the RGB camera is poor in depth position information accuracy.
The existing acquisition box is characterized in that RGB camera shooting target clear color images are arranged in a box body and used for building a front base model, but because texture images shot by the RGB cameras are poor in depth position information precision, only approximate outlines can be obtained, local outlines of users cannot be clearly displayed, and therefore formed three-dimensional imaging is inaccurate.
To solve this problem, in the prior art, for example, patent application number CN202010932310.2, entitled 3D camera and infrared scanning algorithm with point cloud aligned with color textures. The method is applicable to conventional three-dimensional imaging, and when infrared imaging and color textures are aligned during inspection image processing, thermal imaging formed in an inspection box by a user is a set of self thermal imaging and thermal imaging of specular reflection in the inspection box; therefore, the method cannot be directly used for texture filling, so that the problem of inaccurate imaging data of thermal imaging in a inspection box and the problem of how to use mapping data of thermal imaging to enable the imaging data to be combined with color textures to form accurate three-dimensional face imaging are the urgent problems to be solved in the application.
Disclosure of Invention
Therefore, the invention aims to provide a three-dimensional face imaging method and system based on thermal imaging projection, which comprehensively identifies the acquired face images by adopting a mode of combining a depth camera, an RGB camera and a thermal imaging camera with a corresponding deep learning network, avoids the actual face imaging result of the thermal imaging image reflected by a mirror surface, improves the face imaging accuracy and improves the success rate of face diagnosis.
In order to achieve the above object, the present invention provides a three-dimensional face imaging method based on thermal imaging projection, comprising the following steps:
acquiring RGB images, depth images and thermal imaging images of multiple frames of the same human face in a uniform light supplementing environment;
marking the trusted region and the untrusted region of the acquired RGB image, depth image and thermal imaging image of the face to form an RGB mark data set, a depth mark data set and a thermal imaging mark data set;
preprocessing the marked data sets respectively;
extracting the face outline and the corresponding pixel position information in the trusted region from the preprocessed data set to serve as a forward learning data set; extracting the object outline and pixel position information of the unreliable region as a corresponding reverse learning data set;
constructing a deep learning network model, wherein the deep learning network model comprises three independent deep learning networks, namely an RGB network, a deep network and a thermal imaging network;
inputting an RGB image, an RGB mark data set, an RGB forward learning data set and an RGB reverse learning data set into an RGB network, performing depth training, and outputting RGB credibility;
the depth training process comprises the following steps: recognizing the position of a face region in an RGB image, performing two classifications on the acquired RBG image according to the learning of the RGB forward learning data set and the RGB reverse learning data set, and calculating the final RGB credibility as a binary label;
according to the depth training process, the depth image, the depth marking data set, the depth forward learning data set and the depth reverse learning data set are input into a depth network, and the depth credibility is output;
inputting the thermal imaging image, the thermal imaging marking data set, the thermal imaging forward learning data set and the thermal imaging reverse learning data set into a thermal imaging network, and outputting thermal imaging credibility;
when all the obtained three credibility values meet the correspondingly set credibility threshold values, filling RGB features extracted by an RGB network and temperature feature data extracted by a thermal imaging network into a three-dimensional face model; the three-dimensional face model is composed of three-dimensional space features extracted by a depth network.
Further preferably, the performing trusted region and untrusted region marking includes:
marking real face areas in the RGB image, the depth image and the thermal imaging image as trusted areas; and marking the mirror image projection area of the human face in the same image as an unreliable area to obtain an RGB mark data set, a depth mark data set and a thermal imaging mark data set.
Further preferably, the preprocessing is performed on the marked data sets respectively, including gray processing, binarization, pixel enhancement, filtering and ecology closing operation are sequentially performed on the marked RGB mark data set, the marked depth mark data set and the marked thermal imaging data set; three pre-processed data sets are formed.
Further preferably, the RGB network, the depth network and the thermal imaging network each comprise an input layer, an output layer, 8 convolution layers for feature extraction, 8 sampling layers for feature optimization selection, 2 full connection layers for feature representation, 1 excitation layer for fast convergence and 1 loss layer for calculating output and target loss values.
Further preferably, before the multi-frame face image is acquired in the uniform light supplementing environment, unified calibration is performed on all cameras, and the calibration process comprises the following steps:
all cameras shoot a metal cube with uniform temperature in the same acquisition space, and corner point calibration points are stuck on the metal cube;
the camera coordinates of all cameras are converted into world coordinates,
establishing a three-dimensional cube according to coordinates of corner calibration points in a depth image acquired in a depth camera;
and positioning and mapping the RGB image acquired by the RGB camera and the thermal imaging image acquired by the infrared camera on the established three-dimensional cube by taking the coordinates of the corner calibration points in the depth image as the reference.
The invention also provides a three-dimensional face imaging system based on thermal imaging projection, which comprises a inspection box, wherein a plurality of cameras are arranged in the inspection box, and the plurality of cameras comprise an RGB camera, a depth camera and a thermal imaging camera; each camera is connected with the central processing unit;
the RGB camera is used for a depth camera and a thermal imaging camera and is respectively used for acquiring RGB images, depth images and thermal imaging images of multiple frames of the same human face in a uniform light supplementing environment;
the central processing unit comprises an image preprocessing module and a deep learning network model;
the image preprocessing module is used for marking the acquired RGB image, depth image and thermal imaging image of the face in a trusted area and an untrusted area to form an RGB mark data set, a depth mark data set and a thermal imaging mark data set; preprocessing the marked data sets respectively; extracting the face outline and the corresponding pixel position information in the trusted region from the preprocessed data set to serve as a forward learning data set; extracting the object outline and pixel position information of the unreliable region as a corresponding reverse learning data set;
the deep learning network model comprises three independent deep learning networks, namely an RGB network, a deep network and a thermal imaging network;
inputting an RGB image, an RGB mark data set, an RGB forward learning data set and an RGB reverse learning data set into an RGB network, performing depth training, and outputting RGB credibility;
inputting the depth image, the depth mark data set, the depth forward learning data set and the depth reverse learning data set into a depth network, performing depth training, and outputting depth credibility;
inputting the thermal imaging image, the thermal imaging marking data set, the thermal imaging forward learning data set and the thermal imaging reverse learning data set into a thermal imaging network for deep training, and outputting the thermal imaging credibility;
when all the obtained three credibility values meet the correspondingly set credibility threshold values, filling RGB features extracted by an RGB network and temperature feature data extracted by a thermal imaging network into a three-dimensional face model; the three-dimensional face model is composed of three-dimensional space features extracted by a depth network.
Further preferably, an arc-shaped panel is arranged in the inspection box, and a uniform light supplementing lamp is arranged behind the arc-shaped panel.
Further preferably, the RGB camera includes an RGB texture camera disposed at the center of the inspection box;
the depth camera comprises a left binocular camera and a right binocular camera which are arranged at the left side and the right side of the inspection box;
the thermal imaging cameras comprise left thermal imaging cameras and right thermal imaging cameras which are arranged on the left side and the right side of the inspection box;
wherein, each side camera of depth camera and thermal imaging camera all is equipped with gesture control mechanism.
The invention also provides electronic equipment, which comprises a processor and a memory, wherein the memory is used for storing computer program instructions, the processor is used for executing the computer program instructions stored in the memory, and when the program instructions are executed, the steps of the three-dimensional face imaging method based on thermal imaging projection are executed.
The present invention also provides a computer storage medium having stored thereon a computer program which, when executed, implements the steps of the three-dimensional face imaging method based on thermal imaging projection as described above.
According to the three-dimensional face imaging method and system based on thermal imaging projection, the acquired face images are comprehensively identified in a mode of combining a depth camera, an RGB camera and a thermal imaging camera with a corresponding deep learning network, actual face imaging results of the thermal imaging images reflected by the mirror surfaces are avoided, face imaging accuracy is improved, and success rate of face diagnosis is improved.
Drawings
Fig. 1 is a schematic flow chart of a three-dimensional face imaging method based on thermal imaging projection.
Fig. 2 is a schematic structural diagram of a three-dimensional face imaging system based on thermal imaging projection provided by the invention.
Fig. 3 is a network structure diagram of a depth recognition network provided by the present invention.
Fig. 4 is a schematic view of the external structure of the inspection box.
Fig. 5 is a schematic view of the internal structure of the inspection box.
In the figure: 1. inspection box; 2. an RGB camera; 5. a central processing unit; 301. a left binocular camera; 302. a right binocular camera; 401. a left thermal imaging camera; 402. a right thermal imaging camera.
Detailed Description
The invention is described in further detail below with reference to the drawings and the detailed description.
As shown in fig. 1-3, a three-dimensional face imaging method based on thermal imaging projection according to an embodiment of the present invention includes the following steps:
s1, acquiring a data set; acquiring RGB images, depth images and thermal imaging images of multiple frames of the same human face in a uniform light supplementing environment;
s2, marking a data set; marking the trusted region and the untrusted region of the acquired RGB image, depth image and thermal imaging image of the face to form an RGB mark data set, a depth mark data set and a thermal imaging mark data set;
further preferably, performing trusted region and untrusted region labeling includes:
marking real face areas in the RGB image, the depth image and the thermal imaging image as trusted areas; and marking the mirror image projection area of the human face in the same image as an unreliable area to obtain an RGB mark data set, a depth mark data set and a thermal imaging mark data set.
S3, preprocessing a data set; preprocessing the marked data sets respectively;
further preferably, the preprocessing is performed on the marked data sets respectively, including gray processing, binarization, pixel enhancement, filtering and ecology closing operation are sequentially performed on the marked RGB mark data set, the marked depth mark data set and the marked thermal imaging data set; three pre-processed data sets are formed.
Extracting the face outline and the corresponding pixel position information in the trusted region from the preprocessed data set to serve as a forward learning data set; extracting the object outline and pixel position information of the unreliable region as a corresponding reverse learning data set;
s4, identifying by using the deep learning network model.
Constructing a deep learning network model, wherein the deep learning network model comprises three independent deep learning networks, namely an RGB network, a deep network and a thermal imaging network;
inputting an RGB image, an RGB mark data set, an RGB forward learning data set and an RGB reverse learning data set into an RGB network, performing depth training, and outputting RGB credibility;
inputting the depth image, the depth mark data set, the depth forward learning data set and the depth reverse learning data set into a depth network, performing depth training, and outputting depth credibility;
inputting the thermal imaging image, the thermal imaging marking data set, the thermal imaging forward learning data set and the thermal imaging reverse learning data set into a thermal imaging network for deep training, and outputting the thermal imaging credibility;
s5, judging whether the image is credible, if so, performing texture filling, and if not, re-acquiring the image and repeating the process.
When all the obtained three credibility values meet the set credibility threshold value, filling RGB features extracted by an RGB network and temperature feature data extracted by a thermal imaging network into a three-dimensional face model; the three-dimensional face model is composed of three-dimensional space features extracted by a depth network.
As shown in fig. 3, it is further preferred that the RGB network, the depth network, and the thermal imaging network each include an input layer, an output layer, 8 convolution layers for feature extraction, 8 sampling layers for feature optimization selection, 2 full connection layers for representing features, 1 excitation layer for rapid convergence, and 1 loss layer for calculating output and target loss values.
The excitation function is as follows:
wherein z is an input parameter of the convolutional layer; the excitation function indicates that when the parameters input by the input layer are extracted to the convolution layer, the convolution layer is activated to start extracting the characteristic parameters.
The application can adopt CIOU function as loss function for identifying the position of human face
Wherein, IOU represents the calculated value of the IOU loss function, and the area where the two boxes intersect/the area where the two boxes merge; wherein the two boxes are the true position box (typically indicated by a green box) and the predicted position box (typically indicated by a blue box) of the target, respectively. ρ represents the center distance of two frames, b represents the parameter of the predicted center coordinates, i.e. the center of the blue frame, b gt A parameter representing the center of the true target bounding box, i.e., the green box center point.
Alpha andthe aspect ratio of the true position frame selection area and the aspect ratio of the predicted position frame selection area are respectively, wherein w, h and w are expressed in the formula gt 、h gt Representing the height width of the predicted frame and the height width of the real frame, respectively.
For example, inputting an RGB image, an RGB mark data set, an RGB forward learning data set and an RGB reverse learning data set into an RGB network, performing depth training, and outputting RGB credibility;
and recognizing the position of a face region in the RGB image by using a CIOU function, performing two classifications on the acquired RBG image by using the deep learning network according to the learning of the RGB forward learning data set and the RGB reverse learning data set, namely judging the real face and judging the non-real face on the position of the recognized face region, marking the reliable region as 1 when the recognized face belongs to the real face, marking the non-real face as 0 when the recognized face belongs to the non-real face, and calculating the final RGB credibility by taking the reliable region as a binary label.
For example, using BCE loss function to calculate confidence
Wherein y is i Is a binary tag (with a value of 0 or 1), and P (x) is the tag probability of output belonging to y. A lower LOSS value indicates a higher degree of confidence in the image.
Similarly, the depth credibility and the thermal imaging credibility can be calculated according to the above process, and finally when the obtained three credibility meet the set credibility threshold, namely, when the RGB credibility meets the RGB credibility threshold, the depth credibility meets the depth credibility threshold and the thermal imaging credibility meets the thermal imaging credibility threshold, mapping can be performed at this time, namely, RGB features extracted by an RGB network and temperature feature data extracted by a thermal imaging network are filled into the three-dimensional face model.
It should be noted that, before the multi-frame face image is acquired in the uniform light supplementing environment, the method further comprises the step of uniformly calibrating all cameras, and the calibration process comprises the following steps:
all cameras shoot a metal cube with uniform temperature in the same acquisition space, and corner point calibration points are stuck on the metal cube;
converting camera coordinates of all cameras into world coordinates;
establishing a three-dimensional cube according to coordinates of corner calibration points in a depth image acquired in a depth camera;
and positioning and mapping the RGB image acquired by the RGB camera and the thermal imaging image acquired by the infrared camera on the established three-dimensional cube by taking the coordinates of the corner calibration points in the depth image as the reference.
The invention also provides a three-dimensional face imaging system based on thermal imaging projection, which comprises a inspection box 1, wherein a plurality of cameras are arranged in the inspection box, and the plurality of cameras comprise an RGB camera 2, a depth camera and a thermal imaging camera, as shown in fig. 5; as shown in fig. 2, each camera is connected to the central processing unit 5; the depth camera comprises a left binocular camera 301 and a right binocular camera 302 which are arranged at the left side and the right side of the inspection box;
the thermal imaging cameras comprise a left thermal imaging camera 401 and a right thermal imaging camera 402 which are arranged at the left side and the right side of the inspection box; wherein, every side camera of depth camera and thermal imaging camera all is equipped with gesture control mechanism.
The RGB camera is used for a depth camera and a thermal imaging camera and is respectively used for acquiring RGB images, depth images and thermal imaging images of multiple frames of the same human face in a uniform light supplementing environment;
the central processing unit comprises an image preprocessing module and a deep learning network model;
the image preprocessing module is used for marking the acquired RGB image, depth image and thermal imaging image of the face in a trusted area and an untrusted area to form an RGB mark data set, a depth mark data set and a thermal imaging mark data set; preprocessing the marked data sets respectively; extracting the face outline and the corresponding pixel position information in the trusted region from the preprocessed data set to serve as a forward learning data set; extracting the object outline and pixel position information of the unreliable region as a corresponding reverse learning data set;
the deep learning network model comprises three independent deep learning networks, namely an RGB network, a deep network and a thermal imaging network;
inputting an RGB image, an RGB mark data set, an RGB forward learning data set and an RGB reverse learning data set into an RGB network, performing depth training, and outputting RGB credibility;
inputting the depth image, the depth mark data set, the depth forward learning data set and the depth reverse learning data set into a depth network, performing depth training, and outputting depth credibility;
inputting the thermal imaging image, the thermal imaging marking data set, the thermal imaging forward learning data set and the thermal imaging reverse learning data set into a thermal imaging network for deep training, and outputting the thermal imaging credibility;
when all the obtained three credibility values meet the set credibility threshold value, filling RGB features extracted by an RGB network and temperature feature data extracted by a thermal imaging network into a three-dimensional face model; the three-dimensional face model is composed of three-dimensional space features extracted by a depth network.
Further preferably, an arc-shaped panel is arranged in the inspection box, and a uniform light supplementing lamp is arranged behind the arc-shaped panel.
Further preferably, the RGB camera includes an RGB texture camera disposed at the center of the inspection box;
the invention also provides electronic equipment, which comprises a processor and a memory, wherein the memory is used for storing computer program instructions, the processor is used for executing the computer program instructions stored in the memory, and when the program instructions are executed, the steps of the three-dimensional face imaging method based on thermal imaging projection are executed.
The present invention also provides a computer storage medium having stored thereon a computer program which, when executed, implements the steps of the three-dimensional face imaging method based on thermal imaging projection as described above.
It is apparent that the above examples are given by way of illustration only and are not limiting of the embodiments. Other variations or modifications of the above teachings will be apparent to those of ordinary skill in the art. It is not necessary here nor is it exhaustive of all embodiments. While still being apparent from variations or modifications that may be made by those skilled in the art are within the scope of the invention.
Claims (9)
1. The three-dimensional face imaging method based on thermal imaging projection is characterized by comprising the following steps of:
acquiring RGB images, depth images and thermal imaging images of multiple frames of the same human face in a uniform light supplementing environment;
the method comprises the steps of marking an acquired RGB image, depth image and thermal imaging image of a face with a trusted area and an untrusted area, wherein the marking of a real face area in the RGB image, the depth image and the thermal imaging image as a trusted area; marking the mirror image projection area of the face in the same image as an unreliable area to form an RGB mark data set, a depth mark data set and a thermal imaging mark data set;
preprocessing the marked data sets respectively;
extracting the face outline and the corresponding pixel position information in the trusted region from the preprocessed data set to serve as a forward learning data set; extracting the object outline and pixel position information of the unreliable region as a corresponding reverse learning data set;
constructing a deep learning network model, wherein the deep learning network model comprises three independent deep learning networks, namely an RGB network, a deep network and a thermal imaging network;
inputting an RGB image, an RGB mark data set, an RGB forward learning data set and an RGB reverse learning data set into an RGB network, performing depth training, and outputting RGB credibility;
inputting the depth image, the depth mark data set, the depth forward learning data set and the depth reverse learning data set into a depth network, performing depth training, and outputting depth credibility;
inputting the thermal imaging image, the thermal imaging marking data set, the thermal imaging forward learning data set and the thermal imaging reverse learning data set into a thermal imaging network for deep training, and outputting the thermal imaging credibility;
when all the obtained three credibility values meet the correspondingly set credibility threshold values, filling RGB features extracted by an RGB network and temperature feature data extracted by a thermal imaging network into a three-dimensional face model; the three-dimensional face model is composed of three-dimensional space features extracted by a depth network.
2. The three-dimensional face imaging method based on thermal imaging projection according to claim 1, wherein preprocessing is performed on the marked data sets respectively, and comprises sequentially performing gray processing, binarization, pixel enhancement, filtering and ecological closure operation on the marked RGB mark data set, the depth mark data set and the thermal imaging mark data set; three pre-processed data sets are formed.
3. The three-dimensional face imaging method based on thermal imaging projection of claim 1, wherein each of the RGB, depth and thermal imaging networks comprises an input layer, an output layer, 8 convolution layers for feature extraction, 8 sampling layers for feature optimization selection, 2 full connection layers for representing features, 1 excitation layer for rapid convergence, and 1 loss layer for calculating output and target loss values.
4. The three-dimensional face imaging method based on thermal imaging projection of claim 1, further comprising uniformly calibrating all cameras before acquiring the multi-frame face image in the uniform light supplementing environment, wherein the calibration process comprises:
all cameras shoot a metal cube with uniform temperature in the same acquisition space, and corner point calibration points are stuck on the metal cube;
the camera coordinates of all cameras are converted into world coordinates,
establishing a three-dimensional cube according to coordinates of corner calibration points in a depth image acquired in a depth camera;
and positioning and mapping the RGB image acquired by the RGB camera and the thermal imaging image acquired by the infrared camera on the established three-dimensional cube by taking the coordinates of the corner calibration points in the depth image as the reference.
5. The three-dimensional face imaging system based on thermal imaging projection is characterized by comprising a inspection box, wherein a plurality of cameras are arranged in the inspection box, and the plurality of cameras comprise an RGB camera, a depth camera and a thermal imaging camera; each camera is connected with the central processing unit;
the RGB camera, the depth camera and the thermal imaging camera are respectively used for acquiring RGB images, depth images and thermal imaging images of multiple frames of the same human face in a uniform light supplementing environment;
the system comprises an image preprocessing module and a deep learning network model;
the image preprocessing module is used for marking the acquired RGB image, depth image and thermal imaging image of the face with a trusted area and an untrusted area, and marking the real face area in the RGB image, the depth image and the thermal imaging image as a trusted area; marking the mirror image projection area of the face in the same image as an unreliable area to form an RGB mark data set, a depth mark data set and a thermal imaging mark data set; preprocessing the marked data sets respectively; extracting the face outline and the corresponding pixel position information in the trusted region from the preprocessed data set to serve as a forward learning data set; extracting the object outline and pixel position information of the unreliable region as a corresponding reverse learning data set;
the deep learning network model comprises three independent deep learning networks, namely an RGB network, a deep network and a thermal imaging network;
inputting the RGB image, the RGB mark data set, the RGB forward learning data set and the RGB reverse learning data set into an RGB network, performing depth training, and outputting RGB credibility;
the depth training process comprises the following steps: recognizing the position of a face region in an RGB image, performing two classifications on the acquired RBG image according to the learning of the RGB forward learning data set and the RGB reverse learning data set, and calculating the final RGB credibility as a binary label;
according to the depth training process, the depth image, the depth marking data set, the depth forward learning data set and the depth reverse learning data set are input into a depth network, and the depth credibility is output;
inputting the thermal imaging image, the thermal imaging marking data set, the thermal imaging forward learning data set and the thermal imaging reverse learning data set into a thermal imaging network, and outputting thermal imaging credibility;
when all the obtained three credibility values meet the correspondingly set credibility threshold values, filling RGB features extracted by an RGB network and temperature feature data extracted by a thermal imaging network into a three-dimensional face model; the three-dimensional face model is composed of three-dimensional space features extracted by a depth network.
6. The three-dimensional face imaging system based on thermal imaging projection of claim 5, wherein an arc-shaped panel is arranged in the inspection box, and a uniform light filling lamp is arranged behind the arc-shaped panel.
7. The three-dimensional face imaging system based on thermal imaging projection of claim 5, wherein the RGB camera comprises an RGB texture camera centrally located in the inspection box;
the depth camera comprises a left binocular camera and a right binocular camera which are arranged at the left side and the right side of the inspection box;
the thermal imaging cameras comprise left thermal imaging cameras and right thermal imaging cameras which are arranged on the left side and the right side of the inspection box;
wherein, each side the depth camera and the thermal imaging camera are provided with an attitude control mechanism.
8. An electronic device comprising a processor and a memory for storing computer program instructions, the processor for executing the computer program instructions stored in the memory, which when executed, perform the method steps of any of claims 1-4.
9. A computer storage medium, characterized in that it has stored thereon a computer program which, when executed, implements the method steps of any of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311393671.4A CN117152397B (en) | 2023-10-26 | 2023-10-26 | Three-dimensional face imaging method and system based on thermal imaging projection |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311393671.4A CN117152397B (en) | 2023-10-26 | 2023-10-26 | Three-dimensional face imaging method and system based on thermal imaging projection |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117152397A CN117152397A (en) | 2023-12-01 |
| CN117152397B true CN117152397B (en) | 2024-01-26 |
Family
ID=88897098
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311393671.4A Active CN117152397B (en) | 2023-10-26 | 2023-10-26 | Three-dimensional face imaging method and system based on thermal imaging projection |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117152397B (en) |
Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017084428A1 (en) * | 2015-11-17 | 2017-05-26 | 努比亚技术有限公司 | Information processing method, electronic device and computer storage medium |
| CN107430500A (en) * | 2015-03-20 | 2017-12-01 | 微软技术许可有限责任公司 | The content enhancing of electronic paper display devices |
| EP3363348A1 (en) * | 2015-10-15 | 2018-08-22 | Daikin Industries, Ltd. | Physiological state determination device and physiological state determination method |
| CN108717696A (en) * | 2018-05-16 | 2018-10-30 | 上海鹰瞳医疗科技有限公司 | Macula lutea image detection method and equipment |
| CN110197156A (en) * | 2019-05-30 | 2019-09-03 | 清华大学 | Manpower movement and the shape similarity metric method and device of single image based on deep learning |
| CN111178130A (en) * | 2019-11-25 | 2020-05-19 | 重庆特斯联智慧科技股份有限公司 | Face recognition method, system and readable storage medium based on deep learning |
| CN111476899A (en) * | 2020-03-24 | 2020-07-31 | 清华大学 | Three-dimensional reconstruction method for dense texture coordinates of human hand based on single-viewpoint RGB camera |
| CN111951372A (en) * | 2020-06-30 | 2020-11-17 | 重庆灵翎互娱科技有限公司 | Three-dimensional face model generation method and equipment |
| CN112215113A (en) * | 2020-09-30 | 2021-01-12 | 张成林 | Face recognition method and device |
| WO2021113268A1 (en) * | 2019-12-01 | 2021-06-10 | Iven Connary | Systems and methods for generating of 3d information on a user display from processing of sensor data |
| US11216663B1 (en) * | 2020-12-01 | 2022-01-04 | Pointivo, Inc. | Systems and methods for generating of 3D information on a user display from processing of sensor data for objects, components or features of interest in a scene and user navigation thereon |
| CN113920097A (en) * | 2021-10-14 | 2022-01-11 | 国网河南省电力公司电力科学研究院 | Power equipment state detection method and system based on multi-source image |
| WO2022089218A1 (en) * | 2020-10-28 | 2022-05-05 | 苏州奇流信息科技有限公司 | Machine learning model training method and apparatus, and prediction system |
| WO2022241874A1 (en) * | 2021-05-18 | 2022-11-24 | 烟台艾睿光电科技有限公司 | Infrared thermal imaging monocular vision ranging method and related assembly |
| CN116071292A (en) * | 2022-10-08 | 2023-05-05 | 中国人民解放军国防科技大学 | Ophthalmoscope retina image blood vessel identification method based on contrast generation learning |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111426393B (en) * | 2020-04-07 | 2021-11-16 | 北京迈格威科技有限公司 | Temperature correction method, device and system |
| US20210344852A1 (en) * | 2020-05-04 | 2021-11-04 | Rebellion Photonics, Inc. | Apparatuses, systems, and methods for thermal imaging |
-
2023
- 2023-10-26 CN CN202311393671.4A patent/CN117152397B/en active Active
Patent Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107430500A (en) * | 2015-03-20 | 2017-12-01 | 微软技术许可有限责任公司 | The content enhancing of electronic paper display devices |
| EP3363348A1 (en) * | 2015-10-15 | 2018-08-22 | Daikin Industries, Ltd. | Physiological state determination device and physiological state determination method |
| WO2017084428A1 (en) * | 2015-11-17 | 2017-05-26 | 努比亚技术有限公司 | Information processing method, electronic device and computer storage medium |
| CN108717696A (en) * | 2018-05-16 | 2018-10-30 | 上海鹰瞳医疗科技有限公司 | Macula lutea image detection method and equipment |
| CN110197156A (en) * | 2019-05-30 | 2019-09-03 | 清华大学 | Manpower movement and the shape similarity metric method and device of single image based on deep learning |
| CN111178130A (en) * | 2019-11-25 | 2020-05-19 | 重庆特斯联智慧科技股份有限公司 | Face recognition method, system and readable storage medium based on deep learning |
| WO2021113268A1 (en) * | 2019-12-01 | 2021-06-10 | Iven Connary | Systems and methods for generating of 3d information on a user display from processing of sensor data |
| CN111476899A (en) * | 2020-03-24 | 2020-07-31 | 清华大学 | Three-dimensional reconstruction method for dense texture coordinates of human hand based on single-viewpoint RGB camera |
| CN111951372A (en) * | 2020-06-30 | 2020-11-17 | 重庆灵翎互娱科技有限公司 | Three-dimensional face model generation method and equipment |
| CN112215113A (en) * | 2020-09-30 | 2021-01-12 | 张成林 | Face recognition method and device |
| WO2022089218A1 (en) * | 2020-10-28 | 2022-05-05 | 苏州奇流信息科技有限公司 | Machine learning model training method and apparatus, and prediction system |
| US11216663B1 (en) * | 2020-12-01 | 2022-01-04 | Pointivo, Inc. | Systems and methods for generating of 3D information on a user display from processing of sensor data for objects, components or features of interest in a scene and user navigation thereon |
| WO2022241874A1 (en) * | 2021-05-18 | 2022-11-24 | 烟台艾睿光电科技有限公司 | Infrared thermal imaging monocular vision ranging method and related assembly |
| CN113920097A (en) * | 2021-10-14 | 2022-01-11 | 国网河南省电力公司电力科学研究院 | Power equipment state detection method and system based on multi-source image |
| CN116071292A (en) * | 2022-10-08 | 2023-05-05 | 中国人民解放军国防科技大学 | Ophthalmoscope retina image blood vessel identification method based on contrast generation learning |
Non-Patent Citations (1)
| Title |
|---|
| 词汇树融合SIFT的热成像和视觉图像人脸识别;张国平;周改云;马丽;;电视技术(第23期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117152397A (en) | 2023-12-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR102319177B1 (en) | Method and apparatus, equipment, and storage medium for determining object pose in an image | |
| JP6560480B2 (en) | Image processing system, image processing method, and program | |
| CN103927016B (en) | Real-time three-dimensional double-hand gesture recognition method and system based on binocular vision | |
| WO2020024684A1 (en) | Method and device for modeling three-dimensional scene, electronic device, readable storage medium, and computer apparatus | |
| CN111768452B (en) | Non-contact automatic mapping method based on deep learning | |
| WO2019096310A1 (en) | Light field image rendering method and system for creating see-through effects | |
| CN109118581A (en) | Image processing method and device, electronic equipment, computer readable storage medium | |
| CN111401266B (en) | Method, equipment, computer equipment and readable storage medium for positioning picture corner points | |
| WO2020119467A1 (en) | High-precision dense depth image generation method and device | |
| CN110400278A (en) | A kind of full-automatic bearing calibration, device and the equipment of color of image and geometric distortion | |
| US20120120196A1 (en) | Image counting method and apparatus | |
| WO2021017589A1 (en) | Image fusion method based on gradient domain mapping | |
| CN109064533B (en) | 3D roaming method and system | |
| US8498453B1 (en) | Evaluating digital images using head points | |
| CN113128428B (en) | Depth map prediction-based in vivo detection method and related equipment | |
| CN112991517B (en) | Three-dimensional reconstruction method for texture image coding and decoding automatic matching | |
| CN111079470A (en) | Method and device for detecting living human face | |
| CN106360941A (en) | Method for locating fingernails and manicure equipment | |
| CN117152397B (en) | Three-dimensional face imaging method and system based on thermal imaging projection | |
| CN112509110A (en) | Automatic image data set acquisition and labeling framework for land confrontation intelligent agent | |
| CN116612357A (en) | Method, system and storage medium for constructing unsupervised RGBD multi-mode data set | |
| US11699303B2 (en) | System and method of acquiring coordinates of pupil center point | |
| WO2023272495A1 (en) | Badging method and apparatus, badge detection model update method and system, and storage medium | |
| US10339702B2 (en) | Method for improving occluded edge quality in augmented reality based on depth camera | |
| CN116125489A (en) | Indoor object three-dimensional detection method, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |