CN113706707B

CN113706707B - Human body three-dimensional surface temperature model construction method based on multi-source information fusion

Info

Publication number: CN113706707B
Application number: CN202110794691.7A
Authority: CN
Inventors: 曹彦鹏; 王钒; 夏晨杰; 杨将新; 曹衍龙
Original assignee: Zhejiang University ZJU
Current assignee: Zhejiang University ZJU
Priority date: 2021-07-14
Filing date: 2021-07-14
Publication date: 2024-03-26
Anticipated expiration: 2041-07-14
Also published as: CN113706707A

Abstract

The invention discloses a human body three-dimensional surface temperature model construction method based on multi-source information fusion, which adopts a multi-source information fusion strategy, optimizes external parameters of a camera in real time in a frame-to-frame mode, predicts the pose of the camera and optimizes the external parameters by using an alternate iteration strategy, and further improves the accuracy of a reconstruction model on the basis of guaranteeing the real-time performance of a reconstruction algorithm.

Description

Human body three-dimensional surface temperature model construction method based on multi-source information fusion

Technical Field

The invention relates to human body three-dimensional surface temperature model construction, in particular to a human body three-dimensional surface temperature model construction method based on multi-source information fusion.

Background

The object surface temperature information not only characterizes, but also reflects the internal state characteristics of the object to a certain extent. The infrared thermal imaging technology can accurately acquire temperature information of the surface of an object, and has strong robustness in severe working environments such as smoke, low illumination and the like, so that the infrared thermal imaging technology is widely applied to military and civil fields such as security monitoring, electric power detection, fire prevention and fire protection, medical diagnosis, crime tracking and the like.

However, the current mainstream two-dimensional infrared thermal imaging technology has the problems of temperature information loss, difficult spatial positioning and the like, and the object three-dimensional surface temperature model construction technology more completely reserves the temperature information of the object surface through reconstructing the object surface three-dimensional temperature model, realizes the rapid positioning of surface temperature characteristics, and greatly improves the application range and the operation friendliness of the infrared thermal imaging technology.

The existing dense three-dimensional reconstruction algorithm based on the small displacement assumption is easy to fail when the camera moves rapidly; the existing three-dimensional reconstruction algorithm can not solve the problem of multi-source information misregistration in a motion state, and influences the precision of a three-dimensional temperature model and the accuracy of temperature distribution; the existing three-dimensional reconstruction algorithm based on single-source information has good effect and insufficient robustness only in part of fixed scenes.

Disclosure of Invention

In view of the above-mentioned drawbacks of the prior art, the present invention is to solve the technical problem of how to perform efficient and accurate reconstruction of a three-dimensional surface temperature model of a target object in a fast moving state of a camera.

In order to achieve the above purpose, the invention provides a method for constructing a three-dimensional surface temperature model of a human body based on multi-source information fusion, which is characterized by comprising the following steps:

(1) Inputting multi-source images obtained by a depth camera, a visible light camera and an infrared camera;

(2) Initializing a space point pair matching relationship based on an iterative nearest neighbor method, and optimizing the space point pair matching relationship by maximizing temperature consistency between an ith frame of infrared image and an ith-1 frame of infrared image and calculating a plane translation transformation relationship between the images;

(3) By maximizing the ith frame of infrared imageAnd i-1 th frame infrared image +.>Initializing the spatial pose of the camera according to the temperature consistency of the camera;

(4) By maximizing the ith frame point cloud V _i Point cloud with i-1 frameEstimating the spatial pose of the camera;

(5) Further adjusting the camera pose estimated in step (4) based on visible light luminosity consistency;

(6) After real-time calibration is carried out on external parameters of the camera based on temperature consistency in an inter-frame registration mode, returning to the step (4) until the precision meets the requirement, and entering the step (7);

(7) Updating the existing temperature three-dimensional model by using the information of the input image;

(8) The model data at the current camera view angle is acquired by using a ray casting algorithm and is used as reference data for the calculation of the next frame.

Further, in step (2), the translation vector omega between the i-th frame and the i-1 th frame infrared image is made _i ＝[t _u t _v ] ^T Constructing a temperature consistency loss function:

where u is the pixel coordinates in the infrared image plane Ω, I _i And I _i-1 An infrared image representing an i-th frame and an i-1 th frame, respectively; solving for it using Gaussian-Newton method, at the (k+1) th iteration, translates vector ω _i The updating method of (1) comprises the following steps:

in the method, in the process of the invention,and->Represents the k+1st and k-th iteration translation vector omega, respectively _i Is calculated according to the calculation result of (2); combining equation (1) and equation (2), and applying a temperature uniformity loss function E _T Linearization:

in the method, in the process of the invention,infrared image I representing the ith frame _i In the gradient map of the u and v directions, calculating the image gradient by using a 3×3 Sobel operator; r is (r) _T Is to use the kth iteration translation vector +.>A vector formed by residual values of each pixel point in the image is calculated; />Is the r obtained by the kth iterative computation _T V omega is calculated by the loss function after linearization:

using translation vector ω _i For the ith frame point cloud V _i And i-1 th frame point cloud V _i-1 Optimizing the matching relation of the space points in the model; wherein V is _i (u _i ) Is the point cloud V _i Middle pixel point u _i Three-dimensional vertex, point cloud V under current camera coordinate system _i-1 The three-dimensional vertex matched with the three-dimensional vertex is V _i-1 (h _i-1 ) The method comprises the steps of carrying out a first treatment on the surface of the In the initialization process, the camera space position T of the ith frame is firstly set _i Set to T _i-1 Then pass through T _i-1 Will V _i (u _i ) Converting to the i-1 frame camera coordinate system, and projecting it to the i-1 frame pixel coordinate system to obtain pixel pointWill +.>And a translation vector omega _i Superposition to obtain h _i-1 Then is connected with V _i (u _i ) The matching points are

Further, in step (3), a loss function of the temperature consistency constraint is constructed:

in the method, in the process of the invention,for one vertex in the point cloud image generated by the i-1 frame infrared image in the world coordinate system, the vertex matched with the i frame infrared image in the i frame infrared image is V _i (h) The method comprises the steps of carrying out a first treatment on the surface of the M is the number of elements in the temperature effective point set P, < ->For rigid transformation of three-dimensional vertices, κ (v) represents the vertex v= (v) _x ,v _y ,v _z ) The process of projection from three-dimensional space to pixel space:

wherein f _x And f _y Is the focal length of the infrared camera, (c) _x ,c _y ) The main point coordinates of the infrared camera;

representing the delta VT of the spatial transformation matrix T as a 6-dimensional vector using lie-group lie algebra

ξ＝(α,β,λ,t _x ,t _y ,t _z ) ^T The method comprises the steps of carrying out a first treatment on the surface of the The update mode of the kth iteration is:

T _i ^k ＝VTT _i ^k-1 ≈(I+ξ^)T _i ^k-1 (8)；

wherein T is _i ^k-1 And T _i ^k The calculated results of the pose of the k-1 iteration camera and the k iteration camera are respectively, and ζ is the SE (3) lie algebraic form of the vector ζ:

substituting equation (8) and equation (9) into equation (5), linearizing the loss function to:

according to the chain derivation rule, there are:

Wherein J is _κ Jacobian matrix (ψ) as a function κ (ψ) is derived from equation (7), J _ψ (ζ) is a Jacobian matrix of the function ψ (ζ), derived from the formulas (6), (8) and (9); j (J) _fi And r _fi Respectively represent the loss functions E _fi Jacobian matrix and residual terms of (ζ), so solve for ζ according to equation (12) to implement E _fi Minimization of (ζ);

updated T through continuous iteration _i Up to E after the kth and k-1 th iterations _fi And (4) if the value is smaller than the preset threshold value, entering the step (4).

Further, in step (4), the i-th frame of infrared image is displayedAnd i-1 th frame infrared image +.>Adding the temperature consistency of the (c) to the optimized loss function in a weighted manner to construct the optimized loss function as shown in formula (13):

in the method, in the process of the invention,and->Representing a geometrical consistency loss function and a temperature consistency optimization function, respectively, < >>Representing the weight that the temperature consistency constraint occupies in optimizing the loss function.

Further, the geometric consistency loss function is implemented by minimizing the point-to-plane distance of matching point pairs between the current frame point cloud and the model point cloud.

Further, spatial vertex V _i (h) To its matching pointThe distance of the model plane is as follows:

in the method, in the process of the invention,for the normal vector diagram of the model under the viewing angle of the i-1 frame, constructing +.>As shown in formula (15):

Wherein M is the number of elements of the matching point pair set O; and v _i ＝V _i (h) Respectively representing a pair of matching points in the current input point cloud and the i-1 frame model point cloud;

substituting the formula (8) and the formula (9) into the formula (15) to obtainIs a linear result of (2):

in the method, in the process of the invention,and->Respectively indicate->Jacobian matrix and residual terms;

with reference to formula (11), after linearizationThe method comprises the following steps:

solving for ζ by equation (18) to achieve E _cr Is to be minimized:

in the method, in the process of the invention,

through continuous iteration updating T _i Until after the kth and the kth-1 iterationsIf the threshold value is smaller than the preset threshold value, the step (5) is carried out.

Further, in step (5), a multi-source joint optimization loss function is constructed as shown in formula (19):

in the method, in the process of the invention,optimizing an objective function for spatial geometrical consistency, +.>Optimizing the objective function for temperature consistency, +.>Is its corresponding weight, ++>Optimizing the objective function for visible light photometric consistency,/->Is its corresponding weight. Further, the collected visible light image is converted from the three primary color space to the YUV color space, wherein the components U and V represent chromaticity, the component Y represents brightness, and the calculation mode is as follows:

Y＝0.299R+0.587G+0.114B；

wherein R, G and B represent three color components of red, green, and blue, respectively, in a three primary color space;

after the color conversion and luminance extraction are completed, the average luminance B of the visible luminance image is used _v (Y) and maximum luminance difference C _v (Y) realizing the quantification of visible light image quality, wherein the calculation mode is as follows:

C _v (Y)＝max(Y(x))-min(Y(x))x∈Ω；

wherein Ω is a pixel space of the visible light brightness image;

thenThe calculation method of (1) is as follows:

wherein f (B) and g (C) are respectively the average brightness B of the image _v (Y) and maximum luminance difference C _v The corresponding weight component is calculated according to the formula (21) and the formula (22); k (k) _B And k _C Coefficients f (B) and g (C), respectively;

wherein H is the maximum value of the luminance image pixel value, and h=255;

constructionThe method comprises the following steps:

wherein N is the number of elements in the visible light luminosity effective point set Q; substituting the formula (8) and the formula (9) into the formula (21) to linearize them:

according to the chain derivation rule:

wherein J is _κ (ψ) is a Jacobian matrix of the function κ (ψ), derived from equation (7), J _ψ (ζ) is a Jacobian matrix of the function ψ (ζ), derived from the formulas (6), (8) and (9);and->Respectively represent a loss function->Jacobian matrix and residual terms;

with reference to formula (16), after linearizationThe method comprises the following steps:

substituting equation (24), equation (25) and equation (26) into equation (19), solving ζ to achieve E _fr Is to be minimized:

in the method, in the process of the invention,

through continuous iteration updating T _i Up to E after the kth and k-1 th iterations _fr If the threshold value is smaller than the preset threshold value, the step (6) is carried out.

Further, in step (6), based on the depth image, the point cloud generated according to the depth image is transformed into the camera coordinate systems of the infrared camera and the visible light camera through spatial rigid transformation by using the external parameters of the camera, and then the point cloud is projected into the corresponding image coordinate systems to realize the correspondence between the images.

Further, external parameters of the infrared camera relative to the depth cameraAnd at t _di To t _ti Pose transformation of depth camera in between->Then there are:

the coordinates of the vertexes in the point cloud image acquired in the ith frame in the current camera coordinate system are v _i ＝V _i (u) v is represented by the formula (29) _i Converting to an i-1 frame infrared camera coordinate system to obtain a point v _i-1 ：

Wherein T is _i-1 And T _i Camera poses of the i-1 th frame and the i-th frame respectively,representing real-time external parameters of the i-1 frame infrared camera; v according to internal parameters of infrared camera _i-1 Projecting to an image plane to obtain an i-1 frame infrared image and v _i A corresponding pixel point p;

real-time external parameters according to the i-th frame thermal infrared imagerWill v _i Converting the image into a camera coordinate system of an ith frame of infrared camera, projecting the camera coordinate system into a pixel coordinate system through an internal reference matrix to obtain a point h, forming a pair of matching points in front and rear two frames of infrared images by the point h and the point p, and marking a set formed by all M pairs of matching point pairs as S; thereby obtaining the real-time optimization objective function of the external parameters of the infrared camera:

Converting the increment VT of the external parameter matrix of the infrared camera into a six-dimensional vector xi according to the Liqun lie algebra _e ＝(α,β,λ,t _x ,t _y ,t _z ) ^T The method comprises the steps of carrying out a first treatment on the surface of the The updating mode of the k iteration of real-time optimization of the external parameters of the infrared camera is as follows:

in the method, in the process of the invention,and->Calculation results of external parameter variation of infrared camera obtained by the kth-1 time and the kth iteration respectively, < ->Is vector xi _e Is (3) lie algebraic form:

combining formula (29) and formula (30), E _et Linearization is as follows:

wherein J is _κ (ψ ₁ ) As a function of kappa (psi) ₁ ) The jacobian matrix of (a) is derived from equation (7),as a function psi ₁ (ξ _e ) Is derived from formulae (6), (31) and (32); j (J) _et And r _et Respectively represent the loss functions E _et Jacobian matrix and residual terms; solving for ζ according to equation (34) _e To realize E _et (ξ _e ) Is to be minimized;

updating the depth camera at t through continuous iteration _di To t _ti Spatial pose transformation in timeUp to E after the kth and k-1 th iterations _et Is smaller than a preset threshold, and the external parameter matrix of the infrared camera is +.>Matching the temperature image and the depth image by using the optimized external parameters;

the depth camera triggers at the moment t _di By the triggering time t of the visible light camera _vi The space pose between them is transformed intoThe real-time external parameters of the visible light camera are:

Marking the ith frame and the (i-1) th frame visible light brightness images as respectivelyAnd->The set formed by N pairs of matching point pairs in the two images is marked as W, and an external parameter real-time optimization objective function E of the visible light camera is constructed _ev The method comprises the following steps:

the linearization result of formula (35) is:

solving for xi according to (37) _e To realize E _ev (ξ _e ) Is to be minimized:

continuously iteratively updating the depth camera at t _di To t _vi Spatial pose transformation in timeLoss function E after up to the kth and k-1 iterations _ev Is smaller than a preset threshold, and the external parameter matrix of the infrared camera is +.>Use of the optimized outer shellThe partial parameters enable matching of the visible light image and the depth image.

Based on a multisource information fusion technology, the invention provides a set of three-dimensional reconstruction algorithm flow on the basis of an iterative nearest neighbor method, and realizes the rapid and accurate reconstruction of a human body temperature three-dimensional model. The invention improves the precision and the robustness of the three-dimensional reconstruction algorithm. Aiming at the problem of external parameter change in the camera motion process, the invention optimizes the external parameters of the camera in real time in a frame-to-frame mode, and proposes to use an alternate iteration strategy to predict the pose of the camera and optimize the external parameters, thereby further improving the accuracy of the reconstruction model on the basis of ensuring the real-time performance of the reconstruction algorithm. The invention provides a temperature three-dimensional model on the basis of a TSDF model, designs a model updating strategy for adaptively adjusting voxel weight according to a temperature measurement distance and a temperature measurement angle, avoids the problem of fuzzy temperature details which are easy to generate in the multi-view data fusion process, and realizes accurate construction and storage of a three-dimensional temperature field of a human body.

The conception, specific structure, and technical effects of the present invention will be further described with reference to the accompanying drawings to fully understand the objects, features, and effects of the present invention.

Drawings

FIG. 1 is a flow chart of a human body three-dimensional reconstruction algorithm for multi-source information fusion in a preferred embodiment of the present invention;

FIG. 2 is a schematic diagram of a small displacement hypothesis in the prior art;

FIG. 3 is a graph of the optimization effect of spatial point-to-match relationship in a preferred embodiment of the present invention;

FIG. 4 is a flowchart of a multi-source information joint pose estimation algorithm in a preferred embodiment of the present invention;

FIG. 5 is a schematic diagram of temperature uniformity constraint loss function construction in a preferred embodiment of the present invention;

FIG. 6 is a diagram of geometric consistency constraint loss function construction in a preferred embodiment of the present invention;

FIG. 7 is a graph showing the average luminance weight function in a preferred embodiment of the present invention;

FIG. 8 is a graph showing the maximum luminance difference weighting function in a preferred embodiment of the present invention;

FIG. 9 is a schematic diagram of the construction of a visible light coherence loss function in a preferred embodiment of the invention;

FIG. 10 is a graph of the iteration process residual variation in a preferred embodiment of the present invention;

FIG. 11 is a schematic diagram of a platform multisource data collection process in a preferred embodiment of the present invention;

FIG. 12 is a comparison of multi-source image registration results at different motion speeds in a preferred embodiment of the present invention;

FIG. 13 is a schematic diagram of platform single frame multi-source data acquisition in a preferred embodiment of the present invention;

FIG. 14 is a schematic diagram of inter-frame temperature consistency optimization in a preferred embodiment of the present invention;

FIG. 15 is a flowchart of an alternate iterative algorithm in a preferred embodiment of the present invention;

FIG. 16 is a schematic illustration of camera pose changes in alternate iterations in accordance with a preferred embodiment of the present invention;

FIG. 17 is a schematic diagram of an extended TSDF model in a preferred embodiment of the present invention;

FIG. 18 is a schematic diagram of symbol distances in a preferred embodiment of the invention;

FIG. 19 is a graph showing the sdf function in a preferred embodiment of the present invention;

FIG. 20 is a schematic representation of the tsdf function in a preferred embodiment of the invention.

Detailed Description

The following description of the preferred embodiments of the present invention refers to the accompanying drawings, which make the technical contents thereof more clear and easy to understand. The present invention may be embodied in many different forms of embodiments and the scope of the present invention is not limited to only the embodiments described herein.

The human body three-dimensional surface temperature model construction technology acquires human body surface three-dimensional geometric information by using a three-dimensional reconstruction technology, fuses the human body surface three-dimensional geometric information according to the matching relation between the three-dimensional geometric information and the temperature information, stores the human body three-dimensional geometric information in a human body temperature three-dimensional model, and improves the integrity of temperature three-dimensional model construction in a multi-view data fusion mode. In a specific embodiment of the invention, a set of three-dimensional reconstruction flow shown in fig. 1 is provided based on a multi-source information fusion technology, so that the rapid and accurate construction of a three-dimensional temperature field of a human body is completed.

In general, multi-source information has the following characteristics: the time sequence of the temperature information has good consistency but coarse texture, and the matching difficulty is small but the precision is limited; the texture in the visible light information is rich and is easy to interfere, and the matching difficulty is high and the precision is high; the depth information is highly reliable but does not have temporal consistency.

Therefore, the method and the device of the invention optimize and calculate the planar transformation relation between the images through the photometric consistency between the input infrared image and the reference infrared image, thereby completing the initialization of the point-to-match relation between the two group point clouds (step 100 in fig. 1), so as to solve the problem of reconstruction algorithm failure when the coincidence degree of the input point cloud and the reference point cloud is less. And then solving the spatial pose of the platform in the current state by using a temperature measuring platform joint pose estimation algorithm based on multi-source information fusion (step 200 in fig. 1). Based on the characteristics of multi-source information, the platform space pose estimation process is divided into three stages of pose quick initialization based on temperature consistency (step 201 in fig. 1), pose rough estimation based on space geometric consistency constraint (step 202 in fig. 1) and pose fine adjustment based on visible light luminosity consistency and geometric consistency (step 203 in fig. 1), and the quick and accurate estimation of the platform pose is realized in a rough-to-fine mode. Because the data acquisition time of each camera is not synchronous, the movement of the platform in the process of constructing the human body three-dimensional temperature field can cause the change of external parameters of the cameras, thereby affecting the precision of the reconstructed human body temperature three-dimensional model. Therefore, the invention uses the interframe registration mode to calibrate the external parameters of the camera in real time through the temperature consistency optimization (step 300 in fig. 1), and adopts an alternate strategy to circularly develop the pose estimation of the platform and the optimization of the external parameters of the camera (step 400 in fig. 1), so as to inhibit the problem of mismatching of multi-source information caused by the change of the external parameters of the camera. And then updating the existing temperature three-dimensional model according to fusion rules by using the information of the input image according to the result of the platform pose joint optimization (step 500 in fig. 1). Finally, a Ray Casting algorithm (Ray Casting) is used for obtaining model data under the current platform visual angle and taking the model data as reference data for calculating a next frame (step 600 in fig. 1), so that a model reconstruction flow of a frame to a model is realized, and the accuracy and the robustness of the construction of a three-dimensional temperature field of a human body are further improved.

Step 100：

The basic assumption of the iterative nearest neighbor method (Iterative Closest Point, ICP) is: input current frame image I _i With the previous frame image I _i-1 The camera displacement generated between the two images is smaller, so that the camera space pose T corresponding to the previous frame image can be used _i-1 As the current frame camera spatial pose T _i Is set to be a constant value.

As shown in fig. 2, when the camera space pose does not change much, image I _i And I _i-1 Point P corresponding to the same pixel coordinate system _i And P _i-1 The distance on the actual object is relatively short, the pose of the camera can be adjusted through iterative calculation, and the alignment of the input point cloud and the model data point cloud is completed. However, when the camera pose changes greatly and the coincidence point of the input point cloud and the model data point cloud is small, the three-dimensional point P in the model data point cloud is obtained through initializing the matching relation _i-1 Not at the input point cloud P _i It is difficult to find the correct camera pose in subsequent iterations, resulting in failure of the reconstruction algorithm. When the surface structure of the reconstructed object is simple, the geometric repeatability is strong, or the surface structure is too complex, the probability of algorithm failure is further increased. In order to improve the problem and improve the robustness of the reconstruction algorithm, the method optimizes the space point pair matching relation initialization algorithm based on the two-dimensional input image and the model data.

Under the condition of room temperature, the temperature information of the human body is kept relatively stable, and the measurement result of the surface temperature of the human body is less influenced by the illumination condition and the measurement angle, so that the consistency of gray values can be better kept in the front and rear infrared images of the same point on the human body, and the better effect can be achieved by optimizing the initial matching relation of point clouds by using the front and rear infrared images.

The invention simulates the plane transformation of the front and back two frames of infrared images by using translation transformation, and the mode can completely provide enough effective initial matching point pairs, thereby improving the robustness of the human body three-dimensional reconstruction algorithm.

In calculating the parameter t of the image translation vector _u And t _v In this case, the present invention adopts a method of maximizing the temperature uniformity between the i-th frame and the i-1 th frame infrared image. Let the translation vector omega between two images _i ＝[t _u t _v ] ^T Then a temperature uniformity loss function can be constructed as shown in equation (5):

where u is the pixel coordinates in the infrared image plane Ω, I _i And I _i-1 Respectively representing the infrared images of the i-th frame and the i-1 th frame. Loss function E _T Is a nonlinear least squares function that can be solved using the gaussian-newton method. At the k+1st iteration, the vector ω is translated _i The updating method of (1) comprises the following steps:

in the method, in the process of the invention,and->Represents the k+1st and k-th iteration translation vector omega, respectively _i Is calculated by the computer. Combining equation (5) and equation (6) and applying a temperature uniformity loss function E _T Linearization:

in the method, in the process of the invention,infrared image I representing the ith frame _i In the gradient map in both u and v directions, the invention uses a 3×3 sobel operator to calculate the image gradient; r is (r) _T Is to use the kth iteration translation vector +.>A vector formed by residual values of each pixel point in the image is calculated; />Is the r obtained by the kth iterative computation _T Is a jacobian matrix of (c). According to the principle of gauss-newton's method, vω can be calculated from the loss function after linearization:

after determining the plane transformation relation of the i-th frame and the i-1 th frame infrared image, the invention uses the translation vector omega _i For the ith frame point cloud V _i And i-1 th frame point cloud V _i-1 The matching relationship is optimized by the space points in the model.

Fig. 3 shows the comparison of the matching relationship between two adjacent frames before and after optimization. The left image is an i-1 frame infrared image, the right image is an i-1 frame infrared image, the pixel point q in the i-1 frame infrared image is directly corresponding to the pixel point p of the i-1 frame infrared image through a small displacement hypothesis, and the corresponding relation is obviously wrong. After the translation vector u calculated by the method is added, the point corresponding to the pixel point q in the i-1 frame infrared image can be found, so that the matching point pair generated between the input point cloud and the point cloud to be matched in the initializing process is physically closer, the difficulty of a reconstruction task is reduced, the algorithm can be applied to the situation that the point cloud superposition such as large displacement or rapid movement is smaller, and the robustness of the reconstruction algorithm is greatly improved.

Step 200:

after the initialization of the matching relation between the current input point cloud and the model point cloud is completed, the pose of the temperature measuring platform can be estimated. The iterative nearest neighbor method used in the prior art only considers a single space geometric consistency constraint, but is easy to be in local optimum when reconstructing a scene with a complex geometric relation, so that the reconstruction algorithm is invalid. In the reconstruction process of the three-dimensional temperature field of the human body, the multi-source data acquisition platform needs to acquire data by encircling the target human body by 360 degrees, and the change of the visual angle is often accompanied with the change of the surface color of the object and the movement of the shadow, so that the visible light information is difficult to keep consistent in the data acquisition process. In the prior art, a loss function with stronger robustness is provided by utilizing the consistency of the surface temperature of an object, but the sparsity of the texture information of the surface temperature of the object leads to the reduction of the model reconstruction accuracy to a certain extent.

The existing three-dimensional reconstruction algorithm uses a fixed loss function when estimating the spatial pose of the camera, but does not consider that in reconstruction tasks of different scenes, the contribution degree of various information to the pose estimation result of the camera is different, and the influence of various information on the calculation result is different for different iteration steps in the pose estimation of the same camera, so that the algorithm has a better reconstruction effect only for part of scenes. The introduction of the multi-source information can enable the human body three-dimensional temperature field reconstruction system to use proper information sources under different conditions, thereby improving the performance of the three-dimensional reconstruction system.

In order to design a multi-source information fusion strategy, the method analyzes the characteristics of different kinds of information, and optimizes the multi-source data acquisition platform in two aspects of pose estimation algorithm flow and constraint item construction.

The object surface temperature information is less influenced by illumination conditions, measurement angles and environmental condition changes, good consistency can be maintained in a longer image sequence, and the difficulty of pose estimation is reduced to a certain extent, so that the object surface temperature information can be used as guiding information to be added to an initial stage of space pose estimation of a three-dimensional thermal imaging platform, and a calculation result can be quickly converged to a neighboring area of a global optimal solution through temperature consistency constraint.

The depth information provides the spatial position information and the surface geometric information of the measured object, and the spatial geometric consistency constraint of constructing two group point clouds through the depth information proves to be a reliable and effective camera spatial pose estimation mode, and is also a mainstream loss function construction method when an iterative nearest neighbor method is used at present. However, due to the limitation of the precision of the depth camera and the influence of noise, certain errors exist in the object space geometric information acquired by the multi-source information data acquisition platform, so that the pose estimation precision is reduced. Therefore, after the rapid convergence stage is finished, space geometric constraint is added, and a reliable rough estimation result of the pose of the camera can be further obtained.

The visible light images generally contain abundant color texture information, and details in a scene can be well restored, so that high-precision estimation of the camera pose can be realized by constructing visible light luminosity consistency constraint among the images. But the abundant detailed information can bring about the problem of weak iterative guidance, so that the iterative times in the solving process are increased rapidly, and the local optimization is easy to be trapped. Meanwhile, the measurement result of the visible light information on the surface of the object is easily influenced by illumination conditions and measurement angles, the consistency of the measurement result can be maintained in a shorter image sequence, and the capability of changing a long base line is slightly insufficient. In order to avoid adverse effects caused by visible light information on the basis of improving the reconstruction accuracy of the three-dimensional model of the human body temperature, the rough estimation result of the spatial pose of the camera can be added into a loss function after the rough estimation result is obtained, so that the iteration result approaches to a more accurate numerical value.

Based on the above analysis, the present invention computationally divides the human three-dimensional temperature field reconstruction algorithm into three phases as shown in fig. 4: platform pose based on temperature consistency is initialized (201) rapidly, platform pose rough estimation based on space geometric consistency (202) and platform pose fine adjustment based on visible light luminosity consistency is dominant (203).

Step 201:

in the stage of rapid initialization of the platform pose, the invention maximizes the ith frame of infrared imageAnd i-1 th frame infrared image +.>To estimate the spatial pose of the platform. As shown in fig. 5 +.>For one vertex in the point cloud image generated in the world coordinate system under the view angle of the i-1 th frame, the vertex matched with the vertex in the i-1 th frame is V _i (h) The formula (20) is shown.

Wherein M is the number of elements in the temperature effective point set P,for rigid transformation of three-dimensional vertices, κ (v) represents the vertex v= (v) _x ,v _y ,v _z ) A process of projection from three-dimensional space to pixel space.

Wherein f _x And f _y Focal length of thermal infrared imager, (c) _x ,c _y ) Is the main point coordinate of the thermal infrared imager.

Because equation (20) is a nonlinear least squares problem, it can be solved using Gaussian-Newton methods. The space rigid transformation matrix T is a nonlinear variable, and in order to calculate the derivative of T, the invention uses Liqular Legend to represent the increment VT of the space transformation matrix T as a 6-dimensional vector xi= (alpha, beta, lambda, T) _x ,t _y ,t _z ) ^T . The update mode of the kth iteration of the pose estimation is as follows:

T _i ^k ＝VTT _i ^k-1 ≈(I+ξ^)T _i ^k-1 (23)

wherein T is _i ^k-1 And T _i ^k The calculation results of the k-1 th and k-th iteration platform pose are respectively that ζ is SE (3) lie algebraic form of vector ζ:

Substituting equation (23) and equation (24) into equation (20) can linearize the loss function of the stage of rapid initialization of the platform pose as:

according to the chain derivation rule:

wherein J is _κ The Jacobian matrix, which is a function of kappa (ψ), can be derived from equation (22), J _ψ The jacobian matrix, which is a function ψ (ζ), can be derived from equation (21), equation (23) and equation (24). J (J) _fi And r _fi Respectively represent the loss functions E _fi Jacobian matrix and residual terms of (ζ), so ζ can be solved according to equation (27) to implement the optimization function E _fi Minimization of (ζ).

Platform pose T is updated through continuous iteration _i Loss function E after up to the kth and k-1 iterations _fi If the platform pose is smaller than a certain threshold value, the stage of quickly initializing the platform pose is considered to be completed and the stage of roughly estimating the platform pose starts to enter.

Step 202:

in the stage of rough estimation of the pose of the platform, the method mainly maximizes the point cloud V of the ith frame _i Model point cloud at view angle with i-1 frameTo estimate the spatial pose of the platform. In order to prevent the platform pose from deviating in the subsequent iterative computation process, the invention iterates the platform pose to the vicinity of the global optimal solution in the rapid initialization stage, and the i-th frame infrared image +. >And i-1 th frame infrared image +.>Adding the temperature consistency of the model (2) into an optimization objective function in a certain weight mode to establish a geometric and temperature joint optimization mechanism, wherein the joint optimization loss function is shown as a formula (28):

wherein E is _sg And E is _t Respectively representing a geometric consistency loss function and a temperature consistency optimization function, ω represents the weight occupied by the temperature consistency constraint in the loss function and ω is set to 0.1.

The space geometrical consistency constraint is realized by minimizing matching points between the point cloud of the current frame and the point cloud of the modelThe point-to-plane distance of the pair. As shown in FIG. 6, the normal vector diagram of the model at the previous frame view angle isThen the spatial vertex V _i (h) To its matching point->The distance of the model plane is as follows:

thus geometric consistency loss functionThe construction mode of (2) is shown as a formula (30).

Wherein M is the number of elements of the matching point pair set O.For the normal vector diagram of the model under the previous frame view, and v _i ＝V _i (h) And representing a pair of matching points in the current input point cloud and the i-1 frame model point cloud. Substituting equation (23) and equation (24) into equation (30) can obtain the linearization result of the space geometry consistency optimization objective function: />

In the method, in the process of the invention,and->Respectively represent a loss function->Jacobian matrix and residual terms.

Temperature consistency optimization objective function in stage of rough estimation of platform poseSimilar to the pose rapid initialization stage,

solving ζ by (33) to implement a joint optimization function E _cr Is to be minimized:

in the method, in the process of the invention,platform pose T is updated through continuous iteration _i Loss function after up to k and k-1 iterations +.>If the pose position is smaller than a certain threshold value, the stage of rough estimation of the pose of the platform is considered to be completed.

Step 203:

in the stage of fine adjustment of the pose of the platform, the invention utilizes abundant texture information in the visible light image to carry out finer adjustment on the pose of the platform through the consistency constraint of visible light luminosity. In order to ensure the correctness of the iteration direction and the reliability of the iteration result, the invention simultaneously reserves space geometric consistency constraint and temperature consistency constraint in the optimization at this stage, and constructs a multi-source joint optimization loss function as shown in a formula (34):

in the method, in the process of the invention,optimizing an objective function for spatial geometrical consistency, +.>Optimizing the objective function for temperature consistency, +.>Is its corresponding weight, ++>Optimizing the objective function for visible light photometric consistency,/->Is its corresponding weight.

The validity of the visible light information mainly depends on illumination conditions of a reconstructed scene, and in order to avoid reconstruction failure problems caused by poor illumination or illumination change and improve the robustness of a reconstruction algorithm, the invention judges the quality of a visible light image before calculating a visible light consistency loss function, and then adjusts the weight of the visible light consistency loss function according to a judging result.

The visible light information reflected by the surface of the object contains both brightness information and color information. The color information is determined by the reflection characteristic of the object surface, the incident light wave band, the incident angle and the radiation intensity, so that the color information has stronger instability relative to the brightness information, and mismatching of point clouds in pose calculation of a camera is easy to cause, and the brightness information is used as auxiliary information for pose fine adjustment. In order to extract the brightness component in the visible light information and complete the conversion from the visible light image to the brightness image, the invention converts the visible light image acquired by the human three-dimensional thermal imaging platform from the three primary color space to the YUV color space through the color space conversion and performs the degree extraction.

The invention uses the average brightness B of the visible brightness image _v (Y) and maximum luminance difference C _v (Y) realizing the quantification of visible light image quality, the calculation method is as follows:

C _v (Y)＝max(Y(x))-min(Y(x)) x∈Ω (37)

where Ω is the pixel space of the visible light luminance image.

Average brightness B of image _v (Y) reflects the overall brightness level of the visible light image when the average brightness B _v (Y) exceeding the effective brightness range (B) _tl :B _th ) The weight of the visible light coherence loss function should be reduced to prevent reconstruction failure.

Maximum brightness difference C of image _v (Y) represents the range of the brightness distribution in the image when the image maximum brightness difference C _v (Y) is higher than threshold C _t When the visible light luminosity consistency loss function is calculated, the weight and the C of the visible light luminosity consistency loss function _v (Y) is in positive correlation. Therefore, in the platform space pose fine adjustment optimization objective function, the visible light luminosity consistency constraint weightThe calculation method of (1) is as follows:

wherein f (B) and g (C) are respectively the average brightness B of the image _v (Y) and maximum luminance difference C _v The corresponding weight component of (Y) is calculated by the following formulas (39) and (40). k (k) _B And k _C The coefficients f (B) and g (C), respectively, are empirical and will be k, respectively, because the average brightness has a greater impact on image quality _B And k _C Set to 4 and 2.

Where H is the maximum value of the luminance image pixel value, and in the present invention, h=255. Fig. 7 and 8 show the weight component f (B) and the average luminance B, respectively _v (Y), and weight component g (C) and maximum luminance difference C _v (Y) functional relationship between (A) and (B).

Visible light luminosity consistency optimization objective functionThe construction method of (2) is shown in FIG. 9. Using the ith frame platform pose T _i Generating a vertex in a point cloud picture in a world coordinate system under the view angle of the i-1 th frame>Converting the image into an ith frame platform coordinate system through space rigidity transformation, projecting the image into an ith frame image coordinate system by using camera internal parameters to obtain a pixel point u', and restraining an ith-1 frame visible light brightness image Y _i-1 And the ith frame visible light brightness image Y _i The consistency of the medium brightness is used for constructing an optimized objective function, namely:

wherein N is the number of elements in the visible light luminosity effective point set Q. Substituting the formula (23) and the formula (24) into the formula (39) to linearize them:

according to the chain derivation rule:

wherein J is _κ The Jacobian matrix, which is a function of kappa (ψ), can be derived from equation (22), J _ψ The jacobian matrix, which is a function ψ (ζ), can be derived from equation (21), equation (23) and equation (24).And->Respectively represent a loss function->Jacobian matrix and residual terms.

Space geometrical consistency optimization objective function of platform pose fine adjustment stageAnd temperature consistency optimization objective function->The construction of the model is similar to that of the stage of rough estimation of the pose of the platform.

Solving xi by linearization method to realize platform pose fine adjustment stage multisource information combined optimization function E _fr Is to be minimized:

in the method, in the process of the invention,platform pose T is updated through continuous iteration _i Loss function E after up to the kth and k-1 iterations _fr Less than a certain threshold valueThe stage of fine adjustment of the pose of the platform is considered to be completed.

Fig. 10 illustrates residual variation of multiple camera pose estimation algorithms in an iterative calculation process for the same set of data acquired by a human three-dimensional thermal imaging platform. The graph shows that the rapid convergence of the platform pose can be realized by using a T-ICP algorithm of temperature information and space geometric information, but the final solving precision is not high; the RGB-ICP algorithm combining the visible light information and the geometric information has high calculation precision, but the iteration number required by the convergence of the result is obviously increased; the camera pose estimation algorithm based on multi-source information fusion provided by the invention can realize rapid reduction of residual errors in the early stage, and can obtain more accurate results in iteration times far smaller than RGB-ICP.

Step 300:

the pose estimation of the multi-source information fusion platform requires good matching relation among multi-source information, but simultaneous triggering among multi-source sensors is difficult to realize, so that external parameters among the multi-source sensors are changed in the data acquisition process, and errors are introduced to the registration result of the multi-source images. As shown in fig. 11 and 12, when the three-dimensional thermal imaging platform of the human body is fixed or moves slowly, the external parameters change caused by the asynchronous triggering can be ignored, so that the external parameters of the camera obtained by calibration can still be used for effectively completing the registration between the multi-source images. However, when the moving speed of the three-dimensional thermal imaging platform of the human body is increased, the platform generates larger pose change in the same trigger time difference, and the calibrated external parameters of the camera are not enough to obtain the multi-source image registration result meeting the precision requirement.

In order to solve the problem, the invention proposes to optimize the external parameters of the camera in real time by using an interframe registration mode so as to reduce the problem of mismatching of multi-source images caused by camera motion.

Fig. 13 illustrates the acquisition process of an i-th frame multisource image pair of a human three-dimensional thermal imaging platform. The trigger time of the depth camera is t _di The triggering time of the thermal infrared imager is t _ti The triggering time of the visible light camera is t _vi . Due to visible camera and infrared thermal imagingThe behavior of the infrared thermal imaging system is similar in the platform data acquisition process, so that the infrared thermal imaging system is used for analyzing the problem of external parameter change in the multi-source data acquisition process. Marking the relative pose of the depth camera and the i-th frame of the thermal infrared imager under the motion state of the platform asAccording to the illustration in FIG. 13, the camera pose in the platform motion state is composed of two parts, namely the external parameters of the thermal infrared imager relative to the depth camera determined during the platform construction ∈>And at t _di To t _ti Pose transformation of depth camera in between->Then there are:

in the method, in the process of the invention,is a fixed value independent of the motion speed and the spatial position of the platform and can be obtained through the camera calibration process in chapter 2, so that the depth camera is solved at t _di To t _ti Spatial pose transformation in time->Is the key of real-time optimization of external parameters of the infrared camera.

In order to improve the real-time performance of the algorithm, the invention proposes to use the front and back two frames of infrared images and the depth image of the current frame, and complete the interframe registration of the infrared images through temperature consistency constraint, thereby completing the real-time optimization of external parameters of the thermal infrared imager.

The manner in which the present invention constructs the inter-frame temperature consistency constraint optimization objective function is shown in fig. 14. Flat plateThe coordinate of the vertex in the point cloud image acquired by the ith frame of the table in the current platform coordinate system is v _i ＝V _i (u) v is represented by the formula (48) _i Converting into an i-1 frame thermal infrared imager camera coordinate system to obtain a point v _i-1 ：

Wherein T is _i-1 And T _i The platform pose of the i-1 th frame and the i-th frame are respectively,and the real-time external parameters of the i-1 frame thermal infrared imager are represented. V is set according to internal parameters of the thermal infrared imager _i-1 Projecting to an image plane to obtain an i-1 frame infrared image and v _i A corresponding pixel point p.

Then according to the real-time external parameters of the i-th frame thermal infrared imagerWill v _i And converting the point h and the point p into a pair of matching points in the front and rear two frames of infrared images, and marking a set formed by all M pairs of matching point pairs as S. Thereby obtaining the real-time optimization objective function of the external parameters of the thermal infrared imager:

linearizing it according to lie algebra:

wherein J is _κ (ψ ₁ ) As a function of kappa (psi) ₁ ) The jacobian matrix of (c) can be derived from equation (22),as a function psi ₁ (ξ _e ) Can be derived from equation (21), equation (50) and equation (51). J (J) _et And r _et Respectively represent the loss functions E _et Jacobian matrix and residual terms of (a), so that a linearization method can be used to solve for ζ _e To realize the optimization function E _et (ξ _e ) Is described.

Updating the depth camera at t through continuous iteration _di To t _ti Spatial pose transformation in timeLoss function E after up to the kth and k-1 iterations _et Is smaller than a certain threshold, and the external parameter matrix of the thermal infrared imager isAnd the pixel level matching of the temperature image and the depth image can be realized by using the optimized external parameters.

Step 400:

the real-time optimization of the external parameters of the camera based on the inter-frame registration adopts the assumption that the pose of the depth camera is unchanged, namely, the pose of the depth camera is considered to be accurate and determined when the external parameters of the thermal infrared imager and the visible light camera are optimized in real time. However, according to the current algorithm flow, the registered depth image, temperature image and visible light image obtained through real-time optimization of external parameters of the camera only participate in updating the temperature three-dimensional model, but are not used for estimating the pose of the platform. In order to fully utilize the advantages of multi-source information in the reconstruction process of the three-dimensional model of the human body temperature, the invention adopts an alternate iteration strategy to realize the effective combination of camera pose estimation based on multi-source information fusion and real-time optimization of external parameters of a camera.

A specific flow of the alternate iterative algorithm designed by the present invention is shown in fig. 15. After the input of the ith frame multisource information of the space point to the initial matching relation optimization is completed, a platform pose estimation result is quickly converged to be close to a true value through a platform pose quick initialization stage, and then a relatively accurate platform pose T is obtained through two stages of platform pose rough estimation and platform pose fine adjustment _i ⁰ . If T _i ⁰ If the precision of the model (C) cannot meet the requirement, the external parameters of the thermal infrared imager and the visible light camera are optimized in real time by using the camera pose obtained by the current calculation, the matching relation of the temperature image, the visible light image and the depth image is optimized according to the calculation result, and the optimized multi-source image is input into the calculation flow of the rough estimation of the platform pose, so that the alternate iteration of the platform pose estimation and the optimization of the external parameters of the camera is realized.

Fig. 16 shows intuitively the change of the pose of the depth camera, the infrared camera and the visible camera in the p-th alternate iteration process. Wherein A is the true value of the pose of each camera of the ith frame of the platform, and the spatial poses of the depth camera, the thermal infrared imager and the visible light camera are respectively T because the triggering time is asynchronous _di 、T _ti And T _vi 。And->The current pose of the thermal infrared imager and the visible light camera can be respectively represented by the corresponding external parameters +.>And->And the current pose of the depth camera +.>And (3) calculating to obtain:

as can be seen from fig. 16, the implementation of the alternate iterative strategy does not affect the calculation flow of the real-time optimization algorithm of the external parameters of the camera, but because of the addition of the external parameter changing amounts of the thermal infrared imager and the visible light camera, the matching relationship between the pixels of the front and back two frames of temperature images and the visible light image in the platform pose estimation algorithm based on multi-source information fusion is changed, so that the temperature consistency optimization objective function and the visible light luminosity consistency optimization objective function in the algorithm need to be corrected.

The corrected temperature consistency optimization objective function is as follows:

the corrected visible light luminosity consistency optimization objective function is as follows:

step 500:

when the temperature three-dimensional model is initialized, updated, output and the like, the spatial characteristics, the temperature characteristics and the visible light characteristics of the temperature three-dimensional model are required to be represented and stored through a certain spatial representation model. Therefore, the invention improves the technology of the TSDF space representation model based on the space voxel model, and provides an extended TSDF space representation model which is completely suitable for reconstructing the human body temperature three-dimensional model.

The extended TSDF spatial representation model proposed by the present invention is shown in fig. 17. In the initialization stage of the space model, the space size of a reconstructed target object is designated as L multiplied by W multiplied by H according to the construction requirement of a three-dimensional temperature field of a human body, and then N is equally divided into the length, width and height of the target space to obtain an N multiplied by N voxel space, so that the voxelization of the target space is completed. The spatial position and size of each voxel in the voxel space is determined by the spatial coordinates of the voxel center point. The voxel stores truncated symbol distance value tsdf representing model space geometrical information and temperature information I _T And color information I _C = (R, G, B) and its corresponding geometric weight w _G Weight of temperature w _T And color weight w _C 。

The extended TSDF model represents surface geometry information of the three-dimensional model by how far the pixel is from the model surface using a truncated symbol distance function (Truncated Signed Distance Function, TSDF). As shown in fig. 18, when the surface geometry information of the three-dimensional model is updated using the depth image, the coordinates of the center point of a certain spatial voxel arePoint +.>The nearest point is P, then spatial point +.>The sign distance of the located voxel is:

Where d (P) represents a depth coordinate value of the acquisition space point P in the world coordinate system.

From equation (63), it is known that the point with the value of sdf closer to 0 is closer to the surface of the three-dimensional model, and the point isOutside the three-dimensional model surface, the sdf value of its corresponding voxel is positive, when the point +.>When the three-dimensional model is inside, the sdf value of the corresponding voxel is negative, so that the voxel with the sdf value of zero is extracted from the model and is interpolated, and the spatial position and the geometric characteristic of the three-dimensional model surface can be determined. In the actual use process, the characterization capability of points far away from the three-dimensional model surface is very limited, and because the absolute value of the sdf is larger, errors are easily introduced due to boundary problems and the like while the calculation resources are occupied, so that a threshold judgment mechanism is introduced on the basis of the sdf function shown in fig. 19. When the absolute value of the sdf of the voxel is larger than the distance uncertainty mu, the absolute value of the sdf is set to be +mu and-mu according to the positive sign and the negative sign of the sdf, and then normalization processing is carried out on all the sdf to obtain a tsdf function image as shown in fig. 20.

The temperature information of voxels in the model is stored as I in the form of single precision floating point numbers _T The temperature in degrees celsius of the space represented by the voxels is recorded. Because the temperature information is gray information which is unfavorable for human eyes to observe, in order to quickly locate on a real object after observing the characteristics of the temperature three-dimensional model and improve the visual effect of the three-dimensional temperature field model reconstruction result, the invention also records the color information of the three-dimensional model when the space model is constructed. The color information of each spatial voxel is recorded as I _C Three of the R, G, B channels record three color components of red, green and blue of the surface color of the three-dimensional model respectively so as to ensure the authenticity of the object color.

After the reconstruction of the target space region is voxelized, the temperature information I of each voxel in the space is automatically calculated _T And color information I _C Initialized to 0 and (0, 0), the tsdf value is set to-1 while weighting the geometry w in the voxel _G Weight of temperature w _T And color weight w _C Zeroing to obtain an initialized extensionThe TSDF space represents the model. After the platform pose and the camera external parameters of each frame are acquired through an alternate iterative algorithm, the currently acquired three-dimensional model surface information, temperature information and color information are required to be converted into a temporary TSDF model according to the calculation result, and then the temporary TSDF model is fused into a global extended TSDF model to realize the update of the three-dimensional model data.

Marking the platform pose estimation result of the ith frame as T _i The tsdf value calculation formula of the spatial voxel P with the center point coordinate P is:

in the method, in the process of the invention,the representation rounds up the vector u, D _i (x) And representing the depth value of the pixel point corresponding to the point p in the i-th frame depth map. Because the depth value measured by the existing sensor at the edge of the object is inaccurate, the present invention adds the current geometric weight of the corresponding voxel of the edge of the object in the point cloud picture >Set to 0. Contours in the depth map are extracted using equation (67):

in the method, in the process of the invention,is a pixelA window neighborhood of 5×5 around u, δ is an edge judgment threshold. The geometrical weights of the spatial voxels P are:

/>

marking the temperature map acquired by the ith frame asThe temperature information corresponding to the spatial voxel P with the center point coordinate P is:

according to the known prior art, factors such as measurement distance, temperature measurement angle and the like of infrared radiation temperature measurement have influence on measurement results. Therefore, the invention corrects the measurement distance and the measurement angle in the temperature three-dimensional model updating process. The invention discards the temperature information outside the working distance range of the platform in the model updating process, and comprises the following steps:

according to the known prior art, the surface emissivity of the non-electrolyte material long-wave infrared radiation is kept unchanged when the normal angle is smaller than 60 degrees, and the surface emissivity is drastically reduced when the normal angle is smaller than 60 degrees, so that the invention only maintains the temperature information that the normal angle is smaller than 60 degrees:

wherein V is _i And N _i Respectively representing a point cloud image and a normal vector image of the ith frame.

According to the assumption of the consistency of the surface temperature of an object, the temperature value of the same space point under different visual angles is kept relatively stable, so that the temperature abrupt change point is removed when the temperature of the three-dimensional model is updated, and the influence of temperature measurement noise on the temperature information of the model is reduced:

In delta _T For the temperature mutation threshold, the present invention empirically sets it to 10 ℃. The temperature weight of the spatial voxel P can thus be derived:

marking the visible light image acquired by the ith frame as I _ci The color information corresponding to the spatial voxel P with the center point coordinate P is:

as the change of the measurement angle is accompanied with the change of illumination conditions, the measurement result of the surface color of the object is greatly influenced, and when the three-dimensional model color information is updated, only the measurement result with the normal angle smaller than 30 degrees is selected to ensure the accuracy of the three-dimensional model color information:

in order to reduce the influence of measurement noise on the color information of the model, the invention also eliminates color mutation points:

/>

in the method, in the process of the invention,representing the i-th frame visible lightR, G, B components in the image. The color weight of the spatial voxel P can thus be derived:

after the temporary TSDF model obtained by calculating the i-th frame multisource information is obtained, the temporary TSDF model is fused into the global TSDF model by the following formula, and the update of the three-dimensional model data is realized.

w _η The method is a voxel weight maximum value set for increasing the robustness of a reconstruction system and preventing data overflow, and the maximum value is set to 2000 according to the actual requirement of human three-dimensional temperature field reconstruction.

Based on a multisource information fusion technology, the invention provides a set of three-dimensional reconstruction algorithm flow on the basis of an iterative nearest neighbor method, and realizes the rapid and accurate reconstruction of a human body temperature three-dimensional model. The invention designs a multi-source information fusion strategy, and improves the precision and the robustness of the three-dimensional reconstruction algorithm. Aiming at the problem of external parameter change in the camera motion process, the invention optimizes the external parameters of the camera in real time in a frame-to-frame mode, and proposes to use an alternate iteration strategy to predict the pose of the camera and optimize the external parameters, thereby further improving the accuracy of the reconstruction model on the basis of ensuring the real-time performance of the reconstruction algorithm. The invention provides a temperature three-dimensional model on the basis of a TSDF model, designs a model updating strategy for adaptively adjusting voxel weight according to a temperature measurement distance and a temperature measurement angle, avoids the problem of fuzzy temperature details which are easy to generate in the multi-view data fusion process, and realizes accurate construction and storage of a three-dimensional temperature field of a human body.

The foregoing describes in detail preferred embodiments of the present invention. It should be understood that numerous modifications and variations can be made in accordance with the concepts of the invention without requiring creative effort by one of ordinary skill in the art. Therefore, all technical solutions which can be obtained by logic analysis, reasoning or limited experiments based on the prior art by the person skilled in the art according to the inventive concept shall be within the scope of protection defined by the claims.

Claims

1. The human body three-dimensional surface temperature model construction method based on multi-source information fusion is characterized by comprising the following steps:

(8) Acquiring model data under the current camera view angle by using a light projection algorithm and taking the model data as reference data of the next frame calculation;

In step (2), the translation vector omega between the i-th frame and the i-1 th frame infrared image is set _i ＝[t _u t _v ] ^T Constructing a temperature consistency loss function:

in the method, in the process of the invention,infrared image I representing the ith frame _i In the gradient map of the u and v directions, calculating the image gradient by using a 3×3 Sobel operator; r is (r) _T Is to use the kth iteration translation vector +.>A vector formed by residual values of each pixel point in the image is calculated; />Is the r obtained by the kth iterative computation _T Is calculated from the loss function after linearization:

using translation vector ω _i For the ith frame point cloud V _i And i-1 th frame point cloud V _i-1 Optimizing the matching relation of the space points in the model; wherein V is _i (u _i ) Is the point cloud V _i Middle pixel point u _i Three-dimensional vertex, point cloud V under current camera coordinate system _i-1 The three-dimensional vertex matched with the three-dimensional vertex is V _i-1 (h _i-1 ) The method comprises the steps of carrying out a first treatment on the surface of the In the initialization process, the camera space position T of the ith frame is firstly set _i Set to T _i-1 Then pass through T _i-1 Will V _i (u _i ) Converting to the i-1 frame camera coordinate system, and thenProjecting it into the pixel coordinate system of the i-1 frame to obtain a pixel pointWill +.>And a translation vector omega _i Superposition to obtain h _i-1 Then is connected with V _i (u _i ) The matching points are

In step (3), a loss function of the temperature consistency constraint is constructed:

using lie algebra to represent the delta deltat of the spatial transformation matrix T as a 6-dimensional vector ζ= (α, β, γ, T) _x ,t _y ,t _z ) ^T The method comprises the steps of carrying out a first treatment on the surface of the The update mode of the kth iteration is:

T _i ^k ＝ΔTT _i ^k-1 ≈(I+ξ^)T _i ^k-1 (8)；

according to the chain derivation rule, there are:

updated T through continuous iteration _i Up to E after the kth and k-1 th iterations _fi Less than a preset threshold, and entering the step (4);

in step (4), an i-th frame of infrared image is displayedAnd i-1 th frame infrared image +.>Adding the temperature consistency of the (c) to the optimized loss function in a weighted manner to construct the optimized loss function as shown in formula (13):

in the method, in the process of the invention,and->Representing a geometrical consistency loss function and a temperature consistency optimization function, respectively, < >>Representing weights occupied by the temperature consistency constraint in optimizing the loss function;

the geometric consistency loss function is realized by minimizing the point-to-plane distance of a matching point pair between the point cloud of the current frame and the point cloud of the model;

space vertex V _i (h) To its matching pointThe distance of the model plane is as follows:

in the method, in the process of the invention,for the normal vector diagram of the model under the viewing angle of the i-1 frame, constructing +. >As shown in formula (15):

wherein L is the number of elements of the matching point pair set O;and v _i ＝V _i (h) Respectively representing a pair of matching points in the current input point cloud and the i-1 frame model point cloud;

solving for ζ by equation (18) to achieve E _cr Is to be minimized:

in the method, in the process of the invention,

through continuous iteration updating T _i Until after the kth and the kth-1 iterationsIf the threshold value is smaller than the preset threshold value, the step (5) is carried out;

in step (5), constructing a multi-source joint optimization loss function as shown in formula (19):

in the method, in the process of the invention,optimizing an objective function for spatial geometrical consistency, +.>Optimizing the objective function for temperature consistency, +.>Is its corresponding weight, ++>Optimizing the objective function for visible light photometric consistency,/->Is its corresponding weight;

the collected visible light image is converted from a three primary color space to a YUV color space, wherein components U and V represent chromaticity, and component Y represents brightness, and the calculation mode is as follows:

Y＝0.299R+0.587G+0.114B；

C _v (Y)＝max(Y(x))-min(Y(x)) x∈Ω；

wherein Ω is a pixel space of the visible light brightness image;

thenThe calculation method of (1) is as follows:

wherein H is the maximum value of the luminance image pixel value, and h=255;

constructionThe method comprises the following steps:

according to the chain derivation rule:

in the method, in the process of the invention,

2. The method for constructing the three-dimensional surface temperature model of the human body based on multi-source information fusion according to claim 1, wherein in the step (6), based on the depth image, the point cloud generated according to the depth image is converted into the camera coordinate systems of the infrared camera and the visible light camera through space rigidity transformation by using the external parameters of the camera, and then the point cloud is projected into the corresponding image coordinate systems to realize the correspondence between the images.

3. The method for constructing a three-dimensional surface temperature model of a human body based on multi-source information fusion as claimed in claim 2, wherein the external parameters of the infrared camera relative to the depth cameraAnd at t _di To t _ti Pose transformation of depth camera in between->Then there are:

external IR camera based on Liqun lie algebraThe delta deltat of the parameter matrix is converted into a six-dimensional vector xi _e ＝(α，β，γ，t _χ ，t _y ，t _z ) ^T The method comprises the steps of carrying out a first treatment on the surface of the The updating mode of the k iteration of real-time optimization of the external parameters of the infrared camera is as follows:

combining formula (29) and formula (30), E _et Linearization is as follows:

the linearization result of formula (35) is:

continuously iteratively updating the depth camera at t _di To t _vi Spatial pose transformation in timeLoss function E after up to the kth and k-1 iterations _ev Is smaller than a preset threshold, and the external parameter matrix of the infrared camera is +.>And matching the visible light image and the depth image by using the optimized external parameters.