CN114708331B - Calibration method and device for depth camera, electronic equipment and storage medium - Google Patents
Calibration method and device for depth camera, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN114708331B CN114708331B CN202210177896.5A CN202210177896A CN114708331B CN 114708331 B CN114708331 B CN 114708331B CN 202210177896 A CN202210177896 A CN 202210177896A CN 114708331 B CN114708331 B CN 114708331B
- Authority
- CN
- China
- Prior art keywords
- matrix
- key point
- depth camera
- point pairs
- equation set
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
- G06T2207/30201—Face
Landscapes
- Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Studio Devices (AREA)
- Image Processing (AREA)
Abstract
The embodiment of the application relates to the technical field of machine vision, and discloses a method and a device for calibrating a depth camera, electronic equipment and a storage medium, wherein the method comprises the following steps: respectively carrying out face detection on the infrared image and the color image shot by the depth camera to obtain a plurality of key point pairs; wherein the keypoint pairs comprise a first keypoint on the infrared map and a second keypoint on the color map; constructing an overdetermined equation set according to the key point pairs; performing least square solution and singular value decomposition on the over-determined equation set to obtain a basic matrix; decomposing the basic matrix to obtain a resolved rotation matrix and a resolved translation matrix; the rotation matrix of the depth camera is calibrated by using the calculated rotation matrix, and the translation matrix of the depth camera is calibrated by using the calculated translation matrix, so that the alignment precision of the three images can be greatly improved under the condition of not arranging additional device devices, the cost is low, and the robustness in the calibration process is high.
Description
Technical Field
The embodiment of the application relates to the technical field of machine vision, in particular to a method and a device for calibrating a depth camera, electronic equipment and a storage medium.
Background
The depth camera can acquire the depth information of a target object in real time, and is technically supported for complex application scenes such as motion capture identification, face identification, three-dimensional modeling in the field of automatic driving, cruising and obstacle avoidance, part scanning detection and sorting in the field of industry, monitoring in the field of security protection, people counting and the like, so that the depth camera has wide consumption level and industrial level application requirements, and the depth camera is used in the application scenes, so that the color image, the infrared image and the depth image which are shot by the depth camera are required to be aligned in pixels, more characteristic information of the target object can be acquired, the characteristic dimension of a single pixel point is increased, and the effect and the precision of depth imaging are improved.
The algorithm for aligning the color map, the infrared map and the depth map needs to perform pixel-by-pixel calculation on internal parameters (internal parameters) of the color lens and the infrared lens, external parameters (external parameters) of the color lens and the infrared lens and depth data of a target object, the internal parameters and the external parameters of the lens are calibrated before the camera comes out and are burnt into the internal storage of the camera so as to be accessed and obtained when needed, but along with long-time use, the camera is difficult to avoid being influenced by a high-temperature environment, a low-temperature environment and a humid environment and also difficult to encounter the problem of stress release of an assembly structure of the camera, the internal structure of the camera is deformed due to the factors, the relative relation between the infrared lens and the color lens is caused, namely the external parameters of the color lens and the infrared lens are deviated, the deviation can reduce the accuracy of aligning the three maps and further influence the accuracy and reliability of downstream application, and the problem caused by the external parameters of the camera can be relieved to a certain extent by reinforcing the internal structure of the camera, but the cost for reinforcing the structure design of the camera is more complicated and the robustness is not high.
Disclosure of Invention
An object of the embodiments of the present application is to provide a calibration method and apparatus for a depth camera, an electronic device, and a storage medium, which can greatly improve the alignment accuracy of three images without providing additional apparatus devices, and have low cost and high robustness in the calibration process.
In order to solve the above technical problem, an embodiment of the present application provides a calibration method for a depth camera, including the following steps: respectively carrying out face detection on the infrared image and the color image shot by the depth camera to obtain a plurality of key point pairs; wherein the infrared image has the same content as the color image, and the key point pairs comprise a first key point on the infrared image and a second key point on the color image; constructing an overdetermined equation set according to the plurality of key point pairs; performing least square solution and singular value decomposition on the over-determined equation set to obtain a basic matrix; decomposing the basic matrix to obtain a resolved rotation matrix and a resolved translation matrix; calibrating a rotation matrix of the depth camera using the resolved rotation matrix, and calibrating a translation matrix of the depth camera using the resolved translation matrix.
Embodiments of the present application further provide a calibration apparatus for a depth camera, including: the system comprises a face detection module, a construction and calculation module and a calibration module; the face detection module is used for respectively carrying out face detection on an infrared image and a color image shot by a depth camera to obtain a plurality of key point pairs, wherein the infrared image and the color image have the same content, and the key point pairs comprise a first key point on the infrared image and a second key point on the color image; the building and resolving module is used for building an overdetermined equation set according to the key point pairs, performing least square method solving and singular value decomposition on the overdetermined equation set to obtain a basic matrix, and resolving the basic matrix to obtain a resolved rotation matrix and a resolved translation matrix; the calibration module is used for calibrating the rotation matrix of the depth camera by using the calculated rotation matrix and calibrating the translation matrix of the depth camera by using the calculated translation matrix.
An embodiment of the present application further provides an electronic device, including: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the above-described depth camera calibration method.
Embodiments of the present application further provide a computer-readable storage medium storing a computer program, which when executed by a processor, implements the calibration method of the depth camera.
The calibration method, the calibration device, the electronic device and the storage medium of the depth camera provided by the embodiment of the application respectively perform face detection on a group of infrared images and color images with the same content shot by the depth camera to obtain a plurality of key point pairs, the obtained key point pairs comprise first key points on the infrared images and second key points on the color images, an over-determined equation set is constructed according to the obtained key point pairs, least square solution and singular value decomposition are performed on the over-determined equation set to obtain a basic matrix, the server decomposes the basic matrix to obtain a solved rotation matrix and a solved translation matrix, finally the rotation matrix of the depth camera is calibrated by the solved rotation matrix, the translation matrix of the depth camera is calibrated by the solved translation matrix, the calibrated rotation matrix and the calibrated translation matrix can bring changes of a relative relation between an infrared lens and a color lens of the depth camera into a three-image alignment process, namely, the influence caused by external parameter deviation of the color lens and the infrared lens is eliminated, no matter whether the internal structure of the depth camera is performed, or materials with high hardness are selected, the influence of the production of the infrared lens and the infrared lens is greatly reduced, and the influence of the infrared lens is greatly increased by the implementation cost of the high-added calibration device.
In addition, after the obtaining the number of key point pairs, before the constructing the over-determined equation set according to the number of key point pairs, the method includes: determining a first transformation matrix corresponding to the infrared image and a second transformation matrix corresponding to the color image according to the total number of the key point pairs, the coordinates of each key point pair and a preset normalization formula; wherein the coordinates of the keypoint pair comprise the coordinates of the first keypoint and the coordinates of the second keypoint; performing isotropic transformation on the plurality of key point pairs according to the coordinates of each key point pair, the first transformation matrix and the second transformation matrix to obtain a plurality of normalized key point pairs; constructing an overdetermined equation set according to the plurality of key point pairs comprises the following steps: an over-determined equation set is constructed according to a plurality of normalized key point pairs, although a basic matrix can be solved and a rotation matrix and a translation matrix are further solved based on the key point pairs, the process of solving the basic matrix is unstable due to the existence of noise and the existence of numerical value selection errors, so that the original image is firstly subjected to normalization processing and isotropic transformation based on coordinates of each key point pair, and the interference of the noise on the solution of the basic matrix is reduced.
In addition, the performing least square solution and singular value decomposition on the over-determined equation set to obtain a basis matrix includes: performing least square solution and singular value decomposition on the over-determined equation set to obtain a normalized basic matrix; the normalized basic matrix is subjected to inverse normalization according to the first transformation matrix and the second transformation matrix to obtain a basic matrix, an overdetermined equation set constructed based on the normalized characteristic point pairs is solved according to the embodiment of the application, the obtained basic matrix is also normalized and cannot be directly decomposed, inverse normalization is required according to the first transformation matrix and the second transformation matrix to obtain the basic matrix, and therefore the fact that the calculated translation matrix and rotation matrix are correct and accord with the actual situation of the depth camera is guaranteed.
In addition, the performing least square solution and singular value decomposition on the over-determined equation set to obtain a basis matrix includes: constructing a solving matrix Q with N rows and 9 columns, and determining a least square solution of the over-determined equation set according to a least square method and the solving matrix Q; wherein N is the total number of the feature point pairs, and the least square solution is a matrix Q T The feature vector corresponding to the minimum feature value of (1); converting the least square solution into a coarse basis matrix with 3 rows and 3 columns, and performing singular value decomposition on the coarse basis matrix; the coarse basis matrix after singular value decomposition comprises a left singular vector, a singular value matrix and a right singular vector; setting singular values of 3 rd row and 3 rd column in the singular value matrix to be 0 to obtain a basic matrix, and using a least square method and singular value decomposition, the method can greatly reduce the difficulty of obtaining the basic matrix, reduce the operand, and improve the calibration speed of the depth camera, thereby improving the use experience of a user。
Drawings
One or more embodiments are illustrated by the corresponding figures in the drawings, which are not meant to be limiting.
FIG. 1 is a first flowchart of a method of calibrating a depth camera according to one embodiment of the present application;
FIG. 2 is a second flowchart of a calibration method of a depth camera according to another embodiment of the present application;
FIG. 3 is a first flowchart illustrating a least squares solution and singular value decomposition of an over-determined set of equations to obtain a basis matrix, according to an embodiment of the present application;
FIG. 4 is a flow chart of a second method for solving the over-determined set of equations by least squares and singular value decomposition to obtain a basis matrix according to an embodiment of the present application;
FIG. 5 is a schematic diagram of a calibration arrangement for a depth camera according to another embodiment of the present application;
fig. 6 is a schematic structural diagram of an electronic device according to another embodiment of the present application.
Detailed Description
To make the objects, technical solutions and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below with reference to the accompanying drawings. However, it will be appreciated by those of ordinary skill in the art that in the examples of the present application, numerous technical details are set forth in order to provide a better understanding of the present application. However, the technical solution claimed in the present application can be implemented without these technical details and various changes and modifications based on the following embodiments. The following embodiments are divided for convenience of description, and should not constitute any limitation to the specific implementation manner of the present application, and the embodiments may be mutually incorporated and referred to without contradiction.
An embodiment of the present application relates to a calibration method for a depth camera, which is applied to an electronic device, where the electronic device may be a terminal or a server, and the electronic device in this embodiment and the following embodiments are described by taking the server as an example.
A specific flow of the calibration method of the depth camera according to this embodiment may be as shown in fig. 1, and includes:
Specifically, the server may perform face detection on the infrared image and the color image taken by the depth camera, respectively, to obtain a plurality of key point pairs, where the key point pairs include a first key point on the infrared image and a second key point on the color image.
In a specific implementation, when a server calibrates a depth camera, the server may perform face detection on an infrared image and a color image captured by the depth camera according to a preset face detection algorithm, to detect a plurality of key points in the infrared image and a plurality of key points in the color image, because the contents of the infrared image and the color image are the same, and the server performs face detection on the infrared image and the color image, the used face detection algorithms are the same, so that the key points in the infrared image and the color image are in one-to-one correspondence regardless of positions or characteristics, such two key points form a group of key point pairs, each group of key point pairs includes a first key point on the infrared image and a second key point on the color image, wherein the preset face detection algorithm may be set by a technician in the field according to actual needs, and embodiments of the present application are not specifically limited to this.
In one example, the server detects 68 keypoints in each of the infrared map and the color map, respectively, and based on the keypoint pairs, 68 groups of keypoints are formed.
And 102, constructing an over-determined equation set according to the key point pairs.
Specifically, the base matrix F is a 3-by-3 matrix with 9 unknown parameters, and the server can estimate the base matrix by an 8-point method, assuming that the coordinates of a group of key point pairs are p 1 ＝[a 1 ，b 1 ，1]，p 2 ＝[a 2 ，b 2 ，1]From the base matrix, equation one can be derived:
the server may treat the elements of the basis matrix F as a vector: f = [ f = [ f ] 1 ,f 2 ,f 3 ,f 4 ,f 5 ,f 6 ,f 7 ,f 8 ,f 9 ]The above equation can be rewritten as equation two:
[a 1 a 2 ,a 1 b 2 ,a 1 ,b 1 a 2 ,b 1 b 2 ,b 1 ,a 2 ,b 2 ,1]*f＝0
the server can obtain each element of the basis matrix F by solving the equation two, but the basis matrix F obtained by the above equation set is very unstable due to noise, rounding errors of numerical values, and the like.
In a specific implementation, in order to make the estimated basic matrix F more stable, the server may obtain a plurality of over-determined equations respectively according to the detected key point pairs, construct an over-determined equation set based on the over-determined equations, and solve the over-determined equation set to obtain the basic matrix F which is a stable and reliable basic matrix.
And 103, performing least square solution and singular value decomposition on the over-determined equation set to obtain a basic matrix.
Specifically, an overdetermined equation set, namely an equation set with the number of equations larger than the number of unknowns, is not accurately solved, the server can use a least square method, namely a curve fitting mode, to solve a closest solution, namely a least square solution, for the overdetermined equation set, the server firstly carries out the least square method solution on the overdetermined equation set constructed according to a plurality of key point pairs to obtain the least square solution of the overdetermined equation set, and then carries out singular value decomposition (SVD decomposition) on the solved overdetermined equation set, so that a basic matrix is obtained.
And 104, decomposing the basic matrix to obtain a resolved rotation matrix and a resolved translation matrix.
Specifically, after the server calculates the basis matrix F, the basis matrix F may be decomposed according to the relationship between the rotation matrix and the translation matrix, so as to obtain the solved rotation matrix R and the solved translation matrix T.
And 105, calibrating the rotation matrix of the depth camera by using the calculated rotation matrix, and calibrating the translation matrix of the depth camera by using the calculated translation matrix.
Specifically, the solved rotation matrix R and the solved translation matrix T already cover the external parameter changes of the infrared lens and the color lens, that is, there is a certain difference compared with the initial rotation matrix and translation matrix of the depth camera, the server calibrates the rotation matrix of the depth camera with the solved rotation matrix, and calibrates the translation matrix of the depth camera with the solved translation matrix, thereby implementing calibration of the depth camera, and the calibrated depth camera performs shooting again, which can greatly improve the accuracy of three-image alignment.
In one example, the server may directly replace the rotation matrix of the depth camera with the solved rotation matrix and replace the translation matrix of the depth camera with the solved translation matrix.
In this embodiment, the server performs face detection on a group of infrared images and color images with the same content photographed by the depth camera to obtain a plurality of key point pairs, where the obtained key point pairs include a first key point on the infrared image and a second key point on the color image, then constructs an overdetermined equation set according to the obtained key point pairs, and performs least square solution and singular value decomposition on the overdetermined equation set to obtain a base matrix, the server decomposes the base matrix to obtain a solved rotation matrix and a solved translation matrix, and finally calibrates the rotation matrix of the depth camera with the solved rotation matrix, and calibrates the translation matrix of the depth camera with the solved translation matrix, and the calibrated rotation matrix and translation matrix can reinforce the relative relationship between the infrared lens and the color lens of the depth camera in the alignment process of the three images, that is, the influence caused by the external reference deviation of the color lens and the infrared lens is eliminated, and considering that no matter whether the reinforcement is performed in the internal structure of the depth camera, or the production of the camera using a component-separate design is included in the stacked mode, the influence of the external reference deviation caused by the high degree of the external reference, that the implementation cost is greatly increased, and the implementation cost of the high-cost of the infrared camera is not increased.
Another embodiment of the present application relates to a calibration method for a depth camera, and the following details of the implementation of the calibration method for a depth camera according to this embodiment are specifically described, and the following details are provided only for facilitating understanding of the implementation and are not necessary to implement the present solution, and a specific flow of the calibration method for a depth camera according to this embodiment may be as shown in fig. 2, and includes:
And 203, performing isotropic transformation on the plurality of key point pairs according to the coordinates of each key point pair, the first transformation matrix and the second transformation matrix to obtain a plurality of normalized key point pairs.
In specific implementation, although a basic matrix can be solved based on key point pairs, and a rotation matrix and a translation matrix are further solved, because the existence of noise and the existence of numerical value accepting and rejecting errors, the process of solving the basic matrix is unstable, so that a server can reduce the interference of noise on the solved basic matrix by adopting a normalization mode on all key point pairs, the server firstly determines a first transformation matrix corresponding to an infrared image and a second transformation matrix corresponding to a color image according to the total number of the key point pairs, the coordinates of each key point pair and a preset normalization formula, and then carries out isotropic transformation on a plurality of key point pairs according to the coordinates of each key point pair, the first transformation matrix and the second transformation matrix to obtain a plurality of normalized key point pairs, wherein the coordinates of the key point pairs comprise the coordinates of a first key point and the coordinates of a second key point, and the server carries out isotropic transformation on the first key point according to the coordinates of the first key point and the first transformation matrix to obtain a normalized first key point; and carrying out isotropic transformation on the second key points according to the coordinates of the second key points and the second transformation matrix to obtain normalized second key points, thereby obtaining a plurality of normalized key point pairs.
In one example, the server determines the first transformation matrix corresponding to the infrared map and the second transformation matrix corresponding to the color map according to the total number of the key point pairs, the coordinates of each key point pair, and a preset normalization formula, and the determination can be implemented by the following formulas:
in the formula u 1i Is the abscissa, v, of the ith first keypoint 1i Is the ordinate, u, of the ith first keypoint 2i Is the abscissa, v, of the ith second keypoint 2i Is the ordinate of the ith second keypoint, N is the total number of keypoints pairs, H 1 For a first transformation matrix corresponding to the infrared map, H 2 And the second transformation matrix is corresponding to the color map.
In one example, the server performs isotropic transformation on the first keypoint according to the coordinate of the first keypoint and the first transformation matrix to obtain a normalized first keypoint, that is, the coordinate of the first keypoint is multiplied by the first transformation matrix to obtain the coordinate of the normalized first keypoint.
In one example, the server performs isotropic transformation on the second keypoint according to the coordinate of the second keypoint and the second transformation matrix to obtain a normalized second keypoint, that is, the coordinate of the second keypoint is multiplied by the second transformation matrix to obtain the coordinate of the normalized second keypoint.
And step 204, constructing an over-determined equation set according to the plurality of normalized key point pairs.
Specifically, the server can respectively obtain a plurality of over-determined equations according to a plurality of normalized key point pairs, an over-determined equation set is constructed based on the over-determined equations, and the basic matrix F obtained by solving the over-determined equation set is a more stable and reliable basic matrix.
And step 205, performing least square solution and singular value decomposition on the over-determined equation set to obtain a basic matrix.
In one example, the server performs least squares solution and singular value decomposition on the over-determined equation set to obtain the basis matrix, which can be implemented by each sub-step shown in fig. 3, and specifically includes:
and a substep 2051 of performing least square solution and singular value decomposition on the over-determined equation set to obtain a normalized basis matrix.
And substep 2052, performing inverse normalization on the normalized basis matrix according to the first transformation matrix and the second transformation matrix to obtain a basis matrix.
Specifically, the server normalizes the key point pairs before constructing the over-determined equation set, so that the constructed over-determined equation set is actually a normalized over-determined equation set, the server performs least square solution and singular value decomposition on the over-determined equation set to obtain a normalized basic matrix which cannot be directly used, and the server needs to perform inverse normalization on the normalized basic matrix according to the first transformation matrix and the second transformation matrix to obtain the basic matrix, so that the solved translation matrix and rotation matrix are correct and accord with the actual situation of the depth camera.
In one example, the server performs inverse normalization on the normalized basis matrix according to the first transformation matrix and the second transformation matrix to obtain a basis matrix, which can be implemented by the following formula:
in the formula, H 1 Is a first transformation matrix, H 2 For the second transformation matrix, F' is the normalized basis matrix and F is the basis matrix.
And step 206, decomposing the basic matrix to obtain a resolved rotation matrix and a resolved translation matrix.
Step 206 to step 207 are substantially the same as step 104 to step 105, and are not described herein again.
In this embodiment, after obtaining the plurality of key point pairs, before constructing the over-determined equation set according to the plurality of key point pairs, the method includes: determining a first transformation matrix corresponding to the infrared image and a second transformation matrix corresponding to the color image according to the total number of the key point pairs, the coordinates of each key point pair and a preset normalization formula; wherein the coordinates of the key point pair comprise the coordinates of the first key point and the coordinates of the second key point; performing isotropic transformation on the plurality of key point pairs according to the coordinates of each key point pair, the first transformation matrix and the second transformation matrix to obtain a plurality of normalized key point pairs; the constructing of the over-determined equation set according to the plurality of key point pairs comprises the following steps: an over-determined equation set is constructed according to a plurality of normalized key point pairs, although a basic matrix can be solved based on the original key point pairs, and a rotation matrix and a translation matrix are further solved, due to the existence of noise and the existence of numerical value accepting and rejecting errors, the process of solving the basic matrix is unstable, so that the original image is subjected to normalization processing and isotropic transformation based on coordinates of each key point pair, and the interference of the noise on the solving of the basic matrix is reduced.
In one embodiment, the server performs least square solution and singular value decomposition on the over-determined equation set to obtain the basis matrix, which may be implemented by the steps shown in the figure, specifically including:
and 301, constructing a solving matrix Q with N rows and 9 columns, and determining a least square solution of the over-determined equation set according to a least square method and the solving matrix Q.
In specific implementation, when the server carries out least square method solving on the over-determined equation set, a solving matrix Q with N rows and 9 columns is constructed firstly, N is the total number of key point pairs, and the server determines the least square solution of the over-determined equation set to be the matrix Q & ltQ & gt according to the least square method and the solving matrix Q T The minimum eigenvalue of (2) corresponds to the eigenvector.
In one example, when the server performs face detection on a color image and an infrared image, 68 sets of key point pairs are detected, so that when the server performs least square solution on an over-determined equation set, a solution matrix Q with 68 rows and 9 columns needs to be constructed.
Specifically, the server converts the least square solution into a coarse basis matrix F ″ with 3 rows and 3 columns, and performs singular value decomposition on the coarse basis matrix F ″ and the coarse basis matrix F ″ after the singular value decomposition includes a left singular vector, a singular value matrix, and a right singular vector.
In one example, the coarse basis matrix F "may be represented by the following formula: f "= S diag (d) 1 ，d 2 ，d 3 )*V T In the formula, S is a left singular vector, V T As right singular vector, diag (d) 1 ，d 2 ，d 3 ) As a matrix of singular values, d 1 Singular values of row 1, column 1, d 2 Singular values of row 2, column 2, d 3 The singular values in row 3, column 3, and F' are the coarse basis matrices.
Specifically, the base matrix F has an important property that the rank of the base matrix F is 2, so the server sets the singular value in row 3, column 3 in the singular value matrix in the coarse base matrix F ″ to 0, i.e., the base matrix F.
In one example, the basis matrix F may be represented by the following formula: f = S × diag (d) 1 ，d 2 ，0)*V T In the formula, S is a left singular vector, V T As right singular vector, diag (d) 1 ，d 2 0) singular value matrix with singular values of row 3, column 3 to 0, and F is the basis matrix.
In this embodiment, the performing least square solution and singular value decomposition on the overdetermined equation set to obtain a basis matrix includes: constructing a solving matrix Q with N rows and 9 columns, and determining a least square solution of the over-determined equation set according to a least square method and the solving matrix Q; wherein, N is the total number of the characteristic point pairs, and the least square solution is the characteristic vector corresponding to the minimum characteristic value of the matrix QxQT; converting the least square solution into a coarse basis matrix with 3 rows and 3 columns, and performing singular value decomposition on the coarse basis matrix; the coarse basis matrix after singular value decomposition comprises a left singular vector, a singular value matrix and a right singular vector; the singular values of the 3 rd row and the 3 rd column in the singular value matrix are set to be 0 to obtain a basic matrix, and the least square method and the singular value decomposition are used, so that the difficulty in obtaining the basic matrix can be greatly reduced, the calculation amount is reduced, the calibration speed of the depth camera is improved, and the use experience of a user is improved.
The steps of the above methods are divided for clarity, and the implementation may be combined into one step or split some steps, and the steps are divided into multiple steps, so long as the same logical relationship is included, which are within the scope of the present patent; it is within the scope of this patent to add insignificant modifications or introduce insignificant designs to the algorithms or processes, but not to change the core designs of the algorithms and processes.
Another embodiment of the present application relates to a calibration apparatus for a depth camera, and the implementation details of the calibration apparatus for a depth camera of the present embodiment are specifically described below, and the following are provided only for the convenience of understanding, and are not necessary for implementing the present invention, and a schematic diagram of the calibration apparatus for a depth camera of the present embodiment may be as shown in fig. 5, and includes: a face detection module 401, a construction and solution module 402 and a calibration module 403.
The face detection module 401 is configured to perform face detection on an infrared image and a color image captured by a depth camera, respectively, to obtain a plurality of key point pairs, where the infrared image and the color image have the same content, and the key point pairs include a first key point on the infrared image and a second key point on the color image.
The building and resolving module 402 is configured to build an overdetermined equation set according to the plurality of key point pairs, perform least square solution and singular value decomposition on the overdetermined equation set to obtain a basic matrix, and decompose the basic matrix to obtain a resolved rotation matrix and a resolved translation matrix.
The calibration module 403 is configured to calibrate the rotation matrix of the depth camera with the solved rotation matrix, and calibrate the translation matrix of the depth camera with the solved translation matrix.
It should be noted that, all the modules involved in this embodiment are logic modules, and in practical application, one logic unit may be one physical unit, may also be a part of one physical unit, and may also be implemented by a combination of multiple physical units. In addition, in order to highlight the innovative part of the present application, a unit which is not so closely related to solve the technical problem proposed by the present application is not introduced in the present embodiment, but this does not indicate that no other unit exists in the present embodiment.
Another embodiment of the present application relates to an electronic device, as shown in fig. 6, including: at least one processor 501; and a memory 502 communicatively coupled to the at least one processor 501; wherein the memory 502 stores instructions executable by the at least one processor 501, the instructions being executable by the at least one processor 501 to enable the at least one processor 501 to perform the calibration method of the depth camera in the above embodiments.
Where the memory and processor are connected by a bus, the bus may comprise any number of interconnected buses and bridges, the buses connecting together one or more of the various circuits of the processor and the memory. The bus may also connect various other circuits such as peripherals, voltage regulators, power management circuits, and the like, which are well known in the art, and therefore, will not be described any further herein. A bus interface provides an interface between the bus and the transceiver. The transceiver may be one element or a plurality of elements, such as a plurality of receivers and transmitters, providing a means for communicating with various other apparatus over a transmission medium. The data processed by the processor is transmitted over a wireless medium via an antenna, which further receives the data and transmits the data to the processor.
The processor is responsible for managing the bus and general processing and may also provide various functions including timing, peripheral interfaces, voltage regulation, power management, and other control functions. While the memory may be used to store data used by the processor in performing operations.
Another embodiment of the present application relates to a computer-readable storage medium storing a computer program. The computer program realizes the above-described method embodiments when executed by a processor.
That is, as can be understood by those skilled in the art, all or part of the steps in the method for implementing the embodiments described above may be implemented by a program instructing related hardware, where the program is stored in a storage medium and includes several instructions to enable a device (which may be a single chip, a chip, or the like) or a processor (processor) to execute all or part of the steps of the method described in the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a U disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk.
It will be understood by those of ordinary skill in the art that the foregoing embodiments are specific examples of implementations of the present application and that various changes in form and details may be made therein without departing from the spirit and scope of the present application.
Claims (9)
1. A method of calibrating a depth camera, comprising:
respectively carrying out face detection on an infrared image and a color image shot by a depth camera to obtain a plurality of key point pairs; the infrared image and the color image have the same content, the key point pairs comprise a first key point on the infrared image and a second key point on the color image, the infrared image is shot by an infrared lens of the depth camera, and the color image is shot by a color lens of the depth camera;
constructing an overdetermined equation set according to the plurality of key point pairs;
performing least square solution and singular value decomposition on the over-determined equation set to obtain a basic matrix;
decomposing the basic matrix to obtain a resolved rotation matrix and a resolved translation matrix;
calibrating a rotation matrix of the depth camera with the solved rotation matrix and calibrating a translation matrix of the depth camera with the solved translation matrix; the solved rotation matrix and the solved translation matrix are used for eliminating the external parameter deviation between the infrared lens and the color lens;
after the obtaining a plurality of key point pairs, before the constructing an over-determined equation set according to the plurality of key point pairs, the method includes:
determining a first transformation matrix corresponding to the infrared image and a second transformation matrix corresponding to the color image according to the total number of the key point pairs, the coordinates of each key point pair and a preset normalization formula; wherein the coordinates of the key point pair comprise the coordinates of the first key point and the coordinates of the second key point;
performing isotropic transformation on the plurality of key point pairs according to the coordinates of the key point pairs, the first transformation matrix and the second transformation matrix to obtain a plurality of normalized key point pairs;
the constructing of the over-determined equation set according to the plurality of key point pairs comprises the following steps:
and constructing an overdetermined equation set according to the plurality of normalized key point pairs.
2. The method for calibrating a depth camera according to claim 1, wherein the first transformation matrix corresponding to the infrared image and the second transformation matrix corresponding to the color image are determined according to the total number of the key point pairs, the coordinates of each key point pair, and a preset normalization formula by using the following formulas:
wherein u is 1i Is the abscissa, v, of the ith said first keypoint 1i Is the ordinate, u, of the ith said first keypoint 2i Is the abscissa, v, of the ith said second keypoint 2i Is the ordinate of the ith second keypoint, N is that of the keypoint pairTotal number, H 1 Is a first transformation matrix, H, corresponding to the infrared map 2 And the second transformation matrix corresponds to the color map.
3. The method for calibrating a depth camera according to claim 1, wherein the performing least squares solution and singular value decomposition on the over-determined system of equations to obtain a basis matrix comprises:
performing least square solution and singular value decomposition on the over-determined equation set to obtain a normalized basic matrix;
and performing inverse normalization on the normalized basic matrix according to the first transformation matrix and the second transformation matrix to obtain a basic matrix.
4. The method for calibrating a depth camera according to claim 3, wherein the normalized basis matrix is de-normalized according to the first transformation matrix and the second transformation matrix to obtain a basis matrix by the following formula:
wherein H 1 For the first transformation matrix, H 2 For said second transformation matrix, F ′ Is the normalized basis matrix and F is the basis matrix.
5. The method for calibrating a depth camera according to any one of claims 1-2, wherein the performing least squares solution and singular value decomposition on the over-determined system of equations to obtain a basis matrix comprises:
constructing a solving matrix Q with N rows and 9 columns, and determining a least square solution of the over-determined equation set according to a least square method and the solving matrix Q; wherein N is the total number of the key point pairs, and the least square solution is a matrix Q T The feature vector corresponding to the minimum feature value of (4);
converting the least square solution into a coarse basis matrix with 3 rows and 3 columns, and performing singular value decomposition on the coarse basis matrix; the coarse basis matrix after singular value decomposition comprises a left singular vector, a singular value matrix and a right singular vector;
and setting the singular value of the 3 rd row and the 3 rd column in the singular value matrix to be 0 to obtain a basic matrix.
6. The method of calibrating a depth camera of claim 5, wherein the base matrix is represented by the following formula:
F＝S*diag(d 1 ，d 2 ，0)*V T
wherein S is the left singular vector, V T For the right singular vector, diag (d) 1 ，d 2 0) a singular value matrix with singular values of row 3, column 3 to 0 and F the basis matrix.
7. A calibration device for a depth camera, comprising: the system comprises a face detection module, a construction and calculation module and a calibration module;
the face detection module is used for respectively carrying out face detection on an infrared image and a color image shot by a depth camera to obtain a plurality of key point pairs, wherein the infrared image and the color image have the same content, the key point pairs comprise a first key point on the infrared image and a second key point on the color image, the infrared image is shot by an infrared lens of the depth camera, and the color image is shot by a color lens of the depth camera;
the building and resolving module is used for building an overdetermined equation set according to the key point pairs, performing least square method solving and singular value decomposition on the overdetermined equation set to obtain a basic matrix, and decomposing the basic matrix to obtain a resolved rotation matrix and a resolved translation matrix;
the calibration module is used for calibrating the rotation matrix of the depth camera by using the solved rotation matrix and calibrating the translation matrix of the depth camera by using the solved translation matrix; the solved rotation matrix and the solved translation matrix are used for eliminating the external parameter deviation between the infrared lens and the color lens;
after the obtaining the plurality of key point pairs, before the constructing the over-determined equation set according to the plurality of key point pairs, the method includes:
determining a first transformation matrix corresponding to the infrared image and a second transformation matrix corresponding to the color image according to the total number of the key point pairs, the coordinates of each key point pair and a preset normalization formula; wherein the coordinates of the key point pair comprise the coordinates of the first key point and the coordinates of the second key point;
performing isotropic transformation on the plurality of key point pairs according to the coordinates of each key point pair, the first transformation matrix and the second transformation matrix to obtain a plurality of normalized key point pairs;
constructing an overdetermined equation set according to the plurality of key point pairs comprises the following steps:
and constructing an overdetermined equation set according to the plurality of normalized key point pairs.
8. An electronic device, comprising:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform a method of calibrating a depth camera according to any one of claims 1 to 6.
9. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out a method of calibrating a depth camera according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210177896.5A CN114708331B (en) | 2022-02-24 | 2022-02-24 | Calibration method and device for depth camera, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210177896.5A CN114708331B (en) | 2022-02-24 | 2022-02-24 | Calibration method and device for depth camera, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114708331A CN114708331A (en) | 2022-07-05 |
| CN114708331B true CN114708331B (en) | 2023-03-14 |
Family
ID=82166417
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210177896.5A Active CN114708331B (en) | 2022-02-24 | 2022-02-24 | Calibration method and device for depth camera, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114708331B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101763632A (en) * | 2008-12-26 | 2010-06-30 | 华为技术有限公司 | Method for demarcating camera and device thereof |
| CN109785393A (en) * | 2018-12-29 | 2019-05-21 | 西安理工大学 | A kind of Camera Self-Calibration method based on plane motion constraint |
| CN112508999A (en) * | 2020-11-20 | 2021-03-16 | 西北工业大学深圳研究院 | Space target motion state identification method based on cooperative observation image sequence |
| CN112629565A (en) * | 2021-03-08 | 2021-04-09 | 中国人民解放军国防科技大学 | Method, device and equipment for calibrating rotation relation between camera and inertial measurement unit |
| CN113902808A (en) * | 2021-08-31 | 2022-01-07 | 北京影谱科技股份有限公司 | Camera calibration method, device, equipment and storage medium |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR102085228B1 (en) * | 2014-03-27 | 2020-03-05 | 한국전자통신연구원 | Imaging processing method and apparatus for calibrating depth of depth sensor |
| CN111080709B (en) * | 2019-11-22 | 2023-05-05 | 大连理工大学 | Multispectral stereo camera self-calibration algorithm based on track feature registration |
-
2022
- 2022-02-24 CN CN202210177896.5A patent/CN114708331B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101763632A (en) * | 2008-12-26 | 2010-06-30 | 华为技术有限公司 | Method for demarcating camera and device thereof |
| CN109785393A (en) * | 2018-12-29 | 2019-05-21 | 西安理工大学 | A kind of Camera Self-Calibration method based on plane motion constraint |
| CN112508999A (en) * | 2020-11-20 | 2021-03-16 | 西北工业大学深圳研究院 | Space target motion state identification method based on cooperative observation image sequence |
| CN112629565A (en) * | 2021-03-08 | 2021-04-09 | 中国人民解放军国防科技大学 | Method, device and equipment for calibrating rotation relation between camera and inertial measurement unit |
| CN113902808A (en) * | 2021-08-31 | 2022-01-07 | 北京影谱科技股份有限公司 | Camera calibration method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114708331A (en) | 2022-07-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108510551B (en) | Method and system for calibrating camera parameters under long-distance large-field-of-view condition | |
| CN112465877B (en) | Kalman filtering visual tracking stabilization method based on motion state estimation | |
| CN112233189B (en) | Multi-depth camera external parameter calibration method and device and storage medium | |
| CN114445506A (en) | Camera calibration processing method, device, equipment and storage medium | |
| CN112929626A (en) | Three-dimensional information extraction method based on smartphone image | |
| CN115393446A (en) | Temperature coefficient calibration method, system, electronic device and storage medium | |
| CN113822920B (en) | Method for acquiring depth information by structured light camera, electronic equipment and storage medium | |
| JP7044016B2 (en) | Line sensor camera calibration device and method | |
| CN114708331B (en) | Calibration method and device for depth camera, electronic equipment and storage medium | |
| CN111127560B (en) | Calibration method and system for three-dimensional reconstruction binocular vision system | |
| KR102265081B1 (en) | System for determining position and attitude of camera using the inner product of vectors and three-dimensional coordinate transformation | |
| CN116597246A (en) | Model training method, target detection method, electronic device and storage medium | |
| CN114677264B (en) | Image alignment method, system, electronic device and storage medium | |
| CN116188591A (en) | Multi-camera global calibration method and device and electronic equipment | |
| CN115018922A (en) | Distortion parameter calibration method, electronic device and computer readable storage medium | |
| CN114897990A (en) | Camera distortion calibration method and system based on neural network and storage medium | |
| CN110232715B (en) | Method, device and system for self calibration of multi-depth camera | |
| CN113298885A (en) | Binocular calibration method and device, equipment and storage medium | |
| Maset et al. | Errors-in-variables anisotropic extended orthogonal procrustes analysis | |
| Dickscheid et al. | Benchmarking automatic bundle adjustment results | |
| CN112734854A (en) | Camera self-calibration method based on trinocular polar line constraint | |
| CN114119680B (en) | Image acquisition method and device, electronic equipment and storage medium | |
| CN110349214B (en) | Object positioning method, terminal and readable storage medium | |
| CN111583317B (en) | Image alignment method and device and terminal equipment | |
| CN115050059A (en) | Model training method, palm posture estimation method, electronic device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |