CN112381886A - Multi-camera-based three-dimensional scene reconstruction method, storage medium and electronic device - Google Patents
Multi-camera-based three-dimensional scene reconstruction method, storage medium and electronic device Download PDFInfo
- Publication number
- CN112381886A CN112381886A CN202011281518.9A CN202011281518A CN112381886A CN 112381886 A CN112381886 A CN 112381886A CN 202011281518 A CN202011281518 A CN 202011281518A CN 112381886 A CN112381886 A CN 112381886A
- Authority
- CN
- China
- Prior art keywords
- cameras
- point cloud
- camera
- point
- matrix
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 42
- 239000011159 matrix material Substances 0.000 claims abstract description 87
- 238000004590 computer program Methods 0.000 claims description 14
- 238000004364 calculation method Methods 0.000 claims description 9
- 238000012163 sequencing technique Methods 0.000 claims description 6
- 230000000007 visual effect Effects 0.000 claims description 6
- 230000000873 masking effect Effects 0.000 claims description 5
- 235000005121 Sorbus torminalis Nutrition 0.000 claims description 4
- 244000152100 Sorbus torminalis Species 0.000 claims description 4
- 230000009466 transformation Effects 0.000 claims description 4
- 230000010354 integration Effects 0.000 claims description 3
- 238000000844 transformation Methods 0.000 claims description 3
- 230000000875 corresponding effect Effects 0.000 description 13
- 238000010586 diagram Methods 0.000 description 11
- 238000004422 calculation algorithm Methods 0.000 description 9
- 230000000694 effects Effects 0.000 description 7
- 230000003190 augmentative effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06T7/85—Stereo camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
- G06T7/344—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods involving models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
- G06T7/75—Determining position or orientation of objects or cameras using feature-based methods involving models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30204—Marker
- G06T2207/30208—Marker matrix
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30244—Camera pose
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computer Graphics (AREA)
- Geometry (AREA)
- Software Systems (AREA)
- Length Measuring Devices By Optical Means (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a multi-camera-based three-dimensional scene reconstruction method, a storage medium and electronic equipment, which comprise the following steps: acquiring a preliminary position relation calibration matrix among a plurality of cameras; acquiring color images and depth images which are respectively shot by a plurality of cameras for a scene to be reconstructed and placed with a plurality of checkerboards, and calibrating a matrix according to the color images and the depth images of the plurality of cameras and the preliminary position relation among the plurality of cameras to obtain a point cloud calibration matrix among the plurality of cameras; splicing the point clouds of the multiple cameras according to the point cloud calibration matrix among the multiple cameras to obtain a reconstructed three-dimensional scene; under the condition that the cameras are fixed, the point clouds of the cameras are registered and spliced by a machine vision method to obtain a reconstructed three-dimensional scene, so that the reconstruction of the three-dimensional scene can be better and faster realized, and the method is more time-saving, labor-saving and universal.
Description
Technical Field
The invention relates to the technical field of image processing, in particular to a multi-camera-based three-dimensional scene reconstruction method, a storage medium and electronic equipment.
Background
In an augmented reality application, in order to enable an experiencer to obtain an immersive experience, a real scene needs to be subjected to scene reconstruction, so that the experiencer has an immersive feeling. In the prior art, an indoor three-dimensional scene reconstruction method based on a double-layer registration method uses a surf (Speeded Up Robust Features) algorithm to extract and match feature points, then maps the matched points to three-dimensional coordinates, and then uses a Random Sample Consensus (Random Sample Consensus) algorithm and an Iterative Closest Point (ICP) algorithm to perform registration.
Disclosure of Invention
The technical problem to be solved by the invention is as follows: provided are a multi-camera-based three-dimensional scene reconstruction method, a storage medium and an electronic device, which can rapidly realize the reconstruction of a three-dimensional scene.
In order to solve the technical problems, the invention adopts the technical scheme that:
a three-dimensional scene reconstruction method based on multiple cameras comprises the following steps:
acquiring a preliminary position relation calibration matrix among a plurality of cameras;
acquiring color images and depth images which are respectively shot by the plurality of cameras for a scene to be reconstructed and provided with the plurality of checkerboards, and calibrating a matrix according to the color images and the depth images of the plurality of cameras and the preliminary position relation among the plurality of cameras to obtain a point cloud calibration matrix among the plurality of cameras;
and splicing the point clouds of the cameras according to the point cloud calibration matrix among the cameras to obtain the reconstructed three-dimensional scene.
In order to solve the technical problem, the invention adopts another technical scheme as follows:
a computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the above-mentioned multi-camera based three-dimensional scene reconstruction method.
In order to solve the technical problem, the invention adopts another technical scheme as follows:
an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the multi-camera based three-dimensional scene reconstruction method as described above when executing the computer program.
The invention has the beneficial effects that: the method comprises the steps of obtaining a preliminary position relation calibration matrix among a plurality of cameras in advance, then placing a plurality of checkerboards in a three-dimensional scene to be reconstructed, obtaining a point cloud calibration matrix among the plurality of cameras according to color images and point clouds of the checkerboards in different cameras, and finally splicing the point clouds of the plurality of cameras according to the point cloud calibration matrix among the plurality of cameras to obtain a reconstructed three-dimensional scene, namely, under the condition that the cameras are fixed, registering and splicing the point clouds of the plurality of cameras by a machine vision method to obtain the reconstructed three-dimensional scene.
Drawings
Fig. 1 is a schematic flow chart of a multi-camera-based three-dimensional scene reconstruction method according to an embodiment of the present invention;
FIG. 2 is a schematic flow chart illustrating a process of generating a preliminary location relationship calibration matrix according to an embodiment of the present invention;
FIG. 3 is a schematic flow chart of point cloud registration and stitching according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a checkerboard according to embodiments of the present invention;
FIG. 5 is a schematic diagram of four checkerboards according to an embodiment of the present invention;
fig. 6 is a schematic diagram illustrating an effect of masking the searched checkerboard according to the embodiment of the present invention;
fig. 7 is a schematic diagram illustrating an effect of a region of interest of a second color image according to an embodiment of the present invention;
FIG. 8 is a schematic diagram illustrating the effect of a region of interest of a point cloud according to an embodiment of the present invention;
fig. 9 is a schematic diagram illustrating an effect of performing preliminary registration on point clouds according to an embodiment of the present invention;
FIG. 10 is a schematic diagram of a calculation for generating a preliminary location relationship calibration matrix according to an embodiment of the present invention;
fig. 11 is a perspective view of a viewing body according to an embodiment of the present invention;
FIG. 12 is a schematic view of a cutting plane of an XZ plane of a viewing body according to an embodiment of the present invention;
FIG. 13 is a schematic diagram of the location of a point cloud and a clipping plane according to an embodiment of the present invention;
FIG. 14 is a schematic point cloud of a camera according to an embodiment of the present invention;
fig. 15 is a schematic diagram of a point cloud after a camera according to an embodiment of the present invention removes an overlapping point cloud region;
FIG. 16 is a schematic diagram illustrating the effect of two cameras after point cloud stitching according to an embodiment of the present invention;
fig. 17 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Description of reference numerals:
1. an electronic device; 2. a processor; 3. a memory.
Detailed Description
In order to explain technical contents, achieved objects, and effects of the present invention in detail, the following description is made with reference to the accompanying drawings in combination with the embodiments.
Referring to fig. 1 to 16, a multi-camera based three-dimensional scene reconstruction method includes the steps of:
acquiring a preliminary position relation calibration matrix among a plurality of cameras;
acquiring color images and depth images which are respectively shot by the plurality of cameras for a scene to be reconstructed and provided with the plurality of checkerboards, and calibrating a matrix according to the color images and the depth images of the plurality of cameras and the preliminary position relation among the plurality of cameras to obtain a point cloud calibration matrix among the plurality of cameras;
and splicing the point clouds of the cameras according to the point cloud calibration matrix among the cameras to obtain the reconstructed three-dimensional scene.
From the above description, the beneficial effects of the present invention are: the method comprises the steps of obtaining a preliminary position relation calibration matrix among a plurality of cameras in advance, then placing a plurality of checkerboards in a three-dimensional scene to be reconstructed, obtaining a point cloud calibration matrix among the plurality of cameras according to color images and point clouds of the plurality of checkerboards in different cameras, and finally splicing the point clouds of the plurality of cameras according to the point cloud calibration matrix among the plurality of cameras to obtain a reconstructed three-dimensional scene.
Further, the acquiring a preliminary position relationship calibration matrix among a plurality of cameras specifically includes the following steps:
acquiring a first color image shot by each camera for one checkerboard to obtain corner point coordinates of the checkerboard in each first color image;
pairing each corner point coordinate of the first color images of different cameras according to an epipolar geometric constraint principle to obtain a corner point pairing relation among the cameras;
and obtaining a preliminary position relation calibration matrix among the cameras according to the corner pairing relation among the cameras.
From the above description, it can be known that the existing chessboard searching algorithm is adopted to identify the checkerboards, and the coordinates of the corner points of the identified checkerboards are paired according to the epipolar geometric constraint principle, so as to obtain the pairing relationship of different cameras to the same target object, so as to obtain the position relationship of the color images of different cameras, i.e. the preliminary position relationship calibration matrix.
Further, the step of obtaining color images and depth images respectively shot by the plurality of cameras for a scene to be reconstructed, in which the plurality of checkerboards are placed, and obtaining a point cloud calibration matrix among the plurality of cameras according to the color images and the depth images of the plurality of cameras and the preliminary position relationship calibration matrix among the plurality of cameras specifically includes the following steps:
acquiring second color images and depth images which are respectively shot by the cameras for a scene to be reconstructed and provided with a plurality of checkerboards;
finding each checkerboard in each second color image, and obtaining the center point coordinate of each checkerboard in each second color image to obtain the region of interest of each second color image of each camera;
obtaining an interested area of the point cloud of each camera according to a preset position relation matrix between the second color image and the depth image in each camera and the interested area of the second color image;
preliminarily registering the point clouds of all the cameras according to the preliminary position relation calibration matrixes among the cameras and the region of interest of the point cloud of each camera to obtain point cloud preliminary calibration matrixes of all the cameras;
and carrying out iterative operation on the point cloud preliminary calibration matrixes of all the cameras to obtain the point cloud final calibration matrixes of all the cameras.
As can be seen from the above description, for the cameras capable of acquiring the color image and the depth image, the position relationship matrix of the color image and the depth image is preset, after the region of interest of the color image is obtained through a plurality of checkerboards, the region of interest of the point cloud of each camera can be obtained according to the position relationship matrix of the color image and the depth image, and the preliminary registration of the point cloud can be realized by calibrating the matrix according to the preliminary position relationship, and then the precise registration is performed through iterative operation, so that a more accurate final calibration matrix of the point cloud is obtained, and a foundation is laid for the reconstruction accuracy of a subsequent three-dimensional scene.
Further, the finding out each checkerboard in each second color image and obtaining the center point coordinate of each checkerboard in each second color image to obtain the region of interest of the second color image of each camera specifically includes the following steps:
searching each checkerboard in the second color image, and obtaining the corner point coordinates of the searched checkerboard to obtain the center point coordinates of the searched checkerboard;
masking the searched checkerboards, then continuously searching other checkerboards in the second color image and obtaining the center point coordinates of other checkerboards until obtaining the center point coordinates of each checkerboard in each second color image;
and carrying out polar angle sequencing on the center point coordinates of all the chessboard in each second color image, and then sequentially connecting the center point coordinates of the chessboard according to the polar angle sequencing result to obtain a closed area, namely the region of interest of each second color image of each camera.
As can be seen from the above description, for the identification of multiple checkerboards, the camera performs mask masking after identifying the checkerboard and obtaining the coordinates of the central point, so that the camera does not identify the checkerboard any more, but starts to identify other checkerboards, thereby completing the identification and positioning of the multiple checkerboards to obtain the region of interest of the color image of the camera.
Further, the number of the cameras is two, and the iterative operation of the point cloud preliminary calibration matrixes of all the cameras to obtain the point cloud final calibration matrixes of all the cameras comprises the following steps:
respectively setting the point cloud A set and the point cloud B set of the two cameras as a set P and a set Q, and obtaining a matching relation M of the set P and the set Q as follows:
M=match(P,Q)={(p,q):p∈P,q∈Q}
defining the error between the set P and the set Q as:
error(P,Q)=∑(p,q∈M)w(p,q)d(p,q)
wherein w (P, Q) represents the weight of the corresponding point clouds in the set P and the set Q, and d (P, Q) represents the distance between the corresponding point clouds in the set P and the set Q;
performing iterative operation on the point cloud preliminary calibration matrix through an iterative calculation formula to obtain a point cloud final calibration matrix of all the cameras, wherein the iterative calculation formula is as follows:
the i and i +1 represent the number of iterations, theFor iterative integration of the transformations, the TinitAnd preliminarily calibrating a matrix for the point cloud.
As can be seen from the above description, for the calibration matrix, one of the point clouds of the two cameras is converted into the coordinate system of the other point cloud, and then the defined error value is continuously reduced through iterative operation until the optimal final calibration matrix of the point cloud is obtained, i.e., the precise registration of the point clouds of the two cameras is completed.
Further, the step of splicing the point clouds of the plurality of cameras according to the point cloud calibration matrix among the plurality of cameras to obtain the reconstructed three-dimensional scene specifically includes the following steps:
calculating an overlapped point cloud area among the cameras according to a point cloud calibration matrix among the cameras, and removing the overlapped point cloud area of one camera;
and splicing the point clouds of all the cameras after the overlapped point cloud areas are removed according to the point cloud calibration matrix among the cameras to obtain a reconstructed three-dimensional scene.
As can be seen from the above description, because the point cloud data of the plurality of cameras have the overlapping region, when the point cloud is spliced, the overlapping region is removed first, and then the two point clouds are spliced, so that the calculation amount is reduced, the splicing speed is increased, and the reconstruction of the three-dimensional scene is realized better and faster.
Further, the calculating an overlapping point cloud area between the plurality of cameras according to the point cloud calibration matrix between the plurality of cameras and removing the overlapping point cloud area of one of the cameras specifically includes the following steps:
the method comprises the steps that the visual field of a camera is equal to a view cone, and an object in the visual field of the camera is cut by using a near plane and a far plane to form a viewing frustum which comprises a left surface, a right surface, a top surface, a bottom surface, a near plane and a far plane;
the camera is positioned at the origin of a world coordinate system, the sight line direction of the camera is a positive direction along the Z axis, the view matrix of the camera is set as a unit matrix, and the vertex is expressed as: v ═ 1 (x y z w ═ 1)TThe projection matrix M of 4 x 4 is represented as: m ═ Mij);
The vertex obtained by rotating the projection matrix M is v '═ 1 (x' y 'z' w)TNamely:
the · is a dot product, the rowi=(mi1 mi2 mi3 mi4) The ith row of data representing the projection matrix M; if vertex v' is located within the frustum, the following first inequality should be satisfied:
-w'<x'<w'
-w^'<y^'<w^'
-w'<z'<w'
obtaining an inequality corresponding to each interception plane according to the first inequality of the interception body and the position relation of the six interception planes in the interception body, so as to calculate a plane equation coefficient of each interception plane, and obtain a plane equation expression of each interception plane;
according to a plane equation expression of each interception plane corresponding to each camera, the position relation between the point cloud of the camera and each interception plane of other cameras is judged one by one, and the point cloud on the inner side of all the interception planes in any one camera is used as an overlapped point cloud area of the cameras;
removing the overlapping point cloud area of any one of the cameras that includes the overlapping point cloud area.
From the above description, the calculation of the plane equation expression is performed on the six interception planes of the frustum, and then the position relationship between the point clouds of other cameras and the six interception planes of the camera is judged, so as to judge whether the point clouds of other cameras overlap with the frustum of the camera, and if so, the point clouds are considered to be overlapped, so that the overlapped point cloud area is quickly and accurately identified.
Further, the plurality of chequers is four chequers.
From the foregoing description, it will be appreciated that a preferred embodiment is provided.
Another embodiment of the invention provides a computer-readable storage medium having stored thereon a computer program which, when being executed by a processor, implements the multi-camera based three-dimensional scene reconstruction method described above.
Referring to fig. 17, another embodiment of the present invention provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the multi-camera based three-dimensional scene reconstruction method when executing the computer program.
With regard to the specific implementation procedures and corresponding effects of the image scaling methods included in the computer programs in the above two embodiments, reference may be made to the related descriptions in the multi-camera based three-dimensional scene reconstruction method in the foregoing embodiments.
Therefore, the following embodiments are provided with a plurality of cameras in a scene to perform three-dimensional reconstruction on the scene, for example, when an AR (Augmented Reality) system needs to perform three-dimensional scene reconstruction on a real scene, the following embodiments are specifically described as follows:
referring to fig. 1 to fig. 16, a first embodiment of the present invention is:
a multi-camera based three-dimensional scene reconstruction method, as shown in fig. 1, includes the steps of:
s1, acquiring a preliminary position relation calibration matrix among the plurality of cameras;
in this embodiment, as shown in fig. 2, step S1 specifically includes the following steps:
s11, acquiring a first color image shot by each camera for one checkerboard, and acquiring corner point coordinates of the checkerboard in each first color image;
in the present embodiment, as shown in fig. 4, a plurality of first color images including checkerboard may be collected at the same time in different poses, for example, in the present embodiment, the number of the first color images is about 100. And then finding the coordinates of the corner points of the checkerboard by adopting the conventional chessboard searching algorithm.
S12, pairing each corner point coordinate of the first color images of different cameras according to an epipolar geometric constraint principle to obtain a corner point pairing relation among the cameras;
in this embodiment, as shown in FIG. 10, where I1And I2Is the image plane of the color image on two cameras, wherein the centers of the cameras are O1And O2. Wherein P is1For a corner point on the checkerboard at I1Up to corresponding projection point, in the same way as P2For the same corner points on the checkerboard at I2And (4) the corresponding projection point. Point O1、O2And the plane defined by the three points P is a polar plane. O is1O2Connecting line and image plane I1、I2Respectively at the intersection points of e1And e2. Wherein e is1And e2Referred to as pole, O1O2Referred to as the baseline. Polar plane and two image planes I1And I2Cross line l between1And l2The corner points are polar lines, so that whether the corner points respectively identified by the cameras are the same corner point can be judged according to the intersection of the same corner points on the checkerboard at the point P, and the corner point pairing relation among the cameras is obtained.
And S13, obtaining a preliminary position relation calibration matrix among the plurality of cameras according to the corner point pairing relation among the plurality of cameras.
In this embodiment, two cameras are selected for image acquisition of a scene at different viewing angles, and in other equivalent embodiments, three, four, and other multiple cameras can be selected for image acquisition.
S2, acquiring color images and depth images which are respectively shot by a plurality of cameras for a scene to be reconstructed and placed with a plurality of checkerboards, and calibrating a matrix according to the color images and the depth images of the plurality of cameras and the preliminary position relation among the plurality of cameras to obtain a point cloud calibration matrix among the plurality of cameras;
and S3, splicing the point clouds of the multiple cameras according to the point cloud calibration matrix among the multiple cameras to obtain the reconstructed three-dimensional scene.
Referring to fig. 1 to fig. 16, a second embodiment of the present invention is:
based on the foregoing first embodiment, as shown in fig. 3, the three-dimensional scene reconstruction method based on multiple cameras, in step S2, specifically includes the following steps:
s21, acquiring second color images and depth images which are respectively shot by a plurality of cameras for a scene to be reconstructed and provided with a plurality of checkerboards;
in the present embodiment, as shown in fig. 5, four checkerboards are placed in the scene to be reconstructed. Thus, two cameras need to capture a color image and a depth image comprising four checkerboards, wherein the depth image corresponds to the point cloud. In other equivalent embodiments, two, three or five, etc. more checkerboards may be selected.
S22, finding each checkerboard in each second color image, and obtaining the center point coordinates of each checkerboard in each second color image to obtain the region of interest of the second color image of each camera;
wherein, step S22 specifically includes the following steps:
s221, searching the checkerboard in each second color image, and obtaining the corner point coordinates of the searched checkerboard to obtain the center point coordinates of the searched checkerboard;
in this embodiment, the checkerboards are searched according to the existing checkerboard searching algorithm, and first, a checkerboard is identified, for example, the checkerboard at the lower right corner of fig. 5 in this embodiment, so as to obtain the coordinates of the center point of the checkerboard.
S222, masking the searched checkerboards, then continuously searching other checkerboards in the second color image and obtaining the center point coordinates of other checkerboards until the center point coordinates of each checkerboard in each second color image are obtained;
the mask is a binary image composed of 0 and 1, and as shown in fig. 6, the checkerboard in the lower right corner is masked, that is, the checkerboard in the lower right corner is masked by replacing the pixel value 0. Thus, the coordinates of the center points of the four chessboard patterns can be obtained.
S223, polar angle sequencing is carried out on the center point coordinates of all the chessboard in each second color image, and then the center point coordinates of the chessboard are sequentially connected according to the polar angle sequencing result to obtain a closed area, namely the region of interest of the second color image of each camera.
The polar angle sorting is that a fixed point is taken in a plane and called a pole point, a ray is led to be called a polar axis, a length unit and the positive direction of an angle are selected, and then points on the given plane are arranged in a preset direction according to the selected fixed point. Corresponding to this embodiment, as shown in fig. 7, the coordinates of the center points of the four chessboard patterns are sequentially connected to obtain a rectangle, which is the region of interest.
S23, obtaining an interested area of the point cloud of each camera according to a preset position relation matrix between the second color image and the depth image in each camera and the interested area of the second color image;
the RGBD camera capable of acquiring the color image and the depth image stores a preset position relationship matrix between the color image and the depth image in advance, so as to obtain an area of interest of the point cloud of each camera, which is shown in fig. 8.
S24, preliminarily registering the point clouds of all the cameras according to the preliminary position relation calibration matrixes among the cameras and the interesting regions of the point clouds of each camera to obtain point cloud preliminary calibration matrixes of all the cameras;
the effect obtained after the point clouds of the two cameras are preliminarily registered is shown in fig. 9.
And S25, carrying out iterative operation on the point cloud preliminary calibration matrixes of all the cameras to obtain the final point cloud calibration matrixes of all the cameras.
In this embodiment, the iterative operation uses an iterative closest point algorithm, whose mathematical form represents:
wherein,Ap represents the point cloud a and P represents the point cloud a,Bq represents the point cloud B, and the fine registration is to convert the point cloud A to the coordinate system of the point cloud B so that error (T: (T))AP),BQ) is minimal. Thus, step S25 specifically includes the following steps:
respectively setting the point cloud A set and the point cloud B set of the two cameras as a set P and a set Q, and obtaining a matching relation M of the set P and the set Q as follows:
M=match(P,Q)={(p,q):p∈P,q∈Q}
the error between set P and set Q is defined as:
error(P,Q)=∑(p,q∈M)w(p,q)d(p,q)
w (P, Q) represents the weight of the corresponding point clouds in the set P and the set Q, and d (P, Q) represents the distance between the corresponding point clouds in the set P and the set Q;
performing iterative operation on the point cloud preliminary calibration matrix through an iterative calculation formula to obtain a point cloud final calibration matrix of all cameras, wherein the iterative calculation formula is as follows:
i and i +1 denote the number of iterations,for iterative integration of transformations, TinitAnd preliminarily calibrating a matrix for the point cloud.
Thereby, the fine registration of the point clouds of the two cameras is completed.
Referring to fig. 1 to 16, a third embodiment of the present invention is:
based on the foregoing first embodiment, the step S3 specifically includes the following steps:
s31, calculating an overlapped point cloud area among the cameras according to the point cloud calibration matrix among the cameras, and removing the overlapped point cloud area of one of the cameras;
in this embodiment, step S31 specifically includes the following steps:
s311, the visual field of the camera is equal to a view cone, and an object in the visual field of the camera is cut by using a near plane and a far plane to form a viewing frustum which comprises a left plane, a right plane, a top plane, a bottom plane, a near plane and a far plane;
the view frustum including the above six truncated planes is shown in fig. 11, and the view frustum XZ is a screenshot in fig. 12.
S312, locating the camera at the origin of the world coordinate system, setting the viewing direction of the camera as the positive direction along the Z-axis, setting the view matrix of the camera as a unit matrix, and expressing the vertices as: v ═ 1 (x y z w ═ 1)TThe projection matrix M of 4 x 4 is represented as: m ═ Mij);
S313, rotating the projection matrix M to obtain a vertexFor v '═ 1 (x' y 'z' w)TNamely:
where, is dot product, rowi=(mi1 mi2 mi3 mi4) The ith line of data representing the projection matrix M is converted into homogeneous coordinates from the top point of a world coordinate system after the transformation;
s314, if the vertex v' is located in the frustum, the following first inequality should be satisfied:
-w'<x'<w'
-w^'<y^'<w^'
-w'<z'<w'
s315, obtaining an inequality corresponding to each interception plane according to the first inequality of the interception body and the position relation of the six interception planes in the interception body, and calculating a plane equation coefficient of each interception plane to obtain a plane equation expression of each interception plane;
wherein, it is assumed that x' satisfies the following inequality:
-w'<x'
from the above rotational equation:
-(v·row4)<(v·row1)
that is:
0<(v·row4)+(v·row1)
finally, the following can be obtained:
0<v·(row4+row1)
the plane equation for the left section plane can be expressed as:
x(m41+m11)+y(m42+m12)+z(m43+m13)+w(m44+m14)=0
because w is 1, then
x(m41+m11)+y(m42+m12)+z(m43+m13)+m44+m14=0
Wherein, the standard plane equation is:
ax+by+cz+d=0
the coefficients of the plane equation for the left section plane are:
a=m41+m11,b=m42+m12,c=m43+m13,d=m44+m14
repeating the above steps, and obtaining a plane equation coefficient of each interception plane according to inequalities corresponding to different interception planes, as shown in table 1:
TABLE 1 planar equation results of the intercept planes
Therefore, the plane equation expression of six cut planes can be obtained.
S316, according to the plane equation expression of each interception plane corresponding to each camera, the position relation between the point cloud of the camera and each interception plane of other cameras is judged one by one, and the point clouds on the inner sides of all the interception planes in any one camera are used as the overlapped point cloud area of the cameras;
wherein, viewing the frustum from the cross section, the relationship between the point and the truncation plane can be obtained as shown in fig. 13. I.e. one plane may be measured from P and normal on the planeAnd determining that any point P' on the plane satisfies the following conditions:
then, given a point Q arbitrarily, there is:
thereby, an overlapping point cloud area of the two cameras is obtained.
And S317, removing the overlapped point cloud area of any one of the cameras comprising the overlapped point cloud area.
That is, fig. 14 is the point cloud of one of the cameras, and fig. 15 is obtained after the overlapping point cloud area is removed.
And S32, splicing the point clouds of all the cameras after the overlapped point cloud areas are removed according to the point cloud calibration matrix among the cameras to obtain the reconstructed three-dimensional scene.
The fourth embodiment of the invention is as follows:
another embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the multi-camera based three-dimensional scene reconstruction method according to any one of the first to third embodiments.
Referring to fig. 17, a fifth embodiment of the present invention is:
an electronic device 1 comprises a memory 3, a processor 2 and a computer program stored on the memory 3 and operable on the processor 2, wherein the processor 2 implements the multi-camera based three-dimensional scene reconstruction method according to any one of the first to third embodiments when executing the computer program.
In summary, the multi-camera-based three-dimensional scene reconstruction method, the storage medium and the electronic device provided by the invention adopt the existing chessboard searching algorithm to identify the checkerboards, and pair the corner point coordinates of the identified checkerboards according to the epipolar geometric constraint principle to obtain the preliminary position relationship calibration matrix among the multiple cameras; after the interesting region of the color image is obtained through the plurality of checkerboards, the interesting region of the point cloud of each camera can be obtained according to the position relation matrix of the color image and the depth image, the initial registration of the point cloud can be realized according to the initial position relation calibration matrix, and then the accurate registration is carried out through the iterative closest point algorithm, so that a more accurate final point cloud calibration matrix is obtained; and finally, according to a point cloud calibration matrix among the cameras, firstly removing the overlapped point cloud area, and then performing point cloud splicing on the cameras to obtain a reconstructed three-dimensional scene, namely, under the condition that the cameras are fixed, performing registration splicing on the point clouds of the cameras by a machine vision method to obtain the reconstructed three-dimensional scene, so that the reconstruction of the three-dimensional scene can be better and faster realized, and the method is time-saving, labor-saving and universal.
The above description is only an embodiment of the present invention, and not intended to limit the scope of the present invention, and all equivalent changes made by using the contents of the present specification and the drawings, or applied directly or indirectly to the related technical fields, are included in the scope of the present invention.
Claims (10)
1. The three-dimensional scene reconstruction method based on multiple cameras is characterized by comprising the following steps:
acquiring a preliminary position relation calibration matrix among a plurality of cameras;
acquiring color images and depth images which are respectively shot by the plurality of cameras for a scene to be reconstructed and provided with the plurality of checkerboards, and calibrating a matrix according to the color images and the depth images of the plurality of cameras and the preliminary position relation among the plurality of cameras to obtain a point cloud calibration matrix among the plurality of cameras;
and splicing the point clouds of the cameras according to the point cloud calibration matrix among the cameras to obtain the reconstructed three-dimensional scene.
2. The multi-camera based three-dimensional scene reconstruction method of claim 1, wherein the obtaining of the preliminary positional relationship calibration matrix between the plurality of cameras specifically comprises the steps of:
acquiring a first color image shot by each camera for one checkerboard to obtain corner point coordinates of the checkerboard in each first color image;
pairing each corner point coordinate of the first color images of different cameras according to an epipolar geometric constraint principle to obtain a corner point pairing relation among the cameras;
and obtaining a preliminary position relation calibration matrix among the cameras according to the corner pairing relation among the cameras.
3. The multi-camera based three-dimensional scene reconstruction method according to claim 1, wherein the obtaining of the color images and the depth images of the plurality of cameras respectively taken by the scene to be reconstructed with the checkerboards placed therein and the obtaining of the point cloud calibration matrix between the plurality of cameras according to the color images and the depth images of the plurality of cameras and the preliminary position relationship calibration matrix between the plurality of cameras specifically comprises the following steps:
acquiring second color images and depth images which are respectively shot by the cameras for a scene to be reconstructed and provided with a plurality of checkerboards;
finding each checkerboard in each second color image, and obtaining the center point coordinate of each checkerboard in each second color image to obtain the region of interest of each second color image of each camera;
obtaining an interested area of the point cloud of each camera according to a preset position relation matrix between the second color image and the depth image in each camera and the interested area of the second color image;
preliminarily registering the point clouds of all the cameras according to the preliminary position relation calibration matrixes among the cameras and the region of interest of the point cloud of each camera to obtain point cloud preliminary calibration matrixes of all the cameras;
and carrying out iterative operation on the point cloud preliminary calibration matrixes of all the cameras to obtain the point cloud final calibration matrixes of all the cameras.
4. The multi-camera based three-dimensional scene reconstruction method according to claim 3, wherein the finding of each checkerboard in each second color image and obtaining the center point coordinates of each checkerboard in each second color image to obtain the region of interest of each second color image of each camera specifically comprises the following steps:
searching each checkerboard in the second color image, and obtaining the corner point coordinates of the searched checkerboard to obtain the center point coordinates of the searched checkerboard;
masking the searched checkerboards, then continuously searching other checkerboards in the second color image and obtaining the center point coordinates of other checkerboards until obtaining the center point coordinates of each checkerboard in each second color image;
and carrying out polar angle sequencing on the center point coordinates of all the chessboard in each second color image, and then sequentially connecting the center point coordinates of the chessboard according to the polar angle sequencing result to obtain a closed area, namely the region of interest of each second color image of each camera.
5. The multi-camera based three-dimensional scene reconstruction method according to claim 3, wherein there are two cameras, and the step of performing iterative operation on the point cloud preliminary calibration matrices of all the cameras to obtain the point cloud final calibration matrices of all the cameras comprises the following steps:
respectively setting the point cloud A set and the point cloud B set of the two cameras as a set P and a set Q, and obtaining a matching relation M of the set P and the set Q as follows:
M=match(P,Q)={(p,q):p∈P,q∈Q}
defining the error between the set P and the set Q as:
error(P,Q)=∑(p,q∈M)w(p,q)d(p,q)
wherein w (P, Q) represents the weight of the corresponding point clouds in the set P and the set Q, and d (P, Q) represents the distance between the corresponding point clouds in the set P and the set Q;
performing iterative operation on the point cloud preliminary calibration matrix through an iterative calculation formula to obtain a point cloud final calibration matrix of all the cameras, wherein the iterative calculation formula is as follows:
6. The multi-camera based three-dimensional scene reconstruction method according to claim 1, wherein the step of splicing the point clouds of the plurality of cameras according to the point cloud calibration matrix among the plurality of cameras to obtain the reconstructed three-dimensional scene specifically comprises the following steps:
calculating an overlapped point cloud area among the cameras according to a point cloud calibration matrix among the cameras, and removing the overlapped point cloud area of one camera;
and splicing the point clouds of all the cameras after the overlapped point cloud areas are removed according to the point cloud calibration matrix among the cameras to obtain a reconstructed three-dimensional scene.
7. The multi-camera based three-dimensional scene reconstruction method of claim 6, wherein the calculating the overlapped point cloud regions between the plurality of cameras according to the point cloud calibration matrix between the plurality of cameras and removing the overlapped point cloud region of one of the cameras comprises the following steps:
the method comprises the steps that the visual field of a camera is equal to a view cone, and an object in the visual field of the camera is cut by using a near plane and a far plane to form a viewing frustum which comprises a left surface, a right surface, a top surface, a bottom surface, a near plane and a far plane;
the camera is positioned at the origin of a world coordinate system, the sight line direction of the camera is a positive direction along the Z axis, the view matrix of the camera is set as a unit matrix, and the vertex is expressed as: v ═ 1 (x y z w ═ 1)TThe projection matrix M of 4 x 4 is represented as: m ═ Mij);
The vertex obtained by rotating the projection matrix M is v '═ 1 (x' y 'z' w)TNamely:
the · is a dot product, the rowi=(mi1 mi2 mi3 mi4) The ith row of data representing the projection matrix M;
if vertex v' is located within the frustum, the following first inequality should be satisfied:
-w'<x'<w'
-w^'<y^'<w^'
-w'<z'<w'
obtaining an inequality corresponding to each interception plane according to the first inequality of the interception body and the position relation of the six interception planes in the interception body, so as to calculate a plane equation coefficient of each interception plane, and obtain a plane equation expression of each interception plane;
according to a plane equation expression of each interception plane corresponding to each camera, the position relation between the point cloud of the camera and each interception plane of other cameras is judged one by one, and the point cloud on the inner side of all the interception planes in any one camera is used as an overlapped point cloud area of the cameras;
removing the overlapping point cloud area of any one of the cameras that includes the overlapping point cloud area.
8. The multi-camera based three-dimensional scene reconstruction method of one of claims 1 to 7, wherein the plurality of chequers is four chequers.
9. A computer-readable storage medium having stored thereon a computer program, characterized in that: the computer program, when being executed by a processor, implements a method of reconstructing a three-dimensional scene as claimed in any one of claims 1 to 8.
10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the three-dimensional scene reconstruction method according to any one of claims 1 to 8 when executing the computer program.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202011281518.9A CN112381886B (en) | 2020-11-16 | 2020-11-16 | Three-dimensional scene reconstruction method based on multiple cameras, storage medium and electronic equipment |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202011281518.9A CN112381886B (en) | 2020-11-16 | 2020-11-16 | Three-dimensional scene reconstruction method based on multiple cameras, storage medium and electronic equipment |
Publications (2)
Publication Number | Publication Date |
---|---|
CN112381886A true CN112381886A (en) | 2021-02-19 |
CN112381886B CN112381886B (en) | 2024-04-09 |
Family
ID=74584818
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202011281518.9A Active CN112381886B (en) | 2020-11-16 | 2020-11-16 | Three-dimensional scene reconstruction method based on multiple cameras, storage medium and electronic equipment |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN112381886B (en) |
Cited By (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112927353A (en) * | 2021-02-25 | 2021-06-08 | 电子科技大学 | Three-dimensional scene reconstruction method based on two-dimensional target detection and model alignment, storage medium and terminal |
CN113012238A (en) * | 2021-04-09 | 2021-06-22 | 南京星顿医疗科技有限公司 | Method for rapid calibration and data fusion of multi-depth camera |
CN113052917A (en) * | 2021-03-29 | 2021-06-29 | 黑芝麻智能科技(上海)有限公司 | Method for acquiring image coordinates of invisible position of camera, calibration method and system |
CN113362445A (en) * | 2021-05-25 | 2021-09-07 | 上海奥视达智能科技有限公司 | Method and device for reconstructing object based on point cloud data |
CN113504385A (en) * | 2021-06-30 | 2021-10-15 | 上海爱观视觉科技有限公司 | Method and device for measuring speed of a plurality of cameras |
CN113516694A (en) * | 2021-04-13 | 2021-10-19 | 同济大学 | Multi-Kinect-based dynamic visual control method for remote medical scene |
CN114554176A (en) * | 2022-01-24 | 2022-05-27 | 北京有竹居网络技术有限公司 | Depth camera |
CN117830439A (en) * | 2024-03-05 | 2024-04-05 | 南昌虚拟现实研究院股份有限公司 | Multi-camera system pose calibration method and device |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN109064404A (en) * | 2018-08-10 | 2018-12-21 | 西安电子科技大学 | It is a kind of based on polyphaser calibration panorama mosaic method, panoramic mosaic system |
CN109242913A (en) * | 2018-09-07 | 2019-01-18 | 百度在线网络技术(北京)有限公司 | Scaling method, device, equipment and the medium of collector relative parameter |
CN110223377A (en) * | 2019-05-28 | 2019-09-10 | 上海工程技术大学 | One kind being based on stereo visual system high accuracy three-dimensional method for reconstructing |
CN111127625A (en) * | 2019-10-08 | 2020-05-08 | 新拓三维技术(深圳)有限公司 | Foot scanning method, system and device |
CN111150175A (en) * | 2019-12-05 | 2020-05-15 | 新拓三维技术(深圳)有限公司 | Method, device and system for three-dimensional scanning of feet |
-
2020
- 2020-11-16 CN CN202011281518.9A patent/CN112381886B/en active Active
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN109064404A (en) * | 2018-08-10 | 2018-12-21 | 西安电子科技大学 | It is a kind of based on polyphaser calibration panorama mosaic method, panoramic mosaic system |
CN109242913A (en) * | 2018-09-07 | 2019-01-18 | 百度在线网络技术(北京)有限公司 | Scaling method, device, equipment and the medium of collector relative parameter |
CN110223377A (en) * | 2019-05-28 | 2019-09-10 | 上海工程技术大学 | One kind being based on stereo visual system high accuracy three-dimensional method for reconstructing |
CN111127625A (en) * | 2019-10-08 | 2020-05-08 | 新拓三维技术(深圳)有限公司 | Foot scanning method, system and device |
CN111150175A (en) * | 2019-12-05 | 2020-05-15 | 新拓三维技术(深圳)有限公司 | Method, device and system for three-dimensional scanning of feet |
Cited By (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112927353A (en) * | 2021-02-25 | 2021-06-08 | 电子科技大学 | Three-dimensional scene reconstruction method based on two-dimensional target detection and model alignment, storage medium and terminal |
CN112927353B (en) * | 2021-02-25 | 2023-05-19 | 电子科技大学 | Three-dimensional scene reconstruction method, storage medium and terminal based on two-dimensional target detection and model alignment |
CN113052917A (en) * | 2021-03-29 | 2021-06-29 | 黑芝麻智能科技(上海)有限公司 | Method for acquiring image coordinates of invisible position of camera, calibration method and system |
CN113052917B (en) * | 2021-03-29 | 2024-05-07 | 黑芝麻智能科技(上海)有限公司 | Method for acquiring image coordinates of invisible position of camera, calibration method and system |
CN113012238A (en) * | 2021-04-09 | 2021-06-22 | 南京星顿医疗科技有限公司 | Method for rapid calibration and data fusion of multi-depth camera |
CN113012238B (en) * | 2021-04-09 | 2024-04-16 | 南京星顿医疗科技有限公司 | Method for quick calibration and data fusion of multi-depth camera |
CN113516694A (en) * | 2021-04-13 | 2021-10-19 | 同济大学 | Multi-Kinect-based dynamic visual control method for remote medical scene |
CN113362445A (en) * | 2021-05-25 | 2021-09-07 | 上海奥视达智能科技有限公司 | Method and device for reconstructing object based on point cloud data |
CN113362445B (en) * | 2021-05-25 | 2023-05-05 | 上海奥视达智能科技有限公司 | Method and device for reconstructing object based on point cloud data |
CN113504385B (en) * | 2021-06-30 | 2023-07-14 | 安徽爱观视觉科技有限公司 | Speed measuring method and device for plural cameras |
CN113504385A (en) * | 2021-06-30 | 2021-10-15 | 上海爱观视觉科技有限公司 | Method and device for measuring speed of a plurality of cameras |
CN114554176A (en) * | 2022-01-24 | 2022-05-27 | 北京有竹居网络技术有限公司 | Depth camera |
CN117830439A (en) * | 2024-03-05 | 2024-04-05 | 南昌虚拟现实研究院股份有限公司 | Multi-camera system pose calibration method and device |
Also Published As
Publication number | Publication date |
---|---|
CN112381886B (en) | 2024-04-09 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN112381886A (en) | Multi-camera-based three-dimensional scene reconstruction method, storage medium and electronic device | |
CN111783820B (en) | Image labeling method and device | |
CN106780619B (en) | Human body size measuring method based on Kinect depth camera | |
CN106553195B (en) | Object 6DOF localization method and system during industrial robot crawl | |
CN104933718B (en) | A kind of physical coordinates localization method based on binocular vision | |
EP2370954B1 (en) | Method for recognizing objects in a set of images recorded by one or more cameras | |
CN103971378A (en) | Three-dimensional reconstruction method of panoramic image in mixed vision system | |
CN110827392B (en) | Monocular image three-dimensional reconstruction method, system and device | |
CN114119864A (en) | Positioning method and device based on three-dimensional reconstruction and point cloud matching | |
CN110310331A (en) | A kind of position and orientation estimation method based on linear feature in conjunction with point cloud feature | |
CN107492107B (en) | Object identification and reconstruction method based on plane and space information fusion | |
CN105976431B (en) | A kind of three-dimensional surface reconstructing method based on rotation light field | |
CN109613974B (en) | AR home experience method in large scene | |
Li et al. | Spherical stereo for the construction of immersive VR environment | |
CN112348890A (en) | Space positioning method and device and computer readable storage medium | |
CN109314774A (en) | System and method for three-dimensional imaging | |
CN113793392A (en) | Camera parameter calibration method and device | |
CN117372647A (en) | Rapid construction method and system of three-dimensional model for building | |
Siddique et al. | 3d object localization using 2d estimates for computer vision applications | |
Deepu et al. | 3D Reconstruction from Single 2D Image | |
KR20210141922A (en) | How to 3D Reconstruct an Object | |
CN115953471A (en) | Indoor scene multi-scale vector image retrieval and positioning method, system and medium | |
KR20160049639A (en) | Stereoscopic image registration method based on a partial linear method | |
CN109872371A (en) | A kind of monocular vision three-dimensional rebuilding method based on improvement Sift algorithm | |
Doria et al. | Determination of camera position in soccer games |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |