CN112381886B - Three-dimensional scene reconstruction method based on multiple cameras, storage medium and electronic equipment - Google Patents
Three-dimensional scene reconstruction method based on multiple cameras, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN112381886B CN112381886B CN202011281518.9A CN202011281518A CN112381886B CN 112381886 B CN112381886 B CN 112381886B CN 202011281518 A CN202011281518 A CN 202011281518A CN 112381886 B CN112381886 B CN 112381886B
- Authority
- CN
- China
- Prior art keywords
- cameras
- point cloud
- camera
- point
- dimensional scene
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 44
- 239000011159 matrix material Substances 0.000 claims abstract description 85
- 238000004590 computer program Methods 0.000 claims description 14
- 230000000007 visual effect Effects 0.000 claims description 12
- 238000004364 calculation method Methods 0.000 claims description 8
- 230000014509 gene expression Effects 0.000 claims description 8
- 230000000873 masking effect Effects 0.000 claims description 5
- 238000012163 sequencing technique Methods 0.000 claims description 4
- 230000010354 integration Effects 0.000 claims description 3
- 230000000875 corresponding effect Effects 0.000 description 13
- 238000004422 calculation algorithm Methods 0.000 description 10
- 238000010586 diagram Methods 0.000 description 7
- 230000000694 effects Effects 0.000 description 7
- 230000003190 augmentative effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 235000005121 Sorbus torminalis Nutrition 0.000 description 1
- 244000152100 Sorbus torminalis Species 0.000 description 1
- 230000001133 acceleration Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 238000000844 transformation Methods 0.000 description 1
- 239000013598 vector Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06T7/85—Stereo camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
- G06T7/344—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods involving models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
- G06T7/75—Determining position or orientation of objects or cameras using feature-based methods involving models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30204—Marker
- G06T2207/30208—Marker matrix
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30244—Camera pose
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computer Graphics (AREA)
- Geometry (AREA)
- Software Systems (AREA)
- Length Measuring Devices By Optical Means (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a three-dimensional scene reconstruction method based on multiple cameras, a storage medium and electronic equipment, which comprise the following steps: acquiring a primary position relation calibration matrix among a plurality of cameras; acquiring color images and depth images which are respectively shot by a plurality of cameras on a scene to be reconstructed, wherein the scene to be reconstructed is provided with a plurality of checkerboards, and acquiring a point cloud calibration matrix among the plurality of cameras according to the color images and the depth images of the plurality of cameras and a preliminary position relationship calibration matrix among the plurality of cameras; splicing the point clouds of the cameras according to the point cloud calibration matrix among the cameras to obtain a reconstructed three-dimensional scene; according to the invention, under the condition that the cameras are fixed, the point clouds of the cameras are registered and spliced through the machine vision method to obtain the reconstructed three-dimensional scene, so that the reconstruction of the three-dimensional scene can be better and faster realized, and meanwhile, the method is more time-saving, labor-saving and more universal.
Description
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to a three-dimensional scene reconstruction method based on multiple cameras, a storage medium, and an electronic device.
Background
In an augmented reality application, in order for a human experimenter to obtain an immersive experience, a scene reconstruction of the real scene is required so that the human experimenter has an immersive feel. In the prior art, an indoor three-dimensional scene reconstruction method based on a double-layer registration method uses surf (Speeded Up Robust Features, acceleration robust feature) algorithm to extract and match feature points, maps the matching points to three-dimensional coordinates, and uses RANSAC (Random Sample Consensus, random sampling consensus algorithm) and ICP (Iterative Closest Point, iterative closest point algorithm) algorithm to perform registration.
Disclosure of Invention
The technical problems to be solved by the invention are as follows: the three-dimensional scene reconstruction method based on the multiple cameras, the storage medium and the electronic equipment can rapidly achieve three-dimensional scene reconstruction.
In order to solve the technical problems, the invention adopts the following technical scheme:
a three-dimensional scene reconstruction method based on multiple cameras comprises the following steps:
acquiring a primary position relation calibration matrix among a plurality of cameras;
acquiring color images and depth images which are respectively shot by a plurality of cameras on a scene to be reconstructed, wherein the scene to be reconstructed is provided with a plurality of checkerboards, and acquiring a point cloud calibration matrix among the plurality of cameras according to the color images and the depth images of the plurality of cameras and a preliminary position relationship calibration matrix among the plurality of cameras;
and splicing the point clouds of the cameras according to the point cloud calibration matrixes among the cameras to obtain a reconstructed three-dimensional scene.
In order to solve the technical problems, the invention adopts another technical scheme that:
a computer readable storage medium having stored thereon a computer program which when executed by a processor implements the multi-camera based three-dimensional scene reconstruction method described above.
In order to solve the technical problems, the invention adopts another technical scheme that:
an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the multi-camera based three-dimensional scene reconstruction method described above when executing the computer program.
The invention has the beneficial effects that: the method comprises the steps of firstly obtaining a preliminary position relation calibration matrix among a plurality of cameras, then placing a plurality of checkerboards in a three-dimensional scene to be reconstructed, obtaining a point cloud calibration matrix among the plurality of cameras according to color images and point clouds of the plurality of checkerboards in different cameras, finally splicing the point clouds of the plurality of cameras according to the point cloud calibration matrix among the plurality of cameras to obtain the reconstructed three-dimensional scene, namely, under the condition that the cameras are fixed, registering and splicing the point clouds of the plurality of cameras by a machine vision method to obtain the reconstructed three-dimensional scene, so that the reconstruction of the three-dimensional scene can be better and faster realized, and simultaneously, the method is time-saving, labor-saving and universal.
Drawings
Fig. 1 is a flow chart of a three-dimensional scene reconstruction method based on multiple cameras according to an embodiment of the invention;
FIG. 2 is a schematic flow chart of generating a calibration matrix of a preliminary positional relationship according to an embodiment of the present invention;
FIG. 3 is a schematic flow chart of point cloud registration and stitching according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a checkerboard according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of four checkerboards according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of the effect of masking the found checkerboard according to an embodiment of the present invention;
FIG. 7 is a schematic view illustrating the effect of a region of interest of a second color image according to an embodiment of the present invention;
FIG. 8 is a schematic view of the effect of a region of interest of a point cloud according to an embodiment of the present invention;
fig. 9 is a schematic diagram of an effect of performing preliminary registration on a point cloud according to an embodiment of the present invention;
FIG. 10 is a schematic diagram of calculation for generating a calibration matrix of a preliminary positional relationship according to an embodiment of the present invention;
FIG. 11 is a perspective view of a view cut-out according to an embodiment of the present invention;
FIG. 12 is a schematic view of a view of the XZ plane of a viewing volume according to an embodiment of the present invention;
FIG. 13 is a schematic diagram illustrating the positions of a point cloud and a plane of interception according to an embodiment of the present invention;
fig. 14 is a schematic view of a point cloud of a camera according to an embodiment of the present invention;
fig. 15 is a schematic view of a point cloud after a certain camera according to an embodiment of the present invention removes overlapping point cloud areas;
fig. 16 is a schematic view of an effect of two spliced camera point clouds according to an embodiment of the present invention;
fig. 17 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Description of the reference numerals:
1. an electronic device; 2. a processor; 3. a memory.
Detailed Description
In order to describe the technical contents, the achieved objects and effects of the present invention in detail, the following description will be made with reference to the embodiments in conjunction with the accompanying drawings.
Referring to fig. 1 to 16, the multi-camera based three-dimensional scene reconstruction method includes the steps of:
acquiring a primary position relation calibration matrix among a plurality of cameras;
acquiring color images and depth images which are respectively shot by a plurality of cameras on a scene to be reconstructed, wherein the scene to be reconstructed is provided with a plurality of checkerboards, and acquiring a point cloud calibration matrix among the plurality of cameras according to the color images and the depth images of the plurality of cameras and a preliminary position relationship calibration matrix among the plurality of cameras;
and splicing the point clouds of the cameras according to the point cloud calibration matrixes among the cameras to obtain a reconstructed three-dimensional scene.
From the above description, the beneficial effects of the invention are as follows: the method comprises the steps of obtaining a preliminary position relation calibration matrix among a plurality of cameras in advance, then placing a plurality of checkerboards in a three-dimensional scene to be reconstructed, obtaining a point cloud calibration matrix among the plurality of cameras according to color images and point clouds of the plurality of checkerboards in different cameras, and finally splicing the point clouds of the plurality of cameras according to the point cloud calibration matrix among the plurality of cameras to obtain a reconstructed three-dimensional scene, namely under the condition that the cameras are fixed, registering and splicing the point clouds of the plurality of cameras by a machine vision method to obtain the reconstructed three-dimensional scene.
Further, the obtaining the calibration matrix of the preliminary positional relationship among the plurality of cameras specifically includes the following steps:
acquiring a first color image shot by each camera on a checkerboard, and obtaining angular point coordinates of the checkerboard in each first color image;
pairing each angular point coordinate of the first color images of different cameras according to the epipolar geometry constraint principle to obtain angular point pairing relations among a plurality of cameras;
and obtaining a plurality of preliminary position relation calibration matrixes among the cameras according to the corner pairing relations among the cameras.
From the above description, the prior lookup chessboard algorithm is adopted to identify the chessboard, and the angular point coordinates of the identified chessboard are paired according to the epipolar geometry constraint principle, so that the pairing relation of different cameras relative to the same object is obtained, and the position relation of the color images of different cameras, namely the preliminary position relation calibration matrix, is obtained.
Further, the obtaining color images and depth images of the plurality of cameras respectively shot for the scene to be reconstructed, where the plurality of checkerboards are placed, and obtaining a point cloud calibration matrix between the plurality of cameras according to the color images and the depth images of the plurality of cameras and the preliminary position relationship calibration matrix between the plurality of cameras specifically includes the following steps:
acquiring second color images and depth images which are respectively shot by a plurality of cameras on a scene to be reconstructed, wherein the scene to be reconstructed is provided with a plurality of checkerboards;
searching each checkerboard in each second color image, and obtaining the center point coordinate of each checkerboard in each second color image to obtain the interested area of the second color image of each camera;
obtaining a region of interest of a point cloud of each camera according to a preset position relation matrix between the second color image and the depth image in each camera and the region of interest of the second color image;
performing preliminary registration on the point clouds of all cameras according to the preliminary position relation calibration matrix among the cameras and the interested area of the point clouds of each camera to obtain the point cloud preliminary calibration matrix of all the cameras;
and performing iterative operation on the point cloud preliminary calibration matrixes of all cameras to obtain the point cloud final calibration matrixes of all cameras.
From the above description, for the camera capable of acquiring the color image and the depth image, the position relation matrix of the color image and the depth image is preset, after the region of interest of the color image is obtained through a plurality of checkerboards, the region of interest of the point cloud of each camera can be obtained according to the position relation matrix of the color image and the depth image, the preliminary registration of the point cloud can be realized according to the preliminary position relation calibration matrix, and then the fine registration is performed through iterative operation, so that a more accurate final calibration matrix of the point cloud is obtained, and a foundation is laid for the reconstruction precision of the subsequent three-dimensional scene.
Further, the searching each checkerboard in each second color image, and obtaining the center point coordinates of each checkerboard in each second color image, and obtaining the interested area of the second color image of each camera specifically includes the following steps:
searching the checkerboard in each second color image, and obtaining the corner coordinates of the searched checkerboard to obtain the center point coordinates of the searched checkerboard;
masking the searched checkerboard, and then continuing to search other checkerboards in the second color image and obtain the central point coordinates of the other checkerboards until the central point coordinates of each checkerboard in each second color image are obtained;
polar angle sequencing is carried out on the central point coordinates of all the chessboards in each second color image, and then the central point coordinates of the chessboards are sequentially connected according to the polar angle sequencing result, so that a closed area is obtained, namely the interested area of the second color image of each camera.
As can be seen from the above description, for the recognition of a plurality of checkerboards, the camera performs mask masking after recognizing the checkerboard and obtaining the coordinates of the center point, so that the camera can not recognize the checkerboard any more, but starts to recognize other checkerboards, thereby completing the recognition and positioning of the plurality of checkerboards to obtain the region of interest of the color image of the camera.
Further, the number of the cameras is two, the iterative operation is performed on the initial calibration matrix of the point clouds of all the cameras, and the final calibration matrix of the point clouds of all the cameras is obtained, which comprises the following steps:
the point cloud A set and the point cloud B set of the two cameras are respectively set as a set P and a set Q, and the matching relation M of the set P and the set Q is obtained as follows:
M＝match(P,Q)＝{(p,q):p∈P,q∈Q}
the error between the set P and the set Q is defined as:
error(P,Q)＝∑ (p,q∈M) w(p,q)d(p,q)
wherein w (P, Q) represents the weight of the corresponding point cloud in the set P and the set Q, and d (P, Q) represents the distance between the corresponding point clouds in the set P and the set Q;
performing iterative operation on the point cloud preliminary calibration matrix through an iterative calculation formula to obtain the point cloud final calibration matrix of all cameras, wherein the iterative calculation formula is as follows:
the i and i+1 represent the number of iterations, theFor iterative integration of transforms, the T init And (5) primarily calibrating a matrix for the point cloud.
It can be seen from the above description that, for the calibration matrix, one of the two camera point clouds is converted to the coordinate system of the other point cloud, and then the defined error value is continuously reduced through iterative operation until the optimal final calibration matrix of the point clouds is obtained, that is, the fine registration of the point clouds of the two cameras is completed.
Further, the splicing of the point clouds of the cameras according to the point cloud calibration matrix between the cameras, and the obtaining of the reconstructed three-dimensional scene specifically includes the following steps:
calculating overlapping point cloud areas among a plurality of cameras according to point cloud calibration matrixes among the plurality of cameras, and removing the overlapping point cloud areas of one camera;
and splicing the point clouds of all cameras after the overlapping point cloud areas are removed according to the point cloud calibration matrixes among the cameras, so as to obtain a reconstructed three-dimensional scene.
As can be seen from the above description, since the point cloud data of the plurality of cameras has an overlapping region, in the embodiment, when the point cloud splicing is performed, the overlapping region is removed first, and then the two point clouds are spliced, so that the operation amount is reduced, the splicing speed is improved, and the reconstruction of the three-dimensional scene is better and faster realized.
Further, the calculating the overlapping point cloud areas between the cameras according to the point cloud calibration matrix between the cameras, and removing the overlapping point cloud area of one of the cameras specifically includes the following steps:
the visual field of the camera is equivalent to a visual cone, and objects in the visual field of the camera are cut by using a near plane and a far plane to form a visual section, wherein the visual section comprises a left face, a right face, a top face, a bottom face, a near plane and a far plane;
locating the camera at the origin of the world coordinate system, locating the camera in a positive direction along the Z-axis, locating the camera in the direction of the line of sightThe view matrix of the machine is set as an identity matrix, and vertexes are expressed as: v= (xyzw=1) T The projection matrix S of 4*4 is expressed as: m=(s) kl )；
The vertex obtained by rotating the projection matrix S is v '= (x' y 'z' w=1) T The method comprises the following steps:
the row is a point multiplication i ＝(s i1 s i2 s i3 s i4 ) An ith row of data representing the projection matrix S;
if the vertex v' is located within the viewing section, the following first inequality should be satisfied:
-w'<x'<w'
-w^'<y^'<w^'
-w'<z'<w'
obtaining an inequality corresponding to each intercepting plane according to a first inequality of the viewing section and the position relation of six intercepting planes in the viewing section, so as to calculate a plane equation coefficient of each intercepting plane, and obtain a plane equation expression of each intercepting plane;
judging the position relation between the point cloud of the camera and each intercepting plane of other cameras one by one according to the plane equation expression of each intercepting plane corresponding to each camera, and taking the point cloud on the inner side of all intercepting planes in any one camera as an overlapped point cloud area of the camera;
and removing the overlapping point cloud area of any one of the cameras comprising the overlapping point cloud area.
From the above description, it can be known that by performing calculation of plane equation expressions on six interception planes of the view frustum, and then determining the positional relationship between the point clouds of other cameras and the six interception planes of the camera, it is determined whether there is overlapping with the view frustum of the camera in the point clouds of other cameras, if so, the point clouds are considered to be overlapping, so that the overlapping point cloud area is rapidly and accurately identified.
Further, the plurality of checkerboards is four checkerboards.
From the foregoing, it can be seen that a preferred embodiment is provided.
Another embodiment of the present invention provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the multi-camera based three-dimensional scene reconstruction method described above.
Referring to fig. 17, another embodiment of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, where the processor implements the multi-camera-based three-dimensional scene reconstruction method when executing the computer program.
With respect to the specific implementation procedure and corresponding effect of the image scaling method contained in the computer program in the above two embodiments, reference may be made to the relevant description in the multi-camera-based three-dimensional scene reconstruction method in the previous embodiment.
Thus, the following embodiments provide a plurality of cameras within a scene to reconstruct the scene in three dimensions, such as when an AR (Augmented Reality ) system needs to reconstruct a real scene in three dimensions, which is specifically as follows:
referring to fig. 1 to 16, a first embodiment of the present invention is as follows:
the three-dimensional scene reconstruction method based on multiple cameras, as shown in fig. 1, comprises the following steps:
s1, acquiring a primary position relation calibration matrix among a plurality of cameras;
in this embodiment, as shown in fig. 2, step S1 specifically includes the following steps:
s11, acquiring first color images shot by each camera on a checkerboard, and obtaining angular point coordinates of the checkerboard in each first color image;
in this embodiment, as shown in fig. 4, a plurality of first color images including a checkerboard may be acquired at the same time and in different poses, for example, in this embodiment, the number of the first color images is about 100. Then the angular point coordinates of the checkerboard are found by adopting the existing lookup checkerboard algorithm.
S12, matching each corner coordinate of the first color images of different cameras according to the epipolar geometry constraint principle to obtain a corner matching relationship among a plurality of cameras;
in the present embodiment, as shown in FIG. 10, wherein I 1 And I 2 Image planes which are color images on two cameras, wherein the center of each camera is O 1 And O 2 . Wherein P is 1 Is one corner point on the checkerboard at I 1 Corresponding projection points, the same as P 2 Is the same angular point on the checkerboard is at I 2 Corresponding to the projection point. Point O 1 、O 2 And the plane defined by the P three points is the polar plane. O (O) 1 O 2 Connection line and image plane I 1 、I 2 The intersection points of (a) are respectively e 1 And e 2 . Wherein e 1 And e 2 Called poles, O 1 O 2 Referred to as the baseline. Polar plane and two image planes I 1 And I 2 Intersecting line l between 1 And l 2 And the corner points are epipolar lines, so that whether the corner points respectively identified by the cameras are the same corner points can be judged according to the fact that the same corner points on the checkerboard are intersected at the P points, and the corner point pairing relation among the cameras is obtained.
S13, obtaining a preliminary position relation calibration matrix among the cameras according to the corner pairing relation among the cameras.
In this embodiment, two cameras are selected to collect images of a scene under different viewing angles, and in other equivalent embodiments, three or four cameras may be selected to collect images.
S2, acquiring color images and depth images which are respectively shot by a plurality of cameras on a scene to be reconstructed, wherein the scene to be reconstructed is provided with a plurality of checkerboards, and acquiring a point cloud calibration matrix among the plurality of cameras according to the color images and the depth images of the plurality of cameras and a preliminary position relationship calibration matrix among the plurality of cameras;
and S3, splicing the point clouds of the cameras according to the point cloud calibration matrix among the cameras to obtain a reconstructed three-dimensional scene.
Referring to fig. 1 to 16, a second embodiment of the present invention is as follows:
based on the first embodiment, as shown in fig. 3, the step S2 specifically includes the following steps:
s21, acquiring second color images and depth images which are respectively shot by a plurality of cameras on a scene to be reconstructed, wherein the scene to be reconstructed is provided with a plurality of checkerboards;
in this embodiment, as shown in fig. 5, four checkerboards are disposed in the scene to be reconstructed. Thus, two cameras need to capture a color image comprising four checkerboards and a depth image, wherein the depth image corresponds to a point cloud. In other equivalent embodiments, two, three or five or more checkerboards may be selected.
S22, finding each checkerboard in each second color image, and obtaining the center point coordinate of each checkerboard in each second color image to obtain the interested area of the second color image of each camera;
the step S22 specifically includes the following steps:
s221, searching the checkerboard in each second color image, and obtaining corner coordinates of the searched checkerboard to obtain center point coordinates of the searched checkerboard;
in this embodiment, the checkerboard is searched according to the existing checkerboard searching algorithm, and first, one checkerboard is identified, for example, the lower right corner checkerboard in fig. 5 in this embodiment, to obtain the center point coordinate of the checkerboard.
S222, masking the searched checkerboard, and then continuing to search other checkerboards in the second color image and obtain the center point coordinates of the other checkerboards until the center point coordinates of each checkerboard in each second color image are obtained;
the mask is a binary image composed of 0 and 1, and as shown in fig. 6, the lower right checkerboard is masked, i.e. the lower right checkerboard is masked by using pixel value 0 instead. Thus, the coordinates of the center points of the four checkerboards can be obtained.
S223, polar angle ordering is carried out on the central point coordinates of all the chessboards in each second color image, and then the central point coordinates of the chessboards are sequentially connected according to the polar angle ordering result, so that a closed area is obtained, namely the interested area of the second color image of each camera.
The polar angle ordering is to take a fixed point called pole in a plane, draw a ray called polar axis, select a positive direction of length unit and angle, and then arrange the points on a given plane into a preset direction according to a selected fixed point. Corresponding to this embodiment, as shown in fig. 7, the coordinates of the center points of the four checkerboards are sequentially connected to obtain a rectangle, that is, the region of interest.
S23, obtaining the interested region of the point cloud of each camera according to a preset position relation matrix between the second color image and the depth image in each camera and the interested region of the second color image;
the RGBD cameras capable of acquiring the color image and the depth image store a preset positional relationship matrix between the color image and the depth image in advance, so that a region of interest of a point cloud of each camera is obtained, as shown in fig. 8.
S24, performing preliminary registration on the point clouds of all cameras according to the preliminary position relation calibration matrix among the cameras and the interested areas of the point clouds of each camera to obtain a point cloud preliminary calibration matrix of all cameras;
the effect obtained after the point clouds of the two cameras are initially registered is shown in fig. 9.
And S25, performing iterative operation on the point cloud preliminary calibration matrixes of all cameras to obtain a point cloud final calibration matrix of all cameras.
In this embodiment, the iterative operation uses an iterative closest point algorithm, whose mathematical form represents:
wherein, A p denotes the point cloud a and, B q represents a point cloud B, and the fine registration is performed by converting the point cloud A into a coordinate system of the point cloud B to enable error (T # A P), B Q) is minimal. Thus, step S25 specifically includes the steps of:
the point cloud A set and the point cloud B set of the two cameras are respectively set as a set P and a set Q, and the matching relation M of the set P and the set Q is obtained as follows:
M＝match(P,Q)＝{(p,q):p∈P,q∈Q}
the error between set P and set Q is defined as:
error(P,Q)＝∑ (p,q∈M) w(p,q)d(p,q)
wherein w (P, Q) represents the weight of the corresponding point clouds in the set P and the set Q, and d (P, Q) represents the distance between the corresponding point clouds in the set P and the set Q;
performing iterative operation on the point cloud preliminary calibration matrix through an iterative calculation formula to obtain point cloud final calibration matrixes of all cameras, wherein the iterative calculation formula is as follows:
i and i +1 represent the number of iterations,for iterative integration of transformations, T init And (5) primarily calibrating a matrix for the point cloud.
Thereby, a fine registration of the point clouds of the two cameras is completed.
Referring to fig. 1 to 16, a third embodiment of the present invention is as follows:
based on the first embodiment, the step S3 specifically includes the following steps:
s31, calculating an overlapped point cloud area between a plurality of cameras according to a point cloud calibration matrix between the plurality of cameras, and removing the overlapped point cloud area of one camera;
in this embodiment, the step S31 specifically includes the following steps:
s311, the visual field of the camera is equivalent to a view cone, and objects in the visual field of the camera are cut by using a near plane and a far plane to form a view section, wherein the view section comprises a left face, a right face, a top face, a bottom face, a near plane and a far plane;
fig. 11 is a view section including the above six cutting planes, and fig. 12 is a view section XZ-plane screenshot.
S312, the camera is located at the origin of the world coordinate system, the line of sight direction of the camera is the positive direction along the Z axis, the view matrix of the camera is set as the identity matrix, and the vertices are expressed as: v= (xyzw=1) T The projection matrix S of 4*4 is expressed as: s= (S) kl )；
S313, the vertex obtained by rotating the projection matrix S is v '= (x' y 'z' w=1) T The method comprises the following steps:
wherein, is dot product, row i ＝(s i1 s i2 s i3 s i4 ) The ith row of data representing the projection matrix S is converted into homogeneous coordinates from the vertexes of the world coordinate system after the transformation;
s314, if the vertex v' is located in the viewing section, the following first inequality should be satisfied:
-w'<x'<w'
-w^'<y^'<w^'
-w'<z'<w'
s315, obtaining an inequality corresponding to each interception plane according to a first inequality of the interception plane and the position relation of six interception planes in the interception plane, so as to calculate a plane equation coefficient of each interception plane and obtain a plane equation expression of each interception plane;
wherein, it is assumed that x' satisfies the following inequality:
-w'<x'
from the above rotation equation:
-(v·row 4 )<(v·row 1 )
that is:
0<(v·row 4 )+(v·row 1 )
finally, the method can obtain:
0<v·(row 4 +row 1 )
the plane equation for the left cross-sectional plane can be expressed as:
x(m 41 +m 11 )+y(m 42 +m 12 )+z(m 43 +m 13 )+w(m 44 +m 14 )＝0
since w=1, there is
x(m 41 +m 11 )+y(m 42 +m 12 )+z(m 43 +m 13 )+m 44 +m 14 ＝0
The standard plane equation is:
ax+by+cz+d＝0
the plane equation coefficients for the left cross-section plane are:
a＝m 41 +m 11 ，b＝m 42 +m 12 ，c＝m 43 +m 13 ，d＝m 44 +m 14
repeating the steps to obtain the plane equation coefficient of each intercepting plane according to the inequality corresponding to different intercepting planes, wherein the plane equation coefficient is shown in the table 1:
TABLE 1 plane equation results for the intercept plane
From this, plane equation expressions for six intercept planes can be found.
S316, judging the position relation between the point clouds of the cameras and each intercepting plane of other cameras one by one according to the plane equation expression of each intercepting plane corresponding to each camera, and taking the point clouds inside all intercepting planes in any one camera as overlapping point cloud areas of the cameras;
wherein viewing the viewing section from the cross section, the relationship of the plane of interception to the point can be obtained as shown in fig. 13. I.e. a plane can be defined by P and normal vectors on the planeDetermining that any point on the plane P' satisfies:
then given any point Q, there are:
the point Q is outside the clipping plane;
the point Q is on the clipping plane;
the point Q is inside the clipping plane;
thereby, an overlapping point cloud area of the two cameras is obtained.
And S317, removing the overlapping point cloud area of any one of the cameras comprising the overlapping point cloud area.
That is, fig. 14 is a point cloud of one of the cameras, and fig. 15 is obtained after removing the overlapping point cloud area.
And S32, splicing the point clouds of all cameras after the overlapping point cloud areas are removed according to the point cloud calibration matrix among the cameras, and obtaining the reconstructed three-dimensional scene.
The fourth embodiment of the invention is as follows:
another embodiment of the present invention provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the multi-camera-based three-dimensional scene reconstruction method of any one of the above embodiments one to three.
Referring to fig. 17, a fifth embodiment of the present invention is as follows:
an electronic device 1 comprises a memory 3, a processor 2 and a computer program stored on the memory 3 and executable on the processor 2, the processor 2 implementing the multi-camera based three-dimensional scene reconstruction method of any of the above embodiments one to three when executing the computer program.
In summary, the three-dimensional scene reconstruction method, the storage medium and the electronic device based on the multiple cameras provided by the invention adopt the existing searching chessboard algorithm to identify the chessboard, pair the angular point coordinates of the identified chessboard according to the epipolar geometry constraint principle, and obtain the preliminary position relation calibration matrix among the multiple cameras; after the interested areas of the color images are obtained through a plurality of chequers, the interested areas of the point clouds of each camera can be obtained according to the position relation matrix of the color images and the depth images, the initial registration of the point clouds can be realized according to the initial position relation calibration matrix, and then the fine registration is carried out through an iterative nearest point algorithm, so that a more accurate final calibration matrix of the point clouds is obtained; finally, according to the point cloud calibration matrix among the cameras, the overlapping point cloud areas are removed firstly, then the point clouds of the cameras are spliced to obtain a reconstructed three-dimensional scene, namely under the condition that the cameras are fixed, the point clouds of the cameras are registered and spliced by a machine vision method to obtain the reconstructed three-dimensional scene, so that the reconstruction of the three-dimensional scene can be better and faster realized, and meanwhile, the method is time-saving and labor-saving and has universality.
The foregoing description is only illustrative of the present invention and is not intended to limit the scope of the invention, and all equivalent changes made by the specification and drawings of the present invention, or direct or indirect application in the relevant art, are included in the scope of the present invention.
Claims (9)
1. The three-dimensional scene reconstruction method based on the multiple cameras is characterized by comprising the following steps of:
acquiring a primary position relation calibration matrix among a plurality of cameras;
acquiring color images and depth images which are respectively shot by a plurality of cameras on a scene to be reconstructed, wherein the scene to be reconstructed is provided with a plurality of checkerboards, and acquiring a point cloud calibration matrix among the plurality of cameras according to the color images and the depth images of the plurality of cameras and a preliminary position relationship calibration matrix among the plurality of cameras;
splicing the point clouds of the cameras according to the point cloud calibration matrixes among the cameras to obtain a reconstructed three-dimensional scene;
the method for obtaining the point cloud calibration matrix between the cameras comprises the following steps of:
acquiring second color images and depth images which are respectively shot by a plurality of cameras on a scene to be reconstructed, wherein the scene to be reconstructed is provided with a plurality of checkerboards;
searching each checkerboard in each second color image, and obtaining the center point coordinate of each checkerboard in each second color image to obtain the interested area of the second color image of each camera;
obtaining a region of interest of a point cloud of each camera according to a preset position relation matrix between the second color image and the depth image in each camera and the region of interest of the second color image;
performing preliminary registration on the point clouds of all cameras according to the preliminary position relation calibration matrix among the cameras and the interested area of the point clouds of each camera to obtain the point cloud preliminary calibration matrix of all the cameras;
and performing iterative operation on the point cloud preliminary calibration matrixes of all cameras to obtain the point cloud final calibration matrixes of all cameras.
2. The multi-camera based three-dimensional scene reconstruction method according to claim 1, wherein the obtaining the preliminary positional relationship calibration matrix between the plurality of cameras specifically comprises the steps of:
acquiring a first color image shot by each camera on a checkerboard, and obtaining angular point coordinates of the checkerboard in each first color image;
pairing each angular point coordinate of the first color images of different cameras according to the epipolar geometry constraint principle to obtain angular point pairing relations among a plurality of cameras;
and obtaining a plurality of preliminary position relation calibration matrixes among the cameras according to the corner pairing relations among the cameras.
3. The multi-camera based three-dimensional scene reconstruction method according to claim 1, wherein said finding each of said second color images and obtaining center point coordinates of each of said second color images, obtaining a region of interest of said second color image for each of said cameras, comprises the steps of:
searching the checkerboard in each second color image, and obtaining the corner coordinates of the searched checkerboard to obtain the center point coordinates of the searched checkerboard;
masking the searched checkerboard, and then continuing to search other checkerboards in the second color image and obtain the central point coordinates of the other checkerboards until the central point coordinates of each checkerboard in each second color image are obtained;
polar angle sequencing is carried out on the central point coordinates of all the chessboards in each second color image, and then the central point coordinates of the chessboards are sequentially connected according to the polar angle sequencing result, so that a closed area is obtained, namely the interested area of the second color image of each camera.
4. The three-dimensional scene reconstruction method based on multiple cameras as set forth in claim 1, wherein the number of cameras is two, the iterative operation is performed on the point cloud preliminary calibration matrices of all the cameras, and obtaining the point cloud final calibration matrices of all the cameras includes the following steps:
the point cloud A set and the point cloud B set of the two cameras are respectively set as a set P and a set Q, and the matching relation M of the set P and the set Q is obtained as follows:
M＝match(P,Q)＝{(p,q):p∈P,q∈Q}
the error between the set P and the set Q is defined as:
error(P,Q)＝∑ (p,q∈M) w(p,q)d(p,q)
wherein w (P, Q) represents the weight of the corresponding point cloud in the set P and the set Q, and d (P, Q) represents the distance between the corresponding point clouds in the set P and the set Q;
performing iterative operation on the point cloud preliminary calibration matrix through an iterative calculation formula to obtain the point cloud final calibration matrix of all cameras, wherein the iterative calculation formula is as follows:
the i and i+1 represent the number of iterations, theFor iterative integration of transforms, the T init And (5) primarily calibrating a matrix for the point cloud.
5. The multi-camera-based three-dimensional scene reconstruction method according to claim 1, wherein the stitching the point clouds of the cameras according to the point cloud calibration matrix between the cameras, to obtain the reconstructed three-dimensional scene specifically comprises the following steps:
calculating overlapping point cloud areas among a plurality of cameras according to point cloud calibration matrixes among the plurality of cameras, and removing the overlapping point cloud areas of one camera;
and splicing the point clouds of all cameras after the overlapping point cloud areas are removed according to the point cloud calibration matrixes among the cameras, so as to obtain a reconstructed three-dimensional scene.
6. The method for reconstructing a three-dimensional scene based on multiple cameras according to claim 5, wherein calculating overlapping point cloud areas between multiple cameras according to a point cloud calibration matrix between the multiple cameras, and removing the overlapping point cloud areas of one of the cameras specifically comprises the following steps:
the visual field of the camera is equivalent to a visual cone, and objects in the visual field of the camera are cut by using a near plane and a far plane to form a visual section, wherein the visual section comprises a left face, a right face, a top face, a bottom face, a near plane and a far plane;
the camera is located at the origin of the world coordinate system, the sight line direction of the camera is the positive direction along the Z axis, the view matrix of the camera is set as an identity matrix, and the vertex is expressed as: v= (xyzw=1) T The projection matrix S of 4*4 is expressed as: s= (S) kl )；
The vertex obtained by rotating the projection matrix S is v '= (x' y 'z' w=1) T The method comprises the following steps:
the row is a point multiplication i ＝(s i1 s i2 s i3 s i4 ) An ith row of data representing the projection matrix S;
if the vertex v' is located within the viewing section, the following first inequality should be satisfied:
-w'<x'<w'
-w^'<y^'<w^'
-w'<z'<w'
obtaining an inequality corresponding to each intercepting plane according to a first inequality of the viewing section and the position relation of six intercepting planes in the viewing section, so as to calculate a plane equation coefficient of each intercepting plane, and obtain a plane equation expression of each intercepting plane;
judging the position relation between the point cloud of the camera and each intercepting plane of other cameras one by one according to the plane equation expression of each intercepting plane corresponding to each camera, and taking the point cloud on the inner side of all intercepting planes in any one camera as an overlapped point cloud area of the camera;
and removing the overlapping point cloud area of any one of the cameras comprising the overlapping point cloud area.
7. The multi-camera based three-dimensional scene reconstruction method according to any of claims 1 to 6, wherein the plurality of checkerboards is four checkerboards.
8. A computer-readable storage medium having stored thereon a computer program, characterized by: the computer program, when executed by a processor, implements the three-dimensional scene reconstruction method as claimed in any one of claims 1 to 7.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the three-dimensional scene reconstruction method according to any of claims 1-7 when the computer program is executed by the processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011281518.9A CN112381886B (en) | 2020-11-16 | 2020-11-16 | Three-dimensional scene reconstruction method based on multiple cameras, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011281518.9A CN112381886B (en) | 2020-11-16 | 2020-11-16 | Three-dimensional scene reconstruction method based on multiple cameras, storage medium and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112381886A CN112381886A (en) | 2021-02-19 |
| CN112381886B true CN112381886B (en) | 2024-04-09 |
Family
ID=74584818
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011281518.9A Active CN112381886B (en) | 2020-11-16 | 2020-11-16 | Three-dimensional scene reconstruction method based on multiple cameras, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112381886B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112927353B (en) * | 2021-02-25 | 2023-05-19 | 电子科技大学 | Three-dimensional scene reconstruction method, storage medium and terminal based on two-dimensional target detection and model alignment |
| CN113052917B (en) * | 2021-03-29 | 2024-05-07 | 黑芝麻智能科技(上海)有限公司 | Method for acquiring image coordinates of invisible position of camera, calibration method and system |
| CN113012238B (en) * | 2021-04-09 | 2024-04-16 | 南京星顿医疗科技有限公司 | Method for quick calibration and data fusion of multi-depth camera |
| CN113516694A (en) * | 2021-04-13 | 2021-10-19 | 同济大学 | Multi-Kinect-based dynamic visual control method for remote medical scene |
| CN113362445B (en) * | 2021-05-25 | 2023-05-05 | 上海奥视达智能科技有限公司 | Method and device for reconstructing object based on point cloud data |
| CN113504385B (en) * | 2021-06-30 | 2023-07-14 | 安徽爱观视觉科技有限公司 | Speed measuring method and device for plural cameras |
| CN114554176A (en) * | 2022-01-24 | 2022-05-27 | 北京有竹居网络技术有限公司 | Depth camera |
| CN117830439B (en) * | 2024-03-05 | 2024-06-14 | 南昌虚拟现实研究院股份有限公司 | Multi-camera system pose calibration method and device |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109064404A (en) * | 2018-08-10 | 2018-12-21 | 西安电子科技大学 | It is a kind of based on polyphaser calibration panorama mosaic method, panoramic mosaic system |
| CN109242913A (en) * | 2018-09-07 | 2019-01-18 | 百度在线网络技术（北京）有限公司 | Scaling method, device, equipment and the medium of collector relative parameter |
| CN110223377A (en) * | 2019-05-28 | 2019-09-10 | 上海工程技术大学 | One kind being based on stereo visual system high accuracy three-dimensional method for reconstructing |
| CN111127625A (en) * | 2019-10-08 | 2020-05-08 | 新拓三维技术(深圳)有限公司 | Foot scanning method, system and device |
| CN111150175A (en) * | 2019-12-05 | 2020-05-15 | 新拓三维技术(深圳)有限公司 | Method, device and system for three-dimensional scanning of feet |
-
2020
- 2020-11-16 CN CN202011281518.9A patent/CN112381886B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109064404A (en) * | 2018-08-10 | 2018-12-21 | 西安电子科技大学 | It is a kind of based on polyphaser calibration panorama mosaic method, panoramic mosaic system |
| CN109242913A (en) * | 2018-09-07 | 2019-01-18 | 百度在线网络技术（北京）有限公司 | Scaling method, device, equipment and the medium of collector relative parameter |
| CN110223377A (en) * | 2019-05-28 | 2019-09-10 | 上海工程技术大学 | One kind being based on stereo visual system high accuracy three-dimensional method for reconstructing |
| CN111127625A (en) * | 2019-10-08 | 2020-05-08 | 新拓三维技术(深圳)有限公司 | Foot scanning method, system and device |
| CN111150175A (en) * | 2019-12-05 | 2020-05-15 | 新拓三维技术(深圳)有限公司 | Method, device and system for three-dimensional scanning of feet |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112381886A (en) | 2021-02-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112381886B (en) | Three-dimensional scene reconstruction method based on multiple cameras, storage medium and electronic equipment | |
| CN111783820B (en) | Image labeling method and device | |
| CN106503671B (en) | The method and apparatus for determining human face posture | |
| US20200005486A1 (en) | Device pose estimation using 3d line clouds | |
| CN110276317B (en) | Object size detection method, object size detection device and mobile terminal | |
| EP3428875A1 (en) | Methods and apparatuses for panoramic image processing | |
| US20110293142A1 (en) | Method for recognizing objects in a set of images recorded by one or more cameras | |
| CN114119864A (en) | Positioning method and device based on three-dimensional reconstruction and point cloud matching | |
| CN114863037A (en) | Single-mobile-phone-based human body three-dimensional modeling data acquisition and reconstruction method and system | |
| Khoshelham et al. | Generation and weighting of 3D point correspondences for improved registration of RGB-D data | |
| CN108028904B (en) | Method and system for light field augmented reality/virtual reality on mobile devices | |
| CN112085802A (en) | Method for acquiring three-dimensional finger vein image based on binocular camera | |
| CN109613974B (en) | AR home experience method in large scene | |
| CN113537047A (en) | Obstacle detection method, obstacle detection device, vehicle and storage medium | |
| CN107818596B (en) | Scene parameter determination method and device and electronic equipment | |
| Ventura et al. | Structure and motion in urban environments using upright panoramas | |
| CN117372647A (en) | Rapid construction method and system of three-dimensional model for building | |
| CN115953471A (en) | Indoor scene multi-scale vector image retrieval and positioning method, system and medium | |
| Deepu et al. | 3D Reconstruction from Single 2D Image | |
| CN107644394A (en) | A kind of processing method and processing device of 3D rendering | |
| CN109314774A (en) | System and method for three-dimensional imaging | |
| KR20160049639A (en) | Stereoscopic image registration method based on a partial linear method | |
| CN109341530B (en) | Object point positioning method and system in binocular stereo vision | |
| Liu et al. | Improved template matching based stereo vision sparse 3D reconstruction algorithm | |
| CN111210476A (en) | Method and device for simultaneously positioning and establishing image |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |