CN112381886A - Multi-camera-based three-dimensional scene reconstruction method, storage medium and electronic device - Google Patents

Multi-camera-based three-dimensional scene reconstruction method, storage medium and electronic device Download PDF

Info

Publication number
CN112381886A
CN112381886A CN202011281518.9A CN202011281518A CN112381886A CN 112381886 A CN112381886 A CN 112381886A CN 202011281518 A CN202011281518 A CN 202011281518A CN 112381886 A CN112381886 A CN 112381886A
Authority
CN
China
Prior art keywords
cameras
point cloud
camera
point
matrix
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202011281518.9A
Other languages
Chinese (zh)
Other versions
CN112381886B (en
Inventor
邱又海
黄智洲
徐倩茹
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Zhouming Technology Co Ltd
Original Assignee
Shenzhen Zhouming Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Zhouming Technology Co Ltd filed Critical Shenzhen Zhouming Technology Co Ltd
Priority to CN202011281518.9A priority Critical patent/CN112381886B/en
Publication of CN112381886A publication Critical patent/CN112381886A/en
Application granted granted Critical
Publication of CN112381886B publication Critical patent/CN112381886B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G06T7/85—Stereo camera calibration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G06T7/344—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods involving models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G06T7/75—Determining position or orientation of objects or cameras using feature-based methods involving models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30204—Marker
    • G06T2207/30208—Marker matrix
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30244—Camera pose

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computer Graphics (AREA)
  • Geometry (AREA)
  • Software Systems (AREA)
  • Length Measuring Devices By Optical Means (AREA)
  • Image Processing (AREA)

Abstract

The invention discloses a multi-camera-based three-dimensional scene reconstruction method, a storage medium and electronic equipment, which comprise the following steps: acquiring a preliminary position relation calibration matrix among a plurality of cameras; acquiring color images and depth images which are respectively shot by a plurality of cameras for a scene to be reconstructed and placed with a plurality of checkerboards, and calibrating a matrix according to the color images and the depth images of the plurality of cameras and the preliminary position relation among the plurality of cameras to obtain a point cloud calibration matrix among the plurality of cameras; splicing the point clouds of the multiple cameras according to the point cloud calibration matrix among the multiple cameras to obtain a reconstructed three-dimensional scene; under the condition that the cameras are fixed, the point clouds of the cameras are registered and spliced by a machine vision method to obtain a reconstructed three-dimensional scene, so that the reconstruction of the three-dimensional scene can be better and faster realized, and the method is more time-saving, labor-saving and universal.

Description

Multi-camera-based three-dimensional scene reconstruction method, storage medium and electronic device
Technical Field
The invention relates to the technical field of image processing, in particular to a multi-camera-based three-dimensional scene reconstruction method, a storage medium and electronic equipment.
Background
In an augmented reality application, in order to enable an experiencer to obtain an immersive experience, a real scene needs to be subjected to scene reconstruction, so that the experiencer has an immersive feeling. In the prior art, an indoor three-dimensional scene reconstruction method based on a double-layer registration method uses a surf (Speeded Up Robust Features) algorithm to extract and match feature points, then maps the matched points to three-dimensional coordinates, and then uses a Random Sample Consensus (Random Sample Consensus) algorithm and an Iterative Closest Point (ICP) algorithm to perform registration.
Disclosure of Invention
The technical problem to be solved by the invention is as follows: provided are a multi-camera-based three-dimensional scene reconstruction method, a storage medium and an electronic device, which can rapidly realize the reconstruction of a three-dimensional scene.
In order to solve the technical problems, the invention adopts the technical scheme that:
a three-dimensional scene reconstruction method based on multiple cameras comprises the following steps:
acquiring a preliminary position relation calibration matrix among a plurality of cameras;
acquiring color images and depth images which are respectively shot by the plurality of cameras for a scene to be reconstructed and provided with the plurality of checkerboards, and calibrating a matrix according to the color images and the depth images of the plurality of cameras and the preliminary position relation among the plurality of cameras to obtain a point cloud calibration matrix among the plurality of cameras;
and splicing the point clouds of the cameras according to the point cloud calibration matrix among the cameras to obtain the reconstructed three-dimensional scene.
In order to solve the technical problem, the invention adopts another technical scheme as follows:
a computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the above-mentioned multi-camera based three-dimensional scene reconstruction method.
In order to solve the technical problem, the invention adopts another technical scheme as follows:
an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the multi-camera based three-dimensional scene reconstruction method as described above when executing the computer program.
The invention has the beneficial effects that: the method comprises the steps of obtaining a preliminary position relation calibration matrix among a plurality of cameras in advance, then placing a plurality of checkerboards in a three-dimensional scene to be reconstructed, obtaining a point cloud calibration matrix among the plurality of cameras according to color images and point clouds of the checkerboards in different cameras, and finally splicing the point clouds of the plurality of cameras according to the point cloud calibration matrix among the plurality of cameras to obtain a reconstructed three-dimensional scene, namely, under the condition that the cameras are fixed, registering and splicing the point clouds of the plurality of cameras by a machine vision method to obtain the reconstructed three-dimensional scene.
Drawings
Fig. 1 is a schematic flow chart of a multi-camera-based three-dimensional scene reconstruction method according to an embodiment of the present invention;
FIG. 2 is a schematic flow chart illustrating a process of generating a preliminary location relationship calibration matrix according to an embodiment of the present invention;
FIG. 3 is a schematic flow chart of point cloud registration and stitching according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a checkerboard according to embodiments of the present invention;
FIG. 5 is a schematic diagram of four checkerboards according to an embodiment of the present invention;
fig. 6 is a schematic diagram illustrating an effect of masking the searched checkerboard according to the embodiment of the present invention;
fig. 7 is a schematic diagram illustrating an effect of a region of interest of a second color image according to an embodiment of the present invention;
FIG. 8 is a schematic diagram illustrating the effect of a region of interest of a point cloud according to an embodiment of the present invention;
fig. 9 is a schematic diagram illustrating an effect of performing preliminary registration on point clouds according to an embodiment of the present invention;
FIG. 10 is a schematic diagram of a calculation for generating a preliminary location relationship calibration matrix according to an embodiment of the present invention;
fig. 11 is a perspective view of a viewing body according to an embodiment of the present invention;
FIG. 12 is a schematic view of a cutting plane of an XZ plane of a viewing body according to an embodiment of the present invention;
FIG. 13 is a schematic diagram of the location of a point cloud and a clipping plane according to an embodiment of the present invention;
FIG. 14 is a schematic point cloud of a camera according to an embodiment of the present invention;
fig. 15 is a schematic diagram of a point cloud after a camera according to an embodiment of the present invention removes an overlapping point cloud region;
FIG. 16 is a schematic diagram illustrating the effect of two cameras after point cloud stitching according to an embodiment of the present invention;
fig. 17 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Description of reference numerals:
1. an electronic device; 2. a processor; 3. a memory.
Detailed Description
In order to explain technical contents, achieved objects, and effects of the present invention in detail, the following description is made with reference to the accompanying drawings in combination with the embodiments.
Referring to fig. 1 to 16, a multi-camera based three-dimensional scene reconstruction method includes the steps of:
acquiring a preliminary position relation calibration matrix among a plurality of cameras;
acquiring color images and depth images which are respectively shot by the plurality of cameras for a scene to be reconstructed and provided with the plurality of checkerboards, and calibrating a matrix according to the color images and the depth images of the plurality of cameras and the preliminary position relation among the plurality of cameras to obtain a point cloud calibration matrix among the plurality of cameras;
and splicing the point clouds of the cameras according to the point cloud calibration matrix among the cameras to obtain the reconstructed three-dimensional scene.
From the above description, the beneficial effects of the present invention are: the method comprises the steps of obtaining a preliminary position relation calibration matrix among a plurality of cameras in advance, then placing a plurality of checkerboards in a three-dimensional scene to be reconstructed, obtaining a point cloud calibration matrix among the plurality of cameras according to color images and point clouds of the plurality of checkerboards in different cameras, and finally splicing the point clouds of the plurality of cameras according to the point cloud calibration matrix among the plurality of cameras to obtain a reconstructed three-dimensional scene.
Further, the acquiring a preliminary position relationship calibration matrix among a plurality of cameras specifically includes the following steps:
acquiring a first color image shot by each camera for one checkerboard to obtain corner point coordinates of the checkerboard in each first color image;
pairing each corner point coordinate of the first color images of different cameras according to an epipolar geometric constraint principle to obtain a corner point pairing relation among the cameras;
and obtaining a preliminary position relation calibration matrix among the cameras according to the corner pairing relation among the cameras.
From the above description, it can be known that the existing chessboard searching algorithm is adopted to identify the checkerboards, and the coordinates of the corner points of the identified checkerboards are paired according to the epipolar geometric constraint principle, so as to obtain the pairing relationship of different cameras to the same target object, so as to obtain the position relationship of the color images of different cameras, i.e. the preliminary position relationship calibration matrix.
Further, the step of obtaining color images and depth images respectively shot by the plurality of cameras for a scene to be reconstructed, in which the plurality of checkerboards are placed, and obtaining a point cloud calibration matrix among the plurality of cameras according to the color images and the depth images of the plurality of cameras and the preliminary position relationship calibration matrix among the plurality of cameras specifically includes the following steps:
acquiring second color images and depth images which are respectively shot by the cameras for a scene to be reconstructed and provided with a plurality of checkerboards;
finding each checkerboard in each second color image, and obtaining the center point coordinate of each checkerboard in each second color image to obtain the region of interest of each second color image of each camera;
obtaining an interested area of the point cloud of each camera according to a preset position relation matrix between the second color image and the depth image in each camera and the interested area of the second color image;
preliminarily registering the point clouds of all the cameras according to the preliminary position relation calibration matrixes among the cameras and the region of interest of the point cloud of each camera to obtain point cloud preliminary calibration matrixes of all the cameras;
and carrying out iterative operation on the point cloud preliminary calibration matrixes of all the cameras to obtain the point cloud final calibration matrixes of all the cameras.
As can be seen from the above description, for the cameras capable of acquiring the color image and the depth image, the position relationship matrix of the color image and the depth image is preset, after the region of interest of the color image is obtained through a plurality of checkerboards, the region of interest of the point cloud of each camera can be obtained according to the position relationship matrix of the color image and the depth image, and the preliminary registration of the point cloud can be realized by calibrating the matrix according to the preliminary position relationship, and then the precise registration is performed through iterative operation, so that a more accurate final calibration matrix of the point cloud is obtained, and a foundation is laid for the reconstruction accuracy of a subsequent three-dimensional scene.
Further, the finding out each checkerboard in each second color image and obtaining the center point coordinate of each checkerboard in each second color image to obtain the region of interest of the second color image of each camera specifically includes the following steps:
searching each checkerboard in the second color image, and obtaining the corner point coordinates of the searched checkerboard to obtain the center point coordinates of the searched checkerboard;
masking the searched checkerboards, then continuously searching other checkerboards in the second color image and obtaining the center point coordinates of other checkerboards until obtaining the center point coordinates of each checkerboard in each second color image;
and carrying out polar angle sequencing on the center point coordinates of all the chessboard in each second color image, and then sequentially connecting the center point coordinates of the chessboard according to the polar angle sequencing result to obtain a closed area, namely the region of interest of each second color image of each camera.
As can be seen from the above description, for the identification of multiple checkerboards, the camera performs mask masking after identifying the checkerboard and obtaining the coordinates of the central point, so that the camera does not identify the checkerboard any more, but starts to identify other checkerboards, thereby completing the identification and positioning of the multiple checkerboards to obtain the region of interest of the color image of the camera.
Further, the number of the cameras is two, and the iterative operation of the point cloud preliminary calibration matrixes of all the cameras to obtain the point cloud final calibration matrixes of all the cameras comprises the following steps:
respectively setting the point cloud A set and the point cloud B set of the two cameras as a set P and a set Q, and obtaining a matching relation M of the set P and the set Q as follows:
M＝match(P,Q)＝{(p,q):p∈P,q∈Q}
defining the error between the set P and the set Q as:
error(P,Q)＝∑(p,q∈M)w(p,q)d(p,q)
wherein w (P, Q) represents the weight of the corresponding point clouds in the set P and the set Q, and d (P, Q) represents the distance between the corresponding point clouds in the set P and the set Q;
performing iterative operation on the point cloud preliminary calibration matrix through an iterative calculation formula to obtain a point cloud final calibration matrix of all the cameras, wherein the iterative calculation formula is as follows:
Figure BDA0002780952620000061
Figure BDA0002780952620000062
the i and i +1 represent the number of iterations, the
Figure BDA0002780952620000063
For iterative integration of the transformations, the TinitAnd preliminarily calibrating a matrix for the point cloud.
As can be seen from the above description, for the calibration matrix, one of the point clouds of the two cameras is converted into the coordinate system of the other point cloud, and then the defined error value is continuously reduced through iterative operation until the optimal final calibration matrix of the point cloud is obtained, i.e., the precise registration of the point clouds of the two cameras is completed.
Further, the step of splicing the point clouds of the plurality of cameras according to the point cloud calibration matrix among the plurality of cameras to obtain the reconstructed three-dimensional scene specifically includes the following steps:
calculating an overlapped point cloud area among the cameras according to a point cloud calibration matrix among the cameras, and removing the overlapped point cloud area of one camera;
and splicing the point clouds of all the cameras after the overlapped point cloud areas are removed according to the point cloud calibration matrix among the cameras to obtain a reconstructed three-dimensional scene.
As can be seen from the above description, because the point cloud data of the plurality of cameras have the overlapping region, when the point cloud is spliced, the overlapping region is removed first, and then the two point clouds are spliced, so that the calculation amount is reduced, the splicing speed is increased, and the reconstruction of the three-dimensional scene is realized better and faster.
Further, the calculating an overlapping point cloud area between the plurality of cameras according to the point cloud calibration matrix between the plurality of cameras and removing the overlapping point cloud area of one of the cameras specifically includes the following steps:
the method comprises the steps that the visual field of a camera is equal to a view cone, and an object in the visual field of the camera is cut by using a near plane and a far plane to form a viewing frustum which comprises a left surface, a right surface, a top surface, a bottom surface, a near plane and a far plane;
the camera is positioned at the origin of a world coordinate system, the sight line direction of the camera is a positive direction along the Z axis, the view matrix of the camera is set as a unit matrix, and the vertex is expressed as: v ═ 1 (x y z w ═ 1)TThe projection matrix M of 4 x 4 is represented as: m ═ Mij)；
The vertex obtained by rotating the projection matrix M is v '═ 1 (x' y 'z' w)TNamely:
Figure BDA0002780952620000071
the · is a dot product, the rowi＝(mi1 mi2 mi3 mi4) The ith row of data representing the projection matrix M; if vertex v' is located within the frustum, the following first inequality should be satisfied:
-w'<x'<w'
-w^'<y^'<w^'
-w'<z'<w'
obtaining an inequality corresponding to each interception plane according to the first inequality of the interception body and the position relation of the six interception planes in the interception body, so as to calculate a plane equation coefficient of each interception plane, and obtain a plane equation expression of each interception plane;
according to a plane equation expression of each interception plane corresponding to each camera, the position relation between the point cloud of the camera and each interception plane of other cameras is judged one by one, and the point cloud on the inner side of all the interception planes in any one camera is used as an overlapped point cloud area of the cameras;
removing the overlapping point cloud area of any one of the cameras that includes the overlapping point cloud area.
From the above description, the calculation of the plane equation expression is performed on the six interception planes of the frustum, and then the position relationship between the point clouds of other cameras and the six interception planes of the camera is judged, so as to judge whether the point clouds of other cameras overlap with the frustum of the camera, and if so, the point clouds are considered to be overlapped, so that the overlapped point cloud area is quickly and accurately identified.
Further, the plurality of chequers is four chequers.
From the foregoing description, it will be appreciated that a preferred embodiment is provided.
Another embodiment of the invention provides a computer-readable storage medium having stored thereon a computer program which, when being executed by a processor, implements the multi-camera based three-dimensional scene reconstruction method described above.
Referring to fig. 17, another embodiment of the present invention provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the multi-camera based three-dimensional scene reconstruction method when executing the computer program.
With regard to the specific implementation procedures and corresponding effects of the image scaling methods included in the computer programs in the above two embodiments, reference may be made to the related descriptions in the multi-camera based three-dimensional scene reconstruction method in the foregoing embodiments.
Therefore, the following embodiments are provided with a plurality of cameras in a scene to perform three-dimensional reconstruction on the scene, for example, when an AR (Augmented Reality) system needs to perform three-dimensional scene reconstruction on a real scene, the following embodiments are specifically described as follows:
referring to fig. 1 to fig. 16, a first embodiment of the present invention is:
a multi-camera based three-dimensional scene reconstruction method, as shown in fig. 1, includes the steps of:
s1, acquiring a preliminary position relation calibration matrix among the plurality of cameras;
in this embodiment, as shown in fig. 2, step S1 specifically includes the following steps:
s11, acquiring a first color image shot by each camera for one checkerboard, and acquiring corner point coordinates of the checkerboard in each first color image;
in the present embodiment, as shown in fig. 4, a plurality of first color images including checkerboard may be collected at the same time in different poses, for example, in the present embodiment, the number of the first color images is about 100. And then finding the coordinates of the corner points of the checkerboard by adopting the conventional chessboard searching algorithm.
S12, pairing each corner point coordinate of the first color images of different cameras according to an epipolar geometric constraint principle to obtain a corner point pairing relation among the cameras;
in this embodiment, as shown in FIG. 10, where I1And I2Is the image plane of the color image on two cameras, wherein the centers of the cameras are O1And O2. Wherein P is1For a corner point on the checkerboard at I1Up to corresponding projection point, in the same way as P2For the same corner points on the checkerboard at I2And (4) the corresponding projection point. Point O1、O2And the plane defined by the three points P is a polar plane. O is1O2Connecting line and image plane I1、I2Respectively at the intersection points of e1And e2. Wherein e is1And e2Referred to as pole, O1O2Referred to as the baseline. Polar plane and two image planes I1And I2Cross line l between1And l2The corner points are polar lines, so that whether the corner points respectively identified by the cameras are the same corner point can be judged according to the intersection of the same corner points on the checkerboard at the point P, and the corner point pairing relation among the cameras is obtained.
And S13, obtaining a preliminary position relation calibration matrix among the plurality of cameras according to the corner point pairing relation among the plurality of cameras.
In this embodiment, two cameras are selected for image acquisition of a scene at different viewing angles, and in other equivalent embodiments, three, four, and other multiple cameras can be selected for image acquisition.
S2, acquiring color images and depth images which are respectively shot by a plurality of cameras for a scene to be reconstructed and placed with a plurality of checkerboards, and calibrating a matrix according to the color images and the depth images of the plurality of cameras and the preliminary position relation among the plurality of cameras to obtain a point cloud calibration matrix among the plurality of cameras;
and S3, splicing the point clouds of the multiple cameras according to the point cloud calibration matrix among the multiple cameras to obtain the reconstructed three-dimensional scene.
Referring to fig. 1 to fig. 16, a second embodiment of the present invention is:
based on the foregoing first embodiment, as shown in fig. 3, the three-dimensional scene reconstruction method based on multiple cameras, in step S2, specifically includes the following steps:
s21, acquiring second color images and depth images which are respectively shot by a plurality of cameras for a scene to be reconstructed and provided with a plurality of checkerboards;
in the present embodiment, as shown in fig. 5, four checkerboards are placed in the scene to be reconstructed. Thus, two cameras need to capture a color image and a depth image comprising four checkerboards, wherein the depth image corresponds to the point cloud. In other equivalent embodiments, two, three or five, etc. more checkerboards may be selected.
S22, finding each checkerboard in each second color image, and obtaining the center point coordinates of each checkerboard in each second color image to obtain the region of interest of the second color image of each camera;
wherein, step S22 specifically includes the following steps:
s221, searching the checkerboard in each second color image, and obtaining the corner point coordinates of the searched checkerboard to obtain the center point coordinates of the searched checkerboard;
in this embodiment, the checkerboards are searched according to the existing checkerboard searching algorithm, and first, a checkerboard is identified, for example, the checkerboard at the lower right corner of fig. 5 in this embodiment, so as to obtain the coordinates of the center point of the checkerboard.
S222, masking the searched checkerboards, then continuously searching other checkerboards in the second color image and obtaining the center point coordinates of other checkerboards until the center point coordinates of each checkerboard in each second color image are obtained;
the mask is a binary image composed of 0 and 1, and as shown in fig. 6, the checkerboard in the lower right corner is masked, that is, the checkerboard in the lower right corner is masked by replacing the pixel value 0. Thus, the coordinates of the center points of the four chessboard patterns can be obtained.
S223, polar angle sequencing is carried out on the center point coordinates of all the chessboard in each second color image, and then the center point coordinates of the chessboard are sequentially connected according to the polar angle sequencing result to obtain a closed area, namely the region of interest of the second color image of each camera.
The polar angle sorting is that a fixed point is taken in a plane and called a pole point, a ray is led to be called a polar axis, a length unit and the positive direction of an angle are selected, and then points on the given plane are arranged in a preset direction according to the selected fixed point. Corresponding to this embodiment, as shown in fig. 7, the coordinates of the center points of the four chessboard patterns are sequentially connected to obtain a rectangle, which is the region of interest.
S23, obtaining an interested area of the point cloud of each camera according to a preset position relation matrix between the second color image and the depth image in each camera and the interested area of the second color image;
the RGBD camera capable of acquiring the color image and the depth image stores a preset position relationship matrix between the color image and the depth image in advance, so as to obtain an area of interest of the point cloud of each camera, which is shown in fig. 8.
S24, preliminarily registering the point clouds of all the cameras according to the preliminary position relation calibration matrixes among the cameras and the interesting regions of the point clouds of each camera to obtain point cloud preliminary calibration matrixes of all the cameras;
the effect obtained after the point clouds of the two cameras are preliminarily registered is shown in fig. 9.
And S25, carrying out iterative operation on the point cloud preliminary calibration matrixes of all the cameras to obtain the final point cloud calibration matrixes of all the cameras.
In this embodiment, the iterative operation uses an iterative closest point algorithm, whose mathematical form represents:
Figure BDA0002780952620000101
wherein the content of the first and second substances,Ap represents the point cloud a and P represents the point cloud a,Bq represents the point cloud B, and the fine registration is to convert the point cloud A to the coordinate system of the point cloud B so that error (T: (T))AP),BQ) is minimal. Thus, step S25 specifically includes the following steps:
respectively setting the point cloud A set and the point cloud B set of the two cameras as a set P and a set Q, and obtaining a matching relation M of the set P and the set Q as follows:
M＝match(P,Q)＝{(p,q):p∈P,q∈Q}
the error between set P and set Q is defined as:
error(P,Q)＝∑(p,q∈M)w(p,q)d(p,q)
w (P, Q) represents the weight of the corresponding point clouds in the set P and the set Q, and d (P, Q) represents the distance between the corresponding point clouds in the set P and the set Q;
performing iterative operation on the point cloud preliminary calibration matrix through an iterative calculation formula to obtain a point cloud final calibration matrix of all cameras, wherein the iterative calculation formula is as follows:
Figure BDA0002780952620000111
Figure BDA0002780952620000112
i and i +1 denote the number of iterations,
Figure BDA0002780952620000113
for iterative integration of transformations, TinitAnd preliminarily calibrating a matrix for the point cloud.
Thereby, the fine registration of the point clouds of the two cameras is completed.
Referring to fig. 1 to 16, a third embodiment of the present invention is:
based on the foregoing first embodiment, the step S3 specifically includes the following steps:
s31, calculating an overlapped point cloud area among the cameras according to the point cloud calibration matrix among the cameras, and removing the overlapped point cloud area of one of the cameras;
in this embodiment, step S31 specifically includes the following steps:
s311, the visual field of the camera is equal to a view cone, and an object in the visual field of the camera is cut by using a near plane and a far plane to form a viewing frustum which comprises a left plane, a right plane, a top plane, a bottom plane, a near plane and a far plane;
the view frustum including the above six truncated planes is shown in fig. 11, and the view frustum XZ is a screenshot in fig. 12.
S312, locating the camera at the origin of the world coordinate system, setting the viewing direction of the camera as the positive direction along the Z-axis, setting the view matrix of the camera as a unit matrix, and expressing the vertices as: v ═ 1 (x y z w ═ 1)TThe projection matrix M of 4 x 4 is represented as: m ═ Mij)；
S313, rotating the projection matrix M to obtain a vertexFor v '═ 1 (x' y 'z' w)TNamely:
Figure BDA0002780952620000121
where, is dot product, rowi＝(mi1 mi2 mi3 mi4) The ith line of data representing the projection matrix M is converted into homogeneous coordinates from the top point of a world coordinate system after the transformation;
s314, if the vertex v' is located in the frustum, the following first inequality should be satisfied:
-w'<x'<w'
-w^'<y^'<w^'
-w'<z'<w'
s315, obtaining an inequality corresponding to each interception plane according to the first inequality of the interception body and the position relation of the six interception planes in the interception body, and calculating a plane equation coefficient of each interception plane to obtain a plane equation expression of each interception plane;
wherein, it is assumed that x' satisfies the following inequality:
-w'<x'
from the above rotational equation:
-(v·row4)<(v·row1)
that is:
0<(v·row4)+(v·row1)
finally, the following can be obtained:
0<v·(row4+row1)
the plane equation for the left section plane can be expressed as:
x(m41+m11)+y(m42+m12)+z(m43+m13)+w(m44+m14)＝0
because w is 1, then
x(m41+m11)+y(m42+m12)+z(m43+m13)+m44+m14＝0
Wherein, the standard plane equation is:
ax+by+cz+d＝0
the coefficients of the plane equation for the left section plane are:
a＝m41+m11，b＝m42+m12，c＝m43+m13，d＝m44+m14
repeating the above steps, and obtaining a plane equation coefficient of each interception plane according to inequalities corresponding to different interception planes, as shown in table 1:
TABLE 1 planar equation results of the intercept planes
Figure BDA0002780952620000131
Therefore, the plane equation expression of six cut planes can be obtained.
S316, according to the plane equation expression of each interception plane corresponding to each camera, the position relation between the point cloud of the camera and each interception plane of other cameras is judged one by one, and the point clouds on the inner sides of all the interception planes in any one camera are used as the overlapped point cloud area of the cameras;
wherein, viewing the frustum from the cross section, the relationship between the point and the truncation plane can be obtained as shown in fig. 13. I.e. one plane may be measured from P and normal on the plane
Figure BDA0002780952620000141
And determining that any point P' on the plane satisfies the following conditions:
Figure BDA0002780952620000142
then, given a point Q arbitrarily, there is:
Figure BDA0002780952620000143
the point Q is arranged outside the cutting plane;
Figure BDA0002780952620000144
point Q is on the clipping plane;
Figure BDA0002780952620000145
point Q is inside the clipping plane;
thereby, an overlapping point cloud area of the two cameras is obtained.
And S317, removing the overlapped point cloud area of any one of the cameras comprising the overlapped point cloud area.
That is, fig. 14 is the point cloud of one of the cameras, and fig. 15 is obtained after the overlapping point cloud area is removed.
And S32, splicing the point clouds of all the cameras after the overlapped point cloud areas are removed according to the point cloud calibration matrix among the cameras to obtain the reconstructed three-dimensional scene.
The fourth embodiment of the invention is as follows:
another embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the multi-camera based three-dimensional scene reconstruction method according to any one of the first to third embodiments.
Referring to fig. 17, a fifth embodiment of the present invention is:
an electronic device 1 comprises a memory 3, a processor 2 and a computer program stored on the memory 3 and operable on the processor 2, wherein the processor 2 implements the multi-camera based three-dimensional scene reconstruction method according to any one of the first to third embodiments when executing the computer program.
In summary, the multi-camera-based three-dimensional scene reconstruction method, the storage medium and the electronic device provided by the invention adopt the existing chessboard searching algorithm to identify the checkerboards, and pair the corner point coordinates of the identified checkerboards according to the epipolar geometric constraint principle to obtain the preliminary position relationship calibration matrix among the multiple cameras; after the interesting region of the color image is obtained through the plurality of checkerboards, the interesting region of the point cloud of each camera can be obtained according to the position relation matrix of the color image and the depth image, the initial registration of the point cloud can be realized according to the initial position relation calibration matrix, and then the accurate registration is carried out through the iterative closest point algorithm, so that a more accurate final point cloud calibration matrix is obtained; and finally, according to a point cloud calibration matrix among the cameras, firstly removing the overlapped point cloud area, and then performing point cloud splicing on the cameras to obtain a reconstructed three-dimensional scene, namely, under the condition that the cameras are fixed, performing registration splicing on the point clouds of the cameras by a machine vision method to obtain the reconstructed three-dimensional scene, so that the reconstruction of the three-dimensional scene can be better and faster realized, and the method is time-saving, labor-saving and universal.
The above description is only an embodiment of the present invention, and not intended to limit the scope of the present invention, and all equivalent changes made by using the contents of the present specification and the drawings, or applied directly or indirectly to the related technical fields, are included in the scope of the present invention.

Claims (10)

1. The three-dimensional scene reconstruction method based on multiple cameras is characterized by comprising the following steps:
acquiring a preliminary position relation calibration matrix among a plurality of cameras;
acquiring color images and depth images which are respectively shot by the plurality of cameras for a scene to be reconstructed and provided with the plurality of checkerboards, and calibrating a matrix according to the color images and the depth images of the plurality of cameras and the preliminary position relation among the plurality of cameras to obtain a point cloud calibration matrix among the plurality of cameras;
and splicing the point clouds of the cameras according to the point cloud calibration matrix among the cameras to obtain the reconstructed three-dimensional scene.
2. The multi-camera based three-dimensional scene reconstruction method of claim 1, wherein the obtaining of the preliminary positional relationship calibration matrix between the plurality of cameras specifically comprises the steps of:
acquiring a first color image shot by each camera for one checkerboard to obtain corner point coordinates of the checkerboard in each first color image;
pairing each corner point coordinate of the first color images of different cameras according to an epipolar geometric constraint principle to obtain a corner point pairing relation among the cameras;
and obtaining a preliminary position relation calibration matrix among the cameras according to the corner pairing relation among the cameras.
3. The multi-camera based three-dimensional scene reconstruction method according to claim 1, wherein the obtaining of the color images and the depth images of the plurality of cameras respectively taken by the scene to be reconstructed with the checkerboards placed therein and the obtaining of the point cloud calibration matrix between the plurality of cameras according to the color images and the depth images of the plurality of cameras and the preliminary position relationship calibration matrix between the plurality of cameras specifically comprises the following steps:
acquiring second color images and depth images which are respectively shot by the cameras for a scene to be reconstructed and provided with a plurality of checkerboards;
finding each checkerboard in each second color image, and obtaining the center point coordinate of each checkerboard in each second color image to obtain the region of interest of each second color image of each camera;
obtaining an interested area of the point cloud of each camera according to a preset position relation matrix between the second color image and the depth image in each camera and the interested area of the second color image;
preliminarily registering the point clouds of all the cameras according to the preliminary position relation calibration matrixes among the cameras and the region of interest of the point cloud of each camera to obtain point cloud preliminary calibration matrixes of all the cameras;
and carrying out iterative operation on the point cloud preliminary calibration matrixes of all the cameras to obtain the point cloud final calibration matrixes of all the cameras.
4. The multi-camera based three-dimensional scene reconstruction method according to claim 3, wherein the finding of each checkerboard in each second color image and obtaining the center point coordinates of each checkerboard in each second color image to obtain the region of interest of each second color image of each camera specifically comprises the following steps:
searching each checkerboard in the second color image, and obtaining the corner point coordinates of the searched checkerboard to obtain the center point coordinates of the searched checkerboard;
masking the searched checkerboards, then continuously searching other checkerboards in the second color image and obtaining the center point coordinates of other checkerboards until obtaining the center point coordinates of each checkerboard in each second color image;
and carrying out polar angle sequencing on the center point coordinates of all the chessboard in each second color image, and then sequentially connecting the center point coordinates of the chessboard according to the polar angle sequencing result to obtain a closed area, namely the region of interest of each second color image of each camera.
5. The multi-camera based three-dimensional scene reconstruction method according to claim 3, wherein there are two cameras, and the step of performing iterative operation on the point cloud preliminary calibration matrices of all the cameras to obtain the point cloud final calibration matrices of all the cameras comprises the following steps:
respectively setting the point cloud A set and the point cloud B set of the two cameras as a set P and a set Q, and obtaining a matching relation M of the set P and the set Q as follows:
M＝match(P,Q)＝{(p,q):p∈P,q∈Q}
defining the error between the set P and the set Q as:
error(P,Q)＝∑(p,q∈M)w(p,q)d(p,q)
wherein w (P, Q) represents the weight of the corresponding point clouds in the set P and the set Q, and d (P, Q) represents the distance between the corresponding point clouds in the set P and the set Q;
performing iterative operation on the point cloud preliminary calibration matrix through an iterative calculation formula to obtain a point cloud final calibration matrix of all the cameras, wherein the iterative calculation formula is as follows:
Figure FDA0002780952610000031
Figure FDA0002780952610000032
the i and i +1 represent the number of iterations, the
Figure FDA0002780952610000033
For iterative integration of the transformations, the TinitAnd preliminarily calibrating a matrix for the point cloud.
6. The multi-camera based three-dimensional scene reconstruction method according to claim 1, wherein the step of splicing the point clouds of the plurality of cameras according to the point cloud calibration matrix among the plurality of cameras to obtain the reconstructed three-dimensional scene specifically comprises the following steps:
calculating an overlapped point cloud area among the cameras according to a point cloud calibration matrix among the cameras, and removing the overlapped point cloud area of one camera;
and splicing the point clouds of all the cameras after the overlapped point cloud areas are removed according to the point cloud calibration matrix among the cameras to obtain a reconstructed three-dimensional scene.
7. The multi-camera based three-dimensional scene reconstruction method of claim 6, wherein the calculating the overlapped point cloud regions between the plurality of cameras according to the point cloud calibration matrix between the plurality of cameras and removing the overlapped point cloud region of one of the cameras comprises the following steps:
the method comprises the steps that the visual field of a camera is equal to a view cone, and an object in the visual field of the camera is cut by using a near plane and a far plane to form a viewing frustum which comprises a left surface, a right surface, a top surface, a bottom surface, a near plane and a far plane;
the camera is positioned at the origin of a world coordinate system, the sight line direction of the camera is a positive direction along the Z axis, the view matrix of the camera is set as a unit matrix, and the vertex is expressed as: v ═ 1 (x y z w ═ 1)TThe projection matrix M of 4 x 4 is represented as: m ═ Mij)；
The vertex obtained by rotating the projection matrix M is v '═ 1 (x' y 'z' w)TNamely:
Figure FDA0002780952610000034
the · is a dot product, the rowi＝(mi1 mi2 mi3 mi4) The ith row of data representing the projection matrix M;
if vertex v' is located within the frustum, the following first inequality should be satisfied:
-w'<x'<w'
-w^'<y^'<w^'
-w'<z'<w'
obtaining an inequality corresponding to each interception plane according to the first inequality of the interception body and the position relation of the six interception planes in the interception body, so as to calculate a plane equation coefficient of each interception plane, and obtain a plane equation expression of each interception plane;
according to a plane equation expression of each interception plane corresponding to each camera, the position relation between the point cloud of the camera and each interception plane of other cameras is judged one by one, and the point cloud on the inner side of all the interception planes in any one camera is used as an overlapped point cloud area of the cameras;
removing the overlapping point cloud area of any one of the cameras that includes the overlapping point cloud area.
8. The multi-camera based three-dimensional scene reconstruction method of one of claims 1 to 7, wherein the plurality of chequers is four chequers.
9. A computer-readable storage medium having stored thereon a computer program, characterized in that: the computer program, when being executed by a processor, implements a method of reconstructing a three-dimensional scene as claimed in any one of claims 1 to 8.
10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the three-dimensional scene reconstruction method according to any one of claims 1 to 8 when executing the computer program.
CN202011281518.9A 2020-11-16 2020-11-16 Three-dimensional scene reconstruction method based on multiple cameras, storage medium and electronic equipment Active CN112381886B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011281518.9A CN112381886B (en) 2020-11-16 2020-11-16 Three-dimensional scene reconstruction method based on multiple cameras, storage medium and electronic equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011281518.9A CN112381886B (en) 2020-11-16 2020-11-16 Three-dimensional scene reconstruction method based on multiple cameras, storage medium and electronic equipment

Publications (2)

Publication Number Publication Date
CN112381886A true CN112381886A (en) 2021-02-19
CN112381886B CN112381886B (en) 2024-04-09

Family

ID=74584818

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011281518.9A Active CN112381886B (en) 2020-11-16 2020-11-16 Three-dimensional scene reconstruction method based on multiple cameras, storage medium and electronic equipment

Country Status (1)

Country Link
CN (1) CN112381886B (en)

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112927353A (en) * 2021-02-25 2021-06-08 电子科技大学 Three-dimensional scene reconstruction method based on two-dimensional target detection and model alignment, storage medium and terminal
CN113012238A (en) * 2021-04-09 2021-06-22 南京星顿医疗科技有限公司 Method for rapid calibration and data fusion of multi-depth camera
CN113052917A (en) * 2021-03-29 2021-06-29 黑芝麻智能科技(上海)有限公司 Method for acquiring image coordinates of invisible position of camera, calibration method and system
CN113362445A (en) * 2021-05-25 2021-09-07 上海奥视达智能科技有限公司 Method and device for reconstructing object based on point cloud data
CN113504385A (en) * 2021-06-30 2021-10-15 上海爱观视觉科技有限公司 Method and device for measuring speed of a plurality of cameras
CN113516694A (en) * 2021-04-13 2021-10-19 同济大学 Multi-Kinect-based dynamic visual control method for remote medical scene
CN114554176A (en) * 2022-01-24 2022-05-27 北京有竹居网络技术有限公司 Depth camera
CN117830439A (en) * 2024-03-05 2024-04-05 南昌虚拟现实研究院股份有限公司 Multi-camera system pose calibration method and device

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109064404A (en) * 2018-08-10 2018-12-21 西安电子科技大学 It is a kind of based on polyphaser calibration panorama mosaic method, panoramic mosaic system
CN109242913A (en) * 2018-09-07 2019-01-18 百度在线网络技术（北京）有限公司 Scaling method, device, equipment and the medium of collector relative parameter
CN110223377A (en) * 2019-05-28 2019-09-10 上海工程技术大学 One kind being based on stereo visual system high accuracy three-dimensional method for reconstructing
CN111127625A (en) * 2019-10-08 2020-05-08 新拓三维技术(深圳)有限公司 Foot scanning method, system and device
CN111150175A (en) * 2019-12-05 2020-05-15 新拓三维技术(深圳)有限公司 Method, device and system for three-dimensional scanning of feet

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109064404A (en) * 2018-08-10 2018-12-21 西安电子科技大学 It is a kind of based on polyphaser calibration panorama mosaic method, panoramic mosaic system
CN109242913A (en) * 2018-09-07 2019-01-18 百度在线网络技术（北京）有限公司 Scaling method, device, equipment and the medium of collector relative parameter
CN110223377A (en) * 2019-05-28 2019-09-10 上海工程技术大学 One kind being based on stereo visual system high accuracy three-dimensional method for reconstructing
CN111127625A (en) * 2019-10-08 2020-05-08 新拓三维技术(深圳)有限公司 Foot scanning method, system and device
CN111150175A (en) * 2019-12-05 2020-05-15 新拓三维技术(深圳)有限公司 Method, device and system for three-dimensional scanning of feet

Cited By (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112927353A (en) * 2021-02-25 2021-06-08 电子科技大学 Three-dimensional scene reconstruction method based on two-dimensional target detection and model alignment, storage medium and terminal
CN112927353B (en) * 2021-02-25 2023-05-19 电子科技大学 Three-dimensional scene reconstruction method, storage medium and terminal based on two-dimensional target detection and model alignment
CN113052917A (en) * 2021-03-29 2021-06-29 黑芝麻智能科技(上海)有限公司 Method for acquiring image coordinates of invisible position of camera, calibration method and system
CN113052917B (en) * 2021-03-29 2024-05-07 黑芝麻智能科技(上海)有限公司 Method for acquiring image coordinates of invisible position of camera, calibration method and system
CN113012238A (en) * 2021-04-09 2021-06-22 南京星顿医疗科技有限公司 Method for rapid calibration and data fusion of multi-depth camera
CN113012238B (en) * 2021-04-09 2024-04-16 南京星顿医疗科技有限公司 Method for quick calibration and data fusion of multi-depth camera
CN113516694A (en) * 2021-04-13 2021-10-19 同济大学 Multi-Kinect-based dynamic visual control method for remote medical scene
CN113362445A (en) * 2021-05-25 2021-09-07 上海奥视达智能科技有限公司 Method and device for reconstructing object based on point cloud data
CN113362445B (en) * 2021-05-25 2023-05-05 上海奥视达智能科技有限公司 Method and device for reconstructing object based on point cloud data
CN113504385B (en) * 2021-06-30 2023-07-14 安徽爱观视觉科技有限公司 Speed measuring method and device for plural cameras
CN113504385A (en) * 2021-06-30 2021-10-15 上海爱观视觉科技有限公司 Method and device for measuring speed of a plurality of cameras
CN114554176A (en) * 2022-01-24 2022-05-27 北京有竹居网络技术有限公司 Depth camera
CN117830439A (en) * 2024-03-05 2024-04-05 南昌虚拟现实研究院股份有限公司 Multi-camera system pose calibration method and device

Also Published As

Publication number Publication date
CN112381886B (en) 2024-04-09

Similar Documents

Publication Publication Date Title
CN112381886A (en) Multi-camera-based three-dimensional scene reconstruction method, storage medium and electronic device
CN111783820B (en) Image labeling method and device
CN106780619B (en) Human body size measuring method based on Kinect depth camera
US9117269B2 (en) Method for recognizing objects in a set of images recorded by one or more cameras
CN109389634A (en) Virtual shopping system based on three-dimensional reconstruction and augmented reality
CN107590444A (en) Detection method, device and the storage medium of static-obstacle thing
Li et al. Spherical stereo for the construction of immersive VR environment
JP2018197685A (en) Three-dimensional measurement device
CN112348890A (en) Space positioning method and device and computer readable storage medium
CN113793392A (en) Camera parameter calibration method and device
CN117372647A (en) Rapid construction method and system of three-dimensional model for building
Siddique et al. 3d object localization using 2d estimates for computer vision applications
CN109314774A (en) System and method for three-dimensional imaging
KR20210141922A (en) How to 3D Reconstruct an Object
CN111179271A (en) Object angle information labeling method based on retrieval matching and electronic equipment
Imre et al. Calibration of nodal and free-moving cameras in dynamic scenes for post-production
CN115953471A (en) Indoor scene multi-scale vector image retrieval and positioning method, system and medium
KR20160049639A (en) Stereoscopic image registration method based on a partial linear method
CN114511637A (en) Weak-feature object image three-dimensional reconstruction system and method based on strong feature construction
Hong et al. Robust 3D reconstruction and parameter measurement of the foot using multiple depth cameras
Sarkar et al. Feature-augmented Trained Models for 6DOF Object Recognition and Camera Calibration.
CN109872371A (en) A kind of monocular vision three-dimensional rebuilding method based on improvement Sift algorithm
CN111210476A (en) Method and device for simultaneously positioning and establishing image
Doria et al. Determination of camera position in soccer games
Zhang et al. Design of a 3D reconstruction model of multiplane images based on stereo vision

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant