CN112085794A - Space positioning method and three-dimensional reconstruction method applying same - Google Patents
Space positioning method and three-dimensional reconstruction method applying same Download PDFInfo
- Publication number
- CN112085794A CN112085794A CN202010956131.2A CN202010956131A CN112085794A CN 112085794 A CN112085794 A CN 112085794A CN 202010956131 A CN202010956131 A CN 202010956131A CN 112085794 A CN112085794 A CN 112085794A
- Authority
- CN
- China
- Prior art keywords
- image
- point
- point location
- matrix
- relative position
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 41
- 239000011159 matrix material Substances 0.000 claims abstract description 164
- 238000013527 convolutional neural network Methods 0.000 claims description 21
- 238000004364 calculation method Methods 0.000 claims description 19
- 238000012545 processing Methods 0.000 claims description 19
- 238000005457 optimization Methods 0.000 claims description 14
- 101100134058 Caenorhabditis elegans nth-1 gene Proteins 0.000 claims description 12
- 238000012549 training Methods 0.000 claims description 12
- 230000009466 transformation Effects 0.000 claims description 11
- 230000004807 localization Effects 0.000 claims description 8
- 238000010606 normalization Methods 0.000 claims description 7
- 239000013598 vector Substances 0.000 claims description 7
- 238000013507 mapping Methods 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 238000009616 inductively coupled plasma Methods 0.000 description 3
- 238000007781 pre-processing Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 230000010354 integration Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008094 contradictory effect Effects 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 238000009792 diffusion process Methods 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000004576 sand Substances 0.000 description 1
- 230000001629 suppression Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- General Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Mathematical Physics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Computer Graphics (AREA)
- Geometry (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a space positioning method and a three-dimensional reconstruction method applying the space positioning method, wherein the space positioning method comprises the steps of determining a first relative position relation matrix of a first point location and a second point location according to first IMU data of the first point location and second IMU data of the second point location; determining a second relative position relation matrix of a first point location and a second point location according to a feature point matching pair of a first image of the first point location and a second image of the second point location; determining the initial position of the shooting device at the second point location according to the first relative position relation matrix of the first point location and the second relative position relation matrix of the first point location and the second point location; and determining the final position of the shooting device at the second point location according to the initial position of the second point location and the matched pair of the feature points of the first image and the second image.
Description
Technical Field
The invention relates to the technical field of three-dimensional imaging, in particular to a space positioning method and a three-dimensional reconstruction method applying the space positioning method.
Background
In the prior art, an eight-point linear characteristic equation based on a feature matching point pair of two images needs to be sequentially constructed for each point location of a camera when the camera position is calculated based on an SFM algorithm, and an initial position matrix of the camera at a certain point location is solved through an epipolar geometry principle and SVD transformation, so that the calculation amount of the SFM algorithm when the initial position of the camera is calculated at each point location is large, and the problem of low calculation rate exists when the final position of the camera is calculated.
Disclosure of Invention
The invention mainly aims to provide a space positioning method and a three-dimensional reconstruction method applying the space positioning method, and aims to solve the technical problems that the calculation amount of an SFM algorithm in the prior art is large when the initial position of a camera is calculated at each point position, and the calculation speed is low when the final position of the camera is calculated.
In order to solve the technical problems, the technical scheme provided by the invention is as follows:
a method of spatial localization, the method comprising:
determining a first relative position relation matrix of a first point location and a second point location according to first IMU data of the first point location and second IMU data of the second point location;
determining a second relative position relation matrix of a first point location and a second point location according to a feature point matching pair of a first image of the first point location and a second image of the second point location;
the first point location and the second point location refer to different positions of a shooting device in the same scene;
determining the initial position of the shooting device at the second point location according to the first relative position relationship matrix of the first point location and the second relative position relationship matrix of the first point location and the second point location, and so on, and determining the initial position of the shooting device at the nth point location according to the first relative position relationship matrix of the nth-1 point location and the nth point location and the second relative position relationship matrix of the nth-1 point location and the nth point location;
and determining the final position of the shooting device at the second point location according to the initial position of the second point location and the matched pair of the feature points of the first image and the second image, and so on, and determining the final position of the shooting device at the nth point location according to the initial position of the nth point location and the matched pair of the feature points of the (n-1) th image and the nth image.
Preferably, the determining a first relative position relationship matrix of the first point location and the second point location according to the first IMU data of the first point location and the second IMU data of the second point location includes: acquiring first IMU data of a shooting device at a first point location and second IMU data of a second point location; determining a first position matrix of the first point under an IMU coordinate system according to the first IMU data; determining a second position matrix of the second point position under the IMU coordinate system according to the second IMU data; and determining a first relative position relation matrix of the first point location and the second point location under an IMU coordinate system according to the first position matrix and the second position matrix.
Preferably, the calculation formula of the first relative position relationship matrix is:
wherein Δ D represents a first relative position relationship matrix of the first point location and the second point location, DIMU1A first position matrix, D, representing said first point location in an IMU coordinate systemIMU2A second position matrix representing the second point location in an IMU coordinate system.
Preferably, the determining a second relative position relationship matrix of the first point location and the second point location according to the feature point matching pair of the first image of the first point location and the second image of the second point location includes: acquiring a first image shot by a shooting device at a first point and a second image shot by a second point; obtaining a feature point matching pair of the first image and the second image; determining an essential matrix of the shooting device according to the matched feature points in the feature point matching pair; and determining a second relative position relation matrix of the shooting device at the first point position and the second point position according to the intrinsic matrix.
Preferably, the calculation formula of the second relative position relationship matrix is:
wherein Δ P represents a second relative position relationship matrix of the first point location and the second point location, P1A first position matrix, P, for the camera at a first point2The second position matrix of the camera at the second point is assumed to be the unit matrix of the first position matrix of the camera at the first point, i.e. the default camera uses the first point as the origin of the world coordinate system.
Preferably, the obtaining of the matched pairs of feature points of the first image and the second image comprises: constructing a nonlinear scale space corresponding to each of the first image and the second image; calculating HESSIAN matrix values of each pixel point in the pyramid corresponding to the first image and the second image after scale normalization processing on the nonlinear scale space corresponding to each pixel point, and obtaining local extreme points corresponding to the first image and the second image; determining a dominant direction in which the first image and the second image match; determining feature points corresponding to the first image and the second image according to local extreme points corresponding to the first image and the second image and the main direction; and matching the characteristic points corresponding to the first image and the second image.
Preferably, the determining the feature points corresponding to the first image and the second image according to the local extreme points corresponding to the first image and the second image and the main direction includes: determining feature descriptors corresponding to the first image and the second image according to the local extreme points corresponding to the first image and the second image and the main direction; respectively carrying out data standardization processing on the feature descriptors corresponding to the first image and the second image; respectively calculating covariance matrixes of feature descriptors corresponding to the first image and the second image which are subjected to normalization processing through a covariance formula; respectively carrying out feature solution on covariance matrixes corresponding to the first image and the second image to obtain feature values and feature vectors corresponding to the first image and the second image; sorting the feature vectors corresponding to the first image and the second image according to the magnitude of the feature values, selecting the first m feature vectors as principal component data, and mapping the principal component data to a new matrix to obtain final feature descriptors corresponding to the first image and the second image, wherein m is a natural number; and determining the characteristic points corresponding to the first image and the second image according to the local extreme points and the final characteristic descriptors corresponding to the first image and the second image.
Preferably, the determining the initial position of the shooting device at the second point according to the first relative position relationship matrix of the first point location and the second relative position relationship matrix of the first point location and the second point location includes:
determining a calculation formula of a transformation matrix of the first relative position relation matrix and the second relative position relation matrix according to the first relative position relation matrix and the second relative position relation matrix;
ΔP＝K·ΔD·K-1(formula three)
Wherein K represents a transformation matrix of the first relative position relationship matrix and the second relative position relationship matrix, Δ D represents the first relative position relationship matrix, and Δ P represents the second relative position relationship matrix;
the calculation formula of the second relative position relation matrix is as follows:
wherein, DeltaP represents a second relative position relationship matrix of the first point location and the second point location, P1A first position matrix, P, for the camera at a first point2A second position matrix of the shooting device at a second point position is assumed, a first position matrix of the shooting device at a first point position is taken as a unit matrix, namely the default shooting device takes the first point position as an origin of a world coordinate system;
the formula three and the formula four are used for obtaining:
P2＝K·ΔD·K-1·P1(formula five)
An initial position matrix C of the shooting device at the nth point can be deduced through a formula VnComprises the following steps:
Pn·Cn＝ΔP·Pn-1＝[K·ΔDn·K-1]·Pn-1·Cn(formula six)
Wherein n is a natural number greater than 1, and m is a natural number greater than 0.
Preferably, the determining, according to the initial position of the second point location and the matched pair of feature points of the first image and the second image, the final position of the camera at the second point location includes: and transmitting the initial position of the second point location and the feature point matching pair of the first image and the second image as input conditions to a BA optimization model for optimization processing to obtain the final position of the shooting device at the second point location, and repeating the steps, and transmitting the initial position of the (n-1) th point location and the feature point matching pair of the (n-1) th image and the n-th image as input conditions to the BA optimization model for optimization processing to obtain the final position of the shooting device at the n-th point location.
The other technical scheme provided by the invention is as follows:
a method of three-dimensional reconstruction, the method comprising:
inputting the nth image of the nth point location into a convolutional neural network model, wherein the convolutional neural network model is obtained by training a plurality of sets of training data, and each set of training data in the plurality of sets of training data comprises: the system comprises a two-dimensional image and a three-dimensional image, wherein the two-dimensional image and the three-dimensional image are shot by a shooting device in the same scene;
acquiring point cloud information of an nth image output by the convolutional neural network model;
matching the point cloud information of the nth image with the final position of the shooting device at the nth point position to form initial dense point cloud information of the scene, wherein the final position of the shooting device at the nth point position is obtained by the space positioning method;
performing secondary matching on the initial dense point cloud information to form final dense point cloud information of a scene;
and generating a three-dimensional model of the scene according to the final dense point cloud information.
Compared with the prior art, the invention has the following beneficial effects:
determining a first relative position relation matrix of a first point location and a second point location through first IMU data of the first point location and second IMU data of the second point location, and determining a second relative position relation matrix of the first point location and the second point location through feature point matching pairs of a first image of the first point location and a second image of the second point location; and determining the initial position of the shooting device at the second point position through the first relative position relationship matrix of the first point position and the second relative position relationship matrix of the first point position and the second point position, and determining the initial position of the shooting device at the nth point position through the first relative position relationship matrix of the nth-1 point position and the nth point position and the second relative position relationship matrix of the nth-1 point position and the nth point position by analogy. Therefore, when the initial position of the shooting device is calculated, the initial position of the shooting device at the nth point can be determined only by obtaining the first relative position relation matrix of the nth-1 point location and the nth point location and the second relative position relation matrix of the nth-1 point location and the nth point location, so that the calculation amount of calculating the initial position of the shooting device at a certain point can be effectively reduced, and the calculation rate of the embodiment when the final position of the shooting device is calculated is remarkably improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the structures shown in the drawings without creative efforts.
FIG. 1 is a flow diagram of a spatial location method according to one embodiment of the present invention;
FIG. 2 is a flow diagram of a first relative positional relationship matrix according to one embodiment of the present invention;
FIG. 3 is a flow diagram of a second relative position relationship matrix according to one embodiment of the invention;
FIG. 4 is a flow diagram of a matching pair of feature points according to one embodiment of the invention;
fig. 5 is a flow chart of a three-dimensional reconstruction method according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that all the directional indicators (such as up, down, left, right, front, and rear … …) in the embodiment of the present invention are only used to explain the relative position relationship between the components, the movement situation, etc. in a specific posture (as shown in the drawing), and if the specific posture is changed, the directional indicator is changed accordingly.
In addition, the descriptions related to "first", "second", etc. in the present invention are for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In addition, "and/or" in the whole text includes three schemes, taking a and/or B as an example, including a technical scheme, and a technical scheme that a and B meet simultaneously; in addition, technical solutions between various embodiments may be combined with each other, but must be realized by a person skilled in the art, and when the technical solutions are contradictory or cannot be realized, such a combination should not be considered to exist, and is not within the protection scope of the present invention.
As shown in fig. 1, the present embodiment provides a spatial positioning method, which specifically includes the following steps:
s10, determining a first relative position relation matrix of the first point location and the second point location according to the first IMU data of the first point location and the second IMU data of the second point location.
And S20, determining a second relative position relation matrix of the first point location and the second point location according to the feature point matching pair of the first image of the first point location and the second image of the second point location.
The first point location and the second point location refer to different angles of the shooting device in the same scene.
And S30, determining the initial position of the shooting device at the second point according to the first relative position relationship matrix of the first point location and the second relative position relationship matrix of the first point location and the second point location, and so on, and determining the initial position of the shooting device at the nth point location according to the first relative position relationship matrix of the nth-1 point location and the nth point location and the second relative position relationship matrix of the nth-1 point location and the nth point location.
And S40, determining the final position of the shooting device at the second point according to the initial position of the second point and the matching pair of the characteristic points of the first image and the second image, and so on, and determining the final position of the shooting device at the nth point according to the initial position of the nth point and the matching pair of the characteristic points of the n-1 th image and the nth image.
In this embodiment, the initial position of the photographing device at the nth point is determined through the first relative position relationship matrix of the nth point and the second relative position relationship matrix of the nth point and the nth point. Therefore, when the initial position of the photographing device is calculated, the initial position of the photographing device at the nth point can be determined only by obtaining the first relative position relationship matrix of the nth-1 th point and the nth point and the second relative position relationship matrix of the nth-1 th point and the nth point, so that the calculation amount of calculating the initial position of the photographing device at a certain point can be effectively reduced, and the calculation rate of the embodiment when the final position of the photographing device is calculated is remarkably increased. As shown in fig. 2, in an embodiment, S10 specifically includes the following:
and S11, acquiring first IMU data of the shooting device at the first point and second IMU data of the shooting device at the second point.
First preprocessing is performed on the first IMU data and the second IMU data.
Specifically, first preprocessing is performed on the first IMU data and the second IMU data respectively through an EKF algorithm.
S12, a first location matrix of the first point in the IMU coordinate system is determined from the first IMU data.
Specifically, integration processing is performed on the first preprocessed IMU data, and a first position matrix of the first point under the IMU coordinate system is determined.
And S13, determining a second position matrix of the second point under the IMU coordinate system according to the second IMU data.
Specifically, the first preprocessed second IMU data is subjected to integration processing, and a second position matrix of the second point location in the IMU coordinate system is determined.
And S14, determining a first relative position relation matrix of the first point location and the second point location under the IMU coordinate system according to the first position matrix and the second position matrix.
The calculation formula of the first relative position relationship matrix is as follows:
where Δ D represents a first point location and a second point locationFirst relative positional relationship matrix of DIMU1Representing a first position matrix of a first point location in the IMU coordinate system, DIMU2A second location matrix representing a second point location in the IMU coordinate system.
In the prior art, the final position of the shooting device cannot be calculated in a white wall or single-color scene by a conventional SFM algorithm, and because the white wall or the single-color scene cannot realize feature point matching of an image, the initial position of the shooting device cannot be calculated by using epipolar geometry, so that the final position of the shooting device cannot be determined by the initial position of the shooting device. According to the embodiment, the initial position of the shooting device can be calculated in a white wall or single-color scene through the IMU data of each point, so that the final position of the shooting device can be determined through the initial position of the shooting device, and the applicability of the space positioning method is improved. As shown in fig. 3, in an embodiment, S20 specifically includes the following:
and S21, acquiring a first image shot by the shooting device at the first point and a second image shot by the shooting device at the second point.
Specifically, the photographing device in this embodiment is a fisheye lens, and therefore the first image and the second image need to be subjected to second preprocessing, respectively.
The specific content of the second pretreatment is as follows: because the image shot by the fisheye lens is distorted, the multi-frame images collected by the fisheye lens need to be corrected one by one, and then the corrected images are unfolded one by one. If the data collected by the fisheye lens is video stream data, the video stream data needs to be decoded one by one, and then the decoded image is processed.
And S22, obtaining the feature point matching pairs of the first image and the second image.
And S23, determining the essence matrix of the shooting device according to the matched feature points in the feature point matching pair.
Specifically, at least eight feature points (at least eight feature points with the largest feature matching degree are selected) obtained by matching the first image and the second image through an eight-point method form a linear equation, and an essential matrix of the shooting device is calculated through an epipolar geometry principle.
And S24, determining a second relative position relation matrix of the shooting device at the first point and the second point according to the essence matrix.
Specifically, the intrinsic matrix is subjected to transformation decomposition through an SVD algorithm, and a second relative position relation matrix of the shooting device at the first point location and the second point location is obtained.
The calculation formula of the second relative position relation matrix is as follows:
where Δ P represents a second relative position relationship matrix of the first point location and the second point location, P1A first position matrix, P, for the camera at a first point2The second position matrix of the camera at the second point is assumed to be the unit matrix of the first position matrix of the camera at the first point, i.e. the default camera uses the first point as the origin of the world coordinate system.
As shown in fig. 4, in an embodiment, S22 specifically includes the following:
and S221, constructing a nonlinear scale space corresponding to each of the first image and the second image.
In the embodiment, the nonlinear scale space corresponding to each of the first image and the second image is constructed by a nonlinear diffusion filter function and an AOS algorithm.
S222, calculating HESSIAN matrix values of each pixel point in the pyramid corresponding to the first image and the second image after scale normalization processing on the nonlinear scale space corresponding to each pixel point, and obtaining local extreme points corresponding to the first image and the second image.
In this embodiment, a non-maximum suppression algorithm is used to calculate a HESSIAN matrix value of each pixel point in the pyramid corresponding to each of the first image and the second image, which is subjected to scale normalization processing in the non-linear scale space corresponding to each pixel point.
S223, determining the main direction of the first image and the second image.
In this embodiment, the principal direction in which the first image and the second image match is determined by the AKAZE algorithm.
S224, determining the characteristic points corresponding to the first image and the second image according to the local extreme points corresponding to the first image and the second image and the main direction.
In one embodiment, S224 specifically includes the following:
and determining the feature descriptors corresponding to the first image and the second image according to the local extreme points corresponding to the first image and the second image and the main direction.
And respectively carrying out data standardization processing on the characteristic descriptors corresponding to the first image and the second image.
And respectively calculating covariance matrixes of the feature descriptors corresponding to the first image and the second image after the normalization processing through a covariance formula.
And respectively carrying out feature solution on the covariance matrixes corresponding to the first image and the second image to obtain feature values and feature vectors corresponding to the first image and the second image.
And sorting the eigenvectors corresponding to the first image and the second image according to the magnitude of the eigenvalue, selecting the first m eigenvectors as principal component data, and mapping the principal component data into a new matrix to obtain final characteristic descriptors corresponding to the first image and the second image, wherein m is a natural number. By selecting a proper number of principal components to replace the original feature descriptors, the data size of the embodiment in calculating the initial position of the shooting device can be effectively reduced, and the speed of the embodiment in calculating the final position of the shooting device is remarkably improved.
And determining the characteristic points corresponding to the first image and the second image according to the local extreme points corresponding to the first image and the second image and the final characteristic descriptors.
The AKAZE-PCA algorithm is adopted to match the first image and the second image, so that the number of effective matching pairs of the images can be increased, and the accuracy of the image matching area can be improved. The AKAZE-PCA algorithm can effectively eliminate the influence of data between the first image and the second image so as to reduce the redundancy, thereby improving the matching rate of the first image and the second image.
And S225, matching the characteristic points corresponding to the first image and the second image.
In this embodiment, feature points corresponding to the first image and the second image are matched by a KD-tree algorithm.
The KD-tree algorithm based on similarity constraint can constrain the matching conditions of the first image and the second image when feature point matching is carried out, so that the matching speed of the first image and the second image is improved.
Wherein the similarity constraint refers to: the similarity that should be possessed in the corresponding neighborhoods of the two points to be matched on the physical quantities (gray distribution, texture features, gray gradient change, contour, etc.).
Specifically, the matching result of the first image and the second image is corrected through a RANSAC algorithm to eliminate mismatching point pairs, so that the matching accuracy of the first image and the second image is improved.
In one embodiment, S30 specifically includes the following:
determining a calculation formula of a transformation matrix of the first relative position relation matrix and the second relative position relation matrix according to the first relative position relation matrix and the second relative position relation matrix;
ΔP＝K·ΔD·K-1(formula three)
Wherein K represents a transformation matrix of the first relative position relation matrix and the second relative position relation matrix, Δ D represents the first relative position relation matrix, and Δ P represents the second relative position relation matrix;
the calculation formula of the second relative position relation matrix is as follows:
where Δ P represents a second relative position relationship matrix of the first point location and the second point location, P1For taking a racketA first position matrix, P, of the camera at a first point2A second position matrix of the shooting device at a second point position is assumed, a first position matrix of the shooting device at a first point position is taken as a unit matrix, namely the default shooting device takes the first point position as an origin of a world coordinate system;
the formula three and the formula four are used for obtaining:
P2＝K·ΔD·K-1·P1(formula five)
An initial position matrix C of the shooting device at the nth point can be deduced through a formula VnComprises the following steps:
Pn·Cm＝ΔP·Pn-1＝[K·ΔDn·K-1]·Pn-1·Cm(formula six)
Wherein n is a natural number greater than 1, and m is a natural number greater than 0.
In this embodiment, the shooting device is an eight-eye camera, the eight-eye camera has an upper group of lenses and a lower group of lenses, each group of lenses has four fisheye lenses, and the four fisheye lenses respectively collect four corresponding images, so that a 360-degree panoramic image is spliced. Adding binocular vision constraint to the formula six to obtain an initial position matrix of the shooting device based on the binocular vision constraint at the nth point:
wherein, C1、C2Is a relative position matrix of the upper and lower groups of lenses.
BA optimization based on binocular vision constraint can effectively reduce the error of the embodiment when the initial position of the shooting device is calculated, so that the accuracy of the initial position of the shooting device is improved, and the accuracy of three-dimensional modeling is improved.
In one embodiment, S40 specifically includes the following:
and transmitting the initial position of the second point location and the feature point matching pair of the first image and the second image as input conditions to a BA optimization model for optimization processing to obtain the final position of the shooting device at the second point location, and repeating the steps, and transmitting the initial position of the (n-1) th point location and the feature point matching pair of the (n-1) th image and the n-th image as input conditions to the BA optimization model for optimization processing to obtain the final position of the shooting device at the n-th point location.
Specifically, the BA optimization model is constructed based on the LM algorithm.
The three-dimensional reconstruction technology has wide application in the aspects of digital cities, machine vision, cultural relic protection and the like. In the prior art, three-dimensional modeling is generally performed by using a SLAM algorithm, and an IMU + SLAM method is generally used to improve the positioning accuracy of a shooting device. However, according to the SLAM, data acquired by a shooting device is required to be real-time linear ordered data, so that all information in a scene cannot be acquired, and the problems of information loss and incompleteness exist in the subsequent three-dimensional modeling.
As shown in fig. 5, in an embodiment, a three-dimensional reconstruction method is provided, and the method specifically includes the following steps:
s100, inputting the nth image of the nth point into a convolutional neural network model, wherein the convolutional neural network model is obtained by training a plurality of groups of training data, and each group of training data in the plurality of groups of training data comprises: the image processing device comprises a two-dimensional image and a three-dimensional image, wherein the two-dimensional image and the three-dimensional image are shot by a shooting device in the same scene.
Wherein the three-dimensional image contains three-dimensional information.
S200, point cloud information of the nth image output by the convolutional neural network model is obtained.
And S300, matching the point cloud information of the nth image with the final position of the shooting device at the nth point, so as to form initial dense point cloud information of the scene, wherein the final position of the shooting device at the nth point is obtained by the space positioning method in any embodiment.
Specifically, after coordinate transformation is carried out on the point cloud information of the nth image, the point cloud information is placed in a coordinate system where the final position of the nth point location of the shooting device is located, so that pre-matching of the point cloud information and the final position of the shooting device is achieved, and therefore initial dense point cloud information of the scene is formed.
For example, if there is point cloud information of the first image, point cloud information of the second image, point cloud information of the third image, point cloud information of the nth image, the point cloud information of the first image is transformed into coordinates, and then the transformed point cloud information is placed in a coordinate system where a final position of the first point location of the photographing device is located; after coordinate transformation is carried out on the point cloud information of the second image, the point cloud information is placed into a coordinate system of a final position of the second point location of the shooting device; and after coordinate transformation is carried out on the point cloud information of the third image, the point cloud information of the nth image is placed under the coordinate system of the final position of the nth point, so that the point cloud information of each image and the final position of the shooting device at each point are pre-matched, and the initial dense point cloud information of the scene is formed.
S400, performing secondary matching on the initial dense point cloud information to form final dense point cloud information of the scene.
In the present embodiment, the initial dense point cloud is secondarily matched by the ICP algorithm. By carrying out secondary matching on the initial dense point cloud information, the point cloud information repeated by each image can be fitted into one point cloud information to realize the effect of removing the weight, thereby improving the precision of three-dimensional modeling.
For example, if there are point cloud information of the first image and point cloud information of the second image, where both the first image and the second image have the same apple, and if the apple in the first image corresponds to the first point cloud information and the apple in the second image corresponds to the second point cloud information, in order to avoid the occurrence of a duplication phenomenon, the first point cloud information in the first image and the second point cloud information in the second image are fitted into one point cloud information, so as to achieve the deduplication effect, thereby improving the precision of three-dimensional modeling.
And S500, generating a three-dimensional model of the scene according to the final dense point cloud information.
In this embodiment, a three-dimensional model of the scene is generated by combining the final dense point cloud information through a poisson algorithm.
Inputting the nth image of the nth point location into a convolutional neural network model to obtain point cloud information of the nth image output by the convolutional neural network model; and matching the point cloud information with the final position information of the camera to form initial dense point cloud information of the scene, and performing secondary matching on the initial dense point cloud through an ICP (inductively coupled plasma) algorithm to form final dense point cloud of the scene, so that the reconstructed three-dimensional model has the characteristics of strong robustness, complete three-dimensional model, high accuracy and the like. In the embodiment, the poisson algorithm is combined with the final dense point cloud information to generate the three-dimensional model of the scene, so that the accuracy of the reconstructed three-dimensional model can be further improved.
In this embodiment, S100 specifically includes the following contents:
the convolutional neural network model is a full convolutional neural network model, a residual error learning module ResNET is added to an input layer of the full convolutional neural network model, and initialization is carried out by using a pre-trained weight.
Specifically, the residual error between the depth image generated by the convolutional neural network model and the real depth image is calculated through a formula eight, the loss between the generated depth image and the real depth image is calculated, the loss function of the convolutional neural network model is calculated through a formula nine, the weight of the convolutional neural network model is optimized, the loss of the convolutional neural network is reduced to the maximum extent, the problems that the depth image generated by the convolutional neural network model is fuzzy, more object contour information is lost and the like are solved, and therefore the definition of the depth image generated by the convolutional neural network model is improved.
Edepth(p)＝||Dgt(p)-Dpred(p) | (formula eight)
Wherein E isloss(p) is the total loss function of the convolutional neural network model，Edepth(p) is the loss between the true depth image and the generated depth image,
for the square of the first derivative in x and y directions on the generated depth image, a is a regular smoothing termsAnd betasA weight coefficient intermediate to the two constraint terms, p being an image pixel, DgtIs the depth value of the true depth image, DpredAre depth values of the generated depth image.
The above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention, and all modifications and equivalents of the present invention, which are made by the contents of the present specification and the accompanying drawings, or directly/indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (10)
1. A method of spatial localization, the method comprising:
determining a first relative position relation matrix of a first point location and a second point location according to first IMU data of the first point location and second IMU data of the second point location;
determining a second relative position relation matrix of a first point location and a second point location according to a feature point matching pair of a first image of the first point location and a second image of the second point location;
the first point location and the second point location refer to different positions of the shooting device in the same scene;
determining the initial position of the shooting device at the second point location according to the first relative position relationship matrix of the first point location and the second relative position relationship matrix of the first point location and the second point location, and so on, and determining the initial position of the shooting device at the nth point location according to the first relative position relationship matrix of the nth-1 point location and the nth point location and the second relative position relationship matrix of the nth-1 point location and the nth point location;
and determining the final position of the shooting device at the second point location according to the initial position of the second point location and the matched pair of the feature points of the first image and the second image, and so on, and determining the final position of the shooting device at the nth point location according to the initial position of the nth point location and the matched pair of the feature points of the (n-1) th image and the nth image.
2. The spatial location method of claim 1, wherein determining the first relative positional relationship matrix for the first point location and the second point location from the first IMU data for the first point location and the second IMU data for the second point location comprises:
acquiring first IMU data of a shooting device at a first point location and second IMU data of a second point location;
determining a first position matrix of the first point under an IMU coordinate system according to the first IMU data;
determining a second position matrix of the second point position under the IMU coordinate system according to the second IMU data;
and determining a first relative position relation matrix of the first point location and the second point location under an IMU coordinate system according to the first position matrix and the second position matrix.
3. The spatial localization method according to claim 2, wherein the calculation formula of the first relative position relationship matrix is:
wherein Δ D represents a first relative position relationship matrix of the first point location and the second point location, DIMU1A first position matrix, D, representing said first point location in an IMU coordinate systemIMU2A second position matrix representing the second point location in an IMU coordinate system.
4. The spatial localization method according to claim 1, wherein determining the second relative position relationship matrix of the first point location and the second point location according to the feature point matching pair of the first image of the first point location and the second image of the second point location comprises:
acquiring a first image shot by a shooting device at a first point and a second image shot by a second point;
obtaining a feature point matching pair of the first image and the second image;
determining an essential matrix of the shooting device according to the matched feature points in the feature point matching pair;
and determining a second relative position relation matrix of the shooting device at the first point position and the second point position according to the intrinsic matrix.
5. The spatial localization method according to claim 4, wherein the second relative position relationship matrix is calculated by the following formula:
wherein Δ P represents a second relative position relationship matrix of the first point location and the second point location, P1A first position matrix, P, for the camera at a first point2The second position matrix of the camera at the second point is assumed to be the unit matrix of the first position matrix of the camera at the first point, i.e. the default camera uses the first point as the origin of the world coordinate system.
6. The spatial localization method according to claim 4, wherein the obtaining of the matched pairs of feature points of the first image and the second image comprises:
constructing a nonlinear scale space corresponding to each of the first image and the second image;
calculating HESSIAN matrix values of each pixel point in the pyramid corresponding to the first image and the second image after scale normalization processing on the nonlinear scale space corresponding to each pixel point, and obtaining local extreme points corresponding to the first image and the second image;
determining a dominant direction in which the first image and the second image match;
determining feature points corresponding to the first image and the second image according to local extreme points corresponding to the first image and the second image and the main direction;
and matching the characteristic points corresponding to the first image and the second image.
7. The spatial localization method according to claim 6, wherein the determining the feature points corresponding to the first image and the second image according to the local extreme points corresponding to the first image and the second image and the main direction comprises:
determining feature descriptors corresponding to the first image and the second image according to the local extreme points corresponding to the first image and the second image and the main direction;
respectively carrying out data standardization processing on the feature descriptors corresponding to the first image and the second image;
respectively calculating covariance matrixes of feature descriptors corresponding to the first image and the second image which are subjected to normalization processing through a covariance formula;
respectively carrying out feature solution on covariance matrixes corresponding to the first image and the second image to obtain feature values and feature vectors corresponding to the first image and the second image;
sorting the feature vectors corresponding to the first image and the second image according to the magnitude of the feature values, selecting the first m feature vectors as principal component data, and mapping the principal component data to a new matrix to obtain final feature descriptors corresponding to the first image and the second image, wherein m is a natural number;
and determining the characteristic points corresponding to the first image and the second image according to the local extreme points and the final characteristic descriptors corresponding to the first image and the second image.
8. The spatial positioning method according to claim 1, wherein the determining an initial position of the camera at the second point according to the first relative position relationship matrix of the first point and the second relative position relationship matrix of the first point and the second point comprises:
determining a calculation formula of a transformation matrix of the first relative position relation matrix and the second relative position relation matrix according to the first relative position relation matrix and the second relative position relation matrix;
ΔP＝K·ΔD·K-1(formula three)
Wherein K represents a transformation matrix of the first relative position relationship matrix and the second relative position relationship matrix, Δ D represents the first relative position relationship matrix, and Δ P represents the second relative position relationship matrix;
the calculation formula of the second relative position relation matrix is as follows:
wherein Δ P represents a second relative position relationship matrix of the first point location and the second point location, P1A first position matrix, P, for the camera at a first point2A second position matrix of the shooting device at a second point position is assumed, a first position matrix of the shooting device at a first point position is taken as a unit matrix, namely the default shooting device takes the first point position as an origin of a world coordinate system;
the formula three and the formula four are used for obtaining:
P2＝K·ΔD·K-1·P1(formula five)
An initial position matrix C of the shooting device at the nth point can be deduced through a formula VnComprises the following steps:
Pn·Cn＝ΔP·Pn-1＝[K·ΔDn·K-1]·Pn-1·Cn(formula six)
Wherein n is a natural number greater than 1, and m is a natural number greater than 0.
9. The spatial localization method according to claim 1, wherein the determining the final position of the camera at the second point location according to the initial position of the second point location and the matched pair of feature points of the first image and the second image comprises:
and transmitting the initial position of the second point location and the feature point matching pair of the first image and the second image as input conditions to a BA optimization model for optimization processing to obtain the final position of the shooting device at the second point location, and repeating the steps, and transmitting the initial position of the (n-1) th point location and the feature point matching pair of the (n-1) th image and the n-th image as input conditions to the BA optimization model for optimization processing to obtain the final position of the shooting device at the n-th point location.
10. A method of three-dimensional reconstruction, the method comprising:
inputting the nth image of the nth point location into a convolutional neural network model, wherein the convolutional neural network model is obtained by training a plurality of sets of training data, and each set of training data in the plurality of sets of training data comprises: the system comprises a two-dimensional image and a three-dimensional image, wherein the two-dimensional image and the three-dimensional image are shot by a shooting device in the same scene;
acquiring point cloud information of an nth image output by the convolutional neural network model;
matching the point cloud information of the nth image with the final position of the shooting device at the nth point location so as to form initial dense point cloud information of the scene, wherein the final position of the shooting device at the nth point location is obtained by the spatial positioning method of any one of claims 1 to 9;
performing secondary matching on the initial dense point cloud information to form final dense point cloud information of a scene;
and generating a three-dimensional model of the scene according to the final dense point cloud information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010956131.2A CN112085794B (en) | 2020-09-11 | 2020-09-11 | Space positioning method and three-dimensional reconstruction method applying same |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010956131.2A CN112085794B (en) | 2020-09-11 | 2020-09-11 | Space positioning method and three-dimensional reconstruction method applying same |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112085794A true CN112085794A (en) | 2020-12-15 |
| CN112085794B CN112085794B (en) | 2022-05-17 |
Family
ID=73736965
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010956131.2A Active CN112085794B (en) | 2020-09-11 | 2020-09-11 | Space positioning method and three-dimensional reconstruction method applying same |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112085794B (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101316368A (en) * | 2008-07-18 | 2008-12-03 | 西安电子科技大学 | Full view stabilizing method based on global characteristic point iteration |
| CN103345751A (en) * | 2013-07-02 | 2013-10-09 | 北京邮电大学 | Visual positioning method based on robust feature tracking |
| CN109974693A (en) * | 2019-01-31 | 2019-07-05 | 中国科学院深圳先进技术研究院 | Unmanned plane localization method, device, computer equipment and storage medium |
| CN110378995A (en) * | 2019-05-29 | 2019-10-25 | 中德（珠海）人工智能研究院有限公司 | A method of three-dimensional space modeling is carried out using projection feature |
| EP3644015A1 (en) * | 2018-10-26 | 2020-04-29 | Fujitsu Limited | Position estimation system and position estimation method |
| CN111161336A (en) * | 2019-12-18 | 2020-05-15 | 北京城市网邻信息技术有限公司 | Three-dimensional reconstruction method, three-dimensional reconstruction apparatus, and computer-readable storage medium |
| CN111260726A (en) * | 2020-02-07 | 2020-06-09 | 北京三快在线科技有限公司 | Visual positioning method and device |
| CN111583136A (en) * | 2020-04-25 | 2020-08-25 | 华南理工大学 | Method for simultaneously positioning and establishing image of autonomous mobile platform in rescue scene |
| CN111595332A (en) * | 2020-04-13 | 2020-08-28 | 宁波深寻信息科技有限公司 | Full-environment positioning method integrating inertial technology and visual modeling |
-
2020
- 2020-09-11 CN CN202010956131.2A patent/CN112085794B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101316368A (en) * | 2008-07-18 | 2008-12-03 | 西安电子科技大学 | Full view stabilizing method based on global characteristic point iteration |
| CN103345751A (en) * | 2013-07-02 | 2013-10-09 | 北京邮电大学 | Visual positioning method based on robust feature tracking |
| EP3644015A1 (en) * | 2018-10-26 | 2020-04-29 | Fujitsu Limited | Position estimation system and position estimation method |
| CN109974693A (en) * | 2019-01-31 | 2019-07-05 | 中国科学院深圳先进技术研究院 | Unmanned plane localization method, device, computer equipment and storage medium |
| CN110378995A (en) * | 2019-05-29 | 2019-10-25 | 中德（珠海）人工智能研究院有限公司 | A method of three-dimensional space modeling is carried out using projection feature |
| CN111161336A (en) * | 2019-12-18 | 2020-05-15 | 北京城市网邻信息技术有限公司 | Three-dimensional reconstruction method, three-dimensional reconstruction apparatus, and computer-readable storage medium |
| CN111260726A (en) * | 2020-02-07 | 2020-06-09 | 北京三快在线科技有限公司 | Visual positioning method and device |
| CN111595332A (en) * | 2020-04-13 | 2020-08-28 | 宁波深寻信息科技有限公司 | Full-environment positioning method integrating inertial technology and visual modeling |
| CN111583136A (en) * | 2020-04-25 | 2020-08-25 | 华南理工大学 | Method for simultaneously positioning and establishing image of autonomous mobile platform in rescue scene |
Non-Patent Citations (1)
| Title |
|---|
| 闫吉星: "《建筑物LiDAR点云三维重建的全局优化方法》", 《中国优秀博硕士学位论文全文数据库(博士)信息科技辑》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112085794B (en) | 2022-05-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112613609B (en) | Nerve radiation field enhancement method based on joint pose optimization | |
| CN110503680B (en) | Unsupervised convolutional neural network-based monocular scene depth estimation method | |
| CN107564061B (en) | Binocular vision mileage calculation method based on image gradient joint optimization | |
| CN112330729B (en) | Image depth prediction method, device, terminal equipment and readable storage medium | |
| CN112132023A (en) | Crowd counting method based on multi-scale context enhanced network | |
| CN107767339B (en) | Binocular stereo image splicing method | |
| CN108280858B (en) | Linear global camera motion parameter estimation method in multi-view reconstruction | |
| CN113221647B (en) | 6D pose estimation method fusing point cloud local features | |
| CN108171249B (en) | RGBD data-based local descriptor learning method | |
| CN113313732A (en) | Forward-looking scene depth estimation method based on self-supervision learning | |
| CN112767467B (en) | Double-image depth estimation method based on self-supervision deep learning | |
| CN106023147B (en) | The method and device of DSM in a kind of rapidly extracting linear array remote sensing image based on GPU | |
| CN111462274A (en) | Human body image synthesis method and system based on SMP L model | |
| CN112750198A (en) | Dense correspondence prediction method based on non-rigid point cloud | |
| CN114049434A (en) | 3D modeling method and system based on full convolution neural network | |
| CN115205463A (en) | New visual angle image generation method, device and equipment based on multi-spherical scene expression | |
| CN116385660A (en) | Indoor single view scene semantic reconstruction method and system | |
| CN117237431A (en) | Training method and device of depth estimation model, electronic equipment and storage medium | |
| CN108830890B (en) | Method for estimating scene geometric information from single image by using generative countermeasure network | |
| CN113256692B (en) | Rigid body registration method and device based on self-adaptive neighborhood weight learning | |
| CN110580726A (en) | Dynamic convolution network-based face sketch generation model and method in natural scene | |
| CN109934283A (en) | A kind of adaptive motion object detection method merging CNN and SIFT light stream | |
| CN111612887B (en) | Human body measuring method and device | |
| CN115761116B (en) | Three-dimensional face reconstruction method based on perspective projection of monocular camera | |
| CN112085794B (en) | Space positioning method and three-dimensional reconstruction method applying same |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |