CN115841554A

CN115841554A - Three-dimensional reconstruction method, device, equipment and storage medium

Info

Publication number: CN115841554A
Application number: CN202211617187.0A
Authority: CN
Inventors: 黄宏疆; 叶培楚
Original assignee: Guangzhou Xaircraft Technology Co Ltd
Current assignee: Guangzhou Xaircraft Technology Co Ltd
Priority date: 2022-12-15
Filing date: 2022-12-15
Publication date: 2023-03-24

Abstract

The application discloses a three-dimensional reconstruction method, a three-dimensional reconstruction device, three-dimensional reconstruction equipment and a storage medium, and relates to the technical field of three-dimensional reconstruction. The technical scheme provided by the application comprises the following steps: constructing a first three-dimensional map through a global reconstruction algorithm based on an image sequence, and determining a first lost frame in the image sequence; optimizing the first three-dimensional map by an incremental reconstruction algorithm based on the first lost frame to obtain a second three-dimensional map, and determining a second lost frame in the image sequence; and under the condition that the second three-dimensional map does not meet the preset complete condition, constructing a local three-dimensional map corresponding to the second lost frame through a hole filling algorithm, and filling the local three-dimensional map into the second three-dimensional map to obtain a target three-dimensional map. By the technical means, a highly complete three-dimensional map can be constructed, the reconstruction success rate and robustness of three-dimensional reconstruction are effectively improved, and the problem of poor robustness of three-dimensional reconstruction in the prior art is solved.

Description

Three-dimensional reconstruction method, device, equipment and storage medium

Technical Field

The present application relates to the field of three-dimensional reconstruction technologies, and in particular, to a three-dimensional reconstruction method, apparatus, device, and storage medium.

Background

SfM (Structure from motion) is a very popular three-dimensional reconstruction algorithm in recent years, which aims to capture a series of images of the same scene, recover the camera position and orientation corresponding to the images, and construct a three-dimensional point cloud of the scene.

In the prior art, a global rotation matrix of an image is generally determined, a global translation vector of the image is determined according to the global rotation matrix, and the image is triangulated according to the global rotation matrix and the global translation vector to construct a three-dimensional point cloud of the image. The calculation of the global translation vector depends on the output of the global rotation matrix, i.e. the accuracy of the global rotation matrix determines the accuracy of the three-dimensional point cloud. However, the global rotation matrix is easily interfered by outliers, so that the calculation result is not accurate enough, the reconstruction failure rate is high, and the algorithm robustness is poor.

Disclosure of Invention

The application provides a three-dimensional reconstruction method, a three-dimensional reconstruction device, a three-dimensional reconstruction equipment and a storage medium, an initial three-dimensional map is quickly constructed through a global reconstruction algorithm, the shooting positions and postures of all lost frames are sequentially determined through an incremental reconstruction algorithm, the initial three-dimensional map is optimized, and finally a hole filling strategy is used for filling a local three-dimensional map in a cavity area, so that a highly-integrated three-dimensional map is obtained, the reconstruction success rate and robustness of three-dimensional reconstruction are effectively improved, and the problem that the robustness of three-dimensional reconstruction in the prior art is poor is solved.

In a first aspect, the present application provides a three-dimensional reconstruction method, including:

constructing a first three-dimensional map through a global reconstruction algorithm based on an image sequence, and determining a first lost frame in the image sequence;

optimizing the first three-dimensional map by an incremental reconstruction algorithm based on the first lost frame to obtain a second three-dimensional map, and determining a second lost frame in the image sequence;

and under the condition that the second three-dimensional map does not meet the preset complete condition, constructing a local three-dimensional map corresponding to the second lost frame through a hole filling algorithm, and filling the local three-dimensional map into the second three-dimensional map to obtain a target three-dimensional map.

In a second aspect, the present application provides a three-dimensional reconstruction apparatus, comprising:

the global reconstruction module is configured to construct a first three-dimensional map through a global reconstruction algorithm based on an image sequence and determine a first lost frame in the image sequence;

the incremental reconstruction module is configured to optimize the first three-dimensional map through an incremental reconstruction algorithm based on the first lost frame to obtain a second three-dimensional map, and determine a second lost frame in the image sequence;

and the hole filling reconstruction module is configured to construct a local three-dimensional map corresponding to the second lost frame through a hole filling algorithm under the condition that the second three-dimensional map is determined not to meet the preset complete condition, and fill the local three-dimensional map into the second three-dimensional map to obtain a target three-dimensional map.

In a third aspect, the present application provides a three-dimensional reconstruction apparatus, comprising:

one or more processors; a storage device storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement the three-dimensional reconstruction method of the first aspect.

In a fourth aspect, the present application provides a storage medium containing computer executable instructions for performing the three-dimensional reconstruction method according to the first aspect when executed by a computer processor.

In the method, a first three-dimensional map is constructed through a global reconstruction algorithm based on an image sequence, and a first lost frame in the image sequence is determined; optimizing the first three-dimensional map through an incremental reconstruction algorithm based on the first lost frame to obtain a second three-dimensional map, and determining a second lost frame in the image sequence; and under the condition that the second three-dimensional map does not meet the preset complete condition, constructing a local three-dimensional map corresponding to the second lost frame through a hole filling algorithm, and filling the local three-dimensional map into the second three-dimensional map to obtain the target three-dimensional map. By the technical means, the global reconstruction algorithm can be used as initialization operation to quickly construct a three-dimensional map and determine the shooting pose of an image frame. However, the accuracy of the three-dimensional map constructed by the global reconstruction algorithm is low, the shooting pose of a lost frame in an image sequence can be further gradually restored by the incremental reconstruction algorithm, and the three-dimensional map and the shooting pose are jointly optimized every time the shooting pose of a lost frame is restored, so that the accuracy of the three-dimensional map is improved. If the image frame contains an area with serious shielding or complex texture and is limited by the constraints of image resolution and the number of the feature points, and the image frame does not recover the shooting pose after the incremental reconstruction algorithm, the image frame can be up-sampled by a hole filling algorithm and then the feature points are extracted to increase the number of the feature points of the image frame and construct a local three-dimensional map corresponding to the image frame, the local three-dimensional map can be filled into the three-dimensional map constructed by the incremental reconstruction algorithm to obtain a highly complete three-dimensional map, and the reconstruction power and robustness of three-dimensional reconstruction are effectively improved.

Drawings

Fig. 1 is a flowchart of a three-dimensional reconstruction method provided in an embodiment of the present application;

fig. 2 is a flowchart of constructing a first three-dimensional map by using a global SfM algorithm provided in an embodiment of the present application;

FIG. 3 is a flow chart of screening target feature matching pairs provided by an embodiment of the present application;

FIG. 4 is a flow chart of determining relative transformation parameters of adjacent image frames according to an embodiment of the present application;

FIG. 5 is a flowchart of determining a second shooting pose of an image frame according to an embodiment of the present application;

fig. 6 is a flowchart for determining a first shooting pose of a third image frame according to an embodiment of the present application;

FIG. 7 is a flow chart for constructing a first three-dimensional map according to an embodiment of the present disclosure;

FIG. 8 is a flow chart for generating a second three-dimensional map provided by an embodiment of the present application;

FIG. 9 is a flowchart of generating a target three-dimensional map based on a hole filling algorithm according to an embodiment of the present application;

fig. 10 is a schematic structural diagram of a three-dimensional reconstruction apparatus provided in an embodiment of the present application;

fig. 11 is a schematic structural diagram of a three-dimensional reconstruction apparatus according to an embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, specific embodiments of the present application will be described in detail with reference to the accompanying drawings. It is to be understood that the specific embodiments described herein are merely illustrative of the application and are not limiting of the application. It should be further noted that, for the convenience of description, only some but not all of the relevant portions of the present application are shown in the drawings. Before discussing exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart may describe the operations (or steps) as a sequential process, many of the operations can be performed in parallel, concurrently or simultaneously. In addition, the order of the operations may be re-arranged. A process may be terminated when its operations are completed, but may have additional steps not included in the figure. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

The terms first, second and the like in the description and in the claims of the present application are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that embodiments of the application may be practiced in sequences other than those illustrated or described herein, and that the terms "first," "second," and the like are generally used herein in a generic sense and do not limit the number of terms, e.g., the first term can be one or more than one. In addition, "and/or" in the specification and claims means at least one of connected objects, a character "/" generally means that a preceding and succeeding related objects are in an "or" relationship.

The three-dimensional reconstruction method provided in this embodiment may be performed by a three-dimensional reconstruction device, where the three-dimensional reconstruction device may be implemented in a software and/or hardware manner, and the three-dimensional reconstruction device may be formed by two or more physical entities, or may be formed by one physical entity. For example, the three-dimensional reconstruction device can be an unmanned device for performing aerial surveying and mapping operations on a region to be surveyed; the system can also be a server in wireless connection with the unmanned equipment, the unmanned equipment sends the image sequence of the area to be drawn to the server, and the server constructs a three-dimensional map of the area to be drawn according to the image sequence. The unmanned equipment refers to equipment such as an unmanned aerial vehicle and the like which can be automatically executed based on preset tasks.

The three-dimensional reconstruction device is provided with at least one type of operating system, the three-dimensional reconstruction device can be provided with at least one application program based on the operating system, and the application program can be an application program carried by the operating system or an application program downloaded from a third-party device or a server. In this embodiment, the three-dimensional reconstruction apparatus has at least an application program that can execute the three-dimensional reconstruction method.

For convenience of understanding, the present embodiment is described by taking a server as an example of a main body for performing the three-dimensional reconstruction method.

In one embodiment, when the unmanned equipment navigates according to a pre-planned flight route, the image sequence of the area to be mapped is shot according to a preset shooting strategy, and the image sequence is sent to the server. The server determines the shooting pose of each image frame through a global SfM algorithm based on each image frame in the image sequence, constructs a three-dimensional map of an area to be drawn, and then performs global BA (Beam Adjustment) optimization on the three-dimensional map. The global SfM algorithm firstly determines a global rotation matrix of the image frame, then determines a global translation vector of the image frame according to the global rotation matrix, and triangulates the image frame according to the global rotation matrix and the global translation vector to construct a three-dimensional point cloud of the image. The calculation of the global translation vector depends on the output of the global rotation matrix, i.e. the accuracy of the global rotation matrix determines the accuracy of the three-dimensional point cloud. However, the global rotation matrix is easily interfered by outliers, so that the calculation result is not accurate enough, and the error cannot be eliminated even through global BA optimization, so that the reconstruction failure rate of the global SfM algorithm is high, and the robustness is poor.

In order to solve the problem that the global SfM algorithm in the prior art is poor in robustness, the embodiment provides a three-dimensional reconstruction method.

Fig. 1 is a flowchart of a three-dimensional reconstruction method according to an embodiment of the present disclosure. Referring to fig. 1, the three-dimensional reconstruction method specifically includes:

s110, constructing a first three-dimensional map through a global reconstruction algorithm based on the image sequence, and determining a first lost frame in the image sequence.

In this embodiment, the global reconstruction algorithm is a global SfM algorithm, and the first three-dimensional map is a three-dimensional map of the to-be-drawn area, which is constructed by the global SfM algorithm based on the image sequence of the to-be-drawn area. Illustratively, the unmanned equipment shoots an area to be painted according to a pre-planned flight route, sorts all shot image frames according to shooting time stamps to generate an image sequence, and sends the image sequence to the server. The server takes the global SfM algorithm as initialization operation, and a first three-dimensional map is quickly reconstructed based on each image frame in the image sequence.

In this embodiment, fig. 2 is a flowchart of constructing a first three-dimensional map by using a global SfM algorithm provided in this embodiment. As shown in fig. 2, the step of constructing the first three-dimensional map by the global SfM algorithm specifically includes S1101-S1104:

s1101, extracting feature points of each image frame in the image sequence, and determining a target feature matching pair of adjacent image frames.

Illustratively, assume that the image sequence is { I } ₁ ,I ₂ ,...I _i ,…,I _N }，I _i Is the ith image frame in the image sequence and N is the total number of image frames in the image sequence. From the image frame I by SIFT (Scale-invariant feature transform) feature extraction algorithm _i Extracts the image frame I _i Characteristic point of

For from the image frame I _i The j-th extracted feature point is extracted, and M is an image frame I _i Chinese character' ZhongteTotal number of feature points. Image frame I _i And image frame I _i+1 Performing feature matching to determine image frame I _i And image frame I _i+1 The feature matching pair between is

Finger image frame I _i The nth feature point in (1) and the image frame I _i+1 Is a set of feature matching pairs.

In the present embodiment, the target feature matching pair refers to a feature matching pair that is subsequently used to estimate a relative transformation parameter between adjacent image frames. Since the situation of mismatching may occur when feature matching is performed on adjacent image frames, feature matching pairs generated by matching can be screened to obtain matched feature matching pairs, introduction of outliers into a subsequent pose estimation process is avoided, and the three-dimensional reconstruction accuracy is improved. Illustratively, fig. 3 is a flowchart for screening target feature matching pairs provided in an embodiment of the present application. As shown in fig. 3, the step of screening the target feature matching pairs specifically includes S11011 to S11012:

s11011, matching the feature points of the adjacent image frames to determine a first feature matching pair.

S11012, rejecting the mismatching pairs in the first feature matching pairs through a mismatching rejection algorithm to obtain target feature matching pairs.

Illustratively, assume that image frame I is _i And image frame I _i+1 After feature matching, an image frame I is determined _i And image frame I _i+1 The first characteristic matching pair of

Image frame I constructed based on first feature matching pairs _i And image frame I _i+1 And screening mismatching pairs in the first feature matching pairs by using an ACRANSAC (A Contrario RANSAC) algorithm based on the epipolar constraint graph. Assume a mismatching pair is->

Will->

Removing the first feature matching pair to obtain a target feature matching pair

And S1102, determining relative transformation parameters of the adjacent image frames according to the target feature matching pairs of the adjacent image frames.

Illustratively, fig. 4 is a flowchart for determining relative transformation parameters of adjacent image frames according to an embodiment of the present application. As shown in fig. 4, the step of determining the relative transformation parameters of the adjacent image frames specifically includes S11021-S11022:

s11021, constructing a triple of the three continuous image frames based on the target feature matching pairs and the feature points of the three continuous image frames in the image sequence, wherein the three feature points in the triple are matched with each other.

Illustratively, from image frame I _i And image frame I _i+1 Selecting image frame I from target characteristic matching pairs _i First feature point of (2), image frame I _i And the image frame I _i+2 All the feature points in the image are matched to determine the image frame I _i+2 The third characteristic point matched with the first characteristic point is selected, and the first characteristic point is corresponding to the image frame I _i+1 The second feature point matched in the image frame I and the corresponding third feature point form the image frame I _i Image frame I _i+1 And image frame I _i+2 Such that three feature points in the triplet are matched with each other. For example, assume image frame I _i And image frame I _i+1 The target feature matching pair comprises

Then the first characteristic point comprising is acquired>

If the first characteristic point>

And image frame I _i+2 Is characterized by a characteristic point->

Match, will >>

As a first characteristic point->

And the first feature point is acquired from the target feature matching pair>

Corresponding second characteristic point->

Further obtain the triple

And S11022, determining the relative transformation parameters of the corresponding adjacent image frames according to the triples of the three continuous image frames under the condition that the number of the triples of the three continuous image frames is greater than or equal to a preset number threshold.

In this embodiment, the preset number threshold may be understood as the minimum number of corresponding triplets when three consecutive image frames match two by two. Illustratively, when image frame I _i Image frame I _i+1 And image frame I _i+2 Is greater than or equal to a preset number threshold value, the image frame I is indicated _i Image frame I _i+1 And image frame I _i+2 The matching constraint between the three groups is larger, and the accuracy of the relative transformation parameters determined based on the three groups is higher. At this time, the image frame I can be determined according to the first characteristic point and the corresponding second characteristic point in the triple _i And imageFrame I _i+1 Relative transformation parameters between the first characteristic points and the second characteristic points are determined according to the second characteristic points and the corresponding third characteristic points in the triplets to determine the image frame I _i+1 And image frame I _i+2 Relative transformation parameters between them for subsequent processing of the image frame I _i And image frame I _i+1 The first shooting pose and the relative transformation parameter are restored to an image frame I _i+2 The first shooting pose.

In this embodiment, the image frame I is determined _i And image frame I _i+1 Relative transformation parameter between and image frame I _i+1 And image frame I _i+2 After the relative transformation parameters are obtained, the corresponding rotation matrix is separated from the two relative transformation parameters, and the image frame I is determined according to the rotation matrix _i And an image frame I _i+1 Angle of rotation therebetween, and image frame I _i+1 And an image frame I _i+2 The angle of rotation therebetween. Because the rotating angle between the adjacent image frames is not too large, if the sum of the two rotating angles exceeds 2 degrees, the currently calculated relative transformation parameters are not accurate enough, and the image frame I cannot be accurately restored based on the relative transformation parameters subsequently _i+2 The first shooting pose. At this time, image frame I can be displayed _i+2 Marking as the first lost frame in the image sequence and obtaining an image frame I from the image sequence _i+3 To construct an image frame I _i Image frame I _i+1 And image frame I _i+3 The triplet of (2).

When the image frame I _i Image frame I _i+1 And image frame I _i+2 Is greater than or equal to a preset number threshold value, the image frame I is indicated _i Image frame I _i+1 And image frame I _i+2 The matching constraint between the image frames is small, the error of the relative transformation parameter determined based on the triple is large, and the image frame I cannot be accurately restored based on the relative transformation parameter subsequently _i+2 The first shooting pose. At this time, image frame I can be displayed _i+2 Marking as the first lost frame in the image sequence and obtaining an image frame I from the image sequence _i+3 To construct an image frame I _i Image frame I _i+1 And image frame I _i+3 From the triplet, further according to the tripletGroup determination image frame I _i And image frame I _i+1 Relative transformation parameters and image frame I _i+1 And image frame I _i+3 Relative transformation parameters of (1).

It should be noted that, the adjacent image frame or the consecutive image frame mentioned in this embodiment does not necessarily refer to the sequentially adjacent or consecutive image frame in the image sequence, but refers to the sequentially adjacent or consecutive image frame in the plurality of image frames remaining after the lost frame is removed from the image sequence. For example, image frame I _i+2 As the first lost frame, image frame I _i+1 And image frame I _i+3 Becoming an adjacent image frame.

And S1103, determining a second shooting pose of each image frame according to the relative transformation parameters and the first shooting poses of two adjacent image frames.

In the present embodiment, the first photographing pose of two adjacent image frames can be understood as a predetermined photographing pose of two adjacent image frames in the image sequence. Illustratively, shooting postures of two adjacent shooting points on a flight path are preset, and when the unmanned equipment navigates to the two shooting points, two adjacent image frames are shot in the corresponding set shooting postures. The unmanned equipment can determine a first shooting pose of a corresponding image frame according to the position of the shooting point and the corresponding shooting pose, and sends the image frame and the corresponding first shooting pose to the server in a correlation mode, so that the first shooting pose of two adjacent image frames in an image sequence acquired by the server is determined in advance by the unmanned equipment. For example, if the shooting postures of the first shooting point and the second shooting point on the flight line are preset, the image frame I is shot according to the shooting posture set by the first shooting point when the unmanned equipment sails to the first shooting point ₁ When the unmanned equipment sails to a second shooting point, the image frame I is shot according to the shooting posture set by the second shooting point ₂ . The unmanned equipment determines an image frame I according to the position and the shooting posture of the first shooting point ₁ And taking the image frame I ₁ And saving the corresponding first shooting pose in association. The unmanned equipment determines an image frame I according to the position and the shooting posture of the second shooting point ₂ And the first shooting pose of (1), and image frame I ₂ And storing the corresponding first shooting pose in an associated mode. After the unmanned equipment sends the image sequence to the server, the server acquires the image sequence and the image frame I ₁ And an image frame I ₂ The server can be according to the image frame I ₁ And image frame I ₂ And determining the first shooting pose of each image frame by the essence matrix between each adjacent image frame in the image sequence.

It should be noted that, the two adjacent image frames in the image sequence for predetermining the first shooting pose may be any two adjacent image frames, and in this embodiment, the two adjacent image frames for predetermining the first shooting pose are used as the image frame I ₁ And image frame I ₂ The description is made for the sake of example.

In this embodiment, the global SfM algorithm recovers the first shooting pose of each image frame based on the target feature matching of adjacent image frames, and performs global joint optimization on the first shooting pose corresponding to each image frame to obtain a second shooting pose of the image frame with higher precision. Illustratively, fig. 5 is a flowchart for determining a second shooting pose of an image frame according to an embodiment of the present application. As shown in fig. 5, the step of determining the second shooting pose of the image frame specifically includes S11031-S11032:

s11031, determining a first shooting pose of a third image frame according to the first shooting pose of a first image frame and a second image frame in three continuous image frames and a relative transformation parameter between the second image frame and the third image frame in the three continuous image frames.

In this embodiment, the first image frame and the second image frame are image frames corresponding to three consecutive image frames, for which the first photographing pose has been restored, and the third image frame is an image frame corresponding to three consecutive image frames, for which the first photographing pose has not been restored. Illustratively, when image frame I ₁ And image frame I ₂ When the first shooting pose of (1) is predetermined, an image frame I is constructed ₁ Image frame I ₂ And image frame I ₃ According to which the image frame I is judged ₃ If it is the first lost frame, the image sequence is continuedTo acquire an image frame I ₄ If not, according to the image frame I ₁ And image frame I ₂ First shooting pose and image frame I ₂ And image frame I ₃ Relative transformation parameters between the two image frames are determined to determine the image frame I ₃ The first shooting pose. In determining the image frame I ₃ After the first shooting pose, an image frame I is acquired from the image sequence ₄ And constructing an image frame I ₂ Image frame I ₃ And image frame I ₄ And so on, the first lost frame in the image sequence can be determined and the first shooting pose of the rest of the image frames can be recovered.

In this embodiment, fig. 6 is a flowchart for determining the first shooting pose of the third image frame according to an embodiment of the present application. As shown in fig. 6, the step of determining the first shooting pose of the third image frame specifically includes S110311 to S110313:

s110311, carrying out triangularization processing on the target feature matching pairs of the first image frame and the second image frame according to the first shooting poses of the first image frame and the second image frame, and generating a first three-dimensional point.

The embodiment uses the image frame I ₃ The first lost frame is not described as an example. Illustratively, from image frame I ₁ Image frame I ₂ And an image frame I ₃ Acquires the image frame I from all the triples ₁ And image frame I ₂ And triangularizing the target feature matching pair to construct a first three-dimensional point corresponding to the target feature matching pair in a world coordinate system.

S110312, multiplying the relative transformation parameter between the second image frame and the third image frame by the first shooting pose of the second image frame, and projecting the first three-dimensional point to the third image frame through the product to obtain a two-dimensional point of the first three-dimensional point in the third image frame.

Illustratively, image frame I ₂ And image frame I ₃ The relative transformation parameter between the first and second image frames is multiplied by the first shooting pose of the second image frame, and the calculated product can be regarded as the image frame I ₃ The shooting pose of (1). The first three-dimension is determined by the shooting pose and camera internal parametersPoint projection to image frame I ₃ Under the pixel coordinate system of (2), constructing a first three-dimensional point in an image frame I ₃ Two-dimensional point of (2).

S110313, constructing a re-projection residual equation according to the two-dimensional points and the feature points in the third image frame, calculating the re-projection residual equation, and obtaining the first shooting pose of the third image frame.

Illustratively, from image frame I ₁ Image frame I ₂ And image frame I ₃ Acquire image frame I from all triplets of ₃ According to the target feature matching pair corresponding to the two-dimensional point, the image frame I which is the same triple with the target feature matching pair is used as the feature point ₃ Is subtracted from the two-dimensional point. Based on the difference value between each two-dimensional point and the corresponding feature point, a first re-projection residual equation is constructed through a least square algorithm, and the first re-projection residual equation is solved to obtain an image frame I ₃ The first rotation matrix of (1). Determining a first three-dimensional point according to the first rotation matrix and re-projecting the first three-dimensional point to the image frame I ₃ Based on the two-dimensional point and the image frame I ₃ The difference value of the corresponding characteristic points is constructed by the least square algorithm to form a second projection residual equation, and the second projection residual equation is solved to obtain the image frame I ₃ The first translation matrix of (a). From the first rotation matrix and the first translation matrix, an image frame I may be determined ₃ The first shooting pose.

S11032, carrying out global optimization on the first shooting pose of each image frame through a beam adjustment algorithm to obtain a second shooting pose of each image frame.

Illustratively, after the first shooting poses of the other image frames except the first lost frame in the image sequence are determined, the first three-dimensional points and the first shooting poses constructed by all the adjacent image frames are combined to perform global BA optimization, and a second shooting pose with higher precision of the image frame is obtained.

And S1104, constructing a first three-dimensional map according to the feature points of the image frame and the second shooting pose.

For example, all the first three-dimensional points after global BA optimization may be combined into a first three-dimensional map of the area to be mapped. But since the AC RANSAC filtering is performed on the first feature matching pairs of the adjacent image frames in the feature matching process, the number of all the first three-dimensional points is small, and the density of the generated first three-dimensional map is low. Therefore, a first three-dimensional map with dense three-dimensional points can be constructed based on the first feature matching pairs of the adjacent image frames and the second shooting pose of the image frames. In this embodiment, fig. 7 is a flowchart for constructing a first three-dimensional map according to an embodiment of the present application. As shown in fig. 7, the step of constructing the first three-dimensional map specifically includes S11041 to S11043:

and S11041, dividing two groups of first feature matching pairs containing the same feature point into the same feature point set.

Illustratively, assume an image frame I _i And image frame I _i+1 Is a first feature matching pair of

Image frame I _i+1 And image frame I _i+2 Is ≥ a first characteristic matching pair>

Then the characteristic point is taken>

And &>

And dividing the feature points into the same feature point set.

And S11042, triangularizing the feature point set according to the second shooting pose of the image frame to generate a first three-dimensional point cloud.

Illustratively, assume that a feature point is included in a feature point set

And &>

From image frame I _i Image frame I _i+1 And image frame I _i+1 For the second shooting pose, for the characteristic point->

And &>

Triangularization is performed to construct a characteristic point->

And &>

Three-dimensional points in the world coordinate system. And analogizing in sequence, combining the three-dimensional points constructed by each feature point set to obtain a first three-dimensional point cloud.

S11043, carrying out global optimization on the first three-dimensional point cloud and the second shooting pose of the image frame through a beam adjustment algorithm to obtain a third shooting pose of the first three-dimensional map and the image frame.

Illustratively, global BA optimization is carried out by combining the first three-dimensional point cloud and the second shooting pose of each image frame to obtain a third shooting pose of the first three-dimensional map and the image frame. Compared with the first three-dimensional point cloud and the second shooting pose, the first three-dimensional map and the third shooting pose are higher in precision.

And S120, optimizing the first three-dimensional map through an incremental reconstruction algorithm based on the first lost frame to obtain a second three-dimensional map, and determining a second lost frame in the image sequence.

Because some first lost frames which do not restore the shooting pose exist in the image sequence and the accuracy of the first three-dimensional map constructed by the global SfM algorithm is low, the embodiment provides that the first lost frames are added to the first three-dimensional map one by one based on the incremental reconstruction algorithm to generate a second three-dimensional map with higher accuracy. The incremental reconstruction algorithm is an incremental SfM algorithm.

In this embodiment, fig. 8 is a flowchart for generating a second three-dimensional map according to an embodiment of the present application.

As shown in fig. 8, the step of generating the second three-dimensional map specifically includes S1201-S1204:

s1201, determining a matching frame of the first lost frame from the recovered frames of the image sequence, and determining a third shooting pose of the first lost frame according to a third shooting pose of the matching frame.

In this embodiment, the recovery frame is an image frame from which the third shooting pose has been recovered in the image sequence. And respectively carrying out feature matching on the first lost frame and each recovery frame in the image sequence, determining the number of target feature matching pairs between the first lost frame and each recovery frame, and taking the recovery frame with the maximum number of the target feature matching pairs as the matching frame of the first lost frame. And determining a relative transformation parameter between the first lost frame and the corresponding matching frame according to the target characteristic matching pair of the first lost frame and the corresponding matching frame, and multiplying the relative transformation parameter by the third shooting pose of the matching frame to recover the third shooting pose of the first lost frame.

It should be noted that, if the number of the target feature matching pairs of the first lost frame and each of the recovery frames is smaller than the preset number threshold, it indicates that the matching constraint of the recovery frames on the first lost frame is low, and the shooting pose of the first lost frame cannot be recovered based on the target feature matching pairs, and at this time, the first lost frame may be marked as a second lost frame in the image sequence.

And S1202, triangularizing the target feature matching pair of the first lost frame and the matching frame according to the third shooting pose of the first lost frame and the matching frame to generate a second three-dimensional point.

Illustratively, according to the third shooting pose of the first lost frame and the matching frame, triangularization processing is performed on the target feature matching pair of the first lost frame and the matching frame, and a three-dimensional point of a feature point of the target feature matching pair in a world coordinate system is constructed.

And S1203, adding the second three-dimensional point into the first three-dimensional map, and optimizing the first three-dimensional map added with the second three-dimensional point cloud through a beam adjustment algorithm.

Illustratively, after adding the second three-dimensional point to the first three-dimensional map, global BA optimization is performed on the first three-dimensional map added with the second three-dimensional point cloud and the third photographing pose of the first lost frame and all recovered frames. Further, the optimized first lost frame is marked as a recovered frame, a next first lost frame is obtained, a matched frame of the first lost frame is determined from all the recovered frames, and the steps S1201-S1203 are repeatedly executed until the second three-dimensional point added with the last first lost frame is optimized.

And S1204, after optimizing the second three-dimensional point cloud of the last first lost frame, globally optimizing the first three-dimensional map added with each second three-dimensional point through a beam adjustment algorithm to obtain a second three-dimensional map.

Illustratively, global BA optimization is carried out on the first three-dimensional map added with all the second three-dimensional points, the optimized third shooting postures of all the recovery frames and the camera internal parameter, and a least square function is constructed. And resolving a least square function to obtain a second three-dimensional map and a fourth shooting pose of each recovery frame.

In this embodiment, after the first three-dimensional map is constructed by the global SfM algorithm, the three-dimensional scene of the first lost frame is gradually added to the first three-dimensional map by the incremental SfM algorithm, so as to construct the second three-dimensional map with higher precision. And the quantity of the first lost frames is small, even if the incremental SfM algorithm needs to perform global BA optimization once after the three-dimensional scene of the first lost frame is constructed, the time required for generating the second three-dimensional map is much shorter than that of directly adopting the incremental SfM algorithm, so that the map construction efficiency is improved while the high-precision map is constructed.

S130, under the condition that the second three-dimensional map does not meet the preset complete condition, constructing a local three-dimensional map corresponding to the second lost frame through a hole filling algorithm, and filling the local three-dimensional map into the second three-dimensional map to obtain the target three-dimensional map.

In this embodiment, the preset complete condition refers to a condition that is satisfied when the second three-dimensional map can completely describe the three-dimensional scene of the area to be mapped. Illustratively, when the second three-dimensional map meets the preset complete condition and the second three-dimensional map is successfully built, the second three-dimensional map is used as a target three-dimensional map of the area to be drawn; and when the second three-dimensional map does not meet the preset complete condition and the second three-dimensional map is determined to be failed in map building, hole filling operation is carried out on the hole area in the second three-dimensional map so as to fill the hole area in the second three-dimensional map, and the target three-dimensional map of the area to be drawn is obtained. For example, if the ratio of the number of restored frames to the number of image frames of the image sequence in the second three-dimensional map is larger, the integrity of the second two-dimensional map is higher. Therefore, the preset ratio threshold is set as the minimum ratio when the second three-dimensional map can completely describe the three-dimensional scene of the area to be drawn. Determining that the second three-dimensional map does not meet a preset integrity condition under the condition that the ratio of the number of the recovery frames in the second three-dimensional map to the number of the image frames of the image sequence is less than or equal to a preset ratio threshold; and under the condition that the ratio of the number of the recovery frames in the second three-dimensional map to the number of the image frames of the image sequence is greater than a preset ratio threshold value, determining that the second three-dimensional map meets a preset complete condition.

In an embodiment, fig. 9 is a flowchart of generating a target three-dimensional map based on a hole filling algorithm according to an embodiment of the present application. As shown in fig. 9, the step of generating the target three-dimensional map based on the hole filling algorithm specifically includes S1301-S1303:

and S1301, adding the lost frame and the recovery frame in the corresponding neighborhood into the group to be solved according to the distance between the second lost frame and the recovery frame.

For example, each time the unmanned device takes one image frame, the RTK positioning system of the unmanned device determines the geographical coordinates of the unmanned device, and uses the geographical coordinates as the geographical coordinates of the currently taken image frame. The coordinate distance of the second lost frame from each recovered frame can be determined based on the geographic coordinates of the second lost frame and the geographic coordinates of each recovered frame. And based on the coordinate distance between the second lost frame and each recovery frame, sequencing the recovery frames from small to large according to the corresponding coordinate distance, and taking the recovery frames with the first preset number in the front of the sequencing as the recovery frames in the adjacent domain corresponding to the second lost frame. And adding all the second lost frames and all the recovery frames in the corresponding neighborhood into a group to be solved.

It should be noted that, since there may be an association relationship between the second lost frame and the second lost frame, all the second lost frames and the recovery frames in the corresponding neighborhood are added to a group to be solved in this embodiment, so as to completely search out the association relationship between the image frames from the group to be solved.

S1302, determining a matching result between each image frame in the group to be solved, and dividing the two matched image frames into the same connected group according to the matching result.

Illustratively, any image frame in the group to be solved is taken as a fourth image frame, and any image frame in the group to be solved except the fourth image frame is taken as a fifth image frame. And calculating the coordinate distance between the fourth image frame and each fifth image frame according to the geographic coordinates of the fourth image frame and each fifth image frame. And sequencing the fifth image frames according to the descending order of the coordinate distances between the fifth image frames and the fourth image frames, so that the fifth image frame which is sequenced more front is closer to the fourth image frame. And if the fourth image frame is a second lost frame, selecting a second lost frame with a second preset number and a recovery frame with a second preset number from the beginning from the sequence of each fifth image frame to form a group of image pairs with the fourth image frame. And if the fourth image frame is a recovery frame, selecting a third preset number of fifth image frames from the beginning of the sequence of the fifth image frames to form a group of image pairs with the fourth image frame respectively. Wherein the third predetermined number is twice the second predetermined number.

The characteristic points are extracted after each group of image pairs are subjected to up-sampling so as to extract more effective characteristic points, the number of characteristic matching pairs of the image pairs is increased, and the incidence relation between the two image frames is deeply excavated. And performing feature matching on the two image frames based on the feature points of the two image frames of the image pair, and determining the number of target feature matching pairs of the image pair. If the number of the target feature matching pairs of the image pair is greater than a preset first number threshold, it may be determined that the two image frames of the image pair match, otherwise it may be determined that the two image frames do not match. Wherein the first number threshold is the minimum number of feature matching pairs when two image frames set by the present embodiment are matched. It can be understood that if the number of the feature matching pairs of the two image frames exceeds the preset number threshold, it indicates that the two image frames are highly overlapped, and the shooting pose of one image frame forms a larger constraint on the shooting pose of the other image frame, so that it can be determined that an association relationship exists between the two matched image frames.

And traversing the matching result of each image pair, and dividing the two matched image frames into the same connected group. For example, if image frame I _i And image frame I _j Matching, image frame I _j And image frame I _k Match, i.e. image frame I _i And image frame I _k Mismatch, image frame I will also be _i Image frame I _j And an image frame I _k Storing the data into the same connected group. In this case, a local three-dimensional map associated with the second three-dimensional map and covering the void region may be constructed by performing three-dimensional reconstruction based on the image frames in the connected group, so that the local three-dimensional map is subsequently aligned to the second three-dimensional map according to the association relationship between the local three-dimensional map and the second three-dimensional map, thereby completing the void region in the second three-dimensional map.

And S1303, performing three-dimensional reconstruction according to the image frames in the connected group to obtain a local three-dimensional map, and aligning the local three-dimensional map with a second three-dimensional map to obtain a target three-dimensional map.

Illustratively, a fourth capture pose of the second lost frame is determined from a fourth capture pose of a recovered frame of the connected group that is a set of image pairs with the second lost frame. And dividing two groups of target feature matching pairs containing the same feature points into the same feature point set according to the target feature matching pairs of each group of image pairs in the connected group. And performing triangularization processing on the feature point set according to the fourth shooting pose of each image frame in the connected group to generate a second three-dimensional point cloud. And carrying out global BA optimization on the second three-dimensional point cloud and the fourth shooting pose of each image frame in the connected group through a beam adjustment algorithm to obtain a local three-dimensional map and a fifth shooting pose of each image frame.

And acquiring a fourth shooting pose and a fifth shooting pose of each recovery frame in the connected group. And the fourth shooting pose can be regarded as the shooting pose of the recovery frame in the coordinate system of the second three-dimensional map, and the fifth shooting pose can be regarded as the shooting pose of the recovery frame in the coordinate system of the local three-dimensional map. And determining a relative transformation parameter between the fourth shooting pose and the corresponding fifth shooting pose, converting the local three-dimensional map into a coordinate system of a second three-dimensional map through the relative transformation parameter, and converting a second lost frame in the connected group into the coordinate system of the second three-dimensional map so as to restore the fourth shooting pose of the second lost frame and obtain the local three-dimensional map positioned under the coordinate system of the second three-dimensional map. And combining the local three-dimensional map and the second three-dimensional map which are positioned under the same coordinate system to obtain a complete target three-dimensional map of the area to be mapped, thereby completing the hole filling operation of the whole three-dimensional map.

In summary, the three-dimensional reconstruction method provided by the embodiment of the application constructs a first three-dimensional map through a global reconstruction algorithm based on an image sequence, and determines a first lost frame in the image sequence; optimizing the first three-dimensional map through an incremental reconstruction algorithm based on the first lost frame to obtain a second three-dimensional map, and determining a second lost frame in the image sequence; and under the condition that the second three-dimensional map does not meet the preset complete condition, constructing a local three-dimensional map corresponding to the second lost frame through a hole filling algorithm, and filling the local three-dimensional map into the second three-dimensional map to obtain the target three-dimensional map. By the technical means, the global reconstruction algorithm can be used as initialization operation to quickly construct a three-dimensional map and determine the shooting pose of the image frame. However, the accuracy of the three-dimensional map constructed by the global reconstruction algorithm is low, the shooting pose of a lost frame in an image sequence can be further gradually restored by the incremental reconstruction algorithm, and the three-dimensional map and the shooting pose are jointly optimized every time the shooting pose of a lost frame is restored, so that the accuracy of the three-dimensional map is improved. If the image frame contains an area with serious shielding or complex texture and is limited by the constraints of image resolution and the number of characteristic points, and the image frame does not recover the shooting pose after the incremental reconstruction algorithm, the image frame can be up-sampled by a hole filling algorithm and then the characteristic points are extracted to increase the number of the characteristic points of the image frame and construct a local three-dimensional map corresponding to the image frame, the local three-dimensional map is filled into the three-dimensional map constructed by the incremental reconstruction algorithm to obtain a highly complete three-dimensional map, and the reconstruction power and the robustness of three-dimensional reconstruction are effectively improved.

On the basis of the foregoing embodiments, fig. 10 is a schematic structural diagram of a three-dimensional reconstruction apparatus provided in an embodiment of the present application. Referring to fig. 10, the three-dimensional reconstruction apparatus provided in this embodiment specifically includes: a global reconstruction module 21, an incremental reconstruction module 22, and a hole filling reconstruction module 23.

and the hole filling reconstruction module is configured to construct a local three-dimensional map corresponding to the second lost frame through a hole filling algorithm under the condition that the second three-dimensional map is determined not to meet the preset complete condition, and fill the local three-dimensional map into the second three-dimensional map to obtain the target three-dimensional map.

On the basis of the above embodiment, the global reconstruction module includes: the characteristic matching submodule is configured to extract characteristic points of each image frame in the image sequence and determine a target characteristic matching pair of adjacent image frames; a relative transformation determination sub-module configured to determine relative transformation parameters of adjacent image frames according to the target feature matching pairs of the adjacent image frames; the second position and posture determining submodule is configured to determine a second shooting posture of each image frame according to the relative transformation parameters and the first shooting postures of the two adjacent image frames; and the first map construction sub-module is configured to construct a first three-dimensional map according to the feature points of the image frames and the second shooting pose.

On the basis of the above embodiment, the feature matching sub-module includes: the image processing device comprises a feature matching unit, a feature matching unit and a feature matching unit, wherein the feature matching unit is configured to match feature points of adjacent image frames and determine a first feature matching pair; and the mismatching rejection unit is configured to reject the mismatching pairs in the first feature matching pairs through a mismatching rejection algorithm to obtain target feature matching pairs.

On the basis of the above embodiment, the relative transformation determining sub-module includes: the image processing device comprises a triple constructing unit, a triple extracting unit and a triple extracting unit, wherein the triple constructing unit is configured to construct triples of three continuous image frames based on target feature matching pairs and feature points of the three continuous image frames in an image sequence, and the three feature points in the triples are matched with each other; a relative transformation determining unit configured to determine a relative transformation parameter of a corresponding adjacent image frame according to the triples of the three consecutive image frames, in case that the number of the triples of the three consecutive image frames is greater than or equal to a preset number threshold.

On the basis of the above embodiment, the second posture determination submodule includes: a first pose determination unit configured to determine a first photographing pose of a third image frame from first photographing poses of a first image frame and a second image frame of three consecutive image frames and a relative transformation parameter between the second image frame and the third image frame of the three consecutive image frames; and the second position and posture determining unit is configured to perform global optimization on the first shooting posture of each image frame through a beam adjustment algorithm to obtain a second shooting posture of each image frame.

On the basis of the above embodiment, the first posture determination unit includes: the first three-dimensional point generating subunit is configured to triangulate the target feature matching pairs of the first image frame and the second image frame according to the first shooting poses of the first image frame and the second image frame, and generate a first three-dimensional point; the three-dimensional point projection subunit is configured to multiply the relative transformation parameter between the second image frame and the third image frame by the first shooting pose of the second image frame and project the first three-dimensional point into the third image frame through the multiplication to obtain a two-dimensional point of the first three-dimensional point in the third image frame; and the first pose determining subunit is configured to construct a re-projection residual equation according to the two-dimensional points and the feature points in the third image frame, solve the re-projection residual equation and obtain a first shooting pose of the third image frame.

On the basis of the above embodiment, the first map building sub-module includes: the characteristic point set dividing unit is configured to divide two groups of first characteristic matching pairs containing the same characteristic points into the same characteristic point set; a three-dimensional point cloud generating unit configured to triangulate the feature point set according to a second shooting pose of the image frame, and generate a first three-dimensional point cloud; and the first map generation unit is configured to perform global optimization on the first three-dimensional point cloud and the second shooting pose of the image frame through a beam adjustment algorithm to obtain a third shooting pose of the first three-dimensional map and the image frame.

On the basis of the above embodiment, the incremental reconstruction module includes: the third pose determination submodule is configured to determine a matching frame of the first lost frame from the recovered frame of the image sequence, and determine a third shooting pose of the first lost frame according to a third shooting pose of the matching frame; the second three-dimensional point generating submodule is configured to triangulate a target feature matching pair of the first lost frame and the matching frame according to a third shooting pose of the first lost frame and the matching frame, and generate a second three-dimensional point; the optimization submodule is configured to add the second three-dimensional point into the first three-dimensional map and optimize the first three-dimensional map added with the second three-dimensional point cloud through a beam adjustment algorithm; and the second map generation submodule is configured to perform global optimization on the first three-dimensional map added with each second three-dimensional point through a beam adjustment algorithm after the second three-dimensional point cloud of the last first lost frame is optimized, so that a second three-dimensional map is obtained.

On the basis of the above embodiment, the hole filling reconstruction module includes: the to-be-solved group generation submodule is configured to add the lost frame and the recovery frame in the corresponding neighborhood into the to-be-solved group according to the distance between the second lost frame and the recovery frame; the connected group generation submodule is configured to determine a matching result between image frames in a group to be solved, and divide two matched image frames into the same connected group according to the matching result; and the target three-dimensional map generation submodule is configured to perform three-dimensional reconstruction according to the image frames in the connected group to obtain a local three-dimensional map, and align the local three-dimensional map with the second three-dimensional map to obtain the target three-dimensional map.

In the foregoing, the three-dimensional reconstruction apparatus provided in this embodiment of the present application constructs a first three-dimensional map through a global reconstruction algorithm based on an image sequence, and determines a first lost frame in the image sequence; optimizing the first three-dimensional map by an incremental reconstruction algorithm based on the first lost frame to obtain a second three-dimensional map, and determining a second lost frame in the image sequence; and under the condition that the second three-dimensional map does not meet the preset complete condition, constructing a local three-dimensional map corresponding to the second lost frame through a hole filling algorithm, and filling the local three-dimensional map into the second three-dimensional map to obtain the target three-dimensional map. By the technical means, the global reconstruction algorithm can be used as initialization operation to quickly construct a three-dimensional map and determine the shooting pose of the image frame. However, the accuracy of the three-dimensional map constructed by the global reconstruction algorithm is low, the shooting pose of a lost frame in an image sequence can be further gradually restored by the incremental reconstruction algorithm, and the three-dimensional map and the shooting pose are jointly optimized every time the shooting pose of a lost frame is restored, so that the accuracy of the three-dimensional map is improved. If the image frame contains an area with serious shielding or complex texture and is limited by the constraints of image resolution and the number of characteristic points, and the image frame does not recover the shooting pose after the incremental reconstruction algorithm, the image frame can be up-sampled by a hole filling algorithm and then the characteristic points are extracted to increase the number of the characteristic points of the image frame and construct a local three-dimensional map corresponding to the image frame, the local three-dimensional map is filled into the three-dimensional map constructed by the incremental reconstruction algorithm to obtain a highly complete three-dimensional map, and the reconstruction power and the robustness of three-dimensional reconstruction are effectively improved.

The three-dimensional reconstruction device provided by the embodiment of the application can be used for executing the three-dimensional reconstruction method provided by the embodiment, and has corresponding functions and beneficial effects.

Fig. 11 is a schematic structural diagram of a three-dimensional reconstruction apparatus provided in an embodiment of the present application, and referring to fig. 11, the three-dimensional reconstruction apparatus includes: a processor 31, a memory 32, a communication device 33, an input device 34, and an output device 35. The number of processors 31 in the three-dimensional reconstruction device may be one or more, and the number of memories 32 in the three-dimensional reconstruction device may be one or more. The processor 31, the memory 32, the communication device 33, the input device 34 and the output device 35 of the three-dimensional reconstruction apparatus may be connected by a bus or other means.

The memory 32 is a computer-readable storage medium, and can be used for storing software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the three-dimensional reconstruction method according to any embodiment of the present application (for example, the global reconstruction module 21, the incremental reconstruction module 22, and the hole filling reconstruction module 23 in the three-dimensional reconstruction apparatus). The memory 32 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function; the storage data area may store data created according to use of the device, and the like. Further, the memory 32 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, the memory may further include memory located remotely from the processor, and these remote memories may be connected to the device over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The communication device 33 is used for data transmission.

The processor 31 executes various functional applications of the apparatus and data processing by executing software programs, instructions and modules stored in the memory 32, that is, implements the three-dimensional reconstruction method described above.

The input device 34 may be used to receive entered numeric or character information and to generate key signal inputs relating to user settings and function controls of the apparatus. The output device 35 may include a display device such as a display screen.

The three-dimensional reconstruction device provided by the embodiment can be used for executing the three-dimensional reconstruction method provided by the embodiment, and has corresponding functions and beneficial effects.

Embodiments of the present application also provide a storage medium containing computer-executable instructions, which when executed by a computer processor, are configured to perform a three-dimensional reconstruction method, the three-dimensional reconstruction method comprising: constructing a first three-dimensional map through a global reconstruction algorithm based on an image sequence, and determining a first lost frame in the image sequence; optimizing the first three-dimensional map by an incremental reconstruction algorithm based on the first lost frame to obtain a second three-dimensional map, and determining a second lost frame in the image sequence; and under the condition that the second three-dimensional map does not meet the preset complete condition, constructing a local three-dimensional map corresponding to the second lost frame through a hole filling algorithm, and filling the local three-dimensional map into the second three-dimensional map to obtain a target three-dimensional map.

Storage medium-any of various types of memory devices or storage devices. The term "storage medium" is intended to include: mounting media such as CD-ROM, floppy disk, or tape devices; computer system memory or random access memory such as DRAM, DDR RAM, SRAM, EDO RAM, lanbas (Rambus) RAM, etc.; non-volatile memory, such as flash memory, magnetic media (e.g., hard disk or optical storage); registers or other similar types of memory elements, etc. The storage medium may also include other types of memory or combinations thereof. In addition, the storage medium may be located in a first computer system in which the program is executed, or may be located in a different second computer system connected to the first computer system through a network (such as the internet). The second computer system may provide program instructions to the first computer for execution. The term "storage medium" may include two or more storage media residing in different locations, e.g., in different computer systems connected by a network. The storage medium may store program instructions (e.g., embodied as a computer program) that are executable by one or more processors.

Of course, the storage medium provided in the embodiments of the present application and containing the computer-executable instructions is not limited to the above three-dimensional reconstruction method, and may also perform related operations in the three-dimensional reconstruction method provided in any embodiments of the present application.

The three-dimensional reconstruction device, the storage medium, and the three-dimensional reconstruction apparatus provided in the above embodiments may perform the three-dimensional reconstruction method provided in any embodiments of the present application, and reference may be made to the three-dimensional reconstruction method provided in any embodiments of the present application without detailed technical details described in the above embodiments.

The foregoing is considered as illustrative only of the preferred embodiments of the invention and the principles of the technology employed. The present application is not limited to the particular embodiments described herein, and various obvious changes, adaptations and substitutions may be made by those skilled in the art without departing from the scope of the present application. Therefore, although the present application has been described in more detail with reference to the above embodiments, the present application is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present application, and the scope of the present application is determined by the scope of the claims.

Claims

1. A method of three-dimensional reconstruction, comprising:

2. The three-dimensional reconstruction method of claim 1, wherein said constructing a first three-dimensional map based on a sequence of images by a global reconstruction algorithm comprises:

extracting feature points of each image frame in the image sequence, and determining a target feature matching pair of adjacent image frames;

determining relative transformation parameters of the adjacent image frames according to the target feature matching pairs of the adjacent image frames;

determining a second shooting pose of each image frame according to the relative transformation parameters and the first shooting poses of two adjacent image frames;

and constructing the first three-dimensional map according to the feature points of the image frame and the second shooting pose.

3. The three-dimensional reconstruction method of claim 2, wherein said extracting feature points of each image frame in the image sequence and determining a target feature matching pair of adjacent image frames comprises:

matching the feature points of adjacent image frames to determine a first feature matching pair;

and rejecting the mismatching pairs in the first characteristic matching pairs through a mismatching rejection algorithm to obtain the target characteristic matching pairs.

4. The three-dimensional reconstruction method of claim 2, wherein said determining the relative transformation parameters of the adjacent image frames from the target feature matching pairs of the adjacent image frames comprises:

constructing a triple of three continuous image frames based on target feature matching pairs and feature points of the three continuous image frames in the image sequence, wherein the three feature points in the triple are matched with each other;

and under the condition that the number of the triples of the three continuous image frames is greater than or equal to a preset number threshold, determining the relative transformation parameters of the corresponding adjacent image frames according to the triples of the three continuous image frames.

5. The three-dimensional reconstruction method according to claim 2, wherein the determining a second shooting pose of each image frame according to the relative transformation parameters and the first shooting poses of two adjacent image frames comprises:

determining a first shooting pose of a third image frame according to a first shooting pose of a first image frame and a second image frame in three continuous image frames and a relative transformation parameter between the second image frame and the third image frame in the three continuous image frames;

and globally optimizing the first shooting pose of each image frame through a beam adjustment algorithm to obtain a second shooting pose of each image frame.

6. The three-dimensional reconstruction method according to claim 5, wherein the determining a first shooting pose of a third image frame of three consecutive image frames according to a first shooting pose of a first image frame and a second image frame of the three consecutive image frames and a relative transformation parameter between the second image frame and the third image frame comprises:

triangularization processing is carried out on the target feature matching pairs of the first image frame and the second image frame according to the first shooting poses of the first image frame and the second image frame, and a first three-dimensional point is generated;

multiplying a relative transformation parameter between the second image frame and the third image frame by a first shooting pose of the second image frame, and projecting the first three-dimensional point into the third image frame through the product to obtain a two-dimensional point of the first three-dimensional point in the third image frame;

and constructing a re-projection residual equation according to the two-dimensional points and the feature points in the third image frame, resolving the re-projection residual equation, and obtaining a first shooting pose of the third image frame.

7. The three-dimensional reconstruction method according to claim 3, wherein the constructing the first three-dimensional map according to the feature points of the image frame and the second shooting pose comprises:

dividing two groups of first feature matching pairs containing the same feature points into the same feature point set;

triangularization processing is carried out on the feature point set according to the second shooting pose of the image frame, and a first three-dimensional point cloud is generated;

and carrying out global optimization on the first three-dimensional point cloud and the second shooting pose of the image frame through a light beam adjustment algorithm to obtain a third shooting pose of the first three-dimensional map and the image frame.

8. The three-dimensional reconstruction method of claim 1, wherein the optimizing the first three-dimensional map by an incremental reconstruction algorithm based on a first lost frame in the image sequence to obtain a second three-dimensional map comprises:

determining a matching frame of the first lost frame from the recovered frames of the image sequence, and determining a third shooting pose of the first lost frame according to a third shooting pose of the matching frame;

triangularization processing is carried out on the target feature matching pair of the first lost frame and the matching frame according to the third shooting pose of the first lost frame and the matching frame, and a second three-dimensional point is generated;

adding the second three-dimensional point into the first three-dimensional map, and optimizing the first three-dimensional map added with the second three-dimensional point cloud through a beam adjustment algorithm;

and after optimizing the second three-dimensional point cloud of the last first lost frame, performing global optimization on the first three-dimensional map added with each second three-dimensional point through a beam adjustment algorithm to obtain a second three-dimensional map.

9. The three-dimensional reconstruction method of claim 1, wherein the constructing the local three-dimensional map corresponding to the second lost frame through a hole filling algorithm and filling the local three-dimensional map into the second three-dimensional map to obtain a target three-dimensional map comprises:

adding the lost frame and the recovery frame in the corresponding neighborhood into a group to be solved according to the distance between the second lost frame and the recovery frame;

determining a matching result between each image frame in the group to be solved, and dividing two matched image frames into the same connected group according to the matching result;

and performing three-dimensional reconstruction according to the image frames in the connected group to obtain a local three-dimensional map, and aligning the local three-dimensional map with the second three-dimensional map to obtain a target three-dimensional map.

10. A three-dimensional reconstruction apparatus, comprising:

11. A three-dimensional reconstruction apparatus, comprising: one or more processors; a storage device storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement the three-dimensional reconstruction method of any one of claims 1-9.

12. A storage medium containing computer-executable instructions for performing the three-dimensional reconstruction method of any one of claims 1-9 when executed by a computer processor.