CN112967311B - Three-dimensional line graph construction method and device, electronic equipment and storage medium - Google Patents

Three-dimensional line graph construction method and device, electronic equipment and storage medium Download PDF

Info

Publication number
CN112967311B
CN112967311B CN201911275034.0A CN201911275034A CN112967311B CN 112967311 B CN112967311 B CN 112967311B CN 201911275034 A CN201911275034 A CN 201911275034A CN 112967311 B CN112967311 B CN 112967311B
Authority
CN
China
Prior art keywords
line segment
image
dimensional line
frame acquisition
frame
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201911275034.0A
Other languages
Chinese (zh)
Other versions
CN112967311A (en
Inventor
王求元
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhejiang Shangtang Technology Development Co Ltd
Original Assignee
Zhejiang Shangtang Technology Development Co Ltd
Filing date
Publication date
Application filed by Zhejiang Shangtang Technology Development Co Ltd filed Critical Zhejiang Shangtang Technology Development Co Ltd
Priority to CN201911275034.0A priority Critical patent/CN112967311B/en
Publication of CN112967311A publication Critical patent/CN112967311A/en
Application granted granted Critical
Publication of CN112967311B publication Critical patent/CN112967311B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Abstract

The disclosure relates to a three-dimensional line graph construction method and device, electronic equipment and storage medium, wherein the method comprises the following steps: determining a predicted position of at least one two-dimensional line segment in a t-1 frame acquisition image according to an observation position of the at least one two-dimensional line segment in the t-1 frame acquisition image, wherein the acquisition image is a two-dimensional image of a target environment acquired by image acquisition equipment, the two-dimensional line segment corresponds to a three-dimensional line segment in a three-dimensional line graph of the target environment, and t is an integer greater than 1; according to the predicted position of the at least one two-dimensional line segment in the t frame acquisition image, respectively determining the observation position of each two-dimensional line segment in the t frame acquisition image; and updating the three-dimensional line graph of the target environment according to the observation positions of the two-dimensional line segments in the t-th frame acquisition image. According to the embodiment of the disclosure, the three-dimensional line graph can be quickly constructed, and the robustness of the three-dimensional line graph construction is improved.

Description

Three-dimensional line graph construction method and device, electronic equipment and storage medium
Technical Field
The disclosure relates to the field of computer technology, and in particular, to a three-dimensional line graph construction method and device, electronic equipment and a storage medium.
Background
The three-dimensional line graph construction technology, also called as the instant localization and reconstruction technology (SLAM, simultaneous Localization AND MAPPING), is an important research hotspot in computer vision, and is widely applied to the fields of robots, unmanned aerial vehicles, augmented/virtual reality and the like. The SLAM technology obtains real-time self-positioning information and a three-dimensional line drawing construction result of the surrounding environment through input of a monocular/multi-view camera, and endows the machine with the capability of sensing the surrounding environment. In general, a three-dimensional line graph construction method for extracting features through a single frame image and matching features between frames based on a description sub-algorithm needs to consume a great deal of time, and the robustness of three-dimensional line graph construction is poor.
Disclosure of Invention
The invention provides a three-dimensional line graph construction method and device, electronic equipment and storage medium, which can realize rapid construction of a three-dimensional line graph and improve the robustness of the three-dimensional line graph construction.
According to an aspect of the present disclosure, there is provided a three-dimensional line graph construction method including: determining a predicted position of at least one two-dimensional line segment in a t-1 frame acquisition image according to an observation position of the at least one two-dimensional line segment in the t-1 frame acquisition image, wherein the acquisition image is a two-dimensional image of a target environment acquired by image acquisition equipment, the two-dimensional line segment corresponds to a three-dimensional line segment in a three-dimensional line graph of the target environment, and t is an integer greater than 1; according to the predicted position of the at least one two-dimensional line segment in the t frame acquisition image, respectively determining the observation position of each two-dimensional line segment in the t frame acquisition image; and updating the three-dimensional line graph of the target environment according to the observation positions of the two-dimensional line segments in the t-th frame acquisition image.
In one possible implementation, determining a predicted position of at least one two-dimensional line segment in a t-1 th frame captured image according to an observed position of the at least one two-dimensional line segment in the t-1 th frame captured image includes: for any two-dimensional line segment, determining a motion increment of the two-dimensional line segment between a t-1 frame acquisition image and a t frame acquisition image; and determining the predicted position of the two-dimensional line segment in the t-1 th frame acquisition image according to the observed position of the two-dimensional line segment in the t-1 th frame acquisition image and the motion increment of the two-dimensional line segment between the t-1 th frame acquisition image and the t-1 th frame acquisition image.
In one possible implementation, determining a predicted position of at least one two-dimensional line segment in a t-1 th frame captured image according to an observed position of the at least one two-dimensional line segment in the t-1 th frame captured image includes: according to the observation pose of the image acquisition equipment corresponding to the t-1 frame acquisition image, determining the predicted pose of the image acquisition equipment corresponding to the t frame acquisition image; and aiming at any two-dimensional line segment, determining the predicted position of the two-dimensional line segment in the t-th frame acquisition image according to the three-dimensional line segment corresponding to the two-dimensional line segment and the predicted pose of the image acquisition device corresponding to the t-th frame acquisition image.
In one possible implementation, determining a predicted position of at least one two-dimensional line segment in a t-1 th frame captured image according to an observed position of the at least one two-dimensional line segment in the t-1 th frame captured image includes: for any two-dimensional line segment, determining a motion increment of the two-dimensional line segment between a t-1 frame acquisition image and a t frame acquisition image; determining a first prediction position of the two-dimensional line segment in the t-1 th frame acquisition image according to the observation position of the two-dimensional line segment in the t-1 th frame acquisition image and the motion increment of the two-dimensional line segment between the t-1 th frame acquisition image and the t-1 th frame acquisition image; according to the observation pose of the image acquisition equipment corresponding to the t-1 frame acquisition image, determining the predicted pose of the image acquisition equipment corresponding to the t frame acquisition image; determining a second predicted position of the two-dimensional line segment in the t-th frame acquisition image according to the three-dimensional line segment corresponding to the two-dimensional line segment and the predicted pose of the image acquisition device corresponding to the t-th frame acquisition image; and determining the predicted position of the two-dimensional line segment in the t-th frame acquisition image according to the first predicted position and the second predicted position.
In one possible implementation manner, determining, according to an observed pose of the image capturing device corresponding to the t-1 st frame of captured image, a predicted pose of the image capturing device corresponding to the t-1 st frame of captured image includes: determining a motion increment of the image acquisition device between a t-1 frame acquisition image and a t frame acquisition image; and determining the predicted pose of the image acquisition equipment corresponding to the t-1 frame acquisition image according to the observed pose of the image acquisition equipment corresponding to the t-1 frame acquisition image and the motion increment of the image acquisition equipment between the t-1 frame acquisition image and the t frame acquisition image.
In one possible implementation manner, determining a predicted position of the two-dimensional line segment in the t-th frame acquisition image according to the three-dimensional line segment corresponding to the two-dimensional line segment and a predicted pose of the image acquisition device corresponding to the t-th frame acquisition image includes: and projecting the three-dimensional line segment corresponding to the two-dimensional line segment into the t frame acquisition image according to the predicted pose of the image acquisition device corresponding to the t frame acquisition image, so as to obtain the predicted position of the two-dimensional line segment in the t frame acquisition image.
In one possible implementation manner, determining the predicted position of the two-dimensional line segment in the t-th frame acquired image according to the first predicted position and the second predicted position includes: determining the length of a first line segment of the two-dimensional line segment in a t-th frame acquisition image according to the first prediction position; determining the length of a second line segment of the two-dimensional line segment in the t-th frame acquisition image according to the second predicted position; and determining the predicted position of the two-dimensional line segment in the t-th frame acquisition image through line segment length weighting operation according to the first predicted position, the second predicted position, the first line segment length and the second line segment length.
In one possible implementation manner, according to the predicted position of the at least one two-dimensional line segment in the t-th frame acquisition image, determining the observation position of each two-dimensional line segment in the t-th frame acquisition image respectively includes: aiming at any two-dimensional line segment, according to the predicted position of the two-dimensional line segment in the t frame acquisition image, performing local extraction operation on the t frame acquisition image; and under the condition that the observation line segment of the two-dimensional line segment in the t-th frame acquisition image is extracted, determining the observation position of the two-dimensional line segment in the t-th frame acquisition image according to the position of the extracted observation line segment.
In one possible implementation manner, for any two-dimensional line segment, according to a predicted position of the two-dimensional line segment in a t-th frame acquisition image, performing a local extraction operation on the t-th frame acquisition image, including: determining a plurality of seed pixel points corresponding to the two-dimensional line segments in a t-th frame acquisition image, and respectively determining a line support area corresponding to each seed pixel point to obtain a plurality of line support areas; a fitting line segment corresponding to each line supporting area is respectively determined through a line segment fitting algorithm, so that a plurality of fitting line segments are obtained; determining a fitting line segment which is collinear with the predicted line segment of the two-dimensional line segment in the t-th frame acquisition image and has an overlapping part as a candidate line segment in the case that the fitting line segment which is collinear with the predicted line segment of the two-dimensional line segment in the t-th frame acquisition image and has an overlapping part exists, wherein the predicted line segment of the two-dimensional line segment in the t-th frame acquisition image corresponds to the predicted position of the two-dimensional line segment in the t-th frame acquisition image; and according to the candidate line segments, determining the observation line segments of the two-dimensional line segments in the t-th frame acquisition image.
In one possible implementation, the method further includes: under the condition that the observation line segment of the two-dimensional line segment in the t frame acquisition image is not extracted, and the extracted observation line segment exists in the t-k-1 to t-1 frame acquisition images, determining the predicted line segment of the two-dimensional line segment in the t frame acquisition image as the observation line segment of the two-dimensional line segment in the t frame acquisition image, wherein k is an integer and is more than or equal to 0 and less than or equal to k < t; and deleting the two-dimensional line segment in the t-th frame acquisition image under the condition that the observation line segment of the two-dimensional line segment in the t-th frame acquisition image is not extracted and the extracted observation line segment does not exist in all the t-k-1 to t-1-th frame acquisition images.
In one possible implementation manner, updating the three-dimensional line graph of the target environment according to the observation positions of the two-dimensional line segments in the t-th frame acquisition image includes: according to the observation position of the at least one two-dimensional line segment in the t frame acquisition image, under the condition that the at least two-dimensional line segments are collinear and have overlapping parts in the t frame acquisition image, merging the at least two-dimensional line segments in the t frame acquisition image to obtain the updated observation position of the at least one two-dimensional line segment in the t frame acquisition image; and updating the three-dimensional line graph of the target environment according to the observation position of at least one updated two-dimensional line segment in the t-th frame acquisition image.
In one possible implementation, the method further includes: and correcting the predicted pose of the image acquisition equipment corresponding to the t frame acquisition image according to the observation position of each two-dimensional line segment in the t frame acquisition image to obtain the observed pose of the image acquisition equipment corresponding to the t frame acquisition image.
In one possible implementation manner, according to the observation positions of the two-dimensional line segments in the t frame of the acquired image, correcting the predicted pose of the image acquisition device corresponding to the t frame of the acquired image to obtain the observed pose of the image acquisition device corresponding to the t frame of the acquired image, including: obtaining the corrected observation pose of the image acquisition equipment corresponding to each frame of acquired images in the t-m frames to the t frame of acquired images by minimizing a first energy function, wherein the first energy function comprises at least one of the following data items: the method comprises the steps of forming a first data item by a re-projection error item of point characteristics and an information matrix of the point characteristics of each frame of acquired image in t-m frames to t frames of acquired images, forming a second data item by a re-projection error item of line segment characteristics and an information matrix of the line segment characteristics of each frame of acquired image in t-m frames to t frames of acquired images, forming a pose information matrix of image acquisition equipment corresponding to t-m frames to t frames of acquired images and a third data item by a pose smoothing item of the image acquisition equipment corresponding to adjacent frames of acquired images, wherein m is an integer and is less than or equal to 2 and less than t.
In one possible implementation manner, updating the three-dimensional line graph of the target environment according to the observation positions of the two-dimensional line segments in the t-th frame acquisition image includes: and under the condition that the t-th frame acquisition image is a key frame acquisition image, updating the three-dimensional line graph of the target environment according to the observation positions of the two-dimensional line segments in the key frame acquisition images in the 1 st to t-th frame acquisition images.
In one possible implementation, updating the three-dimensional line map of the target environment according to the observed positions of the two-dimensional line segments in the key frame acquisition images in the 1 st to t th frame acquisition images includes: for any two-dimensional line segment, under the condition that a three-dimensional line segment corresponding to the two-dimensional line segment is not determined according to the observation position of the two-dimensional line segment in the t-th frame acquisition image and the two-dimensional line segment exists in at least two key frame acquisition images, line segment triangularization operation is carried out according to the observation position of the two-dimensional line segment in the at least two key frame acquisition images and the central point of the image acquisition device corresponding to the at least two key frame acquisition images, and the three-dimensional line segment corresponding to the two-dimensional line segment in the t-th frame acquisition image is determined.
In one possible implementation, updating the three-dimensional line map of the target environment according to the observed positions of the two-dimensional line segments in the key frame acquisition images in the 1 st to t th frame acquisition images includes: correcting a three-dimensional line segment corresponding to at least one two-dimensional line segment in each frame of the co-view acquisition image and a viewing pose of the image acquisition device corresponding to each frame of the co-view acquisition image by minimizing a second energy function aiming at the co-view acquisition image of a t frame of the key frame acquisition image, wherein the second energy function comprises at least one of the following data items: the method comprises the steps of forming a first data item by a re-projection error item of point characteristics of each frame of the common-view acquired image and an information matrix of the point characteristics, forming a second data item by a re-projection error item of line segment characteristics of each frame of the common-view acquired image and an information matrix corresponding to the line segment characteristics, wherein the common-view acquired image of a t frame of the acquired image is the same or similar to the image content in the t frame of the acquired image.
In one possible implementation, updating the three-dimensional line map of the target environment according to the observed positions of the two-dimensional line segments in the key frame acquisition images in the 1 st to t th frame acquisition images includes: under the condition that loop closure exists in a three-dimensional line graph of the target environment, updating a three-dimensional line segment corresponding to at least one two-dimensional line segment in each key frame acquisition image and the pose of the image acquisition device corresponding to each key frame acquisition image by minimizing a third energy function, wherein the third energy function comprises at least one of the following data items: the method comprises the steps of acquiring a first data item formed by a re-projection error item of a point characteristic of an image acquired by each key frame and an information matrix corresponding to the point characteristic, acquiring a second data item formed by a re-projection error item of a line segment characteristic of the image acquired by each key frame and an information matrix corresponding to the line segment characteristic, and acquiring scale parameters.
According to another aspect of the present disclosure, there is provided a three-dimensional line drawing construction apparatus including: the first determining module is used for determining a predicted position of at least one two-dimensional line segment in a t-1 frame acquisition image according to an observation position of the at least one two-dimensional line segment in the t-1 frame acquisition image, wherein the acquisition image is a two-dimensional image of a target environment acquired by image acquisition equipment, the two-dimensional line segment corresponds to a three-dimensional line segment in a three-dimensional line graph of the target environment, and t is an integer greater than 1; the second determining module is used for respectively determining the observation positions of the two-dimensional line segments in the t-th frame acquisition image according to the predicted positions of the at least one two-dimensional line segment in the t-th frame acquisition image; and the updating module is used for updating the three-dimensional line graph of the target environment according to the observation positions of the two-dimensional line segments in the t-th frame acquisition image.
According to another aspect of the present disclosure, there is provided an electronic device including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the above method.
According to another aspect of the present disclosure, there is provided a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the above-described method.
In the embodiment of the disclosure, the observation position in the t-1 th frame acquisition image is determined according to the observation position of the two-dimensional line segment which is already determined in the t-1 th frame acquisition image as prior information, so that the three-dimensional line graph of the target environment can be updated by utilizing the space-time coherence between the acquisition images, thereby realizing rapid construction of the three-dimensional line graph and improving the robustness of the construction of the three-dimensional line graph.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments, which proceeds with reference to the accompanying drawings.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the technical aspects of the disclosure.
FIG. 1 illustrates a flow chart of a three-dimensional line graph construction method according to an embodiment of the present disclosure;
FIG. 2 shows a schematic diagram of a line flow j corresponding to a three-dimensional line segment L in an embodiment of the disclosure;
Fig. 3 is a schematic diagram illustrating a plurality of seed pixel points corresponding to two-dimensional line segments in a t-th frame acquisition image according to an embodiment of the present disclosure;
FIG. 4 illustrates a schematic diagram of a plurality of line support regions determined based on a plurality of seed pixels in FIG. 3, in accordance with an embodiment of the present disclosure;
FIG. 5 illustrates a schematic view of a plurality of fitted line segments determined based on the plurality of line support regions of FIG. 4 in accordance with an embodiment of the present disclosure;
FIG. 6 illustrates a schematic diagram of performing a line segment triangularization operation based on two keyframe acquired images in accordance with an embodiment of the present disclosure;
FIG. 7 illustrates a schematic diagram of performing operations to cull inaccurate three-dimensional line segments based on key frame captured images in an embodiment of the present disclosure;
FIG. 8 shows a block diagram of a three-dimensional line graph construction apparatus according to an embodiment of the present disclosure;
FIG. 9 illustrates a block diagram of an electronic device of an embodiment of the present disclosure;
fig. 10 shows a block diagram of an electronic device of an embodiment of the disclosure.
Detailed Description
Various exemplary embodiments, features and aspects of the disclosure will be described in detail below with reference to the drawings. In the drawings, like reference numbers indicate identical or functionally similar elements. Although various aspects of the embodiments are illustrated in the accompanying drawings, the drawings are not necessarily drawn to scale unless specifically indicated.
The word "exemplary" is used herein to mean "serving as an example, embodiment, or illustration. Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.
The term "and/or" is herein merely an association relationship describing an associated object, meaning that there may be three relationships, e.g., a and/or B, may represent: a exists alone, A and B exist together, and B exists alone. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, C, may mean including any one or more elements selected from the group consisting of A, B and C.
Furthermore, numerous specific details are set forth in the following detailed description in order to provide a better understanding of the present disclosure. It will be understood by those skilled in the art that the present disclosure may be practiced without some of these specific details. In some instances, methods, means, elements, and circuits well known to those skilled in the art have not been described in detail in order not to obscure the present disclosure.
Fig. 1 shows a flowchart of a three-dimensional line drawing construction method according to an embodiment of the present disclosure. As shown in fig. 1, the method may include:
step S11, according to the observation position of at least one two-dimensional line segment in the t-1 frame acquisition image, determining the predicted position of the at least one two-dimensional line segment in the t frame acquisition image.
The acquisition image is a two-dimensional image of the target environment acquired by the image acquisition device, the two-dimensional line segment corresponds to a three-dimensional line segment in a three-dimensional line graph of the target environment, and t is an integer greater than 1.
And step S12, respectively determining the observation positions of the two-dimensional line segments in the t-th frame acquisition image according to the predicted positions of the at least one two-dimensional line segment in the t-th frame acquisition image.
And S13, updating a three-dimensional line graph of the target environment according to the observation positions of the two-dimensional line segments in the t-th frame acquisition image.
The observation position is the actual position of the two-dimensional line segment and corresponds to the observation line segment; the predicted position is a possible position of the two-dimensional line segment and corresponds to the predicted line segment. The image capturing device may be various devices capable of capturing images of a target environment, such as a camera, to which the present disclosure is not particularly limited.
The observation position of the two-dimensional line segment which is already determined in the t-1 frame acquisition image is used as priori information to determine the observation position in the t frame acquisition image, so that the three-dimensional line graph of the target environment can be updated by utilizing the space-time coherence between the acquisition images, the three-dimensional line graph can be quickly constructed, and meanwhile, the robustness of the three-dimensional line graph construction can be improved.
And respectively determining the observation positions of the two-dimensional line segments in the t-frame acquisition image according to the predicted positions of at least one two-dimensional line segment in the t-frame acquisition image, wherein the determination modes comprise but are not limited to the following three modes.
First kind:
In one possible implementation, determining the predicted position of the at least one two-dimensional line segment in the t-1 th frame of the acquired image according to the observed position of the at least one two-dimensional line segment in the t-1 th frame of the acquired image includes: for any two-dimensional line segment, determining the motion increment of the two-dimensional line segment between a t-1 frame acquisition image and a t frame acquisition image; and determining the predicted position of the two-dimensional line segment in the t-1 frame acquisition image according to the observed position of the two-dimensional line segment in the t-1 frame acquisition image and the motion increment of the two-dimensional line segment between the t-1 frame acquisition image and the t frame acquisition image.
And constructing a line flow model according to the corresponding relation between the three-dimensional line segments in the three-dimensional line graph and the two-dimensional line segments in the two-dimensional image by utilizing the space-time coherence of the two-dimensional line segments between the continuous frame acquisition images. And determining a two-dimensional line segment sequence formed by two-dimensional line segments corresponding to the three-dimensional line segments L in the continuous frame acquisition images as a line stream j aiming at the three-dimensional line segments L in the three-dimensional line graph. Wherein j= { l t-n,lt-n+1,...,lt-1 }, n is an integer and 1.ltoreq.n < t. The two-dimensional line segment corresponding to the three-dimensional line segment L can be determined in the t-n frame to t-1 frame acquisition images, wherein L t-n is the two-dimensional line segment (observation line segment) corresponding to the three-dimensional line segment L determined in the t-n frame acquisition image for the first time, and L t-1 is the two-dimensional line segment (observation line segment) corresponding to the three-dimensional line segment L in the t-1 frame acquisition image (F t-1). Fig. 2 shows a schematic diagram of a line flow j corresponding to a three-dimensional line segment L in an embodiment of the present disclosure.
For any two-dimensional line segment, according to the line flow j corresponding to the two-dimensional line segment and based on the space-time constraint of constant acceleration, the motion increment m l of the two-dimensional line segment between the t-1 frame acquisition image and the t frame acquisition image can be determined by using a least square method, and then according to the observation position indicated by the observation line segment l t-1 of the two-dimensional line segment in the t-1 frame acquisition image and the motion increment m l of the two-dimensional line segment between the t-1 frame acquisition image and the t frame acquisition image, the predicted line segment g t of the two-dimensional line segment in the t frame acquisition image is determined:
gt＝lt-1+mlΔt (1)。
in equation (1), Δt is the time difference between the t-1 st frame captured image and the t-th frame captured image, and the predicted line segment g t may indicate the predicted position of the two-dimensional line segment in the t-th frame captured image.
Second kind:
In one possible implementation, determining the predicted position of the at least one two-dimensional line segment in the t-1 th frame of the acquired image according to the observed position of the at least one two-dimensional line segment in the t-1 th frame of the acquired image includes: according to the observation pose of the image acquisition equipment corresponding to the t-1 frame acquisition image, determining the predicted pose of the image acquisition equipment corresponding to the t frame acquisition image; and aiming at any two-dimensional line segment, determining the predicted position of the two-dimensional line segment in the t-th frame acquisition image according to the three-dimensional line segment corresponding to the two-dimensional line segment and the predicted pose of the image acquisition device corresponding to the t-th frame acquisition image.
In one possible implementation manner, determining the predicted pose of the image acquisition device corresponding to the t-1 st frame of acquired image according to the observed pose of the image acquisition device corresponding to the t-1 st frame of acquired image includes: determining a motion increment of the image acquisition device between a t-1 frame acquisition image and a t frame acquisition image; and determining the predicted pose of the image acquisition device corresponding to the t-1 th frame acquisition image according to the observed pose of the image acquisition device corresponding to the t-1 th frame acquisition image and the motion increment of the image acquisition device between the t-1 th frame acquisition image and the t-1 th frame acquisition image.
According to the observation pose of the image acquisition device corresponding to the 1 st frame to the T-1 st frame acquisition image, based on the space constraint of the motion continuity of the image acquisition device, the motion increment M T of the image acquisition device between the T-1 st frame acquisition image and the T frame acquisition image can be determined by using a least square method, and then the predicted pose T t' of the image acquisition device corresponding to the T frame acquisition image is determined according to the observation pose T t-1 of the image acquisition device corresponding to the T-1 st frame acquisition image and the motion increment M T of the image acquisition device between the T-1 st frame acquisition image and the T frame acquisition image:
Tt'＝Tt-1+MTΔt (2)。
In one possible implementation manner, determining a predicted position of the two-dimensional line segment in the t-th frame acquisition image according to the three-dimensional line segment corresponding to the two-dimensional line segment and a predicted pose of the image acquisition device corresponding to the t-th frame acquisition image includes: and projecting the three-dimensional line segment corresponding to the two-dimensional line segment into the t frame acquisition image according to the predicted pose of the image acquisition device corresponding to the t frame acquisition image, so as to obtain the predicted position of the two-dimensional line segment in the t frame acquisition image.
For any two-dimensional line segment, according to the three-dimensional line segment L corresponding to the two-dimensional line segment and the predicted pose T t' of the image acquisition device corresponding to the T frame acquisition image, determining the predicted position h t of the two-dimensional line segment in the T frame acquisition image through the following formula:
In formula (3), E s and E t are two three-dimensional endpoints of the three-dimensional line segment L in the three-dimensional line graph; e s and e t are two-dimensional endpoints of a predicted line segment in a t-frame acquisition image, which corresponds to the two-dimensional line segment L, and the predicted position h t;Kp of the predicted line segment in the t-frame acquisition image, which corresponds to the predicted line segment, is an internal reference matrix of the image acquisition device, which corresponds to the midpoint characteristic of the acquired image; r t、tt is a rotation matrix parameter and a translation matrix parameter which are obtained based on the predicted pose T t' of the image acquisition device corresponding to the T frame acquisition image; Representing the conversion of homogeneous coordinates to two-dimensional coordinates.
Third kind:
In one possible implementation, determining the predicted position of the at least one two-dimensional line segment in the t-1 th frame of the acquired image according to the observed position of the at least one two-dimensional line segment in the t-1 th frame of the acquired image includes: for any two-dimensional line segment, determining the motion increment of the two-dimensional line segment between a t-1 frame acquisition image and a t frame acquisition image; determining a first predicted position of the two-dimensional line segment in the t-1 th frame acquisition image according to the observed position of the two-dimensional line segment in the t-1 th frame acquisition image and the motion increment of the two-dimensional line segment between the t-1 th frame acquisition image and the t-1 th frame acquisition image; according to the observation pose of the image acquisition equipment corresponding to the t-1 frame acquisition image, determining the predicted pose of the image acquisition equipment corresponding to the t frame acquisition image; determining a second predicted position of the two-dimensional line segment in the t-th frame acquisition image according to the three-dimensional line segment corresponding to the two-dimensional line segment and the predicted pose of the image acquisition device corresponding to the t-th frame acquisition image; and determining the predicted position of the two-dimensional line segment in the t-th frame acquisition image according to the first predicted position and the second predicted position.
For any two-dimensional line segment, the mode of determining the first predicted position of the two-dimensional line segment in the t-th frame acquisition image is the same as the first mode, and is not repeated here; the manner of determining the second predicted position of the two-dimensional line segment in the t-th frame acquired image is the same as the second manner, and is not described herein again.
In one possible implementation, determining the predicted position of the two-dimensional line segment in the t-th frame acquired image according to the first predicted position and the second predicted position includes: determining the length of a first line segment of the two-dimensional line segment in the t-th frame acquisition image according to the first predicted position; determining the length of a second line segment of the two-dimensional line segment in the t-th frame acquisition image according to the second predicted position; and determining the predicted position of the two-dimensional line segment in the t-th frame acquisition image through line segment length weighting operation according to the first predicted position, the second predicted position, the first line segment length and the second line segment length.
For any two-dimensional line segment, comprehensively utilizing the space-time coherence of the two-dimensional line segment between the continuous frame acquisition images and the space coherence of the motion continuity of the image acquisition device, based on a first predicted line segment g t of the two-dimensional line segment in a t-th frame acquisition image and a second predicted line segment h t of the two-dimensional line segment in the t-th frame acquisition image, determining a predicted line segment l' t for indicating the predicted position of the two-dimensional line segment in the t-th frame acquisition image by utilizing the characteristic of line segment collineation through the following line segment length weighting operation:
In equation (4), l g is a first line segment length corresponding to a first predicted line segment g t of the two-dimensional line segment in the t-th frame captured image, and l h is a second line segment length corresponding to a second predicted line segment h t of the two-dimensional line segment in the t-th frame captured image.
In one possible implementation manner, determining the observation positions of the two-dimensional line segments in the t-th frame acquisition image according to the prediction positions of the at least one two-dimensional line segment in the t-th frame acquisition image respectively includes: aiming at any two-dimensional line segment, according to the predicted position of the two-dimensional line segment in the t frame acquisition image, local extraction operation is carried out on the t frame acquisition image; and under the condition that the observation line segment of the two-dimensional line segment in the t frame acquisition image is extracted, determining the observation position of the two-dimensional line segment in the t frame acquisition image according to the position of the extracted observation line segment.
For any two-dimensional line segment, the predicted position of the two-dimensional line segment in the t frame acquisition image is used as priori information, and the local extraction operation is performed on the t frame acquisition image, so that the observed position of the two-dimensional line segment in the t frame acquisition image can be rapidly and accurately determined. The specific extraction process is described in detail below.
In one possible implementation manner, for any two-dimensional line segment, according to a predicted position of the two-dimensional line segment in the t frame acquisition image, performing a local extraction operation on the t frame acquisition image, including: determining a plurality of seed pixel points corresponding to the two-dimensional line segment in a t-th frame acquisition image, and respectively determining a line support area corresponding to each seed pixel point to obtain a plurality of line support areas; a fitting line segment corresponding to each line supporting area is respectively determined through a line segment fitting algorithm, so that a plurality of fitting line segments are obtained; determining a fitting line segment which is collinear with the predicted line segment of the two-dimensional line segment in the t-th frame acquisition image and has an overlapping part as a candidate line segment when the fitting line segment which is collinear with the predicted line segment of the two-dimensional line segment in the t-th frame acquisition image and has an overlapping part exists, wherein the predicted line segment of the two-dimensional line segment in the t-th frame acquisition image corresponds to the predicted position of the two-dimensional line segment in the t-th frame acquisition image; and according to the candidate line segments, determining the observation line segments of the two-dimensional line segments in the t-th frame acquisition image.
In one possible implementation manner, determining a plurality of seed pixel points corresponding to the two-dimensional line segment in the t-th frame acquisition image includes: n pixel points are determined by equally dividing predicted line segments of the two-dimensional line segments in the t-th frame acquisition image, wherein N is an odd number; and according to the N pixel points, an N multiplied by N pixel point grid area is obtained, any pixel point in the N multiplied by N pixel point grid area is a seed pixel point, and the central pixel point of the N pixel points is the central pixel point of the N multiplied by N pixel point grid area.
The specific value of N may be determined according to practical situations, which is not specifically limited in the present disclosure.
Fig. 3 is a schematic diagram of a plurality of seed pixel points corresponding to two-dimensional line segments in a t-th frame acquisition image according to an embodiment of the disclosure. The line segment AB shown in fig. 3 is a predicted line segment corresponding to the predicted position of the two-dimensional line segment in the t-th frame captured image. As shown in fig. 3, the predicted line segment AB is equidistantly divided to obtain 5 pixel points (n=5) on the predicted line segment, where the pixel point O is a center pixel point of the 5 pixel points, and further, a 5×5 pixel point network area is obtained by using the pixel point O as the center pixel point, any pixel point in the 5×5 pixel point network area is a seed pixel point, that is, fig. 3 shows 25 seed pixel points corresponding to a two-dimensional line segment in the t-th frame acquisition image.
In one possible implementation manner, determining the line support area corresponding to each seed pixel point respectively, to obtain a plurality of line support areas includes: and executing region growing operation on the pixel gradient of each seed pixel point, and determining a line support region corresponding to each seed pixel point to obtain a plurality of line support regions.
For any one seed pixel point, performing a region growing operation according to a pixel gradient of the seed pixel point, specifically: and determining the pixel points around the seed pixel point, which are the same as or similar to the pixel gradient direction of the seed pixel point (the difference value of the pixel gradients is smaller than a third threshold value), as a line support area. Fig. 4 shows a schematic diagram of a plurality of line support regions determined based on a plurality of seed pixels in fig. 3, according to an embodiment of the present disclosure. Four different wire support areas are shown in fig. 4. Each line support region can obtain a corresponding fitting line segment through a line segment fitting technology. Fig. 5 shows a schematic diagram of a plurality of fitted line segments determined based on the plurality of line support regions in fig. 4, in accordance with an embodiment of the present disclosure. In fig. 5, 4 fitted line segments are shown: fitting segment GH, fitting segment UV, fitting segment MN and fitting segment XY.
After the plurality of fitting line segments are determined, if the fitting line segments which are collinear with the predicted line segments of the two-dimensional line segments in the t-th frame acquisition image and have overlapping parts exist in the plurality of fitting line segments, the fitting line segments are determined to be candidate line segments, and then the observation line segments of the two-dimensional line segments in the t-th frame acquisition image are determined based on the candidate line segments.
In one possible implementation, determining an observation line segment of the two-dimensional line segment in the t-th frame acquisition image according to the candidate line segment includes: in the case that only one candidate line segment exists, determining the candidate line segment as an observation line segment of the two-dimensional line segment in the t-th frame acquisition image; and under the condition that a plurality of candidate line segments exist, carrying out line segment fusion on the plurality of candidate line segments, and determining the fused line segments as observation line segments of the two-dimensional line segments in the t-th frame acquisition image.
Taking fig. 5 as an example, among the 4 fitting line segments shown in fig. 5, the fitting line segment GH and the fitting line segment UV are each collinear with and have an overlapping portion with the predicted line segment AB corresponding to the predicted position of the two-dimensional line segment in the t-th frame captured image, and therefore, the fitting line segment GH and the fitting line segment UV are determined as candidate line segments. And further, the observation line segment of the two-dimensional line segment in the t-th frame acquisition image is obtained by carrying out line segment fusion on the fitting line segment GH and the fitting line segment UV.
And updating the line flow where the two-dimensional line segments are located according to the observed line segments of the two-dimensional line segments in the t-th frame acquisition image. For example, for any line flow j, the line flow j before updating is j= { l t-n,lt-n+1,...,lt-1 }, according to the observed line segment l t-1 of the two-dimensional line segment in the t-1 th frame acquisition image as prior information, the observed line segment l t of the two-dimensional line segment in the t-1 th frame acquisition image is determined, and then the observed line segment l t is added to the line flow j, so that the updated line flow j is j= { l t-n,lt-n+1,...,lt-1,lt }.
In one possible implementation, the method further includes: and determining that the observation line segment of the two-dimensional line segment in the t-frame acquisition image is not extracted in the case that the fitting line segment which is collinear with the predicted line segment of the two-dimensional line segment in the t-frame acquisition image and has an overlapped part does not exist.
In one possible implementation, the method further includes: under the condition that no observation line segment of the two-dimensional line segment in the t frame acquisition image is extracted, and the extracted observation line segment exists in the t-k-1 to t-1 frame acquisition images, determining a predicted line segment of the two-dimensional line segment in the t frame acquisition image as an observation line segment of the two-dimensional line segment in the t frame acquisition image, wherein k is an integer and is more than or equal to 0 and less than or equal to k < t; and deleting the two-dimensional line segment in the t-th frame acquisition image under the condition that the observation line segment of the two-dimensional line segment in the t-th frame acquisition image is not extracted and the extracted observation line segment does not exist in the t-k-1 to t-1 th frame acquisition images.
After the plurality of fitting line segments are determined, if the fitting line segments which are collinear with the predicted line segments of the two-dimensional line segments in the t-th frame acquisition image and have overlapping parts do not exist in the plurality of fitting line segments, determining that the observation line segments of the two-dimensional line segments in the t-th frame acquisition image are not extracted by executing local extraction operation on the t-th frame acquisition image.
For any two-dimensional line segment, when the two-dimensional line segment is not extracted from the continuous acquisition images smaller than k frames, the predicted line segment can be directly determined as the observation line segment in the continuous acquisition images smaller than k frames, so that the robustness of the three-dimensional line graph construction is improved. For example, when no observation line segment of the two-dimensional line segment in the t frame acquisition image is extracted, and the extracted observation line segment exists in the t-k-1 to t-1 frame acquisition images, the condition that the current t frame acquisition image is cut off, and the continuous acquisition image frame of the non-extracted observation line segment is smaller than k frames is indicated, at this time, the predicted line segment of the two-dimensional line segment in the t frame acquisition image can be directly determined as the observation line segment of the two-dimensional line segment in the t frame acquisition image, and then the line stream where the two-dimensional line segment is located is updated according to the observation line segment of the two-dimensional line segment in the t frame acquisition image.
For any two-dimensional line segment, when the two-dimensional line segment reaches the condition that no observation line segment is extracted from the continuous k frame acquisition images, deleting the two-dimensional line segment from the latest frame acquisition image so as to improve the construction accuracy of the three-dimensional line graph. For example, when no observation line segment of the two-dimensional line segment in the t-th frame acquisition image is extracted, and no extracted observation line segment exists in the t-k-1 th frame acquisition images, the method indicates that the current t-th frame acquisition image is cut off, and the continuous acquisition image frames of the non-extracted observation line segment reach k frames, at this time, the two-dimensional line segment can be deleted in the t-th frame acquisition image, and then the line flow in which the two-dimensional line segment is located is deleted.
In one possible implementation manner, updating the three-dimensional line graph of the target environment according to the observation positions of the two-dimensional line segments in the t-th frame acquisition image includes: according to the observation position of at least one two-dimensional line segment in the t frame acquisition image, under the condition that the at least two-dimensional line segments are collinear and have overlapping parts in the t frame acquisition image, merging the at least two-dimensional line segments in the t frame acquisition image to obtain the updated observation position of the at least one two-dimensional line segment in the t frame acquisition image; and updating the three-dimensional line graph according to the observation position of at least one updated two-dimensional line segment in the t-th frame acquisition image.
In order to solve the influence of line breakage, line false extraction and the like on the construction accuracy of the three-dimensional line graph, under the condition that at least two-dimensional line segments are determined to be collinear in a t-th frame acquisition image and have overlapping parts according to the observation position of the at least two-dimensional line segments in the t-th frame acquisition image, the at least two-dimensional line segments can be combined in the t-th frame acquisition image to obtain the updated observation position of the at least one two-dimensional line segment in the t-th frame acquisition image, and then the three-dimensional line graph of the target environment is updated to improve the construction accuracy of the three-dimensional line graph.
In one possible implementation manner, under the condition that it is determined that at least two-dimensional line segments are collinear in a t-th frame acquisition image and have overlapping portions, and two-dimensional line segments exceeding a fourth threshold number in line streams where the at least two-dimensional line segments are located are collinear, the at least two line streams are combined, that is, two-dimensional line segments corresponding to the at least two line streams are combined in each frame acquisition image corresponding to the at least two line streams, an observation position of at least one updated two-dimensional line segment in each frame acquisition image corresponding to the at least two line streams is obtained, and then a three-dimensional line map of a target environment is updated, so that the accuracy of three-dimensional line map construction is improved.
In one possible implementation, the method further includes: and aiming at any two-dimensional line segment, determining the length of the line segment of the two-dimensional line segment in the t-th frame acquisition image according to the observation positions of the two-dimensional line segment in the 1 st to t-th frame acquisition images.
For any one three-dimensional line segment in the three-dimensional line graph, the line segment length of the corresponding two-dimensional line segment in the acquired image under different acquisition view angles can be changed remarkably. Therefore, for any two-dimensional line segment, the length of each two-dimensional line segment in the line stream where the two-dimensional line segment is located (for example, the length of each line segment determined by the observation position of the two-dimensional line segment in the 1 st to t th frame acquisition images) is comprehensively considered, so as to determine the length of the line segment of the two-dimensional line segment in the t th frame acquisition image.
In one possible implementation, the segment length of the two-dimensional segment in the t-th frame acquisition image is determined by the following formula:
In the formula (5), l t is the length of the line segment of the two-dimensional line segment in the t-th frame acquisition image, The length of the observation line segment in the t frame acquisition image of the two-dimensional line segment,/>And beta is the length change rate of the preset line segment, which is the average value of the line segment lengths of the two-dimensional line segments in the line flow where the two-dimensional line segments are located.
In one possible implementation, the method further includes: and under the condition that the whole image line segment extraction operation is not executed in all the t-N cl -1 frames to the t-1 frame acquisition image, executing the whole image line segment extraction operation on the t frame acquisition image, and determining a new two-dimensional line segment outside at least one two-dimensional line segment in the t frame acquisition image, wherein N cl is an integer and N cl is less than or equal to 1 and less than t-1.
The generation of new line segments is not very frequent based on the temporal continuity of the acquired images, and therefore, the full line segment extraction operation is performed every N cl frames of acquired images. For the current t frame acquisition image, under the condition that the t-N cl -1 frame to the t-1 frame acquisition image do not execute the full-image line segment extraction operation, executing the full-image line segment extraction operation on the t frame acquisition image, determining a new two-dimensional line segment, further increasing the line flow corresponding to the new two-dimensional line segment, and updating the three-dimensional line graph of the target environment.
In one possible implementation, the method further includes: and correcting the predicted pose of the image acquisition equipment corresponding to the t frame acquisition image according to the observation position of each two-dimensional line segment in the t frame acquisition image to obtain the observed pose of the image acquisition equipment corresponding to the t frame acquisition image.
After the observation positions of the two-dimensional line segments in the t-th frame acquisition image are determined, the corresponding relation between the two-dimensional line segments in the t-th frame acquisition image and the three-dimensional line segments in the three-dimensional line graph can be determined based on the corresponding relation between the line flow of the two-dimensional line segments and the three-dimensional line segments in the three-dimensional line graph, and then based on the corresponding relation, the pose of the image acquisition device corresponding to the multi-frame acquisition image in the sliding window can be subjected to joint correction. For example, a sliding window includes 3 frames of captured images: and correcting the observation pose of the image acquisition equipment corresponding to the t-2 frame acquisition image and the t-1 frame acquisition image based on the corresponding relation between the two-dimensional line segment in each frame acquisition image of the 3 frame acquisition images and the three-dimensional line segment in the three-dimensional line graph, so as to obtain the corrected observation pose of the image acquisition equipment corresponding to the t-2 frame acquisition image and the t-1 frame acquisition image, and correcting the predicted pose of the image acquisition equipment corresponding to the t frame acquisition image, so as to obtain the observation pose of the image acquisition equipment corresponding to the t frame acquisition image.
In one possible implementation manner, correcting, according to an observation position of each two-dimensional line segment in the t frame of the acquired image, a predicted pose of the image acquisition device corresponding to the t frame of the acquired image to obtain an observation pose of the image acquisition device corresponding to the t frame of the acquired image, including: obtaining the observation pose of the corrected image acquisition equipment corresponding to each frame of acquired images in the t-m frame to the t frame of acquired images by minimizing a first energy function, wherein the first energy function comprises at least one of the following data items: the method comprises the steps of forming a first data item by a re-projection error item of point characteristics and an information matrix of the point characteristics of each frame of acquired image in t-m frames to t frames of acquired images, forming a second data item by a re-projection error item of line segment characteristics and an information matrix of the line segment characteristics of each frame of acquired image in t-m frames to t frames of acquired images, forming a pose information matrix of image acquisition equipment corresponding to t-m frames to t frames of acquired images and a third data item by a pose smoothing item of image acquisition equipment corresponding to adjacent frames of acquired images, wherein m is an integer and is less than or equal to 2 and less than or equal to t.
For example, the expression of the first energy function C s may be as follows:
In formula (6), e l is a reprojection error term of a line segment feature of the acquired image, Σ l is an information matrix of a line segment feature of the acquired image, e p is a reprojection error term of a point feature of the acquired image, Σ p is an information matrix of a point feature of the acquired image, e T is a pose smoothing term of an image acquisition device corresponding to an adjacent frame acquired image, Σ T is a pose information matrix of the image acquisition device corresponding to the acquired image, () T represents matrix transposition. By performing iterative optimization on the first energy function C s, the pose of the image acquisition device corresponding to the multi-frame acquired image in the sliding window can be corrected in a combined mode.
In one possible implementation, ε l、∈p may be determined by the following formula:
/>
In the formula (7), K p is an internal reference matrix of the image acquisition device corresponding to the point feature in the acquired image, K l is an internal reference matrix of the image acquisition device corresponding to the line segment feature in the acquired image, f x and f y are focal lengths of the image acquisition device, (x 0,y0)T is an origin in the acquired image; t × is an antisymmetric form of t; P and P are two-dimensional points of the acquired image and three-dimensional points of the three-dimensional line graph, respectively; Is a normalized term of the three-dimensional line segment L, where L x and L y are two-dimensional straight line parameters.
In one possible implementation, ε T may be determined by the following formula:
In the formula (8), T t-2 is the observation pose of the image acquisition device corresponding to the T-2 th frame of acquisition image, T t-1 is the observation pose of the image acquisition device corresponding to the T-1 st frame of acquisition image, T t' is the prediction pose of the image acquisition device corresponding to the T-1 st frame of acquisition image, SE (3) is the lie algebra, and -1 represents the inverse matrix of the matrix.
In one possible implementation manner, updating the three-dimensional line graph of the target environment according to the observation positions of the two-dimensional line segments in the t-th frame acquisition image includes: and under the condition that the t-th frame acquisition image is a key frame acquisition image, updating a three-dimensional line graph of the target environment according to the observation positions of the two-dimensional line segments in the key frame acquisition images in the 1 st to t-th frame acquisition images.
Based on the corresponding relation between the two-dimensional line segments and the three-dimensional line graph in each key frame acquisition image, one or more of line segment triangularization operation, inaccurate three-dimensional line segment elimination, three-dimensional line segment fusion, iterative optimization energy function and other operations can be performed on the key frame acquisition image, so that joint correction of the three-dimensional line graph, the pose of the image acquisition device corresponding to each key frame acquisition image and the line flow is realized.
In one possible implementation, updating the three-dimensional line map of the target environment according to the observed positions of the two-dimensional line segments in the key frame acquisition images in the 1 st to t th frame acquisition images includes: for any two-dimensional line segment, under the condition that a three-dimensional line segment corresponding to the two-dimensional line segment is not determined according to the observation position of the two-dimensional line segment in the t-th frame acquisition image and the two-dimensional line segment exists in at least two key frame acquisition images, line segment triangularization operation is carried out according to the observation position of the two-dimensional line segment in the at least two key frame acquisition images and the central point of the image acquisition device corresponding to the at least two key frame acquisition images, and the three-dimensional line segment corresponding to the two-dimensional line segment in the t-th frame acquisition image is determined. For example, the center point of the image capturing device may be the center position of the image capturing device.
Fig. 6 shows a schematic diagram of performing a line segment triangularization operation based on two keyframe acquired images in accordance with an embodiment of the present disclosure. For any two-dimensional line segment, under the condition that a three-dimensional line segment corresponding to the two-dimensional line segment is not determined according to the observation position of the two-dimensional line segment in the t-th frame acquisition image and the two-dimensional line segment exists in the two key frame acquisition images, based on the two key frame acquisition images, determining a three-dimensional line segment L corresponding to the two-dimensional line segment in the t-th frame acquisition image through the following formula:
In the formula (9) and in figure 6, To acquire images based on keyframes/>In the two-dimensional line segment/>And keyframe acquisition image/>Center point/>, of corresponding image acquisition deviceDetermined plane/>To acquire images based on keyframes/>In the two-dimensional line segment/>And keyframe acquisition image/>Center point/>, of corresponding image acquisition deviceDetermined plane/>Based on plane/>And plane/>And determining a three-dimensional line segment L corresponding to the two-dimensional line segment in the t-th frame acquisition image.
In one possible implementation, updating the three-dimensional line map of the target environment according to the observed positions of the two-dimensional line segments in the key frame acquisition images in the 1 st to t th frame acquisition images includes: for any two-dimensional line segment in any key frame acquisition image, determining uncertainty of a three-dimensional line segment corresponding to the two-dimensional line segment in the key frame acquisition image according to a distance between two-dimensional end points E 1、e2 and a distance between three-dimensional end points E 1、E2, wherein the two-dimensional end point E 1、e2 is positioned on a two-dimensional straight line corresponding to the two-dimensional line segment in the key frame acquisition image and a distance between the two-dimensional end point E 1、e2 and a projection point E of a camera center on the two-dimensional straight line corresponding to the two-dimensional line segment is a first threshold, and the three-dimensional end point E 1、E2 is a projection point of a two-dimensional end point E 1、e2 on a three-dimensional line segment corresponding to the two-dimensional line segment in the key frame acquisition image; and deleting the three-dimensional line segments with uncertainty larger than the second threshold.
Fig. 7 is a schematic diagram illustrating performing operations to cull inaccurate three-dimensional line segments based on keyframe captured images in an embodiment of the present disclosure. For any two-dimensional line segment in a key frame acquisition image, determining the uncertainty U e of a three-dimensional line segment corresponding to the two-dimensional line segment in the key frame acquisition image through the following formula:
In formula (10) and fig. 7, E is a two-dimensional endpoint of an observation line segment of the two-dimensional line segment in the keyframe acquisition image F, E 1、e2 is two-dimensional points located on a two-dimensional straight line corresponding to a predicted line segment, distances between E 1、e2 and E are both a first threshold, and E 1、E2 is a projection point of E 1、e2 on a three-dimensional line segment L corresponding to the two-dimensional line segment. The value of the first threshold may be determined according to practical situations, for example, the value of the first threshold is 0.5 pixel, which is not specifically limited in the present disclosure.
For the three-dimensional line segment with the uncertainty larger than the second threshold value, the three-dimensional line segment is indicated to bring larger error to the three-dimensional line graph construction, so that the three-dimensional line segment with the uncertainty larger than the second threshold value is deleted, and the accuracy of the three-dimensional line graph construction is improved. The value of the second threshold may be determined according to practical situations, which is not specifically limited in the present disclosure.
In one possible implementation, updating the three-dimensional line map of the target environment according to the observed positions of the two-dimensional line segments in the key frame acquisition images in the 1 st to t th frame acquisition images includes: and according to the observation positions of the two-dimensional line segments in each key frame acquisition image, under the condition that the three-dimensional line segments corresponding to at least two-dimensional line segments are collinear and have overlapping parts, carrying out line segment fusion on the three-dimensional line segments corresponding to the at least two-dimensional line segments.
In order to improve the accuracy of the three-dimensional line diagram construction, the three-dimensional line sections corresponding to at least two-dimensional line sections can be combined under the condition that the three-dimensional line sections corresponding to the at least two-dimensional line sections are collinear and have overlapping parts, so that the three-dimensional line diagram is updated.
In one possible implementation, updating the three-dimensional line map of the target environment according to the observed positions of the two-dimensional line segments in the key frame acquisition images in the 1 st to t th frame acquisition images includes: correcting a three-dimensional line segment corresponding to at least one two-dimensional line segment in each frame of the co-view acquisition image and a viewing pose of image acquisition equipment corresponding to each frame of the co-view acquisition image by minimizing a second energy function aiming at the co-view acquisition image of the t frame of the key frame acquisition image, wherein the second energy function comprises at least one of the following data items: the method comprises the steps of forming a first data item by a re-projection error item of point characteristics of each frame of the common-view acquired image and an information matrix of the point characteristics, forming a second data item by a re-projection error item of line segment characteristics of each frame of the common-view acquired image and an information matrix corresponding to the line segment characteristics, wherein the common-view acquired image of a t frame of the acquired image is the same or similar to the image content in the t frame of the acquired image.
For example, the expression of the second energy function C s may be as follows:
In formula (11), e l is a reprojection error term of a line segment feature of the co-view captured image, Σ l is an information matrix of a line segment feature of the co-view captured image, e p is a reprojection error term of a point feature of the co-view captured image, and Σ p is an information matrix of a point feature of the co-view captured image. By performing iterative optimization on the second energy function C s, the three-dimensional line segment corresponding to at least one two-dimensional line segment in each frame of the co-view acquired image and the observed pose of the image acquisition device corresponding to each frame of the co-view acquired image can be corrected.
In one possible implementation, updating the three-dimensional line map of the target environment according to the observed positions of the two-dimensional line segments in the key frame acquisition images in the 1 st to t th frame acquisition images includes: correcting the three-dimensional line segment corresponding to at least one two-dimensional line segment in each key frame acquisition image and the pose of the image acquisition device corresponding to each key frame acquisition image by minimizing a third energy function under the condition that a loop closure exists in the three-dimensional line graph of the detection target environment, wherein the third energy function comprises at least one of the following data items: the method comprises the steps of acquiring a first data item formed by a re-projection error item of a point characteristic of an image acquired by each key frame and an information matrix corresponding to the point characteristic, acquiring a second data item formed by a re-projection error item of a line segment characteristic of the image acquired by each key frame and an information matrix corresponding to the line segment characteristic, and acquiring scale parameters.
For example, the expression of the third energy function C s may be as follows:
In the formula (12) of the present invention, Sigma l is the information matrix of the line segment characteristics of the acquired image, which is the reprojection error term of the line segment characteristics of the acquired imageFor the re-projection error term of the point feature of the acquired image, Σ p is the information matrix of the point feature of the acquired image, and s is the scale parameter. By performing iterative optimization on the third energy function C s, the three-dimensional line segment corresponding to at least one two-dimensional line segment in each key frame acquisition image and the pose of the image acquisition device corresponding to each key frame acquisition image can be corrected to eliminate scale drift.
In the embodiment of the disclosure, the observation position in the t-1 th frame of acquisition image is determined according to the observation position of the two-dimensional line segment which is determined in the t-1 th frame of acquisition image as prior information, so that the three-dimensional line graph of the target environment can be updated by utilizing the space-time coherence between the acquisition images, thereby realizing the rapid construction of the three-dimensional line graph in the scenes of jitter, motion blur and the like of some image acquisition devices, obtaining a structured line graph model and improving the robustness of the construction of the three-dimensional line graph.
It will be appreciated that the above-mentioned method embodiments of the present disclosure may be combined with each other to form a combined embodiment without departing from the principle logic, and are limited to the description of the present disclosure. It will be appreciated by those skilled in the art that in the above-described methods of the embodiments, the particular order of execution of the steps should be determined by their function and possible inherent logic.
In addition, the disclosure further provides a three-dimensional line drawing construction device, an electronic device, a computer readable storage medium and a program, which can be used for implementing any three-dimensional line drawing construction method provided by the disclosure, and corresponding technical schemes and descriptions and corresponding records referring to method parts are omitted.
Fig. 8 shows a block diagram of a three-dimensional line drawing construction apparatus according to an embodiment of the present disclosure. As shown in fig. 8, the apparatus 80 includes:
A first determining module 81, configured to determine, according to an observation position of at least one two-dimensional line segment in a t-1 th frame acquisition image, a predicted position of the at least one two-dimensional line segment in the t-1 th frame acquisition image, where the acquisition image is a two-dimensional image of a target environment acquired by an image acquisition device, the two-dimensional line segment corresponds to a three-dimensional line segment in a three-dimensional line map of the target environment, and t is an integer greater than 1;
a second determining module 82, configured to determine, according to the predicted position of at least one two-dimensional line segment in the t-th frame collected image, the observed position of each two-dimensional line segment in the t-th frame collected image;
And the updating module 83 is used for updating the three-dimensional line graph of the target environment according to the observation positions of the two-dimensional line segments in the t-th frame acquisition image.
In one possible implementation, the first determining module 81 includes:
The first determining submodule is used for determining the motion increment of any two-dimensional line segment between the t-1 frame acquisition image and the t frame acquisition image;
the second determining submodule is used for determining the predicted position of the two-dimensional line segment in the t-1 frame acquisition image according to the observed position of the two-dimensional line segment in the t-1 frame acquisition image and the motion increment of the two-dimensional line segment between the t-1 frame acquisition image and the t frame acquisition image.
In one possible implementation, the first determining module 81 includes:
The third determining submodule is used for determining the predicted pose of the image acquisition device corresponding to the t-1 th frame acquisition image according to the observed pose of the image acquisition device corresponding to the t-1 th frame acquisition image;
and the fourth determination submodule is used for determining the predicted position of any two-dimensional line segment in the t-th frame acquisition image according to the three-dimensional line segment corresponding to the two-dimensional line segment and the predicted pose of the image acquisition device corresponding to the t-th frame acquisition image.
In one possible implementation, the first determining module 81 includes:
The first determining submodule is used for determining the motion increment of any two-dimensional line segment between the t-1 frame acquisition image and the t frame acquisition image;
The second determining submodule is used for determining a first prediction position of the two-dimensional line segment in the t-1 frame acquisition image according to the observation position of the two-dimensional line segment in the t-1 frame acquisition image and the motion increment of the two-dimensional line segment between the t-1 frame acquisition image and the t frame acquisition image;
The third determining submodule is used for determining the predicted pose of the image acquisition device corresponding to the t-1 th frame acquisition image according to the observed pose of the image acquisition device corresponding to the t-1 th frame acquisition image;
A fourth determining submodule, configured to determine a second predicted position of the two-dimensional line segment in the t-th frame acquisition image according to the three-dimensional line segment corresponding to the two-dimensional line segment and a predicted pose of the image acquisition device corresponding to the t-th frame acquisition image;
And the fifth determination submodule is used for determining the predicted position of the two-dimensional line segment in the t-th frame acquisition image according to the first predicted position and the second predicted position.
In one possible implementation, the third determining submodule is specifically configured to:
determining a motion increment of the image acquisition device between a t-1 frame acquisition image and a t frame acquisition image;
And determining the predicted pose of the image acquisition device corresponding to the t-1 th frame acquisition image according to the observed pose of the image acquisition device corresponding to the t-1 th frame acquisition image and the motion increment of the image acquisition device between the t-1 th frame acquisition image and the t-1 th frame acquisition image.
In one possible implementation, the fourth determining submodule is specifically configured to:
And projecting the three-dimensional line segment corresponding to the two-dimensional line segment into the t-th frame acquisition image according to the predicted pose of the image acquisition device corresponding to the t-th frame acquisition image, so as to obtain the predicted position of the two-dimensional line segment in the t-th frame acquisition image.
In one possible implementation, the fifth determining submodule is specifically configured to:
Determining the length of a first line segment of the two-dimensional line segment in a t-th frame acquisition image according to the first predicted position;
determining the length of a second line segment of the two-dimensional line segment in the t-th frame acquisition image according to the second predicted position;
And determining the predicted position of the two-dimensional line segment in the t-th frame acquisition image through line segment length weighting operation according to the first predicted position, the second predicted position, the first line segment length and the second line segment length.
In one possible implementation, the second determining module 82 includes:
The local extraction sub-module is used for executing local extraction operation on the t frame acquisition image according to the predicted position of any two-dimensional line segment in the t frame acquisition image;
And the sixth determining submodule is used for determining the observation position of the two-dimensional line segment in the t-th frame acquisition image according to the position of the extracted observation line segment under the condition that the observation line segment of the two-dimensional line segment in the t-th frame acquisition image is extracted.
In one possible implementation, the local extraction submodule is specifically configured to:
determining a plurality of seed pixel points corresponding to the two-dimensional line segment in a t-th frame acquisition image, and respectively determining a line support area corresponding to each seed pixel point to obtain a plurality of line support areas;
a fitting line segment corresponding to each line supporting area is respectively determined through a line segment fitting algorithm, so that a plurality of fitting line segments are obtained;
Determining a fitting line segment which is collinear with the predicted line segment of the two-dimensional line segment in the t-th frame acquisition image and has an overlapping part as a candidate line segment when the fitting line segment which is collinear with the predicted line segment of the two-dimensional line segment in the t-th frame acquisition image and has an overlapping part exists, wherein the predicted line segment of the two-dimensional line segment in the t-th frame acquisition image corresponds to the predicted position of the two-dimensional line segment in the t-th frame acquisition image;
and according to the candidate line segments, determining the observation line segments of the two-dimensional line segments in the t-th frame acquisition image.
In one possible implementation, the apparatus 80 further includes:
the third determining module is used for determining a predicted line segment of the two-dimensional line segment in the t-th frame acquisition image as an observed line segment of the two-dimensional line segment in the t-th frame acquisition image under the condition that the observed line segment of the two-dimensional line segment in the t-th frame acquisition image is not extracted and the extracted observed line segment exists in the t-th-k-1 to t-1 frame acquisition images, wherein k is an integer and 0 is less than or equal to k < t;
And the fourth determining module is used for deleting the two-dimensional line segment in the t-th frame acquisition image under the condition that the observed line segment of the two-dimensional line segment in the t-th frame acquisition image is not extracted and the extracted observed line segment does not exist in the t-k-1 to t-1 th frame acquisition images.
In one possible implementation, the update module 83 includes:
the first updating sub-module is used for merging at least two-dimensional line segments in the t-th frame acquisition image to obtain the updated observation position of at least one two-dimensional line segment in the t-th frame acquisition image under the condition that the at least two-dimensional line segments are collinear and have an overlapped part in the t-th frame acquisition image according to the observation position of the at least one two-dimensional line segment in the t-th frame acquisition image;
and the second updating sub-module is used for updating the three-dimensional line graph of the target environment according to the observation position of at least one updated two-dimensional line segment in the t-th frame acquisition image.
In one possible implementation, the apparatus 80 further includes:
And the fifth determining module is used for correcting the predicted pose of the image acquisition device corresponding to the t frame acquisition image according to the observation position of each two-dimensional line segment in the t frame acquisition image to obtain the observation pose of the image acquisition device corresponding to the t frame acquisition image.
In one possible implementation manner, the fifth determining module is specifically configured to:
Obtaining the observation pose of the corrected image acquisition equipment corresponding to each frame of acquired images in the t-m frame to the t frame of acquired images by minimizing a first energy function, wherein the first energy function comprises at least one of the following data items: the method comprises the steps of forming a first data item by a re-projection error item of point characteristics and an information matrix of the point characteristics of each frame of acquired image in t-m frames to t frames of acquired images, forming a second data item by a re-projection error item of line segment characteristics and an information matrix of the line segment characteristics of each frame of acquired image in t-m frames to t frames of acquired images, forming a pose information matrix of image acquisition equipment corresponding to t-m frames to t frames of acquired images and a third data item by a pose smoothing item of image acquisition equipment corresponding to adjacent frames of acquired images, wherein m is an integer and is less than or equal to 2 and less than or equal to t.
In one possible implementation, the updating module 83 includes:
And the third updating sub-module is used for updating the three-dimensional line graph of the target environment according to the observation positions of the two-dimensional line segments in the key frame acquisition images in the 1 st frame to the t frame acquisition images under the condition that the t frame acquisition image is the key frame acquisition image.
In one possible implementation, the third updating sub-module is specifically configured to:
For any two-dimensional line segment, under the condition that a three-dimensional line segment corresponding to the two-dimensional line segment is not determined according to the observation position of the two-dimensional line segment in the t-th frame acquisition image and the two-dimensional line segment exists in at least two key frame acquisition images, line segment triangularization operation is carried out according to the observation position of the two-dimensional line segment in the at least two key frame acquisition images and the central point of the image acquisition device corresponding to the at least two key frame acquisition images, and the three-dimensional line segment corresponding to the two-dimensional line segment in the t-th frame acquisition image is determined.
In one possible implementation, the third updating sub-module is specifically configured to:
Correcting a three-dimensional line segment corresponding to at least one two-dimensional line segment in each frame of the co-view acquisition image and a viewing pose of image acquisition equipment corresponding to each frame of the co-view acquisition image by minimizing a second energy function aiming at the co-view acquisition image of the t frame of the key frame acquisition image, wherein the second energy function comprises at least one of the following data items: the method comprises the steps of forming a first data item by a re-projection error item of point characteristics of each frame of the common-view acquired image and an information matrix of the point characteristics, forming a second data item by a re-projection error item of line segment characteristics of each frame of the common-view acquired image and an information matrix corresponding to the line segment characteristics, wherein the common-view acquired image of a t frame of the acquired image is the same or similar to the image content in the t frame of the acquired image.
In one possible implementation, the third updating sub-module is specifically configured to:
Under the condition that loop closure exists in a three-dimensional line graph of a detection target environment, updating a three-dimensional line segment corresponding to at least one two-dimensional line segment in each key frame acquisition image and the pose of an image acquisition device corresponding to each key frame acquisition image by minimizing a third energy function, wherein the third energy function comprises at least one of the following data items: the method comprises the steps of acquiring a first data item formed by a re-projection error item of a point characteristic of an image acquired by each key frame and an information matrix corresponding to the point characteristic, acquiring a second data item formed by a re-projection error item of a line segment characteristic of the image acquired by each key frame and an information matrix corresponding to the line segment characteristic, and acquiring scale parameters.
In some embodiments, functions or modules included in an apparatus provided by the embodiments of the present disclosure may be used to perform a method described in the foregoing method embodiments, and specific implementations thereof may refer to descriptions of the foregoing method embodiments, which are not repeated herein for brevity.
The disclosed embodiments also provide a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the above-described method. The computer readable storage medium may be a non-volatile computer readable storage medium.
The embodiment of the disclosure also provides an electronic device, which comprises: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the above method.
The disclosed embodiments also provide a computer program product comprising computer readable code which, when run on a device, causes a processor in the device to execute instructions for implementing the three-dimensional line map construction method provided in any of the embodiments above.
The disclosed embodiments also provide another computer program product for storing computer readable instructions that, when executed, cause a computer to perform the operations of the three-dimensional line drawing construction method provided in any of the above embodiments.
The electronic device may be provided as a terminal, server or other form of device.
Fig. 9 shows a block diagram of an electronic device of an embodiment of the disclosure. For example, electronic device 900 may be a mobile phone, computer, digital broadcast terminal, messaging device, game console, tablet device, medical device, exercise device, personal digital assistant, or the like.
Referring to fig. 9, an electronic device 900 may include one or more of the following components: a processing component 902, a memory 904, a power component 906, a multimedia component 908, an audio component 910, an input/output (I/O) interface 912, a sensor component 914, and a communication component 916.
The processing component 902 generally controls overall operation of the electronic device 900, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 902 may include one or more processors 920 to execute instructions to perform all or part of the steps of the methods described above. Further, the processing component 902 can include one or more modules that facilitate interaction between the processing component 902 and other components. For example, the processing component 902 can include a multimedia module to facilitate interaction between the multimedia component 908 and the processing component 902.
The memory 904 is configured to store various types of data to support operations at the electronic device 900. Examples of such data include instructions for any application or method operating on the electronic device 900, contact data, phonebook data, messages, pictures, videos, and so forth. The memory 904 may be implemented by any type of volatile or nonvolatile memory device or combination thereof, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disk.
The power supply component 906 provides power to the various components of the electronic device 900. Power supply components 906 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for electronic device 900.
The multimedia component 908 comprises a screen between the electronic device 900 and the user that provides an output interface. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensor may sense not only the boundary of a touch or slide action, but also the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 908 includes a front-facing camera and/or a rear-facing camera. When the electronic device 900 is in an operational mode, such as a shooting mode or a video mode, the front camera and/or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have focal length and optical zoom capabilities.
The audio component 910 is configured to output and/or input audio signals. For example, the audio component 910 includes a Microphone (MIC) configured to receive external audio signals when the electronic device 900 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may be further stored in the memory 904 or transmitted via the communication component 916. In some embodiments, the audio component 910 further includes a speaker for outputting audio signals.
The I/O interface 912 provides an interface between the processing component 902 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to: homepage button, volume button, start button, and lock button.
The sensor assembly 914 includes one or more sensors for providing status assessment of various aspects of the electronic device 900. For example, the sensor assembly 914 may detect an on/off state of the electronic device 900, a relative positioning of the components, such as a display and keypad of the electronic device 900, the sensor assembly 914 may also detect a change in position of the electronic device 900 or a component of the electronic device 900, the presence or absence of a user's contact with the electronic device 900, an orientation or acceleration/deceleration of the electronic device 900, and a change in temperature of the electronic device 900. The sensor assembly 914 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 914 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 914 may also include an acceleration sensor, a gyroscopic sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
The communication component 916 is configured to facilitate communication between the electronic device 900 and other devices, either wired or wireless. The electronic device 900 may access a wireless network based on a communication standard, such as WiFi,2G, or 3G, or a combination thereof. In one exemplary embodiment, the communication component 916 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component 916 further includes a Near Field Communication (NFC) module to facilitate short range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra Wideband (UWB) technology, bluetooth (BT) technology, and other technologies.
In an exemplary embodiment, the electronic device 900 may be implemented by one or more Application Specific Integrated Circuits (ASICs), digital Signal Processors (DSPs), digital Signal Processing Devices (DSPDs), programmable Logic Devices (PLDs), field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic elements for executing the methods described above.
In an exemplary embodiment, a non-transitory computer readable storage medium is also provided, such as a memory 904 including computer program instructions executable by the processor 920 of the electronic device 900 to perform the above-described method.
Fig. 10 shows a block diagram of an electronic device of an embodiment of the disclosure. For example, the electronic device 1000 may be provided as a server. Referring to fig. 10, the electronic device 1000 includes a processing component 1022 that further includes one or more processors, and memory resources represented by memory 1032, for storing instructions, such as application programs, executable by the processing component 1022. The application programs stored in memory 1032 may include one or more modules each corresponding to a set of instructions. Further, the processing component 1022 is configured to execute instructions to perform the methods described above.
The electronic device 1000 can also include a power component 1026 configured to perform power management of the electronic device 1000, a wired or wireless network interface 1050 configured to connect the electronic device 1000 to a network, and an input output (I/O) interface 1058. The electronic device 1000 may operate based on an operating system stored in memory 1032, such as Windows Server, mac OS XTM, unixTM, linuxTM, freeBSDTM, or the like.
In an exemplary embodiment, a non-transitory computer readable storage medium is also provided, such as memory 1032 including computer program instructions executable by processing component 1022 of electronic device 1000 to perform the above-described method.
The present disclosure may be a system, method, and/or computer program product. The computer program product may include a computer readable storage medium having computer readable program instructions embodied thereon for causing a processor to implement aspects of the present disclosure.
The computer readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: portable computer disks, hard disks, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static Random Access Memory (SRAM), portable compact disk read-only memory (CD-ROM), digital Versatile Disks (DVD), memory sticks, floppy disks, mechanical coding devices, punch cards or in-groove structures such as punch cards or grooves having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media, as used herein, are not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., optical pulses through fiber optic cables), or electrical signals transmitted through wires.
The computer readable program instructions described herein may be downloaded from a computer readable storage medium to a respective computing/processing device or to an external computer or external storage device over a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers and/or edge servers. The network interface card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium in the respective computing/processing device.
The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as SMALLTALK, C ++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, aspects of the present disclosure are implemented by personalizing electronic circuitry, such as programmable logic circuitry, field Programmable Gate Arrays (FPGAs), or Programmable Logic Arrays (PLAs), with state information of computer readable program instructions, which can execute the computer readable program instructions.
Various aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.
These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable medium having the instructions stored therein includes an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The computer program product may be realized in particular by means of hardware, software or a combination thereof. In an alternative embodiment, the computer program product is embodied as a computer storage medium, and in another alternative embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK), or the like.
The foregoing description of the embodiments of the present disclosure has been presented for purposes of illustration and description, and is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the various embodiments described. The terminology used herein was chosen in order to best explain the principles of the embodiments, the practical application, or the improvement of technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims (19)

1. The three-dimensional line graph construction method is characterized by comprising the following steps of:
Determining a predicted position of at least one two-dimensional line segment in a t-1 frame acquisition image according to an observation position of the at least one two-dimensional line segment in the t-1 frame acquisition image, wherein the acquisition image is a two-dimensional image of a target environment acquired by image acquisition equipment, the two-dimensional line segment corresponds to a three-dimensional line segment in a three-dimensional line graph of the target environment, and t is an integer greater than 1;
according to the predicted position of the at least one two-dimensional line segment in the t frame acquisition image, respectively determining the observation position of each two-dimensional line segment in the t frame acquisition image;
according to the observation positions of the two-dimensional line segments in the t-th frame acquisition image, updating a three-dimensional line graph of the target environment;
Wherein, according to the predicted position of the at least one two-dimensional line segment in the t frame acquisition image, the observing positions of the two-dimensional line segments in the t frame acquisition image are respectively determined, and the method comprises the following steps:
aiming at any two-dimensional line segment, according to the predicted position of the two-dimensional line segment in the t frame acquisition image, performing local extraction operation on the t frame acquisition image;
And under the condition that the observation line segment of the two-dimensional line segment in the t-th frame acquisition image is extracted, determining the observation position of the two-dimensional line segment in the t-th frame acquisition image according to the position of the extracted observation line segment.
2. The method of claim 1, wherein determining the predicted position of the at least one two-dimensional line segment in the t-1 st frame captured image based on the observed position of the at least one two-dimensional line segment in the t-1 st frame captured image comprises:
For any two-dimensional line segment, determining a motion increment of the two-dimensional line segment between a t-1 frame acquisition image and a t frame acquisition image;
And determining the predicted position of the two-dimensional line segment in the t-1 th frame acquisition image according to the observed position of the two-dimensional line segment in the t-1 th frame acquisition image and the motion increment of the two-dimensional line segment between the t-1 th frame acquisition image and the t-1 th frame acquisition image.
3. The method of claim 1, wherein determining the predicted position of the at least one two-dimensional line segment in the t-1 st frame captured image based on the observed position of the at least one two-dimensional line segment in the t-1 st frame captured image comprises:
according to the observation pose of the image acquisition equipment corresponding to the t-1 frame acquisition image, determining the predicted pose of the image acquisition equipment corresponding to the t frame acquisition image;
and aiming at any two-dimensional line segment, determining the predicted position of the two-dimensional line segment in the t-th frame acquisition image according to the three-dimensional line segment corresponding to the two-dimensional line segment and the predicted pose of the image acquisition device corresponding to the t-th frame acquisition image.
4. The method of claim 1, wherein determining the predicted position of the at least one two-dimensional line segment in the t-1 st frame captured image based on the observed position of the at least one two-dimensional line segment in the t-1 st frame captured image comprises:
For any two-dimensional line segment, determining a motion increment of the two-dimensional line segment between a t-1 frame acquisition image and a t frame acquisition image;
Determining a first prediction position of the two-dimensional line segment in the t-1 th frame acquisition image according to the observation position of the two-dimensional line segment in the t-1 th frame acquisition image and the motion increment of the two-dimensional line segment between the t-1 th frame acquisition image and the t-1 th frame acquisition image;
according to the observation pose of the image acquisition equipment corresponding to the t-1 frame acquisition image, determining the predicted pose of the image acquisition equipment corresponding to the t frame acquisition image;
Determining a second predicted position of the two-dimensional line segment in the t-th frame acquisition image according to the three-dimensional line segment corresponding to the two-dimensional line segment and the predicted pose of the image acquisition device corresponding to the t-th frame acquisition image;
And determining the predicted position of the two-dimensional line segment in the t-th frame acquisition image according to the first predicted position and the second predicted position.
5. The method according to claim 3 or 4, wherein determining the predicted pose of the image capturing device corresponding to the t-1 st frame captured image from the observed pose of the image capturing device corresponding to the t-1 st frame captured image comprises:
determining a motion increment of the image acquisition device between a t-1 frame acquisition image and a t frame acquisition image;
And determining the predicted pose of the image acquisition equipment corresponding to the t-1 frame acquisition image according to the observed pose of the image acquisition equipment corresponding to the t-1 frame acquisition image and the motion increment of the image acquisition equipment between the t-1 frame acquisition image and the t frame acquisition image.
6. The method according to claim 3 or 4, wherein determining the predicted position of the two-dimensional line segment in the t-th frame captured image according to the three-dimensional line segment corresponding to the two-dimensional line segment and the predicted pose of the image capturing device corresponding to the t-th frame captured image comprises:
And projecting the three-dimensional line segment corresponding to the two-dimensional line segment into the t frame acquisition image according to the predicted pose of the image acquisition device corresponding to the t frame acquisition image, so as to obtain the predicted position of the two-dimensional line segment in the t frame acquisition image.
7. The method of claim 4, wherein determining the predicted position of the two-dimensional line segment in the t-th frame captured image based on the first predicted position and the second predicted position comprises:
determining the length of a first line segment of the two-dimensional line segment in a t-th frame acquisition image according to the first prediction position;
determining the length of a second line segment of the two-dimensional line segment in the t-th frame acquisition image according to the second predicted position;
And determining the predicted position of the two-dimensional line segment in the t-th frame acquisition image through line segment length weighting operation according to the first predicted position, the second predicted position, the first line segment length and the second line segment length.
8. The method of claim 1, wherein for any two-dimensional line segment, performing a local extraction operation on a t-th frame captured image based on a predicted position of the two-dimensional line segment in the t-th frame captured image, comprises:
Determining a plurality of seed pixel points corresponding to the two-dimensional line segments in a t-th frame acquisition image, and respectively determining a line support area corresponding to each seed pixel point to obtain a plurality of line support areas;
a fitting line segment corresponding to each line supporting area is respectively determined through a line segment fitting algorithm, so that a plurality of fitting line segments are obtained;
determining a fitting line segment which is collinear with the predicted line segment of the two-dimensional line segment in the t-th frame acquisition image and has an overlapping part as a candidate line segment in the case that the fitting line segment which is collinear with the predicted line segment of the two-dimensional line segment in the t-th frame acquisition image and has an overlapping part exists, wherein the predicted line segment of the two-dimensional line segment in the t-th frame acquisition image corresponds to the predicted position of the two-dimensional line segment in the t-th frame acquisition image;
and according to the candidate line segments, determining the observation line segments of the two-dimensional line segments in the t-th frame acquisition image.
9. The method of claim 8, wherein the method further comprises:
Under the condition that the observation line segment of the two-dimensional line segment in the t frame acquisition image is not extracted, and the extracted observation line segment exists in the t-k-1 to t-1 frame acquisition images, determining the predicted line segment of the two-dimensional line segment in the t frame acquisition image as the observation line segment of the two-dimensional line segment in the t frame acquisition image, wherein k is an integer and is more than or equal to 0 and less than or equal to k < t;
And deleting the two-dimensional line segment in the t-th frame acquisition image under the condition that the observation line segment of the two-dimensional line segment in the t-th frame acquisition image is not extracted and the extracted observation line segment does not exist in all the t-k-1 to t-1-th frame acquisition images.
10. The method of any of claims 1-4, wherein updating the three-dimensional map of the target environment based on the observed positions of the respective two-dimensional line segments in the t-th frame acquisition image comprises:
According to the observation position of the at least one two-dimensional line segment in the t frame acquisition image, under the condition that the at least two-dimensional line segments are collinear and have overlapping parts in the t frame acquisition image, merging the at least two-dimensional line segments in the t frame acquisition image to obtain the updated observation position of the at least one two-dimensional line segment in the t frame acquisition image;
And updating the three-dimensional line graph of the target environment according to the observation position of at least one updated two-dimensional line segment in the t-th frame acquisition image.
11. A method according to claim 3, characterized in that the method further comprises:
And correcting the predicted pose of the image acquisition equipment corresponding to the t frame acquisition image according to the observation position of each two-dimensional line segment in the t frame acquisition image to obtain the observed pose of the image acquisition equipment corresponding to the t frame acquisition image.
12. The method according to claim 11, wherein correcting the predicted pose of the image capturing device corresponding to the t-th frame captured image according to the observed position of each two-dimensional line segment in the t-th frame captured image, to obtain the observed pose of the image capturing device corresponding to the t-th frame captured image, comprises:
Obtaining the corrected observation pose of the image acquisition equipment corresponding to each frame of acquired images in the t-m frames to the t frame of acquired images by minimizing a first energy function, wherein the first energy function comprises at least one of the following data items: the method comprises the steps of forming a first data item by a re-projection error item of point characteristics and an information matrix of the point characteristics of each frame of acquired image in t-m frames to t frames of acquired images, forming a second data item by a re-projection error item of line segment characteristics and an information matrix of the line segment characteristics of each frame of acquired image in t-m frames to t frames of acquired images, forming a pose information matrix of image acquisition equipment corresponding to t-m frames to t frames of acquired images and a third data item by a pose smoothing item of the image acquisition equipment corresponding to adjacent frames of acquired images, wherein m is an integer and is less than or equal to 2 and less than t.
13. The method of claim 1, wherein updating the three-dimensional map of the target environment based on the observed positions of the respective two-dimensional line segments in the t-th frame acquisition image comprises:
And under the condition that the t-th frame acquisition image is a key frame acquisition image, updating the three-dimensional line graph of the target environment according to the observation positions of the two-dimensional line segments in the key frame acquisition images in the 1 st to t-th frame acquisition images.
14. The method of claim 13, wherein updating the three-dimensional map of the target environment based on the observed positions of the respective two-dimensional line segments in key frame acquisition images in the 1 st to t-th frame acquisition images comprises:
For any two-dimensional line segment, under the condition that a three-dimensional line segment corresponding to the two-dimensional line segment is not determined according to the observation position of the two-dimensional line segment in the t-th frame acquisition image and the two-dimensional line segment exists in at least two key frame acquisition images, line segment triangularization operation is carried out according to the observation position of the two-dimensional line segment in the at least two key frame acquisition images and the central point of the image acquisition device corresponding to the at least two key frame acquisition images, and the three-dimensional line segment corresponding to the two-dimensional line segment in the t-th frame acquisition image is determined.
15. The method of claim 13, wherein updating the three-dimensional map of the target environment based on the observed positions of the respective two-dimensional line segments in key frame acquisition images in the 1 st to t-th frame acquisition images comprises:
Correcting a three-dimensional line segment corresponding to at least one two-dimensional line segment in each frame of the co-view acquisition image and a viewing pose of the image acquisition device corresponding to each frame of the co-view acquisition image by minimizing a second energy function aiming at the co-view acquisition image of a t frame of the key frame acquisition image, wherein the second energy function comprises at least one of the following data items: the method comprises the steps of forming a first data item by a re-projection error item of point characteristics of each frame of the common-view acquired image and an information matrix of the point characteristics, forming a second data item by a re-projection error item of line segment characteristics of each frame of the common-view acquired image and an information matrix corresponding to the line segment characteristics, wherein the common-view acquired image of a t frame of the acquired image is the same or similar to the image content in the t frame of the acquired image.
16. The method of claim 13, wherein updating the three-dimensional map of the target environment based on the observed positions of the respective two-dimensional line segments in key frame acquisition images in the 1 st to t-th frame acquisition images comprises:
Under the condition that loop closure exists in a three-dimensional line graph of the target environment, updating a three-dimensional line segment corresponding to at least one two-dimensional line segment in each key frame acquisition image and the pose of the image acquisition device corresponding to each key frame acquisition image by minimizing a third energy function, wherein the third energy function comprises at least one of the following data items: the method comprises the steps of acquiring a first data item formed by a re-projection error item of a point characteristic of an image acquired by each key frame and an information matrix corresponding to the point characteristic, acquiring a second data item formed by a re-projection error item of a line segment characteristic of the image acquired by each key frame and an information matrix corresponding to the line segment characteristic, and acquiring scale parameters.
17. A three-dimensional map construction apparatus, comprising:
The first determining module is used for determining a predicted position of at least one two-dimensional line segment in a t-1 frame acquisition image according to an observation position of the at least one two-dimensional line segment in the t-1 frame acquisition image, wherein the acquisition image is a two-dimensional image of a target environment acquired by image acquisition equipment, the two-dimensional line segment corresponds to a three-dimensional line segment in a three-dimensional line graph of the target environment, and t is an integer greater than 1;
The second determining module is used for respectively determining the observation positions of the two-dimensional line segments in the t-th frame acquisition image according to the predicted positions of the at least one two-dimensional line segment in the t-th frame acquisition image;
the updating module is used for updating the three-dimensional line graph of the target environment according to the observation position of each two-dimensional line segment in the t-th frame acquisition image;
Wherein the second determining module includes:
The local extraction sub-module is used for executing local extraction operation on the t frame acquisition image according to the predicted position of any two-dimensional line segment in the t frame acquisition image;
And the sixth determining submodule is used for determining the observation position of the two-dimensional line segment in the t-th frame acquisition image according to the position of the extracted observation line segment under the condition that the extracted two-dimensional line segment is in the t-th frame acquisition image.
18. An electronic device, comprising:
A processor;
a memory for storing processor-executable instructions;
Wherein the processor is configured to invoke the instructions stored in the memory to perform the method of any of claims 1 to 16.
19. A computer readable storage medium having stored thereon computer program instructions, which when executed by a processor, implement the method of any of claims 1 to 16.
CN201911275034.0A 2019-12-12 Three-dimensional line graph construction method and device, electronic equipment and storage medium Active CN112967311B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911275034.0A CN112967311B (en) 2019-12-12 Three-dimensional line graph construction method and device, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911275034.0A CN112967311B (en) 2019-12-12 Three-dimensional line graph construction method and device, electronic equipment and storage medium

Publications (2)

Publication Number Publication Date
CN112967311A CN112967311A (en) 2021-06-15
CN112967311B true CN112967311B (en) 2024-06-07

Family

ID=

Citations (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105096386A (en) * 2015-07-21 2015-11-25 中国民航大学 Method for automatically generating geographic maps for large-range complex urban environment
CN106570507A (en) * 2016-10-26 2017-04-19 北京航空航天大学 Multi-angle consistent plane detection and analysis method for monocular video scene three dimensional structure
JP2017162024A (en) * 2016-03-07 2017-09-14 学校法人早稲田大学 Stereo matching processing method, processing program and processing device
CN108230437A (en) * 2017-12-15 2018-06-29 深圳市商汤科技有限公司 Scene reconstruction method and device, electronic equipment, program and medium
CN108510516A (en) * 2018-03-30 2018-09-07 深圳积木易搭科技技术有限公司 A kind of the three-dimensional line segment extracting method and system of dispersion point cloud
CN108986037A (en) * 2018-05-25 2018-12-11 重庆大学 Monocular vision odometer localization method and positioning system based on semi-direct method
CN109166149A (en) * 2018-08-13 2019-01-08 武汉大学 A kind of positioning and three-dimensional wire-frame method for reconstructing and system of fusion binocular camera and IMU
CN109558879A (en) * 2017-09-22 2019-04-02 华为技术有限公司 A kind of vision SLAM method and apparatus based on dotted line feature
CN109584362A (en) * 2018-12-14 2019-04-05 北京市商汤科技开发有限公司 3 D model construction method and device, electronic equipment and storage medium
CN109978891A (en) * 2019-03-13 2019-07-05 浙江商汤科技开发有限公司 Image processing method and device, electronic equipment and storage medium
EP3508935A1 (en) * 2018-01-05 2019-07-10 iRobot Corporation System for spot cleaning by a mobile robot
CN110125928A (en) * 2019-03-27 2019-08-16 浙江工业大学 A kind of binocular inertial navigation SLAM system carrying out characteristic matching based on before and after frames
WO2019169540A1 (en) * 2018-03-06 2019-09-12 斯坦德机器人（深圳）有限公司 Method for tightly-coupling visual slam, terminal and computer readable storage medium

Patent Citations (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105096386A (en) * 2015-07-21 2015-11-25 中国民航大学 Method for automatically generating geographic maps for large-range complex urban environment
JP2017162024A (en) * 2016-03-07 2017-09-14 学校法人早稲田大学 Stereo matching processing method, processing program and processing device
CN106570507A (en) * 2016-10-26 2017-04-19 北京航空航天大学 Multi-angle consistent plane detection and analysis method for monocular video scene three dimensional structure
CN109558879A (en) * 2017-09-22 2019-04-02 华为技术有限公司 A kind of vision SLAM method and apparatus based on dotted line feature
CN108230437A (en) * 2017-12-15 2018-06-29 深圳市商汤科技有限公司 Scene reconstruction method and device, electronic equipment, program and medium
EP3508935A1 (en) * 2018-01-05 2019-07-10 iRobot Corporation System for spot cleaning by a mobile robot
WO2019169540A1 (en) * 2018-03-06 2019-09-12 斯坦德机器人（深圳）有限公司 Method for tightly-coupling visual slam, terminal and computer readable storage medium
CN108510516A (en) * 2018-03-30 2018-09-07 深圳积木易搭科技技术有限公司 A kind of the three-dimensional line segment extracting method and system of dispersion point cloud
CN108986037A (en) * 2018-05-25 2018-12-11 重庆大学 Monocular vision odometer localization method and positioning system based on semi-direct method
CN109166149A (en) * 2018-08-13 2019-01-08 武汉大学 A kind of positioning and three-dimensional wire-frame method for reconstructing and system of fusion binocular camera and IMU
CN109584362A (en) * 2018-12-14 2019-04-05 北京市商汤科技开发有限公司 3 D model construction method and device, electronic equipment and storage medium
CN109978891A (en) * 2019-03-13 2019-07-05 浙江商汤科技开发有限公司 Image processing method and device, electronic equipment and storage medium
CN110125928A (en) * 2019-03-27 2019-08-16 浙江工业大学 A kind of binocular inertial navigation SLAM system carrying out characteristic matching based on before and after frames

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
Building a 3-D Line-Based Map Using Stereo SLAM;Guoxuan Zhang等;IEEE TRANSACTIONS ON ROBOTICS;第1364-1377页 *
三维重建中线段匹配方法的研究;陈起凤;刘军;李威;雷光元;董广峰;;武汉工程大学学报(第04期);全文 *
基于几何约束及0-1规划的视频帧间线段特征匹配算法;李海丰;胡遵河;范龙飞;姜子政;陈新伟;;计算机应用(第08期);全文 *
基于线特征的单目 SLAM 中的迭代数据关联算法;魏鑫燏等;计算机应用研究;第1-8页 *

Similar Documents

Publication Publication Date Title
CN110287874B (en) Target tracking method and device, electronic equipment and storage medium
CN110378976B (en) Image processing method and device, electronic equipment and storage medium
TWI769635B (en) Network training pedestrian re-identification method and storage medium
TWI767596B (en) Scene depth and camera motion prediction method, electronic equipment and computer readable storage medium
CN110798630B (en) Image processing method and device, electronic equipment and storage medium
US11288531B2 (en) Image processing method and apparatus, electronic device, and storage medium
CN111401230B (en) Gesture estimation method and device, electronic equipment and storage medium
CN111445414B (en) Image processing method and device, electronic equipment and storage medium
CN109584362B (en) Three-dimensional model construction method and device, electronic equipment and storage medium
CN111881827B (en) Target detection method and device, electronic equipment and storage medium
CN109840917B (en) Image processing method and device and network training method and device
CN108900903B (en) Video processing method and device, electronic equipment and storage medium
CN112991381B (en) Image processing method and device, electronic equipment and storage medium
CN112432637B (en) Positioning method and device, electronic equipment and storage medium
CN111680646B (en) Action detection method and device, electronic equipment and storage medium
CN111860373B (en) Target detection method and device, electronic equipment and storage medium
CN108171222B (en) Real-time video classification method and device based on multi-stream neural network
CN112541971A (en) Point cloud map construction method and device, electronic equipment and storage medium
CN113052874B (en) Target tracking method and device, electronic equipment and storage medium
CN114202562A (en) Video processing method and device, electronic equipment and storage medium
CN113283343A (en) Crowd positioning method and device, electronic equipment and storage medium
CN111311588B (en) Repositioning method and device, electronic equipment and storage medium
CN112837372A (en) Data generation method and device, electronic equipment and storage medium
CN112967311B (en) Three-dimensional line graph construction method and device, electronic equipment and storage medium
CN113506324B (en) Image processing method and device, electronic equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
SE01 Entry into force of request for substantive examination
GR01 Patent grant