WO2022247286A1

WO2022247286A1 - Positioning method, apparatus, device, and storage medium

Info

Publication number: WO2022247286A1
Application number: PCT/CN2021/143513
Authority: WO
Inventors: 王帅; 陈丹鹏
Original assignee: 浙江商汤科技开发有限公司
Priority date: 2021-05-27
Filing date: 2021-12-31
Publication date: 2022-12-01
Also published as: CN113256718A; CN113256718B

Abstract

Disclosed in the present application are a positioning method, an apparatus, a device, and a storage medium. The positioning method comprises: obtaining a current image frame photographed by a device; determining an initial pose of the current image frame according to a relative positional relationship between the current image frame and a first historical image frame; utilizing a point-plane constraint model to optimize the initial pose, and obtaining an optimized pose that serves as a visual positioning result of the device, wherein the point-plane constraint model is constructed by utilizing an association relationship between a structured plane and a feature point pair between the current image frame and a second historical image frame.

Description

Positioning method and device, equipment and storage medium

Cross References to Related Applications

This application claims the priority of the Chinese patent application with the application number 202110587271.1 and the title of the invention "Positioning method and device, equipment and storage medium" submitted on May 27, 2021. The entire content of the Chinese patent application is cited by reference way incorporated into this article.

technical field

The present application relates to the technical field of positioning, and in particular to a positioning method and device, equipment and storage medium.

Background technique

Visual positioning technology plays an important role in many fields, such as unmanned driving, robotics and other fields. In visual positioning technology, structured planes are often used to optimize the positioning results of devices. A structured plane refers to a three-dimensional plane established from constructed three-dimensional points. Generally, the positioning result of the device is optimized based on the relationship between the constructed 3D point and the structured plane, and the accuracy of the 3D point will directly affect the optimization result. In other words, if the precision of the three-dimensional points is not high, the precision of the optimization of the positioning result by the structured plane will be low.

Contents of the invention

The present application provides at least one positioning method and device, equipment and storage medium.

The present application provides a positioning method, including: obtaining the current image frame captured by the device; determining the initial pose of the current image frame according to the relative positional relationship between the current image frame and the first historical image frame; using a point-plane constraint model The initial pose is optimized, and the optimized pose is obtained as the visual positioning result of the device; among them, the point-plane constraint model uses the feature point pairs between the current image frame and the second historical image frame, and the structure plane relationship built.

Therefore, a point-plane constraint model is constructed through the feature point pairs between the current image frame and the second historical image frame, and the relationship between the structured plane, and the initial pose of the device is optimized through the point-plane constraint model. Since the point-plane constraint model does not include 3D point parameters, the process of optimizing the initial pose using the point-plane constraint model is not affected by the accuracy of the 3D point, thereby improving the positioning accuracy of the device.

Among them, the structured plane is a three-dimensional plane constructed by using the current image frame captured by the device and the historical image frames before the current image frame; the point-plane constraint model is used to optimize the initial pose before obtaining the optimized pose, including: Obtain the first feature point pair that has an association relationship between the feature point pairs of the current image frame and the second historical image frame and the structured plane; according to the first position of the current image frame and the second historical image frame in the world coordinate system parameter, the two-dimensional coordinates of the first feature point pair, and the second position parameter of the structured plane to construct a point-plane constraint model.

Therefore, by using the first position parameter of the current image frame and the second historical image frame in the world coordinate system, the two-dimensional coordinates of the first feature point pair, and the second position parameter of the structured plane, instead of using three-dimensional points, the The point-plane constraint model is constructed, and in the process of optimizing the initial pose using the point-plane constraint model, it is not affected by the three-dimensional point accuracy, which effectively improves the positioning accuracy of the device.

Among them, the point-plane constraint model includes the point-plane optimization equation, the point-plane optimization equation includes the first item and the second item, and the first item and the second item are respectively located on both sides of the equal sign of the point-plane optimization equation; the first position parameter includes the rotation Matrix and translation matrix, the second position parameter includes direction matrix and distance matrix; use the point-plane constraint model to optimize the initial pose, including: rotation matrix and translation matrix based on the current image frame and the second historical image frame, structured plane The direction matrix and the distance matrix, the two-dimensional coordinates of the historical feature points in the second historical image frame in the first feature point pair, determine the predicted coordinates of the matching feature points corresponding to the historical feature points in the current image frame, wherein, The predicted coordinates are the first item in the point-plane optimization equation; adjust the preset parameters in the point-plane optimization equation so that the first item in the point-plane optimization equation is equal to the second item in the point-plane optimization equation, where the first The binomial is the real two-dimensional coordinates of the corresponding matching feature points in the current image frame, and the preset parameters include the initial pose of the current image frame.

Therefore, by constructing the association relationship between the feature points and the structured plane, optimizing the preset parameters including the initial pose of the current image frame can improve the positioning accuracy of the device.

Wherein, the point-plane constraint model is used to optimize the initial pose, and the optimized pose is obtained as the visual positioning result of the device, including: responding to the second historical image frame being the last historical image frame of the current image frame, the second The pose of the historical image frame, the pose of the current image frame, and the structured plane are optimized to obtain the visual positioning results of the device at two moments; in response to the second historical image frame not being the last historical image frame of the current image frame, The pose and structured plane of the current image frame are optimized to obtain the visual positioning result of the device at the current moment.

Therefore, when the second historical image frame is the previous historical image frame of the current image frame, not only the pose of the current image frame can be optimized, but also the pose of the second historical image frame can be optimized, thereby improving The accuracy of the positioning results of the device at each time.

Wherein, before obtaining the first feature point pair that has an association relationship between the feature point pair of the current image frame and the second historical image frame and the structured plane, it includes: triangulating the current image frame to obtain the corresponding two-dimensional A grid group, wherein the two-dimensional grid group is composed of several two-dimensional grids. The vertices in the two-dimensional grid group are the feature points in the current image frame; the two-dimensional grid group is projected into the world coordinate system to obtain the corresponding three-dimensional mesh group, wherein the three-dimensional mesh group includes several three-dimensional The vertices in the three-dimensional grid group are three-dimensional points corresponding to the feature points in the current image frame; the first three-dimensional grid satisfying the first preset condition in the three-dimensional grid group is obtained to generate a structured plane.

Therefore, a 2D mesh group is obtained by triangulating the current image frame, and a 3D mesh group is obtained by using the 2D mesh group, and then the first 3D mesh that satisfies the first preset condition in the 3D mesh group is obtained , rather than arbitrary three-dimensional grids to generate structured planes, making the constructed structured planes more precise.

Wherein, the three-dimensional grid group includes several three-dimensional grids; obtaining the first three-dimensional grid satisfying the first preset condition in the three-dimensional grid group to generate the structured plane includes: making the distance from the current image frame less than or equal to The three-dimensional grid of the first preset distance is used as a candidate three-dimensional grid; two or more candidates whose difference in direction is less than or equal to the first preset difference, and/or the difference in distance is less than or equal to the second preset difference are selected 3D mesh, as the first 3D mesh.

Therefore, by setting the first preset distance and the first preset difference and/or the second preset difference to select the first three-dimensional grid for generating the structured plane, the precision of the structured plane can be improved.

Wherein, acquiring the first feature point pair that has an association relationship with the structured plane between the feature point pairs of the current image frame and the second historical image frame includes: acquiring the first feature point pair that has an association relationship with the structured plane in the three-dimensional grid group Two three-dimensional grid; determine some first feature point pairs corresponding to the second three-dimensional grid in the feature point pairs of the current image frame and the second historical image frame, and there is an association relationship between the first feature point pair and the structured plane .

Therefore, by constructing the association relationship between the 3D grid and the structured plane to determine the association relationship between the feature points and the structured plane, compared with the construction of the association relationship between the 3D points and the structured plane, it can reduce the false association probability.

Wherein, obtaining the second three-dimensional grid that has an association relationship with the structured plane in the three-dimensional grid group includes: obtaining the first distance between each vertex of all three-dimensional grids in the three-dimensional grid group and the structured plane; selecting all A 3D grid whose first distance between the vertices and the structured plane is all less than or equal to the second preset distance is used as the second 3D grid; or the first distance between all vertices and the structured plane is selected to be less than or equal to the second A 3D grid with a preset distance and a plane composed of all vertices parallel to the structured plane is used as the second 3D grid.

Therefore, by selecting a three-dimensional grid whose distance from the structured plane is smaller than the second preset distance, and then constructing an association relationship between the three-dimensional grid and the structured plane, the accuracy of the association can be improved.

Wherein, before acquiring the second three-dimensional grid that has an association relationship with the structured plane in the three-dimensional grid group, it includes: selecting the first structured plane that satisfies the second preset condition from multiple structured planes, and the second preset The conditions include that the distance from the current image frame is less than or equal to the third preset distance threshold; obtaining the second 3D grid in the 3D grid group that has an associated relationship with the structured plane includes: obtaining the second 3D grid in the 3D grid group that is associated with the structured plane A second three-dimensional grid with associated relations on a structured plane.

Therefore, by selecting the first structured plane that satisfies the second preset condition from the multiple structured planes, the amount of calculation in the process of constructing the association relationship can be reduced.

Wherein, using the point-plane constraint model to optimize the initial pose includes: merging the point-plane constraint model with at least one of the reprojection constraint model and the IMU constraint model to obtain a fusion constraint model; using the fusion constraint model to optimize the initial pose optimization.

Therefore, by constructing a fusion constraint model to optimize the pose of the current image frame of the device, the positioning accuracy of the device can be improved.

The present application provides a positioning device, including: an image acquisition module, used to acquire the current image frame captured by the device; an initial pose acquisition module, used to obtain the relative positional relationship between the current image frame and the first historical image frame, Determine the initial pose of the current image frame; the pose optimization module is used to optimize the initial pose using the point-plane constraint model, and obtain the optimized pose as the visual positioning result of the device; wherein, the point-plane constraint model uses the current The feature point pairs between the image frame and the second historical image frame, and the association relationship between the structured plane are constructed.

The present application provides an electronic device, including a memory and a processor, and the processor is used to execute program instructions stored in the memory, so as to realize the above positioning method.

The present application provides a computer-readable storage medium, on which program instructions are stored, and the above positioning method is implemented when the program instructions are executed by a processor.

In the above scheme, a point-plane constraint model is constructed through the feature point pairs between the current image frame and the second historical image frame, and the relationship between the structured plane, and the initial pose of the device is optimized through the point-plane constraint model . Since the point-plane constraint model does not include 3D point parameters, the process of optimizing the initial pose using the point-plane constraint model is not affected by the accuracy of the 3D point, thereby improving the positioning accuracy of the device.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.

Description of drawings

The accompanying drawings here are incorporated into the specification and constitute a part of the specification. These drawings show embodiments consistent with the application, and are used together with the description to describe the technical solution of the application.

FIG. 1 is a schematic flow diagram of an embodiment of the positioning method of the present application;

FIG. 2 is a schematic flow diagram of another embodiment of the positioning method of the present application;

Fig. 3 is a schematic structural view of an embodiment of the positioning device of the present application;

FIG. 4 is a schematic structural diagram of an embodiment of the electronic device of the present application;

Fig. 5 is a schematic structural diagram of an embodiment of a computer-readable storage medium of the present application.

Detailed ways

The solutions of the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings.

In the following description, for purposes of illustration rather than limitation, specific details, such as specific system architectures, interfaces, and techniques, are set forth in order to provide a thorough understanding of the present application.

The term "and/or" in this article is just an association relationship describing associated objects, which means that there can be three relationships, for example, A and/or B can mean: A exists alone, A and B exist simultaneously, and there exists alone B these three situations. In addition, the character "/" in this article generally indicates that the contextual objects are an "or" relationship. In addition, "many" herein means two or more than two. In addition, the term "at least one" herein means any one of a variety or any combination of at least two of the more, for example, including at least one of A, B, and C, which may mean including from A, Any one or more elements selected from the set formed by B and C.

Please refer to FIG. 1 . FIG. 1 is a schematic flowchart of an embodiment of a positioning method of the present application. Specifically, the positioning method may include the following steps:

Step S11: Obtain the current image frame captured by the device.

The device executing the positioning method may be the device that captures the current image frame, or may not be the device that captures the current image frame. For example, the execution device may acquire the current image frame by establishing a communication connection with the device that captures the current image frame. In the case that the execution device is not the same device as the device that captures the current image frame, there is no limitation on the communication connection between the two. The following uses the same device as the execution device and the device that captures the current image frame as an example for introduction. The device includes a device for taking a current image frame and a sensor. Among them, the sensor is used to measure the motion information of the device. The current image frame may be an image acquired in real time without any image processing, or may be image processed. The image processing here may be cropping, data enhancement and other processing methods.

In some disclosed embodiments, in order to ensure the positioning accuracy of the device, not all image frames captured by the device will be used as the current image frame, and some image frames with lower quality will not be used as the current image frame. The methods for judging whether the image frame captured by the device can be used as the current image frame may include: 1. Extract the feature points in the image frame, and if the number of feature points is greater than or equal to the first preset number, use the image frame as the current image frame; 2. Obtain the number of feature point pairs between the image frame and the historical image frame in the preset time period, and when the number of feature point pairs is greater than or equal to the second preset number, use the image frame as the current image frame. Of course, in addition to the two conditions listed here, it is also possible to judge whether the clarity and brightness of the image meet the requirements. By selecting an image frame that meets the quality requirements as the current image frame, it is possible to reduce the occurrence of poor positioning results due to poor quality of the image frame, thereby ensuring the positioning accuracy of the device.

Step S12: Determine the initial pose of the current image frame according to the relative positional relationship between the current image frame and the first historical image frame.

The first historical image frame is an image frame that has undergone positioning processing, for example, it may be a previous frame of the current image frame. Generally, the relative positional relationship includes relative distance and relative angle. In the embodiment of the present disclosure, a distance is assumed as the relative distance between the current image frame and the historical image frame, and the sensor readings between the current image frame and the first historical image frame are pre-integrated to obtain the relative angle between the two . Since the pose of the first historical image frame is known, the initial pose of the current image frame can be determined through the relative distance and relative angle between the two.

Step S13: Use the point-plane constraint model to optimize the initial pose, and obtain the optimized pose as the visual positioning result of the device; wherein, the point-plane constraint model uses the feature points between the current image frame and the second historical image frame It is built on the association relationship between , and structured plane.

The first historical image frame and the second historical image frame may be the same or different. The structured plane here refers to a three-dimensional plane constructed by using the three-dimensional points observed in the current image frame and each historical image frame. The feature point pairs between the current image frame and the second historical image frame are obtained by matching the feature points in the current image frame with the feature points in the second historical image frame. The association relationship between the feature point pair and the structured plane may be the positional relationship between the feature point pair and the structured plane. The pose of the current image frame can be optimized through the positional relationship between the feature point pairs and the structured plane to obtain more accurate results.

In some disclosed embodiments, the structured plane is a three-dimensional plane constructed by using the current image frame captured by the device and the historical image frames before the current image frame. Before step S13 is executed, a structured plane may be constructed using the current image frame and the first historical image frame. A structured plane can be represented parametrically using certain mathematical expressions, for example, using directions and distances. In the embodiment of the present disclosure, the parametric expression Π of the structured plane is:

Π=[n d]

Among them, Π is a four-dimensional vector, n is a three-dimensional vector, representing the direction; d is a constant, representing the distance. The direction here is for the world coordinate system. For example, n can be a set of vectors for the three coordinate axes, or it can be considered as the direction for the origin of the world coordinate system.

The specific way to build a structured plane may include the following steps:

In the first step, the current image frame is triangulated to obtain the corresponding two-dimensional grid group. Wherein, the two-dimensional grid group is composed of several two-dimensional grids. Each vertex in the two-dimensional grid group is a feature point in the current image frame. That is, performing triangulation on the current image frame is actually performing triangulation on the feature points in the current image frame. Before triangulating the current image frame, the feature points in the current image frame can be extracted, and the feature points in the current image frame can be matched with the feature points in the previous historical image frame to obtain the current image frame with the previous historical image frame A pair of feature points between historical image frames. Combining the initial pose of the current image frame, the pose of the last historical image frame, and the two-dimensional coordinates of the feature point pairs in their respective image frames, the three-dimensional points corresponding to the feature point pairs can be determined. Further, only the feature points located in the feature point pairs are triangulated.

The second step is to project the 2D grid group into the world coordinate system to obtain the corresponding 3D grid group. Wherein, the three-dimensional grid group includes several three-dimensional grids. The vertices in the three-dimensional grid group are three-dimensional points corresponding to the feature points in the current image frame. That is, according to the connection relationship between the feature points in the two-dimensional grid group, the connection relationship between the three-dimensional points corresponding to each feature point is determined, so as to obtain the corresponding three-dimensional grid group. The "several numbers" mentioned in the embodiments of the present disclosure may be 1 or more, for example, 2, 3, 10, 20, 30, 50 and so on. Wherein, a three-dimensional grid may include three three-dimensional points.

The third step is to acquire the first three-dimensional grid satisfying the first preset condition in the three-dimensional grid group to generate the structured plane. Specifically, firstly, a three-dimensional grid whose distance from the current image frame is less than or equal to a first preset distance is used as a candidate three-dimensional grid. Because when the distance between the 3D grid and the current image frame exceeds the first preset distance, the coordinate error of the 3D points in the 3D grid may be relatively large. If the 3D grid is used to construct a structured plane, it may cause The constructed structured plane is less accurate. The first preset distance here can be set according to specific scenarios and requirements, and is not specifically regulated here. Next, two or more candidate 3D grids whose direction difference is less than or equal to the first preset difference value, and/or whose distance difference is less than or equal to the second preset difference value are selected as the first 3D grid. Wherein, the direction difference here is less than or equal to the first preset difference means that the difference between the three-dimensional vectors of the two candidate three-dimensional grids is less than or equal to the first preset difference. Further, in the embodiment of the present disclosure, only when the direction difference between two candidate 3D grids is less than or equal to the first preset difference value, and the distance difference is less than or equal to the second preset difference value, Then these two candidate 3D grids are used as the first 3D grids. For example, when two candidate 3D grids are located on the same vertical plane, these two candidate 3D grids may be used as the first 3D grid, and a plane including the two candidate 3D grids may be generated. Of course, if a partial structured plane has been constructed based on several historical image frames before the current image frame, the constructed structured plane can be expanded by utilizing the information in the current image frame on the constructed partially structured plane. At this time, it is also possible to select a candidate three-dimensional grid whose direction difference from the constructed structured plane is less than or equal to the first preset difference value, and/or whose distance difference is less than or equal to the second preset difference value as the first three-dimensional grid, and form a new structured plane with the selected first three-dimensional grid and the constructed structured plane. In this way, the expansion of the structured plane is realized, and the positioning of the current image frame can refer to the previous information, so that the positioning result is more accurate.

Obtain a two-dimensional grid group by triangulating the current image frame, and use the two-dimensional grid group to obtain a three-dimensional grid group, and then obtain the first three-dimensional grid satisfying the first preset condition in the three-dimensional grid group, and The non-arbitrary three-dimensional grid generates a structured plane, which makes the constructed structured plane more precise. Further, by setting the first preset distance and the first preset difference and/or the second preset difference to select the first three-dimensional grid for generating the structured plane, the precision of the structured plane can be improved.

After the structured plane is constructed, the first feature point pair that has an association relationship with the structured plane among the feature point pairs of the current image frame and the second historical image frame is acquired. In some disclosed embodiments, in order to reduce the calculation load of the execution device and ensure the subsequent positioning accuracy, before obtaining the second 3D grid in the 3D grid group that has an associated relationship with the structured plane, the following steps need to be performed, so as to obtain from many A first structured plane that satisfies the second preset condition is selected from the structured planes. Wherein, the second preset condition includes that the distance to the current image frame is less than or equal to a third preset distance threshold. The third preset distance threshold here can be comprehensively determined according to the computing power of the specific execution device and the positioning accuracy requirements. For example, if the positioning accuracy requirement is high, the third preset distance threshold may be relatively small; if the positioning accuracy requirement is low, the third preset distance threshold may be relatively high. For another example, if the computing power of the execution device is weak, the third preset distance threshold may be relatively low; if the computing power of the execution device is strong, the third preset distance threshold may be relatively high. Therefore, no specific provisions are made here for the determination of the third preset distance threshold. By selecting the first structured plane that satisfies the second preset condition from multiple structured planes, the calculation amount in the process of constructing the association relationship can be reduced.

Among the feature point pairs of the current image frame and the second historical image frame, the method of obtaining the first feature point pair that has an association relationship with the structured plane may include the following steps:

Firstly, a second 3D grid that has an association relationship with the structured plane in the 3D grid group is acquired. Specifically, the first distance between each vertex of all three-dimensional meshes in the three-dimensional mesh group and the structured plane is obtained. And select the 3D grid whose first distance between all vertices and the structured plane is less than or equal to the second preset distance as the second 3D grid. That is, a three-dimensional grid includes three vertices, and only when the first distances between the three vertices and the structured plane are all less than or equal to the second preset distance, the three-dimensional grid can be used as the second three-dimensional grid. Alternatively, a three-dimensional grid whose first distance between all vertices and the structured plane is less than or equal to the second preset distance and whose plane is parallel to the structured plane is selected as the second three-dimensional grid. That is, the second three-dimensional grid is parallel to the structured plane, and the distance between all vertices of the second three-dimensional grid and the structured plane is less than or equal to the second preset distance. Optionally, the second preset distance is not fixed, and the second preset distance may be dynamically adjusted according to the distance of the three-dimensional grid or structured plane relative to the current image frame. For example, the distance between the three-dimensional grid or structured plane and the current image frame is proportional to the second preset distance, that is, the second preset distance can fluctuate within a certain range. For example, if the distance between the A three-dimensional grid and the current image frame is 10 meters, the second preset distance can be determined as 0.1 meters, and the distance between the B three-dimensional grid and the current image frame is 15 meters, then the second preset distance can be determined. The preset distance is 0.15 meters. Specifically, a second three-dimensional grid that has an association relationship with the first structured plane in the three-dimensional grid group is acquired. As mentioned above, the first structured plane is selected from a plurality of structured planes.

Second, several first feature point pairs corresponding to the second three-dimensional grid are determined among the feature point pairs of the current image frame and the second historical image frame. There is an association relationship between the first feature point pair and the structured plane.

By constructing the association relationship between the 3D grid and the structured plane to determine the association relationship between the feature point pair and the structured plane, compared with the construction of the association relationship between the 3D point and the structured plane, it can reduce the probability of wrong association .

After acquiring the first feature point pair that has an association relationship with the structured plane between the feature point pairs of the current image frame and the second historical image frame, according to the first feature point pair of the current image frame and the second historical image frame in the world coordinate system A position parameter, the two-dimensional coordinates of the first feature point pair, and the second position parameter of the structured plane are used to construct a point-plane constraint model. The point-surface constraint model includes point-surface optimization equations. The point-surface optimization equation includes the first item and the second item, and the first item and the second item are respectively located on both sides of the equal sign of the point-surface optimization equation. The first position parameter includes a rotation matrix and a translation matrix, and the second position parameter includes a direction matrix and a distance matrix. Among them, the rotation matrix and translation matrix here are relative to the origin of the world coordinate system, the rotation matrix is used to represent the rotation amount of the current image frame or the second historical image frame in the world coordinate system, and the translation matrix is used to represent the current image frame Or the translation amount of the second historical image frame in the world coordinate system.

Among them, the point-surface optimization equation is as follows:

in,

and

are respectively the 3D coordinates of point f in the coordinate system corresponding to the second historical image frame _c1 and the current image frame _c2 , wherein point f here is a feature point in the first feature point pair that has an association relationship with the structured plane;

and

Respectively used to represent the rotation matrix of the second historical image frame c ₁ and the current image frame c ₂ in the world coordinate system W;

and

are respectively used to represent the translation matrices of the second historical image frame _c1 and the current image frame _c2 in the world coordinate system W. n and d are used to represent the direction matrix and distance matrix of the structured plane, respectively, E is the identity matrix, and T represents the transpose. Simplify the point-surface optimization equation to get equation (2):

Since the three-dimensional coordinates are transformed into normalized camera plane coordinates, and the two-dimensional coordinates of each feature point in the corresponding image frame in the feature point pair are known, the above equation (2) can be

and

Convert to the corresponding two-dimensional coordinates

and

From this, the final point-surface optimization equation is obtained. In this way, a point-plane constraint model is constructed by using the feature point pairs between the current image frame and the second historical image frame, and the relationship between the structured plane. Optionally, the second term is the term on the left side of the equal sign in the final point-surface optimization equation, for example, the second term is

The first item is the item on the right side of the equal sign in the final point-surface optimization equation, for example, the first item is the operation result on the right side of the equal sign.

Construct points by using the first position parameter of the current image frame and the second historical image frame in the world coordinate system, the 2D coordinates of the first feature point pair, and the second position parameter of the structured plane, instead of using 3D points The surface constraint model, in the process of optimizing the initial pose by using the point-surface constraint model, is not affected by the accuracy of the three-dimensional point, which effectively improves the positioning accuracy of the device.

Among them, the point-plane constraint model is used to optimize the initial pose, including: according to the rotation matrix and translation matrix of the current image frame and the second historical image frame, the direction matrix and distance matrix of the structured plane, the first feature point is located at The two-dimensional coordinates of the feature points (hereinafter also referred to as historical feature points) in the second historical image frame determine the matching feature points corresponding to the historical feature points in the current image frame (which can be equivalent to the centering of the first feature points) The predicted coordinates of the feature points located in the current image frame). Among them, the predicted coordinate is the first item in the point-surface optimization equation. And adjust the preset parameters in the point-surface optimization equation, so that the first item in the point-surface optimization equation is equal to the second item in the point-surface optimization equation. Wherein, the second item is the real two-dimensional coordinates of the corresponding matching feature points in the current image frame. Wherein, the preset parameters include the initial pose of the current image frame. The initial pose of the current image frame includes the position and orientation of the current image frame in the world coordinate system. Wherein, the position can be represented by the above-mentioned translation matrix, and the orientation can be represented by the above-mentioned rotation matrix. Combined with the above point-surface optimization equation, the result calculated on the right side of the equation will get the two-dimensional predicted coordinates of point f in the current image frame. In theory, the two-dimensional predicted coordinates of point f should be equal to the real two-dimensional coordinates of point f . However, because the initial pose of the current image frame is generally not very accurate, the two-dimensional predicted coordinates of point f are not equal to its two-dimensional coordinates in the current image frame. Therefore, the preset parameters in the point-surface optimization equation can be adjusted according to the difference between the predicted coordinates and the real two-dimensional coordinates, so that the two-dimensional predicted coordinates of the final point f are equal to the real two-dimensional coordinates, or between the two The error is less than or equal to the preset error. Specifically, the preset parameters include a rotation matrix and a translation matrix of the current image frame, and a direction matrix and a distance matrix of the structured plane. Of course, the rotation matrix and translation matrix of the second historical image frame may also be included. By constructing the association relationship between the feature points and the structured plane, optimizing the preset parameters including the initial pose of the current image frame can improve the positioning accuracy of the device.

In some disclosed embodiments, the optimization of the initial pose using the point-plane constraint model further includes: in response to the second historical image frame being the last historical image frame of the current image frame, the pose of the second historical image frame, the current The pose and structured plane of the image frame are optimized to obtain the visual positioning results of the device at two moments. In response to the fact that the second historical image frame is not the last historical image frame of the current image frame, the pose and structured plane of the current image frame are optimized to obtain a visual positioning result of the device at the current moment. Because the earlier the shooting time of the historical image frame is, the more accurate its corresponding pose will be. Similarly, the later the shooting time of the historical image frame, the accuracy of its corresponding pose will be reduced. In the embodiment of the present disclosure, when the second historical image frame is the previous historical image frame of the current image frame, while optimizing the pose and structured plane of the current image frame, the second historical image frame will also be optimized pose. Therefore, in the embodiment of the present disclosure, when the second historical image frame is the previous historical image frame of the current image frame, not only the pose of the current image frame can be optimized, but also the position of the second historical image frame can be optimized. The attitude is optimized to improve the accuracy of the positioning results of the device at each moment.

In the embodiment of the present disclosure, several frames of historical image frames may be used to simultaneously optimize the initial pose of the current image frame. For example, the current image frame is optimized first by using a previous historical image frame of the current image frame, and then the current image frame is optimized by using a previous historical image frame of the historical image frame. Wherein, the historical image frame can be used to optimize the pose of the current image frame only if there are matching feature point pairs between the current image frame and the historical image frame, and the feature point pair has an association relationship with the structured plane. Further, the historical image frames in the sliding window may be used as the second historical image frames. Generally, the latest historical image frame in the sliding window is the previous historical image frame of the current image frame. In some disclosed embodiments, in order to reduce system time consumption, older point-plane constraints are marginalized. That is, the oldest historical image frame in the sliding window does not participate in the optimization process of the pose of the current image frame.

In some disclosed embodiments, using the point-plane constraint model to optimize the initial pose includes: fusing the point-plane constraint model with at least one of the reprojection constraint model and the IMU constraint model to obtain the fusion constraint model. And use the fusion constraint model to optimize the initial pose of the current image frame. By building a fusion constraint model to optimize the pose of the current image frame of the device, the positioning accuracy of the device can be improved.

Wherein, the process of using the re-projection constraint model to constrain the pose of the current image frame mainly includes adjusting the pose of the current image frame by using the re-projection error, so that the re-projection error meets the preset error requirements. The process of using the IMU constraint model to constrain the pose of the current image frame mainly includes using the IMU integral error to optimize the initial pose of the current image frame. In the embodiment of the present disclosure, the form of the fusion constraint model is as follows:

in,

Among them, X represents the amount to be optimized, including the device pose (the device pose corresponding to the current image frame and/or the device pose corresponding to the second historical image frame), IMU parameters, parameters of 3D points and structured planes. r _p is the prior residual, H _p is its corresponding measurement matrix, B is all IMU measurements, and the residual between the IMU measurements at time k and k+1 is

The corresponding covariance matrix is

C is the feature set observed by the device at all times, and the reprojection residual of the device at point l at time j is

The corresponding covariance matrix is

P is the set of all structured planes. The homography-based plane residual of point l under structured plane k at time i and j of the device is

The corresponding covariance matrix is

In order to better understand the technical solutions proposed by the embodiments of the present disclosure, please refer to FIG. 2 , which is a schematic flowchart of another embodiment of the positioning method of the present application. In the embodiment of the present disclosure, the positioning method includes the following steps:

Step S21: Obtain the current image frame captured by the device.

Wherein, the manner of obtaining the current image frame is as in the above step S11, which will not be repeated here.

Step S22: Determine the initial pose of the current image frame according to the relative positional relationship between the current image frame and the first historical image frame.

Wherein, the manner of determining the initial pose of the current image frame is as in the above-mentioned step S12, which will not be repeated here.

Step S23: Generate a structured plane using the current image frame.

Specifically, the following steps may be included: 1. Perform triangulation on the current image frame to obtain a corresponding two-dimensional grid group. Wherein, the vertices in the two-dimensional grid group are feature points in the current image frame. The second is to project the two-dimensional grid group into the world coordinate system to obtain the corresponding three-dimensional grid group. Wherein, the vertices in the three-dimensional grid group are three-dimensional points corresponding to the feature points in the current image frame. The third is to obtain the first three-dimensional grid satisfying the first preset condition in the three-dimensional grid group to generate the structured plane. Wherein, for the three steps here, reference may be made to the foregoing embodiments in detail, and details are not repeated here.

Step S24: Obtain the first feature point pair that has an association relationship between the feature point pairs of the current image frame and the second historical image frame and the structured plane.

The specific manner of obtaining the first feature point pair that has an association relationship with the structured plane between the feature point pairs of the current image frame and the second historical image frame is as described above, and will not be repeated here.

Step S25: Construct a point-plane constraint model according to the first position parameter of the current image frame and the second historical image frame in the world coordinate system, the two-dimensional coordinates of the feature point pairs, and the second position parameter of the structured plane.

Wherein, the point-plane constraint model and the way of constructing the point-plane constraint model are as above, and will not be repeated here.

Step S26: Use the point-plane constraint model to optimize the initial pose, and obtain the optimized pose as the visual positioning result of the device.

Among them, the method of optimizing the initial pose by using the point-plane constraint model, and obtaining the optimized pose as the visual positioning result of the device is as described above, and will not be repeated here.

In the above scheme, a point-plane constraint model is constructed through the feature point pairs between the current image frame and the second historical image frame, and the relationship between the structured plane, and the initial pose of the device is optimized through the point-plane constraint model , so that the 3D point parameters are not included in the point-plane constraint model, so that in the process of optimizing the initial pose using the point-plane constraint model, it will not be affected by the accuracy of the 3D point, thereby improving the positioning accuracy of the device.

Those skilled in the art can understand that in the above method of specific implementation, the writing order of each step does not mean a strict execution order and constitutes any limitation on the implementation process. The specific execution order of each step should be based on its function and possible The inner logic is OK.

The execution subject of the positioning method may be a positioning device. For example, the positioning method may be executed by a terminal device or a server or other processing equipment, wherein the terminal device may be a user equipment (User Equipment) that has requirements for visual positioning, three-dimensional reconstruction, image registration, etc. , UE), mobile devices, user terminals, terminals, cellular phones, cordless phones, Personal Digital Assistant (PDA), handheld devices, computing devices, vehicle-mounted devices, wearable devices, and self-driving cars. Robots with graphic needs, medical imaging systems with registration needs, glasses, helmets and other products for augmented reality or virtual reality, etc. In some possible implementation manners, the positioning method may be implemented by a processor calling computer-readable instructions stored in a memory.

Please refer to FIG. 3 . FIG. 3 is a schematic structural diagram of an embodiment of the positioning device of the present application. The positioning device 30 includes an image acquisition module 31 , an initial pose acquisition module 32 and a pose optimization module 33 . The image acquisition module 31 is used to acquire the current image frame captured by the device; the initial pose acquisition module 32 is used to determine the initial pose of the current image frame according to the relative positional relationship between the current image frame and the first historical image frame; The pose optimization module 33 is used to optimize the initial pose using the point-plane constraint model, and obtain the optimized pose as the visual positioning result of the device; wherein, the point-plane constraint model uses the current image frame and the second historical image frame It is constructed by the feature point pairs between them and the association relationship with the structured plane.

In some disclosed embodiments, the structured plane is a three-dimensional plane constructed by using the current image frame captured by the device and the historical image frames before the current image frame; the pose optimization module 33 optimizes the initial pose using a point-plane constraint model to obtain Before the optimized pose, it is also used to: obtain the first feature point pair that has an association relationship between the feature point pair of the current image frame and the second historical image frame and the structured plane; according to the current image frame and the second historical image frame The first position parameter of the image frame in the world coordinate system, the two-dimensional coordinates of the first feature point pair, and the second position parameter of the structured plane construct a point-plane constraint model.

In the above scheme, by using the first position parameter of the current image frame and the second historical image frame in the world coordinate system, the two-dimensional coordinates of the first feature point pair, and the second position parameter of the structured plane, instead of using three-dimensional points, To build a point-plane constraint model, in the process of optimizing the initial pose using the point-plane constraint model, it is not affected by the three-dimensional point accuracy, which effectively improves the positioning accuracy of the device.

In some disclosed embodiments, the point-plane constraint model includes a point-plane optimization equation, and the point-plane optimization equation includes a first term and a second term, and the first term and the second term are respectively located on both sides of the equal sign of the point-plane optimization equation; The position parameters include a rotation matrix and a translation matrix, and the second position parameters include a direction matrix and a distance matrix; the pose optimization module 33 optimizes the initial pose using the point-plane constraint model, including: based on the current image frame and the second historical image frame The rotation matrix and translation matrix, the direction matrix and the distance matrix of the structured plane, the two-dimensional coordinates of the historical feature points in the second historical image frame in the first feature point pair, determine the corresponding historical feature point in the current image frame Match the predicted coordinates of the feature points, where the predicted coordinates are the first item in the point-surface optimization equation; adjust the preset parameters in the point-surface optimization equation so that the first item in the point-surface optimization equation is the same as that in the point-surface optimization equation The second term of is equal, where the second term is the real two-dimensional coordinates of the corresponding matching feature points in the current image frame, and the preset parameters include the initial pose of the current image frame.

In the above solution, by constructing the association relationship between the feature points and the structured plane, optimizing the preset parameters including the initial pose of the current image frame, the positioning accuracy of the device can be improved.

In some disclosed embodiments, the pose optimization module 33 uses the point-plane constraint model to optimize the initial pose, and the step of obtaining the optimized pose as the visual positioning result of the device includes: responding to the second historical image frame being the current image The last historical image frame of the frame, the pose of the second historical image frame, the pose of the current image frame, and the structured plane are optimized to obtain the visual positioning results of the device at two moments; in response to the second historical image frame not being For the previous historical image frame of the current image frame, the pose and structured plane of the current image frame are optimized to obtain the visual positioning result of the device at the current moment.

The above solution, in the case that the second historical image frame is the last historical image frame of the current image frame, can not only optimize the pose of the current image frame, but also optimize the pose of the second historical image frame, thereby Improve the accuracy of the positioning results of the device at each time.

In some disclosed embodiments, before the pose optimization module 33 acquires the first feature point pair that has an association relationship between the feature point pair of the current image frame and the second historical image frame and the structured plane, it includes: performing Triangulation to obtain the corresponding two-dimensional grid group, wherein, the two-dimensional grid group is composed of several two-dimensional grids, and each vertex in the two-dimensional grid group is a feature point in the current image frame; The grid group is projected to the world coordinate system to obtain the corresponding 3D grid group, wherein the 3D grid group includes several 3D grids, and the vertices in the 3D grid group are the 3D points corresponding to the feature points in the current image frame. point; acquiring the first 3D grid satisfying the first preset condition in the 3D grid group to generate the structured plane.

In the above solution, the two-dimensional mesh group is obtained by triangulating the current image frame, and the three-dimensional mesh group is obtained by using the two-dimensional mesh group, and then the first three-dimensional network that satisfies the first preset condition in the three-dimensional mesh group is obtained. A structured plane is generated using a grid instead of an arbitrary three-dimensional grid, so that the precision of the constructed structured plane is higher.

In some disclosed embodiments, the three-dimensional mesh group includes several three-dimensional meshes. The pose optimization module 33 obtains the first three-dimensional grid satisfying the first preset condition in the three-dimensional grid group to generate a structured plane, including: dividing the three-dimensional grid whose distance from the current image frame is less than or equal to the first preset distance As a candidate three-dimensional grid; select two or more candidate three-dimensional grids whose difference in direction is less than or equal to the first preset difference, and/or the difference in distance is less than or equal to the second preset difference, as the first three-dimensional network grid.

In the solution above, by setting the first preset distance and the first preset difference and/or the second preset difference to select the first three-dimensional grid to generate the structured plane, the precision of the structured plane can be improved.

In some disclosed embodiments, the pose optimization module 33 acquires the first feature point pair that has an association relationship between the feature point center of the current image frame and the second historical image frame and the structured plane, including: acquiring A second three-dimensional grid having an association relationship with the structured plane; determining a number of first feature point pairs corresponding to the second three-dimensional grid among the feature point pairs of the current image frame and the second historical image frame, the first feature point pairs There is an associative relationship with the structured plane.

The above scheme determines the relationship between the feature point pair and the structured plane by constructing the relationship between the 3D grid and the structured plane, which can reduce errors compared to building the relationship between the 3D point and the structured plane. probability of association.

In some disclosed embodiments, the pose optimization module 33 acquires the second 3D grid that has an association relationship with the structured plane in the 3D grid group, including: acquiring vertices and structured planes of all 3D grids in the 3D grid group The first distance between; select the 3D grid whose first distance between all vertices and the structured plane is less than or equal to the second preset distance as the second 3D grid; or select the distance between all vertices and the structured plane The first distance is less than or equal to the second preset distance, and the three-dimensional grid whose plane composed of all vertices is parallel to the structured plane is used as the second three-dimensional grid.

In the above scheme, by selecting a three-dimensional grid whose distance from the structured plane is smaller than the second preset distance, and then constructing an association relationship between the three-dimensional grid and the structured plane, the accuracy of the association can be improved.

In some disclosed embodiments, before the pose optimization module 33 acquires the second 3D grid that has an association relationship with the structured plane in the 3D grid group, it includes: selecting the first 3D grid that satisfies the second preset condition from multiple structured planes A structured plane, the second preset condition includes that the distance between the current image frame and the current image frame is less than or equal to the third preset distance threshold; obtaining the second three-dimensional grid in the three-dimensional grid group that has an association relationship with the structured plane includes : Obtain the second 3D grid that has an association relationship with the first structured plane in the 3D grid group.

In the above solution, by selecting the first structured plane that satisfies the second preset condition from multiple structured planes, the calculation amount in the process of building the association relationship can be reduced.

In some disclosed embodiments, the pose optimization module 33 uses the point-plane constraint model to optimize the initial pose, including: fusing the point-plane constraint model with at least one of the reprojection constraint model and the IMU constraint model to obtain the fusion constraint model; The initial pose is optimized using the fused constraint model.

In the above solution, the pose of the current image frame of the device is optimized by constructing a fusion constraint model, which can improve the positioning accuracy of the device.

Please refer to FIG. 4 . FIG. 4 is a schematic structural diagram of an embodiment of an electronic device of the present application. The electronic device 40 includes a memory 41 and a processor 42, and the processor 42 is configured to execute program instructions stored in the memory 41, so as to realize the steps in the above positioning method embodiments. In a specific implementation scenario, the electronic device 40 may include, but is not limited to: a microcomputer and a server. In addition, the electronic device 40 may also include mobile devices such as notebook computers and tablet computers, which are not limited here.

Specifically, the processor 42 is used to control itself and the memory 41 to implement the steps in the above positioning method embodiments. The processor 42 may also be called a CPU (Central Processing Unit, central processing unit). The processor 42 may be an integrated circuit chip with signal processing capability. The processor 42 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. Wherein, the general-purpose processor may be a microprocessor or the processor may be any conventional processor or the like. In addition, the processor 42 may also be jointly realized by an integrated circuit chip.

Please refer to FIG. 5 . FIG. 5 is a schematic structural diagram of an embodiment of a computer-readable storage medium of the present application. The computer-readable storage medium 50 stores program instructions 501 that can be executed by the processor, and the program instructions 501 are used to implement the steps in the above positioning method embodiments when executed by the processor.

In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the methods described in the method embodiments above, and its specific implementation can refer to the description of the method embodiments above. For brevity, here No longer.

The above descriptions of the various embodiments tend to emphasize the differences between the various embodiments, the same or similar points can be referred to each other, and for the sake of brevity, details are not repeated herein.

In the several embodiments provided in this application, it should be understood that the disclosed methods and devices may be implemented in other ways. For example, the device implementations described above are only illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, units or components can be combined or integrated. to another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) execute all or part of the steps of the methods in various embodiments of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disc, etc., which can store program codes. .

Claims

A positioning method, characterized in that, comprising:

Get the current image frame captured by the device;

determining the initial pose of the current image frame according to the relative positional relationship between the current image frame and the first historical image frame;

Optimizing the initial pose by using a point-plane constraint model, and obtaining the optimized pose as a visual positioning result of the device;

Wherein, the point-plane constraint model is constructed using the feature point pairs between the current image frame and the second historical image frame, and the association relationship between the structured plane.
The method according to claim 1, further comprising:

Obtaining a first feature point pair that has an association relationship with the structured plane among the feature point pairs of the current image frame and the second historical image frame;

According to the first position parameter of the current image frame and the second historical image frame in the world coordinate system, the two-dimensional coordinates of the first feature point pair, and the second position parameter of the structured plane, construct the The point-surface constraint model,

Wherein, the point-plane constraint model includes a point-plane optimization equation,

The point-surface optimization equation includes a first item and a second item, and the first item and the second item are respectively located on both sides of the equal sign of the point-surface optimization equation;

The first position parameter includes a rotation matrix and a translation matrix,

The second position parameter includes a direction matrix and a distance matrix.
The method according to claim 2, said utilizing a point-surface constraint model to optimize said initial pose, comprising:

According to the rotation matrix and translation matrix of the current image frame and the second historical image frame, the direction matrix and the distance matrix of the structured plane, the first feature point pair is located in the second historical image frame The two-dimensional coordinates of the historical feature points, determining the predicted coordinates of the matching feature points corresponding to the historical feature points in the current image frame, wherein the predicted coordinates are used as the first item;

Adjusting the preset parameters in the point-plane optimization equation, so that the first term is equal to the second term, wherein the second term is the true value of the matching feature point in the current image frame Two-dimensional coordinates, the preset parameters include the initial pose of the current image frame.
The method according to claim 2, wherein the initial pose is optimized using a point-plane constraint model, and the optimized pose is obtained as a visual positioning result of the device, comprising:

In response to the second historical image frame being a previous historical image frame of the current image frame, optimizing the pose of the second historical image frame, the pose of the current image frame, and the structured plane , to obtain the visual positioning results of the device at two moments;

In response to the fact that the second historical image frame is not the previous historical image frame of the current image frame, optimizing the pose of the current image frame and the structured plane to obtain the visual positioning results.
The method according to claim 2, further comprising:

The structured plane is constructed by using the current image frame and historical image frames preceding the current image frame.
The method according to claim 5, wherein constructing the structured plane using the current image frame and historical image frames before the current image frame comprises:

performing triangulation on the current image frame to obtain a two-dimensional grid group comprising a plurality of two-dimensional grids, wherein the vertices in the two-dimensional grid group are feature points in the current image frame;

Projecting the two-dimensional grid group into the world coordinate system to obtain a three-dimensional grid group including a plurality of three-dimensional grids, wherein the vertices of each of the three-dimensional grids are corresponding to the feature points in the current image frame 3D point;

generating the structured plane based on two or more first three-dimensional grids satisfying a first preset condition in the three-dimensional grid group,

Wherein, the first preset condition includes: the distance between the two or more first three-dimensional grids and the current image frame is less than or equal to the first preset distance, and the distance between the two or more first three-dimensional grids The direction difference between them is less than or equal to the first preset difference, and/or the distance difference is less than or equal to the second preset difference.
The method according to claim 6, wherein the generating the structured plane based on two or more first three-dimensional grids satisfying a first preset condition in the three-dimensional grid group comprises:

Using a 3D grid whose distance from the current image frame in the 3D grid group is less than or equal to the first preset distance as a candidate 3D grid;

Select two or more candidate three-dimensional grids whose direction difference is less than or equal to the first preset difference, and/or whose distance difference is less than or equal to the second preset difference, or with the constructed Candidate three-dimensional grids whose direction difference between structured planes is less than or equal to the first preset difference value, and/or whose distance difference is less than or equal to the second preset difference value, are used as the first 3D grid;

A structured plane is generated based on the first three-dimensional grid, or a new structured plane is formed from the constructed structured plane.
The method according to claim 6, wherein said acquiring the first feature point that has an association relationship between the feature point pair of the current image frame and the second historical image frame and the structured plane Yes, including:

Acquiring a second 3D grid in the 3D grid group that has an associated relationship with the structured plane;

Determining several first feature point pairs corresponding to the second three-dimensional grid among the feature point pairs of the current image frame and the second historical image frame.
The method according to claim 8, wherein the acquiring the second 3D grid that has an association relationship with the structured plane in the 3D grid group comprises:

Obtain a first distance between each vertex of all 3D meshes in the 3D mesh group and the structured plane;

A three-dimensional grid whose first distances between all vertices and the structured plane are less than or equal to a second preset distance is selected as the second three-dimensional grid.
The method according to claim 8, wherein the acquiring the second 3D grid that has an association relationship with the structured plane in the 3D grid group comprises:

Obtain a first distance between each vertex of all 3D meshes in the 3D mesh group and the structured plane;

Obtain a first distance between each vertex of all 3D meshes in the 3D mesh group and the structured plane;

A three-dimensional grid whose first distance between all vertices and the structured plane is less than or equal to a second preset distance and whose plane is parallel to the structured plane is selected as the second three-dimensional grid.
The method according to claim 8, wherein the acquiring the second 3D grid that has an association relationship with the structured plane in the 3D grid group comprises:

Selecting a first structured plane that satisfies a second preset condition from the structured planes, the second preset condition includes that the distance from the current image frame is less than or equal to a third preset distance threshold;

Acquiring a second 3D grid in the 3D grid group that has an association relationship with the first structured plane.
The method according to any one of claims 1 to 11, wherein the optimization of the initial pose using a point-surface constraint model includes:

Fusing the point-plane constraint model with at least one of the reprojection constraint model and the IMU constraint model to obtain a fusion constraint model;

The initial pose is optimized by using the fusion constraint model.
A positioning device is characterized in that it comprises:

An image acquisition module, configured to acquire the current image frame captured by the device;

An initial pose acquisition module, configured to determine the initial pose of the current image frame according to the relative positional relationship between the current image frame and the first historical image frame;

A pose optimization module, configured to optimize the initial pose using a point-plane constraint model, and obtain the optimized pose as a visual positioning result of the device; wherein, the point-plane constraint model utilizes the current image The feature point pairs between the frame and the second historical image frame, and the association relationship between the structured plane are constructed.
An electronic device, characterized by comprising a memory and a processor, the processor is configured to execute program instructions stored in the memory, so as to implement the method according to any one of claims 1 to 12.
A computer-readable storage medium on which program instructions are stored, wherein the method according to any one of claims 1 to 12 is implemented when the program instructions are executed by a processor.