CN113361365B

CN113361365B - Positioning method, positioning device, positioning equipment and storage medium

Info

Publication number: CN113361365B
Application number: CN202110606804.6A
Authority: CN
Inventors: 王帅; 陈丹鹏; 王楠; 周立阳; 章国锋
Original assignee: Zhejiang Shangtang Technology Development Co Ltd
Current assignee: Zhejiang Shangtang Technology Development Co Ltd
Priority date: 2021-05-27
Filing date: 2021-05-27
Publication date: 2023-06-23
Anticipated expiration: 2041-05-27
Also published as: WO2022247548A1; CN113361365A

Abstract

The application discloses a positioning method, a positioning device, positioning equipment and a storage medium. The positioning method comprises the following steps: acquiring an a priori structured plane constructed by a plurality of image frames; acquiring a current image frame shot by equipment; determining a first association relationship between the prior structured plane and a matching feature point pair between the current image frame and at least one historical image frame; and acquiring a positioning result of the equipment by using the first association relation. By means of the scheme, the positioning accuracy of the equipment can be improved.

Description

Positioning method, positioning device, positioning equipment and storage medium

Technical Field

The present disclosure relates to the field of positioning technologies, and in particular, to a positioning method and apparatus, a device, and a storage medium.

Background

Visual localization techniques play an important role in many fields, such as in the field of robotics, etc. Among other things, in visual positioning techniques, structured planes are often utilized to optimize the positioning of the device. The structured plane refers to a three-dimensional plane established from the constructed three-dimensional points. However, it is common practice to construct a structured plane in real time and then obtain a more accurate positioning result of the device by using the relationship between the structured plane and the three-dimensional points constructed in real time. Drawbacks of this approach include the limited precision and speed of the planes currently being built in real-time, and the inability to build planes in real-time in environments with weaker textures, affecting the positioning of the device.

Disclosure of Invention

The application provides at least one positioning method, device, equipment and storage medium.

The application provides a positioning method, which comprises the following steps: acquiring an a priori structured plane constructed by a plurality of image frames; acquiring a current image frame shot by equipment; determining a first association relationship between the prior structured plane and a matching feature point pair between the current image frame and at least one historical image frame; and acquiring a positioning result of the equipment by using the first association relation.

Thus, by acquiring a constructed a priori structured plane before acquiring the current image frame, the former is more accurate than a structured plane constructed in real time. And the positioning result of the equipment is obtained by constructing the association relation between the two-dimensional characteristic points in the current image frame and the historical image frame and the prior structured plane, so that the positioning precision of the equipment can be improved.

Wherein determining a first association between the prior structured plane and the matching feature point pairs between the current image frame and at least one historical image frame comprises: respectively acquiring position parameters of three-dimensional points corresponding to the matched characteristic point pairs and the prior structured plane under the same coordinate system; performing point-to-surface matching on the three-dimensional points and the prior structured plane based on the position parameters so as to obtain a second association relationship between the three-dimensional points and the prior structured plane; obtaining a first association relation according to the second association relation; the same coordinate system is a first coordinate system or a second coordinate system, the first coordinate system is a world coordinate system corresponding to the prior structured plane, and the second coordinate system is a world coordinate system corresponding to the current image frame, wherein the first coordinate system and the second coordinate system are the same or different.

Therefore, the construction of the second association relationship is facilitated by acquiring the three-dimensional points corresponding to the matched feature point pairs and the position parameters of the priori structured plane in the same coordinate system, so that the first association relationship is obtained.

Wherein the same coordinate is a first coordinate system; before performing point-to-face matching on the three-dimensional points and the prior structured plane based on the position parameters, the method comprises the following steps: acquiring a first distance between the three-dimensional point and a current image frame, selecting a three-dimensional point with the first distance smaller than or equal to a first preset distance as a first three-dimensional point, wherein the first three-dimensional point is used for carrying out point-plane matching with a priori structured plane; and/or acquiring a first pose of the current image frame under a first coordinate system; and selecting a priori structured plane with a second distance smaller than or equal to a second preset distance from the current image frame as a first priori structured plane, wherein the first priori structured plane is used for carrying out point-to-surface matching with the three-dimensional points to obtain a second association relation.

Therefore, because the number of the prior structured planes is better than the number of the corresponding three-dimensional points of the matched characteristic point pairs, the workload of the equipment is reduced by acquiring the position parameters of the three-dimensional points under the first coordinate system so as to construct the second association relation under the first coordinate system. And the three-dimensional points and the prior structured plane are screened by utilizing the current image frame, so that the three-dimensional points and/or the prior structured plane which are closer to the current image frame are selected, the workload is reduced, and the accuracy of point-plane matching can be improved.

Before the first distance between the three-dimensional point and the current image frame is acquired, the method comprises the following steps: triangulating the current image frame to obtain a two-dimensional grid group related to the current image frame, wherein vertexes in the two-dimensional grid group are characteristic points; projecting the two-dimensional grid set to a second coordinate system to obtain a three-dimensional grid set related to the current image frame, wherein the three-dimensional grid set comprises a plurality of three-dimensional grids, and each vertex in the three-dimensional grids is the three-dimensional point; the distance between the three-dimensional grid and the current image frame is taken as a first distance.

Therefore, by constructing the association relationship between the three-dimensional grid and the prior structured plane, the association relationship obtained by the three-dimensional grid is more accurate and has higher precision than the association relationship between the single three-dimensional point and the prior structured plane.

The method for performing point-plane matching on the three-dimensional points and the prior structured plane based on the position parameters comprises the following steps: obtaining third distances between vertexes of all three-dimensional grids in the three-dimensional grid group and the prior structured plane; selecting three-dimensional grids with first distances between all vertexes and the priori structured plane smaller than or equal to a third preset distance as first three-dimensional grids; or selecting a three-dimensional grid, wherein the third distance between all vertexes and the prior structured plane is smaller than or equal to a third preset distance, and the plane formed by all vertexes is parallel to the prior structured plane, as a first three-dimensional grid; and constructing an association relationship between the first three-dimensional grid and the prior structured plane as a second association relationship.

Therefore, the association relationship between the three-dimensional grid and the prior structured plane is built only by selecting that the third distance between the vertex and the prior structured plane in the three-dimensional grid is smaller than or equal to the third preset distance, or further under the condition that the three-dimensional grid is parallel to the prior structured plane, so that the finally built association relationship is more accurate.

The step of obtaining the position parameters of the three-dimensional points corresponding to the matched characteristic point pairs and the prior structured plane under the same coordinate system, or obtaining the first pose of the current image frame under the first coordinate system comprises the following steps: acquiring position parameters of the three-dimensional points under a second coordinate system, and converting the position parameters into position parameters under a first coordinate system by utilizing conversion parameters; or acquiring a second pose of the current image frame under the second coordinate system, and converting the second pose into a first pose under the first coordinate system by utilizing the conversion parameters.

Therefore, by first acquiring the coordinates of the three-dimensional point and the coordinates of the current image frame in the second coordinate system and converting the coordinates into the first coordinate system by using the conversion parameters, the speed of acquiring the position parameters of the three-dimensional point and the current image frame in the first coordinate system can be increased.

Wherein after determining a first association between the matching feature point pairs between the current image frame and the at least one historical image frame and the prior structured plane, the method comprises: constructing a point-plane constraint model by using a first association relation, wherein the point-plane constraint model is used for acquiring a positioning result of equipment, the point-plane constraint model comprises a point-plane optimization equation, the point-plane optimization equation comprises a first term and a second term, and the first term and the second term are respectively positioned on two sides of an equal sign of the point-plane optimization equation; obtaining a positioning result of the device by using the first association relation, including: determining prediction coordinates of corresponding feature points in the current image frame according to the rotation matrix and the translation matrix of the current image frame and the historical image frame, the direction matrix and the distance matrix of the structured plane, and the two-dimensional coordinates of the feature points in the historical image frame, wherein the prediction coordinates are used as a first item; and adjusting preset parameters in the point-plane optimization equation to enable the first term to be equal to the second term, wherein the second term is two-dimensional coordinates of corresponding feature points in the current image frame, and the preset parameters comprise a rotation matrix and a translation matrix of the current image frame.

Therefore, by constructing the association relation between the two-dimensional feature points and the structural plane and optimizing the preset parameters including the initial pose of the current image frame, the positioning precision of the equipment can be improved.

The method for constructing the point-surface constraint model by utilizing the first association relation comprises the following steps: fusing the point-plane constraint model with at least one of the re-projection constraint model and the IMU constraint model to obtain a fused constraint model; the fusion constraint model is used for obtaining a positioning result of the equipment.

Therefore, the pose of the current image frame of the equipment is optimized by constructing the fusion constraint model, and the positioning precision of the equipment can be improved.

The historical image frames are any historical image frame before the current image frame; obtaining a positioning result of the device by using the first association relation, including: responding to the historical image frame which is the last historical image frame of the current image frame, acquiring a positioning result corresponding to the current image frame of the equipment and updating the positioning result corresponding to the historical image frame of the equipment; and responding to the historical image frame which is not the last historical image frame of the current image frame, and acquiring a positioning result corresponding to the current image frame of the equipment.

Therefore, when the historical image is the last historical image frame of the current image frame, not only the pose of the current image frame can be optimized, but also the pose of the historical image frame can be optimized, so that the accuracy of the positioning results of the equipment at a plurality of moments is improved.

The application provides a positioning device, including: the plane acquisition module is used for acquiring a priori structured plane constructed by a plurality of image frames; the image acquisition module is used for acquiring a current image frame shot by the equipment; the association relation determining module is used for determining a first association relation between a matching characteristic point pair between a current image frame and at least one historical image frame and an priori structured plane, wherein the priori structured plane is constructed before the current image frame shot by the acquisition equipment is executed; and the positioning module is used for acquiring a positioning result of the equipment by utilizing the first association relation.

The application provides an electronic device comprising a memory and a processor for executing program instructions stored in the memory to implement the above-mentioned positioning method.

The present application provides a computer readable storage medium having stored thereon program instructions which, when executed by a processor, implement the above-described positioning method.

By acquiring the constructed prior structured plane before acquiring the current image frame, the scheme has higher precision compared with the structured plane constructed in real time. And the positioning result of the equipment is obtained by constructing the association relation between the two-dimensional characteristic points in the current image frame and the historical image frame and the prior structured plane, so that the positioning precision of the equipment can be improved.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and, together with the description, serve to explain the technical aspects of the application.

FIG. 1 is a flow chart of an embodiment of a positioning method of the present application;

FIG. 2 is a second flow chart of an embodiment of a positioning method according to the present application;

FIG. 3 is a schematic view of an embodiment of a positioning device of the present application;

FIG. 4 is a schematic diagram of an embodiment of an electronic device of the present application;

fig. 5 is a schematic structural view of an embodiment of the computer-readable storage medium of the present application.

Detailed Description

The following describes the embodiments of the present application in detail with reference to the drawings.

In the following description, for purposes of explanation and not limitation, specific details are set forth such as the particular system architecture, interfaces, techniques, etc., in order to provide a thorough understanding of the present application.

The term "and/or" is herein merely an association relationship describing an associated object, meaning that there may be three relationships, e.g., a and/or B, may represent: a exists alone, A and B exist together, and B exists alone. In addition, the character "/" herein generally indicates that the front and rear associated objects are an "or" relationship. Further, "a plurality" herein means two or more than two. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, C, and may mean including any one or more elements selected from the group consisting of A, B and C.

Referring to fig. 1, fig. 1 is a flowchart illustrating an embodiment of a positioning method according to the present application. Specifically, the positioning method may include the steps of:

step S11: an a priori structured plane constructed from a number of image frames is acquired.

The a priori structured planes may be constructed by devices that perform the positioning methods provided by embodiments of the present disclosure, or by other devices. No specific provisions are made herein regarding the apparatus that constructs the a priori structured planes. In the process of constructing the prior structured plane, the environmental image can be shot again under the condition that the moving speed and the rotating speed of the equipment are 0 or approach to 0, and the quality of the environmental image acquired by the method is higher than that of the environmental image shot by the equipment in the moving process, so that the precision is higher according to the prior structured plane obtained by the method relative to the prior structured plane obtained by the method. The specific form of the prior structured plane may be a dense three-dimensional map or a sparse three-dimensional map, and is not specifically defined herein. The embodiments of the present disclosure take a priori structured planes as examples.

The method for obtaining the prior structured plane may be to obtain the storage position of the prior structured plane in the device by reading the storage information of the device, or obtain the prior structured plane from other devices by means of communication connection or the like, or construct the prior structured plane by using the execution device to shoot the multi-frame image frames of the environment. Thus, the manner in which the a priori structured planes are obtained is not specifically defined herein.

Step S12: the current image frame taken by the device is acquired.

The device for performing the positioning method according to the embodiment of the present disclosure may be a device for capturing a current image frame, or may not be a device for capturing a current image frame. For example, the execution device may acquire the current image frame by establishing a communication connection with the device that captured the current image frame, or the like. In the case where the execution device and the device that captured the current image frame are not the same device, the manner of communication connection between the two is not limited. The embodiment of the present disclosure takes the same device as the device for capturing the current image frame as an example. The device in the embodiment of the disclosure comprises a device for shooting a current image frame and a sensor. Wherein the sensor is for measuring motion information of the device. The current image frame may be an image acquired in real time and not subjected to any image processing, or may be subjected to image processing. The image processing here may be clipping, data enhancement, or the like.

In some disclosed embodiments, in order to ensure positioning accuracy of the device, not all image frames photographed by the device are performed as current image frames, and some image frames with lower quality are not used as current image frames photographed by the device. The manner of judging whether the image frame shot by the device becomes the current image frame may be as follows: 1. extracting characteristic points in the image frame, and taking the image frame as a current image frame under the condition that the number of the characteristic points is larger than or equal to a first preset number; 2. the number of the matched characteristic point pairs between the image frame and the historical image frame in the preset time period is acquired, and the image frame is taken as the current image frame when the number of the matched characteristic points is larger than or equal to the second preset number. Of course, it is also possible to judge whether the sharpness, brightness, and the like of the image satisfy the requirements, in addition to the two conditions listed here. By selecting the image frame meeting the quality requirement as the current image frame, the situation that the positioning result is too bad due to poor quality of the image frame can be reduced, and the positioning precision of equipment is ensured.

Step S13: a first association between pairs of matching feature points between the current image frame and at least one historical image frame and an a priori structured plane is determined.

Before acquiring the first association, an initial pose of the current image frame needs to be acquired. In the embodiment of the present disclosure, the manner of acquiring the initial pose of the current image frame may be: the relative positional relationship between the current image frame and a historical image frame is first acquired. The relative positional relationship includes a relative distance and a relative angle. In the disclosed embodiment, a distance is assumed as the relative distance between the current image frame and the historical image frame, and the sensor readings between the current image frame and the historical image frame are pre-integrated to obtain the relative angle therebetween. Since the pose of the historical image frame is known, the initial pose of the current image frame can be determined through the relative distance and the relative angle between the historical image frame and the historical image frame.

Wherein the history image is an image frame that has undergone positioning processing, such as a frame immediately preceding the current image frame. The historical image frames may be one frame or multiple frames. For example, when the number of the historical image frames is 3, a first association relationship between the matching feature point pairs between the current image frame and the three historical image frames and the prior structured plane is respectively obtained, so that the pose of the current image frame is obtained through the information of the multi-frame historical image frames, namely the positioning result of the current image frame is obtained.

Step S14: and acquiring a positioning result of the equipment by using the first association relation.

And according to a first association relation between the matching characteristic point pairs between the current image frame and at least one frame of historical image frame and the prior structured plane. In particular, there may be various ways of obtaining the positioning result of the device by using the first association relationship, for example, the first association relationship between the matching feature point pair between the current image frame and at least one frame of historical image frame and the prior structured plane, and the two-dimensional coordinates of the feature point in the current image frame, determining the predicted coordinates of the feature point corresponding to the feature point in the historical image frame, and obtaining the pose of the current image frame more accurately by obtaining the difference between the predicted coordinates and the real coordinates.

In some disclosed embodiments, the manner of determining the first association between the matching pairs of feature points between the current image frame and the at least one historical image frame and the a priori structured plane may be: and respectively acquiring the position parameters of the three-dimensional points corresponding to the matched characteristic point pairs and the prior structured plane under the same coordinate system. Because the a priori structured plane is already constructed before the positioning method provided by the embodiments of the present disclosure is performed, a world coordinate system applicable to the a priori structured plane is constructed at the same time when the a priori structured plane is constructed, for example, the a point in the a priori structured plane is used as the origin of the world coordinate system, and a corresponding world coordinate system is constructed. In the motion process of the device, the first frame of image shot by the device is generally taken as the origin of the world coordinate system, and the world coordinate system is constructed. If the first frame of image shot by the device is just coincident with the A point of the prior structured plane, the two world coordinates are actually the same coordinate system, and similarly, if the two world coordinates are not coincident, the two world coordinate systems are considered to be different. Generally, when the first frame of image taken by the device is acquired, its position in the world coordinate system corresponding to the a priori structured plane is not known. In the embodiment of the disclosure, the first coordinate system is used as a world coordinate system corresponding to the prior structured plane, and the second coordinate system is used as a world coordinate system corresponding to the current image frame. The first coordinate systems are the same or different. The same coordinate system may refer to a first coordinate system or a second coordinate system. And the construction of the second association relationship is facilitated by acquiring the three-dimensional points corresponding to the matched feature point pairs and the position parameters of the priori structured plane in the same coordinate system, so that the first association relationship is obtained.

In some disclosed embodiments, the same coordinate system refers to a first coordinate system. Acquiring three-dimensional points corresponding to matching feature point pairs and priori structured planesThe way of the position parameters under the same coordinate system comprises: and acquiring the position parameters of the three-dimensional points in the second coordinate system, and converting the position parameters in the second coordinate system into the position parameters in the first coordinate system by utilizing the conversion parameters. The position parameters of the three-dimensional points are generally represented by three-dimensional coordinates. The conversion parameters include rotational conversion parameters and translational conversion parameters. In particular, the form of the conversion parameters can be referred to as the following example:

wherein>

Representing rotation transformation parameters from the second coordinate system to the first coordinate system,/for>

Represents a translation transformation parameter from the second coordinate system to the first coordinate system, G represents the first coordinate system, L represents the second coordinate system,/or->

Representing the conversion of the second coordinate system into the first coordinate system. The coordinates of the three-dimensional point in the first coordinate system are obtained by multiplying the three-dimensional coordinates of the three-dimensional point in the second coordinate system by the product of the rotation conversion parameters and the translation conversion parameters. Because the position parameters of the a priori structured plane in the first coordinate system are known, it is not necessary to solve for them. And the association relationship between the three-dimensional grid and the prior structured plane is built only by selecting that the third distance between the vertex and the prior structured plane in the three-dimensional grid is smaller than or equal to the third preset distance or further under the condition that the three-dimensional grid is parallel to the prior structured plane, so that the finally built association relationship is more accurate. The speed of acquiring the position parameters of the three-dimensional point and the current image frame in the first coordinate system can be improved by firstly acquiring the coordinates of the three-dimensional point and the coordinates of the current image frame in the second coordinate system and converting the coordinates into the first coordinate system by utilizing the conversion parameters.

In the embodiment of the present disclosure, the same coordinate system is a first coordinate system. Considering that the number of the prior structured planes is huge relative to the three-dimensional points corresponding to the matching feature points in the current image frame and the historical image frame, if the position parameters of all the prior structured planes in the second coordinate system are acquired, the workload is huge, and conversely, if the position parameters of only a few three-dimensional points in the first coordinate system are acquired, the method is relatively simple. Of course, under the condition that the computing power of the execution device is strong, or the number of the prior structured planes is relatively small, the three-dimensional points corresponding to the matching feature points and the position parameters of the prior structured planes in the second coordinate system can be obtained.

In some disclosed embodiments, before performing the point-to-face matching of the three-dimensional point with the a priori structured plane based on the location parameters, the method further comprises the steps of: a first distance between the three-dimensional point and the current image frame is obtained. And selecting a three-dimensional point with the first distance smaller than or equal to the first preset distance as a first three-dimensional point. The first three-dimensional point is used for carrying out point-plane matching with the prior structured plane. Alternatively, the manner of acquiring the first distance between the three-dimensional point and the current image frame may be: and firstly, acquiring coordinates of the three-dimensional point under a second coordinate system by using the current image frame and the historical image frame. Specifically, feature points in the current image frame are extracted, the feature points are matched with feature points in the historical image frames to obtain a plurality of matched feature point pairs, and three-dimensional point coordinates corresponding to the matched feature point pairs can be obtained because the initial pose of the current image frame under the second coordinate system is known. The distance from the three-dimensional point to the current image frame is then calculated to obtain a first distance. Because the number of the priori structured planes is better than the number of the corresponding three-dimensional points of the matched characteristic point pairs, the workload of the equipment is reduced by acquiring the position parameters of the three-dimensional points under the first coordinate system so as to construct the second association relationship under the first coordinate system. And the three-dimensional points are screened by utilizing the current image frame, so that the accuracy of point-plane matching can be improved while the workload of point-plane matching is reduced when the three-dimensional points which are closer to the current image frame are selected.

In some disclosed embodiments, the method for obtaining the first distance between the three-dimensional point and the current image frame may further be: triangulation is performed on the current image frame to obtain a two-dimensional grid set for the current image frame. Wherein the vertices in the two-dimensional mesh set are feature points. That is, triangulating the current image frame is effectively triangulating two-dimensional feature points in the current image frame. The two-dimensional grid group consists of a plurality of two-dimensional grids. Further, only two-dimensional feature points located in the matching feature point pair are triangulated. And projecting the two-dimensional grid set to a second coordinate system to obtain a three-dimensional grid set related to the current image frame. Wherein the three-dimensional grid group comprises a plurality of three-dimensional grids. Wherein each vertex in the three-dimensional grid is a three-dimensional point. Several of the embodiments of the present disclosure may be 1 and more, such as 2, 3, 10, 20, 30, 50, etc. Wherein three-dimensional points may be included in one three-dimensional grid. Specifically, according to the connection relation between the characteristic points in the two-dimensional grid set, the connection relation between the three-dimensional points corresponding to the characteristic points is determined, so that the corresponding three-dimensional grid set is obtained. In this case, the distance between the three-dimensional mesh and the current image frame is taken as the first distance. Because the coordinate error of the three-dimensional points in the portion of the three-dimensional grid may be relatively large in case the distance of the three-dimensional grid from the current image frame exceeds the first preset distance. The first preset distance may be set according to a specific scenario and requirement, and is not specifically defined herein. By constructing the association relationship between the three-dimensional grid and the prior structured plane, the association relationship obtained by the three-dimensional grid is more accurate and has higher precision compared with the association relationship between a single three-dimensional point and the prior structured plane.

In some disclosed embodiments, to reduce the calculation amount of the execution device and to ensure the subsequent positioning accuracy, before acquiring the parameter, the method may further include the following steps: a first pose of a current image frame in a first coordinate system is acquired. And selecting a priori structured plane with a second distance smaller than or equal to a second preset distance from the current image frame as the first priori structured plane. The first priori structured plane is used for carrying out point-plane matching with the three-dimensional points to obtain a second association relation. Acquiring a current imageThe manner of the first pose of the frame in the first coordinate system comprises: the method comprises the steps of firstly obtaining a second pose of a current image frame under a second coordinate system, and converting the second pose into a first pose under a first coordinate system by utilizing conversion parameters. The second pose is the initial pose of the current image frame. As described above, the conversion parameters from the second coordinate system into the first coordinate system are

The second pose of the current image frame includes a rotation parameter and a translation parameter. The rotation parameters and translation parameters may be represented in a matrix. The rotation parameter of the current image frame under the second coordinate system is multiplied by the rotation conversion parameter from the second coordinate system to the first coordinate system, so that the rotation parameter of the current image frame under the first coordinate system can be obtained; and the translation parameter of the current image frame under the first coordinate system is equal to the product of the translation parameter of the current image frame under the second coordinate system multiplied by the rotation conversion parameter, and the translation conversion parameter is added to obtain the translation parameter of the current image frame under the first coordinate system. For example, the position parameter of the current image frame in the second coordinate system is +. >

i denotes the current image frame, ">

Representing a second pose of the current image frame in a second coordinate system +.>

Rotation parameters representing the current image frame in the second coordinate system,/or->

Representing translation parameters of the current image frame in the second coordinate system. Rotation parameters of current image frame in first coordinate system

While the current image frame isTranslation parameter in the first coordinate System +.>

Thus, the first pose of the current image frame under the first coordinate system can be obtained. The second preset distance can be comprehensively determined according to the calculation force and the positioning accuracy requirement of the specific execution equipment. For example, if the positioning accuracy requirement is high, the second preset distance is relatively small, if the positioning accuracy requirement is low, the second preset distance is relatively high, and for example, the calculation power of the execution device is weak, the second preset distance is relatively low, and if the calculation power of the execution device is strong, the second preset distance is relatively high, so that no specific regulation is made here regarding the determination of the second preset distance. The prior structured plane is screened by utilizing the current image frame, so that the accuracy of point-plane matching can be improved while the workload of point-plane matching is reduced by selecting the prior structured plane which is closer to the current image frame.

Alternatively, the point-to-face matching of the three-dimensional point with the a priori structured plane based on the location parameters may be the point-to-face matching of the first three-dimensional point with the first a priori structured plane. That is, three-dimensional points and a priori structured planes are screened simultaneously before point-to-plane matching.

And carrying out point-surface matching on the three-dimensional points and the prior structured plane based on the position parameters of the corresponding three-dimensional points and the prior structured plane under the same coordinate system of the matched feature point pairs so as to acquire a second association relation between the three-dimensional points and the prior structured plane. The three-dimensional points and the prior structured plane are subjected to point-surface matching, so that a second association relationship between the three-dimensional points and the prior structured plane can be obtained in various manners, for example, the association relationship between a single three-dimensional point and the prior structured plane can be obtained respectively, and the association relationship between the three-dimensional grid and the prior structured plane can be obtained. Specifically, a third distance between vertices of all three-dimensional meshes in the three-dimensional mesh group and the a priori-structured plane is obtained. Further, the three-dimensional grid set herein may be composed of three-dimensional grids including the first three-dimensional point, or three-dimensional points in all three-dimensional grids of the set in the three-dimensional grid are the first three-dimensional points. The a priori structured plane here may be a first a priori structured plane. Namely, three vertexes are included in the three-dimensional grid, and association relations between the vertexes and the prior structured plane are respectively obtained. And selecting three-dimensional grids with the first distances between all vertexes and the priori structured plane smaller than or equal to the third preset distance as the first three-dimensional grids. Or selecting a three-dimensional grid, wherein the third distance between all vertexes and the prior structured plane is smaller than or equal to the third preset distance, and the plane formed by all vertexes and the prior structured plane are parallel to each other, as the first three-dimensional grid. That is, there may be various requirements for building the association relationship, and the more strict the requirements are, the more accurate the built association relationship is. The plane formed by all the vertexes is the plane in which the three-dimensional grid is located. That is, the three-dimensional grid is parallel to the structured plane and the distance between the two is less than or equal to the second preset distance. Alternatively, the third preset distance is not only one, and may be dynamically adjusted according to the distance of the three-dimensional grid or structured plane relative to the current image frame. For example, the distance between the three-dimensional grid or the structured plane and the current image frame is proportional to the third preset distance, i.e. the third preset distance can fluctuate within a certain range. For example, the distance between the a three-dimensional grid and the current image frame is 5 meters, the third preset distance is determined to be 0.2 meters, and the distance between the B three-dimensional grid and the current image frame is determined to be 10 meters, and then the third preset distance is determined to be 0.25 meters. Of course, this is merely an example, and the specific value of the third preset distance may be set according to a specific application scenario. And constructing an association relationship between the first three-dimensional grid and the prior structured plane as a second association relationship. The association relationship between the first three-dimensional grid and the prior structured plane is taken as the second association relationship, which is actually the association relationship between each three-dimensional point in the first three-dimensional grid and the prior structured plane. By constructing the association relationship between the three-dimensional grid and the prior structured plane, the association relationship obtained by the three-dimensional grid is more accurate and has higher precision compared with the association relationship between a single three-dimensional point and the prior structured plane.

After thatAnd obtaining the first association relation according to the second association relation. Because the first association is constructed in the first coordinate system, the positioning process of the subsequent device is generally performed based on the second coordinate system, so that the association relationship between the matching feature point pair and the prior structured plane in the second coordinate system needs to be acquired. Because the coordinates of the three-dimensional points in the second association relationship under the second coordinate system are known, the position parameters of the prior structured plane in the second association relationship under the first coordinate system are only required to be converted into the position parameters under the second coordinate system. The positional parameter of a generic a priori structured plane may be denoted pi, where pi is: pi= [ n d ]]. Where n is a four-dimensional vector, n is a three-dimensional vector, representing direction, d is a constant, representing distance. The direction here is for the coordinate system, and for example, n may be a vector set for three coordinate axes, or may be regarded as a direction for the origin of the coordinate system. Assuming the position parameter of the prior structured plane under the first coordinate system is pi ^G The corresponding position parameter under the second coordinate system to be solved is pi ^L II type ^G And pi (a Chinese character) ^L The relation between the two is:

therefore, the position parameters of the priori structured plane in the second association relation under the second coordinate system can be obtained, and the first association relation between the matched characteristic point pairs and the priori structured plane under the second coordinate system is obtained.

After the first association is obtained, a point-surface constraint model is built by using the first association. The point-plane constraint model is used for obtaining a positioning result of the equipment. Wherein the point-plane constraint model comprises a point-plane optimization equation. The point-to-plane optimization equation includes a first term and a second term, the first term and the second term being located on opposite sides of an equal sign of the point-to-plane optimization equation, respectively. And determining the predicted coordinates of the corresponding feature points in the current image frame according to the rotation matrix and the translation matrix of the current image frame and the historical image frame, the direction matrix and the distance matrix of the structured plane, and the two-dimensional coordinates of the feature points in the historical image frame. Wherein the predicted coordinates are the first term. The preset parameters in the point-plane optimization equation are adjusted so that the first term and the second term are equal. The second term is two-dimensional coordinates of the corresponding feature point in the current image frame. The preset parameters include a rotation matrix and a translation matrix of the current image frame.

The rotation matrix and the translation matrix are relative to the origin of the coordinate system, and the rotation matrix and the translation matrix are the rotation parameters and the translation parameters.

The point-plane optimization equation is as follows:

wherein, the liquid crystal display device comprises a liquid crystal display device,

and->

Respectively historical image frame c ₁ And the current image frame c ₂ 3D coordinates of f points under a corresponding coordinate system, wherein the f points are characteristic points in a matched characteristic point pair with association relation with the prior structured plane; />

And

respectively for representing historical image frames c ₁ And the current image frame c ₂ A rotation matrix in world coordinate system W; />

And->

Respectively for representing historical image frames c ₁ And the current image frame c ₂ A translation matrix in world coordinate system W. n andd is a direction matrix and a distance matrix respectively representing the prior structured plane, where the direction matrix and the distance matrix respectively correspond to the direction and the distance of the prior structured plane. E is the identity matrix and T is the transpose. Simplifying the point optimization equation to obtain equation (2):

since the three-dimensional coordinates are converted into normalized camera plane coordinates and the two-dimensional coordinates of each feature point in the pair of matching feature points in the corresponding image frame are known, the above equation (2) can be applied

And->

Conversion to corresponding two-dimensional coordinates

And->

Thus, the final point-plane optimization equation is obtained. And constructing a point-plane constraint model by utilizing the association relation between the prior structured plane and the matching characteristic point pairs between the current image frame and the historical image frame.

In combination with the above-mentioned point-plane optimization equation, the result of the calculation on the right of the equation will obtain the predicted coordinate of the f point in the current image frame, and theoretically, the predicted coordinate of the f point should be equal to the true coordinate of the f point. However, the predicted coordinates of the f-point are not equal to their coordinates in the current image frame, typically because the initial pose of the current image frame is less accurate. Therefore, the preset parameters in the point-plane optimization equation can be adjusted through the difference value between the predicted coordinate and the real coordinate, so that the predicted coordinate of the final f point is equal to the real coordinate, or the error between the predicted coordinate and the real coordinate is smaller than or equal to the preset error. Specifically, the preset parameters include a rotation matrix and a translation matrix of the current image frame, a direction matrix and a distance matrix of the structured plane. Of course, a rotation matrix and a translation matrix of the second historical image frame may also be included. By constructing the association relation between the feature points and the structural plane and optimizing the preset parameters including the initial pose of the current image frame, the positioning precision of the equipment can be improved.

By using the first position parameters of the current image frame and the second historical image frame in the world coordinate system, the two-dimensional coordinates of the matched characteristic point pairs and the second position parameters of the prior structured plane, a point-plane constraint model is built without using three-dimensional points, so that the initial pose is not influenced by the precision of the three-dimensional points in the process of optimizing the initial pose by using the point-plane constraint model, and the positioning precision of equipment is further improved.

In some disclosed embodiments, constructing the point-plane constraint model using the first association relationship further comprises: and fusing the point-plane constraint model with at least one of the re-projection constraint model and the IMU constraint model to obtain a fused constraint model. The fusion constraint model is used for acquiring a positioning result of the equipment. Namely, the positioning result of the equipment is obtained by utilizing the fusion constraint model. Specifically, the initial pose of the equipment is optimized according to the fusion constraint model, and a final positioning result of the equipment is obtained. The initial pose here includes a second pose of the current image frame. The pose of the current image frame of the equipment is optimized by constructing the fusion constraint model, so that the positioning precision of the equipment can be improved.

The process of constraining the pose of the current image frame by using the re-projection constraint model mainly comprises the step of adjusting the pose of the current image frame by using a re-projection error, so that the re-projection error meets the error requirement. The IMU constraint model mainly comprises the process of constraining the pose of the current image frame by using IMU integral errors to optimize the initial pose of the current image frame. In the embodiment of the disclosure, the form of the fusion constraint model is as follows:

wherein, X represents the amount to be optimized, and comprises the equipment pose (the equipment pose corresponding to the current image frame and/or the equipment pose corresponding to the second historical image frame), the IMU parameters, the parameters of the three-dimensional points and the structural plane. r is (r) _p Is a priori residual error, H _p Is the corresponding measurement matrix, B is all IMU measured values, and the residual error between the IMU measured values at the k moment and the k+1 moment is

The corresponding covariance matrix is +.>

C is the feature set observed by the device at all times, and the reprojection residual error of the device at the point of j time l is +.>

The corresponding covariance matrix is +.>

P is the set of all structuring planes, and the homography-based plane residual of the point l under the structuring plane k at the moment i and j of the device is +.>

The corresponding covariance matrix is +.>

The fusion model comprises four models, namely a point-plane constraint model, a re-projection constraint model, an IMU constraint model and a global depth constraint model. Other disclosed embodiments incorporate constraint models that include constraint models other than IMU constraint models.

In some disclosed embodiments, in the step of obtaining the positioning result of the device by using the first association relationship, the historical image frame is any historical image frame before the current image frame. The obtaining the positioning result of the device by using the first association relation further comprises: and responding to the historical image frame which is the last historical image frame of the current image frame, acquiring a positioning result corresponding to the equipment with respect to the current image frame and updating the positioning result corresponding to the historical image frame of the equipment. And responding to the historical image frame which is not the last historical image frame of the current image frame, and acquiring a positioning result corresponding to the current image frame of the equipment. Specifically, as described above, the positioning result of the device obtained by using the first association relationship is actually an initial pose of the current image frame photographed by the device is optimized, and the optimized pose is used as the positioning result of the device. Thus, this step is actually: and under the condition that the second historical image frame is the last historical image of the current image frame, optimizing the pose of the historical image frame and the pose of the current image frame to obtain the positioning results of the equipment at two moments. And under the condition that the historical image frame is not the last historical image frame of the current image frame, optimizing the pose of the current image frame to obtain the positioning result of the equipment at the current moment. Because the earlier the shooting time of the historical image frame is, the more accurate the corresponding pose is, and similarly, the later the shooting time of the historical image frame is, the lower the accuracy of the corresponding pose is. Therefore, by using the technical scheme provided by the embodiment of the disclosure, not only the pose of the current image frame can be optimized, but also the pose of the second historical image frame can be optimized, so that the accuracy of the positioning result of the equipment at each moment is improved.

In the embodiment of the disclosure, the initial pose of the current image frame can be optimized simultaneously by using a plurality of frame history image frames. For example, the current image frame is optimized using the previous history image frame of the current image frame, then the current image frame is optimized using the previous history image of the history image frame, and so on. The pose of the current image frame can be optimized by using the historical image frame only if a matching characteristic point pair exists between the current image frame and the historical image frame and an association relation exists between the matching characteristic point pair and the structural plane. Further, the history image frames in the sliding window may be used as the second history image frame. Typically, the latest frame in the sliding window is the last historical image frame of the current image frame. In some disclosed embodiments, older point-plane constraints are marginalized. I.e. the oldest frame of the sliding window does not participate in the optimization process for the pose of the current image frame.

In the embodiment of the disclosure, the construction method of the prior structured plane includes the following steps. As described above, in embodiments of the present disclosure, the a priori structured planes are three dimensional planes. In particular, the a priori structured plane refers to a three-dimensional plane constructed using commonly observed three-dimensional points in several frames of image frames taken by the device. The device for constructing the a priori structured plane and the device for executing the method provided by the embodiment of the disclosure may be the same device or different devices.

First, triangulating the first image frame to obtain a corresponding two-dimensional grid set. Before triangulation is performed on the first image frame, two-dimensional feature points in the first image frame are extracted, and the two-dimensional feature points in the first image frame are matched with feature points in a last historical image frame of the first image frame, so that a successfully matched two-dimensional matching feature point pair is obtained. And determining three-dimensional points corresponding to the two-dimensional matching characteristic point pairs by combining the initial pose of the first image frame with the pose of the last historical image frame and the two-dimensional coordinates of the two-dimensional matching characteristic point pairs in the respective image frames. Wherein each vertex in the two-dimensional grid set is a two-dimensional feature point in the first image frame. That is, triangulating the first image frame is effectively triangulating the two-dimensional feature points in the first image frame. The two-dimensional grid group consists of a plurality of two-dimensional grids. Further, triangulation is performed on only two-dimensional feature points located in the two-dimensional matching feature point pair.

And secondly, projecting the two-dimensional grid set to a second coordinate system to obtain a corresponding three-dimensional grid set. The vertexes in the three-dimensional grid group are three-dimensional points corresponding to the two-dimensional feature points in the first image frame. That is, according to the connection relationship between the two-dimensional feature points in the two-dimensional grid set, the connection relationship between the three-dimensional points corresponding to the two-dimensional points is determined, so that the corresponding three-dimensional grid set is obtained.

Thirdly, a second three-dimensional grid meeting preset conditions in the three-dimensional grid group is obtained to generate a structured plane. Specifically, first, a three-dimensional mesh having a distance from the first image frame less than or equal to a fourth preset distance is taken as a candidate three-dimensional mesh. Because the coordinate error of the three-dimensional points in the portion of the three-dimensional grid may be relatively large in the case where the distance of the three-dimensional grid from the first image frame exceeds the fourth preset distance, constructing the structured plane using the portion of the three-dimensional grid may result in lower accuracy of the constructed structured plane. The fourth preset distance here may be set according to a specific scenario and requirement, and is not specifically defined herein. And then, selecting the three-dimensional grids with the direction difference smaller than or equal to the preset direction difference value and/or the distance difference smaller than or equal to the preset distance difference value as the second three-dimensional grids. Wherein, the difference of directions is less than or equal to the preset direction difference value, which means that the difference of three-dimensional vectors is less than or equal to the preset direction difference value. Further, in the embodiment of the present disclosure, the candidate mesh is taken as the second three-dimensional mesh only if the difference in direction is less than or equal to the preset direction difference value and the difference in distance is less than or equal to the preset distance difference value. For example, when two three-dimensional grids are located on the same vertical plane, the two three-dimensional grids may be regarded as a second three-dimensional grid, and a plane including the two three-dimensional grids may be generated. Of course, if several historical image frames have already constructed a partially structured plane before the first image frame, the constructed structured plane may be expanded on the constructed partially structured plane with information in the first image frame. At this time, when three-dimensional grids with the direction difference smaller than or equal to the preset direction difference and/or the distance difference smaller than or equal to the preset distance difference are selected, three-dimensional grids with the direction difference smaller than or equal to the preset direction difference and/or the distance difference smaller than or equal to the second preset difference between the three-dimensional grids and the constructed structured plane can be simultaneously selected, and the selected three-dimensional grids and the corresponding structured plane form a new structured plane. That is, in this way, an expansion of the structured plane is achieved, and the positioning of the current frame can refer to the previous information, so that the positioning result is more accurate.

The two-dimensional grid group is obtained by triangulating the first image frame, the three-dimensional grid group is obtained by utilizing the two-dimensional grid group, then the three-dimensional grids meeting the preset conditions in the three-dimensional grid group are obtained to generate the structured plane, and the structured plane can not be generated between any two three-dimensional grids, so that the precision of the structured plane is higher.

Further, by setting the second preset difference value and the fourth preset distance to select the three-dimensional grid to generate the structured plane, the accuracy of the structured plane can be improved.

For a better understanding of the manner of acquiring the first association relationship in the embodiment of the present disclosure, refer to fig. 2, fig. 2 is a schematic flow chart of an embodiment of the positioning method of the present application. In an embodiment of the present disclosure, a positioning method includes the steps of:

step S21: an a priori structured plane constructed from a number of image frames is acquired.

The manner of acquiring the a priori structured plane constructed by the plurality of image frames is as described above, and will not be described herein.

Step S22: the current image frame taken by the device is acquired.

The manner of acquiring the current image frame is as in step S12, and is not described herein.

Step S23: and determining a second pose of the current image frame in a second coordinate system according to the relative position relation between the current image frame and the historical image frame.

As described above, before acquiring the first association relationship, the second pose of the current image frame needs to be acquired. In the embodiment of the present disclosure, the manner of acquiring the second pose of the current image frame may be: the relative positional relationship between the current image frame and a historical image frame is first acquired. The relative positional relationship includes a relative distance and a relative angle. In the disclosed embodiment, a distance is assumed as the relative distance between the current image frame and the historical image frame, and the sensor readings between the current image frame and the historical image frame are pre-integrated to obtain the relative angle therebetween. Since the pose of the historical image frame is known, the second pose of the current image frame can be determined through the relative distance and the relative angle between the historical image frame and the historical image frame.

Step S24: and acquiring the position parameters of the three-dimensional point under the first coordinate system.

The manner of obtaining the position parameter of the three-dimensional point in the first coordinate system is as described above, and is not described herein.

Step S25: and performing point-surface matching on the three-dimensional points and the prior structured plane to obtain a second association relation.

Specifically, the three-dimensional points and the prior structured plane are subjected to point-surface matching, and the manner of obtaining the second association relationship is as described above, which is not described herein.

Step S26: and obtaining the first association relation based on the second association relation.

The manner of obtaining the first association relationship based on the second association relationship is as described above, and is not described herein.

Step S27: and acquiring a positioning result of the equipment by using the first association relation.

The manner of obtaining the positioning result of the device by using the first association relationship is as described above, and is not described herein.

It will be appreciated by those skilled in the art that in the above-described method of the specific embodiments, the written order of steps is not meant to imply a strict order of execution but rather should be construed according to the function and possibly inherent logic of the steps.

The positioning method may be performed by a positioning device, for example, the positioning method may be performed by a terminal device or a server or other processing device, where the terminal device may be a User Equipment (UE) with requirements for visual positioning, three-dimensional reconstruction, image registration, etc., a mobile device, a User terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, and an autopilot, a robot with requirements for positioning and mapping, a medical imaging system with requirements for registration, a product such as glasses, helmets, etc., for augmented reality or virtual reality, etc. In some possible implementations, the positioning method may be implemented by way of a processor invoking computer readable instructions stored in a memory.

Referring to fig. 3, fig. 3 is a schematic structural diagram of an embodiment of a positioning device of the present application. The positioning device 30 includes a plane acquisition module 31, an image acquisition module 32, an association relationship determination module 33, and a positioning module 34. The plane acquisition module 31 is configured to acquire an a priori structured plane constructed from a number of image frames; the image acquisition module 32 is used for acquiring a current image frame shot by the device; the association determining module 33 is configured to determine a first association between a pair of matching feature points between the current image frame and at least one historical image frame and an a priori structured plane, where the a priori structured plane is constructed before performing the current image frame captured by the capturing device; the positioning module 34 is configured to obtain a positioning result of the device using the first association relationship.

In some disclosed embodiments, the association determination module 33 determines a first association between the matching pairs of feature points between the current image frame and the at least one historical image frame and the a priori structured plane, including: respectively acquiring position parameters of three-dimensional points corresponding to the matched characteristic point pairs and the prior structured plane under the same coordinate system; performing point-to-surface matching on the three-dimensional points and the prior structured plane based on the position parameters so as to obtain a second association relationship between the three-dimensional points and the prior structured plane; obtaining a first association relation according to the second association relation; the same coordinate system is a first coordinate system or a second coordinate system, the first coordinate system is a world coordinate system corresponding to the prior structured plane, and the second coordinate system is a world coordinate system corresponding to the current image frame, wherein the first coordinate system and the second coordinate system are the same or different.

According to the scheme, the construction of the second association relationship is facilitated by acquiring the three-dimensional points corresponding to the matched characteristic point pairs and the position parameters of the priori structured plane in the same coordinate system, so that the first association relationship is obtained.

In some disclosed embodiments, the same coordinate is a first coordinate system; the association determining module 33 includes, before performing point-to-face matching on the three-dimensional point and the a priori structured plane based on the position parameter: acquiring a first distance between the three-dimensional point and a current image frame, selecting a three-dimensional point with the first distance smaller than or equal to a first preset distance as a first three-dimensional point, wherein the first three-dimensional point is used for carrying out point-plane matching with a priori structured plane; and/or acquiring a first pose of the current image frame under a first coordinate system; and selecting a priori structured plane with a second distance smaller than or equal to a second preset distance from the current image frame as a first priori structured plane, wherein the first priori structured plane is used for carrying out point-to-surface matching with the three-dimensional points to obtain a second association relation.

According to the scheme, the number of the priori structured planes is better than the number of the corresponding three-dimensional points of the matched characteristic point pairs, and the position parameters of the three-dimensional points in the first coordinate system are acquired so as to construct the second association relationship in the first coordinate system, so that the workload of equipment is reduced. And the three-dimensional points and the prior structured plane are screened by utilizing the current image frame, so that the three-dimensional points and/or the prior structured plane which are closer to the current image frame are selected, the workload is reduced, and the accuracy of point-plane matching can be improved.

In some disclosed embodiments, before the association determining module 33 obtains the first distance between the three-dimensional point and the current image frame, the association determining module includes: triangulating the current image frame to obtain a two-dimensional grid group related to the current image frame, wherein vertexes in the two-dimensional grid group are characteristic points; projecting the two-dimensional grid set to a second coordinate system to obtain a three-dimensional grid set related to the current image frame, wherein the three-dimensional grid set comprises a plurality of three-dimensional grids, and each vertex in the three-dimensional grids is a three-dimensional point; the distance between the three-dimensional grid and the current image frame is taken as a first distance.

According to the scheme, the association relationship between the three-dimensional grid and the prior structured plane is more accurate and has higher precision than the association relationship between the single three-dimensional point and the prior structured plane.

In some disclosed embodiments, the association determination module 33 performs a point-to-face matching of the three-dimensional point with the a priori structured plane based on the location parameters, including: obtaining third distances between vertexes of all three-dimensional grids in the three-dimensional grid group and the prior structured plane; selecting three-dimensional grids with first distances between all vertexes and the priori structured plane smaller than or equal to a third preset distance as first three-dimensional grids; or selecting a three-dimensional grid, wherein the third distance between all vertexes and the prior structured plane is smaller than or equal to a third preset distance, and the plane formed by all vertexes is parallel to the prior structured plane, as a first three-dimensional grid; and constructing an association relationship between the first three-dimensional grid and the prior structured plane as a second association relationship.

According to the scheme, the association relationship between the three-dimensional grid and the prior structured plane is built only by selecting that the third distance between the vertexes in the three-dimensional grid and the prior structured plane is smaller than or equal to the third preset distance or further under the condition that the three-dimensional grid is parallel to the prior structured plane, so that the finally built association relationship is more accurate.

In some disclosed embodiments, the association determining module 33 obtains the position parameters of the three-dimensional points corresponding to the matching feature point pairs and the a priori-structured plane in the same coordinate system, or obtains the first pose of the current image frame in the first coordinate system, including: acquiring position parameters of the three-dimensional points under a second coordinate system, and converting the position parameters into position parameters under a first coordinate system by utilizing conversion parameters; or acquiring a second pose of the current image frame under the second coordinate system, and converting the second pose into a first pose under the first coordinate system by utilizing the conversion parameters.

According to the scheme, the coordinates of the three-dimensional point and the coordinates of the current image frame are acquired under the second coordinate system, and the conversion parameters are converted into the first coordinate system, so that the speed of acquiring the position parameters of the three-dimensional point and the current image frame under the first coordinate system can be improved.

In some disclosed embodiments, after the association determination module 33 determines a first association between the matching pairs of feature points between the current image frame and the at least one historical image frame and the a priori structured plane, the positioning module 34 includes: constructing a point-plane constraint model by using a first association relation, wherein the point-plane constraint model is used for acquiring a positioning result of equipment, the point-plane constraint model comprises a point-plane optimization equation, the point-plane optimization equation comprises a first term and a second term, and the first term and the second term are respectively positioned on two sides of an equal sign of the point-plane optimization equation; obtaining a positioning result of the device by using the first association relation, including: determining prediction coordinates of corresponding feature points in the current image frame according to the rotation matrix and the translation matrix of the current image frame and the historical image frame, the direction matrix and the distance matrix of the structured plane, and the two-dimensional coordinates of the feature points in the historical image frame, wherein the prediction coordinates are used as a first item; and adjusting preset parameters in the point-plane optimization equation to enable the first term to be equal to the second term, wherein the second term is two-dimensional coordinates of corresponding feature points in the current image frame, and the preset parameters comprise a rotation matrix and a translation matrix of the current image frame.

According to the scheme, the association relation between the two-dimensional feature points and the structural plane is constructed, and the preset parameters including the initial pose of the current image frame are optimized, so that the positioning accuracy of equipment can be improved.

In some disclosed embodiments, the historical image frame is any historical image frame that is prior to the current image frame; the positioning module 34 obtains a positioning result of the device by using the first association relation, including: responding to the historical image frame which is the last historical image frame of the current image frame, acquiring a positioning result corresponding to the current image frame of the equipment and updating the positioning result corresponding to the historical image frame of the equipment; and responding to the historical image frame which is not the last historical image frame of the current image frame, and acquiring a positioning result corresponding to the current image frame of the equipment.

According to the scheme, under the condition that the historical image is the last historical image frame of the current image frame, not only can the pose of the current image frame be optimized, but also the pose of the historical image frame can be optimized, so that the accuracy of the positioning results of the equipment at a plurality of moments is improved.

In some disclosed embodiments, the positioning module 34 obtains a positioning result of the device using the first association relationship, including: fusing the point-plane constraint model with at least one of the re-projection constraint model and the IMU constraint model to obtain a fused constraint model; the fusion constraint model is used for obtaining a positioning result of the equipment.

According to the scheme, the pose of the current image frame of the equipment is optimized by constructing the fusion constraint model, so that the positioning accuracy of the equipment can be improved.

Referring to fig. 4, fig. 4 is a schematic structural diagram of an embodiment of an electronic device of the present application. The electronic device 40 comprises a memory 41 and a processor 42, the processor 42 being arranged to execute program instructions stored in the memory 41 for carrying out the steps of the above-described positioning method embodiments. In one particular implementation scenario, electronic device 40 may include, but is not limited to: the microcomputer and the server, and the electronic device 40 may also include a mobile device such as a notebook computer and a tablet computer, which is not limited herein.

In particular, the processor 42 is adapted to control itself and the memory 41 to implement the steps in the above-described positioning method embodiments. The processor 42 may also be referred to as a CPU (Central Processing Unit ). The processor 42 may be an integrated circuit chip having signal processing capabilities. The processor 42 may also be a general purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a Field programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. In addition, the processor 42 may be commonly implemented by an integrated circuit chip.

Referring to fig. 5, fig. 5 is a schematic structural diagram of an embodiment of a computer readable storage medium of the present application. The computer readable storage medium 50 stores program instructions 501 executable by the processor, the program instructions 501 for implementing the steps in the positioning method embodiments described above.

In some embodiments, functions or modules included in an apparatus provided by the embodiments of the present disclosure may be used to perform a method described in the foregoing method embodiments, and specific implementations thereof may refer to descriptions of the foregoing method embodiments, which are not repeated herein for brevity.

The foregoing description of various embodiments is intended to highlight differences between the various embodiments, which may be the same or similar to each other by reference, and is not repeated herein for the sake of brevity.

In the several embodiments provided in the present application, it should be understood that the disclosed methods and apparatus may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of modules or units is merely a logical functional division, and there may be additional divisions of actual implementation, e.g., units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical, or other forms.

In addition, each functional unit in each embodiment of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.

The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be embodied essentially or in part or all or part of the technical solution contributing to the prior art or in the form of a software product stored in a storage medium, including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the methods of the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

Claims

1. A positioning method, comprising:

acquiring a priori structured plane constructed by a plurality of image frames, wherein the priori structured plane is a three-dimensional plane constructed by three-dimensional points commonly observed in a plurality of image frames shot by equipment;

Acquiring a current image frame shot by equipment;

determining a first association between the prior structured plane and a matching feature point pair between the current image frame and at least one historical image frame;

and acquiring a positioning result of the equipment by using the first association relation.

2. The method of claim 1, wherein the determining a first association between the prior structured plane and a pair of matching feature points between the current image frame and at least one historical image frame comprises:

respectively acquiring position parameters of the three-dimensional points corresponding to the matching characteristic point pairs and the prior structured plane under the same coordinate system;

performing point-to-surface matching on the three-dimensional points and the prior structured plane based on the position parameters so as to acquire a second association relationship between the three-dimensional points and the prior structured plane;

obtaining the first association relation according to the second association relation;

the same coordinate system is a first coordinate system or a second coordinate system, the first coordinate system is a world coordinate system corresponding to the prior structured plane, and the second coordinate system is a world coordinate system corresponding to the current image frame, wherein the first coordinate system and the second coordinate system are the same or different.

3. The method of claim 2, wherein the same coordinate is a first coordinate system; before the point-to-face matching of the three-dimensional point and the prior structured plane based on the position parameter, the method comprises the following steps:

acquiring a first distance between the three-dimensional point and the current image frame, and selecting a three-dimensional point with the first distance smaller than or equal to a first preset distance as a first three-dimensional point, wherein the first three-dimensional point is used for carrying out point-plane matching with the prior structured plane;

and/or acquiring a first pose of the current image frame under the first coordinate system; and selecting a priori structured plane with a second distance smaller than or equal to a second preset distance from the current image frame as a first priori structured plane, wherein the first priori structured plane is used for carrying out point-plane matching with the three-dimensional points to obtain the second association relation.

4. A method according to claim 3, wherein said obtaining a first distance between the three-dimensional point and the current image frame comprises:

triangulating the current image frame to obtain a two-dimensional grid group related to the current image frame, wherein vertexes in the two-dimensional grid group are the characteristic points;

Projecting the two-dimensional grid set to the second coordinate system to obtain a three-dimensional grid set related to the current image frame, wherein the three-dimensional grid set comprises a plurality of three-dimensional grids, and each vertex in the three-dimensional grids is the three-dimensional point;

and taking the distance between the three-dimensional grid and the current image frame as the first distance.

5. The method of claim 4, wherein the performing a point-to-face matching of the three-dimensional point with the a priori structured plane based on the location parameters comprises:

obtaining third distances between vertexes of all three-dimensional grids in the three-dimensional grid group and the priori structured plane;

selecting three-dimensional grids with first distances between all vertexes and the priori structured plane smaller than or equal to a third preset distance as first three-dimensional grids; or selecting a three-dimensional grid, wherein a third distance between all vertexes and the prior structured plane is smaller than or equal to a third preset distance, and a plane formed by all vertexes and the prior structured plane are parallel to each other, as the first three-dimensional grid;

and constructing an association relationship between the first three-dimensional grid and the prior structured plane as the second association relationship.

6. A method according to claim 3, wherein the obtaining the position parameters of the three-dimensional points corresponding to the matching feature point pairs and the a priori-structured plane in the same coordinate system, or obtaining the first pose of the current image frame in the first coordinate system, comprises:

acquiring the position parameters of the three-dimensional points in the second coordinate system, and converting the position parameters into the position parameters in the first coordinate system by utilizing conversion parameters;

or acquiring a second pose of the current image frame under the second coordinate system, and converting the second pose into a first pose under the first coordinate system by utilizing the conversion parameters.

7. The method of any of claims 1-6, wherein after determining a first association between the matching pairs of feature points between the current image frame and at least one historical image frame and an a priori structured plane, comprising:

constructing a point-plane constraint model by utilizing the first association relation, wherein the point-plane constraint model is used for acquiring a positioning result of equipment, the point-plane constraint model comprises a point-plane optimization equation, the point-plane optimization equation comprises a first term and a second term, and the first term and the second term are respectively positioned at two sides of an equal sign of the point-plane optimization equation;

The obtaining the positioning result of the device by using the first association relation includes:

determining predicted coordinates of corresponding feature points in the current image frame according to a rotation matrix and a translation matrix of the current image frame and the historical image frame, a direction matrix and a distance matrix of the structured plane and two-dimensional coordinates of the feature points in the historical image frame, wherein the predicted coordinates are used as the first item;

and adjusting preset parameters in the point-plane optimization equation to enable the first term to be equal to the second term, wherein the second term is a two-dimensional coordinate of a corresponding feature point in the current image frame, and the preset parameters comprise a rotation matrix and a translation matrix of the current image frame.

8. The method of claim 7, wherein constructing a point-plane constraint model using the first association relationship comprises:

fusing the point-plane constraint model with at least one of a re-projection constraint model and an IMU constraint model to obtain a fused constraint model; the fusion constraint model is used for obtaining a positioning result of the equipment.

9. The method of claim 7, wherein the historical image frame is any historical image frame that is prior to a current image frame; the obtaining the positioning result of the device by using the first association relation includes:

Responding to the historical image frame which is the last historical image frame of the current image frame, acquiring a positioning result corresponding to the equipment with respect to the current image frame and updating the positioning result corresponding to the historical image frame of the equipment;

and responding to the historical image frame which is not the last historical image frame of the current image frame, and acquiring a positioning result corresponding to the current image frame of the equipment.

10. A visual positioning device, comprising:

the plane acquisition module is used for acquiring a priori structured plane constructed by a plurality of image frames, wherein the priori structured plane is a three-dimensional plane constructed by three-dimensional points commonly observed in a plurality of image frames shot by equipment;

the image acquisition module is used for acquiring a current image frame shot by the equipment;

the association relation determining module is used for determining a first association relation between the prior structured plane and the matching characteristic point pairs between the current image frame and at least one historical image frame;

and the positioning module is used for acquiring a positioning result of the equipment by utilizing the first association relation.

11. An electronic device comprising a memory and a processor for executing program instructions stored in the memory to implement the method of any one of claims 1 to 9.

12. A computer readable storage medium having stored thereon program instructions, which when executed by a processor, implement the method of any of claims 1 to 9.