WO2022247548A1 - 定位方法和装置、电子设备及存储介质 - Google Patents

定位方法和装置、电子设备及存储介质 Download PDF

Info

Publication number
WO2022247548A1
WO2022247548A1 PCT/CN2022/088754 CN2022088754W WO2022247548A1 WO 2022247548 A1 WO2022247548 A1 WO 2022247548A1 CN 2022088754 W CN2022088754 W CN 2022088754W WO 2022247548 A1 WO2022247548 A1 WO 2022247548A1
Authority
WO
WIPO (PCT)
Prior art keywords
image frame
plane
current image
coordinate system
point
Prior art date
Application number
PCT/CN2022/088754
Other languages
English (en)
French (fr)
Inventor
王帅
陈丹鹏
王楠
周立阳
章国锋
Original Assignee
上海商汤智能科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 上海商汤智能科技有限公司 filed Critical 上海商汤智能科技有限公司
Publication of WO2022247548A1 publication Critical patent/WO2022247548A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition

Definitions

  • the present application relates to the technical field of positioning, and in particular to a positioning method and device, electronic equipment, and a computer-readable storage medium.
  • Visual positioning technology plays an important role in many fields, such as unmanned driving, robotics and other fields.
  • structured planes are often used to optimize the positioning results of devices.
  • the structured plane refers to a three-dimensional plane established according to the constructed three-dimensional points.
  • the general approach is to construct a structured plane in real time, and then use the relationship between the structured plane and the 3D points constructed in real time to obtain a more accurate positioning result of the device.
  • the disadvantages of this method include the limited precision and speed of real-time construction of planes, and the inability to construct real-time planes in environments with weak textures, which affects the positioning of devices.
  • the present application at least provides a positioning method and device, electronic equipment, and a computer-readable storage medium.
  • the present application provides a positioning method, including: obtaining a priori structured plane; obtaining the current image frame captured by the device; determining the matching feature point pair between the current image frame and at least one historical image frame; determining the matching feature point pair and Prioritizing the first association relationship between the structured planes; using the first association relationship to obtain the positioning result of the device.
  • the accuracy of the former is higher than that of a structured plane constructed in real time.
  • the positioning result of the device is obtained, and the positioning accuracy of the device can be improved.
  • Determining the first association relationship between the matching feature point pair and the prior structured plane includes: respectively obtaining the three-dimensional points corresponding to the matching feature point pair and the position parameters of the prior structured plane in the reference coordinate system; based on the position parameter pair The 3D point is matched with the prior structured plane to obtain the second relationship between the 3D point and the prior structured plane; according to the second relationship, the first relationship is obtained; where the reference coordinate system is the first A coordinate system or a second coordinate system, the first coordinate system is the world coordinate system corresponding to the prior structured plane, and the second coordinate system is the world coordinate system corresponding to the current image frame, wherein the first coordinate system and the second coordinate system same or different.
  • the construction of the second association relationship can be facilitated, thereby obtaining the first association relationship.
  • the reference coordinate system is the first coordinate system, respectively obtaining the three-dimensional points corresponding to the matching feature point pairs and the position parameters of the prior structured plane in the reference coordinate system, including: for the three-dimensional points corresponding to the matching feature point pairs For each 3D point of , obtain the first distance between the 3D point and the current image frame; select the 3D point whose first distance is less than or equal to the first preset distance as the first 3D point; obtain the first 3D point at the first coordinate
  • the location parameters under the system are used for point-plane matching with the prior structured plane.
  • Respectively obtain the 3D points corresponding to the matching feature point pairs and the position parameters of the prior structured plane in the reference coordinate system including: obtaining the first pose of the current image frame in the first coordinate system; based on the first pose, select A priori structured plane whose second distance from the current image frame is less than or equal to a second preset distance is used as a first priori structured plane; obtain the position parameter of the first priori structured plane in the first coordinate system Used for point-plane matching with 3D points.
  • the number of prior structured planes is better than the number of matching feature point pairs corresponding to 3D points, by obtaining the position parameters of 3D points in the first coordinate system, in order to construct the first coordinate system
  • the second association relationship reduces the workload of the device.
  • the selection of 3D points and/or prior structured planes that are closer to the current image frame can reduce the workload while improving the point plane. Matching precision.
  • Obtaining the first distance between the 3D point and the current image frame includes: triangulating the current image frame to obtain a 2D grid group about the current image frame, wherein the vertices in the 2D grid group are matching features The feature points on the current image frame included in the point pair; the two-dimensional grid group is projected under the second coordinate system to obtain the three-dimensional grid group about the current image frame, and the three-dimensional grid group includes several three-dimensional grids, wherein , each vertex in the three-dimensional grid is the three-dimensional point; the distance between the three-dimensional grid and the current image frame is taken as the first distance.
  • the association relationship obtained by the former is more accurate and the precision is higher. high.
  • the point-plane matching of the 3D points and the prior structured plane based on the position parameters includes: selecting the first 3D grid from all the 3D grids in the 3D grid group, wherein each vertex of the first 3D grid is consistent with the prior
  • the first distance between the structured planes is less than or equal to the third preset distance, or the third distance between all vertices of the first three-dimensional grid and the prior structured plane is less than or equal to the third preset distance, and A plane composed of all vertices is parallel to the prior structured plane; an association relationship between the first three-dimensional grid and the prior structured plane is constructed as the second association relationship.
  • the three-dimensional grid and the prior structured plane makes the final association relationship more accurate.
  • Obtaining the corresponding three-dimensional points of the matching feature point pair and the position parameters of the prior structured plane in the reference coordinate system, or obtaining the first pose of the current image frame in the first coordinate system includes: obtaining the three-dimensional point in the second coordinate system, and use the conversion parameters to convert the position parameters into position parameters in the first coordinate system; or, obtain the second pose of the current image frame in the second coordinate system, and use the conversion parameters to convert the second position to The pose is transformed into the first pose in the first coordinate system.
  • Obtaining the positioning result of the device by using the first association relationship includes: using the first association relationship to construct a point-plane constraint model, the point-plane constraint model is used to obtain the positioning result of the device, wherein the point-plane constraint model includes point-plane optimization Equation, the point-plane optimization equation includes the first item and the second item, the first item and the second item are respectively located on both sides of the equal sign of the point-plane optimization equation; according to the rotation matrix and translation matrix of the current image frame and the historical image frame, first Test the direction matrix and distance matrix of the structured plane, and the two-dimensional coordinates of the feature points in the historical image frame to determine the predicted coordinates of the corresponding feature points in the current image frame, where the predicted coordinates are taken as the first item; adjust the point-plane optimization equation
  • the preset parameters in so that the first item is equal to the second item, wherein the second item is the two-dimensional coordinates of the corresponding feature points in the current image frame, and the preset parameters include the rotation matrix and translation
  • Constructing a point-plane constraint model by using the first relationship includes: merging the point-plane constraint model with at least one of a reprojection constraint model and an IMU constraint model to obtain a fusion constraint model; the fusion constraint model is used to obtain device positioning results.
  • the positioning accuracy of the device can be improved.
  • Obtaining the positioning result of the device by using the first association relationship includes: in response to the historical image frame being the last historical image frame of the current image frame, obtaining the positioning result of the device corresponding to the current image frame and updating the positioning result of the device corresponding to the historical image frame ; Responding to the last historical image frame that the historical image frame is not the current image frame, acquire the positioning result of the device corresponding to the current image frame.
  • the historical image frame is the previous historical image frame of the current image frame
  • the pose of the current image frame can be optimized, but also the pose of the historical image frame can be optimized, thereby improving the The accuracy of the positioning result of the device.
  • the present application provides a positioning device, including: a plane acquisition module, used to acquire a priori structured plane; an image acquisition module, used to acquire the current image frame captured by the device; an association relationship determination module, used to determine the current image frame and Matching feature point pairs between at least one historical image frame, and determining a first association relationship between the matching feature point pair and a priori structured plane, where the priori structured plane is constructed before performing the acquisition of the current image frame captured by the device ; A positioning module, configured to obtain a positioning result of the device by using the first association relationship.
  • the present application provides an electronic device, including a memory and a processor, and the processor is used to execute program instructions stored in the memory, so as to realize the above positioning method.
  • the present application provides a computer-readable storage medium, on which program instructions are stored, and the above positioning method is implemented when the program instructions are executed by a processor.
  • the precision of the former is higher than that of the structured plane constructed in real time.
  • the positioning result of the device is obtained, and the positioning accuracy of the device can be improved.
  • Fig. 1 is a schematic flow diagram 1 of an embodiment of the positioning method of the present application
  • FIG. 2 is a second schematic flow diagram of an embodiment of the positioning method of the present application.
  • Fig. 3 is a schematic structural view of an embodiment of the positioning device of the present application.
  • FIG. 4 is a schematic structural diagram of an embodiment of the electronic device of the present application.
  • Fig. 5 is a schematic structural diagram of an embodiment of a computer-readable storage medium of the present application.
  • FIG. 1 is a first schematic flowchart of an embodiment of a positioning method of the present application. Specifically, the positioning method may include the following steps S11 to S14.
  • Step S11 Obtain a priori structured plane.
  • the prior structured plane may be constructed based on several image frames by a device implementing the positioning method provided by the embodiments of the present disclosure, or may be constructed by other devices based on several image frames. No specific provisions are made here regarding the equipment for constructing a priori structured plane.
  • the environmental image can be taken while keeping the movement and rotation speed of the device at 0 or close to 0.
  • the quality of the environmental image obtained in this way is compared with that of the device.
  • the quality of the environment image taken during the movement is higher, therefore, the prior structured plane obtained according to the former has higher accuracy than the prior structured plane obtained according to the latter.
  • the specific form of the prior structured plane may be a dense three-dimensional map or a sparse three-dimensional map, and the specific form of the prior structured plane is not specified here.
  • the embodiments of the present disclosure take the prior structured plane as a three-dimensional plane as an example.
  • the way to obtain the prior structured plane can be to obtain the storage location of the prior structured plane in the device by reading the storage information of the device, or to obtain the prior structured plane from other devices through communication connections, etc.
  • a priori structured plane may be constructed based on multiple image frames obtained by shooting the environment with the execution device. Therefore, the manner of obtaining the prior structured plane is not specified here.
  • Step S12 Obtain the current image frame captured by the device.
  • the execution device of the positioning method proposed by the embodiments of the present disclosure may be a device that captures the current image frame, or may not be the device that captures the current image frame.
  • the execution device may acquire the current image frame by establishing a communication connection with the device that captures the current image frame.
  • the execution device is not the same device as the device that captures the current image frame, there is no limitation on the communication connection between the two.
  • the execution device and the device that captures the current image frame are the same device as an example.
  • the device in the embodiments of the present disclosure may include a device for capturing a current image frame and a sensor, where the sensor is used to measure motion information of the device.
  • the current image frame may be an image acquired in real time without any image processing, or an image after image processing, where the image processing may be cropping, data enhancement, and other processing methods.
  • the method of judging whether the image frame captured by the device will become the current image frame may include: 1. Extracting feature points in the image frame, and when the number of feature points is greater than or equal to the first preset number, the image frame is used as the current image frame. Image frame; 2. Obtain the number of matching feature point pairs between the image frame and historical image frames within a preset time period, and when the number of matching feature point pairs is greater than or equal to the second preset number, the image frame as the current image frame.
  • Step S13 Determine the matching feature point pairs between the current image frame and at least one historical image frame, and determine the first association relationship between the matching feature point pairs and the prior structured plane.
  • the way to obtain the initial pose of the current image frame may be: first obtain the relative positional relationship between the current image frame and a historical image frame, and then based on the pose of the historical image frame and the relative positional relationship Determine the initial pose of the current image frame.
  • Relative positional relationships may include relative distances and relative angles. In the embodiment of the present disclosure, it is assumed that a distance is taken as the relative distance between the current image frame and the historical image frame, and the sensor readings between the current image frame and the historical image frame are pre-integrated to obtain the relative angle between the two . Since the pose of the historical image frame is known, the initial pose of the current image frame can be determined through the relative distance and relative angle between the two.
  • the historical image frame is an image frame that has undergone positioning processing, for example, a previous historical image frame of the current image frame.
  • the historical image frame can be one frame or multiple frames. For example, when the number of historical image frames is 3, the first association relationship between the matching feature point pairs between the current image frame and the three historical image frames and the prior structured plane is respectively obtained to pass through multiple frames of historical image
  • the frame information obtains the pose of the current image frame, that is, obtains the positioning result of the current image frame.
  • Step S14 Obtain the positioning result of the device by using the first association relationship.
  • the positioning result of the device is obtained.
  • the positioning result of the device by using the first association relationship, for example, the first association between the matching feature point pairs between the current image frame and at least one historical image frame and the prior structured plane relationship, and based on the two-dimensional coordinates of the feature points in the current image frame, determine the predicted coordinates of the feature points corresponding to the feature points in the historical image frame, and obtain the current image frame by obtaining the difference between the predicted coordinates and the real coordinates more precise pose.
  • the precision of the former is higher than that of the structured plane constructed in real time.
  • the positioning result of the device is obtained, and the positioning accuracy of the device can be improved.
  • the manner of determining the first association relationship between the matching feature point pair and the prior structured plane may be: separately obtain the 3D points corresponding to the matching feature point pair and the prior structured plane in the reference coordinate system The positional parameters below. Because the prior structured plane has been constructed before executing the positioning method provided by the embodiment of the present disclosure, therefore, when establishing the prior structured plane, the world coordinate system applicable to the prior structured plane will be constructed at the same time, for example , taking point A in the prior structured plane as the origin of the world coordinate system to construct the corresponding world coordinate system. During the movement of the device, the first frame of image captured by the device is generally used as the origin of the world coordinate system, and the world coordinate system is constructed.
  • the two world coordinates are actually the reference coordinate system; similarly, if they do not coincide, the two world coordinates are considered Department is different.
  • the first coordinate system is used as the world coordinate system corresponding to the prior structured plane
  • the second coordinate system is used as the world coordinate system corresponding to the current image frame, wherein the first coordinate system can be the same as the second coordinate system same or different.
  • the aforementioned reference coordinate system may be the first coordinate system or the second coordinate system.
  • the reference coordinate system refers to the first coordinate system.
  • the way to obtain the position parameters of the 3D points corresponding to the matching feature point pairs and the prior structured plane in the reference coordinate system includes: obtaining the position parameters of the 3D points in the second coordinate system, and using the conversion parameters to convert the position parameters in the second coordinate system to The positional parameters under are converted to the positional parameters under the first coordinate system.
  • the position parameters of a three-dimensional point are generally represented by three-dimensional coordinates.
  • the transformation parameters include rotation transformation parameters and translation transformation parameters.
  • the form of conversion parameters can refer to the following example:
  • the coordinates of the three-dimensional point in the first coordinate system can be obtained by multiplying the product obtained by multiplying the three-dimensional coordinates of the three-dimensional point in the second coordinate system by the rotation transformation parameter and adding the translation transformation parameter. Since the location parameters of the structured plane in the first coordinate system are known a priori, there is no need to solve it.
  • the third distance between the vertices in the three-dimensional grid and the prior structured plane can be less than or equal to the third preset distance, or further constructing the three-dimensional grid and the prior structured plane only when the three-dimensional grid is parallel to the prior structured plane
  • the prior relationship between structured planes can make the final relationship more accurate.
  • the reference coordinate system is the first coordinate system.
  • the number of prior structured planes is larger than the number of corresponding three-dimensional point pairs between the current image frame and the historical image frame, if all the prior structured planes are obtained in the second coordinate system
  • the workload is huge; on the contrary, if only the position parameters of several three-dimensional points in the first coordinate system are obtained, it is relatively simple.
  • the computing power of the execution device is strong enough or the number of prior structured planes is relatively small, it is also possible to obtain the 3D points corresponding to the matching feature point pairs and the position of the prior structured plane in the second coordinate system parameter.
  • the following steps are further included: obtaining the first distance between the 3D point and the current image frame, and selecting the first distance
  • the three-dimensional points less than or equal to the first preset distance are used as the first three-dimensional points.
  • the first three-dimensional point is used for point-plane matching with the prior structured plane.
  • a manner of obtaining the first distance between the 3D point and the current image frame may be: firstly using the current image frame and the historical image frame to obtain the coordinates of the 3D point in the second coordinate system.
  • the manner of obtaining the first distance between the 3D point and the current image frame may also be: performing triangulation on the current image frame to obtain a 2D grid group related to the current image frame.
  • the vertices in the two-dimensional grid group are feature points. That is, performing triangulation on the current image frame is actually performing triangulation on the two-dimensional feature points in the current image frame.
  • a 2D grid group consists of several 2D grids. Further, only the two-dimensional feature points located in the matching feature point pairs are triangulated. Project the two-dimensional grid group to the second coordinate system to obtain the three-dimensional grid group about the current image frame.
  • the 3D grid group may include several 3D grids.
  • each vertex in the three-dimensional grid is a three-dimensional point.
  • the number of numbers proposed in the embodiments of the present disclosure may be 1 or more, for example, 2, 3, 10, 20, 30, 50, and so on.
  • a 3D grid can include three 3D points. Specifically, according to the connection relationship between the feature points in the two-dimensional grid group, the connection relationship between the three-dimensional points corresponding to each feature point is determined, so as to obtain the corresponding three-dimensional grid group. In this case, the distance between the 3D grid and the current image frame is taken as the first distance. Because when the distance between the 3D grid and the current image frame exceeds the first preset distance, the coordinate errors of the 3D points in this part of the 3D grid may be relatively large.
  • the first preset distance here can be set according to specific scenarios and requirements, and is not specifically regulated here. Compared with constructing the association relationship between a single 3D point and the prior structured plane by constructing the association relationship between the 3D grid and the prior structured plane, the association relationship obtained by the former is more accurate and has higher precision.
  • the method may further include the following steps: obtaining the first pose of the current image frame in the first coordinate system , and select a priori structured plane whose second distance from the current image frame is less than or equal to a second preset distance based on the first pose as the first priori structured plane.
  • the first prior structured plane is used to perform point-plane matching with the three-dimensional points to obtain the second association relationship.
  • the way to obtain the first pose of the current image frame in the first coordinate system includes: first obtain the second pose of the current image frame in the second coordinate system, and convert the second pose to the first coordinate by using the conversion parameters The first pose under the system.
  • the second pose here is the initial pose of the above-mentioned current image frame.
  • the conversion parameters from the second coordinate system to the first coordinate system are
  • the second pose of the current image frame includes rotation parameters and translation parameters.
  • rotation parameters and translation parameters can be expressed in a matrix.
  • the rotation parameter of the current image frame in the second coordinate system By multiplying the rotation parameter of the current image frame in the second coordinate system by the rotation conversion parameter from the second coordinate system to the first coordinate system, the rotation parameter of the current image frame in the first coordinate system can be obtained; and the current image frame in The translation parameter in the first coordinate system is equal to the product obtained by multiplying the translation parameter of the current image frame in the second coordinate system by the rotation transformation parameter, plus the translation transformation parameter.
  • the position parameter of the current image frame in the second coordinate system is
  • i represents the current image frame
  • the second preset distance here can be comprehensively determined according to the computing power of the specific execution device and the positioning accuracy requirements. For example, if the positioning accuracy requirement is high, the second preset distance is relatively small; if the positioning accuracy requirement is low, the second preset distance is relatively high. For another example, if the computing power of the execution device is weak, the second preset distance is relatively low; if the computing power of the execution device is strong, the second preset distance is relatively high. Therefore, no specific provisions are made here regarding the determination of the second preset distance.
  • the selection of the prior structured plane closer to the current image frame can not only reduce the workload of point-plane matching, but also improve the accuracy of point-plane matching.
  • performing point-plane matching on the three-dimensional point and the prior structured plane based on the position parameter may be performing point-plane matching on the first three-dimensional point and the first prior structured plane. That is, 3D points and prior structured planes are screened simultaneously before point-plane matching.
  • point-plane matching is performed on the 3D point and the prior structured plane to obtain the distance between the 3D point and the prior structured plane.
  • the second relationship There are many ways to perform point-surface matching on the 3D point and the prior structured plane to obtain the second association relationship between the 3D point and the prior structured plane.
  • a single 3D point and the prior structure can be obtained separately It can also obtain the association relationship between the 3D grid and the prior structured plane.
  • the third distance between the vertices of all the three-dimensional meshes in the three-dimensional mesh group and the prior structured plane is acquired.
  • the 3D grid group here may be composed of 3D grids containing the first 3D points, or the 3D points in all the 3D grids in the 3D grid group are the first 3D points.
  • the prior structured plane here may be the first prior structured plane. That is, the three-dimensional grid may include three vertices, and the association relationship between each vertex and the prior structured plane is acquired respectively. A three-dimensional grid whose first distances between all vertices and the prior structured plane are all less than or equal to the third preset distance is selected as the first three-dimensional grid.
  • the first three-dimensional grid selects a three-dimensional grid whose third distance between all vertices and the prior structured plane is less than or equal to the third preset distance, and whose plane composed of all vertices is parallel to the prior structured plane, as the first three-dimensional grid. That is to say, there may be multiple requirements for establishing an association relationship, and the stricter the requirements, the more accurate the association relationship constructed will be.
  • the plane composed of all vertices is actually the plane where the 3D mesh is located. That is, the three-dimensional grid is parallel to the structured plane, and the distance between them is less than or equal to the second preset distance.
  • the third preset distance can be dynamically adjusted according to the distance of the three-dimensional grid or structured plane relative to the current image frame.
  • the distance between the three-dimensional grid or structured plane and the current image frame is proportional to the third preset distance, that is, the third preset distance can fluctuate within a certain range.
  • the third preset distance is determined to be 0.2 meters; the distance between the B three-dimensional grid current image frame is 10 meters, then the third preset distance is determined The distance is 0.25 meters.
  • the specific value of the third preset distance can be set according to specific application scenarios.
  • An association relationship between the first three-dimensional grid and the prior structured plane is constructed as the second association relationship.
  • the second association relationship between the first three-dimensional grid and the prior structured plane is actually the association relationship between each three-dimensional point in the first three-dimensional grid and the prior structured plane.
  • the association relationship obtained by the former is more accurate and has higher precision.
  • the first association relationship is obtained. Because the first association relationship is constructed in the first coordinate system, and the positioning process of the subsequent equipment is generally based on the second coordinate system, it is necessary to obtain the above-mentioned matching feature point pairs in the second coordinate system and the prior structured plane. Association relationship in the second coordinate system. Because the coordinates of the 3D points in the second relationship in the second coordinate system are known, it is only necessary to convert the position parameters of the prior structured plane in the second relationship in the first coordinate system to the second coordinate system The positional parameters of the . Generally, the location parameter of the prior structured plane can be expressed as ⁇ , where ⁇ is:
  • ⁇ is a four-dimensional vector
  • n is a three-dimensional vector, representing the direction
  • d is a constant, representing the distance.
  • the direction here is for the coordinate system.
  • n can be a vector set for the three coordinate axes respectively, and can also be considered as the direction for the origin of the coordinate system.
  • the position parameters of the priori structured plane in the second coordinate system can be obtained in the second correlation, so as to obtain the first position between the above matching feature point pair and the priori structured plane in the second coordinate system. a relationship.
  • a point-plane constraint model is constructed using the first association relationship.
  • the point-plane constraint model is used to obtain the positioning results of the device.
  • the point-surface constraint model may include point-surface optimization equations.
  • the point-surface optimization equation may include the first item and the second item, and the first item and the second item are respectively located on both sides of the equal sign of the point-surface optimization equation.
  • the prediction of the corresponding feature point in the current image frame is determined coordinate. Among them, the predicted coordinates are taken as the first item.
  • the preset parameters may include a rotation matrix and a translation matrix of the current image frame.
  • Both the rotation matrix and translation matrix here are relative to the origin of the coordinate system, and the rotation matrix and translation matrix here are the above-mentioned rotation parameters and translation parameters.
  • the calculation result on the right side of the equation will get the predicted coordinates of point f in the current image frame.
  • the predicted coordinates of point f should be equal to the real coordinates of point f.
  • the preset parameters in the point-surface optimization equation can be adjusted through the difference between the predicted coordinates and the real coordinates, so that the predicted coordinates of the final point f are equal to the real coordinates, or the error between the two is less than or equal to the preset error .
  • the preset parameters may include the rotation matrix and translation matrix of the current image frame, and the direction matrix and distance matrix of the prior structured plane.
  • the preset parameters may also include a rotation matrix and a translation matrix of the second historical image frame.
  • a point-plane constraint model is constructed without using
  • the point-plane constraint model is constructed by 3D points, so that in the process of optimizing the initial pose using the point-plane constraint model, it is not affected by the accuracy of the 3D point, thereby improving the positioning accuracy of the device.
  • constructing the point-plane constraint model using the first association relationship further includes: fusing the point-plane constraint model with at least one of the reprojection constraint model and the IMU constraint model to obtain the fusion constraint model.
  • the fusion constraint model is used to obtain the positioning result of the device. That is, use the fusion constraint model to obtain the positioning result of the device.
  • the initial pose of the device is optimized according to the fusion constraint model to obtain the final positioning result of the device.
  • the initial pose includes the second pose of the current image frame.
  • the process of using the reprojection constraint model to constrain the pose of the current image frame mainly includes: using the reprojection error to adjust the pose of the current image frame so that the reprojection error meets the error requirements.
  • the process of constraining the pose of the current image frame by the IMU constraint model mainly includes: using the IMU integral error to optimize the initial pose of the current image frame.
  • the form of the fusion constraint model is as follows:
  • X represents the amount that needs to be optimized, including the device pose (the device pose corresponding to the current image frame and/or the device pose corresponding to the second historical image frame), IMU parameters, parameters of three-dimensional points and structured planes; r p is the prior residual, H p is its corresponding measurement matrix, B is all IMU measurements, and the residual between the IMU measurements at time k and k+1 is
  • C is the feature set observed by the device at all times, and the reprojection residual of the device at point l at time j is
  • the corresponding covariance matrix is P is the set of all structured planes.
  • the homography-based plane residual of point l under structured plane k at time i and j of the device is
  • the fusion model may include four models, namely, a point-plane constrained model, a reprojection constrained model, an IMU constrained model, and a global depth constrained model.
  • the fused constraint model may include a constraint model other than the IMU constraint model.
  • the aforementioned historical image frame may be any historical image frame before the current image frame.
  • Obtaining the positioning result of the device using the first association relationship further includes: in response to the historical image frame being the previous historical image frame of the current image frame, obtaining the positioning result of the device corresponding to the current image frame and updating the positioning result of the device corresponding to the historical image frame ; Responding to the last historical image frame that the historical image frame is not the current image frame, acquire the positioning result of the device corresponding to the current image frame.
  • this step is actually: in the case that the second historical image frame is the last historical image frame of the current image frame, optimize the pose of the historical image frame and the pose of the current image frame to obtain the The positioning result of the moment; in the case that the historical image frame is not the previous historical image frame of the current image frame, the pose of the current image frame is optimized to obtain the positioning result of the device at the current moment.
  • the provided technical solution can not only optimize the pose of the current image frame, but also optimize the pose of the second historical image frame, thereby improving the accuracy of the positioning result of the device at each moment.
  • several frames of historical image frames may be used to simultaneously optimize the initial pose of the current image frame.
  • the current image frame is optimized first by using a previous historical image frame of the current image frame, and then the current image frame is optimized by using a previous historical image frame of the historical image frame. Only when there are matching feature point pairs between the current image frame and the historical image frame, and there is an association between the matching feature point pair and the structured plane, can the historical image frame be used to optimize the pose of the current image frame.
  • the historical image frames in the sliding window may be used as the second historical image frames.
  • the latest frame in the sliding window is the previous historical image frame of the current image frame.
  • older point-plane constraints are marginalized, that is, the oldest frame of the sliding window does not participate in the optimization process of the pose of the current image frame.
  • the method for constructing a priori structured plane may include the following steps.
  • the prior structured plane is a three-dimensional plane.
  • the prior structured plane refers to a three-dimensional plane constructed by using commonly observed three-dimensional points in several image frames captured by the device.
  • the device for constructing the prior structured plane and the device for executing the method provided by the embodiments of the present disclosure may be the same device or different devices.
  • the first step triangulation is performed on the first image frame to obtain a corresponding two-dimensional grid group.
  • the two-dimensional feature points in the first image frame are extracted, and the two-dimensional feature points in the first image frame are combined with the previous historical image frame of the first image frame
  • the feature points in are matched, and the two-dimensional matching feature point pairs that are successfully matched are obtained.
  • the three-dimensional points corresponding to the two-dimensional matching feature point pairs are determined.
  • each vertex in the two-dimensional grid group is a two-dimensional feature point in the first image frame.
  • performing triangulation on the first image frame is actually performing triangulation on the two-dimensional feature points in the first image frame.
  • a 2D grid group can consist of several 2D grids. Further, only the two-dimensional feature points located in the two-dimensional matching feature point pairs are triangulated.
  • the two-dimensional grid group is projected to the second coordinate system to obtain the corresponding three-dimensional grid group.
  • the vertices in the three-dimensional grid group are three-dimensional points corresponding to the two-dimensional feature points in the first image frame. That is to say, according to the connection relationship between the two-dimensional feature points in the two-dimensional grid group, the connection relationship between the three-dimensional points corresponding to each two-dimensional feature point is determined, so as to obtain the corresponding three-dimensional grid group.
  • the third step is to obtain the first three-dimensional grid satisfying the preset condition in the three-dimensional grid group to generate the structured plane.
  • a three-dimensional grid whose distance from the first image frame is less than or equal to the fourth preset distance is used as a candidate three-dimensional grid, because the distance between the three-dimensional grid and the first image frame exceeds the fourth preset distance.
  • the coordinate error of the 3D points in this part of the 3D grid may be relatively large. If this part of the 3D grid is used to construct a structured plane, the precision of the constructed structured plane may be low.
  • the fourth preset here The distance can be set according to specific scenarios and needs, and no specific regulations are made here; then, between the candidate 3D grids, the direction difference is less than or equal to the preset direction difference and/or the distance difference is less than or equal to the preset distance
  • the 3D mesh of the difference as the second 3D mesh.
  • the direction difference here is less than or equal to the preset direction difference means that the differences of the three-dimensional vectors are all less than or equal to the preset direction difference.
  • the candidate grid is used as the second three-dimensional grid.
  • these two 3D grids can be used as a second 3D grid, and a plane including these two 3D grids can be generated.
  • the constructed structured plane can be expanded using the information in the first image frame on the constructed partially structured plane .
  • a three-dimensional grid whose direction difference between the structured planes is less than or equal to the preset direction difference, and/or a distance difference is less than or equal to the second preset difference, and the selected three-dimensional grid is compared with the corresponding The structured planes form a new structured plane. That is, in this way, the expansion of the structured plane is realized, and the positioning of the current frame can refer to the previous information, so that the positioning result is more accurate.
  • the accuracy of the structured plane can be improved.
  • FIG. 2 is a schematic flowchart 2 of an embodiment of the positioning method of the present application.
  • the positioning method includes the following steps S21 to S27.
  • Step S21 Obtain a priori structured plane.
  • Step S22 Obtain the current image frame captured by the device.
  • step S12 The manner of obtaining the current image frame is as described in step S12 above, and will not be repeated here.
  • Step S23 According to the relative position relationship between the current image frame and the historical image frame, determine the second pose of the current image frame in the second coordinate system.
  • the way to obtain the second pose of the current image frame may be: first obtain the relative positional relationship between the current image frame and a historical image frame, and then based on the pose of the historical image frame and the acquired The relative positional relationship determines the initial pose of the current image frame.
  • the relative positional relationship may include relative distance and relative angle.
  • Step S24 Obtain the position parameters of the three-dimensional point in the first coordinate system.
  • Step S25 Perform point-plane matching on the 3D points and the prior structured plane to obtain the second association relationship.
  • Step S26 Obtain the first association relationship based on the second association relationship.
  • Step S27 Obtain the positioning result of the device by using the first association relationship.
  • the precision of the former is higher than that of the structured plane constructed in real time.
  • the positioning result of the device is obtained, and the positioning accuracy of the device can be improved.
  • the writing order of each step does not mean a strict execution order and constitutes any limitation on the implementation process.
  • the specific execution order of each step should be based on its function and possible
  • the inner logic is OK.
  • the execution subject of the positioning method may be a positioning device.
  • the positioning method may be executed by a terminal device or a server or other processing equipment, wherein the terminal device may be a user equipment (User Equipment) that has requirements for visual positioning, three-dimensional reconstruction, image registration, etc. , UE), mobile devices, user terminals, terminals, cellular phones, cordless phones, Personal Digital Assistant (PDA), handheld devices, computing devices, vehicle-mounted devices, wearable devices, and self-driving cars.
  • PDA Personal Digital Assistant
  • the positioning method may be implemented by a processor calling computer-readable instructions stored in a memory.
  • FIG. 3 is a schematic structural diagram of an embodiment of the positioning device of the present application.
  • the positioning device 30 includes a plane acquisition module 31 , an image acquisition module 32 , an association relationship determination module 33 and a positioning module 34 .
  • the plane acquisition module 31 is used to acquire a priori structured plane;
  • the image acquisition module 32 is used to acquire the current image frame captured by the device;
  • the correlation determination module 33 is used to determine the matching feature points between the current image frame and at least one historical image frame Yes, and determine the first association relationship between the matching feature point pair and the priori structured plane, the priori structured plane is constructed before the current image frame captured by the acquisition device;
  • the positioning module 34 is used to utilize the first association relationship Obtain the positioning result of the device.
  • the precision of the former is higher than that of the structured plane constructed in real time.
  • the positioning result of the device is obtained, and the positioning accuracy of the device can be improved.
  • the association relationship determining module 33 determines the first association relationship between the matching feature point pair and the prior structured plane, including: respectively obtaining the 3D points corresponding to the matching feature point pair and the prior structured plane in The position parameter under the reference coordinate system; based on the position parameter, point-plane matching is performed on the 3D point and the prior structured plane to obtain the second correlation between the 3D point and the prior structured plane; according to the second correlation, the obtained The first association relationship; wherein, the reference coordinate system is the first coordinate system or the second coordinate system, the first coordinate system is the world coordinate system corresponding to the prior structured plane, and the second coordinate system is the world coordinate system corresponding to the current image frame , where the first coordinate system is the same as or different from the second coordinate system.
  • the reference coordinate system is the first coordinate system; the correlation determination module 33 obtains respectively the three-dimensional points corresponding to the matching feature point pairs and the position parameters of the prior structured plane in the reference coordinate system, including: For each three-dimensional point in the corresponding three-dimensional points of a pair of matching feature points, obtain the first distance between the three-dimensional point and the current image frame; select the three-dimensional point whose first distance is less than or equal to the first preset distance as the first three-dimensional point ; Obtain the position parameters of the first three-dimensional point in the first coordinate system for point-plane matching with the prior structured plane.
  • the association relationship determination module 33 respectively obtains the three-dimensional points corresponding to the matching feature point pairs and the position parameters of the prior structured plane in the reference coordinate system, including: obtaining the position parameters of the current image frame in the first coordinate system The first pose; based on the first pose, select a priori structured plane whose second distance from the current image frame is less than or equal to the second preset distance as the first priori structured plane; obtain the first priori The position parameters of the structured plane in the first coordinate system are used for point-plane matching with the three-dimensional points.
  • the number of prior structured planes is better than the number of matching feature point pairs corresponding to 3D points, by obtaining the position parameters of 3D points in the first coordinate system, in order to construct The second association relationship reduces the workload of the device.
  • the selection of 3D points and/or prior structured planes that are closer to the current image frame can reduce the workload while improving the point plane. Matching precision.
  • the correlation determining module 33 acquires the first distance between the 3D point and the current image frame, including: performing triangulation on the current image frame to obtain a 2D grid group about the current image frame, wherein , the vertices in the two-dimensional grid group are the feature points included in the matching feature point pair on the current image frame; the two-dimensional grid group is projected into the second coordinate system to obtain the three-dimensional grid group about the current image frame,
  • the 3D grid group includes several 3D grids, wherein each vertex in the 3D grid is a 3D point; the distance between the 3D grid and the current image frame is taken as the first distance.
  • the association relationship obtained by the former is more accurate and the precision higher.
  • the association relationship determination module 33 performs point-plane matching on the 3D points and the prior structured plane based on the position parameters, including: selecting the first 3D grid from all 3D grids in the 3D grid group, wherein , the first distance between each vertex of the first three-dimensional grid and the prior structured plane is less than or equal to the third preset distance, or the first distance between all vertices of the first three-dimensional grid and the prior structured plane The three distances are less than or equal to the third preset distance, and the plane composed of all vertices is parallel to the prior structured plane; constructing an association relationship between the first three-dimensional grid and the prior structured plane as the second association relationship.
  • the above scheme is constructed by selecting the third distance between the vertices in the three-dimensional grid and the prior structured plane to be less than or equal to the third preset distance, or when the three-dimensional grid is parallel to the prior structured plane.
  • the relationship between the 3D grid and the prior structured plane makes the final relationship more accurate.
  • the association relationship determination module 33 respectively obtains the three-dimensional point corresponding to the matching feature point pair and the position parameter of the prior structured plane in the reference coordinate system, or obtains the position parameter of the current image frame in the first coordinate system.
  • a posture includes: obtaining the position parameter of the three-dimensional point in the second coordinate system, and converting the position parameter into the position parameter in the first coordinate system by using the transformation parameter; or, obtaining the position parameter of the current image frame in the second coordinate system Two poses, and transform the second pose into the first pose in the first coordinate system by using the conversion parameters.
  • the relationship determining module 33 using the first relationship to obtain the positioning result of the device includes: constructing a point-plane constraint model using the first relationship, and the point-plane constraint model is used to obtain the positioning result of the device, wherein the point The surface constraint model includes a point-plane optimization equation, and the point-plane optimization equation includes the first item and the second item, and the first item and the second item are respectively located on both sides of the equal sign of the point-plane optimization equation;
  • the rotation matrix and translation matrix, the direction matrix of the prior structured plane and the distance matrix, and the two-dimensional coordinates of the feature points in the historical image frame determine the predicted coordinates of the corresponding feature points in the current image frame, wherein the predicted coordinates are used as the first item; adjust the preset parameters in the point-surface optimization equation so that the first item is equal to the second item, where the second item is the two-dimensional coordinates of the corresponding feature points in the current image frame, and the preset parameters include the current image frame
  • the positioning accuracy of the device can be improved.
  • the historical image frame is any historical image frame before the current image frame; the positioning module 34 uses the first association relationship to obtain the positioning result of the device, including: responding to the historical image frame being the previous image frame of the current image frame Historical image frame, obtain the positioning result corresponding to the current image frame of the device and update the positioning result corresponding to the historical image frame of the device; in response to the previous historical image frame that the historical image frame is not the current image frame, obtain the corresponding positioning result of the device regarding the current image frame positioning results.
  • the above scheme in the case that the historical image frame is the previous historical image frame of the current image frame, can not only optimize the pose of the current image frame, but also optimize the pose of the historical image frame, thereby improving the The accuracy of the positioning results of the next device.
  • the positioning module 34 uses the first association relationship to obtain the positioning result of the device, including: fusing the point-plane constraint model with at least one of the reprojection constraint model and the IMU constraint model to obtain the fusion constraint model, wherein, The fused constraint model is used to obtain the localization results of the device.
  • the pose of the current image frame of the device is optimized by constructing a fusion constraint model, which can improve the positioning accuracy of the device.
  • the precision of the former is higher than that of the structured plane constructed in real time.
  • the positioning result of the device can be obtained, and the positioning accuracy of the device can be improved.
  • FIG. 4 is a schematic structural diagram of an embodiment of an electronic device of the present application.
  • the electronic device 40 includes a memory 41 and a processor 42, and the processor 42 is configured to execute program instructions stored in the memory 41, so as to realize the steps in the above positioning method embodiments.
  • the electronic device 40 may include, but is not limited to: a microcomputer and a server.
  • the electronic device 40 may also include mobile devices such as notebook computers and tablet computers, which are not limited here.
  • the processor 42 is used to control itself and the memory 41 to implement the steps in the above positioning method embodiments.
  • the processor 42 may also be called a CPU (Central Processing Unit, central processing unit).
  • the processor 42 may be an integrated circuit chip with signal processing capability.
  • the processor 42 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
  • a general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like.
  • the processor 42 may be jointly implemented by an integrated circuit chip.
  • the precision of the former is higher than that of the structured plane constructed in real time.
  • the positioning result of the device is obtained, and the positioning accuracy of the device can be improved.
  • FIG. 5 is a schematic structural diagram of an embodiment of a computer-readable storage medium of the present application.
  • the computer-readable storage medium 50 stores program instructions 501 that can be executed by a processor, and the program instructions 501 are used to implement the steps in the above positioning method embodiments.
  • the precision of the former is higher than that of the structured plane constructed in real time.
  • the positioning result of the device is obtained, and the positioning accuracy of the device can be improved.
  • the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the methods described in the method embodiments above, and its specific implementation can refer to the description of the method embodiments above. For brevity, here No longer.
  • the disclosed methods and devices may be implemented in other ways.
  • the device implementations described above are only illustrative.
  • the division of modules or units is only a logical function division. In actual implementation, there may be other division methods.
  • units or components can be combined or integrated. to another system, or some features may be ignored, or not implemented.
  • the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.
  • each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.
  • the above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.
  • the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.
  • the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) execute all or part of the steps of the methods in various embodiments of the present application.
  • the aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disc, etc., which can store program codes. .

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Artificial Intelligence (AREA)
  • Image Analysis (AREA)
  • Length Measuring Devices By Optical Means (AREA)

Abstract

本申请公开了一种定位方法和装置、电子设备及存储介质。该定位方法包括：获取先验结构化平面；获取设备拍摄的当前图像帧；确定当前图像帧以及至少一历史图像帧之间的匹配特征点对；确定匹配特征点对与先验结构化平面之间的第一关联关系；利用第一关联关系获取设备的定位结果。本申请的定位方法能够提高设备的定位精度。

Description

定位方法和装置、电子设备及存储介质
交叉引用声明
本申请要求于2021年05月27日提交中国专利局的申请号为202110606804.6的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及定位技术领域，特别是涉及一种定位方法和装置、电子设备及计算机可读存储介质。
背景技术
视觉定位技术在很多领域中都起着很重要的作用，例如在无人驾驶、机器人等领域。在视觉定位技术中，很多时候会利用结构化平面来优化对设备的定位结果。这里，结构化平面指的是根据构建的三维点建立的三维平面。一般做法是实时构建结构化平面，然后利用结构化平面与实时构建的三维点之间的关系，获取设备较为准确的定位结果。这种方式的缺点包括目前实时构建的平面精度和速度有限，并且在纹理较弱的环境下无法实时构建出平面，从而影响了设备的定位。
发明内容
本申请至少提供一种定位方法和装置、电子设备及计算机可读存储介质。
本申请提供了一种定位方法，包括：获取先验结构化平面；获取设备拍摄的当前图像帧；确定当前图像帧以及至少一历史图像帧之间的匹配特征点对；确定匹配特征点对与先验结构化平面之间的第一关联关系；利用第一关联关系获取设备的定位结果。
因此，通过在获取当前图像帧之前获取已构建好的先验结构化平面，相比于实时构建的结构化平面而言，前者精度更高。并且通过构建当前图像帧和历史图像帧中的二维特征点与先验结构化平面之间的关联关系，获取设备的定位结果，能够提高设备的定位精度。
确定匹配特征点对与先验结构化平面之间的第一关联关系，包括：分别获取匹配特征点对对应的三维点以及先验结构化平面在基准坐标系下的位置参数；基于位置参数对三维点与先验结构化平面进行点面匹配，以获取三维点与先验结构化平面之间的第二关联关系；依据第二关联关系，得到第一关联关系；其中，基准坐标系为第一坐标系或第二坐标系，第一坐标系为先验结构化平面对应的世界坐标系，第二坐标系为当前图像帧对应的世界坐标系，其中，第一坐标系与第二坐标系相同或不同。
因此，通过获取匹配特征点对对应的三维点以及先验结构化平面在基准坐标系的位置参数，能够方便第二关联关系的构建，从而得到第一关联关系。
基准坐标系为第一坐标系，分别获取匹配特征点对对应的三维点以及先验结构化平 面在基准坐标系下的位置参数，包括：针对多个所述匹配特征点对对应的三维点中的每个三维点，获取三维点与当前图像帧之间的第一距离；选取第一距离小于或等于第一预设距离的三维点作为第一三维点；获取第一三维点在第一坐标系下的位置参数用于与先验结构化平面进行点面匹配。分别获取匹配特征点对对应的三维点以及先验结构化平面在基准坐标系下的位置参数，包括：获取当前图像帧在第一坐标系下的第一位姿；基于第一位姿，选择与当前图像帧之间的第二距离小于或等于第二预设距离的先验结构化平面作为第一先验结构化平面；获取第一先验结构化平面在第一坐标系下的位置参数用于与三维点进行点面匹配。
因此，因为先验结构化平面的数量相对于匹配特征点对对应的三维点的数量而言更好，通过获取三维点在第一坐标系下的位置参数，以便在第一坐标系下构建第二关联关系，减少了设备的工作量。并且，通过利用当前图像帧对三维点以及先验结构化平面进行筛选，使得选取距离当前图像帧较近的三维点和/或先验结构化平面在减少工作量的同时，也能够提高点面匹配的精度。
获取三维点与当前图像帧之间的第一距离，包括：对当前图像帧进行三角剖分，得到关于当前图像帧的二维网格组，其中，二维网格组中的顶点为匹配特征点对包括的在当前图像帧上的特征点；将二维网格组投影到第二坐标系下，得到关于当前图像帧的三维网格组，三维网格组包括若干个三维网格，其中，所述三维网格中的各顶点为所述三维点；将三维网格与当前图像帧之间的距离作为第一距离。
因此，通过构建三维网格与先验结构化平面之间的关联关系，相对于构建单个三维点与先验结构化平面之间的关联关系而言，前者获得的关联关系更为准确，精度更高。
基于位置参数对三维点与先验结构化平面进行点面匹配，包括：从三维网格组中的所有三维网格选取第一三维网格，其中，第一三维网格的各顶点与先验结构化平面之间的第一距离均小于或等于第三预设距离，或者第一三维网格的所有顶点与先验结构化平面之间的第三距离小于或等于第三预设距离、且所有顶点组成的平面与先验结构化平面平行；构建第一三维网格与先验结构化平面之间的关联关系作为第二关联关系。
因此，通过选择三维网格中顶点与先验结构化平面之间的第三距离小于或等于第三预设距离，或者进一步在三维网格与先验结构化平面平行的情况下，才构建三维网格与先验结构化平面之间的关联关系，使得最后构建的关联关系更为准确。
分别获取匹配特征点对对应的三维点以及先验结构化平面在基准坐标系下的位置参数，或获取当前图像帧在第一坐标系下的第一位姿包括：获取三维点在第二坐标系下的位置参数，并利用转换参数将位置参数转换为第一坐标系下的位置参数；或，获取当前图像帧在第二坐标系下的第二位姿，并利用转换参数将第二位姿转换为第一坐标系下的第一位姿。
因此，通过先在第二坐标系下获取三维点的坐标以及当前图像帧的坐标，并利用转换参数转换到第一坐标系下，能够提高获取三维点以及当前图像帧在第一坐标系下的位置参数的速度。
利用所述第一关联关系获取所述设备的定位结果，包括：利用第一关联关系构建点面约束模型，点面约束模型用于获取设备的定位结果，其中，点面约束模型包括点面优化方程，点面优化方程包括第一项和第二项，第一项和第二项分别位于点面优化方程的等号的两边；依据当前图像帧和历史图像帧的旋转矩阵和平移矩阵、先验结构化平面的方向矩阵以及距离矩阵、历史图像帧中的特征点的二维坐标，确定当前图像帧中对应的特征点的预测坐标，其中，预测坐标作为第一项；调整点面优化方程中的预设参数，以使第一项与第二项相等，其中，第二项为当前图像帧中对应的特征点的二维坐标，预设参数包括当前图像帧的旋转矩阵以及平移矩阵。
因此，通过构建二维特征点与结构化平面之间的关联关系，对包含当前图像帧初始位姿的预设参数进行优化，能够提高对设备的定位精度。
利用第一关联关系构建点面约束模型，包括：将点面约束模型与重投影约束模型、IMU约束模型中的至少一个融合，得到融合约束模型；融合约束模型用于获取设备的定位结果。
因此，通过构建融合约束模型对设备的当前图像帧的位姿进行优化，能够提高对设备的定位精度。
利用第一关联关系获取设备的定位结果，包括：响应于历史图像帧为当前图像帧的上一历史图像帧，获取设备关于当前图像帧对应的定位结果并更新设备关于历史图像帧对应的定位结果；响应于历史图像帧不为当前图像帧的上一历史图像帧，获取设备关于当前图像帧对应的定位结果。
因此，在历史图像帧为当前图像帧的上一历史图像帧的情况下，不仅能够对当前图像帧的位姿进行优化，还能对历史图像帧的位姿进行优化，从而提高多个时刻下设备的定位结果的精度。
本申请提供了一种定位装置，包括：平面获取模块，用于获取先验结构化平面；图像获取模块，用于获取设备拍摄的当前图像帧；关联关系确定模块，用于确定当前图像帧以及至少一历史图像帧之间的匹配特征点对，并确定匹配特征点对与先验结构化平面之间的第一关联关系，先验结构化平面是在执行获取设备拍摄的当前图像帧之前构建；定位模块，用于利用第一关联关系获取设备的定位结果。
本申请提供了一种电子设备，包括存储器和处理器，处理器用于执行存储器中存储的程序指令，以实现上述定位方法。
本申请提供了一种计算机可读存储介质，其上存储有程序指令，程序指令被处理器执行时实现上述定位方法。
上述方案，通过在获取当前图像帧之前获取已构建好的先验结构化平面，相比于实时构建的结构化平面而言，前者精度更高。并且通过构建当前图像帧和历史图像帧中的二维特征点与先验结构化平面之间的关联关系，获取设备的定位结果，能够提高设备的定位精度。
应当理解的是，以上的一般描述和后文的细节描述仅是示例性和解释性的，而非限 制本申请。
附图说明
此处的附图被并入说明书中并构成本说明书的一部分，这些附图示出了符合本申请的实施例，并与说明书一起用于说明本申请的技术方案。
图1是本申请定位方法一实施例的流程示意图一；
图2是本申请定位方法一实施例的流程示意图二；
图3是本申请定位装置一实施例的结构示意图；
图4是本申请电子设备一实施例的结构示意图；
图5是本申请计算机可读存储介质一实施例的结构示意图。
具体实施方式
下面结合说明书附图，对本申请实施例的方案进行详细说明。
以下描述中，为了说明而不是为了限定，提出了诸如特定系统结构、接口、技术之类的具体细节，以便透彻理解本申请。
本文中术语“和/或”，仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本文中字符“/”，一般表示前后关联对象是一种“或”的关系。此外，本文中的“多”表示两个或者多于两个。另外，本文中术语“至少一种”表示多种中的任意一种或多种中的至少两种的任意组合，例如，包括A、B、C中的至少一种，可以表示包括从A、B和C构成的集合中选择的任意一个或多个元素。
请参阅图1，图1是本申请定位方法一实施例的流程示意图一。具体而言，该定位方法可以包括如下步骤S11至步骤S14。
步骤S11：获取先验结构化平面。
先验结构化平面可以是由执行本公开实施例提供的定位方法的设备基于若干图像帧构建的，也可以是由其他设备基于若干图像帧构建的。关于构建先验结构化平面的设备此处不做具体规定。在构建先验结构化平面的过程中，可以在保持设备的移动和转动速度均为0或趋近于0的情况下拍摄环境图像，通过此种方式获取到的环境图像的质量相比于设备处于移动过程中拍摄的环境图像的质量而言更高，因此，根据前者得到的先验结构化平面相比于根据后者得到的先验结构化平面而言精度更高。这里，先验结构化平面的具体形式可以是稠密的三维地图，也可以是稀疏的三维地图，关于先验结构化平面的具体形式此处不做具体规定。本公开实施例以先验结构化平面为三维平面为例。
获取先验结构化平面的方式可以是通过读取设备的存储信息得到先验结构化平面在设备中的存储位置，也可以是通过通信连接等方式从其他设备中获取先验结构化平面，还可以是基于利用执行设备对环境进行拍摄得到的多帧图像帧构建先验结构化平面。因此，获取先验结构化平面的方式此处不做具体规定。
步骤S12：获取设备拍摄的当前图像帧。
本公开实施例提出的定位方法的执行设备可以是拍摄当前图像帧的设备，也可以不是拍摄当前图像帧的设备。例如，执行设备可以通过与拍摄当前图像帧的设备建立通信连接等方式获取当前图像帧。在执行设备与拍摄当前图像帧的设备不是同一设备的情况下，二者之间的通信连接的方式不限。本公开实施例以执行设备与拍摄当前图像帧的设备为同一设备为例。本公开实施例中的设备可以包括用于拍摄当前图像帧的设备以及传感器，其中，传感器用于测量设备的运动信息。当前图像帧可以是实时获取的且未经过任何图像处理的图像，也可以是经过图像处理的图像，这里的图像处理可以是裁剪、数据增强等处理方式。
在一些公开实施例中，为保障设备的定位精度，并非执行设备拍摄的所有图像帧均会成为当前图像帧，有的质量较低的图像帧不会作为设备拍摄的当前图像帧。判断设备拍摄的图像帧是否会成为当前图像帧的方式可以包括：1、提取图像帧中的特征点，在特征点的数量大于或等于第一预设数量的情况下，将该图像帧作为当前图像帧；2、获取图像帧与预设时间段内的历史图像帧之间的匹配特征点对的数量，在匹配特征点对的数量大于或等于第二预设数量的情况下，将该图像帧作为当前图像帧。当然，除了这里列举的两个条件之外，还可以判断图像的清晰度、亮度等是否满足要求。通过选择满足质量要求的图像帧作为当前图像帧，能够减少因为图像帧的质量不佳导致的定位结果过差的情况出现，从而保障了对设备的定位精度。
步骤S13：确定当前图像帧以及至少一历史图像帧之间的匹配特征点对，并确定匹配特征点对与先验结构化平面之间的第一关联关系。
在获取第一关联关系之前，需要获取当前图像帧的初始位姿。在本公开实施例中，获取当前图像帧的初始位姿的方式可以是：先获取当前图像帧和一历史图像帧之间的相对位置关系，再基于历史图像帧的位姿和该相对位置关系确定当前图像帧的初始位姿。相对位置关系可以包括相对距离和相对角度。在本公开实施例中，假定将一个距离作为当前图像帧和历史图像帧之间的相对距离，并将当前图像帧和历史图像帧之间的传感器读数进行预积分得到二者之间的相对角度。因历史图像帧的位姿已知，通过二者之间的相对距离和相对角度，即可确定当前图像帧的初始位姿。
这里，历史图像帧是已经经过定位处理的图像帧，例如当前图像帧的上一历史图像帧。历史图像帧可以是一帧，也可以是多帧。例如，当历史图像帧的数量为3时，分别获取当前图像帧和三帧历史图像帧之间的匹配特征点对与先验结构化平面之间的第一关联关系，以通过多帧历史图像帧的信息获取当前图像帧的位姿，即获取当前图像帧的定位结果。
步骤S14：利用第一关联关系获取设备的定位结果。
根据当前图像帧和至少一帧历史图像帧之间的匹配特征点对与先验结构化平面之间的第一关联关系，获取设备的定位结果。具体地，利用第一关联关系获取设备的定位结果的方式可以有很多种，例如当前图像帧和至少一帧历史图像帧之间的匹配特征点对与 先验结构化平面之间的第一关联关系，以及基于当前图像帧中的特征点的二维坐标，确定历史图像帧中与该特征点对应的特征点的预测坐标，通过获取预测坐标与真实坐标之间的差值，获取当前图像帧更为精确的位姿。
上述方案，通过在获取当前图像帧之前获取构建好的先验结构化平面，相比于实时构建的结构化平面而言，前者精度更高。并且通过构建当前图像帧和历史图像帧中的二维特征点与先验结构化平面之间的关联关系，获取设备的定位结果，能够提高设备的定位精度。
在一些公开实施例中，确定匹配特征点对与先验结构化平面之间的第一关联关系的方式可以是：分别获取匹配特征点对对应的三维点以及先验结构化平面在基准坐标系下的位置参数。因为先验结构化平面是在执行本公开实施例提供的定位方法之前就已经构建好的，因此，在建立先验结构化平面的时候会同时构建先验结构化平面适用的世界坐标系，例如，将先验结构化平面中A点作为世界坐标系原点，构建对应的世界坐标系。在设备进行运动过程中，一般将设备拍摄的第一帧图像作为世界坐标系的原点，并构建世界坐标系。若设备拍摄的第一帧图像刚好与先验结构化平面的A点重合，则这两个世界坐标实际上是基准坐标系；同理，在不重合的情况下，则认为这两个世界坐标系不同。一般地，在获取设备拍摄的第一帧图像时，并不知道其在先验结构化平面对应的世界坐标系中的位置。在本公开实施例中，将第一坐标系作为先验结构化平面对应的世界坐标系，第二坐标系作为当前图像帧对应的世界坐标系，其中，第一坐标系可以与第二坐标系相同或不同。上述的基准坐标系可以是第一坐标系，也可以是第二坐标系。通过获取匹配特征点对对应的三维点以及先验结构化平面在基准坐标系的位置参数，能够方便第二关联关系的构建，从而得到第一关联关系。
在一些公开实施例中，基准坐标系指的是第一坐标系。获取匹配特征点对对应的三维点以及先验结构化平面在基准坐标系下的位置参数的方式包括：获取三维点在第二坐标系下的位置参数，并利用转换参数将在第二坐标系下的位置参数转换为第一坐标系下的位置参数。三维点的位置参数一般用三维坐标表示。转换参数包括旋转转换参数以及平移转换参数。具体地，转换参数的形式可参照如下示例：
Figure PCTCN2022088754-appb-000001
其中，
Figure PCTCN2022088754-appb-000002
表示从第二坐标系到第一坐标系的旋转转换参数，
Figure PCTCN2022088754-appb-000003
表示从第二坐标系到第一坐标系的平移转换参数，G表示第一坐标系，L表示第二坐标系，
Figure PCTCN2022088754-appb-000004
表示第二坐标系到第一坐标系的转换。通过将三维点在第二坐标系下的三维坐标乘以旋转转换参数得到的积，加上平移转换参数，能够得到三维点在第一坐标系下的坐标。因为先验结构化平面在第一坐标系下的位置参数已知，所以无需对其进行求解。通过选择三维网格中顶点与先验结构化平面之间的第三距离小于或等于第三预设距离，或者进一步在三维网格与先验结构化平面平行的情况下才构建三维网格与先验结构化平面之间的关联关系，能够使得最后构建的关联关系更为准确。通过先在第二坐标系下获取三维点的坐标以及当前图像帧的坐标，并利用转换参数转换到第一坐标系下，能够提高获取三维点以及当前 图像帧在第一坐标系下的位置参数的速度。
在本公开实施例中，基准坐标系为第一坐标系。考虑到先验结构化平面的数量相对于当前图像帧与历史图像帧之间的匹配特征点对对应的三维点的数量而言更庞大，若获取所有先验结构化平面在第二坐标系中的位置参数，则工作量巨大；相反，若仅获取几个三维点在第一坐标系中的位置参数，则相对简单。当然，在执行设备的计算力够强或者先验结构化平面的数量相对较少的情况下，也可以获取匹配特征点对对应的三维点以及先验结构化平面在第二坐标系中的位置参数。
在一些公开实施例中，在基于位置参数对三维点与先验结构化平面进行点面匹配之前，还包括以下步骤：获取三维点与当前图像帧之间的第一距离，并选取第一距离小于或等于第一预设距离的三维点作为第一三维点。其中，第一三维点用于与先验结构化平面进行点面匹配。可选地，获取三维点与当前图像帧之间的第一距离的方式可以是：先利用当前图像帧和历史图像帧获取三维点在第二坐标系下的坐标。具体地，提取当前图像帧中的特征点，将该特征点与历史图像帧中的特征点进行匹配得到若干匹配特征点对，因为当前图像帧在第二坐标系下的初始位姿已知，即可得到各匹配特征点对对应的三维点坐标。然后计算三维点到当前图像帧之间的距离得到第一距离。因为先验结构化平面的数量相对于匹配特征点对对应的三维点的数量而言更好，通过获取三维点在第一坐标系下的位置参数，以便在第一坐标系下构建第二关联关系，减少了设备的工作量。并且，通过利用当前图像帧对三维点进行筛选，使得选取距离当前图像帧较近的三维点在减少点面匹配工作量的同时，也能够提高点面匹配的精度。
在一些公开实施例中，获取三维点与当前图像帧之间的第一距离的方式还可以是：对当前图像帧进行三角剖分，得到关于当前图像帧的二维网格组。其中，二维网格组中的顶点为特征点。即，对当前图像帧进行三角剖分实际上是对当前图像帧中的二维特征点进行三角剖分。二维网格组由若干的二维网格组成。进一步地，仅对位于匹配特征点对中的二维特征点进行三角剖分。将二维网格组投影到第二坐标系下，得到关于当前图像帧的三维网格组。其中，三维网格组可以包括若干个三维网格。其中，三维网格中的各顶点为三维点。本公开实施例中提出的若干个可以是1个及以上，例如2个、3个、10个、20个、30个、50个等。一个三维网格中可以包括三个三维点。具体地，根据二维网格组中的各特征点之间的连接关系，确定各特征点对应的三维点之间的连接关系，从而得到对应的三维网格组。在这种情况下，将三维网格与当前图像帧之间的距离作为第一距离。因为在三维网格距离当前图像帧的距离超过第一预设距离的情况下，这部分的三维网格中的三维点的坐标误差可能相对较大。这里的第一预设距离可根据具体场景和需求设置，此处不对其做具体规定。通过构建三维网格与先验结构化平面之间的关联关系，相对于构建单个三维点与先验结构化平面之间的关联关系而言，前者获得的关联关系更为准确，精度更高。
在一些公开实施例中，为减轻执行设备的计算量以及为保障后续定位精度，在获取该参数以前，该方法还可包括以下步骤：获取当前图像帧在第一坐标系下的第一位姿， 并基于第一位姿选择与当前图像帧之间的第二距离小于或等于第二预设距离的先验结构化平面作为第一先验结构化平面。其中，第一先验结构化平面用于与三维点进行点面匹配得到第二关联关系。获取当前图像帧在第一坐标系下的第一位姿的方式包括：先获取当前图像帧在第二坐标系下的第二位姿，并利用转换参数将第二位姿转换为第一坐标系下的第一位姿。其中，这里的第二位姿即为上述当前图像帧的初始位姿。如上述，从第二坐标系转换到第一坐标系中的转换参数为
Figure PCTCN2022088754-appb-000005
当前图像帧的第二位姿包括旋转参数和平移参数。一般地，旋转参数和平移参数可以用矩阵的方式表示。将当前图像帧在第二坐标系下的旋转参数乘以第二坐标系到第一坐标系的旋转转换参数，就能够得到当前图像帧在第一坐标系下的旋转参数；而当前图像帧在第一坐标系下的平移参数等于当前图像帧在第二坐标系下的平移参数乘以旋转转换参数得到的积，再加上平移转换参数。例如，当前图像帧在第二坐标系下的位置参数为
Figure PCTCN2022088754-appb-000006
其中，i表示当前图像帧，
Figure PCTCN2022088754-appb-000007
表示当前图像帧在第二坐标系下的第二位姿，
Figure PCTCN2022088754-appb-000008
表示当前图像帧在第二坐标系下的旋转参数，
Figure PCTCN2022088754-appb-000009
表示当前图像帧在第二坐标系下的平移参数。当前图像帧在第一坐标系下的旋转参数
Figure PCTCN2022088754-appb-000010
而当前图像帧在第一坐标系下的平移参数
Figure PCTCN2022088754-appb-000011
由此，即可得到当前图像帧在第一坐标系下的第一位姿。这里的第二预设距离可根据具体执行设备的计算力以及定位精度需求综合确定。例如，若定位精度需求较高，则第二预设距离相对较小；若定位精度需求较低，则第二预设距离相对较高。又例如，若执行设备的计算力较弱，则第二预设距离相对较低；若执行设别的计算力较强，则第二预设距离相对较高。因此，关于第二预设距离的确定此处不做具体规定。通过利用当前图像帧对先验结构化平面进行筛选，使得选取距离当前图像帧较近的先验结构化平面在减少点面匹配工作量的同时，也能够提高点面匹配的精度。
可选地，基于位置参数对三维点与先验结构化平面进行点面匹配可以是对第一三维点和第一先验结构化平面进行点面匹配。即，在进行点面匹配之前，同时对三维点以及先验结构化平面进行筛选。
基于匹配特征点对对应的三维点以及先验结构化平面在基准坐标系下的位置参数，对三维点与先验结构化平面进行点面匹配，以获取三维点与先验结构化平面之间的第二关联关系。对三维点与先验结构化平面进行点面匹配，以获取三维点与先验结构化平面之间的第二关联关系的方式可以有多种，例如，可以分别获取单个三维点与先验结构化平面之间的关联关系，也可以获取三维网格与先验结构化平面之间的关联关系。具体地，获取三维网格组中所有三维网格的顶点与先验结构化平面之间的第三距离。进一步地， 这里的三维网格组可以是由包含第一三维点的三维网格组成，或三维网格组中的所有三维网格中的三维点均为第一三维点。这里的先验结构化平面可以是第一先验结构化平面。即，三维网格中可以包括三个顶点，分别获取各个顶点与先验结构化平面之间的关联关系。选取所有顶点与先验结构化平面之间的第一距离均小于或等于第三预设距离的三维网格作为第一三维网格。或选取所有顶点与先验结构化平面之间的第三距离小于或等于第三预设距离，且所有顶点组成的平面与先验结构化平面平行的三维网格作为第一三维网格。也就是说，可以有多种关联关系构建的要求，要求越严格，则构建的关联关系相对而言就越准确。这里，所有顶点组成的平面其实就是三维网格所在平面。即，三维网格与结构化平面平行，且二者之间的距离小于或等于第二预设距离。可选地，第三预设距离并非只有一个，可根据三维网格或结构化平面相对于当前图像帧的距离动态调整第三预设距离。例如，三维网格或结构化平面与当前图像帧之间的距离与第三预设距离呈正比例关系，即第三预设距离能够在一定的范围内波动。例如，A三维网格与当前图像帧之间的距离为5米，则确定第三预设距离为0.2米；B三维网格当前图像帧之间的距离为10米，则确定第三预设距离为0.25米。当然，这仅是举例，第三预设距离的具体数值可根据具体运用场景设置。构建第一三维网格与先验结构化平面之间的关联关系作为第二关联关系。第一三维网格与先验结构化平面之间的关联关系作为第二关联关系实际上就是第一三维网格中的各三维点与先验结构化平面之间的关联关系。通过构建三维网格与先验结构化平面之间的关联关系，相对于构建单个三维点与先验结构化平面之间的关联关系而言，前者获得的关联关系更为准确，精度更高。
之后，依据第二关联关系，得到第一关联关系。因为第一关联关系是在第一坐标系中构建，而后续设备的定位过程一般都是基于第二坐标系进行，所以需要获取第二坐标系中上述匹配特征点对与先验结构化平面在第二坐标系下的关联关系。因为第二关联关系中三维点在第二坐标系下的坐标已知，所以仅需要将第二关联关系中的先验结构化平面在第一坐标系下的位置参数转换到第二坐标系下的位置参数即可。一般地，先验结构化平面的位置参数可以表示为Π，其中，Π为：
Π＝[n d]，
其中，Π是一个四维的向量，n是一个三维的向量，表示方向；d是一个常量，表示距离。这里的方向是针对坐标系而言的，例如，n可以是分别针对三个坐标轴的向量集合，也可认为是针对坐标系原点的方向。假设先验结构化平面在第一坐标系下的位置参数为Π G，需要求解的在第二坐标系下对应的位置参数为Π L，则Π G与Π L之间的关系为：
Figure PCTCN2022088754-appb-000012
由此，即可得到第二关联关系中先验结构化平面在第二坐标系下的位置参数，从而得到在第二坐标系下，上述匹配特征点对与先验结构化平面之间的第一关联关系。
在获得第一关联关系之后，利用第一关联关系构建点面约束模型。点面约束模型用于获取设备的定位结果。点面约束模型可以包括点面优化方程。点面优化方程可以包括 第一项和第二项，第一项和第二项分别位于点面优化方程的等号的两边。依据当前图像帧和历史图像帧的旋转矩阵和平移矩阵、先验结构化平面的方向矩阵以及距离矩阵、历史图像帧中的特征点的二维坐标，确定当前图像帧中对应的特征点的预测坐标。其中，预测坐标作为第一项。调整点面优化方程中的预设参数，以使第一项与第二项相等。其中，第二项为当前图像帧中对应的特征点的二维坐标。预设参数可以包括当前图像帧的旋转矩阵以及平移矩阵。
这里的旋转矩阵和平移矩阵均是相对于坐标系原点的，这里的旋转矩阵和平移矩阵为上述的旋转参数和平移参数。
点面优化方程如下：
Figure PCTCN2022088754-appb-000013
其中，
Figure PCTCN2022088754-appb-000014
和
Figure PCTCN2022088754-appb-000015
分别是历史图像帧c 1和当前图像帧c 2对应的坐标系下f点的3D坐标，这里的f点是与先验结构化平面具备关联关系的匹配特征点对中的特征点；
Figure PCTCN2022088754-appb-000016
和
Figure PCTCN2022088754-appb-000017
分别用于表示历史图像帧c 1和当前图像帧c 2在世界坐标系W下的旋转矩阵；
Figure PCTCN2022088754-appb-000018
和
Figure PCTCN2022088754-appb-000019
分别用于表示历史图像帧c 1和当前图像帧c 2在世界坐标系W下的平移矩阵；n和d分别用于表示先验结构化平面的方向矩阵和距离矩阵，这里的方向矩阵和距离矩阵分别对应上述的先验结构化平面的方向和距离；E是单位矩阵，T表示转置。对点面优化方程进行简化，得到等式(2)：
Figure PCTCN2022088754-appb-000020
因为三维坐标转换为归一化相机平面坐标，并且匹配特征点对中的各特征点在对应的图像帧中的二维坐标已知，所以可以将上述等式(2)中的
Figure PCTCN2022088754-appb-000021
和
Figure PCTCN2022088754-appb-000022
转换为对应的二维坐标
Figure PCTCN2022088754-appb-000023
和
Figure PCTCN2022088754-appb-000024
由此得到最终的点面优化方程。以此利用当前图像帧同历史图像帧之间的匹配特征点对与先验结构化平面之间的关联关系构建了点面约束模型。
结合上述点面优化方程而言就是，等式右边计算的结果将会得到当前图像帧中的f点的预测坐标，理论上f点的预测坐标应该与f点的真实坐标相等。但是，因为当前图像帧的初始位姿不太准确，导致f点的预测坐标与其在当前图像帧中的坐标不相等。因此，可通过预测坐标与真实坐标之间的差值调整点面优化方程中的预设参数，使得最终f点的预测坐标与真实坐标相等，或者二者之间的误差小于或等于预设误差。具体地，预设参数可以包括当前图像帧的旋转矩阵和平移矩阵、先验结构化平面的方向矩阵和距离矩阵。当然，预设参数还可包括第二历史图像帧的旋转矩阵和平移矩阵。通过构建特征点与结构化平面之间的关联关系，对包含当前图像帧的初始位姿的预设参数进行优化，能够提高对设备的定位精度。
通过使用当前图像帧和第二历史图像帧在世界坐标系下的第一位置参数、匹配特征点对的二维坐标以及先验结构化平面的第二位置参数，构建点面约束模型，无需利用三维点构建点面约束模型，从而在利用点面约束模型对初始位姿进行优化的过程中，不受三维点精度的影响，进而提高了设备的定位精度。
在一些公开实施例中，利用第一关联关系构建点面约束模型还包括：将点面约束模型与重投影约束模型、IMU约束模型中的至少一个融合，得到融合约束模型。其中，融合约束模型用于获取设备的定位结果。即，利用融合约束模型获取设备的定位结果。具体地，根据融合约束模型对设备的初始位姿进行优化，得到设备的最终定位结果。这里的初始位姿包括当前图像帧的第二位姿。通过构建融合约束模型对设备的当前图像帧的位姿进行优化，能够提高对设备的定位精度。
使用重投影约束模型对当前图像帧位姿的约束的过程主要包括：使用重投影误差调整当前图像帧的位姿，使得重投影误差满足误差的要求。IMU约束模型对当前图像帧位姿的约束的过程主要包括：使用IMU积分误差对当前图像帧的初始位姿进行优化。在本公开实施例中，融合约束模型的形式如下：
Figure PCTCN2022088754-appb-000025
其中，
Figure PCTCN2022088754-appb-000026
这里，X表示需要优化的量，包含设备位姿(当前图像帧对应的设备位姿和/或第二历史图像帧对应的设备位姿)、IMU参数、三维点和结构化平面的参数；r p是先验残差，H p是其对应的测量矩阵，B是所有的IMU测量值，k时刻与k+1时刻的IMU测量值之间的残差是
Figure PCTCN2022088754-appb-000027
对应的协方差矩阵为
Figure PCTCN2022088754-appb-000028
C是设备在所有时刻观测到的特征集合，设备在j时刻l点的重投影残差为
Figure PCTCN2022088754-appb-000029
对应协方差矩阵为
Figure PCTCN2022088754-appb-000030
P是所有的结构化平面集合，设备在i,j时刻下l点在结构化平面k下的基于单应性平面残差为
Figure PCTCN2022088754-appb-000031
对应的协方差矩阵为
Figure PCTCN2022088754-appb-000032
此融合模型中可以包括四个模型，即点面约束模型、重投影约束模型、IMU约束模型以及全局深度约束模型。在另一些公开实施例中，融合约束模型可以包括除IMU约束模型以外的约束模型。
在一些公开实施例中，利用第一关联关系获取设备的定位结果的步骤中，上述历史 图像帧可以为当前图像帧以前的任一历史图像帧。利用第一关联关系获取设备的定位结果还包括：响应于历史图像帧为当前图像帧的上一历史图像帧，获取设备关于当前图像帧对应的定位结果并更新设备关于历史图像帧对应的定位结果；响应于历史图像帧不为当前图像帧的上一历史图像帧，获取设备关于当前图像帧对应的定位结果。具体地，如上述，利用第一关联关系获取设备的定位结果实际上是对设备拍摄的当前图像帧的初始位姿进行优化，将优化后的位姿作为设备的定位结果。因此，该步骤实际上是：在第二历史图像帧为当前图像帧的上一历史图像帧的情况下，对历史图像帧的位姿、当前图像帧的位姿进行优化，得到设备在两个时刻的定位结果；在历史图像帧不为当前图像帧的上一历史图像帧的情况下，对当前图像帧的位姿进行优化，得到设备在当前时刻的定位结果。因为历史图像帧的拍摄时间越早，其对应的位姿越准确；同理，历史图像帧的拍摄时间越晚，其对应位姿的准确度就会有所下滑，因此，使用本公开实施例提供的技术方案不仅能够对当前图像帧的位姿进行优化，还能对第二历史图像帧的位姿进行优化，从而提高各时刻设备的定位结果的精度。
在本公开实施例中，可以使用若干帧历史图像帧同时对当前图像帧的初始位姿进行优化。例如，先使用当前图像帧的上一历史图像帧对当前图像帧进行优化，再使用该历史图像帧的上一历史图像帧对当前图像帧进行优化等。只有当前图像帧和历史图像帧之间具有匹配特征点对，并且匹配特征点对与结构化平面之间具备关联关系才能够使用历史图像帧对当前图像帧的位姿进行优化。进一步地，可以使用滑动窗口中的历史图像帧作为第二历史图像帧。一般地，滑动窗口中最新的一帧为当前图像帧的上一历史图像帧。在一些公开实施例中，边缘化较老的点面约束，即滑动窗口的最老的一帧不参与对当前图像帧的位姿的优化过程。
在本公开实施例中，先验结构化平面的构建方法可以包括如下几个步骤。如上述，在本公开实施例中，先验结构化平面是三维平面。具体地，先验结构化平面指的是利用设备拍摄的若干帧图像帧中共同观测到的三维点构建的三维平面。这里，构建先验结构化平面的设备与执行本公开实施例提供的方法的设备可以是同一个设备，也可以是不同的设备。
第一步，对第一图像帧进行三角剖分以得到对应的二维网格组。具体地，在对第一图像帧进行三角剖分之前，提取第一图像帧中的二维特征点，并将第一图像帧中的二维特征点与第一图像帧的上一历史图像帧中的特征点进行匹配，得到匹配成功的二维匹配特征点对。结合第一图像帧的初始位姿与上一历史图像帧的位姿、二维匹配特征点对在各自图像帧中的二维坐标，确定二维匹配特征点对对应的三维点。其中，二维网格组中的各顶点为第一图像帧中的二维特征点。即，对第一图像帧进行三角剖分实际上是对第一图像帧中的二维特征点进行三角剖分。二维网格组可以由若干的二维网格组成。进一步地，仅对位于二维匹配特征点对中的二维特征点进行三角剖分。
第二步，将二维网格组投影到第二坐标系下，得到对应的三维网格组。其中，三维网格组中的顶点为第一图像帧中的二维特征点对应的三维点。也就是说，根据二维网格 组中的各二维特征点之间的连接关系，确定各二维特征点对应的三维点之间的连接关系，从而得到对应的三维网格组。
第三步，获取三维网格组中满足预设条件的第一三维网格以生成结构化平面。具体地，首先，将与第一图像帧的距离小于或等于第四预设距离的三维网格作为候选三维网格，因为在三维网格距离第一图像帧的距离超过第四预设距离的情况下，这部分的三维网格中的三维点的坐标误差可能相对较大，若使用这部分三维网格构建结构化平面，可能导致构建的结构化平面精度较低，这里的第四预设距离可根据具体场景和需求设置，此处不对其做具体规定；接着，选取候选三维网格之间，方向之差小于或等于预设方向差值和/或距离之差小于或等于预设距离差值的三维网格，作为第二三维网格。其中，这里的方向之差小于或等于预设方向差值指的是三维向量的差值均小于或等于预设方向差值。进一步地，在本公开实施例中，只有方向之差小于或等于预设方向差值且距离之差小于或等于预设距离差值的情况下，才将候选网格作为第二三维网格。例如，两个三维网格位于同一竖直平面上时，可以将这两个三维网格作为第二三维网格，并生成一个包括这两个三维网格的平面。当然，若在第一图像帧以前，若干历史图像帧已经构建了部分结构化平面，则可以在已构建的部分结构化平面上利用第一图像帧中的信息对已构建的结构化平面进行扩张。此时，选取三维网格之间，方向之差小于或等于预设方向差值和/或距离之差小于或等于预设距离差值的三维网格时，可以同时执行选取三维网格与已构建的结构化平面之间的方向之差小于或等于预设方向差值，和/或距离之差小于或等于第二预设差值的三维网格，并将选择出的三维网格与对应的结构化平面组成一个新的结构化平面。即，通过此种方式，实现了对结构化平面的扩张，并且对当前帧的定位可以参考之前的信息，使得定位结果更加准确。
通过对第一图像帧进行三角剖分得到二维网格组，并利用二维网格组得到三维网格组，然后获取三维网格组中满足预设条件的三维网格生成结构化平面，并非任意两个三维网格之间都能生成结构化平面，使得构建的结构化平面的精度更高。
进一步地，通过设置第二预设差值和第四预设距离挑选三维网格生成结构化平面，能够提高结构化平面的准确度。
为更好地理解本公开实施例中获取第一关联关系的方式，请参阅图2，图2是本申请定位方法一实施例的流程示意图二。在本公开实施例中，该定位方法包括以下步骤S21至步骤S27。
步骤S21：获取由先验结构化平面。
获取先验结构化平面的方式如上所述，此处不再赘述。
步骤S22：获取设备拍摄的当前图像帧。
获取当前图像帧的方式如上述步骤S12所述，此处不再赘述。
步骤S23：依据当前图像帧与历史图像帧之间的相对位置关系，确定当前图像帧在第二坐标系中的第二位姿。
如上述，在获取第一关联关系之前，需要获取当前图像帧的第二位姿。在本公开实 施例中，获取当前图像帧的第二位姿的方式可以是：先获取当前图像帧和一历史图像帧之间的相对位置关系，再基于历史图像帧的位姿和该获取的相对位置关系确定当前图像帧的初始位姿。这里，相对位置关系可以包括相对距离和相对角度。在本公开实施例中，假定将一个距离作为当前图像帧和历史图像帧之间的相对距离，并将当前图像帧和历史图像帧之间的传感器读数进行预积分得到二者之间的相对角度。因历史图像帧的位姿已知，通过二者之间的相对距离和相对角度，即可确定当前图像帧的第二位姿。
步骤S24：获取三维点在第一坐标系下的位置参数。
获取三维点在第一坐标系下的位置参数的方式如上所述，此处不再赘述。
步骤S25：对三维点和先验结构化平面进行点面匹配，得到第二关联关系。
具体地，对三维点和先验结构化平面进行点面匹配，得到第二关联关系的方式如上所述，此处不再赘述。
步骤S26：基于第二关联关系，得到第一关联关系。
基于第二关联关系，得到第一关联关系的方式如上所述，此处不再赘述。
步骤S27：利用第一关联关系获取设备的定位结果。
利用第一关联关系获取设备的定位结果的方式如上所述，此处不再赘述。
上述方案，通过在获取当前图像帧之前获取已构建好的先验结构化平面，相比于实时构建的结构化平面而言，前者精度更高。并且通过构建当前图像帧和历史图像帧中的二维特征点与先验结构化平面之间的关联关系，获取设备的定位结果，能够提高设备的定位精度。
本领域技术人员可以理解，在具体实施方式的上述方法中，各步骤的撰写顺序并不意味着严格的执行顺序而对实施过程构成任何限定，各步骤的具体执行顺序应当以其功能和可能的内在逻辑确定。
定位方法的执行主体可以是定位装置，例如，定位方法可以由终端设备或服务器或其它处理设备执行，其中，终端设备可以为具有视觉定位、三维重建、图像配准等需求的用户设备(User Equipment，UE)、移动设备、用户终端、终端、蜂窝电话、无绳电话、个人数字处理(Personal Digital Assistant，PDA)、手持设备、计算设备、车载设备、可穿戴设备以及自动驾驶汽车，有定位及建图需求的机器人，有配准需求的医疗成像系统，用于增强现实或虚拟现实的眼镜、头盔等产品等。在一些可能的实现方式中，该定位方法可以通过处理器调用存储器中存储的计算机可读指令的方式来实现。
请参阅图3，图3是本申请定位装置一实施例的结构示意图。定位装置30包括平面获取模块31、图像获取模块32、关联关系确定模块33以及定位模块34。平面获取模块31用于获取先验结构化平面；图像获取模块32用于获取设备拍摄的当前图像帧；关联关系确定模块33用于确定当前图像帧以及至少一历史图像帧之间的匹配特征点对，并确定匹配特征点对与先验结构化平面之间的第一关联关系，先验结构化平面是在执行获取设备拍摄的当前图像帧之前构建；定位模块34用于利用第一关联关系获取设备的定位结果。
上述方案，通过在获取当前图像帧之前获取已构建好的先验结构化平面，相比于实时构建的结构化平面而言，前者精度更高。并且通过构建当前图像帧和历史图像帧之间的二维特征点与先验结构化平面之间的关联关系，获取设备的定位结果，能够提高设备的定位精度。
在一些公开实施例中，关联关系确定模块33确定匹配特征点对与先验结构化平面之间的第一关联关系，包括：分别获取匹配特征点对对应的三维点以及先验结构化平面在基准坐标系下的位置参数；基于位置参数对三维点与先验结构化平面进行点面匹配，以获取三维点与先验结构化平面之间的第二关联关系；依据第二关联关系，得到第一关联关系；其中，基准坐标系为第一坐标系或第二坐标系，第一坐标系为先验结构化平面对应的世界坐标系，第二坐标系为当前图像帧对应的世界坐标系，其中，第一坐标系与第二坐标系相同或不同。
上述方案，通过获取匹配特征点对对应的三维点以及先验结构化平面在基准坐标系的位置参数，能够方便第二关联关系的构建，从而得到第一关联关系。
在一些公开实施例中，基准坐标系为第一坐标系；关联关系确定模块33分别获取匹配特征点对对应的三维点以及先验结构化平面在基准坐标系下的位置参数，包括：针对多个匹配特征点对对应的三维点中的每个三维点，获取三维点与当前图像帧之间的第一距离；选取第一距离小于或等于第一预设距离的三维点作为第一三维点；获取第一三维点在第一坐标系下的位置参数用于与先验结构化平面进行点面匹配。
在一些公开实施例中，关联关系确定模块33分别获取匹配特征点对对应的三维点以及先验结构化平面在基准坐标系下的位置参数，包括：获取当前图像帧在第一坐标系下的第一位姿；基于第一位姿，选择与当前图像帧之间的第二距离小于或等于第二预设距离的先验结构化平面作为第一先验结构化平面；获取第一先验结构化平面在第一坐标系下的位置参数用于与三维点进行点面匹配。
上述方案，因为先验结构化平面的数量相对于匹配特征点对对应的三维点的数量而言更好，通过获取三维点在第一坐标系下的位置参数，以便在第一坐标系下构建第二关联关系，减少了设备的工作量。并且，通过利用当前图像帧对三维点以及先验结构化平面进行筛选，使得选取距离当前图像帧较近的三维点和/或先验结构化平面在减少工作量的同时，也能够提高点面匹配的精度。
在一些公开实施例中，关联关系确定模块33获取三维点与当前图像帧之间的第一距离，包括：对当前图像帧进行三角剖分，得到关于当前图像帧的二维网格组，其中，二维网格组中的顶点为匹配特征点对包括的在当前图像帧上的特征点；将二维网格组投影到第二坐标系下，得到关于当前图像帧的三维网格组，该三维网格组包括若干个三维网格，其中，三维网格中的各顶点为三维点；将三维网格与当前图像帧之间的距离作为第一距离。
上述方案，通过构建三维网格与先验结构化平面之间的关联关系，相对于构建单个三维点与先验结构化平面之间的关联关系而言，前者获得的关联关系更为准确，精 度更高。
在一些公开实施例中，关联关系确定模块33基于位置参数对三维点与先验结构化平面进行点面匹配，包括：从三维网格组中的所有三维网格选取第一三维网格，其中，第一三维网格的各顶点与先验结构化平面之间的第一距离均小于或等于第三预设距离，或者第一三维网格的所有顶点与先验结构化平面之间的第三距离小于或等于第三预设距离、且所有顶点组成的平面与先验结构化平面平行；构建第一三维网格与先验结构化平面之间的关联关系作为第二关联关系。
上述方案，通过选择三维网格中顶点与先验结构化平面之间的第三距离小于或等于第三预设距离，或者进一步在三维网格与先验结构化平面平行的情况下，才构建三维网格与先验结构化平面之间的关联关系，使得最后构建的关联关系更为准确。
在一些公开实施例中，关联关系确定模块33分别获取匹配特征点对对应的三维点以及先验结构化平面在基准坐标系下的位置参数，或获取当前图像帧在第一坐标系下的第一位姿包括：获取三维点在第二坐标系下的位置参数，并利用转换参数将位置参数转换为第一坐标系下的位置参数；或，获取当前图像帧在第二坐标系下的第二位姿，并利用转换参数将第二位姿转换为第一坐标系下的第一位姿。
上述方案，通过先在第二坐标系下获取三维点的坐标以及当前图像帧的坐标，并利用转换参数转换到第一坐标系下，能够提高获取三维点以及当前图像帧在第一坐标系下的位置参数的速度。
在一些公开实施例中，关联关系确定模块33利用第一关联关系获取设备的定位结果包括：利用第一关联关系构建点面约束模型，点面约束模型用于获取设备的定位结果，其中，点面约束模型包括点面优化方程，点面优化方程包括第一项和第二项，第一项和第二项分别位于点面优化方程的等号的两边；依据当前图像帧和历史图像帧的旋转矩阵和平移矩阵、先验结构化平面的方向矩阵以及距离矩阵、历史图像帧中的特征点的二维坐标，确定当前图像帧中对应的特征点的预测坐标，其中，预测坐标作为第一项；调整点面优化方程中的预设参数，以使第一项与第二项相等，其中，第二项为当前图像帧中对应的特征点的二维坐标，预设参数包括当前图像帧的旋转矩阵以及平移矩阵。
上述方案，通过构建二维特征点与结构化平面之间的关联关系，对包含当前图像帧初始位姿的预设参数进行优化，能够提高对设备的定位精度。
在一些公开实施例中，历史图像帧为当前图像帧以前的任一历史图像帧；定位模块34利用第一关联关系获取设备的定位结果，包括：响应于历史图像帧为当前图像帧的上一历史图像帧，获取设备关于当前图像帧对应的定位结果并更新设备关于历史图像帧对应的定位结果；响应于历史图像帧不为当前图像帧的上一历史图像帧，获取设备关于当前图像帧对应的定位结果。
上述方案，在历史图像帧为当前图像帧的上一历史图像帧的情况下，不仅能够对当前图像帧的位姿进行优化，还能对历史图像帧的位姿进行优化，从而提高多个时刻下设备的定位结果的精度。
在一些公开实施例中，定位模块34利用第一关联关系获取设备的定位结果，包括：将点面约束模型与重投影约束模型、IMU约束模型中的至少一个融合，得到融合约束模型，其中，融合约束模型用于获取设备的定位结果。
上述方案，通过构建融合约束模型对设备的当前图像帧的位姿进行优化，能够提高对设备的定位精度。
上述方案，通过在获取当前图像帧之前获取已构建好的先验结构化平面，相比于实时构建的结构化平面而言，前者精度更高。并且通过构建当前图像帧和历史图像帧中的二维特征点与先验结构化平面之间的关联关系，获取设备的定位结果，能够提高设备的定位精度。
请参阅图4，图4是本申请电子设备一实施例的结构示意图。电子设备40包括存储器41和处理器42，处理器42用于执行存储器41中存储的程序指令，以实现上述定位方法实施例中的步骤。在一个具体的实施场景中，电子设备40可以包括但不限于：微型计算机、服务器，此外，电子设备40还可以包括笔记本电脑、平板电脑等移动设备，在此不做限定。
具体而言，处理器42用于控制其自身以及存储器41以实现上述定位方法实施例中的步骤。处理器42还可以称为CPU(Central Processing Unit，中央处理单元)。处理器42可能是一种集成电路芯片，具有信号的处理能力。处理器42还可以是通用处理器、数字信号处理器(Digital Signal Processor，DSP)、专用集成电路(Application Specific Integrated Circuit，ASIC)、现场可编程门阵列(Field-Programmable Gate Array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。另外，处理器42可以由集成电路芯片共同实现。
上述方案，通过在获取当前图像帧之前获取已构建好的先验结构化平面，相比于实时构建的结构化平面而言，前者精度更高。并且通过构建当前图像帧和历史图像帧中的二维特征点与先验结构化平面之间的关联关系，获取设备的定位结果，能够提高设备的定位精度。
请参阅图5，图5为本申请计算机可读存储介质一实施例的结构示意图。计算机可读存储介质50存储有能够被处理器运行的程序指令501，程序指令501用于实现上述定位方法实施例中的步骤。
上述方案，通过在获取当前图像帧之前获取已构建好的先验结构化平面，相比于实时构建的结构化平面而言，前者精度更高。并且通过构建当前图像帧和历史图像帧中的二维特征点与先验结构化平面之间的关联关系，获取设备的定位结果，能够提高设备的定位精度。
在一些实施例中，本公开实施例提供的装置具有的功能或包含的模块可以用于执行上文方法实施例描述的方法，其具体实现可以参照上文方法实施例的描述，为了简洁，这里不再赘述。
上文对各个实施例的描述倾向于强调各个实施例之间的不同之处，其相同或相似之处可以互相参考，为了简洁，本文不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的方法和装置，可以通过其它的方式实现。例如，以上所描述的装置实施方式仅仅是示意性的，例如，模块或单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性、机械或其它的形式。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)或处理器(processor)执行本申请各个实施方式方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、磁碟或者光盘等各种可以存储程序代码的介质。

Claims (13)

  1. 一种定位方法，其特征在于，包括：
    获取先验结构化平面；
    获取设备拍摄的当前图像帧；
    确定所述当前图像帧以及至少一历史图像帧之间的匹配特征点对；
    确定所述匹配特征点对与所述先验结构化平面之间的第一关联关系；
    利用所述第一关联关系获取所述设备的定位结果。
  2. 根据权利要求1所述的方法，其特征在于，所述确定所述匹配特征点对与所述先验结构化平面之间的第一关联关系，包括：
    分别获取所述匹配特征点对对应的三维点以及所述先验结构化平面在基准坐标系下的位置参数；
    基于所述位置参数对所述三维点与所述先验结构化平面进行点面匹配，以获取所述三维点与所述先验结构化平面之间的第二关联关系；
    依据所述第二关联关系，得到所述第一关联关系；
    其中，所述基准坐标系为第一坐标系或第二坐标系，所述第一坐标系为所述先验结构化平面对应的世界坐标系，所述第二坐标系为所述当前图像帧对应的世界坐标系，其中，所述第一坐标系与所述第二坐标系相同或不同。
  3. 根据权利要求2所述的方法，其特征在于，所述基准坐标系为所述第一坐标系，所述分别获取所述匹配特征点对对应的三维点以及所述先验结构化平面在基准坐标系下的位置参数，包括：
    针对多个所述匹配特征点对对应的三维点中的每个三维点，获取所述三维点与所述当前图像帧之间的第一距离；
    选取所述第一距离小于或等于第一预设距离的三维点作为第一三维点；
    获取所述第一三维点在所述第一坐标系下的位置参数用于与所述先验结构化平面进行点面匹配。
  4. 根据权利要求2或3所述的方法，其特征在于，所述分别获取所述匹配特征点对对应的三维点以及所述先验结构化平面在基准坐标系下的位置参数，包括：
    获取所述当前图像帧在所述第一坐标系下的第一位姿；
    基于所述第一位姿，选择与所述当前图像帧之间的第二距离小于或等于第二预设距离的先验结构化平面作为第一先验结构化平面；
    获取所述第一先验结构化平面在所述第一坐标系下的位置参数用于与所述三维点进行点面匹配。
  5. 根据权利要求3所述的方法，其特征在于，所述获取所述三维点与所述当前图像帧之间的第一距离，包括：
    对所述当前图像帧进行三角剖分，得到关于所述当前图像帧的二维网格组，其中，所述二维网格组中的顶点为所述匹配特征点对包括的在所述当前图像帧上的特征点；
    将所述二维网格组投影到所述第二坐标系下，得到关于所述当前图像帧的三维网格组，所述三维网格组包括若干个三维网格，其中，所述三维网格中的各顶点为所述三维点；
    将所述三维网格与所述当前图像帧之间的距离作为所述第一距离。
  6. 根据权利要求5所述的方法，其特征在于，所述基于所述位置参数对所述三维点与所述先验结构化平面进行点面匹配，包括：
    从所述三维网格组中的所有三维网格选取第一三维网格，其中，
    所述第一三维网格的各顶点与所述先验结构化平面之间的第一距离均小于或等于第三预设距离，或者
    所述第一三维网格的所有顶点与所述先验结构化平面之间的第三距离小于或等于第三预设距离、且所有顶点组成的平面与所述先验结构化平面平行；
    构建所述第一三维网格与所述先验结构化平面之间的关联关系作为所述第二关联关系。
  7. 根据权利要求4所述的方法，其特征在于，所述分别获取所述匹配特征点对对应的三维点以及所述先验结构化平面在基准坐标系下的位置参数，或获取所述当前图像帧在所述第一坐标系下的第一位姿包括：
    获取所述三维点在所述第二坐标系下的位置参数，并利用转换参数将所述位置参数转换为所述第一坐标系下的位置参数；
    或，获取所述当前图像帧在所述第二坐标系下的第二位姿，并利用所述转换参数将所述第二位姿转换为所述第一坐标系下的第一位姿。
  8. 根据权利要求1-7任一项所述的方法，其特征在于，所述利用所述第一关联关系获取所述设备的定位结果，包括：
    利用所述第一关联关系构建点面约束模型，所述点面约束模型用于获取设备的定位结果，其中，所述点面约束模型包括点面优化方程，所述点面优化方程包括第一项和第二项，所述第一项和第二项分别位于所述点面优化方程的等号的两边；
    依据所述当前图像帧和所述历史图像帧的旋转矩阵和平移矩阵、所述先验结构化平面的方向矩阵以及距离矩阵、所述历史图像帧中的特征点的二维坐标，确定所述当前图像帧中对应的特征点的预测坐标，其中，所述预测坐标作为所述第一项；
    调整所述点面优化方程中的预设参数，以使所述第一项与所述第二项相等，其中，所述第二项为所述当前图像帧中对应的特征点的二维坐标，所述预设参数包括所述当前图像帧的旋转矩阵以及平移矩阵。
  9. 根据权利要求8所述的方法，其特征在于，所述利用所述第一关联关系构建点面约束模型，包括：
    将所述点面约束模型与重投影约束模型、IMU约束模型中的至少一个融合，得到融合约束模型；所述融合约束模型用于获取所述设备的定位结果。
  10. 根据权利要求1至9任一项所述的方法，其特征在于，所述利用所述第一关联关系获取所述设备的定位结果，包括：
    响应于所述历史图像帧为所述当前图像帧的上一历史图像帧，获取所述设备关于所述当前图像帧对应的定位结果并更新所述设备关于所述历史图像帧对应的定位结果；
    响应于所述历史图像帧不为所述当前图像帧的上一历史图像帧，获取所述设备关于所述当前图像帧对应的定位结果。
  11. 一种视觉定位装置，其特征在于，包括：
    平面获取模块，用于获取先验结构化平面；
    图像获取模块，用于获取设备拍摄的当前图像帧；
    关联关系确定模块，用于确定所述当前图像帧以及至少一历史图像帧之间的匹配特征点对，并确定所述匹配特征点对与所述先验结构化平面之间的第一关联关系；
    定位模块，用于利用所述第一关联关系获取所述设备的定位结果。
  12. 一种电子设备，其特征在于，包括存储器和处理器，所述处理器用于执行所述存储器中存储的程序指令，以实现权利要求1至10任一项所述的方法。
  13. 一种计算机可读存储介质，其上存储有程序指令，其特征在于，所述程序指令被处理器执行时实现权利要求1至10任一项所述的方法。
PCT/CN2022/088754 2021-05-27 2022-04-24 定位方法和装置、电子设备及存储介质 WO2022247548A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110606804.6A CN113361365B (zh) 2021-05-27 2021-05-27 定位方法和装置、设备及存储介质
CN202110606804.6 2021-05-27

Publications (1)

Publication Number Publication Date
WO2022247548A1 true WO2022247548A1 (zh) 2022-12-01

Family

ID=77530987

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/088754 WO2022247548A1 (zh) 2021-05-27 2022-04-24 定位方法和装置、电子设备及存储介质

Country Status (2)

Country Link
CN (1) CN113361365B (zh)
WO (1) WO2022247548A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117115414A (zh) * 2023-10-23 2023-11-24 西安羚控电子科技有限公司 基于深度学习的无gps无人机定位方法及装置

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113361365B (zh) * 2021-05-27 2023-06-23 浙江商汤科技开发有限公司 定位方法和装置、设备及存储介质
CN113808196A (zh) * 2021-09-09 2021-12-17 浙江商汤科技开发有限公司 平面融合定位方法、装置、电子设备及存储介质
CN113847907B (zh) * 2021-09-29 2024-09-13 深圳市慧鲤科技有限公司 定位方法及装置、设备、存储介质
CN114998433A (zh) * 2022-05-31 2022-09-02 Oppo广东移动通信有限公司 位姿计算方法、装置、存储介质以及电子设备

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109544615A (zh) * 2018-11-23 2019-03-29 深圳市腾讯信息技术有限公司 基于图像的重定位方法、装置、终端及存储介质
CN110490131A (zh) * 2019-08-16 2019-11-22 北京达佳互联信息技术有限公司 一种拍摄设备的定位方法、装置、电子设备及存储介质
US20200327683A1 (en) * 2020-02-06 2020-10-15 Tianzhi Yang Generating and evaluating mappings between spatial point sets
CN112146649A (zh) * 2020-09-23 2020-12-29 北京市商汤科技开发有限公司 Ar场景下的导航方法、装置、计算机设备及存储介质
WO2020259360A1 (zh) * 2019-06-28 2020-12-30 Oppo广东移动通信有限公司 定位方法及装置、终端、存储介质
CN113361365A (zh) * 2021-05-27 2021-09-07 浙江商汤科技开发有限公司 定位方法和装置、设备及存储介质

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11199414B2 (en) * 2016-09-14 2021-12-14 Zhejiang University Method for simultaneous localization and mapping
CN108732584B (zh) * 2017-04-17 2020-06-30 百度在线网络技术（北京）有限公司 用于更新地图的方法和装置
CN110246147B (zh) * 2019-05-14 2023-04-07 中国科学院深圳先进技术研究院 视觉惯性里程计方法、视觉惯性里程计装置及移动设备
CN110349213B (zh) * 2019-06-28 2023-12-12 Oppo广东移动通信有限公司 基于深度信息的位姿确定方法、装置、介质与电子设备
CN112785705B (zh) * 2021-01-21 2024-02-09 中国科学技术大学 一种位姿获取方法、装置及移动设备

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109544615A (zh) * 2018-11-23 2019-03-29 深圳市腾讯信息技术有限公司 基于图像的重定位方法、装置、终端及存储介质
WO2020259360A1 (zh) * 2019-06-28 2020-12-30 Oppo广东移动通信有限公司 定位方法及装置、终端、存储介质
CN110490131A (zh) * 2019-08-16 2019-11-22 北京达佳互联信息技术有限公司 一种拍摄设备的定位方法、装置、电子设备及存储介质
US20200327683A1 (en) * 2020-02-06 2020-10-15 Tianzhi Yang Generating and evaluating mappings between spatial point sets
CN112146649A (zh) * 2020-09-23 2020-12-29 北京市商汤科技开发有限公司 Ar场景下的导航方法、装置、计算机设备及存储介质
CN113361365A (zh) * 2021-05-27 2021-09-07 浙江商汤科技开发有限公司 定位方法和装置、设备及存储介质

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117115414A (zh) * 2023-10-23 2023-11-24 西安羚控电子科技有限公司 基于深度学习的无gps无人机定位方法及装置
CN117115414B (zh) * 2023-10-23 2024-02-23 西安羚控电子科技有限公司 基于深度学习的无gps无人机定位方法及装置

Also Published As

Publication number Publication date
CN113361365A (zh) 2021-09-07
CN113361365B (zh) 2023-06-23

Similar Documents

Publication Publication Date Title
WO2022247548A1 (zh) 定位方法和装置、电子设备及存储介质
US11295472B2 (en) Positioning method, positioning apparatus, positioning system, storage medium, and method for constructing offline map database
US20210233275A1 (en) Monocular vision tracking method, apparatus and non-transitory computer-readable storage medium
WO2022247286A1 (zh) 定位方法和装置、设备及存储介质
US11145083B2 (en) Image-based localization
WO2020001168A1 (zh) 三维重建方法、装置、设备和存储介质
JP2020507850A (ja) 画像内の物体の姿の確定方法、装置、設備及び記憶媒体
TWI795885B (zh) 視覺定位方法、設備和電腦可讀儲存介質
CN115690382B (zh) 深度学习模型的训练方法、生成全景图的方法和装置
KR102566300B1 (ko) 실내 측위 방법, 장치, 장비 및 저장 매체
JP2017036970A (ja) 情報処理装置、情報処理方法、プログラム
CN112200157A (zh) 一种降低图像背景干扰的人体3d姿态识别方法及其系统
US8509522B2 (en) Camera translation using rotation from device
CN114266823A (zh) 一种结合SuperPoint网络特征提取的单目SLAM方法
CN113610702B (zh) 一种建图方法、装置、电子设备及存储介质
CN114882106A (zh) 位姿确定方法和装置、设备、介质
JP2018173882A (ja) 情報処理装置、方法、及びプログラム
US20240029350A1 (en) Computing apparatus and model generation method
JP2009186287A (ja) 平面パラメータ推定装置、平面パラメータ推定方法及び平面パラメータ推定プログラム
CN115578432B (zh) 图像处理方法、装置、电子设备及存储介质
TWI822423B (zh) 運算裝置及模型產生方法
WO2022193104A1 (zh) 一种光场预测模型的生成方法及相关装置
JP7074694B2 (ja) 情報端末装置及びプログラム
Qiu et al. MAC-VO: Metrics-aware Covariance for Learning-based Stereo Visual Odometry
CN115170914A (zh) 位姿估计方法、装置、电子设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22810281

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22810281

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 22810281

Country of ref document: EP

Kind code of ref document: A1