WO2021057742A1 - Positioning method and apparatus, device, and storage medium - Google Patents

Abstract

Disclosed in embodiments of the present application are a positioning method and apparatus, a device, and a storage medium. The method comprises: determining a feature point in an image to be processed collected by an image collection device; obtaining attribute information of the feature point; and matching the attribute information of the feature point with attribute information of a plurality of sampling points in a pre-constructed point cloud map to obtain a positioning result of the image collection device, wherein world coordinates in the attribute information of the sampling points are the world coordinates of sampling points in a sample image.

Description

Positioning method and device, equipment and storage medium

Cross-references to related applications

This application is filed based on the Chinese patent application with the application number 201910921484.6 and the filing date on September 27, 2019, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby introduced in this application in full .

Technical field

The embodiments of the present application relate to electronic technology, and relate to, but are not limited to, positioning methods and devices, equipment, and storage media.

Background technique

Among the related technologies for positioning based on image information, currently, people and fixed objects in the images collected by the camera module are mainly identified to determine the position of the person. This solution matches the fixed object in the image with a pre-built indoor map to determine the corresponding position of the fixed object indoors; then, the indoor position of the person is determined according to the position of the fixed object; wherein, the position of the person is determined The overall idea of the location is: to identify the fixed object in the image through the image recognition method, and determine the position of the person according to the relative position relationship between the fixed object and the person in the image and the location of the fixed object in the room.

However, this positioning method is mainly based on the relative positional relationship between the characters and fixed objects in the image; in this way, the electronic device requires recognizable characters and fixed objects in the image when realizing positioning; otherwise, the positioning is Will fail. Therefore, the robustness of this positioning method is poor.

Summary of the invention

In view of this, the positioning method, device, device, and storage medium provided by the embodiments of the present application do not depend on the fixed object and the object to be positioned in the image, so it has better robustness. The technical solutions of the embodiments of the present application are implemented as follows:

The positioning method provided by the embodiments of the present application includes: determining feature points in an image to be processed collected by an image collection device; acquiring attribute information of the feature points; and comparing the attribute information of the feature points with a pre-built point cloud map The attribute information of the multiple sampling points in the data is matched to obtain the positioning result of the image acquisition device.

The positioning device provided by the embodiment of the present application includes: a first determining module configured to determine a feature point in an image to be processed collected by an image acquisition device; an attribute information obtaining module configured to obtain attribute information of the feature point; positioning The module is configured to match the attribute information of the characteristic point with the attribute information of a plurality of sampling points in a pre-built point cloud map to obtain a positioning result of the image acquisition device.

The electronic device provided by the embodiment of the present application includes a memory and a processor, the memory stores a computer program that can run on the processor, and the processor implements the steps in the positioning method when the program is executed.

The embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above-mentioned positioning method are realized.

In the positioning method provided in the embodiment of the present application, according to the attribute information of the feature points in the image to be processed and the attribute information of multiple sampling points in the pre-built point cloud map, the method used to collect the image to be processed can be determined The positioning result of the image acquisition device; in this way, when positioning the object to be positioned carrying the image acquisition device, the positioning method does not depend on the fixed object and the object to be positioned in the image to be processed, so better results can be obtained Robustness.

Description of the drawings

The drawings here are incorporated into the specification and constitute a part of the specification. These drawings show embodiments that conform to the application, and are used in the specification to illustrate the technical solution of the application.

FIG. 1 is a schematic diagram of an implementation process of a positioning method according to an embodiment of this application;

2 is a schematic diagram of determining camera coordinates of multiple first target sampling points according to an embodiment of the application;

FIG. 3 is a schematic diagram of an implementation process of a positioning method according to an embodiment of the application;

4 is a schematic diagram of the implementation process of the method for constructing a point cloud map according to an embodiment of the application;

FIG. 5 is a schematic diagram of a feature point matching pair according to an embodiment of the application;

6A is a schematic structural diagram of a positioning device according to an embodiment of the application;

6B is a schematic structural diagram of another positioning device according to an embodiment of the application;

FIG. 7 is a schematic diagram of a hardware entity of an electronic device according to an embodiment of the application.

detailed description

In order to make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the specific technical solutions of the present application will be described in further detail below in conjunction with the drawings in the embodiments of the present application. The following examples are used to illustrate the application, but are not used to limit the scope of the application.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field of this application. The terminology used herein is only for the purpose of describing the embodiments of the application, and is not intended to limit the application.

In the following description, “some embodiments” are referred to, which describe a subset of all possible embodiments, but it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments, and Can be combined with each other without conflict.

It should be pointed out that the term "first\second\third" involved in the embodiments of this application is used to distinguish different objects, and does not represent a specific order for the objects. Understandably, "first\second\third" Where permitted, the specific order or sequence can be interchanged, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein.

The embodiments of the application provide a positioning method, which can be applied to electronic devices, which can be mobile phones, tablet computers, notebook computers, desktop computers, servers, robots, drones and other devices with information processing capabilities. . The functions implemented by the positioning method can be implemented by a processor in the electronic device calling program code. Of course, the program code can be stored in a computer storage medium. It can be seen that the electronic device includes at least a processor and a storage medium.

FIG. 1 is a schematic diagram of the implementation process of the positioning method according to the embodiment of this application. As shown in FIG. 1, the method may include the following steps S101 to S103:

Step S101: Determine the characteristic points in the image to be processed collected by the image collecting device.

Understandably, the feature point is the pixel point with a certain feature in the image to be processed. When the electronic device implements step S101, it usually takes the corner points in the image to be processed as feature points. Generally speaking, the image to be processed is a two-dimensional image. For example, the image to be processed is a red, green, and blue (Red, Green, Blue, RGB) image. Of course, the image to be processed can also be an image in other formats, for example, the image is grayscale. image.

In the embodiments of the present application, the image acquisition device may be various. For example, the image acquisition device is a monocular camera or a multi-camera (for example, a binocular camera). It should be noted that the electronic device may include an image capture device, that is, the image capture device is installed in the electronic device. For example, the electronic device is a smart phone with at least one camera. Of course, in some embodiments, the electronic device may not include an image capture device, and the image capture device may send the captured image to the electronic device.

Step S102: Obtain the attribute information of the characteristic point.

Understandably, the attribute information of the feature point is unique information of the feature point. The attribute information of the feature point includes at least one of the following: image feature and camera coordinates. In some embodiments, the attribute information of the feature point includes image features and camera coordinates. The image feature can be a feature descriptor of the feature point, or other information that can describe the image feature of the feature point. Understandably, the camera coordinates of the feature point refer to the coordinates of the feature point in the camera coordinate system. The camera coordinates can be two-dimensional coordinates or three-dimensional coordinates.

Step S103: Match the attribute information of the characteristic point with the attribute information of multiple sampling points in a pre-built point cloud map to obtain a positioning result of the image acquisition device.

In physical space, electronic devices can obtain sampling points on the surface of an object through image acquisition, and construct a point cloud map based on the world coordinates of these sampling points. That is, the world coordinates in the attribute information of the sampling point are the world coordinates of the sampling point in the sample image. The point cloud map construction process can be implemented through steps S801 to S805 in the following embodiments. The type of point cloud map can be sparse point cloud or dense point cloud. In the sparse point cloud, the spacing between sampling points is greater than the spacing threshold, and the attribute information of the sampling points can be the world coordinates and image features of the sampling points; in the dense point cloud, the spacing between the sampling points is less than the spacing threshold, sampling The attribute information of the point may include the world coordinates of the sampling point, but not the image characteristics of the sampling point. Of course, the dense point cloud can also include the image features of the sampling points, but this greatly increases the data volume of the dense point cloud, which consumes a lot of storage resources of the memory. For the point cloud map corresponding to the same physical area, the number of sampling points of the sparse point cloud is much smaller than the number of sampling points of the dense point cloud.

It is understandable that the sampling point is actually a feature point in the sample image where it is located, and the attribute information of the sampling point is unique information of the sampling point. It's just that the point is called the sampling point in the point cloud map, and the point is called the feature point in the image, so that the reader can clearly distinguish whether these points are points in the point cloud map or points in the image. The attribute information of the sampling point includes at least one of the following: image characteristics and world coordinates. In some embodiments, the attribute information of the sampling point includes image features and world coordinates. In other embodiments, the attribute information of the sampling point includes world coordinates, but does not include image features. Understandably, the world coordinates of the sampling point refer to the coordinates of the sampling point in the world coordinate system. The world coordinates can be two-dimensional coordinates or three-dimensional coordinates.

In the positioning method provided by the embodiment of the present application, the electronic device can determine the attribute information of the feature points in the image to be processed and the attribute information of multiple sampling points in the pre-built point cloud map to determine that it is used to collect the to-be-processed image. The image is the positioning result of the image capture device. In this way, when the electronic device locates the image acquisition device, its positioning method does not depend on the fixed object and the object to be positioned in the image to be processed, so better robustness can be obtained.

It should be noted that the attribute information of the sampling points in the point cloud map includes image features and does not include image features, and the corresponding positioning methods are different. In the case where the attribute information of the sampling point includes image features and world coordinates, the positioning method includes the following embodiments.

The embodiment of the present application provides a positioning method, and the method may include the following steps S201 to S204:

Step S201, determining feature points in the image to be processed collected by the image collecting device;

Step S202: Acquire the image feature of the feature point and the camera coordinates of the feature point;

Step S203: Match the image features of the feature points with the image features of multiple sampling points in a pre-built point cloud map to obtain a first target sampling point.

Understandably, the purpose of matching is to be able to find out the target sampling points that represent the same spatial location point as the feature point from multiple sampling points. In some embodiments, a sampling point that is the same or similar to the image feature of the feature point in the point cloud map is usually determined as the first target sampling point. For example, the electronic device determines the first target sampling point through step S303 and step S304 in the following embodiment.

Step S204: Determine the positioning result of the image acquisition device according to the camera coordinates of the feature point and the world coordinates of the first target sampling point.

In some embodiments, the point cloud map includes image features and world coordinates of multiple sampling points. Understandably, if the camera coordinates of a plurality of feature points and the world coordinates of the first target sampling point matching each of the feature points are known, the image acquisition device can be determined through steps S404 to S407 in the following embodiment World coordinates and orientation (ie the posture of the image capture device).

In the positioning method provided by the embodiment of the present application, the electronic device can more accurately determine the first matching point from the multiple sampling points based on the image features of the feature points and the image features of the multiple sampling points. A target sampling point, thereby improving positioning accuracy.

The embodiment of the present application further provides a positioning method, and the method may include the following steps S301 to S305:

Step S301, determining feature points in the image to be processed collected by the image collecting device;

Step S302, acquiring the image feature of the feature point and the camera coordinates of the feature point;

Step S303: Determine the similarity between the sampling point and the feature point according to the image feature of the sampling point and the image feature of the feature point.

Understandably, the similarity refers to the similarity between the image features of the sampling points and the image features of the feature points. When the electronic device implements step S303, it may determine the similarity through various methods. For example, the Euclidean distance between the image feature of the sampling point and the image feature of the feature point is determined, and the Euclidean distance is determined as the similarity. In some embodiments, the Hamming distance or cosine similarity between the image feature of the sampling point and the image feature of the feature point may also be determined, and the Hamming distance or cosine similarity is determined as the similarity. There is no limitation on the type of parameters that characterize the similarity. The parameter type may be the Euclidean distance, Hamming distance, or cosine similarity.

Step S304: Determine the sampling point whose similarity with the feature point meets the first condition as the first target sampling point.

In some embodiments, when the electronic device implements step S304, it may determine a sampling point with a similarity between the plurality of sampling points and a characteristic point that is less than or equal to a similarity threshold as the first target sampling point. For example, the sampling point whose Euclidean distance with the feature point is less than or equal to the Euclidean distance threshold is determined as the first target sampling point; or, among the multiple sampling points, the sampling point with the smallest similarity to the feature point is determined The sampling point is determined as the first target sampling point. That is, the first condition is that the similarity is less than or equal to the similarity threshold, or the first condition is that the similarity is the smallest.

Step S305: Determine the positioning result of the image acquisition device according to the camera coordinates of the feature point and the world coordinates of the first target sampling point.

In some embodiments, when the electronic device implements step S305, it can determine at least one of the following positioning results according to the camera coordinates of the feature point and the world coordinates of the first target sampling point matching the feature point: image acquisition The world coordinates of the device and the orientation (ie posture) of the image capture device. For example, through steps S404 to S407 in the following embodiment, the world coordinates of the image capture device and the orientation of the image capture device are determined.

The embodiment of the present application further provides a positioning method, and the method may include the following steps S401 to S407:

Step S401, determining feature points in the image to be processed collected by the image collecting device;

Step S402, acquiring the image feature of the feature point and the camera coordinates of the feature point;

Step S403: Match the image features of the feature points with the image features of multiple sampling points in a pre-built point cloud map to obtain a first target sampling point.

In some embodiments, the point cloud map includes attribute information of multiple sampling points, and the attribute information of each sampling point includes image features and world coordinates.

Step S404: Determine the camera coordinates of the multiple first target sampling points according to the world coordinates of the multiple first target sampling points and the camera coordinates of the feature points matching the multiple first target sampling points.

In some embodiments, the plurality of first target sampling points are at least three first target sampling points. That is, in step S404, according to the camera coordinates of the at least three feature points and the world coordinates of the first target sampling point that matches the three feature points, it can be accurately determined to match the three feature points. The camera coordinates of the first target sampling point.

For example, as shown in FIG. 2, point O is the origin of the camera coordinate system, that is, the optical center of the image capture device, and the multiple first target sampling points are the capitals A and B shown in FIG. The three sampling points of, C, in the image to be processed 20, the feature point that matches the sampling point A is the lowercase feature point a, and the feature point that matches the sampling point B is the lowercase feature point b, which matches the sampling point C The feature point of is the lowercase feature point c.

According to the law of cosines, the following formula (1) can be listed:

OA ² +OB ² -2·OA·OB·cos<a,b>=AB ²

OA ² +OC ² -2·OA·OC·cos<a,c>=AC ² (1);

OB ² +OC ² -2·OB·OC·cos<b,c>=BC ²

In formula (1), <a,b> refers to ∠aOb, <a,c> refers to ∠aOc, and <b,c> refers to ∠bOc. OA, OB, and OC are the distances between the origin of the coordinate system of the target local map and the target points A, B, and C, respectively.

Eliminate the above formula, divide by OC ² at the same time, and set

Then the following formula (2) can be obtained:

Then replace, so

Then the following formula (3) can be obtained:

x ² +y ² -2·x·y·cos<a,b>=u ²

x ² +1-2·x·cos<a,c>=wu (3);

y ² +1-2·y·cos<b,c>=vu

Incorporating the above formula (1) into formula (2) and formula (3), the following formula (4) can be obtained:

In formula (4), w, v, cos<a,c>, cos<b,c>, cos<a,b> are all known quantities, so there are only two unknown quantities x and y, so by the above The two equations in formula (4) can be used to obtain the values of x and y, and then the values of OA, OB and OC can be solved according to the three equations in the following formula (5):

Finally, the camera coordinates of the three sampling points A, B, and C are solved. According to the vector formula (6), we can get:

In formula (6),

The direction is from point O to point a;

The direction is from point O to point b;

The direction is from point O to point c.

Step S405: Determine a first rotation relationship and a first translation relationship of the camera coordinate system relative to the world coordinate system according to the world coordinates of the multiple first target sampling points and the camera coordinates of the multiple first target sampling points.

Understandably, if the world coordinates and camera coordinates of the multiple first target sampling points are known, the first rotation relationship and the first translation relationship of the camera coordinate system relative to the world coordinate system can be determined.

Step S406: Determine a positioning result of the image acquisition device according to the first translation relationship and the first rotation relationship; wherein the positioning result includes coordinate information and orientation information.

In the positioning method provided by the embodiment of the present application, according to the world coordinates of the multiple first target sampling points and the camera coordinates of the multiple first target sampling points, the first of the camera coordinate system relative to the world coordinate system is determined. The rotation relationship and the first translation relationship; in this way, the orientation of the image acquisition device can be determined according to the first translation relationship and the first rotation relationship, so that the positioning method can be applied to more application scenarios. For example, according to the current orientation of the robot, the robot is instructed to perform the next action. For another example, in a navigation application, knowing the user's orientation can more accurately guide the user in which direction to walk/drive. For another example, in an unmanned driving application, knowing the direction of the vehicle can more accurately control which direction the vehicle is traveling in.

In the case where the attribute information of the sampling points in the point cloud map includes world coordinates but does not include image features, the positioning method includes the following embodiments.

The embodiment of the present application provides a positioning method. FIG. 3 is a schematic diagram of the implementation process of the positioning method according to the embodiment of the present application. As shown in FIG. 3, the method may include the following steps S501 to S505:

Step S501, determining feature points in the image to be processed collected by the image collection device;

Step S502, acquiring the camera coordinates of the feature point;

Step S503: According to the iterative strategy, the camera coordinates of the multiple feature points are matched with the world coordinates of the multiple sampling points in the pre-built point cloud map, and the target rotation relationship and target translation of the camera coordinate system relative to the world coordinate system are obtained. relationship.

In some embodiments, the point cloud map includes the world coordinates of the sampling points, but does not include the image features of the sampling points. Understandably, when storing a point cloud map, image features generally occupy a relatively large storage space. For example, an image feature is a feature descriptor. Normally, the feature descriptor of each sampling point has 256 bytes, which requires the electronic device to allocate at least 256 bytes of storage space for each sampling point to store the feature description child. In some embodiments, the point cloud map does not include the image features of multiple sampling points. In this way, the data volume of the point cloud map can be greatly reduced, thereby saving the storage space of the point cloud map in the electronic device.

In the case that the point cloud map does not include the image features of the sampling points, that is, under the premise that the camera coordinates of multiple feature points and the world coordinates of multiple sampling points are known, an iterative strategy is used to try to find the camera coordinate system relative to the world. The target rotation relationship and target translation relationship of the coordinate system can realize the positioning of the image acquisition device and obtain high-precision positioning results.

For the search of the target rotation relationship and the target translation relationship, for example, through steps S603 to S608 in the following embodiment, iteratively find the sampling points that are closest (ie, the best match) to multiple feature points, so as to obtain the target rotation relationship and the target translation relationship.

Step S504: Determine the orientation of the image acquisition device according to the target rotation relationship.

Step S505: Determine the world coordinates of the image acquisition device according to the target translation relationship and the target rotation relationship.

In the positioning method provided by the embodiment of the present application, there is no need to extract the image features of the feature points, and it is not necessary to match the image features of the feature points with the image features of multiple sampling points in the point cloud map. The camera coordinates of multiple feature points are matched with the world coordinates of multiple sampling points to realize the positioning of the image acquisition device. In this way, there is no need to store the image features of multiple sampling points in the point cloud map, thereby greatly saving the storage space of the point cloud map.

The embodiment of the present application further provides a positioning method, and the method may include the following steps S601 to S610:

Step S601: Determine feature points in the image to be processed collected by the image collection device;

Step S602, acquiring the camera coordinates of the feature point;

Step S603: Select an initial target sampling point that matches the characteristic point from a plurality of sampling points in the pre-built point cloud map.

In some embodiments, when the electronic device implements step S603, it can first set the initial rotation relationship and initial translation relationship of the camera coordinate system relative to the world coordinate system; then, according to the camera coordinates, initial rotation relationship and initial rotation relationship of the feature point , The feature point is matched with multiple sampling points, and the initial target sampling point that matches the feature point is selected from the multiple sampling points. In some embodiments, the initial target sampling point can be selected through steps S703 to S705 in the following embodiments.

In fact, step S603 is used to select sampling points that may match the feature point. The initial target sampling point may not be a point that truly matches the feature point; therefore, the following steps S604 to S608 are required to further determine the initial target sampling point Whether it is a point that really matches the feature point.

Step S604, according to the camera coordinates of the multiple feature points and the world coordinates of the initial target sampling points matching the multiple feature points, determine the second rotation relationship and the second rotation relationship of the camera coordinate system relative to the world coordinate system. Translation relationship.

In some embodiments, when the electronic device implements the steps, it can construct an error function based on the camera coordinates of multiple feature points and the world coordinates of the initial target sampling points that match the multiple feature points; then, the current The optimal second rotation relationship and second translation relationship. For example, the camera coordinate comprising n feature point set is expressed as _{P = {p 1, p 2} , ..., p i, ..., p n}, the camera coordinates of the feature point is denoted by p _i, and The set of world coordinates of the initial target sampling points matched by n feature points is expressed as Q={q ₁ ,q ₂ ,...,q _i ,...,q _n }, and the world coordinates of the initial target sampling points are expressed as q _i to represent, then the following formula (7) can be listed:

In the formula, E(R, T) is the error function, and R and T are the second rotation relationship and the second translation relationship to be solved, respectively. Then, the optimal solution of R and T in equation (7) can be solved by the least square method.

Step S605: Determine the first world coordinate of the feature point according to the second rotation relationship, the second translation relationship and the camera coordinates of the feature point.

After obtaining the optimal solution, that is, the second rotation relationship and the second translation relationship, the camera coordinates of the feature point can be converted into the first world coordinates of the feature point. If the selected initial target sampling point and feature point represent the same location point in the actual physical space, or two similar location points, then the first world coordinates determined in step S605 should be the same as the initial target sampling point The world coordinates are the same or similar. Conversely, if the two represent not the same location point, nor two similar location points, then the first world coordinate determined in step S605 is different from the world coordinate of the initial target sampling point and is not similar. Based on this, the matching error of multiple feature points can be determined by the following step S606, and based on the matching error and the first threshold, it is determined whether the initial target sampling point is a point that actually matches the feature point, and then the target conversion relationship and the target translation relationship are determined. .

Step S606: Determine the matching error of the multiple feature points according to the first world coordinates of the multiple feature points and the world coordinates of the initial target sampling points matching the multiple feature points.

In some embodiments, when the electronic device implements step S606, it can determine the matching errors of multiple feature points through step S708 and step S709 in the following embodiments. That is, according to the first world coordinates of the feature point and the world coordinates of the initial target sampling point, the distance between the feature point and the initial target sampling point is determined; then, the distance between the multiple feature points and the matching initial target sampling point is determined. The distance determines the matching error.

Step S607, if the matching error is greater than the first threshold, return to step S603, reselect the initial target sampling point, and re-determine the matching error until the re-determined matching error is less than the first threshold.

Understandably, if the matching error is greater than the first threshold, it means that the currently selected initial target sampling point is not the sampling point that matches the feature point, and the two refer to not the same location point or similar location points in the physical space. At this time, it is necessary to return to step S603, reselect the initial target sampling point, and re-execute steps S604 to S606 based on the reselected initial target sampling point to re-determine the matching error until the re-determined matching error is less than the first threshold , It is determined that the initial target sampling point selected in the current iteration is a point that truly matches the feature point. At this time, the second rotation relationship and the second translation relationship obtained in the current iteration can be determined as the target rotation relationship and the target translation relationship, respectively.

Conversely, in some embodiments, if the matching error is less than or equal to the first threshold, the orientation (ie attitude) of the image acquisition device is determined according to the second rotation relationship obtained in the current iteration, and the second translation relationship obtained in the current iteration is determined. , Determine the coordinates of the image acquisition device in the point cloud map (ie world coordinates).

Step S608, determining a second rotation relationship determined when the matching error is less than or equal to the first threshold value as the target rotation relationship; determining a second rotation relationship determined when the matching error is less than or equal to the first threshold value The translation relationship is determined as the target translation relationship.

Step S609: Determine the orientation of the image acquisition device according to the target rotation relationship.

Step S610: Determine the world coordinates of the image acquisition device according to the target translation relationship and the target rotation relationship.

The embodiment of the present application further provides a positioning method, and the method may include the following steps S701 to S713:

Step S701, determining feature points in the image to be processed collected by the image collection device;

Step S702, acquiring the camera coordinates of the feature point;

Step S703: Acquire the third rotation relationship and the third translation relationship of the camera coordinate system relative to the world coordinate system. In some embodiments, the third rotation relationship and the third translation relationship may be set to an initial value respectively.

Step S704, determining the second world coordinates of the feature point according to the third rotation relationship, the third translation relationship, and the camera coordinates of the feature point;

Step S705: Match the second world coordinates of the feature point with the world coordinates of the multiple sampling points to obtain an initial target sampling point.

In some embodiments, when the electronic device implements step S705, it can determine the distance between the second world coordinates of the feature point and the world coordinates of the sampling point, and then determine the sampling point closest to the feature point as the initial target sampling point. Or, the sampling point whose distance is less than or equal to the distance threshold is determined as the initial target sampling point. In some embodiments, the Euclidean distance between the second world coordinates of the feature point and the world coordinates of the sampling point can be determined, and the Euclidean distance is taken as the distance between the feature point and the sampling point.

Step S706, according to the camera coordinates of the multiple feature points and the world coordinates of the initial target sampling points matching the multiple feature points, determine the second rotation relationship and the second rotation relationship of the camera coordinate system relative to the world coordinate system. Translation relationship

Step S707: Determine the first world coordinate of the feature point according to the second rotation relationship, the second translation relationship and the camera coordinates of the feature point;

Step S708: Determine the distance between the feature point and the initial target sampling point according to the first world coordinates of the feature point and the world coordinates of the initial target sampling point.

In some embodiments, the electronic device may also determine the Euclidean distance between the first world coordinates and the world coordinates of the initial target sampling point, and use the Euclidean distance as the distance between the feature point and the initial target sampling point.

Step S709: Determine the matching error according to the distance between the multiple feature points and the matched initial target sampling point.

In some embodiments, when the electronic device implements step S709, the average distance between the multiple feature points and the matched initial target sampling point may be determined as the matching error. For example, the set of first world coordinates including n feature points is expressed as P′={p′ ₁ ,p′ ₂ ,...,p′ _i ,...,p′ _n }, the first of the feature points The world coordinates are represented by p′ _i , and the set of world coordinates of the initial target sampling points that match the n feature points is represented as Q={q ₁ ,q ₂ ,...,q _i ,...,q _n }, The world coordinate of the initial target sampling point is _{represented by q i} , then the matching error d can be obtained by the following formula (8):

Where ||p′ _i -q _i || ² represents the Euclidean distance between the feature point and the matched initial target sampling point.

Step S710, if the matching error is greater than the first threshold, use the second translation relationship as the third translation relationship, and use the second rotation relationship as the third rotation relationship, return to step S704, and reselect the initial Target sampling points, and re-determine the matching error until the re-determined matching error is less than the first threshold.

Understandably, if the matching error is greater than the first threshold, it means that the acquired third rotation relationship and third translation relationship do not conform to reality. In other words, the obtained initial target sampling point is not a point that really matches the feature point. In this case, the second translation relationship can be used as the third translation relationship, and the second rotation relationship can be used as the third rotation relationship. Step S704 to step S709, until the matching error is less than the first threshold.

Step S711, determining a second rotation relationship determined when the matching error is less than or equal to the first threshold value as the target rotation relationship; determining a second rotation relationship determined when the matching error is less than or equal to the first threshold value The translation relationship is determined as the target translation relationship;

Step S712: Determine the orientation of the image acquisition device according to the target rotation relationship;

Step S713: Determine the world coordinates of the image acquisition device according to the target translation relationship and the target rotation relationship.

In some embodiments, the point cloud map construction process, as shown in FIG. 4, may include the following steps S801 to S805:

Step S801: Obtain multiple sample images.

In some embodiments, when the electronic device implements step S801, the image acquisition device may be used to collect sample images at a preset frame rate. The sample image can be a two-dimensional sample image, for example, a monocular camera is used to collect red, green, and blue (Red, Green, Blue, RGB) images at a fixed frame rate; or it can be from a pre-collected sample image library Obtain the multiple sample images in.

In step S802, the multiple sample images are processed to obtain a first sampling point set, and the first sampling point set includes at least the world coordinates of the sampling points in the multiple sample images.

In the initial stage of point cloud map construction, the image features and camera coordinates of the sampling points can be obtained, but the world coordinates of the sampling points in the sample image are unknown. In some embodiments, multiple sample images may be processed by a three-dimensional reconstruction method, so as to obtain the world coordinates of the sampling points. For example, the structure from motion (SFM) method is used to initialize multiple sample images to obtain the world coordinates of multiple sample points. In some embodiments, the first sampling point set includes the world coordinates of a plurality of sampling points, and does not include the image characteristics of the sampling points. In other embodiments, the first sampling point set includes not only the world coordinates of multiple sampling points, but also the image characteristics of the sampling points.

In some embodiments, when the electronic device implements step S802, it can determine the first sampling point set through steps S902 to 906 in the following embodiments.

Step S803: Obtain other sample images except for the multiple sample images.

Similarly, the electronic device can use the image acquisition device to acquire sample images in real time at a preset frame rate, and execute the following steps S804 and S805; or it can also acquire the other sample images from a pre-established sample image library.

Step S804: Determine the world coordinates of the sampling points in the other sample images according to the first sampling point set and the acquired attribute information of the sampling points in the other sample images to obtain a second sampling point set.

In fact, the time complexity of determining the world coordinates of multiple sampling points through step S802 is relatively high. Therefore, in the initial stage of map construction, after obtaining the world coordinates of multiple sampling points, step S804 is used to determine the world coordinates of the sampling points in the other sample images. In this way, the time cost of map construction can be greatly reduced.

In some embodiments, when the electronic device implements step S804, in the case where the image feature of the sampling point and the world coordinate of the sampling point are included in the first sampling point set, the steps S201 to S201 to that provided in the above-mentioned embodiment can be adopted. In step S204, step S301 to step S305, or step S401 to step S407, the world coordinates of sampling points in other sample images are determined, and the obtained second sampling point set includes the world coordinates and image features of the sampling points in other sample images.

In the case that the world coordinates of the sampling points are included in the first sampling point set, but the image features of the sampling points are not included, the electronic device can perform steps S501 to S505, and steps S601 to S610 similar to those provided in the foregoing embodiment. Or step S701 to step S713, determine the world coordinates of the sampling points in the other sample images, and the obtained second sampling point set includes the world coordinates of the sampling points in the other sample images, but does not include image features.

Step S805: Construct a point cloud map according to the first sampling point set and the second sampling point set.

In some embodiments, when the electronic device implements step S805, the first sampling point set and the second sampling point set may be combined to obtain a point cloud map. In other words, the point cloud map is actually a data set, and the distance between sampling points in the data set is greater than the first threshold.

In the embodiment of the present application, in the initial stage of map construction, after the electronic device obtains a first sampling point set including at least the world coordinates of a plurality of sampling points through multiple sample images, according to the first sampling point set and other acquired samples The attribute information of the sampling points in the image is determined, the world coordinates of the sampling points in other sample images are determined, and the second sampling point set is obtained; in this way, the world coordinates of the sampling points in other sample images can be quickly obtained, thereby reducing the time cost of map construction .

The embodiment of the present application further provides a method for constructing a point cloud map, and the method may include the following steps S901 to S909:

Step S901, acquiring multiple sample images;

Step S902: Obtain the image features and camera coordinates of the sampling points in the sample image.

In some embodiments, when the electronic device implements step S902, the image acquisition device may be used to collect sample images at a preset frame rate, and process the collected sample images in real time, and extract the image characteristics and the image characteristics of the sampling points in the sample images. Camera coordinates.

Step S903: According to the image characteristics of the sampling points in the multiple sample images, select the first target image and the second target image that meet the second condition from the multiple sample images.

In some embodiments, when the electronic device implements step S903, the selected first target image and second target image are generally two sample images with relatively large parallax; in this way, the determination of the first target image or the second target image can be improved. The accuracy of the world coordinates of the sampling points in the image is conducive to the subsequent acquisition of higher positioning accuracy. For example, the electronic device determines the first target image and the second target image through steps S113 to S116 in the following embodiment.

Step S904: Determine a fourth rotation relationship and a fourth translation relationship between the first target image and the second target image.

In some embodiments, when the electronic device implements step S904, the four-point method in the Random Sample Consensus (RANSAC) algorithm may be used to process the first target image and the second target image, and calculate the homography matrix. Thus, the fourth rotational relationship and the fourth translation relationship are obtained.

Step S905: Determine the world coordinates of the sampling points in the first target image according to the fourth rotation relationship, the fourth translation relationship, and the camera coordinates of the sampling points in the first target image. In some embodiments, when the electronic device implements step S905, the world coordinates of the sampling points in the first target image may be obtained by triangulation calculation.

Step S906: Determine a first set of sampling points according to the world coordinates of the sampling points in each of the first target sample images.

It is understandable that the sampling point in the first target sample image and the sampling point of the matched second target sample image are actually at the same location; therefore, in some embodiments, the electronic device may be based on each first target sample image or For the world coordinates of the sampling points in the second target sample image, the first sampling point set may be determined. In some embodiments, the first set of sampling points includes the world coordinates of the sampling points, but does not include the image features of the sampling points. In other embodiments, the first set of sampling points includes the world coordinates of the sampling points and the image features of the sampling points.

Step S907, acquiring other sample images except for the multiple sample images;

Step S908: Determine the world coordinates of the sampling points in the other sample images according to the first sampling point set and the acquired attribute information of the sampling points in the other sample images to obtain a second sampling point set;

Step S909: Construct a point cloud map according to the first sampling point set and the second sampling point set.

The embodiment of the present application further provides a method for constructing a point cloud map, and the method may include the following steps S111 to S122:

Step S111, acquiring multiple sample images;

Step S112, acquiring image features and camera coordinates of sampling points in the sample image;

Step S113: Perform pairwise matching on the multiple sample images according to the image characteristics of the sampling points in the multiple sample images to obtain a first matching pair set of each pair of sample images.

The so-called pairwise matching refers to: each sample image is matched with other sample images. For example, if multiple sample images include sample images 1 to 6, the electronic device can match sample image 1 with sample image 2, match sample image 1 with sample image 3, match sample image 1 with sample image 4, and match sample image 1. Match with sample image 5, match sample image 1 with sample image 6, match sample image 2 with sample image 3, match sample image 2 with sample image 4, match sample image 2 with sample image 5, match sample image 2 with sample image 2 Match with sample image 6,..., match sample image 5 with sample image 6. The obtained first matching pair set includes the matching relationship between the sampling points in the two images, that is, it includes multiple sampling point matching pairs.

In step S114, the matching pairs of sampling points that do not meet the third condition in the first matching pair set are eliminated to obtain a second matching pair set.

When the electronic device implements step S114, the elimination method can use the eight-point method in RANSAC to calculate the basic matrix, and the matching pairs that do not meet the basic matrix are selected to be eliminated; in this way, some matching pairs of sampling points with poor robustness can be eliminated. Thereby improving the robustness of the algorithm.

Step S115, selecting a target matching pair set whose number of matching pairs meets the second condition from each of the second matching pair sets.

Generally speaking, when the number of matching pairs is too large, the parallax of the two images is relatively small; however, when the number of matching pairs is small, the fourth rotation relationship and the fourth translation relationship between the two images cannot be determined. In some embodiments, the second condition may be set as the number of matching pairs is greater than the first value and less than the second value.

Step S116: Determine the two sample images corresponding to the target matching pair set as the first target image and the second target image;

Step S117, determining a fourth rotation relationship and a fourth translation relationship between the first target image and the second target image;

Step S118, determining the world coordinates of the sampling points in the first target image according to the fourth rotation relationship, the fourth translation relationship, and the camera coordinates of the sampling points in the first target image;

Step S119: Determine the first set of sampling points according to the world coordinates of the sampling points in each of the first target sample images;

Step S120, acquiring other sample images except for the multiple sample images;

Step S121, determining the world coordinates of the sampling points in the other sample images according to the first sampling point set and the acquired attribute information of the sampling points in the other sample images to obtain a second sampling point set;

Step S122: Construct a point cloud map according to the first sampling point set and the second sampling point set.

An exemplary application of the embodiments of the present application in an actual application scenario will be described below.

In the embodiments of the present application, an indoor positioning technology based on sparse point clouds is implemented, which can help users locate their positions in real time. This solution can extract image features and construct a sparse point cloud map (that is, an example of the point cloud map) for indoor scenes. The positioning process does not depend on external base station equipment, with low cost, high positioning accuracy and strong robustness. The program consists of two main parts: building a map and visual positioning.

In the embodiment of the present application, the construction of the map part is mainly to collect RGB image information (ie, the sample image) through a monocular camera, and extract image features to construct a sparse point cloud map, which includes at least the following steps S11 to S15:

Step S11, using a monocular camera to collect RGB images at a fixed frame rate;

Step S12, real-time extraction of attribute information in the RGB image (such as image features and camera coordinates of sampling points in the image) in the acquisition process;

Step S13, after collecting a certain number of RGB images, initialize the relative rotation and translation of the images using the SFM method;

Step S14: After the initialization is completed, calculate the three-dimensional world coordinates (ie some embodiments of the world coordinates) of the sparse points (ie sampling points) of the subsequent image through the PnP (Perspective-n-Point) algorithm to obtain a sparse point cloud map;

In step S15, the sparse point cloud map and its corresponding image features are stored, for example, the information is serialized and stored locally as an offline map.

Among them, for the extraction of image features in the RGB image in step S12, the following explanation is given here. The process of feature extraction is actually the process of interpreting and labeling RGB images. In some embodiments, FAST corner points are extracted from RGB images, and the number of corner points extracted is generally fixed at 150 for image tracking; and ORB descriptors are extracted for the corner points, which are used as feature descriptors for sparse points. match. It should be noted that 150 is an empirical value, and the number of corner points to be extracted is not limited in this application. Too few corner points will lead to a high tracking failure rate, while too many corner points will affect the efficiency of the algorithm.

Among them, for the initialization of the relative rotation and translation of the image using the SFM method in step S13, the following explanation is given here. First, after a certain number of images are collected, the relative rotation and translation of the images are initialized by the SFM method, and the three-dimensional world coordinates of the sparse points are obtained. The SFM algorithm includes at least the following steps S131 to S139:

Step S131: Perform pairwise matching on a certain number of images, and establish a matching relationship between the sparse points of the image using the method of Euclidean distance judgment;

Step S132, the matching pairs are eliminated, the elimination method adopts the RANSAC eight-point method to calculate the basic matrix, and the matching pairs that do not satisfy the basic matrix are selected to be eliminated;

Step S133: After the matching relationship is established, a tracking list is generated, and the tracking list refers to a collection of image names of points with the same name;

Step S134, removing invalid matches in the tracking list;

Step S135, searching for the initial image pair, the purpose is to find the image pair with the largest camera baseline, and use the RANSAC algorithm four-point method to calculate the homography matrix. The matching points that satisfy the homography matrix are called interior points, and those that are not satisfied are called exterior points. Find the image pair with the smallest proportion of interior points;

Step S136, searching for the relative rotation and translation of the initialized image pair, the method is to calculate the essential matrix by the RANSAC eight-point method, and obtain the relative rotation and translation between the image pairs by SVD decomposition of the essential matrix;

Step S137: Obtain the three-dimensional world coordinates of the sparse points in the initialization image pair through triangulation calculation;

Step S138: Repeating steps S136 and S137 on other images, the relative rotation and translation of all images, and the three-dimensional world coordinates of sparse points can be obtained;

In step S139, the three-dimensional world coordinates of the rotation, translation, and sparse point between the obtained images are optimized through the beam adjustment method. This is a non-linear optimization process, the purpose is to reduce the error of the SFM results.

Based on steps S11 to S15, an offline map based on the sparse point cloud can be constructed. The map stores the sparse point cloud and its image attribute information (including three-dimensional world coordinates and descriptor information) in a binary format to the local, during the visual positioning process , The map will be loaded and used.

In the embodiment of this application, the visual positioning part mainly collects the current RGB image through the monocular camera, loads the constructed offline map, and uses the descriptor matching to find the matching pair between the current feature point and the sparse point of the map. Finally, Then the PnP algorithm is used to solve the current camera's precise pose in the map to achieve the purpose of positioning. In some embodiments, k may include the following steps S21 to S25:

Step S21, loading a pre-built offline map (ie a sparse point cloud map);

Step S22, using a monocular camera to collect RGB images;

Step S23, extract the attribute information in the current frame image in real time during the acquisition process;

Step S24: Find a matching pair between the current feature point and the sparse point on the map through descriptor matching;

In step S25, after finding enough matching pairs, the accurate pose of the current camera in the map coordinate system is solved through the PnP algorithm.

For the real-time extraction of the attribute information in the current frame image in step S23, reference may be made to the above step S12.

Among them, for finding the matching pair between the current feature point and the sparse point of the map through the descriptor matching in step S24, the algorithm includes at least the following steps S241 to S244:

Step S241, for the Nth (initially 0) feature point F _1N extracted in the current image, set the minimum Euclidean distance d _min =d _TH , and set the matching point

Step S242: Calculate F _1N and the M-th (initially 0) feature point F _2M in the sparse point cloud, and calculate the Euclidean distance d _NM between the feature point descriptors;

Step S243, judge the Euclidean distance d _NM and the minimum Euclidean distance d _min , if d _NM <d _min , then d _min = d _NM ,

Then M=M+1, if the sparse point in the sparse point cloud has not been traversed, skip back to step S242; otherwise, N=N+1, skip back to step S241; if the feature points of the current image are traversed, skip Go to step S244;

In step S244, the matching pairs between the feature points of the current image and the sparse points of the map are sorted out as the algorithm output, and the algorithm ends.

Among them, for solving the precise pose of the current camera in the map coordinate system through the PnP algorithm in step S25, in some embodiments, as shown in FIG. 5:

First, it is judged that a matching pair sequence is formed in step S24 (in this example, the matching pair sequence is {F ₀ , F ₁ , F ₂ }). If the number of elements in the matching pair sequence is greater than TH ₂ , step S25 is performed; otherwise, the algorithm ends . In this embodiment, based on the matched pair sequence, the SolvePnP function in OpenCV is called to find the pose of the current camera in the map coordinate system. The principle of the PnP algorithm is as follows:

The input of the PnP algorithm is three-dimensional (3D) points (that is, the three-dimensional world coordinates of the sparse points in the map coordinate system) and the 2D points obtained by the projection of these 3D points in the current image (that is, the camera of the feature point in the current frame) Coordinate), the output of the algorithm is the pose transformation of the current frame relative to the origin of the map coordinate system (that is, the pose of the current frame in the map coordinate system).

The PnP algorithm does not directly obtain the camera pose matrix according to the matching pair sequence, but first obtains the 3D coordinates of the corresponding 2D point in the current coordinate system; then, according to the 3D coordinates in the map coordinate system and the current coordinate system 3D coordinates to solve the camera pose.

Based on steps S21 to S25, visual features can be used to achieve the positioning purpose in a predefined sparse point cloud map, and to obtain its own position and posture in the map coordinate system (that is, the world coordinate system). The positioning result has high accuracy, does not need to rely on external base station equipment, has low cost and strong robustness.

In the embodiment of the present application, the camera movement is used to obtain the three-dimensional information of the feature point, and the position and posture can be provided in the positioning result at the same time, which improves the positioning accuracy compared with other indoor positioning methods;

In the embodiment of the present application, the stored map is in the form of a sparse point cloud, which is equivalent to sparse sampling of the image, and the map size is compressed to a certain degree compared with the traditional method;

In the embodiment of the present application, only ordinary mobile terminal equipment is needed in the process of mapping and positioning, and other external base station equipment is not required to be introduced, so the cost is low;

In the embodiments of the present application, there is no need to introduce algorithms with high error rates such as object recognition, and the positioning success rate is high, and the robustness is strong.

In the embodiment of the present application, the three-dimensional information of the image feature is fully excavated and combined with the high-precision and high-robustness image matching algorithm for indoor environment positioning. In map construction, the three-dimensional world coordinates and descriptor information of the feature points in the visual image are collected and stored as an offline map in the form of a sparse point cloud. In the positioning method, the descriptor matching method is used to find the matching pair of the current feature point in the sparse point cloud, and then the current position and posture of the current self are accurately calculated through the PnP algorithm. The combination of the two forms a low-cost, high-precision, and robust indoor positioning method.

In map construction, the sparse point cloud stores three-dimensional world coordinates and descriptor information including image feature points. The descriptor information is used for matching with the feature points in the current image during the visual positioning process. Among them, the image feature descriptor can be an ORB descriptor, and the descriptor information of each image feature point occupies 256 bytes of space. For offline maps stored in the form of sparse point clouds, each sparse point must be allocated 256 bytes as the storage space of the feature descriptor, which occupies a large proportion of the size of the final offline map. In order to reduce the size of the offline map, the embodiments of the present application provide the following extension solutions.

In the map building part, the electronic device can serialize and store the three-dimensional world coordinates of the sparse point cloud.

In the visual positioning part, the embodiment of the present application provides a positioning method, which may include the following steps S31 to S35:

Step S31, loading a pre-built offline map;

Step S32, using a monocular camera to collect RGB images;

Step S33, real-time extraction of attribute information (ie, camera coordinates of feature points) in the current frame image during the acquisition process;

Step S34: Calculate the three-dimensional camera coordinates of the feature points in the current image to form a local point cloud;

In step S35, the local point cloud and the sparse point cloud of the map are matched through the Iterative Closest Point (ICP) algorithm to solve the accurate pose of the current camera in the map coordinate system.

Among them, for the matching of the local point cloud and the map sparse point cloud by the ICP algorithm in step S35, the following explanation is given here.

The ICP algorithm is essentially an optimal registration method based on the least squares method. The algorithm repeatedly selects the corresponding point pairs and calculates the optimal rigid body transformation until the convergence accuracy requirements of the correct registration are met. ICP algorithm is the basic principle: to be respectively matched target point cloud point cloud P and Q in the source, according to certain constraints, find the nearest point (p _i, q _i), and then calculate the optimal rotation R Translate T to minimize the error function. The formula of the error function E(R,T) is:

Where n is the number of adjacent point pairs, p _i is a point in the target point cloud P, q _i is the closest point in the source point cloud Q _{corresponding to p i} , R is the rotation matrix (also known as the rotation relationship), and T is Translation vector (also called translation relationship).

Based on step S31 to step S35, visual features can be used to achieve the positioning purpose in the predefined sparse point cloud map, and obtain its own position and posture in the map coordinate system. And the predetermined sparse point cloud map does not need to store additional feature point descriptor information, which reduces the size of the offline map.

Based on the foregoing embodiment, the embodiment of the present application provides a positioning device, which includes each module and each unit included in each module, which can be implemented by a processor in a terminal; of course, it can also be implemented by a specific logic circuit. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA).

FIG. 6A is a schematic structural diagram of a positioning device according to an embodiment of the application. As shown in FIG. 6A, the device 600 includes a first determining module 601, an attribute information acquiring module 602, and a positioning module 603. The first determining module 601 is configured to Determine the characteristic points in the image to be processed collected by the image acquisition device; the attribute information acquisition module 602 is configured to acquire the attribute information of the characteristic points; the positioning module 603 is configured to compare the attribute information of the characteristic points with the pre-built The attribute information of multiple sampling points in the point cloud map is matched to obtain the positioning result of the image acquisition device.

In some embodiments, the attribute information of the feature point includes at least one of the following: the image feature of the feature point, and the camera coordinates of the feature point; the attribute information of the sampling point includes at least one of the following: the The image feature of the sampling point, and the world coordinates of the sampling point.

In some embodiments, the positioning module 603 includes: a matching unit configured to match the image features of the feature point with the image features of the multiple sampling points to obtain the first target sampling point; the positioning unit is configured to To determine the positioning result of the image acquisition device according to the camera coordinates of the feature point and the world coordinates of the first target sampling point.

In some embodiments, the matching unit is configured to determine the similarity between the sampling point and the feature point according to the image feature of the sampling point and the image feature of the feature point; The sampling point whose similarity between the feature points meets the first condition is determined as the first target sampling point.

In some embodiments, the positioning unit is configured to determine the plurality of first target sampling points according to the world coordinates of the plurality of first target sampling points and the camera coordinates of the feature points matching the plurality of first target sampling points. Camera coordinates of a target sampling point; according to the world coordinates of the plurality of first target sampling points and the camera coordinates of the plurality of first target sampling points, determine the first rotation relationship of the camera coordinate system relative to the world coordinate system and A first translation relationship; determine the world coordinates of the image capture device according to the first translation relationship and the first rotation relationship; determine the orientation of the image capture device according to the first rotation relationship.

In some embodiments, the matching unit is configured to match the camera coordinates of the multiple feature points with the world coordinates of the multiple sampling points according to an iterative strategy to obtain the camera coordinate system relative to the world coordinate system. The target rotation relationship and the target translation relationship; the positioning unit is further configured to: determine the orientation of the image acquisition device according to the target rotation relationship; determine the image according to the target translation relationship and the target rotation relationship Collect the world coordinates of the device.

In some embodiments, the matching unit includes: a selection subunit configured to select an initial target sampling point that matches the characteristic point from the plurality of sampling points; a transformation relationship determination subunit configured to select an initial target sampling point that matches the characteristic point from the plurality of sampling points; The camera coordinates of the multiple feature points and the world coordinates of the initial target sampling points that match the multiple feature points, and determine the second rotation relationship and the second translation relationship of the camera coordinate system relative to the world coordinate system; A world coordinate determination subunit, configured to determine the first world coordinate of the feature point according to the second rotation relationship, the second translation relationship, and the camera coordinates of the feature point; a matching error determination subunit, configured To determine the matching error of the multiple feature points according to the first world coordinates of the multiple feature points and the world coordinates of the initial target sampling points that match the multiple feature points; the iterative subunit is configured to: If the matching error is greater than the first threshold, the initial target sampling point is reselected, and the matching error is re-determined until the re-determined matching error is less than the first threshold; the target transformation relationship determination subunit is configured to set the matching error to be less than Or the second rotation relationship determined when the first threshold is equal to or less is determined as the target rotation relationship; the second translation relationship determined when the matching error is less than or equal to the first threshold is determined as the target translation relationship.

In some embodiments, the selection subunit is configured to: obtain a third rotation relationship and a third translation relationship of the camera coordinate system relative to the world coordinate system; The three translation relations and the camera coordinates of the feature point are used to determine the second world coordinates of the feature point; the second world coordinates of the feature point are matched with the world coordinates of the multiple sampling points to obtain the The initial target sampling point.

In some embodiments, the matching error determining subunit is configured to determine the characteristic point and the initial target sampling point according to the first world coordinates of the characteristic point and the world coordinates of the initial target sampling point According to the distance between the multiple feature points and the matching initial target sampling point, the matching error is determined.

The iterative subunit is configured to, if the matching error is greater than a first threshold, use the second translation relationship as the third translation relationship, and the second rotation relationship as the third rotation relationship, and reselect The initial target sampling point.

In some embodiments, as shown in FIG. 6B, the device 600 further includes: an image acquisition module 604, configured to acquire multiple sample images; and an image processing module 605, configured to process the multiple sample images, Obtain a first sampling point set, the first sampling point set includes at least the world coordinates of the sampling points in the multiple sample images; the image acquisition module 604 is further configured to acquire other than the multiple sample images Sample image; the second determination module 606 is configured to determine the world coordinates of the sampling points in the other sample images according to the first sampling point set and the acquired attribute information of the sampling points in the other sample images, to obtain the first Two sampling point sets; the map construction module 607 is configured to construct the point cloud map according to the first sampling point set and the second sampling point set.

In some embodiments, the image processing module 605 includes: an attribute information acquiring unit configured to acquire image features and camera coordinates of sampling points in the sample image; a target image determining unit configured to The image characteristics of the sampling points, select the first target image and the second target image that meet the second condition from the multiple sample images; the transformation relationship determination unit is configured to determine the first target image and the second target image A fourth rotation relationship and a fourth translation relationship between the target images; a world coordinate determination unit configured to be based on the fourth rotation relationship, the fourth translation relationship, and the camera coordinates of the sampling point in the first target image , Determining the world coordinates of the sampling points in the first target image; a set determining unit configured to determine the first sampling point set according to the world coordinates of the sampling points in each of the first target sample images.

In some embodiments, the target image determining unit is configured to: perform pairwise matching on the multiple sample images according to the image characteristics of the sampling points in the multiple sample images to obtain the first pair of sample images. A set of matching pairs; the matching pairs of sampling points that do not meet the third condition in the first matching pair set are eliminated to obtain a second matching pair set; the number of matching pairs is selected from each second matching pair set that meets all The target matching pair set of the second condition; the two sample images corresponding to the target matching pair set are determined as the first target image and the second target image.

The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

It should be noted that, in the embodiments of the present application, if the above positioning method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application can be embodied in the form of software products in essence or the parts that contribute to related technologies. The computer software products are stored in a storage medium and include several instructions to enable An electronic device (which may be a mobile phone, a tablet computer, a notebook computer, a desktop computer, a server, a robot, a drone, etc.) executes all or part of the method described in each embodiment of the present application. The aforementioned storage media include: U disk, mobile hard disk, read only memory (Read ON1ly Memory, ROM), magnetic disk or optical disk and other media that can store program codes. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

Correspondingly, an embodiment of the present application provides an electronic device. FIG. 7 is a schematic diagram of a hardware entity of the electronic device according to an embodiment of the application. As shown in FIG. 7, the hardware entity of the electronic device 700 includes: a memory 701 and a processor. 702. The memory 701 stores a computer program that can run on the processor 702, and the processor 702 implements the steps in the positioning method provided in the foregoing embodiment when the processor 702 executes the program.

The memory 701 is configured to store instructions and applications executable by the processor 702, and can also cache data to be processed or processed by the processor 702 and each module in the electronic device 700 (for example, image data, audio data, voice communication data, and Video communication data) can be implemented by flash memory (FLASH) or random access memory (RaN1dom Access Memory, RAM).

Correspondingly, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps in the positioning method provided in the foregoing embodiment are implemented.

It should be pointed out here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

It should be understood that “one embodiment” or “some embodiments” mentioned throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of the present application. Therefore, appearances of "in one embodiment" or "in some embodiments" in various places throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not correspond to the embodiments of the present application. The implementation process constitutes any limitation. The serial numbers of the foregoing embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

The above description of the various embodiments tends to emphasize the differences between the various embodiments, and the same or similarities can be referred to each other. For the sake of brevity, the details are not repeated herein.

The term "and/or" in this article is only an association relationship that describes associated objects, which means that there can be three relationships, for example, object A and/or object B, which can mean: object A exists alone, and object A and object exist at the same time. B, there are three cases of object B alone.

It should be noted that in this article, the terms "include", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, It also includes other elements not explicitly listed, or elements inherent to the process, method, article, or device. If there are no more restrictions, the element defined by the sentence "including a..." does not exclude the existence of other identical elements in the process, method, article, or device that includes the element.

In the several embodiments provided in this application, it should be understood that the disclosed device and method may be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there may be other divisions in actual implementation, such as: multiple units or components can be combined, or It can be integrated into another system, or some features can be ignored or not implemented. In addition, the coupling, or direct coupling, or communication connection between the components shown or discussed may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms. of.

The units described above as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, the functional units in the embodiments of the present application can be all integrated into one processing unit, or each unit can be individually used as a unit, or two or more units can be integrated into one unit; The unit can be implemented in the form of hardware, or in the form of hardware plus software functional units.

A person of ordinary skill in the art can understand that all or part of the steps in the above method embodiments can be implemented by a program instructing relevant hardware. The foregoing program can be stored in a computer readable storage medium. When the program is executed, the execution includes The steps of the foregoing method embodiment; and the foregoing storage medium includes: various media that can store program codes, such as a mobile storage device, a read-only memory (Read ON1ly Memory, ROM), a magnetic disk or an optical disk.

Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it may also be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product in essence or a part that contributes to related technologies. The computer software product is stored in a storage medium and includes a number of instructions to enable An electronic device (which may be a mobile phone, a tablet computer, a notebook computer, a desktop computer, a server, a robot, a drone, etc.) executes all or part of the method described in each embodiment of the present application. The aforementioned storage media include: removable storage devices, ROMs, magnetic disks or optical discs and other media that can store program codes.

The methods disclosed in the several method embodiments provided in this application can be combined arbitrarily without conflict to obtain new method embodiments.

The features disclosed in the several product embodiments provided in this application can be combined arbitrarily without conflict to obtain new product embodiments.

The features disclosed in the several method or device embodiments provided in this application can be combined arbitrarily without conflict to obtain a new method embodiment or device embodiment.

The above are only the implementation manners of this application, but the protection scope of this application is not limited to this. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in this application. Covered in the scope of protection of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims (20)

1. A positioning method, the method includes:

   Determine the feature points in the image to be processed collected by the image acquisition device;

   Acquiring attribute information of the characteristic point;

   Match the attribute information of the characteristic point with the attribute information of multiple sampling points in the pre-built point cloud map to obtain the positioning result of the image acquisition device; wherein, the world coordinates in the attribute information of the sampling point Is the world coordinate of the sampling point in the sample image.
2. The method according to claim 1, wherein the matching the attribute information of the characteristic point with the attribute information of a plurality of sampling points in a pre-built point cloud map to obtain a positioning result of the image acquisition device, include:

   Matching the image features of the feature points with the image features of the multiple sampling points to obtain a first target sampling point;

   The positioning result of the image acquisition device is determined according to the camera coordinates of the characteristic point and the world coordinates of the first target sampling point.
3. The method according to claim 2, wherein the matching the image features of the feature point with the image features of the multiple sampling points to obtain the first target sampling point comprises:

   Determining the similarity between the sampling point and the feature point according to the image feature of the sampling point and the image feature of the feature point;

   A sampling point whose similarity with the characteristic point among the plurality of sampling points meets a first condition is determined as the first target sampling point.
4. The method according to claim 3, wherein the determining the sampling point whose similarity between the plurality of sampling points and the characteristic point satisfies a first condition as the first target sampling point comprises :

   Determine the sampling point whose similarity between the plurality of sampling points and the characteristic point to which it belongs is less than or equal to the similarity threshold as the first target sampling point; or, compare the plurality of sampling points with the The sampling point with the smallest similarity between the feature points is determined as the first target sampling point.
5. The method according to claim 3 or 4, wherein the determining the similarity between the sampling point and the feature point according to the image feature of the sampling point and the image feature of the feature point comprises:

   The Euclidean distance between the image feature of the sampling point and the image feature of the feature point is determined, and the Euclidean distance is determined as the similarity.
6. The method according to claim 2, wherein the positioning result includes world coordinates and orientation; said determining the position of the image acquisition device according to the camera coordinates of the feature point and the world coordinates of the first target sampling point Positioning results, including:

   Determining the camera coordinates of the multiple first target sampling points according to the world coordinates of the multiple first target sampling points and the camera coordinates of the feature points matching the multiple first target sampling points;

   Determine the first rotation relationship and the first translation relationship of the camera coordinate system relative to the world coordinate system according to the world coordinates of the plurality of first target sampling points and the camera coordinates of the plurality of first target sampling points;

   Determine the world coordinates of the image capture device according to the first translation relationship and the first rotation relationship; determine the orientation of the image capture device according to the first rotation relationship.
7. The method according to claim 1, wherein the attribute information of the feature point includes camera coordinates; the attribute information of the sampling point includes world coordinates; and the positioning result includes world coordinates and orientation;

   The matching the attribute information of the characteristic point with the attribute information of multiple sampling points in a pre-built point cloud map to obtain the positioning result of the image acquisition device includes:

   According to the iterative strategy, the camera coordinates of the multiple feature points are matched with the world coordinates of the multiple sampling points to obtain the target rotation relationship and the target translation relationship of the camera coordinate system relative to the world coordinate system;

   Determine the orientation of the image capture device according to the target rotation relationship; determine the world coordinates of the image capture device according to the target translation relationship and the target rotation relationship.
8. The method according to claim 7, wherein the camera coordinates of a plurality of feature points are matched with the world coordinates of the plurality of sampling points according to an iterative strategy to obtain the target of the camera coordinate system relative to the world coordinate system Rotation relationship and target translation relationship, including:

   Selecting an initial target sampling point matching the characteristic point from the plurality of sampling points;

   According to the camera coordinates of the multiple feature points and the world coordinates of the initial target sampling points matching the multiple feature points, determine the second rotation relationship and the second translation of the camera coordinate system relative to the world coordinate system relationship;

   Determine the first world coordinates of the feature point according to the second rotation relationship, the second translation relationship, and the camera coordinates of the feature point;

   Determine the matching error of the multiple feature points according to the first world coordinates of the multiple feature points and the world coordinates of the initial target sampling points matching the multiple feature points;

   If the matching error is greater than the first threshold, reselect the initial target sampling point and re-determine the matching error until the re-determined matching error is less than the first threshold;

   Determining the second rotation relationship determined when the matching error is less than or equal to the first threshold value as the target rotation relationship; determining the second translation relationship when the matching error is less than or equal to the first threshold value, Determined as the target translation relationship.
9. 8. The method according to claim 8, wherein the selecting an initial target sampling point matching the characteristic point from the plurality of sampling points comprises:

   Acquiring a third rotation relationship and a third translation relationship of the camera coordinate system relative to the world coordinate system;

   Determine the second world coordinates of the feature point according to the third rotation relationship, the third translation relationship, and the camera coordinates of the feature point;

   Matching the second world coordinates of the characteristic point with the world coordinates of the multiple sampling points to obtain the initial target sampling point.
10. 8. The method according to claim 8, wherein the plurality of feature points are determined based on the first world coordinates of the plurality of feature points and the world coordinates of the initial target sampling points matching the plurality of feature points The matching error includes:

    Determine the distance between the feature point and the initial target sampling point according to the first world coordinates of the feature point and the world coordinates of the initial target sampling point;

    The matching error is determined according to the distance between the plurality of feature points and the matched initial target sampling point.
11. The method according to claim 9, wherein, if the matching error is greater than a first threshold, reselecting an initial target sampling point comprises:

    If the matching error is greater than the first threshold, the second translation relationship is used as the third translation relationship, and the second rotation relationship is used as the third rotation relationship, and the initial target sampling point is reselected.
12. The method according to any one of claims 1 to 11, wherein the construction process of the point cloud map comprises:

    Obtain multiple sample images;

    Processing the multiple sample images to obtain a first sampling point set, where the first sampling point set includes at least the world coordinates of the sampling points in the multiple sample images;

    Obtaining sample images other than the plurality of sample images;

    Determining the world coordinates of the sampling points in the other sample images according to the first sampling point set and the acquired attribute information of the sampling points in the other sample images to obtain a second sampling point set;

    The point cloud map is constructed according to the first sampling point set and the second sampling point set.
13. The method according to claim 12, wherein said processing said plurality of sample images to obtain a first set of sampling points comprises:

    Acquiring image features and camera coordinates of sampling points in the sample image;

    According to the image characteristics of the sampling points in the multiple sample images, select the first target image and the second target image that meet the second condition from the multiple sample images;

    Determining a fourth rotation relationship and a fourth translation relationship between the first target image and the second target image;

    Determine the world coordinates of the sampling points in the first target image according to the fourth rotation relationship, the fourth translation relationship, and the camera coordinates of the sampling points in the first target image;

    The first sampling point set is determined according to the world coordinates of the sampling points in each of the first target sample images.
14. The method according to claim 13, wherein the first target image and the second target that meet the second condition are selected from the plurality of sample images according to the image characteristics of the sampling points in the plurality of sample images Images, including:

    Performing pairwise matching on the multiple sample images according to the image characteristics of the sampling points in the multiple sample images to obtain a first matching pair set of each pair of sample images;

    Removing matching pairs of sampling points that do not meet the third condition in the first matching pair set to obtain a second matching pair set;

    Selecting, from each of the second matching pair sets, a target matching pair set whose number of matching pairs meets the second condition;

    The two sample images corresponding to the target matching pair set are determined as the first target image and the second target image.
15. A positioning device includes:

    The first determining module is configured to determine the characteristic points in the image to be processed collected by the image collecting device;

    The attribute information obtaining module is configured to obtain the attribute information of the characteristic point;

    The positioning module is configured to match the attribute information of the characteristic point with the attribute information of a plurality of sampling points in a pre-built point cloud map to obtain a positioning result of the image acquisition device.
16. The device according to claim 15, wherein the positioning module comprises:

    A matching unit configured to match the image features of the feature point with the image features of the multiple sampling points to obtain a first target sampling point;

    The positioning unit is configured to determine the positioning result of the image acquisition device according to the camera coordinates of the characteristic point and the world coordinates of the first target sampling point.
17. The device according to claim 16, wherein the matching unit is configured to:

    Determining the similarity between the sampling point and the feature point according to the image feature of the sampling point and the image feature of the feature point;

    A sampling point whose similarity with the feature point satisfies a first condition is determined as the first target sampling point.
18. The device according to claim 16, wherein the positioning result includes world coordinates and orientation; the matching unit is configured to: according to an iterative strategy, compare the camera coordinates of the multiple feature points with the world of the multiple sampling points. Coordinates are matched, and the target rotation relationship and target translation relationship of the camera coordinate system relative to the world coordinate system are obtained;

    The positioning unit is configured to: determine the orientation of the image capture device according to the target rotation relationship; determine the world coordinates of the image capture device according to the target translation relationship and the target rotation relationship.
19. An electronic device, comprising a memory and a processor, the memory storing a computer program that can run on the processor, and the processor implements the positioning method of any one of claims 1 to 14 when the processor executes the program step.
20. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the positioning method according to any one of claims 1 to 14.

Images