CN114565663A - Positioning method and device - Google Patents

Abstract

The present application provides a positioning method and apparatus. In this embodiment of the present application, by determining in the image database at least one second image that matches the first image captured by the first terminal, and the poses of the first image and the at least one second image in the local coordinate system of the first terminal, And according to the pose of the first image in the first local coordinate system, the pose of at least one second image in the first local coordinate system, and the pose of the at least one second image in the world coordinate system, output The pose of the first image in the world coordinate system realizes the positioning of the first image. Therefore, the embodiment of the present application can obtain the relationship between the local coordinate system and the world coordinate system based on the pose of a part of the image in the local coordinate system and the world coordinate system, so as to locate the first image in the world coordinate system, without the need for each image. 3D point clouds are collected from each image. The present application can be applied to the fields of virtual reality VR, augmented reality AR and the like.

Description

Method and device for positioning

technical field

The present application relates to the technical field of augmented reality (AR), and more particularly, to a method and apparatus for positioning.

Background technique

With the gradual maturity of the 5th generation (5G) communication technology and the development of mobile phone camera hardware and computing capabilities, intelligent application services based on visual AR technology are becoming more and more abundant. AR technology is a technology that ingeniously integrates virtual information with the real world. It widely uses various technical means such as multimedia, three-dimensional modeling, real-time tracking and registration, intelligent interaction, and sensing. Models, music, videos and other virtual information are simulated and applied to the real world, and the virtual information and real world information complement each other, thereby realizing the "enhancement" of the real world.

In order to realize a wide coverage of AR applications, it is necessary to build a map feature library for pose acquisition. For example, first, professional acquisition equipment, such as road survey vehicles, unmanned aerial vehicles, survey laser scanners, optical cameras, etc., can be used to acquire three-dimensional (3dimensions, 3D) point clouds (x, y, 3D) of the surrounding environment in the offline library construction stage z) Information, build a map feature library. Then, in the online positioning stage, a picture of the environment can be taken, the two-dimensional (2D) feature points of the captured picture can be extracted, and matched with the map feature library constructed in the offline library construction stage, a series of 2D-3D feature points can be retrieved. Match point pairs, and then obtain the pose of the terminal through an attitude calculation algorithm, such as PnP (point-n-point).

However, the above solution relies on the construction of a complete and highly unified map feature library containing point cloud information. However, the construction of a map feature library containing point cloud information requires professionals and collection equipment, which is time-consuming, labor-intensive, and expensive. Therefore, an efficient and low-cost positioning solution is urgently needed.

SUMMARY OF THE INVENTION

The present application provides a positioning method and device, which can obtain the relationship between the local coordinate system and the world coordinate system based on the pose of a part of the image in the local coordinate system and the pose in the world coordinate system, so that the image to be positioned is located in the world. Positioning in a coordinate system without acquiring a 3D point cloud for each image.

In a first aspect, a positioning method is provided, and the method can be applied to a terminal or a cloud. In this method, a first image captured by a first terminal may be acquired first, and according to the first information of the first image and the first information of the image in the image database, at least one image matching the first image is determined in the image database. a second image, wherein the first information is used to indicate the global feature of the image, the image database includes the first information of the multi-frame image, the second information of the multi-frame image and the pose of the multi-frame image in the world coordinate system, Wherein, the second information is used to indicate local features of the image. Then, the poses of the first image and the at least one second image in the first local coordinate system of the first terminal may be determined according to the second information of the first image and the second information of the at least one second image. After that, according to the pose of the first image under the first local coordinate system, the pose of the at least one second image under the first local coordinate system, and the pose of the at least one second image under the world coordinate system, output the first image The pose of an image in the world coordinate system.

Therefore, in this embodiment of the present application, by determining in the image database at least one second image that matches the first image captured by the first terminal, and the position of the first image and the at least one second image in the local coordinate system of the first terminal pose, and according to the pose of the first image in the first local coordinate system, the pose of at least one second image in the first local coordinate system, and the pose of the at least one second image in the world coordinate system , and output the pose of the first image in the world coordinate system, so as to realize the positioning of the first image. Therefore, the embodiment of the present application can obtain the relationship between the local coordinate system and the world coordinate system based on the pose of a part of the image in the local coordinate system and the pose in the world coordinate system, so that the image to be positioned (for example, the first image) is Positioning in the world coordinate system without acquiring a 3D point cloud for each image.

In this embodiment of the present application, the pose of the first image in the world coordinate system is the pose of the first terminal in the world coordinate system when the first image is captured, and may also be referred to as the pose of the camera.

In some embodiments, the pose of the at least one second image in the world coordinate system and the second information may be acquired from an image database, which is not limited in this application.

As an example, in this embodiment of the present application, the first terminal may be a mobile device such as a mobile phone and an autonomous vehicle, which is not limited in the present application.

Compared with the 3D point cloud feature library (which includes global 3D point cloud features) of the environment in the existing solution, and the calculation solution of realizing pose based on feature point matching, the embodiment of the present application does not use the 3D point cloud of the image. feature, but uses the pose of the image for localization. Since the construction of the 3D point cloud feature library of the environment needs to rely on professionals and acquisition equipment, which is time-consuming and labor-intensive, and the cost is high, and this application does not need to acquire 3D point cloud features, on the one hand, the application can use low-cost equipment, such as Cameras with low resolution and small field of view (such as mobile phone cameras) can build image databases without relying on professionals and professional equipment, and can reduce the data volume of image databases, thereby reducing the cost of building image databases; On the one hand, the present application can shorten the time for constructing the database, which contributes to the efficient construction of the image database. Therefore, the positioning solution based on the pose of the image and the image database of the present application can contribute to high-efficiency and low-cost positioning.

In some embodiments, the above-mentioned at least one second image may form an image set, and the image set may be referred to as a similar image set of the first image.

In some embodiments, the above-mentioned first image and at least one second image may form an image set, which may be referred to as a partial image set. The above-mentioned first local coordinate system, that is, the camera coordinate system of the first terminal, may be a relative coordinate system constructed with any one frame of image in the local image set as the origin of the first local coordinate system.

As a possible implementation manner, the first local coordinate system and The mapping relationship of the world coordinate system (or can also be called a transformation relationship). Then, according to the mapping relationship, the pose of the first image under the first local coordinate system can be transformed, the pose of the first image under the world coordinate system is determined, and the pose is output.

As an example, the mapping relationship between the world coordinate system and the first local coordinate system may be described by a rotation matrix or a translation vector, which is not limited in this application.

In this embodiment of the present application, the world coordinate system may also be referred to as an absolute coordinate system, a global coordinate system, or the like. As an example, the world coordinate system can be used as a reference coordinate system in the environment to describe the position of the camera (which can also be the terminal including the camera), and can also be used to describe the position of any object in the environment.

As a possible implementation manner, the image database may include multiple mapping relationships, wherein one mapping relationship may indicate the identification of an image, the first information of the image, the second information of the image, and the image in the world coordinate system The corresponding relationship between the lower poses is not limited in this application.

In conjunction with the first aspect, in some implementations of the first aspect, the method may further include establishing the above-mentioned image database.

With reference to the first aspect, in some implementations of the first aspect, a video shot by the second terminal may be acquired, and poses of multiple frames of images in the video in the second local coordinate system of the second terminal may be acquired. Then, the pose of the multi-frame images in the video under the world coordinate system may be determined according to the third information and the pose of the multi-frame images in the video under the second local coordinate system. Wherein, the third information is used to indicate the position of at least part of the images in the video in a map, and the map is associated with the world coordinate system. Afterwards, the first information and the second information of the multi-frame images in the video may be acquired. In this way, the establishment of the image database can be realized.

Therefore, in this embodiment of the present application, by acquiring the poses of multiple frames of images in the video shot by the second terminal in the second local coordinate system of the second terminal, and combining the relevant positions of the multiple frames of images in the video in the map, to determine the positions of the multiple frames of images in the video in the world coordinate system, and to acquire the first information and the second information of these images to establish the image data volume. Therefore, when constructing an image database, the present application only needs to obtain the pose of the image in the world coordinate system, as well as the first and second information of the image, but does not need to obtain the 3D point cloud features of the environment (for example, the global 3D point cloud feature).

It should be noted that, in this embodiment of the present application, the first terminal and the second terminal may be the same terminal device or different terminal devices, which are not limited.

In some embodiments, after the pose of the first image in the world coordinate system is acquired, the first information of the first image, the second information of the first image, and the world coordinate system of the first image may be added to the image database. The pose in the coordinate system is used to update the image database.

As an example, the third information may be used to indicate the position of the first frame of image in the video on the map, and/or the position of the last frame of image on the map. Exemplarily, the map may be a map obtained through a global positioning system (global positioning system, GPS), or a map obtained through a global satellite positioning system (global navigation satellite system, GNSS), which is not limited in this application.

As a possible implementation manner, the video can be collected based on the camera of the second terminal, and the second part of each image frame in the video can be obtained through a simultaneous localization and mapping (SLAM) algorithm in the second part of the second terminal. The pose in the coordinate system. Further, the pose of each image in the video in the world coordinate system can be calculated by combining the position of the first frame of image in the video in the map and the position of the last frame of image in the map. Here, the second local coordinate system, that is, the camera coordinate system of the second terminal, may be a relative coordinate system constructed with any one frame of image in the video as the origin of the second local coordinate system.

Exemplarily, video collection may be performed according to a planned collection route. For example, the user can hold the terminal, start from the starting point of the route, and face the environment along the route to continuously shoot until the end of the route is finished, so as to obtain the video corresponding to the route. For some or all of the video images in the video, the SLAM algorithm can be used to obtain the pose of the image in the local coordinate system of the camera. At the same time, the terminal can also obtain the position of the starting point of the route on the map, and/or the position of the end point of the route on the map, so as to calculate the pose of the image in the video in the world coordinate system.

With reference to the first aspect, in some implementations of the first aspect, the first position of the first image may also be acquired, and according to the first position of the first image, at least one image database may be determined in the image database. image. Wherein, the distance between the position of each image in the at least one image and the first position of the first image is less than a first threshold, and the first position comes from the GPS module or Wi-Fi of the first terminal (wireless fidelity, WiFi) module.

Wherein, as the first information of the first image is matched with the first information of the image in the image database, a specific method of determining at least one second image matching the first image in the image database In an implementation manner, the first information of the first image may be matched with the first information of the at least one image, and at least one second image matching the first image is acquired in the at least one image.

It should be noted that the above-mentioned first position is a "coarse positioning" position, and its accuracy is relatively low, and the error may be, for example, 3 to 10 meters.

Therefore, in this embodiment of the present application, by determining at least one image in the image database according to the first position of the first image, and then matching the first information of the first image with the first information of the at least one image, in the at least one image At least one second image matching the first image is acquired from the image. In this way, it is not necessary to match the first information of the first image with the first information of all images in the image database, thereby reducing the calculation amount of the terminal, reducing the time-consuming of matching, and improving the efficiency of acquiring the second image.

As an example, the above-mentioned at least one image obtained in the image data according to the first position may be an image set, and the image set may be referred to as an image candidate set.

As a possible implementation manner of determining at least one second image according to the first information of the first image and the first information of the images in the image database or the image candidate set, the first information of the first image and the image database or The similarity of the first information of the images in the image candidate set, and then the at least one second image is determined according to the similarity, which is not limited in this application.

As a possible implementation, after obtaining the similarity between the image in the image database or the image candidate set and the first image, the calculated similarity may be sorted to obtain the first m images with the highest similarity, as The second image, where m bits are an integer greater than 1. As an example, m may be 20. Or, in some other implementation manners, an image whose similarity with the first image is greater than a preset threshold may be used as the second image, which is not limited in this application.

Here, the process of calculating the similarity between the first information of the first image and the first information of the images in the image database or the image candidate set may be referred to as matching the first image with the images in the image database or the image candidate set. This application This is not limited.

With reference to the first aspect, in some implementations of the first aspect, as the first information of the first image and the first information of the image in the image database, it is determined in the image database that it is the same as the first image. An implementation manner of the at least one second image that matches the image, according to the first information of the first image and the first information of the image in the image database, to determine the number of images that match the first image in the image database. and then delete the images whose distance from the cluster center of the plurality of third images is greater than the second threshold from among the plurality of third images, so as to obtain the at least one second image. Wherein, the cluster center is determined according to the positions of the plurality of third images in the world coordinate system.

Exemplarily, an image whose distance from the cluster centers of the plurality of third images is greater than the second threshold is an outlier image in the plurality of third images.

Here, the image set composed of the plurality of third images may also be referred to as a similar image set, and the similar image set includes a similar image set composed of the above at least one second image. In addition, when there is no outlier image among the plurality of third images, the operation of deleting the image may not be performed. At this time, the third image is the second image.

Since images at different locations may have very similar textures, there may be errors in the similarity calculation process, which may lead to outlier images in a set of similar images. Therefore, by eliminating these outlier images in the present application, errors in the images in the similar image set can be eliminated, thereby helping to obtain a more accurate similar image set. Here, acquiring a more accurate set of similar images can help to make the pose of the second image acquired subsequently in the first local coordinate system more accurate, thereby helping to improve the pose of the first image in the world coordinate system accuracy.

With reference to the first aspect, in some implementations of the first aspect, as the first information of the first image and the first information of the image in the image database, it is determined in the image database that it is the same as the first image. An implementation manner of at least one second image for image matching, which can determine a plurality of fourth images matching the first image in the image database according to the first information of the first image and the first information of the image in the image database , and then delete the images whose angle is smaller than the third threshold value and/or the distance is smaller than the fourth threshold value among the plurality of fourth images, so as to obtain the at least one second image. The angle is the difference between the poses of the at least two images in the world coordinate system, and the distance is the difference between the poses of the at least two images in the world coordinate system. As an example, the angle may be a difference in pitch, yaw or roll of poses of at least two images in the world coordinate system.

Exemplarily, among the plurality of fourth images, an image whose angle is smaller than the third threshold and/or whose distance is smaller than the fourth threshold is a redundant image among the plurality of second images.

Here, the image set composed of the plurality of fourth images may also be referred to as a similar image set, and the similar image set includes a similar image set composed of the above at least one second image. In addition, when there is no redundant image among the plurality of fourth images, the operation of deleting the image may not be performed. At this time, the fourth image is the second image.

Since the images whose angles are smaller than the preset threshold and/or whose distances are smaller than the preset threshold have a high degree of overlap in spatial distribution, redundant images exist in the image-similar image set. Therefore, in the embodiment of the present application, by deleting the images whose angle is smaller than the preset value and/or the distance is smaller than the preset value in the similar image set, the overlapping degree of the remaining second images in the similar image set can be moderate, and the surrounding space can be uniformly covered , which in turn helps to obtain a more refined set of similar images. Here, acquiring a more refined set of similar images can help reduce the amount of calculation of the pose of the second image acquired in the first local coordinate system, thereby helping to improve the accuracy of acquiring the first image in the world coordinate system. Pose efficiency.

In some embodiments, after the outlier images are deleted from the plurality of second images, redundant images may be further deleted, which is not limited in this application.

In a second aspect, an embodiment of the present application provides a positioning device for executing the method in the first aspect or any possible implementation manner of the first aspect. Specifically, the device includes a device for executing the first aspect. or a module of the method in any possible implementation manner of the first aspect. The apparatus may include an acquisition unit, a processing unit and an output unit.

an acquiring unit, configured to acquire the first image captured by the first terminal.

a processing unit, configured to determine at least one second image matching the first image in the image database according to the first information of the first image and the first information of the image in the image database, wherein the The first information is used to indicate the global feature of the image, and the image database includes the first information of the multi-frame image, the second information of the multi-frame image, and the pose of the multi-frame image in the world coordinate system , wherein the second information is used to indicate local features of the image.

The processing unit is further configured to determine, according to the second information of the first image and the second information of the at least one second image, where the first image and the at least one second image are located. The pose in the first local coordinate system of the first terminal is described.

an output unit, configured to: according to the pose of the first image under the first local coordinate system, the pose of the at least one second image under the first local coordinate system, and the at least one second image The pose of the image in the world coordinate system, and the pose of the first image in the world coordinate system is output.

With reference to the second aspect, in some implementations of the second aspect, an establishment unit is further included for establishing the image database.

With reference to the second aspect, in some implementations of the second aspect, the establishing unit is specifically configured to acquire a video shot by a second terminal, and acquire multiple frames of images in the video in a second part of the second terminal. The pose in the coordinate system, and according to the third information and the pose of the multi-frame images in the video in the second local coordinate system, determine the position of the multi-frame images in the video in the world coordinate system posture. Wherein, the third information is used to indicate the position of at least part of the images in the video in a map, and the map is associated with the world coordinate system.

The establishing unit is further configured to acquire the first information and the second information of the multi-frame images in the video.

With reference to the second aspect, in some implementations of the second aspect, the acquiring unit is further configured to acquire a first position of the first image, where the first position comes from a GPS module of the first terminal or WiFi module.

Wherein, the acquiring unit may receive data sent by the GPS module or the WiFi module, such as the above-mentioned first position.

Optionally, the obtaining unit may also send a request message to the GPS module or the WiFi module, where the request message is used to request the GPS module or the WiFi module to send the first position of the first image collected by the GPS module or the WiFi module to the obtaining unit. In response to the request message, the GPS module or the WiFi module may send the first location to the acquisition unit.

The processing unit is further configured to determine at least one image in the image database according to the first position of the first image, and the position of each image in the at least one image is the same as the first position of the first image. The distance between the locations is less than the first threshold.

The processing unit is further configured to match the first information of the first image with the first information of the at least one image, and obtain at least one second image matching the first image in the at least one image. image.

With reference to the second aspect, in some implementations of the second aspect, the processing unit is specifically configured to: according to the first information of the first image and the first information of the image in the image database, in the image database Determine a plurality of third images matching the first image, and delete the plurality of third images, the distance from the cluster center of the plurality of third images is greater than the second threshold, to obtain The at least one second image, wherein the cluster centers are determined according to the positions of the plurality of third images in the world coordinate system.

With reference to the second aspect, in some implementations of the second aspect, the processing unit is specifically configured to: according to the first information of the first image and the first information of the image in the image database, in the image database determining a plurality of fourth images matching the first image, and deleting the images whose angle is smaller than the third threshold value and/or the distance is smaller than the fourth threshold value among the plurality of fourth images, so as to obtain the at least one first image Two images, wherein the angle is the difference between the poses of the at least two images in the world coordinate system, and the distance is the difference between the poses of the at least two images in the world coordinate system.

In a third aspect, an embodiment of the present application provides a positioning device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more programs The execution of the one or more processors causes the one or more processors to implement the method as described above in the first aspect or in any possible implementation manner of the first aspect.

In a fourth aspect, embodiments of the present application provide a computer-readable medium for storing a computer program, where the computer program includes instructions for executing the method in the first aspect or any possible implementation manner of the first aspect.

In a fifth aspect, the embodiments of the present application further provide a computer program product containing instructions, when the computer program product runs on a computer, the computer is made to execute the method in the first aspect or any possible implementation manner of the first aspect .

It should be understood that the beneficial effects obtained by the description of the second to fifth aspects of the present application and the corresponding implementation manners can be referred to the beneficial effects obtained by the first aspect of the present application and the corresponding implementation manners, and will not be repeated.

Description of drawings

1 is a schematic block diagram of a positioning device provided by an embodiment of the present application;

2 is a schematic block diagram of a system architecture of a solution applied to an embodiment of the present application;

3 is a schematic flowchart of a positioning method provided by an embodiment of the present application;

FIG. 4 is an example of an indoor collection route according to an embodiment of the present application;

Fig. 5 is an example of the extracted video for a video in the embodiment of the present application;

Fig. 6 is an example of the visualization result of the image database provided by the embodiment of the present application;

FIG. 7 is an example of retrieving a second image according to an embodiment of the present application;

8 is a specific example of performing regional clustering on multiple frames of second images;

FIG. 9 is an example of further deleting redundant images after deleting outlier images in a plurality of second images;

FIG. 10 is a specific example of the pose calculation provided by the embodiment of the present application;

FIG. 11 is a specific example of the real-time positioning result of the embodiment of the present application;

12 is a schematic flowchart of another positioning method provided by an embodiment of the present application;

13 is a schematic block diagram of another positioning apparatus according to an embodiment of the present application;

FIG. 14 is a schematic structural diagram of another positioning device according to an embodiment of the present application.

Detailed ways

The technical solutions in the present application will be described below with reference to the accompanying drawings.

First, the related terms involved in this application are briefly introduced.

Pose: that is, the information used to indicate the position and pose of an image or a content in an image. Wherein, the content can be objects, people, buildings, animals, etc., which is not limited in this application. As an example, the pose may be 6 degrees of freedom (6DOF). The position can be represented by coordinates (X, Y, Z) in Euclidean space, and the attitude is represented by the rotation coordinates pitch, yaw, and roll.

Simultaneous localization and mapping (SLAM): Start moving from an unknown position in an unknown environment, and perform self-positioning according to position estimation and map during the movement process, and build incremental maps based on self-positioning. Achieve autonomous positioning and navigation. Through SLAM tracking and positioning, the pose of the camera in the relative space (that is, the relative coordinate system) can be obtained.

FIG. 1 is a schematic block diagram of a positioning apparatus 100 provided by an embodiment of the present application. The apparatus 100 may be applied to a terminal, such as a mobile phone, a wearable device, a virtual reality (VR) device, an AR device, a vehicle-mounted smart terminal, etc. The apparatus 100 may also be applied to a cloud, such as a server, to which the embodiments of the present application Not limited. As an example, the apparatus 100 may be used for visual positioning.

As shown in FIG. 1 , the apparatus 100 includes an image retrieval module 110 and a pose calculation module 120 .

The image retrieval module 110 is configured to obtain the first image (also called the image to be positioned, the image to be positioned, etc.) captured by the first terminal, and, according to the first information of the first image and the first image of the image in the image database, A message that determines at least one second image in the image database that matches the first image.

The first information is used to indicate the global feature of the image, and may also be referred to as global feature information. The image database includes first information of the multi-frame images, second information of the multi-frame images, and poses of the multi-frame images in the world coordinate system. The second information is used to indicate local features of the image, and may also be referred to as local feature information.

In some embodiments, the above-mentioned at least one second image may form an image set, and the image set may be referred to as a similar image set of the first image.

In this embodiment of the present application, the image database may also be referred to as an image pose library, an image pose feature library, a pose feature library, etc., which is not limited thereto.

The pose calculation module 120 may determine the pose of the first image and the at least one second image in the first local coordinate system of the first terminal according to the second information of the first image and the second information of the at least one second image. . After that, according to the pose of the first image under the first local coordinate system, the pose of the at least one second image under the first local coordinate system, and the pose of the at least one second image under the world coordinate system, output the first image The pose of an image in the world coordinate system.

As a possible implementation manner, the first local coordinate system and The mapping relationship of the world coordinate system (or can also be called a transformation relationship). Then, according to the mapping relationship, the pose of the first image under the first local coordinate system can be transformed, the pose of the first image under the world coordinate system is determined, and the pose is output.

As an example, the mapping relationship between the world coordinate system and the first local coordinate system may be described by a rotation matrix or a translation vector, which is not limited in this application.

In this embodiment of the present application, the world coordinate system may also be referred to as an absolute coordinate system. As an example, the world coordinate system can be used as a reference coordinate system in the environment to describe the position of the camera (which can also be the terminal including the camera), and can also be used to describe the position of any object in the environment.

As a possible implementation manner, the image database may include multiple mapping relationships, wherein one mapping relationship may indicate the identification of an image, the first information of the image, the second information of the image, and the image in the world coordinate system The corresponding relationship between the lower poses is not limited in this application.

As an example, the pose calculation module 120 may acquire the pose of the at least one second image in the world coordinate system and the second information from the image database, which is not limited in this application.

In this embodiment of the present application, the pose of the first image in the world coordinate system is the pose of the first terminal in the world coordinate system when the first image is captured, and may also be referred to as the pose of the camera.

In some embodiments, the above-mentioned first image and at least one second image may form an image set, which may be referred to as a partial image set. The above-mentioned first local coordinate system, that is, the camera coordinate system of the first terminal, may be a relative coordinate system constructed by taking any one frame of image in the local image set as the origin of the first local coordinate system. As an example, SLAM may be used to construct the first local coordinate system, which is not limited in this application.

Therefore, in this embodiment of the present application, by determining in the image database at least one second image that matches the first image captured by the first terminal, and the position of the first image and the at least one second image in the local coordinate system of the first terminal pose, and according to the pose of the first image in the first local coordinate system, the pose of at least one second image in the first local coordinate system, and the pose of the at least one second image in the world coordinate system , and output the pose of the first image in the world coordinate system, so as to realize the positioning of the first image. Therefore, the embodiment of the present application can obtain the relationship between the local coordinate system and the world coordinate system based on the pose of a part of the image in the local coordinate system and the pose in the world coordinate system, so that the image to be positioned (for example, the first image) is Positioning in the world coordinate system without acquiring a 3D point cloud for each image.

Compared with the 3D point cloud feature library (which includes global 3D point cloud features) of the environment in the existing solution, and the calculation solution of realizing pose based on feature point matching, the embodiment of the present application does not use the 3D point cloud of the image. feature, but uses the pose of the image for localization. Since the construction of the 3D point cloud feature library of the environment needs to rely on professionals and acquisition equipment, which is time-consuming and labor-intensive, and the cost is high, and this application does not need to acquire 3D point cloud features, on the one hand, the application can use low-cost equipment, such as Cameras with low resolution and small field of view (such as mobile phone cameras) can build image databases without relying on professionals and professional equipment, and can reduce the data volume of image databases, thereby reducing the cost of building image databases; On the one hand, the present application can shorten the time for constructing the database, which is conducive to the efficient construction of the image database. Therefore, the positioning solution based on the pose of the image and the image database of the present application can contribute to high-efficiency and low-cost positioning.

FIG. 2 is a schematic block diagram of a system architecture 200 to which the solution of the embodiment of the present application is applied. As shown in FIG. 2 , the system architecture 200 may include a hardware abstraction layer data interface 210 , a pose acquisition module 220 , an application service 230 and a data processing module 240 . The pose acquisition module 220 may include an image retrieval module 210 and a pose calculation module 220 . The image retrieval module 210 may further include a feature extraction unit 2211 and an image retrieval unit 2212, and the pose calculation module 220 may further include a local pose calculation unit 2221 and a coordinate conversion unit 2222. The data processing module 240 may include an image database building module 241 and a video 242.

It should be understood that FIG. 2 shows modules or units of a system architecture applicable to the embodiment of the present application, but these modules or units are only examples, and the embodiment of the present application may further include other parts or parts of each part in FIG. 2 . Variations, or possibly not all of the modules or units in FIG. 2 are intended to be included.

In FIG. 2 , the pose acquisition module 220 can be used as an example of the above apparatus 100 , the image retrieval module 210 can be used as an example of the above image retrieval module 110 , and the pose calculation module 222 can be used as one of the pose calculation modules 120 For example, this application does not limit it.

In some embodiments, the pose acquisition module 220 and the image database construction module 241 may exist in the form of binary software packages. In addition, the pose acquisition module 220 can be deployed in the framework layer of the operating system of the terminal, and provides positioning information for service applications at the application layer through an interface, such as an application programming interface (API).

It should be noted that the embodiments of the present application may be implemented based on hardware such as a built-in global positioning system (global positioning system, GPS)/magnetometer, gyroscope, wireless fidelity (WiFi) chip, and camera chip. The hardware drivers or data read/write modules corresponding to these hardwares or chips can interact with the data and control through the hardware abstraction layer data interface 210 according to the standard system interface and the upper-layer positioning software service program.

In some embodiments, the positioning solution provided by the embodiments of the present application can realize pose calculation through the framework of offline library building-online positioning stage.

In the offline library construction stage, the image database construction module 241 may construct an image database, for example, based on the video 242, construct an image database. As an example, the image database construction module 241 can obtain the pose of each image frame in the video 242 in the local coordinate system of the camera through the SLAM algorithm, and fuse the position of the video 242 in the map to obtain the position in the world coordinate system pose sequence of image frames. Further, the image database building module 241 can also extract global features (also referred to as global feature information, first information) and local features (also referred to as local feature information, second information) of the image frames in the video 242 . Here, the local coordinate system may be a local coordinate system constructed by taking any image in the video 242 as the origin of the second local coordinate system.

The above-mentioned video 242 may be referred to as a data source for constructing an image database, for example, a mobile phone video.

As an example, after the image database building module 241 builds the image database, it can output the image feature library. Optionally, the image database may be stored in a memory, which is not limited in this application.

Therefore, when constructing an image database in the embodiment of the present application, it is only necessary to obtain the pose of the image in the world coordinate system, as well as the global feature information and local feature information of the image, and it is not necessary to obtain the 3D point cloud features that are highly unified with the environment ( such as global 3D point cloud features). Therefore, on the one hand, the present application can use low-cost equipment, such as cameras with low resolution and small field of view (such as mobile phone cameras) to construct an image database, without relying on professionals and professional equipment, and can analyze the image database. The amount of data is reduced, thereby reducing the cost of constructing the image database; on the other hand, the present application can shorten the time for constructing the database, which is conducive to the efficient construction of the image database. Therefore, the positioning solution based on the pose of the image and the image database of the present application can contribute to high-efficiency and low-cost positioning.

In the online positioning stage, the image can be acquired in real time, and the pose of the image can be acquired according to the image.

As an example, the hardware abstraction layer data interface 210 may be used to obtain image data collected by a camera, such as obtaining an image to be positioned, a first image, and the like. Optionally, the hardware abstraction layer data interface 210 may also be used to acquire sensor signals (such as magnetometer, gyroscope parameters, etc.), WiFi chip parameters, which are not limited. As an example, the hardware abstraction layer data interface 210 may obtain the above information from a standard API extracted from the hardware abstraction layer of the operating system of the terminal, which is not limited in this application.

After acquiring the first image, the hardware abstraction layer data interface 210 may send the first image to the pose acquiring module 220 . After the pose acquisition module 220 acquires the first image, the feature extraction unit 2211 can extract the global feature information of the first image, and the image retrieval unit 2212 can, according to the global feature information of the first image and the global feature information of the image in the image database, At least one second image matching the first image is determined in the image database. As an example, the set of second images may be referred to as a set of similar images.

The local pose calculation unit 2221 may determine the pose of the first image and the at least one second image in the local coordinate system according to the local feature information of the first image and the local feature information of the at least one second image. Here, the local coordinate system may be, for example, the above-mentioned first local coordinate system. For details, reference may be made to the description of the first local coordinate system, which will not be repeated here. After that, the coordinate conversion unit 2222 may determine the distance between the local coordinate system and the world coordinate system according to the pose of the at least one second image in the world coordinate system and the pose of the at least one second image in the local coordinate system mapping relationship, and then, according to the mapping relationship, transform the pose of the first image in the local coordinate system to obtain the pose of the first image in the world coordinate system.

In some optional embodiments, after acquiring the pose of the first image, the pose obtaining module 220 may send the pose to the application service 230 . The application service 230 can use the pose to provide the user with a pose location service.

In some embodiments, the application service 230 may also initiate a request for the pose location service to the pose acquisition module 220 . In response to the request, the pose obtaining module 220 can obtain the image from the hardware abstraction layer data interface 210, and locate the image to obtain the pose of the image in the world coordinate system.

As an example, the application service 230 may be various types of location based services (location based services, LBS), AR/VR application services, etc., for example, including but not limited to dedicated positioning applications, various e-commerce shopping applications, various types of social communication Application, various car applications, online to offline (online to offline, O2O) on-site service application, exhibition hall self-service tour application, family separation prevention application, emergency rescue service application, audio-visual entertainment application, game application, etc. need to be provided Precise positioning applications. A typical application scenario is, for example, AR navigation, that is, superimposing navigation icons on the real world captured by the camera of the terminal to provide more indicative navigation information.

In the system architecture 200 shown in FIG. 2 , the pose obtaining module 220 may be implemented on a client (eg, a terminal) to support various application services on the client. As an example, in response to the positioning service request initiated by the application service 230, as the positioning software of the system service, the pose obtaining module 220 starts running, and uses the positioning method provided by the embodiment of the present application to obtain the image pose on the smartphone in real time.

As a possible implementation, the system architecture 200 may support a client-server model. As an example, the data processing module 140 may be stored in a server (eg, a server), and the video data acquired by a client (eg, a terminal) may be uploaded to the server through a network connection. After the server completes the construction of the image database, the image database or part of the data in the image database can be downloaded to the client through a network connection.

As another possible implementation manner, the system architecture 200 may support a client mode, that is, positioning processing may be performed directly offline. At this time, the data processing module 140 can be stored in the client, forming an offline pure client mode.

FIG. 3 shows a schematic flowchart of a positioning method 300 provided by an embodiment of the present application. As an example, the method 300 includes an offline library construction stage and an online positioning stage for description. The method 300 will be described below in conjunction with the system architecture 200 in FIG. 2 .

It should be understood that FIG. 3 shows steps or operations of the positioning method, but these steps or operations are only examples, and the embodiments of the present application may also perform other operations or variations of the respective operations in FIG. 3 . Furthermore, the various steps in FIG. 3 may be performed in a different order than that presented in FIG. 3 , and it is possible that not all operations in FIG. 3 are intended to be performed.

As shown in FIG. 3 , the method 300 may include steps 301 to 310 . The offline composition phase may include steps 301 to 304 , and the online positioning phase may include steps 305 to 310 .

Step 301, video capture. Here, the captured video can be used as the input of the offline library construction stage.

As an example, video capture can be done indoors. As a possible implementation manner, a collection route may be planned according to an indoor floor plan, for example, multiple collection routes may be planned, and these collection routes may be located on one floor or multiple floors, which is not limited.

Figure 4 shows an example of an indoor acquisition route. Exemplarily, the user may start from the starting point of the route with a terminal (eg, a mobile phone), and face the environment along the route to continuously shoot until the end of the route is completed.

While collecting the video, the terminal can also obtain the location of the route on the map. For example, when starting to shoot a video, the user can manually start the positioning module in the terminal to obtain the location of the starting point of the route on the map. When reaching the end of the route, the user can manually start the positioning module to obtain the location of the end of the route on the map. . Alternatively, when starting to shoot a video, the positioning module in the terminal can automatically obtain the position of the starting point of the route on the map, and when reaching the end point of the route, the positioning module can automatically obtain the position of the end point of the route on the map. Here, the position of the starting point of the route on the map can be referred to as the position of the first frame of the image in the captured video, and the position of the end point of the route in the map can be referred to as the last image in the captured video. The location of the frame image on the map.

In this embodiment of the present application, the position information of the starting point and/or the ending point in the map may be used to assist the calculation, for example, to calculate the pose of the image in the video in the world coordinate system. Exemplarily, the positioning module may be, for example, at least one of GPS/magnetometer, gyroscope, WiFi chip, and the like.

It should be noted that the indoor scene is used as an example for description here. This solution is also applicable to the outdoor scene. That is, in step 301, video collection can also be performed outdoors, for example, an outdoor collection route can be planned, which is not limited in this application. .

Step 302, frame extraction of the video image.

As an example, in the process of using the terminal to capture and acquire a video image, the terminal may also automatically perform frame extraction processing on the video image to acquire multiple frames of images. FIG. 5 shows an example of extracted multi-frame images for a video.

In some embodiments, when the client-server mode is adopted, the terminal can upload the acquired video to the server. When the client mode is adopted, the terminal can store the video and process the video.

Step 303, build an image database.

As an example, step 303 may be performed by the image database building module 241 on the data processing module 240 in FIG. 2 described above. Referring to FIG. 3 , step 303 may further include step 3031 and step 3032 .

Step 3031, SLAM path tracking.

As an example, for each frame of video image, the SLAM algorithm can be used to obtain the pose of the image in the local coordinate system of the camera (ie, the terminal). Here, the local coordinate system is the relative coordinate system of the camera in the process of constructing the image database.

In other embodiments, an AR engine (AREngien) algorithm may also be used to obtain the pose of each frame of image in the local coordinate system of the camera, which is not limited in this application.

After obtaining the pose of each frame of image in the local coordinate system of the camera, the image coordinate registration can be performed in combination with the position of the image in the map, and the pose of the image in the camera coordinate system can be converted to the world coordinate system. For example, the location of the starting point of the video collection route on the map (that is, the location of the first frame in the video on the map), and/or the location of the end point of the video recording route on the map (that is, the last frame of the video) image location in the map) for image coordinate registration.

FIG. 6 shows an example of visualization results of an image database. As shown in Figure 6, the collection route can be selected on the left side of the interface. When a collection route is selected, the collection route can be displayed on the right side of the interface, for example, the location of the collection route on the map can be displayed. As an example, FIG. 6 shows the 3-segment collection route collected on the 4th floor of Building A. Among them, as an example, the name of the collection route 1 can be 0a1c55c4-9303-46a5-b785-3ad77d80d809_20200415-090323, the name of the collection route 2 can be 0a1c55c4-9303-46a5-b785-3ad77d80d809_20200415-091659, and the name of the collection route 3 can be 0a1c55c4-9303-46a5-b785-3ad77d80d809_20200415-093745.

Optionally, the folder where the image data corresponding to the collection route is located can also be opened through the visual display interface. For example, the file where the image data corresponding to the selected collection route is located can be opened through the "Open Folder" button in the upper right corner of Figure 6. folder.

Exemplarily, after selecting the track tree-building A-4th floor-track-0ale55c4-9303-46a5-b785-3ad77d80d809_202004150-090323 track and choosing to open the folder, the following table 1 can be displayed. Among them, pos_x, pos_y, pos_z represent position information; quat_x, quat_y, quat_z, quat_w represent the direction represented by quaternion, which can be converted into attitude information, such as yaw, roll, pitch.

Table 1

FileName pos_x pos_y pos_z quat_x quat_y quat_z quat_w 000001 1.8536 0.0189 76.6551 0.4322 -0.5769 -0.5329 0.4430 000002 2.8263 0.0043 76.6297 0.4190 -0.5702 -0.5573 0.4341 000003 3.3308 0.0002 76.5465 0.4543 -0.5904 -0.53556 0.3975 …

Optionally, in the visualization result of the image database, image frames obtained on each collection route, global specific information or local feature information, etc. may also be displayed, which is not limited in this application.

Step 3032, global/local feature extraction.

Specifically, global feature information and local feature information can be extracted from each frame of image collected by the terminal. The global feature information is image retrieval feature information, such as NetVLAD, BoVW, etc., which is not limited. The local feature information is, for example, scale-invariant feature transform (SIFT), ORB, SUPERPOINT, D2Net, etc., without limitation.

Step 304, outputting the image database.

As an example, the image database may include the pose, global features and local features of each frame of images in the video collected along the planned path in the world coordinate system. As an example, an image in an image database can be represented as (R, T, F _l , F _g ), where R represents position information, T represents angle information, F _l represents global features, and F _g represents local features.

Step 305, image acquisition. Here, the collected image is the first image, which may also be called an image to be positioned or others, which is not limited in this application.

As an example, a user may use the terminal to take an image. Correspondingly, the hardware abstraction layer data interface 210 in the smart phone can acquire the image data collected by the camera, and send the image data to the pose acquisition module 220 .

It should be noted that the terminal in step 305 and the terminal in step 301 may be the same terminal or different terminals, which are not limited in this application.

Optionally, in step 306, the initial position is acquired.

As an example, when image acquisition is performed in step 305 , the hardware abstraction layer data interface 210 may also acquire the positioning information collected by the WiFi chip or the GPS module, and send the positioning information to the pose acquisition module 220 . The positioning information may be referred to as the initial position of the first image.

It should be noted that the position provided by the WiFi chip or the GPS module is a "coarse positioning" position, and its accuracy is low, and the error may be, for example, 3 to 10 meters. This position can be used as the initial position of the first image.

In addition, the hardware abstraction layer data interface 210 can also send the information of the gyroscope and the magnetometer to the pose obtaining module 220, so that the pose obtaining module 220 can obtain the positioning information according to the WiFi chip or the GPS module, as well as the information of the gyroscope and the magnetometer. The acquisition information jointly determines the initial position of the first image.

Optionally, step 307, floor identification. As an example, when the initial position of the first image acquired in step 306 includes height information, the floor corresponding to the first image may be identified based on the initial position.

Step 308, image retrieval.

As an example, the image retrieval module 221 in FIG. 2 may, based on the global feature information of the first image obtained in step 305 and the global feature information of the image in the image database, determine in the image database at least one image matching the first image. a second image. In this embodiment of the present application, determining at least one second image matching the first image in the image data can also be described as retrieving at least one second image matching the first image in an image database.

As an example, the set of at least one second image may be referred to as a set of similar images.

In some possible implementations, the image retrieval module 221 may perform image retrieval based on the first image and the initial position obtained in step 306 .

In some possible implementations, the image retrieval module 221 may perform image retrieval based on the first image, the initial position obtained in step 306 , and the floor obtained in step 307 .

Continuing to refer to FIG. 3 , as an example, step 308 may include the following steps 3081 to 3083 . As an example, step 3081 may be performed by the feature extraction unit 2211 , and steps 3082 and 3083 may be performed by the image retrieval unit 2212 .

Step 3081, global feature extraction.

As an example, global feature information of the first image may be extracted using a deep learning algorithm. For example, the NetVLAD layer can be added after the convolutional neural network framework to realize the extraction of the global descriptor (an example of global feature information). The global feature of the image is the overall attribute of the image, for example, it may include color feature, texture feature, shape feature, etc., which is not limited.

Step 3082, preliminary retrieval of images.

As a possible implementation manner, at least one second image matching the first image may be retrieved in the image database according to the global feature information obtained in step 3081 to obtain a similar image set.

As another possible implementation manner, at least one image may be determined in the image database according to the initial position obtained in step 306, wherein the distance between the position of each image in the at least one image and the initial position of the first image less than the preset threshold. Then, at least one second image matching the first image is retrieved in the at least one image to obtain a set of similar images.

As a specific example, an area (for example, it may be called a buffer area) may be constructed according to the initial position of the first image. For example, a circular buffer zone may be constructed with the initial position of the first image as the center and a radius of 5 meters (or 10 meters, or 30 meters, which is not limited). Then, at least one second image matching the first image is retrieved from the images located in the buffer in the image database to obtain a set of similar images. As an example, the set of images located within the buffer may be referred to as the image candidate set.

Therefore, in this embodiment of the present application, by determining at least one image in the image database according to the first position of the first image, and then matching the first information of the first image with the first information of the at least one image, in the at least one image At least one second image matching the first image is acquired from the images. In this way, it is not necessary to match the first information of the first image with the first information of all images in the image database, thereby reducing the calculation amount of the terminal, reducing the time-consuming of matching, and improving the efficiency of acquiring the second image.

As a possible implementation manner, the similarity between the global feature information of the first image and the global feature information of the images in the image database or the image candidate set may be calculated, and the second image may be determined according to the similarity.

Exemplarily, the similarity can be calculated using the following formula (1):

Among them, similarity(X, Y) represents the similarity between X and Y, X represents the global feature information of the first image, Y represents the global feature information of the image to be matched in the feature library, and x _i represents each feature code of X component, _yi represents each component in the feature code of Y, n represents the total number of components in a feature code, and n is a positive integer.

After obtaining the similarity between each frame of images and the first image, the calculated similarity may be sorted to obtain the first m images with the highest similarity as the second image, where m is an integer greater than 1. As an example, m may be 20. Or, in some other implementation manners, an image whose similarity with the first image is greater than a preset threshold may be used as the second image.

)的全局特征信息，“○”表示聚类区域中最邻近第一图像的图像(可称为最邻近图像)的全局特征信息，“●”表示缓冲区中次邻近第一图像的图像(可称为次邻近图像)的全局特征信息。当最邻近图像与第一图像的距离和次邻近图像与第一图像的距离比值大于阈值时，则表示该最邻近图像与第一图像匹配。可选的，可以根据该方法，依次在该缓冲区中确定20个能够与第一图像匹配的图像，作为第二图像。FIG. 7 shows an example of retrieving the second image. As shown in FIG. 7 , “×” represents the global feature information (may be expressed as C _k ^VLAD ) of the image of the cluster center in the one cluster area, and the first image is included in the cluster area. All images in the clustered region can be retrieved to obtain the second image. Among them, "★" represents the first image (which can be expressed as

), "○" represents the global feature information of the image closest to the first image in the clustering area (may be referred to as the nearest neighbor image), "●" represents the image next to the first image in the buffer area (may be called the closest neighbor image) known as the global feature information of the next adjacent image). When the ratio between the distance between the closest adjacent image and the first image and the distance between the second adjacent image and the first image is greater than the threshold, it means that the closest adjacent image matches the first image. Optionally, according to the method, 20 images that can be matched with the first image may be sequentially determined in the buffer as the second image.

Optionally, in step 3083, pose-aware image refinement.

As an example, when the number of acquired second images is multiple, image refinement may be further performed on the multiple second images. For example, the plurality of second images may be refined based on their poses in the world coordinate system.

As a possible implementation manner, outlier images in the second image (ie, similar image set) may be deleted based on the pose of the second image in the world coordinate system. Here, the outlier image in the second image refers to an image whose distance from the cluster centers of the plurality of second images is greater than a preset threshold, wherein the cluster centers are based on the plurality of second images in the world coordinate system The location is determined.

Exemplarily, the position of each second image in the world coordinate system may be obtained from the image database, and according to the position, the cluster centers of the plurality of second images may be determined, and the cluster centers of the plurality of second images may be determined in the plurality of second images. Images whose distance from the cluster center is greater than the preset threshold are regarded as outlier images.

It should be noted that the above-mentioned process of determining an image whose distance from the cluster center is greater than the second threshold (that is, determining an outlier image) is to determine whether there is a distance from the cluster center in the plurality of second images that is greater than the second threshold. image (that is, whether there is an outlier image). As a possible judgment result, there may be no outlier images in the plurality of second images. As another possible judgment result, there may be at least one frame of outlier images in the plurality of second images.

In some embodiments, when there is no outlier image in the plurality of second images, the operation of deleting the image may not be performed.

Since images at different locations may have very similar textures, there may be errors in the similarity calculation process, which may lead to outlier images in a set of similar images. Therefore, by eliminating these outlier images in the present application, errors in the images in the similar image set can be eliminated, thereby helping to obtain a more accurate similar image set. Here, acquiring a more accurate set of similar images can help to make the pose of the second image acquired subsequently in the first local coordinate system more accurate, thereby helping to improve the pose of the first image in the world coordinate system accuracy.

FIG. 8 shows a specific example of performing region clustering on multiple frames of second images. Among them, after clustering calculation of multiple frames of second images in the similar image set, it can be obtained that most of the second images are gathered in areas 1 and 2, and only a small part of the second images are scattered in areas 3 and 4. At this time, the second images corresponding to the clustering points in the areas 2 and 3 may be deleted, and the second images corresponding to the clustering points in the areas 1 and 2 are used as the second images in the updated similar image set.

As another possible implementation manner, redundant images in the second image (ie, the set of similar images) may be deleted based on the pose of the second image in the world coordinate system. As an example, the pose of each second image may be acquired from an image database, and then, according to the pose, images whose angles are smaller than a preset angle threshold and/or whose distances are smaller than a preset distance threshold are determined in the multiple frames of second images , as redundant images in the plurality of second images. The angle is the difference between the poses of the at least two images in the world coordinate system, and the distance is the difference between the poses of the at least two images in the world coordinate system.

As an example, the angle may be a difference in pitch, yaw or roll of poses of at least two images in the world coordinate system.

It should be noted that the above process of determining an image with an angle smaller than a preset angle threshold and/or a distance smaller than the preset distance threshold (ie, determining redundant images) is to determine whether there is an angle smaller than the preset angle in the plurality of second images. The process of thresholding and/or images whose distance is less than a preset distance threshold (ie, whether there are redundant images). As a possible judgment result, there may be no redundant images in the plurality of second images. As another possible judgment result, at least one frame of redundant image may exist in the plurality of second images.

In some embodiments, when there are no redundant images in the plurality of second images, the operation of deleting images may not be performed.

Since the images whose angle is less than the preset threshold and/or the distance is less than the preset threshold have a higher degree of overlap in the spatial distribution, it can be considered that the images whose angle is less than the preset threshold and/or the distance is less than the preset threshold have a higher degree of overlap in the spatial distribution. high, so that there are redundant images in the image set with similar images. Therefore, in the embodiment of the present application, by deleting the images whose angle is smaller than the preset value and/or the distance is smaller than the preset value in the similar image set, the overlapping degree of the remaining second images in the similar image set can be moderate, and the surrounding space can be uniformly covered , which in turn helps to obtain a more refined set of similar images. Here, acquiring a more refined set of similar images can help reduce the amount of calculation of the pose of the second image acquired in the first local coordinate system, thereby helping to improve the accuracy of acquiring the first image in the world coordinate system. Pose efficiency.

In some embodiments, after the outlier images are removed from the plurality of second images, redundant images may be further removed. FIG. 9 shows an example of further removing redundant images after removing outlier images in the plurality of second images. The position of the circle may represent the position of the second image, and the arrow may represent the posture of the second image. As a specific example, the angle threshold can be set to ±15°, and the distance threshold can be set to 0.5m. Correspondingly, when the difference (that is, the angle) of the postures of any two or more frames of the at least two frames of the second image is less than 15° and/or the position distance is less than 0.5m, it can be considered that the two or more frames The two images are redundant images (for example, they may correspond to the images corresponding to the positions of the white circles in FIG. 9 ). Correspondingly, redundant images in the second image will be eliminated by filtering by angle and position. At this time, the angle between the remaining second images is greater than or equal to 15° and/or the position distance is greater than or equal to 0.5m, which can cover the surrounding space uniformly and has a moderate degree of overlap.

As a specific example, after steps 3081 and 3082, 20 frames of second images in the original similar image set can be reduced to 8 frames of second images. Exemplarily, these 8 frames of images can be represented as (R ₁ , T ₁ ), (R ₂ , T ₂ ), (R ₃ , T ₃ ), (R ₄ , T ₄ ), (R ₅ , T ₅ ), respectively ), (R ₆ , T ₆ ), (R ₇ , T ₇ ), (R ₈ , T ₈ ). At this time, the first image can be represented as (R _q , T _q ).

Continuing to refer to FIG. 3 , after the image retrieval module 221 acquires the similar image set, the above-mentioned first image and at least one second image may be sent to the pose calculation module 222 .

Step 309, the camera pose is solved.

Exemplarily, after the pose solving module 222 acquires the first image and at least one second image, the camera pose can be solved. Referring to FIG. 3 , step 309 may further include steps 3091 and 3092 .

Step 3091, local pose calculation.

As an example, the local pose calculation unit 2221 in the image retrieval module 221 may perform a local pose calculation based on the first image and the at least one second image.

Specifically, the local pose calculation unit 2221 can determine, according to the local feature information of the first image and the local feature information of the at least one second image, that the first image and the at least one second image are in the local coordinate system of the first terminal pose. Exemplarily, the local pose calculation unit 2221 may use the first image and at least one second image as the local image set, and obtain the information of each image in the local image set according to the local feature information of each image in the local image set. The relative pose in the local coordinate system. Here, the local coordinate system may be a relative coordinate system of the camera constructed with any one frame of image in the local image set as the origin of the local coordinate system. Among them, the local feature of the image is the local expression of the image feature, which can reflect the local characteristics of the image.

As a possible implementation, the local pose calculation unit 2221 can match each image in the local image set according to the local feature information in the local image set, and obtain the pose of each image in the local image set in the local coordinate system, And the point cloud information that each image is mapped to three-dimensional space. Here, the pose of each image in the local image set in the local coordinate system and the point cloud information mapped into the three-dimensional space can be referred to as the local structure information of the local image set. In some embodiments, the process of obtaining the local structure information of the partial image set may be referred to as a process of restoring the local structure information of each image in the partial image set.

In the embodiment of the present application, by restoring the local structure information of the images in the local image set, on the one hand, the construction of a global 3D point cloud feature library can be avoided, and the cost of building the library can be saved, and on the other hand, the use of the global feature library to align a large number of images can be avoided. Therefore, a more accurate pose of each image in the local image set can be obtained.

Step 3092, coordinate transformation.

Exemplarily, the coordinate conversion unit 2222 may acquire the pose of the second image in the similar image set in the world coordinate system, and determine the local coordinate system according to the pose of the second image in the local coordinate system obtained in step 3091 above. and the mapping relationship between the world coordinate system (ie, the transformation relationship). Then, according to the mapping relationship, coordinate transformation is performed on the pose of the first image in the local coordinate system to obtain the pose of the first image in the world coordinate system.

As a possible implementation manner, the coordinate conversion unit 2222 may acquire the pose of the second image in the world coordinate system from the image database output in step 304 .

FIG. 10 shows a specific example of performing pose calculation. Among them, Figure (a) represents the partial image set input during pose calculation, including n frames of second images in the similar image set, which are represented as (R ₁ , T ₁ )…(R _n ,T _n ) , and the first image, denoted as (R _q , T _q ). As an example, in FIG. 10 , n is 8 for description, but the present application is not limited to this.

At the same time, when the pose calculation is performed, it is also necessary to input the pose and local feature information of the n-frame second images in the similar image set in the world coordinate system. As an example, the pose and local feature information of the n frames of second images in the world coordinate system may be obtained from an image database. Figure (e) shows an example of the distribution of image trajectories of 8 frames of second images in the world coordinate system, wherein the 8 boxes constituting the image trajectories correspond to the poses of the 8 frames of second images respectively.

As an example, for the input partial image set, the pose of each image in the partial image set in the local coordinate system can be estimated by using the SFM algorithm.

First, image feature points may be extracted for the first image in the local image set to obtain local feature information of the first image. Then, the local feature information of the first image and the local feature information of the second image are matched. (b) of FIG. 9 shows a specific example of three matched feature points in three frames of images.

Then, incremental camera parameter solving can be performed on feature points that match between images in the local image set. As an example, Fig. 9 (c) shows an example of incremental camera parameters corresponding to three frames of images in the partial image set, wherein the upper image in (c) includes 7 feature points. (c) The left image at the bottom of the figure is a frame of image in the local image set, including 4 feature points (for example, the 4 feature points on the left in the above 7 feature points). The incremental formula of the left image The parameter is P ₀ =K[I|0], where K represents the camera memory, I represents the initial direction (0, 0, 0, 1), and 0 represents the initial position (0, 0, 0). That is, the local coordinate system of the local image set can be constructed with the left image as the origin. (c) The middle image at the bottom of the figure is a frame of image in the local image set, including the 4 feature points (for example, the 4 feature points in front of the above 7 feature points). The incremental parameters of the middle image are: P ₁ =K[R ₁ |t ₁ ], where R ₁ represents the position of the intermediate image and t ₁ represents the angle of the intermediate image. (c) The lower right image in the figure is a frame of image in the local image set, including the 4 feature points (for example, the 4 feature points on the right in the above 7 feature points). The incremental formula of the right image The parameter is P _i =K[R _i |t _i ], where R _i represents the position of the intermediate image, and t _i represents the angle of the intermediate image.

Afterwards, the pose and 3D point cloud of each image in the local image set in the local coordinate system can be obtained according to the obtained incremental camera parameters. (d) Figure shows a specific example of the image trajectory distribution and 3D point cloud of 9 frames of images in the local coordinate system. Among them, the 9 boxes constituting the image trajectory correspond to the poses of the 9 frames of images respectively. Among them, the black box represents the pose of the first image in the local coordinate system, and the white box represents the pose of the eight frames of the second image in the local coordinate system.

After that, the pose (or image trajectory distribution) of each image in the local image set in the local coordinate system and the pose (or image trajectory distribution) of the second image in the similar image set in the world coordinate system can be input into The coordinate conversion unit 2222 performs coordinate conversion by the coordinate conversion unit 2222 . Figure (f) shows a specific example of coordinate conversion. The mapping relationship between the local coordinate system and the world coordinate system can be obtained according to the poses of the 8 frames of second images in the world coordinate system and the poses of the 8 frames of second images in the local coordinate system. As an example, the mapping relationship may be a similar transformation matrix (R _i , t _i , α _i ) from the local coordinate system to the world coordinate system. Further, by multiplying the pose of the first image in the local coordinate system by the similarity transformation matrix (R _i , t _i , α _i ), the pose of the first image in the world coordinate system can be obtained.

310. Output the positioning result.

Specifically, after the pose of the first image in the world coordinate system is obtained, the pose can be output, that is, the positioning result of the first image is output, thereby completing the positioning of the first image.

FIG. 11 shows a specific example of the real-time positioning result. Among them, the real scene in the figure is the current location of the terminal, the small circle in the lower map represents the visualization result of the current location in the two-dimensional map, and the arrow represents the direction information.

Therefore, in this embodiment of the present application, by determining in the image database at least one second image that matches the first image captured by the first terminal, and the position of the first image and the at least one second image in the local coordinate system of the first terminal pose, and according to the pose of the first image in the first local coordinate system, the pose of at least one second image in the first local coordinate system, and the pose of the at least one second image in the world coordinate system , and output the pose of the first image in the world coordinate system, so as to realize the positioning of the first image. Therefore, the embodiment of the present application can obtain the relationship between the local coordinate system and the world coordinate system based on the pose of a part of the image in the local coordinate system and the pose in the world coordinate system, so that the image to be positioned (for example, the first image) is Positioning in the world coordinate system without acquiring a 3D point cloud for each image.

Based on the above positioning solution, the embodiments of the present application can make full use of existing terminal devices, such as user-level receiver devices (such as mobile phones) or navigation-level receiver devices (such as vehicles), to collect images, extract image features, and improve robustness. image retrieval and matching to achieve positioning (such as 6DoF positioning). In the process of image retrieval, the embodiment of the present application can further consider the spatial distribution of the retrieved images on the basis of image retrieval based on the global feature information, and construct a double-threshold image refinement algorithm jointly with poses, so as to detect outlier images and redundant images. Deletion (also known as secondary retrieval) is performed to obtain a refined set of similar images. For a local image set, the embodiment of the present application can execute the SFM algorithm, recover the relative pose of the image in the local image set in the local coordinate system based on the local structure information, and combine the position of the image in the world coordinate system in the pose feature library The relative pose of the image to be positioned is converted into the world coordinate system to solve the camera pose.

The embodiments of the present application can solve the problem of difficulty in building a point cloud database for LBS/AR/VR application service requirements and inability to achieve large-scale promotion. Specifically, the embodiments of the present application propose solutions for indoor and outdoor positioning without relying on a 3D point cloud feature library. Compared with the positioning scheme that needs to build a point cloud database, the embodiments of the present application can have the advantages of high positioning accuracy and low cost, and break the high cost barrier of traditional global 3D point cloud feature database construction. In addition, the embodiment of the present application does not need to align each image to construct a global 3D point cloud database, but only needs to obtain the pose of the image, and extract the global and local features of the image to construct the image pose library. In the online positioning stage of this embodiment of the present application, image retrieval is first performed on the image to be positioned in the database, and then the local structure information of the retrieved image and the image to be positioned is restored, and the relative image of the image to be positioned and the retrieved image are obtained. The relative pose in the coordinate system is finally combined with the pose feature library to realize the transformation from the relative coordinate system to the world coordinate system, so as to realize the solution of the camera pose.

According to the above embodiment, an example of the positioning accuracy of the present application in place A (for example, Nanjing) and place B (for example, Xi'an) is shown in Table 2 below. Among them, the error of position positioning is at the centimeter level, the proportion of angle positioning accuracy within 3° is 48.1% to 67.7%, and the proportion of angle positioning accuracy within 10% is more than 90%.

Table 2

<1° <3° <10° X(m) Y(m) Z(m) A place 10% 48.1% 91.9% 0.17 0.01 0.13 place B 7.5% 67.7% 93.7% -0.01 0.004 -0.019

The reason why the above technical effects can be achieved is that the image pose library (one model) proposed in this application, the image retrieval algorithm based on pose perception, and the camera pose solution algorithm (two algorithms) based on local structure information have solid theoretical foundations. Fundamentals and practical foundations. The theoretical basis is that the embodiments of the present application can search for image datasets evenly distributed in the image database based on the global feature information and location of the image to be located, provide stable input data for SMF to restore local structure information, and ensure as many images as possible. While participating in reconstruction, redundant redundant information is not generated. Combined with the global prior information provided by the image pose library, the transformation relationship (ie the mapping relationship) from the local coordinate system to the global coordinate system can be obtained to achieve accurate pose calculation. The basis of practice is that a terminal (such as a mobile phone), as an easy-to-obtain image acquisition device, combined with an easy-to-operate acquisition method such as video acquisition, can greatly speed up the efficiency of data acquisition and simplify the data acquisition process and labor costs.

FIG. 12 shows a schematic flowchart of a positioning method 1200 provided by an embodiment of the present application. Wherein, the method 1200 may be applied to the terminal or the cloud. As an example, the method 1200 may be performed by the positioned apparatus 100 in FIG. 1 , or by the system 200 in FIG. 2 . As shown in FIG. 12 , the method 1200 includes steps 1210 to 1240 .

1210. Acquire a first image captured by the first terminal.

1220. Determine at least one second image matching the first image in the image database according to the first information of the first image and the first information of the image in the image database, wherein the first image The information is used to indicate the global feature of the image, and the image database includes the first information of the multi-frame image, the second information of the multi-frame image, and the pose of the multi-frame image in the world coordinate system, wherein, The second information is used to indicate local features of the image.

1230. According to the second information of the first image and the second information of the at least one second image, determine whether the first image and the at least one second image are in the first terminal. The pose in the first local coordinate system.

1240. According to the pose of the first image in the first local coordinate system, the pose of the at least one second image in the first local coordinate system, and the pose of the at least one second image in the world The pose in the coordinate system, and output the pose of the first image in the world coordinate system.

Therefore, in this embodiment of the present application, by determining in the image database at least one second image that matches the first image captured by the first terminal, and the position of the first image and the at least one second image in the local coordinate system of the first terminal pose, and according to the pose of the first image in the first local coordinate system, the pose of at least one second image in the first local coordinate system, and the pose of the at least one second image in the world coordinate system , and output the pose of the first image in the world coordinate system, so as to realize the positioning of the first image. Therefore, the embodiment of the present application can obtain the relationship between the local coordinate system and the world coordinate system based on the pose of a part of the image in the local coordinate system and the pose in the world coordinate system, so that the image to be positioned (for example, the first image) is Positioning in the world coordinate system without acquiring a 3D point cloud for each image.

Compared with the 3D point cloud feature library (which includes global 3D point cloud features) of the environment in the existing solution, and the calculation solution of realizing pose based on feature point matching, the embodiment of the present application does not use the 3D point cloud of the image. feature, but uses the pose of the image for localization. Since the construction of the 3D point cloud feature library of the environment needs to rely on professionals and acquisition equipment, which is time-consuming and labor-intensive, and the cost is high, and this application does not need to acquire 3D point cloud features, on the one hand, the application can use low-cost equipment, such as Cameras with low resolution and small field of view (such as mobile phone cameras) can build image databases without relying on professionals and professional equipment, and can reduce the data volume of image databases, thereby reducing the cost of building image databases; On the one hand, the present application can shorten the time for constructing the database, which is conducive to the efficient construction of the image database. Therefore, the positioning solution based on the pose of the image and the image database of the present application can contribute to high-efficiency and low-cost positioning.

In some possible implementations, method 1200 also includes establishing an image database.

In some possible implementations, the video shot by the second terminal may be acquired, and the poses of multiple frames of images in the video in the second local coordinate system of the second terminal may be acquired. Then, the pose of the multi-frame images in the video under the world coordinate system may be determined according to the third information and the pose of the multi-frame images in the video under the second local coordinate system. Wherein, the third information is used to indicate the position of at least part of the images in the video in a map, and the map is associated with the world coordinate system. Afterwards, the first information and the second information of the multi-frame images in the video may be acquired. In this way, the establishment of the image database can be realized.

In some possible implementations, method 1200 may further include acquiring a first position of the first image, and determining at least one image in the image database according to the first position of the first image. Wherein, the distance between the position of each image in the at least one image and the first position of the first image is less than a first threshold, and the first position comes from the GPS module or the WIFI module of the first terminal.

Wherein, as the first information of the first image is matched with the first information of the image in the image database, a specific method of determining at least one second image matching the first image in the image database In an implementation manner, the first information of the first image may be matched with the first information of the at least one image, and at least one second image matching the first image is acquired in the at least one image.

In some possible implementations, a plurality of third images matching the first image may be determined in the image database according to the first information of the first image and the first information of the image in the image database, and then The at least one second image is obtained by deleting an image whose distance from the cluster center of the plurality of third images is greater than a second threshold from among the plurality of third images. Wherein, the cluster center is determined according to the positions of the plurality of third images in the world coordinate system.

Here, an image whose distance from the cluster centers of the plurality of third images is greater than the second threshold may be referred to as an outlier image. That is, the third image may include the outlier image and the aforementioned at least one second image. In some possible descriptions, the third image may be described as being a plurality of second images in the image database that match the first image and have not removed outlier images.

In some possible implementations, a plurality of fourth images matching the first image may be determined in the image database according to the first information of the first image and the first information of the images in the image database, and then the plurality of fourth images may be deleted. In the fourth image, the angle is smaller than the third threshold and/or the distance is smaller than the fourth threshold, so as to obtain the at least one second image. The angle is the difference between the poses of the at least two images in the world coordinate system, and the distance is the difference between the poses of the at least two images in the world coordinate system. As an example, the angle may be a difference in pitch, yaw or roll of poses of at least two images in the world coordinate system.

Here, images whose angle is smaller than the third threshold and/or whose distance is smaller than the fourth threshold may be referred to as redundant images. That is, the fourth image may include the redundant image and the aforementioned at least one second image. In some possible descriptions, the fourth image may be described as being a plurality of second images in the image database that match the first image and have not removed redundant images.

Specifically, for all relevant contents of the steps involved in the positioning method 1200 shown in FIG. 12 , please refer to the relevant functions of the modules in FIG. 1 or FIG. 2 above, or the description of the positioning method 300 shown in FIG. 3 . , and will not be repeated here.

The positioning method provided by the embodiment of the present application has been described in detail above with reference to FIGS. 1 to 12 , and the positioning device according to the embodiment of the present application will be introduced below with reference to FIG. 13 and FIG. 14 . It should be understood that the positioning device in FIGS. 13 and 14 can perform each step in the positioning method in the embodiment of the present application. In order to avoid repetition, repetition is appropriately omitted when introducing the positioning device in FIGS. 13 and 14 below. description of.

FIG. 13 is a schematic block diagram of an apparatus 1300 for positioning according to an embodiment of the present application. The apparatus 1300 includes an acquisition unit 1310 , a processing unit 1320 and an output unit 1330 .

Specifically, when the positioning apparatus 1300 executes the positioning method, the obtaining unit 1310 is configured to obtain the first image captured by the first terminal.

The processing unit 1320 is configured to determine at least one second image matching the first image in the image database according to the first information of the first image and the first information of the image in the image database, wherein, The first information is used to indicate the global characteristics of the image, and the image database includes the first information of the multi-frame images, the second information of the multi-frame images, and the position of the multi-frame images in the world coordinate system. pose, wherein the second information is used to indicate local features of the image.

The processing unit 1320 is further configured to determine, according to the second information of the first image and the second information of the at least one second image, that the first image and the at least one second image are The pose of the first terminal in the first local coordinate system.

The output unit 1330 is configured to, according to the pose of the first image under the first local coordinate system, the pose of the at least one second image under the first local coordinate system, and the at least one first The pose of the second image in the world coordinate system, and the pose of the first image in the world coordinate system is output.

In some possible implementations, the apparatus 1300 further includes an establishment unit for establishing the image database.

In some possible implementations, the establishment unit is specifically used for:

Acquiring the video shot by the second terminal; acquiring the pose of the multi-frame images in the video in the second local coordinate system of the second terminal; according to the third information and the multi-frame images in the video in the The pose in the second local coordinate system, determining the pose of the multi-frame images in the video in the world coordinate system, wherein the third information is used to indicate the position of at least part of the images in the video in the map position, the map is associated with the world coordinate system; and the first information and the second information of the multi-frame images in the video are acquired.

In some possible implementations, the obtaining unit 1310 is further configured to obtain a first position of the first image, where the first position comes from a GPS module or a WiFi module of the first terminal.

Wherein, the acquiring unit may receive data sent by the GPS module or the WiFi module, such as the above-mentioned first position.

Optionally, the obtaining unit may also send a request message to the GPS module or the WiFi module, where the request message is used to request the GPS module or the WiFi module to send the first position of the first image collected by the GPS module or the WiFi module to the obtaining unit. In response to the request message, the GPS module or the WiFi module may send the first location to the acquisition unit.

The processing unit 1320 is further configured to determine at least one image in the image database according to the first position of the first image, and the position of each image in the at least one image is the same as the first position of the first image. A distance between locations is less than a first threshold.

The processing unit 1320 is further configured to match the first information of the first image with the first information of the at least one image, and obtain at least one first image matching the first image in the at least one image. Second image.

In some possible implementations, the processing unit 1320 is specifically used for:

According to the first information of the first image and the first information of the image in the image database, determine a plurality of third images matching the first image in the image database; delete the plurality of third images , the distance from the cluster centers of the plurality of third images is greater than a second threshold to obtain the at least one second image, wherein the cluster centers are based on the plurality of third images in The position in the world coordinate system is determined.

In some possible implementations, the processing unit 1320 is specifically used for:

According to the first information of the first image and the first information of the image in the image database, determine a plurality of fourth images matching the first image in the image database; delete the plurality of fourth images among the images whose angle is smaller than the third threshold and/or the distance is smaller than the fourth threshold to obtain the at least one second image. The angle is the difference between the poses of the at least two images in the world coordinate system, and the distance is the difference between the poses of the at least two images in the world coordinate system.

Specifically, for all relevant content (for example, implementation examples or technical effects) of the units involved in the positioning apparatus 1300 shown in FIG. 13 , reference may be made to the relevant functions of each module in FIG. The relevant description of the positioning method 300 shown in the figure is omitted here.

FIG. 14 is a schematic structural diagram of a positioning apparatus 1400 according to an embodiment of the present application. As shown in FIG. 14 , the device 1400 includes a communication module 1410 , a sensor 1420 , a user input module 1430 , an output module 1440 , a processor 1450 , an audio and video input module 1460 , a memory 1470 and a power supply 1480 .

The communication module 1410 may include at least one module that enables communication between the device 1400 and other devices (eg, other computer systems or mobile terminals). For example, the communication module 1410 may include one or more of a wired network interface, a broadcast receiving module, a mobile communication module, a wireless Internet module, a local area communication module, and a location (or positioning) information module, among others. There are various implementations of these various modules in the prior art, which are not described in this application.

The sensor 1420 may sense the current state of the device 1400, such as position, contact with the user, orientation, acceleration/deceleration, and the like. For example, the sensor 1420 may transmit the sensed current state of the device 1400 to the GPS module or the WiFi module.

The user input module 1430 is used for receiving input digital information, character information or contact touch operation/non-contact gesture, as well as receiving signal input related to user settings and function control of the device. User input module 1430 includes a touch panel and/or other input devices.

The output module 1440 includes a display panel for displaying information input by the user, information provided to the user, various menu interfaces of the system, and the like. Optionally, the display panel may be configured in the form of a liquid crystal display (liquid crystal display, LCD) or an organic light-emitting diode (organic light-emitting diode, OLED). In other embodiments, the touch panel may cover the display panel to form a touch display screen. In addition, the output module 1440 may further include an audio output module, an alarm, a haptic module, and the like.

Exemplarily, the output module 1440 may implement the related functions of the output unit 1330 in the apparatus 1300, for example, it may be used to output the pose of the first image in the world coordinate system. As a specific example, the display panel of the output module 1440 may display the visualization result of the current location of the terminal in the map.

The audio and video input module 1460 is used for inputting audio signals or video signals. The audio and video input module 1460 may include a camera and a microphone. Exemplarily, the camera may be used to capture the first image, which is not limited in this application.

The power supply 1480 may receive external power and internal power under the control of the processor 1450 and provide power required for the operation of various components of the system.

Processor 1450 may refer to one or more processors, for example, processor 1450 may include one or more central processing units, or a central processing unit and a graphics processor, or an application processor and a co-processor (eg a microcontroller unit). When the processor 1450 includes multiple processors, the multiple processors may be integrated on the same chip, or may be independent chips. A processor may include one or more physical cores, where a physical core is the smallest processing module.

Exemplarily, the processor 1450 may implement the functions of the processing unit 1320 in the apparatus 1300 . Optionally, the processor 1450 may also implement the function of the establishment unit in the apparatus 1300, which is not limited in this application.

Exemplarily, the processor 1450 may also be used to implement the function of the obtaining unit 1310 in the apparatus 1300, such as obtaining the first image captured by the terminal from a camera unit (eg, a camera), and/or obtaining the first image from a GPS module or a WiFi module. The first position, etc., is not limited in this application.

的安卓

系统等用于移动终端的系统。前述实施例提供的方法可以通过软件的方式实现，可以认为是应用程序1471和/或操作系统程序1472的具体实现。The memory 1470 stores computer programs including an operating system program 1472, an application program 1471, and the like. Typical operating systems such as Microsoft's Windows, Apple's MacOS, etc. are used for desktop or notebook systems, and Google's developed

android

system etc. for mobile terminals. The methods provided in the foregoing embodiments may be implemented by means of software, which may be considered as the specific implementation of the application program 1471 and/or the operating system program 1472 .

The memory 1470 may be one or more of the following types: flash memory, hard disk type memory, micro multimedia card type memory, card memory (eg SD or XD memory), random access memory (random access memory, RAM), static random access memory (static RAM, SRAM), read only memory (read only memory, ROM), electrically erasable programmable read-only memory (electrically erasable programmable read-only memory, EEPROM), programmable read only memory Memory (programmable ROM, PROM), magnetic memory, magnetic disk or optical disk. In some other embodiments, the memory 1470 can also be a network storage device on the Internet, and the system can perform operations such as updating or reading on the memory 1470 on the Internet.

The processor 1450 is used to read the computer program in the memory 1470, and then execute the method defined by the computer program. For example, the processor 1450 reads the operating system program 1472 to run the operating system and implement various functions of the operating system in the system, or read the operating system program 1472. One or more applications are taken 1471 to run on the system.

The memory 1470 also stores other data 1473 other than computer programs, such as image databases and the like referred to in this application.

The connection relationship of each module in FIG. 14 is only an example, and the method provided by any embodiment of the present application can also be applied to a positioning device in other connection manners, for example, all modules are connected through a bus.

The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented in software, the above-described embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer may be a general purpose computer, special purpose computer, computer network, or other programmable device. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be downloaded from a website site, computer, server, or data center Transmission to another website site, computer, server or data center by wire (eg, infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, a data center, or the like that contains one or more sets of available media. The usable media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVDs), or semiconductor media. The semiconductor medium may be a solid state drive.

Embodiments of the present application further provide a computer-readable medium, on which a computer program is stored, and when the computer program is executed by a computer, implements the steps of the positioning method in any of the foregoing embodiments.

The embodiments of the present application further provide a computer program product, which implements the steps of the positioning method in any of the foregoing embodiments when the computer program product is executed by a computer.

Each embodiment in this application can be used independently or in combination, which is not limited here.

Additionally, various aspects or features of the present application may be implemented as a method, apparatus, or article of manufacture using standard programming and/or engineering techniques. The term "article of manufacture" as used in this application encompasses a computer program accessible from any computer-readable device, carrier or media. For example, computer readable media may include, but are not limited to: magnetic storage devices (eg, hard disks, floppy disks, or magnetic tapes, etc.), optical disks (eg, compact discs (CDs), digital versatile discs (DVDs) etc.), smart cards and flash memory devices (eg, erasable programmable read-only memory (EPROM), cards, stick or key drives, etc.). Additionally, various storage media described herein can represent one or more devices and/or other machine-readable media for storing information. The term "machine-readable medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing, and/or carrying instructions and/or data.

It should be noted that, in the various embodiments provided in this application, there is no time-limited relationship between each step, and each step can be used as a solution, and can also be combined with one or more other steps to form a solution. Not limited.

The various embodiments in this application can be used independently or in combination, for example, any one or more steps in each of the different embodiments can be combined to form an independent embodiment, which is not limited here.

It should be understood that, in the embodiments shown above, the first and the second are only for the convenience of distinguishing different objects, and should not constitute any limitation to the present application.

It should also be understood that, in the embodiments of the present application, the size of the sequence numbers of the above-mentioned processes does not mean the sequence of execution, and the execution sequence of each process should be determined by its functions and internal logic, rather than the implementation of the embodiments of the present application. The process constitutes any qualification.

It should also be understood that "and/or", describing the association relationship of associated objects, means that there can be three kinds of relationships, for example, A and/or B, which can mean that A exists alone, A and B exist at the same time, and B exists alone. a situation. The character "/" generally indicates that the associated objects are an "or" relationship. "At least one" refers to one or more than one; "at least one of A and B", similar to "A and/or B", describes the association relationship of associated objects, indicating that there can be three kinds of relationships, for example, A and B At least one of the three cases can be represented: A exists alone, A and B exist simultaneously, and B exists alone.

Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this application.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described systems, devices and units may refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as independent products, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence, or the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, removable hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk and other media that can store program codes .

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited to this. should be covered within the scope of protection of this application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims (13)

1. A method of positioning, comprising:

acquiring a first image shot by a first terminal;

determining at least one second image matched with the first image in the image database according to the first information of the first image and the first information of the images in the image database, wherein the first information is used for indicating the global features of the images, and the image database comprises the first information of multiframe images, the second information of the multiframe images and the poses of the multiframe images in a world coordinate system, wherein the second information is used for indicating the local features of the images;

determining the poses of the first image and the at least one second image in a first local coordinate system of the first terminal according to the second information of the first image and the second information of the at least one second image;

and outputting the pose of the first image in the world coordinate system according to the pose of the first image in the first local coordinate system, the pose of the at least one second image in the first local coordinate system and the pose of the at least one second image in the world coordinate system.

2. The method of claim 1, further comprising:

and establishing the image database.

3. The method of claim 2, wherein the establishing the image database comprises:

acquiring a video shot by a second terminal;

acquiring the pose of a multi-frame image in the video under a second local coordinate system of the second terminal;

determining the pose of the multi-frame image in the video in a world coordinate system according to third information and the pose of the multi-frame image in the video in the second local coordinate system, wherein the third information is used for indicating the position of at least part of the image in a map, and the map is associated with the world coordinate system;

and acquiring the first information and the second information of a plurality of frames of images in the video.

4. The method according to any one of claims 1-3, further comprising:

acquiring a first position of the first image, wherein the first position is from a Global Positioning System (GPS) module or a wireless fidelity (WiFi) module of the first terminal;

determining at least one image in the image database according to the first position of the first image, wherein the distance between the position of each image in the at least one image and the first position of the first image is less than a first threshold value;

wherein the matching the first information of the first image with the first information of images in an image database, determining at least one second image in the image database that matches the first image, comprises:

and matching the first information of the first image with the first information of the at least one image, and acquiring at least one second image matched with the first image in the at least one image.

5. The method of any of claims 1-4, wherein determining, in the image database, at least one second image that matches the first image based on the first information of the first image and first information of images in an image database comprises:

determining a plurality of third images matched with the first image in an image database according to the first information of the first image and the first information of the images in the image database;

deleting the images, of the plurality of third images, of which the distance from the cluster center of the plurality of third images is greater than a second threshold value, to obtain the at least one second image, wherein the cluster center is determined according to the positions of the plurality of third images in the world coordinate system.

6. The method of any one of claims 1-5, wherein determining, in the image database, at least one second image that matches the first image based on the first information of the first image and first information of images in an image database comprises:

determining a plurality of fourth images matched with the first image in an image database according to the first information of the first image and the first information of the images in the image database;

and deleting images with angles smaller than a third threshold and/or distances smaller than a fourth threshold from the plurality of fourth images to obtain the at least one second image, wherein the angles are the difference of the poses of the at least two images in the world coordinate system, and the distances are the difference of the positions of the poses of the at least two images in the world coordinate system.

7. An apparatus for positioning, comprising:

the device comprises an acquisition unit, a processing unit and a display unit, wherein the acquisition unit is used for acquiring a first image shot by a first terminal;

the processing unit is used for determining at least one second image matched with the first image in an image database according to the first information of the first image and the first information of the images in the image database, wherein the first information is used for indicating the global features of the images, and the image database comprises the first information of multi-frame images, the second information of the multi-frame images and the poses of the multi-frame images in a world coordinate system, wherein the second information is used for indicating the local features of the images;

the processing unit is further configured to determine, according to the second information of the first image and the second information of the at least one second image, poses of the first image and the at least one second image in a first local coordinate system of the first terminal;

an output unit configured to output the pose of the first image in the world coordinate system according to the pose of the first image in the first local coordinate system, the pose of the at least one second image in the first local coordinate system, and the pose of the at least one second image in the world coordinate system.

8. The apparatus according to claim 7, further comprising a building unit configured to build the image database.

9. The apparatus according to claim 8, wherein the establishing unit is specifically configured to:

acquiring a video shot by a second terminal;

acquiring the pose of a multi-frame image in the video under a second local coordinate system of the second terminal;

determining the pose of the multi-frame image in the video in a world coordinate system according to third information and the pose of the multi-frame image in the video in the second local coordinate system, wherein the third information is used for indicating the position of at least part of the image in a map, and the map is associated with the world coordinate system;

and acquiring the first information and the second information of a plurality of frames of images in the video.

10. The apparatus according to any one of claims 7 to 9,

the acquisition unit is further configured to acquire a first position of the first image, where the first position is from a Global Positioning System (GPS) module or a wireless fidelity (WiFi) module of the first terminal;

the processing unit is further configured to determine at least one image in the image database according to the first position of the first image, wherein a distance between the position of each image of the at least one image and the first position of the first image is smaller than a first threshold;

the processing unit is further configured to match the first information of the first image with the first information of the at least one image, and acquire at least one second image matched with the first image in the at least one image.

11. The apparatus according to any one of claims 7 to 10, wherein the processing unit is specifically configured to:

determining a plurality of third images matched with the first image in an image database according to the first information of the first image and the first information of the images in the image database;

deleting the images, of the plurality of third images, of which the distance from the cluster center of the plurality of third images is greater than a second threshold value, to obtain the at least one second image, wherein the cluster center is determined according to the positions of the plurality of third images in the world coordinate system.

12. The apparatus according to any one of claims 7 to 11, wherein the processing unit is specifically configured to:

determining a plurality of fourth images matched with the first image in an image database according to the first information of the first image and the first information of the images in the image database;

and deleting images with an angle smaller than a third threshold and/or a distance smaller than a fourth threshold from the plurality of fourth images to obtain the at least one second image, wherein the angle is a difference value of the poses of the at least two images in the world coordinate system, and the distance is a difference value of the positions of the poses of the at least two images in the world coordinate system.

13. A terminal device, comprising:

one or more processors;

a memory for storing one or more programs;

when executed by the one or more processors, cause the one or more processors to implement the apparatus of any of claims 1-6.

Images