WO2022016803A1

WO2022016803A1 - Visual positioning method and apparatus, electronic device, and computer readable storage medium

Info

Publication number: WO2022016803A1
Application number: PCT/CN2020/139166
Authority: WO
Inventors: 金诚; 冯友计; 章国锋
Original assignee: 浙江商汤科技开发有限公司
Priority date: 2020-07-22
Filing date: 2020-12-24
Publication date: 2022-01-27
Also published as: CN111859003A; TW202205206A; CN111859003B

Abstract

A visual positioning method and apparatus, an electronic device, and a storage medium. The method comprises: extracting a feature vector of a feature point of a query image (S11); searching, according to the feature vector of the feature point of the query image, for a database feature point that matches the feature point of the query image, the database feature point representing a feature point of a database image (S12); and determining a visual positioning result of the query image according to the matched database feature point (S13).

Description

Visual positioning method and apparatus, electronic device and computer-readable storage medium

CROSS-REFERENCE TO RELATED APPLICATIONS

The embodiments of the present disclosure are based on a Chinese patent application with an application number of 202010710996.0 and an application date of July 22, 2020, and claim the priority of the Chinese patent application. The entire content of the Chinese patent application is incorporated herein by reference.

technical field

The embodiments of the present disclosure relate to the technical field of computer vision, and in particular, to a visual positioning method and apparatus, an electronic device, and a computer-readable storage medium.

Background technique

With the continuous development of information technology, positioning technology is increasingly important to people's lives. Traditional positioning technologies mainly include positioning technology based on GPS (Global Position System, global positioning system), positioning technology based on wireless local area network or Bluetooth, and positioning technology based on ultra-wideband. These traditional positioning technologies all have certain limitations. Among them, the GPS signal penetration ability is poor, and it is difficult to achieve effective and accurate positioning in a densely built environment or an indoor environment. In addition, even in an open scene, in order to achieve high-precision positioning, professional GPS equipment with a high cost is required, so it is difficult to achieve consumer-level applications. The positioning technology based on wireless local area network or Bluetooth needs to pre-arrange relevant devices in the to-be-located area, the layout process is cumbersome, the reliability and accuracy are poor, and the positioning range is small. UWB-based positioning technology can achieve relatively high accuracy, but it requires at least three receivers, and the space between the transmitter and receiver needs to be kept open, which limits the application scenarios of UWB-based positioning technology . For large scenarios, the UWB-based positioning technology often needs to multiply the number of receivers, resulting in poor system reliability. In addition, the above-mentioned positioning technology can usually only obtain position information, and it is difficult to obtain reliable attitude information.

In recent years, with the continuous maturity of visual positioning technology, vision-based positioning methods have been used more and more. Compared with the traditional positioning technology, the visual positioning technology not only has a simpler way of obtaining information, but also does not require much modification to the positioning scene. In addition, the visual positioning technology can not only locate the position information, but also locate the attitude information, so that the positioning results can not only serve the needs of conventional position information acquisition, but also realize more intelligent applications, such as augmented reality. However, in the related art, the speed of visual localization is low.

SUMMARY OF THE INVENTION

The embodiments of the present disclosure provide a technical solution for visual positioning.

According to an aspect of the embodiments of the present disclosure, a visual positioning method is provided, including:

Extract the feature vector of the feature points of the query image;

According to the feature vector of the feature points of the query image, search for the database feature points matching the feature points of the query image, wherein the database feature points represent the feature points of the database image;

According to the matched database feature points, the visual positioning result of the query image is determined.

By extracting the feature vectors of the feature points of the query image, searching for the database feature points matching the feature points of the query image according to the feature vectors of the feature points of the query image, and determining the database feature points according to the matched database feature points. The visual positioning result of the query image is described, so that there is no need to retrieve and obtain a local map in the visual positioning, feature point matching is performed directly, and the visual positioning result of the query image is determined according to the database feature points matching the feature points of the query image, so that the positioning process It is more direct and effective, with lower memory consumption, which can reduce the time-consuming of visual positioning, and the positioning process is more reliable.

In a possible implementation manner, the extracting feature vectors of feature points of the query image includes:

Transforming the query image to obtain at least one transformed image corresponding to the query image;

Feature extraction is performed on at least two images of the query image and the at least one transformed image to obtain feature vectors of feature points of the query image.

In this implementation manner, by performing transformation processing on the query image, and then using at least two images in the query image and the at least one transformed image to perform feature extraction, the obtained feature vectors of the feature points of the query image can be It reflects the richer and more comprehensive information in the query image, and the extracted feature vector has strong robustness to environmental changes such as illumination, thereby helping to improve the accuracy of visual positioning.

In a possible implementation manner, performing feature extraction on at least two images in the query image and the at least one transformed image to obtain a feature vector of feature points of the query image, including:

Inputting at least two images of the query image and the at least one transformed image into a first neural network respectively, and outputting feature maps of the at least two images via the first neural network;

Performing grouped convolution on the feature maps of the at least two images to obtain at least two grouped convolution results;

Feature fusion is performed on the at least two grouped convolution results to obtain feature vectors of feature points of the query image.

In this implementation, by performing group convolution on the feature maps of the at least two images, the deep features of the query image can be obtained, thereby improving the robustness of subsequent feature point matching and improving the reliability of visual positioning. sex.

In a possible implementation manner, the searching for a database feature point matching the feature point of the query image according to the feature vector of the feature point of the query image includes:

Decompose the feature vector of the feature points of the query image to obtain a plurality of sub-feature vectors of the feature points of the query image, wherein the dimension of the sub-feature vectors of the feature points of the query image is smaller than the dimension of the query image. The dimension of the feature vector of the feature point;

Finding a database class center that matches a plurality of sub-feature vectors of the feature points of the query image, wherein the database class center represents the class center of the sub-feature vectors of the database feature points;

A first group of database feature points that match the feature points of the query image is determined according to the database class centers that match a plurality of sub-feature vectors of the feature points of the query image.

In this implementation, the feature vector of the feature points of the query image is decomposed into lower-dimensional sub-feature vectors before matching, thereby improving the speed of determining the database feature points matching the feature points of the query image.

In a possible implementation manner, before the searching for a database class center matching a plurality of sub-feature vectors of the feature points of the query image, the method further includes:

Extract feature vectors of multiple database feature points;

For any database feature point among the plurality of database feature points, the feature vector of the any database feature point is decomposed to obtain a plurality of sub-feature vectors of the any database feature point, wherein the any one The dimension of the sub-feature vector of the database feature point is less than the dimension of the feature vector of any of the database feature points;

Clustering the sub-feature vectors of the multiple database feature points to obtain a database class center;

A corresponding relationship between any of the database feature points and the database class center is established.

In this implementation, only the database class center and the corresponding relationship between the database feature points and the database class center need to be stored, and the feature vector (high-dimensional vector) of the database feature point does not need to be stored, thereby saving storage space, saving computing memory, and improving Find speed.

In a possible implementation manner, determining the first group of database feature points matching the feature points of the query image according to the database class centers matched with multiple sub-feature vectors of the feature points of the query image, including :

Determine the candidate database feature points corresponding to the feature points of the query image according to the database class centers matching the multiple sub-feature vectors of the feature points of the query image;

Geometric verification is performed on the candidate database feature points, and a first group of database feature points matching the feature points of the query image is determined.

In this implementation, the candidate database feature points corresponding to the feature points of the query image are determined according to the database class centers that match with multiple sub-feature vectors of the feature points of the query image, and then the candidate database feature points are determined. Geometric verification is performed to determine the first group of database feature points that match the feature points of the query image, so that the first group of database feature points that match the feature points of the query image can be quickly and accurately determined.

In a possible implementation manner, performing geometric verification on the candidate database feature points to determine the first group of database feature points matching the feature points of the query image, including:

determining the similarity transformation matrix between the feature points of the candidate database and the feature points of the corresponding query image;

In a plurality of preset matrix intervals, determine the matrix interval to which the similarity transformation matrix belongs;

Determining a matrix interval where the number of similar transformation matrices in the plurality of matrix intervals satisfies the first quantity condition as a target matrix interval;

According to the candidate database feature points corresponding to the similarity transformation matrix in the target matrix interval, the first group of database feature points matching the feature points of the query image is determined.

In this implementation, geometric verification is performed by means of matrix interval voting, so that the feature points matching the feature points of the query image can be quickly determined, thereby improving the visual positioning speed.

In a possible implementation manner, determining the first group of database feature points matching the feature points of the query image according to the candidate database feature points corresponding to the similarity transformation matrix in the target matrix interval, including :

Determine the database image to which the candidate database feature point belongs, wherein the candidate database feature point represents the candidate database feature point corresponding to the similarity transformation matrix in the target matrix interval;

The first group of database feature points is determined according to candidate database feature points in the database image whose candidate database feature points satisfy the second quantity condition.

According to this implementation, the candidate database feature points can be filtered to obtain the first set of database feature points, thereby determining the visual positioning result of the query image based on the first set of database feature points, which helps to improve the accuracy of the determined visual positioning result. accuracy.

In a possible implementation manner, the searching for a database feature point matching the feature point of the query image according to the feature vector of the feature point of the query image, further includes:

determining the three-dimensional coordinates corresponding to the first group of database feature points;

Determine the second group of database feature points corresponding to the three-dimensional coordinates; the second group of database feature points are database feature points other than the first group of database feature points in the multiple groups of database feature points corresponding to the three-dimensional coordinates;

According to the first set of database feature points and the second set of database feature points, a visual positioning result of the query image is determined.

This implementation can increase the number of associated point pairs through reverse search, thereby improving the robustness of visual positioning.

According to an aspect of the embodiments of the present disclosure, a visual positioning device is provided, comprising:

a first extraction part configured to extract feature vectors of feature points of the query image;

a search section configured to search for a database feature point matching the feature point of the query image according to the feature vector of the feature point of the query image, wherein the database feature point represents the feature point of the database image;

The determining part is configured to determine the visual positioning result of the query image according to the matched database feature points.

In a possible implementation manner, the first extraction part is configured as:

In a possible implementation, the lookup section is configured as:

In a possible implementation, the apparatus further includes:

The second extraction part is configured to extract feature vectors of multiple database feature points;

The decomposition part is configured to decompose the feature vector of any database feature point for any database feature point in the plurality of database feature points to obtain a plurality of sub-feature vectors of the any database feature point, Wherein, the dimension of the sub-feature vector of the feature point of any database is less than the dimension of the feature vector of the feature point of any database;

The clustering part is configured to cluster the sub-feature vectors of the plurality of database feature points to obtain the database class center;

The establishing part is configured to establish the corresponding relationship between the feature point of any database and the database class center.

In a possible implementation, the lookup section is configured as:

According to an aspect of the embodiments of the present disclosure, there is provided an electronic device, comprising: one or more processors; a memory configured to store executable instructions; wherein the one or more processors are configured to invoke the executable instructions stored in the memory to perform the above method.

According to an aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium having computer program instructions stored thereon, the computer program instructions implementing the above method when executed by a processor.

According to an aspect of the embodiments of the present disclosure, a computer program is provided, including computer-readable codes, and when the computer-readable codes are executed in an electronic device, the processor in the electronic device implements the above when executed. method.

In the embodiment of the present disclosure, by extracting the feature vector of the feature points of the query image, searching for the database feature points matching the feature points of the query image according to the feature vector of the feature points of the query image, and according to the matching feature points The database feature points of the query image are determined, and the visual positioning result of the query image is determined, so that there is no need to retrieve and obtain a local map in the visual positioning, and feature point matching is performed directly. Therefore, the positioning process is more direct and effective, the memory consumption is lower, the time-consuming of visual positioning can be reduced, and the positioning process is more reliable.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not limiting of the disclosed embodiments.

Other features and aspects of embodiments of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings.

Description of drawings

The accompanying drawings, which are incorporated into and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, are configured to illustrate the technical solutions of the embodiments of the present disclosure.

FIG. 1 shows a flowchart of a visual positioning method provided by an embodiment of the present disclosure.

FIG. 2 shows an exemplary schematic diagram of performing transformation processing on a query image to obtain multiple transformed images corresponding to the query image provided by an embodiment of the present disclosure.

FIG. 3 is a schematic diagram illustrating an exemplary input of a query image and multiple transformed images into a vanilla CNN, and outputting feature maps of the query image and multiple transformed images via the vanilla CNN provided by an embodiment of the present disclosure.

FIG. 4 shows an exemplary embodiment of the present disclosure that performs grouped convolution on the feature points of a query image and multiple transformed images through two convolutional neural networks, and performs bilinear pooling on the grouped convolution results to obtain Schematic diagram of the eigenvectors of the feature points.

FIG. 5 shows a block diagram of a visual positioning apparatus provided by an embodiment of the present disclosure.

FIG. 6 shows a block diagram of an electronic device 800 provided by an embodiment of the present disclosure.

FIG. 7 shows a block diagram of an electronic device 1900 provided by an embodiment of the present disclosure.

detailed description

Various exemplary embodiments, features and aspects of embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numbers in the figures denote elements that have the same or similar functions. While various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

The term "and/or" in this article is only an association relationship to describe the associated objects, indicating that there can be three kinds of relationships, for example, A and/or B, it can mean that A exists alone, A and B exist at the same time, and A and B exist independently B these three cases. In addition, the term "at least one" herein refers to any combination of any one of the plurality or at least two of the plurality, for example, including at least one of A, B, and C, and may mean including from A, B, and C. Any one or more elements selected from the set of B and C.

In addition, in order to better illustrate the embodiments of the present disclosure, numerous implementation details are given in the following detailed description. It should be understood by those skilled in the art that embodiments of the present disclosure may be practiced without certain details. In some instances, methods, means, components and circuits well known to those skilled in the art have not been described in detail so as not to obscure the subject matter of the embodiments of the present disclosure.

In the related art, in the case of visual positioning, a plurality of database images closest to the query image are searched to obtain a local map, and then feature points of the query image are matched in the local map to obtain the visual positioning result of the query image. The process of this visual positioning method consumes a lot of computing memory, and the visual positioning speed is slow.

In order to solve the technical problem similar to the above, in the embodiment of the present disclosure, by extracting the feature vector of the feature points of the query image, and according to the feature vector of the feature points of the query image, the feature points related to the query image are searched for The matched database feature points, and according to the matched database feature points, the visual positioning result of the query image is determined, so that there is no need to retrieve and obtain a local map in the visual positioning, and feature point matching is performed directly. The point-matched database feature points determine the visual positioning result of the query image, so that the positioning process is more direct and effective, the memory consumption is lower, the time-consuming of visual positioning can be reduced, and the positioning process is more reliable.

FIG. 1 shows a flowchart of a visual positioning method provided by an embodiment of the present disclosure. The execution subject of the visual positioning method may be a visual positioning device. For example, the visual positioning method may be executed by a terminal device or a cloud server or other processing device. The terminal device may be a robot, a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a Personal Digital Assistant (PDA), a handheld device, a computing device, a vehicle-mounted device Or wearable devices, etc. In some possible implementations, the visual positioning method may be implemented by a processor invoking computer-readable instructions stored in a memory. As shown in FIG. 1 , the visual positioning method includes steps S11 to S13.

In step S11, feature vectors of feature points of the query image are extracted.

In the embodiment of the present disclosure, the feature points of the query image may correspond to the pixels of the query image, that is, the position of any feature point of the query image in the query image may be uniquely determined according to the pixel corresponding to the feature point. Thus, the extracted feature vector of the feature points of the query image can provide pixel-level description information. The number of feature points of the query image may be less than or equal to the number of pixels of the query image. In one example, the number of feature points of the query image may be less than the number of pixels of the query image. For example, the value range of the number of feature points of the query image may be 500-2500.

In a possible implementation manner, the extracting the feature vector of the feature points of the query image includes: transforming the query image to obtain at least one transformed image corresponding to the query image; Feature extraction is performed on at least two images in the at least one transformed image to obtain feature vectors of feature points of the query image.

As an example of this implementation manner, the transformation processing may be at least one of rotation, scaling, mirroring, distortion, and the like. FIG. 2 shows a schematic diagram of transforming a query image to obtain a plurality of transformed images corresponding to the query image.

As an example of this implementation, a query image may be transformed to obtain multiple transformed images corresponding to the query image, and feature extraction is performed on the query image and multiple transformed images corresponding to the query image to obtain the query image. The feature vector of the feature points of the query image.

As an example of this implementation, the performing feature extraction on at least two images in the query image and the at least one transformed image to obtain a feature vector of feature points of the query image includes: extracting the query image The image and at least two images in the at least one transformed image are respectively input into the first neural network, and the feature maps of the at least two images are output through the first neural network; the feature maps of the at least two images are Perform group convolution to obtain at least two grouped convolution results; perform feature fusion on the at least two grouped convolution results to obtain feature vectors of feature points of the query image.

In this example, the first neural network may be a convolutional neural network, such as vanilla CNN or the like. FIG. 3 shows a schematic diagram of inputting a query image (not shown in FIG. 3 ) and a plurality of transformed images into a vanilla CNN, and outputting feature maps of the query image and the plurality of transformed images via the vanilla CNN.

According to this example, dense image description features can be obtained, that is, feature vectors that can extract a large number of feature points (eg, 500-2500 feature points). The feature vector of the feature points of the query image extracted by this example may be called a GIFT (Group Invariant Feature Transform, group invariant feature transform) feature. In this example, by performing group convolution on the feature maps of the at least two images, the deep features of the query image can be obtained; while in the related art, the neural network usually only uses the information of a single image to extract features, which will There is a pooling process, which will cause the loss of a certain amount of image information, while GIFT uses multiple transformed images of the image to obtain the deep structure of the feature through grouping convolution, instead of the original pooling process. It is more comprehensive, so that the robustness of subsequent feature point matching can be improved, and the reliability of visual positioning can be improved.

In an example, performing grouped convolution on the feature maps of the at least two images to obtain at least two grouped convolution results includes: dividing the feature maps of the at least two images into a first feature map group and the second feature map group; input the first feature map group into the second neural network, and output the grouped convolution result of the first feature map group through the second neural network; A third neural network is input, and the grouped convolution result of the second feature map group is output via the third neural network.

The first feature map group includes part of the feature maps in the feature maps of the at least two images, and the second feature map group includes another part of the feature maps in the feature maps of the at least two images.

In this example, the second neural network and the third neural network may both be convolutional neural networks. In this example, the grouped convolution results of the first feature map group and the second feature map group are respectively processed by the second neural network and the third neural network, so that the deep structure of the feature can be obtained, so that the obtained feature information It is more comprehensive and the overall matching effect is more robust.

In this example, the number of neural networks used for group convolution may also be more than three.

In an example, the performing feature fusion on the at least two items of grouped convolution results to obtain feature vectors of feature points of the query image includes: performing bilinear pooling on the at least two items of grouped convolution results The transformation operation is performed to obtain the feature vector of the feature points of the query image.

In other examples, the fusion may also be performed in a manner such as concat, which is not limited in this embodiment of the present disclosure.

Figure 4 shows that the feature points of the query image and multiple transformed images are grouped by convolution through two convolutional neural networks, and bilinear pooling is performed on the grouped convolution results to obtain the feature vector of the feature points of the query image. Schematic. As shown in FIG. 4 , an activation function can be used for processing after each convolutional layer of the second neural network and the third neural network. For example, the activation function can be ReLU (Rectified Linear Unit, linear rectification unit).

In step S12, a database feature point matching the feature point of the query image is searched according to the feature vector of the feature point of the query image, wherein the database feature point represents the feature point of the database image.

In a possible implementation manner, the database feature points matching the feature points of the query image may be determined according to the similarity between the feature vectors of the feature points.

In a possible implementation manner, the searching for the database feature points matching the feature points of the query image according to the feature vector of the feature points of the query image includes: The vector is decomposed to obtain a plurality of sub-feature vectors of the feature points of the query image, wherein the dimension of the sub-feature vectors of the feature points of the query image is less than the dimension of the feature vectors of the feature points of the query image; find A database class center matched with a plurality of sub-feature vectors of the feature points of the query image, wherein the database class center represents the class center of the sub-feature vectors of the database feature points; The database class center of the feature vector matching determines the first group of database feature points matching the feature points of the query image.

For example, the feature vector of the feature points of the query image is [s ₁ ,s ₂ ,s ₃ ,s ₄ ,s ₅ ,s ₆ ,s ₇ ,s ₈ ,s ₉ ], and the feature vector of the feature points of the query image is decomposed For 3 sub-eigenvectors, get [s ₁ ,s ₂ ,s ₃ ], [s ₄ ,s ₅ ,s ₆ ] and [s ₇ ,s ₈ ,s ₉ ]. Of course, in practical applications, the dimension of the feature vector of the feature points of the query image can be much higher. The embodiments of the present disclosure do not limit the dimension of the feature vector of the feature points of the query image, the number of sub-feature vectors of the feature points of the query image, and the dimension of the sub-feature vectors of the feature points of the query image. After obtaining the sub-feature vectors [s ₁ , s ₂ , s ₃ ], [s ₄ , s ₅ , s ₆ ] and [s ₇ , s ₈ , s ₉ ], search for the database closest to each sub-feature vector respectively Class center, for example, _{the database class center closest to the sub-feature vector [s 1} ,s ₂ ,s ₃ ] is the database class center 2, and the database class closest to the sub-feature vector [s ₄ ,s ₅ ,s ₆ ] The center is database class center 5, and the database class center closest to the _{sub-feature vector [s 7} ,s ₈ ,s _{9 ] is database class center 8.} According to the correspondence between the database feature points and the database class centers, the database feature points matching the feature points of the query image can be determined.

As an example of this implementation, before the searching for a database class center matching a plurality of sub-feature vectors of the feature points of the query image, the method further includes: extracting feature vectors of a plurality of database feature points; Any database feature point in the plurality of database feature points, decompose the feature vector of the any database feature point, and obtain a plurality of sub-feature vectors of the any database feature point, wherein the any database feature The dimension of the sub-feature vector of the point is less than the dimension of the feature vector of the feature point of any database; the sub-feature vector of the feature points of the multiple databases is clustered to obtain the database class center; the database of any one is established Correspondence between feature points and database class centers.

In this example, a plurality of database images may be included in the database, wherein the database images represent images in the database. For each database image, the feature vector of the database feature points in the database image can be extracted in a manner similar to the manner of extracting the feature vector of the feature points of the query image described above. For example, the value range of the number of database feature points extracted from each database image may be 500-2500.

In this example, for any database feature point, the feature vector of any database feature point may be decomposed into a plurality of sub-feature vectors. For example, if the feature vector of a feature point in a database is [1,3,2,3,4,5,3,2,1], the feature vector of the feature point in the database can be decomposed into 3 sub-feature vectors [1,3 ,2], [3,4,5] and [3,2,1]. Of course, in practical applications, the dimension of the feature vector of the database feature point may be much higher, which is not limited in this embodiment of the present disclosure.

In this example, methods such as K-means, KD tree, or vocabulary tree can be used to cluster sub-feature vectors of multiple database feature points to obtain database class centers.

In this example, after the database class center is obtained, the correspondence between the database feature points and the database class center is recorded.

In one example, the correspondence between the database class center and the database feature points corresponding to all sub-feature vectors in the class to which the database class center belongs may be recorded. For example, if the sub-feature vectors of database feature point 5, database feature point 6, and database feature point 7 belong to the class to which database class center 1 belongs, then database class center 1 and database feature point 5, database feature point 6, and database feature point can be recorded. The correspondence between 7, for example, can be recorded as (1:5,6,7).

In another example, the correspondence between a database feature point and each database class center corresponding to the database feature point may be recorded. For example, database feature point 1 corresponds to database class center 2, database class center 5, and database class center 8, then the correspondence between database feature point 1 and database class center 2, database class center 5, and database class center 8 can be recorded , for example, it can be written as (1:2,5,8).

In this example, it is only necessary to store the database class center and the correspondence between database feature points and database class centers, instead of storing the feature vector (high-dimensional vector) of the database feature point, which can save storage space, save computing memory, and improve search efficiency. speed.

In one example, an indexer can be built from all database class centers.

As an example of this implementation, the determining the first group of database feature points matching the feature points of the query image according to the database class centers matched with multiple sub-feature vectors of the feature points of the query image includes: Determine the candidate database feature points corresponding to the feature points of the query image according to the database class centers matching the multiple sub-feature vectors of the feature points of the query image; perform geometric verification on the candidate database feature points, and determine The first set of database feature points matching the feature points of the query image.

In this example, according to the corresponding relationship between the database feature points and the database class centers, and the database class centers that match with multiple sub-feature vectors of the feature points of the query image, the corresponding feature points of the query image can be determined. Candidate database feature points.

In an example, a Cartesian product method can be used to determine the candidate database feature points corresponding to the feature points of the query image. For example, the feature vector of the feature point A of the query image corresponds to 3 sub-feature vectors A1, A2 and A3, the database class center matching the sub-feature vector A1 is the database class center P1, and the database class center matching the sub-feature vector A2 is The database class center P2, the database class center that matches the sub-feature vector A3 is the database class center P3, then, the database feature points corresponding to all the sub-feature vectors in the class to which the database class centers P1, P2, and P3 belong can be determined as The candidate database feature point corresponding to the feature point A of the query image. For example, database feature points corresponding to all sub-feature vectors in the class to which database class center P1 belongs include database feature points D1, D2, D5, and D6, and database feature points corresponding to all sub-feature vectors in the class to which database class center P2 belongs Including database feature points D1, D7, D8, D9, database feature points corresponding to all sub-feature vectors in the class to which database class center P3 belongs, including database feature points D3, D4, D10, D1-D10 can be determined as all The candidate database feature point corresponding to the feature point A of the query image.

In another example, the feature vector of the feature point A of the query image corresponds to 3 sub-feature vectors A1, A2 and A3, the database class center matching the sub-feature vector A1 is the database class center P1, and the database class center matching the sub-feature vector A2 The database class center is the database class center P2, and the database class center matching the sub-feature vector A3 is the database class center P3, then, the database feature points corresponding to the database class centers P1, P2 and P3 can be determined as the query image The feature point A corresponds to the candidate database feature point. For example, the database feature points corresponding to all sub-feature vectors in the class to which database class center P1 belongs include database feature points D1, D2, D3, D5, and D6, and the database corresponding to all sub-feature vectors in the class to which database class center P2 belongs The feature points include database feature points D1, D2, D3, D7, D8, and D9. The database feature points corresponding to all sub-feature vectors in the class to which the database class center P3 belongs include database feature points D1, D3, D4, and D10. D1 and D3 are respectively determined as candidate database feature points corresponding to the feature point A of the query image.

In this example, the number of candidate database feature points corresponding to each feature point of the query image may be one or more, for example, 25 at most. That is, for any feature point of the query image, multiple database feature points closest to the feature point may be used as candidate database feature points corresponding to the feature point.

In this example, the candidate database feature points corresponding to the feature points of the query image are determined according to the database class centers that match with multiple sub-feature vectors of the feature points of the query image, and then the candidate database feature points are analyzed. The geometric verification is to determine the first group of database feature points that match the feature points of the query image, so that the first group of database feature points that match the feature points of the query image can be quickly and accurately determined.

In one example, performing geometric verification on the candidate database feature points and determining a first group of database feature points matching the feature points of the query image includes: determining the candidate database feature points and the corresponding query image The similarity transformation matrix between the feature points; in a plurality of preset matrix intervals, determine the matrix interval to which the similarity transformation matrix belongs; The matrix interval is determined as the target matrix interval; according to the candidate database feature points corresponding to the similarity transformation matrix in the target matrix interval, the first group of database feature points matching the feature points of the query image is determined.

In this example, the feature point of any candidate database feature point corresponding to the query image represents the feature point of the query image that matches the candidate database feature point. In this example, for any candidate database feature point, a similarity transformation matrix between the candidate database feature point and the corresponding query image feature point can be constructed according to the coordinates, scale and rotation angle of the candidate database feature point. For example, the similarity transformation matrix consists of 4 elements

The preset matrix interval includes the value range of each element in the similarity transformation matrix. For example, the preset matrix interval can be expressed as

According to the value of each element in the similarity transformation matrix and the value range of each element in the preset matrix interval, the matrix interval to which each similarity transformation matrix belongs can be determined, so that the number of similar transformation matrices in each matrix interval can be determined. .

For example, the first quantity condition may be that the number of similar transformation matrices is greater than or equal to the second preset value. The matrix interval is determined as the target matrix interval.

For another example, the first quantity condition may be the M matrix intervals that belong to the largest number of similar transformation matrices, where M is a positive integer. According to the number of similar transformation matrices in each matrix interval, M matrix intervals with the largest number of similar transformation matrices can be determined, and the M matrix intervals with the largest number of similar transformation matrices can be respectively determined as target matrix intervals.

For another example, the first quantity condition may be that the number of similar transformation matrices is greater than or equal to the second preset value, and belongs to the M matrix intervals with the largest number of similar transformation matrices. For example, when the number of similar transformation matrices in a certain matrix interval is greater than or equal to the second preset value, and the matrix interval belongs to the M matrix intervals with the largest number of similar transformation matrices, the matrix interval can be determined as the target matrix interval.

In the above example, geometric verification is performed by means of matrix interval voting, so that the feature points matching the feature points of the query image can be quickly determined, thereby improving the visual positioning speed.

As an example of this implementation, the determining the first group of database feature points matching the feature points of the query image according to the candidate database feature points corresponding to the similarity transformation matrix in the target matrix interval includes: determining a backup Select the database image to which the database feature point belongs, wherein the candidate database feature point represents the candidate database feature point corresponding to the similarity transformation matrix in the target matrix interval; the database image that satisfies the second quantity condition according to the candidate database feature point The candidate database feature points in the first group of database feature points are determined. Accordingly, the candidate database feature points can be filtered to obtain the first set of database feature points, thereby determining the visual positioning result of the query image based on the first set of database feature points, which helps to improve the accuracy of the determined visual positioning result. .

Here, the second quantity condition may be greater than or equal to the first preset value, for example, the first preset value is equal to 12.

For example, if the candidate database feature points in any database image satisfy the second quantity condition, it may be determined that the candidate database feature points in any database image belong to the first group of database feature points.

For another example, when the number of database images whose feature points of the candidate database meet the second quantity condition is greater than N, the database image with the most feature points of the candidate database may be selected from the database images whose feature points of the candidate database satisfy the second quantity condition. The first N database images, and the candidate database feature points in the N database images are determined as belonging to the first group of database feature points. where N is a positive integer, for example, N equals 30. In the case that the number of database images whose feature points of the candidate database meet the second quantity condition is less than or equal to N, the candidate database feature points in all database images whose feature points of the candidate database meet the second quantity condition may be determined as the first group of database feature points.

Wherein, when the number of database images whose feature points of the candidate database meet the second quantity condition is 0, it may be determined that the visual positioning result of the query image is a positioning failure.

As another example of this implementation, methods such as random sampling consensus (RANdom Sample Consensus, RANSAC) can be used for geometric verification, that is, by iterative sampling, the most reasonable basic matrix is estimated for the pixel correspondences in the two pictures. , when the number of interior points that satisfy the corresponding relationship of this matrix reaches a certain value, the geometric verification is passed. This method has high precision and can improve the accuracy of geometric verification.

As an example of this implementation, the searching for a database feature point matching the feature point of the query image according to the feature vector of the feature point of the query image further includes: determining that the first group of database feature points corresponds to three-dimensional coordinates; determine the second group of database feature points corresponding to the three-dimensional coordinates; the second group of database feature points are those other than the first group of database feature points in the multiple groups of database feature points corresponding to the three-dimensional coordinates database feature points; determining the visual positioning result of the query image according to the first group of database feature points and the second group of database feature points.

In this example, according to the correspondence between the database feature points and the three-dimensional coordinates, the three-dimensional coordinates corresponding to the first group of database feature points may be determined. A three-dimensional coordinate can correspond to multiple database feature points. According to the correspondence between the three-dimensional coordinates and the database feature points, all the database feature points corresponding to the three-dimensional coordinates can be determined, and the database feature points corresponding to the three-dimensional coordinates other than the first group of database feature points can be determined. The database feature points of are determined as the second group of database feature points.

According to the above content, in this example, the 2D feature points in the database picture that match the 2D feature points in the query picture are determined, and then the 2D feature points in the query picture are obtained according to the correspondence between the 2D feature points and the 3D feature points of the database picture. The corresponding relationship between points and 3D feature points, compared with the related art, which only uses the matching of 2D feature points to 3D feature points for positioning, the embodiment of the present disclosure can increase the number of associated point pairs through reverse search, Thus, the robustness of visual positioning can be improved. Wherein, the associated point pair represents the database feature point and the corresponding three-dimensional coordinate.

In step S13, the visual positioning result of the query image is determined according to the matched database feature points.

In a possible implementation manner, the visual positioning result of the query image may include pose information corresponding to the query image. The pose information may include one or both of position information and attitude information. The position information may be represented by coordinates, and the attitude information may be represented by angles. In one example, the visual positioning result of the query image may include six degrees of freedom pose information of the query image.

In the embodiment of the present disclosure, after obtaining the matched database feature points, a method such as Perspective-n-Point (PnP) may be used to determine the visual positioning result of the query image. For example, an efficient perspective n-point (Efficient Perspective-n-Point, EPnP), perspective 3-point (Perspective-3-Point, P3P) or direct least squares (Direct Least-Squares, DLS) method can be used to determine the Visual localization results for query images.

In one example, on the basis of solving the PnP, if the number of interior points is less than the third preset value, it may be determined that the visual positioning result of the query image is the positioning failure. For example, the third preset value may be equal to 12. When the number of interior points is greater than or equal to the third preset value, the pose information corresponding to the query image can be obtained. Among them, the interior points can represent the feature points that are correctly matched in the case of solving the pose. For example, the inliers can be determined according to the inlier_mask of the RANSAC algorithm. In an example, the pose information corresponding to the query image can also be optimized by a nonlinear optimizer to obtain the final visual positioning result.

In an application scenario, a query image may be collected by the user equipment, and the user equipment sends a visual positioning request to the cloud server, where the visual positioning request carries the query image; the cloud server uses the visual positioning method provided by the embodiment of the present disclosure to perform processing, to obtain the visual positioning result of the query image, and return the visual positioning result of the query image to the user equipment. Wherein, the user equipment may be a device with a camera function, such as a mobile phone. The visual positioning request may also include camera intrinsic parameter information of the user equipment, for example, may include the focal length and the position of the principal point.

The embodiments of the present disclosure can be applied to various application scenarios such as positioning and navigation systems, high-precision maps, and augmented reality products. For example, the embodiments of the present disclosure can be used to provide visual positioning and navigation services in large indoor scenes such as shopping malls, airports, and museums, and solve the problem that effective positioning cannot be performed in indoor scenes because there is no GPS signal. For another example, in outdoor scenes, high-precision maps can be enhanced to achieve higher-precision positioning combined with GPS signals, and visual positioning services can be provided in places with weak outdoor GPS signals. For another example, since the embodiments of the present disclosure can quickly obtain the six-degree-of-freedom position and attitude information of the user equipment, the embodiments of the present disclosure can be applied to augmented reality applications.

An application scenario of the embodiment of the present disclosure will be described below by taking the visual positioning of a robot in a shopping mall as an example. In this application scenario, multiple photos of shopping malls can be taken first as database images. For any database image, transform the database image to obtain multiple transformed images corresponding to the database image; perform feature extraction on the database image and multiple transformed images corresponding to the database image to obtain the features of the database image The feature vector of the point, that is, the feature vector of the database feature point. For any database feature point, the feature vector of the database feature point can be decomposed to obtain multiple sub-feature vectors of the database feature point. The sub-feature vectors of all database feature points are clustered to obtain multiple database class centers. For any database feature point, establish the corresponding relationship between the database feature point and the database class center. When the robot needs to locate, it can use the currently collected image as a query image. Transform the query image to obtain multiple transformed images corresponding to the query image; perform feature extraction on the query image and the multiple transformed images corresponding to the query image to obtain feature vectors of feature points of the query image. The feature vector of the feature points of the query image is decomposed to obtain multiple sub-feature vectors of the feature points of the query image. Find the database class centers that match the multiple sub-feature vectors of the feature points of the query image, and determine the first set of database features that match the feature points of the query image according to the database class centers that match the multiple sub-feature vectors of the feature points of the query image. point. Three-dimensional coordinates corresponding to the first group of database feature points are determined, and a second group of database feature points corresponding to the three-dimensional coordinates are determined. According to the first group of database feature points and the second group of database feature points, the visual positioning result of the query image is determined, that is, the current position information of the robot in the shopping mall and the current posture information of the robot are determined.

It can be understood that the above method embodiments mentioned in the embodiments of the present disclosure can be combined with each other to form a combined embodiment without violating the principle and logic.

Those skilled in the art can understand that, in the above-mentioned method of the specific embodiment, the writing order of each step does not mean a strict execution order but constitutes any limitation on the implementation process, and the execution order of each step should be based on its function and possible intrinsic Logical OK.

In addition, the embodiments of the present disclosure also provide visual positioning devices, electronic devices, computer-readable storage media, and programs, all of which can be used to implement any of the visual positioning methods provided by the embodiments of the present disclosure, and the corresponding technical solutions and descriptions and refer to the methods. Some of the corresponding records will not be repeated.

FIG. 5 shows a block diagram of a visual positioning apparatus provided by an embodiment of the present disclosure. As shown in FIG. 5 , the visual positioning device includes: a first extraction part 51 configured to extract feature vectors of the feature points of the query image; a search part 52 configured to extract feature vectors of the feature points of the query image according to the feature vectors of the query image , searching for the database feature points matching the feature points of the query image, wherein the database feature points represent the feature points of the database image; the determining part 53 is configured to determine the query based on the matched database feature points The visual localization result of the image.

In a possible implementation manner, the first extraction part 51 is configured to: perform transformation processing on the query image to obtain at least one transformed image corresponding to the query image; Feature extraction is performed on at least two images in the images to obtain feature vectors of feature points of the query image.

In a possible implementation manner, the first extraction part 51 is configured to: input at least two images of the query image and the at least one transformed image into a first neural network respectively, A neural network outputs the feature maps of the at least two images; performs grouped convolution on the feature maps of the at least two images to obtain at least two grouped convolution results; performs feature on the at least two grouped convolution results Fusion to obtain the feature vector of the feature points of the query image.

In a possible implementation manner, the searching part 52 is configured to: decompose the feature vector of the feature points of the query image to obtain multiple sub-feature vectors of the feature points of the query image, wherein the The dimension of the sub-feature vector of the feature point of the query image is less than the dimension of the feature vector of the feature point of the query image; find the database class center that matches the multiple sub-feature vectors of the feature point of the query image, wherein all the The database class center represents the class center of the sub-feature vector of the database feature point; according to the database class center matched with the multiple sub-feature vectors of the feature point of the query image, determine the first group matching the feature point of the query image. Database feature points.

In a possible implementation manner, the apparatus further includes: a second extraction part configured to extract feature vectors of multiple database feature points; a decomposition part configured to extract feature vectors for any of the multiple database feature points; A database feature point, decompose the feature vector of any database feature point to obtain a plurality of sub-feature vectors of any database feature point, wherein the dimension of the sub-feature vector of any database feature point is less than The dimension of the feature vector of the feature points of any database; the clustering part is configured to cluster the sub-feature vectors of the multiple database feature points to obtain the database class center; the establishment part is configured to establish all Describe the correspondence between any database feature point and the database class center.

In a possible implementation manner, the searching part 52 is configured to: determine the candidate database corresponding to the feature points of the query image according to the database class centers matching a plurality of sub-feature vectors of the feature points of the query image Feature points; perform geometric verification on the candidate database feature points, and determine a first group of database feature points matching the feature points of the query image.

In a possible implementation manner, the searching part 52 is configured to: determine the similarity transformation matrix between the feature points of the candidate database and the feature points of the corresponding query image; in a plurality of preset matrix intervals, Determine the matrix interval to which the similarity transformation matrix belongs; determine the matrix interval in which the number of similar transformation matrices in the plurality of matrix intervals satisfies the first quantity condition as the target matrix interval; according to the similarity transformation matrix in the target matrix interval For the corresponding candidate database feature points, the first group of database feature points matching the feature points of the query image is determined.

In a possible implementation manner, the searching part 52 is configured to: determine the database image to which the candidate database feature point belongs, wherein the candidate database feature point indicates that the similarity transformation matrix in the target matrix interval corresponds to The candidate database feature points of the candidate database; the first group of database feature points is determined according to the candidate database feature points in the database images whose candidate database feature points satisfy the second quantity condition.

In a possible implementation manner, the searching part 52 is configured to: determine three-dimensional coordinates corresponding to the first group of database feature points; determine a second group of database feature points corresponding to the three-dimensional coordinates; The group database feature points are database feature points other than the first group of database feature points in the multiple groups of database feature points corresponding to the three-dimensional coordinates; according to the first group of database feature points and the second group of database feature points , and determine the visual positioning result of the query image.

In some embodiments, the functions or included parts of the apparatus provided in the embodiments of the present disclosure may be configured to execute the methods described in the above method embodiments. For implementation, reference may be made to the above method embodiments. For brevity, here No longer.

In the embodiments of the present disclosure and other embodiments, a "part" may be a part of a circuit, a part of a processor, a part of a program or software, etc., of course, a unit, a module or a non-modularity.

Embodiments of the present disclosure further provide a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the foregoing method is implemented. Wherein, the computer-readable storage medium may be a non-volatile computer-readable storage medium, or may be a volatile computer-readable storage medium.

Embodiments of the present disclosure also provide a computer program product, including computer-readable codes. When the computer-readable codes are run on a device, a processor in the device executes a method for implementing the visual positioning method provided by any of the above embodiments. instruction.

Embodiments of the present disclosure further provide another computer program product for storing computer-readable instructions, which, when executed, cause the computer to perform the operations of the visual positioning method provided by any of the foregoing embodiments.

Embodiments of the present disclosure further provide an electronic device, including: one or more processors; a memory for storing executable instructions; wherein the one or more processors are configured to invoke executable instructions stored in the memory instruction to execute the above method.

The electronic device may be provided as a terminal, server or other form of device.

FIG. 6 shows a block diagram of an electronic device 800 provided by an embodiment of the present disclosure. For example, electronic device 800 may be a mobile phone, computer, digital broadcast terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, etc. terminal.

6, electronic device 800 may include one or more of the following components: processing component 802, memory 804, power supply component 806, multimedia component 808, audio component 810, input/output (I/O) interface 812, sensor component 814 , and the communication component 816 .

The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 can include one or more processors 820 to execute instructions to perform all or some of the steps of the methods described above. Additionally, processing component 802 may include one or more modules that facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

Memory 804 is configured to store various types of data to support operation at electronic device 800 . Examples of such data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, videos, and the like. Memory 804 may be implemented by any type of volatile or nonvolatile storage device or combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), Magnetic Memory, Flash Memory, Magnetic or Optical Disk.

Power supply assembly 806 provides power to various components of electronic device 800 . Power supply components 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800 .

Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, swipe, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or swipe action, but also detect the duration and pressure associated with the touch or swipe action. In some embodiments, the multimedia component 808 includes a front-facing camera and/or a rear-facing camera. When the electronic device 800 is in an operation mode, such as a shooting mode or a video mode, the front camera and/or the rear camera may receive external multimedia data. Each of the front and rear cameras can be a fixed optical lens system or have focal length and optical zoom capability.

Audio component 810 is configured to output and/or input audio signals. For example, audio component 810 includes a microphone (MIC) that is configured to receive external audio signals when electronic device 800 is in operating modes, such as call mode, recording mode, and voice recognition mode. The received audio signal may be further stored in memory 804 or transmitted via communication component 816 . In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

The I/O interface 812 provides an interface between the processing component 802 and a peripheral interface module, which may be a keyboard, a click wheel, a button, or the like. These buttons may include, but are not limited to: home button, volume buttons, start button, and lock button.

Sensor assembly 814 includes one or more sensors for providing status assessment of various aspects of electronic device 800 . For example, the sensor assembly 814 can detect the on/off state of the electronic device 800, the relative positioning of the components, such as the display and the keypad of the electronic device 800, the sensor assembly 814 can also detect the electronic device 800 or one of the electronic device 800 Changes in the position of components, presence or absence of user contact with the electronic device 800 , orientation or acceleration/deceleration of the electronic device 800 and changes in the temperature of the electronic device 800 . Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. The electronic device 800 may access wireless networks based on communication standards, such as Wi-Fi, 2G, 3G, 4G/LTE, 5G, or a combination thereof. In one exemplary embodiment, the communication component 816 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

In an exemplary embodiment, electronic device 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable A programmed gate array (FPGA), controller, microcontroller, microprocessor or other electronic component implementation is used to perform the above method.

In an exemplary embodiment, a non-volatile computer-readable storage medium, such as a memory 804 comprising computer program instructions executable by the processor 820 of the electronic device 800 to perform the above method is also provided.

FIG. 7 shows a block diagram of an electronic device 1900 provided by an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server. 7, the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource, represented by memory 1932, for storing instructions, such as applications, executable by the processing component 1922. An application program stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Additionally, the processing component 1922 is configured to execute instructions to perform the above-described methods.

The electronic device 1900 may also include a power supply assembly 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input output (I/O) interface 1958 . Electronic device 1900 may operate based on an operating system stored in memory 1932, such as Windows

or similar.

In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as memory 1932 comprising computer program instructions executable by processing component 1922 of electronic device 1900 to perform the above-described method.

In an exemplary embodiment, there is provided a computer program comprising computer readable code which, when executed in an electronic device, implements the above method when executed by a processor in the electronic device.

Embodiments of the present disclosure may be systems, methods and/or computer program products. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the present disclosure.

A computer-readable storage medium may be a tangible device that can hold and store instructions for use by the instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. Examples (a non-exhaustive list) of computer readable storage media may include: portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory) ), static random access memory (SRAM), portable compact disc read only memory (CD-ROM), digital versatile disc (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards with instructions stored thereon Or the protruding structure in the groove, and any suitable combination of the above. Computer-readable storage media, as used herein, are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (eg, light pulses through fiber optic cables), or through electrical wires transmitted electrical signals.

The computer readable program instructions described herein may be downloaded to various computing/processing devices from a computer readable storage medium, or to an external computer or external storage device over a network such as the Internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer-readable program instructions from a network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .

The computer program instructions for carrying out the operations of the disclosed embodiments may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or programmed in one or more Source or object code written in any combination of languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as the "C" language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (eg, using an Internet service provider through the Internet connect). In some embodiments, custom electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), can be personalized by utilizing state information of computer readable program instructions. Computer readable program instructions are executed to implement various aspects of the embodiments of the present disclosure.

Aspects of embodiments of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer or other programmable data processing apparatus to produce a machine that causes the instructions when executed by the processor of the computer or other programmable data processing apparatus , resulting in means for implementing the functions/acts specified in one or more blocks of the flowchart and/or block diagrams. These computer readable program instructions can also be stored in a computer readable storage medium, these instructions cause a computer, programmable data processing apparatus and/or other equipment to operate in a specific manner, so that the computer readable medium on which the instructions are stored includes An article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks of the flowchart and/or block diagrams.

Computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other equipment to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other equipment to produce a computer-implemented process , thereby causing instructions executing on a computer, other programmable data processing apparatus, or other device to implement the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more functions for implementing the specified logical function(s) executable instructions. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or actions , or can be implemented in a combination of dedicated hardware and computer instructions.

The computer program product can be implemented in hardware, software or a combination thereof. In some embodiments of the present disclosure, the computer program product is embodied as a computer storage medium, and in other embodiments of the present disclosure, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK), etc. Wait.

Various embodiments of the present disclosure have been described above, and the foregoing descriptions are exemplary, not exhaustive, and not limiting of the disclosed embodiments. Numerous modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the various embodiments, the practical application or improvement over the technology in the marketplace, or to enable others of ordinary skill in the art to understand the various embodiments disclosed herein.

Industrial Applicability

Embodiments of the present disclosure relate to a visual positioning method and apparatus, an electronic device, and a computer-readable storage medium. The method includes: extracting a feature vector of a feature point of a query image; searching for a database feature point matching the feature point of the query image according to the feature vector of the feature point of the query image, wherein the database feature point represents feature points of the database image; determine the visual positioning result of the query image according to the matched database feature points. By extracting the feature vectors of the feature points of the query image, searching for the database feature points matching the feature points of the query image according to the feature vectors of the feature points of the query image, and determining the database feature points according to the matched database feature points. The visual positioning result of the query image is described, so there is no need to retrieve and obtain a local map in the visual positioning, and feature point matching is performed directly, and the visual positioning result of the query image is determined according to the database feature points matching the feature points of the query image, so that the positioning process It is more direct and effective, with lower memory consumption, which can reduce the time-consuming of visual positioning, and the positioning process is more reliable.

Claims

A visual positioning method comprising:

Extract the feature vector of the feature points of the query image;

According to the feature vector of the feature points of the query image, search for the database feature points matching the feature points of the query image, wherein the database feature points represent the feature points of the database image;

According to the matched database feature points, the visual positioning result of the query image is determined.
The method according to claim 1, wherein the extracting the feature vector of the feature points of the query image comprises:

Transforming the query image to obtain at least one transformed image corresponding to the query image;

Feature extraction is performed on at least two images of the query image and the at least one transformed image to obtain feature vectors of feature points of the query image.
The method according to claim 2, wherein, performing feature extraction on at least two images in the query image and the at least one transformed image to obtain feature vectors of feature points of the query image, comprising:

Inputting at least two images of the query image and the at least one transformed image into a first neural network respectively, and outputting feature maps of the at least two images via the first neural network;

Performing grouped convolution on the feature maps of the at least two images to obtain at least two grouped convolution results;

Feature fusion is performed on the at least two grouped convolution results to obtain feature vectors of feature points of the query image.
The method according to any one of claims 1 to 3, wherein, according to the feature vector of the feature points of the query image, searching for the database feature points matching the feature points of the query image comprises:

Decompose the feature vector of the feature points of the query image to obtain a plurality of sub-feature vectors of the feature points of the query image, wherein the dimension of the sub-feature vectors of the feature points of the query image is smaller than the dimension of the query image. The dimension of the feature vector of the feature point;

Finding a database class center that matches a plurality of sub-feature vectors of the feature points of the query image, wherein the database class center represents the class center of the sub-feature vectors of the database feature points;

A first group of database feature points matching the feature points of the query image is determined according to the database class centers matching the plurality of sub-feature vectors of the feature points of the query image.
The method according to claim 4, wherein before the searching for a database class center matching a plurality of sub-feature vectors of the feature points of the query image, the method further comprises:

Extract feature vectors of multiple database feature points;

For any database feature point among the plurality of database feature points, the feature vector of the any database feature point is decomposed to obtain a plurality of sub-feature vectors of the any database feature point, wherein the any one The dimension of the sub-feature vector of the database feature point is less than the dimension of the feature vector of any of the database feature points;

Clustering the sub-feature vectors of the multiple database feature points to obtain a database class center;

A corresponding relationship between any of the database feature points and the database class center is established.
The method according to claim 4 or 5, wherein the first group of databases matching the feature points of the query image is determined according to the database class centers matching a plurality of sub-feature vectors of the feature points of the query image Feature points, including:

Determine the candidate database feature points corresponding to the feature points of the query image according to the database class centers matching the multiple sub-feature vectors of the feature points of the query image;

Geometric verification is performed on the candidate database feature points, and a first group of database feature points matching the feature points of the query image is determined.
The method according to claim 6, wherein, performing geometric verification on the candidate database feature points to determine the first group of database feature points matching the feature points of the query image, comprising:

determining the similarity transformation matrix between the feature points of the candidate database and the feature points of the corresponding query image;

In a plurality of preset matrix intervals, determine the matrix interval to which the similarity transformation matrix belongs;

Determining a matrix interval where the number of similar transformation matrices in the plurality of matrix intervals satisfies the first quantity condition as a target matrix interval;

According to the candidate database feature points corresponding to the similarity transformation matrix in the target matrix interval, the first group of database feature points matching the feature points of the query image is determined.
The method according to claim 7, wherein the first group of database feature points matching the feature points of the query image is determined according to the candidate database feature points corresponding to the similarity transformation matrix in the target matrix interval ,include:

Determine the database image to which the candidate database feature point belongs, wherein the candidate database feature point represents the candidate database feature point corresponding to the similarity transformation matrix in the target matrix interval;

The first group of database feature points is determined according to candidate database feature points in the database image whose candidate database feature points satisfy the second quantity condition.
The method according to any one of claims 4 to 8, wherein, according to the feature vector of the feature points of the query image, searching for the database feature points matching the feature points of the query image, further comprising:

determining the three-dimensional coordinates corresponding to the first group of database feature points;

Determine the second group of database feature points corresponding to the three-dimensional coordinates; the second group of database feature points are database feature points other than the first group of database feature points in the multiple groups of database feature points corresponding to the three-dimensional coordinates;

According to the first set of database feature points and the second set of database feature points, a visual positioning result of the query image is determined.
A visual positioning device, comprising:

a first extraction part configured to extract feature vectors of feature points of the query image;

a search section configured to search for a database feature point matching the feature point of the query image according to the feature vector of the feature point of the query image, wherein the database feature point represents the feature point of the database image;

The determining part is configured to determine the visual positioning result of the query image according to the matched database feature points.
An electronic device comprising:

one or more processors;

memory configured to store executable instructions;

wherein the one or more processors are configured to invoke executable instructions stored in the memory to perform the method of any one of claims 1-9.
A computer-readable storage medium having computer program instructions stored thereon, the computer program instructions implementing the method of any one of claims 1 to 9 when executed by a processor.
A computer program, comprising computer-readable codes, when the computer-readable codes are executed in an electronic device, a processor in the electronic device implements the method described in any one of claims 1 to 9 when executed. method.