TW202217662A - Visual positioning method, training method of related models, electronic device and computer-readable storage medium - Google Patents

Abstract

The present application discloses a visual positioning method, a training method of related models, an electronic device and a computer-readable storage medium, the training method of matching prediction model includes: using sample image and map data to construct sample matching data, in which the sample matching data includes several groups of point pairs and the actual matching value of each group of point pairs, and two points of each group of point pairs are from the sample image and map data respectively; the matching prediction model is used to predict several groups of point pairs, and the prediction matching value of the point pairs is obtained; the loss value of the matching prediction model is determined by using the actual matching value and the prediction matching value; and the parameters of the matching prediction model are adjusted by using the loss value. The above scheme can improve the accuracy and instantaneity of visual positioning.

Description

Visual positioning method and related model training method, electronic device and computer-readable storage medium

The present invention relates to the technical field of computer vision, in particular to a visual positioning method and a training method of a related model, an electronic device and a computer-readable storage medium.

Visual positioning can be divided into various ways according to the expression of map data. Among them, the structure-based method, also known as the feature-based method, has received extensive attention due to its high accuracy and excellent generalization performance.

At present, when using a feature-based method for visual positioning, it is necessary to obtain multiple point pairs between image data and map data by matching. However, the reliability of using local similarity to establish a matching relationship is weak, especially in large-scale scenes or scenes with repeated structures/textures, which are prone to false matching, thus affecting the accuracy of visual localization. Although the use of random sampling consistency (Random Sample Consensus, RANSAC) can eliminate false matching, but because RANSAC performs equal probability sampling on each sample point, when there are too many outliers in the initial matching, RANSAC has a time-consuming and low-accuracy problem. problems, thereby affecting the immediacy and accuracy of visual positioning. In view of this, how to improve the accuracy and immediacy of visual positioning has become an urgent problem to be solved.

The present invention provides a visual positioning method and a training method of a related model, an electronic device and a computer-readable storage medium.

A first aspect of the present invention provides a training method for a matching prediction model, including: using sample images and map data to construct sample matching data, wherein the sample matching data includes several groups of point pairs and the actual matching value of each group of point pairs, The two points of each group of point pairs come from sample images and map data respectively; use the matching prediction model to predict and process several groups of point pairs to obtain the predicted matching value of the point pair; use the actual matching value and the predicted matching value to determine the matching prediction The loss value of the model; using the loss value, adjust the parameters of the matching prediction model.

Therefore, the sample matching data is obtained by constructing the sample image and map data, and the sample matching data includes several groups of point pairs and the actual matching value of each group of point pairs. The two points of each group of point pairs come from the sample image and the map respectively. Then use the matching prediction model to perform prediction processing on several groups of point pairs to obtain the predicted matching value of the point pair, and then use the actual matching value and the predicted matching value to determine the loss value of the matching prediction model, and use the loss value to match the prediction model. Therefore, the matching prediction model can be used to establish a matching relationship, so that the matching prediction model can be used to predict the matching value between point pairs in visual positioning, so the point pair with high matching value can be preferentially sampled based on the predicted matching value. , to determine the photographing pose parameters of the image to be positioned, which can help improve the accuracy and immediacy of visual positioning.

Wherein, using sample images and map data to construct sample matching data includes: obtaining several image points from the sample image, and obtaining several map points from the map data to form several sets of point pairs; wherein, several sets of point pairs It includes at least one set of matching point pairs matched between the included image points and map points; for each set of matching point pairs: using the pose parameters of the sample image to project the map points into the dimension to which the sample image belongs, to obtain The projected point of the map point; and based on the difference between the image point and the projected point, the actual matching value of the matched point pair is determined.

Therefore, by obtaining a number of image points from the sample image and obtaining a number of map points from the map data, several sets of point pairs are formed, and the several sets of point pairs include at least one set of the included image points and map points. The matching point pairs that match between the two, so it is possible to generate samples for training the matching prediction model, and for each set of matching point pairs, use the pose parameters of the sample image to project the map points to the dimension of the sample image to get The projection point of the map point, so as to determine the actual matching value of the matching point pair based on the difference between the image point and the projection point, so the matching prediction model can learn the geometric features of the matching point pair during the training process, which is conducive to improving the matching Predictive model accuracy.

The several sets of point pairs include at least one set of non-matching points that do not match between the included image points and map points, and using sample images and map data to construct sample matching data also includes: Match value is set to preset value

Therefore, several groups of point pairs include at least one group of non-matching point pairs that do not match between the included image points and map points, and different from the matching point pairs, the actual matching value of the non-matching point pairs is set to a preset value, Thus, the robustness of the matching prediction model can be improved.

Wherein, acquiring several image points from the sample image and acquiring several map points from the map data to form several sets of point pairs, including: dividing the image points in the sample image into the first image point and the first image point Two image points, wherein the first image point has a matching map point in the map data, and the second image point does not have a matching map point in the map data; for each first image point, from A number of first map points are allocated in the map data, and the first image point and each first map point are respectively used as a first point pair, wherein the first map point includes a map point matching the first image point ; and, for each second image point, assign a number of second map points from the map data, and respectively use the second image point and each second map point as a second point pair; from the first point pair And the second point pair is extracted to obtain several groups of point pairs.

Therefore, by dividing the image points in the sample image into a first image point and a second image point, and the first image point has a matching map point in the map, the second image point is in the image There is no matching image point in the data, and for the first image point, a number of first map points are allocated from the map data, and the first image point and each first map point are regarded as a first point pair. , and the first map point includes map points matching the first image point, and for each second image point, a number of second map points are allocated from the map data, and the second image point and each The second map point is used as a second point pair, and several groups of point pairs are extracted from the first point pair and the second point pair, so that several groups of points that are abundant and include non-matching point pairs and matching point pairs can be constructed. Yes, it is used for training the matching prediction model, so it can help to improve the accuracy of the matching prediction model.

Wherein, using the pose parameter of the sample image to project the map point into the dimension to which the sample image belongs, and obtaining the projected point of the map point includes: calculating the pose parameter of the sample image based on the matching point pair; using the pose parameter to The map point is projected into the dimension to which the sample image belongs, and the projected point of the map point is obtained.

Therefore, by using the matching point pairs to calculate the pose parameters of the sample image, and using the pose parameters to project the map points to the dimension of the sample image, the projected points of the map points can be obtained, which can help to improve the relationship between the projected points and the image. The accuracy of the difference between image points can be beneficial to improve the accuracy of the matching prediction model.

Wherein, determining the actual matching value of the matching point pair based on the difference between the image point and the projection point includes: using a preset probability distribution function to convert the difference into a probability density value as the actual matching value of the matching point pair.

Therefore, by using the preset probability distribution function to convert the difference into a probability density value as the actual matching value of the matching point pair, it is beneficial to accurately describe the difference between the projection point and the image point, which can help to improve the matching Predictive model accuracy.

Among them, the sample matching data is a bipartite graph, and the bipartite graph includes several groups of point pairs and connecting edges connecting each group of point pairs, and the connecting edges are marked with the actual matching values of the corresponding point pairs; the matching prediction model includes the dimension to which the sample image belongs. The corresponding first point feature extraction sub-model, the second point feature extraction sub-model corresponding to the dimension to which the map data belongs, and the edge feature extraction sub-model; use the matching prediction model to perform prediction processing on several groups of point pairs, and obtain the prediction of point pairs The matching value includes: using the first point feature extraction sub-model and the second point feature extraction sub-model to perform feature extraction on the bipartite graph to obtain the first feature and the second feature; using the edge feature extraction sub-model to extract the first feature and the second feature. Feature extraction is performed to obtain a third feature; using the third feature, the predicted matching value of the point pair corresponding to the connecting edge is obtained.

Therefore, by performing point feature extraction and edge feature extraction respectively on the bipartite graph, the matching prediction model can more effectively perceive the spatial geometric structure of the matching, which can help to improve the accuracy of the matching prediction model.

Wherein, the structure of the first point feature extraction sub-model and the second point feature extraction sub-model is any of the following: including at least one residual block, including at least one residual block and at least one spatial transformation network; and/or, edge The feature extraction submodel includes at least one residual block.

Therefore, by setting the structure of the first point feature extraction sub-model and the second point feature extraction sub-model to any one of the following: including at least one residual block, including at least one residual block and at least one spatial transformation network, and The edge feature extraction sub-model is set to include at least one residual block, so it can facilitate the optimization of the matching prediction model and improve the accuracy of the matching prediction model.

Wherein, several sets of point pairs include at least one set of matching point pairs that match between the included image points and map points and at least one set of non-matching point pairs that do not match between the included image points and map points; using The actual matching value and the predicted matching value, and determining the loss value of the matching prediction model includes: using the predicted matching value and the actual matching value of the matching point pair to determine the first loss value of the matching prediction model; and using the predicted matching value of the non-matching point pair. and the actual matching value to determine the second loss value of the matching prediction model; the first loss value and the second loss value are weighted to obtain the loss value of the matching prediction model.

Therefore, the first loss value of the matching prediction model is determined by using the predicted matching value and the actual matching value of the matching point pair, and the second loss value of the matching prediction model is determined using the predicted matching value and the actual loss value of the non-matching point pair. , so that the first loss value and the second loss value are weighted to obtain the loss value of the matching prediction model, which can help the matching prediction model to effectively perceive the spatial geometric structure of the matching, thereby improving the accuracy of the matching prediction model.

Wherein, before determining the first loss value of the matching prediction model by using the predicted matching value and the actual matching value of the matching point pair, the method further includes: separately counting the first number of matching point pairs and the second number of non-matching point pairs; Using the predicted matching value and the actual matching value of the matching point pair to determine the first loss value of the matching prediction model includes: using the difference between the predicted matching value and the actual matching value of the matching point pair, and the first number, determining the first loss value of the matching prediction model. Loss value; using the predicted matching value and the actual matching value of the unmatched point pair to determine the second loss value of the matching prediction model includes: using the difference between the predicted matching value and the actual matching value of the unmatched point pair, and the second loss value of the matching prediction model. quantity to determine the second loss value.

Therefore, by counting the first number of matching point pairs and the second number of non-matching point pairs, the first loss is determined by using the difference between the predicted matching value and the actual matching value of the matching point pair, and the first number value, and use the difference between the predicted matching value and the actual matching value of the non-matching point pair, and the second quantity, to determine the second loss value, which can help to improve the accuracy of the loss value of the matching prediction model, which can be beneficial to Improve the accuracy of matching prediction models.

The dimension to which the sample image belongs is 2D or 3D, and the dimension to which the map data belongs is 2D or 3D.

Therefore, by setting the dimensions to which the sample images and map data belong, it is possible to train a matching prediction model for 2D-2D, or a matching prediction model for 2D-3D, or a matching prediction model for 2D-3D. It can be used in 3-3-dimensional matching prediction model, so that the applicable scope of the matching prediction model can be improved.

A second aspect of the present invention provides a visual positioning method, comprising: constructing matching data to be identified by using images to be positioned and map data, wherein the matching data to be identified includes several sets of point pairs, and two points of each set of point pairs are respectively From the image and map data to be positioned; use the matching prediction model to perform prediction processing on several groups of point pairs to obtain the predicted matching value of the point pair; based on the predicted matching value of the point pair, determine the pose parameters of the photographic device of the image to be positioned .

Therefore, by using the to-be-located image and map data, the to-be-identified matching data is constructed, and the to-be-identified matching data includes several sets of point pairs, and the two points of each set of point pairs are respectively from the to-be-located image and the map data, so as to utilize the matching data to be identified. The prediction model performs prediction processing on several groups of point pairs, so as to obtain the predicted matching value of the point pair, and then determines the pose parameters of the photographic device of the image to be positioned based on the predicted matching value of the point pair, which improves the accuracy of visual positioning. Immediacy.

Wherein, determining the pose parameters of the photographic device of the image to be positioned based on the predicted matching values of the point pairs includes: sorting several groups of point pairs in descending order of the predicted matching values; using the previously preset number of groups of points Yes, determine the pose parameters of the imaging device of the image to be positioned.

Therefore, by sorting several groups of point pairs in descending order of the predicted matching values, and using the previously preset number of point pairs to determine the pose parameters of the photographic device of the image to be positioned, it is beneficial to use the sorting The latter point pairs are incrementally sampled, and the point pairs with high matching values are preferentially sampled, so the geometric prior can guide the solution of the pose parameters, thereby improving the accuracy and immediacy of visual positioning.

Wherein, the matching prediction model is obtained by using the training method of the matching prediction model in the first aspect.

Therefore, performing visual positioning with the matching prediction model obtained by the training method for the matching prediction model can improve the accuracy and immediacy of the visual positioning.

A third aspect of the present invention provides a training device for matching prediction models, including: a sample construction module, a prediction processing module, a loss determination module and a parameter adjustment module, and the sample construction module is used for using sample images and map data. , construct the sample matching data, wherein the sample matching data includes several groups of point pairs and the actual matching value of each group of point pairs, and the two points of each group of point pairs come from the sample image and map data respectively; the prediction processing module is used to use The matching prediction model performs prediction processing on several groups of point pairs to obtain the predicted matching value of the point pair; the loss determination module is used to use the actual matching value and the predicted matching value to determine the loss value of the matching prediction model; the parameter adjustment module is used to use The loss value, which adjusts the parameters to match the prediction model.

A fourth aspect of the present invention provides a visual positioning device, comprising: a data construction module, a prediction processing module and a parameter determination module, the data construction module is used for using the image to be positioned and the map data to construct the matching data to be identified, Among them, the matching data to be identified includes several groups of point pairs, and the two points of each group of point pairs are respectively from the image to be located and the map data; the prediction processing module is used to use the matching prediction model to perform prediction processing on several groups of point pairs, and obtain The predicted matching value of the point pair; the parameter determination module is used to determine the pose parameters of the photographic device of the image to be positioned based on the predicted matching value of the point pair.

A fifth aspect of the present invention provides an electronic device, comprising a memory and a processor coupled to each other, the processor is configured to execute program instructions stored in the memory, so as to implement the training method for a matching prediction model in the first aspect, or The visual positioning method in the second aspect above is implemented.

A sixth aspect of the present invention provides a computer-readable storage medium on which program instructions are stored. When the program instructions are executed by a processor, the training method of the matching prediction model in the first aspect above is implemented, or the training method in the second aspect above is implemented. Visual positioning method.

A seventh aspect of the present invention provides a computer program, including computer-readable codes, in the case where the computer-readable codes are executed in an electronic device and executed by a processor in the electronic device, the above-mentioned first aspect is realized. The training method of the matching prediction model, or implement the visual positioning method in the second aspect above.

The above solution can use the matching prediction model to establish a matching relationship, so that the matching prediction model can be used to predict the matching value between point pairs in visual positioning, so the point pair with high matching value can be preferentially sampled based on the predicted matching value, and establish. The matching relationship can be beneficial to improve the accuracy and immediacy of visual positioning.

The solutions of the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

In the following description, for purposes of illustration and not limitation, specific details such as specific system structures, interfaces, techniques, etc. are set forth in order to provide a thorough understanding of the present invention.

The terms "system" and "network" are often used interchangeably herein. The term "and/or" in this article is only a relationship to describe related objects, which means that there can be three relationships, for example, A and/or B, which can mean that A exists alone, A and B exist at the same time, and B exists alone. three situations. In addition, the character "/" in this text generally indicates that the contextually related objects are in an "or" relationship. Also, "multiple" herein means two or more than two.

Please refer to FIG. 1. FIG. 1 is a schematic flowchart of an embodiment of a training method for a matching prediction model of the present invention. The training method of the matching prediction model may include the following steps.

Step S11: Using the sample image and map data, construct sample matching data.

In the embodiment of the present invention, the sample matching data includes several groups of point pairs and actual matching values of each group of point pairs, and the two points of each group of point pairs come from sample images and map data respectively.

In one implementation scenario, the map data may be constructed from sample images. The dimension to which the sample image belongs may be 2-dimensional or 3-dimensional, and the dimension to which the map data belongs may be 2-dimensional or 3-dimensional, which is not limited herein. For example, if the sample image is a two-dimensional image, the two-dimensional image can be processed by three-dimensional reconstruction methods such as SFM (Structure From Motion) to obtain map data such as a sparse point cloud model. Including three-dimensional information, for example, the sample image may also be an RGB-D image (ie, a color image and a depth image), which is not limited here. The map data can be composed of a simple two-dimensional image, or a three-dimensional point cloud map, or a combination of a two-dimensional image and a three-dimensional point cloud, which is not limited here.

In this embodiment of the present invention, the execution body of the training method for matching prediction models may be a training device for matching prediction models, which is described as a training device hereinafter; for example, the training method for matching prediction models may be processed by a terminal device or a server or other processing methods Device execution, where the terminal device may be User Equipment (UE), mobile device, user terminal, terminal, cellular phone, wireless phone, Personal Digital Assistant (PDA), handheld device, computing device , vehicle equipment, wearable devices, etc. In some possible implementations, the training method of the matching prediction model may be implemented by the processor calling computer-readable instructions stored in the memory.

In an implementation scenario, the sample matching data can be a bipartite graph, which is also called a bipartite graph, which is an undirected graph composed of a point set and an edge set, and the point set can be divided into two mutually disjoint sub-graphs. The two points associated with each edge in the edge set belong to the two disjoint subsets. Among them, when the sample matching data is a bipartite graph, it includes several groups of point pairs and connecting edges connecting each group of point pairs, and the connecting edges are marked with the actual matching value of the corresponding point pair, which is used to describe the matching degree of the corresponding point pair, for example , the actual matching value can be a value between 0 and 1; here, when the actual matching value is 0.1, it can indicate that the matching degree between the corresponding point pairs is low, and the points from the sample image in the point pairs are the same as those from the map data. The probability that a point corresponds to the same point in the space is low; when the actual matching value is 0.98, it can indicate that the matching degree between the corresponding point pairs is high, and the points from the sample image in the point pairs correspond to the points from the map data. The probability of the same point in space is high. Please refer to FIG. 2. FIG. 2 is a schematic state diagram of an embodiment of a training method for a matching prediction model of the present invention. As shown in FIG. 2, the left side is the sample matching data represented by a bipartite graph. The upper side and the lower side of the bipartite graph are two It is a set of points that do not intersect with each other, and the points connecting the two point sets are the connecting edges, and the connecting edges are marked with actual matching values (not shown).

In an implementation scenario, in order to improve the diversification of the sample matching data, the training device may further perform data enhancement on the sample matching data. For example, the training device may randomly rotate the coordinates of the three-dimensional points in the sample matching data to three axes respectively; or, it may also perform normalization processing on the three-dimensional points in the sample matching data, which is not limited herein.

Step S12: Use the matching prediction model to perform prediction processing on several groups of point pairs to obtain the predicted matching values of the point pairs.

Please continue to refer to FIG. 2, still taking the sample matching data represented by a bipartite graph as an example, the matching prediction model may include a first point feature extraction sub-model corresponding to the dimension to which the sample image belongs, and a first point feature extraction sub-model corresponding to the dimension to which the map data belongs. Two-point feature extraction sub-model, and edge feature extraction sub-model. For example, when the sample image is a two-dimensional image and the map data includes two-dimensional images, the first point feature extraction sub-model and the second point feature extraction sub-model are two-dimensional point feature extraction sub-models, then the matching prediction obtained by training The model can be used for 2D-3D matching prediction; or, when the sample image is a 3D image and the map data includes a 3D point cloud, the first point feature extraction submodel and the second point feature extraction submodel are 3D point feature extraction. submodel, the matching prediction model obtained by training can be used for 3D-3D matching prediction; or, when the sample image is a 2D image and the map data includes a 3D point cloud, the first point feature extraction submodel is a 2D point The feature extraction sub-model and the second point feature extraction sub-model are 3D point feature extraction sub-models, and the matching prediction model obtained by training can be used for 2D-3D matching prediction; here, the matching prediction model can be set according to the actual application , which is not limited here.

In an implementation scenario, the training device may use the first point feature extraction sub-model and the second point feature extraction sub-model to perform feature extraction on the bipartite graph to obtain the first feature and the second feature; and then use the edge feature extraction sub-model to perform feature extraction on the second feature The first feature and the second feature are extracted to obtain the third feature; the third feature is used to obtain the predicted matching value of the corresponding point of the connecting edge; as shown in Figure 2, the predicted matching value corresponding to each connecting edge in the bipartite graph is shown.

In one implementation scenario, when the first point feature extraction sub-model and the second point feature extraction sub-model are two-dimensional point feature extraction sub-models, at least one residual block (resblock) may be included, for example, one residual block, 2 residual blocks, 3 residual blocks, etc., each residual block (resblock) consists of multiple basic blocks (base blocks), and each basic block (base block) consists of a layer of 1*1 Convolutional layer, batch normalization layer (batch normalization), context normalization layer (context normalization) composition. When the first point feature extraction sub-model and the second point feature extraction sub-model are three-dimensional point feature extraction sub-models, at least one residual block (resblock) and at least one spatial transformation network (eg, t-net) may be included, For example, one residual block, two residual blocks, three residual blocks, etc., are not limited here. The number of spatial transformation networks can be one or two. Here, the spatial transformation networks can be located at the beginning and end of the model, which is not limited here. For the structure of the residual block (resblock), reference may be made to the structure in the foregoing implementation scenario, and details are not described herein again. The edge feature extraction sub-model may include at least one residual block, for example, 1 residual block, 2 residual blocks, 3 residual blocks, etc., which are not limited here, and the structure of the residual block (resblock) can be Referring to the structure in the foregoing implementation scenario, details are not repeated here.

Step S13: Determine the loss value of the matching prediction model by using the actual matching value and the predicted matching value.

In an implementation scenario, the training device may count the difference between the actual matching value and the predicted matching value, so as to determine the loss value of the matching prediction model. Here, the training device can count the sum of the differences between the predicted matching values of all point pairs and their actual matching values, and then use the sum and the number of all point pairs to obtain the average of the predicted matching values of all point pairs, as matching The loss value of the prediction model.

和實際匹配值

，確定匹配預測模型的第一損失值

，並利用非匹配點對的預測匹配值

和實際匹配值

，確定匹配預測模型的第二損失值

，從而通過對第一損失值

和第二損失值

進行加權處理，得到匹配預測模型的損失值

，參見公式（1）：

……（1） 上述公式（1）中，

表示匹配預測模型的損失值，

表示匹配點對所對應的第一損失值，

表示非匹配點對所對應的第二損失值，

和

分別表示第一損失值

的權重、第二損失值

的權重。 In another implementation scenario, the several sets of matching point pairs may include at least one set of matching point pairs between the included image points and map points, that is, the image points and map points included in the matching point pairs are in space the same point of different points in , the training device can use the predicted matching values

and the actual match value

, determine the first loss value that matches the prediction model

, and use the predicted matching values for non-matching point pairs

and the actual match value

, determine the second loss value that matches the prediction model

, so that by taking the first loss value

and the second loss value

Perform weighting processing to get the loss value that matches the prediction model

, see formula (1):

...(1) In the above formula (1),

represents the loss value of the matching prediction model,

represents the first loss value corresponding to the matching point pair,

represents the second loss value corresponding to the unmatched point pair,

and

respectively represent the first loss value

The weight of , the second loss value

the weight of.

和非匹配點對的第二數量

，從而可以利用匹配點對的預測匹配值和實際匹配值之間的差值，以及第一數量，確定第一損失值，參見公式（2）：

……（2） 上述公式（2）中，

表示第一損失值，

表示第一數量，

分別表示匹配點對的實際匹配值和預測匹配值。 In an implementation scenario, the training device may also count the first number of matching point pairs separately

and the second number of non-matching point pairs

, so that the difference between the predicted matching value and the actual matching value of the matching point pair, as well as the first quantity, can be used to determine the first loss value, see formula (2):

...(2) In the above formula (2),

represents the first loss value,

represents the first quantity,

represent the actual matching value and the predicted matching value of the matching point pair, respectively.

……（3） 上述公式（3）中，

表示第二損失值，

表示第二數量，

分別表示非匹配點對的實際匹配值和預測匹配值；此外，非匹配點對的實際匹配值

還可以統一設置為一預設數值（例如，0）。 The training device may also use the difference between the predicted matching value and the actual matching value of the unmatched point pair, as well as the second quantity, to determine the second loss value, see formula (3):

...(3) In the above formula (3),

represents the second loss value,

represents the second quantity,

represent the actual matching value and predicted matching value of the unmatched point pair, respectively; in addition, the actual matching value of the unmatched point pair

It can also be uniformly set to a preset value (eg, 0).

Step S14: Using the loss value, adjust the parameters of the matching prediction model.

In the embodiment of the present invention, the training device may adopt methods such as Stochastic Gradient Descent (SGD), Batch Gradient Descent (BGD), Mini-Batch Gradient Descent (MBGD), etc. The loss value adjusts the parameters of the matching prediction model; among them, batch gradient descent means that all samples are used to update parameters in each iterative operation; stochastic gradient descent means that one sample is used for each iterative operation. Parameter update; mini-batch gradient descent refers to using a batch of samples to update parameters in each repeated operation, which will not be repeated here.

In an implementation scenario, a training end condition may also be set, and when the training end condition is satisfied, the training device may end the training of the matching prediction model. The training end conditions may include: the loss value is less than a preset loss threshold, and the loss value is no longer reduced; the current training times reaches a preset times threshold (for example, 500 times, 1000 times, etc.), which is not limited here.

In the above scheme, the sample matching data is obtained by constructing the sample image and map data, and the sample matching data includes several groups of point pairs and the actual matching value of each group of point pairs, and the two points of each group of point pairs come from the sample image and Map data, so as to use the matching prediction model to perform prediction processing on several groups of point pairs to obtain the predicted matching value of the point pair, and then use the actual matching value and the predicted matching value to determine the loss value of the matching prediction model, and use the loss value to predict the matching. The parameters of the model are adjusted, so the matching prediction model can be used to establish a matching relationship, so that the matching prediction model can be used to predict the matching value between point pairs in visual positioning, so the points with high matching value can be preferentially sampled based on the predicted matching value. Yes, it can help to improve the accuracy and immediacy of visual positioning.

Please refer to FIG. 3 , which is a schematic flowchart of an embodiment of step S11 in FIG. 1 . The training device can construct the sample matching data through the following steps.

Step S111: Acquire several image points from the sample image, and acquire several map points from the map data to form several groups of point pairs.

Several sets of point pairs include at least one set of matched point pairs that match between the included image points and map points; that is, at least one set of included image points and map points in the several sets of point pairs corresponds to the spatial matching point pairs at the same point. Taking the sample image as a two-dimensional image and the map data as a sparse point cloud model obtained by SFM reconstruction as an example, several groups of point pairs contain at least one triangulated point and the triangulated point corresponds to the sparse point cloud 3D points in the model.

In an implementation scenario, several groups of point pairs may further include at least one group of non-matching point pairs that do not match between the included image points and map points, that is, several groups of point pairs may further include at least one A group contains image points and map points that correspond to unmatched pairs of different points in space. Still taking the sample image as a two-dimensional image and the map data as an example of a sparse point cloud model obtained by SFM reconstruction, several groups of point pairs can also include untriangulated points and any point in the sparse point cloud model. In order to form a set of non-matching point pairs, noise can be added to the sample matching data, and the robustness of the matching prediction model can be improved.

In an implementation scenario, please refer to FIG. 4 , which is a schematic flowchart of an embodiment of step S111 in FIG. 3 . The training device can obtain several sets of point pairs through the following steps.

Step S41: Divide the image points in the sample image into a first image point and a second image point.

In this embodiment of the present invention, the first image point has a matching map point in the map data, and the second image point does not have a matching map point in the map data. Still taking the sample image as a two-dimensional image and the map data as a sparse point cloud model obtained by SFM reconstruction as an example, the first image point can be the triangulated feature point in the sample image, and the second image point can be is an untriangulated feature point in the sample image; in other application scenarios, it can be deduced by analogy, which is not limited here.

In one implementation scenario, the image points in the sample image are feature points of the sample image. In another implementation scenario, the coordinates of the feature points may also be converted to a normalized plane.

Step S42: For each first image point, assign a number of first map points from the map data, and use the first image point and each first map point as a first point pair, wherein the first map point Points contains map points that match the first image point.

For each first image point, a number of first map points are allocated from the map data, and the first image point and each first map point are respectively regarded as a first point pair, and the first map point includes and The first image point matches the map point. In an implementation scenario, the number of first map points allocated to each first image point may be the same or different. In another implementation scenario, before assigning the first map points, a number of first image points may also be randomly selected from the first image points obtained by division, and the extracted first image points may be executed from the map The steps of allocating a number of first map points in the data and using the first image point and each first map point as a first point pair are not limited herein. In an implementation scenario, N points may be randomly selected from the first image points obtained by division, and for each first image point of the N first image points obtained by the extraction, randomly selected from the map data K first map points are allocated, and the randomly allocated K first map points include map points matching the first image point.

Step S43: For each second image point, assign a number of second map points from the map data, and use the second image point and each second map point as a second point pair respectively.

For each second image point, a number of second map points are allocated from the map data, and the second image point and each second map point are respectively used as a second point pair. In an implementation scenario, the number of second map points allocated to each second image point may be the same or different. In another implementation scenario, before allocating the second map points, a number of second image points may be randomly selected from the second image points obtained by division, and the extracted second image points may be executed from the map The steps of allocating a plurality of second map points in the data and using the second image point and each second map point as a second point pair are not limited herein. In one implementation scenario, M points may be randomly selected from the divided second image points, and for each second image point of the M second image points obtained by the extraction, randomly selected from the map data Allocate K second map points.

In an implementation scenario, in order to clarify whether each first point pair and each second point pair are matching point pairs, it is also possible to traverse each first point pair and each second point pair, and use the first identification A word (eg, 1) marks matching point pairs, and a second identifying word (eg, 0) marks non-matching point pairs.

The above-mentioned steps S42 and S43 can be performed sequentially, for example, step S42 is performed first, and then step S43 is performed; or, step S43 is performed first, and then step S42 is performed; This is not limited.

Step S44: Extracting several groups of point pairs from the first point pair and the second point pair.

In the embodiment of the present invention, several groups of point pairs may be obtained by randomly extracting from the first point pair and the second point pair, as a sample matching data. In an implementation scenario, the first point pair and the second point pair may also be randomly selected several times to obtain several sample matching data. In another implementation scenario, a plurality of sample images and map data can also be obtained, and the above steps are repeatedly performed for each sample image and map data to obtain a plurality of sample matching data, thereby increasing the number of samples, which is beneficial to Improve the accuracy of matching prediction models.

Step S112: for each set of matching point pairs: using the pose parameters of the sample image to project the map point into the dimension to which the sample image belongs to obtain the projected point of the map point; and based on the difference between the image point and the projected point , to determine the actual matching value of the matching point pair.

For each set of matched point pairs, the map point can be projected into the dimension to which the sample image belongs by using the pose parameters of the corresponding sample image to obtain the projected point of the map point. Taking the sample image as a two-dimensional image and the map data as a sparse point cloud model obtained by SFM reconstruction as an example, the training device can use the pose parameters to reproject the three-dimensional points to obtain their projected points.

In an implementation scenario, a preset probability distribution function can be used to convert the difference between the image point and its projected point into a probability density value, which is used as the actual matching value of the matching point pair. In an implementation scenario, the preset probability distribution function may be a standard Gaussian distribution function, so that the difference in the value range from negative infinity to positive infinity can be converted into a corresponding probability density value, and the greater the absolute value of the difference, The smaller the corresponding probability density value, the lower the matching degree of the corresponding point pair, the smaller the absolute value of the difference, the smaller the corresponding probability density value, the higher the matching degree of the corresponding point pair, when the absolute value of the difference is 0 , the corresponding probability density value is the largest.

In an implementation scenario, before using the pose parameters to project the map points to the dimension to which the sample images belong, the training device may also calculate the pose parameters of the sample images based on the matched point pairs; ) to calculate the pose parameters, so as to use the pose parameters to project the map point into the dimension to which the sample image belongs to obtain the projected point of the map point.

In an implementation scenario, the actual matching value of the non-matching point pair may also be set to a preset value, for example, the actual matching value of the non-matching point pair is set to 0.

Different from the foregoing embodiments, several groups of point pairs are formed by acquiring several image points from the sample image and several map points from the map data, and the several groups of point pairs include at least one group of included images The matching point pairs matched between points and map points can generate samples for training the matching prediction model, and for each set of matching point pairs, the pose parameters of the sample images are used to project the map points to the location where the sample images belong. In the dimension, the projection point of the map point is obtained, so as to determine the actual matching value of the matching point pair based on the difference between the image point and the projection point, so that the matching prediction model can learn the geometric features of the matching point pair during the training process. It is beneficial to improve the accuracy of the matching prediction model.

Please refer to FIG. 5 , which is a schematic flowchart of an embodiment of a visual positioning method of the present invention. The visual positioning method may include the following steps.

Step S51: Using the image to be located and the map data to construct matching data to be identified.

In the embodiment of the present invention, the matching data to be identified includes several groups of point pairs, and the two points of each group of point pairs are respectively from the image to be located and the map data. The dimension to which the image to be located and the map data belong may be 2-dimensional or 3-dimensional, which is not limited herein. For example, the image to be positioned may be a two-dimensional image, or, the image to be positioned may also be an RGB-D image, which is not limited here; the map data may be composed of a simple two-dimensional image, or may be composed of The composition of a 3D point cloud map can also be a combination of a 2D image and a 3D point cloud, which is not limited here.

Step S52: Use the matching prediction model to perform prediction processing on several groups of point pairs to obtain the predicted matching values of the point pairs.

The matching prediction model is a neural network model trained in advance through sample matching data. In an implementation scenario, the matching prediction model may be obtained by training through the steps in any of the foregoing embodiments of the matching prediction model training method, wherein the training steps may refer to the steps in the foregoing embodiments, which will not be repeated here.

By using the matching prediction model to perform prediction processing on several groups of point pairs, the predicted matching values of the point pairs in the matching data to be identified can be obtained. In one implementation scenario, the matching data to be identified is a bipartite graph, the bipartite graph includes several groups of point pairs and connecting edges connecting each group of point pairs, and the matching prediction model includes a first point feature corresponding to the dimension to which the image to be located belongs. The extraction sub-model, the second point feature extraction sub-model corresponding to the dimension to which the map data belongs, and the edge feature extraction sub-model, so that the bipartite graph can be characterized by using the first point feature extraction sub-model and the second point feature extraction sub-model Extraction to obtain the first feature and the second feature, and use the edge feature extraction sub-model to perform feature extraction on the first feature and the second feature to obtain the third feature, and then use the third feature to obtain the prediction of the point pair corresponding to the connecting edge match value. Here, reference may be made to the steps in the foregoing embodiments, which will not be repeated here.

Step S53: Determine the pose parameters of the photographing device of the image to be positioned based on the predicted matching value of the point pair.

According to the predicted matching value of the point pair in the matching data to be identified, the point pair with a relatively high predicted matching value can be preferentially used to determine the pose parameters of the photographic device of the image to be positioned. In an implementation scenario, the PnP (Perspective-n-Point) problem can be constructed by using n point pairs with relatively high predicted matching values, so as to solve the PnP problem by means such as EPnP (Efficient PnP), and then obtain the to-be-located problem. The pose parameters of the photographic device of the image. In another implementation scenario, several sets of point pairs may also be sorted in descending order of predicted matching values, and the previously preset number of sets of point pairs may be used to determine the pose parameters of the photographic device of the image to be positioned. The first preset number can be set according to the actual situation, for example, point pairs whose predicted matching value is not 0 among the sorted groups of point pairs are used as the first preset number of point pairs; If the predicted matching value in the group point pair is greater than the lower limit value, it is used as the first preset number of group point pairs; the first preset number can be set according to the actual application, which is not limited here. Here, a method such as PROSAC (PROgressive SAmple Consensus, progressive consistent sampling) may also be used to process the sorted point pairs to obtain the pose parameters of the photographic device of the image to be positioned. In an implementation scenario, the pose parameters of the photographic device of the image to be positioned may include 6 degrees of freedom (DoF) of the photographic device in the map coordinate system to which the map data belongs, including: pose , that is, the coordinates, and the deflection yaw (pitch angle) around the x-axis, the deflection pitch (yaw angle) around the y-axis, and the deflection roll (roll angle) around the z-axis.

In the above scheme, the matching data to be identified is constructed by using the image to be located and the map data, and the matching data to be identified includes several sets of point pairs, and the two points of each set of point pairs are respectively from the image to be located and the map data, so as to use the matching data to be identified. The matching prediction model performs prediction processing on several groups of point pairs to obtain the predicted matching value of the point pair, and then determines the pose parameters of the photographic device of the image to be positioned based on the predicted matching value of the point pair, so the matching can be used in visual positioning. The prediction model predicts the matching value between point pairs to establish a matching relationship, which can help to improve the accuracy and immediacy of visual positioning.

Please refer to FIG. 6 . FIG. 6 is a schematic block diagram of an embodiment of a training apparatus 60 for matching prediction models of the present invention. The training device 60 for matching prediction models includes a sample construction part 61, a prediction processing part 62, a loss determination part 63 and a parameter adjustment part 64. The sample construction part 61 is configured to use sample images and map data to construct sample matching data, wherein the sample The matching data includes several groups of point pairs and the actual matching value of each group of point pairs, and the two points of each group of point pairs are respectively from the sample image and the map data; the prediction processing part 62 is configured to use the matching prediction model to perform the matching prediction model on the several groups of point pairs. Prediction processing to obtain the predicted matching value of the point pair; the loss determination part 63 is configured to use the actual matching value and the predicted matching value to determine the loss value of the matching prediction model; the parameter adjustment part 64 is configured to use the loss value to adjust the parameters of the matching prediction model .

The above solution can use the matching prediction model to establish a matching relationship, so that the matching prediction model can be used in the visual positioning to predict the matching value between the point pairs, so the point pair with the high matching value can be preferentially sampled based on the predicted matching value, and then the matching value can be sampled. It is beneficial to improve the accuracy and immediacy of visual positioning.

In some embodiments, the sample construction section 61 includes a point pair acquisition subsection configured to acquire several image points from the sample image and several map points from the map material to form several sets of point pairs; wherein several The set of point pairs includes at least one set of matching point pairs matched between the included image points and map points, and the sample construction section 61 includes a first matching value determination subsection configured to, for each set of matching point pairs: use the sample image The pose parameters of the map point are projected into the dimension to which the sample image belongs, and the projected point of the map point is obtained; and the actual matching value of the matching point pair is determined based on the difference between the image point and the projected point.

Different from the foregoing embodiments, several groups of point pairs are formed by acquiring several image points from the sample image and several map points from the map data, and the several groups of point pairs include at least one group of included images The matching point pairs matched between points and map points can generate samples for training the matching prediction model, and for each set of matching point pairs, the pose parameters of the sample images are used to project the map points to the location where the sample images belong. In the dimension, the projection point of the map point is obtained, so as to determine the actual matching value of the matching point pair based on the difference between the image point and the projection point, so that the matching prediction model can learn the geometric features of the matching point pair during the training process. It is beneficial to improve the accuracy of the matching prediction model.

In some embodiments, the sets of point pairs include at least one set of non-matched point pairs that do not match between the included image points and map points, and the sample construction section 61 includes a second matching value determination subsection configured to The actual matching value of the matching point pair is set to the preset value.

Different from the foregoing embodiments, several groups of point pairs include at least one group of non-matching point pairs that do not match between the included image points and map points, and different from the matching point pairs, the actual matching values of the non-matching point pairs are set. The preset value can help to improve the robustness of the matching prediction model.

In some embodiments, the point pair acquisition subsection includes an image point division section configured to divide the image points in the sample image into first image points and second image points, wherein the first image point There is a matching map point in the map material, the second image point does not have a matching map point in the map material, the point pair acquisition subsection includes a first point pair acquisition section, and is configured for each first image point, assign a number of first map points from the map data, and respectively use the first image point and each first map point as a first point pair, wherein the first map point includes matching with the first image point the map point, the point pair acquisition subsection includes a second point pair acquisition section, configured to allocate a number of second map points from the map data for each second image point, and respectively associate the second image point with each The second map point is used as a second point pair, and the point pair acquisition subsection includes a point pair extraction section configured to extract several groups of point pairs from the first point pair and the second point pair.

Different from the foregoing embodiments, by dividing the image points in the sample image into a first image point and a second image point, and the first image point has a matching map point in the map, the second image There is no matching image point in the image data, and for the first image point, a number of first map points are allocated from the map data, and the first image point and each first map point are regarded as one. The first point pair, and the first map point includes a map point matching the first image point, and for each second image point, a number of second map points are allocated from the map data, and the second image The point and each second map point are used as a second point pair, and several groups of point pairs are extracted from the first point pair and the second point pair, so that a rich number of point pairs can be constructed and include non-matching point pairs and matching point pairs Several groups of point pairs are used to train the matching prediction model, so it can help to improve the accuracy of the matching prediction model.

In some embodiments, the first matching value determination subsection includes a pose calculation section configured to calculate pose parameters of the sample image based on the matching point pairs, and the first matching value determination subsection includes a projection section configured to utilize the position The pose parameter projects the map point into the dimension to which the sample image belongs to obtain the projected point of the map point.

Different from the previous embodiments, by using the matching point pair to calculate the pose parameters of the sample image, and using the pose parameters to project the map points to the dimension to which the sample image belongs, the projection points of the map points are obtained, which can be beneficial to Improving the accuracy of the difference between the projection point and the image point can be beneficial to improve the accuracy of the matching prediction model.

In some embodiments, the first matching value determination subsection includes a probability density conversion section configured to convert the difference into a probability density value using a preset probability distribution function as the actual matching value of the matching point pair

Different from the foregoing embodiments, by using a preset probability distribution function to convert the difference into a probability density value as the actual matching value of the matching point pair, it can be beneficial to accurately describe the difference between the projection point and the image point, thereby enabling It is beneficial to improve the accuracy of the matching prediction model.

In some embodiments, the sample matching data is a bipartite graph, and the bipartite graph includes several groups of point pairs and connecting edges connecting each group of point pairs, and the connecting edges are marked with the actual matching values of the corresponding point pairs; the matching prediction model includes and the sample graph Like the first point feature extraction sub-model corresponding to the dimension to which it belongs, the second point feature extraction sub-model and the edge feature extraction sub-model corresponding to the dimension to which the map data belongs, the prediction processing part 62 includes a point feature extraction sub-part, which is configured to be respectively Using the first point feature extraction sub-model and the second point feature extraction sub-model to perform feature extraction on the bipartite graph to obtain the first feature and the second feature, the prediction processing part 62 includes an edge feature extraction sub-part, which is configured to use the edge feature extraction sub-part The model performs feature extraction on the first feature and the second feature to obtain a third feature. The prediction processing part 62 includes a prediction sub-part configured to use the third feature to obtain the predicted matching value of the point pair corresponding to the connecting edge.

Different from the foregoing embodiments, by performing point feature extraction and edge feature extraction on the bipartite graph respectively, the matching prediction model can more effectively perceive the spatial geometric structure of the matching, thereby improving the accuracy of the matching prediction model.

In some embodiments, the structure of the first point feature extraction sub-model and the second point feature extraction sub-model is any of the following: including at least one residual block, including at least one residual block and at least one spatial transformation network; and /or, the edge feature extraction sub-model includes at least one residual block.

Different from the foregoing embodiments, the structures of the first point feature extraction sub-model and the second point feature extraction sub-model are set to any one of the following: including at least one residual block, including at least one residual block and at least one spatial transformation network, and the edge feature extraction sub-model is set to include at least one residual block, so it can facilitate the optimization of the matching prediction model and improve the accuracy of the matching prediction model.

In some embodiments, the sets of point pairs include at least one set of matched point pairs that match between the included image points and map points and at least one set of non-matches that do not match between the included image points and map points point pair, the loss determination section 63 includes a first loss determination subsection configured to use the predicted matching value and the actual matching value of the matching point pair to determine a first loss value matching the prediction model, and the loss determination section 63 includes a second loss determination subsection part 63 configured to use the predicted matching value and the actual matching value of the non-matching point pair to determine a second loss value of the matching prediction model, the loss determination part 63 includes a loss weighting sub-section configured to compare the first loss value and the second loss value Perform weighting processing to obtain the loss value that matches the prediction model.

Different from the previous embodiment, the first loss value of the matching prediction model is determined by using the predicted matching value and the actual matching value of the matching point pair, and the predicted matching value and the actual loss value of the non-matching point pair are used to determine the matching prediction model. The second loss value, so that the first loss value and the second loss value are weighted to obtain the loss value of the matching prediction model, so it can help the matching prediction model to effectively perceive the matching spatial geometry, thereby improving the matching prediction model. accuracy.

In some embodiments, the loss determination part 63 further includes a quantity statistics subsection, configured to count the first number of matched point pairs and the second number of non-matched point pairs, respectively; the first loss determination subsection is configured to use matching The difference between the predicted matching value and the actual matching value of the point pair, and the first number, determine the first loss value; the second loss determining subsection is configured to use the predicted matching value and the actual matching value of the non-matching point pair The difference between , and the second quantity, determines the second loss value.

Different from the foregoing embodiments, by counting the first number of matching point pairs and the second number of non-matching point pairs, the difference between the predicted matching value and the actual matching value of the matching point pair, and the first number, are used, Determining the first loss value, and using the difference between the predicted matching value and the actual matching value of the unmatched point pair, and the second quantity, determining the second loss value can help to improve the accuracy of the loss value of the matching prediction model, Thus, the accuracy of the matching prediction model can be improved.

In some embodiments, the dimension to which the sample image belongs is 2D or 3D, and the dimension to which the map data belongs is 2D or 3D.

Different from the foregoing embodiments, by setting the dimensions to which the sample images and map data belong, a matching prediction model for 2D-2D can be trained, or a matching prediction model for 2D-3D can be trained, or The matching prediction model for 3D-3D can be obtained by training, so that the applicable scope of the matching prediction model can be improved.

Please refer to FIG. 7 . FIG. 7 is a schematic block diagram of an embodiment of a visual positioning device 70 of the present invention. The visual positioning device 70 includes a data construction part 71, a prediction processing part 72 and a parameter determination part 73. The data construction part 71 is configured to use the image and map data to be located to construct the matching data to be identified, wherein the matching data to be identified includes several groups. point pairs, the two points of each group of point pairs are respectively from the image to be located and the map data; the prediction processing part 72 is configured to use the matching prediction model to perform prediction processing on several groups of point pairs to obtain the predicted matching values of the point pairs; parameter determination Section 73 is configured to determine pose parameters of the photographic device of the image to be positioned based on the predicted matching values of the point pairs.

In the above solution, the matching prediction model can be used to establish the matching relationship, so that the matching prediction model can be used to predict the matching value between the point pairs in the visual positioning to establish the matching relationship, which can help to improve the accuracy and immediacy of the visual positioning.

In some embodiments, the parameter determination section 73 includes a point pair sorting subsection configured to sort several groups of point pairs in descending order of predicted matching values, and the parameter determination section 73 further includes a parameter determination subsection configured as Using the previously preset number of point pairs, the pose parameters of the photographic device of the image to be positioned are determined.

In the embodiments of the present invention and other embodiments, a "part" may be a part of a circuit, a part of a processor, a part of a program or software, etc., of course, a unit, a module or a non-modular form.

Different from the foregoing embodiments, by sorting several groups of point pairs in descending order of predicted matching values, and using the previously preset number of groups of point pairs to determine the pose parameters of the photographic device of the image to be positioned, it is possible to It is beneficial to use the sorted point pairs for incremental sampling, and preferentially sample the point pairs with high matching values. Therefore, the geometric prior can guide the solution of the pose parameters, thereby improving the accuracy and immediacy of visual positioning.

In some embodiments, the matching prediction model is obtained by training the matching prediction model training device in any of the above-mentioned embodiments of the matching prediction model training device.

Different from the foregoing embodiments, performing visual positioning through the matching prediction model obtained by the training device for matching prediction models in any of the above-mentioned embodiments of the training device for matching prediction models can improve the accuracy and immediacy of visual positioning.

Please refer to FIG. 8 , which is a block diagram illustrating an embodiment of an electronic device 80 of the present invention. The electronic device 80 includes a memory 81 and a processor 82 coupled to each other, and the processor 82 is configured to execute program instructions stored in the memory 81 to implement the steps in any of the above-mentioned embodiments of the training method for matching the prediction model, or to implement The steps in any of the above-mentioned embodiments of the visual positioning method. In an implementation scenario, the electronic device 80 may include, but is not limited to, mobile devices such as a mobile phone and a matching computer, which are not limited herein.

The processor 82 is configured to control itself and the memory 81 to implement the steps in any of the above-mentioned embodiments of the training method for matching prediction models, or to implement the steps in any of the above-mentioned embodiments of the visual positioning method. The processor 82 may also be referred to as a CPU (Central Processing Unit, central processing unit). The processor 82 may be an integrated circuit chip with signal processing capabilities. The processor 82 may also be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), or a field-programmable gate array (FPGA) Or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. Additionally, the processor 82 may be commonly implemented by an integrated circuit die.

The above solution can use the matching prediction model to establish a matching relationship, so that the matching prediction model can be used in the visual positioning to predict the matching value between the point pairs, so the point pair with the high matching value can be preferentially sampled based on the predicted matching value, and then the matching value can be sampled. It is beneficial to improve the accuracy and immediacy of visual positioning.

Please refer to FIG. 9 , which is a block diagram illustrating an embodiment of a computer-readable storage medium 90 of the present invention. The computer-readable storage medium 90 stores program instructions 901 that can be run by the processor. The program instructions 901 are used to implement the steps in any of the above-mentioned embodiments of the training method for matching prediction models, or to implement any of the above-mentioned embodiments of the visual positioning method. A step of.

The above solution can use the matching prediction model to establish a matching relationship, so that the matching prediction model can be used in the visual positioning to predict the matching value between the point pairs, so the point pair with the high matching value can be preferentially sampled based on the predicted matching value, and then the matching value can be sampled. It is beneficial to improve the accuracy and immediacy of visual positioning.

In the several embodiments provided by the present invention, it should be understood that the disclosed method and apparatus may be implemented in other manners. For example, the device implementations described above are only illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other divisions. For example, units or elements may be combined or integrated. to another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

Units described as separate components may or may not be physically separated, and components shown as units may or may not be physical units, that is, may be located in one place, or may be distributed over network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this implementation manner.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware, or can be implemented in the form of software functional units.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, Several instructions are included to cause a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes: U disk, removable hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or CD, etc. medium.

Industrial Applicability In the embodiment of the present invention, the matching prediction model can be used to establish a matching relationship, so that the matching prediction model can be used to predict the matching value between point pairs in visual positioning, so the point pair with high matching value can be preferentially sampled based on the predicted matching value. , and establish a matching relationship, which can help to improve the accuracy and immediacy of visual positioning.

60: A training device for matching prediction models 61: Sample Construction Section 62: Prediction processing part 63: Loss Determination Section 64: Parameter adjustment part 70: Visual positioning device 71: Data Construction Section 72: Prediction processing part 73: Parameter determination part 80: Electronic equipment 81: Memory 82: Processor 90: Computer-readable storage media 901: Program command S11~S14, S111~S112, S41~S44, S51~S53: Steps

1 is a schematic flowchart of an embodiment of a training method for a matching prediction model of the present invention; 2 is a state schematic diagram of an embodiment of a training method for a matching prediction model of the present invention; 3 is a schematic flowchart of an embodiment of step S11 in FIG. 1; FIG. 4 is a schematic flowchart of an embodiment of step S111 in FIG. 3; 5 is a schematic flowchart of an embodiment of the visual positioning method of the present invention; 6 is a schematic block diagram of an embodiment of a training device for matching prediction models of the present invention; 7 is a schematic block diagram of an embodiment of the visual positioning device of the present invention; 8 is a schematic block diagram of an embodiment of an electronic device of the present invention; FIG. 9 is a schematic block diagram of an embodiment of a computer-readable storage medium of the present invention.

S11~S14: Steps

Claims (15)

A training method for matching predictive models, including: Use sample images and map data to construct sample matching data, wherein the sample matching data includes several groups of point pairs and the actual matching values of each group of point pairs, and the two points of each group of point pairs are respectively from the sample image and said map data; Use a matching prediction model to perform prediction processing on the several groups of point pairs to obtain the predicted matching values of the point pairs; Using the actual matching value and the predicted matching value, determining the loss value of the matching prediction model; Using the loss value, the parameters of the matching prediction model are adjusted. The training method according to claim 1, wherein, using sample images and map data to construct sample matching data includes: A number of image points are obtained from the sample image, and a number of map points are obtained from the map material to form a number of sets of point pairs; wherein the sets of point pairs include at least one set of included image points Matching point pairs that match between map points; For each set of the matching point pairs: use the pose parameters of the sample image to project the map point into the dimension to which the sample image belongs to obtain the projected point of the map point; and based on the map The difference between the image point and the projected point determines the actual matching value of the matching point pair. The training method according to claim 2, wherein the several groups of point pairs include at least one group of non-matching point pairs that do not match between the included image points and map points, and the sample images and map data are used for the training method. , the construction of sample matching data also includes: The actual matching value of the non-matching point pair is set as a preset value. The training method according to claim 2 or 3, wherein the acquiring several image points from the sample image and acquiring several map points from the map data to form several sets of point pairs, including: The image points in the sample image are divided into a first image point and a second image point, wherein the first image point has the map point matching it in the map data, so the second image point does not have a matching map point in the map data; For each of the first image points, a number of first map points are allocated from the map data, and the first image point and each of the first map points are respectively used as a first point pair, wherein the first map point includes the map point that matches the first image point; and, for each of the second image points, assigning a plurality of second map points from the map data, and respectively using the second image point and each of the second map points as a second point pair; Several sets of point pairs are extracted from the first point pair and the second point pair. The training method according to claim 2 or 3, wherein the map point is projected into the dimension to which the sample image belongs by using the pose parameter of the sample image to obtain the projection of the map point Points include: Based on the matched point pair, calculate the pose parameter of the sample image; Using the pose parameter to project the map point into the dimension to which the sample image belongs, to obtain the projected point of the map point; and/or, The determining the actual matching value of the matching point pair based on the difference between the image point and the projection point includes: The difference is converted into a probability density value using a preset probability distribution function as the actual matching value of the matching point pair. The training method according to any one of claim 1 to 3, wherein the sample matching data is a bipartite graph, and the bipartite graph includes several groups of point pairs and connecting edges connecting each group of point pairs, and the connecting edges The actual matching value corresponding to the point pair is marked; the matching prediction model includes a first point feature extraction sub-model corresponding to the dimension to which the sample image belongs, and a second point corresponding to the dimension to which the map data belongs Feature extraction sub-model and edge feature extraction sub-model; The performing prediction processing on the several groups of point pairs by using the matching prediction model, and obtaining the predicted matching values of the point pairs includes: Using the first point feature extraction sub-model and the second point feature extraction sub-model to perform feature extraction on the bipartite graph, respectively, to obtain the first feature and the second feature; Using the edge feature extraction sub-model to perform feature extraction on the first feature and the second feature to obtain a third feature; Using the third feature, the predicted matching value of the point pair corresponding to the connecting edge is obtained. The training method according to claim 6, wherein the structures of the first point feature extraction sub-model and the second point feature extraction sub-model are any of the following: including at least one residual block, including at least one residual blocks and at least one spatial transformation network; and/or, The edge feature extraction submodel includes at least one residual block. The training method according to any one of claim 1 to 3, wherein the several sets of point pairs include at least one set of matching point pairs that match between image points and map points contained in at least one set and at least one set of contained point pairs. non-matching point pairs that do not match between image points and map points; The determining the loss value of the matching prediction model by using the actual matching value and the predicted matching value includes: Using the predicted matching value and the actual matching value of the matching point pair to determine a first loss value of the matching prediction model; and using the predicted matching value and the actual matching value of the non-matching point pair to determine the second loss value of the matching prediction model; The first loss value and the second loss value are weighted to obtain the loss value of the matching prediction model. The training method according to claim 8, wherein before the first loss value of the matching prediction model is determined by using the predicted matching value and the actual matching value of the matching point pair, the method further comprises: include: respectively count the first number of the matching point pairs and the second number of the non-matching point pairs; The determining the first loss value of the matching prediction model by using the predicted matching value and the actual matching value of the matching point pair includes: Using the difference between the predicted matching value and the actual matching value of the matching point pair, and the first number, determining the first loss value; Using the predicted matching value and the actual matching value of the non-matching point pair to determine the second loss value of the matching prediction model includes: The second loss value is determined using the difference between the predicted match value and the actual match value for the pair of non-matching points, and the second number. The training method according to any one of claims 1 to 3, wherein the dimension to which the sample image belongs is 2-dimensional or 3-dimensional, and the dimension to which the map data belongs is 2-dimensional or 3-dimensional. A visual positioning method comprising: Using the to-be-located image and map data, the to-be-identified matching data is constructed, wherein the to-be-identified matching data includes several sets of point pairs, and the two points of each set of point pairs are respectively from the to-be-located image and the map data ; Use a matching prediction model to perform prediction processing on the several groups of point pairs to obtain the predicted matching values of the point pairs; Based on the predicted matching values of the point pairs, a pose parameter of the photographic device of the to-be-located image is determined. The visual positioning method according to claim 11, wherein the determining the pose parameters of the photographic device of the image to be positioned based on the predicted matching value of the point pair comprises: sorting the groups of point pairs in descending order of the predicted matching values; Using the previously preset number of sets of the point pairs, the pose parameters of the photographing device of the to-be-positioned image are determined. The visual positioning method according to claim 11 or 12, wherein the matching prediction model is obtained by using the training method of the matching prediction model described in any one of claim 1 to 10. An electronic device, comprising a memory and a processor coupled to each other, the processor is used to execute program instructions stored in the memory, so as to realize the training method of the matching prediction model according to any one of request items 1 to 10 , or the visual positioning method described in any one of claim items 11 to 13. A computer-readable storage medium on which program instructions are stored, and when the program instructions are executed by a processor, the training method of the matching prediction model described in any one of claim items 1 to 10, or any one of claim items 11 to 13 The visual positioning method described in item.

Images