KR20220051162A - Visual positioning methods, training methods for related models, and related devices and devices - Google Patents

Abstract

The present invention discloses a visual positioning method, a training method of a related model, and related apparatus and apparatus. Here, the training method of the matching prediction model comprises the steps of constructing sample matching data using a sample image and map data - the sample matching data includes a plurality of groups of point pairs and actual matching values of each group of point pairs, each The two points of the point pair in the group are derived from sample image and map data, respectively; performing prediction processing on a plurality of groups of point pairs using a matching prediction model to obtain predictive matching values of the point pairs; determining a loss value of the matching prediction model using the actual matching value and the predicted matching value; and adjusting a parameter of the matching prediction model using the loss value. According to the above means, the accuracy and immediacy of the visual positioning is improved.

Description

Visual positioning methods, training methods for related models, and related devices and devices

Cross-referencing of related applications

The present invention is filed based on a Chinese patent application with an application number of 202011110569.5 and an filing date of October 16, 2020, and claims the priority of the Chinese patent application, the entire contents of which are incorporated herein by reference.

The present invention relates to the field of computer visual technology, and more particularly, to a method for visual positioning, a method for training a related model, and related apparatus and apparatus.

Visual positioning can be classified in various ways depending on the representation method of map data. Among them, the structure-based method, also called a feature-based method, is receiving widespread attention due to its high accuracy and excellent generalization performance.

When visual positioning is performed using the current feature-based method, it is necessary to obtain a plurality of point pairs between image data and map data through matching. However, when using local similarity to build a matching relationship, the reliability is low, especially in large scenes or scenes with repetitive structures/repeating textures, matching errors easily occur, affecting the accuracy of visual positioning. Random Sample Consensus (RANSAC) can eliminate matching errors, but RANSAC samples with equal probability for each sample point, so if there are too many external points during initial matching, RANSAC is time consuming and reduces accuracy. A low problem exists that affects the immediacy and accuracy of visual positioning. From this point of view, how to improve the accuracy and immediacy of visual positioning is an urgent task.

The present invention provides a visual positioning method, a training method of a related model, and related apparatus and apparatus.

A first aspect of the present invention provides a method for training a matching prediction model. The training method of the matching prediction model includes: constructing sample matching data using a sample image and map data; the sample matching data includes a plurality of groups of point pairs and actual matching values of each group of point pairs, each group The two points of the point pair of − are derived from sample image and map data, respectively; performing prediction processing on a plurality of groups of point pairs using a matching prediction model to obtain predictive matching values of the point pairs; determining a loss value of the matching prediction model using the actual matching value and the predicted matching value; and adjusting a parameter of the matching prediction model using the loss value.

Therefore, sample matching data is constructed and obtained using the sample image and map data, and the sample matching data includes a plurality of groups of point pairs and actual matching values of each group of point pairs, and two points of each group of point pairs. are derived from sample image and map data, respectively. Therefore, predictive processing is performed on a plurality of groups of point pairs using the matching prediction model to obtain the predicted matching value of the point pair, and further, the loss value of the matching prediction model is determined using the actual matching value and the predicted matching value, Since the parameters of the matching prediction model are adjusted using the loss value, a matching relationship can be built using the matching prediction model, and accordingly, the matching value between the point pairs can be predicted using the matching prediction model in the visual positioning process. Based on the matching value obtained by prediction, it is possible to determine the imaging pose parameter of the image to be positioned by preferentially sampling the point pair having a high matching value, and furthermore, it may be advantageous to improve the accuracy and immediateness of visual positioning.

Here, the step of constructing the sample matching data by using the sample image and the map data includes: obtaining a plurality of image points from the sample image and obtaining a plurality of map points from the map data to configure a plurality of groups of point pairs the pair of points in the group includes a pair of matching points that match between the image points and the map points included in at least one group; and for each group of matching point pairs, project the map point to the dimension to which the sample image belongs by using the pose parameter of the sample image to obtain the projection point of the map point, and based on the difference between the image point and the projection point, the matching point determining an actual match value of the pair.

Accordingly, a plurality of image points are obtained from the sample image and a plurality of map points are obtained from the map data to configure a plurality of groups of point pairs, and the plurality of groups of point pairs includes matching between image points and map points included in at least one group. Since matching point pairs are included, samples for training a matching prediction model can be generated, and for each group of matching point pairs, the map point is projected onto the dimension to which the sample image belongs by using the pose parameter of the sample image. By obtaining the projection point of the map point, the matching prediction model can learn the geometric characteristics of the matching point pair in the training process by determining the actual matching value of the matching point pair based on the difference between the image point and the projection point. It is beneficial to improve the accuracy of

Here, the pairs of points in the plurality of groups include non-matching points that do not match between the image points and map points included in at least one group, and the step of constructing sample matching data using the sample image and map data includes: The method further includes setting the actual matching value of the pair to a preset value.

Accordingly, a plurality of groups of point pairs includes a mismatching point pair that does not match between an image point and a map point included in at least one group. Therefore, it may be advantageous to improve the robustness of the matching prediction model.

Here, the step of obtaining a plurality of image points from the sample image and obtaining a plurality of map points from the map data to form a plurality of groups of point pairs includes dividing the image points in the sample image into a first image point and a second image point. doing, a map point matching the first image point exists in the map data, and a map point matching the second image point does not exist in the map data; allocating, for each first image point, a plurality of first map points in the map data, and using the first image point and each first map point as one first pair of points each - a first map point contains a map point that matches the first image point; allocating, for each second image point, a plurality of second map points in the map data, and using the second image point and each second map point as one second point pair; and extracting and obtaining a plurality of groups of point pairs from the first point pair and the second point pair.

Accordingly, an image point in the sample image is divided into a first image point and a second image point, a map point matching the first image point exists in the map, and an image point matching the second image point exists in the image data. without, for the first image point, allocating a plurality of first map points in the map data, using the first image point and each first map point as one first point pair, respectively, and the first map point includes map points matching the first image point, and for each second image point, assigns a plurality of second map points from the map data, and sets the second image point and each second map point to one each. is used as the second point pair of , and by extracting and obtaining a plurality of groups of point pairs from the first and second point pairs Since the matching prediction model can be used for training by building

Here, the step of obtaining the projection point of the map point by projecting the map point to a dimension to which the sample image belongs by using the pose parameter of the sample image includes: calculating the pose parameter of the sample image based on a pair of matching points; and projecting the map point to a dimension to which the sample image belongs by using the pose parameter to obtain a projection point of the map point.

Therefore, by using the matching point pair to calculate the pose parameter of the sample image, and by using the pose parameter to project the map point to the dimension to which the sample image belongs to obtain the projection point of the map point, the difference accuracy between the projection point and the image point It may be advantageous for improvement, and furthermore, it may be advantageous for improving the accuracy of the matching prediction model.

Here, the step of determining the actual matching value of the matching point pair based on the difference between the image point and the projection point includes converting the difference into a probability density value using a preset probability distribution function and using it as the actual matching value of the matching point pair including the steps of

Therefore, since the difference is converted into a probability density value using a preset probability distribution function and used as an actual matching value of a pair of matching points, it may be advantageous to accurately explain the difference between the projection point and the image point, and thus the accuracy of the matching prediction model may be beneficial for improvement.

Here, the sample matching data is a bipartite graph, wherein the bipartite graph includes a plurality of groups of point pairs and a connecting edge connecting each group of point pairs, and the connecting edge is labeled with an actual matching value corresponding to the point pair; the matching prediction model includes a first point feature extraction submodel corresponding to a dimension to which the sample image belongs, a second point feature extraction submodel corresponding to a dimension to which map data belongs, and an edge feature extraction submodel; The step of performing prediction processing on a plurality of groups of point pairs using the matching prediction model to obtain a predicted matching value of the point pair includes using the first point feature extraction sub-model and the second point feature extraction sub-model, respectively. performing feature extraction on the graph to obtain first and second features; obtaining a third feature by performing feature extraction on the first feature and the second feature using the edge feature extraction sub-model; and obtaining a predictive matching value of a point pair corresponding to a connecting edge by using the third feature.

Accordingly, by performing point feature extraction and edge feature extraction respectively on the bipartite graph, the matching prediction model can more effectively recognize the spatial geometry of the matching, and furthermore, it can be advantageous for improving the accuracy of the matching prediction model.

Here, the structures of the first point feature extraction sub-model and the second point feature extraction sub-model include a structure including at least one residual block, a structure including at least one residual block and at least one spatial transformation network. either; and/or, the edge feature extraction sub-model includes at least one residual block.

Accordingly, the structure of the first point feature extraction sub-model and the second point feature extraction sub-model is any one of a structure including at least one residual block, a structure including at least one residual block and at least one spatial transformation network. By setting one and setting the edge feature extraction sub-model to a structure including at least one residual block, it may be advantageous to optimize the matching prediction model, and also improve the accuracy of the matching prediction model.

wherein the plurality of groups of point pairs includes a matching point pair that matches between an image point and a map point included in at least one group and a non-matching point pair that does not match between an image point and a map point included in at least one group; The determining of the loss value of the matching prediction model by using the actual matching value and the predicted matching value may include: determining a first loss value of the matching prediction model by using the predicted matching value and the actual matching value of the matching point pair; determining a second loss value of the matching prediction model by using the predicted matching value and the actual matching value of the non-matching point pair; and performing weighting processing on the first loss value and the second loss value to obtain a loss value of the matching prediction model.

Therefore, the first loss value of the matching prediction model is determined using the predicted matching value and the actual matching value of the matching point pair, and the second loss of the matching prediction model is determined using the predicted matching value and the actual loss value of the non-matching point pair. By determining the value, weight processing is performed on the first loss value and the second loss value to obtain a loss value of the matching prediction model, so that the matching prediction model may be advantageous for effectively recognizing the spatial geometry of the matching, and thus the matching Improve the accuracy of predictive models.

Here, before the step of determining the first loss value of the matching prediction model by using the predicted matching value and the actual matching value of the matching point pair, the method includes: a first number of matching point pairs and a second number of non-matching point pairs further comprising stating each determining a first loss value by using the difference value and the first number, and determining a second loss value of the matching prediction model by using the predicted matching value and the actual matching value of the mismatching point pair, and determining a second loss value using a second number and a difference value between the predicted matching value and the actual matching value of the non-matching point pair.

Therefore, by stating the first number of matching point pairs and the second number of non-matching point pairs, the first loss value is determined using the difference value and the first number between the predicted matching value and the actual matching value of the matching point pair, and , since the second loss value is determined using the second number and the difference between the predicted matching value and the actual matching value of the non-matching point pair, it may be advantageous to improve the accuracy of the loss value of the matching prediction model, and thus the accuracy of the matching prediction model may be beneficial for improvement.

Here, the dimension to which the sample image belongs is 2D or 3D, and the dimension to which the map data belongs is 2D or 3D.

Therefore, by setting the dimension to which the sample image and map data belong, a matching prediction model for 2D-2D is obtained through training, a matching prediction model for 2D-3D is obtained through training, or through training Since a matching prediction model for 3D-3D can be obtained, the application range of the matching prediction model can be improved.

A second aspect of the present invention provides a method of visual positioning. The visual positioning method comprises the steps of constructing matching data to be identified using the image and map data to be positioned - the matching data to be identified includes a plurality of groups of point pairs, and two points of each group of point pairs to be positioned derived from image and map data - ; performing prediction processing on a plurality of groups of point pairs using a matching prediction model to obtain predictive matching values of the point pairs; and determining a pose parameter of an image pickup device of an image to be positioned based on the predictive matching value of the point pair.

Accordingly, the matching data to be identified is constructed using the image and map data to be positioned, the matching data to be identified includes a plurality of groups of point pairs, and two points of the point pair of each group are respectively obtained from the image and map data to be positioned. Therefore, using a matching prediction model, prediction processing is performed on a plurality of groups of point pairs to obtain a predicted matching value of the point pair, and further, the pose parameter of the imaging device of the image to be positioned based on the predicted matching value of the point pair By determining , the accuracy and immediacy of visual positioning is improved.

Here, the determining of the pose parameter of the imaging device of the image to be positioned based on the predictive matching value of the point pair may include: arranging a plurality of groups of point pairs in a descending order of the predictive matching value; and determining a pose parameter of an image pickup device of an image to be positioned by using a previously preset number group of point pairs.

Therefore, since a plurality of groups of point pairs are arranged in descending order of predictive matching values, and a pose parameter of an image pickup device of an image to be positioned is determined using a previously preset number group of point pairs, incremental sampling using the sorted point pairs It may be advantageous to preferentially sample a point pair having a high matching value by performing . Therefore, it can be guided geometrically and a priori to obtain the pose parameters, which can improve the accuracy and immediacy of visual positioning.

Here, the matching prediction model is obtained by the training method of the matching prediction model in the first aspect.

Therefore, since visual positioning is performed on the matching prediction model obtained by the training method of the matching prediction model, the accuracy and immediateness of the visual positioning can be improved.

A third aspect of the present invention provides an apparatus for training a matching prediction model. The training apparatus of the matching prediction model includes a sample building module, a prediction processing module, a loss determination module and a parameter adjustment module, wherein the sample building module builds sample matching data using a sample image and map data, but the sample matching data is a plurality of groups of point pairs and actual matching values of each group of point pairs, wherein two points of each group of point pairs are respectively derived from sample images and map data; the prediction processing module performs prediction processing on a plurality of groups of point pairs by using the matching prediction model to obtain prediction matching values of the point pairs; the loss determining module determines a loss value of the matching prediction model using the actual matching value and the predicted matching value; The parameter adjustment module adjusts the parameters of the matching prediction model using the loss values.

A fourth aspect of the present invention provides a visual positioning device. The visual positioning device includes a data building module, a prediction processing module, and a parameter determining module, wherein the data building module is configured to construct matching data to be identified using the image to be positioned and the map data, wherein the matching data to be identified is a plurality of a group of point pairs, wherein two points of each group point pair are derived from image and map data to be positioned, respectively; the prediction processing module performs prediction processing on a plurality of groups of point pairs using the matching prediction model to obtain prediction matching values of the point pairs; The parameter determination module determines a pose parameter of the imaging device of the image to be positioned based on the predictive matching value of the point pair.

A fifth aspect of the present invention provides an electronic device. The electronic device includes a memory coupled to each other and a processor, wherein the processor executes program instructions stored in the memory to implement the training method of the matching prediction model in the first aspect or the visual positioning method in the second aspect.

A sixth aspect of the present invention provides a computer-readable storage medium. The computer-readable storage medium stores program instructions, which, when executed by a processor, implement the training method of the matching prediction model in the first aspect or the visual positioning method in the second aspect.

A seventh aspect of the present invention provides a computer program. The computer program includes computer readable code, and when the computer readable code is executed in an electronic device and is executed by a processor in the electronic device, the method for training a matching predictive model in the first aspect or the second aspect Implement a visual positioning method in the middle.

According to the means, since a matching relationship can be built using the matching prediction model, the matching value between the point pairs can be predicted using the matching prediction model in the visual positioning process. Accordingly, it is possible to establish a matching relationship by preferentially sampling a point pair having a high matching value based on the matching value obtained by prediction, and furthermore, it may be advantageous to improve the accuracy and immediacy of visual positioning.

1 is a flow schematic diagram of an embodiment of a method for training a matching prediction model of the present invention.
2 is a state schematic diagram of an embodiment of a method for training a matching prediction model of the present invention.
3 is a flow schematic diagram of an embodiment of step S11 in FIG. 1 .
4 is a flow schematic diagram of an embodiment of step S111 in FIG. 3 .
5 is a flow diagram of an embodiment of the visual positioning method of the present invention.
6 is a schematic diagram of a framework of an embodiment of a training apparatus for a matching prediction model of the present invention.
7 is a schematic diagram of a framework of an embodiment of the visual positioning device of the present invention.
8 is a schematic diagram of a framework of an embodiment of an electronic device of the present invention.
9 is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present invention.

Hereinafter, the technical solutions of the embodiments of the present invention will be described in detail in conjunction with the drawings in the specification.

In the following description, specific details such as specific system structures, interfaces, and techniques are set forth for purposes of explanation and not limitation, for a thorough understanding of embodiments of the present invention.

In this text, the terms “system” and “network” are often used interchangeably in this text. In the text, the term “and/or” is only for describing the relation of the related object, and means that there are three types of relation. For example, "A and/or B" means three cases in which only A exists, A and B exist simultaneously, and only B exists. In addition, in the text, the symbol “/” generally means that the related object before and after is “or”. In addition, “plural” in the text means two or more than two.

Referring to FIG. 1, FIG. 1 is a flow schematic diagram of an embodiment of a method for training a matching prediction model of the present invention. The training method of the matching prediction model may include the following steps.

In step S11, sample matching data is constructed using the sample image and map data.

In an embodiment of the present invention, the sample matching data includes a plurality of groups of point pairs and actual matching values of each group of point pairs, and two points of each group of point pairs are derived from sample image and map data, respectively.

In one embodiment, the map data may be constructed through a sample image. Here, the dimension to which the sample image belongs may be two-dimensional or three-dimensional, and the dimension to which the map data belongs may be two-dimensional or three-dimensional, but the present invention is not limited thereto. For example, when the sample image is a two-dimensional image, map data such as a sparse point cloud model may be obtained by processing the two-dimensional image by a three-dimensional reconstruction method such as structure from motion (SFM). In addition, the sample image may further include three-dimensional information. For example, the sample image may also be an RGB-D image (ie, a color image and a depth image), but the present disclosure is not limited thereto. The map data may consist of a simple two-dimensional image, a three-dimensional point cloud map, or a combination of a two-dimensional image and a three-dimensional point cloud, but the present invention is not limited thereto.

In an embodiment of the present invention, the execution subject of the training method of the matching prediction model may be a training device of the matching prediction model, which will be described as a training device below; For example, the training method of the matching prediction model may be performed by a terminal device, a server or other processing device, wherein the terminal device is a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, It may be a wireless telephone, a Personal Digital Assistant (PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, or the like. In some possible implementation manners, the training method of the matching prediction model may be implemented by a processor that calls computer readable instructions stored in a memory.

In one embodiment, the sample matching data may be a bipartite graph, also called a bipartite graph. This is an undirected graph composed of a point set and an edge set, a point set can be divided into two subsets without an intersection point, and two points associated with each edge among the edge set belong to the two subsets each without an intersection point. Here, when the sample matching data is a bipartite graph, it includes a plurality of groups of point pairs and a connecting edge connecting the point pairs of each group, and the connecting edge is labeled with an actual matching value corresponding to the point pair, Used to describe the corresponding matching degree, for example, the actual matching value may be a value between 0 and 1; Here, when the actual matching value is 0.1, it means that the degree of matching between the corresponding point pairs is low, and the probability that the point from the sample image and the point from the map data in the point pair correspond to the same point in space is low. there is; When the actual matching value is 0.98, this may mean that the degree of matching between the corresponding point pairs is high, and the probability that the point from the sample image and the point from the map data in the point pair correspond to the same point in space is high. Referring to FIG. 2, FIG. 2 is a state schematic diagram of an embodiment of a method for training a matching prediction model of the present invention. As shown in FIG. 2 , the left side is sample matching data displayed as a bipartite graph, the upper and lower sides of the bipartite graph are two point sets without intersection points, and connecting the points of the two point sets is a connecting edge, and a connecting edge is labeled with an actual matching value (not shown).

In one embodiment, in order to improve the diversification of the sample matching data, the training device may also perform data enhancement on the sample matching data. For example, the training apparatus may randomly rotate three axes at the coordinates of the three-dimensional point in the sample matching data, or normalize the three-dimensional point in the sample matching data, but the present invention is not limited thereto.

In step S12, prediction processing is performed on a plurality of groups of point pairs using the matching prediction model to obtain predictive matching values of the point pairs.

Continuing to refer to FIG. 2 , for example, when the sample matching data is also displayed as a bipartite graph, the matching prediction model corresponds to the first point feature extraction sub-model corresponding to the dimension to which the sample image belongs, and the dimension to which the map data belongs. It may include a second point feature extraction sub-model and an edge feature extraction sub-model. For example, if the sample image is a two-dimensional image, the map data includes a two-dimensional image, and the first point feature extraction submodel and the second point feature extraction submodel are two-dimensional point feature extraction submodels, The matching prediction model obtained by the above can be used for 2-D-3D matching prediction; Or, if the sample image is a three-dimensional image, and the map data includes a three-dimensional point cloud, the first point feature extraction sub-model and the second point feature extraction sub-model are three-dimensional point feature extraction sub-models, obtained by training The matched prediction model can be used for 3-D matching prediction; Alternatively, when the sample image is a two-dimensional image and the map data is a three-dimensional point cloud, the first point feature extraction sub-model is a two-dimensional point feature extraction sub-model, and the second point feature extraction sub-model is a three-dimensional point feature extraction sub model If it is a model, the matching prediction model obtained by training can be used for two-dimensional and three-dimensional matching prediction; Here, the matching prediction model may be set according to actual application, and the present invention is not limited thereto.

In one embodiment, the training apparatus performs feature extraction on the bipartite graph using the first point feature extraction submodel and the second point feature extraction submodel to obtain the first feature and the second feature; performing feature extraction on the first feature and the second feature using the edge feature extraction sub-model to obtain a third feature; a predictive matching value of a point corresponding to a connecting edge may be obtained using the third feature; 2 shows prediction matching values corresponding to each connected edge in a bipartite graph.

In one embodiment, when the first point feature extraction sub-model and the second point feature extraction sub-model are two-dimensional point feature extraction sub-models, at least one residual block (resblock), for example, one residual It may include a block, two residual blocks, three residual blocks, etc., and each residual block (resblock) is composed of a plurality of base blocks (base block), and each base block (base block) is one It consists of a 1*1 convolutional layer, a batch normalization layer, and a context normalization layer. When the first point feature extraction submodel and the second point feature extraction submodel are three-dimensional point feature extraction submodels, at least one residual block (resblock) and at least one spatial transformation network (eg, t-net); For example, it may include one residual block, two residual blocks, three residual blocks, and the like, but is not limited thereto. There may be one or two spatial transformation networks, where the spatial transformation networks may be located at the beginning and the end of the model, but the present disclosure is not limited thereto. For the structure of the residual block, reference may be made to the structure in the above-described embodiment, and a detailed description thereof will be omitted. The edge feature extraction sub-model may include at least one residual block, for example, one residual block, two residual blocks, three residual blocks, and the like, but is not limited thereto. For the structure of the residual block, reference may be made to the structure in the above-described embodiment, and a detailed description thereof will be omitted.

In step S13, a loss value of the matching prediction model is determined using the actual matching value and the predicted matching value.

In one embodiment, the training apparatus may determine a loss value of the matching prediction model by stating the difference between the actual matching value and the predicted matching value. Here, the training device stats the sum of the difference values between the predicted matching values of all point pairs and the actual matching values, and then using the statistics and the number of all point pairs, the average value of the predicted matching values of all point pairs is predicted by matching It can be calculated as the loss value of the model.

및 실제 매칭 값

을 이용하여 매칭 예측 모델의 제1 손실 값

을 결정하고, 비매칭 포인트 쌍의 예측 매칭 값

및 실제 매칭 값

을 이용하여 매칭 예측 모델의 제2 손실 값

을 결정함으로써, 제1 손실 값

및 제2 손실 값

에 대해 가중 처리를 수행하여 매칭 예측 모델의 손실 값

을 획득할 수 있다. 공식 (1)을 참조하면, In another embodiment, the plurality of groups of matching point pairs may include matching point pairs matched between image points and map points included in at least one group, that is, the image points and map points included in the matching point pairs are spatially is the same point within, and a plurality of groups of matching point pairs may also include mismatching point pairs that do not match between image points and map points included in at least one group, that is, image points and map points included in the non-matching point pair. A point is a different point in space, in which case the training device predicts a matching value of a pair of matching points.

and actual matching values

The first loss value of the matching prediction model using

to determine the predictive matching value of the non-matching point pair.

and actual matching values

The second loss value of the matching prediction model using

By determining the first loss value

and a second loss value

Loss value of the matching prediction model by performing weighted processing on

can be obtained. Referring to formula (1),

……(1)

… … (One)

는 매칭 예측 모델의 손실 값을 나타내고,

는 매칭 포인트 쌍에 대응되는 제1 손실 값을 나타내며,

는 비매칭 포인트 쌍에 대응되는 제2 손실 값을 나타내고,

및

는 각각 제1 손실 값

의 가중치, 제2 손실 값

의 가중치를 나타낸다. In the above formula (1),

represents the loss value of the matching prediction model,

represents the first loss value corresponding to the matching point pair,

represents the second loss value corresponding to the non-matching point pair,

and

are the first loss values, respectively

weight of , the second loss value

represents the weight of

및 비매칭 포인트 쌍의 제2 개수

를 각각 통계하여 매칭 포인트 쌍의 예측 매칭 값과 실제 매칭 값 간의 차이 값 및 제1 개수를 이용하여 제1 손실 값을 결정할 수 있다. 공식 (2)를 참조하면,In one embodiment, the training device also includes a first number of matching point pairs.

and a second number of non-matching point pairs.

The first loss value may be determined by using the first number and the difference value between the predicted matching value and the actual matching value of the matching point pair by stating each . Referring to formula (2),

……(2)

… … (2)

는 제1 손실 값을 나타내고,

는 제1 개수를 나타내며,

는 각각 매칭 포인트 쌍의 실제 매칭 값 및 예측 매칭 값을 나타낸다. In the above formula (2),

represents the first loss value,

represents the first number,

denotes the actual matching value and the predicted matching value of the matching point pair, respectively.

The training apparatus may also determine the second loss value by using the second number and the difference value between the predicted matching value and the actual matching value of the non-matching point pair. Referring to formula (3),

……(3)

… … (3)

는 제2 손실 값을 나타내고,

는 제2 개수를 나타내며,

는 각각 비매칭 포인트 쌍의 실제 매칭 값 및 예측 매칭 값을 나타내고; 이 밖에, 비매칭 포인트 쌍의 실제 매칭 값

은 또한 하나의 기설정된 값(예를 들어, 0)으로 통일적으로 설정할 수도 있다. In the above formula (3),

represents the second loss value,

represents the second number,

denotes the actual matching value and the predicted matching value of the non-matching point pair, respectively; In addition, the actual matching value of the non-matching point pair

may also be uniformly set to one preset value (eg, 0).

In step S14, the parameters of the matching prediction model are adjusted using the loss values.

In an embodiment of the present invention, the training device includes stochastic gradient descent (SGD), batch gradient descent (BGD), mini-batch gradient descent (MBGD) etc., and the loss value can be used to adjust the parameters of the matching prediction model; Here, batch gradient descent means updating parameters using all samples in each iteration; Stochastic gradient descent means updating parameters using one sample at each iteration; The mini-batch gradient descent method means that parameters are updated using one batch of samples at each iteration, and a detailed description thereof is omitted here.

In one embodiment, a training termination condition may also be set. When the training termination condition is satisfied, the training apparatus may end training for the matching prediction model. Here, the training termination condition is a condition in which the loss value is less than a preset loss threshold and the loss value does not decrease any more; It may include a condition in which the current number of training reaches a preset number threshold (eg, 500 times, 1000 times, etc.), but is not limited thereto.

According to the above means, sample matching data is constructed and obtained by using the sample image and map data, wherein the sample matching data includes a plurality of groups of point pairs and actual matching values of each group of point pairs, and each group of point pairs The two points of are derived from the sample image and map data, respectively. Therefore, predictive processing is performed on a plurality of groups of point pairs using the matching prediction model to obtain the predicted matching value of the point pair, and further, the loss value of the matching prediction model is determined using the actual matching value and the predicted matching value, Since the parameters of the matching prediction model are adjusted using the loss value, a matching relationship can be built using the matching prediction model, and accordingly, the matching value between the point pairs can be predicted using the matching prediction model in the visual positioning process. Based on the matching value obtained by prediction, it is possible to determine the imaging pose parameter of the image to be positioned by preferentially sampling the point pair having a high matching value, and furthermore, it may be advantageous to improve the accuracy and immediateness of visual positioning.

Referring to FIG. 3 , FIG. 3 is a flow schematic diagram of an embodiment of step S11 in FIG. 1 . Here, the training apparatus may build and acquire sample matching data through the following steps.

In step S111, a plurality of image points are obtained from the sample image and a plurality of map points are obtained from the map data to configure a plurality of groups of point pairs.

The plurality of groups of point pairs includes matching point pairs that match between the image points and map points included in at least one group, that is, the image points and map points included in the plurality of groups of point pairs correspond to the same point in space. At least one group of matching point pairs is included. Taking the case where the sample image is a two-dimensional image and the map data is a sparse point cloud model obtained by the SFM reconstruction method, for example, there is one triangulated point in a plurality of groups of point pairs and the triangulated point in the sparse point cloud model. At least a three-dimensional point corresponding to the point is included.

In one embodiment, the plurality of groups of point pairs may include mismatching point pairs that do not match between the image points and map points included in at least one group, that is, the plurality of groups of point pairs includes images included in at least one group. Non-matching point pairs in which the point and the map point correspond to different points in space may also be included. Again, taking the case where the sample image is a two-dimensional image and the map data is a sparse point cloud model obtained by the SFM reconstruction method, a plurality of groups of point pairs are not triangulated to constitute a group of non-matching point pairs. Any point in point and sparse point cloud models may also be included. Accordingly, noise can be added to the sample matching data, and the robustness of the matching prediction model can be improved.

In one embodiment, referring to FIG. 4 , FIG. 4 is a flowchart of an embodiment of step S111 in FIG. 3 . Here, the training apparatus may acquire a plurality of groups of point pairs through the following steps.

In step S41, an image point in the sample image is divided into a first image point and a second image point.

In an embodiment of the present invention, there is a map point matching the first image point in the map data, and there is no map point matching the second image point in the map data. Again, for example, when the sample image is a two-dimensional image and the map data is a sparse point cloud model obtained by the SFM reconstruction method, the first image point may be a triangulated feature point among the sample images, and the second image point may be non-triangulated feature points in the sample image; In other application scenes, it may be inferred according to this, but the present invention is not limited thereto.

In one embodiment, the image points in the sample image are feature points of the sample image. In another embodiment, the coordinates of the feature points may be transformed into a normalization plane.

In step S42, for each first image point, a plurality of first map points are allocated in the map data, and the first image point and each first map point are each used as one first pair of points, One map point includes a map point that matches the first image point.

For each first image point, assign a plurality of first map points in the map data, use the first image point and each first map point as one first point pair, respectively, the first map point includes A map point matching the first image point is included. In an embodiment, the number of first map points allocated to each first image point may be the same or different. In another embodiment, before allocating the first map point, a plurality of first image points are randomly extracted from the first image point obtained by segmentation, and for the first image point obtained by the extraction, map data may perform the step of allocating a plurality of first map points and using each of the first image point and each first map point as one first point pair, but the present invention is not limited thereto. In one embodiment, N points are randomly extracted from the first image points obtained by segmentation, and, for each first image point of the N first image points obtained by the extraction, K The first map points may be randomly assigned, and the randomly assigned K first map points include map points matching the first image point.

In step S43, for each second image point, a plurality of second map points are allocated in the map data, and the second image point and each second map point are used as one second point pair.

For each second image point, a plurality of second map points are allocated in the map data, and the second image point and each second map point are each used as one second point pair. In an embodiment, the number of second map points allocated to each second image point may be the same or different. In another implementation scene, before allocating the second map point, a plurality of second image points are randomly extracted from the second image point obtained by segmentation, and for the second image point obtained by the extraction, the map data allocating a plurality of second map points and using each of the second image point and each second map point as one second point pair may be performed, but the present disclosure is not limited thereto. In one embodiment, M points are randomly extracted from the second image points obtained by segmentation, and for each second image point among the M second image points obtained by the extraction, K The second map point may be randomly assigned.

In one embodiment, each first point pair and each second point pair are traversed, so as to easily know whether each first point pair and each second point pair are a matching point pair, and a first identifier (eg, 1) may be used to indicate a matching point pair, and a second identifier (eg, 0) may be used to indicate a non-matching point pair.

Step S42 and step S43 may be sequentially performed, for example, first performing step S42 and then performing step S43; or first performing step S43 and then performing step S42; In addition, the steps S42 and S43 may be performed synchronously, but the present invention is not limited thereto.

In step S44, a plurality of groups of point pairs are extracted and obtained from the first point pair and the second point pair.

In an embodiment of the present invention, a plurality of groups of point pairs may be obtained as one sample matching data by randomly extracting the first point pair and the second point pair. In one embodiment, a plurality of sample matching data may be obtained by randomly extracting the first point pair and the second point pair a plurality of times. In another embodiment, a plurality of sample images and map data may be obtained, and the above steps may be repeatedly performed for each sample image and map data to obtain a plurality of sample matching data. Therefore, it is possible to increase the number of samples, which is advantageous in improving the accuracy of the matching prediction model.

In step S112, for each group of matching point pairs, the map point is projected onto the dimension to which the sample image belongs by using the pose parameter of the sample image to obtain the projection point of the map point, and based on the difference between the image point and the projection point to determine the actual matching value of the matching point pair.

For each pair of matching points in each group, the map point may be projected onto a dimension to which the sample image belongs by using the pose parameter of the corresponding sample image to obtain a projection point of the map point. Again, for example, when the sample image is a two-dimensional image and the map data is a sparse point cloud model obtained by the SFM reconstruction method, the training device can obtain a projection point by re-projecting a three-dimensional point using a pose parameter. there is.

In one embodiment, the difference between the image point and the projection point may be converted into a probability density value using a preset probability distribution function to be used as an actual matching value of a matching point pair. In one embodiment, since the preset probability distribution function may be a standard Gaussian distribution function, a difference having a range of negative infinity and positive infinity may be converted into a corresponding probability density value, and the larger the absolute value of the difference, the corresponding The probability density value becomes smaller, which means that the absolute value of the difference is smaller as the matching degree of the point pair is lower, and the smaller the corresponding probability density value, the higher the matching degree of the point pair is correspondingly , and when the absolute value of the difference is 0, the corresponding probability density value is the maximum value.

In one embodiment, before projecting the map point to the dimension to which the sample image belongs by using the pose parameter, the training device may calculate a pose parameter of the sample image based on the pair of matching points; Here, since the pose parameter can be calculated using BA (Bundle Adjustment), the map point is projected onto the dimension to which the sample image belongs by using the pose parameter to obtain the projection point of the map point.

In one embodiment, the actual matching value of the non-matching point pair may be set to a preset value, for example, the actual matching value of the non-matching point pair is set to 0.

Unlike the above-described embodiment, a plurality of image points are obtained from a sample image and a plurality of map points are obtained from map data to configure a plurality of groups of point pairs, and the plurality of groups of point pairs includes at least one image point included in a group. Since matching point pairs matching between and map points are included, samples for training a matching prediction model can be generated, and for each group of matching point pairs, map points are sampled using pose parameters of the sample image. By projecting to the dimension to which it belongs and obtaining the projection point of the map point, the actual matching value of the matching point pair is determined based on the difference between the image point and the projection point, so that the matching prediction model learns the geometrical features of the matching point pair in the training process. Therefore, it is advantageous to improve the accuracy of the matching prediction model.

Referring to Figure 5, Figure 5 is a flow diagram of an embodiment of the visual positioning method of the present invention. The visual positioning method may include the following steps.

In step S51, matching data to be identified is constructed using the image to be positioned and the map data.

In an embodiment of the present invention, the matching data to be identified includes a plurality of groups of point pairs, and two points of each group of point pairs are derived from image and map data to be positioned, respectively. Here, the dimension to which the image and map data to be positioned belong may be two-dimensional or three-dimensional, but the present invention is not limited thereto. For example, the image to be positioned may be a two-dimensional image or the image to be positioned may be an RGB-D image, but the present disclosure is not limited thereto. The map data may consist of a simple two-dimensional image, a three-dimensional point cloud map, or a combination of a two-dimensional image and a three-dimensional point cloud, but the present invention is not limited thereto.

In step S52, prediction processing is performed on a plurality of groups of point pairs using the matching prediction model to obtain predictive matching values of the point pairs.

The matching prediction model may be a neural network model obtained by pre-training through sample matching data. In one embodiment, the matching prediction model may be obtained by training through the steps in the embodiments of the training method for any matching prediction model described above. Here, the training step may refer to steps in the above-described embodiments, and detailed description thereof is omitted herein.

A prediction matching value of a point pair among matching data to be identified may be obtained by performing prediction processing on a plurality of groups of point pairs using the matching prediction model. In one embodiment, the matching data to be identified is a bipartite graph, the bipartite graph includes a plurality of groups of point pairs and a connecting edge connecting each group of point pairs, and the matching prediction model corresponds to a dimension to which the image to be positioned belongs. Since it includes the first point feature extraction submodel, the second point feature extraction submodel corresponding to the dimension to which the map data belongs, and the edge feature extraction submodel, the first point feature extraction submodel and the second point feature extraction submodel are used to obtain the first feature and the second feature by performing feature extraction on the bipartite graph, and performing feature extraction on the first feature and the second feature using the edge feature extraction sub-model to obtain the third feature, Furthermore, a prediction matching value of a point pair corresponding to a connecting edge may be obtained using the third feature. This may refer to steps in the above-described embodiments, and detailed descriptions thereof are omitted herein.

In step S53, a pose parameter of the imaging device of the image to be positioned is determined based on the predictive matching value of the point pair.

Among the matching data to be identified, the pose parameter of the imaging device of the image to be positioned may be determined using a point pair having a high predictive matching value through the predictive matching value of the point pair. In one embodiment, since a perspective-n-point (PnP) problem can be constructed using n point pairs having a high predictive matching value, the PnP problem is solved using a method such as Efficient PnP (EPnP), Furthermore, a pose parameter of an image pickup device of an image to be positioned is acquired. In another embodiment, a plurality of groups of point pairs may be arranged in a descending order of predictive matching values, and a pose parameter of an imaging device of an image to be positioned may be determined using a previously preset number of point pairs of groups. Here, the previously preset number can be set according to the actual situation, for example, using a point pair whose prediction matching value is not 0 in a plurality of sorted point pairs as a point pair of a previously preset number group; Alternatively, a point pair having a predictive matching value greater than a lower limit value in the sorted plurality of groups of point pairs may be used as a point pair of a previously preset number of groups; The previously preset number may be set according to actual application, but is not limited thereto. Here, a pose parameter of an image pickup device of an image to be positioned may be obtained by processing the aligned point pair using a method such as PROSAC (PROgressive sample consensus). In one embodiment, the pose parameter of the imaging device of the image to be positioned may include six degrees of freedom (DoF) of the imaging device in a map coordinate system to which the map data belongs, including a pose, that is, Includes coordinates and deflection yaw around the x-axis, pitch deflection around the y-axis, and roll deflection around the z-axis.

According to the means, the matching data to be identified is constructed using the image to be positioned and the map data, the matching data to be identified includes a plurality of groups of point pairs, and two points of the point pair of each group are respectively the image to be positioned and the Since it is derived from map data, prediction processing is performed on a plurality of groups of point pairs using a matching prediction model to obtain a predicted matching value of the point pair, and furthermore, an image pickup device of an image to be positioned based on the predicted matching value of the point pair By determining the pose parameter of , it is possible to predict a matching value between pairs of points and build a matching relationship using a matching prediction model in the process of visual positioning, which can be advantageous in improving the accuracy and immediacy of visual positioning.

Referring to FIG. 6 , FIG. 6 is a schematic diagram of a framework of an embodiment of the apparatus 60 for training a matching prediction model of the present invention. The training apparatus 60 of the matching prediction model includes a sample construction part 61 , a prediction processing part 62 , a loss determining part 63 and a parameter adjustment part 64 , and the sample construction part 61 is a sample image and construct sample matching data using the map data, wherein the sample matching data includes a plurality of groups of point pairs and actual matching values of each group of point pairs, and two points of each group of point pairs are respectively sample images and map data; the prediction processing part 62 is configured to perform prediction processing on a plurality of groups of point pairs by using the matching prediction model to obtain predictive matching values of the point pairs; the loss determining part 63 is configured to determine a loss value of the matching prediction model using the actual matching value and the predicted matching value; The parameter adjustment portion 64 is configured to adjust a parameter of the matching prediction model using the loss value.

According to the means, since a matching relationship can be built using the matching prediction model, the matching value between the point pairs can be predicted using the matching prediction model in the visual positioning process. Therefore, it is possible to preferentially sample a point pair having a high matching value based on the matching value obtained by prediction, and furthermore, it may be advantageous to improve the accuracy and immediacy of visual positioning.

In some embodiments, the sample building part 61 includes a point pair acquiring sub part, configured to acquire a plurality of image points from a sample image and acquire a plurality of map points from map data to form a plurality of groups of point pairs; , the plurality of groups of point pairs includes matching point pairs that are matched between image points and map points included in at least one group, and the sample building part 61 includes, for each group of matching point pairs, a pose parameter of a sample image a first matching value determining sub, configured to project the map point onto the dimension to which the sample image belongs, to obtain the projection point of the map point, and to determine the actual matching value of the pair of matching points based on the difference between the image point and the projection point includes part.

Unlike the above-described embodiment, a plurality of image points are obtained from a sample image and a plurality of map points are obtained from map data to configure a plurality of groups of point pairs, and the plurality of groups of point pairs includes at least one image point included in a group. Since matching point pairs matching between and map points are included, samples for training a matching prediction model can be generated, and for each group of matching point pairs, map points are sampled using pose parameters of the sample image. By projecting to the dimension to which it belongs and obtaining the projection point of the map point, the actual matching value of the matching point pair is determined based on the difference between the image point and the projection point, so that the matching prediction model learns the geometrical features of the matching point pair in the training process. Therefore, it is advantageous to improve the accuracy of the matching prediction model.

In some embodiments, the plurality of groups of pairs of points include pairs of non-matching points that do not match between the image points and map points included in at least one group, and the sample building portion 61 determines the actual matching values of the pairs of non-matching points. and a second matching value determining sub-portion, configured to set to a preset value.

Unlike the above-described embodiment, a plurality of groups of point pairs includes a mismatching point pair that does not match between an image point and a map point included in at least one group, and unlike the matching point pair, the actual matching value of the mismatching point pair Since is set to a preset value, it may be advantageous to improve the robustness of the matching prediction model.

In some embodiments, the point pair acquiring sub-portion includes an image point dividing part configured to divide an image point in the sample image into a first image point and a second image point, wherein the map data includes a map matching the first image point a point exists, there is no map point matching the second image point in the map data, and the point pair acquisition sub-part allocates, for each first image point, a plurality of first map points in the map data, , a first point pair acquisition portion configured to each use the first image point and each first map point as one first point pair, wherein the first map point includes a map point that matches the first image point and the point pair acquisition sub-part allocates, for each second image point, a plurality of second map points in the map data, and uses the second image point and each second map point as one second point pair a second point pair obtaining portion configured to: wherein the point pair obtaining sub portion includes a point pair extraction portion configured to extract and obtain a plurality of groups of point pairs from the first point pair and the second point pair.

Unlike the above-described embodiment, an image point in the sample image is divided into a first image point and a second image point, a map point matching the first image point exists in the map, and the image data matches the second image point There is no image point to be used, and for the first image point, a plurality of first map points are allocated from the map data, and the first image point and each first map point are each used as one first point pair, , the first map point includes a map point that matches the first image point, for each second image point, assigns a plurality of second map points in the map data, the second image point and each second image point Each map point is used as a second pair of points, and by extracting and obtaining a plurality of groups of point pairs from the first and second point pairs, the number of Since the matching prediction model can be used for training by constructing a plurality of groups of point pairs, it may be advantageous to improve the accuracy of the matching prediction model.

In some embodiments, the first matching value determining sub-portion includes a pose calculating part configured to calculate a pose parameter of the sample image based on the pair of matching points, wherein the first matching value determining sub-portion includes: and a projection portion configured to project the map point onto a dimension to which the sample image belongs to obtain a projection point of the map point.

Unlike the above-described embodiment, the projection point and the projection point are obtained by calculating the pose parameter of the sample image using a pair of matching points, and projecting the map point to the dimension to which the sample image belongs by using the pose parameter. It may be advantageous to improve the accuracy of the difference between image points, and further, it may be advantageous to improve the accuracy of the matching prediction model.

In some embodiments, the first matching value determining sub-portion includes a probability density transformation part configured to convert the difference into a probability density value using a preset probability distribution function to be used as an actual matching value of a matching point pair.

Unlike the above-described embodiment, since the difference is converted into a probability density value using a preset probability distribution function and used as an actual matching value of a pair of matching points, it may be advantageous to accurately explain the difference between the projection point and the image point, and thus It may be advantageous to improve the accuracy of the matching prediction model.

In some embodiments, the sample matching data is a bipartite graph, the bipartite graph comprising a plurality of groups of point pairs and connecting edges connecting each group of point pairs, wherein the connecting edges are labeled with actual matching values corresponding to the point pairs, and ; The matching prediction model includes a first point feature extraction submodel corresponding to a dimension to which the sample image belongs, a second point feature extraction submodel corresponding to a dimension to which map data belongs, and an edge feature extraction submodel, and a prediction processing part 62 ) includes a point feature extraction sub-portion configured to perform feature extraction on the bipartite graph using the first point feature extraction sub-model and the second point feature extraction sub-model, respectively, to obtain the first feature and the second feature, , the prediction processing part 62 includes an edge feature extraction sub-part configured to perform feature extraction on the first feature and the second feature to obtain a third feature by using the edge feature extraction sub-model, the prediction processing part (62) includes a prediction sub-portion configured to obtain a prediction matching value of a point pair corresponding to a connecting edge by using the third feature.

Unlike the above-described embodiment, by performing point feature extraction and edge feature extraction respectively on the bipartite graph, the matching prediction model can more effectively recognize the spatial geometry of the matching, and furthermore, it can be advantageous for improving the accuracy of the matching prediction model. there is.

In some embodiments, the structures of the first point feature extraction sub-model and the second point feature extraction sub-model include a structure including at least one residual block, at least one residual block, and at least one spatial transformation network. any one of the structures; and/or, the edge feature extraction sub-model includes at least one residual block.

Unlike the above-described embodiment, the structure of the first point feature extraction sub-model and the second point feature extraction sub-model is a structure including at least one residual block, at least one residual block, and at least one spatial transformation network. By setting it to any one of the including structures and setting the edge feature extraction sub-model to a structure including at least one residual block, it may be advantageous to optimize the matching prediction model, and also improve the accuracy of the matching prediction model.

In some embodiments, the plurality of groups of pairs of points include matching point pairs that match between image points and map points included in at least one group and pairs of non-matching points between image points and map points included in at least one group that do not match. and the loss determining part 63 includes a first loss determining sub part, configured to determine a first loss value of the matching prediction model using the predicted matching value and the actual matching value of the matching point pair, the loss determining part ( 63) includes a second loss determining sub-part, configured to determine a second loss value of the matching predictive model by using the predicted matching value and the actual matching value of the non-matching point pair, wherein the loss determining part 63 includes the first and a loss weighting sub-portion, configured to perform weighting processing on the loss value and the second loss value to obtain a loss value of the matching prediction model.

Unlike the above-described embodiment, the first loss value of the matching prediction model is determined using the predicted matching value and the actual matching value of the matching point pair, and the matching prediction is made using the predicted matching value and the actual loss value of the non-matching point pair. By determining the second loss value of the model, weight processing is performed on the first loss value and the second loss value to obtain a loss value of the matching prediction model, which will be advantageous for the matching prediction model to effectively recognize the spatial geometry of the matching. and thus improve the accuracy of the matching prediction model.

In some embodiments, the loss determining portion 63 includes: a count statistics sub-portion configured to statistic respectively a first number of matching point pairs and a second number of non-matching point pairs; a first loss determining sub-portion, configured to determine a first loss value using the first number and the difference value between the predicted matching value and the actual matching value of the matching point pair; and a second loss determining sub-portion, configured to determine a second loss value using the second number and the difference value between the predicted matching value and the actual matching value of the non-matching point pair.

Unlike the above-described embodiment, by stating the first number of matching point pairs and the second number of non-matching point pairs, the first number and the difference value between the predicted matching value of the matching point pair and the actual matching value are used to determine the first Since the loss value is determined and the second loss value is determined using the second number and the difference between the predicted matching value and the actual matching value of the non-matching point pair, it may be advantageous to improve the accuracy of the loss value of the matching prediction model, and thus It may be advantageous to improve the accuracy of the matching prediction model.

In some embodiments, the dimension to which the sample image belongs is two-dimensional or three-dimensional, and the dimension to which the map data belongs is two-dimensional or three-dimensional.

Unlike the above-described embodiment, a matching prediction model for 2D-2D is obtained through training by setting the dimension to which the sample image and map data belong, or a matching prediction model for 2D-3D is obtained through training Alternatively, since a matching prediction model for 3D-3D can be obtained through training, the application range of the matching prediction model can be improved.

Referring to FIG. 7 , FIG. 7 is a schematic diagram of a framework of an embodiment of the visual positioning device 70 of the present invention. The visual positioning device 70 includes a data building part 71 , a prediction processing part 72 and a parameter determining part 73 , and the data building part 71 uses the image to be positioned and the map data to identify a match construct data, wherein the matching data to be identified includes a plurality of groups of point pairs, and two points of each group of point pairs are respectively derived from image and map data to be positioned; the prediction processing part 72 is configured to perform prediction processing on a plurality of groups of point pairs by using the matching prediction model to obtain predictive matching values of the point pairs; The parameter determining part 73 is configured to determine a pose parameter of the image pickup device of the image to be positioned based on the predictive matching value of the point pair.

According to the above means, since a matching relationship can be built using the matching prediction model, it is possible to predict the matching value between pairs of points and build a matching relationship using the matching prediction model in the process of visual positioning, so that the accuracy of visual positioning and immediate It can be beneficial for performance improvement.

In some embodiments, the parameter determining portion 73 includes a point pair sorting sub-portion configured to sort the plurality of groups of point pairs in a descending order of predictive matching values, wherein the parameter determining portion 73 includes a previously preset number of groups of points and a parameter determining sub-portion configured to determine a pose parameter of the imaging device of the image to be positioned using the pair.

In embodiments and other embodiments of the present invention, a “part” may be a partial circuit, a partial processor, a partial program or software, and of course, a unit, a module, or a non-module.

Unlike the above-described embodiment, a plurality of groups of point pairs are arranged in a descending order of predictive matching values, a pose parameter of an image pickup device of an image to be positioned is determined using a previously preset number of point pairs of groups, and the aligned point pairs Since it may be advantageous to preferentially sample point pairs with high matching values by performing incremental sampling using

In some embodiments, the matching predictive model is obtained by training using the matching predictive model training device in the above any matching predictive model training device embodiments.

Unlike the above-described embodiment, by performing visual positioning on the matching prediction model obtained through the training apparatus of the matching prediction model in the training apparatus of the arbitrary matching prediction model in the embodiment, the accuracy and immediacy of the visual positioning can be improved. can

Referring to FIG. 8, FIG. 8 is a schematic diagram of a framework of an embodiment of an electronic device 80 of the present invention. The electronic device 80 includes a memory 81 and a processor 82 coupled to each other, the processor 82 comprising a step in any of the above embodiments of the training method of a matching prediction model or any of the above embodiments of the visual positioning method. Execute the program instructions stored in the memory 81 to implement the steps. In one embodiment, the electronic device 80 may include a mobile device such as a mobile phone or a matching computer, but is not limited thereto.

Here, the processor 82 controls itself and the memory 81 to implement a step in any of the above training method embodiments of the matching prediction model or a step in any of the above embodiments of the visual positioning method. The processor 82 may also be referred to as a Central Processing Unit (CPU). The processor 82 may be an integrated circuit chip having signal processing capabilities. Processor 82 may also be a general purpose processor, Digital Signal Processor (DSP), Application Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA), or other programmable logic device. , discrete gate or transistor logic devices, discrete hardware components. A general purpose processor may be a microprocessor, or the processor may be any conventional processor or the like. In addition, the processor 82 may be jointly implemented by an integrated circuit chip.

According to the means, since a matching relationship can be built using the matching prediction model, the matching value between the point pairs can be predicted using the matching prediction model in the visual positioning process. Therefore, it is possible to preferentially sample a point pair having a high matching value based on the matching value obtained by prediction, and furthermore, it may be advantageous to improve the accuracy and immediacy of visual positioning.

Referring to FIG. 9, FIG. 9 is a schematic diagram of a framework of an embodiment of a computer-readable storage medium 90 of the present invention. The computer-readable storage medium 90 stores program instructions 901 executable by a processor, the program instructions 901 executing a step in the embodiment of the training method of any matching prediction model or implementing the any visual positioning method Implement the steps in the example.

According to the above means, since a matching relationship can be established using the matching prediction model, the matching value between the point pairs can be predicted using the matching prediction model in the visual positioning process. Therefore, it is possible to preferentially sample a point pair having a high matching value based on the matching value obtained by prediction, and furthermore, it may be advantageous to improve the accuracy and immediacy of visual positioning.

In some embodiments provided herein, it is to be understood that the disclosed methods and apparatus may be implemented in other ways. For example, the device embodiments described above are merely exemplary, and for example, the partition of a module or unit is only a logical function partition, and there may be other partitioning methods in actual implementation, for example, the unit or component is another may be integrated into the system of, or some features may be omitted or not implemented. Further, the described or discussed coupling or direct coupling or communication connection between each other may be through some interface, and the indirect coupling or communication connection of a device or unit may be electrical, mechanical, or other form.

A unit described as a separation member may or may not be physically separated, and a member denoted as a unit may or may not be a physical unit, and may be located in one place or distributed across multiple network units. can According to the actual demand, some or all of the units may be selected to implement the purpose of the solution of the present embodiment.

In addition, in each embodiment of the present invention, each functional unit may be integrated into one processing unit, or each unit may physically exist separately, and two or more units may be integrated into one unit; The integrated unit may be implemented in the form of hardware or may be implemented in the form of hardware and software functional units.

When the integrated unit is implemented in the form of a software functional unit and sold or used as a separate product, it may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention essentially or a part contributing to the prior art or a part of the technical solution may be implemented in the form of a software product, and the computer software product is stored in a single storage medium. stored, including some instructions, that may cause one computer device (which may be a personal computer, server or network device, etc.) or processor to perform all or some steps of the method according to each embodiment of the present invention there is. The above-described storage medium includes various media capable of storing program codes, such as a USB memory, an external hard drive, a read-only memory (ROM), a random access memory (RAM), a diskette, or a CD.

In an embodiment of the present invention, since a matching relationship can be built using the matching prediction model, the matching value between the point pairs can be predicted using the matching prediction model in the visual positioning process. Therefore, it is possible to establish a matching relationship by preferentially sampling a point pair having a high matching value based on the matching value obtained by prediction, and furthermore, it may be advantageous to improve the accuracy and immediacy of visual positioning.

Claims (18)

A method for training a matching prediction model, comprising:
constructing sample matching data using sample image and map data, wherein the sample matching data includes a plurality of groups of point pairs and actual matching values of each group of point pairs, wherein the two points of each group of point pairs are respectively derived from the sample image and the map data;
performing prediction processing on the plurality of groups of point pairs using a matching prediction model to obtain predictive matching values of the point pairs;
determining a loss value of the matching prediction model using the actual matching value and the predicted matching value; and
and adjusting a parameter of the matching prediction model by using the loss value. According to claim 1,
The step of constructing sample matching data using the sample image and map data includes:
obtaining a plurality of image points from the sample image and obtaining a plurality of map points from the map data to configure a plurality of groups of point pairs, wherein the plurality of groups of point pairs includes at least one image point and a map point included in a group contains matching point pairs that match between - ; and
For each pair of matching points in each group, project the map point to the dimension to which the sample image belongs by using the pose parameter of the sample image to obtain a projection point of the map point, and between the image point and the projection point A method of training a matching prediction model, comprising determining an actual matching value of the pair of matching points based on a difference. 3. The method of claim 2,
The plurality of groups of point pairs includes a non-matching point pair that does not match between image points and map points included in at least one group, and the step of constructing sample matching data using the sample image and map data comprises:
The method of training a matching prediction model further comprising the step of setting the actual matching value of the non-matching point pair to a preset value. 4. The method of claim 2 or 3,
The step of obtaining a plurality of image points from the sample image and obtaining a plurality of map points from the map data to configure a plurality of groups of point pairs comprises:
dividing an image point in the sample image into a first image point and a second image point; the map data includes the map point matching the first image point, and the map data includes the second image point and no matching map point exists;
allocating, for each said first image point, a plurality of first map points in said map data, and using said first image point and each said first map point each as a first pair of points; the first map point includes the map point matching the first image point;
allocating, for each second image point, a plurality of second map points in the map data, and using the second image point and each second map point as one second point pair; and
and extracting and obtaining a plurality of groups of point pairs from the first point pair and the second point pair. 5. The method according to any one of claims 2 to 4,
The step of obtaining the projection point of the map point by projecting the map point to a dimension to which the sample image belongs by using the pose parameter of the sample image includes calculating the pose parameter of the sample image based on the pair of matching points. step; and projecting the map point to a dimension to which the sample image belongs by using the pose parameter to obtain a projection point of the map point; and
The step of determining the actual matching value of the matching point pair based on the difference between the image point and the projection point includes converting the difference into a probability density value using a preset probability distribution function to actually match the matching point pair including the step of using it as a value; At least one of the training methods of the matching prediction model. 6. The method according to any one of claims 1 to 5,
the sample matching data is a bipartite graph, wherein the bipartite graph includes a plurality of groups of point pairs and a connecting edge connecting each group of point pairs, wherein the connecting edge is labeled with an actual matching value corresponding to the point pair; the matching prediction model includes a first point feature extraction submodel corresponding to a dimension to which the sample image belongs, a second point feature extraction submodel corresponding to a dimension to which the map data belongs, and an edge feature extraction submodel;
Acquiring the prediction matching value of the point pair by performing prediction processing on the plurality of groups of point pairs using the matching prediction model,
performing feature extraction on the bipartite graph using the first point feature extraction sub-model and the second point feature extraction sub-model, respectively, to obtain first features and second features;
obtaining a third feature by performing feature extraction on the first feature and the second feature using the edge feature extraction sub-model; and
and obtaining a prediction matching value of a point pair corresponding to the connected edge by using the third feature. 7. The method of claim 6,
The structures of the first point feature extraction submodel and the second point feature extraction submodel include a structure including at least one residual block, a structure including at least one residual block, and at least one spatial transformation network. any one of the structures comprising; and
wherein the edge feature extraction sub-model includes at least one residual block; At least one of the training methods of the matching prediction model. 8. The method according to any one of claims 1 to 7,
the plurality of groups of point pairs include matching point pairs matching between image points and map points included in at least one group and non-matching point pairs not matching between image points and map points included in at least one group;
The step of determining the loss value of the matching prediction model by using the actual matching value and the predicted matching value includes:
determining a first loss value of the matching prediction model by using the predicted matching value and the actual matching value of the matching point pair;
determining a second loss value of the matching prediction model by using the predicted matching value and the actual matching value of the non-matching point pair; and
and obtaining a loss value of the matching prediction model by performing weighting on the first loss value and the second loss value. 9. The method of claim 8,
Before determining the first loss value of the matching prediction model by using the predicted matching value and the actual matching value of the matching point pair, the training method of the matching prediction model includes:
The method further comprising: stating the first number of matching point pairs and the second number of non-matching point pairs, respectively;
The step of determining the first loss value of the matching prediction model by using the predicted matching value and the actual matching value of the matching point pair,
determining the first loss value using the first number and a difference value between the predicted matching value and the actual matching value of the matching point pair;
Determining the second loss value of the matching prediction model by using the predicted matching value and the actual matching value of the non-matching point pair,
and determining the second loss value by using the second number and a difference value between the predicted matching value and the actual matching value of the non-matching point pair. 10. The method according to any one of claims 1 to 9,
The dimension to which the sample image belongs is two-dimensional or three-dimensional, and the dimension to which the map data belongs is two-dimensional or three-dimensional. A visual positioning method comprising:
constructing matching data to be identified using the image to be positioned and the map data, wherein the matching data to be identified comprises a plurality of groups of point pairs, and two points of each group of point pairs are respectively the image to be positioned and the map. Derived from data - ;
performing prediction processing on the plurality of groups of point pairs using a matching prediction model to obtain predictive matching values of the point pairs; and
and determining a pose parameter of an imaging device of the image to be positioned based on the predictive matching value of the point pair. 12. The method of claim 11,
The step of determining a pose parameter of the imaging device of the image to be positioned based on the predictive matching value of the point pair comprises:
sorting the plurality of groups of point pairs in a descending order of the predictive matching values; and
and determining a pose parameter of an imaging device of the image to be positioned by using the pair of points in a previously preset number group. 13. The method of claim 11 or 12,
The matching prediction model is a visual positioning method obtained by using the training method of the matching prediction model according to any one of claims 1 to 10. A training apparatus for a matching prediction model, comprising:
A sample building module for constructing sample matching data by using a sample image and map data, wherein the sample matching data includes a plurality of groups of point pairs and actual matching values of each group of point pairs, and two points of each group of point pairs are derived from the sample image and the map data, respectively;
a prediction processing part configured to perform prediction processing on the plurality of groups of point pairs by using a matching prediction model to obtain predictive matching values of the point pairs;
a loss determining portion, configured to determine a loss value of the matching prediction model by using the actual matching value and the predictive matching value; and
and a parameter adjustment part configured to adjust a parameter of the matching prediction model by using the loss value. A visual positioning device comprising:
Construct matching data to be identified with the image to be positioned and the map data, wherein the matching data to be identified includes a plurality of groups of point pairs, and two points of each group of point pairs are derived from the image to be positioned and the map data, respectively. become;
the prediction processing part is configured to perform prediction processing on the plurality of groups of point pairs by using a matching prediction model to obtain prediction matching values of the point pairs;
The parameter determining part is configured to determine a pose parameter of the imaging device of the to-be-positioned image based on the predictive matching value of the point pair. An electronic device comprising a memory and a processor coupled to each other, the electronic device comprising:
The processor executes the program instructions stored in the memory to implement the training method of the matching prediction model according to any one of claims 1 to 10 or the visual positioning method according to any one of claims 11 to 13. A computer-readable storage medium having program instructions stored thereon, comprising:
The program instructions, when executed by a processor, implement a method for training a matching predictive model according to any one of claims 1 to 10 or a method for visual positioning according to any one of claims 11 to 13. A computer program comprising computer readable code, comprising:
14. When the computer readable code is executed in an electronic device and executed by a processor in the electronic device, the method for training a matching prediction model according to any one of claims 1 to 10 or to claim 11 to 13 A computer program that implements a visual positioning method according to

Images