JP2022149076A - Position estimation device, method and program - Google Patents

Abstract

To estimate a position of a query image on the basis of a position and a direction of its similar image.SOLUTION: A query image acquisition unit 10 acquires a query image Iq. A similar image extraction unit 20 extracts a similar image which is similar to the query image Iq from a database 2. A clustering unit 30 performs clustering into a plurality of image groups for a large number of similar images on the basis of its position. An image group selection unit 40 calculates dispersion of directions of a similar image for each image group, and selects a portion of image groups in which the dispersion is small. A position calculation unit 50 calculates a position of the query image on the basis of a position of the similar image included in the image group for each selected image group. A position determination unit 60 evaluates likelihood of a position of the query image calculated for each image group to determine a maximum likelihood position as the position of the query image.SELECTED DRAWING: Figure 1

Description

The present invention relates to a position estimation device, method and program, and more particularly to a position estimation device, method and program for estimating the position based on the position and orientation of an image similar to an image whose position is to be estimated.

The 3D positions of characteristic points (image feature points) in landscape images are stored in a database in advance, and the camera image taken by the user with a mobile terminal is used as a query image to calculate the degree of similarity with the landscape image in the database. A technique for estimating the position of a mobile terminal based on the position and orientation of a landscape image with a high degree of accuracy is known as, for example, VPS (Visual Positioning Service/System).

Patent Document 1 discloses a technique for estimating the position of an image having a high degree of similarity with an input image as the position of the input image.

Patent Document 2 discloses a position estimation method that uses both local feature amounts and global feature amounts so that local similarity does not become dominant when estimating the position and orientation of an object based on an image. ing.

Patent Document 3 discloses a system that continuously grasps the degree of similarity between the input image and the reference pixels in a system that estimates the position of the input image by comparing the input image with a large number of reference pixels associated with shooting positions. is disclosed.

Japanese Patent Application Laid-Open No. 2011-215716 Japanese Patent Application Laid-Open No. 2018-036770 JP 2020-008984 A

Patent Document 1 only searches for images similar to the input image. In Patent Document 2, the similarity is determined only by the similarity of the local feature amount and the global feature amount of the image, so even if the shooting position and orientation are different, if the feature amount is similar, it is determined that the image is at the same position. put away. In Patent Document 3, similar images are referred to when performing position calculation, but it is simply stated that priority is given to images with higher similarity, and differences in positions are not taken into consideration.

In this way, all of the techniques disclosed in the above documents extract each image on the assumption that even if multiple similar images are extracted, each image means roughly the same location as the query image. , there is a technical problem that the calculation results are affected when similar images are mixed but at different positions.

An object of the present invention is to solve the above technical problems by selecting an image group to be used for estimating the position of a query image by considering not only the similarity of the image features but also the similarity of the shooting position and direction. To improve the accuracy of position estimation by preventing position estimation based on similar images photographed at different positions or similar images of different subjects photographed.

To achieve the above object, the present invention provides a position estimation device for estimating the position of a query image based on the positions and orientations of its similar images, wherein means for extracting a plurality of similar images similar to the query image; means for classifying a plurality of similar images into a plurality of image groups based on the similarity of photographing positions and directions; Means for selecting a small subset of images and means for estimating the position of the query image based on the position of each similar image in the selected image group.

According to the present invention, a group of images used for estimating the position of a query image is selected in consideration of not only the similarity of image features but also the similarity of shooting positions and directions. Therefore, it is possible to prevent the position of the query image from being estimated based on images with similar image features but taken at different positions, or images with similar image features but different subjects taken at different positions. , it is possible to improve the accuracy of position estimation.

1 is a functional block diagram showing the configuration of a position estimation system to which the present invention is applied; FIG. It is a functional block diagram showing the configuration of a position estimation device. FIG. 10 is a diagram showing a method of selecting an image group to be used for estimating the position of a query image; FIG. 10 is a diagram showing a method of obtaining the variance of the orientation of an image; FIG. 10 is a diagram showing an example of an index for evaluating the likelihood of similar images; It is a flow chart showing the operation of one embodiment of the present invention.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a functional block diagram showing the configuration of the main parts of a position estimation system to which the present invention is applied. It is configured.

The image DB2 manages a large number of landscape images used to generate the VPS map, along with position information and orientation information where the images were taken. A user who wishes to estimate his/her own position takes a picture of the surrounding scenery with the camera function of a mobile terminal 3 such as a smartphone, and sends a position request attached with this camera image as a query image Iq to the position estimation device 1 via a network. do.

The position estimation device 1 extracts a landscape image similar to the query image Iq attached to the position request from the image DB 2, and estimates the position of the mobile terminal 3 based on the position information and orientation information associated with the landscape image. presume. The position estimation result is returned to the mobile terminal 3 via the network. Part or all of the position estimation device 1 and the image DB 2 may be installed in the mobile terminal 3 .

FIG. 2 is a functional block diagram showing the configuration of the main parts of the position estimation device 1, which includes a query image acquisition unit 10, a similar image extraction unit 20, a clustering unit 30, an image group selection unit 40, a position calculation unit 50, A position determination unit 60 and a position response unit 70 are main components.

Such a position estimation device 1 can be configured by installing an application (program) that implements each function in at least one general-purpose computer or server equipped with a CPU, ROM, RAM, bus, interface, and the like. Alternatively, a part of the application can be configured as a dedicated machine or a single-function machine that is made into hardware or software.

The query image acquisition unit 10 acquires the query image Iq attached to the location request received from the mobile terminal 3. FIG. This query image Iq is provided to the similar image extraction unit 20 and the position determination unit 60. FIG.

The similar image extraction unit 20 extracts feature points and their image feature amounts from the query image Iq, and extracts each feature point extracted in advance from each landscape image stored in the image DB 2 and whose feature amount is calculated. Calculate the similarity based on the image feature quantity. Alternatively, the degree of similarity may be determined by cosine distance or the like based on global feature values such as GeM (Generalized-Mean pooling). A landscape image having a relatively or absolutely large number Nfc of feature points whose degree of similarity exceeds a predetermined threshold is extracted as a similar image.

In the present embodiment, the top N best (relative) landscape images with a large number of feature points Nfc exceeding the threshold, or all (absolute) landscape images with a feature amount number Nfc exceeding a predetermined threshold Nfc_ref are regarded as similar images. extracted.

The clustering unit 30 classifies a large number of extracted similar images into a plurality of image groups (clusters) based on the closeness of their three-dimensional positions. Alternatively, the three-dimensional rotation may be converted to an angle (one variable) on the horizontal plane, and clustering may be performed using a four-dimensional variable in which three-dimensional position coordinates are added.

The image group selection unit 40 selects a partial image group to be used for estimating the position of the query image from a large number of image groups. In this embodiment, the image group selection unit 40 includes an orientation variance calculation unit 41, and calculates the variance of the orientations of similar images for each selected image group.

The image group selection unit 40, as shown in FIG. 3, excludes an image group in which the dispersion of image orientations is relatively or absolutely large, and selects the remaining image group for estimating the position of the query image. In the illustrated example, since the variance of the orientation of the images clustered in image group C is significantly larger than the variance of the orientation of the images clustered in image groups A and B, image group C is excluded and image group A , B are selected as image groups Im _k (k is an image group identifier) for position estimation.

FIG. 4 is a diagram for explaining a method of calculating the variance of the orientation of an image. If the components of the quaternion representing the direction of the image are w, x, y, and z, and the unit direction vector V representing the direction of the image is projected onto the XZ plane, the elements can be calculated by the following equations (1) to (3). .

vx = 2* _x *z + 2*y*w (1)
v _z = 1 - 2*x*x - 2*y*y (2)
V = (v _x , v _z ) (3)

The angular difference th with the X-axis can be expressed by the inner product of vectors, and since the unit vector of the x-axis with a length of 1 is (1,0), it can be calculated by the following equations (4) and (5).

cost = vx (4)
th = arc_cos(costh) (5)

Let V _sum be the _sum of the unit vectors of each similar image included in each image group Im _k , and let th _sum be its angle. The angle difference th _diff with respect to the direction vector is obtained by the following equation (6).

th _diff = th _sum - th (6)

Then, let thdiff ∈Imk be the difference in angle th between the images included in each image group _Imk , and if the variance VARk is less than or equal to a predetermined threshold, it is determined that the image group has a small orientation variance and is oriented in a certain direction. do.

The position calculation unit 50 calculates the position of the query image for each selected image group _Imk based on the similar images clustered in the image group _Imk . In this embodiment, the projection relationship between each feature point of the query image Iq and each corresponding feature point (corresponding point) of each similar image I _i (where i is a similar image identifier) is estimated as, for example, a homography matrix.

Next, a reprojection error when each feature point of the query image _Iq is projected onto each similar image Ii based on the projection relation is calculated as, for example, a Euclidean distance. Furthermore, using a known library such as Ceres Solver (http://ceres-solver.org/), the minimization problem with the reprojection error as the objective function is solved to determine the position of the query image Iq from the image group Im _k Calculate for each

The position determining unit 60 includes a feature point counting unit 61, a bias calculating unit 62, and a _cost difference calculating unit 63 that calculate index values used to evaluate the likelihood of similar images. The position of the image Iq is evaluated based on each index value, and the final position of the query image Iq is determined based on this evaluation result.

As shown in FIG. 5, the feature point counting unit 61 calculates the number of feature points similar to each feature point of the query image Iq in each similar image _Ii for each of the selected image groups _Imk , and further sums the number of feature points Count N3d _k .

As shown in FIG. 5, the bias calculation unit 62 projects the feature points similar to the feature points of the query image Iq in each similar image _Ii for each of the selected image groups _Imk onto the query image Iq. Compute the variance Dtd _k of the points on the query image. The higher the variance Dtd _k is, the more the feature points are dispersed on the image. It becomes an index for evaluation.

The cost difference calculation unit 63 obtains the average value of reprojection errors when each feature point of the query image Iq is projected onto each similar image _Ii for each of the selected image groups _Imk as the initial cost _Cinitk . Further, the average value of reprojection errors when each feature point of the query image Iq is projected onto the query image whose position is estimated by solving the minimization problem by the position calculation unit 50 is obtained as the post-minimization cost Cop _k . . Then, the cost difference Cdiff _k between the initial cost Cinit _k and the post-minimization cost Copt _k is calculated. The cost difference Cdiff _k is an index for determining whether or not the minimization problem has been successfully solved, and it can be determined that the position of the query image has been calculated more accurately as the cost difference Cdiff _k increases.

The position determining unit 60 evaluates the likelihood of a position higher for an image group with a larger total sum N3d _k , an image group with a larger variance Dtd _ik , and an image group with a larger difference Cdiff _ik based on an evaluation function that evaluates the likelihood of a position. , evaluate the position of the query image Iq computed for each group of images Im _k .

In the present embodiment, each index value is normalized so that the maximum value is 1, and for example, in order to be weighted and evaluated, the above three index values are applied to the following equation (7) to determine the likelihood. and determine the position of maximum likelihood to be the position of the query image Iq.

( _Cdiffk ×0.9+ _N3dk ×0.1)× _Dtdik (7)

The location response unit 70 transmits the determined location of the query image Iq to the mobile terminal 3 as a response to the location request.

FIG. 6 is a flowchart showing the operation of one embodiment of the present invention. In step S1, the query image acquisition unit 10 acquires the query image Iq attached to the position request received from the user. In step S2, the similar image extraction unit 20 extracts the top N landscape images having the highest degree of similarity with the query image Iq from among the large number of landscape images accumulated in the image DB2 as similar images.

In step S3, the clustering unit 30 clusters the N similar images into a plurality of image groups based on their positions. In the present embodiment, the explanation is continued assuming that clustering is performed by the k-means method.

In step S4, the orientation variance calculator 41 calculates the orientation variance of the similar image for each image group. In step S5, the image group selection unit 40 eliminates a part of the image group whose orientation variance is absolutely or relatively large, and selects the remaining image group as the image group _Imk to be used for position estimation.

In step S6, the position calculation unit 50 calculates the position of the query image Iq that minimizes the reprojection error for each selected image group _Imk . In step S7, the feature point counting unit 61, the bias calculating unit 62, and the cost difference calculating unit 63 calculate the feature point sum N3d _k , the position variance Dtd _k , and the cost difference Cdiff _k for each image group Im _k . be.

In step S8, the feature point sum N3d _k , the position variance Dtd _k and the cost difference Cdiff _k are applied to the above equation (7) to calculate the likelihood of the position calculated for each image group Im _k . In step S9, the position of maximum likelihood is determined to be the final position of the query image Iq. In step S10, the determined position of the query image Iq is transmitted to the mobile terminal 3. FIG.

According to this embodiment, a group of images used for estimating the position of a query image is selected in consideration of not only the similarity of image features but also the similarity of shooting positions and directions. Therefore, it is possible to prevent the position of the query image from being estimated based on images with similar image features but taken at different positions, or images with similar image features but different subjects taken at different positions. , it is possible to improve the accuracy of position estimation.

In addition, according to the above-described embodiments, it is possible to provide highly accurate position estimation at low cost via communication infrastructure, so that various services can be provided to many people overcoming geographical or economic disparities. become. As a result, the UN-led Sustainable Development Goals (SDGs) include Goal 9 “Build resilient infrastructure and promote inclusive and sustainable industrialization” and Goal 11 “Make cities inclusive, safe and resilient.” and make it sustainable.

1... Position estimation device, 2... Image database (DB), 3... Mobile terminal, 10... Query image acquisition unit, 20... Similar image extraction unit, 30... Clustering unit, 40... Image group selection unit, 41... Orientation variance calculation Part, 50... Position calculation part, 60... Position determination part, 61... Feature point counting part, 62... Deviation calculation part, 63... Cost difference calculation part, 70... Position response part

Claims (9)

In a position estimation device that estimates the position of a query image based on the positions and orientations of its similar images,
means for extracting a plurality of similar images similar to the query image;
means for classifying the plurality of similar images into a plurality of image groups based on their positions;
means for calculating the variance of the orientations of similar images included in the image group for each image group, and selecting a part of the image group with an absolute or relatively small variance;
and means for estimating the position of the query image based on the position of each similar image in the selected image group. 2. The position estimation device according to claim 1, wherein said means for extracting similar images extracts similar images having an absolute or relatively large number of feature points similar to each feature point of the query image. The means for estimating the position of the query image comprises:
means for calculating the position of a query image based on the position of a similar image included in each selected image group;
3. The position estimation device according to claim 1, further comprising means for determining the position of the query image based on the likelihood of the position of the query image calculated for each image group. The means for calculating the position of the query image estimates a projection relationship between the query image and each similar image for each of the selected image groups, and calculates the position of the query image that minimizes the reprojection error based on the projection relationship. 4. The position estimation device according to claim 3, wherein: The means for determining the position obtains the sum of the number of feature points of each similar image that are similar to the feature points of the query image for each of the selected image groups, and based on the similar images of the image group with the larger sum 5. The position estimating device according to claim 4, wherein the likelihood of the calculated position is evaluated higher. The means for determining the position projects the feature points of each similar image that are similar to the feature points of the query image to the query image for each of the selected image groups, and determines the similarity of the image group having a larger variance of the projected points. 6. The position estimating device according to claim 4, wherein the plausibility of the position calculated based on the image is evaluated higher. The means for determining the position obtains the sum of differences between the position of the query image that minimizes the reprojection error and the position of each similar image before the minimization for each of the selected image groups, and the sum is more than 7. The position estimation device according to any one of claims 4 to 6, wherein the plausibility of positions calculated based on similar images of a large group of images is evaluated more highly. In a position estimation method in which a computer estimates the position of a query image based on the positions and orientations of its similar images,
Extract multiple similar images similar to the query image,
classifying the plurality of similar images into a plurality of image groups based on their positions;
calculating the variance of orientations of similar images included in the image group for each image group;
Selecting a partial group of images in which the variance is absolutely or relatively small;
A position estimation method, comprising estimating the position of a query image based on the position of each similar image in the selected image group. In a location estimation program that estimates the location of a query image based on the locations and orientations of its similar images,
extracting a plurality of similar images similar to the query image;
classifying the plurality of similar images into a plurality of image groups based on their positions;
a procedure for calculating, for each image group, the variance of orientations of similar images included in the image group;
A procedure of selecting a partial group of images with the absolute or relatively small variance;
and estimating the position of a query image based on the position of each similar image in the selected image group.

Images