WO2020197495A1

WO2020197495A1 - Method and system for feature matching

Info

Publication number: WO2020197495A1
Application number: PCT/SG2020/050159
Authority: WO
Inventors: Miaolong Yuan; Zhengguo Li; Kong Wah Wan; Wei Yun Yau
Original assignee: Agency for Science Technology and Research Singapore
Current assignee: Agency for Science Technology and Research Singapore
Priority date: 2019-03-26
Filing date: 2020-03-24
Publication date: 2020-10-01
Anticipated expiration: 2021-09-26
Also published as: SG11202110473TA; WO2020197494A1

Abstract

A method of verifying a plurality of match candidates between a first image and a second image, wherein a match candidate comprises a first feature in the first image and a second feature in the second image, wherein the first and second features are hypothesised to be projections of the same point in three -dimensional space, the method comprising: for each of the plurality of match candidates: determining an individual similarity transformation between a position of the first feature in the first image and a position of the second feature in the second image; and mapping the determined individual similarity transformation into a Hough space; partitioning the Hough space into a plurality of partitions; determining a plurality of groups, wherein a group is comprised of all of the match candidates with individual similarity transformations that fall into the same partition; for each group, determining a local similarity transformation; and verifying a match candidate by: calculating an error generated by describing the relative positions of the first feature in the first image and the second feature in the second image with one of the determined local similarity transformations, and determining that the error is below an error threshold.

Description

METHOD AND SYSTEM FOR FEATURE MATCHING

Cross Reference to Related Application(s) The present disclosure claims the benefit of Singapore Patent Application No.

10201902701U filed on 26 March 2019 and Singapore Patent Application No.

10201902800S filed on 28 March 2019, which are both incorporated in their entirety by reference herein. Technical Field

The present disclosure relates to visual place recognition, particularly visual place recognition using feature matching between images for robotic navigation. Background

Feature matching serves as a core technique for visual place recognition. Spatial verification is often the final step for checking geometric consistencies in order to eliminate outliers. The traditional RANSAC technique, a hypothesize-and-verify scheme, is still one of the most popular method for spatial verification. However, its performance is poor if the ratio of inliers is too low and the running time grows exponentially with the percentage of the outliers. In robotics, place recognition plays a key role for visual simultaneous localization and mapping (SLAM). It is a challenging task, as the appearance of the environments can change drastically in robot subsequent visits, such as various weather and seasons, lighting conditions or moving objects in dynamic environments. Another issue is that multiple places in an environment may look similar, which is a well-known perceptual aliasing problem.

Feature matching is a core component for place recognition and loop closure detection. Over a decade, the well-known bag-of-words (BoW) technique has been widely used to generate putative correspondences. The BoW method represents an image as a set of visual words, which are obtained by quantizing local feature spaces. It usually considers occurrences of visual features of each image while ignoring their spatial relationships during visual word assignments, thus leading to a large percentage of outliers. Therefore, a subsequent spatial verification is required to filter out outliers which are not geometrically consistent. Random sample consensus (RANSAC), a hypothesize-and-verify scheme, is still one of the most popular methods. RANSAC is an iterative method to estimate parameters of a mathematical model from a set of observed data that contains outliers, in which outliers are to be accorded no influence on the values of the estimates. Therefore, it also can be interpreted as an outlier detection method. However, its matching performance is poor when the ratio of inliers is too low and the running time grows exponentially with the percentage of the outliers.

In recent years, Hough voting schemes have received considerable attention. By exploiting local feature shapes, a correspondence between two images can be related by a similarity or an affine transformation. The transformations of each group of correspondences (one cluster) in a Hough transformation space exhibit a property of similarity. By using Hough voting, similar transformations are likely fall into the same bins and the most promising transformations can thus be identified directly in the Hough space. This computation is linear-time in the number of putative

correspondences. It itself boosts spatial verification. The major advantage of the Hough voting strategy is that it is extremely fast while maintaining matching performance. Although Hough voting allows us to realize spatial verification and is extremely fast, the voting scheme still achieves low accuracy compared to the traditional RANSAC- based hypothesize-and-verify scheme. It also remains sensitive to feature detection errors because of the inflexible quantization of correspondences. Recent work shows that better performance can be achieved by incorporating a second-step verification scheme, i.e., vote-and-verify. The purpose of the vote-and- verify scheme is to estimate a single, global similarity/affine transformation for second-step spatial verification based on the most promising transformations identified by Hough voting. However, this method still fails to correctly identify many matches. Therefore, in order to address or alleviate at least one of the aforementioned problems and/or disadvantages, there is a need to provide an improved method for feature matching between images.

Summary

In a first aspect, a method of verifying a plurality of match candidates between a first image and a second image, wherein a match candidate comprises a first feature in the first image and a second feature in the second image, wherein the first and second features are hypothesised to be projections of the same point in three-dimensional space, the method comprising: for each of the plurality of match candidates:

determining a similarity transformation between a position of the first feature in the first image and a position of the second feature in the second image and mapping the determined individual similarity transformation into a Hough space; in a Hough space; partitioning the Hough space into a plurality of partitions; determining a plurality of groups, wherein a group is comprised of all of the match candidates with similarity transformations that fall into the same partition; for each group: determining a local similarity transformation; and verifying a candidate match by: calculating an error generated by describing the relative positions of the first feature in the first image and the second feature in the second image with one of the determined local similarity transformations, and determining that the error is below an error threshold. A match candidate refers to a pair of image features in the first and second images that are hypothesised to correspond to the same point in 3-D space. They are putative correspondences between the first image and the second image. Verifying a match candidate refers to identifying whether the feature in the first image and the feature in the second do, in fact, correspond to the same point in 3-D space, i.e. that the hypothesis is correct.

A similarity transformation may comprise an affine transformation between the location of the feature in the first image and the location of the feature in the second image.

An individual similarity transformation is the similarity transformation between the first feature of the match candidate in the first image and the second feature of the match candidate in the second image. The similarity transformation may be an affine transformation. Note that, in this disclosure, the term feature (as in "first feature" and "second feature") refers to an image feature, or, equivalently, a feature point in an image. The terms feature, image feature and feature point are employed interchangeably herein.

Mapping the similarity transformations into a Hough space, or equivalently, voting the correspondences into Hough space, or identifying the similarity transformations in a Hough space may comprise identifying or mapping the correspondences into the parameter space of the similarity transformations. Partitioning the Hough space into a plurality of partitions may comprise dividing up the parameter space of the similarity transformations, or equivalently, quantizing the parameter space of the similarity transformations. Each dimension of the parameter space may correspond to a different parameter of the similarity transformation. Every dimension of the parameter space may be divided up or only some of the dimensions of the parameter space may be divided up. The dimensions may be divided up equally or differently.

Determining the plurality of groups of correspondences in the Hough space may comprise identifying clusters in the Hough space. A single local similarity

transformation may be determined for each group or cluster of correspondences. A cluster or group may comprise only correspondences falling within a single partition or it may comprise correspondences falling within several neighbouring partitions.

Several correspondences may fall into the same partition but not be considered a cluster or group. Only correspondences falling within particular partitions may be considered a group or cluster.

Verifying a match candidate may comprise ensuring that one of the local similarity transformations adequately represents the relationship between the feature in the first image and the feature in the second image. Adequately representing may mean that the similarity transformation for that match candidate is an inlier of the local similarity transformation. Adequately representing may mean that it the local similarity transformation is applied to the position of the feature in the first, the corresponding position of the feature in the second image is obtained within a margin of error and/or vice-versa.

Determining the groups in the Hough space may comprise calculating a score for each of the partitions. Determining the groups in the Hough space may comprise selecting from those partitions with a score which exceeds a threshold. The score may include contributions from different levels of partitioning of the Hough space. The score may only consider the occupancy of each partition at the finest level of partitioning. The score may be employed in combination with the occupancy of each partition at the finest level of partitioning. The local similarity transformation may comprise an average of all of the match similarity transformations in a group. The local similarity transformation may comprise the mean of all of the similarity transformations in a group. Calculating an error generated by describing the relative positions of the feature in the first image and the corresponding feature in the second image with one of the determined local similarity transformations may comprise calculating a two-way projection error. Calculating an error generated by describing the relative positions of the feature in the first image and the corresponding feature in the second image with one of the determined local similarity transformations may transformations may or may not comprise applying a Random sample consensus (RANSAC) algorithm and determining that the similarity transformation of the match candidate is an inlier of one of the determined local similarity transformations. The Hough space may be partitioned into dimensions corresponding to translation in a first, or x-direction, translation in a second, or y-direction, scale and orientation. The dimension corresponding to orientation may be partitioned into many partitions, or only two partitions or it may not be partitioned at all. The Hough space may be partitioned according to the parameterization of the similarity transformations. Match candidates can be identified by applying a bag-of-words technique to the images. The bag-of-words technique enables efficient identification of match candidates. A pair of image features may be identified if they share the visual word. The first image and the second image may be represented by a histogram of a set of features using a visual vocabulary. The first feature and the second feature may be hypothesized to be the same, and therefore to be a match candidate, by virtue of their being represented by the same feature of the visual vocabulary.

In an aspect, a method of determining if a geographic location is captured in both a first image and a second image is provided, the method comprising: generating a plurality of match candidates between a first image and a second image, wherein a match candidate a first feature in the first image and a second feature in the second image, wherein the first and second features are hypothesised to be projections of the same point in three-dimensional space; for each of the plurality of match candidates: determining a similarity transformation between a position of the feature in the first image and a position of the feature in the second image in a Hough space; partitioning the Hough space into a plurality of partitions; determining a plurality of groups, wherein a group is comprised of all of the match candidates with similarity

transformations that fall into the same partition; for each group: determining a local similarity transformation; and verifying a match candidate by: calculating an error generated by describing the relative positions of the feature in the first image and the corresponding feature in the second image with one of the determined local similarity transformations, and determining that the error is below an error threshold;

aggregating all of the verified match candidates between the first image and the second image; and determining that the same geographic location is captured in both the first image and the second image if the aggregated number of verified matches exceeds a pre-defined threshold.

Determining if a geographic location is captured in both a first image and a second image may comprise identifying a visited place.

A geographic location is captured in both a first and a second image if the same place is captured in a first and second image. The same place may be determined to be captured in the first and second image if the number of verified matches exceeds a threshold.

At least one of the first or second images may be a reference image that forms part of, for example, a stored visual map. At least one of the first or second images may be a reference image that is, for example, stored as part of a database. At least one of the first or second images may not be a reference image and may be captured in real time. At least one of the first or second images may be captured in real time by a camera. Geographic location data of at least one of the first or second images may be available. Geographic location data may comprise information regarding the location from which the image was obtained and/or information regarding the location or locations captured in the image.

Geographic location data of one of the images may be employed to determine the geographic location of the other image.

In an aspect, a method of determining a geographic location of a mobile device is provided, the method comprising: obtaining a first image; retrieving a second, stored image, retrieving details of the geographic location captured in the second image; and determining if the same geographic location is captured in both the first image and the second image by generating a plurality of match candidates between a first image and a second image; for each of the plurality of match candidates: determining a similarity transformation between a position of the feature in the first image and a position of the corresponding feature in the second image in a Hough space; partitioning the Hough space into a plurality of partitions; determining a plurality of groups, wherein a group is comprised of all of the hypothesized matches with similarity transformations that fall into the same partition; for each group: determining a local similarity transformation; and verifying a match candidate by: calculating an error generated by describing the relative positions of the feature in the first image and the corresponding feature in the second image with one of the determined local similarity transformations, and determining that the error is below an error threshold;

aggregating all of the verified matches between the first image and the second image; and determining that the same geographic location is captured in both the first image and the second image if the aggregated number of verified matches exceeds a pre- defined threshold.

The determined geographic location may or may not be employed to alter the trajectory of the device. The mobile device may be a mobile robotic device.

In an aspect, a system for verifying a plurality of matches between a first image and a second image is provided, the system comprising an input for receiving a first image and a second image; and a processor configured to: verify a plurality of match candidates between a first image and a second image, the method comprising: for each of the plurality of match candidates: determining a similarity transformation between a position of the feature in the first image and a position of the

corresponding feature in the second image in a Hough space; partitioning the Hough space into a plurality of partitions; determining a plurality of groups, wherein a group is comprised of all of the hypothesized matches with similarity transforms that fall into the same partition; for each group: determining a local similarity transformation; and verifying a match candidate by: calculating an error generated by describing the relative positions of the feature in the first image and the corresponding feature in the second image with one of the determined local similarity transformations, and determining that the error is below an error threshold. The system may or may not form part of a navigation device. The system may or may not form part of a mobile robotic device. The mobile robotic device may or may not further comprise a camera configured to obtain one or more images for use by the system. In an aspect, a computer readable medium is provided, the computer readable medium being configured to cause a processor to verify a plurality of match candidates between a first image and a second image, the method of verification comprising: for each of the plurality of match candidates: determining a similarity transformation between a position of feature in the first image and a position of the corresponding feature in the second image in a Hough space; partitioning the Hough space into a plurality of partitions; determining a plurality of groups, wherein a group is comprised of all of the hypothesized matches with similarity transformations that fall into the same partition; for each group: determining a local similarity transformation; and verifying a match candidate by: calculating an error generated by describing the relative positions of the feature in the first image and the feature in the second image with one of the determined local similarity transformations, and determining that the error is below an error threshold. The computer readable medium may be tangible or non-tangible.

In an aspect, a computer readable medium is provided, the computer readable medium being configured to cause a processor to determine if a geographic location is captured in both a first image and a second image, the method of determining comprising: generating a plurality of match candidates between a first image and a second image; for each of the plurality of match candidates: determining a match similarity transformation between a position of the feature in the first image and a position of the corresponding feature in the second image in a Hough space;

partitioning the Hough space into a plurality of partitions; determining a plurality of groups, wherein a group is comprised of all of the hypothesized matches with similarity transformations that fall into the same partition; for each group: determining a local similarity transformation; and verifying a match candidate by: calculating an error generated by describing the relative positions of the feature in the first image and the corresponding feature in the second image with one of the determined local similarity transformations, and determining that the error is below an threshold; aggregating all of the verified matches between the first image and the second image; and determining that the same geographic location is captured in both the first image and the second image if the aggregated number of verified matches exceeds a pre- defined threshold. The computer readable medium may be tangible or non-tangible.

In an aspect, a computer readable medium is provided, the computer readable medium being configured to cause a processor to determine a geographic location captured in a first image, the method of determining comprising: obtaining a first image; retrieving a second, stored image, retrieving details of the geographic location captured in the second image; and determining if the same geographic location is captured in both the first image and the second image by generating a plurality of match candidates between a first image and a second image; for each of the plurality of match candidates: determining a match similarity transformation between a position of the feature in the first image and a position of the corresponding feature in the second image in a Hough space; partitioning the Hough space into a plurality of partitions; determining a plurality of groups, wherein a group is comprised of all of the hypothesized matches with similarity transformations that fall into the same partition; for each group: determining a local similarity transformation; and verifying a match candidate by: calculating an error generated by describing the relative positions of the feature in the first image and the corresponding in the second image with one of the determined local similarity transformations, and determining that the error is below an threshold; aggregating all of the verified matches between the first image and the second image; and determining that the same geographic location is captured in both the first image and the second image if the aggregated number of verified matches exceeds a pre-defined threshold. The computer readable medium may be tangible or non-tangible.

In an embodiment, a method for visual recognition in robotic navigation and localisation is provided, the method comprising: acquiring image of space

(environment, surrounding); recognition, determining regions with a similar geometric relationship of local image features by generating correspondence based on visual vocabulary database where each correspondence defines a similarity transformation, identifying local similarity transformations of each group of correspondences using a Hough transform, wherein similarity transformations of correspondences in one group are likely to fall into the same bins while the ones in other groups of correspondences are likely in other bins , and aggregating the inliers for matching and verifying.

In an embodiment, the method further includes identifying the sets of similarity transformations of groups of correspondences in the Hough transformation space if the count of bins is above a predefined threshold, estimating the corresponding local similarity transformations and verifying the corresponding number of local similarity transformations against putative correspondences and inliers based on a two-way re projection error threshold.

In an embodiment, a method of verifying a plurality of hypothesized matches between a first image and a second image is provided, wherein a hypothesized match comprises a first feature that appears the first image and a second feature that appears in the second image, the first feature and the second feature being hypothesized to be the same, the method comprising:

for each of the plurality of hypothesized matches, determining a match similarity transformation between a position of the first feature in the first image and a position of the second feature in the second image; grouping at least some of the plurality of hypothesized matches into a plurality of groups according to their respective match similarity transformations; for each group, determining a local similarity

transformation; and verifying a hypothesized match by: calculating an error generated by describing the hypothesized match with one of the determined local similarity transformations; and determining that the error is below an error threshold, wherein grouping at least some of the plurality of hypothesized matches into a plurality of groups according to their respective similarity transformations comprises: plotting the determined match similarity transformations in a Hough space; and partitioning the Hough space into a plurality of partitions, and wherein a group is comprised of all of the hypothesized matches with match similarity transforms that fall into the same partition, wherein the Hough space is 4 dimensional, with each dimension

corresponding to one of: translation in a first direction, translation in a second direction, scale and orientation, and wherein the dimension corresponding to orientation is partitioned into no more than two partitions. In an aspect, a method of determining a location in space is provided, the method comprising: capturing a first query image at a first location; capturing a second query image at a second location, wherein a spatial relationship between the first location and the second location is known; comparing the first query image and the second query image with a plurality of stored reference images, wherein a location of each of the stored reference images is known; determining which of the stored reference images most closely matches the first query image; determining which of the stored reference images most closely matches the second query image; determining a similarity transformation between features that appear in the first query image and features which appear in the stored reference image which most closely matches the first query image; determining a similarity transformation between features that appear in the second query image and features which appear in the stored reference image which most closely matches the second query image; and calculating the location of the first image from the spatial relationship between the first location and the second location, the location of the stored reference image that most closely matches the first query image, the location of the stored reference image that most closely matches the second query image, the similarity transformation between features that appear in the first query image and features which appear in the stored reference image which most closely matches the first query image, and the similarity transformation between features that appear in the first query image and features which appear in the stored reference image which most closely matches the first query image. The method may further comprise determining the spatial relationship between the first location and the second location from wheel odometry.

Determining which of the stored reference images most closely matches the first and second query images may comprise: determining a plurality of matches between each of the stored reference image and the respective query image, wherein a match is a feature which appears in both images; and determining which reference image corresponds to the largest number of matches.

Determining a plurality of matches between the each of the stored reference images and the respective query image may comprise, for each stored reference image:

determining a plurality of hypothesized matches between the stored reference image and the respective query image, wherein a hypothesized match comprises a first feature that appears the respective query image and a second feature that appears in the reference image, the first feature and the second feature being hypothesized to be the same; and spatially verifying the hypothesized match.

Spatially verifying the hypothesized match may comprise: for each of the plurality of hypothesized matches, determining a match similarity transformation between a position of the first feature in the query image and a position of the second feature in the reference image; grouping at least some of the plurality of hypothesized matches into a plurality of groups according to their respective match similarity

transformations; for each group, determining a local similarity transformation; and verifying a hypothesized match by: calculating an error generated by describing the hypothesized match with one of the determined local similarity transformations; and determining that the error is below an error threshold, and wherein the similarity transformation between features that appear in the respective query image and features which appear in the stored reference image which most closely matches the respective query image comprises one or more of the determined local similarity transformations

Grouping at least some of the plurality of hypothesized matches into a plurality of groups according to their respective similarity transformations may comprise: plotting the determined match similarity transformations in a Hough space; and partitioning the Hough space into a plurality of partitions, and wherein a group is comprised of all of the hypothesized matches with match similarity transforms that fall into the same partition.

In an aspect, a navigation system for a device is provided, the system comprising: an input for receiving a first query image and a second query image and information regarding where the first and second query images were captured; a memory configured to store a plurality of reference images and information regarding a geographic location of each stored reference image; and

a processor configured to determine the geographic location of the mobile device by performing a method according to an embodiment, wherein the navigation system is configured to alter a trajectory of the mobile device based on the determination by the processor.

Brief Description of the Drawings Methods and systems according to embodiments will now be described with reference to the accompanying figures, in which:

Figure 1 shows a schematic of mobile robotic device comprising a system for feature matching between images according to an embodiment;

Figure 2 shows a flowchart of a method according to an embodiment of feature matching;

Figure 3 shows a simplified illustration of a method of voting correspondences into a Hough space according to an embodiment;

Figure 4 shows a flowchart of a method according to an embodiment of determining if a location is captured in both a first image and a second image;

Figure 5 shows a flowchart of a method according to an embodiment of determining the geographic location of a mobile robotic device; Figure 6 shows a flowchart of a method according to an embodiment of determining a geographic location of a mobile device;

Figure 7 shows a schematic representation of the calculation performed in Figure 6; Figure 8 shows a flowchart of a method according to an embodiment of determining geographic location of a mobile device;

Figure 9 shows a flowchart of a method according to an embodiment of determining geographic location of a mobile device;

Figure 10 shows a flowchart of a method according to an embodiment of determining geographic location of a mobile device;

Figure 11 shows experimental results for a comparison of feature matching results between a method according to an embodiment and methods known in the art;

Figure 12 shows experimental results for a comparison of feature matching results between a method according to an embodiment and a method known in the art; Figure 13 shows experimental results for a comparison of feature matching results between a method according to an embodiment and a method known in the art;

Figure 14 shows experimental results for a comparison of accuracy between a method according to an embodiment and a method known in the art;

Figure 15 shows the experimental results for running time for a method known in the art;

Figure 16 shows experimental results for a method according to an embodiment and a method known in the art; Figure 17 shows experimental results for a method according to an embodiment and a method known in the art;

Figure 18 shows experimental results for a method according to an embodiment and a method known in the art;

Figure 19 shows experimental results for a method according to an embodiment and a method known in the art;

Figure 20 shows experimental results for a method of determining geographic location according to an embodiment;

Figure 21 shows experimental results for a method of determining geographic location according to an embodiment; and

Figure 22 shows experimental results for a method of determining geographic location according to an embodiment.

Detailed Description

For purposes of brevity and clarity, descriptions of embodiments of the present disclosure are directed to a system and method for feature matching, in accordance with the drawings. While aspects of the present disclosure will be described in conjunction with the embodiments provided herein, it will be understood that they are not intended to limit the present disclosure to these embodiments. On the contrary, the present disclosure is intended to cover alternatives, modifications and equivalents to the embodiments described herein, which are included within the scope of the present disclosure as defined by the appended claims. Furthermore, in the following detailed description, specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be recognized by an individual having ordinary skill in the art, i.e. a skilled person, that the present disclosure may be practiced without specific details, and/or with multiple details arising from combinations of aspects of particular embodiments. In a number of instances, well-known systems, methods, procedures, and components have not been described in detail so as to not unnecessarily obscure aspects of the embodiments of the present disclosure.

In embodiments of the present disclosure, depiction of a given element or

consideration or use of a particular element number in a particular figure or a reference thereto in corresponding descriptive material can encompass the same, an equivalent, or an analogous element or element number identified in another figure or descriptive material associated therewith.

References to "an embodiment / example", "another embodiment / example", "some embodiments / examples", "some other embodiments / examples", and so on, indicate that the embodiment(s) / example(s) so described may include a particular feature, structure, characteristic, property, element, or limitation, but that not every embodiment / example necessarily includes that particular feature, structure, characteristic, property, element or limitation. Furthermore, repeated use of the phrase "in an embodiment / example" or "in another embodiment / example" does not necessarily refer to the same embodiment / example.

The terms "comprising", "including", "having", and the like do not exclude the presence of other features / elements / steps than those listed in an embodiment. Recitation of certain features / elements / steps in mutually different embodiments does not indicate that a combination of these features / elements / steps cannot be used in an embodiment. As used herein, the terms "a" and "an" are defined as one or more than one. The use of "/" in a figure or associated text is understood to mean "and/or" unless otherwise indicated. The term "set" is defined as a non-empty finite organization of elements that mathematically exhibits a cardinality of at least one (e.g. a set as defined herein can correspond to a unit, singlet, or single-element set, or a multiple-element set), in accordance with known mathematical definitions. The recitation of a particular numerical value or value range herein is understood to include or be a recitation of an approximate numerical value or value range. The recitation of a particular numerical value or value range herein is understood to include or be a recitation of an approximate numerical value or value range.

In this disclosure, a match candidate refers to a pair of image features in the first and second images that are hypothesised to correspond to the same point in 3-D space. For example, if the corner of a building projected as an image feature in the first image and another image feature in the second image then that this pair of image features is a match. Figure 1 shows a computer system 100 configured to perform feature matching between two images according to an embodiment. The computer system 100 comprises a processor 101, such as a CPU which performs the feature matching according to an embodiment. The system may comprise an input 105 by which it may receive one or more of the images to be to be matched. The input 105 may comprise a camera, a hard drive, a disk reader, an ethernet connection or any other means for receiving an image or data relating to an image. The input 105 may instead or further be configured to receive instructions which cause the processor 101 to perform image matching according to an embodiment. Although only one input 105 is shown in Figure 1, the person skilled in the art will appreciate that the system 100 may have multiple inputs, one or more of which may be configured to received data in the form of images and one or more of which may be configured to receive instructions for the processor 101. In an embodiment, the computer system 100 may also comprise a memory 103. According to embodiments, the memory 103 may be configured to store images. The memory 103 may be further configured to store geographical information about the locations captured in those images. The memory 103 may additionally, or alternatively be configured to store instructions configured to cause the processor to perform a method of feature matching or localization according to an embodiment and/or a visual vocabulary database. Such a database will be discussed below. The computer system 100 may also comprise an output 107 by which data relating to the feature matching performed by the processor 101 is output. Examples of output 107 may include a screen, an external hard drive, or a disk writer, or any other means suitable for outputting data. In an embodiment, the output 107 may include instructions for controlling the movement of a mobile device. In an embodiment, the system 100 forms part of a mobile device 10, such as a mobile robotic device. In this embodiment, the output 107 may comprise the control module for the movement of the mobile device. The mobile device may comprise means (not shown) of determining its location or approximate or relative location such as a GPS module, a processor configured to calculate wheel odometry, etc.

Figure 2 shows the steps a method of verifying matches between the features of two images according to an embodiment. In an embodiment, the images are of the space around a vehicle or robotic device, such as images of the environment or surroundings and/or objects. In an embodiment, the images may be acquired at different points in time, for example the first image may be captured by a robotic device at a first point in time and the second image may be captured after the vehicle has travelled for a period of time. In an embodiment, the image may be a historical image which is stored in the memory 103 as part of a visual map or other visual reference.

Each of the steps of the method of Figure 2 will now be discussed in detail. In step S201, putative correspondences (candidate matches) between the features of two images are determined. In an embodiment, this is done using the well-known bag-of-words (BoW) technique. In other embodiments, other matching techniques could be employed. Examples include (but are not limited to) the brute-force matcher, FLANN (Fast Library for Approximate Nearest Neighbours) Matcher. As discussed above, the BoW method represents an image as a histogram of a set of visual words, which are obtained by quantizing local feature spaces. The bag-of-words approach usually considers occurrences of visual features of each image while ignoring their spatial relationships during word assignments. In an embodiment, this is based on an offline trained vocabulary stored in the memory 103. In an embodiment, the features of an image are first detected using a method known in the art such as ORB which represents the image by a set P of local features, as described by E. Rublee, V. Rabaud, K. Konolige and G. Bradski in "ORB: An efficient alternative to SIFT or SURF", IEEE International Conference on Computer Vision, 2564- 2571, 2011". Each of the local features is then converted to a visual word u. In an embodiment, given two images P and Q, a pair of features p P and q Q are considered as a putative correspondence c = ( p,q ) if they share the same visual word, i.e., u(p ) = u(q ) . In step S203, a similarity transform is determined for each correspondence c = ( p,q ) .

Given a local image feature p P, such as an ORB feature, we assume that its position (x_p, y_p) , scale s , orientation q_r and its visual word u(p ) are given. The local shape and position of each feature can be described by a 3 x 3 matrix given by:

t(p) = (x_p, y_p) is the position in the x, y axis. Intuitively, F(p) defines a similarity transformation from the current image framework to a normalized image patch, centred at the origin (0, 0) with scale s =1 and orientation Q =0.

A similarity transformation (i.e., a relative transformation) from p to q is given by:

Where M(c) = s(c)R(c ) and t(c ) = t(p) -M(c)t(q) , and in which s(c) = s_q / s_p and

R(c ) = R(q)R(p) ¹ denote relative scaling and rotation from p to q, respectively.

In step S205, the similarity transforms, F(c) are clustered. In an embodiment, this is done by mapping the similarity transform for each correspondence in a Hough space and determining which of the correspondences fall into the same bins (thereby forming a cluster) in the Hough space. This process according to an embodiment now be explained in detail. F(c) can be written as a 4D transformation vector by

F(c) = (x(c), y(c), 5(c), q(c)) (3) where 0(c) = 0_{q -} 0_p and (x(c),y(c))^T = t(c ) . It follows that each correspondence c = (p,q) can be mapped as a point in a 4-D Hough transformation space, with each dimension corresponding to one of the parameters of F(c) shown in Equation (3). In order to enable efficient implementation, the parameter space for each of the four parameters (i.e. translation (x(c),y(c)) , scale 5(c) and orientation q(c) ) is independently quantized into n_x, n , n_s and n_q bins, respectively. The

transformation space L may also be partitioned at different resolution levels

with quantization of the parameter space at each resolution level. In an

embodiment, only the finest level l_Q is considered in order to determine inliers as this is the most geometrically consistent.

In order to cluster the correspondences, each of the four parameters is normalized for each correspondence as:

s(c) = (log(s(c) + log(5_max)) / 2 log(5_max) (4) x(c ) = (x(c) + w * 5(c)) / 2 w * 5(c) (5)

y(c ) = ( y(c) + h * 5(c)) / 2 h * 5(c) (6)

9(c) = (9(c) + 2p) / 4p (7)

where w and h are width and height of the image, and 5_max is the maximal scale of the local feature shape. Once the normalized parameters are calculated, their values are then mapped ("voted") into the Hough space.

The score of each bin in Hough space is then calculated. It is expected that the most groups of correspondences at the finest level, i.e., l₀ , will be the most geometrically consistent and thus considered as inliers. In an embodiment, only the score of each bin at the finest level (1=0) is taken into account. The score of the bins at level l₀ are calculated using the following equation:

where b_Ii denotes the count (i.e. occupancy) of the i^th bin at level / and 2^-aI denotes the contributions to the overall score of its corresponding bin at level / .

In an embodiment, a=l. In an embodiment, a cluster is identified where the score of a given bin in Hough space exceeds a threshold p. The value of this threshold will be discussed in detail below. In step S207, a local similarity transform is determined for each identified cluster.

In an embodiment, the local similarity transform for an individual cluster is the mean of all of the similarity transforms in that cluster, i.e. the mean of all of the transforms that were voted into the same bin. Thus, in the example of figure 3, the means of the three F_t(i = 1,...,3) are employed to identify three separate transforms /,(/ = 1,...,3) for a second-step verification and keeping all inliers.

In Step S209, each local similarity transform determined in step S207, is employed to verify the putative correspondences determined in step S201.

In an embodiment, verification is done using a two-way re-projection error threshold d_ep and determining which correspondences are inliers for each individual local similarity transform. Minimizing the reprojection error estimates the errors from the point correspondences between two images. The reprojection error is a geometric error corresponding to the distance between a projected point and the corresponding point on the same image. It is used to quantify how closely an estimate of a feature recreates the point's true projection. In an embodiment, if a correspondence is found to be an inlier for one of the local similarity transforms, it is regarded as verified and the features in the two images are considered to be a match. In another embodiment, RANSAC (see above) is employed for verification instead of or as well as the two-way re-projection error. Thus, in the above described embodiment, Hough voting is employed to determine local similarity transforms, i.e. a plurality of transforms, each of which describes the geometric relationship of a different group of correspondences. Calculating a plurality of local similarity transforms as opposed to a single, global similarity transformation is advantageous because Hough voting is sensitive to mismatches due to uniform quantization or to feature detection errors, as will now be explained.

Considering a 3D point \X, Y, Z] in the world frame,

correspond to its positions in two different local camera frames. The relationship between them is given as

where the rotation matrix and translation vector depend on the movement of a robot and are independent of features. Let ( f_x, f_y ) and (c_x, c_y) be the focal lengths and principle point, respectively, of the first camera in pixels, and let be the focal lengths and principle

point, respectively, of the second camera in pixels. Let be the

positions of the features in the image of the first camera and the image of the second camera, respectively. Using the pin-hole camera model, it can be derived that

where the scale factor , rotation matrix and translation vector are computed

as:

Consider the following two cases: Case 1: values of t_x and t are close to zero. In this case, Equation 13 can be reduced to:

From Equations 10, 11 and 1 it can be concluded that: (1) all correspondences with approximately same depth ratios are grouped together and share one affine transformation; and (2) if the depth ratios of all correspondences are approximately egual, all correspondences share one affine transformation. In other words, a global affine transformation is enough. Otherwise, local affine transformations are reguired. Usually, it is impossible for the depth ratios of all correspondences to always be approximately same. Thus, local affine transformations are desirable for robust feature matching.

Case 2: At least one of t_x and t is different from zero. In this case, two conclusions can be drawn from Equations 11-13 as follows: (1) all correspondences with approximately same depths in two images are grouped together and they share one affine transformation; and (2) if the depths of all correspondences in two images are approximately eguai, all correspondences share one affine transformation. In other words, a global affine transformation is enough. Otherwise, local affine

transformations are reguired.

As we would not expect the depths, nor the depth ratios of most correspondences to be approximately equal, it follows that in the majority of cases, a plurality of different, local, similarity transformations will be required in order to optimally capture all of the correspondences between two images.

The parameters employed in the methods of Figures 2 and 4 are summarised below in Table 1.

Table 1: System parameters As will be appreciated by the person skilled in the art, selection of the values of δ_ep and p_ep according to embodiments affects the sensitivity of the system. In embodiments, the selection of the parameters given above is made experimentally according to operational requirements and the components of the device and/or intended environment in which the device will operate.

The value of p will also affect the system accuracy and reliability. In Hough voting, each correspondence is hard-mapped to a single transformation and therefore actual correspondences might not be identified if their transformations fall into unrelated bins due to feature noises. On the other hand, the transformations of correct correspondences often distribute irregularly due to feature noises. This effect may result in accuracy degradation due to irregular distribution of matches in the transformation space. False positives may also be identified if they fall into a same bin and dominate it. In methods according to the above described embodiments, if the count of a bin is above p_ep , the transformations falling into this bin are considered as being active. A local hypothesis is estimated using the mean of these transformations for verification. In general, as p decreases, number of matches identified will increase because fewer correspondences are required to fall into a bin for a local transformation to be calculated. However, as p decreases more false matches arise, reducing the reliability of the matching method.

In an embodiment, the quantization of the four-dimensional Hough space, parameterized by n_x , n , n_q and n_s , is determined experimentally. In one embodiment, the space is quantized evenly with n_x , n , n_q and n_s set as, for example, 16, 16, 16, and 8, respectively.

In other embodiments, rotation may be largely ignored in the Hough transformation space, as will now be explained. From Equation (10) above, it can be shown that

Where (x^ y,) and (x , y_t) be the positions of features in the image of a first camera and the image of a second camera, respectively.

From Eqn. (9), it can be shown that

Subsequently, it follows that

where the constant matrix and vecto are computed as

It is worth noting that the matrix R is not a rotational matrix. Let the singular value decomposition of be denoted as

The scale factor s(c) , rational matrix R (c) , and translational vector t(c) are then given as

From Eqn. (25) it can be seen that the value of 0(c) is independent of the features. This implies that one bin is enough for the rotational angles 0(c) 's if their values are accurate. In other words, the quantization on 0(c) c an be ignored from a theoretical point of view. However, in practice, noise will arise in the computed values of 0(c) 's, thus, in an embodiment, two bins are adopted for rotation in order to reduce the effect of noise in the computed values of q_q and q_q .

In this embodiment therefore, quantization is only performed on translation

(t _x(c),t (c)) , and scale s(c) for each similarity transformation of each correspondence c = (p, q) , independently. In this embodiment, therefore, example values of n_x , n , n_q and n_s might be 16, 16, 2 and 8, respectively, with n_q = 2 in order to reduce the effect of noise.

Experimental results to support neglecting the rotational angles according to the embodiment described above is shown in Table 2. In these examples, the matching between ten pairs of images were tested using the proposed simplified Hough quantization approach outlined above. Six different choices on the number of bins for the rotation angle were tested, namely 1, 2, 4, 8, 16, and 32 for each pair. The number of matches obtained between the two images of each set are given in Table 2 for each rotational quantization value. Clearly, all six quantization values result in almost the same number of matched pairs in most testing sets. Therefore, it is clear that quantization on the rotation angle can be largely removed from the Hough quantization. In an embodiment, two bins are employed in order to compensate for noise effects.

Table 2: Number of matched pairs for different choices on the number of bins for the rotation angle In an embodiment, local similarity transforms as described above are employed in a method of place recognition. The steps of this method are shown in Figure 4 according to an embodiment. Steps S401-S409 proceed as steps S201-S209, described above in relation to Figure 2. However, they are recapped briefly below:

In step S401, the BoW technique is employed to initialize putative correspondences C based on an offline trained vocabulary. In step S403, each correspondence c is mapped into a 4-D Hough transformation space according to Equation 1.

In step S405, each correspondence c C is voted into the bins in Hough space. There then follows a two-step process for identifying clusters of similarity transformations in the Hough space:

1) In the first step the score (Eqn. 8) is employed to identify a given number of clusters in the Hough space. For example, the clusters corresponding to the 15 largest scores may be identified. 2) In the second step, those clusters with a count b_0i less than a threshold _e

_p are discarded. This ensures that any identified clusters with a very small number of transformations, such as 1 or 2, are discarded as they may be unreliable. Following these steps, transformations of groups of correspondences

in the Hough transformation space H are identified from those

clusters that have not been discarded.

In Step S407, the corresponding N local similarity transformations

related to each set F_i(i are calculated using their means.

In Step S409, for each local the putative correspondences C are verified

and a set of inliers C_i is output determined using a two-way re-projection error threshold

In Step S413, the sets of inliers for set are aggregated using

In Step S415, If the number of C^* is above a threshold it is determined that a

visited place is found. Determining a visited place according to embodiments described above may be advantageous for use in the self-recovery of robotic devices. For example, a robotic device might get lost due to abrupt wheel slippage, unsteady movements on uneven floors, collision with obstacles, blocked perception sensors or kidnapping. When this happens, the robot robotic device must recover from the incident by itself.

Determining a visited place as described above can be used to self-recover the lost robot with the use of lased-based metric maps and visual place recognition solutions. These will now be described in detail. Figure 5 shows a method of altering the trajectory of a mobile device according to an embodiment.

In step S1601, a first image is obtained. In an embodiment, this image is obtained by taking a photo with a camera 105 built into the device.

In step S1603, a second image is retrieved. In an embodiment, this is an image which is stored in the memory 103 of the device. For example, the image may form part of a visual map stored in the mobile device. Geographical data regarding the location captured in the image is also retrieved in this step. In an embodiment, this data is stored in the memory 103. In step S1605, the method of Figure 4 is employed to determine if the first image corresponds to a visited place.

In an embodiment, the steps of S1603 and S1605 may be repeated for a plurality of stored images until a visited place is found.

Thus, as soon as a robotic device detects that it is lost, self-exploration can be employed in order to enable the lost device to explore the vicinity while performing place recognition according to embodiments described above. Once the device determines during self-exploration that a visited place is found, the proximate global location can then be obtained using stored data regarding the visited place and the device can re-localize according to the position stored in a database.

While methods according to the above described embodiments enable a mobile device to determine which of a set of reference images most closely corresponds to its present location, in almost all cases, there will be a gap between the location of the mobile device and that of the reference image. The sparser the set of reference images, the less memory they will take up but the more imprecise the localization will be. In order to overcome this tension between the need for precise localization while minimizing memory requirements, a method for determining a precise three- dimensional location from sparse map reference images is provided according to an embodiment. Figure 6 shows a method of determining the 3-D position, or pose, of a device using visual place recognition according to an embodiment.

In step S1201, at least two images, are captured by the device, each in

a slightly different position.

In step S1203, the three-dimensional relationship between the position of each of the captured images is determined. In an embodiment, this comprises a 3-D rotation matrix and a 3-D translation vector, i.e., In an embodiment, these

values are determined by wheel odometry. Note that there is no scale ambiguity with regard to t_csince its value can be computed exactly using the corresponding robot poses.

In step S1205, the captured images are compared with the selected

vocabulary and matched with a reference image R . In practice, this process consists

of determining feature matches between the image and the stored reference

images and determining which reference image produces the largest number of matches. In an embodiment, the feature matching is performed according to the method of Figure 5. As discussed above, feature matching comprises determining putative correspondences between images and then spatially verifying the correspondences. In order to spatially verify the correspondences, similarity transformations between features are determined. In step S1207, these similarity transformations are employed to compute the 2D rotation matrix and the 2D translation vector between the pair of images

In step S1209, the 3D rotation matrix and the 3D translation vector between and

are be computed from the image poses which are stored in the visual map, i.e., When the reference images are the same,

become the identity matrix and the zero vector, respectively. Note that there is no scale ambiguity regarding to

since its value can be computed exactly using the location information obtained during capture.

In Step S1211, the 3-D relationship between and

the two dimensional relationship between

employed to determine the three dimensional relationship between

The 3D geometrical relationship between features in the four images

can be expressed as:

Where The values of s_k-l and

S_k are usually determined by considering all or part of the above four equations via the least mean square.

In step S1213, the position of the reference image and the 3-D relationship

between are employed to determine the position in 3-D space of

thereby determining the exact 3-D pose of the device.

As is well known in the art, as only 2D information is available from images, a scale ambiguity may arise when converting to 3-D space. For example, the (x,y,d) and (2x,2y,2d) are matched to the same point in an image. The more query images are obtained and matched with reference images, the more accurate a scale estimation can be carried out and therefore the accuracy of the pose determined in step S1213 can be improved.

The method of Figure 6 is schematically represented in Figure 7. The known geometric relationship 1501 between reference images and the known geometric relationship 1503 between query images in order to determine the geometric relationship 1505 between query images and reference images.

As will be clear to the person skilled in the art, in order to implement the method of Figures 6 and 7, it is necessary to include the 3D positions of feature points in the world frame of each key frame of the visual map. In an embodiment, ORB-SLAM is applied to construct the visual map complete with 3-D positions of features. A description of ORB-SLAM is given in Raul Mur-Artal, J. M. M. Montiel, Juan D. Tardos, ORB-SLAM: a Versatile and Accurate Monocular SLAM System, IEEE TRO, 2015. In an embodiment, pose information from AMCL may also be incorporated into the ORB- SLAM to improve its accuracy. Advantageously, this reduces the scale ambiguity of the

ORB-SLAM. The method of determining the 3D position of the query image according to embodiments advantageously enables the number of reference images stored as part of a visual map to be reduced, thereby reducing memory requirements. In an embodiment, the pose determination using visual place recognition (VPR) according to the embodiment of Figures 6 and 7 is employed as part of a framework for localization in combination with Adaptive Monte Carlo Localization (AMCL) and/or wheel odometry. Table 3 shows the different characteristics of VPR, AMCL and wheel odometry.

Table 3

As shown in Table 3, VPR is more complex than AMCL and wheel odometry due to the potential for large numbers of correspondences. However, in other areas such as dynamic obstacles and drift, VPR is advantageous over AMCL and wheel odometry. In embodiments described below, one or more of these techniques is combined in order to exploit the strengths of each. A method of pose estimation employing VPR according to an embodiment is employed in combination with other techniques is shown in Figure 8. In step S1401, the location of a device is estimated using AMCL or wheel odometry. In an embodiment, the most reliable is chosen to estimate the location.

In step S1403, VPR is performed to determine the exact location. In an embodiment, the location determined in step S1401, is used as the searching neighborhood for reference images and in order to determine the vocabulary for use with VPR.

In step S1405, the pose is corrected using a PnP based method. Perspective-n-Point (PnP) methods are known in the art and are the process of estimating the pose of a calibrated camera given a set of n 3D points in the world and their corresponding 2D projections in an image.

In an embodiment, a VPR method according to an embodiment is employed in order to enhance AMCL. Such an embodiment is shown in Figure 9. In step S1501, the location of a device is determined according to AMCL. In step S1503, the location of the device is determined using VPR with pose correction according to an embodiment. In an embodiment, the location determined in step S1401, is used as the searching neighborhood for reference images and in order to determine the vocabulary for use with VPR, as described above.

In step S1505, the starting pose of the AMCL is reset using the result obtained in step

S1503.

In an embodiment, a VPR method according to an embodiment is employed in order to enhance wheel odometry. Such an embodiment is shown in Figure 10.

In step S2601, the location of a device is determined according to wheel odometry.

In step S2603, the location of the device is determined using VPR according to an embodiment. In an embodiment, the location determined in step S2601, is used as the searching neighborhood for reference images and in order to determine the vocabulary for use with VPR, as described above.

In step S2605, the starting pose of the wheel odometry is reset using the result obtained in step S2603.

Comparison with prior-art methods The performance of methods of feature matching according to embodiments will now be compared with that of methods known in the art. Feature matching was implemented in a machine with an Intel(R) Core(TM) i7-4770R CPU @ 3.20GHz/16GB RAM under Ubuntu 14.04. It included a BoW place recognition module for initializing putative correspondences using a visual vocabulary. The ORB detector was used to detect up to 2000 keypoints in each frame. The ORB detector was selected as it enables features to be extracted extremely fast with local feature shape information including position, orientation and scale. The visual vocabulary was generated offline using a hierarchical vocabulary tree technique with the ORB descriptors extracted from an independent large set of images. The size of the vocabulary used was 142M. The person skilled in the art would appreciate that if the images are general enough, the same vocabulary can be used for various environments without scarifying any performance.

The scale factor in the ORB detector was set as 1.2 and there were 8 scales, calculated by s_j = s_l-l * 1.2, i =1, ...8 for each octave. Thus, the maximal scale s_max was 3.2. The 4D transformation space was quantized evenly, by setting n_x , n , n_q and n_s as 16, 16, 16, and 8, respectively, in all cases. 5 levels were set for a hierarchical partition of the transformation space and -a in the level-dependent contributions in Equation 8 was set to 1.0. Before voting, each correspondence was converted into the Hough transformation space and normalized using Equations 4-7. The effect of the scale of the keypoints on the normalization of the translation in similarity transformation space was taken into account, in accordance with Equations 5-6. Figure 11 shows a comparison between feature matching according to an embodiment of the present invention (Figure 11(a)) and feature matching using conventional vote- and-verify (Figure 11(b)) and RANSAC (Figure 11(c)) -based methods for spatial verification on a dataset known as the Oxford City Centre dataset. Matching is performed between two images 505 and 507. The similarity transformations identified for corresponding features in the two figures by each method are indicated by straight lines between the relevant features in each figure. In Figure 11(a), the similarity transformations of the two groups of correspondences 501 and 503 (denoted by white ellipses) fall into two bins at the finest level and they were most likely two different groups of transformations in two local regions. In contrast, only one global

transformation was identified by the vote-and-verify approach, as shown in Fig. 11(b), namely the transformation corresponding to correspondences 501. Similarly, only the transformation of correspondences 503 was identified using RANSAC, as shown in Figure 11(c). Thus, in both vote-and-verify and RANSAC, a group of correspondences was filtered out. As discussed above, the transformations in one local region are coherent and can be approximated by their means. Hence, if a single global transformation is used for verification, such as in vote and verify and RANSAC, some correspondences in other regions are also be filtered out by the global one. Figures 12 shows a comparison of feature matching using local hypotheses verification (Figure 12(a)) and global hypothesis verification (Figure 12(b)). Matching is performed between two images 607 and 609. The similarity transformations identified for corresponding features in the two figures by each method are indicated by straight lines between the relevant features in each figure. As can be seen in Figure 12(b), using global verification verification, all of the correspondences 601 and most of the correspondences 603 were filtered out, as the global transformation was represented by the large part of the correspondences 605 on the right side, thus reducing the matching performance. These examples indicate the fact that one global

transformation such as an affine transformation or even an 8-DOF fundamental matrix might not be sufficient for high accurate feature matching. Advantageously, as shown in Figure 12(a), local hypothesis verification, as employed in the embodiments described above, enabled some sparse features to be identified (such as those in region 601). The identification of such sparse features may be important in enabling a robot to navigate and localize itself.

Figures 13(a) and (b) likewise show a comparison of feature matching using local hypotheses verification and global hypothesis verification, respectively. Matching is performed between two images 707 and 709. The similarity transformations identified for corresponding features in the two figures by each method are indicated by straight lines between the relevant features in each figure. As can be seen from these figures, the sparse landmark features 701, 703 and 705 circled by ellipses in Fig. 13(a) can be identified using the methods according to embodiments described above, which can be used for robot navigation and localization. However, these features were filtered out when only one global hypothesis was used for spatial verification as shown in Figure 13(b). Such features may therefore not be identified in methods that employ global verification such as the vote-and-verify method.

The accuracy and computation time of methods according to embodiments described above were further benchmarked against existing solutions on several public datasets. First, methods according to embodiments described above were compared with three state-of-the-art place recognition methods in robotics, namely FABMAP2.0, ORB-SLAM and HTMAP. As different feature detectors are used in FABMAP2.0, ORB-SLAM and HTMAP, methods according to embodiments were compared with them in three aspects: (1) comparison with RANSAC in terms of computation time and accuracy using the same feature detector and the same vocabulary in BoW. This is because both RANSAC and BoW techniques were used in these three methods. (2) comparison in terms of maximum recall at a 100% precision. (3) Finally, comparison with the vote- and-verify method, which is known in the art. The evaluation was conducted on five popular datasets including Oxford City Centre and the so-called New College datasets and three KITTI datasets. The City Centre and the New College datasets include 2474 and 2146 images of size 640 x 480, respectively. The ground truth poses of the robot were derived from interpolated GPS. These two datasets are very challenging as the City Centre dataset was captured in a highly dynamic outdoor environment and the New College dataset includes high perceptual aliasing conditions. We selected three sequences KITTI 00, KITTI 05, and KITTI 06, including 4541, 2761 and 1101 images of size 1241x376, for experimental comparison. The KITTI sequences do not provide a ground truth for loop closure detection. The ground truth is determined by means of a 0-1 binary matrix where the rows and columns correspond to the images at different time indices and an element is set to 1 if a loop closure is detected.

(1) Comparison with RANSAC

In this section, the computation time and accuracy of a feature matching method according to an embodiment is compared with a BoW place recognition module in which a traditional RANSAC-based spatial verification is embedded. The default parameter setting was employed, and an 8-DOF fundamental matrix was iteratively estimated in order to identify an optimal one for final verification.

In the following, the precision and recall of a matching technique are defined based on the number of true positives (TP) (i.e. putative correspondences correctly identified to be matches), true negatives (TN) (putative correspondences correctly identified not to be matches), false positives (FP) (putative correspondences incorrectly identified to be matches), and false negatives (FN) (putative correspondences incorrectly identified not to be matches), defined as:

Recall =#TP/(#TP +#FN) (31)

Precision =#TP/(#TP +#FP ) (32)

For comparison, the same values of the threshold t are employed in order to detect whether a loop is found.

Turning now to execution times of spatial verification, Table 3 shows the average running time in ms of each stage on five datasets where 2000 ORB features were extracted. The datasets employed were City Centre, New College, KITTI 00, KITTI 05, and KITTI 06. In the experiments, the same values of the system parameters were set for both the ORB detector and BoW with the same vocabulary.

For each image, there are three major stages that require time to process, i.e., feature detection, initialization of correspondences using BoW and spatial verification. The times for extracting 2000 features were around 16ms, while times to initialize putative correspondences using the BoW method were around 12ms. After initializing the putative correspondences, the final task is geometric consistency checking in order to eliminate outliers. The last two columns in Table 4 show the execution times for spatial verification using a method according to an embodiment ("Our Method") and RANSAC. It can be observed that the execution times of the method according to an

embodiment were less than 2ms and much faster than the RANSAC-based method. The reason is that in methods according to the above described embodiments, the hypotheses for verification can be identified directly in the Hough space while it requires a certain large amount of computational time to iteratively search for a "best" transformation for verification using RANSAC technique. Especially for the KITTI datasets, the RANSAC spatial verification required more than 100ms to iteratively search for an optimal global transformation for verification. This is because KITTI datasets are largely static environments and therefore large number of the putative correspondences are initialized, resulting in a large amount of computational time to iteratively estimate a hypothesis.

Table 4.

As mentioned above, the performance of RANSAC is poor if the ratio of inliers is too low. Further, the running time of RANSAC grows exponentially with the percentage of the outliers. It is therefore necessary to tune the number of iterations in order to obtain a trade-off between better performance and computational time. However, it is difficult to know the actual percentage of the outliers in advance. Hence, the number of iterations is normally set as sufficiently large as possible. With the increase of the number of iterations, the accuracy will increase correspondingly, as shown in Fig. 14.

In Figure 14(a), line 1001 shows the results for RANSAC on the City Centre dataset while line 100S shows the results for RANSAC on the New College dataset. Stars 1005 and 1007 show corresponding results for a method according to an embodiment for the City Centre and New College datasets, respectively.

In Figure 14(b), line 1009 shows results for RANSAC on the KITTI 00 dataset while line 1011 shows the results for RANSAC on the KITTI 06 dataset and 1013 the results for RANSAC on KITTI 05. Stars 1015, 1017 and 1019 show corresponding results for a method according to an embodiment for KITTI 05, KITTI 06 and KITTI 00, respectively.

Note that as multiple hypotheses can be identified directly in Hough transformation space, no iterations are required in methods according to embodiments. Hence, the recalls are plotted as stars onto the Recall-axis directly. It can be seen that even after 1000 iterations of RANSAC, the recall results using the method according to an embodiment still outperformed those of RANSAC. Further, with the increase of the number of iterations, the running times for RANSAC increased correspondingly, as shown in Figure 15 (reference numerals unchanged from Figure 14). It can be observed that even when only 100 iterations were used, 16.51ms, 7.929ms, 132.91ms,

137.603ms and 114.5ms was still required for verification on the five datasets. In particular, more than 100ms was still required for the KITTI datasets. Hence, it is clear that the running time of RANSAC is environment-dependant. For different applications, it may be possible to tune this parameter for better performance for RANSAC-based approaches. In methods according to the present embodiment, however, no such tuning is necessary as the hypotheses are identified directly in Hough space. (2) Comparison with FABMAP2.0, ORB-SLAM and HTMAP

To compare with FABMAP2.0, ORB-SLAM and HTMAP with methods according to embodiment, the maximum recall that can be obtained under 100% precision was considered. This corresponds to no false positive being detected in each dataset. This is important that in robotics as false positive may result in an inconsistent map or localization/re-localization failures. In addition, the loop closure acceptance threshold, t_er , was set as 75, 125, 80, 140 and 125 for the New College, City Centre, KITTI 00,

KITTI 05 and KITTI 06 datasets, respectively, to identify whether a loop was found. Table 5 shows the maximum recall at a 100% precision for FABMAP2.0, ORB-SLAM, HTMAP and a method according to an embodiment ("Ours") for all five datasets. It should be noted that in the results of Table 3, the same values of

and ground truth have been used for evaluation. High recall was obtained for the method according to an embodiment for all three KITTI datasets. The recall was above 96% for all five datasets under a 100% precision condition. In particular, the recall for KITTI 00 was close to 100%. This is because the environments used for the KITTI datasets were relatively static and with few dynamical objects, resulting in very high recalls. In contrast, the maximum recalls for the more challenging New College and City Centre datasets were 84.39% and 62.57%, respectively. The results show that the recalls obtained from the method according to an embodiment were better than the ones obtained from the stare— of-the-art solutions with the exception of the recall on the City Centre using HTMAP (79.69% vs 62.57%). However, the method according to the present embodiment is more than three times faster than HTMAP.

Table 5

(3) Comparison with the Vote-and-Verify method

The major advantage of methods based on Hough voting is that they are extremely fast while still maintaining high accuracy. Indeed, the computational time of Hough voting based methods can almost be ignored. In this section, therefore, methods according to embodiments are compared with the vote— and-verify approach in terms of accuracy. The same values of the parameters are employed for comparison using the challenging Oxford City Centre and New College datasets. Table 6 shows the maximum recall at a 100% precision for the vote-and-verify method and the method according to an embodiment with p =6 . From the experiments conducted on the City Centre and New College datasets, rates of 62.57% and 84.39% were obtained, respectively. In particular, the recall of the method according to an embodiment was almost 10% higher than the vote-and-verify approach on the New College dataset.

Table 6 Figs. 16 and 17 show the effect of t on recall using the City Centre and New College datasets respectively, with the vote-and-verify method shown by lines 1301 and the method according to an embodiment shown by lines 1303. As can be seen, even though the value of t was increased from 70 to 200 for the City Centre dataset and from 120 to 220 for the New College dataset, the average recalls obtained from the method according to an embodiment were 3.23% and 10.96% higher than the recalls obtained from the vote-and-verify method using the City Centre and the New College datasets, respectively.

The impact of the parameter p , i.e, the threshold of the count of bins to identify local similarity transformation in Hough space when comparing the method according to an embodiment and vote-and-verify is shown in Figures 18 and 19. Results for the vote-and-verify method are shown by lines 1501 and 1505 on the City Centre and New College datasets, respectively. Results for a method according to an embodiment are shown by lines 1503 and 1507 for the for the City Centre and New College datasets, respectively. As discussed above, in embodiments, if the count of a bin is above p_ep , the transformations which fall into this bin are considered to be active and a local hypothesis is estimated using the mean of these transformations for verification. As the value of p_ep increases, and fewer and fewer bins will meet the required score, and it follows multiple local hypotheses verification will eventually converge to a single global hypothesis verification approach, i.e., the vote-and-verify approach. Fig. 18 shows that when p_ep is above 11 for the City Centre dataset or above 34 for the New

College dataset, the recalls of both the method according to an embodiment and the vote-and-verify method were the same. The vote-and-verify method is therefore a special case of methods according to embodiments. Note that when 5 £ p £ 12 for the City Centre dataset and 5 < p_ep < 35 for the New College dataset, the method according to an embodiment outperformed the vote-and-verify method. Fig 19 shows the precision of each technique as it varies with p (lines 1503 and 1507 are labelled with values of p ). It can be seen from Fig. 13 that if p_ep < 5, although the recall still increased using the method according to an embodiment, the precision also decreased, meaning that were more false matches. In particular when p_ep =1 for the City Centre dataset, the precision decreased drastically. In contrast, using one global hypothesis for final verification in the vote-and— verify approach, even when p is small, there were no false positives using City Centre and New College datasets.

However, the recalls were less than those obtained using the method according to an embodiment.

Figures 20 to 22 show experimental results using pose refinement according to embodiments described above. In these examples, the performance was evaluated using the Kitti benchmark suite 2012 dataset 00, 02, and 08, shown in Figures 20, 21 and 22, respectively. The dataset had the ground truth pose for each captured image measured by a GPS system, such that it was able to minimize the error of

transformation 7> and T_c. Two image sequences (1382 x 512 pixels) from the center and right grayscale camera of the dataset were used in the experiment. The center image sequence was taken to train as reference images together with the associated ground truth pose, and the right image sequence was used for testing the result of pose estimate and refinement. To build up the visual map, four different configurations were set up: taking the reference image for every 3rd, 5th, 10th, 15th frame from the dataset. The more frequently the reference image is taken from the database, the higher rate of visual matching can be achieved. In addition, for pose refinement, five different choices of consecutive matched pairs were applied to estimate the scale factor (N=2, 3, 4, 5 or 6).

The experimental results are shown in Fig. 22, where root mean square error (RMSE) is adopted to measure the accuracy. The bar labelled as DBOW indicates the RMSE of the matched VPR pose against the ground truth without pose refinement, i.e. this result indicates the difference between the position of the mobile device and the reference image that is determined to be its closest match. Bars labelled with their

corresponding value of scale factor (N= x) show the results of pose estimation according to embodiments described above. Figure 22 shows that the accuracy of pose estimation is indeed improved by employing pose refinement according to an embodiment. Furthermore, the scale ambiguity decreases as more matched pairs (N) are adopted in Eqn. 30. The pose refinement method according to an embodiment is also compared with the PnP-based approach indicated by the bars labelled "PNP. In order to estimate the feature points in 3D, two past consecutive images were used to triangulate and form a point cloud. Then the 2D corresponding feature points in the current image were matched to the 3D point using RANSAC method to minimize re projection error. This is different from conventional methods in which the 3D positions of feature points associated with the keyframe are obtained by using the bundle adjustment. Although a better result is achieved by examining the RMSE value, as shown in Table 7, the total number of frames that the PnP-based approach was able to perform pose estimation for was much lower than the pose refinement approach according to an embodiment.

Note that the overall performance of pose estimation improves as more reference images are taken from the dataset. This is because, when the visual map is denser, there is a higher change that the query image will be matched by the VPR according to an embodiment.

Table 7 Small percentage of images with refined poses

Thus, the above results show that methods according to the embodiments described above outperform methods known in the art. Methods according to embodiments exploit the locality property of similarity among different groups of correspondences, thus leading to better performance during place recognition. These methods have been implemented on a mobile robot and tested in a typical office environment.

Methods according to embodiments are particularly suitable for applications where real-time performance is critical of importance. For example, they can be applied into real-time loop closure detection and visual localization for mobile robot navigation.

In summary, methods according to embodiments convert space, place or environment recognition problem into a Hough space problem, which enables faster and leaner computation. Further, clustering of correspondence is kept local, and multiple clusters of local correspondence are incorporated in order to overcome the burden of computation, and loss of information for matching.

Although the above description of embodiments is directed to robotic vision for navigation and localization, methods according to embodiments may also be employed in other applications, including but not limited to self-driving car navigation, 3-D reconstruction and image stitching.

In the foregoing detailed description, embodiments of the present disclosure are described with reference to the provided figures. The description of the various embodiments herein is not intended to call out or be limited only to specific or particular representations of the present disclosure, but merely to illustrate non- limiting examples of the present disclosure. The present disclosure serves to address at least one of the mentioned problems and issues associated with the prior art. Although only some embodiments of the present disclosure are disclosed herein, it will be apparent to a person having ordinary skill in the art in view of this disclosure that a variety of changes and/or modifications can be made to the disclosed embodiments without departing from the scope of the present disclosure. Therefore, the scope of the disclosure as well as the scope of the following claims is not limited to

embodiments described herein.

Claims

1. A method of verifying a plurality of match candidates between a first image and a second image, wherein a match candidate comprises a first feature in the first image and a second feature in the second image, wherein the first and second features are hypothesised to be projections of the same point in three-dimensional space, the method comprising:

for each of the plurality of match candidates:

determining an individual similarity transformation between a position of the first feature in the first image and a position of the second feature in the second image; and

mapping the determined individual similarity transformation into a

Hough space;

partitioning the Hough space into a plurality of partitions;

determining a plurality of groups,

wherein a group is comprised of all of the match candidates with individual similarity transformations that fall into the same partition; for each group, determining a local similarity transformation; and

verifying a match candidate by:

calculating an error generated by describing the relative positions of the first feature in the first image and the second feature in the second image with one of the determined local similarity transformations, and determining that the error is below an error threshold.

2. The method of claim 1, further comprising: determining a number of individual similarity transformations falling within a partition, and wherein

a group is comprised of all the match candidates having individual similarity transforms that fall into the same partition, wherein the number of individual similarity transformations falling within the partition exceeds a threshold.

3. The method of claim 1, further comprising:

determining a score for each of the plurality of partitions, wherein the score of a partition depends on a number of individual similarity transformations falling within the partition;

determining a given number of partitions with the highest scores;

determining a number of individual similarity transformations falling within each partition of the given number of partitions with the highest scores, and wherein a group is comprised of all the match candidates with individual similarity

transformations that fall into the same partition of one of the given number of partitions with the highest scores, wherein the number of individual similarity transformations falling within the partition exceeds a threshold.

4. The method of any one of the preceding claims, wherein determining a local similarity transformation for a group comprises calculating a mean of all of the individual similarity transformations of the match candidates in that group.

5. The method of any one of the preceding claims, wherein calculating an error generated by describing the relative positions of the first feature in the first image and the second feature in the second image with one of the determined local similarity transformations comprises calculating a two-way projection error.

6. The method of any one of of the preceding claims, wherein determining that the error is below an error threshold comprises determining that the individual similarity transformation of the match candidate is an inlier of one of the determined local similarity transformations.

7. The method of any of the preceding claims, wherein the Hough space is 4 dimensional, with each dimension corresponding to one of: translation in a first direction, translation in a second direction, scale, and orientation, and wherein the dimension corresponding to orientation is partitioned into no more than two partitions.

8. The method of any of claims 1 to 7, wherein the Hough space is 3 dimensional, with each dimension corresponding to one of: translation in a first direction, translation in a second direction and scale.

9. The method of claim 7 or 8, wherein each dimension of the Hough space is independently partitioned.

10. The method of any of the preceding claims, wherein the first image and the second image are each represented by a histogram of a set of features of a visual vocabulary using a bag-of-words technique, and wherein match candidates between the first image and the second image are determined by identifying image features that correspond to the same feature of the visual vocabulary in both images.

11. A method of determining if a geographic location is captured in both a first image and a second image, the method comprising:

generating a plurality of match candidates between a first image and a second image; verifying the plurality of match candidates according to the method of any of the preceding claims;

12. A method of determining a geographic location of a mobile device, the method comprising:

capturing a first query image at a first location;

capturing a second query image at a second location, wherein a spatial relationship between the first location and the second location is known; comparing the first query image and the second query image with a plurality of stored reference images, wherein a location of each of the stored reference images is known; determining which of the stored reference images most closely matches the first query image; determining which of the stored reference images most closely matches the second query image; determining a similarity transformation between features that appear in the first query image and features which appear in the stored reference image which most closely matches the first query image; determining a similarity transformation between features that appear in the second query image and features which appear in the stored reference image which most closely matches the second query image; and calculating the location of the first image from the spatial relationship between the first location and the second location, the location of the stored reference image that most closely matches the first query image, the location of the stored reference image that most closely matches the second query image, the similarity transformation between features that appear in the first query image and features which appear in the stored reference image which most closely matches the first query image, and the similarity transformation between features that appear in the first query image and features which appear in the stored reference image which most closely matches the first query image, wherein

determining which of the stored reference images most closely matches the first and second query images comprises: determining a plurality of matches between each of the stored reference images and the respective query image, wherein a match comprises a first feature appearing in a reference image and a second feature appearing in a query image, wherein the first and second features are both projections of the same point in three-dimensional space; and determining which reference image corresponds to the largest number of matches, and wherein

determining a plurality of matches between the each of the stored reference images and the respective query image comprises, for each stored reference image:

determining a plurality of match candidates between the stored reference image and the respective query image and verifying the match candidates using the method of any one of claims 1 to 10.

13. The method of claim 12, wherein

capturing the first and second query images comprises capturing a first and second image using a camera.

14. The method of either of claims 12 or 13, further comprising:

altering a trajectory of the device based on the determination of the geographic location of the mobile device.

15. A system for verifying a plurality of matches between a first image and a second image, the system comprising:

an input for receiving a first image and a second image; and

a processor configured to perform the method of any of claims 1 to 10.

16. A navigation system for a mobile device, the system comprising:

an input for receiving a first query image and a second query image and information regarding where the first and second query images were captured;

a memory configured to store a plurality of reference images and information regarding a geographic location of each stored reference image; and

a processor configured to determine the geographic location of the mobile device by performing the method of claim 12,

wherein the navigation system is configured to alter a trajectory of the mobile device based on the determination by the processor.

17. A mobile robotic device comprising:

a camera; and

the system of claim 16.

18. A computer readable medium configured to cause a processor to perform the method of any one of claims 1 to 10.

19. A computer readable medium configured to cause a processor to perform the method of claim 11.

20. A computer readable medium configured to cause a processor to perform the method of claim 12.