RU2759773C1 - Method and system for determining the location of the user - Google Patents

Abstract

FIELD: information technology.SUBSTANCE: method for determining the location of the mobile apparatus of the user comprises the stages of: receiving a user request for determining the location thereof, containing at least one image from the camera of the mobile apparatus of the user and the GNSS coordinates of the mobile apparatus; determining the expected area of location of the user based on the received GNSS coordinates of the mobile apparatus by finding the three-dimensional model of the environment; determining a global descriptor for said image; determining at least one similar image in the determined three-dimensional model of the environment based on the obtained global descriptor; calculating singular points and calculating two-dimensional local descriptors therefor on the image of the user request; comparing the local descriptors; based on the result of comparison, defining a subset of singular points wherefor a projection of three-dimensional points is present on the three-dimensional model of the environment; determining the three-dimensional coordinates and the rotation of the camera of the mobile apparatus relative to the coordinate system of the three-dimensional model of the environment.EFFECT: increase in the accuracy and speed of determining the location of the mobile apparatus of the user.13 cl, 6 dwg

Description

FIELD OF TECHNOLOGY

[0001] The present technical solution generally relates to the field of digital data processing, and in particular, methods and systems for the purpose of determining the location of a user in visual positioning systems (eng. Visual Positioning System).

LEVEL OF TECHNOLOGY

[0002] Developments in the field of determining the location of the user using mobile computing devices have always been a priority area of development in view of the rapid development of the consumer digital technology sector. The most commonly used approaches are software applications (for example, Yandex.Maps, Google Maps, Navitel, etc.) that offer various digital maps for positioning a user by obtaining his GNSS (Global Navigation Satellite System) coordinates, for example, using receivers GPS / GLONASS, massively embedded in mobile devices.

[0003] These approaches have a significant drawback in terms of the lack of the ability to analyze and determine the location of the user in the conditions of real objects of the surrounding space, using their capture by the camera of the mobile device.

[0004] To solve this problem, visual positioning systems known from the prior art can be used, which allow processing images of the surrounding space using the camera of a mobile device for the purpose of determining the location of the user, taking into account his position and direction of view.

[0005] These approaches are based on the principle of determining the localization of the camera of a mobile device, in particular, there are structural methods (see, for example, Active Search: Torsten Sattler, Bastian Leibe, Leif Kobbelt, Improving Image-Based Localization by Active Correspondence Search, ECCV 2012 , 2012, pp 752-765.) And deep learning methods (PoseNet: A. Kendall, M. Grimes, R. Cipolla, Posenet: A convolutional network for real-time 6-dof camera relocalization, In Proceedings of the IEEE international conference on computer vision, 2015, pp. 2938-2946).

[0006] Structural methods can achieve a sufficiently high accuracy rate, but are computationally demanding, which leads to a longer time to perform the required calculations. On the other hand, deep learning methods allow performing the required calculations rather quickly, but the quality of positioning of these methods does not allow obtaining a sufficient indicator of the positioning accuracy of the camera of a mobile device.

[0007] The task of absolute localization of the camera in a known environment is a key task in augmented reality projects, as well as an auxiliary task in navigation systems for mobile robots (SLAM task). The main requirements for this approach are as follows:

• consistently high accuracy,

• minimum requirements for computing resources and information storage systems (the ability to solve this problem directly on mobile devices),

• high speed of solving the problem.

[0008] To meet these requirements, it is proposed to create a new approach that provides the localization of the camera of a mobile device in a known environment to determine the accurate and fast location of the user based on information received from the camera of the mobile device.

SUMMARY OF THE INVENTION

[0009] The present technical solution is aimed at solving the technical problem of quickly and accurately determining the location of the user in a known environment using information captured from the camera of his mobile device.

[0010] The technical result coincides with the solution of the technical problem and is aimed at improving the accuracy and speed of determining the location of the user's mobile device.

[0011] The claimed result is achieved by a computer-implemented method for determining the location of the user, performed using a processor and containing the steps at which:

a) receive a user request to determine its location, containing at least one image received from the camera of the user's mobile device, and the image contains at least part of one of the objects of the surrounding space;

b) define a global descriptor for said image contained in the user request;

c) based on the received global descriptor, determine

the user's location area by finding a three-dimensional model of the environment stored in a database of pre-formed three-dimensional models of the environment, and each three-dimensional model contains at least a set of images of the environment from different angles, and each image has specific points for which the two-dimensional local descriptors are calculated;

at least one similar image in a certain three-dimensional model of the environment, and the definition of a similar image is performed using a search algorithm for nearest neighbors;

d) compute the feature points and calculate two-dimensional local descriptors for them on the image of the user request;

e) comparing the local descriptors obtained in step d) with the local descriptors of at least one image of the three-dimensional model of the environment identified in step c);

f) determine a subset of special points from the result of the comparison in step e), for which there is a reproduction of three-dimensional points on a three-dimensional model of the environment;

g) determine the three-dimensional coordinates and rotation of the camera of the mobile device relative to the coordinate system of the three-dimensional model of the environment based on the identified projections for the subset of special points in step f).

[0012] In one particular embodiment of the method, the two-dimensional local descriptors are determined using a scale invariant feature transformation (SIFT) algorithm.

[0013] In another particular embodiment of the method, three-dimensional coordinates are calculated for each feature point.

[0014] In another particular embodiment of the method, for each image forming a three-dimensional model of the environment, at least one global descriptor is determined.

[0015] In another particular embodiment of the method, global descriptors are defined using at least one artificial neural network (ANN).

[0016] In another particular embodiment of the method, a search index is generated based on the global descriptors.

[0017] In another particular embodiment of the method, the search index is stored in a database in a binary format as an element of a three-dimensional model of the environment.

[0018] In another particular embodiment of the method, the user request further comprises data received from the inertial measurement device of the user's mobile device.

[0019] In another particular embodiment of the method, for a three-dimensional model, correspondence points are determined on two-dimensional images with three-dimensional model points.

[0020] In another particular embodiment of the method, based on the corresponding points, a set of points on the three-dimensional model is extracted to estimate the position of the camera of the mobile device.

[0021] In another particular embodiment of the method, the GNSS coordinates of the mobile device are additionally used.

[0022] In another particular embodiment of the method, each 3D model further comprises GNSS coordinates.

[0023] The claimed invention is also implemented using a system for determining the location of a user, which contains a server and a mobile device of the user, in which

the user's mobile device is configured to

- obtaining at least one image from the camera of the mobile device, and the image contains at least part of one of the objects of the surrounding space;

- generating a user request for location determination, containing the mentioned at least one image;

the server is configured

- definition of a global descriptor for the said image contained in the user request;

- based on the obtained global descriptor, the definition of a three-dimensional model of the environment stored in a database of pre-formed three-dimensional models of the environment, and each model contains a set of images of the environment from different angles, and each image has special points for which two-dimensional local descriptors are calculated;

at least one similar image in the found three-dimensional model of the environment, and the definition of the images is performed using the algorithm for finding nearest neighbors;

- definitions based on the received global image descriptor of the user request

- comparison of local descriptors with local descriptors of at least one image of a three-dimensional model of the environment;

- calculation of special points and two-dimensional local descriptors for them on the image of the user request;

- determination of a subset of singular points based on the results of the performed comparison of local descriptors, for which there is a reprojection of three-dimensional points on a three-dimensional model of the environment;

- determination of three-dimensional coordinates and rotation of the camera of the mobile device relative to the coordinate system of the three-dimensional model of the environment based on the identified projections for a subset of special points.

[0024] In one of the particular embodiments of the system, the user's mobile device is selected from the group: smartphone, tablet, portable game console, laptop, augmented reality device.

[0025] In one of the particular embodiments of the system, data exchange between the user's mobile device and the server is carried out via the Internet.

[0026] In one of the particular embodiments of the system, the user's mobile device, when generating a request, additionally takes into account the GNSS coordinates of the device.

[0027] In one of the particular embodiments of the system, the search for three-dimensional space models is carried out taking into account the GNSS coordinates of the user's mobile device.

BRIEF DESCRIPTION OF DRAWINGS

[0028] FIG. 1 illustrates a block diagram of the implementation of the claimed method.

[0029] FIG. 2A illustrates an example of a 3D outdoor environment model.

[0030] FIG. 2B illustrates an example of a 3D indoor environment model.

[0031] FIG. 3A illustrates the principle of communication between a user and a server.

[0032] FIG. 3B illustrates an example of comparing feature points between images of the environment model and a frame from the camera of the user's device.

[0033] FIG. 4 illustrates a general view of a computing device for implementing the claimed invention.

CARRYING OUT THE INVENTION

[0034] As shown in FIG. 1, the claimed method (100) for determining the user's location is carried out by performing successive steps. The method is based on the formation of three-dimensional models (3D) of the environment for the purposes of their use with the subsequent localization of the position of users by analyzing images captured using mobile devices.

[0035] FIG. 2A-FIG. 2B shows examples of the created 3D model of the environment (20) for a group of images from different angles (201) - (203). Building an environment model consists of the following steps. With the help of software that provides the creation of three-dimensional reconstructions, for example, software created on the basis of the ARKit, COLMAP, OpenMVG, pytorch, TensorFlow frameworks, etc., a sequence of images is created, in which all elements of the described environment are visible from different angles, for example , buildings, groups of buildings, streets, interior layout of buildings, objects of the environment, architectural elements, furniture, interior items, etc. The program also saves information about the position of the camera for each image (this information is used for the subsequent validation of calculations). As such information, the 6-Dof position of the user is used, in particular, his three-dimensional position and angles of rotation in three-dimensional space (x, y, z).

[0036] Shooting images (201) - (203) is carried out according to a certain method: in the form of successive rotations of the camera in the horizontal plane and the subsequent transition to the next shooting point. This improves the accuracy of the calculations.

[0037] The resulting 3D model (20) is a point cloud generated during said survey. Special points (2011) - (2013) are highlighted on each image, for which two-dimensional (2D) local descriptors are determined using the scale-invariant feature transformation (SIFT) algorithm. Also, examples of algorithms such as ORB, SURF, LIFT, BRIEF, SuperPoint, etc. can be applied.

[0038] The calculated local descriptors are stored in a database in a predetermined format, for example, COLMAP (structure-from-motion system). With the help of COLMAP, a point cloud of a three-dimensional model is constructed (20), according to which the possibility of calculating the reproduction of the position of two-dimensional points (2011) - (2013) in a three-dimensional coordinate system of a three-dimensional model (20) is realized. Various distinguishable parts of environmental objects can be used as special two-dimensional points (2011) - (2013), for example, parts of buildings, building elements (arches, signs, columns, statues, etc.), etc.

[0039] For each image, global descriptors are computed using a machine learning model such as an artificial neural network NetVLAD, pre-trained on the Pittsburgh 30k dataset. The global descriptor is a vector representing the basic information encoded in the image.

[0040] A search index is generated from the global descriptors using one of the applicable methods, for example, FAISS, Nmslib, Falconn, etc. A search index is a data structure that contains information about images and is used in image search systems. The search index is stored in a binary format as an element of the 3D model of the environment (20).

[0041] Next, a process of performing method (100) with reference to FIG. 3A-3B. At step (101), the user (30), using the mobile device (300), initiates a request to determine his location, which is transmitted to the server (302) via a data network (305), for example, the Internet. The user request contains one or more images (301) captured by the camera of the device (300). The image (301) generally contains at least a part of one of the objects of the surrounding space, for example, a building, an element of interior decoration, architecture, etc. The image (301) can also additionally take into account the GNSS coordinates of the mobile device obtained using the corresponding antenna module of the device (300) (GPS, GLONASS, BeiDou, etc.). Additionally, the user request may contain information obtained from the inertial sensors of the mobile device (300), for example, gyroscope readings.

[0042] In step (102), the server (302) determines the global descriptor for the obtained user image (301). Determination of the global descriptor for the image (301), which contains in whole or in part an object (or part of it) of the environment, can be carried out using the previously mentioned artificial neural network NetVlad or any other method suitable for performing this function.

[0043] At step (103), based on the obtained global descriptor, the server (302) determines the intended area of the user (30). This determination is carried out on the basis of comparing the obtained global descriptor with pre-formed three-dimensional environment models (20) stored in the database (303).

[0044] For each model (30), a search index is generated based on the global descriptors, the images (201) - (203) included in them from different angles with the calculated local 2D descriptors at the corresponding special points (2011) - (2013).

[0045] At step (104), when one or more suitable models (20) are found when they are analyzed for similarity to the global descriptor of a custom image (301), one or more images (201) - (203) are searched for forming the aforementioned 3D environment model (20), for the purpose of subsequent identification of similar objects of the environment. Algorithmically, a number of similar images of the corresponding 3D model of the environment are identified (20). Image determination is performed by determining the proximity measure (L2-norm) using a method based on image search using the nearest neighbors algorithm, for example, using the FAISS method (abbreviated from the Facebook AI Research Similarity Search).

[0046] In step (105), special points (ZON) - (3013) are calculated for the user request image (301) and the two-dimensional local descriptor points are calculated using the above SIFT algorithm.

[0047] Next, a comparison is made (step 106) of the calculated 2D local descriptors obtained in step (105) with the image descriptors (201) - (203) of the 3D environment model (20). As shown in FIG. 3B, based on the results of the comparison of local descriptors at step (107), a subset of two-dimensional points is determined on at least one identified image (201) - (203) of a three-dimensional environment model (20).

[0048] In step (107), an analysis is performed to match the image descriptors (201) for the environment model (20) stored in the database (303) and the user's image (301). At this stage, a subset of singular points is determined, which are contained in the user's image (301) as a result of comparison with images (201) - (203), for which there is a reproduction of three-dimensional points on a three-dimensional model of the environment (20). space from different angles, which makes it possible to more accurately determine similar images containing these points (see, for example, Zhang. Camera Models and Image Reprojection // ECE661 Computer Vision Homework 6. 2008).

[0049] Since for images (201) - (203), forming a 3D model of the environment (20), 2D-3D correspondences between the points of the three-dimensional model (20) and their two-dimensional projections (2011) - (2013) on a certain image ( 201) - (203), then by comparing the two-dimensional descriptors, it becomes possible to determine the three-dimensional coordinates (projections) for the special points (3011) - (3013) on the user image (301).

[0050] In step (108), the camera position of the device (300) is determined using the perspective-n-point (abbreviated PNP) algorithm. The PNP algorithm provides a solution to the problem of estimating the position of a calibrated camera from a set of n three-dimensional points in three-dimensional space and their corresponding 2D projections on the image. As such an algorithm, the P3P + RANSAC algorithm known from the prior art (XS Gao, X.-R. Hou, J. Tang, H.-F. Chang, Complete Solution Classification for the Perspective-Three-Point Problem, IEEE Transactions on Pattern Analysis and Machine Intelligence, volume 25, issue 8, pp 930-943, 2003).

[0051] Using the PNP algorithm, three-dimensional coordinates and rotation data of the camera of the mobile device (300) relative to the coordinate system of the three-dimensional model of the environment (20) are determined. This determination is performed based on the detected projections for the subset of feature points (3011) - (3013) of the user image (301). This allows you to clarify the location of the user (30) and obtain the corresponding data on the server (302).

[0052] The position of the camera based on the results of the method (100) is understood as [R, t] - rotation and displacement of the camera of the device (300) relative to the coordinate system of the point cloud forming a three-dimensional model (20), where

[R, t] is a combined matrix in which

R => {α, β, γ} - angles of rotation of the virtual camera relative to the axes Ox, Oy, Oz,

t => {x, y, z} - offset of the position of the virtual camera, predicted from the user image.

[0053] As one of the examples of the use of the present solution is the placement of AR content when pointing the camera of the device (300) at objects of the surrounding world. Due to the fact that AR content is bound in the point cloud coordinate system on the environment model (20), then knowing the position of the camera and its rotation of the device (300) through 2D-3D reprojection, it is possible to obtain a display of AR content on a mobile device (300) ...

[0054] Additionally, the acquired GNSS coordinates of the mobile device (300) may be taken into account. Thanks to the obtained GNSS coordinates, at stage (101) it becomes possible to clarify the location of the object in the world coordinate system. Each point cloud that forms the ST model (20) also has a binding to positions in the world coordinate system, thus the approximate values of longitude and latitude become known. Using a priori knowledge of the location due to GNSS, at stage (103), the model (20) is selected, in which further search by global descriptors (104) will take place. Model (20) is selected based on finding the shortest distance from the longitude and latitude values of the user's device (300) and the longitude / latitude of the model itself (20). After finding candidates (104) and obtaining a two-dimensional mapping of descriptors in step (105), the GNSS position is also used during the PnP optimization process in step (108), during which a constraint is added on a possible position in space equal to the radius (several meters) from position of the GNSS device (130). This improves the convergence of the method, as well as refines the position, getting the best result.

[0055] The claimed method (100) for determining the position of the user (30) can be used as a correction when applying the SLAM (imultaneous localization and mapping) algorithm indoors. This method is used in mobile autonomous tools to build a map in an unknown space or to update a map in a pre-known space while monitoring the current location and the distance traveled.

[0056] In general, the process of using the method (100) indoors with the SLAM algorithm is as follows. Images are formed using ARKit / ARCore and sent with the corresponding positions to the server (302), storing the positions for subsequent correction. By applying method (100) to each picture, position prediction is performed. Taking into account the positions received from the indoor device, incorrectly predicted positions are discarded, the best position is selected by the number of inliners (English, inliner, points that satisfy the model) after the PnP stage and sent to the indoor device, for example, user device (300) (30) ...

[0057] Next, the current position of the user device (300) is replaced with a new position received from the server (302), taking into account the stored position and its own offset while waiting for a response from the server (302). This improves the accuracy of determining the position of the device (300) of the user indoors.

[0058] FIG. 4 shows a general view of the computing device (400). On the basis of the device (400), a user device (300), a server (302) and other unrepresented devices can be implemented, which can participate in the general information architecture of the claimed solution.

[0059] In the General case, the computing device (400) contains one or more processors (401) united by a common bus of information exchange, memory means such as RAM (402) and ROM (403), input / output interfaces (404), devices input / output (405), and a device for networking (406).

[0060] The processor (401) (or multiple processors, multi-core processor) can be selected from a range of devices currently widely used, for example, Intel ™, AMD ™, Apple ™, Samsung Exynos ™, MediaTEK ™, Qualcomm Snapdragon ™ and etc.

[0061] RAM (402) is a random access memory and is intended for storing machine-readable instructions executed by the processor (401) for performing the necessary operations for logical processing of data. RAM (402) typically contains executable instructions of the operating system and associated software components (applications, software modules, etc.).

[0062] ROM (403) is one or more persistent storage devices such as a hard disk drive (HDD), solid state data storage device (SSD), flash memory (EEPROM, NAND, etc.), optical storage media ( CD-R / RW, DVD-R / RW, BlueRay Disc, MD), etc.

[0063] Various types of I / O interfaces (404) are used to organize the operation of the components of the device (400) and to organize the operation of external connected devices. The choice of the appropriate interfaces depends on the specific version of the computing device, which can be, but are not limited to: PCI, AGP, PS / 2, IrDa, FireWire, LPT, COM, SATA, IDE, Lightning, USB (2.0, 3.0, 3.1, micro, mini, type C), TRS / Audio jack (2.5, 3.5, 6.35), HDMI, DVI, VGA, Display Port, RJ45, RS232, etc.

[0064] To ensure the interaction of the user with the computing device (400), various means (405) I / O information are used, for example, a keyboard, display (monitor), touch display, touch-pad, joystick, mouse manipulator, light pen, stylus, touch panel, trackball, speakers, microphone, augmented reality, optical sensors, tablet, light indicators, projector, camera, biometric identification (retina scanner, fingerprint scanner, voice recognition module), etc.

[0065] The networking means (406) allows the device (400) to transmit data via an internal or external computer network, for example, Intranet, Internet, LAN, and the like. One or more means (406) may be used, but not limited to: Ethernet card, GSM modem, GPRS modem, LTE modem, 5G modem, satellite communication module, NFC module, Bluetooth and / or BLE module, Wi-Fi module and dr.

[0066] Additionally, satellite navigation aids can also be used as part of the device (400), for example, GPS, GLONASS, BeiDou, Galileo.

[0067] The presented materials of the technical solution disclose preferred examples of implementation and should not be construed as limiting other, particular examples of its implementation, which do not go beyond the scope of the claimed legal protection, which are obvious to specialists in the relevant field of technology.

Claims (32)

1. A computer-implemented method for determining the location of a user's mobile device, performed using a processor and containing the stages at which: a) receive a user request to determine its location, containing at least one image received from the camera of the user's mobile device, and the image contains at least part of one of the objects of the surrounding space, as well as the coordinates of the GNSS of the mobile device; b) determine the intended user area based on the obtained GNSS coordinates of the mobile device by finding a three-dimensional model of the environment stored in a database of pre-formed three-dimensional models of the environment, each model containing the coordinates of the GNSS and a set of images of the corresponding environment from different angles, with each image has singular points for which two-dimensional local descriptors are calculated; c) define a global descriptor for the said image contained in the user request; d) determining, based on the obtained global descriptor of the user's request image, at least one similar image in the three-dimensional model of the environment determined in step b), and the determination of the images is performed using the nearest neighbors search algorithm; e) compute the feature points and compute two-dimensional local descriptors for them on the user request image; f) comparing the local descriptors obtained in step e) with the local descriptors of at least one image of the three-dimensional model of the environment identified in step d); g) determine a subset of special points from the comparison at step f), for which there is a reproduction of three-dimensional points on a three-dimensional model of the environment; h) determine the three-dimensional coordinates and rotation of the camera of the mobile device relative to the coordinate system of the three-dimensional model of the environment based on the identified projections for the subset of the special points in step g). 2. The method according to claim 2, characterized in that the two-dimensional local descriptors are determined using a scale-invariant feature transformation (SIFT) algorithm. 3. The method according to claim 1, characterized in that three-dimensional coordinates are calculated for each feature point. 4. The method according to claim 3, characterized in that for each image forming a three-dimensional model of the environment, at least one global descriptor is defined. 5. The method according to claim 4, characterized in that the global descriptors are determined using at least one artificial neural network (ANN). 6. The method according to claim 4, characterized in that a search index is formed on the basis of the global descriptors. 7. The method according to claim 6, characterized in that the search index is stored in the database in a binary format as an element of a three-dimensional model of the environment. 8. The method according to claim 1, characterized in that the user request further comprises data received from the inertial measuring device of the user's mobile device. 9. The method according to claim 4, characterized in that correspondence points are determined for the three-dimensional model on two-dimensional images with three-dimensional model points. 10. The method according to claim 9, characterized in that, based on the corresponding points, a set of points on the three-dimensional model is selected to estimate the position of the camera of the mobile device. 11. A system for determining the location of a user's mobile device, containing a server and a user's mobile device, in which the user's mobile device is configured to - obtaining at least one image from the camera of the mobile device, and the image contains at least part of one of the objects of the surrounding space; - generating a user request to determine the location, containing the aforementioned at least one image obtained from the camera of the user's mobile device, as well as the coordinates of the GNSS of the mobile device; the server is configured - determining the intended area of the user's location based on the obtained GNSS coordinates of the mobile device by finding a three-dimensional model of the environment stored in a database of pre-formed three-dimensional models of the environment, and each model contains the coordinates of the GNSS and a set of images of the corresponding environment from different angles, while each image has special points for which two-dimensional local descriptors are calculated; - definition of a global descriptor for the said image contained in the user request; - determining, based on the obtained global descriptor of the user's request image, at least two similar images in the found three-dimensional model of the environment, and the definition of the images is performed using the algorithm for finding the nearest neighbors; - comparison of local descriptors with local descriptors of at least one image of a three-dimensional model of the environment; - calculation of special points and two-dimensional local descriptors for them on the image of the user request; - determination of a subset of singular points based on the results of the performed comparison of local descriptors, for which there is a reprojection of three-dimensional points on a three-dimensional model of the environment; - determination of three-dimensional coordinates and rotation of the camera of the mobile device relative to the coordinate system of the three-dimensional model of the environment based on the identified projections for a subset of special points. 12. The system according to claim 11, characterized in that the user's mobile device is selected from the group: smartphone, tablet, portable game console, laptop, augmented reality device. 13. The system according to claim 11, characterized in that the exchange of data between the user's mobile device and the server is carried out via the Internet.

Images