EP4204766A1 - Apparatuses and methods for positioning a device - Google Patents

Apparatuses and methods for positioning a device

Info

Publication number
EP4204766A1
EP4204766A1 EP20799694.3A EP20799694A EP4204766A1 EP 4204766 A1 EP4204766 A1 EP 4204766A1 EP 20799694 A EP20799694 A EP 20799694A EP 4204766 A1 EP4204766 A1 EP 4204766A1
Authority
EP
European Patent Office
Prior art keywords
compact
signature
signatures
request
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP20799694.3A
Other languages
German (de)
French (fr)
Inventor
Nathan PIASCO
Dzmitry Tsishkou
Tao Yin
Laurent SMADJA
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Yinwang Intelligenttechnologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Publication of EP4204766A1 publication Critical patent/EP4204766A1/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G01MEASURING; TESTING
    • G01CMEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/26Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network
    • G01C21/34Route searching; Route guidance
    • G01C21/36Input/output arrangements for on-board computers
    • G01C21/3602Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera
    • GPHYSICS
    • G01MEASURING; TESTING
    • G01CMEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/26Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network
    • G01C21/28Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network with correlation of data from several navigational instruments
    • G01C21/30Map- or contour-matching
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00Pattern recognition
    • G06F18/20Analysing
    • G06F18/24Classification techniques
    • G06F18/241Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2415Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00Machine learning
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/04Architecture, e.g. interconnection topology
    • G06N3/045Combinations of networks
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/04Architecture, e.g. interconnection topology
    • G06N3/0464Convolutional networks [CNN, ConvNet]
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/08Learning methods
    • G06N3/084Backpropagation, e.g. using gradient descent
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/08Learning methods
    • G06N3/09Supervised learning
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/70Determining position or orientation of objects or cameras
    • G06T7/73Determining position or orientation of objects or cameras using feature-based methods
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/70Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/761Proximity, similarity or dissimilarity measures
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/70Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/7715Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00Scenes; Scene-specific elements
    • G06V20/50Context or environment of the image
    • G06V20/56Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10016Video; Image sequence
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/30Subject of image; Context of image processing
    • G06T2207/30244Camera pose
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/30Subject of image; Context of image processing
    • G06T2207/30248Vehicle exterior or interior
    • G06T2207/30252Vehicle exterior; Vicinity of vehicle

Definitions

  • the following description relates to the field of positioning systems. More specifically, the following description relates to apparatuses and methods for positioning a device.
  • the positioning system is used to measure a location of the device.
  • the location information may be used in many applications. For example, navigation and other map related applications require that the position of the device is known so that navigation instructions can be formed.
  • a device determines the current location by receiving signals transmitted along a line of sight from a plurality of satellites.
  • the device such as a mobile phone or a vehicle, has electronic receivers for receiving the signals that can be used in positioning of the device.
  • a global navigation satellite system does not require Internet reception; however, in some circumstances it may be used in assisting the positioning process.
  • positioning signals are received along a line of sight.
  • positioning is possible only when there is a line of sight to at least four satellites.
  • buildings, mountains or other obstacles are preventing visibility to at least four satellites, and the positioning is not possible.
  • This is particularly a problem underground, for example in parking garages and road tunnels, or inside buildings.
  • the visibility problem has been addressed by developing indoor positioning systems.
  • the mobile communication network may be used.
  • Special purpose indoor positioning systems are capable of providing the exact location. However, they are normally based on geolocated beacons, which transmit signals using Bluetooth, Wi-Fi or other suitable radio technology. These beacons are expensive and require maintenance.
  • Another option is to use a plurality of different sensors and maps, either separately or together.
  • a vehicle can use a map and odometry to support navigation when entering a tunnel; however, it typically drifts quite quickly.
  • a mobile phone or a tablet computer is used instead of an integrated navigation device, the odometry information is not accessible. Thus, there is a need for improved positioning systems.
  • apparatuses and methods for determining a position of a device are disclosed.
  • the positioning is based on acquiring images that have unique position identifying features, which are either natural or man built. These position identifying features are used in generating a multi-dimensional signature.
  • the multi-dimensional signature is processed so that a compact signature with reduced dimensions is achieved.
  • the position of the device can be retrieved using the compact signature.
  • a method for determining a position of an apparatus comprises acquiring an image; extracting a position-identifying feature from the acquired image; generating a multi-dimensional signature of the acquired image using the position-identifying feature; generating a compact signature by reducing dimensions of the multidimensional signature of the acquired image; and retrieving the position of the apparatus using the compact signature. It is beneficial to use position-identifying features from acquired images to determine the location of a device or a vehicle. This provides a possibility for accurate positioning also in locations where satellite-based positioning is not possible.
  • acquiring the image comprises acquiring the image in a road direction. It is beneficial to use images that are acquired in the road direction either forward or backward. This facilitates accurately measuring the distance between images and reduces the motion blur.
  • the method further comprises acquiring a plurality of images, extracting a position-identifying feature from each of the acquired images; generating a multi-dimensional signature of each of the acquired images using the positionidentifying feature; generating a compact signature by reducing dimensions of each of the multidimensional signatures; and retrieving the position of the apparatus using a plurality of compact signatures. It is beneficial to use a plurality of images as it improves the accuracy of the method in case of occlusion in one of the images.
  • the method further comprises measuring a geographical distance between the first and the last image. It is beneficial to measure the distance between the first and the last image, as the distance information improves the accuracy of the method by determining the proper length of the multidimensional signature.
  • retrieving the position of the apparatus comprises: generating a request comprising the compact signature; matching the compact signature of the request with a database comprising a plurality of compact signatures, wherein each of the compact signatures is associated with a position; and determining the position of an apparatus based on positions of the compact signatures matching the at least one signature. It is beneficial to have a database in the device so that the positioning can be done without a network connection, as it may not be available when the positioning is needed.
  • matching comprises finding a sequence of compact signatures that is the most similar to the compact signatures of the request. It is beneficial to try to find a sequence that is the most similar to the compact signatures of the request.
  • the compact signature of the request may also have features that are not positionidentifying, and the compact signature does not need to be an exact match.
  • retrieving the position comprises generating a request comprising the compact signature; transmitting the request to an external database suitable for matching the compact signature of the request with a database comprising a plurality of compact signatures associated with the position of an apparatus based on positions of the compact signatures matching the at least one signature; and receiving the position of the apparatus as a response to the request. It is beneficial to use an external database that might be larger and also easier to maintain so that it is always updated.
  • the position identifying feature is extracted using a machine learning arrangement configured to find a relevant position-identifying feature. It is beneficial to use a machine learning arrangement in detecting position-identifying features, as the machine learning arrangement may be trained further. This improves positioning in all possible locations instead of only the trained position.
  • a computer program comprising computer program code.
  • the computer program code is configured to cause performing a method as described above when the computer program code is executed in a computing apparatus. It is beneficial to implement the positioning method as software using a circuitry for determining the position.
  • an apparatus comprising a camera configured to acquire an image; and processing circuitry configured to: extract a position identifying feature from the acquired image; generate a multi-dimensional signature of the acquired image using the position identifying feature; generate a compact signature by reducing dimensions of the multidimensional signature of the acquired image; and retrieve the position of the apparatus using at least one compact signature including the compact signature. It is beneficial to use position-identifying features from acquired images to determine the location of a device or a vehicle. This provides a possibility for accurate positioning also in locations where satellite-based positioning is not possible.
  • the camera is configured to acquire the image in a road orientation. It is beneficial to use to use images that are acquired in the road direction either forward or backward. This facilitates accurately measuring the distance between images and reduces the motion blur.
  • the camera is configured to acquire a plurality of images; and the processing circuitry is configured to extract a position-identifying feature from each of the acquired images; generate a multi-dimensional signature of each of the acquired images using the position-identifying feature; generate a compact signature by reducing dimensions of each of the multidimensional signatures; and retrieve the position of the apparatus using a plurality of compact signatures. It is beneficial to use a plurality of images as it improves the accuracy of the method in case of occlusion in one of the images.
  • the processing circuitry is further configured to: measure a geographical distance between the first and the last image. It is beneficial to measure the distance between the first and the last image, as the distance information improves the accuracy of the method by determining the proper length of the multidimensional signature.
  • the processing circuitry when retrieving the position of the apparatus, is further configured to: generate a request comprising the compact signature; match one or more compact signatures of the request with a database comprising a plurality of compact signatures, wherein each of the compact signatures is associated with a position; and determine the position of an apparatus based on positions of the compact signatures matching the at least one signature. It is beneficial to have a database in the device so that the positioning can be done without a network connection, as it may not be available when the positioning is needed.
  • the processing circuitry when matching, is further configured to find a sequence of compact signatures that is the most similar to the compact signatures of the request. It is beneficial to try to find a sequence that is the most similar to the compact signatures of the request.
  • the compact signature of the request may also have features that are not position-identifying, and the compact signature does not need to be an exact match.
  • the processing circuitry when retrieving the position of an apparatus, is further configured to: generate a request comprising the compact signature; transmit the request to an external service suitable for matching one or more compact signatures of the request with a database comprising a plurality of compact signatures associated with the position of an apparatus based on positions of the compact signatures matching at least one compact signature; and receive the position of the apparatus as a response to the request. It is beneficial to use an external database that might be larger and also easier to maintain so that it is always updated.
  • the processing circuitry is configured to extract the position identifying feature using a machine learning arrangement configured to find a relevant position identifying feature. It is beneficial to use a machine learning arrangement in detecting position-identifying features, as the machine learning arrangement may be trained further. This improves positioning in all possible locations instead of only the trained position.
  • a server for positioning comprises a database comprising a plurality of compact signatures associated with a position; and a processing circuitry configured to: receive a request comprising at least one compact signature; match at least one compact signature of the request with a database comprising a plurality of compact signatures, wherein each of the compact signatures is associated with a position; and determine the position of a requesting apparatus based on positions of the compact signatures matching the at least one compact signature.
  • the external server may have a database that is larger and also easier to maintain so that it is always updated.
  • matching comprises finding a sequence of compact signatures that is the most similar to the compact signatures of the request. It is beneficial to try to find a sequence that is the most similar to the compact signatures of the request.
  • the compact signature of the request may also have features that are not positionidentifying, and the compact signature does not need to be an exact match.
  • Fig. 1 shows an example of a method for generating compact signatures
  • Fig. 2 shows an example of a scheme for generating compact signatures
  • Fig. 3 shows an example of a training process for machine learning involved in road signature generation
  • Fig. 4 shows an example of a scheme for an acquisition vehicle
  • Fig. 5 shows an example of an arrangement for positioning a car.
  • a disclosure in connection with a described method may also hold true for a corresponding device or system configured to perform the method and vice versa.
  • a corresponding device may include a unit to perform the described method step, even if such unit is not explicitly described or illustrated in the figures.
  • the features of the various exemplary aspects described herein may be combined with each other, unless specifically noted otherwise.
  • Fig. 1 discloses an example of a method for retrieving a position of an apparatus, such as a mobile phone, vehicle or any similar apparatus comprising a camera and an access to a database comprising position associated compact signatures or other similar references that can be derived from an image.
  • the method is initiated by acquiring an image, step 100.
  • the image is acquired using a conventional camera, for example a camera unit of a mobile phone or a camera of a vehicle.
  • the acquired image is then processed in order to extract position-identifying features from the image, step 110.
  • the features are extracted, it is possible that also non-position identifying features are extracted. It is possible to use additional analysis tools for removing them; however, the positioning is possible when acquired images include enough position identifying features.
  • even one position identifying feature may be enough.
  • some of the road signs are unique, and when they are extracted the position can be accurately determined, even if the image comprises additional features.
  • the extracting step may include identifying the extracted features. For example, when the feature is identified to be a vehicle, it may be discarded as it is highly unlikely that a vehicle can be used for determining the position.
  • a multi-dimensional signature is generated, step 120.
  • the multi-dimensional signature provides very accurate identifying information of the extracted feature. It is used as a source for generating a compact signature, step 130.
  • the position of the device is retrieved using one or more generated compact signatures, step 140.
  • the method may be implemented in one device comprising a processing circuitry.
  • the processing circuitry comprises at least one processor for executing computer programs and at least one memory for storing computer programs and related data.
  • the memory may be volatile or non-volatile depending on the task required.
  • the processor is configured to execute computer programs that are configured to cause performing the method as explained.
  • two or more devices may be used as explained in the following disclosure.
  • the example method of Fig. 1 may be completely performed in the device. This implementation may be considered as an offline implementation.
  • a remote service such as a cloud service or a positioning server.
  • Step 100 of acquiring images is performed in the device that is being positioned. As modern cameras are of high resolution, it is typically not desired to send the acquired images over the network to a remote service.
  • position-identifying features are typically extracted in the requesting device.
  • generating multi-dimensional signatures and also generating compact signatures is typically done at the requesting device. This is beneficial, as compact signatures are smaller in size, and thus the request transmitted to a cloud or positioning service does not require as much bandwidth as it would require with images or multidimensional signatures.
  • the position can be retrieved from an internal database stored in a memory of the device, or it can be retrieved from an external service.
  • Retrieving comprises generating an appropriate request and receiving a response.
  • retrieving comprises processing the request and transmitting a response.
  • these are internal steps and signals within the device.
  • these tasks are performed by the external service, and the device does not need any further information on them.
  • Fig. 2 discloses an example of a scheme for generating compact signatures.
  • the scheme may be used in several different devices or apparatuses, for example in mobile phones or cars.
  • the following disclosure uses a car as an example; however, the scheme may be applied in any device having a camera unit for acquiring images and an access to a database, which may be located in a device itself or in a network service, such as a cloud service or server.
  • one or more images are acquired, step 200.
  • these images are acquired in a road orientation or direction. This orientation may be forward or backward.
  • One image may be sufficient for coarse positioning.
  • a method according to the scheme is used, for example in a navigation application, a plurality of images increases the accuracy of the method.
  • the distance traveled between the first and the last or between each consecutive image may be measured and used in positioning.
  • the acquired images can be ordinary images acquired with a camera that is already integrated into the acquiring device. For example, modern cars and mobile phones typically have one or more cameras.
  • the acquired images are transmitted to an image encoder 210. In the image encoder, the image is processed for finding position identifying features in the image.
  • a position identifying feature is a feature that can be used in positioning.
  • the position identifying feature is an object, landmark or the like.
  • the position identifying feature does not change or move rapidly over time.
  • Examples of position identifying features include trees, rock formations, lakes, buildings, road signs, roads and streets as such and the like.
  • Examples of other features include the sun, persons, animals and other cars. These features typically move and change their place, so they cannot be used in determining the position.
  • the image encoder 210 provides a set of image features 220 that includes one or more position identifying features.
  • the image encoder comprises a convolutional neural network (CNN) followed by a global pooling layer.
  • CNN outputs k feature maps that are summarized to one value by a feature map by the global pooling layer.
  • the obtained k dimensional vector is finally normalized.
  • the outcome 220 from the image encoder 210 is fed to the signature generator 230.
  • the position identifying features 220, together with the distance information, are then used in generating a signature.
  • the road signature generation module uses as inputs a continuous sequence of images from time t to t + n and the distance traveled d t+n from the position of the camera at time t and the position of the camera at time t + n.
  • the attention mechanism inside the signature generator uses a temporal filter to combine the input image features into road signature vectors.
  • the signature generator 230 returns a set of I vectors 240 that is called a road signature.
  • Each vector of the signature has a dimension k and two consecutive vectors are separated by a distance of c/ A .
  • the set of I vectors 240 is then fed to dimension reduction 250.
  • the dimension reduction is an unsupervised machine learning method that is configured to reduce the dimension of the road signature.
  • Dimension reduction is applied to each road signature vector to reduce its size from k to m, with m « k.
  • the outcome is a compact road signature 260, which is a set of I vectors, wherein each of the vectors has a dimension m, which is considerably less than k, and two consecutive vectors are separated by a distance of c/ A .
  • signature generation process is a fully differentiable operation, and thus road signatures can be optimized end to end for localization purposes and all the road signature generator components may be optimized jointly during the training.
  • the training process is described below.
  • Fig. 3 an example of a training process for machine learning involved in the road signature generator is disclosed.
  • the road signature generator can be trained with the following principles.
  • the example of Fig. 3 shows two separate pipelines involving road signature generation.
  • the first pipeline involves receiving a reference sequence of images and distance 300 as a reference input, which is processed by a road signature generator 302.
  • a road signature generator 302. As a result, one or more reference compact signatures 304 are computed.
  • the process of generating road signatures may be similar to the process described above in the example of Fig. 2.
  • the second pipeline receives a target sequence of images and distance 310 as a target input, which is processed by a road signature generator 304.
  • one or more target compact signatures 314 are computed.
  • the first pipeline uses a reference input of a continuous sequence of images from time t to t + n and the distance traveled d t+n from the position of the camera at time t and the position of the camera at time t + n.
  • the second pipeline uses a target input of a continuous sequence of images from time f’to t’ + m and the distance traveled df +m from the position of the camera at time t’ and the position of the camera at time t ’ + m.
  • the target sequence is recorded on the same road as the reference sequence at a different time t t
  • the target sequence is longer than the reference sequence: df +m > dt+n.
  • the road signature generators 302 and 312 both use a Siamese network sharing the same set of parameter weights to compute the reference and the target signatures from the image sequences. These Siamese networks return the compact signature of the reference and the target sequence of images. As the distance traveled in the target sequence is longer than the distance traveled in the reference sequence, the target signature is longer than the reference signature.
  • a road signature matcher 320 computes signal similarity 322 between the reference and the target signature using sliding-window functions, for example Zero-mean Normalized Cross Correlation.
  • the road signature matcher 320 returns a similarity measurement 330 of various sizes depending on the signature length. High similarity results in a low value in the similarity measurement signal.
  • the road signature matcher 320 creates a ground truth signal from the ground truth spatial and temporal alignment of the reference and the target sequence 360. It is computed by generating a narrow 1 D Gaussian function centered at the overlapping position of the reference and target sequence. Ground truth alignment of the reference and target sequence are obtained from GPS measurement or by content-based image retrieval.
  • a training signal is obtained by computing the cross-entropy loss 350 between the ground truth signal and the similarity measurement obtained from the road signature matcher 320. Before computing the loss, the similarity measurement obtained from the road signature matcher 320 is converted to probability 340 using a Softmax function 335 on the opposed signal. A gradient of the loss is computed regarding the input sequences and back-propagated through the whole pipeline to correct the weight parameters of the road signature generator.
  • Fig. 4 discloses a method for an acquisition vehicle 400 that is used in collecting location associated data and building a database to be used in image-based positioning.
  • the acquisition vehicle of the example of Fig. 4 comprises an acquisition subsystem including an inertial navigation system, a satellite-based 410 navigation system antenna and an odometry system.
  • the inertial navigation system is coupled to a camera, oriented along the longitudinal axis of the vehicle.
  • a subsystem may be used to ensure the synchronization of the image capture process and the inertial navigation system output.
  • the additional subsystem saves images in a dedicated memory.
  • the acquisition subsystem acquires images precisely geolocated and geotagged along the traveled road. From the recorded sequence of images 420 and the traveled distance, the road signature generator 430 computes a compact road signature.
  • the road signature computation can be done on the embedded vehicle to reduce the inboard storage usage or on a remote server to reduce the computational load on the acquisition vehicle.
  • the road signature generator 430 produces a compact road signature 440 precisely longitudinally localized on the road being traveled by the acquisition vehicle.
  • the precisely localized compact road signature 440 is stored in a geographic database 450.
  • the geographic database 450 is then used as a source of information, or the information stored can be packaged for later use.
  • Fig. 5 an example of an arrangement for positioning a car 500 is shown.
  • the user is in a vehicle, equipped with a device that combines a GNSS application and a camera module, with a geodatabase in a memory.
  • the GNSS application will trigger an online process which will determine the position of the car using an image-based position system.
  • the car 500 acquires images 510 from its camera module as well as the traveled distance, or vehicle speed to derive the traveled distance, of the image sequence.
  • the vehicle uses a road signature generator 512 to compute a compact road signature 514 from the online-captured image sequence.
  • the compact road signature 514 computed online by the user device or vehicle is called a target road signature.
  • a geographic database 520 of precomputed compact road signatures may be stored or partially stored on the user vehicle or device, and it is queried when entering a GNSS restricted area.
  • the geographical database is accessed using an internet connection of the vehicle.
  • the geographic database 520 is used to retrieve a reference georeferenced compact road signature. This is done from a last valid GNSS position received before entering the GNSS restricted area.
  • the user device or vehicle retrieves from the embedded geographic database 520 the precomputed compact road signature 522 geolocated on the road being traveled by the user.
  • This compact road signature 522 precisely longitudinally located on the road, is called a reference compact road signature.
  • the vehicle 500 proceeds to target and reference signatures matching using a road signature matcher 530.
  • the road signature matcher 530 produces a similarity signal between the target and the reference signatures for similarity measurement 532.
  • the pick in the similarity signal represents the optimal alignment between the target and the reference signatures. Thus, it provides the relative position of the target signature to the reference signature that produces the highest similarity measurement.
  • the road signature georeferencing module 540 of the user device or vehicle uses the similarity signal from the road signature matcher 530 and the precise position of the reference compact road signature to compute the precise position 550 of the target compact road signature along the driven road and to derive the current position of the user vehicle.
  • the device location may be shown in an application that normally relies on satellite positioning.
  • the arrangements using positioning as described above may be implemented in hardware, such as a mobile telephone, tablet computer, computer, telecommunication network base station or any other network connected device, or as a method.
  • the method may be implemented as a computer program.
  • the computer program is then executed in a computing device.
  • the apparatus such as apparatus for positioning, is configured to perform one of the methods described above.
  • the apparatus comprises necessary hardware components. These may include at least one processor, at least one memory, at least one network connection, a bus and the like. Instead of dedicated hardware components, it is possible to share for example memories or processors with other components or access at a cloud service, centralized computing unit or other resource that can be used over a network connection.
  • a computer program may be stored/distributed on a suitable medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems.
  • a suitable medium such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Data Mining & Analysis (AREA)
  • Computing Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Multimedia (AREA)
  • Medical Informatics (AREA)
  • Automation & Control Theory (AREA)
  • Databases & Information Systems (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Probability & Statistics with Applications (AREA)
  • Navigation (AREA)
  • Traffic Control Systems (AREA)

Abstract

Apparatuses and methods are provided for determining a position of a device. The positioning is based on acquiring images that have unique position identifying features, which are either natural or man built. These position identifying features are used in generating a multi-dimensional signature. The multi-dimensional signature is processed so that a compact signature with reduced dimensions is achieved. The position of the device can be retrieved using the compact signature.

Description

APPARATUSES AND METHODS FOR POSITIONING A DEVICE
TECHNICAL FIELD
Generally, the following description relates to the field of positioning systems. More specifically, the following description relates to apparatuses and methods for positioning a device.
BACKGROUND
Modern cars, mobile phones and many other devices are often equipped with a positioning system. The positioning system is used to measure a location of the device. The location information may be used in many applications. For example, navigation and other map related applications require that the position of the device is known so that navigation instructions can be formed.
The most common way to derive the exact location information is to use satellite-based navigation systems. In satellite navigation, a device determines the current location by receiving signals transmitted along a line of sight from a plurality of satellites. The device, such as a mobile phone or a vehicle, has electronic receivers for receiving the signals that can be used in positioning of the device. Typically, a global navigation satellite system does not require Internet reception; however, in some circumstances it may be used in assisting the positioning process.
As mentioned above, positioning signals are received along a line of sight. Thus, positioning is possible only when there is a line of sight to at least four satellites. This limits positioning in many locations to outdoors only. Furthermore, sometimes buildings, mountains or other obstacles are preventing visibility to at least four satellites, and the positioning is not possible. This is particularly a problem underground, for example in parking garages and road tunnels, or inside buildings. The visibility problem has been addressed by developing indoor positioning systems. When coarse positioning is sufficient, the mobile communication network may be used. Special purpose indoor positioning systems are capable of providing the exact location. However, they are normally based on geolocated beacons, which transmit signals using Bluetooth, Wi-Fi or other suitable radio technology. These beacons are expensive and require maintenance. Another option is to use a plurality of different sensors and maps, either separately or together. For example, a vehicle can use a map and odometry to support navigation when entering a tunnel; however, it typically drifts quite quickly. Furthermore, when a mobile phone or a tablet computer is used instead of an integrated navigation device, the odometry information is not accessible. Thus, there is a need for improved positioning systems.
SUMMARY
In the following disclosure, apparatuses and methods for determining a position of a device are disclosed. The positioning is based on acquiring images that have unique position identifying features, which are either natural or man built. These position identifying features are used in generating a multi-dimensional signature. The multi-dimensional signature is processed so that a compact signature with reduced dimensions is achieved. The position of the device can be retrieved using the compact signature.
In a first aspect a method for determining a position of an apparatus is disclosed. The method comprises acquiring an image; extracting a position-identifying feature from the acquired image; generating a multi-dimensional signature of the acquired image using the position-identifying feature; generating a compact signature by reducing dimensions of the multidimensional signature of the acquired image; and retrieving the position of the apparatus using the compact signature. It is beneficial to use position-identifying features from acquired images to determine the location of a device or a vehicle. This provides a possibility for accurate positioning also in locations where satellite-based positioning is not possible.
In an implementation of the first aspect, acquiring the image comprises acquiring the image in a road direction. It is beneficial to use images that are acquired in the road direction either forward or backward. This facilitates accurately measuring the distance between images and reduces the motion blur.
In an implementation of the first aspect, the method further comprises acquiring a plurality of images, extracting a position-identifying feature from each of the acquired images; generating a multi-dimensional signature of each of the acquired images using the positionidentifying feature; generating a compact signature by reducing dimensions of each of the multidimensional signatures; and retrieving the position of the apparatus using a plurality of compact signatures. It is beneficial to use a plurality of images as it improves the accuracy of the method in case of occlusion in one of the images.
In an implementation of the first aspect, the method further comprises measuring a geographical distance between the first and the last image. It is beneficial to measure the distance between the first and the last image, as the distance information improves the accuracy of the method by determining the proper length of the multidimensional signature.
In an implementation of the first aspect, retrieving the position of the apparatus comprises: generating a request comprising the compact signature; matching the compact signature of the request with a database comprising a plurality of compact signatures, wherein each of the compact signatures is associated with a position; and determining the position of an apparatus based on positions of the compact signatures matching the at least one signature. It is beneficial to have a database in the device so that the positioning can be done without a network connection, as it may not be available when the positioning is needed.
In an implementation of the first aspect, matching comprises finding a sequence of compact signatures that is the most similar to the compact signatures of the request. It is beneficial to try to find a sequence that is the most similar to the compact signatures of the request. The compact signature of the request may also have features that are not positionidentifying, and the compact signature does not need to be an exact match.
In an implementation of the first aspect, retrieving the position comprises generating a request comprising the compact signature; transmitting the request to an external database suitable for matching the compact signature of the request with a database comprising a plurality of compact signatures associated with the position of an apparatus based on positions of the compact signatures matching the at least one signature; and receiving the position of the apparatus as a response to the request. It is beneficial to use an external database that might be larger and also easier to maintain so that it is always updated.
In an implementation of the first aspect, the position identifying feature is extracted using a machine learning arrangement configured to find a relevant position-identifying feature. It is beneficial to use a machine learning arrangement in detecting position-identifying features, as the machine learning arrangement may be trained further. This improves positioning in all possible locations instead of only the trained position.
In a second aspect a computer program comprising computer program code is disclosed. The computer program code is configured to cause performing a method as described above when the computer program code is executed in a computing apparatus. It is beneficial to implement the positioning method as software using a circuitry for determining the position.
In a third aspect an apparatus is disclosed. The apparatus comprises a camera configured to acquire an image; and processing circuitry configured to: extract a position identifying feature from the acquired image; generate a multi-dimensional signature of the acquired image using the position identifying feature; generate a compact signature by reducing dimensions of the multidimensional signature of the acquired image; and retrieve the position of the apparatus using at least one compact signature including the compact signature. It is beneficial to use position-identifying features from acquired images to determine the location of a device or a vehicle. This provides a possibility for accurate positioning also in locations where satellite-based positioning is not possible.
In an implementation of the third aspect, the camera is configured to acquire the image in a road orientation. It is beneficial to use to use images that are acquired in the road direction either forward or backward. This facilitates accurately measuring the distance between images and reduces the motion blur.
In an implementation of the third aspect, the camera is configured to acquire a plurality of images; and the processing circuitry is configured to extract a position-identifying feature from each of the acquired images; generate a multi-dimensional signature of each of the acquired images using the position-identifying feature; generate a compact signature by reducing dimensions of each of the multidimensional signatures; and retrieve the position of the apparatus using a plurality of compact signatures. It is beneficial to use a plurality of images as it improves the accuracy of the method in case of occlusion in one of the images.
In an implementation of the third aspect, the processing circuitry is further configured to: measure a geographical distance between the first and the last image. It is beneficial to measure the distance between the first and the last image, as the distance information improves the accuracy of the method by determining the proper length of the multidimensional signature.
In an implementation of the third aspect, when retrieving the position of the apparatus, the processing circuitry is further configured to: generate a request comprising the compact signature; match one or more compact signatures of the request with a database comprising a plurality of compact signatures, wherein each of the compact signatures is associated with a position; and determine the position of an apparatus based on positions of the compact signatures matching the at least one signature. It is beneficial to have a database in the device so that the positioning can be done without a network connection, as it may not be available when the positioning is needed.
In an implementation of the third aspect, when matching, the processing circuitry is further configured to find a sequence of compact signatures that is the most similar to the compact signatures of the request. It is beneficial to try to find a sequence that is the most similar to the compact signatures of the request. The compact signature of the request may also have features that are not position-identifying, and the compact signature does not need to be an exact match.
In an implementation of the third aspect, when retrieving the position of an apparatus, the processing circuitry is further configured to: generate a request comprising the compact signature; transmit the request to an external service suitable for matching one or more compact signatures of the request with a database comprising a plurality of compact signatures associated with the position of an apparatus based on positions of the compact signatures matching at least one compact signature; and receive the position of the apparatus as a response to the request. It is beneficial to use an external database that might be larger and also easier to maintain so that it is always updated.
In an implementation of the third aspect, the processing circuitry is configured to extract the position identifying feature using a machine learning arrangement configured to find a relevant position identifying feature. It is beneficial to use a machine learning arrangement in detecting position-identifying features, as the machine learning arrangement may be trained further. This improves positioning in all possible locations instead of only the trained position.
In a fourth aspect a server for positioning is disclosed. The server comprises a database comprising a plurality of compact signatures associated with a position; and a processing circuitry configured to: receive a request comprising at least one compact signature; match at least one compact signature of the request with a database comprising a plurality of compact signatures, wherein each of the compact signatures is associated with a position; and determine the position of a requesting apparatus based on positions of the compact signatures matching the at least one compact signature. It is beneficial to use an external server receiving requests from a positioning device. The external server may have a database that is larger and also easier to maintain so that it is always updated.
In an implementation of the fourth aspect, matching comprises finding a sequence of compact signatures that is the most similar to the compact signatures of the request. It is beneficial to try to find a sequence that is the most similar to the compact signatures of the request. The compact signature of the request may also have features that are not positionidentifying, and the compact signature does not need to be an exact match.
The foregoing and other objects are achieved by the subject matter of the independent claims. Further implementation forms are apparent from the dependent claims, the description and the figures.
The principles discussed in the present description can be implemented in hardware and/or software. BRIEF DESCRIPTION OF THE DRAWINGS
Further example embodiments will be described with respect to the following figures, wherein:
Fig. 1 shows an example of a method for generating compact signatures;
Fig. 2 shows an example of a scheme for generating compact signatures;
Fig. 3 shows an example of a training process for machine learning involved in road signature generation;
Fig. 4 shows an example of a scheme for an acquisition vehicle; and
Fig. 5 shows an example of an arrangement for positioning a car.
In the figures, identical reference signs will be used for identical or at least functionally equivalent features.
DETAILED DESCRIPTION OF THE EMBODIMENTS
In the following description, reference is made to the accompanying drawings, which form part of the disclosure, and in which are shown, by way of illustration, specific aspects in which the present apparatuses and methods may be provided. It is understood that other aspects may be utilized and structural or logical changes may be made without departing from the scope of the claims. Thus, the following detailed description is not to be taken in a limiting sense.
For instance, it is understood that a disclosure in connection with a described method may also hold true for a corresponding device or system configured to perform the method and vice versa. For example, if a specific method step is described, a corresponding device may include a unit to perform the described method step, even if such unit is not explicitly described or illustrated in the figures. Further, it is understood that the features of the various exemplary aspects described herein may be combined with each other, unless specifically noted otherwise.
Fig. 1 discloses an example of a method for retrieving a position of an apparatus, such as a mobile phone, vehicle or any similar apparatus comprising a camera and an access to a database comprising position associated compact signatures or other similar references that can be derived from an image. In the example of Fig. 1 , the method is initiated by acquiring an image, step 100. The image is acquired using a conventional camera, for example a camera unit of a mobile phone or a camera of a vehicle. The acquired image is then processed in order to extract position-identifying features from the image, step 110. When the features are extracted, it is possible that also non-position identifying features are extracted. It is possible to use additional analysis tools for removing them; however, the positioning is possible when acquired images include enough position identifying features. In some cases, even one position identifying feature may be enough. For example, some of the road signs are unique, and when they are extracted the position can be accurately determined, even if the image comprises additional features. The extracting step may include identifying the extracted features. For example, when the feature is identified to be a vehicle, it may be discarded as it is highly unlikely that a vehicle can be used for determining the position. From the extracted features, a multi-dimensional signature is generated, step 120. The multi-dimensional signature provides very accurate identifying information of the extracted feature. It is used as a source for generating a compact signature, step 130. Finally, the position of the device is retrieved using one or more generated compact signatures, step 140.
In the above, an example of a method is disclosed. The method may be implemented in one device comprising a processing circuitry. The processing circuitry comprises at least one processor for executing computer programs and at least one memory for storing computer programs and related data. The memory may be volatile or non-volatile depending on the task required. The processor is configured to execute computer programs that are configured to cause performing the method as explained. Instead of one device, two or more devices may be used as explained in the following disclosure.
The example method of Fig. 1 may be completely performed in the device. This implementation may be considered as an offline implementation. In an online implementation, one or more steps are executed by a remote service, such as a cloud service or a positioning server. Step 100 of acquiring images is performed in the device that is being positioned. As modern cameras are of high resolution, it is typically not desired to send the acquired images over the network to a remote service. In order to reduce network traffic, position-identifying features are typically extracted in the requesting device. Furthermore, generating multi-dimensional signatures and also generating compact signatures is typically done at the requesting device. This is beneficial, as compact signatures are smaller in size, and thus the request transmitted to a cloud or positioning service does not require as much bandwidth as it would require with images or multidimensional signatures. The position can be retrieved from an internal database stored in a memory of the device, or it can be retrieved from an external service. Retrieving comprises generating an appropriate request and receiving a response. Furthermore, retrieving comprises processing the request and transmitting a response. In an offline implementation, these are internal steps and signals within the device. In an online implementation, these tasks are performed by the external service, and the device does not need any further information on them.
Fig. 2 discloses an example of a scheme for generating compact signatures. The scheme may be used in several different devices or apparatuses, for example in mobile phones or cars. The following disclosure uses a car as an example; however, the scheme may be applied in any device having a camera unit for acquiring images and an access to a database, which may be located in a device itself or in a network service, such as a cloud service or server.
In the scheme, firstly one or more images are acquired, step 200. In the example of a car, these images are acquired in a road orientation or direction. This orientation may be forward or backward. One image may be sufficient for coarse positioning. However, when a method according to the scheme is used, for example in a navigation application, a plurality of images increases the accuracy of the method. Furthermore, the distance traveled between the first and the last or between each consecutive image may be measured and used in positioning. The acquired images can be ordinary images acquired with a camera that is already integrated into the acquiring device. For example, modern cars and mobile phones typically have one or more cameras. The acquired images are transmitted to an image encoder 210. In the image encoder, the image is processed for finding position identifying features in the image. A position identifying feature is a feature that can be used in positioning. Thus, the position identifying feature is an object, landmark or the like. The position identifying feature does not change or move rapidly over time. Examples of position identifying features include trees, rock formations, lakes, buildings, road signs, roads and streets as such and the like. Examples of other features include the sun, persons, animals and other cars. These features typically move and change their place, so they cannot be used in determining the position. As an outcome, the image encoder 210 provides a set of image features 220 that includes one or more position identifying features.
The image encoder comprises a convolutional neural network (CNN) followed by a global pooling layer. The CNN outputs k feature maps that are summarized to one value by a feature map by the global pooling layer. The obtained k dimensional vector is finally normalized.
In the example of Fig. 2, the outcome 220 from the image encoder 210 is fed to the signature generator 230. The position identifying features 220, together with the distance information, are then used in generating a signature. The road signature generation module uses as inputs a continuous sequence of images from time t to t + n and the distance traveled dt+n from the position of the camera at time t and the position of the camera at time t + n. The number of /vectors generated is controlled by the spatial resolution factor c/A of the signature generator and can be computed using the relation: I = . The attention mechanism inside the signature generator uses a temporal filter to combine the input image features into road signature vectors.
The signature generator 230 returns a set of I vectors 240 that is called a road signature. Each vector of the signature has a dimension k and two consecutive vectors are separated by a distance of c/A. The set of I vectors 240 is then fed to dimension reduction 250. In the example of Fig. 2, the dimension reduction is an unsupervised machine learning method that is configured to reduce the dimension of the road signature. Dimension reduction is applied to each road signature vector to reduce its size from k to m, with m « k. The outcome is a compact road signature 260, which is a set of I vectors, wherein each of the vectors has a dimension m, which is considerably less than k, and two consecutive vectors are separated by a distance of c/A. In the above described processes of image encoding, signature generation and dimension reduction are differentiable operations. That means the signature generation process is a fully differentiable operation, and thus road signatures can be optimized end to end for localization purposes and all the road signature generator components may be optimized jointly during the training. The training process is described below.
The process as described above is an example of a process that may be used in the examples discussed below. Other similar processes providing an association between one or more acquired images and a location may be used.
In Fig. 3, an example of a training process for machine learning involved in the road signature generator is disclosed. When producing discriminative compact signatures for precise localization, the road signature generator can be trained with the following principles.
The example of Fig. 3 shows two separate pipelines involving road signature generation. The first pipeline involves receiving a reference sequence of images and distance 300 as a reference input, which is processed by a road signature generator 302. As a result, one or more reference compact signatures 304 are computed. The process of generating road signatures may be similar to the process described above in the example of Fig. 2. The second pipeline receives a target sequence of images and distance 310 as a target input, which is processed by a road signature generator 304. As a result, one or more target compact signatures 314 are computed.
The first pipeline uses a reference input of a continuous sequence of images from time t to t + n and the distance traveled dt+n from the position of the camera at time t and the position of the camera at time t + n. The second pipeline uses a target input of a continuous sequence of images from time f’to t’ + m and the distance traveled df+m from the position of the camera at time t’ and the position of the camera at time t ’ + m. The target sequence is recorded on the same road as the reference sequence at a different time t t The target sequence is longer than the reference sequence: df+m > dt+n. The road signature generators 302 and 312 both use a Siamese network sharing the same set of parameter weights to compute the reference and the target signatures from the image sequences. These Siamese networks return the compact signature of the reference and the target sequence of images. As the distance traveled in the target sequence is longer than the distance traveled in the reference sequence, the target signature is longer than the reference signature.
A road signature matcher 320 computes signal similarity 322 between the reference and the target signature using sliding-window functions, for example Zero-mean Normalized Cross Correlation. The road signature matcher 320 returns a similarity measurement 330 of various sizes depending on the signature length. High similarity results in a low value in the similarity measurement signal. The road signature matcher 320 creates a ground truth signal from the ground truth spatial and temporal alignment of the reference and the target sequence 360. It is computed by generating a narrow 1 D Gaussian function centered at the overlapping position of the reference and target sequence. Ground truth alignment of the reference and target sequence are obtained from GPS measurement or by content-based image retrieval. A training signal is obtained by computing the cross-entropy loss 350 between the ground truth signal and the similarity measurement obtained from the road signature matcher 320. Before computing the loss, the similarity measurement obtained from the road signature matcher 320 is converted to probability 340 using a Softmax function 335 on the opposed signal. A gradient of the loss is computed regarding the input sequences and back-propagated through the whole pipeline to correct the weight parameters of the road signature generator.
Fig. 4 discloses a method for an acquisition vehicle 400 that is used in collecting location associated data and building a database to be used in image-based positioning. The acquisition vehicle of the example of Fig. 4 comprises an acquisition subsystem including an inertial navigation system, a satellite-based 410 navigation system antenna and an odometry system. The inertial navigation system is coupled to a camera, oriented along the longitudinal axis of the vehicle. Additionally, a subsystem may be used to ensure the synchronization of the image capture process and the inertial navigation system output. The additional subsystem saves images in a dedicated memory. The acquisition subsystem acquires images precisely geolocated and geotagged along the traveled road. From the recorded sequence of images 420 and the traveled distance, the road signature generator 430 computes a compact road signature. The road signature computation can be done on the embedded vehicle to reduce the inboard storage usage or on a remote server to reduce the computational load on the acquisition vehicle. The road signature generator 430 produces a compact road signature 440 precisely longitudinally localized on the road being traveled by the acquisition vehicle. The precisely localized compact road signature 440 is stored in a geographic database 450. The geographic database 450 is then used as a source of information, or the information stored can be packaged for later use.
In Fig. 5, an example of an arrangement for positioning a car 500 is shown. The user is in a vehicle, equipped with a device that combines a GNSS application and a camera module, with a geodatabase in a memory. When entering GNSS restricted areas, the GNSS application will trigger an online process which will determine the position of the car using an image-based position system. The car 500 acquires images 510 from its camera module as well as the traveled distance, or vehicle speed to derive the traveled distance, of the image sequence. The vehicle uses a road signature generator 512 to compute a compact road signature 514 from the online-captured image sequence. The compact road signature 514 computed online by the user device or vehicle is called a target road signature.
A geographic database 520 of precomputed compact road signatures may be stored or partially stored on the user vehicle or device, and it is queried when entering a GNSS restricted area. In another implementation, the geographical database is accessed using an internet connection of the vehicle.
The geographic database 520 is used to retrieve a reference georeferenced compact road signature. This is done from a last valid GNSS position received before entering the GNSS restricted area. The user device or vehicle retrieves from the embedded geographic database 520 the precomputed compact road signature 522 geolocated on the road being traveled by the user. This compact road signature 522, precisely longitudinally located on the road, is called a reference compact road signature.
The vehicle 500 proceeds to target and reference signatures matching using a road signature matcher 530. The road signature matcher 530 produces a similarity signal between the target and the reference signatures for similarity measurement 532. The pick in the similarity signal represents the optimal alignment between the target and the reference signatures. Thus, it provides the relative position of the target signature to the reference signature that produces the highest similarity measurement.
Finally, the road signature georeferencing module 540 of the user device or vehicle uses the similarity signal from the road signature matcher 530 and the precise position of the reference compact road signature to compute the precise position 550 of the target compact road signature along the driven road and to derive the current position of the user vehicle. Finally, the device location may be shown in an application that normally relies on satellite positioning.
As explained above, the arrangements using positioning as described above may be implemented in hardware, such as a mobile telephone, tablet computer, computer, telecommunication network base station or any other network connected device, or as a method. The method may be implemented as a computer program. The computer program is then executed in a computing device.
The apparatus, such as apparatus for positioning, is configured to perform one of the methods described above. The apparatus comprises necessary hardware components. These may include at least one processor, at least one memory, at least one network connection, a bus and the like. Instead of dedicated hardware components, it is possible to share for example memories or processors with other components or access at a cloud service, centralized computing unit or other resource that can be used over a network connection.
The apparatus for positioning and the corresponding method have been described in conjunction with various embodiments herein. However, other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention from a study of the drawings, the disclosure, and the appended claims. In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. A computer program may be stored/distributed on a suitable medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems.

Claims

1 . A method for determining a position of an apparatus, the method comprising: acquiring an image; extracting a position-identifying feature from the acquired image; generating a multi-dimensional signature of the acquired image using the positionidentifying feature; generating a compact signature by reducing dimensions of the multidimensional signature of the acquired image; and retrieving the position of the apparatus using the compact signature.
2. The method according to any of claim 1 , wherein acquiring the image comprises acquiring the image in a road direction.
3. The method according to claim 1 or 2, wherein the method further comprises: acquiring a plurality of images, extracting a position-identifying feature from each of the acquired images; generating a multi-dimensional signature of each of the acquired images using the position-identifying feature; generating a compact signature by reducing dimensions of each of the multidimensional signatures; and retrieving the position of the apparatus using a plurality of compact signatures.
4. The method according to claim 3, wherein the method further comprises measuring a geographical distance between the first and the last image.
5. The method according to any of claims 1 - 4, wherein retrieving the position of the apparatus comprises: generating a request comprising the compact signature; matching the compact signature of the request with a database comprising a plurality of compact signatures, wherein each of the compact signatures is associated with a position; and determining the position of an apparatus based on positions of the compact signatures matching the at least one signature.
6. The method according to claim 5, wherein matching comprises finding a sequence of compact signatures that is the most similar to the compact signatures of the request.
7. The method according to any of claims 1 - 4, wherein retrieving the position comprises generating a request comprising the compact signature; transmitting the request to an external database suitable for matching the compact signature of the request with a database comprising a plurality of compact signatures associated with the position of an apparatus based on positions of the compact signatures matching at least one signature; and receiving the position of the apparatus as a response to the request.
8. The method according to any of claims 1 - 7, wherein the position identifying feature is extracted using a machine learning arrangement configured to find a relevant position identifying feature.
9. A computer program comprising computer program code, wherein the computer program code is configured to cause performing a method according to any of claims 1 - 8 when the computer program code is executed in a computing apparatus.
10. An apparatus comprising: a camera configured to acquire an image; and processing circuitry configured to: extract a position identifying feature from the acquired image; generate a multi-dimensional signature of the acquired image using the position identifying feature; generate a compact signature by reducing dimensions of the multidimensional signature of the acquired image; and retrieve the position of the apparatus using at least one compact signature including the compact signature.
11 . The apparatus according to claim 10, wherein the camera is configured to acquire the image in a road direction.
12. The apparatus according to claim 10 or 11 , wherein the camera is configured to acquire a plurality of images; and the processing circuitry is configured to extract a position-identifying feature from each of the acquired images; generate a multi-dimensional signature of each of the acquired images using the position-identifying feature; generate a compact signature by reducing dimensions of each of the multidimensional signatures; and retrieve the position of the apparatus using a plurality of compact signatures.
13. The apparatus according to claim 12, wherein the processing circuitry is further configured to: measure a geographical distance between the first and the last image.
14. The apparatus according to any of claims 10 - 13, wherein when retrieving the position of the apparatus, the processing circuitry is further configured to: generate a request comprising the compact signature; match one or more compact signatures of the request with a database comprising a plurality of compact signatures, wherein each of the compact signatures is associated with a position; and determine the position of an apparatus based on positions of the compact signatures matching the at least one signature.
18
15. The apparatus according to claim 14, wherein when matching, the processing circuitry is further configured to find a sequence of compact signatures that is the most similar to the compact signatures of the request.
16. The apparatus according to any of claims 10 - 13, wherein when retrieving the position of an apparatus, the processing circuitry is further configured to: generate a request comprising the compact signature; transmit the request to an external service suitable for matching one or more compact signatures of the request with a database comprising a plurality of compact signatures associated with the position of an apparatus based on positions of the compact signatures matching at least one compact signature; and receive the position of the apparatus as a response to the request.
17. The apparatus according to any of claims 10 - 16, wherein the processing circuitry is configured to extract the position identifying feature using a machine learning arrangement configured to find a relevant position identifying feature.
18. A server for positioning comprising: a database comprising a plurality of compact signatures associated with a position; and a processing circuitry configured to: receive a request comprising at least one compact signature; match at least one compact signature of the request with a database comprising a plurality of compact signatures, wherein each of the compact signatures is associated with a position; and determine the position of a requesting apparatus based on positions of the compact signatures matching the at least one compact signature.
19. The server according to claim 18, wherein matching comprises finding a sequence of compact signatures that is the most similar to the compact signatures of the request.
19
EP20799694.3A 2020-10-29 2020-10-29 Apparatuses and methods for positioning a device Pending EP4204766A1 (en)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/EP2020/080338 WO2022089743A1 (en) 2020-10-29 2020-10-29 Apparatuses and methods for positioning a device

Publications (1)

Publication Number Publication Date
EP4204766A1 true EP4204766A1 (en) 2023-07-05

Family

ID=73037990

Family Applications (1)

Application Number Title Priority Date Filing Date
EP20799694.3A Pending EP4204766A1 (en) 2020-10-29 2020-10-29 Apparatuses and methods for positioning a device

Country Status (3)

Country Link
EP (1) EP4204766A1 (en)
CN (1) CN114710970B (en)
WO (1) WO2022089743A1 (en)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160187144A1 (en) * 2014-12-26 2016-06-30 Here Global B.V. Selecting Feature Geometries for Localization of a Device
US20180157939A1 (en) * 2016-12-05 2018-06-07 Avigilon Corporation System and method for appearance search

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9042907B2 (en) * 2012-03-09 2015-05-26 Blackberry Limited Location identification of a portable electronic device based on characteristics of an operating environment of the portable electronic device
CN103927387B (en) * 2014-04-30 2017-06-16 成都理想境界科技有限公司 Image indexing system and its correlation technique and device
CN111380545B (en) * 2015-02-10 2024-11-12 御眼视觉技术有限公司 Method, server, autonomous vehicle and medium for autonomous vehicle navigation
WO2018017793A1 (en) * 2016-07-21 2018-01-25 Intelligent Technologies International, Inc. System and method for creating, updating, and using maps generated by probe vehicles
US9940729B1 (en) * 2016-11-18 2018-04-10 Here Global B.V. Detection of invariant features for localization
US10571280B2 (en) * 2017-05-09 2020-02-25 Toyota Research Institute, Inc. Systems and methods for localizing a vehicle using a roadway signature
US10416681B2 (en) * 2017-07-12 2019-09-17 Mitsubishi Electric Research Laboratories, Inc. Barcode: global binary patterns for fast visual inference
CN110658539B (en) * 2018-06-29 2022-03-18 比亚迪股份有限公司 Vehicle positioning method, device, vehicle and computer readable storage medium
CN111241218B (en) * 2018-11-28 2024-07-12 驭势科技(北京)有限公司 A method and device for identifying a location
CN110929589B (en) * 2019-10-31 2023-07-07 浙江大华技术股份有限公司 Method, apparatus, computer apparatus and storage medium for identifying vehicle characteristics

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160187144A1 (en) * 2014-12-26 2016-06-30 Here Global B.V. Selecting Feature Geometries for Localization of a Device
US20180157939A1 (en) * 2016-12-05 2018-06-07 Avigilon Corporation System and method for appearance search

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
See also references of WO2022089743A1 *
WONG ALEXANDER ET AL: "Robust Hough-Based Symbol Recognition Using Knowledge-Based Hierarchical Neural Networks", 1 January 2008 (2008-01-01), XP055819058, Retrieved from the Internet <URL:https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.217.3744&rep=rep1&type=pdf> [retrieved on 20210629] *

Also Published As

Publication number Publication date
CN114710970B (en) 2026-02-06
WO2022089743A1 (en) 2022-05-05
CN114710970A (en) 2022-07-05

Similar Documents

Publication Publication Date Title
CN109059906B (en) Vehicle positioning method and device, electronic equipment and storage medium
CN116097128B (en) Method and apparatus for determining vehicle position
KR20190082071A (en) Method, apparatus, and computer readable storage medium for updating electronic map
EP2450667A1 (en) Vision system and method of analyzing an image
KR102522923B1 (en) Apparatus and method for estimating self-location of a vehicle
CN114565674B (en) Method and device for purely visually positioning urban structured scene of automatic driving vehicle
EP3746744B1 (en) Methods and systems for determining geographic orientation based on imagery
CN112689234B (en) Indoor vehicle positioning method, device, computer equipment and storage medium
US20120176494A1 (en) System and method for virtual range estimation
CN111159459B (en) Landmark positioning method, landmark positioning device, computer equipment and storage medium
CN109596121A (en) A kind of motor-driven station Automatic Targets and space-location method
US20220122316A1 (en) Point cloud creation
WO2020049089A1 (en) Methods and systems for determining the position of a vehicle
JP7665207B2 (en) Method for creating a correction function to improve the accuracy of a GPS device - Patents.com
JP6856750B2 (en) Method and equipment
WO2021168795A1 (en) Navigation method and device
KR20200002219A (en) Indoor navigation apparatus and method
CN114283397A (en) Global relocation method, device, equipment and storage medium
US11783504B2 (en) Geolocation system
CN111480165A (en) Method for creating a feature-based localization map for a vehicle taking into account the feature structure of an object
CN114710970B (en) Apparatus and method for locating devices
WO2021199024A1 (en) Automatic labelling of objects in images
CN110674327A (en) More accurate positioning method and device
CN117606506A (en) A vehicle positioning method, device, electronic equipment and medium
CN118730136B (en) A method, device, equipment and medium for downloading maps of an autonomous driving vehicle

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: UNKNOWN

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20230330

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: SHENZHEN YINWANG INTELLIGENTTECHNOLOGIES CO., LTD.

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20250702