WO2019205853A1 - 相机姿态追踪过程的重定位方法、装置、设备及存储介质 - Google Patents

相机姿态追踪过程的重定位方法、装置、设备及存储介质 Download PDF

Info

Publication number
WO2019205853A1
WO2019205853A1 PCT/CN2019/079355 CN2019079355W WO2019205853A1 WO 2019205853 A1 WO2019205853 A1 WO 2019205853A1 CN 2019079355 W CN2019079355 W CN 2019079355W WO 2019205853 A1 WO2019205853 A1 WO 2019205853A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
feature point
key frame
entry storage
descriptor
Prior art date
Application number
PCT/CN2019/079355
Other languages
English (en)
French (fr)
Inventor
林祥凯
凌永根
暴林超
刘威
Original Assignee
腾讯科技（深圳）有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 腾讯科技（深圳）有限公司 filed Critical 腾讯科技（深圳）有限公司
Priority to EP19792551.4A priority Critical patent/EP3786894A4/en
Publication of WO2019205853A1 publication Critical patent/WO2019205853A1/zh
Priority to US16/900,634 priority patent/US11189037B2/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/587—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using geographical or spatial information, e.g. location
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22—Indexing; Data structures therefor; Storage structures
    • G06F16/2228—Indexing structures
    • G06F16/2255—Hash tables
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/51—Indexing; Data structures therefor; Storage structures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/55—Clustering; Classification
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/22—Matching criteria, e.g. proximity measures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/20—Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G06T7/75—Determining position or orientation of objects or cameras using feature-based methods involving models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30244—Camera pose
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
    • G06T2219/20—Indexing scheme for editing of 3D models
    • G06T2219/2004—Aligning objects, relative positioning of parts

Definitions

  • the embodiments of the present application relate to the field of augmented reality, and in particular, to a method, a device, a device, and a storage medium for relocating a camera attitude tracking process.
  • Visual SLAM refers to the technique of estimating the movement of the body while the camera is the main body, without the prior information of the environment, establishing a model of the environment during the movement.
  • SLAM can be used in the field of AR (Augmented Reality), robotics and unmanned driving.
  • the first frame image captured by the camera is usually used as a marker image (Anchor).
  • the device tracks the feature points commonly shared between the current image and the mark image, and calculates the pose change of the camera in the real world according to the change of the feature point position between the current image and the mark image.
  • the feature point loss (Lost) in the current image may occur, and the tracking cannot be continued.
  • the current image needs to be relocated using the SLAM relocation method.
  • the embodiment of the present invention provides a relocation method, a device, a device, and a storage medium for a camera attitude tracking process, which can solve the problem that the SLAM relocation method in the related art is directly used in the AR usage scenario.
  • the technical solution is as follows:
  • a relocation method of a camera attitude tracking process is provided, which is applied to a device having a camera for sequentially performing camera attitude tracking of a plurality of marker images, the method comprising:
  • the initial pose parameter being used to instruct the camera to acquire the The camera pose when the first image is marked;
  • the target camera pose is that the camera is collecting the current Camera pose when image
  • Retargeting obtains a target pose parameter corresponding to the target camera pose according to the initial pose parameter and the pose change amount.
  • a repositioning device for a camera pose tracking process for use in a device having a camera for sequentially performing camera pose tracking of a plurality of marker images, the device comprising :
  • An image acquisition module configured to acquire a current image acquired after the i-th mark image of the plurality of mark images, where i is an integer greater than one;
  • An information acquiring module configured to acquire an initial feature point and an initial pose parameter of the first one of the plurality of mark images when the current image meets a relocation condition, where the initial pose parameter is used to indicate a camera pose when the camera captures the first marker image;
  • a feature point tracking module configured to perform feature point tracking on the current image with respect to the first marker image, to obtain a target feature point that matches the initial feature point;
  • a change amount calculation module configured to calculate, according to the initial feature point and the target feature point, a pose change amount when the camera changes from the first camera pose to a target camera pose, where the target camera pose is a camera pose when the camera captures the current image;
  • a relocation module configured to retarget the target pose parameter corresponding to the target camera pose according to the initial pose parameter and the pose change amount.
  • an electronic device including a memory and a processor
  • At least one instruction is stored in the memory, the at least one instruction being loaded and executed by the processor to implement the relocation method as described above.
  • a computer readable storage medium having stored therein at least one instruction loaded by a processor and executed to implement relocation as described above method.
  • relocation can be realized in the Anchor-SLAM algorithm for tracking a plurality of consecutive marker images, thereby reducing the number of The possibility of tracking the interruption of the process, because the key frame image in the key frame image library can be multiple, if the key frame image library includes different key frame images covering the surrounding area of the camera acquisition scene, there is a great probability of relocation Success, which increases the chances of success in the relocation process.
  • FIG. 1 is a schematic diagram of a scenario of an AR application scenario provided by an exemplary embodiment of the present application
  • FIG. 2 is a schematic diagram of a scenario of an AR application scenario provided by an exemplary embodiment of the present application
  • FIG. 3 is a schematic diagram of a schematic diagram of an Anchor-Switching AR System algorithm provided by an exemplary embodiment of the present application
  • FIG. 4 is a structural block diagram of an electronic device provided by an exemplary embodiment of the present application.
  • FIG. 5 is a flowchart of a method for relocating a camera attitude tracking process according to an exemplary embodiment of the present application
  • FIG. 6 is a schematic diagram of comparison of a first marker image, a current image, and a first-order key frame image provided by an exemplary embodiment of the present application;
  • FIG. 7 is a flowchart of a method for relocating a camera attitude tracking process according to an exemplary embodiment of the present application.
  • FIG. 8 is a schematic diagram of a data structure of hash index information provided by an exemplary embodiment of the present application.
  • FIG. 9 is a flowchart of a key frame image adding process provided by an exemplary embodiment of the present application.
  • FIG. 10 is a flowchart of a hash classification process provided by an exemplary embodiment of the present application.
  • FIG. 11 is a flowchart of a key frame image query process provided by an exemplary embodiment of the present application.
  • FIG. 12 is a flowchart of a key frame image deletion process provided by an exemplary embodiment of the present application.
  • FIG. 13 is a flowchart of a feature point tracking method provided by an exemplary embodiment of the present application.
  • FIG. 14 is a schematic diagram of a principle of a feature point tracking method according to an exemplary embodiment of the present application.
  • FIG. 15 is a flowchart of a method for relocating a camera attitude tracking process according to an exemplary embodiment of the present application.
  • 16 is a block diagram of a relocation device of a camera pose tracking process provided by an exemplary embodiment of the present application.
  • FIG. 17 is a block diagram of a feature point tracking device provided by an exemplary embodiment of the present application.
  • FIG. 18 is a block diagram of an electronic device provided by an exemplary embodiment of the present application.
  • AR Augmented Reality
  • a method of calculating camera pose parameters in the real world (or 3D world, real world) in real time during camera acquisition of images, according to the camera pose parameters collected by the camera The technique of adding virtual elements to an image.
  • Virtual elements include, but are not limited to, images, video, and 3D models.
  • the goal of AR technology is to interact with the virtual world on the screen in the real world.
  • the camera pose parameters include a displacement vector used to characterize the displacement distance of the camera in the real world and a rotation matrix used to characterize the angle of rotation of the camera in the real world.
  • the device adds a virtual character image to the image captured by the camera.
  • the image captured by the camera changes, and the orientation of the avatar changes. It simulates that the avatar is still in the image, and the camera changes with position and posture.
  • the effect of capturing images and avatars gives the user a realistic three-dimensional picture.
  • Anchor-Switching AR System is based on a plurality of marker image (Anchor) camera attitude tracking to determine the camera pose parameters in a natural scene, and then superimpose the virtual world AR system on the image captured by the camera according to the camera pose parameters.
  • Anchor marker image
  • IMU Inertial Measurement Unit
  • an IMU consists of three single-axis accelerometers and three single-axis gyros.
  • the accelerometer is used to detect the acceleration signal of each object in each coordinate axis of the three-dimensional coordinate system, and then calculate the displacement vector; Used to detect the rotation matrix of an object in a three-dimensional coordinate system.
  • the IMU includes a gyroscope, an accelerometer, and a geomagnetic sensor.
  • the three-dimensional coordinate system is established as follows: 1.
  • the X-axis is defined by the vector product Y*Z. In the current position of the device, the X-axis points to the east in a direction tangent to the ground; 2.
  • the Y-axis is At the current position of the device, it points in the direction tangent to the ground to the north pole of the earth's magnetic field; 3.
  • the Z axis points to the sky and is perpendicular to the ground.
  • the present application provides a relocation method suitable for the Anchor-Switching AR System algorithm.
  • the Anchor-Switching AR System algorithm divides the motion process of the camera into at least two tracking processes for tracking, and each tracking process corresponds to a respective marker image.
  • a preset condition for example, the feature point that can be matched is less than a preset threshold
  • the previous image of the current image is determined as the i+1th marker image, and the i+1th segment tracking process is turned on.
  • i is a positive integer.
  • FIG. 3 is a schematic diagram showing the principle of an Anchor-Switching AR System algorithm provided by an exemplary embodiment of the present application.
  • an object 320 is present, the device 340 provided with the camera is moved by the user, and a multi-frame image 1-6 including the object 320 is captured during the movement.
  • the device determines image 1 as the first marker image (born-anchor or born-image) and records the initial pose parameter, which may be acquired by the IMU, and then performs feature point tracking on the image 2 relative to the image 1.
  • the pose parameter of the camera when the image 2 is captured is calculated; the image 3 is tracked with respect to the image 1 , and the camera is calculated according to the initial pose parameter and the feature point tracking result.
  • the pose parameter when the image 3 is captured; the feature 4 is tracked with respect to the image 1 , and the pose parameter of the camera when the image 4 is captured is calculated based on the initial pose parameter and the feature point tracking result.
  • the image 5 is tracked with respect to the image 1 , and if the feature point tracking effect is worse than the preset condition (for example, the number of matching feature points is small), the image 4 is determined as the second marker image, and the image 5 is Performing feature point tracking with respect to the image 4, calculating the amount of displacement change of the camera between the captured image 4 and the image 5, and calculating the amount of displacement change between the captured image 4 and the image 1 and the initial pose parameter, The pose parameter of the camera when taking image 5. Then, the image 6 is tracked with respect to the image 4, and so on. If the feature point tracking effect of the current image is deteriorated, the previous frame image of the current image can be determined as a new mark image, and the new mark is switched. Feature point tracking is performed again after the image.
  • the preset condition for example, the number of matching feature points is small
  • feature point tracking may employ an algorithm based on a visual odometer principle, such as a feature point method or a direct method.
  • a visual odometer principle such as a feature point method or a direct method.
  • the Anchor-Switching AR System tracking process may be lost. Loss phenomenon means that enough feature points cannot be matched in the current image, resulting in tracking failure.
  • the device includes a processor 420, a memory 440, a camera 460, and an IMU 480.
  • Processor 420 includes one or more processing cores, such as a 4-core processor, an 8-core processor, and the like.
  • the processor 420 is configured to execute at least one of instructions, code, code segments, and programs stored in the memory 440.
  • the processor 420 is electrically connected to the memory 440.
  • processor 420 is coupled to memory 440 via a bus.
  • Memory 440 stores one or more instructions, code, code segments, and/or programs. The instructions, code, code segments and/or programs, when executed by processor 420, are used to implement the SLAM relocation method provided in the following embodiments.
  • the processor 420 is also electrically coupled to the camera 460.
  • processor 420 is coupled to camera 460 via a bus.
  • Camera 460 is a sensor device having image acquisition capabilities. Camera 460 may also be referred to as a camera, a photosensitive device, and the like. Camera 460 has the ability to continuously acquire images or acquire images multiple times.
  • camera 460 is located inside or outside the device.
  • the processor 420 is also electrically connected to the IMU 480.
  • the IMU 480 is configured to acquire the pose parameters of the camera every predetermined time interval, and record the time stamp of each set of pose parameters at the time of acquisition.
  • the camera's pose parameters include: displacement vector and rotation matrix. Among them, the rotation matrix acquired by IMU480 is relatively accurate, and the displacement vector acquired may have a large error due to the actual environment.
  • FIG. 5 a flow diagram of a method of relocating a camera pose tracking process provided in an exemplary embodiment of the present application is shown. This embodiment is exemplified by the application of the relocation method to the apparatus shown in FIG. 4 for performing camera attitude tracking of a plurality of marker images in sequence.
  • the method includes:
  • Step 502 Acquire a current image acquired after the i-th mark image in the plurality of mark images.
  • the camera in the device collects a frame image at a preset time interval to form an image sequence.
  • the camera acquires a frame of image forming image sequence according to a preset time interval during motion (translation and/or rotation).
  • the device determines the first frame image in the image sequence (or one frame image in the first few frames of images that meets the predetermined condition) as the first marker image, and performs the subsequently acquired image on the first marker image.
  • Feature point tracking, and calculating a camera pose parameter according to the feature point tracking result if the feature point tracking effect of the current frame image is worse than a preset condition, determining the previous frame image of the current frame image as the second marker image, The subsequently acquired image is subjected to feature point tracking with respect to the second marker image, and the camera pose parameter of the camera is calculated according to the feature point tracking result, and so on.
  • the device can sequentially perform camera attitude tracking of a plurality of marked images in sequence.
  • the camera When in the i-th tracking process corresponding to the i-th mark image, the camera captures the current image.
  • the current image is a certain frame image acquired after the i-th mark image, where i is an integer greater than one.
  • Step 504 Acquire an initial feature point and an initial pose parameter of the first one of the plurality of mark images when the current image meets the relocation condition, where the initial pose parameter is used to indicate when the camera captures the first mark image.
  • the device determines if the current image meets the relocation criteria.
  • the relocation condition is used to indicate that the tracking process of the current image with respect to the i-th flag image fails, or the re-location condition is used to indicate that the accumulated error in the history tracking process has been higher than the preset condition.
  • the device tracks the current image relative to the i-th mark image, if there is no feature point matching the i-th mark image in the current image, or the i-th mark in the current image When the feature points of the image matching are less than the first number, it is determined that the tracking process of the current image with respect to the i-th flag image fails, and the relocation condition is met.
  • the device determines that the number of frames between the current image and the last relocated image is greater than the second number, determining that the accumulated error in the history tracking process is higher than a preset condition, or When it is determined that the number of mark images between the i-th mark image and the first mark image is greater than the third number, it is determined that the cumulative error in the history tracking process has been higher than the preset condition.
  • This embodiment does not limit the specific condition content of the relocation condition.
  • the device When the current image meets the relocation condition, the device attempts to track the current image relative to the first marker image. At this time, the device acquires an initial feature point in the first marked image of the cache and an initial pose parameter, which is used to indicate a camera pose when the camera captures the first marker image.
  • Step 506 Perform feature point tracking on the current image with respect to the first marker image to obtain a target feature point that matches the initial feature point.
  • each set of matching feature point pairs includes two initial feature points and target feature points that match each other.
  • the feature point tracking can use a visual odometer-based tracking algorithm, which is not limited in this application.
  • feature point tracking uses a KLT (Kanade-Lucas) optical flow tracking algorithm; in another embodiment, feature point tracking is based on a SIFT (Scale-Invariant Feature Transform) algorithm.
  • the feature point tracking is performed by the ORB feature point descriptor extracted by the SIFT feature point descriptor and the ORB (Oriented FAST and Rotated BRIEF) algorithm.
  • the specific algorithm for feature point tracking is not limited in this application, and the feature point tracking process may adopt a feature point method or a direct method.
  • the device performs feature point extraction on the first marker image to obtain N initial feature points; the device further performs feature point extraction on the current image to obtain M candidate feature points; and then M candidate feature points are successively Matching with N initial feature points to determine at least one set of matching feature point pairs.
  • Each set of matching feature point pairs includes: an initial feature point and a target feature point.
  • the initial feature point is a feature point on the first marker image, and the target feature point is a candidate feature point having the highest matching degree with the initial feature point on the current image.
  • the number of initial feature points is greater than or equal to the number of target feature points.
  • the number of initial feature points is 450 and the target feature points are 320.
  • Step 508 Calculate, according to the initial feature point and the target feature point, a pose change amount when the camera changes from the initial pose parameter to the target pose parameter, where the target pose parameter is used to represent a camera pose when the camera captures the current image;
  • the device calculates the homography matrix homography between the two image images according to the initial feature point and the target feature point; decomposes the homography matrix homography to obtain a pose when the camera changes from the initial posture parameter to the target posture parameter.
  • the homography matrix describes the mapping relationship between two planes. If the feature points in the natural scene (real environment) fall on the same physical plane, the motion estimation can be performed through the homography matrix.
  • the device decomposes the homography matrix by ransac to obtain a rotation matrix R relocalize and a translation vector T relocalize .
  • R relocalize camera is changed from the initial posture upon rotation matrix parameters to the target attitude parameters
  • T relocalize camera is changed from the initial posture parameters to the displacement vectors at the target attitude parameters.
  • Step 510 Retargeting according to the initial pose parameter and the pose change amount to obtain the target pose parameter.
  • the device transforms the initial pose parameter by the amount of pose change, and then re-positions the target pose parameter to calculate the camera pose when the camera captures the current image.
  • the relocation method provided by the embodiment can perform the Anchor-Switching in which a plurality of marker images are continuously tracked by relocating the current image and the first marker image when the current image meets the relocation condition.
  • Relocation in the AR System algorithm reduces the possibility of interruption of the tracking process, so that the SLAM relocation method in the related art is not applicable to the problem of the modified SLAM algorithm.
  • the relocation process is to reposition the current image relative to the first marker image
  • the first marker image can be considered to have no cumulative error, so the embodiment can also eliminate the tracking process of multiple marker images. Cumulative error.
  • the relocation method shown in FIG. 5 is to reposition the current image as the first marker image, and the method may have the possibility of relocation failure in some scenarios.
  • the difference between the shooting posture corresponding to the current image and the shooting posture of the first marking image is large, directly matching the current image with the first marking image to achieve relocation fails.
  • (1) of FIG. 6 is the first marker image
  • (2) of FIG. 6 is the current image, since the overlap region between the current image and the first marker image is too small to reach Sufficient matching, so relocating the current image and the first marker image directly will fail.
  • the result (relocation result) respectively calculates the rotation matrix and the translation vector, and connects the two matching results by switching the marker image (using the first-order key frame image as a marker image), thereby obtaining the current image relative to the first
  • the rotation matrix and translation vector of the marker image are used, so that the relocation is successful.
  • FIG. 7 a flowchart of a relocation method of a camera pose tracking process provided by another exemplary embodiment of the present application is shown. This embodiment is exemplified by applying the relocation method to the device shown in FIG. 4.
  • the device is used to perform camera pose tracking of a plurality of marker images in sequence.
  • the method includes:
  • Step 701 Acquire a current image acquired after the i-th mark image in the plurality of mark images, i>1;
  • the camera in the device collects a frame image at a preset time interval to form an image sequence.
  • the camera acquires a frame of image forming image sequence according to a preset time interval during motion (translation and/or rotation).
  • the device determines the first frame image in the image sequence (or one frame image in the first few frames of images that meets the predetermined condition) as the first marker image, and performs the subsequently acquired image on the first marker image.
  • Feature point tracking, and calculating a camera pose parameter according to the feature point tracking result if the feature point tracking effect of the current frame image is worse than a preset condition, determining the previous frame image of the current frame image as the second marker image, The subsequently acquired image is subjected to feature point tracking with respect to the second marker image, and the camera pose parameter of the camera is calculated according to the feature point tracking result, and so on.
  • the device can sequentially perform camera attitude tracking of a plurality of marked images in sequence.
  • the camera When in the i-th tracking process corresponding to the i-th mark image, the camera captures the current image.
  • the current image is a certain frame image acquired after the i-th mark image, where i is an integer greater than one.
  • Step 702 When the current image meets the relocation condition, select a target key frame image from the key frame image library according to the hash index information; the key frame image library stores hash index information of at least one key frame image, and the key frame The image is an image that is cached during the camera pose tracking process and has a first relocation success relative to other images;
  • the device will select some representative images in the relocated image as key frame images to save to the key frame image library.
  • the key frame image library is a database for storing related information of key frame images.
  • the related information includes: hash index information of the key frame image and a first relocation result corresponding to the key frame image.
  • the related information further includes: an image of the key frame image itself.
  • the device determines if the current image meets the relocation criteria.
  • the relocation condition is used to indicate that the tracking process of the current image with respect to the i-th flag image fails, or the re-location condition is used to indicate that the accumulated error in the history tracking process has been higher than the preset condition.
  • the device tracks the current image relative to the i-th mark image, if there is no feature point matching the i-th mark image in the current image, or the i-th mark in the current image When the feature points of the image matching are less than the first number, it is determined that the tracking process of the current image with respect to the i-th flag image fails, and the relocation condition is met.
  • the device determines that the number of frames between the current image and the last relocated image is greater than the second number, determining that the accumulated error in the history tracking process is higher than a preset condition, or When it is determined that the number of mark images between the i-th mark image and the first mark image is greater than the third number, it is determined that the cumulative error in the history tracking process has been higher than the preset condition.
  • This embodiment does not limit the specific condition content of the relocation condition.
  • the device picks out the target key frame image from the key frame image library.
  • the target key frame image is the image (or the closest image or the most similar image) that is closest to the image of the current image in the key frame image library.
  • the key frame image library stores hash index information of the key frame image
  • the device selects the target key frame image from the key frame image library according to the hash index information.
  • Step 703 Perform a second relocation of the current image with respect to the target key frame image.
  • the process of relocating includes the following steps:
  • the key frame image library stores key frame feature points and first camera pose tracking results on each key frame image.
  • the key frame feature points are characterized by SIFT feature point descriptors or ORB feature point descriptors.
  • the first repositioning result is a camera pose parameter (referred to as a key frame attitude parameter) when the camera captures a key frame image
  • the first camera pose tracking result is a camera pose parameter when the camera captures a key frame image relative to the camera acquisition.
  • the amount of change in pose (rotation matrix and displacement vector) of the camera pose parameter when marking an image is referred to as a key frame attitude parameter.
  • each set of matching feature point pairs includes two mutually matching key frame feature points and target feature points.
  • the feature point tracking can use a visual odometer-based tracking algorithm, which is not limited in this application.
  • feature point tracking uses a KLT (Kanade-Lucas) optical flow tracking algorithm; in another embodiment, feature point tracking is based on an ORB (Oriented FAST and Rotated BRIEF) algorithm.
  • KLT Kerade-Lucas
  • ORB Oriented FAST and Rotated BRIEF
  • the ORB feature point descriptor performs feature point tracking.
  • the specific algorithm for feature point tracking is not limited in this application, and the feature point tracking process may adopt a feature point method or a direct method.
  • the feature points may also be characterized by other binary representations, which is not limited in this application.
  • the device extracts feature points from the target key frame image to obtain N key frame feature points; the device also performs feature point extraction on the current image to obtain M.
  • Each set of matching feature point pairs includes: a key frame feature point and a target feature point.
  • the key frame feature point is a feature point on the target key frame image
  • the target feature point is a candidate feature point with the highest matching degree with the key frame feature point on the current image.
  • the number of key frame feature points is greater than or equal to the number of target feature points.
  • the number of key frame feature points is 480 and the target feature points are 350.
  • the device calculates a homography matrix homography between the two frame images according to the key frame feature points and the target feature points; and decomposes the homography matrix homography to obtain a camera when the camera changes from the key frame attitude parameter to the target posture parameter.
  • the amount of change in pose is R cm and T cm .
  • R cm is the rotation matrix when the camera changes from the key frame attitude parameter to the target attitude parameter
  • T cm is the displacement vector when the camera changes from the key frame attitude parameter to the target attitude parameter
  • Step 704 Calculate a target pose parameter when the camera captures the current image according to the first relocated positioning result and the second relocated positioning result.
  • R mf is the rotation matrix when the camera changes from the initial attitude parameter to the key frame attitude parameter.
  • T mf is the displacement vector when the camera changes from the initial attitude parameter to the key frame attitude parameter. The following formula is used to calculate the target when the camera captures the current image. Attitude parameters:
  • R and T are target pose parameters and S mf is the scale of the target key frame image.
  • the relocation method provided by the embodiment can reposition the current image and the key frame image in the key frame image library when the current image meets the relocation condition, and can perform continuous mapping on multiple labeled images.
  • Retargeting is implemented in the tracked Anchor-Switching AR System algorithm, which reduces the possibility of interruption of the tracking process. Since the key frame image in the key frame image library can be multiple, if the key frame image library includes a cover camera acquisition scene. Different key frame images in the surrounding area have a great probability of being relocated successfully, thereby increasing the probability of success in the relocation process.
  • the keyframe image in the keyframe database is gradually added and deleted during the entire camera pose tracking process.
  • the key frame images are all images that have been successfully relocated.
  • the key frame image includes at least one of the following two images:
  • n is an integer greater than one.
  • Hash index information
  • Hash index information (English: hash bundle) is also known as hash bundle, hash index array, hash store and other names. Since there are many feature point descriptors on each key frame image, such as 100-500 feature point descriptors, each feature point descriptor can be a SIFT feature point descriptor or an ORB feature point descriptor. Taking the feature point descriptor as an example of an ORB feature point descriptor, the ORB feature point descriptor is a 256-bit binary descriptor with rotation invariance.
  • the hash index information includes an m column *n row entry storage entry, and each first feature point descriptor on the plurality of key frame images is stored in the m column *n row entry storage bit according to the hash classification rule. The first entry is in the storage bit.
  • the hash index information includes 4*8 entry storage bits, and each column entry storage bit may be a hash table, and there are four hash tables in total: HSI 1, Hash Table 2, Hash Table 3, and Hash Table 4. There are 8 entry storage bits in hash table 1, 8 entry storage bits in hash table 2, 8 entry storage bits in hash table 3, and 8 entry storage bits in hash table 4. Each entry storage bit has its own entry identifier. For example, the entry identifiers of the eight entry storage bits in the hash table 1 are: 000, 001, 010, 011, 100, 101, 110, 111.
  • Each entry storage bit can be thought of as a "drawer” and the entry identifier can be thought of as a "drawer number.”
  • the index information of the first feature point descriptor is allocated by the hash classification rule to one of the eight entry storage bits, and the entry identifier of the entry storage bit is The hash value of the i-th segment of a feature point descriptor matches.
  • the index information of the feature point descriptor includes: the feature point descriptor itself, and the identifier of the key frame image to which the feature point descriptor belongs.
  • the index information of the feature point descriptor further includes: an order of the key frame image, that is, a first-order key frame image, a second-order key frame image, a third-order key frame image, and the like.
  • n f i n is: 001101011000
  • the feature point descriptor f i n described is equally divided into four sub-segments: 001,101,011,000 .
  • the index information of the feature point descriptor f i n is stored in the storage bit of the second row entry of the first column, and the index information includes: (n, i, 001101011000); If the hash value of the sub-segment 2 is 101, the index information of the feature point descriptor f i n is stored in the storage bit of the sixth column entry; since the hash value of the description sub-segment 3 is 011, then The index information of the feature point descriptor f i n is stored in the storage row of the 3rd row of the 3rd row; since the hash value of the description subsegment 4 is 000, the feature point descriptor is stored in the storage bit of the 1st row of the 4th column Index information of f i n .
  • the device needs to add a keyframe image to the keyframe image library during the historical relocation process.
  • the method further includes the following steps:
  • Step 901 Acquire a candidate image that is successfully relocated recently.
  • the key frame image library can be empty, and the device uses only the first marker image as the marker image at the time of relocation. As the entire camera tracking process is performed, more and more images are successfully relocated with the first marker image (or the keyframe image that has been added to the keyframe image library).
  • the device relocates the successfully image as a candidate image that may be added to the key frame image library each time the relocation succeeds.
  • the relocation succeeds in that the relocation is successful with respect to the first marker image, or the relocation is successful with respect to the existing keyframe image in the keyframe image library.
  • the device picks up a part of the image from the candidate image as a key frame image and adds it to the key frame image library.
  • the candidate image may also be selected in other manners. For example, as long as the first marker image can be successfully matched, even if the image is not successfully relocated, the candidate image may be used as a candidate image for expanding the key frame image library.
  • Step 902 determining whether the candidate image satisfies the adding condition, the adding condition includes: the first distance between the candidate image and the first marked image is greater than a first threshold, and/or between the candidate image and the last added key frame image The second distance is greater than the second threshold;
  • the key frame image added this time needs to be at a certain distance from the first marker image, because if the two images are relatively close, there is no significant difference from the effect of directly relocating with the first marker image.
  • the device calculates a first distance between the candidate image and the first marker image, the first distance being used to represent image similarity or camera pose similarity between the candidate image and the first marker image.
  • the specific calculation manner of the first distance is not limited in this application.
  • for the initial feature points in the first mark image there are target feature points in the candidate image that match the initial feature points, one The initial feature point and the corresponding target feature point constitute a matching feature point pair, and the L2 distance (the Euclidean distance corresponding to the L2 norm) is calculated according to each set of matching feature points, and the average value of all L2 distances is taken as the candidate image.
  • the first distance between the image and the first marker image Taking the first threshold as 50 pixels as an example, when the first distance between the candidate image and the first marker image is greater than 50 pixels, the candidate image is added to the key frame image library.
  • the key frame image added this time needs to have a certain distance from the last added key frame image, because if the two images are relatively close, there is no significant difference between the two images being relocated using the last added key frame image. .
  • the device calculates a second distance between the candidate image and the last added key frame image, and the second distance is used to represent image similarity or camera pose similarity between the candidate image and the last added key frame image.
  • the specific calculation manner of the second distance is not limited in this application.
  • the number of relocated images between the candidate image and the last added key frame image is used as the second distance, and the second For example, if the threshold value is 10, if the number of relocated images between the candidate image and the last added key frame image exceeds 10 frames, the candidate image is added to the key frame image library.
  • Step 903 when the candidate image satisfies the adding condition, adding the candidate image to the key frame image library;
  • each of the first feature point descriptors of the key frame image and the first relocated positioning result are stored to the key frame image library.
  • Each of the first feature point descriptors may be stored in the hash index information, and the first relocated positioning result may be represented by a key frame attitude parameter when the camera captures the key frame image, or may be changed by the camera from the initial posture parameter. The amount of change in attitude to the key frame attitude parameter is expressed.
  • the positioning result of the first relocation is represented by a rotation matrix and a displacement vector.
  • the step includes the following sub-steps, as shown in FIG. 10:
  • Step 903a for each first feature point descriptor on the key frame image, dividing the first feature point descriptor into m descriptor segments;
  • the key frame image can be multiple sheets, such as 1-1000 sheets.
  • the device For each first feature point descriptor on each key frame image, the device divides the first feature point descriptor into m descriptor segments. Optionally, the device divides the first feature point descriptor into m descriptor sub-segments according to an equal division principle. Where m is smaller than the total number of bits of the first feature point descriptor, and m is a power of 2. Table 1 shows schematically the process.
  • Step 903b Obtain a first hash value of the i-th descriptor sub-segment for the i-th descriptor sub-segment of the m descriptor sub-segments;
  • the i-th description sub-segment is directly used as the first hash value, or after the calculation of the preset hash function, the first hash value is obtained.
  • the first hash value is obtained.
  • step 903c in the m column *n row entry storage bit, the entry storage bit located in the jth row of the i th column is determined as the first entry storage bit.
  • the device selects the j-th row entry storage bit corresponding to the first hash value as the first entry storage bit in the i-th column entry storage bit.
  • the entry storage bit of the jth row of the i th column is an entry storage bit that matches the first hash value among the n entry storage bits of the i th column.
  • Table 2 schematically shows the process of storing three first feature point descriptors in the hash index information. Among them (point B1, image B) represents the index information of the first feature point description sub-B1 on the key frame image B.
  • Step 904 When the candidate image does not satisfy the adding condition, the candidate image is not added to the key frame image library.
  • the candidate image is not added Keyframe image library.
  • the relocation method provided in this embodiment can select a candidate key image by adding a condition, so that the key frame image in the key frame image library can cover the real scene as much as possible. Different areas to ensure the success rate of the current image relative to the key frame image for the second relocation.
  • the relocation method provided in this embodiment can also reduce the first feature point descriptors that need to be matched in the key frame image query process by storing the first feature point descriptors of the key frame image in the hash index information.
  • the number of the first feature point descriptors does not need to be matched by all the first feature point descriptors, and only needs to be matched by a plurality of first feature point descriptors in the same item storage bit according to the hash classification rule. To the most similar first feature point descriptor.
  • step 702 includes the following sub-steps, as shown in Figure 11:
  • Step 702a acquiring each second feature point descriptor on the current image
  • the device extracts each second feature point descriptor on the current image, and the second feature point descriptor is extracted in the same manner as the first feature point descriptor.
  • the "first" in the first feature point descriptor in the application is used to indicate that the feature point descriptor belongs to the key frame image
  • the "second" in the second feature point descriptor is used to indicate that the feature point descriptor belongs to the current image.
  • the device may use an ORB feature point descriptor extracted based on an ORB (Oriented FAST and Rotated BRIEF) algorithm as a second feature point descriptor.
  • ORB Oriented FAST and Rotated BRIEF
  • other binary representation feature point descriptors may be used.
  • Step 702b for each second feature point descriptor, determining, in the m column *n row entry storage bit, the second entry storage bit corresponding to the second feature point descriptor according to the hash classification rule;
  • This step can include the following substeps:
  • the entry storage bit of the jth row of the i th column is an entry storage bit that matches the second hash value among the n entry storage bits of the i th column.
  • the second entry storage bit and the first entry storage bit are only described in different manners, representing the result of two hash classification processes, but may be substantially in the m column *n row entry storage bits. The same entry storage bit.
  • Step 702c In the first feature point descriptor stored in the second entry storage bit, determine a target first feature point descriptor with the highest similarity with the second feature point descriptor, and record the target first feature point descriptor. Similar scores and associated keyframe images;
  • the second entry storage bit has stored a plurality of first feature point descriptors in the plurality of key frame images.
  • These first feature point descriptors are feature point descriptors that are classified into the same class according to hash classification rules.
  • the second feature point descriptor For the second feature point descriptor on the current image, if the ith descriptor sub-segment of the second feature point descriptor is classified into the second entry storage bit, the second feature point descriptor and the The degree of similarity of each of the first feature points in the two entry storage bits.
  • the degree of similarity is characterized by the ratio of the same number of bits to the total number of bits.
  • the second feature point descriptor on the current image is: 010110100001, it is divided into four descriptor segments: 010 110 100 001.
  • the third column entry storage bit of the first column is the second entry storage bit, and the second feature point descriptor and the first one stored in the third column entry storage bit of the first column are calculated.
  • the similarity score of the feature point descriptor, and the first feature point descriptor of the highest similarity score is determined as the target first feature point descriptor.
  • the similarity score of the target first feature point descriptor and the associated key frame image are recorded. Since it is divided into four descriptor sub-segments, each second feature point descriptor will have four target first feature point descriptors, and there may be duplicates between the four target first feature point descriptors.
  • the above process is performed, and at most k*m target first feature point descriptors, similar scores corresponding to each target first feature point descriptor, and each The target first feature point describes the key frame image to which the child belongs.
  • Step 702d accumulating at least one similarity score belonging to the same key frame image, and determining a key frame image having the highest similarity score summation as the target key frame image.
  • the first feature point description of each target belonging to the same key frame image will be described.
  • the similarity scores of the sub-estimates are summed to obtain the similarity sum sums of the respective key frame images.
  • the key frame image in which the highest similarity score is summed is determined as the target key frame image corresponding to the current image.
  • the method provided in this embodiment can quickly query the target key frame image matching the current image from the key frame image library by using the hash index information, because each first feature point of the key frame image is The descriptors are stored in the hash index information in a distributed manner, and the number of first feature point descriptors that need to be matched when the target key frame image matching the current image is searched can be reduced, and all the first feature point descriptors are not required to be rounded. The match is matched once, which speeds up the entire matching process.
  • the method provided by the embodiment of the present application can be applied to the AR application of the mobile terminal, so that the limited storage space on the mobile terminal is reduced, because the data of the hash index information is small and does not occupy too much storage space. Occupy.
  • the key frame image in the key frame image library should not be too much, and too many key frame images will cause a computational burden on the relocation process, making it difficult to search for an ideal image.
  • the keyframe image in the keyframe image library does not exceed 1000 sheets.
  • the following steps may also be included, as shown in FIG. 12:
  • Step 905 When the number of key frame images in the key frame image library reaches a maximum value, the second key frame image is randomly selected from the key frame image library;
  • Step 906 deleting the second key frame image from the key frame image library.
  • the step includes the following substeps:
  • the identifiers of the first feature point descriptor and the second key frame image are deleted from the first entry storage bit.
  • the index information of the first feature point descriptor is deleted from the first entry storage bit.
  • the method provided in this embodiment also deletes a key frame image closest to the candidate image when the key frame image in the key frame image library reaches a maximum value (for example, 1000 sheets), so that the key frame image library is retained as much as possible.
  • a maximum value for example, 1000 sheets
  • the device preferentially performs a third relocation of the current image with respect to the first marker image; when the third relocation fails, the current image is Try to relocate keyframe images in the keyframe image library. That is, zero-order relocation is preferentially performed. If the zero-order relocation is successful, the zero-order relocation result is used. If the zero-order relocation is unsuccessful, the first-order relocation is performed.
  • the device performs two relocation processes in parallel.
  • the current image is second repositioned relative to the key frame image in the keyframe image library; on the other hand, the current image is third repositioned relative to the first marker image. That is, the first-order relocation and the zero-order relocation are independent and can be performed in parallel.
  • a successful relocation result is adopted; if both relocations are successful, the zero-order priority is used first. Position the results.
  • the result of directly using the relocation may cause a jump
  • the device may calculate the target attitude parameter input filter (Kalman filter) Or a complementary filter), using the target attitude parameter as an observation value, estimating a predicted value by using the motion model, and using the predicted value as the target attitude parameter actually used, that is, performing subsequent processing on the premise that the observation value is large probability. Therefore, the AR system obtains a smooth result, and no jump occurs in the displayed user screen, thereby ensuring a better user experience.
  • the above-described fast indexing method of hash index information can also be used to accelerate the feature point matching process between two images. Assuming that there is a first image and a second image, the first image is the ith marker image and the second image is the current image to be matched.
  • the embodiment of the present application further provides the following feature point matching method.
  • FIG. 13 shows a flowchart of a feature point matching method provided by an exemplary embodiment of the present application.
  • the method can be applied to the device shown in FIG. 4, for example, in the process of performing feature point matching on the i-th mark image, and can also be used in other computer devices performing image processing.
  • the method includes:
  • Step 1301 Acquire hash index information of the first image, where the hash index information includes m column *n row entry storage bits, and each first feature point descriptor on the first image is classified and stored in the m column according to a hash classification rule. n rows of storage bits in respective first entry storage bits;
  • the first image be the ith mark image, and store, in advance, each first feature point descriptor on the first image according to a hash classification rule and store the first entry corresponding to each of the m column *n row entry storage bits. In the bit.
  • a plurality of key frame images jointly construct the same hash index information.
  • the first image separately constructs a hash index information. That is, in the hash index information, only the hash index information of the first image is stored, as shown in FIG. In other words, when the first image is a mark image, each mark image corresponds to index information of a respective hash.
  • the storage process of the hash index information of the first image reference may be made to the description of the foregoing step 903 (FIG. 10), and details are not described herein again.
  • Step 1302 Acquire each second feature point descriptor of the second image.
  • the second image be an image to be tracked by feature points, such as the current image.
  • the device can use the SIFT feature point descriptor and the ORB (Oriented FAST and Rotated BRIEF) algorithm based on the SIFT (Scale-Invariant Feature Transform) algorithm to extract the ORB feature point descriptor.
  • SIFT Scale-Invariant Feature Transform
  • Step 1303 for each second feature point descriptor, determining a second entry storage bit corresponding to the second feature point descriptor in the m column *n row entry storage bit according to the hash classification rule;
  • This step can include the following substeps:
  • the entry storage bit of the jth row of the i th column is an entry storage bit that matches the second hash value among the n entry storage bits of the i th column.
  • the second entry storage bit and the first entry storage bit are only described in different manners, representing the result of two hash classification processes, but may be substantially in the m column *n row entry storage bits. The same entry storage bit.
  • Step 1304 Determine, in each first feature point descriptor stored in the second entry storage bit, a first feature point descriptor that matches the second feature point descriptor.
  • the second entry storage bit has stored a plurality of first feature point descriptors in the plurality of key frame images.
  • These first feature point descriptors are feature point descriptors that are classified into the same class according to hash classification rules.
  • the second feature point descriptor For the second feature point descriptor on the current image, if the ith descriptor sub-segment of the second feature point descriptor is classified into the second entry storage bit, the second feature point descriptor and the The degree of similarity of each of the first feature points in the two entry storage bits.
  • the degree of similarity is characterized by the ratio of the same number of bits to the total number of bits.
  • the first feature point descriptor having the highest degree of similarity is determined as the first feature point descriptor matching the second feature point descriptor.
  • the feature point matching method can reduce the number of candidate first feature point descriptors that need to be matched by the second feature point descriptor by using the hash index information, and does not need to be all on the first image.
  • the first feature point descriptors are all matched, and only the first feature point descriptor in a part of the entry storage bits needs to be matched, so that the matching process can be accelerated.
  • the device acquires a current image captured by the camera; S102, if the current image is a candidate image for expanding the key frame image library, matching the current image with the born-image ( That is, relocation), obtaining a first relocation result; S103, if the current image is successfully matched with the born-image, the current image is used as a first-order relocation successful image; S104, for the first-order relocation successful image, For a self-building hash-bundle, that is, hash index information, the related details can refer to the embodiment shown in FIG.
  • S105 for the first-order relocation successful image, can be added to the total database (all key frame image sharing) Hash-bundle), according to the situation increase, delete and query, the detailed process can refer to the above key frame image addition, deletion and query process;
  • S106 if the current image is used for relocation of the image, find the most in the total database An approximate first-order key frame image;
  • S107 the current image is simultaneously matched with the born-image and the closest first-order key frame image (ie, relocated); S108, according to The matching result is relocated to obtain a second relocation result.
  • the relocation method of the camera attitude tracking process described above can be used in an AR program, by which the camera pose on the terminal can be tracked according to real-world scene information in real time, and tracked according to the tracking.
  • the result adjusts and modifies the display position of the AR element in the AR application.
  • the AR program running on the mobile phone shown in FIG. 1 or FIG. 2 as an example, when it is necessary to display a still cartoon character standing on a book, no matter how the user moves the mobile phone, it only needs to change according to the camera posture on the mobile phone. Modifying the display position of the cartoon character will keep the standing position of the cartoon character on the book unchanged.
  • FIG. 16 is a block diagram of a relocating apparatus of a camera pose tracking process provided by an exemplary embodiment of the present application.
  • the relocation device may be implemented as a whole or a part of an electronic device (or mobile terminal) by software, hardware or a combination of the two.
  • the relocating device includes: an image obtaining module 1620, a hash searching module 1640, and a relocating module 1660. And parameter calculation module 1680.
  • the image acquisition module 1620 is configured to acquire a current image acquired after the i-th mark image of the plurality of mark images, i>1;
  • the hash search module 1640 is configured to: when the current image meets the relocation condition, select a target key frame image from the key frame image library according to the hash index information; and store at least one key frame in the key frame image library Hash index information corresponding to the image, where the key frame image is an image that is cached in the camera pose tracking process and is successfully re-positioned with respect to other images;
  • a relocation module 1660 configured to perform second relocation of the current image with respect to the target key frame image
  • the parameter calculation module 1680 is configured to calculate, according to the positioning result of the first relocation and the positioning result of the second relocation, a camera pose parameter when the camera captures the current image.
  • the hash index information includes m columns * n row entry storage bits, and each first feature point descriptor on the key frame image is classified and stored in the m according to a hash classification rule.
  • the hash search module 1640 is configured to acquire each second feature point descriptor on the current image; for each of the second feature point descriptors, in the m column according to the hash classification rule. Determining, in the n-line entry storage bit, a second entry storage bit corresponding to the second feature point descriptor; determining, in the first feature point descriptor stored in the second entry storage bit, determining And a target first feature point descriptor having the highest similarity with the second feature point descriptor, and recording a similarity score of the target first feature point descriptor and an associated key frame image; and belonging to the same key frame image At least one similarity score is accumulated, and a key frame image having the highest similarity score sum sum is determined as the target key frame image.
  • the hash search module 1640 is configured to divide the second feature point descriptor into m descriptor sub-segments; for the i-th description in the m descriptor sub-segments a sub-segment, obtaining a second hash value of the i-th descriptor sub-segment; determining, in the m-column *n row entry storage bit, an entry storage bit located in the j-th row of the i-th column as the second entry storage Bit
  • the entry storage bit of the jth row of the i th column is an entry storage bit that matches the second hash value among the n entry storage bits of the i th column.
  • the apparatus further includes: a key frame adding module 1630;
  • the image obtaining module 1620 is further configured to acquire a candidate image that is successfully relocated recently;
  • the key frame adding module 1630 is further configured to determine whether the candidate image satisfies an adding condition, where the adding condition includes: a first distance between the candidate image and the first mark image is greater than a first threshold, And/or, the second distance between the candidate image and the last added key frame image is greater than a second threshold; when the candidate image satisfies the adding condition, the candidate image is taken as the first key frame image Join the keyframe image library.
  • the key frame adding module 1630 is configured to acquire each first feature point descriptor of the first key frame image; for each of the first feature point descriptors, according to the Determining, in the m column *n row entry storage bit, a first entry storage bit corresponding to the first feature point descriptor; the first feature point descriptor and the An identification of the first key frame image is stored in the first entry storage bit.
  • the key frame adding module 1630 is configured to divide the first feature point descriptor into m descriptor segments; for the i-th description in the m descriptor segments a sub-segment, obtaining a first hash value of the i-th descriptor sub-segment; determining, in the m-column *n row entry storage bit, an entry storage bit located in the j-th row of the i-th column as the first entry storage Bit
  • the entry storage bit of the jth row of the i th column is an entry storage bit that matches the first hash value among the n entry storage bits of the i th column.
  • the apparatus further includes:
  • the key frame deleting module 1650 is configured to randomly select a second key frame image from the key frame image library when the number of key frame images in the key frame image library reaches a maximum value; and the second key frame The image is deleted from the keyframe image library.
  • the key frame deletion module 1650 is configured to acquire respective first feature point descriptors of the second key frame image; for each of the first feature point descriptors, according to the Determining, in the m column *n row entry storage bit, a first entry storage bit corresponding to the first feature point descriptor; the first feature point descriptor and the The identification of the second key frame image is deleted from the first entry storage bit.
  • the relocation module 1660 is configured to perform third relocation of the current image relative to the first marker image when the current image meets a relocation condition
  • the hash search module 1640 is configured to perform the step of selecting the target key frame image from the key image library according to the hash index information when the third relocation fails.
  • the relocation device of the camera attitude tracking process provided by the above embodiment is only exemplified by the division of the above functional modules when implementing relocation. In actual applications, the functions may be assigned differently according to needs.
  • the function module is completed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.
  • the relocation device and the relocation method embodiment are provided in the same concept, and the specific implementation process is described in detail in the method embodiment, and details are not described herein again.
  • FIG. 17 shows a block diagram of a feature point matching apparatus provided by an exemplary embodiment of the present application.
  • the feature point matching device may be implemented as a whole or a part of an electronic device (or mobile terminal) by software, hardware or a combination of the two.
  • the feature point matching device includes: a first obtaining module 1720, a second acquiring module 1740, and a The hashing module 1760 and the feature point matching module 1780.
  • the first obtaining module 1720 is configured to obtain hash index information of the first image, where the hash index information includes m columns *n row entry storage bits, and each first feature point descriptor on the first image is followed by The hash classification rule is stored in each of the corresponding first entry storage bits in the m column *n row entry storage bits;
  • a second acquiring module 1740 configured to acquire each second feature point descriptor of the second image
  • a hash classification module 1760 configured to determine, for each of the second feature point descriptors, the second feature point descriptor in the m column *n row entry storage bits according to the hash classification rule Corresponding second entry storage bit;
  • a feature point matching module 1780 configured to determine, in each of the first feature point descriptors stored in the second item storage bit, a first feature point descriptor that matches the second feature point descriptor .
  • the hash classification module 1760 is configured to divide the second feature point descriptor into m descriptor sub-segments; for the i-th description in the m descriptor sub-segments a sub-segment, obtaining a second hash value of the i-th descriptor sub-segment; determining, in the m-column *n row entry storage bit, an entry storage bit located in the j-th row of the i-th column as the second entry storage Bit
  • the entry storage bit of the jth row of the i th column is an entry storage bit that matches the second hash value among the n entry storage bits of the i th column.
  • the feature point matching device provided by the foregoing embodiment matches the feature points between the two frames of images
  • only the division of the above functional modules is illustrated.
  • the functions may be allocated according to needs. Different functional modules are completed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.
  • the feature point matching device and the feature point matching method embodiment provided in the foregoing embodiments are in the same concept, and the specific implementation process is described in detail in the method embodiment, and details are not described herein again.
  • FIG. 18 is a block diagram showing the structure of an electronic device 1800 provided by an exemplary embodiment of the present application.
  • the electronic device 1800 can be: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), and a MP4 (Moving Picture Experts Group Audio Layer IV). Audio level 4) Player, laptop or desktop computer.
  • Terminal 1800 may also be referred to as a user device, a portable electronic device, a laptop electronic device, a desktop electronic device, and the like.
  • the electronic device 1800 includes a processor 1801 and a memory 1802.
  • the processor 1801 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like.
  • the processor 1801 may be configured by at least one of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), and a PLA (Programmable Logic Array). achieve.
  • the processor 1801 may also include a main processor and a coprocessor.
  • the main processor is a processor for processing data in an awake state, which is also called a CPU (Central Processing Unit); the coprocessor is A low-power processor for processing data in standby.
  • the processor 1801 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and rendering of the content that the display needs to display.
  • the processor 1801 may further include an AI (Artificial Intelligence) processor for processing computational operations related to machine learning.
  • AI Artificial Intelligence
  • Memory 1802 can include one or more computer readable storage media, which can be non-transitory. Memory 1802 can also include high speed random access memory, as well as non-volatile memory, such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer readable storage medium in memory 1802 is for storing at least one instruction for execution by processor 1801 to implement the camera pose provided by the method embodiments of the present application. The method of relocating the tracking process.
  • the electronic device 1800 optionally further includes: a peripheral device interface 1803 and at least one peripheral device.
  • the processor 1801, the memory 1802, and the peripheral device interface 1803 may be connected by a bus or a signal line.
  • Each peripheral device can be connected to the peripheral device interface 1803 via a bus, signal line or circuit board.
  • the peripheral device includes at least one of a radio frequency circuit 1804, a touch display screen 1805, a camera 1806, an audio circuit 1807, a positioning component 1808, and a power source 1809.
  • the peripheral device interface 1803 can be used to connect at least one peripheral device associated with an I/O (Input/Output) to the processor 1801 and the memory 1802.
  • the processor 1801, the memory 1802, and the peripheral device interface 1803 are integrated on the same chip or circuit board; in some other embodiments, any one of the processor 1801, the memory 1802, and the peripheral device interface 1803 or The two can be implemented on a separate chip or circuit board, which is not limited in this embodiment.
  • the RF circuit 1804 is configured to receive and transmit an RF (Radio Frequency) signal, also referred to as an electromagnetic signal.
  • the RF circuit 1804 communicates with the communication network and other communication devices via electromagnetic signals.
  • the RF circuit 1804 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal.
  • the RF circuit 1804 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and the like.
  • Radio frequency circuitry 1804 can communicate with other electronic devices via at least one wireless communication protocol.
  • the wireless communication protocols include, but are not limited to, the World Wide Web, a metropolitan area network, an intranet, generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and/or WiFi (Wireless Fidelity) networks.
  • the RF circuit 1804 may also include NFC (Near Field Communication) related circuitry, which is not limited in this application.
  • the display 1805 is used to display a UI (User Interface).
  • the UI can include graphics, text, icons, video, and any combination thereof.
  • the display 1805 also has the ability to acquire touch signals over the surface or surface of the display 1805.
  • the touch signal can be input to the processor 1801 as a control signal for processing.
  • the display 1805 can also be used to provide virtual buttons and/or virtual keyboards, also referred to as soft buttons and/or soft keyboards.
  • the display screen 1805 can be one, and the front panel of the electronic device 1800 is disposed; in other embodiments, the display screen 1805 can be at least two, respectively disposed on different surfaces of the electronic device 1800 or in a folded design.
  • display screen 1805 can be a flexible display screen disposed on a curved surface or folded surface of electronic device 1800. Even the display screen 1805 can be set to a non-rectangular irregular pattern, that is, a profiled screen.
  • the display 1805 can be made of a material such as an LCD (Liquid Crystal Display) or an OLED (Organic Light-Emitting Diode).
  • Camera component 1806 is used to capture images or video.
  • camera assembly 1806 includes a front camera and a rear camera.
  • the front camera is placed on the front panel of the electronic device and the rear camera is placed on the back of the electronic device.
  • the rear camera is at least two, which are respectively a main camera, a depth camera, a wide-angle camera, and a telephoto camera, so as to realize the background blur function of the main camera and the depth camera, and the main camera Combine with a wide-angle camera for panoramic shooting and VR (Virtual Reality) shooting or other integrated shooting functions.
  • camera assembly 1806 can also include a flash.
  • the flash can be a monochrome temperature flash or a two-color temperature flash.
  • the two-color temperature flash is a combination of a warm flash and a cool flash that can be used for light compensation at different color temperatures.
  • the audio circuit 1807 can include a microphone and a speaker.
  • the microphone is used to collect sound waves of the user and the environment, and convert the sound waves into electrical signals for processing into the processor 1801 for processing, or input to the RF circuit 1804 for voice communication.
  • the microphone can also be an array microphone or an omnidirectional acquisition microphone.
  • the speaker is then used to convert electrical signals from the processor 1801 or the RF circuit 1804 into sound waves.
  • the speaker can be a conventional film speaker or a piezoelectric ceramic speaker.
  • audio circuit 1807 can also include a headphone jack.
  • the positioning component 1808 is configured to locate a current geographic location of the electronic device 1800 to implement navigation or LBS (Location Based Service).
  • the positioning component 1808 can be a positioning component based on a US-based GPS (Global Positioning System), a Chinese Beidou system, or a Russian Galileo system.
  • a power supply 1809 is used to power various components in the electronic device 1800.
  • the power source 1809 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery.
  • the rechargeable battery may be a wired rechargeable battery or a wireless rechargeable battery.
  • a wired rechargeable battery is a battery that is charged by a wired line
  • a wireless rechargeable battery is a battery that is charged by a wireless coil.
  • the rechargeable battery can also be used to support fast charging technology.
  • electronic device 1800 also includes one or more sensors 1810.
  • the one or more sensors 1810 include, but are not limited to, an acceleration sensor 1811, a gyro sensor 1812, a pressure sensor 1818, a fingerprint sensor 1814, an optical sensor 1815, and a proximity sensor 1816.
  • the acceleration sensor 1811 can detect the magnitude of the acceleration on the three coordinate axes of the coordinate system established by the electronic device 1800.
  • the acceleration sensor 1811 can be used to detect components of gravity acceleration on three coordinate axes.
  • the processor 1801 can control the touch display screen 1805 to display the user interface in a landscape view or a portrait view according to the gravity acceleration signal collected by the acceleration sensor 1811.
  • the acceleration sensor 1811 can also be used for the acquisition of game or user motion data.
  • the gyro sensor 1812 can detect the body direction and the rotation angle of the electronic device 1800, and the gyro sensor 1812 can cooperate with the acceleration sensor 1811 to collect the 3D action of the user on the electronic device 1800. Based on the data collected by the gyro sensor 1812, the processor 1801 can implement functions such as motion sensing (such as changing the UI according to the user's tilting operation), image stabilization at the time of shooting, game control, and inertial navigation.
  • functions such as motion sensing (such as changing the UI according to the user's tilting operation), image stabilization at the time of shooting, game control, and inertial navigation.
  • the pressure sensor 1818 can be disposed on a side border of the electronic device 1800 and/or a lower layer of the touch display screen 1805.
  • the pressure sensor 1818 is disposed on the side frame of the electronic device 1800, the user's holding signal to the electronic device 1800 can be detected, and the processor 1801 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 1818.
  • the operability control on the UI interface is controlled by the processor 1801 according to the user's pressure operation on the touch display screen 1805.
  • the operability control includes at least one of a button control, a scroll bar control, an icon control, and a menu control.
  • the fingerprint sensor 1814 is configured to collect the fingerprint of the user, and the processor 1801 identifies the identity of the user according to the fingerprint collected by the fingerprint sensor 1814, or the fingerprint sensor 1814 identifies the identity of the user according to the collected fingerprint. Upon identifying that the user's identity is a trusted identity, the processor 1801 authorizes the user to perform related sensitive operations including unlocking the screen, viewing encrypted information, downloading software, paying and changing settings, and the like.
  • Fingerprint sensor 1814 can be provided with the front, back or side of electronic device 1800. When the physical device or vendor logo is provided on the electronic device 1800, the fingerprint sensor 1814 can be integrated with the physical button or the manufacturer logo.
  • Optical sensor 1815 is used to collect ambient light intensity.
  • the processor 1801 can control the display brightness of the touch display 1805 based on the ambient light intensity acquired by the optical sensor 1815. Illustratively, when the ambient light intensity is high, the display brightness of the touch display screen 1805 is raised; when the ambient light intensity is low, the display brightness of the touch display screen 1805 is lowered.
  • the processor 1801 can also dynamically adjust the shooting parameters of the camera assembly 1806 based on the ambient light intensity acquired by the optical sensor 1815.
  • Proximity sensor 1816 also referred to as a distance sensor, is typically disposed on the front panel of electronic device 1800. Proximity sensor 1816 is used to capture the distance between the user and the front of electronic device 1800. In one embodiment, when the proximity sensor 1816 detects that the distance between the user and the front side of the electronic device 1800 is gradually decreasing, the processor 1801 controls the touch display screen 1805 to switch from the bright screen state to the interest screen state; when the proximity sensor 1816 When it is detected that the distance between the user and the front side of the electronic device 1800 is gradually increased, the processor 1801 controls the touch display screen 1805 to switch from the state of the screen to the bright state.
  • FIG. 18 does not constitute a limitation to electronic device 1800, may include more or fewer components than illustrated, or may be combined with certain components, or with different component arrangements.
  • a person skilled in the art may understand that all or part of the steps of implementing the above embodiments may be completed by hardware, or may be instructed by a program to execute related hardware, and the program may be stored in a computer readable storage medium.
  • the storage medium mentioned may be a read only memory, a magnetic disk or an optical disk or the like.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Data Mining & Analysis (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Software Systems (AREA)
  • Databases & Information Systems (AREA)
  • Computer Graphics (AREA)
  • Computer Hardware Design (AREA)
  • Library & Information Science (AREA)
  • Architecture (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Evolutionary Biology (AREA)
  • Geometry (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Artificial Intelligence (AREA)
  • Studio Devices (AREA)
  • Image Analysis (AREA)

Abstract

本申请公开了一种相机姿态追踪过程的重定位方法、装置、设备及存储介质，属于增强现实领域。所述方法包括：获取所述多个标记图像中第i个标记图像后采集的当前图像；当所述当前图像符合重定位条件时，根据哈希索引信息从关键帧图像库中选择出目标关键帧图像；将所述当前图像相对于所述目标关键帧图像进行第二重定位；根据所述第一重定位的定位结果和所述第二重定位的定位结果，计算得到所述相机采集所述当前图像时的相机姿态参数。关键帧图像库中的关键帧图像可以为多个，如果存在覆盖相机采集场景的四周区域的不同关键帧图像，则有极大概率能够重定位成功，从而提高了重定位过程的成功几率。

Description

相机姿态追踪过程的重定位方法、装置、设备及存储介质
本申请要求于2018年04月27日提交的申请号为201810392368.5、发明名称为“相机姿态追踪过程的重定位方法、装置、设备及存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请实施例涉及增强现实领域，特别涉及一种相机姿态追踪过程的重定位方法、装置、设备及存储介质。
背景技术
视觉SLAM(simultaneous Localization and mapping，同时定位与地图构建)是指搭载相机的主体，在没有环境先验信息的情况下，于运动过程中建立环境的模型，同时估计自己的运动的技术。SLAM可以应用在AR(Augmented Reality，增强现实)领域、机器人领域和无人驾驶领域中。
以单目视觉SLAM为例，通常将相机采集的第一帧图像作为标记图像(Anchor)。在相机后续采集到当前图像时，设备对当前图像与标记图像之间共同具有的特征点进行追踪，根据当前图像与标记图像之间的特征点位置变化计算得到相机在现实世界中的位姿变化。但某些场景下会发生当前图像中的特征点丢失(Lost)，无法继续追踪的情况。此时，需要使用SLAM重定位方法对当前图像进行重定位。
发明内容
本申请实施例提供了一种相机姿态追踪过程的重定位方法、装置、设备及存储介质，可以解决AR使用场景中直接使用相关技术中的SLAM重定位方法的效果较差的问题。所述技术方案如下：
根据本申请的一个方面，提供了一种相机姿态追踪过程的重定位方法，应用于具有相机的设备中，所述设备用于按序执行多个标记图像的相机姿态追踪，所述方法包括：
获取所述多个标记图像中第i个标记图像之后采集的当前图像，i为大于1的整数；
当所述当前图像符合重定位条件时，获取所述多个标记图像中的第一个标记图像的初始特征点和初始位姿参数，所述初始位姿参数用于指示所述相机采集所述第一个标记图像时的相机姿态；
将所述当前图像相对于所述第一个标记图像进行特征点追踪，得到与所述初始特征点匹配的目标特征点；
根据所述初始特征点和所述目标特征点，计算所述相机从所述第一相机姿态改变至目标相机姿态时的位姿变化量，所述目标相机姿态是所述相机在采集所述当前图像时的相机姿态；
根据所述初始位姿参数和所述位姿变化量，重定位得到所述目标相机姿态对应的目标位姿参数。
根据本申请的另一方面，提供了一种相机姿态追踪过程的重定位装置，应用于具有相机的设备中，所述装置用于按序执行多个标记图像的相机姿态追踪，所述装置包括：
图像获取模块，用于获取所述多个标记图像中第i个标记图像之后采集的当前图像，i为大于1的整数；
信息获取模块，用于当所述当前图像符合重定位条件时，获取所述多个标记图像中的第一个标记图像的初始特征点和初始位姿参数，所述初始位姿参数用于指示所述相机采集所述第一个标记图像时的相机姿态；
特征点追踪模块，用于将所述当前图像相对于所述第一个标记图像进行特征点追踪，得到与所述初始特征点匹配的目标特征点；
变化量计算模块，用于根据所述初始特征点和所述目标特征点，计算所述相机从所述第一相机姿态改变至目标相机姿态时的位姿变化量，所述目标相机姿态是所述相机在采集所述当前图像时的相机姿态；
重定位模块，用于根据所述初始位姿参数和所述位姿变化量，重定位得到所述目标相机姿态对应的目标位姿参数。
根据本申请实施例的另一方面，提供了一个电子设备，所述电子设备包括存储器和处理器；
所述存储器中存储有至少一条指令，所述至少一条指令由所述处理器加载并执行以实现如上所述的重定位方法。
根据本申请实施例的另一方面，提供了一种计算机可读存储介质，所述存储介质中存储有至少一条指令，所述至少一条指令由处理器加载并执行以实现如上所述的重定位方法。
本申请实施例提供的技术方案带来的有益效果至少包括：
通过在当前图像符合重定位条件时，将当前图像与关键帧图像库中的关键帧图像进行重定位，能够在对连续多个标记图像进行追踪的Anchor-SLAM算法中实现重定位，从而减少了追踪过程中断的可能性，由于关键帧图像库中的关键帧图像可以为多个，如果关键帧图像库包括有覆盖相机采集场景的四周区域的不同关键帧图像，则有极大概率能够重定位成功，从而提高了重定位过程的成功几率。
附图说明
为了更清楚地说明本申请实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1是本申请一个示例性实施例提供的AR应用场景的场景示意图；
图2是本申请一个示例性实施例提供的AR应用场景的场景示意图；
图3是本申请一个示例性实施例提供的Anchor-Switching AR System算法的原理示意图；
图4是本申请一个示例性实施例提供的电子设备的结构框图；
图5是本申请一个示例性实施例提供的相机姿态追踪过程的重定位方法的流程图；
图6是本申请一个示例性实施例提供的第一个标记图像、当前图像和一阶关键帧图像的对比示意图；
图7是本申请一个示例性实施例提供的相机姿态追踪过程的重定位方法的流程图；
图8是本申请一个示例性实施例提供的哈希索引信息的数据结构示意图；
图9是本申请一个示例性实施例提供的关键帧图像添加过程的流程图；
图10是本申请一个示例性实施例提供的哈希分类过程的流程图；
图11是本申请一个示例性实施例提供的关键帧图像查询过程的流程图；
图12是本申请一个示例性实施例提供的关键帧图像删除过程的流程图；
图13是本申请一个示例性实施例提供的特征点追踪方法的流程图；
图14是本申请一个示例性实施例提供的特征点追踪方法的原理示意图；
图15是本申请一个示例性实施例提供的相机姿态追踪过程的重定位方法的流程图；
图16是本申请一个示例性实施例提供的相机姿态追踪过程的重定位装置的框图；
图17是本申请一个示例性实施例提供的特征点追踪装置的框图；
图18是本申请一个示例性实施例提供的电子设备的框图。
具体实施方式
为使本申请的目的、技术方案和优点更加清楚，下面将结合附图对本申请实施方式作进一步地详细描述。
首先对本申请涉及的若干个名词进行简介:
AR(Augmented Reality，增强现实)：一种在相机采集图像的过程中，实时地计算相机在现实世界(或称三维世界、真实世界)中的相机姿态参数，根据该相机姿态参数在相机采集的图像上添加虚拟元素的技术。虚拟元素包括但不限于：图像、视频和三维模型。AR技术的目标是在屏幕上把虚拟世界套接在现实世界上进行互动。该相机姿态参数包括位移向量和旋转矩阵，位移向量用于表征相机在现实世界中发生的位移距离，旋转矩阵用于表征相机在现实世界中发生的旋转角度。
例如，参见图1和参见图2，设备在相机拍摄到的图像中添加了一个虚拟人物形象。随着相机在现实世界中的运动，相机拍摄到的图像会发生变化，虚拟人物的拍摄方位也发生变化，模拟出了虚拟人物在图像中静止不动，而相机随着位置和姿态的变化同时拍摄图像和虚拟人物的效果，为用户呈现了一幅真实立体的画面。
Anchor-Switching AR System：是基于多个标记图像(Anchor)的相机姿态追踪来确定在自然场景下的相机姿态参数，进而根据相机姿态参数在相机采集的图像上叠加虚拟世界的AR系统。
IMU(Inertial Measurement Unit，惯性测量单元)：是用于测量物体的三轴姿态角(或角速率)以及加速度的装置。一般的，一个IMU包含了三个单轴的加速度计和三个单轴的陀螺，加速度计用于检测物体在三维坐标系中每个坐标轴上的加速度信号，进而计算得到位移向量；而陀螺用于检测物体在三维坐标系中的旋转矩阵。可选地，IMU包括陀螺仪、加速度计和地磁传感器。
示意性的，三维坐标系的建立方式为：1、X轴使用向量积Y*Z来定义，在X轴在设备当前的位置上，沿与地面相切的方向指向东方；2、Y轴在设备当前的位置上，沿与地面相切的方向指向地磁场的北极；3、Z轴指向天空并垂直于地面。
在AR(Augmented Reality，增强现实)领域进行相机姿态追踪时，比如使用手机拍摄桌面进行AR游戏的场景，由于AR使用场景存在其场景特殊性，通常会对现实世界中的某个固定平面进行持续性拍摄(比如某个桌面或墙面)，直接使用相关技术中的SLAM重定位方法的效果较差，尚需提供一种适用于AR领域的重定位解决方案。
本申请提供了一种适用于Anchor-Switching AR System算法的重定位方法。Anchor- Switching AR System算法在确定相机姿态的过程中，将相机的运动过程划分为至少两段追踪过程进行追踪，每段追踪过程对应各自的标记图像。示意性的，当第i个标记图像对应的追踪过程中，当当前图像相对于第i个标记图像的追踪效果差于预设条件(比如能够匹配到的特征点少于预设阈值)时，将当前图像的上一个图像确定为第i+1个标记图像，开启第i+1段追踪过程。其中，i为正整数。示意性的参考图3，其示出了本申请一个示例性实施例提供的Anchor-Switching AR System算法的原理示意图。在现实世界中存在物体320，设置有相机的设备340被用户手持进行移动，在移动过程中拍摄得到包括物体320的多帧图像1-6。设备将图像1确定为第1个标记图像(born-anchor或born-image)并记录初始位姿参数，该初始位姿参数可以是IMU采集的，然后将图像2相对于图像1进行特征点追踪，根据初始位姿参数和特征点追踪结果计算出相机在拍摄图像2时的位姿参数；将图像3相对于图像1进行特征点追踪，根据初始位姿参数和特征点追踪结果计算出相机在拍摄图像3时的位姿参数；将图像4相对于图像1进行特征点追踪，根据初始位姿参数和特征点追踪结果计算出相机在拍摄图像4时的位姿参数。
然后，将图像5相对于图像1进行特征点追踪，如果特征点追踪效果差于预设条件(比如匹配的特征点数量较少)，则将图像4确定为第2个标记图像，将图像5相对于图像4进行特征点追踪，计算出相机在拍摄图像4至图像5之间的位移变化量，再结合相机在拍摄图像4至图像1之间的位移变化量以及初始位姿参数，计算出相机在拍摄图像5时的位姿参数。然后再将图像6相对于图像4进行特征点追踪，依次类推，若当前图像的特征点追踪效果变差时，即可将当前图像的上一帧图像确定为新的标记图像，切换新的标记图像后重新进行特征点追踪。
可选地，特征点追踪可以采用基于视觉里程计原理的算法，比如特征点法或直接法。但是若相机在追踪过程中处于发生较为剧烈的运动、朝向强光源、朝向白色墙壁等各种异常场景时，上述Anchor-Switching AR System追踪过程可能会发生丢失(Lost)现象。丢失现象是指在当前图像中无法匹配到足够多的特征点，导致追踪失败。
参考图4，其示出了本申请一个示例性实施例提供的电子设备的结构框图。该设备包括：处理器420、存储器440、相机460和IMU 480。
处理器420包括一个或多个处理核心，比如4核心处理器、8核心处理器等。处理器420用于执行存储器440中存储的指令、代码、代码片段和程序中的至少一种。
处理器420与存储器440电性相连。可选地，处理器420通过总线与存储器440相连。存储器440存储有一个或多个指令、代码、代码片段和/或程序。该指令、代码、代码片段和/或程序在被处理器420执行时，用于实现如下实施例中提供的SLAM重定位方法。
处理器420还与相机460电性相连。可选地，处理器420通过总线与相机460相连。相机460是具有图像采集能力的传感器件。相机460还可称为摄像头、感光器件等其它名称。相机460具有连续采集图像或多次采集图像的能力。可选地，相机460设置在设备内部或设备外部。
处理器420还与IMU480电性相连。可选地，IMU480用于每隔预定时间间隔采集相机的位姿参数，并记录每组位姿参数在采集时的时间戳。相机的位姿参数包括：位移向量和旋转矩阵。其中，IMU480采集的旋转矩阵相对准确，采集的位移向量受实际环境可能会有较大的误差。
参考图5，其示出了本申请一个示例性实施例中提供的相机姿态跟踪过程的重定位方法的流程图。本实施例以该重定位方法应用于图4所示的设备中来举例说明，该设备用于按序执行多个标记图像的相机姿态追踪。该方法包括:
步骤502，获取多个标记图像中第i个标记图像之后采集的当前图像；
设备内的相机按照预设时间间隔采集一帧帧图像，形成图像序列。可选地，相机是在运动(平移和/或旋转)过程中，按照预设时间间隔采集一帧帧图像形成图像序列。
可选地，设备将图像序列中的第一帧图像(或前几帧图像中符合预定条件的一帧图像)确定为第一个标记图像，将后续采集的图像相对于第一个标记图像进行特征点追踪，并根据特征点追踪结果计算相机的相机姿态参数；若当前帧图像的特征点追踪效果差于预设条件时，将当前帧图像的上一帧图像确定为第二个标记图像，将后续采集的图像相对于第二个标记图像进行特征点追踪，并根据特征点追踪结果计算相机的相机姿态参数，依次类推。设备可以按序进行连续多个标记图像的相机姿态追踪。
当处于第i个标记图像对应的第i个追踪过程时，相机会采集到当前图像。当前图像是第i个标记图像之后采集的某一帧图像，其中，i为大于1的整数。
步骤504，当当前图像符合重定位条件时，获取多个标记图像中的第一个标记图像的初始特征点和初始位姿参数，初始位姿参数用于指示相机采集第一个标记图像时的相机姿态；
设备会确定当前图像是否符合重定位条件。重定位条件用于指示当前图像相对于第i个标记图像的追踪过程失败，或者，重定位条件用于指示历史追踪过程中的累积误差已经高于预设条件。
在一个可选的实施例中，设备对当前图像相对于第i个标记图像进行追踪，若当前图像中不存在与第i个标记图像匹配的特征点，或者，当前图像中与第i个标记图像匹配的特征点少于第一数量时，确定当前图像相对于第i个标记图像的追踪过程失败，符合重定位条件。
在另一个可选的实施例中，设备确定当前图像与上一次重定位的图像之间的帧数大于第二数量时，确定历史追踪过程中的累积误差已经高于预设条件，或者，设备确定第i个标记图像和第一个标记图像之间的标记图像数量大于第三数量时，确定历史追踪过程中的累计误差已经高于预设条件。
本实施例对重定位条件的具体条件内容不加以限定。
当当前图像符合重定位条件时，设备尝试将当前图像相对于第一个标记图像进行特征点追踪。此时，设备获取缓存的第一个标记图像中的初始特征点以及初始位姿参数，该初始位姿参数用于指示相机采集第一个标记图像时的相机姿态。
步骤506，将当前图像相对于第一个标记图像进行特征点追踪，得到与初始特征点匹配的目标特征点；
可选地，每组匹配特征点对中包括两个互相匹配的初始特征点和目标特征点。
特征点追踪可采用基于视觉里程计的追踪算法，本申请对此不加以限定。在一个实施例中，特征点追踪采用KLT(Kanade-Lucas)光流追踪算法；在另一个实施例中，特征点追踪采用基于SIFT(Scale-Invariant Feature Transform，尺度不变特征转换)算法提取的SIFT特征点描述子、ORB(Oriented FAST and Rotated BRIEF，快速特征点提取和描述)算法提取的ORB特征点描述子进行特征点跟踪。本申请对特征点追踪的具体算法不加以限定，特征点追踪过程可以采用特征点法或直接法。
在一个实施例中，设备对第一个标记图像进行特征点提取，得到N个初始特征点；设备还对当前图像进行特征点提取，得到M个候选特征点；然后将M个候选特征点逐一与N个初始特征点进行匹配，确定出至少一组匹配特征点对。每组匹配特征点对包括：一个初始特征点和一个目标特征点。初始特征点是第1个标记图像上的特征点，目标特征点是当前图像上与该初始特征点匹配度最高的候选特征点。
可选地，初始特征点的数量大于或等于目标特征点的数量。比如，初始特征点的数量是450个，目标特征点为320个。
步骤508，根据初始特征点和目标特征点，计算相机从初始姿态参数改变至目标姿态参数时的位姿变化量，目标姿态参数用于表征相机在采集当前图像时的相机姿态；
可选地，设备根据初始特征点和目标特征点计算两帧图像之间的单应性矩阵homography；对单应性矩阵homography进行分解，得到相机从初始姿态参数改变至目标姿态参数时的位姿变化量R relocalize和T relocalize。
单应性矩阵描述了两个平面之间的映射关系，若自然场景(现实环境)中的特征点都落在同一物理平面上，则可以通过单应性矩阵进行运动估计。当存在至少四对相匹配的初始特征点和目标特征点时，设备通过ransac对该单应性矩阵进行分解，得到旋转矩阵R relocalize和平移向量T relocalize。
其中，R relocalize是相机从初始姿态参数改变至目标姿态参数时的旋转矩阵，T relocalize是相机从初始姿态参数改变至目标姿态参数时的位移向量。
步骤510，根据初始位姿参数和位姿变化量，重定位得到目标位姿参数。
设备将初始位姿参数利用位姿变化量进行变换后，重定位得到目标位姿参数，从而计算得到相机在采集当前图像时的相机姿态。
综上所述，本实施例提供的重定位方法，通过在当前图像符合重定位条件时，将当前图像与第一个标记图像进行重定位，能够在连续多个标记图像进行追踪的Anchor-Switching AR System算法中实现重定位，从而减少了追踪过程中断的可能性，从而解决相关技术中的SLAM重定位方法并不适用于变种后的SLAM算法的问题。
另外，由于重定位过程是将当前图像相对于第一个标记图像进行重定位，第一个标记图像可以认为是没有累积误差的，所以本实施例还能消除多个标记图像的追踪过程所产生的累积误差。
图5所示出的重定位方法是将当前图像相当于第一个标记图像进行重定位，该方法在某些场景下会存在重定位失败的可能性。比如当前图像对应的拍摄姿态与第一个标记图像的拍摄姿态之间的差别较大时，直接将当前图像和第一个标记图像建立匹配来实现重定位会失败。示意性的，设图6的(1)图是第一个标记图像，图6的(2)图是当前图像，由于当前图像和第一个标记图像之间的重合区域太少，达不到足够的匹配度，因此直接将当前图像和第一个标记图像进行重定位会失败。本申请实施例中，引入图6的(3)图作为一阶关键帧图像，将当前图像相对于一阶关键帧图像的匹配结果，以及一阶关键帧图像相对于第一个标记图像的匹配结果(重定位结果)分别计算出旋转矩阵和平移向量，通过切换标记图像(将一阶关键帧图像作为一个标记图像)的方式将两个匹配结果联系在一起，进而得到当前图像相对于第一个标记图像的旋转矩阵和平移向量，从而重定位成功。
参考图7，其示出了本申请另一个示例性实施例提供的相机姿态跟踪过程的重定位方法的流程图。本实施例以该重定位方法应用于图4所示的设备中来举例说明。该设备用于按序执行多个标记图像的相机姿态追踪。该方法包括：
步骤701，获取多个标记图像中第i个标记图像后采集的当前图像，i＞1；
设备内的相机按照预设时间间隔采集一帧帧图像，形成图像序列。可选地，相机是在运动(平移和/或旋转)过程中，按照预设时间间隔采集一帧帧图像形成图像序列。
可选地，设备将图像序列中的第一帧图像(或前几帧图像中符合预定条件的一帧图像)确定为第一个标记图像，将后续采集的图像相对于第一个标记图像进行特征点追踪，并根据特征点追踪结果计算相机的相机姿态参数；若当前帧图像的特征点追踪效果差于预设条件时，将当前帧图像的上一帧图像确定为第二个标记图像，将后续采集的图像相对于第二个标记图像进行特征点追踪，并根据特征点追踪结果计算相机的相机姿态参数，依次类推。设备可以按序进行连续多个标记图像的相机姿态追踪。
当处于第i个标记图像对应的第i个追踪过程时，相机会采集到当前图像。当前图像是第i个标记图像之后采集的某一帧图像，其中，i为大于1的整数。
步骤702，当当前图像符合重定位条件时，根据哈希索引信息从关键帧图像库中选择出目标关键帧图像；关键帧图像库中存储有至少一个关键帧图像的哈希索引信息，关键帧图像是在相机姿态追踪过程中缓存的相对于其它图像存在第一重定位成功的图像；
在相机姿态追踪过程中会存在不止一次重定位过程。设备会在重定位成功的图像中挑选一些有代表性的图像作为关键帧图像保存至关键帧图像库。关键帧图像库是用于存储关键帧图像的相关信息的数据库。可选地，该相关信息包括：关键帧图像的哈希索引信息和关键帧图像对应的第一重定位结果。可选地，该相关信息还包括：关键帧图像的图像本身。
设备会确定当前图像是否符合重定位条件。重定位条件用于指示当前图像相对于第i个标记图像的追踪过程失败，或者，重定位条件用于指示历史追踪过程中的累积误差已经高于预设条件。
在一个可选的实施例中，设备对当前图像相对于第i个标记图像进行追踪，若当前图像中不存在与第i个标记图像匹配的特征点，或者，当前图像中与第i个标记图像匹配的特征点少于第一数量时，确定当前图像相对于第i个标记图像的追踪过程失败，符合重定位条件。
在另一个可选的实施例中，设备确定当前图像与上一次重定位的图像之间的帧数大于第二数量时，确定历史追踪过程中的累积误差已经高于预设条件，或者，设备确定第i个标记图像和第一个标记图像之间的标记图像数量大于第三数量时，确定历史追踪过程中的累计误差已经高于预设条件。
本实施例对重定位条件的具体条件内容不加以限定。
当当前图像符合重定位条件时，设备从关键帧图像库中挑选出目标关键帧图像。可选地，目标关键帧图像是在关键帧图像库中与当前图像的图像距离最接近的图像(或最接近的图像或最相似的图像)。
在本申请实施例中，关键帧图像库中存储有关键帧图像的哈希索引信息，设备根据哈希索引信息从关键帧图像库中挑选目标关键帧图像。
步骤703，将当前图像相对于目标关键帧图像进行第二重定位；
当设备在关键帧图像库中挑选出目标关键帧图像之后，将当前图像相对于目标关键帧图像进行重定位。可选地，重定位的过程包括如下步骤：
1、获取目标关键帧图像的关键帧特征点和第一重定位的定位结果；
关键帧图像库中存储有每个关键帧图像上的关键帧特征点和第一相机姿态追踪结果。可选地，关键帧特征点采用SIFT特征点描述子或ORB特征点描述子来表征。第一重定位的定位结果是相机采集关键帧图像时的相机姿态参数(简称关键帧姿态参数)，或者，第一相机姿态追踪结果是相机采集关键帧图像时的相机姿态参数相对于相机采集第一个标记图像时的相机姿态参数的姿态变化量(旋转矩阵和位移向量)。
2、将当前图像相对于目标关键图像库进行特征点追踪，得到与关键帧特征点匹配的目标特征点；
可选地，每组匹配特征点对中包括两个互相匹配的关键帧特征点和目标特征点。
特征点追踪可采用基于视觉里程计的追踪算法，本申请对此不加以限定。在一个实施例中，特征点追踪采用KLT(Kanade-Lucas)光流追踪算法；在另一个实施例中，特征点追踪采用基于ORB(Oriented FAST and Rotated BRIEF，快速特征点提取和描述)算法提取的ORB特征点描述子进行特征点跟踪。本申请对特征点追踪的具体算法不加以限定，特征点追踪过程可以采用特征点法或直接法。特征点还可以采用其它二进制表示的特征点描述子，本申请对此不加以限定。
在一个实施例中，由于目标关键帧图像是已经重定位成功的图像，设备对目标关键帧图像进行特征点提取，得到N个关键帧特征点；设备还对当前图像进行特征点提取，得到M个候选特征点；然后将M个候选特征点逐一与N个初始特征点进行匹配，确定出至少一组匹配特征点对。每组匹配特征点对包括：一个关键帧特征点和一个目标特征点。关键帧特征点是目标关键帧图像上的特征点，目标特征点是当前图像上与该关键帧特征点匹配度最高的候选特征点。
可选地，关键帧特征点的数量大于或等于目标特征点的数量。比如，关键帧特征点的数量是480个，目标特征点为350个。
3、根据关键帧特征点和目标特征点，计算相机从关键帧姿态参数改变至目标姿态参数时的位姿变化量，目标姿态参数用于表征相机在采集当前图像时的相机姿态。
可选地，设备根据关键帧特征点和目标特征点计算两帧图像之间的单应性矩阵homography；对单应性矩阵homography进行分解，得到相机从关键帧姿态参数改变至目标姿态参数时的位姿变化量R cm和T cm。
其中，R cm是相机从关键帧姿态参数改变至目标姿态参数时的旋转矩阵，T cm是相机从关键帧姿态参数改变至目标姿态参数时的位移向量。
步骤704，根据第一重定位的定位结果和第二重定位的定位结果，计算得到相机采集当前图像时的目标姿态参数。
假设第一重定位的定位结果包括R mf和T mf。R mf是相机从初始姿态参数改变至关键帧姿态参数时的旋转矩阵，T mf是相机从初始姿态参数改变至关键帧姿态参数时的位移向量，采用如下公式计算得到相机采集当前图像时的目标姿态参数：
Figure PCTCN2019079355-appb-000001
其中，R和T是目标姿态参数，S mf是目标关键帧图像的尺度。
综上所述，本实施例提供的重定位方法，通过在当前图像符合重定位条件时，将当前图像与关键帧图像库中的关键帧图像进行重定位，能够在对连续多个标记图像进行追踪的 Anchor-Switching AR System算法中实现重定位，从而减少了追踪过程中断的可能性，由于关键帧图像库中的关键帧图像可以为多个，如果关键帧图像库包括有覆盖相机采集场景的四周区域的不同关键帧图像，则有极大概率能够重定位成功，从而提高了重定位过程的成功几率。
关键帧图像库(keyframe Database)中的关键帧图像(keyframe)是在整个相机姿态追踪过程中逐渐增加和删除的。关键帧图像均为重定位成功的图像。可选地，关键帧图像包括如下两种图像中的至少一种：
1、相对于多个标记图像中的第一个标记图像进行第一重定位成功的一阶关键帧图像；
2、相对于关键帧图像库中的n-1阶关键帧图像进行第一重定位成功的n阶关键帧图像，n为大于1的整数。
哈希索引信息:
哈希索引信息(英文：hash bundle)又称哈希束、哈希索引阵列、哈希仓库等其它名称。由于每个关键帧图像上有很多个特征点描述子，比如100-500个特征点描述子，每个特征点描述子可以是SIFT特征点描述子或ORB特征点描述子。以特征点描述子是ORB特征点描述子为例，ORB特征点描述子是一种256比特的二进制的描述子，具有旋转不变性。哈希索引信息包括m列*n行条目存储位(entry)，多个关键帧图像上的各个第一特征点描述子按照哈希分类规则分类存储于m列*n行条目存储位中对应的第一条目存储位中。
参考图8，以m＝4，n＝8为例，哈希索引信息包括4*8个条目存储位，每一列条目存储位可以是一张哈希表，共存在四个哈希表：哈希表1、哈希表2、哈希表3和哈希表4。哈希表1中具有8个条目存储位、哈希表2中具有8个条目存储位、哈希表3中具有8个条目存储位、哈希表4中具有8个条目存储位。每个条目存储位具有各自的条目标识，比如哈希表1中的8个条目存储位的条目标识依次为：000、001、010、011、100、101、110、111。每个条目存储位可以认为是一个“抽屉”，条目标识可以认为是“抽屉号”。在每i个哈希表中，第一特征点描述子的索引信息会被哈希分类规则分配至8个条目存储位中的一个条目存储位中进行存储，该条目存储位的条目标识与第一特征点描述子的第i个分段的哈希值相匹配。特征点描述子的索引信息包括：特征点描述子本身、特征点描述子所属关键帧图像的标识。可选地，该特征点描述子的索引信息中还包括：所属关键帧图像的阶数，也即一阶关键帧图像、二阶关键帧图像、三阶关键帧图像等。
示意性的，假设关键帧图像n上的第i个特征点描述子f i n为：001101011000，该特征点描述子f i n被等分为4个描述子片段：001、101、011、000。由于描述子片段1的哈希值为001，则在第1列第2行条目存储位中存储特征点描述子f i n的索引信息，该索引信息包括：(n，i，001101011000)；由于描述子片段2的哈希值为101，则在第2列第6行条目存储位中存储特征点描述子f i n的索引信息；由于描述子片段3的哈希值为011，则在第3列第4行条目存储位中存储特征点描述子f i n的索引信息；由于描述子片段4的哈希值为000，则在第4列第1行条目存储位中存储特征点描述子f i n的索引信息。
上述以特征点描述子为12比特的信息、m＝4、n＝8来举例说明，但是在不同的实施例中可以采用不同的取值。比如，特征点描述子为256比特的信息、n＝2 (256/m)，当m＝16时，n＝65536。
关键帧图像的添加阶段:
基于图7的可选实施例中，设备需要在历史重定位过程中向关键帧图像库添加关键帧图像。如图9所示，该方法还包括如下步骤：
步骤901，获取最近一个重定位成功的候选图像；
在初始状态下，关键帧图像库可以为空，设备仅使用第一个标记图像作为重定位时的标记图像。随着整个相机追踪过程的执行，会出现越来越多与第一个标记图像(或已加入关键帧图像库的关键帧图像)进行重定位成功的图像，
设备在每次重定位成功时，将重定位成功的图像作为可能添加至关键帧图像库的候选图像。可选地，重定位成功包括：相对于第一个标记图像进行重定位成功，或者，相对于关键帧图像库中已有的关键帧图像进行重定位成功。
设备从候选图像中挑选出一部分图像作为关键帧图像，添加至关键帧图像库中。
需要说明的是，候选图像也可以采用其它方式进行选取，比如只要能够与第一个标记图像匹配成功，即便不是重定位成功的图像，也可以作为候选图像用于对关键帧图像库的扩展。
步骤902，确定候选图像是否满足添加条件，添加条件包括：候选图像和第一个标记图像之间的第一距离大于第一阈值，和/或，候选图像与上一次加入的关键帧图像之间的第二距离大于第二阈值；
可选地，本次添加的关键帧图像需要与第一个标记图像存在一定距离，因为如果两个图像比较接近，那么与直接用第一个标记图像进行重定位的效果没有明显差别。
设备计算候选图像与第一个标记图像之间的第一距离，第一距离用于表示候选图像和第一个标记图像之间的图像相似度或者相机姿态相似度。本申请对第一距离的具体计算方式不加以限定，在一个可选的实施例中，对于第一个标记图像中的初始特征点，候选图像中存在与初始特征点匹配的目标特征点，一个初始特征点和对应的目标特征点构成一个匹配特征点对，根据每一组匹配特征点分别计算L2距离(对应L2范数的欧几里得距离)，将所有L2距离的平均值作为候选图像和第一个标记图像之间的第一距离。以第一阈值为50个像素点为例，当候选图像和第一个标记图像之间的第一距离大于50个像素点时，将候选图像添加至关键帧图像库。
可选地，本次添加的关键帧图像需要与上一次添加的关键帧图像存在一定距离，因为如果两个图像比较接近，那么与使用上一次添加的关键帧图像进行重定位的效果没有明显差别。
设备计算候选图像与上一次添加的关键帧图像之间的第二距离，第二距离用于表示候选图像和上一次添加的关键帧图像之间的图像相似度或相机姿态相似度。本申请对第二距离的具体计算方式不加以限定，在一个可选的实施例中，采用候选图像和上一次添加的关键帧图像之间的重定位图像个数作为第二距离，以第二阈值为10为例，若候选图像和上一次添加的关键帧图像之间的重定位图像个数超过10帧，则将候选图像添加至关键帧图像库中。
步骤903，当候选图像满足添加条件时，将候选图像加入关键帧图像库；
当确定候选图像满足添加条件时，将候选图像作为关键帧图像添加至关键帧图像库。可选地，将关键帧图像的各个第一特征点描述子和第一重定位的定位结果存储至关键帧图像库。其中，各个第一特征点描述子可以存储至哈希索引信息中，第一重定位的定位结果可以使用相机采集关键帧图像时的关键帧姿态参数来表示，也可以采用相机从初始姿态参数改变至关键帧姿态参数时的姿态变化量来表示。本实施例中，以第一重定位的定位结果采用旋转矩阵和位移向量来表示进行举例说明。
可选地，本步骤包括如下子步骤，如图10所示：
步骤903a，对于关键帧图像上的每个第一特征点描述子，将第一特征点描述子划分为m个描述子片段；
关键帧图像可以为多张，比如1-1000张。每个关键帧图像上又存在多个第一特征点，该第一特征点采用第一特征点描述子来表征，该第一特征点描述子为256比特的ORB特征点描述子。
对于每个关键帧图像上的每个第一特征点描述子，设备将第一特征点描述子划分为m个描述子片段。可选地，设备将第一特征点描述子按照等分原则划分为m个描述子片段。其中，m小于第一特征点描述子的总比特数，且m为2的幂。表一示意性的示出了该过程。
表一
索引信息 第一特征点描述子 拆分后的描述子片段
关键帧图像A，点A1 010000110101 010 000 110 101
关键帧图像A，点A2 001110101011 001 110 101 011
关键帧图像B，点B1 001110101010 001 110 101 010
步骤903b，对于m个描述子片段中的第i个描述子片段，获取第i个描述子片段的第一哈希值；
对于第i个描述子片段，将第i个描述子片段直接作为第一哈希值，或者经过预设哈希函数的计算后，得到第一哈希值。其中，0≤i≤m-1或1≤i≤m。
步骤903c，在m列*n行条目存储位中，确定位于第i列第j行的条目存储位作为第一条目存储位。
对于第i个描述子片段，设备在第i列条目存储位中，选择出与第一哈希值对应的第j行条目存储位作为第一条目存储位。
其中，第i列第j行的条目存储位是位于第i列的n个条目存储位中与第一哈希值匹配的条目存储位。
表二示意性的示出了将三个第一特征点描述子存储在哈希索引信息中的过程。其中的(点B1，图像B)代表关键帧图像B上的第一特征点描述子B1的索引信息。
表二
Figure PCTCN2019079355-appb-000002
Figure PCTCN2019079355-appb-000003
步骤904，当候选图像不满足添加条件时，将候选图像不加入关键帧图像库。
当候选图像与第一个标记图像之间的第一距离小于第一阈值，或者，候选图像与上一次添加的关键帧图像之间的第二距离小于第二阈值，则不将该候选图像加入关键帧图像库。
综上所述，本实施例提供的重定位方法，通过添加条件对候选图像进行选择，能够选择出具有代表性的关键帧图像，使得关键帧图像库中的关键帧图像能够尽量覆盖现实场景中的不同区域，从而保证当前图像相对于关键帧图像进行第二重定位时的成功率。
本实施例提供的重定位方法，还通过将关键帧图像的各个第一特征点描述子分散地存储在哈希索引信息中，能够减少关键帧图像查询过程中需要匹配的第一特征点描述子的个数，不需要将所有第一特征点描述子都轮询匹配一遍，只需要按照哈希分类规则在同一个条目存储位中的多个第一特征点描述子中，就可以大概率匹配到最相似的第一特征点描述子。
关键帧图像的查询阶段:
在基于图7的可选实施例中，步骤702包括如下子步骤，如图11所示：
步骤702a，获取当前图像上的各个第二特征点描述子；
设备提取当前图像上的各个第二特征点描述子，第二特征点描述子和第一特征点描述子的提取方式相同。本申请中第一特征点描述子中的“第一”用于表示该特征点描述子属于关键帧图像，第二特征点描述子中的“第二”用于表示该特征点描述子属于当前图像。
设备可以采用基于ORB(Oriented FAST and Rotated BRIEF，快速特征点提取和描述)算法提取的ORB特征点描述子，作为第二特征点描述子。或者，也可以采用其它二进制表示的特征点描述子。
步骤702b，对于每个第二特征点描述子，根据哈希分类规则在m列*n行条目存储位中确定出与第二特征点描述子对应的第二条目存储位中；
本步骤可以包括如下子步骤：
1、将第二特征点描述子划分为m个描述子片段；
2、对于m个描述子片段中的第i个描述子片段，获取第i个描述子片段的第二哈希值；
3、在m列*n行条目存储位中，确定位于第i列第j行的条目存储位作为第二条目存储位；
其中，第i列第j行的条目存储位是位于第i列的n个条目存储位中与第二哈希值匹配的条目存储位。
需要说明的是，第二条目存储位和第一条目存储位仅是采用不同的描述方式，代表两次哈希分类过程的结果，但实质上可以是m列*n行条目存储位中的同一个条目存储位。
步骤702c，在第二条目存储位中存储的第一特征点描述子中，确定出与第二特征点描述子相似度最高的目标第一特征点描述子，记录目标第一特征点描述子的相似分数和所属关键帧图像；
可选地，第二条目存储位已存储有多个关键帧图像中的多个第一特征点描述子。这些第 一特征点描述子是按照哈希分类规则被分类至同一类的特征点描述子。
对于当前图像上的第二特征点描述子，若该第二特征点描述子的第i个描述子片段被分类至该第二条目存储位，则可以计算该第二特征点描述子与第二条目存储位中每个第一特征点描述子的相似程度。可选地，采用相同比特数量与总比特数量的比值来表征该相似程度。
比如，假设当前图像上的第二特征点描述子为：010110100001，则划分为四个描述子片段：010 110 100 001。对于描述子片段1：001，确定第1列第3行条目存储位是第二条目存储位，计算第二特征点描述子和存储在第1列第3行条目存储位中的各个第一特征点描述子的相似分数，将最高相似分数的第一特征点描述子确定为目标第一特征点描述子。记录该目标第一特征点描述子的相似分数和所属关键帧图像。由于划分为四个描述子片段，所以每个第二特征点描述子会对应有四个目标第一特征点描述子，这四个目标第一特征点描述子之间还可能存在重复。
对于当前图像上的k个第二特征点描述子，均执行上述过程，最多能够得到k*m个目标第一特征点描述子、每个目标第一特征点描述子对应的相似分数以及每个目标第一特征点描述子所属的关键帧图像。
步骤702d，对属于同一个关键帧图像的至少一个相似分数进行累加，将具有最高相似分数累加和的关键帧图像确定为目标关键帧图像。
对于当前图像的最多k*m个目标第一特征点描述子，由于很多目标第一特征点描述子可能属于同一个关键帧图像，因此将属于同一个关键帧图像的各个目标第一特征点描述子的相似分数进行累加，得到多个关键帧图像各自的相似分数累加和。将其中具有最高相似分数累加和的关键帧图像，确定为与当前图像对应的目标关键帧图像。
综上所述，本实施例提供的方法，能够利用哈希索引信息快速地从关键帧图像库中，查询到与当前图像匹配的目标关键帧图像，由于将关键帧图像的各个第一特征点描述子分散地存储在哈希索引信息中，能够减少查询与当前图像匹配的目标关键帧图像时需要匹配的第一特征点描述子的个数，不需要将所有第一特征点描述子都轮询匹配一遍，从而加速了整个匹配过程。
同时，由于哈希索引信息的数据量较少，不会占用太多的存储空间，因此本申请实施例提供的方法可以适用于移动终端的AR应用程序中，从而减少对移动终端上有限存储空间的占用。
关键帧图像的删除阶段:
关键帧图像库中的关键帧图像不应当太多，太多的关键帧图像会造成重定位过程的计算量负担，导致难以搜索到理想的图像。在一个示意性的例子中，关键帧图像库中的关键帧图像不超过1000张。在基于图7的可选实施例中，还可包括如下步骤，如图12所示：
步骤905，当关键帧图像库中的关键帧图像数量达到最大值时，从关键帧图像库中随机选择出第二关键帧图像；
步骤906，将所述第二关键帧图像从关键帧图像库中删除。
可选地，本步骤包括如下子步骤：
1、获取第二关键帧图像的各个第一特征点描述子；
2、对于每个第一特征点描述子，根据哈希分类规则在m列*n行条目存储位中确定出与第一特征点描述子对应的第一条目存储位中；
3、将第一特征点描述子和第二关键帧图像的标识，从第一条目存储位中进行删除。
也即，将第一特征点描述子的索引信息，从第一条目存储位中删除。
本实施例提供的方法，还通过在关键帧图像库中的关键帧图像达到最大值(比如1000张)时，将与候选图像最近的一个关键帧图像进行删除，使得关键帧图像库中尽量保留具有代表性的关键帧图像，从而保证在关键帧图像库中查询关键帧图像时的效率。
在基于图7的一个可选实施例中，若当前图像符合重定位条件，则设备优先将当前图像相对于第一个标记图像进行第三重定位；当第三重定位失败时，将当前图像尝试与关键帧图像库中的关键帧图像进行重定位。也即，优先进行零阶重定位，若零阶重定位成功，则使用零阶重定位结果，若零阶重定位不成功，则进行一阶重定位。
在基于图7的另一个可选的实施例中，若当前图像符合重定位条件，则设备并行进行两个重定位过程。一方面，将当前图像相对于关键帧图像库中的关键帧图像进行第二重定位；另一方面，将当前图像相对于第一个标记图像进行第三重定位。也即，一阶重定位和零阶重定位是独立的，可以并行的执行，当其中一个重定位成功时，采用成功的重定位结果；如果两个重定位均成功，则优先使用零阶重定位结果。
在基于图7的一个可选的实施例中，若上述方法应用于AR系统中，直接使用重定位的结果可能会导致跳变，设备可将计算得到目标姿态参数输入滤波器(卡尔曼滤波器或互补滤波器)，将目标姿态参数作为观测值，利用运动模型估计一个预测值，使用预测值作为实际使用的目标姿态参数，也即大概率相信观测值的前提下进行后续处理。从而使得AR系统得到平滑的结果，所显示用户画面中不会出现跳变，保证更好的用户体验。
在基于多个标记图像的相机姿态追踪过程中，还经常存在将当前图像相对于第i个标记图像进行特征点追踪的过程。上述哈希索引信息的快速索引方法还能够用于对两张图像之间的特征点匹配过程的加速。假设存在第一图像和第二图像，第一图像是第i个标记图像，第二图像是待匹配的当前图像。本申请实施例还提供有如下特征点匹配方法。
请参考图13，其示出了本申请一个示例性实施例提供的特征点匹配方法的流程图。该方法可以应用于图4所示的设备中，例如可以应用对第i个标记图像进行特征点匹配的过程中，也可以用于其他进行图像处理的计算机设备中。该方法包括：
步骤1301，获取第一图像的哈希索引信息，哈希索引信息包括m列*n行条目存储位，第一图像上的各个第一特征点描述子按照哈希分类规则分类存储于m列*n行条目存储位中各自对应的第一条目存储位中；
设第一图像是第i个标记图像，预先将第一图像上的各个第一特征点描述子按照哈希分类规则分类存储与m列*n行条目存储位中各自对应的第一条目存储位中。与上述实施例不同的是，上述实施例中是多张关键帧图像共同构建同一个哈希索引信息(hash bundle)，本实施例中是第一图像单独构建一个哈希索引信息。也即该哈希索引信息中，仅存储有第一图像的哈希索引信息，如图14所示。或者说，当第一图像是标记图像时，每个标记图像对应有各自的哈希的索引信息。该第一图像的哈希索引信息的存储过程可以参考上述步骤903的描述(图10)，本文不再赘述。
步骤1302，获取第二图像的各个第二特征点描述子；
设第二图像是待进行特征点追踪的图像，比如当前图像。
设备可以采用基于SIFT(Scale-Invariant Feature Transform，尺度不变特征转换)算法提取的SIFT特征点描述子、ORB(Oriented FAST and Rotated BRIEF，快速特征点提取和描述)算法提取的ORB特征点描述子，作为第二特征点描述子。
步骤1303，对于每个第二特征点描述子，按照哈希分类规则在m列*n行条目存储位中确定出与第二特征点描述子对应的第二条目存储位；
本步骤可以包括如下子步骤：
1、将第二特征点描述子划分为m个描述子片段；
2、对于m个描述子片段中的第i个描述子片段，获取第i个描述子片段的第二哈希值；
3、在m列*n行条目存储位中，确定位于第i列第j行的条目存储位作为第二条目存储位；
其中，第i列第j行的条目存储位是位于第i列的n个条目存储位中与第二哈希值匹配的条目存储位。
需要说明的是，第二条目存储位和第一条目存储位仅是采用不同的描述方式，代表两次哈希分类过程的结果，但实质上可以是m列*n行条目存储位中的同一个条目存储位。
步骤1304，在第二条目存储位中存储的各个第一特征点描述子中，确定出与第二特征点描述子匹配的第一特征点描述子。
可选地，第二条目存储位已存储有多个关键帧图像中的多个第一特征点描述子。这些第一特征点描述子是按照哈希分类规则被分类至同一类的特征点描述子。
对于当前图像上的第二特征点描述子，若该第二特征点描述子的第i个描述子片段被分类至该第二条目存储位，则可以计算该第二特征点描述子与第二条目存储位中每个第一特征点描述子的相似程度。可选地，采用相同比特数量与总比特数量的比值来表征该相似程度。将相似程度最高的第一特征点描述子，确定为与第二特征点描述子匹配的第一特征点描述子。
综上所述，本实施例提供的特征点匹配方法，能够利用哈希索引信息减少第二特征点描述子需要匹配的候选第一特征点描述子的数量，不需要对第一图像上的所有第一特征点描述子均进行匹配，仅需要匹配一部分条目存储位中的第一特征点描述子即可，因此能够加速匹配过程。
在一个示意性的例子中，如图15所示，S101，设备获取相机采集的当前图像；S102，若当前图像是用于扩展关键帧图像库的候选图像，将当前图像和born-image匹配(也即重定位)，得到第一重定位结果；S103，若当前图像相对于born-image匹配成功，则将当前图像作为一阶重定位成功的图像；S104，对于一阶重定位成功的图像，对自身建一个匹配用的hash-bundle，也即哈希索引信息，相关细节可以参考图13所示实施例；S105，对于一阶重定位成功的图像，可以加入总数据库(所有关键帧图像共用的hash-bundle)，根据情况增加、删除和查询，详细过程可参考上述的关键帧图像增加、删除和查询过程；S106，若当前图像是用于重定位的图像，则在总数据库中查找最接近的一阶关键帧图像；S107，当前图像同时与born-image和最接近的一阶关键帧图像进行匹配(也即重定位)；S108，根据匹配结果进行重定位，得到第二重定位结果。
在一个示意性的例子中，上述相机姿态追踪过程的重定位方法可以用于AR程序中，通过该重定位方法能够实时根据现实世界的场景信息，对终端上的相机姿态进行追踪，并根据追踪结果调整和修改AR应用程序中的AR元素的显示位置。以图1或图2所示的运行在手机上的AR程序为例，当需要显示一个站立在书籍上的静止卡通人物时，不论用户如何移动 该手机，只需要根据该手机上的相机姿态变化修改该卡通人物的显示位置，即可使该卡通人物在书籍上的站立位置保持不变。
以下为本申请的装置实施例，对于装置实施例中未详细描述的细节，可以参考如上所述的方法实施例。
请参考图16，其示出了本申请一个示例性实施例提供的相机姿态追踪过程的重定位装置的框图。该重定位装置可以通过软件、硬件或者两者的结合实现成为电子设备(或称移动终端)的全部或一部分，该重定位装置包括：图像获取模块1620、哈希查找模块1640、重定位模块1660和参数计算模块1680。
图像获取模块1620，用于获取所述多个标记图像中第i个标记图像后采集的当前图像，i＞1；
哈希查找模块1640，用于当所述当前图像符合重定位条件时，根据哈希索引信息从关键帧图像库中选择出目标关键帧图像；所述关键帧图像库中存储有至少一个关键帧图像对应的哈希索引信息，所述关键帧图像是在所述相机姿态追踪过程中缓存的相对于其它图像进行第一重定位成功的图像；
重定位模块1660，用于将所述当前图像相对于所述目标关键帧图像进行第二重定位；
参数计算模块1680，用于根据所述第一重定位的定位结果和所述第二重定位的定位结果，计算得到所述相机采集所述当前图像时的相机姿态参数。
在一个可选的实施例中，所述哈希索引信息包括m列*n行条目存储位，所述关键帧图像上的各个第一特征点描述子按照哈希分类规则分类存储于所述m列*n行条目存储位中对应的第一条目存储位中；
所述哈希查找模块1640，用于获取所述当前图像上的各个第二特征点描述子；对于每个所述第二特征点描述子，根据所述哈希分类规则在所述m列*n行条目存储位中确定出与所述第二特征点描述子对应的第二条目存储位中；在所述第二条目存储位中存储的所述第一特征点描述子中，确定出与所述第二特征点描述子相似度最高的目标第一特征点描述子，记录所述目标第一特征点描述子的相似分数和所属关键帧图像；对属于同一个所述关键帧图像的至少一个相似分数进行累加，将具有最高相似分数累加和的关键帧图像确定为所述目标关键帧图像。
在一个可选的实施例中，所述哈希查找模块1640，用于将所述第二特征点描述子划分为m个描述子片段；对于所述m个描述子片段中的第i个描述子片段，获取第i个描述子片段的第二哈希值；在所述m列*n行条目存储位中，确定位于第i列第j行的条目存储位作为所述第二条目存储位；
其中，所述第i列第j行的条目存储位是位于第i列的n个条目存储位中与所述第二哈希值匹配的条目存储位。
在一个可选的实施例中，所述装置还包括：关键帧添加模块1630；
所述图像获取模块1620，还用于获取最近一个重定位成功的候选图像；
所述关键帧添加模块1630，还用于确定所述候选图像是否满足添加条件，所述添加条件包括：所述候选图像和所述第一个标记图像之间的第一距离大于第一阈值，和/或，所述候选图像与上一次加入的关键帧图像之间的第二距离大于第二阈值；当所述候选图像满足所述添 加条件时，将所述候选图像作为第一关键帧图像加入所述关键帧图像库。
在一个可选的实施例中，所述关键帧添加模块1630，用于获取所述第一关键帧图像的各个第一特征点描述子；对于每个所述第一特征点描述子，根据所述哈希分类规则在所述m列*n行条目存储位中确定出与所述第一特征点描述子对应的第一条目存储位中；将所述第一特征点描述子和所述第一关键帧图像的标识，存储至所述第一条目存储位中。
在一个可选的实施例中，所述关键帧添加模块1630，用于将所述第一特征点描述子划分为m个描述子片段；对于所述m个描述子片段中的第i个描述子片段，获取第i个描述子片段的第一哈希值；在所述m列*n行条目存储位中，确定位于第i列第j行的条目存储位作为所述第一条目存储位；
其中，所述第i列第j行的条目存储位是位于第i列的n个条目存储位中与所述第一哈希值匹配的条目存储位。
在一个可选的实施例中，所述装置还包括：
关键帧删除模块1650，用于当所述关键帧图像库中的关键帧图像数量达到最大值时，从所述关键帧图像库中随机选择出第二关键帧图像；将所述第二关键帧图像从所述关键帧图像库中删除。
在一个可选的实施例中，所述关键帧删除模块1650，用于获取所述第二关键帧图像的各个第一特征点描述子；对于每个所述第一特征点描述子，根据所述哈希分类规则在所述m列*n行条目存储位中确定出与所述第一特征点描述子对应的第一条目存储位中；将所述第一特征点描述子和所述第二关键帧图像的标识，从所述第一条目存储位中进行删除。
在一个可选的实施例中，所述重定位模块1660，用于当所述当前图像符合重定位条件时，将所述当前图像相对于第一个标记图像进行第三重定位；
所述哈希查找模块1640，用于当所述第三重定位失败时，执行根据所述哈希索引信息从所述关键图像库中选择出所述目标关键帧图像的步骤。
需要说明的是：上述实施例提供的相机姿态追踪过程的重定位装置在实现重定位时，仅以上述各功能模块的划分进行举例说明，实际应用中，可以根据需要而将上述功能分配由不同的功能模块完成，即将设备的内部结构划分成不同的功能模块，以完成以上描述的全部或者部分功能。另外，上述实施例提供的重定位装置与重定位方法实施例属于同一构思，其具体实现过程详见方法实施例，这里不再赘述。
请参考图17，其示出了本申请一个示例性实施例提供的特征点匹配装置的框图。该特征点匹配装置可以通过软件、硬件或者两者的结合实现成为电子设备(或称移动终端)的全部或一部分，该特征点匹配装置包括：第一获取模块1720、第二获取模块1740、哈希分类模块1760和特征点匹配模块1780。
第一获取模块1720，用于获取第一图像的哈希索引信息，所述哈希索引信息包括m列*n行条目存储位，所述第一图像上的各个第一特征点描述子按照哈希分类规则分类存储于所述m列*n行条目存储位中各自对应的第一条目存储位中；
第二获取模块1740，用于获取第二图像的各个第二特征点描述子；
哈希分类模块1760，用于对于每个所述第二特征点描述子，按照所述哈希分类规则在所述m列*n行条目存储位中确定出与所述第二特征点描述子对应的第二条目存储位；
特征点匹配模块1780，用于在所述第二条目存储位中存储的各个所述第一特征点描述子 中，确定出与所述第二特征点描述子匹配的第一特征点描述子。
在一个可选的实施例中，所述哈希分类模块1760，用于将所述第二特征点描述子划分为m个描述子片段；对于所述m个描述子片段中的第i个描述子片段，获取第i个描述子片段的第二哈希值；在所述m列*n行条目存储位中，确定位于第i列第j行的条目存储位作为所述第二条目存储位；
其中，所述第i列第j行的条目存储位是位于第i列的n个条目存储位中与所述第二哈希值匹配的条目存储位。
需要说明的是：上述实施例提供的特征点匹配装置在两帧图像之间匹配特征点时，仅以上述各功能模块的划分进行举例说明，实际应用中，可以根据需要而将上述功能分配由不同的功能模块完成，即将设备的内部结构划分成不同的功能模块，以完成以上描述的全部或者部分功能。另外，上述实施例提供的特征点匹配装置与特征点匹配方法实施例属于同一构思，其具体实现过程详见方法实施例，这里不再赘述。
图18示出了本申请一个示例性实施例提供的电子设备1800的结构框图。该电子设备1800可以是：智能手机、平板电脑、MP3播放器(Moving Picture Experts Group Audio Layer III，动态影像专家压缩标准音频层面3)、MP4(Moving Picture Experts Group Audio Layer IV，动态影像专家压缩标准音频层面4)播放器、笔记本电脑或台式电脑。终端1800还可能被称为用户设备、便携式电子设备、膝上型电子设备、台式电子设备等其他名称。
通常，电子设备1800包括有：处理器1801和存储器1802。
处理器1801可以包括一个或多个处理核心，比如4核心处理器、8核心处理器等。处理器1801可以采用DSP(Digital Signal Processing，数字信号处理)、FPGA(Field－Programmable Gate Array，现场可编程门阵列)、PLA(Programmable Logic Array，可编程逻辑阵列)中的至少一种硬件形式来实现。处理器1801也可以包括主处理器和协处理器，主处理器是用于对在唤醒状态下的数据进行处理的处理器，也称CPU(Central Processing Unit，中央处理器)；协处理器是用于对在待机状态下的数据进行处理的低功耗处理器。在一些实施例中，处理器1801可以在集成有GPU(Graphics Processing Unit，图像处理器)，GPU用于负责显示屏所需要显示的内容的渲染和绘制。一些实施例中，处理器1801还可以包括AI(Artificial Intelligence，人工智能)处理器，该AI处理器用于处理有关机器学习的计算操作。
存储器1802可以包括一个或多个计算机可读存储介质，该计算机可读存储介质可以是非暂态的。存储器1802还可包括高速随机存取存储器，以及非易失性存储器，比如一个或多个磁盘存储设备、闪存存储设备。在一些实施例中，存储器1802中的非暂态的计算机可读存储介质用于存储至少一个指令，该至少一个指令用于被处理器1801所执行以实现本申请中方法实施例提供的相机姿态追踪过程的重定位方法。
在一些实施例中，电子设备1800还可选包括有：外围设备接口1803和至少一个外围设备。处理器1801、存储器1802和外围设备接口1803之间可以通过总线或信号线相连。各个外围设备可以通过总线、信号线或电路板与外围设备接口1803相连。示意性的，外围设备包括：射频电路1804、触摸显示屏1805、摄像头1806、音频电路1807、定位组件1808和电源1809中的至少一种。
外围设备接口1803可被用于将I/O(Input/Output，输入/输出)相关的至少一个外围设备连接到处理器1801和存储器1802。在一些实施例中，处理器1801、存储器1802和外围设 备接口1803被集成在同一芯片或电路板上；在一些其他实施例中，处理器1801、存储器1802和外围设备接口1803中的任意一个或两个可以在单独的芯片或电路板上实现，本实施例对此不加以限定。
射频电路1804用于接收和发射RF(Radio Frequency，射频)信号，也称电磁信号。射频电路1804通过电磁信号与通信网络以及其他通信设备进行通信。射频电路1804将电信号转换为电磁信号进行发送，或者，将接收到的电磁信号转换为电信号。可选地，射频电路1804包括：天线系统、RF收发器、一个或多个放大器、调谐器、振荡器、数字信号处理器、编解码芯片组、用户身份模块卡等等。射频电路1804可以通过至少一种无线通信协议来与其它电子设备进行通信。该无线通信协议包括但不限于：万维网、城域网、内联网、各代移动通信网络(2G、3G、4G及5G)、无线局域网和/或WiFi(Wireless Fidelity，无线保真)网络。在一些实施例中，射频电路1804还可以包括NFC(Near Field Communication，近距离无线通信)有关的电路，本申请对此不加以限定。
显示屏1805用于显示UI(User Interface，用户界面)。该UI可以包括图形、文本、图标、视频及其它们的任意组合。当显示屏1805是触摸显示屏时，显示屏1805还具有采集在显示屏1805的表面或表面上方的触摸信号的能力。该触摸信号可以作为控制信号输入至处理器1801进行处理。此时，显示屏1805还可以用于提供虚拟按钮和/或虚拟键盘，也称软按钮和/或软键盘。在一些实施例中，显示屏1805可以为一个，设置电子设备1800的前面板；在另一些实施例中，显示屏1805可以为至少两个，分别设置在电子设备1800的不同表面或呈折叠设计；在再一些实施例中，显示屏1805可以是柔性显示屏，设置在电子设备1800的弯曲表面上或折叠面上。甚至，显示屏1805还可以设置成非矩形的不规则图形，也即异形屏。显示屏1805可以采用LCD(Liquid Crystal Display，液晶显示屏)、OLED(Organic Light-Emitting Diode,有机发光二极管)等材质制备。
摄像头组件1806用于采集图像或视频。可选地，摄像头组件1806包括前置摄像头和后置摄像头。通常，前置摄像头设置在电子设备的前面板，后置摄像头设置在电子设备的背面。在一些实施例中，后置摄像头为至少两个，分别为主摄像头、景深摄像头、广角摄像头、长焦摄像头中的任意一种，以实现主摄像头和景深摄像头融合实现背景虚化功能、主摄像头和广角摄像头融合实现全景拍摄以及VR(Virtual Reality，虚拟现实)拍摄功能或者其它融合拍摄功能。在一些实施例中，摄像头组件1806还可以包括闪光灯。闪光灯可以是单色温闪光灯，也可以是双色温闪光灯。双色温闪光灯是指暖光闪光灯和冷光闪光灯的组合，可以用于不同色温下的光线补偿。
音频电路1807可以包括麦克风和扬声器。麦克风用于采集用户及环境的声波，并将声波转换为电信号输入至处理器1801进行处理，或者输入至射频电路1804以实现语音通信。出于立体声采集或降噪的目的，麦克风可以为多个，分别设置在电子设备1800的不同部位。麦克风还可以是阵列麦克风或全向采集型麦克风。扬声器则用于将来自处理器1801或射频电路1804的电信号转换为声波。扬声器可以是传统的薄膜扬声器，也可以是压电陶瓷扬声器。当扬声器是压电陶瓷扬声器时，不仅可以将电信号转换为人类可听见的声波，也可以将电信号转换为人类听不见的声波以进行测距等用途。在一些实施例中，音频电路1807还可以包括耳机插孔。
定位组件1808用于定位电子设备1800的当前地理位置，以实现导航或LBS(Location Based Service，基于位置的服务)。定位组件1808可以是基于美国的GPS(Global Positioning  System，全球定位系统)、中国的北斗系统或俄罗斯的伽利略系统的定位组件。
电源1809用于为电子设备1800中的各个组件进行供电。电源1809可以是交流电、直流电、一次性电池或可充电电池。当电源1809包括可充电电池时，该可充电电池可以是有线充电电池或无线充电电池。有线充电电池是通过有线线路充电的电池，无线充电电池是通过无线线圈充电的电池。该可充电电池还可以用于支持快充技术。
在一些实施例中，电子设备1800还包括有一个或多个传感器1810。该一个或多个传感器1810包括但不限于：加速度传感器1811、陀螺仪传感器1812、压力传感器1818、指纹传感器1814、光学传感器1815以及接近传感器1816。
加速度传感器1811可以检测以电子设备1800建立的坐标系的三个坐标轴上的加速度大小。比如，加速度传感器1811可以用于检测重力加速度在三个坐标轴上的分量。处理器1801可以根据加速度传感器1811采集的重力加速度信号，控制触摸显示屏1805以横向视图或纵向视图进行用户界面的显示。加速度传感器1811还可以用于游戏或者用户的运动数据的采集。
陀螺仪传感器1812可以检测电子设备1800的机体方向及转动角度，陀螺仪传感器1812可以与加速度传感器1811协同采集用户对电子设备1800的3D动作。处理器1801根据陀螺仪传感器1812采集的数据，可以实现如下功能：动作感应(比如根据用户的倾斜操作来改变UI)、拍摄时的图像稳定、游戏控制以及惯性导航。
压力传感器1818可以设置在电子设备1800的侧边框和/或触摸显示屏1805的下层。当压力传感器1818设置在电子设备1800的侧边框时，可以检测用户对电子设备1800的握持信号，由处理器1801根据压力传感器1818采集的握持信号进行左右手识别或快捷操作。当压力传感器1818设置在触摸显示屏1805的下层时，由处理器1801根据用户对触摸显示屏1805的压力操作，实现对UI界面上的可操作性控件进行控制。可操作性控件包括按钮控件、滚动条控件、图标控件、菜单控件中的至少一种。
指纹传感器1814用于采集用户的指纹，由处理器1801根据指纹传感器1814采集到的指纹识别用户的身份，或者，由指纹传感器1814根据采集到的指纹识别用户的身份。在识别出用户的身份为可信身份时，由处理器1801授权该用户执行相关的敏感操作，该敏感操作包括解锁屏幕、查看加密信息、下载软件、支付及更改设置等。指纹传感器1814可以被设置电子设备1800的正面、背面或侧面。当电子设备1800上设置有物理按键或厂商Logo时，指纹传感器1814可以与物理按键或厂商Logo集成在一起。
光学传感器1815用于采集环境光强度。在一个实施例中，处理器1801可以根据光学传感器1815采集的环境光强度，控制触摸显示屏1805的显示亮度。示意性的，当环境光强度较高时，调高触摸显示屏1805的显示亮度；当环境光强度较低时，调低触摸显示屏1805的显示亮度。在另一个实施例中，处理器1801还可以根据光学传感器1815采集的环境光强度，动态调整摄像头组件1806的拍摄参数。
接近传感器1816，也称距离传感器，通常设置在电子设备1800的前面板。接近传感器1816用于采集用户与电子设备1800的正面之间的距离。在一个实施例中，当接近传感器1816检测到用户与电子设备1800的正面之间的距离逐渐变小时，由处理器1801控制触摸显示屏1805从亮屏状态切换为息屏状态；当接近传感器1816检测到用户与电子设备1800的正面之间的距离逐渐变大时，由处理器1801控制触摸显示屏1805从息屏状态切换为亮屏状态。
本领域技术人员可以理解，图18中示出的结构并不构成对电子设备1800的限定，可以 包括比图示更多或更少的组件，或者组合某些组件，或者采用不同的组件布置。
上述本申请实施例序号仅仅为了描述，不代表实施例的优劣。
本领域普通技术人员可以理解实现上述实施例的全部或部分步骤可以通过硬件来完成，也可以通过程序来指令相关的硬件完成，所述的程序可以存储于一种计算机可读存储介质中，上述提到的存储介质可以是只读存储器，磁盘或光盘等。
以上所述仅为本申请的较佳实施例，并不用以限制本申请，凡在本申请的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本申请的保护范围之内。

Claims (24)

  1. 一种相机姿态追踪过程的重定位方法，其特征在于，应用于具有相机的设备中，所述设备用于按序执行多个标记图像的相机姿态追踪过程，所述方法包括：
    获取所述多个标记图像中第i个标记图像后采集的当前图像，i＞1；
    当所述当前图像符合重定位条件时，根据哈希索引信息从关键帧图像库中选择出目标关键帧图像；所述关键帧图像库中存储有至少一个关键帧图像对应的哈希索引信息，所述关键帧图像是在所述相机姿态追踪过程中缓存的相对于其它图像进行第一重定位成功的图像；
    将所述当前图像相对于所述目标关键帧图像进行第二重定位；
    根据所述第一重定位的定位结果和所述第二重定位的定位结果，计算得到所述相机采集所述当前图像时的相机姿态参数。
  2. 根据权利要求1所述的方法，其特征在于，所述哈希索引信息包括m列*n行条目存储位，所述关键帧图像上的各个第一特征点描述子按照哈希分类规则分类存储于所述m列*n行条目存储位中对应的第一条目存储位中；
    所述根据哈希索引信息从关键帧图像库中选择出目标关键帧图像，包括：
    获取所述当前图像上的各个第二特征点描述子；
    对于每个所述第二特征点描述子，根据所述哈希分类规则在所述m列*n行条目存储位中确定出与所述第二特征点描述子对应的第二条目存储位中；
    在所述第二条目存储位中存储的所述第一特征点描述子中，确定出与所述第二特征点描述子相似度最高的目标第一特征点描述子，记录所述目标第一特征点描述子的相似分数和所属关键帧图像；
    对属于同一个所述关键帧图像的至少一个相似分数进行累加，将具有最高相似分数累加和的关键帧图像确定为所述目标关键帧图像。
  3. 根据权利要求2所述的方法，其特征在于，所述根据所述哈希分类规则在所述m列*n行条目存储位中确定出与所述第二特征点描述子对应的目标条目存储位，包括：
    将所述第二特征点描述子划分为m个描述子片段；
    对于所述m个描述子片段中的第i个描述子片段，获取第i个描述子片段的第二哈希值；
    在所述m列*n行条目存储位中，确定位于第i列第j行的条目存储位作为所述第二条目存储位；
    其中，所述第i列第j行的条目存储位是位于第i列的n个条目存储位中与所述第二哈希值匹配的条目存储位。
  4. 根据权利要求1至3任一所述的方法，其特征在于，所述方法还包括：
    获取最近一个重定位成功的候选图像；
    确定所述候选图像是否满足添加条件，所述添加条件包括：所述候选图像和所述第一个标记图像之间的第一距离大于第一阈值，和/或，所述候选图像与上一次加入的关键帧图像之间的第二距离大于第二阈值；
    当所述候选图像满足所述添加条件时，将所述候选图像作为第一关键帧图像加入所述关键帧图像库。
  5. 根据权利要求4所述的方法，其特征在于，所述将所述候选图像作为第一关键帧图像加入所述关键帧图像库，包括：
    获取所述第一关键帧图像的各个第一特征点描述子；
    对于每个所述第一特征点描述子，根据所述哈希分类规则在所述m列*n行条目存储位中确定出与所述第一特征点描述子对应的第一条目存储位中；
    将所述第一特征点描述子和所述第一关键帧图像的标识，存储至所述第一条目存储位中。
  6. 根据权利要求5所述的方法，其特征在于，所述根据所述哈希分类规则在所述m列*n行条目存储位中确定出与所述第一特征点描述子对应的第一条目存储位中，包括：
    将所述第一特征点描述子划分为m个描述子片段；
    对于所述m个描述子片段中的第i个描述子片段，获取第i个描述子片段的第一哈希值；
    在所述m列*n行条目存储位中，确定位于第i列第j行的条目存储位作为所述第一条目存储位；
    其中，所述第i列第j行的条目存储位是位于第i列的n个条目存储位中与所述第一哈希值匹配的条目存储位。
  7. 根据权利要求4所述的方法，其特征在于，所述方法还包括：
    当所述关键帧图像库中的关键帧图像数量达到最大值时，从所述关键帧图像库中随机选择出第二关键帧图像；
    将所述第二关键帧图像从所述关键帧图像库中删除。
  8. 根据权利要求7所述的方法，其特征在于，所述将所述待删关键帧图像从所述关键帧图像库中删除，包括：
    获取所述第二关键帧图像的各个第一特征点描述子；
    对于每个所述第一特征点描述子，根据所述哈希分类规则在所述m列*n行条目存储位中确定出与所述第一特征点描述子对应的第一条目存储位中；
    将所述第一特征点描述子和所述第二关键帧图像的标识，从所述第一条目存储位中进行删除。
  9. 根据权利要求1至8任一所述的方法，其特征在于，所述当所述当前图像符合重定位条件时，根据哈希索引信息从关键帧图像库中选择出目标关键帧图像，包括：
    当所述当前图像符合重定位条件时，将所述当前图像相对于第一个标记图像进行第三重定位；
    当所述第三重定位失败时，执行根据所述哈希索引信息从所述关键图像库中选择出所述目标关键帧图像的步骤。
  10. 一种特征点匹配方法，其特征在于，所述方法包括：
    获取第一图像的哈希索引信息，所述哈希索引信息包括m列*n行条目存储位，所述第一图像上的各个第一特征点描述子按照哈希分类规则分类存储于所述m列*n行条目存储位中各自对应的第一条目存储位中；
    获取第二图像的各个第二特征点描述子；
    对于每个所述第二特征点描述子，按照所述哈希分类规则在所述m列*n行条目存储位中确定出与所述第二特征点描述子对应的第二条目存储位；
    在所述第二条目存储位中存储的各个所述第一特征点描述子中，确定出与所述第二特征点描述子匹配的第一特征点描述子。
  11. 根据权利要求10所述的方法，其特征在于，所述按照所述哈希分类规则在所述m列*n行条目存储位中确定出与所述第二特征点描述子对应的目标条目存储位，包括：
    将所述第二特征点描述子划分为m个描述子片段；
    对于所述m个描述子片段中的第i个描述子片段，获取第i个描述子片段的第二哈希值；
    在所述m列*n行条目存储位中，确定位于第i列第j行的条目存储位作为所述第二条目存储位；
    其中，所述第i列第j行的条目存储位是位于第i列的n个条目存储位中与所述第二哈希值匹配的条目存储位。
  12. 一种相机姿态追踪过程的重定位装置，其特征在于，应用于具有相机的设备中，所述设备用于按序执行多个标记图像的相机姿态追踪过程，所述装置包括：
    图像获取模块，用于获取所述多个标记图像中第i个标记图像后采集的当前图像，i＞1；
    哈希查找模块，用于当所述当前图像符合重定位条件时，根据哈希索引信息从关键帧图像库中选择出目标关键帧图像；所述关键帧图像库中存储有至少一个关键帧图像对应的哈希索引信息，所述关键帧图像是在所述相机姿态追踪过程中缓存的相对于其它图像进行第一重定位成功的图像；
    重定位模块，用于将所述当前图像相对于所述目标关键帧图像进行第二重定位；
    参数计算模块，用于根据所述第一重定位的定位结果和所述第二重定位的定位结果，计算得到所述相机采集所述当前图像时的相机姿态参数。
  13. 根据权利要求12所述的装置，其特征在于，所述哈希索引信息包括m列*n行条目存储位，所述关键帧图像上的各个第一特征点描述子按照哈希分类规则分类存储于所述m列*n行条目存储位中对应的第一条目存储位中；
    所述哈希查找模块，用于获取所述当前图像上的各个第二特征点描述子；对于每个所述第二特征点描述子，根据所述哈希分类规则在所述m列*n行条目存储位中确定出与所述第二特征点描述子对应的第二条目存储位中；在所述第二条目存储位中存储的所述第一特征点描述子中，确定出与所述第二特征点描述子相似度最高的目标第一特征点描述子，记录所述目标第一特征点描述子的相似分数和所属关键帧图像；对属于同一个所述关键帧图像的至少一个相似分数进行累加，将具有最高相似分数累加和的关键帧图像确定为所述目标关键帧图像。
  14. 根据权利要求13所述的装置，其特征在于，
    所述哈希查找模块，用于将所述第二特征点描述子划分为m个描述子片段；对于所述m个描述子片段中的第i个描述子片段，获取第i个描述子片段的第二哈希值；在所述m列*n行条目存储位中，确定位于第i列第j行的条目存储位作为所述第二条目存储位；
    其中，所述第i列第j行的条目存储位是位于第i列的n个条目存储位中与所述第二哈希值匹配的条目存储位。
  15. 根据权利要求13至14任一所述的装置，其特征在于，所述装置还包括：关键帧添加模块；
    所述图像获取模块，还用于获取最近一个重定位成功的候选图像；
    所述关键帧添加模块，还用于确定所述候选图像是否满足添加条件，所述添加条件包括：所述候选图像和所述第一个标记图像之间的第一距离大于第一阈值，和/或，所述候选图像与上一次加入的关键帧图像之间的第二距离大于第二阈值；当所述候选图像满足所述添加条件时，将所述候选图像作为第一关键帧图像加入所述关键帧图像库。
  16. 根据权利要求15所述的装置，其特征在于，
    所述关键帧添加模块，用于获取所述第一关键帧图像的各个第一特征点描述子；对于每个所述第一特征点描述子，根据所述哈希分类规则在所述m列*n行条目存储位中确定出与所述第一特征点描述子对应的第一条目存储位中；将所述第一特征点描述子和所述第一关键帧图像的标识，存储至所述第一条目存储位中。
  17. 根据权利要求16所述的装置，其特征在于，
    所述关键帧添加模块，用于将所述第一特征点描述子划分为m个描述子片段；对于所述m个描述子片段中的第i个描述子片段，获取第i个描述子片段的第一哈希值；在所述m列*n行条目存储位中，确定位于第i列第j行的条目存储位作为所述第一条目存储位；
    其中，所述第i列第j行的条目存储位是位于第i列的n个条目存储位中与所述第一哈希值匹配的条目存储位。
  18. 根据权利要求15所述的装置，其特征在于，所述装置还包括：
    关键帧删除模块，用于当所述关键帧图像库中的关键帧图像数量达到最大值时，从所述关键帧图像库中随机选择出第二关键帧图像；将所述第二关键帧图像从所述关键帧图像库中删除。
  19. 根据权利要求18所述的装置，其特征在于，
    所述关键帧删除模块，用于获取所述第二关键帧图像的各个第一特征点描述子；对于每个所述第一特征点描述子，根据所述哈希分类规则在所述m列*n行条目存储位中确定出与所述第一特征点描述子对应的第一条目存储位中；将所述第一特征点描述子和所述第二关键帧图像的标识，从所述第一条目存储位中进行删除。
  20. 根据权利要求12至19任一所述的装置，其特征在于，
    所述重定位模块，用于当所述当前图像符合重定位条件时，将所述当前图像相对于第一个标记图像进行第三重定位；
    所述哈希查找模块，用于当所述第三重定位失败时，执行根据所述哈希索引信息从所述关键图像库中选择出所述目标关键帧图像的步骤。
  21. 一种特征点匹配装置，其特征在于，所述装置包括：
    第一获取模块，用于获取第一图像的哈希索引信息，所述哈希索引信息包括m列*n行条目存储位，所述第一图像上的各个第一特征点描述子按照哈希分类规则分类存储于所述m列*n行条目存储位中各自对应的第一条目存储位中；
    第二获取模块，用于获取第二图像的各个第二特征点描述子；
    哈希分类模块，用于对于每个所述第二特征点描述子，按照所述哈希分类规则在所述m列*n行条目存储位中确定出与所述第二特征点描述子对应的第二条目存储位；
    特征点匹配模块，用于在所述第二条目存储位中存储的各个所述第一特征点描述子中，确定出与所述第二特征点描述子匹配的第一特征点描述子。
  22. 根据权利要求21所述的装置，其特征在于，
    所述哈希分类模块，用于将所述第二特征点描述子划分为m个描述子片段；对于所述m个描述子片段中的第i个描述子片段，获取第i个描述子片段的第二哈希值；在所述m列*n行条目存储位中，确定位于第i列第j行的条目存储位作为所述第二条目存储位；
    其中，所述第i列第j行的条目存储位是位于第i列的n个条目存储位中与所述第二哈希值匹配的条目存储位。
  23. 一种电子设备，其特征在于，所述电子设备包括存储器和处理器；
    所述存储器中存储有至少一条指令，所述至少一条指令由所述处理器加载并执行以实现如权利要求1至11任一所述的相机姿态追踪过程中的重定位方法。
  24. 一种计算机可读存储介质，其特征在于，所述存储介质中存储有至少一条指令，所述至少一条指令由处理器加载并执行以实现如权利要求1至11任一所述的相机姿态追踪过程中的重定位方法。
PCT/CN2019/079355 2018-04-27 2019-03-22 相机姿态追踪过程的重定位方法、装置、设备及存储介质 WO2019205853A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP19792551.4A EP3786894A4 (en) 2018-04-27 2019-03-22 METHOD, DEVICE AND APPARATUS FOR REPOSITIONING IN A CAMERA ORIENTATION TRACKING PROCESS, AND INFORMATION HOLDER
US16/900,634 US11189037B2 (en) 2018-04-27 2020-06-12 Repositioning method and apparatus in camera pose tracking process, device, and storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810392368.5A CN108615247B (zh) 2018-04-27 2018-04-27 相机姿态追踪过程的重定位方法、装置、设备及存储介质
CN201810392368.5 2018-04-27

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US16/900,634 Continuation US11189037B2 (en) 2018-04-27 2020-06-12 Repositioning method and apparatus in camera pose tracking process, device, and storage medium

Publications (1)

Publication Number Publication Date
WO2019205853A1 true WO2019205853A1 (zh) 2019-10-31

Family

ID=63661267

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/079355 WO2019205853A1 (zh) 2018-04-27 2019-03-22 相机姿态追踪过程的重定位方法、装置、设备及存储介质

Country Status (4)

Country Link
US (1) US11189037B2 (zh)
EP (1) EP3786894A4 (zh)
CN (1) CN108615247B (zh)
WO (1) WO2019205853A1 (zh)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110866496A (zh) * 2019-11-14 2020-03-06 合肥工业大学 基于深度图像的机器人定位与建图方法和装置
CN111340707A (zh) * 2020-02-27 2020-06-26 创新奇智(重庆)科技有限公司 基于关键帧的图像拼接系统及拼接方法
CN112489083A (zh) * 2020-12-07 2021-03-12 之江实验室 基于orb-slam算法的图像特征点跟踪匹配方法
CN113674412A (zh) * 2021-08-12 2021-11-19 浙江工商大学 基于位姿融合优化的室内地图构建方法、系统及存储介质
US20220300751A1 (en) * 2021-03-17 2022-09-22 Kabushiki Kaisha Toshiba Image processing device and image processing method

Families Citing this family (24)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108734736B (zh) 2018-05-22 2021-10-26 腾讯科技（深圳）有限公司 相机姿态追踪方法、装置、设备及存储介质
CN113987228A (zh) * 2018-06-20 2022-01-28 华为技术有限公司 一种数据库构建方法、一种定位方法及其相关设备
CN109544615B (zh) * 2018-11-23 2021-08-24 深圳市腾讯信息技术有限公司 基于图像的重定位方法、装置、终端及存储介质
CN111322993B (zh) * 2018-12-13 2022-03-04 杭州海康机器人技术有限公司 一种视觉定位方法和装置
CN109579847B (zh) * 2018-12-13 2022-08-16 歌尔股份有限公司 同步定位与地图构建中关键帧提取方法、装置和智能设备
CN111696157B (zh) * 2019-03-12 2024-06-18 北京京东尚科信息技术有限公司 图像重定位的确定方法、系统、设备和存储介质
CN109947886B (zh) * 2019-03-19 2023-01-10 腾讯科技（深圳）有限公司 图像处理方法、装置、电子设备及存储介质
CN110261823B (zh) * 2019-05-24 2022-08-05 南京航空航天大学 基于单个led灯的可见光室内通信定位方法及系统
CN110170167B (zh) * 2019-05-28 2023-02-28 上海米哈游网络科技股份有限公司 一种画面显示方法、装置、设备及介质
CN112446915B (zh) * 2019-08-28 2024-03-29 北京初速度科技有限公司 一种基于图像组的建图方法及装置
CN111339228B (zh) * 2020-02-18 2023-08-11 Oppo广东移动通信有限公司 一种地图更新方法、设备、云服务器和存储介质
CN111311588B (zh) * 2020-02-28 2024-01-05 浙江商汤科技开发有限公司 重定位方法及装置、电子设备和存储介质
CN111402412B (zh) * 2020-04-16 2023-06-09 Oppo广东移动通信有限公司 数据采集方法及装置、设备、存储介质
CN111862148B (zh) * 2020-06-05 2024-02-09 中国人民解放军军事科学院国防科技创新研究院 实现视觉跟踪的方法、装置、电子设备及介质
CN111950642B (zh) * 2020-08-17 2024-06-21 联想(北京)有限公司 一种重定位方法及电子设备
CN112348888B (zh) * 2020-09-09 2023-06-20 北京市商汤科技开发有限公司 显示设备的定位方法、装置、显示设备及计算机存储介质
CN113538574B (zh) * 2021-01-04 2022-09-27 腾讯科技（深圳）有限公司 位姿定位方法、装置、设备及计算机可读存储介质
CN112750164B (zh) * 2021-01-21 2023-04-18 脸萌有限公司 轻量化定位模型的构建方法、定位方法、电子设备
EP4057226A1 (en) * 2021-03-09 2022-09-14 Samsung Electronics Co., Ltd. Method and apparatus for estimating pose of device
CN115082549A (zh) * 2021-03-10 2022-09-20 北京图森智途科技有限公司 位姿估计方法及装置、以及相关设备和存储介质
CN115082516A (zh) * 2021-03-15 2022-09-20 北京字跳网络技术有限公司 一种目标跟踪方法、装置、设备及介质
CN116862979A (zh) * 2022-03-25 2023-10-10 北京字跳网络技术有限公司 重定位方法及相关设备
CN116258769B (zh) * 2023-05-06 2023-07-25 亿咖通(湖北)技术有限公司 一种定位校验方法、装置、电子设备和存储介质
CN116528062B (zh) * 2023-07-05 2023-09-15 合肥中科类脑智能技术有限公司 多目标追踪方法

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20120038616A (ko) * 2010-10-14 2012-04-24 한국전자통신연구원 마커리스 실감형 증강현실 제공 방법 및 시스템
CN106885574A (zh) * 2017-02-15 2017-06-23 北京大学深圳研究生院 一种基于重跟踪策略的单目视觉机器人同步定位与地图构建方法
CN106934827A (zh) * 2015-12-31 2017-07-07 杭州华为数字技术有限公司 三维场景的重建方法和装置

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7688381B2 (en) * 2003-04-08 2010-03-30 Vanbree Ken System for accurately repositioning imaging devices
US20100259610A1 (en) * 2009-04-08 2010-10-14 Celsia, Llc Two-Dimensional Display Synced with Real World Object Movement
CN101996217B (zh) * 2009-08-24 2012-11-07 华为技术有限公司 一种存储数据的方法及其存储设备
US20120159368A1 (en) * 2010-12-17 2012-06-21 Ebay Inc. Search history navigation
US8892542B2 (en) * 2011-02-24 2014-11-18 Nec Laboratories America, Inc. Contextual weighting and efficient re-ranking for vocabulary tree based image retrieval
JP6457648B2 (ja) * 2015-01-27 2019-01-23 ノキア テクノロジーズ オサケユイチア 位置特定およびマッピングの方法
CN105808709B (zh) * 2016-03-04 2019-10-29 智慧眼科技股份有限公司 人脸识别快速检索方法及装置
WO2017172778A1 (en) * 2016-03-28 2017-10-05 Sri International Collaborative navigation and mapping
CN106780601B (zh) * 2016-12-01 2020-03-27 北京未动科技有限公司 一种空间位置追踪方法、装置及智能设备
CN107193279A (zh) * 2017-05-09 2017-09-22 复旦大学 基于单目视觉和imu信息的机器人定位与地图构建系统
CN107301402B (zh) * 2017-06-30 2020-06-16 锐捷网络股份有限公司 一种现实场景关键帧的确定方法、装置、介质及设备

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20120038616A (ko) * 2010-10-14 2012-04-24 한국전자통신연구원 마커리스 실감형 증강현실 제공 방법 및 시스템
CN106934827A (zh) * 2015-12-31 2017-07-07 杭州华为数字技术有限公司 三维场景的重建方法和装置
CN106885574A (zh) * 2017-02-15 2017-06-23 北京大学深圳研究生院 一种基于重跟踪策略的单目视觉机器人同步定位与地图构建方法

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3786894A4

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110866496A (zh) * 2019-11-14 2020-03-06 合肥工业大学 基于深度图像的机器人定位与建图方法和装置
CN111340707A (zh) * 2020-02-27 2020-06-26 创新奇智(重庆)科技有限公司 基于关键帧的图像拼接系统及拼接方法
CN112489083A (zh) * 2020-12-07 2021-03-12 之江实验室 基于orb-slam算法的图像特征点跟踪匹配方法
US20220300751A1 (en) * 2021-03-17 2022-09-22 Kabushiki Kaisha Toshiba Image processing device and image processing method
US11921823B2 (en) * 2021-03-17 2024-03-05 Kabushiki Kaisha Toshiba Image processing device and image processing method
CN113674412A (zh) * 2021-08-12 2021-11-19 浙江工商大学 基于位姿融合优化的室内地图构建方法、系统及存储介质
CN113674412B (zh) * 2021-08-12 2023-08-29 浙江工商大学 基于位姿融合优化的室内地图构建方法、系统及存储介质

Also Published As

Publication number Publication date
EP3786894A1 (en) 2021-03-03
US11189037B2 (en) 2021-11-30
US20200302615A1 (en) 2020-09-24
CN108615247B (zh) 2021-09-14
CN108615247A (zh) 2018-10-02
EP3786894A4 (en) 2022-01-26

Similar Documents

Publication Publication Date Title
WO2019205853A1 (zh) 相机姿态追踪过程的重定位方法、装置、设备及存储介质
CN110555883B (zh) 相机姿态追踪过程的重定位方法、装置及存储介质
CN108596976B (zh) 相机姿态追踪过程的重定位方法、装置、设备及存储介质
CN108615248B (zh) 相机姿态追踪过程的重定位方法、装置、设备及存储介质
US11158083B2 (en) Position and attitude determining method and apparatus, smart device, and storage medium
CN110544280B (zh) Ar系统及方法
CN108876854B (zh) 相机姿态追踪过程的重定位方法、装置、设备及存储介质
WO2019205851A1 (zh) 位姿确定方法、装置、智能设备及存储介质
CN109947886B (zh) 图像处理方法、装置、电子设备及存储介质
US11210810B2 (en) Camera localization method and apparatus, terminal, and storage medium
CN108682037B (zh) 相机姿态追踪过程的重定位方法、装置、设备及存储介质
JP2022511427A (ja) 画像特徴点の動き情報の決定方法、タスク実行方法およびデバイス
CN108776822B (zh) 目标区域检测方法、装置、终端及存储介质
US20230047514A1 (en) Method and apparatus for detection and tracking, and storage medium
WO2019134305A1 (zh) 确定姿态的方法、装置、智能设备、存储介质和程序产品
CN110941327A (zh) 虚拟对象的显示方法和装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19792551

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

WWE Wipo information: entry into national phase

Ref document number: 2019792551

Country of ref document: EP