WO2019205868A1 - 相机姿态追踪过程的重定位方法、装置、设备及存储介质 - Google Patents

相机姿态追踪过程的重定位方法、装置、设备及存储介质 Download PDF

Info

Publication number
WO2019205868A1
WO2019205868A1 PCT/CN2019/079781 CN2019079781W WO2019205868A1 WO 2019205868 A1 WO2019205868 A1 WO 2019205868A1 CN 2019079781 W CN2019079781 W CN 2019079781W WO 2019205868 A1 WO2019205868 A1 WO 2019205868A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
target image
end program
pose parameter
pose
Prior art date
Application number
PCT/CN2019/079781
Other languages
English (en)
French (fr)
Inventor
林祥凯
凌永根
暴林超
刘威
Original Assignee
腾讯科技（深圳）有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 腾讯科技（深圳）有限公司 filed Critical 腾讯科技（深圳）有限公司
Priority to EP19794077.8A priority Critical patent/EP3786895A4/en
Publication of WO2019205868A1 publication Critical patent/WO2019205868A1/zh
Priority to US16/919,059 priority patent/US11276183B2/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G06T7/74—Determining position or orientation of objects or cameras using feature-based methods involving reference images or patches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G06T7/248—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments involving reference images or patches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/269—Analysis of motion using gradient-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30204—Marker
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30244—Camera pose

Definitions

  • the embodiments of the present application relate to the field of augmented reality, and in particular, to a method, a device, a device, and a storage medium for relocating a camera attitude tracking process.
  • the main body of the camera needs to track the camera's motion posture in the real environment during the movement without the prior information of the environment.
  • the motion process of the camera is divided into at least two tracking processes for tracking, and each tracking process corresponds to a respective marker image (Anchor).
  • the first frame image captured by the camera is used as the first mark image.
  • the device tracks the feature points common between the current image and the first mark image, according to the current image and the first image.
  • the change of the feature point position between one marker image calculates the camera pose change of the camera in the real world.
  • the previous image of the current image is switched to the i+1th marker image, and the i+1th segment tracking process is turned on. Where i is a positive integer.
  • the embodiment of the present application provides a relocation method, device, device, and storage medium for a camera attitude tracking process, which can be used to solve the relocation process, which is time consuming, and may not only block the main thread, but also may be in several frame images.
  • the problem of relocation results will be obtained later.
  • the technical solution is as follows:
  • a relocation method of a camera attitude tracking process is provided, which is applied to a device running a front end program and a back end program, the front end program being used to sequentially execute camera gestures of a plurality of mark images Tracking process, the method comprising:
  • the front end program acquires a target image acquired after the i th mark image of the plurality of mark images, where i is a positive integer;
  • the front end program switches the target image to an i+1th mark image, and sends the target image to the backend program;
  • the front-end program acquires a current image acquired after the target image, and performs feature point tracking on the current image with respect to the target image to obtain a first pose parameter, wherein the first pose parameter is used to represent the slave pose a camera pose change amount between the target image and the current image;
  • the backend program relocates the target image to obtain a second pose parameter, and sends the second pose parameter to the front end program, the second pose parameter is used to represent the image from the first mark a camera pose change amount to the target image;
  • the front end program calculates a current pose parameter of the current image according to the first pose parameter and the second pose parameter.
  • a relocation device for a camera attitude tracking process comprising: a front end program module and a back end program module, wherein the front end program module is configured to execute in sequence Camera attitude tracking process for marking images;
  • the front-end program module is further configured to acquire a target image acquired after the i-th mark image of the plurality of mark images, where i is a positive integer;
  • the front end program module is further configured to switch the target image to an i+1th mark image when the target image meets a relocation condition, and send the target image to the backend program module;
  • the front-end program module is further configured to acquire a current image acquired after the target image, and perform feature point tracking on the current image with respect to the target image to obtain a first pose parameter, the first pose parameter Means for indicating a camera pose change amount from the target image to the current image;
  • the backend program module is configured to reposition the target image to obtain a second pose parameter, and send the second pose parameter to the front end program, where the second pose parameter is used to indicate a camera pose change amount between a mark image and the target image;
  • the front end program module is configured to calculate a current pose parameter of the current image according to the first pose parameter and the second pose parameter.
  • an electronic device including a memory and a processor
  • At least one instruction is stored in the memory, the at least one instruction being loaded and executed by the processor to implement the relocation method as described above.
  • a computer readable storage medium having stored therein at least one instruction loaded by a processor and executed to implement a relocation method as described above.
  • the target image is switched to the i+1th mark image by the front end program, and the front end program performs the camera attitude tracking based on the i+1th mark image in parallel in the process of repositioning the target image by the front end program.
  • the pose parameter if the relocation result (the second pose parameter) is obtained when the front end program processes the current image, the first pose parameter and the second pose parameter are used to correct the positioning result of the current image, thereby solving the backend program.
  • the relocation result before several images obtained can not be directly used for the relocation of the current image; even if the relocation result of the backend program is long, it can pass The two pose changes are superimposed to correct the positioning result of the current image, thereby ensuring a better repositioning effect.
  • FIG. 1 is a schematic diagram of a scenario of an AR application scenario provided by an exemplary embodiment of the present application
  • FIG. 2 is a schematic diagram of a scenario of an AR application scenario provided by an exemplary embodiment of the present application
  • FIG. 3 is a schematic diagram of a schematic diagram of an Anchor-Switching AR System algorithm provided by an exemplary embodiment of the present application
  • FIG. 4 is a structural block diagram of an electronic device provided by an exemplary embodiment of the present application.
  • FIG. 5 is a timing diagram of a relocation process provided by an exemplary embodiment of the present application.
  • FIG. 6 is a flowchart of a method for relocating a camera pose tracking process provided by an exemplary embodiment of the present application
  • FIG. 7 is a computational schematic diagram of a normal tracking process provided by an exemplary embodiment of the present application.
  • FIG. 8 is a schematic diagram of calculation of a relocation process provided by an exemplary embodiment of the present application.
  • FIG. 9 is a flowchart of a method for relocating a camera attitude tracking process according to an exemplary embodiment of the present application.
  • FIG. 10 is a schematic diagram of comparison of a first marker image, a target image, and a first-order key frame image provided by an exemplary embodiment of the present application;
  • FIG. 11 is a flowchart of a method for relocating a camera pose tracking process provided by an exemplary embodiment of the present application.
  • FIG. 12 is a schematic diagram of a principle of a relocation method according to an exemplary embodiment of the present application.
  • FIG. 13 is a schematic diagram showing the working principle of a backend program provided by an exemplary embodiment of the present application.
  • FIG. 14 is a schematic diagram of a working principle of a front end program according to an exemplary embodiment of the present application.
  • FIG. 15 is a block diagram of a relocation device of a camera pose tracking process provided by an exemplary embodiment of the present application.
  • FIG. 16 is a block diagram of an electronic device provided by an exemplary embodiment of the present application.
  • AR A technique for calculating camera pose parameters in the real world (or three-dimensional world, real world) in real time during camera acquisition of images, and adding virtual elements to images captured by the camera according to the camera pose parameters.
  • Virtual elements include, but are not limited to, images, video, and 3D models.
  • the goal of AR technology is to interact with the virtual world on the screen in the real world.
  • the camera pose parameters include a rotation matrix and a displacement vector, the rotation matrix is used to characterize the rotation angle of the camera in the real world, and the displacement vector is used to characterize the displacement distance of the camera in the real world.
  • the device adds a virtual character image to the image captured by the camera.
  • the image captured by the camera changes, and the orientation of the avatar changes. It simulates that the avatar is still in the image, and the camera changes with position and posture.
  • the effect of capturing images and avatars gives the user a realistic three-dimensional picture.
  • Anchor-Switching AR System is based on the camera attitude tracking connected to multiple marker images (Anchor) to determine the camera pose parameters in the natural scene, and then superimpose the virtual world AR system on the images captured by the camera according to the camera pose parameters.
  • IMU Inertial Measurement Unit
  • an IMU consists of three single-axis accelerometers and three single-axis gyros.
  • the accelerometer is used to detect the acceleration signal of each object in each coordinate axis of the three-dimensional coordinate system, and then calculate the displacement vector (may be The cumulative error is followed by an increase in time; and the gyro is used to detect the rotation matrix of the object in a three-dimensional coordinate system.
  • the IMU includes a gyroscope, an accelerometer, and a geomagnetic sensor.
  • the three-dimensional coordinate system is established as follows: 1.
  • the X-axis is defined by the vector product Y*Z. In the current position of the device, the X-axis points to the east in a direction tangent to the ground; 2.
  • the Y-axis is At the current position of the device, it points in the direction tangent to the ground to the north pole of the earth's magnetic field; 3.
  • the Z axis points to the sky and is perpendicular to the ground.
  • the target image fails to perform feature point tracking with respect to the i-th marker image, and the camera attitude tracking process cannot be smoothly performed, the target image is compared with other images ( The first mark image or other representative key frame image is re-performed with feature points so that when the feature point tracking is successful, the camera captures the camera pose when the target image is captured according to the retrieved tracking process.
  • the present application provides a relocation method suitable for the Anchor-Switching AR System algorithm.
  • the Anchor-Switching AR System algorithm divides the camera's motion process into at least two tracking processes for tracking. Each tracking process corresponds to the respective marker image. Specifically, when the tracking effect corresponding to the i-th marker image is worse than the preset condition (for example, the feature point that can be matched is less than a preset threshold), when the tracking effect corresponding to the i-th marker image is different, The previous image of the current image is determined to be the i+1th mark image, and the i+1th segment tracking process is turned on. Where i is a positive integer.
  • FIG. 3 is a schematic diagram showing the principle of an Anchor-Switching AR System algorithm provided by an exemplary embodiment of the present application.
  • an object 320 is present, the device 340 provided with the camera is moved by the user, and a multi-frame image 1-6 including the object 320 is captured during the movement.
  • the device determines the image 1 as the first marker image (born-anchor or born-image) and records the initial pose parameter, which may be the pose parameter when the camera acquired by the IMU captures the image 1.
  • the image 2 is tracked with respect to the image 1 , and the pose parameter of the camera when the image 2 is captured is calculated according to the initial pose parameter and the feature point tracking result; the feature point tracking of the image 3 relative to the image 1 is performed according to the initial The pose parameter and the feature point tracking result calculate the pose parameter of the camera when the image 3 is captured; the image 4 is tracked with respect to the image 1 , and the camera is captured according to the initial pose parameter and the feature point tracking result 4 The pose parameter of the time.
  • the image 5 is tracked with respect to the image 1 , and if the feature point tracking effect is worse than the preset condition (for example, the number of matching feature points is small), the image 4 is determined as the second marker image, and the image 5 is Performing feature point tracking with respect to the image 4, calculating the amount of displacement change of the camera between the captured image 4 and the image 5, and calculating the amount of displacement change between the captured image 4 and the image 1 and the initial pose parameter, The pose parameter of the camera when taking image 5. Then, the image 6 is tracked with respect to the image 4, and so on. If the feature point tracking effect of the current image is deteriorated, the previous frame image of the current image can be determined as a new mark image, and the new image is switched. Feature point tracking is performed after the image is marked.
  • the preset condition for example, the number of matching feature points is small
  • feature point tracking may employ an algorithm based on a visual odometer principle, such as a feature point method or a direct method.
  • a visual odometer principle such as a feature point method or a direct method.
  • the Anchor-Switching AR System tracking process may be lost. Loss phenomenon means that enough feature points cannot be matched in the target image, resulting in feature point tracking failure.
  • the target image needs to be relocated.
  • the relocation process takes a lot of time, not only may block the main thread, but may also get the relocation result after several frames of images.
  • the electronic device includes a processor 420, a memory 440, a camera 460, and an IMU 480.
  • Processor 420 includes one or more processing cores, such as a 4-core processor, an 8-core processor, and the like.
  • the processor 420 is configured to execute at least one of instructions, code, code segments, and programs stored in the memory 440.
  • the processor 420 is electrically connected to the memory 440.
  • processor 420 is coupled to memory 440 via a bus.
  • Memory 440 stores one or more instructions, code, code segments, and/or programs. The instructions, code, code segments, and/or programs, when executed by processor 420, are used to implement a relocation method of the camera pose tracking process provided in the following embodiments.
  • the processor 420 is also electrically coupled to the camera 460.
  • processor 420 is coupled to camera 460 via a bus.
  • Camera 460 is a sensor device having image acquisition capabilities. Camera 460 may also be referred to as a camera, a photosensitive device, and the like. Camera 460 has the ability to continuously acquire images or acquire images multiple times.
  • camera 460 is located inside or outside the device.
  • the processor 420 is also electrically connected to the IMU 480.
  • the IMU 480 is configured to acquire the pose vector of the camera every predetermined time interval and record the time stamp of each set of pose parameters at the time of acquisition.
  • the camera's pose parameters include: displacement vector and rotation matrix. Among them, the rotation matrix acquired by IMU480 is relatively accurate, and the displacement vector acquired may have a large error due to the actual environment.
  • the embodiment of the present application provides a relocation architecture of a "front end program + back end program".
  • the front-end program runs in the main thread, and the back-end program runs in another thread.
  • the front end program is used to sequentially execute a camera pose tracking process of a plurality of mark images;
  • the back end program is used to reposition the target image when the target image satisfies the relocation condition.
  • Inter-process communication is implemented between the front-end program and the back-end program through a Result module.
  • the inventor found that if the relocation process is performed in the front-end program, since the relocation process usually takes a long time, the normal operation of the blocking front-end program may occur; Executed in the backend program, since the backend program cannot control the success time of the relocation result, the relocation result before several frames of images may be obtained, which cannot be directly used for relocation of the current image.
  • FIG. 5 it is assumed that when the front end program processes the image frame m (referred to as a target image in the following embodiment), the image frame m is lost, and the backend program starts repositioning the image frame m.
  • the present application provides the following examples.
  • FIG. 6 is a flowchart of a method for relocating a camera attitude tracking process provided by an embodiment of the present application. This embodiment is exemplified by applying the relocation method to the device shown in FIG. 4. The method includes:
  • Step 601 The front end program acquires a target image acquired after the i-th mark image in the plurality of mark images, where i is a positive integer;
  • the camera in the device collects a frame image at a preset time interval to form an image sequence.
  • the camera acquires a frame of image forming image sequence according to a preset time interval during motion (translation and/or rotation).
  • the front end program determines the first frame image in the image sequence (or one frame image in the first few frames of images that meets the predetermined condition) as the first marker image, and compares the subsequently acquired image with the first marker image.
  • the frame image is determined as a second marker image, and the subsequently acquired image is subjected to feature point tracking with respect to the second marker image, and the camera pose parameter of the camera is calculated according to the feature point tracking result, and so on.
  • the front-end program can sequentially perform camera attitude tracking of a plurality of marked images in sequence.
  • Step 602 when the target image meets the relocation condition, the front end program switches the target image to the i+1th mark image;
  • the device determines if the current image meets the relocation criteria.
  • the relocation condition is used to indicate that the tracking process of the current image with respect to the i-th flag image fails, or the re-location condition is used to indicate that the accumulated error in the history tracking process has been higher than the preset condition.
  • the device tracks the current image relative to the i-th mark image, if there is no feature point matching the i-th mark image in the current image, or the i-th mark in the current image When the feature points of the image matching are less than the first number, it is determined that the tracking process of the current image with respect to the i-th flag image fails, and the relocation condition is met.
  • the device determines that the number of frames between the current image and the last relocated image is greater than the second number, determining that the accumulated error in the history tracking process is higher than a preset condition, or When it is determined that the number of mark images between the i-th mark image and the first mark image is greater than the third number, it is determined that the cumulative error in the history tracking process has been higher than the preset condition.
  • This embodiment does not limit the specific condition content of the relocation condition.
  • the front end program determines that the target image satisfies the relocation condition, the front end program switches the target image to the i+1th mark image, and temporarily captures the camera pose parameters R old and T old (possible error) collected by the IMU as the i th +1 marker image reference pose parameters.
  • R old is used to represent the rotation matrix between the first marker image and the i+1th marker image (target image)
  • T old is used to indicate that the camera is from the first marker image to the i+1th
  • Step 603 the front end program sends the target image to the backend program
  • the front end program uses the result module to send the target image to the backend program.
  • the result module is the storage space shared by the front-end program and the back-end program.
  • the backend program is used to reposition the target image relative to other images.
  • Other images include: the first marker image, and/or a keyframe image that was once successfully relocated and representative.
  • Step 604 the front end program acquires the current image acquired after the target image, and performs feature point tracking on the current image with respect to the target image to obtain a first pose parameter;
  • the front end program After the target image is provided to the backend program, the front end program acquires the current image acquired after the i+1th mark image (target image), and performs feature point tracking on the current image with respect to the target image to obtain a first pose parameter.
  • the first pose parameter is used to represent the amount of camera pose change from the target image to the current image.
  • the first pose parameter includes a first rotation matrix R ca and a first displacement vector T ca , and R ca is used to represent a rotation matrix between the i+1th marker image (target image) and the current image of the camera, for T ca Indicates the displacement vector between the camera from the i+1th marker image (target image) to the current image.
  • Step 605 The backend program repositions the target image to obtain a second pose parameter.
  • step 604 the backend program repositions the target image relative to other images to obtain a second pose parameter.
  • the second pose parameter is used to represent the amount of camera pose change from the first marker image to the target image.
  • the second pose parameter includes a second rotation matrix R relocalize and a second displacement vector T relocalice .
  • R relocalice is used to indicate the rotation matrix of the camera from the first marker image to the i+ 1th marker image (target image), which is the correction value for the above R old obtained by relocation;
  • T relocalice is used to indicate that the camera is from the first
  • the displacement vector between the one marker image and the (i+1)th marker image (target image) is a correction value for the above T old obtained by relocation.
  • Step 606 The backend program sends a second pose parameter to the front end program.
  • the second pose parameter that is relocated is sent to the second pose parameter through the result module.
  • Step 607 The front end program calculates a current pose parameter of the current image according to the first pose parameter and the second pose parameter.
  • the front end program calculates the current pose parameters R m+k and T m+k according to the following formula:
  • the first pose parameter includes a first rotation matrix R ca and a first displacement vector T ca
  • the second pose parameter includes a second rotation matrix R relocalize and a second displacement vector T relocalice , where s is a target image scale.
  • s is a preset value, or s is that the front end program maps the initial feature points on the first marked image to the target image to obtain the projected feature points, and calculates the target image according to the average depth of the projected feature points. Corresponding scale.
  • the relocation method switches the target image to the i+1th tag image through the front end program, and is re-positioned by the front end program in parallel in the process of relocating the target image by the backend program.
  • the camera pose tracking of the i+1th marker image obtains the first pose parameter. If the relocation result (second pose parameter) is obtained when the front end program processes the current image, the first pose parameter and the second pose are used.
  • the parameter corrects the positioning result of the current image, thereby solving the problem that the backend program cannot control the success of the relocation result, and the relocation result before several images obtained can not be directly used for the relocation of the current image;
  • the relocation result of the end program is longer, and the correction of the positioning result of the current image can be realized by superimposing the two pose changes, thereby ensuring a better repositioning effect.
  • Fig. 7 In the process of camera attitude tracking based on the i-th marker image, the i-th marker image has a pose change amount R _i_old and T _i_old relative to the first marker image, if the image frame m The feature point tracking is successfully performed with respect to the i-th mark image, and the pose change amounts R ca1 and T ca1 are obtained, and the camera pose parameter of the image frame m is calculated by the following formula:
  • s i is a scale corresponding to the ith marker image.
  • the camera pose parameter of the image frame m+k is calculated by the following formula:
  • Fig. 8 In the process of camera attitude tracking based on the i-th marker image, if the target image is lost, the front-end program switches the target image to the i+1th marker image, and the reference collected by the IMU is used. The pose parameter is used as the pose change amount R old and T old of the i+1th mark image with respect to the first mark image. If the rear-end program successfully relocates the target image, the relocation results R relocalize and T relocalice are obtained. Then, the camera pose parameter of the current image (image frame m+k) is calculated by the following formula:
  • the backend program repositions the target image relative to the first marker image to obtain a second pose parameter.
  • the above step 605 includes the following sub-steps:
  • Step 605a acquiring an initial feature point and an initial pose parameter of the first marker image, where the initial pose parameter is used to indicate a camera pose when the camera captures the first marker image;
  • the backend program attempts to track the current image relative to the first marker image.
  • the device acquires an initial feature point in the first marked image of the cache and an initial pose parameter, which is used to indicate a camera pose when the camera captures the first marker image.
  • Step 605b performing feature point tracking on the target image with respect to the first marker image, to obtain a target feature point that matches the initial feature point;
  • each set of matching feature point pairs includes two initial feature points and target feature points that match each other.
  • the feature point tracking can use a visual odometer-based tracking algorithm, which is not limited in this application.
  • feature point tracking uses a KLT (Kanade-Lucas) optical flow tracking algorithm; in another embodiment, feature point tracking is based on a SIFT (Scale-Invariant Feature Transform) algorithm.
  • SIFT Scale-Invariant Feature Transform
  • ORB Oriented FAST and Rotated BRIEF
  • the specific algorithm for feature point tracking is not limited in this application, and the feature point tracking process may adopt a feature point method or a direct method.
  • the backend program performs feature point extraction on the first marker image to obtain N initial feature points; the device also performs feature point extraction on the target image to obtain M candidate feature points; and then M candidate features.
  • the points are matched one by one with the N initial feature points to determine at least one set of matching feature point pairs.
  • Each set of matching feature point pairs includes: an initial feature point and a target feature point.
  • the initial feature point is a feature point on the first marker image
  • the target feature point is a candidate feature point on the target image that has the highest matching degree with the initial feature point.
  • the number of initial feature points is greater than or equal to the number of target feature points.
  • the number of initial feature points is 450 and the target feature points are 320.
  • Step 605c Calculate, according to the initial feature point and the target feature point, a second pose parameter when the camera changes from the initial pose parameter to the target pose parameter, where the target pose parameter is used to represent a camera pose of the camera when acquiring the target image;
  • the device calculates the homography matrix homography between the two image images according to the initial feature point and the target feature point; decomposes the homography matrix homography to obtain a pose when the camera changes from the initial posture parameter to the target posture parameter.
  • the homography matrix describes the mapping relationship between two planes. If the feature points in the natural scene (real environment) fall on the same physical plane, the motion estimation can be performed through the homography matrix.
  • the device decomposes the homography matrix by ransac to obtain a rotation matrix R relocalize and a translation vector T relocalize .
  • R relocalize is the rotation matrix when the camera changes from the initial attitude parameter to the target posture parameter
  • T relocalize is the displacement parameter when the camera changes from the initial posture parameter to the target posture parameter
  • step 605d the second pose parameter is obtained by repositioning according to the initial pose parameter and the pose change amount.
  • the second pose parameters R relocalize and T relocalice are obtained by repositioning , thereby calculating the camera pose of the camera when the target image is acquired.
  • the relocation method shown in FIG. 8 is to reposition the target image as the first marker image, and the method may have the possibility of relocation failure in some scenarios.
  • the difference between the shooting posture corresponding to the target image and the shooting posture of the first marking image is large, directly establishing the matching of the target image and the first marking image to achieve the relocation fails.
  • (1) of FIG. 10 is the first marker image
  • FIG. 10 (2) is the target image. Since the overlapping area between the target image and the first marker image is too small, the image is not obtained. Sufficient matching, so relocating the target image and the first marker image directly will fail.
  • the (3) diagram of FIG. 10 is introduced as a first-order key frame image, and the target image is relocated with respect to the first-order key frame image, which can improve the success probability of relocation.
  • the backend program repositions the target image relative to the first marker image to obtain a second pose parameter.
  • the above step 605 includes the following sub-steps:
  • Step 6051 selecting a target key frame image from a key frame image library
  • the key frame image library stores image information of at least one key frame image, and the key frame image is an image that is successfully relocated in the history camera attitude tracking process and has a first repositioning result with respect to the first marker image;
  • the device will select some representative images in the relocated image as key frame images to save to the key frame image library.
  • the keyframe image library is a database for storing keyframe images.
  • the key frame image library stores image information of a plurality of key frame images.
  • the image information includes: an image of the key frame image itself, a feature point in the key frame image, and a first relocation result corresponding to the key frame image.
  • the image information further includes: a first global descriptor of the key frame image, wherein the first global descriptor is used to represent the image feature of the key frame image in a brief amount of data.
  • the first global descriptor is used to uniquely represent the image features of the key frame image with a brief amount of data.
  • the backend program picks up the target keyframe image from the keyframe image library.
  • the target key frame image is an image that is closest to the image distance of the target image in the key frame image library.
  • the image distance can be represented by any one of the image similarity, the image feature similarity, and the distance between the matched feature points, which is not limited by the embodiment of the present application.
  • the device selects the target key frame image from the key frame image library by the similarity between the first global descriptor of the key frame image and the second global descriptor of the target image; in another embodiment In the middle, since the IMU can collect the reference pose parameters (possibly errors) when the camera captures the target image, the device passes the similarity between the first pose parameter of the key frame image and the reference pose parameter of the target image from the key frame image library. To pick the target keyframe image.
  • Step 6052 relocating the target image with respect to the target key frame image to obtain a second relocation result
  • the target image is repositioned relative to the target key frame image.
  • the process of relocating includes the following steps:
  • the key frame image library stores key frame feature points and first camera pose tracking results on each key frame image.
  • the key frame feature points are characterized by SIFT feature descriptors or ORB feature descriptors.
  • the first relocation result is a camera pose parameter (referred to as a key frame attitude parameter) when the camera captures a key frame image, or the first relocation result is a camera pose parameter when the camera captures a key frame image, and the first marker is collected with respect to the camera.
  • each set of matching feature point pairs includes two mutually matching key frame feature points and target feature points.
  • the feature point tracking can use a visual odometer-based tracking algorithm, which is not limited in this application.
  • the feature point tracking uses a KLT optical flow tracking algorithm; in another embodiment, the feature point tracking uses the SIFT feature descriptor extracted by the SIFT algorithm and the ORB feature descriptor extracted by the ORB algorithm for feature point tracking.
  • the specific algorithm for feature point tracking is not limited in this application, and the feature point tracking process may adopt a feature point method or a direct method.
  • the device since the target key frame image is an image that has been successfully relocated, the device has extracted feature points of the target key frame image in the historical process to obtain N key frame feature points; the device also performs feature points on the target image. Extracting, obtaining M candidate feature points; then matching the M candidate feature points one by one with the N initial feature points to determine at least one set of matching feature point pairs.
  • Each set of matching feature point pairs includes: a key frame feature point and a target feature point.
  • the key frame feature point is a feature point on the target key frame image
  • the target feature point is a candidate feature point on the target image that has the highest matching degree with the key frame feature point.
  • the number of key frame feature points is greater than or equal to the number of target feature points.
  • the number of key frame feature points is 480 and the target feature points are 350.
  • the device calculates a homography matrix homography between the two frame images according to the key frame feature point and the target feature point; and decomposes the homography matrix homography to obtain a camera changing from the key frame attitude parameter to the second posture parameter
  • the pose change amounts R cm and T cm , and the pose change amounts R cm and T cm are used as the second relocation result.
  • R cm is the rotation matrix when the camera changes from the key frame attitude parameter to the second posture parameter
  • T cm is the displacement vector when the camera changes from the key frame attitude parameter to the second posture parameter
  • Step 6053 Calculate, according to the first relocation result and the second relocation result, a second posture parameter when the camera acquires the target image.
  • the first relocation result includes R mf and T mf .
  • R mf is the rotation matrix when the camera changes from the initial attitude parameter to the key frame attitude parameter
  • T mf is the displacement vector when the camera changes from the initial attitude parameter to the key frame attitude parameter
  • the second relocation result includes R cm and T cm .
  • R cm is the rotation matrix when the camera changes from the key frame attitude parameter to the second posture parameter
  • T cm is the displacement vector when the camera changes from the key frame attitude parameter to the second posture parameter.
  • the key frame image library includes: a first-order key frame image successfully performed with respect to the first mark image for first-order relocation, and/or, with respect to the n-1 order key frame image in the key frame image library.
  • the foregoing is only exemplified by the first-order relocation image.
  • For the first relocation result of the n-th order key frame image multiple pose changes can be multiplied, which is not described in this application.
  • the front end program maps the initial feature points on the first marker image to the target image to obtain the projected feature points, and calculates the scale corresponding to the target image according to the average depth of the projected feature points. as follows:
  • the front end program pre-fetches and caches the two-dimensional coordinates of the initial feature points in the first marked image.
  • the two-dimensional coordinates are represented in homogeneous order.
  • the front-end program transforms the two-dimensional coordinates of the initial feature points into a three-dimensional space by the following formula, and obtains the first three-dimensional coordinates X born of the initial feature points in the three-dimensional space.
  • f x , f y , c x , and c y are built-in parameters of the camera.
  • the two-dimensional coordinate x born of the initial feature point is a homogeneous representation of the layer-keyPoints on the first marker image
  • the three-dimensional point X born is a non-homogeneous representation. Assume that the initial depth d of the first marker image is 1.
  • the first three-dimensional coordinate X born is subjected to three-dimensional rotational translation by the following formula, and a second three-dimensional coordinate X current corresponding to the initial feature point on the target image is obtained ;
  • R is the rotation matrix in the reference pose change collected by the IMU
  • T is the displacement vector in the reference pose change acquired by the IMU.
  • the front end program projects the second three-dimensional coordinate X current to the target image by the following formula, and obtains the two-dimensional coordinate x current of the projected feature point in the target image:
  • f x , f y , c x , and c y are built-in parameters of the camera.
  • a front end program and a back end program pass a Result Module
  • the result module is a storage space shared by the front end program and the back end program.
  • the result module includes: a relocalization list, a result-flag, and a result storage bit.
  • the result module further includes: an expand-map list and an Abandon Time.
  • the relocation list is used to store the target image that needs to be relocated. Optionally, there is at most one target image in the relocation list at the same time. When the current program needs to perform relocation, the target image is written to the relocation list.
  • the result tag is used to store the working state of the backend program, including: ready, have-result, and still-sunning.
  • Ready indicates that the backend program is started and idle; have-result indicates that the backend program has completed the last relocation, and there are two cases of relocation success (OK) or relocation failure (Fail); still-sunning Indicates that it is being calculated and has not been calculated.
  • the result storage bit is used to store the second pose parameters R relocalize and T relocalice when the relocation is successful .
  • the expanded map list is used to store candidate images of the keyframe image library.
  • the current program passes the image to the extended graph list for the backend program to extend the keyframe image library.
  • the last switch time is used to store the time when the front-end program last switched the marker image. For example, when the front end program sets the target image to the i+1th mark image, the switching time of the i+1th mark image is recorded to the Abandon Time.
  • the backend program is a thread that continues to run in the background.
  • the backend program runs in a different thread from the foreground program, and the interprocess communication is implemented through the result module.
  • the backend program runs the steps in Figure 13 through the while(1) loop:
  • Step 1200 waiting for a new image
  • step 1201 it is determined that the relocation list is empty.
  • the relocation list When the relocation list is empty, it is determined that the foreground program does not have a target image that needs to be relocated, and the process proceeds to step 1202. When the relocation list is not empty, it is determined that the foreground program has a target image that needs to be relocated, and the process proceeds to step 1204.
  • Step 1202 Determine whether the extended graph list is empty?
  • Step 1203 expanding the key frame image library
  • the backend program expands the key frame image in the keyframe database by using the candidate image in the extended graph list, so that the key frame image in the key frame image library is added or updated.
  • Step 1204 setting the result label to be running (still-running);
  • the backend program sets the result flag to still-running and then begins relocating the target image.
  • Step 1205 Perform relocation on the target image that needs to be relocated.
  • the backend program relocates the target image relative to the first marker image, and relocates the target image relative to the keyframe image library when the relocation fails.
  • the backend program repositions the target image in parallel with respect to the first marker image and repositions the target image relative to the keyframe image library.
  • step 1206 it is determined whether the relocation process is successful.
  • Step 1207 whether the relocation success time is earlier than the last handover time
  • the last switch time used to indicate when the front end program last switched the marked image is the last switch time used to indicate when the front end program last switched the marked image.
  • the relocation success time When the relocation success time is earlier than the last switching time, it indicates that the foreground program has switched the image one or more times after switching the target image to the i+1th tag image, and the relocation result is invalid. Go to step 1208; when the relocation success time is later than the last handover time, the relocation result is valid, and the process proceeds to step 1209.
  • Step 1208 setting the result label to fail
  • the backend program sets the result flag to fail.
  • step 1209 the result label is set to be successful.
  • the backend program sets the result flag to OK and writes the second pose parameters R relocalize and T relocalice to the result module.
  • the front-end program runs in the main thread and is responsible for sequentially performing a camera tracking process of multiple markup images.
  • the back-end program and the front-end program run in different threads, and the process module implements the inter-process process. Communication.
  • flag1 and flag2 There are two tags set in the front-end program: flag1 and flag2, flag1 is used to indicate whether relocation is required, and flag2 is used to indicate that relocation fails, and the marker needs to be switched again.
  • the front-end program is used to perform the following steps, as shown in Figure 14:
  • Step 1300 Camera attitude tracking based on a plurality of marker images
  • the front end program is responsible for the camera pose tracking process based on multiple markup images. Illustratively, assume that the front end program is performing the camera pose tracking process for the ith marker image.
  • Step 1301 whether there is a tracking result
  • the target image acquired after the front end program acquires the i-th mark image, and after performing the feature point tracking on the target image with respect to the i-th mark image, determines whether there is a tracking result. If there is a tracking result, the process proceeds to step 1302; if the tracking result does not exist, the process proceeds to step 1303.
  • Step 1302 setting flag1 to false
  • step 1300 The camera attitude tracking process of step 1300 is continued.
  • the front-end program sets flag1 to true, indicating that the target image needs to be relocated.
  • Step 1304 detecting a state of flag1
  • the front-end program also detects the state of flag1 in each frame of image processing; if the state of flag1 is false, step 1305 is executed, and the front-end program sends the target image to the extended graph list for the back-end program to expand the key frame image library. If the state of flag1 is true, the process proceeds to step 1306, and the front-end program considers that the target image being processed needs to be relocated, and detects the state of flag2.
  • step 1307 If the state of flag2 is no, it indicates that the last relocation succeeds, and the process proceeds to step 1307. If the state of flag2 is YES, it indicates that the last relocation failed, and the process proceeds to step 1313.
  • Step 1307 detecting whether the running status of the backend program is ready?
  • the front-end program reads the result label in the result module. If the result label is ready, the back-end program is in an idle state, and the target image can be relocated.
  • Step 1308, switching the i+1th tag image, sending the latest modification time, and sending the target image to the relocation list;
  • the front end program switches the target image to the i+1th tag image and writes the switching time to the abandoned time in the result module.
  • Step 1309 attempting to obtain a relocation result of the backend program
  • the front end program reads the result label in the result module. If the result label is still-running, the process proceeds to step 1310. If the result label is have result_ok, the process proceeds to step 1311. If the result label is have result_fail, the process proceeds to step 1312.
  • Step 1310 doing other transactions
  • the front end program can perform the feature point tracking process on the current image after the target image.
  • Step 1311 setting flag1 to fail
  • setting flag1 to failure indicates that the re-positioning is not required, and the camera tracking process based on the i+1th mark image is performed.
  • Step 1312 setting flag2 to be successful
  • step 1313 the image is switched to the i+2th tag image, and the latest modification time is transmitted.
  • the current end program acquires the current image after the target image, if flag1 is true and flag2 is true, it indicates that the last relocation process of the backend program failed, for example, the relocation process of the target image fails, and the front end program needs to target
  • the current image after the image is sent to the relocation list so that the backend program initiates the relocation process for the current image.
  • the front end program switches the current image to the i+2th mark image, and writes the switching time to the abandoned time in the result module again.
  • the backend program extends the keyframe image in the keyframe image library using candidate images in the expanded map list, including the following steps:
  • Step 1 Obtain a candidate image that is successfully relocated recently
  • the key frame image library can be empty, and the backend program uses only the first marker image as the marker image at the time of relocation. As the entire camera tracking process is performed, more and more images are successfully relocated with the first marker image (or the keyframe image that has been added to the keyframe image library).
  • the backend program will relocate the successfully image as a candidate image that may be added to the keyframe image library each time the relocation succeeds.
  • the relocation succeeds in that the relocation is successful with respect to the first marker image, or the relocation is successful with respect to the existing keyframe image in the keyframe image library.
  • the backend program picks up a portion of the image from the candidate image as a keyframe image and adds it to the keyframe image library.
  • Step 2 determining whether the candidate image satisfies the adding condition, the adding condition includes: the first distance between the candidate image and the first marked image is greater than the first threshold, and/or between the candidate image and the last added key frame image The second distance is greater than the second threshold;
  • the key frame image added this time needs to be at a certain distance from the first marker image, because if the two images are relatively close, there is no significant difference from the effect of directly relocating with the first marker image.
  • the backend program calculates a first distance between the candidate image and the first marker image, the first distance being used to represent image similarity or camera pose similarity between the candidate image and the first marker image.
  • the specific calculation manner of the first distance is not limited in this application.
  • for the initial feature points in the first mark image there are target feature points in the candidate image that match the initial feature points, one The initial feature point and the corresponding target feature point constitute a matching feature point pair, and the L2 distance (the Euclidean distance corresponding to the L2 norm) is calculated according to each set of matching feature points, and the average value of all L2 distances is taken as the candidate image.
  • the first distance between the image and the first marker image Taking the first threshold as 50 pixels as an example, when the first distance between the candidate image and the first marker image is greater than 50 pixels, the candidate image is added to the key frame image library.
  • the key frame image added this time needs to have a certain distance from the last added key frame image, because if the two images are relatively close, there is no significant difference between the two images being relocated using the last added key frame image. .
  • the backend program calculates a second distance between the candidate image and the last added keyframe image, and the second distance is used to represent image similarity or camera pose similarity between the candidate image and the last added keyframe image.
  • the specific calculation manner of the second distance is not limited in this application.
  • the number of relocated images between the candidate image and the last added key frame image is used as the second distance, and the second For example, if the threshold value is 10, if the number of relocated images between the candidate image and the last added key frame image exceeds 10 frames, the candidate image is added to the key frame image library.
  • Step 3 when the candidate image satisfies the adding condition, adding the candidate image to the key frame image library;
  • the candidate image When it is determined that the candidate image satisfies the addition condition, the candidate image is added as a key frame image to the key frame image library.
  • the first global descriptor of the key frame image, the key frame feature point, and the first relocated positioning result are stored in the key frame image library.
  • the first global descriptor is used to represent the image feature of the key frame image with a brief data amount, and the first relocated positioning result may be represented by a key frame attitude parameter when the camera captures the key frame image, or may be performed by using a camera.
  • the amount of posture change when the initial posture parameter is changed to the key frame posture parameter is expressed.
  • Step 4 When the candidate image does not satisfy the adding condition, the candidate image is not added to the key frame image library.
  • the candidate image is not added Keyframe image library.
  • the key frame image satisfying the above adding condition may be used to expand the key frame image library, but the key frame image in the key frame image library should not be too much, and too many key frame images may cause calculation amount of the relocation process.
  • the burden makes it difficult to search for the ideal image.
  • the keyframe image in the keyframe image library does not exceed 1000 sheets.
  • the similarity between the candidate image and the existing key frame image is calculated.
  • the key frame image with the greatest similarity is deleted from the key frame image library, and the candidate image is added to the key frame image library.
  • the relocation method of the camera attitude tracking process described above may be used in an AR program, by which the camera pose on the electronic device can be tracked in real time according to real world scene information, and according to The tracking result adjusts and modifies the display position of the AR element in the AR application.
  • the AR program running on the mobile phone shown in FIG. 1 or FIG. 2 as an example, when it is necessary to display a still cartoon character standing on a book, no matter how the user moves the mobile phone, it only needs to change according to the camera posture on the mobile phone. Modifying the display position of the cartoon character will keep the standing position of the cartoon character on the book unchanged.
  • FIG. 15 is a block diagram of a relocating apparatus of a camera pose tracking process provided by an exemplary embodiment of the present application.
  • the relocating device may be implemented as a whole or a part of an electronic device (or mobile electronic device) by software, hardware or a combination of the two, the relocating device comprising: a front end program module 1520 and a back end program module 1540, the front end The program module 1520 is configured to sequentially perform a camera attitude tracking process of the plurality of mark images;
  • the front end program module 1520 is further configured to acquire a target image acquired after the i th mark image of the plurality of mark images, where i is a positive integer;
  • the front end program module 1520 is further configured to: when the target image meets a relocation condition, switch the target image to an i+1th tag image, and send the target image to the backend program module. ;
  • the front-end program module 1520 is further configured to acquire a current image acquired after the target image, and perform feature point tracking on the current image with respect to the target image to obtain a first pose parameter, the first pose a parameter for indicating a camera pose change amount from the target image to the current image;
  • the backend program module 1540 is configured to reposition the target image to obtain a second pose parameter, and send the second pose parameter to the front end program, where the second pose parameter is used to indicate a camera gesture change amount between the first marker image and the target image;
  • the front end program module 1520 is configured to calculate a current pose parameter of the current image according to the first pose parameter and the second pose parameter.
  • the backend program module 1540 is configured to reposition the target image relative to the first marker image to obtain the second pose parameter; and/or, The backend program module 1540 is configured to reposition the target image with respect to a key frame image in a key frame image library to obtain the second pose parameter; the key frame image is successfully relocated and relatively An image of the relocation result exists in the first marker image.
  • the key frame image library includes:
  • Performing a first-order key frame image with a first-order relocation succeeding with respect to the first mark image; and/or performing n-th order relocation with respect to an n-1 order key frame image in the key frame image library An n-th order key frame image, where n is an integer greater than one.
  • the front end program module 1520 is configured to calculate a current pose parameter of the current image according to the first pose parameter and the second pose parameter, including:
  • the current pose parameters R n and T n are calculated as follows by the following formula:
  • the first pose parameter includes a first rotation matrix R ca and a first displacement vector T ca
  • the second pose parameter includes a second rotation matrix R relocalize and a second displacement vector T relocalice
  • s is The scale corresponding to the target image.
  • the front end program module 1520 is configured to map an initial feature point on the first mark image to the target image to obtain a projected feature point, according to the projected feature point.
  • the average depth calculation yields a scale corresponding to the target image.
  • the apparatus further includes a result module 1560, where the result module 1560 is a storage space shared by the front end program module and the back end program module;
  • the front end program module 1520 is configured to write the target image into the relocated image list in the result module 1560;
  • the backend program module 1540 is configured to read the target image from the relocated image list.
  • the front end program module 1520 is configured to detect whether a result status bit in the result module 1560 is a ready state; the front end program module 1520 is configured to: when the detection result is the result When the status bit is the ready state, the target image is written to the relocated image list in the result module 1560.
  • the backend program module 1540 is configured to write the second pose parameter into a result storage bit in the result module
  • the result module 1560 is the front end program module. 1520 and a storage space shared by the backend program module 1540.
  • the front end program module 1520 is configured to read the second pose parameter from the result storage bit.
  • the front end program module 1520 is configured to write the target image into an extended graph list in the result module; the backend program module 1540 is configured to be in the target image.
  • the target image is read from the expanded map list and added to the key frame image library.
  • the relocation device of the camera attitude tracking process provided by the above embodiment is only exemplified by the division of the above functional modules when implementing relocation. In actual applications, the functions may be assigned differently according to needs.
  • the function module is completed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.
  • the relocation device and the relocation method embodiment are provided in the same concept, and the specific implementation process is described in detail in the method embodiment, and details are not described herein again.
  • FIG. 16 is a block diagram showing the structure of an electronic device 1600 provided by an exemplary embodiment of the present application.
  • the electronic device 1600 can be: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a dynamic image expert compression standard Audio level 4) Player, laptop or desktop computer.
  • Electronic device 1600 may also be referred to as a user device, a portable electronic device, a laptop electronic device, a desktop electronic device, and the like.
  • the electronic device 1600 includes a processor 1601 and a memory 1602.
  • the processor 1601 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like.
  • the processor 1601 may be configured by at least one of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), and a PLA (Programmable Logic Array). achieve.
  • the processor 1601 may also include a main processor and a coprocessor.
  • the main processor is a processor for processing data in an awake state, which is also called a CPU (Central Processing Unit); the coprocessor is A low-power processor for processing data in standby.
  • the processor 1601 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and rendering of the content that the display needs to display.
  • the processor 1601 may further include an AI (Artificial Intelligence) processor for processing computational operations related to machine learning.
  • AI Artificial Intelligence
  • Memory 1602 can include one or more computer readable storage media, which can be non-transitory.
  • the memory 1602 can also include high speed random access memory, as well as non-volatile memory such as one or more disk storage devices, flash storage devices.
  • the non-transitory computer readable storage medium in memory 1602 is for storing at least one instruction for execution by processor 1601 to implement the camera pose provided by the method embodiments of the present application. The method of relocating the tracking process.
  • the electronic device 1600 optionally further includes: a peripheral device interface 1603 and at least one peripheral device.
  • the processor 1601, the memory 1602, and the peripheral device interface 1603 may be connected by a bus or a signal line.
  • Each peripheral device can be connected to the peripheral device interface 1603 via a bus, signal line or circuit board.
  • the peripheral device includes at least one of a radio frequency circuit 1604, a touch display screen 1605, a camera 1606, an audio circuit 1607, a positioning component 1608, and a power source 1609.
  • Peripheral device interface 1603 can be used to connect at least one peripheral device associated with I/O (Input/Output) to processor 1601 and memory 1602.
  • the processor 1601, the memory 1602, and the peripheral device interface 1603 are integrated on the same chip or circuit board; in some other embodiments, any one of the processor 1601, the memory 1602, and the peripheral device interface 1603 or The two can be implemented on a separate chip or circuit board, which is not limited in this embodiment.
  • the RF circuit 1604 is configured to receive and transmit an RF (Radio Frequency) signal, also referred to as an electromagnetic signal.
  • Radio frequency circuit 1604 communicates with the communication network and other communication devices via electromagnetic signals.
  • the RF circuit 1604 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal.
  • the radio frequency circuit 1604 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and the like.
  • Radio frequency circuitry 1604 can communicate with other electronic devices via at least one wireless communication protocol.
  • the wireless communication protocols include, but are not limited to, the World Wide Web, a metropolitan area network, an intranet, generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and/or WiFi (Wireless Fidelity) networks.
  • the RF circuit 1604 may also include NFC (Near Field Communication) related circuitry, which is not limited in this application.
  • the display 1605 is used to display a UI (User Interface).
  • the UI can include graphics, text, icons, video, and any combination thereof.
  • the display 1605 also has the ability to capture touch signals over the surface or surface of the display 1605.
  • the touch signal can be input to the processor 1601 as a control signal for processing.
  • the display 1605 can also be used to provide virtual buttons and/or virtual keyboards, also referred to as soft buttons and/or soft keyboards.
  • the display screen 1605 can be one, and the front panel of the electronic device 1600 is disposed; in other embodiments, the display screen 1605 can be at least two, respectively disposed on different surfaces of the electronic device 1600 or in a folded design.
  • display screen 1605 can be a flexible display screen disposed on a curved surface or folded surface of electronic device 1600. Even the display screen 1605 can be set to a non-rectangular irregular pattern, that is, a profiled screen.
  • the display 1605 can be made of a material such as an LCD (Liquid Crystal Display) or an OLED (Organic Light-Emitting Diode).
  • Camera component 1606 is used to capture images or video.
  • camera assembly 1606 includes a front camera and a rear camera.
  • the front camera is placed on the front panel of the electronic device and the rear camera is placed on the back of the electronic device.
  • the rear camera is at least two, which are respectively a main camera, a depth camera, a wide-angle camera, and a telephoto camera, so as to realize the background blur function of the main camera and the depth camera, and the main camera Combine with a wide-angle camera for panoramic shooting and VR (Virtual Reality) shooting or other integrated shooting functions.
  • camera assembly 1606 can also include a flash.
  • the flash can be a monochrome temperature flash or a two-color temperature flash.
  • the two-color temperature flash is a combination of a warm flash and a cool flash that can be used for light compensation at different color temperatures.
  • the audio circuit 1607 can include a microphone and a speaker.
  • the microphone is used to collect sound waves of the user and the environment, and convert the sound waves into electrical signals for processing into the processor 1601 for processing, or input to the RF circuit 1604 for voice communication.
  • the microphones may be multiple, and are respectively disposed at different parts of the electronic device 1600.
  • the microphone can also be an array microphone or an omnidirectional acquisition microphone.
  • the speaker is then used to convert electrical signals from the processor 1601 or the RF circuit 1604 into sound waves.
  • the speaker can be a conventional film speaker or a piezoelectric ceramic speaker.
  • the audio circuit 1607 can also include a headphone jack.
  • the location component 1608 is used to locate the current geographic location of the electronic device 1600 to implement navigation or LBS (Location Based Service).
  • the positioning component 1608 can be a positioning component based on a US-based GPS (Global Positioning System), a Chinese Beidou system, or a Russian Galileo system.
  • a power supply 1609 is used to power various components in the electronic device 1600.
  • the power source 1609 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery.
  • the rechargeable battery may be a wired rechargeable battery or a wireless rechargeable battery.
  • a wired rechargeable battery is a battery that is charged by a wired line
  • a wireless rechargeable battery is a battery that is charged by a wireless coil.
  • the rechargeable battery can also be used to support fast charging technology.
  • electronic device 1600 also includes one or more sensors 1610.
  • the one or more sensors 1610 include, but are not limited to, an acceleration sensor 1611, a gyro sensor 1612, a pressure sensor 1613, a fingerprint sensor 1614, an optical sensor 1615, and a proximity sensor 1616.
  • the acceleration sensor 1611 can detect the magnitude of the acceleration on the three coordinate axes of the coordinate system established by the electronic device 1600.
  • the acceleration sensor 1611 can be used to detect components of gravity acceleration on three coordinate axes.
  • the processor 1601 can control the touch display screen 1605 to display the user interface in a landscape view or a portrait view according to the gravity acceleration signal collected by the acceleration sensor 1611.
  • the acceleration sensor 1611 can also be used for the acquisition of game or user motion data.
  • the gyro sensor 1612 can detect the body direction and the rotation angle of the electronic device 1600, and the gyro sensor 1612 can cooperate with the acceleration sensor 1611 to collect the 3D motion of the user on the electronic device 1600. Based on the data collected by the gyro sensor 1612, the processor 1601 can implement functions such as motion sensing (such as changing the UI according to the user's tilting operation), image stabilization at the time of shooting, game control, and inertial navigation.
  • functions such as motion sensing (such as changing the UI according to the user's tilting operation), image stabilization at the time of shooting, game control, and inertial navigation.
  • the pressure sensor 1613 can be disposed on a side border of the electronic device 1600 and/or a lower layer of the touch display screen 1605.
  • the pressure sensor 1613 When the pressure sensor 1613 is disposed on the side frame of the electronic device 1600, the user's holding signal to the electronic device 1600 can be detected, and the processor 1601 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 1613.
  • the operability control on the UI interface is controlled by the processor 1601 according to the user's pressure operation on the touch display screen 1605.
  • the operability control includes at least one of a button control, a scroll bar control, an icon control, and a menu control.
  • the fingerprint sensor 1614 is configured to collect the fingerprint of the user, and the processor 1601 identifies the identity of the user according to the fingerprint collected by the fingerprint sensor 1614, or the fingerprint sensor 1614 identifies the identity of the user according to the collected fingerprint. Upon identifying that the user's identity is a trusted identity, the processor 1601 authorizes the user to perform related sensitive operations including unlocking the screen, viewing encrypted information, downloading software, paying and changing settings, and the like.
  • Fingerprint sensor 1614 can be provided with the front, back or side of electronic device 1600. When a physical button or vendor logo is provided on the electronic device 1600, the fingerprint sensor 1614 can be integrated with a physical button or vendor logo.
  • Optical sensor 1615 is used to collect ambient light intensity.
  • the processor 1601 can control the display brightness of the touch display 1605 based on the ambient light intensity acquired by the optical sensor 1615. Specifically, when the ambient light intensity is high, the display brightness of the touch display screen 1605 is raised; when the ambient light intensity is low, the display brightness of the touch display screen 1605 is lowered.
  • the processor 1601 can also dynamically adjust the shooting parameters of the camera assembly 1606 based on the ambient light intensity acquired by the optical sensor 1615.
  • Proximity sensor 1616 also referred to as a distance sensor, is typically disposed on the front panel of electronic device 1600. Proximity sensor 1616 is used to capture the distance between the user and the front of electronic device 1600. In one embodiment, when the proximity sensor 1616 detects that the distance between the user and the front side of the electronic device 1600 is gradually decreasing, the processor 1601 controls the touch display screen 1605 to switch from the bright screen state to the interest screen state; when the proximity sensor 1616 When it is detected that the distance between the user and the front side of the electronic device 1600 gradually becomes larger, the processor 1601 controls the touch display screen 1605 to switch from the state of the screen to the bright state.
  • FIG. 16 does not constitute a limitation to electronic device 1600, may include more or fewer components than illustrated, or may combine certain components, or employ different component arrangements.
  • the application further provides a computer readable storage medium, where the storage medium stores at least one instruction, at least one program, a code set or a set of instructions, the at least one instruction, the at least one program, the code set or The set of instructions is loaded and executed by the processor to implement the method embodiments described above to provide a relocation method in a camera pose tracking process.
  • the present application also provides a computer program product that, when run on an electronic device, causes the electronic device to perform the relocation method in the camera pose tracking process described in the various method embodiments above.
  • a person skilled in the art may understand that all or part of the steps of implementing the above embodiments may be completed by hardware, or may be instructed by a program to execute related hardware, and the program may be stored in a computer readable storage medium.
  • the storage medium mentioned may be a read only memory, a magnetic disk or an optical disk or the like.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Studio Devices (AREA)
  • Image Analysis (AREA)

Abstract

本申请公开了一种相机姿态追踪过程的重定位方法、装置、设备及存储介质，属于增强现实领域。所述方法包括：当目标图像符合重定位条件时，前端程序将目标图像切换为第i+1个标记图像，并将目标图像发送给后端程序；前端程序获取在目标图像后采集的当前图像，将当前图像相对于目标图像进行特征点追踪得到第一位姿参数；后端程序将目标图像进行重定位得到第二位姿参数，向前端程序发送第二位姿参数；前端程序根据第一位姿参数和第二位姿参数，计算得到当前图像的当前位姿参数。本申请能够达到即便后端程序对目标图像的重定位结果的时间较长，也能够实现对当前图像的定位结果的修正，从而保证较好的重定位效果。

Description

相机姿态追踪过程的重定位方法、装置、设备及存储介质
本申请要求于2018年04月27日提交的申请号为201810392872.5、发明名称为“相机姿态追踪过程的重定位方法、装置、设备及存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请实施例涉及增强现实领域，特别涉及一种相机姿态追踪过程的重定位方法、装置、设备及存储介质。
背景技术
在AR(Augmented Reality，增强现实)领域，搭载相机的主体需要在没有环境先验信息的情况下，于运动过程中追踪相机在现实环境中的运动姿态。
在相关技术中将相机的运动过程划分为至少两段追踪过程进行追踪，每段追踪过程对应各自的标记图像(Anchor)。通常将相机采集的第一帧图像作为第1个标记图像，在相机后续采集到当前图像时，设备对当前图像与第1个标记图像之间共同具有的特征点进行追踪，根据当前图像与第1个标记图像之间的特征点位置变化计算得到相机在现实世界中的相机姿态变化。当当前图像相对于第i个标记图像的追踪效果差于预设条件时，将当前图像的上一个图像切换为第i+1个标记图像，开启第i+1段追踪过程。其中，i为正整数。
发明内容
本申请实施例提供了一种相机姿态追踪过程的重定位方法、装置、设备及存储介质，可以用于解决重定位过程耗时较高，不仅可能会阻塞主线程，还可能会在若干帧图像以后才得到重定位结果的问题。所述技术方案如下：
根据本申请的一个方面，提供了一种相机姿态追踪过程的重定位方法，应用于运行有前端程序和后端程序的设备中，所述前端程序用于按序执行多个标记图像的相机姿态追踪过程，所述方法包括：
所述前端程序获取所述多个标记图像中第i个标记图像后采集的目标图像，i为正整数；
当所述目标图像符合重定位条件时，所述前端程序将所述目标图像切换为第i+1个标记图像，并将所述目标图像发送给所述后端程序；
所述前端程序获取在所述目标图像后采集的当前图像，将所述当前图像相对于所述目标图像进行特征点追踪得到第一位姿参数，所述第一位姿参数用于表示从所述目标图像至所述当前图像之间的相机姿态变化量；
所述后端程序将所述目标图像进行重定位得到第二位姿参数，向所述前端程序发送所述第二位姿参数，所述第二位姿参数用于表示从第一个标记图像至所述目标图像之间的相机姿态变化量；
所述前端程序根据所述第一位姿参数和所述第二位姿参数，计算得到所述当前图像的当前位姿参数。
根据本申请的另一方面，提供了一种相机姿态追踪过程的重定位装置，其特征在于，所 述装置包括：前端程序模块和后端程序模块，所述前端程序模块用于按序执行多个标记图像的相机姿态追踪过程；
所述前端程序模块，还用于获取所述多个标记图像中第i个标记图像后采集的目标图像，i为正整数；
所述前端程序模块，还用于当所述目标图像符合重定位条件时，将所述目标图像切换为第i+1个标记图像，并将所述目标图像发送给所述后端程序模块；
所述前端程序模块，还用于获取在所述目标图像后采集的当前图像，将所述当前图像相对于所述目标图像进行特征点追踪得到第一位姿参数，所述第一位姿参数用于表示从所述目标图像至所述当前图像之间的相机姿态变化量；
所述后端程序模块，用于将所述目标图像进行重定位得到第二位姿参数，向所述前端程序发送所述第二位姿参数，所述第二位姿参数用于表示从第一个标记图像至所述目标图像之间的相机姿态变化量；
所述前端程序模块，用于根据所述第一位姿参数和所述第二位姿参数，计算得到所述当前图像的当前位姿参数。
根据本申请的另一方面，提供了一种电子设备，所述电子设备包括存储器和处理器；
所述存储器中存储有至少一条指令，所述至少一条指令由所述处理器加载并执行以实现如上所述的重定位方法。
根据本申请的另一方面，提供了一种计算机可读存储介质，所述存储介质中存储有至少一条指令，所述至少一条指令由处理器加载并执行以实现如上所述的重定位方法。
本申请实施例提供的技术方案带来的有益效果至少包括：
通过前端程序将目标图像切换为第i+1个标记图像，在后端程序对目标图像进行重定位的过程中，由前端程序并行进行基于第i+1个标记图像的相机姿态追踪得到第一位姿参数，若在前端程序处理当前图像时得到重定位结果(第二位姿参数)，则使用第一位姿参数和第二位姿参数修正当前图像的定位结果，从而解决了后端程序无法控制重定位结果成功的时间，则得到的好几帧图像之前的重定位结果不能直接用于当前图像的重定位的问题；达到了即便后端程序的重定位结果的时间较长，也能够通过叠加两个位姿变化量，来实现对当前图像的定位结果的修正，从而保证较好的重定位效果。
附图说明
为了更清楚地说明本申请实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1是本申请一个示例性实施例提供的AR应用场景的场景示意图；
图2是本申请一个示例性实施例提供的AR应用场景的场景示意图；
图3是本申请一个示例性实施例提供的Anchor-Switching AR System算法的原理示意图；
图4是本申请一个示例性实施例提供的电子设备的结构框图；
图5是本申请一个示例性实施例提供的重定位过程的时序示意图；
图6是本申请一个示例性实施例提供的相机姿态追踪过程的重定位方法的流程图；
图7是本申请一个示例性实施例提供的正常追踪过程的计算原理图；
图8是本申请一个示例性实施例提供的重定位过程的计算原理图；
图9是本申请一个示例性实施例提供的相机姿态追踪过程的重定位方法的流程图；
图10是本申请一个示例性实施例提供的第一个标记图像、目标图像和一阶关键帧图像的对比示意图；
图11是本申请一个示例性实施例提供的相机姿态追踪过程的重定位方法的流程图；
图12是本申请一个示例性实施例提供的重定位方法的原理示意图；
图13是本申请一个示例性实施例提供的后端程序的工作原理示意图；
图14是本申请一个示例性实施例提供的前端程序的工作原理示意图；
图15是本申请一个示例性实施例提供的相机姿态追踪过程的重定位装置的框图；
图16是本申请一个示例性实施例提供的电子设备的框图。
具体实施方式
为使本申请的目的、技术方案和优点更加清楚，下面将结合附图对本申请实施方式作进一步地详细描述。
首先对本申请涉及的若干个名词进行简介:
AR：一种在相机采集图像的过程中，实时地计算相机在现实世界(或称三维世界、真实世界)中的相机姿态参数，根据该相机姿态参数在相机采集的图像上添加虚拟元素的技术。虚拟元素包括但不限于：图像、视频和三维模型。AR技术的目标是在屏幕上把虚拟世界套接在现实世界上进行互动。该相机姿态参数包括旋转矩阵和位移向量，旋转矩阵用于表征相机在现实世界中发生的旋转角度，位移向量用于表征相机在现实世界中发生的位移距离。
例如，参见图1和参见图2，设备在相机拍摄到的图像中添加了一个虚拟人物形象。随着相机在现实世界中的运动，相机拍摄到的图像会发生变化，虚拟人物的拍摄方位也发生变化，模拟出了虚拟人物在图像中静止不动，而相机随着位置和姿态的变化同时拍摄图像和虚拟人物的效果，为用户呈现了一幅真实立体的画面。
Anchor-Switching AR System：是基于连接多个标记图像(Anchor)的相机姿态追踪来确定在自然场景下的相机姿态参数，进而根据相机姿态参数在相机采集的图像上叠加虚拟世界的AR系统。
IMU(Inertial Measurement Unit，惯性测量单元)：是用于测量物体的三轴姿态角(或角速率)以及加速度的装置。一般的，一个IMU包含了三个单轴的加速度计和三个单轴的陀螺，加速度计用于检测物体在三维坐标系中每个坐标轴上的加速度信号，进而计算得到位移向量(可能会跟随时间的增加累积误差)；而陀螺用于检测物体在三维坐标系中的旋转矩阵。可选地，IMU包括陀螺仪、加速度计和地磁传感器。
示意性的，三维坐标系的建立方式为：1、X轴使用向量积Y*Z来定义，在X轴在设备当前的位置上，沿与地面相切的方向指向东方；2、Y轴在设备当前的位置上，沿与地面相切的方向指向地磁场的北极；3、Z轴指向天空并垂直于地面。
重定位：在基于第i个标记图像的相机姿态追踪过程中，若目标图像相对于第i个标记图像进行特征点追踪失败，无法顺利进行相机姿态追踪过程时，将目标图像相对于其它图像(第1个标记图像或其它有代表性的关键帧图像)重新进行特征点追踪，以便在特征点追踪成功时，根据重新得到的追踪过程实现相机采集对目标图像时的相机姿态进行定位的技术。
本申请提供了一种适用于Anchor-Switching AR System算法的重定位方法。Anchor-Switching AR System算法在确定相机姿态的过程中，将相机的运动过程划分为至少两段追踪过程进行追踪，每段追踪过程对应各自的标记图像。具体地，当第i个标记图像对应的追踪过程中，当当前图像相对于第i个标记图像的追踪效果差于预设条件(比如能够匹配到的特征点少于预设阈值)时，将当前图像的上一个图像确定为第i+1个标记图像，开启第i+1段追踪过程。其中，i为正整数。示意性的参考图3，其示出了本申请一个示例性实施例提供的Anchor-Switching AR System算法的原理示意图。在现实世界中存在物体320，设置有相机的设备340被用户手持进行移动，在移动过程中拍摄得到包括物体320的多帧图像1-6。设备将图像1确定为第1个标记图像(born-anchor或born-image)并记录初始位姿参数，该初始位姿参数可以是IMU采集的相机采集图像1时的位姿参数。然后将图像2相对于图像1进行特征点追踪，根据初始位姿参数和特征点追踪结果计算出相机在拍摄图像2时的位姿参数；将图像3相对于图像1进行特征点追踪，根据初始位姿参数和特征点追踪结果计算出相机在拍摄图像3时的位姿参数；将图像4相对于图像1进行特征点追踪，根据初始位姿参数和特征点追踪结果计算出相机在拍摄图像4时的位姿参数。
然后，将图像5相对于图像1进行特征点追踪，如果特征点追踪效果差于预设条件(比如匹配的特征点数量较少)，则将图像4确定为第2个标记图像，将图像5相对于图像4进行特征点追踪，计算出相机在拍摄图像4至图像5之间的位移变化量，再结合相机在拍摄图像4至图像1之间的位移变化量以及初始位姿参数，计算出相机在拍摄图像5时的位姿参数。然后再将图像6相对于图像4进行特征点追踪，以此类推，若当前图像的特征点追踪效果变差时，即可将当前图像的上一帧图像确定为新的标记图像，切换新的标记图像后重新进行特征点追踪。
可选地，特征点追踪可以采用基于视觉里程计原理的算法，比如特征点法或直接法。但是若相机在追踪过程中处于发生较为剧烈的运动、朝向强光源、朝向白色墙壁等各种异常场景时，上述Anchor-Switching AR System追踪过程可能会发生丢失(Lost)现象。丢失现象是指在目标图像中无法匹配到足够多的特征点，导致特征点追踪失败。此时，需要对目标图像进行重定位。但重定位过程耗时较高，不仅可能会阻塞主线程，还可能会在若干帧图像以后才得到重定位结果。
参考图4，其示出了本申请一个示例性实施例提供的电子设备的结构框图。该电子设备包括：处理器420、存储器440、相机460和IMU 480。
处理器420包括一个或多个处理核心，比如4核心处理器、8核心处理器等。处理器420用于执行存储器440中存储的指令、代码、代码片段和程序中的至少一种。
处理器420与存储器440电性相连。可选地，处理器420通过总线与存储器440相连。存储器440存储有一个或多个指令、代码、代码片段和/或程序。该指令、代码、代码片段和/或程序在被处理器420执行时，用于实现如下实施例中提供的相机姿态追踪过程的重定位方法。
处理器420还与相机460电性相连。可选地，处理器420通过总线与相机460相连。相机460是具有图像采集能力的传感器件。相机460还可称为摄像头、感光器件等其它名称。相机460具有连续采集图像或多次采集图像的能力。可选地，相机460设置在设备内部或设备外部。
处理器420还与IMU480电性相连。可选地，IMU480用于每隔预定时间间隔采集相机的位姿向量，并记录每组位姿参数在采集时的时间戳。相机的位姿参数包括：位移向量和旋转矩阵。其中，IMU480采集的旋转矩阵相对准确，采集的位移向量受实际环境可能会有较大的误差。
本申请实施例提供了“前端程序+后端程序”的重定位架构。其中，前端程序运行在主线程中，后端程序运行在另一线程中。前端程序用于按序执行多个标记图像的相机姿态追踪过程；后端程序用于在目标图像满足重定位条件时，对目标图像进行重定位。前端程序和后端程序之间通过结果模块(Result module)实现进程间的通信。在实现本申请的过程中，发明人发现若将重定位过程放在前端程序中执行，由于重定位过程的时间通常较长，可能会出现阻塞前端程序的正常运行；但若将重定位过程放在后端程序中执行，由于后端程序无法控制重定位结果成功的时间，则得到的可能是好几帧图像之前的重定位结果，不能直接用于当前图像的重定位。如图5所示，假设在前端程序处理图像帧m(下文实施例中称为目标图像)时，图像帧m发生了丢失现象，后端程序开始对图像帧m进行重定位。后端程序得到图像帧m的重定位结果时，也即重定位结束时，前端程序已经开始处理图像帧m+k了，图像帧m的重定位结果无法直接作为修正图像帧m+k的结果。为此，本申请提供有如下实施例。
请参考图6，其示出了本申请一个实施例提供的相机姿态追踪过程的重定位方法的流程图。本实施例以该重定位方法应用于图4所示的设备中来举例说明。该方法包括：
步骤601，前端程序获取多个标记图像中第i个标记图像后采集的目标图像，i为正整数；
设备内的相机按照预设时间间隔采集一帧帧图像，形成图像序列。可选地，相机是在运动(平移和/或旋转)过程中，按照预设时间间隔采集一帧帧图像形成图像序列。
可选地，前端程序将图像序列中的第一帧图像(或前几帧图像中符合预定条件的一帧图像)确定为第一个标记图像，将后续采集的图像相对于第一个标记图像进行特征点追踪，并根据特征点追踪结果计算相机的相机姿态参数；若当前图像的特征点追踪效果差于预设条件(比如匹配到的特征点数量较少)时，将当前图像的上一帧图像确定为第二个标记图像，将后续采集的图像相对于第二个标记图像进行特征点追踪，并根据特征点追踪结果计算相机的相机姿态参数，以此类推。前端程序可以按序进行连续多个标记图像的相机姿态追踪。
步骤602，当目标图像符合重定位条件时，前端程序将目标图像切换为第i+1个标记图像；
设备会确定当前图像是否符合重定位条件。重定位条件用于指示当前图像相对于第i个标记图像的追踪过程失败，或者，重定位条件用于指示历史追踪过程中的累积误差已经高于预设条件。
在一个可选的实施例中，设备对当前图像相对于第i个标记图像进行追踪，若当前图像中不存在与第i个标记图像匹配的特征点，或者，当前图像中与第i个标记图像匹配的特征点少于第一数量时，确定当前图像相对于第i个标记图像的追踪过程失败，符合重定位条件。
在另一个可选的实施例中，设备确定当前图像与上一次重定位的图像之间的帧数大于第二数量时，确定历史追踪过程中的累积误差已经高于预设条件，或者，设备确定第i个标记图像和第一个标记图像之间的标记图像数量大于第三数量时，确定历史追踪过程中的累计误差已经高于预设条件。
本实施例对重定位条件的具体条件内容不加以限定。
若前端程序确定目标图像满足重定位条件，则前端程序将目标图像切换为第i+1个标记图像，并将IMU采集的相机位姿参数R old和T old(可能存在误差)暂时作为第i+1个标记图像的参考位姿参数。其中，R old用于表示相机从第1个标记图像至第i+1个标记图像(目标图像)之间的旋转矩阵，T old用于表示相机从第1个标记图像至第i+1个标记图像(目标图像)之间的位移向量。
步骤603，前端程序将目标图像发送给后端程序；
可选地，前端程序利用结果模块将目标图像发送给后端程序。该结果模块是前端程序和后端程序共享的存储空间。后端程序用于对目标图像相对于其它图像进行重定位。其它图像包括：第1个标记图像，和/或，曾经重定位成功且具有代表性的关键帧图像。
步骤604，前端程序获取在目标图像后采集的当前图像，将当前图像相对于目标图像进行特征点追踪得到第一位姿参数；
在将目标图像提供给后端程序后，前端程序获取第i+1个标记图像(目标图像)后采集的当前图像，将当前图像相对于目标图像进行特征点追踪，得到第一位姿参数。
其中，第一位姿参数用于表示从目标图像至当前图像之间的相机姿态变化量。第一位姿参数包括第一旋转矩阵R ca和第一位移向量T ca，R ca用于表示相机从第i+1个标记图像(目标图像)至当前图像之间的旋转矩阵，T ca用于表示相机从第i+1个标记图像(目标图像)至当前图像之间的位移向量。
步骤605，后端程序将目标图像进行重定位得到第二位姿参数；
与步骤604并列的，后端程序将目标图像相对于其它图像进行重定位得到第二位姿参数。
其中，第二位姿参数用于表示从第一个标记图像至目标图像之间的相机姿态变化量。第二位姿参数包括第二旋转矩阵R relocalize和第二位移向量T relocalice。R relocalice用于表示相机从第1个标记图像至第i+1个标记图像(目标图像)的旋转矩阵，是通过重定位得到的对上述R old的修正值；T relocalice用于表示相机从第1个标记图像至第i+1个标记图像(目标图像)之间的位移向量，是通过重定位得到的对上述T old的修正值。
步骤606，后端程序向前端程序发送第二位姿参数；
可选地，当后端程序对目标图像进行重定位过程后，将重定位得到的第二位姿参数，通过结果模块发送至第二位姿参数。
步骤607，前端程序根据第一位姿参数和第二位姿参数，计算得到当前图像的当前位姿参数。
可选地，前端程序按照如下公式计算得到当前位姿参数R m+k和T m+k如下：
Figure PCTCN2019079781-appb-000001
其中，第一位姿参数包括第一旋转矩阵R ca和第一位移向量T ca，所述第二位姿参数包括第二旋转矩阵R relocalize和第二位移向量T relocalice，s为目标图像对应的尺度。
可选地，s为预设值，或者，s为前端程序将第一个标记图像上的初始特征点映射至目标图像上得到投影特征点，根据投影特征点的平均深度计算得到所述目标图像对应的尺度。
综上所述，本实施例提供的重定位方法，通过前端程序将目标图像切换为第i+1个标记图像，在后端程序对目标图像进行重定位的过程中，由前端程序并行进行基于第i+1个标记图像的相机姿态追踪得到第一位姿参数，若在前端程序处理当前图像时得到重定位结果(第二位姿参数)，则使用第一位姿参数和第二位姿参数修正当前图像的定位结果，从而解决了后 端程序无法控制重定位结果成功的时间，则得到的好几帧图像之前的重定位结果不能直接用于当前图像的重定位的问题；达到了即便后端程序的重定位结果的时间较长，也能够通过叠加两个位姿变化量，来实现对当前图像的定位结果的修正，从而保证较好的重定位效果。
为了从另一角度来阐述上述重定位原理，结合图7和图8来对正常追踪过程和重定位成功过程进行说明。
正常追踪过程(图7)：在基于第i个标记图像进行相机姿态追踪过程中，设第i个标记图像相对于第1个标记图像存在位姿变化量R _i_old和T _i_old，若图像帧m相对于第i个标记图像进行特征点追踪成功，得到位姿变化量R ca1和T ca1，则通过如下公式，计算得到图像帧m的相机位姿参数：
Figure PCTCN2019079781-appb-000002
其中，s i为所述第i个标记图像对应的尺度。
若图像帧m+k相对于第i个标记图像进行特征点追踪成功，得到位姿变化量R ca2和T ca2，则通过如下公式，计算得到图像帧m+k的相机位姿参数：
Figure PCTCN2019079781-appb-000003
重定位成功(图8)：在基于第i个标记图像进行相机姿态追踪过程中，目标图像发生了lost，则前端程序将目标图像切换为第i+1个标记图像，利用IMU采集到的参考位姿参数作为第i+1个标记图像相对于第1个标记图像的位姿变化量R old和T old，若后端程序对目标图像的重定位成功，得到重定位结果R relocalize和T relocalice，则通过如下公式，计算得到当前图像(图像帧m+k)的相机位姿参数：
Figure PCTCN2019079781-appb-000004
在基于图6所示的可选实施例中，后端程序将目标图像相对于第一个标记图像进行重定位，得到第二位姿参数。如图9所示，上述步骤605包括如下子步骤：
步骤605a，获取第一个标记图像的初始特征点和初始位姿参数，初始位姿参数用于指示相机采集第一个标记图像时的相机姿态；
后端程序尝试将当前图像相对于第一个标记图像进行特征点追踪。此时，设备获取缓存的第一个标记图像中的初始特征点以及初始位姿参数，该初始位姿参数用于指示相机采集第一个标记图像时的相机姿态。
步骤605b，将目标图像相对于第一个标记图像进行特征点追踪，得到与初始特征点匹配的目标特征点；
可选地，每组匹配特征点对中包括两个互相匹配的初始特征点和目标特征点。
特征点追踪可采用基于视觉里程计的追踪算法，本申请对此不加以限定。在一个实施例中，特征点追踪采用KLT(Kanade-Lucas)光流追踪算法；在另一个实施例中，特征点追踪采用基于SIFT(Scale-Invariant Feature Transform，尺度不变特征转换)算法提取的SIFT特征描述子、ORB(Oriented FAST and Rotated BRIEF，快速特征点提取和描述)算法提取的ORB特征描述子进行特征点跟踪。本申请对特征点追踪的具体算法不加以限定，特征点追踪过程可以采用特征点法或直接法。
在一个实施例中，后端程序对第一个标记图像进行特征点提取，得到N个初始特征点；设备还对目标图像进行特征点提取，得到M个候选特征点；然后将M个候选特征点逐一与N个初始特征点进行匹配，确定出至少一组匹配特征点对。每组匹配特征点对包括：一个初始特征点和一个目标特征点。初始特征点是第1个标记图像上的特征点，目标特征点是目标图像上与该初始特征点匹配度最高的候选特征点。
可选地，初始特征点的数量大于或等于目标特征点的数量。比如，初始特征点的数量是450个，目标特征点为320个。
步骤605c，根据初始特征点和目标特征点，计算相机从初始姿态参数改变至目标姿态参数时的第二位姿参数，目标姿态参数用于表征相机在采集目标图像时的相机姿态；
可选地，设备根据初始特征点和目标特征点计算两帧图像之间的单应性矩阵homography；对单应性矩阵homography进行分解，得到相机从初始姿态参数改变至目标姿态参数时的位姿变化量R relocalize和T relocalize。
单应性矩阵描述了两个平面之间的映射关系，若自然场景(现实环境)中的特征点都落在同一物理平面上，则可以通过单应性矩阵进行运动估计。当存在至少四对相匹配的初始特征点和目标特征点时，设备通过ransac对该单应性矩阵进行分解，得到旋转矩阵R relocalize和平移向量T relocalize。
其中，R relocalize是相机从初始姿态参数改变至目标姿态参数时的旋转矩阵，T relocalize是相机从初始姿态参数改变至目标姿态参数时的位移参数。
步骤605d，根据初始位姿参数和位姿变化量，重定位得到第二位姿参数。
设备将初始位姿参数利用位姿变化量进行变换后，重定位得到第二位姿参数R relocalize和T relocalice，从而计算得到相机在采集目标图像时的相机姿态。
图8所示出的重定位方法是将目标图像相当于第一个标记图像进行重定位，该方法在某些场景下会存在重定位失败的可能性。比如目标图像对应的拍摄姿态与第一个标记图像的拍摄姿态之间的差别较大时，直接将目标图像和第一个标记图像建立匹配来实现重定位会失败。示意性的，设图10的(1)图是第一个标记图像，图10的(2)图是目标图像，由于目标图像和第一个标记图像之间的重合区域太少，达不到足够的匹配度，因此直接将目标图像和第一个标记图像进行重定位会失败。本申请的可选实施例中，引入图10的(3)图作为一阶关键帧图像，将目标图像相对于一阶关键帧图像进行重定位，能够提高重定位的成功概率。
在基于图6的实施例中，后端程序将目标图像相对于第一个标记图像进行重定位，得到第二位姿参数。如图11所示，上述步骤605包括如下子步骤：
步骤6051，从关键帧图像库中选择出目标关键帧图像；
关键帧图像库中存储有至少一个关键帧图像的图像信息，关键帧图像是在历史相机姿态追踪过程中重定位成功且相对于第一个标记图像存在第一重定位结果的图像；
在相机姿态追踪过程中会存在不止一次重定位过程。设备会在重定位成功的图像中挑选一些有代表性的图像作为关键帧图像保存至关键帧图像库。关键帧图像库是用于存储关键帧图像的数据库。关键帧图像库中存储有多个关键帧图像的图像信息。该图像信息包括：关键帧图像的图像本身、关键帧图像中的特征点和关键帧图像对应的第一重定位结果。可选地，图像信息还包括：关键帧图像的第一全局描述子，第一全局描述子用于以简要的数据量表示关键帧图像的图像特征。或者，第一全局描述子用于以简要的数据量唯一性地表示关键帧图 像的图像特征。
当对目标图像进行重定位时，后端程序从关键帧图像库中挑选出目标关键帧图像。可选地，目标关键帧图像是在关键帧图像库中与目标图像的图像距离最接近的图像。在不同实施例中，图像距离能够采用图像相似度、图像特征相似度、相匹配的特征点之间的距离中的任意一种形式来表征，本申请实施例对此不加以限定。
在一个实施例中，设备从关键帧图像库中通过关键帧图像的第一全局描述子和目标图像的第二全局描述子之间的相似度，来挑选目标关键帧图像；在另一个实施例中，由于IMU能够采集到相机采集目标图像时的参考姿态参数(可能存在误差)，设备从关键帧图像库中通过关键帧图像的第一姿态参数和目标图像的参考姿态参数之间的相似度，来挑选目标关键帧图像。
步骤6052，将目标图像相对于目标关键帧图像进行重定位，得到第二重定位结果；
当设备在关键帧图像库中挑选出目标关键帧图像之后，将目标图像相对于目标关键帧图像进行重定位。可选地，重定位的过程包括如下步骤：
1、获取目标关键帧图像的关键帧特征点和第一重定位结果；
关键帧图像库中存储有每个关键帧图像上的关键帧特征点和第一相机姿态追踪结果。可选地，关键帧特征点采用SIFT特征描述子或ORB特征描述子来表征。第一重定位结果是相机采集关键帧图像时的相机姿态参数(简称关键帧姿态参数)，或者，第一重定位结果是相机采集关键帧图像时的相机姿态参数相对于相机采集第一个标记图像时的相机姿态参数的姿态变化量(旋转矩阵和位移向量)。
2、将目标图像相对于目标关键图像库进行特征点追踪，得到与关键帧特征点匹配的目标特征点；
可选地，每组匹配特征点对中包括两个互相匹配的关键帧特征点和目标特征点。
特征点追踪可采用基于视觉里程计的追踪算法，本申请对此不加以限定。在一个实施例中，特征点追踪采用KLT光流追踪算法；在另一个实施例中，特征点追踪采用基于SIFT算法提取的SIFT特征描述子、ORB算法提取的ORB特征描述子进行特征点跟踪。本申请对特征点追踪的具体算法不加以限定，特征点追踪过程可以采用特征点法或直接法。
在一个实施例中，由于目标关键帧图像是已经重定位成功的图像，设备在历史过程已经对目标关键帧图像进行特征点提取，得到N个关键帧特征点；设备还对目标图像进行特征点提取，得到M个候选特征点；然后将M个候选特征点逐一与N个初始特征点进行匹配，确定出至少一组匹配特征点对。每组匹配特征点对包括：一个关键帧特征点和一个目标特征点。关键帧特征点是目标关键帧图像上的特征点，目标特征点是目标图像上与该关键帧特征点匹配度最高的候选特征点。
可选地，关键帧特征点的数量大于或等于目标特征点的数量。比如，关键帧特征点的数量是480个，目标特征点为350个。
3、根据关键帧特征点和目标特征点，计算相机从关键帧姿态参数改变至第二姿态参数时的位姿变化量，第二姿态参数用于表征相机在采集目标图像时的相机姿态。
可选地，设备根据关键帧特征点和目标特征点计算两帧图像之间的单应性矩阵homography；对单应性矩阵homography进行分解，得到相机从关键帧姿态参数改变至第二姿态参数时的位姿变化量R cm和T cm，将位姿变化量R cm和T cm作为第二重定位结果。
其中，R cm是相机从关键帧姿态参数改变至第二姿态参数时的旋转矩阵，T cm是相机从关 键帧姿态参数改变至第二姿态参数时的位移向量。
步骤6053，根据第一重定位结果和第二重定位结果，计算得到相机采集目标图像时的第二姿态参数。
假设第一重定位结果包括R mf和T mf。R mf是相机从初始姿态参数改变至关键帧姿态参数时的旋转矩阵，T mf是相机从初始姿态参数改变至关键帧姿态参数时的位移向量，第二重定位结果包括R cm和T cm。R cm是相机从关键帧姿态参数改变至第二姿态参数时的旋转矩阵，T cm是相机从关键帧姿态参数改变至第二姿态参数时的位移向量。采用如下公式计算得到相机采集目标图像时的目标姿态参数：
Figure PCTCN2019079781-appb-000005
其中，S mf是目标关键帧图像的尺度。
需要说明的是，关键帧图像库包括：相对于第一个标记图像进行一阶重定位成功的一阶关键帧图像，和/或，相对关键帧图像库中的n-1阶关键帧图像进行n阶重定位成功的n阶关键帧图像，n为大于1的整数。上述仅以一阶重定位图像进行举例说明，对于n阶关键帧图像的第一重定位结果，可以将多个位姿变化量进行累乘得到，本申请对此不再赘述。
在基于图6的可选实施例中，前端程序将第一个标记图像上的初始特征点映射至目标图像上得到投影特征点，根据投影特征点的平均深度计算得到目标图像对应的尺度的过程如下：
1、获取第一个标记图像中的初始特征点的二维坐标；
前端程序预先提取和缓存有第一个标记图像中的初始特征点的二维坐标。该二维坐标采用齐次表示。
2、对初始特征点的二维坐标进行反投影，得到初始特征点在三维空间中的第一三维坐标X born；
前端程序通过如下公式将初始特征点的二维坐标变换至三维空间，得到这些初始特征点在三维空间中的第一三维坐标X born。
Figure PCTCN2019079781-appb-000006
其中，f x、f y、c x、c y为相机的内置参数。初始特征点的二维坐标x born是第一个标记图像上的layer-keyPoints的齐次表示，三维点X born是非齐次表示。假设第一个标记图像的初始深度d为1。
3、将第一三维坐标X born通过如下公式进行三维旋转平移，得到初始特征点在目标图像上对应的第二三维坐标X current；
X current＝R*X born+T；
其中，R是IMU采集的参考位姿变化量中的旋转矩阵，T是IMU采集的参考位姿变化量中的位移向量。
4、将第二三维坐标X current投影至目标图像，得到投影特征点在目标图像中的二维坐标；
前端程序通过如下公式将第二三维坐标X current投影至目标图像，得到投影特征点在目标图像中的二维坐标x current：
Figure PCTCN2019079781-appb-000007
其中，f x、f y、c x、c y为相机的内置参数。
5、根据投影特征点在目标图像中的平均深度，计算得到目标图像的尺度s。
在基于图6的可选实施例中，前端程序和后端程序之间通过结果模块(Result Module)，结果模块是前端程序和后端程序共享的存储空间。如图12所示，结果模块包括：重定位列表(Relocalization list)、结果标签(Result-flag)、结果存储位。可选地，结果模块还包括：扩展图列表(Expand-map list)、最近一次切换时间(Abandon Time)。
重定位列表用于存储需要进行重定位的目标图像。可选地，同一时间内重定位列表中最多有一张目标图像。当前端程序需要进行重定位时，将目标图像写入重定位列表。
结果标签用于存储后端程序的工作状态，包括：ready(准备)、have-result(得到结果)、still-sunning(正在计算)三种。Ready表示后端程序为已启动且空闲的状态；have-result表示后端程序对最近一次重定位已经计算完毕，存在重定位成功(OK)或重定位失败(Fail)两种情形；still-sunning表示正在计算且尚未计算完毕。
结果存储位用于在重定位成功时，存储第二位姿参数R relocalize和T relocalice。
扩展图列表用于存储关键帧图像库的候选图像。当前端程序在正常相机追踪过程时，将图像传入扩展图列表，供后端程序对关键帧图像库进行扩展。
最近一次切换时间用于存储前端程序最近一次切换标记图像时的时间。比如，前端程序将目标图像设置为第i+1个标记图像时，记录该第i+1个标记图像的切换时间至Abandon Time。
在一个结合图12的实施例里，后端程序是一个持续在后台运行的线程，后端程序与前台程序运行在不同的线程中，通过结果模块实现进程间的通信。后端程序通过while(1)循环运行图13中的步骤：
步骤1200，等待新图像；
后端程序监测结果模块中的重定位列表以及扩展图列表。
步骤1201，判断重定位列表为空？
当重定位列表为空时，确定前台程序没有需要重定位的目标图像，进入步骤1202；当重定位列表不为空时，确定前台程序存在需要重定位的目标图像，进入步骤1204。
步骤1202，判断扩展图列表是否为空？
当扩展图列表不为空，进入步骤1203；当扩展图列表为空时，回到步骤1200。
步骤1203，扩展关键帧图像库；
后端程序利用扩展图列表中的候选图像对关键帧图像库(keyframe database)中的关键帧图像进行扩展，使得关键帧图像库中的关键帧图像得到添加或更新。
步骤1204，设置结果标签为正在运行(still-running)；
后端程序将result flag设置为still-running，然后开始对目标图像进行重定位。
步骤1205，对需要进行重定位的目标图像进行重定位；
参考图9或图11的描述，本实施例不再赘述。可选地，后端程序对目标图像先相对于第一个标记图像进行重定位，重定位失败时再对目标图像相对于关键帧图像库进行重定位。或者，后端程序并行地对目标图像相对于第一个标记图像进行重定位，以及对目标图像相对于关键帧图像库进行重定位。
步骤1206，判断重定位过程是否成功？
当重定位成功时，进入步骤1207；当重定位失败时，进入步骤1208；
步骤1207，重定位成功时间是否早于最近一次切换时间；
最近一次切换时间，用于指示前端程序最近一次切换标记图像时的时间。
当重定位成功时间早于最近一次切换时间时，说明前台程序在将目标图像切换为第i+1个标记图像后，已经又切换了一次或多次标记图像，本次重定位结果为无效状态，进入步骤1208；当重定位成功时间晚于最近一次切换时间时，本次重定位结果为有效状态，进入步骤1209。
步骤1208，设置结果标签为失败；
后端程序将result flag设置为fail。
步骤1209，设置结果标签为成功。
后端程序将result flag设置为OK，并将第二位姿参数R relocalize和T relocalice写入结果模块中。
在一个结合图12的实施例里，前端程序运行在主线程中，负责依序进行多个标记图像的相机追踪过程，后端程序与前端程序运行在不同的线程中，通过结果模块实现进程间的通信。前端程序中设置有两个标签：flag1和flag2，flag1用于表示是否需要进行重定位，flag2用于表示重定位失败，需要再次切换marker。前端程序用于执行如下步骤，如图14：
步骤1300，基于多个标记图像的相机姿态追踪；
前端程序负责基于多个标记图像的相机姿态追踪过程。示意性的，假设前端程序正在进行第i个标记图像的相机姿态追踪过程。
步骤1301，是否存在追踪结果；
前端程序获取第i个标记图像之后采集的目标图像，在将目标图像相对于第i个标记图像进行特征点追踪后，判断是否存在追踪结果。若存在追踪结果，则进入步骤1302；若不存在追踪结果，则进入步骤1303。
步骤1302，设置flag1为假；
继续进行步骤1300的相机姿态追踪过程。
步骤1303，设置flag1为真；
前端程序设置flag1为真，表示需要对目标图像进行重定位。
步骤1304，检测flag1的状态；
前端程序还会在每帧图像处理时，检测flag1的状态；若flag1的状态为假，则执行步骤1305，前端程序将目标图像发送至扩展图列表，供后端程序在扩展关键帧图像库时使用；若flag1的状态为真，则进入步骤1306，前端程序认为正在处理的目标图像需要进行重定位，检测flag2的状态。
若flag2的状态为否，表示上一次重定位成功，进入步骤1307；若flag2的状态为是，表示最近一次重定位失败，进入步骤1313。
步骤1307，检测后端程序的运行状态是否为ready？
前端程序读取结果模块中的结果标签，若结果标签为ready，说明后端程序处于空闲态，可以对目标图像进行重定位。
步骤1308，切换第i+1个标记图像，发送最近一次修改时间，将目标图像发送至重定位列表；
前端程序将目标图像切换为第i+1个标记图像，并将切换时间写入结果模块中的abandoned time。
步骤1309，尝试获取后端程序的重定位结果；
前端程序读取结果模块中的结果标签，若结果标签为still-running，则进入步骤1310；若结果标签为have result_ok，则进入步骤1311；若结果标签为have result_fail，则进入步骤1312。
步骤1310，做其它事务；
在后端程序对目标图像进行重定位的计算过程中，前端程序可以对目标图像之后的当前图像进行特征点追踪过程。
步骤1311，设置flag1为失败；
当对目标图像的重定位成功时，设置flag1为失败，表示不需要进行重定位，进行基于第i+1个标记图像的相机追踪过程。
步骤1312，设置flag2为成功；
当对目标图像的重定位失败时，设置flag2为成功。
步骤1313，切换至第i+2个标记图像，发送最近一次修改时间。
当前端程序获取到目标图像之后的当前图像时，若flag1为真且flag2为真时，表示后端程序的最近一次重定位过程失败，比如对目标图像的重定位过程失败，前端程序需要将目标图像之后的当前图像发送至重定位列表，以便后端程序对当前图像启动重定位过程。此时，前端程序将当前图像切换为第i+2个标记图像，再次将切换时间写入结果模块中的abandoned time。
在基于图12的可选实施例中，后端程序利用扩展图列表中的候选图像对关键帧图像库中的关键帧图像进行扩展，包括如下步骤：
步骤1，获取最近一个重定位成功的候选图像；
在初始状态下，关键帧图像库可以为空，后端程序仅使用第一个标记图像作为重定位时的标记图像。随着整个相机追踪过程的执行，会出现越来越多与第一个标记图像(或已加入关键帧图像库的关键帧图像)进行重定位成功的图像，
后端程序在每次重定位成功时，将重定位成功的图像作为可能添加至关键帧图像库的候选图像。可选地，重定位成功包括：相对于第一个标记图像进行重定位成功，或者，相对于关键帧图像库中已有的关键帧图像进行重定位成功。
后端程序从候选图像中挑选出一部分图像作为关键帧图像，添加至关键帧图像库中。
步骤2，确定候选图像是否满足添加条件，添加条件包括：候选图像和第一个标记图像之间的第一距离大于第一阈值，和/或，候选图像与上一次加入的关键帧图像之间的第二距离大于第二阈值；
可选地，本次添加的关键帧图像需要与第一个标记图像存在一定距离，因为如果两个图像比较接近，那么与直接用第一个标记图像进行重定位的效果没有明显差别。
后端程序计算候选图像与第一个标记图像之间的第一距离，第一距离用于表示候选图像和第一个标记图像之间的图像相似度或者相机姿态相似度。本申请对第一距离的具体计算方式不加以限定，在一个可选的实施例中，对于第一个标记图像中的初始特征点，候选图像中存在与初始特征点匹配的目标特征点，一个初始特征点和对应的目标特征点构成一个匹配特征点对，根据每一组匹配特征点分别计算L2距离(对应L2范数的欧几里得距离)，将所有L2距离的平均值作为候选图像和第一个标记图像之间的第一距离。以第一阈值为50个像素点为例，当候选图像和第一个标记图像之间的第一距离大于50个像素点时，将候选图像添加至关键帧图像库。
可选地，本次添加的关键帧图像需要与上一次添加的关键帧图像存在一定距离，因为如果两个图像比较接近，那么与使用上一次添加的关键帧图像进行重定位的效果没有明显差别。
后端程序计算候选图像与上一次添加的关键帧图像之间的第二距离，第二距离用于表示候选图像和上一次添加的关键帧图像之间的图像相似度或相机姿态相似度。本申请对第二距离的具体计算方式不加以限定，在一个可选的实施例中，采用候选图像和上一次添加的关键帧图像之间的重定位图像个数作为第二距离，以第二阈值为10为例，若候选图像和上一次添加的关键帧图像之间的重定位图像个数超过10帧，则将候选图像添加至关键帧图像库中。
步骤3，当候选图像满足添加条件时，将候选图像加入关键帧图像库；
当确定候选图像满足添加条件时，将候选图像作为关键帧图像添加至关键帧图像库。可选地，将关键帧图像的第一全局描述子、关键帧特征点、第一重定位的定位结果存储至关键帧图像库。其中，第一全局描述子用于以简要的数据量表示关键帧图像的图像特征，第一重定位的定位结果可以使用相机采集关键帧图像时的关键帧姿态参数来表示，也可以采用相机从初始姿态参数改变至关键帧姿态参数时的姿态变化量来表示。
步骤4，当候选图像不满足添加条件时，将候选图像不加入关键帧图像库。
当候选图像与第一个标记图像之间的第一距离小于第一阈值，或者，候选图像与上一次添加的关键帧图像之间的第二距离小于第二阈值，则不将该候选图像加入关键帧图像库。
可选地，满足上述添加条件的关键帧图像可以用来扩展关键帧图像库，但是关键帧图像库中的关键帧图像不应当太多，太多的关键帧图像会造成重定位过程的计算量负担，导致难以搜索到理想的图像。在一个示意性的例子中，关键帧图像库中的关键帧图像不超过1000张。
当候选图像满足添加条件且关键帧图像库中的关键帧图像数量达到最大值(比如1000张)时，计算该候选图像和已有关键帧图像之间的相似度。将具有最大相似度的关键帧图像从关键帧图像库中删除，将候选图像加入关键帧图像库中。
在一个示意性的例子中，上述相机姿态追踪过程的重定位方法可以用于AR程序中，通过该重定位方法能够实时根据现实世界的场景信息，对电子设备上的相机姿态进行追踪，并根据追踪结果调整和修改AR应用程序中的AR元素的显示位置。以图1或图2所示的运行在手机上的AR程序为例，当需要显示一个站立在书籍上的静止卡通人物时，不论用户如何移动该手机，只需要根据该手机上的相机姿态变化修改该卡通人物的显示位置，即可使该卡通人物在书籍上的站立位置保持不变。
以下为本申请的装置实施例，对于装置实施例中未详细描述的细节，可以参考如上所述的方法实施例。
请参考图15，其示出了本申请一个示例性实施例提供的相机姿态追踪过程的重定位装置 的框图。该重定位装置可以通过软件、硬件或者两者的结合实现成为电子设备(或称移动电子设备)的全部或一部分，该重定位装置包括：前端程序模块1520和后端程序模块1540，所述前端程序模块1520用于按序执行多个标记图像的相机姿态追踪过程；
所述前端程序模块1520，还用于获取所述多个标记图像中第i个标记图像后采集的目标图像，i为正整数；
所述前端程序模块1520，还用于当所述目标图像符合重定位条件时，将所述目标图像切换为第i+1个标记图像，并将所述目标图像发送给所述后端程序模块；
所述前端程序模块1520，还用于获取在所述目标图像后采集的当前图像，将所述当前图像相对于所述目标图像进行特征点追踪得到第一位姿参数，所述第一位姿参数用于表示从所述目标图像至所述当前图像之间的相机姿态变化量；
所述后端程序模块1540，用于将所述目标图像进行重定位得到第二位姿参数，向所述前端程序发送所述第二位姿参数，所述第二位姿参数用于表示从第一个标记图像至所述目标图像之间的相机姿态变化量；
所述前端程序模块1520，用于根据所述第一位姿参数和所述第二位姿参数，计算得到所述当前图像的当前位姿参数。
在一个可选的实施例中，所述后端程序模块1540，用于将所述目标图像相对于所述第一个标记图像进行重定位，得到所述第二位姿参数；和/或，所述后端程序模块1540，用于将所述目标图像相对于关键帧图像库中的关键帧图像进行重定位，得到所述第二位姿参数；所述关键帧图像是重定位成功且相对于所述第一个标记图像存在重定位结果的图像。
在一个可选的实施例中，所述关键帧图像库包括：
相对于所述第一个标记图像进行一阶重定位成功的一阶关键帧图像；和/或，相对于所述关键帧图像库中的n-1阶关键帧图像进行n阶重定位成功的n阶关键帧图像，n为大于1的整数。
在一个可选的实施例中，所述前端程序模块1520，用于根据所述第一位姿参数和所述第二位姿参数，计算得到所述当前图像的当前位姿参数，包括：
按照如下公式计算得到所述当前位姿参数R n和T n如下：
Figure PCTCN2019079781-appb-000008
其中，所述第一位姿参数包括第一旋转矩阵R ca和第一位移向量T ca，所述第二位姿参数包括第二旋转矩阵R relocalize和第二位移向量T relocalice，s为所述目标图像对应的尺度。
在一个可选的实施例中，所述前端程序模块1520，用于将所述第一个标记图像上的初始特征点映射至所述目标图像上得到投影特征点，根据所述投影特征点的平均深度计算得到所述目标图像对应的尺度。
在一个可选的实施例中，所述装置还包括结果模块1560，所述结果模块1560是所述前端程序模块和所述后端程序模块共享的存储空间；
所述前端程序模块1520，用于将所述目标图像写入所述结果模块1560中的重定位图像列表；
所述后端程序模块1540，用于从所述重定位图像列表中读取所述目标图像。
在一个可选的实施例中，所述前端程序模块1520，用于检测所述结果模块1560中的结果状态位是否为准备状态；所述前端程序模块1520，用于当检测结果为所述结果状态位是准 备状态时，将所述目标图像写入所述结果模块1560中的重定位图像列表。
在一个可选的实施例中，所述后端程序模块1540，用于将所述第二位姿参数写入所述结果模块中的结果存储位，所述结果模块1560是所述前端程序模块1520和所述后端程序模块1540共享的存储空间；所述前端程序模块1520，用于从所述结果存储位中读取所述第二位姿参数。
在一个可选的实施例中，所述前端程序模块1520，用于将所述目标图像写入所述结果模块中的扩展图列表；所述后端程序模块1540，用于在所述目标图像重定位成功且符合预设条件时，从所述扩展图列表读取所述目标图像添加至关键帧图像库中。
需要说明的是：上述实施例提供的相机姿态追踪过程的重定位装置在实现重定位时，仅以上述各功能模块的划分进行举例说明，实际应用中，可以根据需要而将上述功能分配由不同的功能模块完成，即将设备的内部结构划分成不同的功能模块，以完成以上描述的全部或者部分功能。另外，上述实施例提供的重定位装置与重定位方法实施例属于同一构思，其具体实现过程详见方法实施例，这里不再赘述。
图16示出了本申请一个示例性实施例提供的电子设备1600的结构框图。该电子设备1600可以是：智能手机、平板电脑、MP3播放器(Moving Picture Experts Group Audio Layer III，动态影像专家压缩标准音频层面3)、MP4(Moving Picture Experts Group Audio Layer IV，动态影像专家压缩标准音频层面4)播放器、笔记本电脑或台式电脑。电子设备1600还可能被称为用户设备、便携式电子设备、膝上型电子设备、台式电子设备等其他名称。
通常，电子设备1600包括有：处理器1601和存储器1602。
处理器1601可以包括一个或多个处理核心，比如4核心处理器、8核心处理器等。处理器1601可以采用DSP(Digital Signal Processing，数字信号处理)、FPGA(Field－Programmable Gate Array，现场可编程门阵列)、PLA(Programmable Logic Array，可编程逻辑阵列)中的至少一种硬件形式来实现。处理器1601也可以包括主处理器和协处理器，主处理器是用于对在唤醒状态下的数据进行处理的处理器，也称CPU(Central Processing Unit，中央处理器)；协处理器是用于对在待机状态下的数据进行处理的低功耗处理器。在一些实施例中，处理器1601可以在集成有GPU(Graphics Processing Unit，图像处理器)，GPU用于负责显示屏所需要显示的内容的渲染和绘制。一些实施例中，处理器1601还可以包括AI(Artificial Intelligence，人工智能)处理器，该AI处理器用于处理有关机器学习的计算操作。
存储器1602可以包括一个或多个计算机可读存储介质，该计算机可读存储介质可以是非暂态的。存储器1602还可包括高速随机存取存储器，以及非易失性存储器，比如一个或多个磁盘存储设备、闪存存储设备。在一些实施例中，存储器1602中的非暂态的计算机可读存储介质用于存储至少一个指令，该至少一个指令用于被处理器1601所执行以实现本申请中方法实施例提供的相机姿态追踪过程的重定位方法。
在一些实施例中，电子设备1600还可选包括有：外围设备接口1603和至少一个外围设备。处理器1601、存储器1602和外围设备接口1603之间可以通过总线或信号线相连。各个外围设备可以通过总线、信号线或电路板与外围设备接口1603相连。具体地，外围设备包括：射频电路1604、触摸显示屏1605、摄像头1606、音频电路1607、定位组件1608和电源1609中的至少一种。
外围设备接口1603可被用于将I/O(Input/Output，输入/输出)相关的至少一个外围设备连接到处理器1601和存储器1602。在一些实施例中，处理器1601、存储器1602和外围设备接口1603被集成在同一芯片或电路板上；在一些其他实施例中，处理器1601、存储器1602和外围设备接口1603中的任意一个或两个可以在单独的芯片或电路板上实现，本实施例对此不加以限定。
射频电路1604用于接收和发射RF(Radio Frequency，射频)信号，也称电磁信号。射频电路1604通过电磁信号与通信网络以及其他通信设备进行通信。射频电路1604将电信号转换为电磁信号进行发送，或者，将接收到的电磁信号转换为电信号。可选地，射频电路1604包括：天线系统、RF收发器、一个或多个放大器、调谐器、振荡器、数字信号处理器、编解码芯片组、用户身份模块卡等等。射频电路1604可以通过至少一种无线通信协议来与其它电子设备进行通信。该无线通信协议包括但不限于：万维网、城域网、内联网、各代移动通信网络(2G、3G、4G及5G)、无线局域网和/或WiFi(Wireless Fidelity，无线保真)网络。在一些实施例中，射频电路1604还可以包括NFC(Near Field Communication，近距离无线通信)有关的电路，本申请对此不加以限定。
显示屏1605用于显示UI(User Interface，用户界面)。该UI可以包括图形、文本、图标、视频及其它们的任意组合。当显示屏1605是触摸显示屏时，显示屏1605还具有采集在显示屏1605的表面或表面上方的触摸信号的能力。该触摸信号可以作为控制信号输入至处理器1601进行处理。此时，显示屏1605还可以用于提供虚拟按钮和/或虚拟键盘，也称软按钮和/或软键盘。在一些实施例中，显示屏1605可以为一个，设置电子设备1600的前面板；在另一些实施例中，显示屏1605可以为至少两个，分别设置在电子设备1600的不同表面或呈折叠设计；在再一些实施例中，显示屏1605可以是柔性显示屏，设置在电子设备1600的弯曲表面上或折叠面上。甚至，显示屏1605还可以设置成非矩形的不规则图形，也即异形屏。显示屏1605可以采用LCD(Liquid Crystal Display，液晶显示屏)、OLED(Organic Light-Emitting Diode,有机发光二极管)等材质制备。
摄像头组件1606用于采集图像或视频。可选地，摄像头组件1606包括前置摄像头和后置摄像头。通常，前置摄像头设置在电子设备的前面板，后置摄像头设置在电子设备的背面。在一些实施例中，后置摄像头为至少两个，分别为主摄像头、景深摄像头、广角摄像头、长焦摄像头中的任意一种，以实现主摄像头和景深摄像头融合实现背景虚化功能、主摄像头和广角摄像头融合实现全景拍摄以及VR(Virtual Reality，虚拟现实)拍摄功能或者其它融合拍摄功能。在一些实施例中，摄像头组件1606还可以包括闪光灯。闪光灯可以是单色温闪光灯，也可以是双色温闪光灯。双色温闪光灯是指暖光闪光灯和冷光闪光灯的组合，可以用于不同色温下的光线补偿。
音频电路1607可以包括麦克风和扬声器。麦克风用于采集用户及环境的声波，并将声波转换为电信号输入至处理器1601进行处理，或者输入至射频电路1604以实现语音通信。出于立体声采集或降噪的目的，麦克风可以为多个，分别设置在电子设备1600的不同部位。麦克风还可以是阵列麦克风或全向采集型麦克风。扬声器则用于将来自处理器1601或射频电路1604的电信号转换为声波。扬声器可以是传统的薄膜扬声器，也可以是压电陶瓷扬声器。当扬声器是压电陶瓷扬声器时，不仅可以将电信号转换为人类可听见的声波，也可以将电信号转换为人类听不见的声波以进行测距等用途。在一些实施例中，音频电路1607还可以包括耳机插孔。
定位组件1608用于定位电子设备1600的当前地理位置，以实现导航或LBS(Location Based Service，基于位置的服务)。定位组件1608可以是基于美国的GPS(Global Positioning System，全球定位系统)、中国的北斗系统或俄罗斯的伽利略系统的定位组件。
电源1609用于为电子设备1600中的各个组件进行供电。电源1609可以是交流电、直流电、一次性电池或可充电电池。当电源1609包括可充电电池时，该可充电电池可以是有线充电电池或无线充电电池。有线充电电池是通过有线线路充电的电池，无线充电电池是通过无线线圈充电的电池。该可充电电池还可以用于支持快充技术。
在一些实施例中，电子设备1600还包括有一个或多个传感器1610。该一个或多个传感器1610包括但不限于：加速度传感器1611、陀螺仪传感器1612、压力传感器1613、指纹传感器1614、光学传感器1615以及接近传感器1616。
加速度传感器1611可以检测以电子设备1600建立的坐标系的三个坐标轴上的加速度大小。比如，加速度传感器1611可以用于检测重力加速度在三个坐标轴上的分量。处理器1601可以根据加速度传感器1611采集的重力加速度信号，控制触摸显示屏1605以横向视图或纵向视图进行用户界面的显示。加速度传感器1611还可以用于游戏或者用户的运动数据的采集。
陀螺仪传感器1612可以检测电子设备1600的机体方向及转动角度，陀螺仪传感器1612可以与加速度传感器1611协同采集用户对电子设备1600的3D动作。处理器1601根据陀螺仪传感器1612采集的数据，可以实现如下功能：动作感应(比如根据用户的倾斜操作来改变UI)、拍摄时的图像稳定、游戏控制以及惯性导航。
压力传感器1613可以设置在电子设备1600的侧边框和/或触摸显示屏1605的下层。当压力传感器1613设置在电子设备1600的侧边框时，可以检测用户对电子设备1600的握持信号，由处理器1601根据压力传感器1613采集的握持信号进行左右手识别或快捷操作。当压力传感器1613设置在触摸显示屏1605的下层时，由处理器1601根据用户对触摸显示屏1605的压力操作，实现对UI界面上的可操作性控件进行控制。可操作性控件包括按钮控件、滚动条控件、图标控件、菜单控件中的至少一种。
指纹传感器1614用于采集用户的指纹，由处理器1601根据指纹传感器1614采集到的指纹识别用户的身份，或者，由指纹传感器1614根据采集到的指纹识别用户的身份。在识别出用户的身份为可信身份时，由处理器1601授权该用户执行相关的敏感操作，该敏感操作包括解锁屏幕、查看加密信息、下载软件、支付及更改设置等。指纹传感器1614可以被设置电子设备1600的正面、背面或侧面。当电子设备1600上设置有物理按键或厂商Logo时，指纹传感器1614可以与物理按键或厂商Logo集成在一起。
光学传感器1615用于采集环境光强度。在一个实施例中，处理器1601可以根据光学传感器1615采集的环境光强度，控制触摸显示屏1605的显示亮度。具体地，当环境光强度较高时，调高触摸显示屏1605的显示亮度；当环境光强度较低时，调低触摸显示屏1605的显示亮度。在另一个实施例中，处理器1601还可以根据光学传感器1615采集的环境光强度，动态调整摄像头组件1606的拍摄参数。
接近传感器1616，也称距离传感器，通常设置在电子设备1600的前面板。接近传感器1616用于采集用户与电子设备1600的正面之间的距离。在一个实施例中，当接近传感器1616检测到用户与电子设备1600的正面之间的距离逐渐变小时，由处理器1601控制触摸显示屏1605从亮屏状态切换为息屏状态；当接近传感器1616检测到用户与电子设备1600的正面之 间的距离逐渐变大时，由处理器1601控制触摸显示屏1605从息屏状态切换为亮屏状态。
本领域技术人员可以理解，图16中示出的结构并不构成对电子设备1600的限定，可以包括比图示更多或更少的组件，或者组合某些组件，或者采用不同的组件布置。
本申请还提供一种计算机可读存储介质，所述存储介质中存储有至少一条指令、至少一段程序、代码集或指令集，所述至少一条指令、所述至少一段程序、所述代码集或指令集由所述处理器加载并执行以实现上述方法实施例提供相机姿态追踪过程中的重定位方法。
本申请还提供了一种计算机程序产品，当其在电子设备上运行时，使得电子设备执行上述各个方法实施例所述的相机姿态追踪过程中的重定位方法。
上述本申请实施例序号仅仅为了描述，不代表实施例的优劣。
本领域普通技术人员可以理解实现上述实施例的全部或部分步骤可以通过硬件来完成，也可以通过程序来指令相关的硬件完成，所述的程序可以存储于一种计算机可读存储介质中，上述提到的存储介质可以是只读存储器，磁盘或光盘等。
以上所述仅为本申请的较佳实施例，并不用以限制本申请，凡在本申请的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本发明的保护范围之内。

Claims (20)

  1. 一种相机姿态追踪过程的重定位方法，其特征在于，应用于运行有前端程序和后端程序的设备中，所述前端程序用于按序执行多个标记图像的相机姿态追踪过程，所述方法包括：
    所述前端程序获取所述多个标记图像中第i个标记图像后采集的目标图像，i为正整数；
    当所述目标图像符合重定位条件时，所述前端程序将所述目标图像切换为第i+1个标记图像，并将所述目标图像发送给所述后端程序；
    所述前端程序获取在所述目标图像后采集的当前图像，将所述当前图像相对于所述目标图像进行特征点追踪得到第一位姿参数，所述第一位姿参数用于表示从所述目标图像至所述当前图像之间的相机姿态变化量；
    所述后端程序将所述目标图像进行重定位得到第二位姿参数，向所述前端程序发送所述第二位姿参数，所述第二位姿参数用于表示从第一个标记图像至所述目标图像之间的相机姿态变化量；
    所述前端程序根据所述第一位姿参数和所述第二位姿参数，计算得到所述当前图像的当前位姿参数。
  2. 根据权利要求1所述的方法，其特征在于，所述后端程序将所述目标图像进行重定位得到第二位姿参数，包括：
    所述后端程序将所述目标图像相对于所述第一个标记图像进行重定位，得到所述第二位姿参数；
    和/或，
    所述后端程序将所述目标图像相对于关键帧图像库中的关键帧图像进行重定位，得到所述第二位姿参数；所述关键帧图像是重定位成功且相对于所述第一个标记图像存在重定位结果的图像。
  3. 根据权利要求2所述的方法，其特征在于，所述关键帧图像库包括：
    相对于所述第一个标记图像进行一阶重定位成功的一阶关键帧图像；
    和/或，
    相对于所述关键帧图像库中的n-1阶关键帧图像进行n阶重定位成功的n阶关键帧图像，n为大于1的整数。
  4. 根据权利要求1至3任一所述的方法，其特征在于，所述前端程序用于根据所述第一位姿参数和所述第二位姿参数，计算得到所述当前图像的当前位姿参数，包括：
    按照如下公式计算得到所述当前位姿参数R n和T n如下：
    Figure PCTCN2019079781-appb-100001
    其中，所述第一位姿参数包括第一旋转矩阵R ca和第一位移向量T ca，所述第二位姿参数包括第二旋转矩阵R relocalize和第二位移向量T relocalice，s为所述目标图像对应的尺度。
  5. 根据权利要求4所述的方法，其特征在于，所述方法还包括：
    所述前端程序将所述第一个标记图像上的初始特征点映射至所述目标图像上得到投影特征点，根据所述投影特征点的平均深度计算得到所述目标图像对应的尺度。
  6. 根据权利要求1至3任一所述的方法，其特征在于，所述前端程序将所述目标图像发送给所述后端程序，包括：
    所述前端程序将所述目标图像写入结果模块中的重定位图像列表，所述结果模块是所述前端程序和所述后端程序共享的存储空间；
    所述后端程序从所述重定位图像列表中读取所述目标图像。
  7. 根据权利要求6所述的方法，其特征在于，所述前端程序将所述目标图像写入结果模块中的重定位图像列表，包括：
    所述前端程序检测所述结果模块中的结果状态位是否为准备状态；
    所述前端程序在检测结果为所述结果状态位是准备状态时，将所述目标图像写入所述结果模块中的重定位图像列表。
  8. 根据权利要求6所述的方法，其特征在于，所述后端程序向所述前端程序发送所述第二位姿参数，包括：
    所述后端程序将所述第二位姿参数写入所述结果模块中的结果存储位，所述结果模块是所述前端程序和所述后端程序共享的存储空间；
    所述前端程序从所述结果存储位中读取所述第二位姿参数。
  9. 根据权利要求6所述的方法，其特征在于，所述方法还包括：
    所述前端程序将所述目标图像写入所述结果模块中的扩展图列表；
    所述后端程序在所述目标图像重定位成功且符合预设条件时，从所述扩展图列表读取所述目标图像添加至关键帧图像库中。
  10. 一种相机姿态追踪过程的重定位装置，其特征在于，所述装置包括：前端程序模块和后端程序模块，所述前端程序模块用于按序执行多个标记图像的相机姿态追踪过程；
    所述前端程序模块，还用于获取所述多个标记图像中第i个标记图像后采集的目标图像，i为正整数；
    所述前端程序模块，还用于当所述目标图像符合重定位条件时，将所述目标图像切换为第i+1个标记图像，并将所述目标图像发送给所述后端程序模块；
    所述前端程序模块，还用于获取在所述目标图像后采集的当前图像，将所述当前图像相对于所述目标图像进行特征点追踪得到第一位姿参数，所述第一位姿参数用于表示从所述目标图像至所述当前图像之间的相机姿态变化量；
    所述后端程序模块，用于将所述目标图像进行重定位得到第二位姿参数，向所述前端程序发送所述第二位姿参数，所述第二位姿参数用于表示从第一个标记图像至所述目标图像之间的相机姿态变化量；
    所述前端程序模块，用于根据所述第一位姿参数和所述第二位姿参数，计算得到所述当前图像的当前位姿参数。
  11. 根据权利要求10所述的装置，其特征在于，
    所述后端程序模块，用于将所述目标图像相对于所述第一个标记图像进行重定位，得到所述第二位姿参数；
    和/或，
    所述后端程序模块，用于将所述目标图像相对于关键帧图像库中的关键帧图像进行重定位，得到所述第二位姿参数；所述关键帧图像是重定位成功且相对于所述第一个标记图像存在重定位结果的图像。
  12. 根据权利要求11所述的装置，其特征在于，所述关键帧图像库包括：
    相对于所述第一个标记图像进行一阶重定位成功的一阶关键帧图像；
    和/或，
    相对于所述关键帧图像库中的n-1阶关键帧图像进行n阶重定位成功的n阶关键帧图像，n为大于1的整数。
  13. 根据权利要求10至12任一所述的装置，其特征在于，
    所述前端程序模块，用于根据所述第一位姿参数和所述第二位姿参数，计算得到所述当前图像的当前位姿参数，包括：
    按照如下公式计算得到所述当前位姿参数R n和T n如下：
    Figure PCTCN2019079781-appb-100002
    其中，所述第一位姿参数包括第一旋转矩阵R ca和第一位移向量T ca，所述第二位姿参数包括第二旋转矩阵R relocalize和第二位移向量T relocalice，s为所述目标图像对应的尺度。
  14. 根据权利要求13所述的装置，其特征在于，所述装置还包括：
    所述前端程序将所述第一个标记图像上的初始特征点映射至所述目标图像上得到投影特征点，根据所述投影特征点的平均深度计算得到所述目标图像对应的尺度。
  15. 根据权利要求10至12任一所述的装置，其特征在于，所述装置还包括结果模块，所述结果模块是所述前端程序模块和所述后端程序模块共享的存储空间；
    所述前端程序模块，用于将所述目标图像写入所述结果模块中的重定位图像列表；
    所述后端程序模块，用于从所述重定位图像列表中读取所述目标图像。
  16. 根据权利要求15所述的装置，其特征在于，
    所述前端程序模块，用于检测所述结果模块中的结果状态位是否为准备状态；
    所述前端程序模块，用于在检测结果为所述结果状态位是准备状态时，将所述目标图像写入所述结果模块中的重定位图像列表。
  17. 根据权利要求15所述的装置，其特征在于，
    所述后端程序模块，用于将所述第二位姿参数写入所述结果模块中的结果存储位，所述结果模块是所述前端程序和所述后端程序共享的存储空间；
    所述前端程序模块，用于从所述结果存储位中读取所述第二位姿参数。
  18. 根据权利要求15所述的装置，其特征在于，
    所述前端程序模块，用于将所述目标图像写入所述结果模块中的扩展图列表；
    所述后端程序模块，用于在所述目标图像重定位成功且符合预设条件时，从所述扩展图列表读取所述目标图像添加至关键帧图像库中。
  19. 一种电子设备，其特征在于，所述电子设备包括存储器和处理器；
    所述存储器中存储有至少一条指令，所述至少一条指令由所述处理器加载并执行以实现如权利要求1至9任一所述的重定位方法。
  20. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质中存储有至少一条指令，所述至少一条指令由处理器加载并执行以实现如权利要求1至9任一所述的重定位方法。
PCT/CN2019/079781 2018-04-27 2019-03-27 相机姿态追踪过程的重定位方法、装置、设备及存储介质 WO2019205868A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP19794077.8A EP3786895A4 (en) 2018-04-27 2019-03-27 METHOD, DEVICE AND APPARATUS FOR REPOSITIONING IN A CAMERA ORIENTATION TRACKING PROCESS, AND INFORMATION HOLDER
US16/919,059 US11276183B2 (en) 2018-04-27 2020-07-01 Relocalization method and apparatus in camera pose tracking process, device, and storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810392872.5A CN108876854B (zh) 2018-04-27 2018-04-27 相机姿态追踪过程的重定位方法、装置、设备及存储介质
CN201810392872.5 2018-04-27

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US16/919,059 Continuation US11276183B2 (en) 2018-04-27 2020-07-01 Relocalization method and apparatus in camera pose tracking process, device, and storage medium

Publications (1)

Publication Number Publication Date
WO2019205868A1 true WO2019205868A1 (zh) 2019-10-31

Family

ID=64326671

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/079781 WO2019205868A1 (zh) 2018-04-27 2019-03-27 相机姿态追踪过程的重定位方法、装置、设备及存储介质

Country Status (4)

Country Link
US (1) US11276183B2 (zh)
EP (1) EP3786895A4 (zh)
CN (1) CN108876854B (zh)
WO (1) WO2019205868A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11410329B2 (en) * 2019-09-24 2022-08-09 Toshiba Tec Kabushiki Kaisha Information processing device, method performed thereby, and non-transitory computer readable medium

Families Citing this family (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108876854B (zh) 2018-04-27 2022-03-08 腾讯科技（深圳）有限公司 相机姿态追踪过程的重定位方法、装置、设备及存储介质
CN108734736B (zh) 2018-05-22 2021-10-26 腾讯科技（深圳）有限公司 相机姿态追踪方法、装置、设备及存储介质
CN110008856B (zh) * 2019-03-20 2021-08-17 联想(北京)有限公司 定位方法和电子设备
CN113785299B (zh) * 2019-05-03 2023-12-19 丰田汽车欧洲股份有限公司 用于发现对象的图像获取设备
CN110307838B (zh) * 2019-08-26 2019-12-10 深圳市优必选科技股份有限公司 机器人重定位方法、装置、计算机可读存储介质及机器人
US11315326B2 (en) * 2019-10-15 2022-04-26 At&T Intellectual Property I, L.P. Extended reality anchor caching based on viewport prediction
CN111339228B (zh) * 2020-02-18 2023-08-11 Oppo广东移动通信有限公司 一种地图更新方法、设备、云服务器和存储介质
CN111552757B (zh) * 2020-04-30 2022-04-01 上海商汤临港智能科技有限公司 生成电子地图的方法、装置、设备及存储介质
CN112164114B (zh) * 2020-09-23 2022-05-20 天津大学 一种基于天际线匹配的室外主动相机重定位方法
CN114022560A (zh) * 2021-10-14 2022-02-08 浙江商汤科技开发有限公司 标定方法及相关装置、设备

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6787750B1 (en) * 2000-06-29 2004-09-07 Siemens Corporate Research, Inc. Method and apparatus for robust optical tracking with beacon markers
CN104243833A (zh) * 2014-09-30 2014-12-24 安科智慧城市技术（中国）有限公司 一种相机姿态的调整方法及装置
CN106885574A (zh) * 2017-02-15 2017-06-23 北京大学深圳研究生院 一种基于重跟踪策略的单目视觉机器人同步定位与地图构建方法
CN108876854A (zh) * 2018-04-27 2018-11-23 腾讯科技（深圳）有限公司 相机姿态追踪过程的重定位方法、装置、设备及存储介质

Family Cites Families (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10802147B2 (en) * 2016-05-18 2020-10-13 Google Llc System and method for concurrent odometry and mapping
US20170374256A1 (en) * 2016-06-24 2017-12-28 Qualcomm Incorporated Method and apparatus for rolling shutter compensation
CN106595601B (zh) * 2016-12-12 2020-01-07 天津大学 一种无需手眼标定的相机六自由度位姿精确重定位方法
CN106708048B (zh) * 2016-12-22 2023-11-28 清华大学 机器人的天花板图像定位方法和系统
CN107300917B (zh) * 2017-05-23 2019-07-09 北京理工大学 一种基于分层架构的视觉slam后端优化方法
CN107633536B (zh) * 2017-08-09 2020-04-17 武汉科技大学 一种基于二维平面模板的相机标定方法及系统
CN109542093B (zh) * 2017-09-22 2022-06-07 华为技术有限公司 一种处理数据的方法和装置
CN110555883B (zh) 2018-04-27 2022-07-22 腾讯科技（深圳）有限公司 相机姿态追踪过程的重定位方法、装置及存储介质
CN110599549B (zh) 2018-04-27 2023-01-10 腾讯科技（深圳）有限公司 界面显示方法、装置及存储介质
CN110555882B (zh) 2018-04-27 2022-11-15 腾讯科技（深圳）有限公司 界面显示方法、装置及存储介质
CN108682038B (zh) 2018-04-27 2021-12-14 腾讯科技（深圳）有限公司 位姿确定方法、装置及存储介质
CN108596976B (zh) 2018-04-27 2022-02-22 腾讯科技（深圳）有限公司 相机姿态追踪过程的重定位方法、装置、设备及存储介质
CN108615248B (zh) 2018-04-27 2022-04-05 腾讯科技（深圳）有限公司 相机姿态追踪过程的重定位方法、装置、设备及存储介质

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6787750B1 (en) * 2000-06-29 2004-09-07 Siemens Corporate Research, Inc. Method and apparatus for robust optical tracking with beacon markers
CN104243833A (zh) * 2014-09-30 2014-12-24 安科智慧城市技术（中国）有限公司 一种相机姿态的调整方法及装置
CN106885574A (zh) * 2017-02-15 2017-06-23 北京大学深圳研究生院 一种基于重跟踪策略的单目视觉机器人同步定位与地图构建方法
CN108876854A (zh) * 2018-04-27 2018-11-23 腾讯科技（深圳）有限公司 相机姿态追踪过程的重定位方法、装置、设备及存储介质

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3786895A4 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11410329B2 (en) * 2019-09-24 2022-08-09 Toshiba Tec Kabushiki Kaisha Information processing device, method performed thereby, and non-transitory computer readable medium

Also Published As

Publication number Publication date
US20200334836A1 (en) 2020-10-22
EP3786895A1 (en) 2021-03-03
CN108876854B (zh) 2022-03-08
US11276183B2 (en) 2022-03-15
EP3786895A4 (en) 2022-01-19
CN108876854A (zh) 2018-11-23

Similar Documents

Publication Publication Date Title
WO2019205868A1 (zh) 相机姿态追踪过程的重定位方法、装置、设备及存储介质
WO2019205842A1 (zh) 相机姿态追踪过程的重定位方法、装置及存储介质
WO2019205853A1 (zh) 相机姿态追踪过程的重定位方法、装置、设备及存储介质
US11481923B2 (en) Relocalization method and apparatus in camera pose tracking process, device, and storage medium
CN108596976B (zh) 相机姿态追踪过程的重定位方法、装置、设备及存储介质
CN110544280B (zh) Ar系统及方法
WO2019205850A1 (zh) 位姿确定方法、装置、智能设备及存储介质
US11640235B2 (en) Additional object display method and apparatus, computer device, and storage medium
WO2019205851A1 (zh) 位姿确定方法、装置、智能设备及存储介质
WO2019154231A1 (zh) 图像处理方法、电子设备及存储介质
US11210810B2 (en) Camera localization method and apparatus, terminal, and storage medium
CN108682037B (zh) 相机姿态追踪过程的重定位方法、装置、设备及存储介质
CN108776822B (zh) 目标区域检测方法、装置、终端及存储介质
WO2019134305A1 (zh) 确定姿态的方法、装置、智能设备、存储介质和程序产品
CN112087728B (zh) 获取Wi-Fi指纹空间分布的方法、装置和电子设备
KR102084161B1 (ko) 이미지를 보정하는 전자 장치 및 그 제어 방법

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19794077

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

WWE Wipo information: entry into national phase

Ref document number: 2019794077

Country of ref document: EP