WO2019223468A1 - 相机姿态追踪方法、装置、设备及系统 - Google Patents

相机姿态追踪方法、装置、设备及系统 Download PDF

Info

Publication number
WO2019223468A1
WO2019223468A1 PCT/CN2019/083437 CN2019083437W WO2019223468A1 WO 2019223468 A1 WO2019223468 A1 WO 2019223468A1 CN 2019083437 W CN2019083437 W CN 2019083437W WO 2019223468 A1 WO2019223468 A1 WO 2019223468A1
Authority
WO
WIPO (PCT)
Prior art keywords
initial
image
camera
change amount
posture
Prior art date
Application number
PCT/CN2019/083437
Other languages
English (en)
French (fr)
Inventor
林祥凯
凌永根
暴林超
朱晓龙
乔亮
刘威
Original Assignee
腾讯科技（深圳）有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 腾讯科技（深圳）有限公司 filed Critical 腾讯科技（深圳）有限公司
Priority to EP19808400.6A priority Critical patent/EP3798983B1/en
Publication of WO2019223468A1 publication Critical patent/WO2019223468A1/zh
Priority to US17/001,658 priority patent/US11321870B2/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G06T7/74—Determining position or orientation of objects or cameras using feature-based methods involving reference images or patches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G06T7/248—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments involving reference images or patches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/02—Affine transformations
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/269—Analysis of motion using gradient-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/292—Multi-camera tracking
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20076—Probabilistic image processing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30204—Marker
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30244—Camera pose

Definitions

  • the present application relates to the field of augmented reality, and in particular, to a camera attitude tracking method, device, device, and storage medium.
  • a device equipped with a camera needs to track the camera's movement posture in the real environment during the movement without the prior information of the environment.
  • the device establishes a three-dimensional coordinate system for the real environment.
  • the device uses the first image collected initially as a labeled image, and performs feature point tracking on subsequent images relative to the labeled image, and calculates a homography matrix between the subsequent image and the labeled image according to the feature point tracking result.
  • the homography matrix is decomposed to obtain the camera attitude of the camera when acquiring subsequent images according to the decomposition result.
  • each device has different three-dimensional coordinate systems established in the real environment.
  • the camera attitude tracked by each device is different. If two devices are required to perform camera attitude tracking based on the same reference, the environment map (including a large number of image frames and three-dimensional points) constructed on the first device needs to be synchronized to the second device, and the synchronization process needs to transmit a large amount of data. Not conducive to implementation on electronic devices using mobile networks.
  • a camera attitude tracking method, device, device, and storage medium are provided.
  • a camera attitude tracking method is provided, which is applied to a second device provided with a camera, and the method includes:
  • the second device calculates and obtains a current camera attitude parameter when the camera collects the current image according to the initial attitude parameter and the camera attitude change amount.
  • a camera attitude tracking device which is applied to a second device provided with a camera, and the device includes:
  • a data receiving module configured to receive an initial marker image and an initial posture parameter sent by the first device
  • An image acquisition module configured to acquire a current image collected by the camera
  • a change amount acquisition module configured to obtain a change amount of a camera pose of the current image relative to the initial labeled image
  • An attitude calculation module is configured to calculate and obtain a current camera attitude parameter when the camera collects the current image according to the initial attitude parameter and the camera attitude change amount.
  • an electronic device includes a memory and a processor.
  • the memory stores computer-readable instructions.
  • the computer-readable instructions are executed by the processor, Causing the processor to perform the following steps:
  • a current camera attitude parameter when the camera collects the current image is calculated and obtained.
  • An embodiment of the present application provides a computer-readable storage medium.
  • the computer-readable instructions are executed by one or more processors, the one or more processors execute the following steps:
  • a current camera attitude parameter when the camera collects the current image is calculated and obtained.
  • FIG. 1 is a schematic diagram of an AR application scenario provided by an exemplary embodiment of the present application.
  • FIG. 2 is a schematic diagram of an AR application scenario provided by an exemplary embodiment of the present application.
  • FIG. 3 is a structural block diagram of an AR system provided by an exemplary embodiment of the present application.
  • FIG. 4 is a structural block diagram of an AR system provided by another exemplary embodiment of the present application.
  • FIG. 5 is a structural block diagram of an electronic device according to another exemplary embodiment of the present application.
  • FIG. 6 is a flowchart of a camera attitude tracking method provided by an exemplary embodiment of the present application.
  • FIG. 7 is a schematic diagram of the Anchor-Switching AR System algorithm provided by an exemplary embodiment of the present application.
  • FIG. 8 is a flowchart of a camera attitude tracking method according to another exemplary embodiment of the present application.
  • FIG. 9 is a flowchart of a camera attitude tracking method according to another exemplary embodiment of the present application.
  • FIG. 10 is a block diagram of a camera attitude tracking device provided by an exemplary embodiment of the present application.
  • FIG. 11 is a block diagram of an electronic device according to an exemplary embodiment of the present application.
  • FIG. 12 shows an internal structure diagram of an electronic device in an embodiment of the present application.
  • AR a technology that calculates camera pose parameters of the camera in the real world (or three-dimensional world, real world) in real-time during the process of camera acquisition of images, and adds virtual elements to the images collected by the camera according to the camera pose parameters .
  • Virtual elements include, but are not limited to: images, videos, and three-dimensional models.
  • the goal of AR technology is to connect the virtual world on the screen to the real world to interact.
  • the camera attitude parameters include a rotation matrix and a displacement vector.
  • the rotation matrix is used to represent the rotation angle of the camera in the real world
  • the displacement vector is used to represent the displacement distance of the camera in the real world.
  • the device adds a avatar to the image captured by the camera.
  • the images captured by the camera will change, and the shooting position of the virtual character will also change. It is simulated that the virtual character is stationary in the image, and the camera changes with the change of position and attitude at the same time.
  • the effect of shooting images and virtual characters presents the user with a real three-dimensional picture.
  • Anchor-Switching AR System An AR system that determines camera pose parameters in natural scenes based on camera pose tracking connecting multiple labeled images (Anchor), and then superimposes a virtual world on the images collected by the camera according to the camera pose parameters.
  • IMU Inertial Measurement Unit
  • an IMU contains three single-axis accelerometers and three single-axis gyroscopes.
  • the accelerometer is used to detect the acceleration signal of an object on each coordinate axis in the three-dimensional coordinate system, and then calculate the displacement vector (maybe (Accumulates the error with time); and the gyro is used to detect the rotation matrix of the object in the three-dimensional coordinate system.
  • the IMU includes a gyroscope, an accelerometer, and a geomagnetic sensor.
  • the three-dimensional coordinate system is established as follows: 1.
  • the X-axis is defined by a vector product Y * Z, and at the current position of the X-axis in the device, it points to the east in a direction tangent to the ground; 2.
  • the Z axis points to the sky and is perpendicular to the ground.
  • the target image fails to track the feature points relative to the i-th mark image and the camera pose tracking process cannot be successfully performed, the target image is relative to other images ( The initial mark image or other representative key frame image) performs feature point tracking again, so that when the feature point tracking is successful, the technology of camera positioning for the current image of the camera according to the retrieved tracking process is implemented.
  • FIG. 3 shows a structural block diagram of an AR system provided by an exemplary embodiment of the present application.
  • the AR system includes: a first device 120 and a second device 140.
  • the first device 120 is, for example, an MP3 player (Moving Picture Experts Group Audio Layer III, Moving Picture Expert Compression Standard Audio Level 3), MP4 (Moving Picture Experts Group Audio Audio Layer IV), such as a smart phone, a tablet computer, and an intelligent operating system. Motion picture experts compress standard audio level 4) Electronic equipment such as players, portable gaming terminals.
  • a target AR program is run on the first device 120, and the target AR program may be an AR program used by multiple persons in cooperation.
  • the program type of the target AR program includes, but is not limited to, at least one selected from the group consisting of an AR teaching program, an AR game program, an AR translation program, an AR social program, an AR sports program, and an AR driving assistance program.
  • the first device 120 is connected to the second device 140 through a data line, a wired network, Bluetooth, or WIFI (WIreless Fidelity). That is, the first device 120 and the second device 140 can directly perform data communication.
  • a data line a wired network, Bluetooth, or WIFI (WIreless Fidelity). That is, the first device 120 and the second device 140 can directly perform data communication.
  • WIFI WIreless Fidelity
  • the second device 140 is, for example, an MP3 player (Moving Picture Experts Group Audio Layer III, Motion Picture Expert Compression Standard Audio Level 3), MP4 (Moving Picture Experts Group Audio Audio Layer IV) with a smart operating system, a smart phone, a tablet computer, and an intelligent operating system. Motion picture experts compress standard audio level 4) Electronic equipment such as players, portable gaming terminals. A target AR program is also installed in the second device 140.
  • MP3 player Motion Picture Experts Group Audio Layer III, Motion Picture Expert Compression Standard Audio Level 3
  • MP4 Motion Picture Experts Group Audio Audio Layer IV
  • Motion picture experts compress standard audio level 4
  • Electronic equipment such as players, portable gaming terminals.
  • a target AR program is also installed in the second device 140.
  • the target AR programs installed on the first device 120 and the second device 140 may be the same or different.
  • the number of devices in the AR system can be more than two.
  • the first device 120 and the second device 140 can be any two devices among the multiple devices. This application does not limit this.
  • FIG. 4 shows a structural block diagram of an AR system provided by another exemplary embodiment of the present application.
  • the AR system includes: a first device 120, a server 160, and a second device 140.
  • the AR system in this embodiment adds a server 160. among them:
  • the first device 120 is connected to the server 160 through a network.
  • the network can be a wireless network or a wired network.
  • a target AR program is installed and executed on the first device 120.
  • the second device 140 is connected to the server 160 through a network.
  • the network can be a wireless network or a wired network.
  • the target AR program is also installed and run on the second device 120.
  • the server 160 may be a server, a server cluster composed of multiple servers, a cloud computing center, a virtualized computing center, or a distributed computing center.
  • the server 160 may be used to provide a background service of the target AR program.
  • the first device 120 and the second device 140 are "clients" of the server; in other embodiments, the first device 120 and the second device 140 cooperate with the server using a distributed computing strategy Calculation, this embodiment does not limit this.
  • FIG. 5 shows a structural block diagram of an electronic device according to an exemplary embodiment of the present application.
  • the electronic device may be the first device 120 and / or the second device 140 described above.
  • the electronic device includes: a processor 320, a memory 340, a camera 360, and an IMU 380.
  • the processor 320 includes one or more processing cores, such as a 4-core processor, an 8-core processor, and the like.
  • the processor 320 is configured to execute at least one of an instruction, a code, a code fragment, and a program stored in the memory 340.
  • the processor 320 is electrically connected to the memory 340.
  • the processor 320 is connected to the memory 340 through a bus.
  • the memory 340 stores one or more instructions, codes, code fragments, and / or programs. When the instructions, codes, code fragments, and / or programs are executed by the processor 320, they are used to implement the camera attitude tracking method provided in the following embodiments.
  • the processor 320 is also electrically connected to the camera 360.
  • the processor 320 is connected to the camera 360 through a bus.
  • the camera 360 is a sensor device having an image acquisition capability.
  • the camera 360 may also be called other names such as a camera, a photosensitive device, and the like.
  • the camera 360 has the ability to capture images continuously or multiple times.
  • the camera 360 is disposed inside or outside the electronic device.
  • the processor 320 is also electrically connected to the IMU380.
  • IMU380 is used to collect the pose vector of the camera every predetermined time interval, and record the time stamp of each set of pose parameters at the time of collection.
  • the pose parameters of the camera include: displacement vector and rotation matrix. Among them, the rotation matrix collected by the IMU380 is relatively accurate, and the collected displacement vector may have a large error due to the actual environment.
  • the specific form of the electronic device may be a smart phone, a tablet computer, AR glasses, an AR helmet, a portable notebook computer, an e-book reader, MP3, MP4, a handheld game terminal, and the like. This embodiment of the present application does not limit this.
  • FIG. 6 shows a flowchart of a camera attitude tracking method provided by an exemplary embodiment of the present application. This method can be used in an AR system as shown in FIG. 1 or FIG. 2. The method includes:
  • Step 401 The first device starts an initialization process of an AR scenario.
  • a target AR program supporting a multiplayer mode runs on the first device and the second device.
  • the first device After the target AR program is started, for example, after an AR scene or an AR game starts, the first device performs the initialization process of the AR scene.
  • the initialization process includes:
  • the first device collects frame-by-frame images in a real environment through a built-in camera
  • the first device is provided with a first camera. After the target AR program is started, the first camera continuously collects frame-by-frame images in the real environment, and then sends the images to the processor for processing.
  • the first device uses the first frame image (or an image specified by the user) as the initial mark image;
  • the first device uses the first frame image as an initial labeled image (English: born-image), or the first device uses an image specified by the user as the initial labeled image.
  • the first device receives an initial attitude parameter set by the user
  • the initial pose parameter (English: born-pose) is set by the user, or in other words, set by the user through the target AR program.
  • the setting process of the initial posture parameter may be explicitly perceived by the user, or may be explicitly not perceived by the user.
  • the target AR program is a game program that superimposes a virtual doll in the real world.
  • the image when the virtual doll is superimposed is set as the initial mark image, and the superimposed image is superimposed.
  • the pose parameter during the virtual doll is set as the initial pose parameter, and the setting process is not perceived by the user.
  • the first device performs a camera posture tracking process according to the initial marker image and the initial posture parameters.
  • the first device performs a camera posture tracking process according to the initial marker image and the initial posture parameters.
  • the camera attitude tracking process may use the same or similar camera attitude tracking technology as the stand-alone form.
  • Step 402 The first device records an initial marker image and an initial attitude parameter.
  • Step 403 The first device sends an initial labeled image and an initial posture parameter to the second device.
  • the first device and the second device are installed with the same target AR program.
  • the first device and the second device have the same target AR program installed, and a preset condition is satisfied between the two target AR programs.
  • the preset conditions include, but are not limited to: two user accounts are friend relationships, two user accounts are in the same AR scenario, two user accounts are in the same AR mode, and The two user accounts are in at least one selected from the group consisting of the same AR game room.
  • a target data channel is directly established between the first device and the second device, and the first device sends an initial marker image and an initial attitude parameter to the second device through the target data channel.
  • the target data channel includes: at least one selected from the group consisting of WIFI, Bluetooth, a wired network, and a data line.
  • the data forwarding between the first device and the second device needs to be relayed by the server.
  • the first device first sends the initial labeled image and the initial posture parameter to the server, and the server sends the initial labeled image and the initial posture parameter. Forward to the second device.
  • Step 404 The second device receives the initial marker image and the initial posture parameter sent by the first device.
  • the second device receives the initial marker image and the initial posture parameter sent by the first device through the target data channel or the server.
  • the initial labeled image is the first labeled image of the first device during the camera posture tracking process
  • the initial posture parameter is the camera posture parameter corresponding to the first labeled image.
  • Both the initial marker image and the initial posture parameters are data generated by the first device, and not data generated by the second device.
  • Step 405 The second device acquires the current image collected by the camera.
  • a second camera is set in the second device. After the target AR program is started, the second camera continuously collects frame-by-frame images in the real environment, and then sends the images to the processor for processing.
  • Step 406 The second device acquires a camera posture change amount of the current image relative to the initial labeled image.
  • the second device performs camera posture tracking on the current image collected by the second device relative to the initial labeled image sent by the first device, and calculates a camera posture change amount of the current image relative to the initial labeled image.
  • the camera of the second device does not reach the initial labeled image.
  • the initial labeled image is assumed to be an image collected by the second device for camera posture tracking.
  • the first device and the second device will do the framing shooting of the same real scene, such as the framing shooting of the same desktop. Therefore, the images captured by the two have a certain correlation.
  • the initial labeled image acquired by the device is assumed to be an image acquired by the second device for use.
  • Step 407 The second device calculates and obtains the current camera attitude parameter when the camera collects the current image according to the initial attitude parameter and the camera attitude change amount.
  • the second device calculates the camera attitude parameters R and T when the second camera collects the current image according to the following formula:
  • R cf is the rotation matrix in the camera pose change of the current image relative to the initial labeled image
  • T cf is the displacement vector in the camera pose change of the current image relative to the initial labeled image
  • R born is the The rotation matrix
  • T born is the displacement vector in the initial pose parameters.
  • the camera attitude tracking method receives the initial marker image and initial pose parameters sent by the first device through the second device, and calculates the current image collected by the camera of the second device relative to the camera of the initial marker image.
  • the attitude change amount is calculated based on the initial attitude parameter and the camera attitude change amount to calculate the camera attitude parameter when the camera collects the current image, so that the second device performs camera attitude tracking on the image acquired by itself relative to the initial labeled image of the first device. That is, the first device and the second device use the same initial marker image and initial posture parameters for camera attitude tracking to ensure that the camera attitude tracking results of the two are the same, thereby solving the need to synchronize the environment map constructed on the first device. On the second device, there is a problem of transmitting a large amount of data.
  • the steps performed by the first device may be separately implemented as the camera attitude tracking method on the first device side
  • the steps performed by the second device may be separately implemented as the camera attitude tracking method on the second device side.
  • the first device uses the Anchor-Switching AR system algorithm for camera attitude tracking.
  • the Anchor-Switching AR system algorithm determines the camera's pose by dividing the entire motion process of the first camera into at least two tracking processes for tracking, and each tracking process corresponds to its own labeled image. Specifically, first, based on the initial mark image (also referred to as the first mark image), camera posture tracking is performed on the current image captured by the camera. The camera attitude tracking process may use the current tracking image during the tracking process corresponding to the i-th mark image.
  • the previous image of the current image is determined as the i + 1-th labeled image, and the first i + 1 segment tracking process.
  • i is a positive integer.
  • the first device 120 provided with a first camera is held by a user to move, and multiple frames of the image 1-6 including the object 50 are captured during the movement.
  • the device determines image 1 as an initial labeled image (English: born-image or born-anchor) and records an initial pose parameter (English: born-pose).
  • the initial pose parameter may be an initial pose parameter set by a user.
  • the pose parameter when the camera collected by IMU collects image 1. Then perform feature point tracking on image 2 relative to image 1, and calculate the pose parameters of the camera when capturing image 2 according to the initial pose parameters and the feature point tracking results; perform feature point tracking on image 3 relative to image 1, according to the initial
  • the pose parameters and feature point tracking results are used to calculate the pose parameters of the camera when capturing image 3.
  • Image 4 is compared to image 1 to perform feature point tracking. Based on the initial pose parameters and feature point tracking results, the camera is capturing images 4 Pose parameter at the time.
  • image 5 is compared with image 1 for feature point tracking. If the feature point tracking effect is worse than a preset condition (for example, the number of matching feature points is small), then image 4 is determined as the second marker image, and image 5 is Perform feature point tracking with respect to image 4 to calculate the displacement change of the camera between captured images 4 to 5 and then combine the displacement change of the camera between captured images 4 to 1 and the initial pose parameters to calculate The pose parameter of the camera when taking image 5. Then, feature point tracking is performed on image 6 relative to image 4, and so on. If the feature point tracking effect of the current image becomes poor, the previous frame image of the current image can be determined as the new marker image, and the new one can be switched. After the image is marked, feature point tracking is performed again.
  • a preset condition for example, the number of matching feature points is small
  • the feature point tracking may use an algorithm based on the principle of visual odometry, such as a feature point method or a direct method.
  • the second device after receiving the initial marker image and the initial posture parameter, performs camera attitude tracking on the current image collected by the second camera in the second device with respect to the initial marker image.
  • this step may be implemented in at least one of the following two ways.
  • the second device repositions the current image relative to the initial labeled image to obtain the camera attitude change amount.
  • the second device obtains a first posture change amount of the i-th mark image relative to the initial mark image, i> 1; performs feature point tracking on the current image relative to the i-th mark image to obtain a second posture change amount; According to the first posture change amount and the second posture change amount, the camera posture change amount is calculated and obtained.
  • the first implementation manner is described below using FIG. 8, and the second implementation manner is introduced using FIG. 9.
  • the second device uses the first method to obtain a camera posture change amount of the current image relative to the initial labeled image.
  • the current image is the first few frames captured by the second camera in the second device (or a loss occurred during camera pose tracking)
  • the second device will The received initial mark image is set as the first mark image of the local machine, and the second device repositions the current image with respect to the initial mark image to obtain a camera attitude change amount.
  • Step 406 includes, but is not limited to, the following steps:
  • Step 406a the second device obtains the initial feature points of the initial labeled image
  • the first device when the first device sends the initial labeled image to the second device, it also simultaneously sends the initial feature points in the initial labeled image, and the second device directly reads the initial feature points of the initial labeled image.
  • the initial feature points are feature points extracted from the initial labeled image, and the initial feature points may be multiple, such as 10-500.
  • the second device extracts the initial feature points in the initial labeled image.
  • the feature extraction algorithm used when the first device extracts feature points may be FAST (Features from Accelerated Segment Test), Shi-Tomasi corner detection algorithm, Harris Corner detection (Harris corner detection) algorithm, SIFT (Scale-Invariant Feature Transformation) algorithm, ORB (Oriented FAST and RotatedBRIEF, fast feature point extraction and description) algorithm, etc.
  • An ORB feature point includes two parts: a FAST corner point (Key-point) and a BRIER descriptor (Binary, Robust, Independent, Elementary, and Feature Descriptor).
  • the ORB feature is faster to calculate and is therefore suitable for implementation on mobile devices.
  • the FAST corner point refers to the position of the ORB feature point in the image.
  • FAST corners are mainly used to detect the obvious changes in the local pixel gray level, which is famous for its fast speed. The idea of FAST corners: If a pixel is significantly different from the pixels in the neighborhood (too bright or too dark), the pixel may be a corner.
  • the BRIEF descriptor is a binary representation of a vector that describes the information about the pixels around the keypoint in an artificial way.
  • the description vector of the BRIEF descriptor consists of multiple 0s and 1s, where 0s and 1s encode the size relationship between the two pixels near the FAST corner.
  • Step 406b the second device performs feature point tracking on the current image relative to the initial labeled image to obtain a target feature point that matches the initial feature point;
  • the tracking of feature points can be based on a visual odometry tracking algorithm, which is not limited in this application.
  • the feature point tracking uses the KLT (Kanade-Lucas) optical flow tracking algorithm; in another embodiment, the feature point tracking uses the ORB (Oriented FAST and RotatedBRIEF, fast feature point extraction and description) algorithm extraction ORB feature descriptor performs feature point tracking.
  • KLT Kerade-Lucas
  • ORB Oriented FAST and RotatedBRIEF, fast feature point extraction and description
  • the specific algorithm for feature point tracking is not limited in this application, and the feature point tracking process may adopt a feature point method or a direct method.
  • the second device performs feature point extraction on the current image to obtain M candidate feature points; and then matches the M candidate feature points one by one with the N initial feature points to determine at least one set of matching feature point pairs.
  • Each set of matching feature point pairs includes: an initial feature point and a target feature point.
  • the initial feature point is a feature point on the initial labeled image
  • the target feature point is a candidate feature point on the current image that has the highest degree of matching with the initial feature point.
  • the number of initial feature points is greater than or equal to the number of target feature points.
  • the number of initial feature points is 450 and the target feature points are 320 groups.
  • Step 406c the second device calculates a homography matrix between the initial labeled image and the current image according to the initial feature point and the target feature point;
  • the homography matrix describes the mapping relationship between the two imaging planes of the camera, that is, the mapping relationship between the imaging plane of the initial labeled image and the imaging plane of the current image. If the natural scene (real environment The feature points in) all lie on the same physical plane, then the camera pose motion estimation can be performed through the homography matrix.
  • Step 406d the second device decomposes the homography matrix to obtain the camera attitude change amount.
  • the device decomposes the homography matrix through ransac to obtain the camera attitude change amount.
  • the camera attitude change amount includes a rotation matrix R cf and a translation vector T cf.
  • the second device may switch (or set) the current image to the second marker image, and compare other images acquired by the second camera in the subsequent phase relative to the first image. Camera tracking of two marker images. For related principles, refer to FIG. 5.
  • the second device obtains the camera attitude change amount by relocating the current image relative to the initial labeled image, so that when the labeled image does not yet exist or is lost,
  • various information of the camera pose tracking process is constructed, and the same benchmark as the first device is used to track the camera pose, thereby realizing the camera pose tracking process of the AR scene in multiplayer mode Synchronization.
  • the second device uses a second method to obtain a camera posture change amount of the current image relative to the initial labeled image. If the current image is an image acquired by the second device after the camera posture tracking process is performed for a period of time, the second device has been tracking the camera posture according to the Anchor-Switching AR System algorithm for a period of time. At this time, the image based on the i-th mark Camera attitude tracking process, i is an integer greater than 1.
  • Step 406 includes, but is not limited to, the following steps, as shown in FIG. 9:
  • Step 4061 the second device acquires a first posture change amount of the i-th labeled image relative to the initial labeled image, i> 1;
  • the i-th mark image is a second mark image, a third mark image, a fourth mark image, and the like. That is, the i-th mark image is a mark image after the initial mark image.
  • the i-th mark image is an image that has been successfully tracked in the history process
  • the second device has a first posture change amount of the i-th mark image relative to the initial mark image buffered.
  • the first attitude change amount includes a first rotation matrix R mf and a first displacement vector T mf .
  • Step 4062 The second device performs feature point tracking on the current image relative to the i-th mark image to obtain a second posture change amount.
  • the tracking of feature points can be based on a visual odometry tracking algorithm, which is not limited in this application.
  • the feature point tracking uses a KLT optical flow tracking algorithm; in another embodiment, the feature point tracking uses an ORB feature descriptor extracted based on the ORB algorithm for feature point tracking.
  • the specific algorithm for feature point tracking is not limited in this application, and the feature point tracking process may adopt a feature point method or a direct method.
  • this step includes the following sub-steps:
  • the second device performs feature point tracking with respect to a reference feature point on the i-th labeled image to obtain a target feature point that matches the reference feature point;
  • the second device When the second device sets the i-th labeled image, it extracts N reference feature points on the i-th labeled image.
  • the second device When feature point tracking is performed on the current image with respect to the reference feature points on the i-th labeled image, the second device extracts feature points from the current image to obtain M candidate feature points; then, the M candidate feature points are N and N one by one.
  • the two reference feature points are matched to determine at least one set of matching feature point pairs.
  • Each set of matching feature point pairs includes: a reference feature point and a target feature point.
  • the reference feature point is a feature point on the i-th labeled image
  • the target feature point is a candidate feature point on the current image that has the highest degree of matching with the reference feature point.
  • the number of reference feature points is greater than or equal to the number of target feature points.
  • the number of initial feature points is 500 and the target feature points are 430 groups.
  • the second device calculates a homography matrix between the i-th labeled image and the current image according to the reference feature point and the target feature point;
  • the homography matrix describes the mapping relationship between the two imaging planes of the camera, that is, the mapping relationship between the imaging plane of the initial labeled image and the imaging plane of the current image. If the natural scene (real environment The feature points in) all lie on the same physical plane, then the camera pose motion estimation can be performed through the homography matrix.
  • the second device decomposes the homography matrix to obtain a second attitude change amount
  • the device decomposes the homography matrix through ransac to obtain a second attitude change amount.
  • the second attitude change amount includes a rotation matrix R cm and a translation vector T cm .
  • step 4063 the second device calculates and obtains the camera posture change amount according to the first posture change amount and the second posture change amount.
  • the second device calculates the camera attitude change amounts R cf and T cf according to the following formula:
  • S mf is the scale of the i-th mark image.
  • S mf is a preset value, or S mf is to map the initial feature points on the initial labeled image to the i-th labeled image to obtain the projected feature points, and calculate the i-th based on the average depth of the projected feature points The corresponding scale of the image.
  • the second device calculates the camera by using the first attitude change amount and the second attitude change amount after tracking the feature point of the current image relative to the i-th mark image.
  • the attitude change amount is calculated based on the camera attitude change amount and the initial attitude parameter to obtain the current camera attitude parameter, so that the second device can perform the camera attitude tracking for a period of time based on the current image collected by itself and the first device
  • the initial marker image constructs various information of the camera pose tracking process, and uses the same reference as the first device to perform camera pose tracking, thereby realizing the synchronization of the camera pose tracking process in the multiplayer mode AR scene.
  • the second device maps the initial feature points on the first labeled image to the i-th labeled image to obtain the projected feature points, and calculates the i-th based on the average depth of the projected feature points.
  • the process of marking the corresponding scale of an image is as follows:
  • the second device obtains two-dimensional coordinates of an initial feature point in the initial labeled image
  • the second device previously extracts and buffers the two-dimensional coordinates of the initial feature points in the initial labeled image.
  • the two-dimensional coordinates are represented in a homogeneous manner.
  • the second device back-projects the two-dimensional coordinates of the initial feature points to obtain the first three-dimensional coordinates X born of the initial feature points in the three-dimensional space;
  • the second device transforms the two-dimensional coordinates of the initial feature points into the three-dimensional space by using the following formula to obtain the first three-dimensional coordinates X born of the initial feature points in the three-dimensional space.
  • f x , f y , c x , and c y are built-in parameters of the camera.
  • the two-dimensional coordinate x born of the initial feature point is a homogeneous representation on the first labeled image
  • the three-dimensional point x born is a non-homogeneous representation.
  • the initial depth d of the first labeled image is 1.
  • the second device performs three-dimensional rotation and translation on the first three-dimensional coordinate X born through the following formula to obtain a second three-dimensional coordinate X current corresponding to the initial feature point on the i-th labeled image ;
  • R mf is a rotation matrix in the first pose change amount
  • T mf is a displacement vector in the first pose change amount
  • the second device projects the second three-dimensional coordinate X current to the i-th mark image to obtain the two-dimensional coordinates of the projection feature point in the i-th mark image;
  • f x , f y , c x , and c y are built-in parameters of the second camera.
  • the second device calculates the scale s mf of the i-th labeled image according to the average depth of the projection feature points in the i-th labeled image.
  • Both the first device and the second device assume that the depth of the three-dimensional point corresponding to the initial labeled image is 1, and the initial posture parameters are also the same. Therefore, the three-dimensional world established by the first device and the second device to the real world is necessarily the same, so two The three-dimensional coordinate system is directly aligned between the devices.
  • the camera attitude tracking method provided by the embodiment of the present application can be applied to the AR field to track the camera's movement posture in the real environment without the prior information of the environment.
  • Typical application scenarios include:
  • the first user uses any one of the devices to perform the initialization process to obtain the initial labeled image and initial posture parameters, and then synchronizes the initial labeled image and initial posture parameters to other used by the second user.
  • Device, and other devices perform camera attitude tracking based on the initial marker image and the initial attitude parameters.
  • multiple devices can implement real-time multiplayer AR games based on the same camera tracking benchmark.
  • the first user uses the first device to bury the treasure (virtual treasure) at location A in the real world.
  • the second device uses the second device to find the treasure at location A, and can use the AR on the second device.
  • the program opens and removes the treasure.
  • steps in the embodiments of the present application are not necessarily performed sequentially in the order indicated by the step numbers. Unless explicitly stated in this document, the execution of these steps is not strictly limited, and these steps can be performed in other orders. Moreover, at least a part of the steps in each embodiment may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily performed at the same time, but may be performed at different times. The execution of these sub-steps or stages The order is not necessarily performed sequentially, but may be performed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.
  • an electronic device is further provided.
  • the electronic device includes a camera attitude tracking device, and the camera attitude tracking device includes various modules, and each module may be implemented in whole or in part by software, hardware, or a combination thereof.
  • FIG. 10 shows a block diagram of a camera attitude tracking device provided by an exemplary embodiment of the present application.
  • the device can be implemented as all or part of an electronic device (second device) through software, hardware, or a combination of the two.
  • the device includes:
  • a receiving module 1020 configured to receive an initial marker image and an initial posture parameter sent by a first device
  • An image acquisition module 1040 configured to acquire a current image collected by the camera
  • a change amount acquisition module 1060 configured to obtain a change amount of a camera posture of the current image with respect to the initial labeled image
  • An attitude calculation module 1080 is configured to calculate, according to the initial attitude parameter and the camera attitude change amount, a current camera attitude parameter when the camera collects the current image.
  • the change amount acquisition module 1060 is configured to reposition the current image relative to the initial marker image to obtain the change amount of the camera posture; and / or, the change A quantity acquisition module 1060 is configured to obtain a first posture change amount of the i-th mark image relative to the initial mark image, i> 1; perform feature point tracking on the current image with respect to the i-th mark image, A second posture change amount is obtained; and the camera posture change amount is calculated and obtained according to the first posture change amount and the second posture change amount.
  • the variation obtaining module 1060 is configured to obtain an initial feature point of the initial labeled image; perform feature point tracking on the current image with respect to the initial labeled image, and obtain The target feature point matching the initial feature point; calculating a homography matrix between the initial labeled image and the current image according to the initial feature point and the target feature point; decomposing the homography matrix to obtain The camera attitude change amount.
  • the first attitude change amount includes a first rotation matrix R mf and a first displacement vector T mf
  • the second attitude change amount includes a second rotation matrix R cm and a second displacement Vector T cm
  • the variation acquisition module 1060 is configured to calculate the camera posture variation R cf and T cf according to the following formula:
  • S mf is the scale of the i-th mark image.
  • the attitude calculation module 1080 is configured to calculate the camera attitude parameters R and T according to the following formula:
  • R cf is the rotation matrix in the camera attitude change amount
  • T cf is the displacement vector in the camera attitude change amount
  • R born is the rotation matrix in the initial pose parameter
  • T born is the initial pose The displacement vector in the parameter.
  • the receiving module 1020 is configured to receive the initial marker image and the initial attitude parameter sent by the first device through a target data channel, where the target data channel includes: At least one selected from the group consisting of a fidelity network, Bluetooth, a wired network, and a data line; and / or, the receiving module 1020 is configured to receive the initial marker image and the initial posture parameter sent by a server, The initial marker image and the initial posture parameter are sent by the first device to the server.
  • the camera attitude tracking device provided in the foregoing embodiment only uses the above-mentioned division of function modules as an example for performing camera attitude tracking.
  • the above functions may be allocated by different function modules as required. Finished, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.
  • the camera attitude tracking device and the camera attitude tracking method embodiment provided by the foregoing embodiments belong to the same concept. For specific implementation processes, see the method embodiments.
  • FIG. 11 shows a structural block diagram of a terminal 1100 provided by an exemplary embodiment of the present application.
  • the terminal 1100 can be: smartphone, tablet, MP3 player (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Compression Standard Audio Level 3), MP4 (Moving Picture Experts Group Audio Audio Layer Encoding IV, Moving Picture Experts Compressing Standard Audio Level 4) Player, laptop or desktop computer.
  • the terminal 1100 may also be called other names such as user equipment, portable terminal, laptop terminal, and desktop terminal.
  • the terminal 1100 includes a processor 1101 and a memory 1102.
  • the processor 1101 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like.
  • the processor 1101 may use at least one hardware form among DSP (Digital Signal Processing, Digital Signal Processing), FPGA (Field-Programmable Gate Array, Field Programmable Gate Array), and PLA (Programmable Logic Array). achieve.
  • the processor 1101 may also include a main processor and a co-processor.
  • the main processor is a processor for processing data in the awake state, also referred to as a CPU (Central Processing Unit).
  • the co-processor is Low-power processor for processing data in standby.
  • the processor 1101 may be integrated with a GPU (Graphics Processing Unit, image processor), and the GPU is responsible for rendering and drawing content required to be displayed on the display screen.
  • the processor 1101 may further include an AI (Artificial Intelligence) processor, and the AI processor is configured to process computing operations related to machine learning.
  • AI Artificial Intelligence
  • the memory 1102 may include one or more computer-readable storage media, which may be non-transitory.
  • the memory 1102 may further include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash storage devices.
  • the non-transitory computer-readable storage medium in the memory 1102 is used to store at least one instruction for execution by the processor 1101 to implement the camera gesture provided by the method embodiment in this application Tracking method.
  • the terminal 1100 may further include a peripheral device interface 1103 and at least one peripheral device.
  • the processor 1101, the memory 1102, and the peripheral device interface 1103 may be connected through a bus or a signal line.
  • Each peripheral device can be connected to the peripheral device interface 1103 through a bus, a signal line, or a circuit board.
  • the peripheral device includes at least one of a radio frequency circuit 1104, a touch display screen 1105, a camera 1106, an audio circuit 1107, a positioning component 1108, and a power supply 1109.
  • the peripheral device interface 1103 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1101 and the memory 1102.
  • the processor 1101, the memory 1102, and the peripheral device interface 1103 are integrated on the same chip or circuit board; in some other embodiments, any one of the processor 1101, the memory 1102, and the peripheral device interface 1103 or Both can be implemented on separate chips or circuit boards, which is not limited in this embodiment.
  • the radio frequency circuit 1104 is used to receive and transmit an RF (Radio Frequency) signal, also called an electromagnetic signal.
  • the radio frequency circuit 1104 communicates with a communication network and other communication devices through electromagnetic signals.
  • the radio frequency circuit 1104 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals.
  • the radio frequency circuit 1104 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and the like.
  • the radio frequency circuit 1104 can communicate with other terminals through at least one wireless communication protocol.
  • the wireless communication protocols include, but are not limited to, the World Wide Web, metropolitan area networks, intranets, mobile communication networks (2G, 3G, 4G, and 5G) of various generations, wireless local area networks, and / or WiFi (Wireless Fidelity) networks.
  • the radio frequency circuit 1104 may further include circuits related to Near Field Communication (NFC), which is not limited in this application.
  • NFC Near Field Communication
  • the display screen 1105 is used to display a UI (User Interface).
  • the UI may include graphics, text, icons, videos, and any combination thereof.
  • the display screen 1105 also has the ability to collect touch signals on or above the surface of the display screen 1105.
  • the touch signal can be input to the processor 1101 as a control signal for processing.
  • the display screen 1105 may also be used to provide a virtual button and / or a virtual keyboard, which is also called a soft button and / or a soft keyboard.
  • one display screen 1105 may be provided, and the front panel of the terminal 1100 is provided.
  • At least two display screens 1105 may be provided on different surfaces of the terminal 1100, or in a folded design.
  • the display screen 1105 may be a flexible display screen disposed on a curved surface or a folded surface of the terminal 1100.
  • the display screen 1105 can also be set as a non-rectangular irregular figure, that is, a special-shaped screen.
  • the display 1105 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).
  • the camera assembly 1106 is used to capture images or videos.
  • the camera assembly 1106 includes a front camera and a rear camera.
  • the front camera is located on the front panel of the terminal, and the rear camera is located on the back of the terminal.
  • the camera assembly 1106 may further include a flash.
  • the flash can be a monochrome temperature flash or a dual color temperature flash.
  • a dual color temperature flash is a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.
  • the audio circuit 1107 may include a microphone and a speaker.
  • the microphone is used to collect sound waves of the user and the environment, and convert the sound waves into electrical signals and input them to the processor 1101 for processing, or input them to the radio frequency circuit 1104 to implement voice communication. For the purpose of stereo collection or noise reduction, there may be multiple microphones, which are respectively disposed at different parts of the terminal 1100.
  • the microphone can also be an array microphone or an omnidirectional acquisition microphone.
  • the speaker is used to convert electrical signals from the processor 1101 or the radio frequency circuit 1104 into sound waves.
  • the speaker can be a traditional film speaker or a piezoelectric ceramic speaker.
  • the speaker When the speaker is a piezoelectric ceramic speaker, it can not only convert electrical signals into sound waves audible to humans, but also convert electrical signals into sound waves inaudible to humans for ranging purposes.
  • the audio circuit 1107 may further include a headphone jack.
  • the positioning component 1108 is used for positioning the current geographic position of the terminal 1100 to implement navigation or LBS (Location Based Service).
  • the positioning component 1108 may be a positioning component based on a US-based GPS (Global Positioning System), a Beidou system in China, or a Galileo system in Russia.
  • the power supply 1109 is used to power various components in the terminal 1100.
  • the power source 1109 may be an alternating current, a direct current, a disposable battery, or a rechargeable battery.
  • the rechargeable battery may be a wired rechargeable battery or a wireless rechargeable battery.
  • the wired rechargeable battery is a battery charged through a wired line
  • the wireless rechargeable battery is a battery charged through a wireless coil.
  • the rechargeable battery can also be used to support fast charging technology.
  • the terminal 1100 further includes one or more sensors 1110.
  • the one or more sensors 1110 include, but are not limited to, an acceleration sensor 1111, a gyro sensor 1112, a pressure sensor 1113, a fingerprint sensor 1114, an optical sensor 1115, and a proximity sensor 1111.
  • the acceleration sensor 1111 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 1100.
  • the acceleration sensor 1111 may be used to detect components of the acceleration of gravity on three coordinate axes.
  • the processor 1101 may control the touch display screen 1105 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 1111.
  • the acceleration sensor 1111 may also be used for collecting motion data of a game or a user.
  • the gyro sensor 1112 can detect the body direction and rotation angle of the terminal 1100, and the gyro sensor 1112 can cooperate with the acceleration sensor 1111 to collect a 3D motion of the user on the terminal 1100. Based on the data collected by the gyro sensor 1112, the processor 1101 can implement the following functions: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.
  • the pressure sensor 1113 may be disposed on a side frame of the terminal 1100 and / or a lower layer of the touch display screen 1105.
  • a user's holding signal to the terminal 1100 can be detected, and the processor 1101 can perform left-right hand recognition or quick operation according to the holding signal collected by the pressure sensor 1113.
  • the processor 1101 controls the operable controls on the UI interface according to the user's pressure operation on the touch display screen 1105.
  • the operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.
  • the fingerprint sensor 1114 is used to collect a user's fingerprint, and the processor 1101 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 1114, or the fingerprint sensor 1114 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as trusted, the processor 1101 authorizes the user to perform related sensitive operations, such as unlocking the screen, viewing encrypted information, downloading software, paying, and changing settings.
  • the fingerprint sensor 1114 may be disposed on the front, back, or side of the terminal 1100. When a physical button or a manufacturer's logo is set on the terminal 1100, the fingerprint sensor 1114 can be integrated with the physical button or the manufacturer's logo.
  • the optical sensor 1115 is used to collect the ambient light intensity.
  • the processor 1101 may control the display brightness of the touch display screen 1105 according to the ambient light intensity collected by the optical sensor 1115. Specifically, when the ambient light intensity is high, the display brightness of the touch display screen 1105 is increased; when the ambient light intensity is low, the display brightness of the touch display screen 1105 is reduced.
  • the processor 1101 may also dynamically adjust the shooting parameters of the camera component 1106 according to the ambient light intensity collected by the optical sensor 1115.
  • the proximity sensor 1116 also called a distance sensor, is usually disposed on the front panel of the terminal 1100.
  • the proximity sensor 1116 is used to collect the distance between the user and the front of the terminal 1100.
  • the processor 1101 controls the touch display screen 1105 to switch from the bright screen state to the closed screen state; when the proximity sensor 1116 detects When the distance between the user and the front of the terminal 1100 gradually becomes larger, the processor 1101 controls the touch display screen 1105 to switch from the rest screen state to the bright screen state.
  • FIG. 11 does not constitute a limitation on the terminal 1100, and may include more or fewer components than shown in the figure, or combine certain components, or adopt different component arrangements.
  • FIG. 12 shows an internal structure diagram of an electronic device in one embodiment.
  • the electronic device includes the computer device including a processor, a memory, a network interface, and an input device connected through a system bus.
  • the memory includes a non-volatile storage medium and an internal memory.
  • the non-volatile storage medium of the electronic device stores an operating system and can also store computer-readable instructions.
  • the processor can implement a camera attitude tracking method.
  • the internal memory may also store computer-readable instructions.
  • the processor may cause the processor to perform a camera attitude tracking method.
  • the input device may be a touch layer covered on a display screen, or a button, a trackball, or a touchpad provided on a computer equipment housing, or an external keyboard, a touchpad, or a mouse.
  • FIG. 12 is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the electronic device to which the solution of the present application is applied.
  • the specific electronic device may include There are more or fewer parts than shown in the figure, or some parts are combined, or there are different parts arrangements.
  • the camera attitude tracking device provided in this application may be implemented in the form of a computer-readable instruction, and the computer-readable instruction may run on an electronic device as shown in FIG. 12.
  • the memory of the electronic device may store various program modules constituting the camera attitude tracking device, for example, a receiving module 1020, an image acquisition module 1040, a variation acquisition module 1060, and an attitude calculation module 1080 shown in FIG.
  • the computer-readable instructions constituted by each program module cause the processor to execute the steps in the camera attitude tracking method of each embodiment of the present application described in this specification.
  • An embodiment of the present application provides a computer-readable storage medium.
  • the storage medium stores computer-readable instructions.
  • the computer-readable instructions are loaded and executed by a processor to implement the camera attitude tracking method shown in FIG. 6.
  • Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory.
  • Volatile memory can include random access memory (RAM) or external cache memory.
  • RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous chain (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
  • SRAM static RAM
  • DRAM dynamic RAM
  • SDRAM synchronous DRAM
  • DDRSDRAM dual data rate SDRAM
  • ESDRAM enhanced SDRAM
  • SLDRAM synchronous chain (Synchlink) DRAM
  • SLDRAM synchronous chain (Synchlink) DRAM
  • Rambus direct RAM
  • DRAM direct memory bus dynamic RAM
  • RDRAM memory bus dynamic RAM

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Computer Graphics (AREA)
  • Computer Hardware Design (AREA)
  • Software Systems (AREA)
  • Human Computer Interaction (AREA)
  • Studio Devices (AREA)
  • User Interface Of Digital Computer (AREA)
  • Image Analysis (AREA)

Abstract

本申请实施例公开了一种相机姿态追踪方法、装置、设备及系统，属于增强现实领域。所述方法包括：接收第一设备发送的初始标记图像和初始姿态参数；获取所述相机采集的当前图像；获取所述当前图像相对于所述初始标记图像的相机姿态变化量；根据所述初始姿态参数和所述相机姿态变化量，计算得到所述相机采集所述当前图像时的当前相机姿态参数。

Description

相机姿态追踪方法、装置、设备及系统
本申请要求于2018年05月22日提交中国专利局，申请号为201810495056.7，申请名称为“相机姿态追踪方法、装置、设备及存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及增强现实领域，特别涉及一种相机姿态追踪方法、装置、设备及存储介质。
背景技术
在AR(Augmented Reality，增强显示)领域，搭载相机的设备需要在没有环境先验信息的情况下，于运动过程中追踪相机在现实环境中的运动姿态。在相机姿态追踪过程中，设备会对现实环境建立三维坐标系。
相关技术中针对单个设备的相机姿态追踪已经存在成熟的解决方案。示意性的，设备以初始采集到的第一个图像作为标记图像，对后续图像相对于标记图像进行特征点追踪，根据特征点追踪结果计算后续图像和标记图像之间的单应性矩阵，对单应性矩阵进行分解，从而根据分解结果得到相机在采集后续图像时的相机姿态。
在AR应用程序中存在多个设备协同的多人模式，由于每个设备的硬件参数以及相机拍摄角度各不相同，每个设备对现实环境中所建立的三维坐标系是不同的，从而导致每个设备追踪到的相机姿态是不同的。若需要两个设备基于相同的基准进行相机姿态追踪，则需要将第一设备上构建的环境地图(包含大量图像帧和三维点)同步至第二设备上，同步过程需要传输大量的数据量，不利于采用移动网络的电子设备上的实施。
发明内容
根据本申请提供的各种实施例，提供一种相机姿态追踪方法、装置、设备及存储介质。
根据本申请的一个方面，提供了一种相机姿态追踪方法，应用于设置有相 机的第二设备中，所述方法包括：
所述第二设备接收第一设备发送的初始标记图像和初始姿态参数；
所述第二设备获取所述相机采集的当前图像；
所述第二设备获取所述当前图像相对于所述初始标记图像的相机姿态变化量；
所述第二设备根据所述初始姿态参数和所述相机姿态变化量，计算得到所述相机采集所述当前图像时的当前相机姿态参数。
根据本申请的另一方面，提供了一种相机姿态追踪装置，应用于设置有相机的第二设备中，所述装置包括：
数据接收模块，用于接收第一设备发送的初始标记图像和初始姿态参数；
图像获取模块，用于获取所述相机采集的当前图像；
变化量获取模块，用于获取所述当前图像相对于所述初始标记图像的相机姿态变化量；
姿态计算模块，用于根据所述初始姿态参数和所述相机姿态变化量，计算得到所述相机采集所述当前图像时的当前相机姿态参数。
根据本申请的另一方面，提供了一种电子设备，所述电子设备包括存储器和处理器,所述存储器中存储有计算机可读指令，所述计算机可读指令被所述处理器执行时，使得所述处理器执行如下步骤：
接收第一设备发送的初始标记图像和初始姿态参数；
获取所述相机采集的当前图像；
获取所述当前图像相对于所述初始标记图像的相机姿态变化量；
根据所述初始姿态参数和所述相机姿态变化量，计算得到所述相机采集所述当前图像时的当前相机姿态参数。
本申请实施例提供了一种计算机可读存储介质，所述计算机可读指令被一个或多个处理器执行时，使得一个或多个处理器执行如下步骤：
接收第一设备发送的初始标记图像和初始姿态参数；
获取所述相机采集的当前图像；
获取所述当前图像相对于所述初始标记图像的相机姿态变化量；
根据所述初始姿态参数和所述相机姿态变化量，计算得到所述相机采集所述当前图像时的当前相机姿态参数。
本申请的一个或多个实施例的细节在下面的附图和描述中提出。本申请的 其它特征、目的和优点将从说明书、附图以及权利要求书变得明显。
附图说明
为了更清楚地说明本申请实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1是本申请一个示例性实施例提供的AR应用场景的场景示意图；
图2是本申请一个示例性实施例提供的AR应用场景的场景示意图；
图3是本申请一个示例性实施例提供的AR系统的结构框图；
图4是本申请另一个示例性实施例提供的AR系统的结构框图；
图5是本申请另一个示例性实施例提供的电子设备的结构框图；
图6是本申请一个示例性实施例提供的相机姿态追踪方法的流程图；
图7是本申请一个示例性实施例提供的Anchor-Switching AR System算法的原理示意图；
图8是本申请另一个示例性实施例提供的相机姿态追踪方法的流程图；
图9是本申请另一个示例性实施例提供的相机姿态追踪方法的流程图；
图10是本申请一个示例性实施例提供的相机姿态追踪装置的框图；
图11是本申请一个示例性实施例提供的电子设备的框图。
图12示出了本申请一个实施例中电子设备的内部结构图。
具体实施方式
为了使本申请的目的、技术方案及优点更加清楚明白，以下结合附图及实施例，对本申请进行进一步详细说明。应当理解，此处所描述的具体实施例仅仅用以解释本申请，并不用于限定本申请。基于本申请中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
首先对本申请涉及的若干个名词进行简介:
AR：一种在相机采集图像的过程中，实时地计算相机在现实世界(或称三维世界、真实世界)中的相机姿态参数，根据该相机姿态参数在相机采集的图 像上添加虚拟元素的技术。虚拟元素包括但不限于：图像、视频和三维模型。AR技术的目标是在屏幕上把虚拟世界套接在现实世界上进行互动。该相机姿态参数包括旋转矩阵和位移向量，旋转矩阵用于表征相机在现实世界中发生的旋转角度，位移向量用于表征相机在现实世界中发生的位移距离。
例如，参见图1和参见图2，设备在相机拍摄到的图像中添加了一个虚拟人物形象。随着相机在现实世界中的运动，相机拍摄到的图像会发生变化，虚拟人物的拍摄方位也发生变化，模拟出了虚拟人物在图像中静止不动，而相机随着位置和姿态的变化同时拍摄图像和虚拟人物的效果，为用户呈现了一幅真实立体的画面。
Anchor-Switching AR System：是基于连接多个标记图像(Anchor)的相机姿态追踪来确定在自然场景下的相机姿态参数，进而根据相机姿态参数在相机采集的图像上叠加虚拟世界的AR系统。
IMU(Inertial Measurement Unit，惯性测量单元)：是用于测量物体的三轴姿态角(或角速率)以及加速度的装置。一般的，一个IMU包含了三个单轴的加速度计和三个单轴的陀螺，加速度计用于检测物体在三维坐标系中每个坐标轴上的加速度信号，进而计算得到位移向量(可能会跟随时间的增加累积误差)；而陀螺用于检测物体在三维坐标系中的旋转矩阵。可选地，IMU包括陀螺仪、加速度计和地磁传感器。
示意性的，三维坐标系的建立方式为：1、X轴使用向量积Y*Z来定义，在X轴在设备当前的位置上，沿与地面相切的方向指向东方；2、Y轴在设备当前的位置上，沿与地面相切的方向指向地磁场的北极；3、Z轴指向天空并垂直于地面。
重定位：在基于第i个标记图像的相机姿态追踪过程中，若目标图像相对于第i个标记图像进行特征点追踪失败，无法顺利进行相机姿态追踪过程时，将目标图像相对于其它图像(初始标记图像或其它有代表性的关键帧图像)重新进行特征点追踪，以便在特征点追踪成功时，根据重新得到的追踪过程实现相机采集对当前图像时的相机姿态进行定位的技术。
图3示出了本申请一个示例性实施例提供的AR系统的结构框图。该AR系统包括：第一设备120和第二设备140。
第一设备120是诸如智能手机、平板电脑、安装有智能操作系统的MP3播 放器(Moving Picture Experts Group Audio Layer III，动态影像专家压缩标准音频层面3)、MP4(Moving Picture Experts Group Audio Layer IV，动态影像专家压缩标准音频层面4)播放器、便携式游戏终端之类的电子设备。第一设备120上运行有目标AR程序，该目标AR程序可以是多人协同使用的AR程序。该目标AR程序的程序类型包括但不限于从AR教学程序、AR游戏程序、AR翻译程序、AR社交程序、AR运动程序和AR驾驶辅助程序组成的群组中选取的至少一种。
第一设备120通过数据线、有线网络、蓝牙或WIFI(WIreless Fidelity，无线保真网络)与第二设备140相连。也即，第一设备120和第二设备140之间可以直接进行数据通信。
第二设备140是诸如智能手机、平板电脑、安装有智能操作系统的MP3播放器(Moving Picture Experts Group Audio Layer III，动态影像专家压缩标准音频层面3)、MP4(Moving Picture Experts Group Audio Layer IV，动态影像专家压缩标准音频层面4)播放器、便携式游戏终端之类的电子设备。第二设备140中也安装有目标AR程序。
可选地，第一设备120和第二设备140上安装的目标AR程序可以是相同的，也可以是不同的。在一些实施例中，第一设备120和第二设备140之间没有主次之分；在另一些实施例中，第一设备120是主设备，第二设备140是从设备。
本领域技术人员还可以预见到，该AR系统中的设备数量可以为两个以上，当存在多个设备时，第一设备120和第二设备140可以是多个设备中的任意两个设备，本申请对此不加以限定。
图4示出了本申请另一个示例性实施例提供的AR系统的结构框图。该AR系统包括：第一设备120、服务器160和第二设备140。
与图1不同的是，本实施例中的AR系统增加了服务器160。其中：
第一设备120通过网络与服务器160相连。该网络可以是无线网络或有线网络。第一设备120上安装和运行有目标AR程序。
第二设备140通过网络与服务器160相连。该网络可以是无线网络或有线网络。第二设备120上也安装和运行有目标AR程序。
服务器160可以是一台服务器、多台服务器构成的服务器集群、云计算中心、虚拟化计算中心或分布式计算中心。服务器160可以用来提供目标AR程序的后台服务。
在一些实施例中，第一设备120和第二设备140是服务器的“客户机”；在另一些实施例中，第一设备120和第二设备140与服务器之间采用分布式计算策略进行协同计算，本实施例对此不加以限定。
图5示出了本申请一个示例性实施例提供的电子设备的结构框图。该电子设备可以是上述的第一设备120和/或第二设备140。该电子设备包括：处理器320、存储器340、相机360和IMU 380。
处理器320包括一个或多个处理核心，比如4核心处理器、8核心处理器等。处理器320用于执行存储器340中存储的指令、代码、代码片段和程序中的至少一种。
处理器320与存储器340电性相连。可选地，处理器320通过总线与存储器340相连。存储器340存储有一个或多个指令、代码、代码片段和/或程序。该指令、代码、代码片段和/或程序在被处理器320执行时，用于实现如下实施例中提供的相机姿态追踪方法。
处理器320还与相机360电性相连。可选地，处理器320通过总线与相机360相连。相机360是具有图像采集能力的传感器件。相机360还可称为摄像头、感光器件等其它名称。相机360具有连续采集图像或多次采集图像的能力。可选地，相机360设置在电子设备的内部或设备外部。
可选地，处理器320还与IMU380电性相连。IMU380用于每隔预定时间间隔采集相机的位姿向量，并记录每组位姿参数在采集时的时间戳。相机的位姿参数包括：位移向量和旋转矩阵。其中，IMU380采集的旋转矩阵相对准确，采集的位移向量受实际环境可能会有较大的误差。
可选地，该电子设备的具体形式可以是智能手机、平板电脑、AR眼镜、AR头盔、便携式笔记本电脑、电子书阅读器、MP3、MP4、掌上游戏终端等等。本申请实施例对此不加以限定。
图6示出本申请一个示例性实施例提供的相机姿态追踪方法的流程图。该方法可以用于如图1或图2所示的AR系统中。该方法包括：
步骤401，第一设备启动AR场景的初始化过程；
第一设备和第二设备中运行有支持多人模式的目标AR程序。
在目标AR程序启动后，比如一个AR场景或一局AR游戏开始后，第一设 备进行AR场景的初始化过程。可选地，该初始化过程包括：
1、第一设备通过内置的相机采集现实环境中的一帧帧图像；
第一设备内设置有第一相机，在目标AR程序启动后，第一相机不断采集现实环境中的一帧帧图像，然后将图像发给处理器进行处理。
2、第一设备将第一帧图像(或用户指定的图像)作为初始标记图像；
第一设备将第一帧图像作为初始标记图像(英文：born-image)，或者，第一设备将用户指定的图像作为初始标记图像。
3、第一设备接收用户设定的初始姿态参数；
可选地，初始姿态参数(英文：born-pose)是由用户设定的，或者说，由用户通过目标AR程序设定的。该初始姿态参数的设定过程可以是用户显式感知的，也可以是用户显式不感知的。
比如，目标AR程序是在现实世界中叠加虚拟玩偶的游戏程序，用户控制目标AR程序在相机采集的图像中叠加一个虚拟玩偶时，将叠加虚拟玩偶时的图像设定为初始标记图像，将叠加虚拟玩偶时的姿态参数设定为初始姿态参数，而该设定过程是用户不感知的。
4、第一设备根据初始标记图像和初始姿态参数进行相机姿态追踪过程。
第一设备根据初始标记图像和初始姿态参数，进行相机姿态追踪过程。该相机姿态追踪过程可以采用与单机形式相同或相似的相机姿态追踪技术。
步骤402，第一设备记录初始标记图像和初始姿态参数；
步骤403，第一设备向第二设备发送初始标记图像和初始姿态参数；
可选地，第一设备和第二设备安装有相同的目标AR程序。或者，第一设备和第二设备安装有相同的目标AR程序，且两个目标AR程序之间满足预设条件。以两个目标AR程序登录有各自的帐号为例，该预设条件包括但不限于：从两个用户帐号是好友关系、两个用户帐号处于同一AR场景、两个用户帐号处于同一AR模式和两个用户帐号处于同一个AR游戏房间组成的群组中选取的至少一种。
在一些实施例中，第一设备和第二设备之间直接建立有目标数据通道，第一设备通过目标数据通道向第二设备发送初始标记图像和初始姿态参数。目标数据通道包括：从WIFI、蓝牙、有线网络和数据线组成的群组中选取的至少一种。
在另一些实施例中，第一设备和第二设备之间的数据转发需要经过服务器 的中转，第一设备先将初始标记图像和初始姿态参数发送至服务器，服务器将初始标记图像和初始姿态参数转发给第二设备。
步骤404，第二设备接收第一设备发送的初始标记图像和初始姿态参数；
第二设备通过目标数据通道或者服务器接收第一设备发送的初始标记图像和初始姿态参数。
其中，初始标记图像是第一设备在相机姿态追踪过程中的第一个标记图像，初始姿态参数是与第一个标记图像对应的相机姿态参数。初始标记图像和初始姿态参数均由第一设备产生的数据，并非第二设备产生的数据。
步骤405，第二设备获取相机采集的当前图像；
第二设备内设置有第二相机，在目标AR程序启动后，第二相机不断采集现实环境中的一帧帧图像，然后将图像发给处理器进行处理。
步骤406，第二设备获取当前图像相对于初始标记图像的相机姿态变化量；
第二设备将自身采集的当前图像相对于第一设备发送的初始标记图像进行相机姿态追踪，计算得到当前图像相对于初始标记图像的相机姿态变化量。
需要说明的是，第二设备的相机并未到初始标记图像，本步骤是将初始标记图像假设为第二设备采集的图像进行相机姿态追踪。因为在典型的AR场景中，第一设备和第二设备会对同一现实场景进行取景拍摄，比如对同一个桌面进行取景拍摄，所以两者拍摄到的图像存在一定的相关性，可以将第一设备采集的初始标记图像假定为第二设备采集的图像进行使用。
步骤407，第二设备根据初始姿态参数和相机姿态变化量，计算得到相机采集当前图像时的当前相机姿态参数。
可选地，第二设备按照如下公式计算得到第二相机采集当前图像时的相机姿态参数R和T：
Figure PCTCN2019083437-appb-000001
其中，R cf是当前图像相对于初始标记图像的相机姿态变化量中的旋转矩阵，T cf是当前图像相对于初始标记图像的相机姿态变化量中的位移向量，R born是初始姿态参数中的旋转矩阵，T born是初始姿态参数中的位移向量。
综上所述，本实施例提供的相机姿态追踪方法，通过第二设备接收第一设备发送的初始标记图像和初始姿态参数，计算第二设备的相机采集的当前图像相对于初始标记图像的相机姿态变化量，根据初始姿态参数和相机姿态变化量，计算得到相机采集当前图像时的相机姿态参数，使得第二设备将自身采集的图 像相对于第一设备的初始标记图像进行相机姿态追踪，也即第一设备和第二设备使用相同的初始标记图像和初始姿态参数进行相机姿态追踪，保证两者的相机姿态追踪结果是相同的，从而解决了若需要将第一设备上构建的环境地图同步至第二设备上，需要传输大量的数据量的问题。
由于第一设备和第二设备之间仅需要传输一个初始标记图像和一组初始姿态参数，因此两个设备之间需要交互的数据量非常小，适用于多个移动终端之间的AR场景下的实时同步，不会占用太多网络通信带宽。
需要说明的是，上述实施例中由第一设备执行的步骤可以单独实现成为第一设备侧的相机姿态追踪方法，由第二设备执行的步骤可以单独实现成为第二设备侧的相机姿态追踪方法。
在基于图6的可选实施例中，第一设备采用Anchor-Switching AR System算法进行相机姿态追踪。Anchor-Switching AR System算法在确定相机姿态的过程中，将第一相机的整个运动过程划分为至少两段追踪过程进行追踪，每段追踪过程对应各自的标记图像。具体地，首先基于初始标记图像(也称第1个标记图像)对相机拍摄到的当前图像进行相机姿态追踪，相机姿态追踪过程可以采用当第i个标记图像对应的追踪过程中，若当前图像相对于第i个标记图像的追踪效果差于预设条件(比如能够匹配到的特征点少于预设阈值)时，将当前图像的上一个图像确定为第i+1个标记图像，开启第i+1段追踪过程。其中，i为正整数。示意性的参考图7。在现实世界中存在物体50，设置有第一相机的第一设备120被用户手持进行移动，在移动过程中拍摄得到包括物体50的多帧图像1-6。设备将图像1确定为初始标记图像(英文：born-image或born-anchor)并记录初始位姿参数(英文：born-pose)，该初始位姿参数可以是用户设定的初始位姿参数，IMU采集的相机采集图像1时的位姿参数。然后将图像2相对于图像1进行特征点追踪，根据初始位姿参数和特征点追踪结果计算出相机在拍摄图像2时的位姿参数；将图像3相对于图像1进行特征点追踪，根据初始位姿参数和特征点追踪结果计算出相机在拍摄图像3时的位姿参数；将图像4相对于图像1进行特征点追踪，根据初始位姿参数和特征点追踪结果计算出相机在拍摄图像4时的位姿参数。
然后，将图像5相对于图像1进行特征点追踪，如果特征点追踪效果差于预设条件(比如匹配的特征点数量较少)，则将图像4确定为第2个标记图像， 将图像5相对于图像4进行特征点追踪，计算出相机在拍摄图像4至图像5之间的位移变化量，再结合相机在拍摄图像4至图像1之间的位移变化量以及初始位姿参数，计算出相机在拍摄图像5时的位姿参数。然后再将图像6相对于图像4进行特征点追踪，以此类推，若当前图像的特征点追踪效果变差时，即可将当前图像的上一帧图像确定为新的标记图像，切换新的标记图像后重新进行特征点追踪。
可选地，特征点追踪可以采用基于视觉里程计原理的算法，比如特征点法或直接法。
在基于图6的可选实施例中，第二设备在接收到初始标记图像和初始姿态参数后，将第二设备内的第二相机采集的当前图像相对于初始标记图像进行相机姿态追踪。视当前图像的图像采集时机和/或图像内容不同，本步骤可以采用如下两种方式中的至少一种方式实现。
第一，第二设备将当前图像相对于初始标记图像进行重定位，得到相机姿态变化量。
第二，第二设备获取第i个标记图像相对于初始标记图像的第一姿态变化量，i＞1；将当前图像相对于第i个标记图像进行特征点追踪，得到第二姿态变化量；根据第一姿态变化量和第二姿态变化量，计算得到相机姿态变化量。
以下采用图8对上述第一种实现方式进行介绍，采用图9对上述第二实现方式进行介绍。
在基于图6提供的可选实施例中，第二设备采用第一种方式获取当前图像相对于初始标记图像的相机姿态变化量。在当前图像是第二设备中的第二相机采集的前几帧图像(或者相机姿态追踪过程中发生丢失)时，由于此时第二设备还没有指定第一个标记图像，因此第二设备将接收到的初始标记图像设定为本机的第1个标记图像，第二设备将当前图像相对于初始标记图像进行重定位，得到相机姿态变化量。可选地，若当前图像相对于初始标记图像重定位成功，则将当前图像切换(或设置)为第二个标记图像，将当前图像之后采集的图像相对于第二个标记图像进行特征点追踪。步骤406包括但不限于如下步骤：
步骤406a，第二设备获取初始标记图像的初始特征点；
可选地，第一设备向第二设备发送初始标记图像时，还同时发送该初始标 记图像中的初始特征点，第二设备直接读取初始标记图像的初始特征点。初始特征点是从初始标记图像上提取到的特征点，初始特征点可以是多个，比如10-500个。
可选地，若第一设备没有向第二设备发送初始标记图像中的初始特征点，则第二设备提取初始标记图像中的初始特征点。可选地，第一设备提取特征点时采用的特征提取算法可以为FAST(Features from Accelerated Segment Test，加速段测试特征点)检测算法、Shi-Tomasi(史托马西)角点检测算法、Harris Corner Detection(Harris角点检测)算法、SIFT(Scale-Invariant Feature Transform，尺度不变特征转换)算法、ORB(Oriented FAST and Rotated BRIEF，快速特征点提取和描述)算法等。
由于SIFT特征的实时计算难度较大，为了保证实时性，设备可以提取第一个标记图像中的ORB特征点。一个ORB特征点包括FAST角点(Key-point)和BRIER描述子(Binary Robust Independent Elementary Feature Descirptor)两部分。ORB特征的计算速度较快，因此适用于移动设备上实施。
FAST角点是指该ORB特征点在图像中所在的位置。FAST角点主要检测局部像素灰度变化明显的地方，以速度快著称。FAST角点的思想时：如果一个像素与邻域的像素差别较大(过亮或过暗)，则该像素可能是一个角点。
BRIEF描述子是一个二进制表示的向量，该向量按照某种人为设计的方式描述了该关键点周围像素的信息。BRIEF描述子的描述向量由多个0和1组成，这里的0和1编码了FAST角点附近的两个像素的大小关系。
步骤406b，第二设备将当前图像相对于初始标记图像进行特征点追踪，得到与初始特征点匹配的目标特征点；
特征点追踪可采用基于视觉里程计的追踪算法，本申请对此不加以限定。在一个实施例中，特征点追踪采用KLT(Kanade-Lucas)光流追踪算法；在另一个实施例中，特征点追踪采用基于ORB(Oriented FAST and Rotated BRIEF，快速特征点提取和描述)算法提取的ORB特征描述子进行特征点跟踪。本申请对特征点追踪的具体算法不加以限定，特征点追踪过程可以采用特征点法或直接法。
在一个实施例中，第二设备对当前图像进行特征点提取，得到M个候选特征点；然后将M个候选特征点逐一与N个初始特征点进行匹配，确定出至少一组匹配特征点对。每组匹配特征点对包括：一个初始特征点和一个目标特征点。 初始特征点是初始标记图像上的特征点，目标特征点是当前图像上与该初始特征点匹配度最高的候选特征点。
可选地，初始特征点的数量大于或等于目标特征点的数量。比如，初始特征点的数量是450个，目标特征点为320组。
步骤406c，第二设备根据初始特征点和目标特征点，计算初始标记图像和当前图像之间的单应性矩阵；
单应性矩阵描述了相机的两个成像平面之间的映射关系，也即初始标记图像的成像平面和当前图像的成像平面之间的映射关系，若两张图像拍摄到的自然场景(现实环境)中的特征点都落在同一物理平面上，则可以通过单应性矩阵进行相机姿态的运动估计。
步骤406d，第二设备分解单应性矩阵得到相机姿态变化量。
当存在至少四对相匹配的初始特征点和目标特征点时，设备通过ransac对该单应性矩阵进行分解，得到相机姿态变化量。可选地，该相机姿态变化量包括旋转矩阵R cf和平移向量T cf。
在一种可能的实现方式中，若当前图像的重定位成功，则第二设备可以将当前图像切换(或设置)为第2个标记图像，将第二相机在后续采集的其它图像相对于第2个标记图像进行相机姿态追踪。相关原理可以参考图5所示。
综上所述，本实施例提供的相机姿态追踪方法，第二设备通过将当前图像相对于初始标记图像进行重定位计算得到相机姿态变化量，从而能够在尚不存在标记图像或发生丢失时，根据自身采集的当前图像和第一设备采集的初始标记图像构建相机姿态追踪过程的各项信息，采用与第一设备相同的基准进行相机姿态追踪，从而实现多人模式AR场景的相机姿态追踪过程的同步。
在基于图6提供的可选实施例中，第二设备采用第二种方式获取当前图像相对于初始标记图像的相机姿态变化量。若当前图像是第二设备在相机姿态追踪过程进行一段时间后采集的图像，则第二设备已经按照Anchor-Switching AR System算法进行相机姿态追踪了一段时间，此时正在进行基于第i个标记图像的相机姿态追踪过程，i为大于1的整数。步骤406包括但不限于如下步骤，如图9所示：
步骤4061，第二设备获取第i个标记图像相对于初始标记图像的第一姿态变化量，i＞1；
可选地，第i个标记图像是第2个标记图像、第3个标记图像、第4个标记图像等等。也即，第i个标记图像是初始标记图像之后的标记图像。
第i个标记图像是历史过程已经追踪成功的图像，第二设备上缓存有第i个标记图像相对于初始标记图像的第一姿态变化量。第一姿态变化量包括第一旋转矩阵R mf和第一位移向量T mf。
步骤4062，第二设备将当前图像相对于第i个标记图像进行特征点追踪，得到第二姿态变化量；
特征点追踪可采用基于视觉里程计的追踪算法，本申请对此不加以限定。在一个实施例中，特征点追踪采用KLT光流追踪算法；在另一个实施例中，特征点追踪采用基于ORB算法提取的ORB特征描述子进行特征点跟踪。本申请对特征点追踪的具体算法不加以限定，特征点追踪过程可以采用特征点法或直接法。
在一个实施例中，本步骤包括如下子步骤：
1、第二设备将当前图像相对于第i个标记图像上的参考特征点进行特征点追踪，得到与参考特征点匹配的目标特征点；
第二设备在设置第i个标记图像时，会提取第i个标记图像上的N个参考特征点。在将当前图像相对于第i个标记图像上的参考特征点进行特征点追踪时，第二设备对当前图像进行特征点提取，得到M个候选特征点；然后将M个候选特征点逐一与N个参考特征点进行匹配，确定出至少一组匹配特征点对。每组匹配特征点对包括：一个参考特征点和一个目标特征点。参考特征点是第i个标记图像上的特征点，目标特征点是当前图像上与该参考特征点匹配度最高的候选特征点。
可选地，参考特征点的数量大于或等于目标特征点的数量。比如，初始特征点的数量是500个，目标特征点为430组。
2、第二设备根据参考特征点和目标特征点，计算第i个标记图像和当前图像之间的单应性矩阵；
单应性矩阵描述了相机的两个成像平面之间的映射关系，也即初始标记图像的成像平面和当前图像的成像平面之间的映射关系，若两张图像拍摄到的自然场景(现实环境)中的特征点都落在同一物理平面上，则可以通过单应性矩阵进行相机姿态的运动估计。
3，第二设备分解单应性矩阵得到第二姿态变化量；
当存在至少四对相匹配的参考特征点和目标特征点时，设备通过ransac对该单应性矩阵进行分解，得到第二姿态变化量。可选地，该第二姿态变化量包括旋转矩阵R cm和平移向量T cm。
步骤4063，第二设备根据第一姿态变化量和第二姿态变化量，计算得到相机姿态变化量。
第二设备按照如下公式计算得到相机姿态变化量R cf和T cf：
Figure PCTCN2019083437-appb-000002
其中，S mf是第i个标记图像的尺度。
可选地，S mf为预设值，或者，S mf是将初始标记图像上的初始特征点映射至第i个标记图像上得到投影特征点，根据投影特征点的平均深度计算得到第i个图像对应的尺度。
综上所述，本实施例提供的相机姿态追踪方法，第二设备通过将当前图像相对于第i个标记图像进行特征点追踪后，利用第一姿态变化量和第二姿态变化量计算得到相机姿态变化量，根据相机姿态变化量和初始姿态参数计算得到当前相机姿态参数，从而能够使第二设备在已经进行了一段时间的相机姿态追踪时，根据自身采集的当前图像和第一设备采集的初始标记图像构建相机姿态追踪过程的各项信息，采用与第一设备相同的基准进行相机姿态追踪，从而实现多人模式AR场景的相机姿态追踪过程的同步。
在基于图9的可选实施例中，第二设备将第一个标记图像上的初始特征点映射至第i个标记图像上得到投影特征点，根据投影特征点的平均深度计算得到第i个标记图像对应的尺度的过程如下：
1、第二设备获取初始标记图像中的初始特征点的二维坐标；
第二设备预先提取和缓存有初始标记图像中的初始特征点的二维坐标。该二维坐标采用齐次表示。
2、第二设备对初始特征点的二维坐标进行反投影，得到初始特征点在三维空间中的第一三维坐标X born；
第二设备通过如下公式将初始特征点的二维坐标变换至三维空间，得到这些初始特征点在三维空间中的第一三维坐标X born。
Figure PCTCN2019083437-appb-000003
其中，f x、f y、c x、c y为相机的内置参数。初始特征点的二维坐标x born是第一个标记图像上的齐次表示，三维点x born是非齐次表示。假设第一个标记图像的初始深度d为1。
3、第二设备将第一三维坐标X born通过如下公式进行三维旋转平移，得到初始特征点在第i个标记图像上对应的第二三维坐标X current；
X current＝R*X born+T；
其中，R mf是第一位姿变化量中的旋转矩阵，T mf是第一位姿变化量中的位移向量。
4、第二设备将第二三维坐标X current投影至第i个标记图像，得到投影特征点在第i个标记图像中的二维坐标；
通过如下公式将第二三维坐标X current投影至第i个标记图像，得到投影特征点在第i个标记图像中的二维坐标x current：
Figure PCTCN2019083437-appb-000004
其中，f x、f y、c x、c y为第二相机的内置参数。
5、第二设备根据投影特征点在第i个标记图像中的平均深度，计算得到第i个标记图像的尺度s mf。
第一设备和第二设备均假设初始标记图像对应的三维点的深度为1，同时初始姿态参数也相同，因此第一设备和第二设备对现实世界所建立的三维世界必然相同，因此两个设备之间直接实现了三维坐标系的对齐。
本申请实施例提供的相机姿态追踪方法，能够应用于AR领域中实现没有环境先验信息的情况下，于运动过程中追踪相机在现实环境中的运动姿态。典型的应用场景包括：
实时多人同时在线的AR游戏：
多个设备接入同一个游戏或游戏室，第一用户使用其中任意一台设备进行 初始化过程得到初始标记图像和初始姿态参数，然后将初始标记图像和初始姿态参数同步至第二用户使用的其它设备，其它设备基于该初始标记图像和初始姿态参数进行相机姿态追踪。最终实现多个设备基于相同的相机追踪基准，进行实时的多人AR游戏。
基于AR的藏宝游戏：
第一用户使用第一设备在现实世界的地点A埋下宝藏(虚拟宝藏)，其它用户经过现实世界的地点A时，使用第二设备在地点A寻找宝藏，并且可以使用第二设备上的AR程序打开和取走该宝藏。
应该理解的是，本申请各实施例中的各个步骤并不是必然按照步骤标号指示的顺序依次执行。除非本文中有明确的说明，这些步骤的执行并没有严格的顺序限制，这些步骤可以以其它的顺序执行。而且，各实施例中至少一部分步骤可以包括多个子步骤或者多个阶段，这些子步骤或者阶段并不必然是在同一时刻执行完成，而是可以在不同的时刻执行，这些子步骤或者阶段的执行顺序也不必然是依次进行，而是可以与其它步骤或者其它步骤的子步骤或者阶段的至少一部分轮流或者交替地执行。
在一个实施例中，还提供了一种电子设备，该电子设备包括相机姿态追踪装置，相机姿态追踪装置中包括各个模块，每个模块可全部或部分通过软件、硬件或其组合来实现。
以下为本申请的装置实施例，对于装置实施例中未详细描述的技术细节，可以参考上述方法实施例中的相关描述，本申请对此不加以限定。
图10示出了本申请一个示例性实施例提供的相机姿态追踪装置的框图。该装置可以通过软件、硬件或者两者的结合实现成为电子设备(第二设备)的全部或一部分，该装置包括：
接收模块1020，用于接收第一设备发送的初始标记图像和初始姿态参数；
图像获取模块1040，用于获取所述相机采集的当前图像；
变化量获取模块1060，用于获取所述当前图像相对于所述初始标记图像的相机姿态变化量；
姿态计算模块1080，用于根据所述初始姿态参数和所述相机姿态变化量，计算得到所述相机采集所述当前图像时的当前相机姿态参数。
在一个可选的实施例中，所述变化量获取模块1060，用于将所述当前图像相对于所述初始标记图像进行重定位，得到所述相机姿态变化量；和/或，所述 变化量获取模块1060，用于获取第i个标记图像相对于所述初始标记图像的第一姿态变化量，i＞1；将所述当前图像相对于所述第i个标记图像进行特征点追踪，得到第二姿态变化量；根据所述第一姿态变化量和所述第二姿态变化量，计算得到所述相机姿态变化量。
在一个可选的实施例中，所述变化量获取模块1060，用于获取所述初始标记图像的初始特征点；将所述当前图像相对于所述初始标记图像进行特征点追踪，得到与所述初始特征点匹配的目标特征点；根据所述初始特征点和所述目标特征点，计算所述初始标记图像和所述当前图像之间的单应性矩阵；分解所述单应性矩阵得到所述相机姿态变化量。
在一个可选的实施例中，设所述第一姿态变化量包括第一旋转矩阵R mf和第一位移向量T mf，所述第二姿态变化量包括第二旋转矩阵R cm和第二位移向量T cm，
所述变化量获取模块1060，用于按照如下公式计算得到所述相机姿态变化量R cf和T cf：
Figure PCTCN2019083437-appb-000005
其中，S mf是所述第i个标记图像的尺度。
在一个可选的实施例中，所述姿态计算模块1080，用于按照如下公式计算得到所述相机姿态参数R和T：
Figure PCTCN2019083437-appb-000006
其中，R cf是所述相机姿态变化量中的旋转矩阵，T cf是所述相机姿态变化量中的位移向量，R born是所述初始姿态参数中的旋转矩阵，T born是所述初始姿态参数中的位移向量。
在一个可选的实施例中，所述接收模块1020，用于接收所述第一设备通过目标数据通道发送的所述初始标记图像和所述初始姿态参数，所述目标数据通道包括：从无线保真网络、蓝牙、有线网络和数据线组成的群组中选取的至少一种；和/或，所述接收模块1020，用于接收服务器发送的所述初始标记图像和所述初始姿态参数，所述初始标记图像和所述初始姿态参数是所述第一设备发送给所述服务器的。
需要说明的是：上述实施例提供的相机姿态追踪装置在进行相机姿态追踪时，仅以上述各功能模块的划分进行举例说明，实际应用中，可以根据需要而 将上述功能分配由不同的功能模块完成，即将设备的内部结构划分成不同的功能模块，以完成以上描述的全部或者部分功能。另外，上述实施例提供的相机姿态追踪装置与相机姿态追踪方法实施例属于同一构思，其具体实现过程详见方法实施例。
图11示出了本申请一个示例性实施例提供的终端1100的结构框图。该终端1100可以是：智能手机、平板电脑、MP3播放器(Moving Picture Experts Group Audio Layer III，动态影像专家压缩标准音频层面3)、MP4(Moving Picture Experts Group Audio Layer IV，动态影像专家压缩标准音频层面4)播放器、笔记本电脑或台式电脑。终端1100还可能被称为用户设备、便携式终端、膝上型终端、台式终端等其他名称。
通常，终端1100包括有：处理器1101和存储器1102。
处理器1101可以包括一个或多个处理核心，比如4核心处理器、8核心处理器等。处理器1101可以采用DSP(Digital Signal Processing，数字信号处理)、FPGA(Field－Programmable Gate Array，现场可编程门阵列)、PLA(Programmable Logic Array，可编程逻辑阵列)中的至少一种硬件形式来实现。处理器1101也可以包括主处理器和协处理器，主处理器是用于对在唤醒状态下的数据进行处理的处理器，也称CPU(Central Processing Unit，中央处理器)；协处理器是用于对在待机状态下的数据进行处理的低功耗处理器。在一些实施例中，处理器1101可以在集成有GPU(Graphics Processing Unit，图像处理器)，GPU用于负责显示屏所需要显示的内容的渲染和绘制。一些实施例中，处理器1101还可以包括AI(Artificial Intelligence，人工智能)处理器，该AI处理器用于处理有关机器学习的计算操作。
存储器1102可以包括一个或多个计算机可读存储介质，该计算机可读存储介质可以是非暂态的。存储器1102还可包括高速随机存取存储器，以及非易失性存储器，比如一个或多个磁盘存储设备、闪存存储设备。在一些实施例中，存储器1102中的非暂态的计算机可读存储介质用于存储至少一个指令，该至少一个指令用于被处理器1101所执行以实现本申请中方法实施例提供的相机姿态追踪方法。
在一些实施例中，终端1100还可选包括有：外围设备接口1103和至少一个外围设备。处理器1101、存储器1102和外围设备接口1103之间可以通过总 线或信号线相连。各个外围设备可以通过总线、信号线或电路板与外围设备接口1103相连。具体地，外围设备包括：射频电路1104、触摸显示屏1105、摄像头1106、音频电路1107、定位组件1108和电源1109中的至少一种。
外围设备接口1103可被用于将I/O(Input/Output，输入/输出)相关的至少一个外围设备连接到处理器1101和存储器1102。在一些实施例中，处理器1101、存储器1102和外围设备接口1103被集成在同一芯片或电路板上；在一些其他实施例中，处理器1101、存储器1102和外围设备接口1103中的任意一个或两个可以在单独的芯片或电路板上实现，本实施例对此不加以限定。
射频电路1104用于接收和发射RF(Radio Frequency，射频)信号，也称电磁信号。射频电路1104通过电磁信号与通信网络以及其他通信设备进行通信。射频电路1104将电信号转换为电磁信号进行发送，或者，将接收到的电磁信号转换为电信号。可选地，射频电路1104包括：天线系统、RF收发器、一个或多个放大器、调谐器、振荡器、数字信号处理器、编解码芯片组、用户身份模块卡等等。射频电路1104可以通过至少一种无线通信协议来与其它终端进行通信。该无线通信协议包括但不限于：万维网、城域网、内联网、各代移动通信网络(2G、3G、4G及5G)、无线局域网和/或WiFi(Wireless Fidelity，无线保真)网络。在一些实施例中，射频电路1104还可以包括NFC(Near Field Communication，近距离无线通信)有关的电路，本申请对此不加以限定。
显示屏1105用于显示UI(User Interface，用户界面)。该UI可以包括图形、文本、图标、视频及其它们的任意组合。当显示屏1105是触摸显示屏时，显示屏1105还具有采集在显示屏1105的表面或表面上方的触摸信号的能力。该触摸信号可以作为控制信号输入至处理器1101进行处理。此时，显示屏1105还可以用于提供虚拟按钮和/或虚拟键盘，也称软按钮和/或软键盘。在一些实施例中，显示屏1105可以为一个，设置终端1100的前面板；在另一些实施例中，显示屏1105可以为至少两个，分别设置在终端1100的不同表面或呈折叠设计；在再一些实施例中，显示屏1105可以是柔性显示屏，设置在终端1100的弯曲表面上或折叠面上。甚至，显示屏1105还可以设置成非矩形的不规则图形，也即异形屏。显示屏1105可以采用LCD(Liquid Crystal Display，液晶显示屏)、OLED(Organic Light-Emitting Diode,有机发光二极管)等材质制备。
摄像头组件1106用于采集图像或视频。可选地，摄像头组件1106包括前置摄像头和后置摄像头。通常，前置摄像头设置在终端的前面板，后置摄像头 设置在终端的背面。在一些实施例中，后置摄像头为至少两个，分别为主摄像头、景深摄像头、广角摄像头、长焦摄像头中的任意一种，以实现主摄像头和景深摄像头融合实现背景虚化功能、主摄像头和广角摄像头融合实现全景拍摄以及VR(Virtual Reality，虚拟现实)拍摄功能或者其它融合拍摄功能。在一些实施例中，摄像头组件1106还可以包括闪光灯。闪光灯可以是单色温闪光灯，也可以是双色温闪光灯。双色温闪光灯是指暖光闪光灯和冷光闪光灯的组合，可以用于不同色温下的光线补偿。
音频电路1107可以包括麦克风和扬声器。麦克风用于采集用户及环境的声波，并将声波转换为电信号输入至处理器1101进行处理，或者输入至射频电路1104以实现语音通信。出于立体声采集或降噪的目的，麦克风可以为多个，分别设置在终端1100的不同部位。麦克风还可以是阵列麦克风或全向采集型麦克风。扬声器则用于将来自处理器1101或射频电路1104的电信号转换为声波。扬声器可以是传统的薄膜扬声器，也可以是压电陶瓷扬声器。当扬声器是压电陶瓷扬声器时，不仅可以将电信号转换为人类可听见的声波，也可以将电信号转换为人类听不见的声波以进行测距等用途。在一些实施例中，音频电路1107还可以包括耳机插孔。
定位组件1108用于定位终端1100的当前地理位置，以实现导航或LBS(Location Based Service，基于位置的服务)。定位组件1108可以是基于美国的GPS(Global Positioning System，全球定位系统)、中国的北斗系统或俄罗斯的伽利略系统的定位组件。
电源1109用于为终端1100中的各个组件进行供电。电源1109可以是交流电、直流电、一次性电池或可充电电池。当电源1109包括可充电电池时，该可充电电池可以是有线充电电池或无线充电电池。有线充电电池是通过有线线路充电的电池，无线充电电池是通过无线线圈充电的电池。该可充电电池还可以用于支持快充技术。
在一些实施例中，终端1100还包括有一个或多个传感器1110。该一个或多个传感器1110包括但不限于：加速度传感器1111、陀螺仪传感器1112、压力传感器1113、指纹传感器1114、光学传感器1115以及接近传感器1111。
加速度传感器1111可以检测以终端1100建立的坐标系的三个坐标轴上的加速度大小。比如，加速度传感器1111可以用于检测重力加速度在三个坐标轴上的分量。处理器1101可以根据加速度传感器1111采集的重力加速度信号， 控制触摸显示屏1105以横向视图或纵向视图进行用户界面的显示。加速度传感器1111还可以用于游戏或者用户的运动数据的采集。
陀螺仪传感器1112可以检测终端1100的机体方向及转动角度，陀螺仪传感器1112可以与加速度传感器1111协同采集用户对终端1100的3D动作。处理器1101根据陀螺仪传感器1112采集的数据，可以实现如下功能：动作感应(比如根据用户的倾斜操作来改变UI)、拍摄时的图像稳定、游戏控制以及惯性导航。
压力传感器1113可以设置在终端1100的侧边框和/或触摸显示屏1105的下层。当压力传感器1113设置在终端1100的侧边框时，可以检测用户对终端1100的握持信号，由处理器1101根据压力传感器1113采集的握持信号进行左右手识别或快捷操作。当压力传感器1113设置在触摸显示屏1105的下层时，由处理器1101根据用户对触摸显示屏1105的压力操作，实现对UI界面上的可操作性控件进行控制。可操作性控件包括按钮控件、滚动条控件、图标控件和菜单控件中的至少一种。
指纹传感器1114用于采集用户的指纹，由处理器1101根据指纹传感器1114采集到的指纹识别用户的身份，或者，由指纹传感器1114根据采集到的指纹识别用户的身份。在识别出用户的身份为可信身份时，由处理器1101授权该用户执行相关的敏感操作，该敏感操作包括解锁屏幕、查看加密信息、下载软件、支付及更改设置等。指纹传感器1114可以被设置终端1100的正面、背面或侧面。当终端1100上设置有物理按键或厂商Logo时，指纹传感器1114可以与物理按键或厂商Logo集成在一起。
光学传感器1115用于采集环境光强度。在一个实施例中，处理器1101可以根据光学传感器1115采集的环境光强度，控制触摸显示屏1105的显示亮度。具体地，当环境光强度较高时，调高触摸显示屏1105的显示亮度；当环境光强度较低时，调低触摸显示屏1105的显示亮度。在另一个实施例中，处理器1101还可以根据光学传感器1115采集的环境光强度，动态调整摄像头组件1106的拍摄参数。
接近传感器1116，也称距离传感器，通常设置在终端1100的前面板。接近传感器1116用于采集用户与终端1100的正面之间的距离。在一个实施例中，当接近传感器1116检测到用户与终端1100的正面之间的距离逐渐变小时，由处理器1101控制触摸显示屏1105从亮屏状态切换为息屏状态；当接近传感器 1116检测到用户与终端1100的正面之间的距离逐渐变大时，由处理器1101控制触摸显示屏1105从息屏状态切换为亮屏状态。
本领域技术人员可以理解，图11中示出的结构并不构成对终端1100的限定，可以包括比图示更多或更少的组件，或者组合某些组件，或者采用不同的组件布置。
图12示出了一个实施例中电子设备的内部结构图。如图12所示，该电子设备包括该计算机设备包括通过系统总线连接的处理器、存储器、网络接口、输入装置。其中，存储器包括非易失性存储介质和内存储器。该电子设备的非易失性存储介质存储有操作系统，还可存储有计算机可读指令，该计算机可读指令被处理器执行时，可使得处理器实现相机姿态追踪方法。该内存储器中也可储存有计算机可读指令，该计算机可读指令被处理器执行时，可使得处理器执行相机姿态追踪方法。输入装置可以是显示屏上覆盖的触摸层，也可以是计算机设备外壳上设置的按键、轨迹球或触控板，还可以是外接的键盘、触控板或鼠标等。
本领域技术人员可以理解，图12示出的结构，仅仅是与本申请方案相关的部分结构的框图，并不构成对本申请方案所应用于其上的电子设备的限定，具体的电子设备可以包括比图中所示更多或更少的部件，或者组合某些部件，或者具有不同的部件布置。
在一个实施例中，本申请提供的相机姿态追踪装置可以实现为一种计算机可读指令的形式，计算机可读指令可在如图12所示的电子设备上运行。电子设备的存储器中可存储组成该相机姿态追踪装置的各个程序模块，比如，图10所示的接收模块1020、图像获取模块1040、变化量获取模块1060、姿态计算模块1080。各个程序模块构成的计算机可读指令使得处理器执行本说明书中描述的本申请各个实施例的相机姿态追踪方法中的步骤。
本申请实施例提供了一种计算机可读存储介质，所述存储介质中存储有计算机可读指令，所述计算机可读指令由处理器加载并执行以实现图6所示的相机姿态追踪方法。
上述本申请实施例序号仅仅为了描述，不代表实施例的优劣。
本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程，是可以通过计算机程序来指令相关的硬件来完成，所述的程序可存储于一非易 失性计算机可读取存储介质中，该程序在执行时，可包括如上述各方法的实施例的流程。其中，本申请所提供的各实施例中所使用的对存储器、存储、数据库或其它介质的任何引用，均可包括非易失性和/或易失性存储器。非易失性存储器可包括只读存储器(ROM)、可编程ROM(PROM)、电可编程ROM(EPROM)、电可擦除可编程ROM(EEPROM)或闪存。易失性存储器可包括随机存取存储器(RAM)或者外部高速缓冲存储器。作为说明而非局限，RAM以多种形式可得，诸如静态RAM(SRAM)、动态RAM(DRAM)、同步DRAM(SDRAM)、双数据率SDRAM(DDRSDRAM)、增强型SDRAM(ESDRAM)、同步链路(Synchlink)DRAM(SLDRAM)、存储器总线(Rambus)直接RAM(RDRAM)、直接存储器总线动态RAM(DRDRAM)、以及存储器总线动态RAM(RDRAM)等。
以上实施例的各技术特征可以进行任意的组合，为使描述简洁，未对上述实施例中的各个技术特征所有可能的组合都进行描述，然而，只要这些技术特征的组合不存在矛盾，都应当认为是本说明书记载的范围。

Claims (32)

  1. 一种相机姿态追踪方法，应用于设置有相机的第二设备中，所述方法包括：
    第二设备接收第一设备发送的初始标记图像和初始姿态参数；
    所述第二设备获取所述相机采集的当前图像；
    所述第二设备获取所述当前图像相对于所述初始标记图像的相机姿态变化量；
    所述第二设备根据所述初始姿态参数和所述相机姿态变化量，计算得到所述相机采集所述当前图像时的当前相机姿态参数。
  2. 根据权利要求1所述的方法，其特征在于，所述第二设备获取所述当前图像相对于所述初始标记图像的相机姿态变化量，包括：
    所述第二设备将所述当前图像相对于所述初始标记图像进行重定位，得到所述相机姿态变化量。
  3. 根据权利要求1所述的方法，其特征在于，所述第二设备获取所述当前图像相对于所述初始标记图像的相机姿态变化量，包括：
    所述第二设备获取第i个标记图像相对于所述初始标记图像的第一姿态变化量，i＞1；所述第二设备将所述当前图像相对于所述第i个标记图像进行特征点追踪，得到第二姿态变化量；所述第二设备根据所述第一姿态变化量和所述第二姿态变化量，计算得到所述相机姿态变化量。
  4. 根据权利要求2至3任一所述的方法，其特征在于，所述第二设备将所述当前图像相对于所述初始标记图像进行重定位，得到所述相机姿态变化量，包括：
    所述第二设备获取所述初始标记图像的初始特征点；
    所述第二设备将所述当前图像相对于所述初始标记图像进行特征点追踪，得到与所述初始特征点匹配的目标特征点；
    所述第二设备根据所述初始特征点和所述目标特征点，计算所述初始标记图像和所述当前图像之间的单应性矩阵；
    所述第二设备分解所述单应性矩阵得到所述相机姿态变化量。
  5. 根据权利要求2至3任一所述的方法，其特征在于，所述第二设备根据所述第一姿态变化量和所述第二姿态变化量，计算得到所述相机姿态变化量，包括：
    设所述第一姿态变化量包括第一旋转矩阵R mf和第一位移向量T mf，所述第二姿态变化量包括第二旋转矩阵R cm和第二位移向量T cm，按照如下公式计算得到所述相机姿态变化量R cf和T cf：
    Figure PCTCN2019083437-appb-100001
    其中，S mf是所述第i个标记图像的尺度。
  6. 根据权利要求1至5任一所述的方法，其特征在于，所述第二设备根据所述初始姿态参数和所述相机姿态变化量，计算得到所述相机采集所述当前图像时的相机姿态参数，包括：
    按照如下公式计算得到所述相机姿态参数R和T：
    Figure PCTCN2019083437-appb-100002
    其中，R cf是所述相机姿态变化量中的旋转矩阵，T cf是所述相机姿态变化量中的位移向量，R born是所述初始姿态参数中的旋转矩阵，T born是所述初始姿态参数中的位移向量。
  7. 根据权利要求1至5任一所述的方法，其特征在于，所述第二设备接收第一设备发送的初始标记图像和初始姿态参数，包括：
    所述第二设备接收所述第一设备通过目标数据通道发送的所述初始标记图像和所述初始姿态参数，所述目标数据通道包括：从无线保真网络、蓝牙、有线网络和数据线组成的群组中选取的至少一种。
  8. 根据权利要求1至5任一所述的方法，其特征在于，所述第二设备接收第一设备发送的初始标记图像和初始姿态参数，包括：
    所述第二设备接收服务器发送的所述初始标记图像和所述初始姿态参数，所述初始标记图像和所述初始姿态参数是所述第一设备发送给所述服务器的。
  9. 一种电子设备，包括存储器和处理器，所述存储器中存储有计算机可读指令，所述计算机可读指令被所述处理器执行时，使得所述处理器执行如下步 骤：
    接收第一设备发送的初始标记图像和初始姿态参数；
    获取所述相机采集的当前图像；
    获取所述当前图像相对于所述初始标记图像的相机姿态变化量；
    根据所述初始姿态参数和所述相机姿态变化量，计算得到所述相机采集所述当前图像时的当前相机姿态参数。
  10. 根据权利要求9所述的电子设备，其特征在于，所述获取所述当前图像相对于所述初始标记图像的相机姿态变化量，包括：
    将所述当前图像相对于所述初始标记图像进行重定位，得到所述相机姿态变化量。
  11. 根据权利要求9所述的电子设备，其特征在于，所述获取所述当前图像相对于所述初始标记图像的相机姿态变化量，包括：
    获取第i个标记图像相对于所述初始标记图像的第一姿态变化量，i＞1；将所述当前图像相对于所述第i个标记图像进行特征点追踪，得到第二姿态变化量；根据所述第一姿态变化量和所述第二姿态变化量，计算得到所述相机姿态变化量。
  12. 根据权利要求10至11任一所述的电子设备，其特征在于，所述将所述当前图像相对于所述初始标记图像进行重定位，得到所述相机姿态变化量，包括：
    获取所述初始标记图像的初始特征点；
    将所述当前图像相对于所述初始标记图像进行特征点追踪，得到与所述初始特征点匹配的目标特征点；
    根据所述初始特征点和所述目标特征点，计算所述初始标记图像和所述当前图像之间的单应性矩阵；
    分解所述单应性矩阵得到所述相机姿态变化量。
  13. 根据权利要求10至11任一所述的电子设备，其特征在于，所述根据所述第一姿态变化量和所述第二姿态变化量，计算得到所述相机姿态变化量， 包括：
    设所述第一姿态变化量包括第一旋转矩阵Rmf和第一位移向量Tmf，所述第二姿态变化量包括第二旋转矩阵Rcm和第二位移向量Tcm，按照如下公式计算得到所述相机姿态变化量Rcf和Tcf：
    Figure PCTCN2019083437-appb-100003
    其中，Smf是所述第i个标记图像的尺度。
  14. 根据权利要求9至13任一所述的电子设备，其特征在于，所述根据所述初始姿态参数和所述相机姿态变化量，计算得到所述相机采集所述当前图像时的相机姿态参数，包括：
    按照如下公式计算得到所述相机姿态参数R和T：
    Figure PCTCN2019083437-appb-100004
    其中，Rcf是所述相机姿态变化量中的旋转矩阵，Tcf是所述相机姿态变化量中的位移向量，Rborn是所述初始姿态参数中的旋转矩阵，Tborn是所述初始姿态参数中的位移向量。
  15. 根据权利要求9至13任一所述的电子设备，其特征在于，所述接收第一设备发送的初始标记图像和初始姿态参数，包括：
    接收所述第一设备通过目标数据通道发送的所述初始标记图像和所述初始姿态参数，所述目标数据通道包括：从无线保真网络、蓝牙、有线网络和数据线组成的群组中选取的至少一种。
  16. 根据权利要求9至13任一所述的电子设备，其特征在于，所述接收第一设备发送的初始标记图像和初始姿态参数，包括：
    接收服务器发送的所述初始标记图像和所述初始姿态参数，所述初始标记图像和所述初始姿态参数是所述第一设备发送给所述服务器的。
  17. 一个或多个存储有计算机可读指令的非易失性存储介质，所述计算机可读指令被一个或多个处理器执行时，使得一个或多个处理器执行如下步骤：
    接收第一设备发送的初始标记图像和初始姿态参数；
    获取所述相机采集的当前图像；
    获取所述当前图像相对于所述初始标记图像的相机姿态变化量；
    根据所述初始姿态参数和所述相机姿态变化量，计算得到所述相机采集所述当前图像时的当前相机姿态参数。
  18. 根据权利要求17所述的存储介质，其特征在于，所述获取所述当前图像相对于所述初始标记图像的相机姿态变化量，包括：
    将所述当前图像相对于所述初始标记图像进行重定位，得到所述相机姿态变化量。
  19. 根据权利要求17所述的存储介质，其特征在于，所述获取所述当前图像相对于所述初始标记图像的相机姿态变化量，包括：
    获取第i个标记图像相对于所述初始标记图像的第一姿态变化量，i＞1；将所述当前图像相对于所述第i个标记图像进行特征点追踪，得到第二姿态变化量；根据所述第一姿态变化量和所述第二姿态变化量，计算得到所述相机姿态变化量。
  20. 根据权利要求18至19任一所述的存储介质，其特征在于，所述将所述当前图像相对于所述初始标记图像进行重定位，得到所述相机姿态变化量，包括：
    获取所述初始标记图像的初始特征点；
    将所述当前图像相对于所述初始标记图像进行特征点追踪，得到与所述初始特征点匹配的目标特征点；
    根据所述初始特征点和所述目标特征点，计算所述初始标记图像和所述当前图像之间的单应性矩阵；
    分解所述单应性矩阵得到所述相机姿态变化量。
  21. 根据权利要求18至19任一所述的存储介质，其特征在于，所述根据所述第一姿态变化量和所述第二姿态变化量，计算得到所述相机姿态变化量，包括：
    设所述第一姿态变化量包括第一旋转矩阵R mf和第一位移向量T mf，所述第 二姿态变化量包括第二旋转矩阵R cm和第二位移向量T cm，按照如下公式计算得到所述相机姿态变化量R cf和T cf：
    Figure PCTCN2019083437-appb-100005
    其中，S mf是所述第i个标记图像的尺度。
  22. 根据权利要求17至21任一所述的存储介质，其特征在于，所述根据所述初始姿态参数和所述相机姿态变化量，计算得到所述相机采集所述当前图像时的相机姿态参数，包括：
    按照如下公式计算得到所述相机姿态参数R和T：
    Figure PCTCN2019083437-appb-100006
    其中，R cf是所述相机姿态变化量中的旋转矩阵，T cf是所述相机姿态变化量中的位移向量，R born是所述初始姿态参数中的旋转矩阵，T born是所述初始姿态参数中的位移向量。
  23. 根据权利要求17至21任一所述的存储介质，其特征在于，所述接收第一设备发送的初始标记图像和初始姿态参数，包括：
    接收所述第一设备通过目标数据通道发送的所述初始标记图像和所述初始姿态参数，所述目标数据通道包括：从无线保真网络、蓝牙、有线网络和数据线组成的群组中选取的至少一种。
  24. 根据权利要求17至21任一所述的存储介质，其特征在于，所述接收第一设备发送的初始标记图像和初始姿态参数，包括：
    接收服务器发送的所述初始标记图像和所述初始姿态参数，所述初始标记图像和所述初始姿态参数是所述第一设备发送给所述服务器的。
  25. 一种相机姿态追踪装置，其特征在于，应用于设置有相机的第二设备中，所述装置包括：
    接收模块，用于接收第一设备发送的初始标记图像和初始姿态参数；
    图像获取模块，用于获取所述相机采集的当前图像；
    变化量获取模块，用于获取所述当前图像相对于所述初始标记图像的相机姿态变化量；
    姿态计算模块，用于根据所述初始姿态参数和所述相机姿态变化量，计算得到所述相机采集所述当前图像时的当前相机姿态参数。
  26. 根据权利要求25所述的装置，其特征在于，所述变化量获取模块，用于将所述当前图像相对于所述初始标记图像进行重定位，得到所述相机姿态变化量。
  27. 根据权利要求25所述的装置，其特征在于，所述变化量获取模块，用于获取第i个标记图像相对于所述初始标记图像的第一姿态变化量，i＞1；将所述当前图像相对于所述第i个标记图像进行特征点追踪，得到第二姿态变化量；根据所述第一姿态变化量和所述第二姿态变化量，计算得到所述相机姿态变化量。
  28. 根据权利要求26至27任一所述的装置，其特征在于，所述变化量获取模块，用于获取所述初始标记图像的初始特征点；将所述当前图像相对于所述初始标记图像进行特征点追踪，得到与所述初始特征点匹配的目标特征点；根据所述初始特征点和所述目标特征点，计算所述初始标记图像和所述当前图像之间的单应性矩阵；分解所述单应性矩阵得到所述相机姿态变化量。
  29. 根据权利要求26至27任一所述的装置，其特征在于，设所述第一姿态变化量包括第一旋转矩阵R mf和第一位移向量T mf，所述第二姿态变化量包括第二旋转矩阵R cm和第二位移向量T cm，所述变化量获取模块，用于按照如下公式计算得到所述相机姿态变化量R cf和T cf：
    Figure PCTCN2019083437-appb-100007
    其中，S mf是所述第i个标记图像的尺度。
  30. 根据权利要求25至29任一所述的装置，其特征在于，所述姿态计算模块，用于按照如下公式计算得到所述相机姿态参数R和T：
    Figure PCTCN2019083437-appb-100008
    其中，R cf是所述相机姿态变化量中的旋转矩阵，T cf是所述相机姿态变化量中的位移向量，R born是所述初始姿态参数中的旋转矩阵，T born是所述初始姿态 参数中的位移向量。
  31. 根据权利要求25至29任一所述的装置，其特征在于，所述接收模块，用于接收所述第一设备通过目标数据通道发送的所述初始标记图像和所述初始姿态参数，所述目标数据通道包括：从无线保真网络、蓝牙、有线网络和数据线组成的群组中选取的至少一种。
  32. 根据权利要求25至29任一所述的装置，其特征在于，所述接收模块，用于接收服务器发送的所述初始标记图像和所述初始姿态参数，所述初始标记图像和所述初始姿态参数是所述第一设备发送给所述服务器的。
PCT/CN2019/083437 2018-05-22 2019-04-19 相机姿态追踪方法、装置、设备及系统 WO2019223468A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP19808400.6A EP3798983B1 (en) 2018-05-22 2019-04-19 Camera orientation tracking method and apparatus, device, and system
US17/001,658 US11321870B2 (en) 2018-05-22 2020-08-24 Camera attitude tracking method and apparatus, device, and system

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810495056.7 2018-05-22
CN201810495056.7A CN108734736B (zh) 2018-05-22 2018-05-22 相机姿态追踪方法、装置、设备及存储介质

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/001,658 Continuation US11321870B2 (en) 2018-05-22 2020-08-24 Camera attitude tracking method and apparatus, device, and system

Publications (1)

Publication Number Publication Date
WO2019223468A1 true WO2019223468A1 (zh) 2019-11-28

Family

ID=63937866

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/083437 WO2019223468A1 (zh) 2018-05-22 2019-04-19 相机姿态追踪方法、装置、设备及系统

Country Status (4)

Country Link
US (1) US11321870B2 (zh)
EP (1) EP3798983B1 (zh)
CN (2) CN108734736B (zh)
WO (1) WO2019223468A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114827338A (zh) * 2021-01-29 2022-07-29 北京外号信息技术有限公司 用于在设备的显示媒介上呈现虚拟对象的方法和电子装置

Families Citing this family (26)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108734736B (zh) * 2018-05-22 2021-10-26 腾讯科技（深圳）有限公司 相机姿态追踪方法、装置、设备及存储介质
CN109544615B (zh) * 2018-11-23 2021-08-24 深圳市腾讯信息技术有限公司 基于图像的重定位方法、装置、终端及存储介质
CN109847350A (zh) * 2019-03-25 2019-06-07 深圳初影科技有限公司 基于ar技术的游戏实现方法、游戏系统、及存储介质
CN109992111B (zh) * 2019-03-25 2021-02-19 联想(北京)有限公司 增强现实扩展方法和电子设备
CN109908589B (zh) * 2019-03-25 2022-09-02 深圳初影科技有限公司 基于ar技术的游戏实现方法、游戏系统、及存储介质
CN110286768B (zh) * 2019-06-27 2022-05-17 Oppo广东移动通信有限公司 虚拟物体显示方法、终端设备及计算机可读存储介质
CN110276794B (zh) * 2019-06-28 2022-03-01 Oppo广东移动通信有限公司 信息处理方法、信息处理装置、终端设备及服务器
CN110473259A (zh) * 2019-07-31 2019-11-19 深圳市商汤科技有限公司 位姿确定方法及装置、电子设备和存储介质
CN110989825B (zh) * 2019-09-10 2020-12-01 中兴通讯股份有限公司 增强现实互动实现方法及系统、增强现实设备、存储介质
CN110866977B (zh) * 2019-10-31 2023-06-16 Oppo广东移动通信有限公司 增强现实处理方法及装置、系统、存储介质和电子设备
CN111078003B (zh) * 2019-11-27 2021-10-22 Oppo广东移动通信有限公司 数据处理方法、装置、电子设备及存储介质
CN111179435B (zh) * 2019-12-24 2024-02-06 Oppo广东移动通信有限公司 增强现实处理方法及装置、系统、存储介质和电子设备
CN111338474B (zh) * 2020-02-19 2022-11-08 Oppo广东移动通信有限公司 虚拟对象位姿校准方法及装置、存储介质和电子设备
CN113781548B (zh) * 2020-06-10 2024-06-14 华为技术有限公司 多设备的位姿测量方法、电子设备及系统
CN111582385B (zh) * 2020-05-11 2023-10-31 杭州易现先进科技有限公司 Slam质量的量化方法、系统、计算机设备和存储介质
CN111798489B (zh) * 2020-06-29 2024-03-08 北京三快在线科技有限公司 一种特征点跟踪方法、设备、介质及无人设备
CN112242008B (zh) * 2020-10-17 2021-09-07 郑州西亚斯学院 一种用于透明容器的ar交互视觉感知方法及系统
CN112650422B (zh) * 2020-12-17 2022-07-29 咪咕文化科技有限公司 设备的ar交互方法、装置、电子设备及存储介质
CN112767455B (zh) * 2021-01-08 2022-09-02 合肥的卢深视科技有限公司 一种双目结构光的校准方法及系统
CN114812381B (zh) * 2021-01-28 2023-07-18 华为技术有限公司 电子设备的定位方法及电子设备
CN114088062B (zh) * 2021-02-24 2024-03-22 上海商汤临港智能科技有限公司 目标定位方法及装置、电子设备和存储介质
US11961201B2 (en) * 2021-05-21 2024-04-16 Meta Platforms Technologies, Llc Viewing 3D photo galleries in VR
CN113223007A (zh) * 2021-06-28 2021-08-06 浙江华睿科技股份有限公司 视觉里程计的实现方法、装置及电子设备
CN115797815B (zh) * 2021-09-08 2023-12-15 荣耀终端有限公司 Ar翻译的处理方法及电子设备
CN117036663A (zh) * 2022-04-18 2023-11-10 荣耀终端有限公司 视觉定位方法、设备和存储介质
WO2024061238A1 (zh) * 2022-09-21 2024-03-28 海信电子科技（深圳）有限公司 一种估计手柄位姿的方法及虚拟显示设备

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130148851A1 (en) * 2011-12-12 2013-06-13 Canon Kabushiki Kaisha Key-frame selection for parallel tracking and mapping
CN103971400A (zh) * 2013-02-06 2014-08-06 阿里巴巴集团控股有限公司 一种基于标识码的三维交互的方法和系统
CN107194968A (zh) * 2017-05-18 2017-09-22 腾讯科技（上海）有限公司 图像的识别跟踪方法、装置、智能终端和可读存储介质
CN107590453A (zh) * 2017-09-04 2018-01-16 腾讯科技（深圳）有限公司 增强现实场景的处理方法、装置及设备、计算机存储介质
CN107833237A (zh) * 2016-09-15 2018-03-23 汤姆逊许可公司 用于模糊视频中的虚拟对象的方法和设备
CN108734736A (zh) * 2018-05-22 2018-11-02 腾讯科技（深圳）有限公司 相机姿态追踪方法、装置、设备及存储介质

Family Cites Families (26)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4991395B2 (ja) * 2007-05-28 2012-08-01 キヤノン株式会社 情報処理方法及び情報処理装置
US10354407B2 (en) * 2013-03-15 2019-07-16 Spatial Cam Llc Camera for locating hidden objects
US9020187B2 (en) * 2011-05-27 2015-04-28 Qualcomm Incorporated Planar mapping and tracking for mobile devices
WO2013086678A1 (zh) * 2011-12-12 2013-06-20 北京航空航天大学 平面模型下点匹配与位姿同步确定方法及计算机程序产品
JP5872923B2 (ja) * 2012-02-22 2016-03-01 株式会社マイクロネット Ａｒ画像処理装置及び方法
WO2015051834A1 (en) * 2013-10-09 2015-04-16 Metaio Gmbh Method and system for determining a pose of a camera
CN106664393A (zh) * 2014-07-31 2017-05-10 索尼公司 信息处理装置、信息处理方法以及图像显示系统
US20160133230A1 (en) * 2014-11-11 2016-05-12 Bent Image Lab, Llc Real-time shared augmented reality experience
EP4224424A1 (en) * 2014-11-21 2023-08-09 Apple Inc. Method and system for determining spatial coordinates of a 3d reconstruction of at least part of a real object at absolute spatial scale
CN106296598B (zh) * 2016-07-29 2019-11-26 厦门美图之家科技有限公司 三维姿态处理方法、系统及拍摄终端
CN106843456B (zh) * 2016-08-16 2018-06-29 深圳超多维光电子有限公司 一种基于姿态追踪的显示方法、装置和虚拟现实设备
JP2018055589A (ja) * 2016-09-30 2018-04-05 セイコーエプソン株式会社 プログラム、物体の追跡方法、表示装置
CN108022264B (zh) * 2016-11-01 2023-06-13 北京墨土科技有限公司 相机位姿确定方法及设备
CN106780601B (zh) * 2016-12-01 2020-03-27 北京未动科技有限公司 一种空间位置追踪方法、装置及智能设备
CN106873784A (zh) * 2017-03-29 2017-06-20 联想(北京)有限公司 一种显示控制方法、装置及电子设备
GB2561368B (en) * 2017-04-11 2019-10-09 Nokia Technologies Oy Methods and apparatuses for determining positions of multi-directional image capture apparatuses
US10515458B1 (en) * 2017-09-06 2019-12-24 The United States Of America, As Represented By The Secretary Of The Navy Image-matching navigation method and apparatus for aerial vehicles
CN108615247B (zh) 2018-04-27 2021-09-14 深圳市腾讯计算机系统有限公司 相机姿态追踪过程的重定位方法、装置、设备及存储介质
CN110555882B (zh) 2018-04-27 2022-11-15 腾讯科技（深圳）有限公司 界面显示方法、装置及存储介质
CN108682037B (zh) 2018-04-27 2022-03-08 腾讯科技（深圳）有限公司 相机姿态追踪过程的重定位方法、装置、设备及存储介质
CN110599549B (zh) 2018-04-27 2023-01-10 腾讯科技（深圳）有限公司 界面显示方法、装置及存储介质
CN108682038B (zh) 2018-04-27 2021-12-14 腾讯科技（深圳）有限公司 位姿确定方法、装置及存储介质
CN108615248B (zh) 2018-04-27 2022-04-05 腾讯科技（深圳）有限公司 相机姿态追踪过程的重定位方法、装置、设备及存储介质
CN108648235B (zh) 2018-04-27 2022-05-17 腾讯科技（深圳）有限公司 相机姿态追踪过程的重定位方法、装置及存储介质
CN108596976B (zh) 2018-04-27 2022-02-22 腾讯科技（深圳）有限公司 相机姿态追踪过程的重定位方法、装置、设备及存储介质
CN108876854B (zh) 2018-04-27 2022-03-08 腾讯科技（深圳）有限公司 相机姿态追踪过程的重定位方法、装置、设备及存储介质

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130148851A1 (en) * 2011-12-12 2013-06-13 Canon Kabushiki Kaisha Key-frame selection for parallel tracking and mapping
CN103971400A (zh) * 2013-02-06 2014-08-06 阿里巴巴集团控股有限公司 一种基于标识码的三维交互的方法和系统
CN107833237A (zh) * 2016-09-15 2018-03-23 汤姆逊许可公司 用于模糊视频中的虚拟对象的方法和设备
CN107194968A (zh) * 2017-05-18 2017-09-22 腾讯科技（上海）有限公司 图像的识别跟踪方法、装置、智能终端和可读存储介质
CN107590453A (zh) * 2017-09-04 2018-01-16 腾讯科技（深圳）有限公司 增强现实场景的处理方法、装置及设备、计算机存储介质
CN108734736A (zh) * 2018-05-22 2018-11-02 腾讯科技（深圳）有限公司 相机姿态追踪方法、装置、设备及存储介质

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3798983A4 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114827338A (zh) * 2021-01-29 2022-07-29 北京外号信息技术有限公司 用于在设备的显示媒介上呈现虚拟对象的方法和电子装置

Also Published As

Publication number Publication date
CN108734736A (zh) 2018-11-02
EP3798983A1 (en) 2021-03-31
CN110544280A (zh) 2019-12-06
US20200388051A1 (en) 2020-12-10
US11321870B2 (en) 2022-05-03
EP3798983A4 (en) 2021-07-21
CN108734736B (zh) 2021-10-26
CN110544280B (zh) 2021-10-08
EP3798983B1 (en) 2024-02-07

Similar Documents

Publication Publication Date Title
WO2019223468A1 (zh) 相机姿态追踪方法、装置、设备及系统
US11205282B2 (en) Relocalization method and apparatus in camera pose tracking process and storage medium
US11481923B2 (en) Relocalization method and apparatus in camera pose tracking process, device, and storage medium
US11158083B2 (en) Position and attitude determining method and apparatus, smart device, and storage medium
US11798190B2 (en) Position and pose determining method, apparatus, smart device, and storage medium
US11436779B2 (en) Image processing method, electronic device, and storage medium
CN108810538B (zh) 视频编码方法、装置、终端及存储介质
CN109308727B (zh) 虚拟形象模型生成方法、装置及存储介质
US11517099B2 (en) Method for processing images, electronic device, and storage medium
JP7026819B2 (ja) カメラの位置決め方法および装置、端末並びにコンピュータプログラム
US11276183B2 (en) Relocalization method and apparatus in camera pose tracking process, device, and storage medium
WO2020215858A1 (zh) 基于虚拟环境的物体构建方法、装置、计算机设备及可读存储介质
WO2022052620A1 (zh) 图像生成方法及电子设备
CN110956571A (zh) 基于slam进行虚实融合的方法及电子设备
CN114093020A (zh) 动作捕捉方法、装置、电子设备及存储介质
CN114422736B (zh) 一种视频处理方法、电子设备及计算机存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19808400

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2019808400

Country of ref document: EP

Effective date: 20201222