JP2016526313A - Monocular visual SLAM using global camera movement and panoramic camera movement - Google Patents

Abstract

Disclosed are systems, devices and methods for monocular visual simultaneous localization and mapping that handle global 6DOF camera movement and panoramic camera movement. A 3D map of the environment is received that includes features with a finite or infinite depth observed in regular or panoramic key frames. The camera is tracked at 6 DOF from a finite feature set, an infinite feature set, or a mixed feature set. When panoramic camera movement toward an unmapped scene area is detected, a reference panoramic key frame with infinite features is generated and inserted into the 3D map. If the panoramic camera movement extends towards the unmapped scene area, the reference key frame is extended using other subordinate panoramic key frames. Panorama keyframes are robustly localized at 6DOF for finite 3D map features. Localized panoramic key frames include 2D observations of infinite map features that are aligned with 2D observations in other localized key frames. The 2D-2D code is triangulated to obtain a new finite 3D map feature.

Description

This application claims the benefit of US Provisional Application No. 61 / 817,808, filed Apr. 30, 2013, which is expressly incorporated herein by reference.

  The subject matter disclosed herein generally relates to simultaneous localization and mapping.

  The Simultaneous Localization and Mapping (SLAM) system processes the input of a single camera and moves it to the third order of the environment as the camera moves in 6 DOF (Six Degrees of Freedom). Original (3D: Three Dimensional) models (eg, SLAM maps) can be created continuously. The visual SLAM system can simultaneously track the position and orientation (pose) of the camera relative to the 3D model. A visual SLAM system based on key frames can process discretely selected frames from an incoming camera image stream or feed. A visual SLAM system based on key frames assumes a general camera motion and applies a structural technique from motion to generate a 3D feature map.

  A visual SLAM system requires sufficient parallax, possibly induced by translational or global camera movement between keyframe pairs, to triangulate 3D map features. Thus, for key frames that have already been selected, the selection algorithm may reject the candidate frame, and only relative rotation will reduce camera motion. Rotation-only camera motion relative to unmapped regions may cause the visual SLAM system to stop due to a lack of newly selected keyframes. Camera tracking will eventually fail due to map inability. As a result, the visual SLAM system will be forced into relocalization mode to resume tracking. Improved tracking and mapping techniques are therefore desirable.

  Embodiments disclosed herein can be associated with methods for monocular visual simultaneous localization and mapping. In one embodiment, a machine-implemented method for processing an image receives a 3D map of the environment. In one embodiment, the 3D map includes features having a finite depth observed in two or more key frames, where each key frame is a panoramic key frame or a regular key frame. The 3D map also includes features with infinite depth observed within one or more panoramic key frames. In one embodiment, the method tracks the camera in 6 degrees of freedom (6 DOF) from a finite depth feature or an infinite depth feature of a 3D map observed in an image frame from an input image feed.

  Embodiments disclosed herein can be associated with an apparatus for monocular visual simultaneous localization and mapping. The apparatus can include means for receiving a 3D map of the environment. In one embodiment, the 3D map includes features having a finite depth observed in two or more key frames, where each key frame is a panoramic key frame or a regular key frame. The 3D map also includes features with infinite depth observed within one or more panoramic key frames. In one embodiment, the apparatus can include means for tracking the camera in 6 degrees of freedom (6 DOF) from a finite depth feature or an infinite depth feature of a 3D map observed in an image frame from an input image feed. .

  The embodiments disclosed herein can be associated with a device for monocular simultaneous visual localization and mapping, the device comprising hardware and software for receiving a 3D map of the environment. The device can process instructions for receiving a three-dimensional (3D) map of the environment. In one embodiment, the 3D map includes features having a finite depth observed in two or more key frames, where each key frame is a panoramic key frame or a regular key frame. The 3D map also includes features with infinite depth observed within one or more panoramic key frames. In one embodiment, the device may process instructions for tracking the camera in 6 degrees of freedom (6 DOF) from a finite depth feature or an infinite depth feature of a 3D map observed in an image frame from an input image feed. it can.

  Embodiments disclosed herein can be associated with a non-transitory storage medium that stores instructions for performing receipt of a 3D map of an environment in response to being executed by a processor in the device. The medium can store instructions for receiving a three-dimensional (3D) map of the environment. In one embodiment, the 3D map includes features having a finite depth observed in two or more key frames, where each key frame is a panoramic key frame or a regular key frame. The 3D map also includes features with infinite depth observed within one or more panoramic key frames. In one embodiment, the medium can store instructions for tracking the camera in 6 degrees of freedom (6 DOF) from a finite depth feature or an infinite depth feature of a 3D map observed in an image frame from an input image feed.

  Other features and advantages will be apparent from the accompanying drawings and from the detailed description.

1 is a block diagram of a system in which aspects of the present invention may be practiced in one embodiment. 2 is a flow diagram of a hybrid SLAM, in one embodiment. FIG. 6 illustrates a first stage of a hybrid SLAM map representation between keyframes and features in one embodiment. FIG. 6 illustrates a second stage of a hybrid SLAM map representation between keyframes and features in one embodiment. 3 is a flow diagram of hybrid SLAM initialization in one embodiment. FIG. 6 illustrates 6DOF and panoramic mapping and tracking steps in which an overall camera motion and a pure rotation camera motion alternate in one embodiment. FIG. 6 is a state diagram for different states of key frame selection during mapping in one embodiment. FIG. 2 is a block diagram of a hybrid SLAM system that includes a tracking component and a mapping component in one embodiment.

  The word “exemplary” or “example” is used herein to mean “serving as an example, instance, or illustration”. Any aspect or embodiment described herein as "exemplary" or "example" is not necessarily to be construed as preferred or advantageous over other aspects or embodiments.

  In one embodiment, the 6DOF SLAM and panoramic SLAM functions can be combined to accept both fully triangulated keyframes for regular 6DOF operation, as well as keyframes where the constraint is only rotation. It can be a SLAM system based on a hybrid keyframe of robust exercise. In one embodiment, Hybrid SLAM (HSLAM: Hybrid SLAM) can successfully handle pure rotations and provide the user with a seamless tracking experience. In one embodiment, HSLAM mapping utilizes 6DOF and panoramic key frames to estimate a new portion of a three-dimensional (3D) map (eg, a wide area SLAM map). HSLAM can continuously track the 3D map away from the mapped part of the scene throughout the rotation, and can also capture information from camera images observed during the rotation-only motion (e.g. panoramic tracking). Can be used to update 3D maps. In one embodiment, HSLAM can be implemented using a single camera sensor as a type of monocular visual SLAM. As described below, HSLAM operations can be performed by device 100 under the control of a processor to perform the functions described herein.

FIG. 1 is a block diagram illustrating a system in which embodiments of the present invention can be practiced.
The system may be a device 100 that may include a general purpose processor 161, an image processing module 171, a 6DOF SLAM module 173, a panorama module 175, and a memory 164. The device 100 can also include multiple device sensors coupled to one or more buses 177, or signal lines further coupled to at least the image processing module 171, 6DOF SLAM module 173, and panoramic SLAM175 module. is there. Modules 171, 173, and 175 are shown separately from processor 161 and / or hardware 162 for clarity, but these are based on instructions in software 165 and firmware 163, It can be combined and / or implemented in processor 161 and / or hardware 162. The control unit 160 can be configured to implement the method for performing the hybrid SLAM described below. For example, the control unit 160 can be configured to implement the functions of the mobile device 100 described in FIG.

  Device 100 is a mobile device, wireless device, cell phone, augmented reality device (AR), personal digital assistant, wearable device (e.g. glasses, watch, hat or similar body-worn device), mobile computer, tablet A personal computer, a laptop computer, a data processing device / system, or any type of device with processing capabilities.

  In one embodiment, device 100 is a mobile / portable platform. The device 100 can include means for capturing an image, such as a camera 114, and optionally includes a motion sensor 111, such as an accelerometer, gyroscope, electronic compass, or other similar motion sensing element. It is also possible. Device 100 can also capture images on a forward-facing camera or a rear-facing camera (eg, camera 114). Device 100 can further include a user interface 150 that includes means such as a display 112 for displaying augmented reality images. The user interface 150 may also include a keyboard, keypad 152, or other input device that allows a user to enter information into the device 100. If desired, the keyboard or keypad 152 can be removed by integrating a virtual keypad into the display 112 having a touch screen / sensor. For example, when the device 100 is a mobile platform such as a cellular phone, the user interface 150 may also include a microphone 154 and a speaker 156. Device 100 includes other elements not relevant to the present disclosure, such as satellite position system receivers, power devices (e.g., batteries), and other configurations typically coupled to portable and non-portable electronic devices. Can contain elements.

  The device 100 can function as a mobile device or a wireless device and is based on any suitable wireless communication technology or otherwise via one or more wireless communication links through a supporting wireless network Can communicate. For example, in some aspects, device 100 may be a client or server and may be coupled to a wireless network. In some aspects, the network may comprise a body area network or a personal area network (eg, an ultra wideband network). In some aspects, the network may comprise a local area network or a wide area network. Does the wireless device support one or more of various wireless communication technologies, protocols, or standards such as 3G, LTE, Advanced LTE, 4G, CDMA, TDMA, OFDM, OFDMA, WiMAX and Wi-Fi , Otherwise it can be used. Similarly, a wireless device may support or otherwise use one or more of a variety of corresponding modulation or multiplexing schemes. Mobile wireless devices can communicate wirelessly with other mobile devices, cellular phones, other wired and wireless computers, Internet websites, and so on.

  As described above, the device 100 may be a portable electronic device (eg, a smartphone, a dedicated augmented reality (AR) device, a gaming device, or other device having AR processing and display capabilities). Devices that implement the AR system described herein can be used in a variety of environments (eg, shopping malls, roads, offices, homes, or places where users can use their devices). . A user can interface with multiple features of his device 100 in a wide variety of situations. In the AR context, a user can use his device to see a real world representation through his device display. Users process real-time images / video using their device's camera and process the image in a way that overlays additional or alternative information on the real-world image / video displayed on the device Allows you to interact with your AR-enabled device. Users can replace or change real-world objects or scenes in real time on the device display when viewing the AR implementation on their devices. Virtual objects (eg, text, images, video) can be inserted into the representation of the scene depicted on the device display.

  In one embodiment, the HSLAM may perform 6DOF SLAM, including global SLAM map tracking and mapping as described above. HSLAM can maintain a single SLAM map (ie, a wide area SLAM map), and both 6DOF SLAM and panoramic SLAM can access and update the wide area SLAM map.

  In some embodiments, HSLAM can generate key frames from captured images through 6DOF SLAM (eg, as a dedicated 6DOF module 173). If HSLAM determines that the captured image matches the threshold translation, it can generate a key frame from the previous key frame that is already combined with the global SLAM map.

  In one embodiment, 6DOF SLAM (eg, 6DOF tracking) can combine features observed from keyframes into a global SLAM map. 6DOF SLAM (eg, 6DOF tracking) can use this feature combination to determine the camera position and orientation (ie, pose) associated with each camera image. 6DOF mapping can also update / maintain wide area SLAM maps. As explained above, the global SLAM map maintained by 6DOF SLAM may contain 3D feature points triangulated from two or more keyframes (e.g. a pair of keyframes or multiple pairs of keyframes). it can. For example, key frames can be selected from an image or video stream or feed to represent an observed scene. HSLAM can calculate each 6DOF camera pose combined with the image for each keyframe. The calculated pose may be referred to herein as a key frame pose (consisting of 3DOF key frame position and 3DOF key frame orientation).

  As used herein, panoramic SLAM refers to stitching together multiple captured images into a combined collection of images acquired using rotational only camera motion. HSLAM using panoramic SLAM (eg panoramic tracking of panorama module 175) can calculate 3 rotational degrees of freedom (3DOF) when compared to 6DOF of 6DOF SLAM (ie calculated by 6DOF module 173) . HSLAM can associate panoramic key frames with each other using relative rotation. HSLAM can bypass or skip feature point triangulation if the minimum threshold disparity or translation does not match. For example, if the camera position remains the same and only pure rotation has occurred from the preceding key frame, the minimum threshold parallax or translation will not match.

  HSLAM can compare the current key frame with the already captured key frame to determine the parallax level or translation level. Thus, panoramic feature points can be considered as rays (ie, infinite features, infinite depth features, features with no estimated depth or features with infinite depth). In one embodiment, 3D points generated from 6DOF SLAM are referred to as finite depth features (eg, a feature can have a defined or estimated depth).

  Conventional 6DOF SLAM cannot handle pure rotational camera movement. Tracking can be lost, and in some situations, incorrectly measured finite features can harm a map (eg, a wide area SLAM map). On the other hand, the panorama SLAM, which has traditionally handled rotational motion during translation, may be encoded as additional rotation, which also reduces map quality.

  In one embodiment, HSLAM combines the advantages of 6DOF SLAM and panoramic SLAM into a hybrid system that can dynamically switch between 6DOF SLAM and panoramic SLAM depending on the nature of the movement. For example, the user can perform an exercise that is gross or pure rotation. HSLAM can handle temporary rotation away from the mapped part of the scene that users often practice. HSLAM can also incorporate observations of scenes made during rotational movement into subsequent 3D mapping steps if sufficient additional information becomes available.

  In one embodiment, HSLAM may use 6DOF tracking to determine camera poses for one or more image frames or video frames. HSLAM can determine the camera pose by projecting features from the 3D map into an image or video frame and updating the camera pose from the validated 2D-3D code. HSLAM can also select new keyframes for insertion into the map. HSLAM can insert a new keyframe into the 3D map if the current camera position (ie translation) is well away from all existing keyframe positions. HSLAM also allows new keyframes if the coverage of the current frame with known features is below a threshold (for example, a new or previously unmapped region of the 3D map is represented in the current frame). Can also be inserted into the map. In addition, HSLAM inserts keyframes when the current camera orientation is sufficiently far from the existing keyframe orientation and the current camera position is translated a minimum distance from the existing keyframe position. can do.

  Alternatively, 6DOF SLAM can bypass or skip keyframe insertion if the orientation is changing but the position is not changing enough. If the orientation changes and the position does not change, 6DOF SLAM can account for pure rotational movement. 6DOF SLAM cannot triangulate during pure rotation or insert new finite features into the map.

  In one embodiment, HSLAM can trigger a real-time switch from 6DOF SLAM to panoramic SLAM when threshold tracking conditions are met. For example, the threshold tracking condition is sufficient orientation change (e.g., rotating the camera view), maintaining the camera position (e.g., where the camera location is fixed or approximately the same as the previously captured image) And existing coverage is narrow (eg, the captured image region is a new or unmapped region in the 3D map). For example, narrow existing coverage indicates that if less than 50 percent of the current image is covered by known feature points, tracking will eventually be lost if the camera view continues to rotate toward the new area Can be determined based on what HSLAM detects. In other embodiments, the HSLAM may use a generalized Geometric Robust Information Criteria (or GRIC) score to trigger a switch from 6DOF SLAM to panoramic SLAM.

  When HSLAM switches to panoramic SLAM, it can generate new keyframes containing infinite features. HSLAM can insert infinite feature keyframes into a keyframe database. For example, a keyframe database can be combined into a global SLAM map. In one embodiment, infinite feature keyframes can be marked or identified as “panoramic” keyframes. As used herein, a panoramic keyframe is a keyframe that includes infinite features or features that have no calculated depth.

  HSLAM can continue to track infinite features and insert additional panoramic keyframes while the threshold tracking condition or threshold GRIC score is met. In one embodiment, HSLAM can assume that the keyframe position of all panoramic keyframes is the same position as the last 6DOF keyframe that was considered before switching from 6DOF tracking to panoramic SLAM. .

  In one embodiment, HSLAM can use a pose refinement algorithm to collectively process a mixed set of finite and infinite features. Initializing the pose first, the pose refinement algorithm calculates an updated 6DOF / 3DOF pose and its corresponding two-dimensional (2D) image measurements from a set of finite and infinite map features. be able to. Incremental pose updates are calculated by iteratively optimizing reprojection errors for both finite and infinite map features.

  In one embodiment, a 6DOF pose is calculated if a threshold number of finite features can be utilized. In some embodiments, a set of features simply constructed from infinite features can result in 3DOF poses instead of 6DOF poses. In one embodiment, the HSLAM pose refinement algorithm can seamlessly switch between panoramic SLAM and 6DOF SLAM (eg, 6DOF tracking). HSLAM can be tracked temporarily using an infinite point and switched to a finite point if it is available (eg if a finite feature point can be determined from an image of a captured scene) . In one embodiment, if tracking is lost, HSLAM may perform relocalization using a global SLAM map. If tracking is lost, HSLAM can perform full relocalization using Small Blurry Images (SBI) for all available keyframes. Alternatively, HSLAM can perform relocalization using descriptor matching. HSLAM can attempt to re-localize the global SLAM map using 6DOF keyframes as well as panoramic keyframes.

  FIG. 2 shows a flow diagram of a hybrid SLAM in one embodiment. At block 205, one embodiment (eg, HSLAM) receives a 3D map of the environment. For example, HSLAM can process wide area SLAM maps. A 3D map can have features with a finite depth observed in more than one keyframe. Each key frame may be a panoramic key frame or a regular key frame. A 3D map can have features with infinite depth observed within one or more panoramic key frames.

  At block 210, the embodiment tracks the camera at 6 DOF from the finite or infinite feature of the 3D map observed in the current frame. The camera movement may be a global camera movement or a pure rotation camera movement. In one embodiment, HSLAM can estimate 6 DOF poses from finite features and can estimate 3 DOF poses from infinite features. Tracking panoramic camera movement can continue beyond the pre-existing boundaries of the received 3D map (eg, a wide area SLAM map). For example, an embodiment can use panoramic SLAM to track and map new regions and add them to the received 3D map. The wide area SLAM map may include one or more of keyframes, triangulated feature points, and the coupling (observation) between keyframes and feature points.

  A key frame may consist of a captured image (eg, an image frame captured by device camera 114) and the camera parameters used to generate the captured image. As used herein, camera parameters include camera position and orientation (pose). A wide area SLAM map can include finite and infinite features. In one embodiment, HSLAM can incorporate a camera image resulting from a rotation-only motion into an existing 3D map if the camera image does not meet a sufficient parallax threshold or translation threshold.

  In one embodiment, when HSLAM detects a pure rotating camera motion, it selects the first panoramic key frame as the reference key frame (ie, the reference panoramic key frame). The first panoramic key frame can be localized with respect to the 3D map. For example, when HSLAM detects a transition from 6DOF camera movement to panoramic camera movement, it can select the first key frame received. HSLAM can select additional panoramic keyframes (eg, dependent keyframes) that are probably not localized. Additional panoramic key frames can later be localized to the 3D map as part of the mapping process. HSLAM can localize additional key frames by generating a match with existing map features (eg, using active search and descriptor matching techniques). When localized, HSLAM will infinitely feature panoramic keyframes (i.e., infinite depth features), (a) match them with the features of other localized keyframes, and (b) the resulting 2D -2D codes can be transformed by triangulation (eg matching infinite features) to obtain additional 3D map features. In turn, new 3D map features can be used to localize other panoramic key frames that are not localized.

  FIG. 3 illustrates a first stage of a hybrid SLAM map representation between keyframes and features in one embodiment. This first stage shows a 6DOF key frame 320 observing a finite map feature 305. The local panorama map 350 can be positioned in a 3D map (eg, a wide area SLAM map) via a reference panorama key frame 330 having a finite feature 305 observation and an infinite feature 310 observation, while the remaining dependent panorama key frame 315. Can observe the infinite feature 310.

  FIG. 4 illustrates a second stage of a hybrid SLAM map representation between keyframes and features in one embodiment. In the second stage, (a) between an additional 6 DOF keyframe 410 and a localized panoramic keyframe (e.g., reference panoramic keyframe 430), or (b) a different local panoramic map (e.g., panorama map `` A '') Infinite features 310 can be triangulated from corresponding observations aligned between localized panoramic key frames (eg, reference panoramic key frame 430) from 440 and panoramic map “B” 450). Additional features may allow localization of other panoramic key frames (eg, dependent panoramic key frame 415).

  Robust localization of panoramic key frames may be an iterative process to find new 2D observations of finite 3D map features within panoramic key frames. Once sufficient 2D observations for finite 3D map features are established, panoramic frames can be localized using all 6DOF poses and converted to regular (ie non-panoramic) keyframes. After conversion to regular key frames, HSLAM can triangulate additional infinite feature points (eg, 2D features), thereby localizing other panoramic key frames as well.

  FIG. 5 shows a flowchart of hybrid SLAM initialization in one embodiment. At block 505, an embodiment (eg, HSLAM) can receive the captured image. For example, the captured image may be a camera image or an image from a video feed.

  At block 510, the embodiment may initialize the HSLAM by generating an initial 3D map or adding information to an existing 3D map 515 and outputting a camera position and orientation (pose) 520. . HSLAM initialization may include processing the captured image or images to create a 3D map (eg, a wide area SLAM map) with a consistent scale. In some embodiments, at the beginning of loading an application onto device 100, HSLAM can load a model-based detector and tracker to generate an initial map. Upon detecting a known planar image target, HSLAM can generate a first 6DOF keyframe. HSLAM can keep track of image targets and perform 2D-2D code inter-frame alignment. If enough signs can be triangulated robustly, the second 6DOF keyframe is selected. Thus, two regular 6DOF keyframes and the resulting finite map features can constitute an initial 3D map.

  The 3D map can be composed of finite point features and infinite point features with 2D image observation in regular 6DOF keyframes and panoramic keyframes. Each captured image can have an associated camera pose at the time that the respective image was captured by the camera. In one embodiment, HSLAM can extend the ability to track 6DOF to track wide area SLAM maps during pure camera rotation. In one embodiment, HSLAM may also incorporate key frames generated during pure camera rotation into a global SLAM map.

  The current camera pose can be estimated by a simple constant collapse motion model. HSLAM filters features for visibility from inferred camera poses, infinite features of the current panorama map (eg panorama map) and superimposes feature reprojection on infinite features when finite is preferred By doing so, a set of features for matching can be selected from all the global SLAM map features. Embodiments can then actively search for each selected feature in the current frame using NCC as the score function. A match with a sufficiently high NCC score can be added to the code set processed by the unified relative pose refiner. The pose refiner can output either an updated 6DOF pose or a 3DOF pose. If the incremental pose estimation fails, re-localization is entered where a 6DOF pose can be output.

  FIG. 6 illustrates 6DOF and panoramic mapping and tracking phases, according to one embodiment, with alternating total camera motion and pure rotation camera motion. Camera motion (eg, global camera motion) can be tracked in 6DOF from a 3D map 605 (eg, a wide area SLAM map). Refine and extend 3D maps using drop keyframes. Switching to rotation only camera motion 625 creates a local panorama map 610 using drop key frames. Camera tracking can be performed using panoramic map features and 3D map features. Global camera motion can destroy tracking, resulting in 6DOF camera pose relocalization 635. The overall camera motion can be returned on the 3D map and can be transitioned 640 smoothly by tracking finite and infinite features.

  FIG. 7 shows a state diagram for different states of key frame selection during mapping. After HSLAM initialization 510, the system starts operating in full 6DOF mapping mode 755. If a pure rotational motion is detected 760, a new panorama map is generated (eg, 3DOF mapping 765). Pure rotational motion can be detected by HSLAM based on the history of tracked 6DOF poses. The tracked 6DOF poses can be stored in memory in chronological order. HSLAM can calculate the parallax angle between the current pose and the stored pose and abandon all poses with large parallax (eg, greater than 5 degrees). The 6DOF measurement 770 can return the system to the full 6DOF mapping mode 755. If tracking fails, relocalization 775 can recover all 6 DOF poses.

  FIG. 8 is a block diagram of a hybrid SLAM system that includes a tracking component and a mapping component in one embodiment. A component may be a thread, an engine, or a module implemented as hardware or software. In one embodiment, HSLAM can estimate 6DOF poses from sufficient finite and infinite features that allow tracking of total camera motion and pure rotational camera motion. Upon determining that the pure rotating camera motion is heading towards an unmapped scene area, HSLAM can continue the hybrid 3D and panorama map tracking 815 and can also assign the hybrid keyframe 845 to the 3D map 865 . Upon determining that the pure rotating camera motion is heading towards the unmapped scene area, HSLAM can switch to pure panorama tracking 820 and assign a panorama key frame 850 to the local panorama map 870. If it is determined that the overall camera motion is toward the scene area to be mapped, the HSLAM can transition back on the wide area SLAM map (eg, 3D map 865). If you decide that the overall camera motion is heading towards an unmapped scene area, tracking will fail and relocalization is required, or select regular 6DOF keyframes based on sufficient disparity and narrow coverage Can do. In either case, HSLAM can move back on the 3D map, track 810, and assign a 6DOF keyframe 840 to the 3D map 865.

  In one embodiment, the HSLAM pose tracking and keyframe selection component 825 processes a calibrated single camera captured image (e.g., a video stream or feed), and against a 3D map 865 (e.g., a wide area SLAM map). Only global camera motion and rotation can track camera motion.

  The tracking component can dynamically and seamlessly switch between full 6D tracking mode and panoramic tracking mode depending on the current movement performed by the user. The tracking component can handle temporary rotation away from the mapped portion of the scene that the user often practices. The tracking component can detect these rotations and select a special “panorama” keyframe that is used to create a local panorama map. The local panorama map is positioned within a single consistent 3D map. Global camera motion and rotational only camera motion can be tracked against a global SLAM map that can include finite and infinite features. In one embodiment, HSLAM enables robust frame rate camera pose tracking and relocalization. Pose estimation can combine measurements for both finite features (known 3D locations) and infinite features, and HSLAM can automatically calculate pose updates 830 for either 6DOF or 3DOF . In one embodiment, if incremental pose tracking fails, HSLAM can be re-localized based on the micro-blurred image.

  HSLAM can extract features from keyframe images. As used herein, features (eg, feature points or important points) are important or noteworthy portions of an image. Features extracted from the captured image can represent completely different points along a three-dimensional space (e.g. coordinates on axes X, Y and Z), and all feature points are related features You can have a location. The feature in the keyframe is either aligned with the feature of the already captured keyframe or has failed to match (i.e., is the same as or corresponds to the feature of the already captured keyframe. ing). Feature detection may be an image processing operation that examines all pixels to determine if a feature exists at a particular pixel. Feature detection can process the entire captured image, alternatively a specific part or part of the captured image.

  Once a feature is detected for each captured image or video frame, a local image patch around that feature can be extracted. Features can be extracted using well-known techniques such as Scale Invariant Feature Transform (SIFT) that localize the features and generate their descriptions. Other techniques such as Speed Up Robust Features (SURF), Gradient Location-Orientation histogram (GLOH), Normalized Cross Correlation (NCC), or other comparable techniques can be used as needed. If it is determined that the number of features extracted for an image exceeds a threshold (eg, 100 features or another number of points), the images and features can be saved as key frames.

  The mapping component 875 can improve map quality through data combining 855 refinement and bundle adjustment optimization 860. HSLAM can perform keyframe selection to select 6DOF and panoramic keyframes 840-850 for inclusion in the 3D map 865. The mapping component 875 can send 3D map data 835 to the tracking component 825 to assist in relocalization. In addition, HSLAM can localize panoramic keyframes to extend 3D maps and triangulate infinite features.

  In one embodiment, HSLAM executes individual mapping components (e.g., threads, engines or modules such as mapping component 875 described above) to improve the quality of wide area SLAM maps (i.e. 3D maps). Can do. For example, the mapping component 875 can perform one or more types of optimization 860 (eg, 3D bundle adjustment). HSLAM can also estimate all 6 DOF poses for panoramic keyframes and triangulate infinite features to extend the 3D map.

  As part of data combination refinement 855, HSLAM searches for new keyframe-feature observations to further constrain existing feature locations and keyframe poses. HSLAM can apply active search and descriptor matching techniques to establish 2D-2D codes. HSLAM can also detect and abandon isolated observations and features.

  HSLAM can robustly localize panoramic keyframes for finite map features. Since poses cannot be accurately estimated from infinite features in all 6 DOFs, panoramic keyframes can initialize poses that are considered unreliable from panorama tracking. However, by establishing a match to existing finite map features, HSLAM can estimate all 6 DOF poses. Therefore, HSLAM effectively converts panoramic key frames to regular 6DOF key frames.

  HSLAM can leverage the information stored in the local panorama map for 3D mapping by triangulating infinite feature observations. HSLAM can use descriptor matching to find 2D-2D codes between robustly localized keyframes in separate local panoramic maps, eg, looking at the same scene region. A sign that passes the verification test constitutes an additional finite map feature. Therefore, HSLAM can effectively convert infinite features into finite features.

  HSLAM may perform map optimization 860 using bundle adjustment. Bundle adjustment updates the 6DOF pose of localized keyframes and the 3D position of finite map features by minimizing the cost function based on keyframe-feature observations. Non-localized panoramic key frames and infinite features cannot be part of the optimization. However, HSLAM can maintain map consistency by adjusting the positioning of the panoramic map within the optimized 3D map.

  In one embodiment, upon determining that the camera pose can be fully constrained within 6DOF, HSLAM can mark or tag each keyframe as a 6DOF keyframe. For example, when sufficient finite feature points are part of the pose estimation, as described below with respect to pose tracking. In addition, HSLAM can select regular 6DOF keyframes if it generates sufficient disparity for existing keyframes while the keyframes image a new part of the scene. Using parallax can ensure robust feature triangulation.

  Parallax is the scale-dependent triangulation angle of 3D point locations (eg, finite 3D map features) observed from two camera views (eg, current camera view, keyframe camera view). HSLAM is the disparity of the current camera view as a function of the average scene depth (for example, the average depth of finite map features observed in the current frame) and the distance between the current camera location and the existing keyframe camera location. The angle can be approximated. Coverage is the ratio of the image frame area covered by a finite map feature that is projected into a camera view (eg, current camera view, keyframe camera view). HSLAM can divide an image frame into regular grids with cells, and can project finite map features using camera poses. A grid cell with the least number of features included is considered to be covered. Coverage is the ratio of the number of covered lattice cells to all lattice cells.

  HSLAM can select regular 6DOF keyframes based on sufficient disparity and narrow coverage. Parallax is necessary for robust feature triangulation. Coverage indicates whether the current frame pose is robustly constrained with the projected map features. Narrow coverage indicates that the camera is observing an unmapped scene area.

  If HSLAM detects that the coverage is narrow and the disparity between the current frame and the existing keyframes is not sufficient, tracking will fail if 3D map features can no longer be observed in the current frame there is a possibility. If the camera motion is close to pure rotation, HSLAM can trigger the selection of localized panoramic keyframes. Narrow coverage can indicate that the camera point is heading towards an unmapped scene area. However, HSLAM cannot generate regular 6DOF keyframes due to the small parallax of purely rotating camera motion. Therefore, HSLAM can generate panoramic key frames localized to the 3D map.

  HSLAM can detect pure rotating camera motion based on the history of tracked 6DOF poses. The tracked 6DOF poses are stored in the history in chronological order. HSLAM can calculate the parallax angle between the current pose and history and abandon all poses with sufficiently large parallax. The remaining history poses can have a 3D location similar to the current frame. Finally, when HSLAM finds a pose in the history that has a small parallax and a large angle with respect to the current frame, HSLAM calculates the angle between the viewing directions and detects the pure rotation. Can do.

  HSLAM can continue selecting panoramic key frames based on narrow coverage and sufficient rotation. Narrow coverage can indicate that the camera continues to explore scene areas that are not mapped. HSLAM can calculate the rotation as the differential angle between the current frame viewing direction and the current panoramic map keyframe pose. HSLAM can implicitly move back to a more holistic movement when viewing a portion of the 3D map again. In overall operation, HSLAM can apply the same criteria to generate a new 6DOF keyframe.

  As described above, the device 100 may be a portable electronic device (e.g., a smartphone, a dedicated augmented reality (AR) device, a gaming device, a wearable device such as glasses, or other device having AR processing and display capabilities. ). Devices that implement the AR system described herein can be used in a variety of environments, such as shopping malls, roads, rooms, or any place where a user can take a portable device. In the AR context, the user can use the device 100 to view a real world representation through the display of his device. The user receives a real world image / video using his device's camera and overlays or superimposes additional or alternative information on the real world image / video displayed on the device, You can interact with your AR-enabled device. Users can replace or change real-world objects or scenes in real time on the device display when viewing the AR implementation on their devices. Virtual objects (eg, text, images, video) can be inserted into the representation of the scene depicted on the device display.

  As device 100 and camera 114 move, the display will update the extension of the target (eg, one or more objects or scenes) in the wide area SLAM map in real time. As the device moves away from the initial reference image position, the device can capture additional images from the alternate view. Extracting features and triangulating from additional keyframes can achieve improved extended accuracy (e.g., better fit the perimeter of the object and better represent the object in the scene) It can appear realistic and the target can be placed more accurately with respect to the camera 114 pose).

  In one embodiment, objects or graphics can be inserted or integrated into the video stream (or image) captured by the camera 114 and displayed on the display 112. HSLAM can optionally prompt the user for additional information to extend the goal. For example, the user can extend the target representation by adding user content. The user content may be an image, 3D object, video, text, or other content type that can be integrated, superimposed, or replaced with a representation of the target.

  The display can be updated from the original scene in real time with seamless tracking. For example, the text on the sign can be replaced with alternative text, or the 3D object can be strategically placed in the scene and displayed on the device 100. As the user changes the position and orientation of the camera 114, the graphic or object can be adjusted or expanded to match the relative movement of the camera 114. For example, if the virtual object is inserted into an expanded realistic display, camera movement away from the virtual object can reduce the size of the virtual object in proportion to the distance the camera 114 has moved. For example, if you step back four steps from the virtual object, the size of the virtual object will be greatly reduced compared to a half-step step back from the virtual object, and the same applies to all other variables. The motion figure or animation can be animated within the scene expressed by HSLAM. For example, an animated object can be “moved” within a scene depicted in an expanded realistic display.

  Those skilled in the art will recognize that the embodiments described herein can be implemented in ways other than AR (eg, robot positioning).

  HSLAM can be implemented as software, firmware, hardware, modules or engines. In one embodiment, the HSLAM description can be implemented by a general purpose processor 161 in the device 100 to achieve the desired function. In one embodiment, HSLAM can be implemented as an engine or module that can include an image processing module 171, a 6DOF module 173, and a panorama module 175 as sub-components. In other embodiments, one or more features of the described subcomponents can be combined or divided into different individual components, modules or engines.

  The teachings herein can be incorporated into various apparatuses (eg, devices) (eg, implemented in or performed by various apparatuses (eg, devices)). In one embodiment, the ITC can be implemented as an engine or module that is executed by a processor to receive an image or video as input. One or more aspects taught herein include a phone (eg, a cellular phone), a personal data assistant (“PDA”), a tablet, a mobile computer, a laptop computer, a tablet, an entertainment device (eg, a music device or video). Devices), headsets (e.g. headphones, earpieces, etc.), media devices (e.g. biometric sensors, heart rate monitors, pedometers, EKG devices, etc.), user I / O devices, computers, servers, point-of-sale devices, entertainment It can be incorporated into a device, set top box or any other suitable device. These devices have different power and data requirements and may result in different power profiles generated for each feature or set of features.

  In some aspects, the wireless device may comprise an access device (eg, a Wi-Fi access point) for the communication system. Such an access device can provide connectivity to another network (eg, a wide area network such as the Internet or a cellular network) through the transceiver 140, eg, via a wired or wireless communication link. Thus, an access device can allow access to another network or some other function by another device (eg, a Wi-Fi station). Furthermore, it should be understood that one or both of the devices may be portable or, in some cases, relatively non-portable.

  Those of skill in the art will understand that information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols and chips that can be referred to throughout the above description are represented by voltage, current, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof. be able to

  The various example logic blocks, modules, engines, circuits, and algorithm steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. Will be further understood by those skilled in the art. To clearly illustrate this interchangeability of hardware and software, various example components, blocks, modules, engines, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Those skilled in the art can implement the described functions in a variable manner for each particular application, but such implementation decisions should not be construed as departing from the scope of the present invention.

  Various exemplary logic blocks, modules, and circuits described in connection with the embodiments disclosed herein are general purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), fields A programmable gate array (FPGA) or other programmable logic device, individual gate or transistor logic, individual hardware components, or any combination thereof designed to perform the functions described herein Can be implemented or implemented. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors associated with a DSP core, or any other such configuration. It is also possible.

  The method or algorithm steps described in connection with the embodiments disclosed herein may be implemented directly in hardware, in software modules executed by a processor, or in a combination of the two. Can be Software modules reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art be able to. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium can reside in an ASIC. The ASIC can reside in the user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.

  In one or more exemplary embodiments, the functions or modules described are among hardware (eg, hardware 162), software (eg, software 165), firmware (eg, firmware 163), or any combination thereof. Can be implemented. When implemented in software as a computer program product, the functions or modules can be stored or transmitted as one or more instructions or code on a non-transitory computer-readable medium. Computer-readable media can include both computer storage media and computer communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media may be any available media that can be accessed by a computer or data processing device / system. By way of example, and not limitation, such non-transitory computer readable media can be RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage device, or instructions or data structure Any other medium that may be used to carry or store the desired program code in the form of and that may be accessible by a computer may be included. Also, any connection is properly termed a computer-readable medium. For example, software can use a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology such as infrared, wireless, and microwave, from a website, server, or other remote source When transmitted, coaxial technology, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the media definition. As used herein, a disk and a disc are a compact disc (CD), a laser disc (registered trademark), an optical disc, a digital versatile disc (DVD), a floppy disc (registered trademark), and Including Blu-ray discs, the disk normally reproduces data magnetically, and the disc optically reproduces data with a laser. Combinations of the above should also be included within the scope of non-transitory computer readable media.

  The above description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be used without departing from the spirit or scope of the invention. It can be applied to the embodiment. Accordingly, the present invention is not intended to be limited to the embodiments shown herein, but is consistent with the broadest scope consistent with the principles and novel features disclosed herein. Shall.

100 devices
111 Motion sensor
112 display
114 camera
140 transceiver
150 User interface
152 Keypad
154 microphone
156 Speaker
160 Control unit
161 processor
162 hardware
163 Firmware
164 memory
165 software
170 Hybrid SLAM
171 Image processing module
173 6DOF SLAM
175 Panorama SLAM
177 Bus
305 Finite map feature, Finite feature
310 Infinite features
315, 415 subordinate panoramic keyframes
320, 410 6DOF key frame
330, 430 standard panoramic keyframes
350, 610 Local panorama map
440 Panorama Map “A”
450 Panorama Map “B”
510 HSLAM initialization
605, 865 3D map
625 rotation only camera movement
635 6DOF camera pose relocalization
640 smooth transition
755 All 6DOF mapping mode
760 Pure rotational motion detection
765 3DOF mapping
770 6DOF measurement
775 relocalization
810 tracking
815 Hybrid 3D and panoramic map tracking
820 pure panorama tracking
825 HSLAM pause tracking and keyframe selection components
830 Pause update
835 3D map data
840 panoramic key frame, 6DOF key frame
845 hybrid key frame
850 panoramic keyframes
855 Data Join
860 Bundle adjustment optimization
870 Local panorama map
875 mapping components

Claims (28)

A machine-implemented method for simultaneous monocular visual localization and mapping comprising:
Receiving a three-dimensional (3D) map of the environment, wherein the 3D map is
A feature having a finite depth observed within two or more key frames, each key frame being a panoramic key frame or a regular key frame;
Comprising a feature having an infinite depth observed within one or more panoramic key frames;
Tracking the camera in 6 degrees of freedom (6 DOF) from a finite depth feature or an infinite depth feature of the 3D map observed in an image frame from an input image feed. When transitioning from the 6DOF camera movement to a panoramic camera movement toward an unmapped scene area, selecting a reference panoramic keyframe;
Incorporating the reference panoramic keyframe into the 3D map by adding finite depth feature observation and infinite depth feature observation to the 3D map;
Initializing a local panoramic map positioned in the 3D map, comprising:
Assigning a reference panorama keyframe to the local panorama map;
2. The machine-implemented method of claim 1, further comprising: using the 6DOF pose of the reference panoramic keyframe to position the local panorama map within the 3D map. Selecting one or more subordinate panoramic keyframes toward a scene area where continuous panoramic camera movement is not mapped, the one or more subordinate panoramic keyframes subordinate to a reference panoramic keyframe; ,
Incorporating the one or more subordinate panoramic keyframes into the 3D map by adding infinite depth feature observations to the 3D map;
The machine-implemented method of claim 1, further comprising expanding the local panorama map by adding the one or more dependent panoramic key frames to the local panorama map. Further comprising localizing the one or more panoramic key frames relative to the 3D map, the localizing step comprising:
Finding a two-dimensional (2D) observation of the finite depth feature in the one or more panoramic key frames;
Determining a 3D-2D code between the 3D map and the 2D observation of the finite depth feature;
2. Estimating a 6DOF camera position and orientation of the one or more panoramic keyframes using the 3D-2D code. Converting an infinite depth feature from a first localized panoramic keyframe to a new finite depth feature for the 3D map, the converting step comprising:
Finding a 2D observation of the infinite depth feature in a second localized keyframe, wherein the second localized keyframe is a localized panoramic keyframe or a localized regular keyframe When,
Determining a 2D-2D code from the 2D observation of the second localized keyframe;
2. The machine-implemented method of claim 1, comprising: triangulating the new finite depth feature based on the 2D-2D code and the 6DOF camera position and orientation of the key frame pair. The step of tracking comprises:
Establishing a sign between the finite depth feature and the infinite depth feature of the 3D map, and an image frame from an input image feed;
The machine-implemented method of claim 1, further comprising: estimating a 6DOF camera position and orientation based on the established code. The step of tracking comprises:
Observing only infinite depth features in the image frame from the input image feed, switching from 6DOF camera movement tracking to panoramic camera movement tracking;
The machine-implemented method of claim 1, further comprising: observing a finite depth feature in the image frame from the input image feed and switching from panoramic camera movement tracking to 6DOF camera movement tracking. A machine readable non-transitory storage medium containing executable program instructions that will cause a data processing device to perform a method for simultaneous monocular visual localization and mapping, the method comprising:
Receiving a three-dimensional (3D) map of the environment, wherein the 3D map is
A feature having a finite depth observed within two or more key frames, each key frame being a panoramic key frame or a regular key frame;
Comprising a feature having an infinite depth observed within one or more panoramic key frames;
Tracking the camera in 6 degrees of freedom (6 DOF) from a finite depth feature or an infinite depth feature of the 3D map observed in an image frame from an input image feed. When transitioning from the 6DOF camera movement to a panoramic camera movement toward an unmapped scene area, selecting a reference panoramic keyframe;
Incorporating the reference panoramic keyframe into the 3D map by adding finite depth feature observation and infinite depth feature observation to the 3D map;
Initializing a local panoramic map positioned in the 3D map, comprising:
Assigning a reference panorama keyframe to the local panorama map;
9. The medium of claim 8, further comprising: positioning the local panorama map within the 3D map using a 6DOF pose of the reference panorama key frame. Selecting one or more subordinate panoramic keyframes toward a scene area where continuous panoramic camera movement is not mapped, the one or more subordinate panoramic keyframes subordinate to a reference panoramic keyframe; ,
Incorporating the one or more subordinate panoramic keyframes into the 3D map by adding infinite depth feature observations to the 3D map;
9. The medium of claim 8, further comprising expanding the local panorama map by adding the one or more dependent panorama key frames to the local panorama map. Further comprising localizing the one or more panoramic key frames relative to the 3D map, the localizing step comprising:
Finding a two-dimensional (2D) observation of the finite depth feature in the one or more panoramic key frames;
Determining a 3D-2D code between the 3D map and the 2D observation of the finite depth feature;
9. Estimating a 6DOF camera position and orientation of the one or more panoramic keyframes using the 3D-2D code. Converting an infinite depth feature from a first localized panoramic keyframe to a new finite depth feature for the 3D map, the converting step comprising:
Finding a 2D observation of the infinite depth feature in a second localized keyframe, wherein the second localized keyframe is a localized panoramic keyframe or a localized regular keyframe When,
Determining a 2D-2D code from the 2D observation of the second localized keyframe;
9. The medium of claim 8, comprising: triangulating the new finite depth feature based on the 2D-2D code and the 6DOF camera position and orientation of the key frame pair. The step of tracking comprises:
Establishing a sign between the finite depth feature and the infinite depth feature of the 3D map, and an image frame from an input image feed;
9. The medium of claim 8, further comprising: estimating a 6DOF camera position and orientation based on the established code. The step of tracking comprises:
Observing only infinite depth features in the image frame from the input image feed, switching from 6DOF camera movement tracking to panoramic camera movement tracking;
9. The medium of claim 8, further comprising switching from panoramic camera movement tracking to 6DOF camera movement tracking upon observing a finite depth feature in the image frame from the input image feed. A data processing device for simultaneous monocular visual localization and mapping comprising:
A processor;
A storage device coupled to the processor, wherein when executed by the processor, the processor
Receiving a three-dimensional (3D) map of the environment, wherein the 3D map is
A feature having a finite depth observed within two or more key frames, each key frame being a panoramic key frame or a regular key frame;
Comprising a feature having an infinite depth observed within one or more panoramic key frames;
Tracking the camera in six degrees of freedom (6 DOF) from a finite depth feature or an infinite depth feature of the 3D map observed in an image frame from an input image feed, and configured to store instructions A data processing device comprising: a storage device capable of: The processor is
When transitioning from the 6DOF camera movement to a panoramic camera movement toward an unmapped scene area, selecting a reference panoramic keyframe;
Incorporating the reference panoramic keyframe into the 3D map by adding finite depth feature observation and infinite depth feature observation to the 3D map;
Initializing a local panoramic map positioned in the 3D map, comprising:
Assigning a reference panorama keyframe to the local panorama map;
16. The device of claim 15, further comprising: steps including: positioning the local panorama map within the 3D map using a 6DOF pose of the reference panoramic keyframe. The processor is
Selecting one or more subordinate panoramic keyframes toward a scene area where continuous panoramic camera movement is not mapped, the one or more subordinate panoramic keyframes subordinate to a reference panoramic keyframe; ,
Incorporating the one or more subordinate panoramic keyframes into the 3D map by adding infinite depth feature observations to the 3D map;
16. The device of claim 15, further comprising instructions that: performing the step of expanding the local panorama map by adding the one or more dependent panorama key frames to the local panorama map. The processor further comprising instructions that will perform the step of localizing the one or more panoramic key frames relative to the 3D map, the localizing step comprising:
Finding a two-dimensional (2D) observation of the finite depth feature in the one or more panoramic key frames;
Determining a 3D-2D code between the 3D map and the 2D observation of the finite depth feature;
16. The device of claim 15, comprising: performing a step of estimating a 6DOF camera position and orientation of the one or more panoramic key frames using the 3D-2D code. The processor further comprising instructions to perform the step of converting an infinite depth feature from a first localized panoramic keyframe to a new finite depth feature for the 3D map, the converting step comprising:
Finding a 2D observation of the infinite depth feature in a second localized keyframe, wherein the second localized keyframe is a localized panoramic keyframe or a localized regular keyframe When,
Determining a 2D-2D code from the 2D observation of the second localized keyframe;
16. The device of claim 15, comprising instructions to: triangulate the new finite depth feature based on the 2D-2D code and the 6DOF camera position and orientation of the keyframe pair. The step of tracking comprises:
Establishing a sign between the finite depth feature and the infinite depth feature of the 3D map, and an image frame from an input image feed;
16. The device of claim 15, further comprising instructions that: perform a step of estimating a 6DOF camera position and orientation based on the established code. The step of tracking comprises:
Observing only infinite depth features in the image frame from the input image feed, switching from 6DOF camera movement tracking to panoramic camera movement tracking;
The device of claim 15, further comprising: upon observing a finite depth feature in the image frame from the input image feed, switching from panoramic camera movement tracking to 6DOF camera movement tracking. A device for simultaneous monocular visual localization and mapping comprising:
Means for receiving a three-dimensional (3D) map of an environment, said 3D map comprising:
A feature having a finite depth observed within two or more key frames, each key frame being a panoramic key frame or a regular key frame;
Means comprising: an infinite depth feature observed within one or more panoramic key frames;
Means for tracking the camera in 6 degrees of freedom (6 DOF) from a finite depth feature or an infinite depth feature of the 3D map observed in an image frame from an input image feed. Means for selecting a reference panoramic keyframe when transitioning from the 6DOF camera movement to a panoramic camera movement toward an unmapped scene area;
Means for incorporating the reference panoramic keyframe into the 3D map by adding finite depth feature observation and infinite depth feature observation to the 3D map;
Means for initializing a local panorama map positioned in the 3D map,
Means for assigning a reference panoramic keyframe to the local panorama map;
23. The apparatus of claim 22, further comprising: means for positioning the local panorama map within the 3D map using a 6DOF pose of the reference panorama key frame. Means for selecting one or more subordinate panoramic keyframes toward a scene area where continuous panoramic camera movement is not mapped, said one or more subordinate panoramic keyframes subordinate to a reference panoramic keyframe Means,
Means for incorporating the one or more subordinate panoramic keyframes into the 3D map by adding infinite depth feature observations to the 3D map;
23. The apparatus of claim 22, further comprising: means for extending the local panorama map by adding the one or more dependent panorama key frames to the local panorama map. Means for localizing the one or more panoramic key frames relative to the 3D map, the means for localizing comprising:
Means for finding a two-dimensional (2D) observation of the finite depth feature in the one or more panoramic key frames;
Means for determining a 3D-2D code between the 3D map and the 2D observation of the finite depth feature;
23. The apparatus of claim 22, comprising means for estimating a 6DOF camera position and orientation of the one or more panoramic key frames using the 3D-2D code. Means for converting an infinite depth feature from a first localized panoramic keyframe to a new finite depth feature for the 3D map, the means for converting comprising:
Means for finding a 2D observation of the infinite depth feature in a second localized keyframe, wherein the second localized keyframe is a localized panoramic keyframe or a localized regular keyframe With some means,
Means for determining a 2D-2D code from the 2D observation of the second localized keyframe;
23. The apparatus of claim 22, comprising: means for triangulating the new finite depth feature based on the 2D-2D code and the 6DOF camera position and orientation of the key frame pair. The means for tracking comprises:
Means for establishing a code between the finite depth feature and the infinite depth feature of the 3D map, and an image frame from an input image feed; and estimating a 6DOF camera position and orientation based on the established code 23. The apparatus of claim 22, further comprising: means for. The means for tracking comprises:
Means to switch from 6DOF camera movement tracking to panoramic camera movement tracking when only infinite depth features in the image frame from the input image feed are observed;
23. The apparatus of claim 22, further comprising: means for switching from panoramic camera movement tracking to 6DOF camera movement tracking upon observing a finite depth feature in the image frame from the input image feed.

Images