DE102019207888A1 - Methods and systems for exploiting motion conflicts per pixel for extracting primary and secondary movements in augmented reality systems - Google Patents

Abstract

A method for motion estimation in an Augmented Reality (AR) system includes receiving inertial sensor data and image data during movement of the AR system to generate a probability map based on the inertial sensor data and the image data, wherein the probability map corresponds to a frame in the image data and containing probability values indicating that each pixel in the one frame is located in an inertial coordinate frame or local coordinate frame with a convolutional neural network encoder / decoder, identifying visual observations of at least one landmark in the local coordinate frame based on the image data and of the probability map, and generating an estimate of secondary motion in the local coordinate frame based on a first previous state in a Hidden Markov Model (HMM) corresponding to the local coordinate frame and the visual n observations of the at least one landmark in the local coordinate frame corresponds.

Description

CLAIM OF PRIORITY

This application claims priority over the provisional one US Application No. 62 / 678,383 entitled "On Exploiting Per-Pixel Motion Conflicts to Extract Secondary Motions" filed May 31, 2018, the entire contents of which are hereby incorporated herein by reference.

FIELD OF THE INVENTION

This disclosure relates generally to the field of augmented reality systems for human-computer interaction, and more particularly to systems and methods for detecting primary and secondary motion in augmented reality systems.

GENERAL PRIOR ART

Unless otherwise stated herein, the materials described in this section are not prior art to the claims of this application and will not become part of the prior art by inclusion in this section.

Augmented reality (AR) systems use computerized devices to provide a man-machine interface that allows a human to view a real-world physical environment while providing an indication of virtual graphics that are features of the physical environment "augment". One common type of AR device includes a transparent pair of glasses with one or more sensors and a video projection device carried by a human operator. The transparent glasses allow people to see the physical environment of the real world, and the AR device projects graphical data onto the glasses or directly into the user's eyes at locations corresponding to objects, locations or other features in the physical environment. In some cases, the graphics display information or otherwise add graphic elements to "augment" a physical object in the physical environment, while in other cases the graphics provide a two-dimensional or three-dimensional representation of information or a virtual object in the physical Environment does not really exist. While augmented reality systems share some features with "virtual reality" (VR) systems, a distinction between AR and VR systems is that AR systems provide visual representation and graphical interaction with a physical environment of the real world that is not generated by a computer and is not under the control of a computer while VR systems provide graphical displays of fully computer generated environments. Thus, many operations in AR systems require further processing to enable the AR system to measure parameters of the physical environment around the AR system to provide accurate augmented graphics.

A function of an AR system performs a "localization" to identify the locations of the sensors in the AR system with respect to the environment around the system. In particular, many AR systems use a camera system and inertial sensors, such as MEMs accelerometers and gyroscopes, to perform the localization. The prior art systems combine the outputs of the camera and the inertial sensor to perform the localization. Because the sensors are generally integrated into a device carried by a user, locating the sensor locations also provides for localization of the user in the environment. Most prior art AR systems assume a static environment and handle only primary motion that corresponds to the inertial coordinate frame. That is, these systems can perform localization in a static, non-moving environment ("inertial coordinate frame") and then use the input of the AR sensors to determine the motion of the AR system and the user with respect to the static inertial coordinate frame ( "Primary movement"). A simple example of this is to measure the movement of a user wearing an AR system in a stationary space that provides the inertial coordinate frame.

Existing AR systems are much less effective at handling dynamic environments where there are multiple types of motion and multiple coordinate frames that produce inconsistent or "conflicting" sensor data inputs. As a frequent, non-limiting example, when a user wears an AR device in a moving motor vehicle, the real inertial reference coordinate frame, such as the non-moving road, appears to be in the move the optical input of the camera system sensors, while the inertial sensors in the AR device need not register any movement at all, when the wearer of the device sits still and does not move relative to the vehicle interior while the vehicle is moving at a uniform speed (which does not generate acceleration). , The vehicle interior is referred to as a "local coordinate frame" because the local movement of the user and the AR system is relative to the vehicle interior, even if the entire vehicle and the AR system also move relative to the inertial coordinate frame of the road. Further, any movement of the user and the AR system within the vehicle generates inertial motion data that does not match the perceived motion of the camera system because the movement of the vehicle relative to the inertial coordinate frame of the road substantially depends on the movement of a user relative to the local coordinate frame of the vehicle is different. Even the input data from the camera system is usually inconsistent since a portion of each generated image of the video data contains the local coordinate frame of the vehicle interior, which is static relative to the user, as long as the user is not moving, while another portion of the image is the external environment's inertial coordinate frame which seems to move relative to the local coordinate frame of the vehicle.

The above problems with existing AR systems reduce the accuracy of the primary motion of the AR sensors in the AR system and provide further challenges to the accurate tracking of "secondary motions" related to movements of the sensors in the AR system refer to a local coordinate frame. Using the above example of a moving vehicle, secondary movements occur in the local coordinate frame of the vehicle interior, such as the movement of the AR system itself, or from another object within the vehicle interior. Accurate tracking of the secondary motion requires close tracking of the primary motion, which is difficult in a local coordinate frame due to the sensor conflicts described above. For example, techniques are known in the art to track motion of moving objects in video data in an inertial coordinate frame, but when the camera moves in an unknown manner because of inaccurate primary motion tracking due to conflicting sensor data Accurate motion tracking of the camera relative to an external object in a local coordinate frame with the video data is significantly more difficult or impractical. Further, although some techniques are known in the art to improve the accuracy of primary motion tracking in situations involving multiple coordinate frames, the techniques rely on the identification and rejection of potentially conflicting sensor data representing the accuracy of primary motion detection improved based on the inertial coordinate frame, but prevents accurate detection of secondary motion for the local coordinate frame. Of course, while there are systems and methods for tracking relative motion using active sensors such as RADAR and LIDAR, these active sensors are not practical for use in many AR systems.

As described above, prior art AR systems have problems in performing localization and tracking of primary and secondary motion in situations where video and inertial sensor data experiences conflicts of relative motion due to an inertial coordinate and a local coordinate frame generated in the sensor data. As a result, improvements to AR systems and methods of operating them to improve primary and secondary motion detection accuracy would be advantageous.

BRIEF SUMMARY OF THE INVENTION

The embodiments described herein provide motion estimation systems and methods in AR systems that provide motion-conflict-detecting visual inertial (-VIO) tomography that allows an AR system to perform localization for an inertial coordinate frame and a local coordinate frame to improve perform primary motion detection and secondary motion detection.

In one embodiment, a motion estimation method has been developed in an Augmented Reality (AR) system. The method includes receiving inertial sensor data with an inertial sensor system during movement of the AR system; Receiving multiple frames of image data during movement of the AR system with a camera system; generating a probability map based on the inertial sensor data and a plurality of frames of the image data with a processor comprising a convolutional neural network encoder / decoder, wherein the probability map corresponds to a frame of the plurality of image data frames and including a plurality Probability values corresponding to a plurality of pixels in the one frame, each probability value in the plurality of probability values indicating a probability that the corresponding pixel is in an inertia coordinate frame or a local coordinate frame; identifying visual observations of at least one landmark with the processor in the local coordinate frame based on the plurality of frames of the image data and the probability map; and generating an estimate of secondary motion for the AR system with the processor with reference to the local coordinate frame based on a first previous state in a hidden Markov model (HMM) corresponding to the local coordinate frame and the visual observations of the at least one Landmark in the local coordinate frame.

In another embodiment, an Augmented Reality (AR) system has been developed. The AR system includes an inertial sensor system, a camera system, a display device, and a processor operatively connected to the inertial sensor system, the camera system, and the display device. The processor is configured to receive inertial sensor data generated by the inertial sensor system during movement of the AR system; Receiving a plurality of frames of image data generated by the camera system during movement of the AR system; Generating a probability map based on the inertial sensor data and the plurality of frames of the image data using a convolutional neural network encoder / decoder, wherein the probability map corresponds to a frame of the plurality of frames of image data, and containing a plurality of probability values corresponding to a plurality of pixels in the one frame each of which probability value in the plurality of probability values indicates a probability that the corresponding pixel is in an inertia coordinate frame or a local coordinate frame; Identifying visual observations of at least one landmark in the local coordinate frame based on the plurality of frames of the image data and the probability map; and generating an estimate of secondary motion for the AR system with reference to the local coordinate frame based on a first previous state in a hidden Markov model (HMM) corresponding to the local coordinate frame and the visual observations of the at least one landmark in the local one coordinate frame.

list of figures

• 1 Figure 12 is a schematic diagram of an Augmented Reality (AR) system configured to detect primary and secondary movement.
• 2 is a diagram of a hidden Markov model (HMM) that makes up the AR system 1 used to track primary and secondary movement.
• 3 is a diagram of a process for operating the AR system 1 for estimating the primary and secondary motion using the HMM 2 ,
• 4 Figure 12 is a diagram of a convolutional neural network (CNN) encoder / decoder that includes an epipolar constrained layer that exposes the AR system 1 used in the process 3 ,
• 5 FIG. 13 is a diagram showing the operation of the epipolar restricted layer in the Encoder / Decoder for a Convolutional Neural Network 4 ,
• 6 is a receiver operating characteristic (ROC) curve that expresses the ratios of true-disposable detection to false-positive detection for the CNN encoder / decoder 4 in comparison with a CNN encoder / decoder which does not have an epipolar restricted layer.
• 7 FIG. 12 is a diagram illustrating the accuracy of the visual inertial odometry in an environment containing inertial and local coordinate frames using the AR system. FIG 1 ,
• 8A Figure 12 is a diagram illustrating the identification of an inertial coordinate frame and a global coordinate frame with landmarks and augmented graphics generated in the AR system 1 ,
• 8B Another is a diagram showing the operation of the AR system 1 represents augmented graphics with exact locations in an inertial coordinate frame and a local coordinate frame.

DETAILED DESCRIPTION

For the purposes of promoting an understanding of the principles of the embodiments disclosed herein, reference is now made to the drawings and descriptions in the following written specification. The references are not intended to limit the scope of the content. This disclosure also includes all changes and modifications of the illustrated embodiments, and includes other applications of the principles of the disclosed embodiments that would normally be apparent to one of ordinary skill in the art to which this disclosure pertains.

As used herein, the term "primary motion" refers to movement of an augmented reality (AR) device that includes a camera system and an inertial sensor system with respect to an inertial coordinate frame. In the context of an AR device, the inertial coordinate frame is a fixed coordinate system that is assumed to be static (immobile) while the AR device itself and potentially other objects are moving relative to the inertial coordinate frame. Non-limiting practical examples of inertial coordinate frames described in the following examples include the surface of the earth, fixed objects that do not move relative to the surface of the earth, walls, floors, ceilings and other static structures in buildings and the like.

As used herein, the term "secondary motion" refers to movement of an AR device that includes a camera system and an inertial sensor system with respect to an inertial coordinate frame. A non-limiting example of a local coordinate frame may include the movement of the AR system relative to an interior of a vehicle or to an object held in the hand of the user of the AR system, such as a book, clipboard. Notebook computers and the like included. While the AR system uses both the video data from a camera system and the inertial data from inertial sensors to track primary motion, the AR system uses only video data from the secondary motion tracking camera system, as the type of accelerometers and gyroscopes in inertial sensors use Inertia coordinate frame is linked.

As used herein, the term "conflict" refers to a situation where input data from two different sensors in an augmented reality system provide non-consistent ("conflicting") data for one or both of primary motion and secondary motion. For example, a camera system may generate pixel data indicative of primary motion in one direction, while an inertial sensor indicates primary motion in a different direction, resulting in conflict. The embodiments described herein detect conflicts of image data from a camera system per pixel, and use the conflict information to improve the accuracy of the primary and secondary motion detection.

As used herein, the term "landmark" refers to an object that detects an AR system in image sensor data that does not move over time relative to a coordinate frame, which may be either an inertial coordinate frame or a local coordinate frame. Examples of landmarks in an inertial coordinate frame include a tree, light poles, a shield, or other static object that does not move relative to the ground in the inertial coordinate frame. Examples of landmarks in a local coordinate frame include fixed elements of a dashboard and other fixed features of the interior of a vehicle in a situation where the vehicle interior forms the local coordinate frame.

A landmark is identified based on one or more visual observations that are generated based on a feature extraction process that is applied to a series of image frames that a camera system in the AR system generates in use. In the embodiments described herein, visual observations are generated using the Features from Accelerated Segment Test (FAST) algorithm or another feature extraction algorithm. While not described in detail herein, the FAST algorithm recognizes features corresponding to the vertices of the objects in image data by generating a circle of pixels around a candidate pixel for a corner, and identifying based on the pixel intensity level of the candidate pixel and pixels Surround the candidate that the candidate is actually a corner. Machine learning techniques, including decision trees, neural networks, and other image processing techniques, can be used to efficiently perform variations of the FAST algorithm. The embodiments described herein combine FAST with another technique known as Binary Robust Invariant Scalable Keypoints (BRISK) that performs key-point recognition based on the characteristics of FAST to generate the visual observations z as key-point descriptors. While the visual observations z are generated over a series of frames of image data, each visual observation z is assigned to a group of pixels in each frame, and as described in greater detail below, a deep neural network classifier generates a classification for each pixel as the inertial coordinate frame or the local coordinate frame.

In a practical AR system, the landmarks are not a priori known and must be estimated from the image sensor input data simultaneously with the processes that estimate the primary and secondary motion of the AR system as part of a process called simultaneous localization and allocation (SLAM) becomes. For SLAM, the estimation of the primary and secondary movements is associated with the localization process, while the landmark estimation is associated with the assignment process. As described in more detail below, the AR systems and methods described herein generate visual landmark observations, wherein the visual observations are generated based on multiple video frames acquired over time as the AR system moves relative to the inertial and spatial landmarks moved to the local coordinate frame. As the landmarks do not move, a change in the position of a landmark over time in the visual observations actually corresponds to a movement of the AR system, and the AR uses visual observations of the landmarks to correct the estimates using a hidden Markov mark. Model and the inertial sensor inputs generated movements. The embodiments of the AR system further classify the individual pixels in frames of image data than belonging to the inertial or local coordinate frame. The AR system then identifies landmarks in visual observations of multiple images and, based on the classifications of pixels from each image, classifies whether the landmarks belong to the inertial coordinate frame or the local coordinate frame. This classification process allows the estimation of secondary motion and improves the accuracy of primary motion estimation.

1 is a schematic diagram of an AR system 100 , The AR system 100 contains a set of AR glasses 104 configured to be worn on the head of a user and an AR control unit 140 that are either completely in the AR glasses 104 is integrated or embodied as one or more computing devices operative with the components in the AR goggles 104 are connected.

The AR glasses 104 contains glasses 108A and 108B , a camera system 112 that in the embodiment of 1 stereoscopic cameras 112A and 112B contains, and one or more inertial sensors 116 , The glasses 108A and 108B are transparent and allow a user to see the surrounding physical environment in a similar way to ordinary glasses or sunglasses. The glasses 108A and 108B either provide a transparent display that generates a graphic in an AR system to augment features in the physical environment without obscuring the view of the physical environment, or provide surfaces for projection of graphics from an integrated projection device in the AR glasses 104 ready. In yet another embodiment, the AR goggles are included 104 a projection device that can deliver graphics directly into the user's eyes, and these embodiments optionally leave the glasses 108A and 108B out. The camera system 112 creates stereoscopic frames with image data in video data to the physical environment around the AR glasses 104 record. The camera system 112 contains two cameras 112A and 112B Both frames of image data in a video stream with a physical separation between the two cameras 112A and 112B generate stereoscopic frames of image data. While the camera system 112 Providing stereoscopic image data, the primary and secondary motion estimation processes described herein can also be performed using monoscopic image data generated using a single camera. The inertial sensors 116 For example, a multiple axis accelerometer or multiple single axis accelerometers that measure linear acceleration in three directions include one or more gyroscopes that measure spin to provide inertial sensor data to measure the linear and angular accelerations that the AR goggles 104 learns during operation. In some embodiments, the inertial sensors are 116 as an inertial measurement unit (IMU) that uses microelectromechanical system (MEMs) devices to translate the accelerometers and gyroscopes.

The AR control unit 140 contains a processor 150 that with a memory 160 connected is. The processor 150 in the AR control unit 150 is operatively connected to input devices that provide user input to the AR system 100 provide, with graphical output display devices such as projectors or displays, the glasses 108A and 108B use or project graphics directly into the eyes of a user, the camera system 112 and the inertial sensors 116 ,

The processor 150 is a digital logic device embodying, for example, one or more of a microprocessor, microcontroller, field programmable gate array (FPGA), digital signal processor (DSP), application specific integrated circuit (ASIC), and the like. In an illustrative embodiment, the processor is 150 implemented as a system on a chip (SoC) that includes a central processing unit (CPU) with one or more cores, and a graphical processing unit (GPU) that implements hardware to speed the generation of graphics for display to the user , In some embodiments, the CPU, the GPU, or a separate digital logic device resides in the processor 150 Also hardware acceleration of conclusions using an encoder / decoder for a deep convolutional neural network around which the AR system 100 used to resolve conflicts in pixels of image frames generated by the camera system 112 be generated in situations where the AR system 100 is in an environment that contains both an inertia coordinate frame and a local coordinate frame.

The memory 160 includes one or more non-volatile memory devices, such as a solid-state or magnetic memory device, and one or more volatile memory devices, such as random access memory (RAM), for storing data in the AR system 100 , In 1 saves the memory 160 program instructions 164 that the processor 150 to perform the operations described herein. The memory 160 stores image data 168 that contain multiple frames of image data in a video stream, the camera system 112 during operation of the AR system 100 generated. The memory 160 stores inertial sensor data 172 containing the inertial sensors 116 during operation of the AR system 100 produce. The memory 160 saves the image data 168 and the inertial sensor data 172 with timestamps to the AR system 100 synchronizing frames of the image data from the camera system 112 with the inertial sensor data 172 to enable in the operation of the AR system 100 occur simultaneously. The memory 160 also stores an Encoder / Decoder for a Deep Convolutional Neural Network (CNN) 180 that uses an epipolar-restricted location to create a probability map 184 per pixel, which identifies the probability that each pixel in a frame of image data 168 in situations where the AR system receives conflicting sensor data, either the inertia coordinate frame or the local coordinate frame. The memory 160 also stores a Hidden Markov Model (HMM) 200 that the AR system 100 used equally for the primary motion estimation and secondary motion estimation, using a state model based on a previously estimated state for the primary and secondary motion and existing time inputs from the image data 168 , the inertial sensor data 172 , and the output of the probability map 184 per pixel. The memory 160 also stores AR graphics data 188 , The AR graphics data 188 For example, they include two-dimensional images, three-dimensional geometric models and textures for the three-dimensional geometric models, shader programs for execution by shader hardware in a GPU of the processor 150 , and other data, the AR system 100 used to generate graphics to provide an augmented reality output to the user. As explained in more detail below, the enhancements enable the primary motion estimation and the secondary motion estimation in the AR system 100 the processor 150 generating graphics with exact placement in the inertial coordinate frame and the local coordinate frame using the AR graphics data 188 ,

While 1 a single AR control unit 140 For illustrative purposes, alternative embodiments may show multiple computing devices that perform the functions of the AR system 100 using a local computing device carried by the user and one or more computing devices communicating with the local computing device via a wired or wireless data network. For example, an AR system, in some embodiments, uses external server computing devices to perform some of the functions described herein, while a smaller local processing device incorporated into the AR glasses 104 is integrated or carried separately by the user performing other functions described herein. So should all references to the AR control unit 140 and the operation of the processor 150 and memory 160 are understood as being equally applicable to a single computing device or a combination of multiple computing devices that are configured to work together to implement the embodiments described herein.

2 is a diagram of a Hidden Markov Model (HMM) 200 that the AR system 100 used to estimate the tracks of the primary movement and the secondary movement. 2 shows the HMM 200 containing a single group of states and inputs 204 contains, which occurs when the AR system 100 no movement conflict in group 204 experiences, and two separate groups of states and inputs 240 and 280 that estimate the primary movement (group 240 ) and the estimate of secondary movement (group 280 ) when the system experiences a conflict of sensor data because both an inertial coordinate frame and a local coordinate frame are present in the frame of the image data from the camera system. More detailed contains the group 204 multiple states representing the track in a single inertial coordinate frame, starting in an initial state 210A (Xo) and including other states 210B (X ₁ ) and 210C (X _m- ), where the state 210C (X _m- ) represents the final state estimate generated prior to the introduction of a local coordinate frame and the estimate of primary and secondary motion. In the group 204 from 2 The existing skew observations based on data from the inertial sensors are by nodes 208A (u ₁ ) and 208B (u _m- ) shown. In operation, the system determines 100 the next state of the series 210B - 210C based on the previous state and the existing input from the inertial sensor observations made in 208A-208B are shown. This is the state 210C based on the previous state 210B (Ignoring older states before the state 210B occurred) and the inertial observations 208B which is the current input for the state 210C represent, predicted. The inertial observations 208A (u ₁ ) and 208B (u _m- ) provide inputs to the HMM 200 to switch from one state to the next state to generate updated estimates of the primary motion.

In isolation, the combination of states and the inertial observations form a type of inertial navigation system that can provide accurate primary motion prediction in an unrealistic scenario in which the inertial observations 208A and 208B always provide perfect information about the movement of the AR system. In practical systems, however, even small errors of inertial observations accumulate 208A and 208B over time and the predicted motion values in the states produce results that are not sufficiently accurate to be used in practice with an AR system. Therefore, the HMM uses 200 also visual observations 212A ( z ₁ ) and 212B ( z ₂ ) generated based on a plurality of frames of video data and landmarks included in the multiple frames of the video data as by the landmark node 214 ( L ₁ ) are identified. Unlike the primary states of motion in nodes 210A - 210C , is the landmark node 214 a single node because the position of a landmark does not change over time. However, the precise set of observed landmarks may change over time due to changes in visual observations, such as when the cameras in an AR system acquire images of a Schildland mark over time, followed by further visual observations that do not include the Schildland mark when the cameras in the AR system no longer see the landmark. The final estimate of the movement is based on the prediction from the HMM 200 based on the previous state and existing inertia inputs, and a subsequent error minimization process using the visual observations z of the landmarks L to produce accurate motion estimates.

The AR system 100 works by using the HMM group 204 in situations where there is no conflict between sensor data from the inertial sensors and the camera system. The Markov assumption that a state X _k only from the entrance u _k and the previous state X _k-1 depends, however, does not apply as soon as a motion conflict occurs. When such a conflict occurs, the AR system receives 100 a user input that identifies an object that provides a local coordinate frame to the AR system 100 to provide estimates for the primary movement and the secondary movement alike. How closer in 8A and 8B As illustrated, a non-limiting embodiment of an object that provides the local coordinate frames is the dashboard of the interior of a motor vehicle. The AR system 100 then uses the two state groups in the HMM 200 to appreciate primary movement (group 240 ) and secondary movement (group 280 ). The primary movement group 240 Illustrated including the states 242A (X ^W ₁ ) and 242B ( X ^W ₂ ), and further states are generated over time during the process of primary motion estimation. The primary movement group 240 Also includes the inertial sensor inputs 244A ( u ₃ ) and 244B ( u ₄ ), which are the primary states of motion X continues over time. The visual observations 248A ( z ^W ₁ ) and 248B ( z ^W ₂ ), the landmarks from the landmark estimation node 250 ( L ^W ₁ ) correspond to where the landmarks 250 Contain landmarks in the inertial coordinate frame that correspond to primary motion and landmarks in the local coordinate frame that correspond to secondary motion. The visual observations 248A and 248B correspond to landmarks in the image data determined in a separate process described below, but each landmark is also assigned to one or more pixels in the original image data. In particular, the embodiments described herein use a "rare" feature recognition process that recognizes a comparatively small number of landmarks in the inertial and local coordinate frames, and each landmark is reassigned to a subset of the pixels in the original images.

In the HMM 200 contains the group 280 conditions 282A ( X ^Sn ₁ ) and 282B ( X ^Sn ₂ ), which correspond to a secondary movement. Unlike when the primary movement updates the group 280 the esteemed Secondary motion states based only on visual observations of landmarks located in the local coordinate frame.

2 also shows a probability map 184 (M) per pixel corresponding to the image data inputs for the states in the HMM groups 240 and 280 be applied. In the embodiment of 2 become the visual observations 248A and 248B generated based on landmarks in all image data that may contain the inertial and the local coordinate frames. To resolve conflicts in the sensor data where the pixels of some landmarks correspond to the inertial coordinate frame while the pixels of other landmarks correspond to the local coordinate frame, the probability map provides a weighting value per pixel for each pixel indicating the likelihood that the pixel will either be the one Inertial coordinate frame or the local coordinate frame. Because each of the visual observations 248A and 248B a set of pixels in each frame of image data can be assigned to the probability map 184 per pixel, to an AR system, classifying the visual observations of the landmarks as into the inertial coordinate frame for the primary motion estimation in the group 240 or in the local coordinate frame for the secondary motion estimation in the group 280 falling. For example, the probability map contains 184 per pixel in a configuration, a numerical value in a range of [0,0, 1,0] for each pixel, where a value of 0,0 indicates complete security that one pixel is in the inertial coordinate frame, while a value of 1,0 full security indicates that a pixel is in the local coordinate frame. Interval values for each pixel may indicate different degrees of confidence that a pixel is included in the coordinate frame (eg, a value of 0.5 indicates that it is equally likely that the pixel will be in each coordinate frame).

The probability weighting card 184 per pixel improves the accuracy of the primary and secondary motion estimation in the HMM 200 By making sure that the HMM 200 the largest weighting pixels of visual observations 248A and 248B assigns with the highest probability, the inertial coordinate frame for the primary motion estimation states 242A and 242B while the pixels most likely to correspond to visual observations in the local coordinate frame belong to the second motion estimation states 282A and 282B be used. As described in more detail below, the AR system uses 100 Encoder / decoder framework for a deep convolutional neural network with an epipolar restricted location for generating the probability map 184 per pixel.

im örtlichen Koordinatenrahmen für sekundäre Bewegungsschätzung beschrieben: $$\begin{array}{l}{\widehat{X}}^W=argma{x}_{X_k^w}P\left(X_m-\right)P\left(X_{k-1}^W|X_{m-}\right)P\left(X_k^W|X_{k-1}^W,u_k,z_k,M\right){\widehat{X}}_k^{S_n}=\\ argma{x}_{X_k^{S_n}}P\left(X_m-\right)P\left(X_{k-1}^{S_n}|X_{m-}\right)P\left(X_k^{S_n}|X_{k-1}^{S_n},z_k,M\right)\end{array}$$

The AR system 100 uses a Maximum à Posteriori (MAP) criteria optimization to estimate the state of the system in the HMM 200 , The optimization process minimizes residuals (errors) û in measurement-based estimates made by the inertial sensors 116 were generated, and the residual values _k Z in estimates that are based on visual observations in the image data obtained by the camera system 112 were generated. The probability map 184 per pixel (M) is used with the MAP optimization process to determine the correct association of residues to either the inertial coordinate frame (W) or the local coordinate frame (FIG. S _n ). The MAP optimization process wants to minimize the residuals, and the correct mapping of pixels and their corresponding visual features z to the inertial or local coordinate frame allows the AR system 100 to reduce the residual values to a greater degree than is practicable if it is mistakenly assumed that all the sensor data correspond to the inertial coordinate frame. The MAP process will be described mathematically below to reduce the residuals X ^W in the inertial coordinate frame for primary motion estimation and ${\widehat{X}}_k^{S_n}$

described in the local coordinate frame for secondary motion estimation: $$\begin{array}{l}{\widehat{X}}^W=arGma{x}_{X_k^w}P\left(X_m-\right)P\left(X_{k-1}^W|X_{m-}\right)P\left(X_k^W|X_{k-1}^W.u_k.z_k.M\right){\widehat{X}}_k^{S_n}=\\ arGma{x}_{X_k^{S_n}}P\left(X_m-\right)P\left(X_{k-1}^{S_n}|X_{m-}\right)P\left(X_k^{S_n}|X_{k-1}^{S_n}.z_k.M\right)\end{array}$$

stellen die Übergangswahrscheinlichkeit vom kombinierten Zustandsschätzer in Gruppe 204 auf den primären Bewegungszustandsschätzer in Gruppe 240 bzw. den sekundären Bewegungszustandsschätzer in Gruppe 280 dar. In einer Umgebung, in der Landmarken nicht stationäre sind, können die visuellen Beobachtungen nicht ausschließlich basierend auf dem aktuellen Zustand X_k des Systems mit dem passenden Zustandsschätzer assoziiert werden. Neben dem aktuellen Zustand wurde auch die Bewegung, die mit der Landmarke assoziiert ist, bestimmt. Das AR-System 100 schätzt die Assoziation unter Verwendung der Wahrscheinlichkeitskarte pro Pixel M zum Klassifizieren der Wahrscheinlichkeit, dass jedes Pixel, und die Landmarken in den visuellen Beobachtungen, die Pixeln entsprechen, sich in dem Trägheitskoordinatenrahmen oder dem örtlichen Koordinatenrahmen befinden.In the HMM 200 and the above equations corresponds to the final state X _m- in the group 204 a final estimated motion trajectory created prior to the occurrence of the motion conflict with two different coordinate frames. The final state X _m- in the group 204 of the HMM becomes the initial state for the primary motion estimation in group 240 and the secondary motion estimation in group 280 which differ from each other by the differences of the primary and secondary movements. The formulas $P\left(X_m-\right)P\left(X_{k-1}^W|X_{m-}\right)\text{and}P\left(X_m-\right)P\left(X_{k-1}^{S_n}|X_m-\right)$

put the Transition probability of the combined state estimator in group 204 on the primary motion state estimator in group 240 or the secondary motion state estimator in group 280 In an environment where landmarks are not stationary, the visual observations can not be based solely on the current state X _k of the system associated with the appropriate state estimator. In addition to the current state, the movement associated with the landmark was also determined. The AR system 100 estimates the association using the probability map per pixel M for classifying the probability that each pixel, and the landmarks in the visual observations corresponding to pixels are in the inertial coordinate frame or the local coordinate frame.

3 shows a process 300 the AR system 100 out 1 translates to primary motion estimation and secondary motion estimation using the HMM 200 from 2 in situations where a motion conflict occurs due to the presence of an inertial coordinate frame and a local coordinate frame in the sensor data. In the following discussion, a reference to the process refers 300 performing a function or action on the operation of a processor such as the processor 150 to execute stored program statements to implement the function or action associated with other components of an AR system. As described in more detail below, the process includes 300 generating a conflict probability map per pixel and estimating the primary motion and secondary motion tracks using the conflict probability map per pixel to identify landmarks in frames of the input image data corresponding to the inertial coordinate frame and the local coordinate frame. The AR system 100 also generates graphics to augment the physical environment around the AR system 100 based on the estimates of the tracks of the primary movement and secondary movement. The process 300 is also referred to as a multi-motion "Motion-Conflict Visual-Inertial Odometry" (MC-VIO) process because of the process 300 Motion conflict in inertial and local coordinate frames of the image data identified and the motion conflict used to provide improved estimates of the primary movement and the secondary movement in a visual inertial odometry process.

Generation of the conflict probability map per pixel

During the process 300 the processor receives 150 the picture frame data 168 and the inertial sensor data 172 from the camera system 112 or the inertial sensors 116 in the AR glasses 104 , The processor 150 uses the image frame data 168 and the inertial sensor data 172 as inputs to the deep neural network encoder / decoder 180 to generate the probability map 184 per pixel (block 304 ).

4 shows the CNN encoder / decoder 180 with epipolar limited position in more detail. The CNN encoder / decoder 180 contains a CNN encoder 408 , an epipolar restricted location 416 and a CNN decoder 432 , which is the initial probability map 184 generated per pixel. The CNN encoder 408 generates outputs 412A and 412B based on the inputs of two input picture frames 404A and 404B , which are two images taken at two different times, including a picture box 404A a previous time (time t) and a current picture frame 404B (Time t + 1). In some embodiments, the input image frames are two consecutive frames in a video data sequence generated by the camera system 112 is generated, although in other embodiments, the time difference between the two picture frames may be greater than the time period between consecutive picture frames in the video data.

The CNN encoder 408 Extracts dimension features from the image frames 404A and 404B using a "Thai" encoding process, which means that the image frames 404A and 404B both the CNN encoder 408 be provided separately to the coded outputs 412A and 412B to generate the input picture frame 404A respectively. 404B correspond. In particular, the CNN encoder 408 Once trained using a training process, the same CNN encoder structure with the trained weighted values in the folded layers is used to encode the encoded outputs 412A and 412B for the input picture frame 404A and 404B to create. In some embodiments, the processor uses 150 separate instances of the coding block 408 for the parallel execution of the coding process.

The Encoder / Decoder for a Convolutional Neural Network 180 which is also referred to as a "deep" neural network encoder / decoder because of the CNN encoder 408 and the CNN decoder 432 both multiple layers providing a folded layer, an activation function (the directed linear unit, ReLu, in the embodiment of 4 used), and a downsampling (in the encoder 408 ) or Upsampling function (in the decoder 412 ) in every situation. In the example of 4 contains the CNN encoder 408 five folded layers. Each layer is followed by a batchnormalization layer and your ReLU activation, with the exception of the last layer. The CNN encoder 408 removes the ReLU activation function from the last folded layer to hold the negative values before the in-product operations in the feature. A maximum of four pooling positions were applied between the folded layers, which successively downsampled the intermediate outputs of the folded layers to the outputs 412A and 412B to create. The two coded outputs 412A and 412B contain coded functions from the two original input picture frames 404A and 404B , The term "reception field" refers to a vector of elements in the coded outputs 412A and 412B which correspond to a group of pixels in the original frame data. The final probability card output 184 classifies pixels based on a combination of the coded outputs 412A and 412B using the epipolar restricted location 416 which are the coded outputs 412A and 412B as inputs receives.

In the embodiments described herein, the epipolar confined posture 416 Restrictions between the coded outputs 412A and 412B of two sets of image frame data generated over time (times t and t + 1) with an example of a video system that generates image frames at a rate of 30 frames per second, the successive frames of image data at a rate of about one frame per 33.3 milliseconds. By the movement of the AR system 100 Over time, the two frames of image data also show a spatial transformation that corresponds to the motion of the AR system between times t and t + 1. An epipolar line refers to an element that appears to be a single element (receiving field) in one of two encoded outputs, while due to the movement of the element over time between the two image frames from the perspective of the camera system 112 seems to be a line in the coded output.

As a simplified example of an epipolar line includes a first image and a corresponding coded output from the CNN encoder 408 the end of a dowel rod, which is seen end-to-end, is a single receive field, the first encoded output, containing all information available to the dowel bar extending away from the camera in a depth dimension. A second image, which is generated after the camera advances, shows the same dowel rod from a different angle, with the encoded output from the CNN encoder 408 represents the dowel rod as a line of elements, since the second image shows more information about the dowel rod, and the line that extends through the dowel rod represents all the information in a three-dimensional space that the second image encodes for the single element taken from the first picture. Thus, a receive field corresponding to the dowel rod in the first encoded output now corresponds to each receive field in the second encoded output along the line and a single zero-dimensional element now points to a one-dimensional array of elements. In the AR system 100 set the inertial sensor data from the inertial sensors 116 Information ready to the processor 150 allow to identify the changes in the angle between the first and second images of the dowel rod in this example, and more generally each set of two images from the camera system 112 adjust. Similar epipolar lines are also available in other sets of two image inputs, based on the motion of the AR system 100 between the generation of the two images, and the AR system 100 uses the geometric relationship between the movement of the camera and the coded outputs of the two images to identify the epipolar lines.

In 4 represents the correspondence between the respective field blocks 420 a processing operation using the processor 150 a correspondence between relative receive field elements in the coded outputs 412A and 412B based on the movement of the AR system 100 identified in the sensor data identified by the inertial sensors 116 received during the period from t to t + 1, that of the movement of the AR system 100 between the times when the camera system 112 generates the frames of the image data corresponds. Over a short period of time between the two frames, the inertial sensor data provides a sufficiently accurate measurement of camera system motion 112 in the AR device 100 ready to the processor 150 to enable the epipolar lines in the respective fields of the coded output 412B identify the individual receive fields in the first coded output 412A correspond. So uses the AR system 100 both video frames from the camera system 112 as inputs to the CNN conflict detector 180 and uses inertial data from the inertial sensors 116 for identifying the association between respective fields and the epipolar lines in the epipolar restricted location 416 , The epipolar limited location 416 "Constrains" each receive field from the first encoded output 412A so that it only has a single line of receive fields in the second encoded output 412B instead of potentially each receive field element in the first coded output 412A one of the receive field elements in the second coded output 412B correspondingly demonstrate what the accuracy of the subsequent CNN decoder 432 improved.

eines einzelnen Empfangsfelds an den Koordinaten i,j in dem codierten Ausgang 412A (Rt), das ein Empfangsfeld ist, das eine, zweidimensionalen Array von Pixeln 504 aus dem ersten Eingangsbildrahmen 404A entspricht. Die Tiefenabmessung des Empfangsfeldvektors 520 $R_t^{i,j}$

stellt eine Reihe von codierten Merkmalen dar, die der CNN-Encoder 408 erzeugt, wobei sich jedes codierte Merkmal in dem Empfangsfeldvektor 520 auf den Array der Pixel 504 in dem ersten Bildrahmen bezieht. So entspricht jedes Empfangsfeld räumlich einer zweidimensionalen Region des ursprünglichen Eingangsbilds, aber die codierten Ausgänge 412A und 412B sind dreidimensional, weil jedes Empfangsfeld mehrere codierte Merkmale für dieselbe räumliche Region bereitstellt. In der illustrativen Ausführungsform aus 4 erzeugt der CNN-Encoder 408 insgesamt 256 Merkmale für jedes Empfangsfeldelement in den Ausgängen 412A und 412B. 5 zeigt eine epipolare Zeile 524 $\left(R_{t+1}^{u,v}\right),$

die der Prozessor 150 in dem codierten Ausgang 412B des zweiten Eingangsbildrahmens 404B identifiziert. Die epipolare Zeile 524 enthält eine lineare Anordnung von Empfangsfeldern, die eine zweidimensionale Matrix codierter Merkmale bildet. Die epipolare Zeile 524 wird mathematisch dargestellt als L = {u, v | au + bv + c = 0} in a u, v Koordinatenraum, wo die Terme a, b, und c Koeffizienten der epipolaren Zeile sind, die der Prozessor 150 unter Verwendung von Trägheitssensordaten identifiziert, um die geometrische Beziehung zwischen den codierten Ausgängen 412A und 412B basierend auf der Bewegung des AR-Systems zwischen der Erzeugung der beiden Eingangsbilder 404A und 404B zu identifizieren. Die lineare Anordnung von Empfangsfeldern in der epipolaren Zeile 524 entspricht einer Bahn von Pixeln 512 in dem zweiten Bilddatenrahmen 404B. 5 shows the structure and operation of the epipolar layer 416 more accurate. The epipolar limited location 416 uses the inertial information for the purpose of detecting a motion conflict. As described above, the epipolar lines reduce the search dimension of potential matches from two dimensions to one dimension. In 5 corresponds to a simple reception field 520 in the first coded output 412A from the previous time t of a whole epipolar line 524 of receive fields in the second encoded output 412B from the current time t + 1. As in 4 and 5 shown are the coded outputs 412A and 412B 3-dimensional structures showing a series of coded representations of a corresponding reduced resolution input image, each reduced resolution representation containing encoded features provided by the CNN encoder 408 be generated. 5 shows a vector 520 $\left(R_t^{i.j}\right)$

a single receive field at the coordinates i, j in the coded output 412A ( Rt ), which is a reception field, which is a, two-dimensional array of pixels 504 from the first input picture frame 404A equivalent. The depth dimension of the receive field vector 520 $R_t^{i.j}$

represents a set of coded features that the CNN encoder 408 wherein each coded feature is in the receive field vector 520 on the array of pixels 504 in the first picture frame. Thus, each receive field spatially corresponds to a two-dimensional region of the original input image, but the coded outputs 412A and 412B are three-dimensional because each receive field provides multiple encoded features for the same spatial region. In the illustrative embodiment 4 generates the CNN encoder 408 a total of 256 features for each receive field element in the outputs 412A and 412B , 5 shows an epipolar line 524 $\left(R_{t+1}^{u.v}\right).$

the processor 150 in the coded output 412B of the second input picture frame 404B identified. The epipolar line 524 contains a linear array of receive fields that forms a two-dimensional array of coded features. The epipolar line 524 is represented mathematically as L = {u, v | au + bv + c = 0} in au, v coordinate space, where the terms a, b, and c are coefficients of the epipolar line that the processor 150 using inertial sensor data to identify the geometric relationship between the coded outputs 412A and 412B based on the movement of the AR system between the generation of the two input images 404A and 404B to identify. The linear arrangement of reception fields in the epipolar line 524 corresponds to a path of pixels 512 in the second image data frame 404B ,

Der Prozessor 150 führt dieselbe Operation für jedes der jeweiligen Felder in dem ersten codierten Ausgang 412A mit epipolaren Linien von Empfangsfeldern aus dem zweiten codierten Ausgang 412B durch, um den endgültigen Ausgang 532 der epipolar eingeschränkten Lage 416 zu erzeugen, die die ursprünglich codierten Ausgänge 412A und 412B kombiniert, um den Eingang für den CNN-Decoder 432 zu bilden. Der letzte Ausgang der epipolaren Lage 532 bildet den Eingang für das den Encoder des neuralen Netzwerks in dem Encoder/Decoder des neuralen Netzwerks als Block H_rf × W_rf × max(W_rf, H_rf) der Elemente,EC^i,j wobei H_rf die Höhe der codierten Eingänge 412A/412B und W_rf die Breite der codierten Eingänge 412A/412B ist.During the process 300 the processor generates 150 an inner product (dot product) of the receiving field 520 from the first coded output 412A and the epipolar row of receive fields 524 from the second coded output to produce the combined output element 528 , which is another vector, the inner product of the receiving field 520 and the array of receive fields 524 equivalent. Each output vector (EC) of the epipolar restricted layer is represented mathematically as: $e{C}^{i.j}=\left\{R_t^{i.j}\cdot R_{t+1}^{u.v}|au+bv+c=0\right\},$

The processor 150 performs the same operation for each of the respective fields in the first encoded output 412A with epipolar lines of receive fields from the second encoded output 412B through to the final exit 532 the epipolar restricted situation 416 to generate the originally encoded outputs 412A and 412B combined to the input for the CNN decoder 432 to build. The last exit of the epipolar situation 532 forms the input for the neural network encoder in the neural network encoder / decoder as block H _rf × W _rf × max (W _rf , H _rf ) of the elements, EC ^{i, j} in which H _rf the height of the coded inputs 412A / 412B and W _rf the width of the coded inputs 412A / 412B is.

In CNN encoder / decoder 180 receives the CNN decoder 432 the combined output of the epipolar restricted position 416 , The CNN decoder 432 contains five of the folded layers, which successively the entrance of the epipolar restricted location 416 unfold and four upsampling layers to produce an output having a corresponding resolution, such as the original input image 404B , The CNN decoder 432 applies a 3 × 3 × 1 folded layer with a sigmoidal activation function to the final probability map 184 per pixel. The CNN decoder 432 generates the probability map 184 per pixel with a probability value indicating the probability that each pixel in the second image 404B either in the inertia coordinate frame or in the local coordinate frame. As described above, each item contains in the probability map 184 Per pixel, a numeric value in the range [0,0, 1,0] to indicate the probability that the respective pixel in the picture frame 404B which corresponds to the inertia coordinate frame or the local coordinate frame.

6 is a receiver operating characteristic (ROC) curve 600 which represent the ratios between true positive detection (vertical axis) and false positive detection (horizontal axis) in the probability maps per pixel, the CNN encoder / decoder 180 produced using the epipolar restricted layer (Plot 608 ), compared to another CNN encoder / decoder that does not use the epipolar restricted layer (Plot 604 ). Like in the curve 600 shown exceeds the true positive rate for the plot 608 80% (0.8) before the false positive rate reaches 20% (0.2), giving greater accuracy to the CNN encoder / decoder 180 indicating the epipolar-restricted location indicates. The plot 604 however, it shows a lower rate of true positive detection, while producing a higher proportion of false positive detection that occurs because the CNN encoder / decoder embodiment used the plot 604 does not contain the epipolar restricted location.

The motion conflict probability card 184 per pixel improves the estimated primary movement and the secondary movement in at least two ways. First, it allows better discarding of outliers in the primary motion estimation. Second, it helps to determine the visual measurements that can be used for secondary motion estimation. One of the biggest challenges in multi-motion MC-VIO is to keep the computational complexity of the algorithm low while performing a robust secondary motion estimation with limited measurements.

The process 300 uses the CNN encoder / decoder 180 for generating the probability map per pixel 180 after training the CNN encoder / decoder 180 what's ahead of the process 300 he follows. In one embodiment, the training process uses a variant of a stochastic progression descent training process that automatically adjusts the learning rate per dimension. The training process minimizes pixel-wise binary cross entropy loss based on the following minimization method: min Σ _i Σ _j -ŷ _ij log ( yi , j) + (1-ŷ _ij ) log (1-y _i , j). During a test phase, the CNN encoder / decoder says 180 the probability map 184 per pixel for each frame of a test video sequence and represents the probability map 184 per pixel for the VIO system to check the accuracy of the primary and secondary motion estimation using given test data.

Major motion estimation

ist: $X_k^W:={\left[{}^W{p}_{WS}{}^T,q_{WS}{}^T{,}^S{v}_{WS}{}^T,b_g^T,b_a^T{,}^W{l}_0{}^T,\dots {,}^W{l}_n{}^T\right]}_k^T\in {ℝ}^3\times S^3\times {ℝ}^9\times {ℝ}^{4n}.$

Der Term ^Wp_WS bezieht sich auf die Pose des AR-Systems, der Term q_WS bezieht sich auf die Orientierung des AR-Systems, ^sv_WS bezieht sich auf die Geschwindigkeit und die Terme $b_g^T,b_a^T$

stellen die lineare Beschleunigung und Drehbeschleunigungstendenzen in den Trägheitssensordaten dar. Die Terme ^Wp_WS , q_WS , ^Sv_WS verwenden die W-Benennung um anzuzeigen, dass diese Elemente relativ zu dem Trägheitskoordinatenrahmen sind. Die ^Wl₀ ..., ^Wl_n stellen visuelle Landmarken dar, die in dem HMM 200 in Knoten 250 geschätzt sind. Die Benennung „ T “, die in diesem Ausdruck und in anderen Ausdrücken verwendet wird, gibt eine Matrixtransposition an.During the process 300 uses the primary motion estimation process (block 310 ) the picture frame data 168 and the inertial sensor data 172 as inputs to perform primary motion estimation of the AR system 100 relative to the inertial coordinate frame. The primary states of motion X ^W ₁ . X ^W ₂ in the group 240 Provide a simplified view of the primary states of motion. In the term described herein, the term W refers to the inertial coordinate frame ("world" frame). The input to the primary state estimator contains the visual observations z and the inertial sensor measurements u = [ω, ã]. More specifically, the primary motion trajectory conditions are as ^W X _{0: N} shown, each state $X_k^W$

is: $X_k^W:={\left[{}^W{p}_{WS}{}^T.q_{WS}{}^T{.}^S{v}_{WS}{}^T.b_G^T.b_a^T{.}^W{l}_0{}^T....{.}^W{l}_n{}^T\right]}_k^T\in {ℝ}^3\times S^3\times {ℝ}^9\times {ℝ}^{4n},$

The term ^W p _WS refers to the pose of the AR system, the term q _WS refers to the orientation of the AR system, ^s v _WS refers to the speed and the terms $b_G^T.b_a^T$

represent the linear acceleration and spin tendencies in the inertial sensor data. The terms ^W p _WS . q _WS . ^S v _WS use the W designation to indicate that these elements are relative to the inertial coordinate frame. The ^W l ₀ ..., ^W l _n represent visual landmarks that are in the HMM 200 in knots 250 are appreciated. The designation " T "Used in this term and in other terms indicates a matrix transposition.

As in 3 shown contains the primary motion estimation process of Block 310 a feature extraction (block 316 ), guided reconciliation (block 320 ), a minimal solver (block 324 ) and the primary motion optimization process (Block 328 ). The primary motion estimation process generates the output 332 containing the estimated pose ( ^W p) and the estimated orientation ( ^W q) for the primary motion.

In the feature extraction of block 316 the processor extracts 150 visual measurements z from the frames of image data 168 , In one embodiment, the processor uses 150 the FAST and BRISK Techniques as described above in the feature extraction process for identifying the visual observations.

The process 300 continues with Guided Matching, which generates a set of landmarks using the features of several images in addition to the inertial sensor data and information contained in the states of the HMM 200 stored (block 320 ). The term "guided" in guided alignment refers to the use of a previous state ^W X _k-1 in the HMM 200 which is to serve as the basis for a "continuation" estimation process that has a different "a priori" state ^W X _k in the HMM 200 generated. The condition apriori ^W X _k is often not the correct state for estimating the primary motion, but generally provides sufficient information that can be used in a matching process that identifies landmarks in the image data that make up the AR system 100 used later in conjunction with the probability map per pixel to determine the final state in the HMM 200 to identify who contains the estimated primary movement.

$${\dot{q}}_{WB}=\frac{1}{2}\mathrm{\Omega }\left({}^B{\tilde{\omega }}_{WB}-b_g\right)q_{WB}$$

$${}^{{}^{s\text{'}}{v}_{WS}=\left({}^s{\tilde{a}}_{WS}-b_a\right){+}^{W}g;}$$

$${\dot{b}}_g=n_{bg};$$

$${\dot{b}}_a=\frac{1}{\tau }{b}_a+n_{ba}.$$

The processor 150 sets the previous primary motion state estimation ^W X _k-1 continue to estimate the new condition a priori ^W X _k , The processor 150 uses the following equations with the inertial sensor data elements ^B ω _WB and ^s Forward to carry out the continuation: $$w_{\dot{p}}=\left(C_{WB}\right)\left({}^{B}v\right)$$

$${\dot{q}}_{WB}=\frac{1}{2}\mathrm{\Omega }\left({}^B{\tilde{\omega }}_{WB}-b_G\right)q_{WB}$$

$${}^{{}^{s\text{'}}{v}_{WS}=\left({}^s{\tilde{a}}_{WS}-b_a\right){+}^{W}G;}$$

$${\dot{b}}_G=n_{bG};$$

$${\dot{b}}_a=\frac{1}{\tau }{b}_a+n_{ba},$$

The continuation equation has taken as input the inertial measurements (ω, ã) that are in the body frame B were recorded. The processor 150 uses the state a priori to guide the matcher, making visual matches between pictures at two different time stamps (time matches), and in embodiments using a stereoscopic camera system 112 use between two images created at the same time stamp (static matches). The processor 150 then uses the matches as visual measurements z. If enough visual measurements of a landmark are available, the processor performs 150 a triangulation initialization of the landmark in the state estimator. The triangulation process identifies landmarks based on changes in the relative positions of the match of visual observations across a series of image frames. Once again, a landmark does not move, but the movement of the camera system 112 in the AR system 100 generates changes in the perceived locations of the visual observations of the landmarks over time, and the processor 150 Performs the triangulation process to landmarks in the HMM 200 to identify.

The process 300 drives with a minimal solver process (block 324 ) the processor 150 used to identify remnants (errors) between predictions and observations. In the process 300 identifies the processor 150 Remains (errors) based on observed inertial measurements as:
$$e_s^k\left(X_k.X_{k+1}.z_k.u_{k-1:k}\right)=\left[\begin{array}{c}{}^W{\widehat{p}}_k{-}^W{p}_k\\ 2\left({\widehat{q}}_k\oplus q_k{}^{-1}\right)\\ {}^s{\widehat{v}}_k-b_{Gk}\\ {\widehat{b}}_{Gk}-b_{Gk}\\ {\widehat{b}}_{ak}-b_{ak}\end{array}\right]$$

The processor 150 also identifies that remnants of the observed visual measurements are represented as: $e_r^{i.j.k}:=z^{i.j.k}-{\pi }_i\left(T_{CB}{\widehat{T}}_{BW}{}^W{l}_j\right),$

In der vorstehenden Gleichung stellt der Term ${\displaystyle {\sum }_{k=1}^K{\displaystyle {\sum }_i{\displaystyle {\sum }_{j\in J\left(k,i\right)}{e}_r^{i,k,j^T}}{w}_r{e}_r^{i,k,j}}}$

einen Reprojektionsfehler dar, und der Term ${\displaystyle {\sum }_{k=2}^K{e}_s^{k^T}{w}_s{e}_s^k}$

stellt einen Vorhersagefehler da. Der Reprojektionsfehler stellt einen Fehler dar, der aufgrund der Differenz zwischen der tatsächlichen dreidimensionalen Position einer Landmarke im Vergleich mit der geschätzten Position auftritt, die das AR-System 100 aus der Triangulation erzeugt, da die Schätzung unter Verwendung einer Reihe von zweidimensionalen Bildern erfolgt, die als Projektionen des dreidimensionalen physischen Raums um das AR-System 100 wirken. Der Vorhersagefehler stellt den Fehler zwischen der anfänglichen Vorhersage von dem HMM 200 basierend auf dem vorherigen Zustand und den aktuellen Trägheitssensoreingängen sowie der tatsächlichen primären Bewegung dar. Der Term „Energie“ entspricht nicht einer physischen Energiemenge, sondern bezieht sich stattdessen auf eine Messung der Gesamtfehlerstufe als Energiemenge, wobei der Fensterbatchoptimierungsprozess Zustände in dem HMM 200 durchsucht, die Verbindung mit dem vorherigen Zustand $X_{k-1}^W$

aufweisen, um den nächsten Zustand $X_k^W$

zu finden, der den kleinsten Energie- (Fehler-) Wert erzeugt. Das AR-System 100 wählt dann den nächsten Zustand $X_k^W$

in dem HMM 200 und verwendet die eingebetteten Werte ^Wp_WS , q_WS in dem gewählten Zustand $X_k^W,$

die der Pose bzw. Ausrichtung entsprechen, als die geschätzten Ausgangslaufbahndaten 332.The processor 150 uses the remainders to perform a windowbatch optimization process (block 328 ) to minimize the following energy: $\left(X_k^W\right):={\displaystyle {\Sigma }_{k=1}^K{\displaystyle {\Sigma }_i{\displaystyle {\Sigma }_{j\in J\left(k.i\right)}{e}_r^{i.k.j^T}}{w}_r{e}_r^{i.k.j}+{\displaystyle {\Sigma }_{k=2}^K{e}_s^{k^T}{w}_s{e}_s^k,}}}$

In the above equation, the term ${\displaystyle {\Sigma }_{k=1}^K{\displaystyle {\Sigma }_i{\displaystyle {\Sigma }_{j\in J\left(k.i\right)}{e}_r^{i.k.j^T}}{w}_r{e}_r^{i.k.j}}}$

a reprojection error, and the term ${\displaystyle {\Sigma }_{k=2}^K{e}_s^{k^T}{w}_s{e}_s^k}$

represents a prediction error. The reprojection error represents an error that occurs due to the difference between the actual three-dimensional position of a landmark compared to the estimated position that the AR system 100 generated from the triangulation, since the estimation is done using a series of two-dimensional images that are projections of the three-dimensional physical space around the AR system 100 Act. The prediction error represents the error between the initial prediction from the HMM 200 The term "energy" does not correspond to a physical amount of energy, but instead refers to a measurement of the total error level as an amount of energy, wherein the windowbatch optimization process states in the HMM 200 searches the connection with the previous state $X_{k-1}^W$

exhibit to the next state $X_k^W$

finding the smallest energy (error) value. The AR system 100 then choose the next state $X_k^W$

in the HMM 200 and uses the embedded values ^W p _WS . q _WS in the selected state $X_k^W.$

that correspond to the pose, as the estimated exit raceway data 332 ,

In addition to motion estimation

As in 3 2, the secondary motion estimation process (block 350 ) the initialization (block 354 ), the secondary motion adjustment (block 358 ), a secondary minimal solver (block 362 ) and the secondary motion optimization process (block 366 ). The secondary motion estimation process 350 Also includes a landmark estimation processor 340 , the estimates of the landmarks in the inertia coordinate frame to the led matcher 320 and estimates of the landmarks in the local coordinate frame to the secondary motion matcher 358 , The secondary motion estimation process of Block 350 generates a secondary motion estimation output 370 who made an estimate of the pose ( ^V p _VB ) and an estimate of the orientation ( q _VB ) contains.

wobei der sekundäre Zustand definiert ist als: $X_k^{S_n}:={\left[{}^V{p}_{VB}{}^T,q_{VB}{}^T{,}^V{l}_0{}^T,\dots {,}^V{l}_n{}^T\right]}_k^T\in {ℝ}^3\times S^3\times {ℝ}^{4n}.$

In der hierin beschriebenen Benennung bezieht sich der Term V auf den örtlichen Koordinatenrahmen Wobei ^Vp_VB and q_VB die geschätzte Pose bzw. Ausrichtung bezüglich des örtlichen Koordinatenrahmens darstellt und ^Vl₀, ..., ^Vl_n Landmarken darstellen, die als in den örtlichen Koordinatenrahmen fallend gelten, wofür eine Wahrscheinlichkeitskarte pro Pixel verwendet wird. Wie oben beschrieben, enthalten die sekundären Zustände in dem HMM 200 keine Parameter, die auf den Trägheitssensordaten 172 basierend, weil die sekundäre Bewegung in dem örtlichen Koordinatenrahmen nur basierend auf Abschnitten der Bildrahmendaten 168 geschätzt wird, die dem örtlichen Koordinatenrahmen wie unter Verwendung der Wahrscheinlichkeitskarte 184 pro Pixel identifiziert entsprechen.The secondary movement appreciates the condition $X_k^{S_n}.$

where the secondary state is defined as: $X_k^{S_n}:={\left[{}^V{p}_{VB}{}^T.q_{VB}{}^T{.}^V{l}_0{}^T....{.}^V{l}_n{}^T\right]}_k^T\in {ℝ}^3\times S^3\times {ℝ}^{4n},$

In the term described herein, the term V refers to the local coordinate frame ^V p _VB and q _VB represents the estimated pose with respect to the local coordinate frame and represent ^V l ₀ , ..., ^V l _n landmarks that are considered to fall within the local coordinate frame, using one probability map per pixel. As described above, the secondary states in the HMM include 200 no parameters based on the inertial sensor data 172 because the secondary motion in the local coordinate frame is based only on portions of the frame data 168 It is estimated that the local coordinate frame as using the probability map 184 correspond to each pixel identified.

In the process 300 initializes the processor 150 the secondary motion estimator for the state X _m- (State 210C in the HMM 200 out 2 ), which determines the final state before the start of secondary motion tracking (Block 354 ).

The process 300 will continue if the processor 150 a secondary motion matching process (block 358 ). The secondary motion matching process 358 is similar to the guided matching process from block 320 in the primary motion estimation process, but in the secondary motion match process, the processor uses 150 the probability map 184 per pixel (M) for identifying landmarks having a high probability of matching the local coordinate frame to perform the secondary motion matching process. This increases the accuracy of the secondary motion estimation by reducing or eliminating the influence of landmarks from the inertial coordinate frame in the secondary motion estimation.

auf den sekundären Bewegungsschätzer. Die marginale Wahrscheinlichkeit, dass eine Landmarke sich basierend auf allen visuellen Beobachtungen z_j der Landmarke Li in dem örtlichen Koordinatenrahmen befindet, wird angegeben durch: $P\left({}^V{l}_i|S_n,M\right)={\displaystyle {\sum }_j^N\frac{P\left(z_j|l_i,M\right)}{P\left(z_j\right)}}.$

Dieser Prozess entfernt implizit die Landmarken von dem primären Bewegungsschätzungsprozess von Block 310 oben. So nutzt zwar der primäre Bewegungsschätzungsprozess die Wahrscheinlichkeitskarte 184 pro Pixel nicht direkt, aber der Prozessor 150 nutzt die Wahrscheinlichkeitskarte 184 pro Pixel zum Aufteilen der Landmarken zwischen den primären und sekundären Bewegungsschätzungsprozessen, die die Genauigkeit beider Prozesse verbessern. Der Prozessor 150 führt einen Zeitabgleich der Landmarken in der sekundären Karte mit dem aktuellen Rahmen durch, um weitere visuelle Messungen zu erzeugen.During the process 300 leads the processor 150 the landmark frame estimate to determine if a landmark is related to the inertial coordinate frame for the primary motion estimation process of Block 310 or the local coordinate frame for the secondary motion estimation process of Block 350 (Block 340 ) belongs. If the processor 150 finds a marginal likelihood of having a landmark belonging to the local coordinate frame greater than a given probability bound, such as greater than a 50% probability, then the processor points 150 the map of the secondary map and moves all associated residues, $e_r^{i.j.k}$

on the secondary motion estimator. The marginal probability of a landmark being based on all visual observations z _j the landmark Li located in the local coordinate frame is indicated by: $P\left({}^V{l}_i|S_n.M\right)={\displaystyle {\Sigma }_j^N\frac{P\left(z_j|l_i.M\right)}{P\left(z_j\right)}},$

This process implicitly removes the landmarks from the block's primary motion estimation process 310 above. For example, the primary motion estimation process uses the probability map 184 not directly per pixel, but the processor 150 uses the probability map 184 per pixel for splitting the landmarks between the primary and secondary motion estimation processes that improve the accuracy of both processes. The processor 150 Time-align the landmarks in the secondary map with the current border to produce more visual measurements.

Dies ist ein Prozess, der ähnlich wie der primäre Bewegungs-Minimal-Solver von Block 324 oben ist, wobei jedoch der Prozessor 150 im sekundären Bewegungsschätzungsprozess von Block 350 nur den Rest $e_r^{i,j,k}$

identifiziert, der Fehlern zwischen den visuellen Beobachtungen z^i,j,k und den geschätzten Landmarken entspricht, da das AR-System 100 keine Trägheitsdaten verwendet, um den sekundären Bewegungsschätzungsprozess auszuführen.A generalized P3P was combined with RANSAC to estimate the pose associated with the secondary map. The process 300 leads the processor 150 a secondary motion-minimal solver process (block 362 ) to identify the rest: $e_r^{i.j.k}:=z^{i.j.k}-{\pi }_i\left(T_{CB}{\widehat{T}}_{BS}{}^S{l}_j\right),$

This is a process similar to the primary motion-minimal solver of Block 324 above, but with the processor 150 in the secondary motion estimation process of Block 350 only the rest $e_r^{i.j.k}$

Identifies the errors between the visual observations z ^{i, j, k} and the estimated landmarks, since the AR system 100 no inertial data is used to perform the secondary motion estimation process.

The processor 150 uses the remainder to perform a window batch optimization process for secondary motion estimation (Block 366 ) to minimize the following energy:

In dieser Gleichung ist der einzige Fehler der Reprojektionsfehler und es gibt keinen Vorhersagefehlerterm aus den Trägheitssensormessungen. Das AR-System 100 wählt dann den nächsten Zustand $X_k^{S_n}$

in den sekundären Zustanden des HMM 200 und verwendet die eingebetteten Werte ^Vp_WS , q_WS in dem gewählten Zustand $X_k^{S_n},$

die der Pose bzw. Ausrichtung entsprechen, als die geschätzten Ausgangslaufbahndaten 370. $\left(X_k^{S_n}\right):={\displaystyle {\Sigma }_{k=1}^K{\displaystyle {\Sigma }_i{\displaystyle {\Sigma }_{j\in J\left(k.i\right)}{e}_r^{i.k.j^T}}{w}_r{e}_r^{i.k.j},}}$

In this equation, the only error is the re-projection error and there is no prediction error term from the inertial sensor measurements. The AR system 100 then choose the next state $X_k^{S_n}$

in the secondary states of the HMM 200 and uses the embedded values ^V p _WS . q _WS in the selected state $X_k^{S_n}.$

that correspond to the pose, as the estimated exit raceway data 370 ,

Improvements in the operation of the AR system

As described above, the AR system uses 100 the process 300 for generating the estimates of the primary motion track and the secondary motion track. The accurate estimation of these tracks makes it possible for the AR system 100 , visual inertial odometry with greater accuracy to generate augmented graphics for display to a user with greater accuracy in environments containing inertial and local coordinate frames.

7 shows improvements in visual inertial (VIO) in more detail. In a VIO operation, the AR system keeps track 100 its location in an environment using the estimated tracks of the inertial coordinate frame. In two practical examples 7 The VIO process tracks the movement of the AR system 100 while a user is walking in a circle in an office while looking at another person who is also moving. In another example, a user in a motor vehicle drives in a circle around a parking lot, starting at a parking lot, and returns to the same parking lot. In both examples, the AR system gets 100 Sensor data from the inertial coordinate frame of the office or parking lot and the local coordinate frame from either the other person in view of the camera system 112 or the interior of the motor vehicle. 7 shows errors in the VIO operation as deviations from the user's path when the user walks around an office or the user in the vehicle when the vehicle is driving around a parking lot.

More precisely, shows 7 cards 704 and 708 VIO data generated using two state-of-the-art VIO systems, including OKVIS ( 704 ) and Wiley-Babu ( 708 ) VIO systems. This VIO system generates tracking errors due to the presence of a moving person in the field of view of a camera sensor with representative images, which in the prior art from the photo 720 for the card 704 and the photo 724 for the card 708 Cause errors. 7 shows big tracking errors 706 and 710 in the cards 704 respectively. 708 that reduce the effectiveness of state-of-the-art VIO systems. In contrast, the map shows 712 VIO using the AR system 100 out 1 using the multi-motion MC-VIO process 300 to perform primary and secondary motion estimation with greater accuracy when the same sets of inertial and image input data are present, include the two images 720 and 724 that have caused errors in previous state-of-the-art VIO systems.

In addition to the results 7 For example, the following table shows the absolute tracking error (ATE) in meters and the relative maze error (RPE) in meters per second for the embodiments described herein as compared to OKVIS and prior art Wiley-Babu MV-VIO-AR systems of three motion sequences containing inertial and local coordinate frames in the input sensor data. As shown below, the embodiment of System 100 that the process 300 performs excellently on nearly all metrics and fits the Wiley-Babu implementation in the RPE standard deviation. record Breath] RPE [m / s] OKVIS Wiley Babu System 100 OKVIS Wiley Babu System 100 Seq ¹ 4,979 4,681 0,568 0,052 0.047 0,018 Seq ² 4,873 6,391 0,142 0,052 0,051 0,010 Seq ³ 27.22 2,740 1,932 0.252 0,065 0,029 medium 12.35 4,604 0.881 0,119 0.054 0.019 Standard deviation 10.51 1.492 0.763 0.094 0,008 0,008

The improvements to the accuracy of the VIO described above enable the AR system 100 also to generate augmented graphics for display to a user with accurate placement of the augmented graphics in portions of the physical environment corresponding to both the inertial coordinate frame and the local coordinate frame. 8A and 8B show the operation of the AR system 100 to create augmented graphics in more detail. 8A shows the view of a physical environment containing a road and other fixed features (inertial coordinate frame) as well as the interior of a vehicle (local coordinate frame). The pictures can be taken using the camera system 112 in the AR system 100 be generated. The photo 802 contains an overlay with a graphical representation of the probability map 184 per pixel representing a region 184 which corresponds to the inertial coordinate frame of the road, and another region 804 that corresponds to the local coordinate frame of the vehicle interior. While the AR system 100 generally no graphical representation of the probability map 184 generated per pixel, presents the photo 802 an illustrative embodiment of the probability map 184 ready per pixel. Furthermore, the photo contains 820 graphic displays for landmarks representing the system 100 based on the frames of the image data from the camera system 112 and the inertial sensor data from the inertial sensors 116 recognizes, with an example of a landmark 820 , which is used only for the primary motion estimation in the inertial coordinate frame, and the landmark 824 , which is used only for the secondary motion estimation in the local coordinate frame.

8A also shows graphic elements that the AR system 100 generated in alignment with either the inertial coordinate frame or the local coordinate frame to augment the view of the physical environment. In the photos 850A and 850B creates the AR system 100 a graphical 3D view of a virtual vehicle 832 parked at the roadside, in the inertia coordinate frame. In the AR system 100 the processor uses 150 the AR graphics data 188 for generating the graphical representation of the vehicle 832 using the primary motion estimation from the process 300 to the virtual vehicle 832 to place at a location on the road which, even at a consistent location from the user's point of view, remains aligned with the inertial coordinate frame when the user is in a moving vehicle. The user views the virtual vehicle 832 through the glasses 108A / 108B along with the physical environment that is depicted in 8A , The AR system 100 can also generate virtual graphics aligned with the coordinate frame. As in the photos 854A and 854B represented, the processor generates 150 a virtual sphere 836 which remains in the local coordinate frame at a fixed location relative to the dashboard of the vehicle interior. The processor uses again 150 the graphics data 188 to the virtual sphere 836 and uses the secondary estimated motion to display the virtual sphere 836 at a consistent location within the local coordinate frame as the user moves relative to the local coordinate frame and the inertial coordinate frame. 8B contains the other photos 860A to 860E , the landmarks, the virtual vehicle 832 and the virtual vehicle 836 during operation of the AR system 100 demonstrate. While this is in 8A and 8B not explicitly shown, the AR system 100 the graphic of the virtual vehicle 832 in the inertia coordinate frame and the virtual sphere 836 in the local coordinate frame due to the improvements to the primary motion estimation and the secondary motion estimation provided by the embodiments described herein simultaneously.

It is to be understood that variations of the above-disclosed and other features and functions or alternatives thereof may be combined as desired in many other various systems, applications or methods. Various currently unforeseen or unexpected alternatives, modifications, variations, or improvements may be made hereinafter by those skilled in the art who should also be encompassed by the following claims.

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• US 62/678383 [0001]

Claims (18)

A motion estimation method in an Augmented Reality (AR) system, comprising: Receiving inertial sensor data with an inertial sensor system during movement of the AR system; Receiving multiple frames of image data with a camera system during movement of the AR system; Generating a probability map based on the inertial sensor data and the plurality of frames of the image data with a processor implementing a convolutional neural network encoder / decoder, the probability map corresponding to a frame of the plurality of frames of the image data and including a plurality of probability values including a plurality of pixels in corresponding to one frame, each probability value from the plurality of probability values indicating a probability that the corresponding pixel is an inertial coordinate frame or a local coordinate frame; Identifying visual observations of at least one landmark in the local coordinate frame with the processor based on the plurality of frames of the image data and the probability map; and Generating an estimate of the secondary motion for the AR system with the processor with reference to the local coordinate frame based on a first previous state in a Hidden Markov Model (HMM) that matches the local coordinate frame and the visual observations of the at least one landmark in the local coordinate frame corresponds. Method according to Claim 1 , further comprising: generating a graphical element aligned with the local coordinate frame based on the estimate of the secondary motion for the AR system with a display device in the AR system. Method according to Claim 1 , further comprising: identifying visual observations of at least one landmark in the inertial coordinate frame with the processor based on the plurality of image data and the probability map, wherein the at least one landmark in the inertial coordinate frame is different than the at least one landmark in the local coordinate frame; and generating an estimate of the primary motion for the AR system with the processor with reference to the inertial coordinate frame based on a second previous state in the HMM corresponding to the inertial coordinate frame, the inertial sensor data, and the visual observations of the at least one landmark in the inertial coordinate frame. Method according to Claim 3 , further comprising: generating a graphical element aligned with the inertial coordinate frame based on the estimate of the primary motion for the AR system with a display device in the AR system. Method according to Claim 3 , further comprising: generating a first graphical element aligned with the local coordinate frame based on the secondary motion estimation estimate for the AR system with a display device in the AR system; and generating a second graphical element aligned with the inertial coordinate frame based on the estimate of the primary motion for the AR system with the display device in the AR system. Method according to Claim 1 , further comprising: receiving, with the camera system, the plurality of frames of the image data including a first frame of the image data and a second frame of the image data, the second frame of the image data occurring after the first frame of the image data, generating a first encoded output Using the first frame of the image data as an input to a convolutional neural network encoder and a second encoded output using the second frame of image data as the input to the Convolutional Neural Network encoder to the processor encoding an encoder of a Convolutional Neural Network in the encoder / Decoder of a Convolutional Neural Network; Generating an input to a convolutional neural network decoder in the neural network encoder / decoder using an epipolar constrained layer based on first multiple receive fields in the first encoded output, second multiple receive fields in the second encoded output, and multiple epipolar lines; generated by the plurality of receive fields in the second encoded output, with the processor; and Generation of the probability map, which corresponds to the second frame of the image data, with the processor, which converts the decoder of the neural network in the Encoder / Decoder of a Convolutional Neural Network. Method according to Claim 6 wherein the generation of the input to the Convolutional Neural Network decoder in the Convolutional Neural Network Encoder / Decoder further comprises: generating the input to the convolutional neural network decoder in the neural network encoder / decoder with the epipolar restricted processor Position with the epipolar line in a two-dimensional matrix of coordinates u, v as: au + bv + c = 0 where each element EC ^{i, j} at the coordinates i, j of the epipolar constrained layer: $e{C}^{i.j}=\left\{R_t^{i.j}\cdot R_{t+1}^{u.v}|au+bv+c=0\right\}$

in which $R_t^{i.j}$

a one-dimensional vector of a reception field from the first plurality of reception fields in the first coded output and $R_{t+1}^{u.v}$

the two-dimensional matrix u, v is of a plurality of reception fields corresponding to one epipolar line of the plurality of epipolar lines in the second encoded output, and a, b, and c are coefficients of one epipolar line; and generating the input to the convolutional neural network encoder in the neural network encoder / decoder as a block H _rf × W _rf × max (W _rf , H _rf ) of the elements EC ^{i, j} with the processor, where H _{rf is} a Height of an encoded feature and W _{rf is} the width of the encoded feature, Method according to Claim 6 , further comprising: identifying, with the processor, each epipolar line of the plurality of epipolar lines based on inertial sensor data received from the inertial sensor system and a movement of the AR system during a time between the generation of the first frame of the image data and the generation of the second Show frame of image data. Method according to Claim 1 wherein the generation of the secondary motion estimation for the AR system further comprises generating an estimate of a pose of the AR system with the processor and an alignment of the AR system with reference to the local coordinate frame. Augmented reality (AR) system comprising: an inertial sensor system; a camera system; a display device; and a processor operably coupled to the inertial sensor system, the camera system, and the display device, the processor configured to: Receiving inertial sensor data generated by the inertial sensor system during movement of the AR system; receive multiple frames of image data generated by the camera system during movement of the AR system; a probability map based on the inertial sensor data and the plurality of frames of the image data using a Convolutional Neural Network encoder / decoder, the probability map corresponding to the one frame of the plurality of frames of the image data and including a plurality of probability values comprising a plurality of pixels in the one frame each probability value from the plurality of probability values indicates a probability that the corresponding pixel is located in an inertial coordinate frame or a local coordinate frame, Identifying visual observations of at least one landmark in the local coordinate frame based on the plurality of frames of the image data and the probability map; and Generating an estimate of the secondary motion for the AR system with reference to the local coordinate frame based on a first previous state in a Hidden Markov Model (HMM) that matches the local coordinate frame and the visual observations of the at least one landmark in the local coordinate frame Coordinate frame corresponds. System after Claim 10 wherein the processor is further configured to: generate a graphical element with the display device, wherein the graphical element is aligned with the local coordinate frame based on the estimate of the secondary motion for the AR system. System after Claim 10 wherein the processor is further configured to: identify visual observations of at least one landmark in the inertial coordinate frame based on the plurality of frames of the image data and the probability map, wherein the at least one landmark in the inertial coordinate frame is different than the at least one landmark in the local coordinate frame ; and generating an estimate of the primary motion for the AR system with the processor with reference to the inertial coordinate frame based on a second previous state in the HMM corresponding to the inertial coordinate frame, the inertial sensor data, and the visual observations of the at least one landmark in the inertial coordinate frame. System after Claim 12 wherein the processor is further configured to: generate a graphical element with the display device, wherein the graphical element is aligned with the inertial coordinate frame based on the estimate of the primary motion for the AR system. System after Claim 12 wherein the processor is further configured to: generate a first graphical element with the display device, wherein the first graphical element is aligned with the local coordinate frame based on the estimate of the secondary motion for the AR system; and generating a second graphical element with the display device, wherein the second graphical element is aligned with the inertial coordinate frame based on the estimate of the primary motion for the AR system. System after Claim 10 wherein the processor is further configured to: receive the plurality of frames of the image data with the camera system, the plurality of frames of the image data including a first frame of the image data and a second frame of the image data, the second frame of the image data subsequent to the first frame of the image data occurs; Generating a first encoded output using the first frame of the image data as input to a convolutional neural network encoder in the convolutional neural network encoder / decoder and a second encoded output using the second frame of the image data as input to the convolutional neural encoder Network; Generating an input to a convolutional neural network decoder in the neural network encoder / decoder using an epipolar restricted layer based on first multiple receive fields in the first encoded output, second multiple receive fields in the second encoded output, and multiple epipolar lines; generated by the plurality of receive fields in the second encoded output; and generating the probability map corresponding to the second frame of the image data using the neural network decoder in the convolutional neural network encoder / decoder. System after Claim 15 wherein the processor is further configured to: generate the input to the convolutional neural network decoder in the neural network encoder / decoder from the epipolar constrained layer with the epipolar line in a two-dimensional array of coordinates u, v as: au + bv + c = 0 where each element EC ^{i, j} at the coordinates i, j of the epipolar constrained layer: $e{C}^{i.j}=\left\{R_t^{i.j}\cdot R_{t+1}^{u.v}|au+bv+c=0\right\}$

in which $R_t^{i.j}$

a one-dimensional vector of a reception field from the first plurality of reception fields in the first coded output and $R_{t+1}^{u.v}$

the two-dimensional matrix u, v is of a plurality of reception fields corresponding to one epipolar line of the plurality of epipolar lines in the second encoded output, and a, b, and c are coefficients of one epipolar line; and generating the input to the convolutional neural network encoder in the neural network encoder / decoder as a block H _rf × W _rf × max (W _rf , H _rf ) of the elements EC ^{i, j} , where H _{rf is} a height of an encoded one Feature is and W _{rf is} the width of the encoded feature, System after Claim 15 wherein the processor is further configured to: identify each epipolar line from the plurality of epipolar lines based on inertial sensor data received from the inertial sensor system and a movement of the AR system during a time between the generation of the first frame of the image data and the generation of the second frame of the image data. System after Claim 10 wherein the processor is further configured to: generate the estimate of the secondary motion for the AR system with the processor including an estimate of a pose of the AR system with the processor and an alignment of the AR system with reference to the local coordinate frame.

Images