JP2002532770A - Method and system for determining a camera pose in relation to an image - Google Patents

Abstract

Summary A completely passive and self-contained system for determining platform pose information is disclosed. The system comprises a motion detector and an imager, both working together in a known temporal relationship, whereby each image generated by the imager is a set of motion data provided by the motion detector. Corresponding to In a preferred embodiment, the motion sensing device and the imaging device work together and / or synchronously. The imaging device detects the surrounding scene and its features are extracted and tracked to determine the motion of the imaging device. Therefore, no prior information about the scene or special scene preparation is required. In addition, a statistical estimation process such as a Kalman filter is used to assist feature tracking. To determine pose information, the feature and motion data propagated by the strapdown navigation process are provided to a statistical estimation process. Errors from the statistical estimation process are used to update feature and motion data. As a result, the pose information output from the statistical estimation process has high accuracy, ignoring the accuracy and characteristics of the motion data, similarly to the related devices.

Description

DETAILED DESCRIPTION OF THE INVENTION

BACKGROUND OF THE INVENTIONField of the Invention  The present invention relates to the field of object pose determination, and more particularly to image analysis and motion detectors.
The position and orientation of the rigid body including the camera under arbitrary motion conditions using the combination
A method and system for determiningRelated prior art  Platform location using surrounding images taken from the platform
And the problem of accurately determining orientation is an important division of the navigation field
Various solutions have been provided for decades. Platform carried by human
Camera subsystem, vehicle with roof mounted camera,
Aircraft with cameras, or cameras and attached equipment that form images of the surrounding environment
Other moving states configured for In simple cases, for a local space
The exact position and orientation of the platform with the camera
Is referred to as a “pause”) by reference images for fixed features of the surrounding environment.
It is determined. Perception of up, down, rotation angle between different images measured by motion detection component
Collecting image perspectives is an automatic
Simplify placement.

[0002] The traditional mechanization of this approach has hundreds of years of history for the use of sextants where manual observation of known patterns of stars with knowledge of the horizon has been used as a navigational reference. Further mechanization of celestial-assisted navigation was made as a component of both early and current strategic missiles. Here, knowledge of vehicle poses is generally known through the use of a self-contained inertial navigation system, and this initial pose is updated using star observations from the camera.

[0003] A second traditional solution to integration with navigation is the Digital Scene Matching Area Correlation (DSMAC) guidance method, which is used routinely for cruise missile guidance. DSMAC uses a set of reference scenes stored along with the onboard inertial navigation system. As the missile flies near the reference scene, the camera looks at suspicious areas of the scene and makes corrections by correlating the observed scene from the onboard digital camera with the reference. Corrections are then made to the inertial navigation solution. A similar image-assisted guidance solution is routinely used for aircraft guidance and navigation updates are made at known visual landmarks.

A common constraint on these navigation schemes is that the visual reference cues used for navigation are known prior to navigation. Constellation-assisted navigation does not work without a known star pattern, and DSMAC requires vigorous advance preparation of the reference scene.

[0005] US Patent Nos. 4,070,307, 5,576,964, 4,494,200, and 434751
No. 1,4179,693 describes a navigation method using the features of roads and terrain graphically, the location of which is known from a map. Constellation aided navigation is described in U.S. Pat. No. 4,746,976. No. 5,525,883, 5,784,282 describes navigation using specially designed visual targets pre-positioned in a scene, and US Pat.
No. 419 describes a system for geolocating any point on the terrain having a vehicle trajectory determined primarily by GPS. Pre-planning is required for all these prior art systems.

[0006] Furthermore, US Pat. No. 5,699,444 disposes camera positions and features in a scene from multiple cameras from multiple features. The system uses only the video sensor without the need for a separate motion sensing component. The overall performance of the system is adversely affected because pose information is available for all views and feature selection is not provided during the manual process. In addition, the only graphical solution to pose determination is constrained by the geometric features of the various view features. US Patent No. 55
No. 11153 improves on other methods by providing a high rate sequence of images that is processed to provide iterative estimation for feature placement, thus providing the possibility of automating the feature tracking process. However, US Pat. No. 5,511,153 discloses that at least seven features are tracked sequentially, with each feature represented as a single range parameter rather than a full 3D position vector. In addition, such an image-only solution imposes a serious constraint on acceptable camera movement, with nodal position of the camera, rapid rotation, looming, and a camera with a narrow field of view. They tend to fail easily. Finally, there is no independent means of predicting feature locations in future frames, such as in U.S. Pat. No. 5,511,153, so that the feature tracking search window will be Must be large enough to include motion, resulting in enormous computation time.

However, in practice there are many applications or areas where prior knowledge of the scene is not available when accurate navigation guidance is required. Thus, there is a great need for a system in which subsequent decisions of the platform can be combined automatically and efficiently with surrounding scenes without prior knowledge of the scene. SUMMARY OF THE INVENTION The present invention relies on visual information from images to aid device-based navigation systems that achieve a particularly high level of accuracy in pose determination under conditions where one or more devices fail to provide accurate data. Extraction and processing technology. Navigation systems using the invention disclosed herein are not subject to the large drift and noise components of motion detection data, and despite the large uncertainties of the camera / lens / optics used for image collection. It can be used to obtain highly accurate platform pose information.

According to one aspect of the present invention, the image is securely connected to or controlled by a cluster of motion sensing devices on a platform, including but not limited to a moving system (eg, a vehicle) and a flight system (eg, a missile). Collected by a camera pointed at the desired orientation. One example of a motion detector is an inertial measurement unit (IMU) consisting of devices that measure acceleration and rotational speed along three orthogonal axes that are fixedly secured to the platform. This IMU is augmented with a GPS sensor that measures the position and speed of the platform with respect to the datum frame.
A salient feature operator is applied to the image obtained from the camera by detecting regions within the image that are relatively unique in the sense of image processing. The image templates for these regions are stored and the visual feature tracking operator attempts to identify corresponding regions in subsequent images. If the corresponding features are obtained from a stationary object or terrain (eg a building), the platform position is determined with respect to a fixed coordinate system for these features.

[0009] Motion sensors, such as accelerometers and velocity gyros, included in the IMU device cluster are subsystems of each respective sensor. Recent developments in the field of micromechanical electromechanical systems (MEMS) have changed the concept of the IMU in size and performance. However, MEMS IMU components are much less accurate than conventional IMU components used for navigation. Furthermore, digital camera components have evolved into highly integrated single-chip cameras that include all digitization of light sensing and image processing components. These cameras can be mounted on miniature lenses to reach very small digital camera components. The new camera optics, along with the MEMS IMU parts, have no surveying accuracy. That is, the rays are not mapped with high precision on the light sensitive two-dimensional array. To further complicate this problem, the lens / optics change the focal length, so that the optical parameters vary highly throughout the image acquisition path. In both the IMU and camera situations, accurate system mechanization must address similar inaccuracies in component subsystems.

One feature of the present invention is the inherent tolerance for inaccuracies from subsystems. The present invention includes a model that is substantially and easily extensible for errors in sensing components parameterized with statistically based coefficients. Errors in these coefficients are taken into account in the processing, and in typical circumstances, the errors are estimated with high accuracy.

A system employing the present invention includes a motion detection and camera device, including self. Navigation using these sensors relates to the local scene, and the axis of the pose information is arbitrarily related to the early detected features. All subsequent features and platform data are provided for this initial coordinate system. According to one embodiment, geodetic pose information is provided. Integration of GPS and IMU information is well known to those skilled in the art, and such integration may be optionally included to work in conjunction with the present invention. As a result, only sparse GPS information is required to be available in the urban environment to allow poses to be referenced to geographic coordinates. Unlike prior art systems, this embodiment provides pose determination in geographic coordinates even during long periods of GPS unavailability, such as indoors, in tunnels, and overcrowded urban environments.

It is known that certain naturally occurring scenes include features classified by their image formation. Such features may be the intersection of straight lines (eg, corners of a building) or the occurrence of perpendiculars (
(E.g., building edges or tree trunks) or objects that reflect other internationally planned features of the scene. The present invention also includes the optional possibility of limited features matching the spherical class.

In some navigation scenes, for example, repeated navigation within an area is required. In this example, it is appropriate to start navigation by practicing the invention as described above in learning mode to collect the required navigation feature archive. Thereafter, the same pre-learned features are used without having to add new features. Archives are also sent between multiple platforms that want to navigate the area. Similarly, being easily recognizable in the image is appropriate for inserting any feature into any non-searched arrangement (cone, colored sphere, post). These features are then preferably addressed in a feature selection process. However, the placement and feature tracking of these preferred features is handled exactly as is done with naturally occurring features.

The present invention can be implemented in various ways including a computer readable medium including program code for automatically obtaining pose information for a platform including a method, system, motion sensor, and imaging device. Different embodiments or implementations have one or more of the following unique advantages and advantages.

An advantage and advantage of the present invention is that the pose for any moving platform is determined with high accuracy without external information or previous mission planning. Furthermore, the present invention has inherent resilience to subsystem errors to allow for less accurate inertial sensing and use of camera components. Ultimately, accurate pose information allows the use of a wide range of image set metrics for 3D scene reconstruction and measurement.

[0016] Other advantages, advantages, objects and features of the invention will be described hereinafter with reference to the drawings, in which:
And the implementation of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS These and other features and advantages of the present invention will be set forth in the following description and claims, with reference to the drawings, in which:
Will be more clearly understood.Signs and terms  In the following detailed description of the invention, numerous specific details are set forth in order to provide a consistent understanding of the invention.
Used to provide. However, the invention may be practiced without these detailed features.
What can be done will be apparent to those skilled in the art. Other examples include well-known methods, procedures,
Components and circuits are described in detail to avoid unnecessarily obstructing features of the present invention.
I won't tell.

The following detailed description of the invention describes procedures, steps, and logic blocks that represent data processing in a computing device.
It is mostly expressed in terms of lock, processing, and other symbolic expressions. These processes
The descriptions and expressions in the text are used by those skilled in the art to most effectively convey the substance of the invention to others skilled in the art.
This is the means used for The system and computer readout described in detail below
Methods along removable media are self-contained processes or steps that lead to the desired result.
It is a sequence. These steps or processes are performed in the required physical manipulations of physical quantities.
is there. Although not usually necessary, these quantities can be calculated using computer systems or electronic
Stored, transferred, combined, compared, displayed and otherwise manipulated within the device
Take the form of electrical signals that can be made. Bits, values, and elements of these signals
References to symbols, operations, messages, terms, numbers, etc. are in the ordinary sense of use.
Provides convenience in principle for reasons. All of these similar terms are appropriate
Note that these are merely convenient labels associated with the physical quantities and affixed to these quantities
I want to be. Except as otherwise apparent from the following description, the
Consider using terms such as, "processing" or "calculation" or "confirmation" or "comparison".
Calculates data expressed as physical quantities in registers and memory of the computing device.
Or other data that is also represented as a physical quantity in another electronic device and manipulates it.
Operation and processing of the computing device.System outline and data collection  Referring to the drawings, like numerals represent like parts throughout the several views. FIG.
Indicates a configuration in which the present invention is implemented. The subject 100 is typically a landscape
Around which a system 110 using the present invention is not planned.
Transported along an arbitrary route 101. Subjects include natural scenes, terrain, and man-made structures.
However, the present invention is not limited to this. Examples of system 110 include vehicles, aircraft, and human controls.
Includes built-in and self-cruising missiles. Typically, system 100 includes an imaging device.
And a motion detection device 104. In one embodiment, the imaging device or imaging
The device 103 is a digital camera that generates a digital image, and the motion sensor 104 is
Six freedoms of measurement (ie pose information) of system 110 with respect to a common coordinate space
An inertial measurement unit (IMU) that provides all degrees. The imaging device 103 is
An elephant image archive 102 is formed, where each image is created at the time the image was taken.
Tagged with a stupid pose. To image such an object, an imager 103
Is mounted to work in cooperation with the IMU device 104, and thus the video frame
Imager around or relatively inside a series of images or scenes in 105 format
A series of images is generated by gradually moving. For example, the imager is in a flooded area
Attach to the aircraft or a camera when a specific set of images from the shape is required
Is a car shop when image features are used to aid car navigation
Camera mounted on or with an IMU mounted on the root is 3D reconstruction
Is planned to be transported by a human operator. A series of specific areas
Images are thus generated as the vehicle moves over urban terrain. In the present invention
Is a motion detector like IMU104 that detects rotational speed and acceleration along orthogonal axes
The device is combined with the imaging process. Furthermore, the timing of discrete IMU measurements is
Exactly synchronized with the timing of each of the frames.

Synchronization of image frames and motion detection data can be achieved in various ways by those skilled in the art. For example, camera frames are triggered exactly by the IMU sample clock, or both camera frames and IMU samples are triggered in time using a common clock.

To facilitate the description of the present invention, assume that object 100 is a building. Thus, a user or operator travels around object 100 to form a series of images that provide a view around object 100. Imager 103 is a video camera, the focal length of which can be changed when surrounding images are generated. The imager 103 is associated with a digital sampling process that generates a digital image on a suitable medium 105 along with a time tag that associates the image with the IMU data. The imager 103 and the associated IMU device 104 are coupled to a computer system 106 that includes means for receiving the image and accompanying motion sensing device data during or after imaging.

Thus, a suitable medium 105 is actually a dedicated data bus. After the camera mechanization or acquisition phase, a digital image is generated. FIG. 1 shows a digital video (DV) camera that makes video recordings directly on magnetic tape. Alternatively, the camera is used to generate a digitized analog signal via a frame grabber in a real-time or post-processing stage. The configuration of the imager 104 itself does not affect the operation of the present invention. In the following, the imager 104 uses commonly used color formats, coordinate systems,
Or a series of digital images C ₁ , C ₂ ,. . . To generate a C _N.

One commonly used color space is the RGB color space, where each of the image color pixels is a vector C (i, j) = [R (i, j), G (i, j), B (
i, j)] ^T , where (i, j) is an image pixel C (i, j) and R, G, B are three intensity images of a color image C, respectively. The R, G, B color image data representation need not be the best color space for any desired calculation, and there are many other color spaces that are particularly useful for one or other purposes.

Computer system 106 is a computing system including, but not limited to, a desktop computer, laptop computer, or portable device integrated with an imaging system. FIG. 2 is a block diagram illustrating an exemplary internal configuration of the computer system 106. As shown in FIG.
06 is a central processing unit (CPU) interfaced with the data bus 120
122 and a device interface 124. The CPU 122 executes certain instructions to manage all devices and interfaces coupled to the data bus 120 for synchronized operation, and the device interface 124 communicates to external devices such as the imaging system 103 and the IMU device 104. Combined, and thus the image data and the IMU data are received over a data bus 120 into a memory or record. Display interface 12 is interfaced to data bus 120.
6, network interface 128, printer interface 130,
A floppy disk drive interface 138. Generally, a compiled and linked version of the present invention is loaded into record 136 through floppy disk drive interface 138, network interface 128, device interface 124, or other interface coupled to data bus 120.

A main memory 132 such as a random access memory (RAM) is also
2 provides instructions and access to memory storage 136 for data and other instructions
Interface to the data bus 120 for communication. In particular, the present invention
Stored applications, such as compiled and linked versions
When executing the program instructions, the CPU 122 performs
Operate the image data. ROM (read only memory) 134 is keyboard 1
40, display 126, pointing device 142 for basic input / output operation
Provided to store an immutable sequence of instructions such as a system (BIOS)
You.Feature extraction and tracking  One feature of the present invention is to extract and track only the most prominent features in the image sequence,
An automatic machine that estimates the motion of the imager automatically by using them in cooperation with IMU measurement
It is to provide structure. The feature used in the present invention is that one frame
Is characterized as the smallest visual change to the
Is placed most accurately in the image by the method. For example, a salient feature is the autocorrelation surface
Characterized by sharp peaks, or features such as corners at each of the image frames
You. There are many techniques to extract salient features and place them in subsequent frames
The detailed method should not be considered a limitation of the present invention.

To speed up the feature extraction and tracking process, the present invention provides an initial image or
Detects features only in images that appear to be tracked with some missing features
Use a prominent feature operator to Applicable with prominent feature operators
For images following the extracted image, the invention detects
Using multi-resolution hierarchical feature tracking to establish features corresponding to broken features
You. Search space for corresponding features is estimated from navigation processing subsystem
It starts at the point indicated by the specified feature arrangement.Extract prominent features  According to one embodiment, the salient features extracted are typically like corners in the image.
It is a feature. FIG. 3 shows a density image 20 including a white area 204 and a black (dark) area 206.
0 shows a 3D drawing 202. Drawing 202 corresponds to white area 204.
A raised surface 208 and a flat surface 210 corresponding to the black area 206 are shown. Corner
212 is a salient feature of the problem, and this position change can be determined most accurately and
Typically, there is minimal impact from one frame to the next.

The salient feature detection process is designed to detect all salient features in the image. The salient feature detection process is to apply a salient feature operator to the image to detect salient features. According to one embodiment, the feature detection operator or feature operator O (I) on image I is a function of the Hessian matrix of the local area of the image based on the Laplacian operator performed on that area. Specifically, the salient feature operator O (I) is determined as follows: _If = O (I) = Det [H (I)]-[lambda] G (I) where _If is given by O (I) This is a result determined as a feature image obtained from a prominent feature detection process. Det () is the determinant of the matrix H, and λ
Is a controllable scaling constant: G (I) = Ixx + Iyy.

The Hessian matrix is further expressed as:

[0027]

(Equation 1)

[0028]

(Equation 2) Where x and y are the horizontal and vertical directions and the second derivative, respectively: Is is the image I obtained by performing an image convolution with a 2D Gaussian kernel, typically 11 × 11 to 15 × 15 pixels in size. Here is a smooth version of.

One of the unique features of the salient features operator described herein is 214 in FIG.
, 208, 210, while only areas such as corners, such as 212, can be enhanced. This is useful because only features such as corners are constrained on two axes. Local maximum noticeable image I _f after being processed by the feature operator image I is conspicuous then be extracted, which corresponds to the prominent features. Typically, image I is an intensity image that is an intensity component in the HIS color space or a luminance component obtained from the original color image.

In general, each salient feature is represented as a template, such as an 11 × 11 or 13 × 13 image template. The characteristics or attributes of a salient feature template consist of the location of the features of the image, color information, and its intensity. The position indicates where the detected salient feature or template is located in the image, generally represented by coordinates (i, j). The color information carries the color information of the template centered at (i, j). Intensity or comprises information is extracted as how strongly I _f salient features or calculated.

In operation, there are N color images sequentially received from the imager. As each color image is received, it is first converted to a color space, where the luminance or intensity components are separated from the chrominance components. As will be apparent to those skilled in the art, color image transformation is only required when the original color image is represented in a format that is not suitable for the feature extraction process. For example, many color images are in the RGB color interval, and thus the luminance components are preferably converted to a color space that is merged into one image. The above feature operator is then preferably applied to the luminance component to form a plurality of salient features indexed as a plurality of templates and held in a table. Each of the templates records the characteristics or attributes of each feature.

By the time N color images have been processed, there are N corresponding feature tables
And each is comprised of a number of salient features. The table then shows how each of the features
Feature tracking used to detect if moving from one image frame to another
Organized as maps, referred to herein as map maps.Tracking prominent features  Tracking goals are sequential image files that correspond to the same physical point in the world.
The purpose is to find the image features of the frames. Computation of salient features in image frames
Data tracking is an image with the same visual features as the center of prominent features
Inaccurate matches are often found because they are based on finding a region. this is
Usually due to changes in imaging conditions or perspective appearance or occurrence of repeated patterns
. If the feature tracker estimates an incorrect feature correspondence, this error will be
Adversely affect dose estimation. Therefore, removing the fit of these erroneous features,
It is extremely important to maintain the maximum possible number of exact matches. Features of the present invention
According to the characteristics, multi-resolution feature tracking, based on epipolar geometry
Combination of technologies such as tracking and navigation aid tracking
Finds only exact feature correspondences in image sequences using the least possible calculations
Designed to work.Multi-resolution feature tracking  In a preferred embodiment, the multi-resolution hierarchical feature structure defines features for tracking.
Used to extract. More specifically, FIG. 4A shows two successive images 402
, 404 are continuously received from the imager. Outstanding Features Operator
After the data is applied to the image 402, one feature 406 is detected and the feature is recorded.
Is done. When the second image 404 comes in, the multi-resolution hierarchical image
A head is formed.

FIG. 4B shows an exemplary multi-resolution hierarchical feature structure 408 for extracting features 406 of the image 404. There are multiple image layers 410 (eg, L layers) of the image structure 408. Each of the image layers 410 is continuously generated from the original image 404 by a decimation process around the feature arrangement. For example, layer 410-L may be layer 410- (L-
This is caused by decimation of 1). The decimation factor is typically a constant, preferably equal to two. By knowing the given properties of the features found in the image 402 and that the two images 402, 404 are two consecutive images, the features of the image 404 and its arrangement 405 do not change dramatically. Thus, a rough search for the feature is determined in the second image, centered on the original location of the feature. More specifically, when the feature 406 is located at the coordinates (152, 234) of the image 402, the window for searching for the same feature is the image 404 (152,
234).

The multi-resolution hierarchical feature structure 408 indicates that as the number of layers 410 increases upward, the resolution of each layer 410 decreases. In other words, the search area is essentially enlarged when the size of the search window is still the same. As shown, the search window 412 covers a relatively larger area on layer 410-L than on layer 410- (L-1). In operation, layer 410-L is first used to find the general location of features in search window 412. One of the available ways to find the corresponding arrangement of functions in successive images is to use a template matching process. The template is typically determined as a square face receiving area (11x11 to 15x15) centered on the original feature locations extracted by the salient feature operators or the expected locations of the new frame features. The exact position of the corresponding sub-pixel of the match is then the maximum of the normalized cross-correlation of the two corresponding image regions (ideally a "1" for a perfect match) Layer 410- (L-1) is then used to update the general arrangement of features in the nearest area of the same window size, and finally layer 410 has the exact features of the features. Used to accurately determine location (x, y) The use of feature structures is known to have many advantages over prior art feature extraction approaches: essentially more efficient feature templates A large representation is achieved, which allows for effective and accurate tracking of features and is particularly suitable for hierarchical tracking mechanisms.

There are K prominent features in the image, where K ranges from 1 to 1000. Thus, there are K features, such as one in FIG. 4B. FIG. 4C shows K feature structures 420 from a single image, each of the feature structures 420 for a feature. As a result of the feature extraction, a set of attributes F (...) describing each of the K features is formed, and includes information on the position, intensity, and color of the feature.

A set of N image frames and K attributes F _i (...), I = 1, 2,. . .
At K, FIG. 4D collectively represents all the features found for the N images, and is used to track the features to estimate the motion of the imager, where a “feature tracking map” or simply Show what is called a feature map. FIG. 4E shows a flowchart of the feature extraction process. Both FIGS. 4D-4E are described in conjunction for a full understanding of the feature detection and tracking process of the present invention.

At 452, a color image is continuously received from the imager. A major component, preferably a luminance or intensity component, is extracted 454 from the color image. In one embodiment, the color image is simply converted to another color space that provides another luminance component. 4
At 56, the process looks for a feature template stored there, eg, in a memory area. If there are enough feature templates in the memory area, it means that the process needs to proceed to feature tracking on the next image, otherwise a new feature must be extracted at 458 It is necessary to check whether or not. In operation, the first image received always performs a feature extraction operation with a prominent feature operator because there is no feature or feature template stored to perform the feature tracking process. The process now proceeds to 460.

At 460, the feature extraction process generates K features of the received image (eg, frame # 1). As shown in FIG. 4D, K of received image frame # 1 is
Features exist. The properties of the K features, preferably as feature templates, are stored in a memory space for the subsequent feature extraction process.

When the next image arrives at 462, the process preferably proceeds to 464 to form a multi-resolution hierarchical image pyramid with the newly arrived image as a base. As described above, and as shown in FIG. 4C, the tracking process searches for an arrangement of image pyramids that most closely resembles each layer of the feature template stored in the feature structure. For each of the K multi-resolution hierarchical feature structures, K
Fewer corresponding features are placed 466 from each corresponding layer in the image pyramid, and K feature locations are then collected and added to features for frame 2. Similarly,
For the next n1 frame, the process proceeds to 462 via iteration 456 to extract K features from each of the n1 frames.

When an image is generated, the imager is moved around the object considerably with respect to the initial position from where the first image was captured. The K features are those that do not need to be found in the subsequently generated images. Due to perspective changes and the motion of the imager, these features have changed out of view or completely, so that they can no longer be tracked. For example, a roof corner of a house loses its salient features or is out of view when viewed from a particular perspective. Therefore, the representation 430 of K features for the n1 image of FIG. 4D is one of the features of the present invention.
The generation of additional new features occurs when the number of features falls below a predetermined threshold (T). At 456, no distinctive number is found in the incoming image and the process proceeds to K
Proceed to 458 to determine if new features need to be extracted to form the individual features. As noted above, the characteristic number is a change in perspective or closure (occlus
new features must be extracted and added to maintain a sufficient amount of features tracked in the image when dropped by ion). The process restarts feature detection at 460, i.e., applying a salient feature operator to the image to generate the missing set of salient features to form it. The process is shown by way of example in FIG. 4D to restart feature detection at frame n1.

As shown in FIGS. 4E and 4D, a feature template that matches a sequential image remains as an original set of tracking features of subsequent images, and does not change from one frame to another. Typically, establishing feature correspondence between subsequent image frames is achieved in two ways. One is to achieve this with a directly consecutive image pair, the other is by fixing the first frame as a reference and finding the corresponding arrangement of all other frames with respect to this reference frame. In one embodiment, the second approach is the reverse of the first approach, where significant drift can be integrated over several image frames, because it minimizes the bias or drift possible in searching for accurate features. It is. However, the second approach only allows the persistence of short-lived features over several frames, since the scene captured by the camera is subject to large changes in view when the camera covers a large displacement, thereby ultimately The tracking process is advanced to 472.

To maintain feature tracking over many subsequent frames using the second approach, a feature template update mechanism is included in 474. If the corresponding feature placement is not found in the most recent frame 492 with respect to the features in the reference frame 490 as shown in FIG. 4F, the template for the missing feature is replaced by the one located in the most recent frame 494. Where they were well tracked, for example at 494. The templates updated at 474 in FIG. 4E have the advantage that they are tracked well even if they have significant perspective changes by minimizing the integrated drift typical for the first approach. .

Of course, the feature representation is made after every certain number of frames. FIG. 4D shows the frame number
n1, n2, n3, n4, n5, and n6 are sets of features 432 for the image, respectively.
436 are shown. Frame numbers n1, n2, n3, n4, n5 and n6 are between
It need not have the same number as the frame. The imager moves further, more images
Generate an image, some of whose features are reproduced in several subsequent images,
440 are reused depending on the preferred implementation. At 470, the process
All frames are processed and the features are obtained. As a result, the example of FIG. 4D
Thus, a feature map is obtained.Feature tracking based on epipolar constraints  While the multi-resolution approach significantly reduces the search area for trackers,
The feature uses a scale similar to that of the feature template only, and
It does not use 3D point geometric constraints in
There is no proof.

As is well known to those skilled in the art, epipolar constraints determine one-dimensional constraints on the projected image layout of stationary points in a 3D environment observed by an imaging device. This constraint, given point corresponding two image frames ^{p1 = (x1, y1,1) T} , p2 = (x2, y2,1) that given pair of ^T is, they formula p1 ^T * F
* P2 = 0 must be satisfied, where F is the elementary matrix. This matrix is calculated from seven or more corresponding point pairs. This matrix is calculated to obtain epipolar constraints for all remaining correspondences of a particular pair of feature points in the image frame. To ensure that the original seven point pairs match exactly, a technique based on RANSAC (RANdom Sample Concensus) well known to those skilled in the art is applied. After this constraint has been calculated, the rest of the feature match is found by searching along these epipolar lines only. This approach greatly reduces the search area and reduces the problem of fitting false features outside of the search area.

The combination of the epipolar constraints of a pair of three image frames results in a much stronger constraint called trifocal constraints known to those skilled in the art. This latter constraint is 2
The placement of the feature points of the third frame, giving this placement of one other frame, is completely determined. FIG. 4G shows a flowchart of epipolar geometry enhancement between the current image frame # n480 and the previous image frame, which includes multiple enhancements 48.
Obtained from the confirmation of trifocal constraint in the arrangement of feature points in frame #n, which gives the arrangement of 2 and earlier frames.

The Kalman filter navigation process is used as a process following the extraction of appropriate features from an image. As used herein, Kalman navigation refers to navigation based on Kalman filtering that is well known to those skilled in the art.
It is often desirable to estimate the state of a system given a set of measurements made over a period of time. The state of the system refers to a set of variables that describe the unique characteristics of the system at a particular point in time. The Kalman filter is a useful technique for updating system state estimates or previous estimates by using covariance information of both the state variables and their indirect measurements, for example, using indirect measurements of state variables.

Mismatched features cause serious problems for Kalman navigation where they are not static, ie move independently. Using epipolar and trifocal constraints in tracking to eliminate most of these bad or unwanted correspondences between the current frame (frame #n) and the previous frame without knowing the camera platform movement Is possible. The Kalman filter is one of many statistical estimation processes used to provide navigation feedback for feature tracking. For simplicity, a Kalman filter will be used to describe the invention according to one embodiment.

Combining Hierarchical Feature Tracking with Epipolar Constrained Tracking
Initialization of the Kalman filter navigation process or the extraction of newly extracted features
Provides a powerful tool for accurate tracking of features during estimation, thereby
It provides optimal convergence during the critical initialization phase of the Kalman filter.Navigation assist feature tracking  As mentioned above, Kalman filter processing is platform location and visual processing
Form an optimal estimate of the feature location giving the set of features provided by Performance is the best
Achieved by major feature persistence and recollection of achieved features.

Using the platform position and feature position estimation, the 3D location can be used to accurately estimate the image position of these salient features already known, or for newly collected features relative to the epipolar line. It is possible to constrain tracking. As soon as the motion estimation of the camera is accurate, it is possible to calculate the epipolar constraint from this parameter as well, and then use this constraint on those newly collected features whose 3D placement is to be determined.

The combination of these techniques is key to helping the feature tracker. The estimated platform position and orientation along with the current feature location estimation allows for the estimation of pixel values in the next image formation. Hence, the Kalman filter provides a natural link to the feature tracking process. This navigational feedback to the tracker not only greatly reduces the risk of mismatching of the wrong features, but it also reduces the search area and thereby speeds up the tracking process. The more accurate the Kalman filter navigation estimate, the more tracking aids and hence the more accurate the tracking process.

The advantage of integrating feature tracking and Kalman filter navigation processing is one of the most important features of the present invention. It not only improves the accuracy and robustness of the navigation method, but also makes it possible to use significantly lower quality navigation and imaging components than similar attempts in prior art systems.

FIG. 5A shows a flowchart of the integration of feature tracking and navigation according to one embodiment of the present invention, which can be understood in connection with FIG. 6, which shows a functional block diagram of a system employing the present invention. The Kalman filter 604 estimates the platform state using information from the motion sensor shown in FIG. 5B. The platform position and orientation estimated along with the current feature position estimation allow estimation of pixel values in the next image formation 603. This information is provided to the feature estimation process. The search area for the next feature occurrence is derived by the Kalman filter. If the expected feature is not found for the estimated number of frames, the feature is declared lost and its image template and estimated location are archived 610.

As mentioned above, one of the advantages of the navigation estimation of the present invention is that each image collection
Image-only solution by estimating camera translation and rotational motion between
Improvement), which is often seen in prior art systems.
Avoid major causes of feature tracking failures. This allows the visualization of the imager
Tolerance for large angular movements between frames, including features that leave and return to the field
It becomes possible. The remaining feature motion of the subsequent frame is epipolar geometry
The result of the estimation error of the feature position from the relative platform determined by
It is important. This rigid connection between navigation and feature tracking processing
In both the reduced search window size as well as the reliability of feature tracking
Provides major benefits.Management of Kalman filter characteristics  The dynamic nature of feature tracking is based on feature management (ma
Management process), which is shown in FIG. 5A. Feature Manager
Plans to keep the current number of features of the Kalman filter at a constant value.

If a feature is not detected in a series of N frames, the feature is declared lost. When a feature is lost, a new feature is selected 601. It is designed to re-collect the archived features 606 that are expected to be visible. The expected visibility is determined by the stored feature placement, the current platform placement, and the previously stored feature template aspects. That is, if the line-of-sight from platform to feature is before the template is available from the archive, this feature is declared to be temporarily (provisionally) visible. You. The feature tracker then attempts to get this archive feature.
If successful, the feature is reinserted into the Kalman filter as an active feature. If the archived features are not available, or if the archived features were not obtained successfully, a new feature is inserted into the filter 608 and initialized to be in the preset range along the LOS ray. Is done.

In one embodiment, the number of Kalman filter features is set from 1 to 20. Characteristic
The upper limit is only determined by the processing power in the system. All features are given
Is not always visible in a given frame, but full covariance is
Maintained for all features held in the filter.Navigation data processing  For a rigid body moving with respect to the earth, the following equation of motion is
Describe how it is calculated from the rolling speed.

[0056]

(Equation 3) Direction cosine matrix from where [nu ^e = body frame relative position vector C _b ^e = ECEF frame in the velocity vector r ^e = ECEF coordinates of the earth is fixed to the Earth at the center (ECEF) coordinate system
f ^b = specific force of body axis (body axis) (
Inertial angle speed from the non-gravitational acceleration) (IMU accelerometer clearly measured) Omega _e = earth rotation speed g ^e = body of the Earth gravity vector ω _{^b /} ⁱ _b = body axis in ECEF coordinates by the device (specifically measured by IMU velocity gyro sensor) ω _{e / i e} ⁼ the position and orientation obtained inertial angle speed from the earth ECEF axes is relative to the axis coordinates fixed to the earth, the equation Regarding the rotation of the earth. The present invention requires careful attention for two reasons. First, a speed gyro having the ability to detect the rotation speed of the earth is used. Second, it allows the use of GPS, which provides measurements on earth fixed (ECEF) coordinates at the center of the earth as an optional input.

In the above calculation process, the IMU measurements are sent to a strapdown navigation algorithm well known to those skilled in the art. Calculate the position, velocity, and attitude with respect to ECEF coordinates by the strapdown navigation algorithm. This trajectory formed by the nonlinear differential equation shown above forms the basis for a linearized Kalman filter algorithm. The state vector for the Kalman filter includes errors in position, speed, and attitude with respect to the system related to the strapdown navigation trajectory estimation.

Due to errors inherent in IMU measurements and errors in initializing the strapdown navigation solution, the strapdown navigation solution drifts significantly from the true trajectory. Some external help must be applied to maintain pose estimation close to the true solution. IM with traditional help
On U systems, external help comes from GPS or other independent navigation systems. In conventional visual aided navigation, such as satellite-assisted navigation or DSMAC, the help comes from known geographical attributes of visual features such as known geodetic points or known geodetic aspects. . For example, in satellite-assisted navigation, the observation of stars relative to the earth is known as being along the line of sight of the prior art. For DSMAC, the features of the visual reference scene are known as geodetic earth latitude, longitude, and altitude. In the present invention, the above unknown features are obtained and used.

As shown in FIG. 6, the center estimation processing is extended Kalman filter processing 61
4 is used. Kalman filtering is controlled by the following factors: (1) The "whole value" strapdown algorithm 604 runs in parallel with the Kalman filter. The strapdown algorithm is IMU60
From 0, the position and orientation are transmitted from high-speed measurements of acceleration and rotation speed. (2) The Kalman filter determines the error for the strapdown navigator. (3) Kalman filter state vector is a strapdown device error model 60
7 based. (4) The equipment error is fed back to the strapdown navigation model 604 to prevent large drift induced by nonlinear effects. (5) The Kalman filter uses partial derivatives of nonlinear system dynamics with respect to the system state to form the propagation 605 and update 611 stages.

External help is achieved by updating the measurement to a basic extended Kalman filter. These updates are made completely asynchronously from the IMU measurements, but their temporal relationships must be accurately known.

The visual processing provides a feature measurement 610 represented by the pixel coordinates of the feature in the camera's field of view. The Kalman filter provides a platform and features 609 for the mathematical determination of pixel measurements for position and orientation. Each feature is modeled as three sets of XYZ in the ECEF space of the navigation process. Thus, if there are actually a total of 10 features, then a total of 30 new states are needed in the Kalman filter 610. The formulation is flexible with respect to the number of features supported by the Kalman filter.

As shown in FIG. 7, a feature 702 is an image 70 observed from a digital camera.
0 is a discrete point detected. Digital Camera Boresight 7
09 is generally known for the IMU axis set 701. This relationship is fixed or the camera is moved in a strictly known manner about the IMU axis. Pixel location 700 in an image frame tracked over time by the visual subsystem
Provides motion information to aid in strapdown navigation algorithms.
It is important to model the camera 609 geometry with respect to the camera image correction process and the IMU axis, and to ensure that the temporal relationship of the camera frames is known with respect to the IMU samples.

The most critical aspect of a feature is that each feature must be static in the surrounding sense, and most importantly, the “feature correspondence” is accurate. Feature correspondence means that the feature observed in the frame at the location corresponds to the same physical point as the feature observed in the previous frame. Features are automatically detected based on the characteristics of the scene content, and thousands of features will be processed by the Kalman filter. Feature processing 603,
606 is operated in parallel by the visual processor.

For the purpose of modeling feature measurements, each feature is accurately represented by three components of position in a reference frame 704 fixed to the earth. Thus, for each feature currently morphologically processed by the Kalman filter, there are three unknown time invariants. These unknown quantities are added to the state vector formed by the IMU instrument error.

Kalman filtering requires propagation 605 of the state vector covariance matrix using methods well known to those skilled in the art. This propagation computational load is typically assumed to increase with the cube of the state vector length. In order to avoid excessive computational load, the Kalman filter uses the current feature 6 in the field of view (FOV) and state vector.
Archived and de-archived by the presence of 10 61
3 Includes the ability to maintain an adaptive state vector.

Features are “archived” as they are observed leaving the camera FOV. Feature archival means storing the current feature location estimate, feature component covariance matrix, feature component to platform correction matrix, and feature reference image template. The de-archiving process replaces features in the Kalman filter formulation and resets feature locations and correction characteristics.

[0067] Imaging sensor surface 700 (pixel) Y, the unit vector u ^c for the feature of the relationship and the camera axis between Z feature positions is given by the following equation.

[0068]

(Equation 4) The following can also be written by inspection.

[0069]

(Equation 5) The physical measurement of the feature location is made from the interpretation of the pattern formed by the gray scale values from the CCD device. This measurement process includes random inter-frame noise related to image processing technology. Errors also arise from the CCD physical layout, the CCD signal sampling process, and the lens / optical path where light enters the CCD array. Nominally,
The net resulting pixel space represents samples from the preferred rectangular grid. The exact grid dimensions (in pixels) and the approximate pixel spacing of the physical dimensions are available from the manufacturer.

Empirical models are often used to represent consistent measurement errors for focal plane feature locations.

[0071]

(Equation 6) Where Y, Z = true physical displacement of the feature ray at the focal plane Y ', Z' = measured pixel count at the feature location Y _PIX , Z _PIX = number of pixels of the Y, Z dimension ε _11, ε ₂₂ = Error in pixel spacing 706 of both dimensions ε ₁₂ , ε ₂₁ = non-rectangular pixel skew terms 707 K ₁ , K ₂ = first and second order radial distortion terms 708 This is based on the basic camera for point features Describes the measurement and relates to the "error state" estimated by the filter. These error states are platform position 705 ([delta] r _p ^e) of the error (3) wherein position 704 ([delta] r _f ^e) of the error (3) of-alignment from the camera to the IMU 701 ([Delta] [phi]) error (3) • Error (3) in ECEF (e-frame) alignment ( Δθ ).

To complete the extended Kalman filter decision for the camera update process, the camera measurements (Y, Z) must be linearized with respect to these error conditions. This linearization is shown by the following equation.

[0073]

(Equation 7) Here, from Equations 5 and 6,

[0074]

(Equation 8) Is obtained.

The expressions for the measurements related to the required partial derivatives for the state are given as follows:

[0076]

(Equation 9) It will be apparent to those skilled in the art that a similar partial derivative for the "Z" feature measurement is identical to the above expression except that "Y" is replaced by "Z".

These equations represent a complete 3D linearized model of the feature measurement process. Feature updates in the Kalman filter can be performed using any selective GPS pseudorange or delta pseudorange (ps).
This is done using these partial derivatives completely asynchronously with the eudo-range) process. The extended Kalman filter feature update process uses a method well known to those skilled in the art. Selective GPS and / or feature update processing occurs because they are available, and the Kalman filter ensures optimal processing based on assumed statistics and fitted models.

In practice, at least three reference features from the archive of candidate features are kept “active”. If you need this set of active features from 10 to 20
(Or more). The active features are maintained with a Kalman filter and have their covariance characteristics updated, where the feature archive has thousands of candidate features at various stages of estimation accuracy available for use.

The feature is also well positioned so that its position is no longer updated by the Kalman filter, and the feature becomes a “landmark”. When the standard deviation of a feature becomes small enough, further attempts to update the feature actually lead to numerical instability in the filter calculations. Certain applications (eg, navigation of closed-robots)
Therefore, it is expected that the local scenes are all considered as landmarks, so that their characteristics are sufficiently arranged to be “calibrated”.

The Kalman filter requires an initial estimation of the feature location at each new feature occurrence.
This appears to be inconsistent with having no previous information about the feature set. The solution is to initialize each feature to lie along a ray defined by the center of the template projected through the camera geometry model, and assume a range along that ray. Typically, this range is 1
It is assumed to have a value of 00m. The processing result is independent of this initial estimate, within a factor of about ten. This means that the true range is anywhere between 10 m and 1000 m. An alternative solution is used in which the Kalman filter process is repeated with a process that proceeds in an assumed initial range of a few seconds. By observing the behavior of the mean square Kalman filter residual, it can be estimated that the initial estimation is poor. Iteration is done by re-initializing different estimation ranges until acceptable convergence is obtained. This property is not required for general urban navigation situations.

According to another embodiment, GPS is considered as a selective measure for the visual data processing system of the present invention. The GPS measurement update process handles each satellite measurement separately. Typically, there are eight (or more) satellites in the view, each satellite providing code-based range measurements as well as carrier phase-based delta range measurements. Both range and delta range measurements include bias errors associated with unknown parameters of the GPS receiver clock. GPS-based Kalman filtering therefore includes two measurement update stages for each of the satellites in the view at a rate of once per second.

FIG. 8 shows a set of exemplary states including sensor error model parameters and dynamic platform motion parameters. These states are shown in FIG.
And can be updated with each measurement.

(Through Arthur: Copied from Background Version of Original Version) One of the key aspects of the present invention is that the pose of the platform is placed in the scene without any prior knowledge of the scene. Are determined with respect to the surrounding scene without special pre-investigated targets. The invention is achieved by a tight integration of inertial navigation, image processing, and photogrammetric processing. Furthermore, due to the use of two different navigation modalities, equipment that mechanizes another navigation solution can be calibrated as part of the navigation process so that very low precision equipment can be used.

The invention described above avoids the limitations of the prior art systems by including a second motion detection modality. The present invention requires an independent measurement of the motion with respect to the camera pose at each image acquisition time. By integrating pause information between frames, a history of pause times is formed. The initialization of this solution uses the rough pose obtained from either the reference feature or some external geodetic information. Independently derived pause time histories drift from true poses unless updated by observing reference features fixed to the surrounding scene. The camera subsystem images its surroundings and the image analysis method is used to automatically select a set of features that are appropriate for the navigation reference point. This automated process uses the nature of the features within the context of their localized image features and the spatial diversity of the set of features in the scene.

The set of features is tracked by the camera / image processing system through successive image frames. Feature tracking is simplified because the feature location of the next image point is estimated by an independent navigation process and the assumption that the features are static in the scene (non-stationary features are automatically detected and discarded). ). Camera view (
FOV) feature location measurements are compared to estimates to provide a convergent prediction of the location of selected features in the surrounding scene. Because the platform moves around the scene, the features become continuously better placed and the platform pose is determined with respect to a fixed coordinate system for the scene. Feature set (set)
Can be accidentally lost from the camera's FOV, but its last known properties (image reference template and position estimate) are kept in a database. The archived features are recollected and used again as reference features without the need for relocation. By navigating within the scene, a feature archive is built and navigation accuracy is even more accurate. Excellent camera position estimation in the horizontal plane can be recovered using only a single feature observation.

The above process involves redundant information about platform pose dynamics obtained from image and motion measurements. This redundancy is most useful because the additional information can be learned with respect to errors in the detected components. Mathematical and statistical modeling of accelerometer, velocity gyro, and digital camera sensor errors can be used to estimate the respective errors in parallel with the feature location and platform pose estimation. The purpose of this self-calibration is very important in the present invention. This is because many applications require very small, low power, low cost sensing components. These factors avoid the use of precision components. The present invention is insensitive to significant accelerometer and gyro errors, misalignment, and the scale, skew, and radial distortion inherent in low precision camera systems. This insensitivity is the result of modeling these known error phenomena and estimating the required model coefficients simultaneously with the pose information.

The present invention is significantly different from the prior art and it will be apparent to one skilled in the art to provide a system for automatically obtaining pose information for a platform including a motion detection system and an imaging device. is there. The motion detection system and the imaging device are configured to cooperate to provide platform pose information without prior knowledge of the environment in which the platform navigates.

The present invention has been described in some detail with certain details. This disclosure of the embodiments is for illustrative purposes only, and numerous changes in the arrangement and combination of parts may be made without departing from the spirit and scope of the invention as claimed. Further, the disclosed invention can be implemented in various ways, including as methods, systems, and computer readable media.
Accordingly, the scope of the present invention is defined by the appended claims rather than the above examples.

[Brief description of the drawings]

FIG. 1 shows a system for implementing the invention.

FIG. 2 is a block diagram of a preferred internal structure of a computer system used in the system of FIG.

FIG. 3 shows a 3D image of an intensity image including a white area and a black area.

FIG. 4A shows images sequentially received from two exemplary continuous and imagers.

FIG. 4B illustrates an exemplary multi-resolution hierarchical feature structure for extracting one feature of the image of FIG. 4A.

FIG. 4C shows K image structures from a single image, each of which is for a feature.

FIG. 4D shows an example of what is referred to herein as a “feature tracking map” or simply as a feature map.

FIG. 4E shows a flowchart of a feature extraction process.

FIG. 4F illustrates updating a template with feature tracking within a set of consecutive images.

FIG. 4G shows a flowchart of an epipolar geometry enhancement process between a current image frame and a previous image frame.

FIG. 5A shows a flowchart of a process of integrating feature tracking and navigation according to one embodiment of the present invention.

FIG. 5B shows characteristics of information derived from a motion sensor.

FIG. 6 shows the entire processing functional diagram of the navigation computation element of the system.

FIG. 7 shows that features are discrete points detected in an image viewed from a digital camera.

FIG. 8 illustrates a set of exemplary states received from an inertial device and further propagated by the Kalman filtering of FIG.

──────────────────────────────────────────────────続 き Continuation of front page (81) Designated country EP (AT, BE, CH, CY, DE, DK, ES, FI, FR, GB, GR, IE, IT, LU, MC, NL, PT, SE ), JP (72) Inventor Cain, Jim United States of America 02452 Massachusetts 02452 Waltham Lexington Street 501 (72) Inventor Yates, Charlie United States of America 32547 Efty Walton Beach Central Le Avenue 924 (72) Inventor Tene, Arthur United States of America 95125 San Jose, California Coastland Avenue 2226 (72) Inventor Fejes, Thunder United States of America 95124 San Jose Cliday Avenue 4859 (72) Inventor Chen, Jinlong United States of America 95054 Santa Clara Cheney Street 4664 (72) Inventor Jablonski, Mark United States of America 94114 San Francisco 17th Street 3625A F-term (reference) 2F105 AA01 BB17 5J062 AA03 BB01 BB02 CC07 5L096 AA02 AA06 BA08 CA02 CA04 DA02 FA67 FA69 GA05 HA04 HA05 JA11

Claims (48)

[Claims] 1. A method comprising: receiving a series of images of a surrounding scene from an imaging device as the platform navigates around the surrounding scene; obtaining platform pose information through an estimation process with respect to features extracted from each of the images. A motion detector and an imaging device operative in a temporal relationship known as a motion detector that supplies motion data with respect to the platform such that each image corresponds to a respective set of motion data. How to get platform pose information including: 2. The method of claim 1, wherein each of the features reflects certain characteristics of an object of the surrounding scene, and further comprising processing at least one of the images to extract the features. 3. The method of claim 2, further comprising the step of continuously tracking features in each of the images following the at least one processed image. 4. The method of claim 3, further comprising the step of: cooperating with the motion measurement device to determine a respective arrangement of the features with respect to the first coordinate space. 5. The method of claim 2, wherein each of the features is prominent and has only a minimal change from one image to another in a sequence of images. 6. The method of claim 3, wherein certain characteristics of the object are included in the tracking of the feature such that tracking of the feature is made more efficient. 7. The method of claim 6, wherein certain characteristics of the object include predefined characteristics of the object intentionally placed in the scene. 8. The method of claim 7, wherein the predetermined characteristics include at least one of (i) a cross-section of the linear feature, (ii) a vertical state of the linear feature, and (iii) a color-based feature. Method. 9. The method of claim 1, wherein the processing of at least one of the images to extract a feature of interest comprises detecting each of the features by applying a salient feature operator, wherein the salient feature operator comprises 3. The method of claim 2, wherein when applied to at least one, enhances regions such as corners while suppressing at least one edge-like and uniform region of the image. 10. The method of claim 9 wherein said feature operator operates on a smooth version of at least one image based on a function of a Hessian matrix of Laplacian operators. 11. The continuous tracking of features of each image comprises detecting features in each of the images following the at least one processed image along an epipolar line with features extracted from the at least one processed image. The method according to claim 9. 12. The image following at least one processed image by using an estimation process such that the continuous tracking of features of each image is such that the search area for each of each feature of the continuous images is sufficiently narrowed. 10. The method of claim 9, comprising estimating the location of the feature at each of the following. 13. The method of claim 12, wherein the estimating process is based on a Kalman filter. 14. The method of claim 1, further comprising calculating a set of parameters from the motion data using a strapdown navigation process. 15. The method according to claim 14, wherein the estimation process is based on a Kalman filter. 16. The set of parameters is (i) location of the platform, (ii)
16. The method of claim 15, including at least one of velocity, (iii) attitude information. 17. The method of claim 16, wherein each of the parameters need not be exact. 18. The method of claim 17, further comprising updating the set of parameters with input from the external positioning system when the external positioning system becomes available.
The described method. 19. The method of claim 18, wherein the external positioning system is a global positioning system (GPS). 20. The method of claim 19, wherein the parameters further include features from the image. 21. The step of obtaining platform pose information further comprises the step of obtaining an imaging model of the imaging device and providing parameters to an estimation process for estimating the platform pose information, wherein the imaging model comprises: 21. The method of claim 20, wherein the features reflect how the features are transformed from the object around the image. 22. The method of claim 21, wherein the step of obtaining an imaging model further comprises updating the imaging model in response to error data from the estimation process such that the imaging model is constantly updated. 23. The method of claim 1, wherein the platform is selected from the group consisting of a vehicle, an aircraft, a boat, a human operator, and a missile, each of the groups provided with a motion sensor integrated with the imaging device. . 24. Forming a series of surrounding images from the imaging device as the platform navigates; processing at least one of the images to extract features that reflect certain characteristics of the surrounding objects; Successively tracking each of the image features following the at least one processed image following the performed feature; obtaining a set of parameters from the motion data through a strapdown navigation process; The steps of obtaining pose information for the platform by using an estimation processing operation, wherein the imaging device is informed by a motion detection device that supplies motion data with respect to the platform such that each image corresponds to a respective set of motion data. Estimation processing is performed based on the time relationship specified, and the platform pose information is statistically calculated from the estimation processing. It estimated coupled with the strap-down navigation processing to be a method for obtaining a pause information platforms, including motion detection device and the imaging device under parameters. 25. The method of claim 24, wherein certain properties are included in the processing so that the processing is performed more efficiently. 26. The method of claim 24, wherein said processing of at least one of the images to extract features comprises detecting each of the features by applying a feature operator. 27. The feature operator is a salient feature operator that, when applied to at least one of the images, enhances a region such as a corner, while highlighting at least one edge of the image. 27. The method of claim 26, wherein such and uniform areas are suppressed. 28. The method of claim 26, wherein the successive tracking of features of each image is performed by using an estimation process such that a search area for each of each feature of the successive images is sufficiently narrowed. 26. The method of claim 25, comprising estimating a location of the feature at each of the following. 29. The method of claim 28, wherein the estimating process is based on a Kalman filter. 30. Maintaining a feature list including the extracted features; and updating the feature list each time one of the extracted features disappears from one of the series of images. 25. The method of claim 24, wherein said tracking is performed continuously. 31. The updating of the feature list comprises processing one of the subsequent images to extract new features to be inserted into the feature list such that the number of features in the feature list is kept constant. 31. The method of claim 30, further comprising. 32. Determining position information for each feature with respect to a coordinate space in which the motion detection device operates; providing motion data to an estimation process along with the position information to estimate an error in the imaging model; 32. The method of claim 31, further comprising the steps of: receiving error data from the estimation process and updating the imaging model to have the least error, wherein the imaging model indicates a mapping relationship from the object to the features of the image. 33. The method of claim 32, further comprising the step of receiving error data from the estimation process and updating the position information of each feature such that the position information has a minimum error. 34. The method of claim 33, wherein the motion data includes at least one of (i) rotation data and (ii) translation data for the platform. 35. The method of claim 24, wherein the motion data need not be accurate, and the resulting pose information operates with the motion data along features extracted using an estimation process. 36. A motion detection device comprising a global positioning system (GP)
36. The method of claim 35, wherein S) is a sensing device and provides pseudorange and pseudorange speed from the imaging device to the GPS satellite. 37. The method of claim 35, wherein the motion sensing device is an inertial measurement unit (IMU) and the sensors included are at least one gyro and one accelerometer that provide rotation and translation data, respectively. 38. A motion detector integrated with the platform and providing motion data about the platform; generating a series of images of the scene, each image corresponding to one set of motion data; An imaging device operating in a known and temporal relationship therewith; a computing system receiving motion data and images and coupled to the motion detecting device and the imaging device, the computing system comprising a processor and an application module And a memory space for storing code for the application module when executed by the processor: processing at least one of the images to extract features that reflect certain characteristics of surrounding objects; Each of the features of the image following the at least one processed image following the feature that was Obtaining a set of parameters from the motion data through a strapdown navigation process; obtaining platform pose information by using a featured estimation process operation as part of the input; Operating in a known temporal relationship with a motion detector that supplies motion data with respect to the platform to correspond to each set of motion data, the estimating process is such that platform pose information is statistically estimated from the estimating process. A system that combines parameters with the strapdown navigation process to obtain pose information for the platform of the scene without prior knowledge of the scene, receiving parameters. 39. The system of claim 38, wherein each feature is extracted by using a feature operator according to a characteristic feature. 40. Each feature is a salient feature, wherein the salient operator is a salient feature operator, wherein the salient feature operator, when applied to at least one of the images, enhances a corner-like region; 40. The system of claim 39, while suppressing at least one edge and uniform area of the image. 41. The method of claim 26, wherein the successive tracking of features of each image is performed by using an estimation process such that a search area for each of the features of each successive image is sufficiently narrowed. 41. The system of claim 40, comprising estimating an arrangement of features at each of the following. 42. The system according to claim 41, wherein the estimation processing is based on a Kalman filter. 43. Maintaining a feature list including the extracted features; and updating the feature list each time one of the extracted features disappears from one of the series of images. 39. The system of claim 38, wherein said tracking is continuous. 44. The updating of the feature list further comprises the step of processing one of the following images to extract new features to be inserted into the feature list such that the number of features in the feature list is kept constant. 44. The system of claim 43. 45. Determining position information for each feature with respect to a coordinate space in which the motion detection device operates; providing motion data to an estimation process along with the position information to estimate an error in the imaging model; 39. The system of claim 38, further comprising updating the imaging model in response to error data from the estimation process to have the least error, wherein the imaging model indicates a mapping relationship from the object to the image features. 46. The system of claim 45, wherein the application module further comprises: receiving the error data from the estimation process and updating the position information for each feature such that the position information has the least error. 47. The system of claim 46, wherein the motion data includes at least one of (i) rotation data and (ii) translation data for the platform. 48. The system of claim 47, wherein the motion data does not need to be accurate, and the obtained pose information operates with the motion data along features extracted using an estimation process.