WO2007048674A1

WO2007048674A1 - Sytem and method for camera-based tracking

Info

Publication number: WO2007048674A1
Application number: PCT/EP2006/066754
Authority: WO
Inventors: Reinhard Koch
Original assignee: Siemens Aktiengesellschaft
Priority date: 2005-10-25
Filing date: 2006-09-26
Publication date: 2007-05-03
Also published as: DE102005051396A1

Abstract

The invention relates to a system and method for continuous determination of pose information which describes a position and orientation of an image recording unit with respect to a real environment. In order to allow efficient optical tracking, it is proposed that, during an initialization phase, the pose information be determined by comparison of the at least one image, recorded during the initialization phase using the image recording unit of an object in the real environment with a three-dimensional model of the object, with the pose information being obtained by evaluation of movement information during a tracking phase which follows the initialization phase, with the movement information characterizing position changes of elements of the real environment within images of the real environment recorded using the image recording unit during the tracking phase, and that, in the situation in which the pose information cannot be determined during the tracking phase, a reinitialization phase is started, during which the pose information is determined by searching for known features in the real environment in at least one image, recorded during the reinitializaiton phase, of the real environment.

Description

description

System and method for camera-based tracking

The invention relates to a method and a system for the continuous determination of pose information, which describe a position and an orientation of an image acquisition unit with respect to a real environment, as well as an augmented reality system comprising such a system for the determination of pose information.

Such a system or such a method are used, for example, in augmented reality (AR) applications. Augmented reality, augmented reality, is a form of human-technology interaction that can be used by humans. B. via a data glasses information fades into his field of view and thus the reality perceived by him erwei ^¬ tert. This happens context-dependent, ie matching to and derived from the object under consideration, eg. As a component, a tool, a machine or to its location. In ^¬ game this can be a safety notice during a assembly / disassembly process.

AR devices can be used in the application domain producing industry, medicine or in the consumer sector.

In the manufacturing industry, applications ranging from simple operator control and observation processes to complex service activities can be supported. In operations, examinations and treatments in the medical environment, such methods and devices serve a user to improve the quality of work. In the consumer sector, applications such as Navigation of persons, Informationsbreitstellung etc. be realized.

For the provision of information usually worn on the body devices are needed. Via a receiving and transmitting device, the coupling to company data realized. A head-worn semi-transparent display is frequently used for visualization of the information, which the user simultaneous Betrach ^¬ of the information and its real environment processing is made possible.

In order to achieve this virtual extension of the field of view of the user with exact position, tracking methods are used. These can consist of both hardware and software. The tracking methods serve to determine the position and the angle of view of the user. With this information, a positionally accurate extension of the field of view of An ^¬ wenders is possible. In this case, optical, inertial, akusti ^¬ specific, magnetic, or other methods are used. Frequently, the calculations required for the positionally accurate virtual extension of the field of view are performed on a stationary remote server.

For example, DE 10 2004 061 841 A1 discloses optical position detection systems which determine the position and / or orientation of objects in a measurement space. A position detection system can be embodied, in particular, as an optical tracking system which, by means of computer-aided image processing, determines the position of objects located in the measurement space and / or the position of the camera via the recognition of the objects.

The invention is based on the object of specifying an efficient optical tracking method.

This object is achieved by a method for the continuous determination of poses information describing a position and an orientation of an image sensing unit with respect to a real environment, - wherein aufgenom- during an initialization phase the Poseninfor ^¬ mation by comparing at least one during the initialization phase of the image capture unit mens image of an object of the real environment with egg ^¬ nem three-dimensional model of the object ^¬ who determined, during a subsequent to the initialization tracking phase tracking the pose information is obtained by evaluating motion information, the movement information changes position of elements of the real environment within during Trackingpha ^¬ se with the image capture unit captured images of the real environment feature and where in the case that the pose information during the tracking phase can not be determined, a Reinitiali- is sierungsphase started while the mation the Poseninfor ^¬ by searching known characteristics of the Real environment can be determined in at least one recorded during the reinitialization phase image of the real environment.

Furthermore, the object is achieved by a system for the continuous Chen determining poses information train a position and an orientation of an image sensing unit in Be ^¬ describe a real environment, the system comprising: the imaging unit, - a memory for a three-dimensional model of a Ob ^¬ jektes the real environment, first means for determining the Poseninformationen during an initialization phase by comparing at least egg ^¬ nes during the initialization phase with the Bildfas- sung unit recorded image of the object with the three-dimensional model, second means for determining the Poseninformationen by evaluating motion information during a tracking phase subsequent to the initialization phase, wherein the movement information changes position of elements of the real environment within during the tracing ckingphase with the image capture unit from received ^¬ images of the real environment feature, and third means for determining the pose information during a reset phase for the case that the pose information during the Tracking Phase is not ermit ^¬ can be telt, wherein the third means such ges ^¬ are taltet, that the information poses during the re initialization ^¬ at least one tialisierungsphase during cleaning the captured image of the real environment Conversely ^¬ be determined by searching for known characteristics of the real environment in.

The invention is based on the finding that an effi cient ^¬ tracking is ensured when applied in various phases of different methodologies for determining the position and orientation of the image capturing unit with respect to the real environment. In the inventive method, the SEN ^¬ initialization is first performed by ^¬. During the initialization phase the pose is first determined that the Bil ^¬ derfassungseinheit defines the position and orientation in relation to the real environment. For this purpose, a three dimensional model of an object is attracted ^¬ forth, with the associated real object in Erfas ^¬ is sungsbereich the image capture unit. The image of the object taken with the image acquisition unit is compared with the three-dimensional model of the object. Based on this comparison, an initial pose for the image ^{acquisition unit} can be determined. For example, in the comparison, the outlines of the 3D model are compared with outlines of the object detected within the image. If these can be brought into coincidence, the pose of the image acquisition unit can also be determined.

The poses determination above by comparing at least a very robust image of the object with a three dimensional Mo ^¬ dell of the object. Even with a partial Covering the real object, the process generally works very reliable.

However, the method used to determine poses in the initialization phase is relatively computationally intensive. In many applications, such as augmented reality, the image capture unit is continuously moved with respect to the elements of the real environment, so the pose changes accordingly. Therefore, it is desirable to reduce the amount of computation required during the initialization phase. This happens according to the invention during the tracking phase in that the Poseninformatio ^¬ NEN are determined by evaluating the movement information. The motion information only describe onsänderungen positive, which are traversed by the elements of the real environment within the during the tracking phase with the image capture unit ^¬ captured images. For this purpose, for example, striking points of the real environment in ^¬ within the images are tracked. The positional change that experi these elements within two successive recordings in particular by movement of the imaging unit ren ^¬ be determined. From these naturally 2-dimensional position changes, finally, a change in the pose with respect to all six degrees of freedom can be determined. If, for example, at least five elements or five points of the real environment are observed in this way, the two-dimensional change of these points between two images taken by the image acquisition unit can be used to infer the change of the pose in three-dimensional space, which was ^¬ through images during recording of these two Ab.

This determination of the pose based on the movement information carried out during the tracking phase is far less computationally intensive than the previously described pose determination based on a comparison of an image with a three-dimensional image. len object model. Therefore, a higher dynamic is achieved during the tracking phase.

However, during the very rapid determination of poses within the tracking phase, it may happen that the movement information for determining the pose can no longer be used. This is the case, for example, when the image acquisition unit undergoes a very large change in position or orientation within a very short time interval. In particular, when the image capture unit is moved completely out of the modeled area of the real environment in the meantime, the tracking phase can not be continued without further ado. In ^such cases, a reinitialization phase is initiated in which a new starting pose is determined. In this Reiniti ^¬ alisierungsphase known features of the real environment are sought within the detection range of the camera to be able with the means to draw conclusions about the position and Orientie ^¬ tion of the imaging unit with respect to the real environment draw and thus the required poses Informa ^¬ to determine the Once these features are found and the corresponding pose information has been generated, you can switch back to the very fast tracking phase.

An embodiment of the invention is particularly advantageous in which the features of the real environment sought during the reinitialization phase are determined during the initialization ^phase and / or tracking phase. In this way, the features are automatically generated by the system, and do not have to be previously provided by other means. With the aid of the image acquisition unit, the system can thus "learn" during the initialization and tracking phase which features of the real environment correlate with which poses. In a further advantageous embodiment of the invention, an edge detection algorithm is used to determine the Posenin ^¬ formations during the initialization phase. This attempts edges of the three-dimensional Objektmo- dells approximately phase with edges within the captured by the image capture unit image of the object during the initialization ^¬ to bring in cover, to thereby return ^¬ connections to draw on the pose of the image capture unit. A prerequisite for this purpose that the imaging unit already roughly a pose has reached in which a Annae ^¬ hernde correspondence of the edges of the three-dimensional model with the images taken by the image acquisition unit edge is present. It is therefore necessary for this purpose, the Bil ^¬ derfassungseinheit currency position in one suitable for this initialization to move.

In a further advantageous embodiment of the invention, a SIFT algorithm is used to determine the Poseninformatio ^¬ NEN during Reinitialisierungsphase. Such a SIFT algorithm makes the determination of the pose information by searching the known features of the real environment in the image of the real environment taken during the reinitialization phase. The advantage of using such an algorithm in the reinitialization phase is that the image acquisition unit does not necessarily have to be moved to a suitable initial position in order to be able to determine the position or orientation of the image acquisition unit. This is usually in the initialization ^¬ approximate phase of the case as a Posen determination based on a comparison of the three-dimensional object model with the two-dimensional image requires that the image capture ^¬ unit a pose during the recording of the image has the ei ^¬ nen such a comparison, allows. During the Reinitia ^¬ lisierungsphase is simply trying to features that were examples play as early as the initialization or Trackingspha- se determined, within the during the re-initialisation tion phase of the real environment.

Also, the SIFT algorithm is similar to the edge tracking relatively computationally intensive, but has the distinct advantage that this information is not certain pose of Bilderkennungsein ^¬ ness is required when recording the image. For example, if the image capturing unit moved by a user for a short time from the modeled area and subsequently pivoted back into the field, the pose, the image acquisition unit are determined from the erkann ^¬ th features directly. When such a method is used, for example, for augmented reality applications, a user of the system will hardly notice the intermittent tracking loss which occurs when he moves his head and thus the head-mounted image capture unit out of the modeled environment.

In a further advantageous embodiment of the invention, augmented reality information is displayed with exact position in a field of view of the user on the basis of the pose information. For example, a user wears a pair of data glasses and a camera attached to the user's head, which serves as an image capture unit. Where the position and orien- tation of the camera relative to the real environment ermit ^¬ is telt, can access the viewing angle of the user rückge ^¬ closed are and thus the position-accurate integration of augmented reality take place information in the field of view to ^¬ wenders.

An embodiment of the invention is therefore particularly advantageous in which an augmented reality system has a system with means for carrying out method steps according to one of the abovementioned embodiments of the invention. In the following the invention will be described and explained in more detail with reference to the embodiments illustrated in the figures. Show it:

1 shows a schematic representation of a method for the continuous determination of pose information,

2 shows a determination of poses during the Initialisierungspha ^¬ se and

3 shows a determination of poses during the reinitialization phase.

1 shows a schematic representation of a method for the continuous determination of pose information, which describe a position and an orientation of an image acquisition unit with respect to a real environment. Such a method is used in applications, for example, Augmented Reality An ^¬ to the position and the angle of egg nes user with respect to a consideration of his real environment to determine. Such a Posenbe ^¬ mood is required for the positionally accurate insertion of additional information in the field of view of the user, which is performed in augmented reality applications, eg for user guidance purposes.

During an initialization phase 1, the position and orientation of the imaging unit are initially with ^¬ stuffs the image recognition determined. An attempt is made to match outlines of the real environment with outlines or edges of a model of the object. Are edges within an image from ^¬ again recognized the real environment in three-dimensional model of the object, the image capture unit can thus be concluded that the pose.

However, such an adjustment is very compute-intensive, so that during the initialization phase only about three images can be evaluated per second. Therefore, after the initial pose determination, a tracking phase 2 is started, in which only movements of individual elements within images taken during the tracking phase are used to determine poses. Once the first pose of the initialization phase are withdrawn together with the associated image be ^¬, constricting image can be determined to what extent the elements of the real environment have moved within the consecutive images during the tracking phase in a nachfol ^¬. Corresponding motion information is generated for the observed elements, which can be used to determine a change in pose made between the two images of the real environment. This type of determination poses is much less computationally intensive than the previously described sierungsphase poses determination during Initiali ^¬. Because of this, a frame rate in the range of twenty frames per second can be achieved.

During the tracking phase, therefore, movements of individual image elements within successive images of the real environment are used in order to determine from these the change in the pose of the image acquisition unit which is usually the cause of this. As elements, for example, certain pixels of the image acquisition unit are selected with certain properties whose movements within the images are tracked from image to image.

As long as a reliable poses determination based on the BEWE ^¬ occupancy information is possible, the procedural described continues reindeer. If, however, for example by rapid head ^¬ the user turn the imaging unit completely out of the modeled environment field moves and then into moving back into the surrounding area, then a Posen determining how it is performed in the tracking phase, not be continued without difficulty. It is a re- initialization phase 3 is necessary as soon as the image capture unit ^¬ back to the modeled environment area overall is pivoted. During the reset phase 3 is true ei ^¬ ne new Ausgangspose as the basis for tracking phase 2 be ^¬. Here, the image capture ^¬ is in the detection range unit in accordance with known characteristics sought. These are noticeably male detected, a recalculation can pose Runaway ^¬ leads are. Examples of such features are certain colors which are characteristic of certain areas of the real environment. In contrast to the initialization phase 1 it ^¬ 3 requests the reset phase is no alignment of the imaging unit in a corresponding Initialpose which allows an alignment of the real environment with the three-dimensional ambient or object model. Therefore, a user of the system, for example, for augmented reality applications does not have to interrupt his work noticeably.

2 shows a determination poses during the initialization phase ^¬ approximately. In the detection range of the image capturing unit is a drive controller 4, by the three-dimensional model described in a memory of a system for continu- ous provision of poses information train a position and an orientation of an image sensing unit in Be ^¬ a real environment, is present. From the three-dimensional model of the drive control characteristic outlines 5 are extracted. Furthermore, one or more images of the drive control 4 are taken with a camera embodied as CCD image sensing unit of the system up ^¬. In order to determine the pose, an attempt is made to recognize the contours 5 in an image of the real environment or of the real drive control 4. In order to make this possible, should the image capturing unit in a pose brought present in that image of the real drive control in rough agreement with the three-dimensional model of the controller, so that recognition of the contours 5 of the Mo ^¬ dells in the image, for example by means of a Kantenerken- algorithm is possible. FIG. 3 shows a determination of poses during the reinitialization phase. Such re-initialization is example ^¬ as necessary if the pose of the image capture unit has greatly changed during the tracking phase between two consecutive images of the real environment. In such a case, a pose determination based on the motion information u. U. no longer possible and it must be determined a new starting position for the tracking phase. A reinitialization phase is generally necessary even if the detection area of the image acquisition unit is completely moved out of the area of the real environment, which is present as a model in a memory of a corresponding system for determining the pose.

During the reinitialization phase in the image area the

Image acquisition unit, which is executed in this example as a CMOS camera, features of an object of the real environment, which is here to act in the well-known from FIG 2 Antriebssteue ^¬ tion 4 sought. These features are determined using a well-known from the prior art SIFT algorithm in the images of the drive control 4. From the recognized features finally the pose is determined.

The characteristic features that are used to determine poses are stored in a grid 6 of the three-dimensional object model. This can be closed when finding these features on their position within the object model and finally the pose can be determined.

The characteristics can be stored for the continuous determination of poses information in a memory of a geeig ^¬ Neten system in advance. Alternatively, the features can also be "collected" during operation of the system, ie during the initialization and tracking phase.

Claims

claims

1. A method for the continuous determination of poses information indicating a position and an orientation of a BiI- derfassungseinheit with respect to a real environment beschrei ^¬ ben, wherein, during an initialization phase the Poseninfor ^¬ mation by comparing at least one during the initialization phase of the image capture unit aufgenom- menen an object of the real environment with egg ^¬ determined image nem three-dimensional model of the object ^¬, wherein recovered during the initialization phase ^¬ sequent tracking phase subsequent poses information by evaluating th motion information to, the movement information changes in position of elements of the real environment within of images recorded during the Trackingpha ^¬ se with the image capture unit images of the real environment and where - in the event that the Poseninformationen can not be determined during the tracking phase, a Re initialization phase is started, during which the Poseninfor ^¬ mations by searching known features of the real environment in at least one recorded during the reinitialization phase image of the real environment are determined.

2. The method of claim 1, wherein the sought during the Reinitialisierungsphase features of the real environment during the initialization phase and / or tracking phase are determined.

3. The method according to claim 1 or 2, wherein an edge detection algorithm is used to determine the position information during the initialization phase.

4. The method according to any one of the preceding claims, wherein a SIFT algorithm is used to determine the Poseninformati ^¬ ons during the Reinitialisierungsphase.

5. The method according to any one of the preceding claims, wherein on the basis of the pose information augmented reality information ^¬ position accurately in a field of view of a user a blinded ^¬ .

6. System for the continuous determination of poses information describing a position and an orientation of an image sensing unit with respect to a real environment, the system comprising: the imaging unit, - a memory for a three-dimensional model of a Whether ^¬ jektes the real environment, first means for determining the pose information during an initialization phase by comparing at least ei ^¬ nes sungseinheit captured during the initialization phase with the Bilderfas- image of an object of the re ^¬ alen environment with the three-dimensional model, second means for determining the pose information by evaluating motion information during the initialization phase subsequent tracking phase, wherein the movement information characterize changes in position of elements of the real environment within recorded during the tracking phase with the image acquisition unit Ab ^¬ images of the real environment n, and third means for determining the pose information during a reset phase for the case that the pose information during the Tracking Phase is not ermit ^¬ can be telt, wherein the third means such ges ^¬ are taltet that poses information during re ^¬ initialization phase by Searching known features of the real environment in at least one recorded during the Reinitialisierungsphase image of the real Umge ^¬ exercise be determined.

7. The system of claim 6, wherein the system fourth means for determining the searched during the reset phase characteristics of the real Conversely ^¬ bung during the initialization phase and / or having tracking phase.

8. System according to claim 6 or 7, wherein the first means are provided for executing an edge detection algorithm for determining the pose information during the initialization phase.

9. The system of claim 6 to 8, wherein the third means are provided for carrying out an SIFT algorithm for determining the Poseninformationen during Reinitialisierungsphase.

10. augmented reality system with a system according to one of claims 6 to 9.