DE102016006232A1 - Method and system for aligning a virtual model with a real object - Google Patents

Abstract

A method according to the invention for aligning a virtual model (2) on a real object (1), in particular in an augmented reality, wherein a first real marker (3) is arranged on the real object, comprises the steps:
- detecting (S20) a position (r) of the first real marker;
- and
- aligning (S20) the virtual model and a virtual marker (4 ') relative to each other according to the position of the first real marker relative to the real object;
wherein the position (b) of the virtual marker relative to the virtual model has a deviation (a) from the position (v) of the first real marker relative to the real object; as well as the steps:
- determining (S20) a position (a) of a second real marker (3 ') relative to the first real marker; and
- determining (S30) the deviation on the basis of the determined position of the second real marker relative to the first real marker.

Description

The present invention relates to a method and a system for aligning a virtual model with a real object and to a computer program or computer program product for carrying out the method.

From your own DE 10 2007 059 478 B4 For example, a method of aligning a virtual model, which is a representation of a real object, to this real object is known.

Referring to 1 is doing by a camera 5 a position r of a real marker 3 that is attached to a real object 1 is arranged, recorded.

The DE 10 2007 059 478 B4 suggests a virtual model 2 that is a representation of the real object 1 is, and a virtual marker 4 relative to each other according to the position of the marker 3 relative to the real object 1 align.

This will change the orientation of the virtual model 2 on the real object 1 facilitates, in particular the determination of a position m of a model-fixed coordinate system M in a world coordinate system I, in which a position r of the real marker 3 is detected such that this position m coincides with the position of an object-fixed coordinate system O, in particular by linking the transformations vor = m with the known transformation v between a marker-fixed coordinate system R and the object-fixed coordinate system O. In this way, the real object 1 in augmented reality with the virtual model 2 be deposited.

Is preferred in the DE 10 2007 059 478 B4 the position v of the virtual marker 4 relative to the virtual model 2 equal to the position v of the real marker 4 relative to the real object 1 , as in 1 for the virtual marker 4 indicated by dashed lines.

However, it can happen that the real marker 3 in such a position relative to the real object 1 that align the virtual model 2 and virtual marker 4 relative to each other is low, in operation by the camera 5 poorly detectable and / or only bad on the real object 1 is alignable, for example, if this is in a modification of 1 no intersecting edges 1.1 . 1.2 or the like.

An object of the present invention is in particular that of the DE 10 2007 059 478 B4 known method, system or computer program (product) to improve.

This object is achieved by a method having the features of claim 1, a system having the features of claim 6 and a computer program (product) according to claim 10 and 11, respectively. The subclaims relate to advantageous developments.

• – Erfassen einer Position des ersten realen Markers; und
• – Ausrichten des virtuellen Modells und eines virtuellen Markers relativ zueinander entsprechend, d. h. auf Basis, der Position des ersten realen Markers relativ zu dem realen Objekt, wobei die Position des virtuellen Markers relativ zu dem virtuellen Modell gegenüber der Position des ersten realen Markers relativ zu dem realen Objekt eine Abweichung aufweist. Insbesondere diesbezüglich wird ergänzend auf die eingangs genannte DE 10 2007 059 478 B4 Bezug genommen, deren Inhalt ausdrücklich vollinhaltlich zum Gegenstand der vorliegenden Offenbarung gemacht wird.

According to one embodiment of the present invention, a method for aligning a virtual model with a real object, in particular in augmented reality, wherein a first real marker is arranged on the real object, comprises the steps:

• Detecting a position of the first real marker; and
• Aligning the virtual model and a virtual marker relative to one another, ie, based on the position of the first real marker relative to the real object, the position of the virtual marker relative to the virtual model relative to the position of the first real marker relative to the first real object has a deviation. In particular, in this regard, in addition to the above-mentioned DE 10 2007 059 478 B4 Reference is made, the content of which is expressly the entire contents of the present disclosure.

• – Ermitteln einer Position eines zweiten realen Markers relativ zu dem ersten realen Marker; und
• – Ermitteln der Abweichung (der Position des virtuellen Markers relativ zu dem virtuellen Modell gegenüber der Position des ersten realen Markers relativ zu dem realen Objekt) auf Basis dieser ermittelten Position des zweiten realen Markers relativ zu dem ersten realen Marker, insbesondere derart, dass die Abweichung gleich der Position des zweiten relativ zu dem ersten realen Marker ist.

According to an embodiment of the present invention, the method comprises the steps of:

• Determining a position of a second real marker relative to the first real marker; and
• - Determining the deviation (the position of the virtual marker relative to the virtual model relative to the position of the first real marker relative to the real object) based on this determined position of the second real marker relative to the first real marker, in particular such that the deviation is equal to the position of the second relative to the first real marker.

As a result, in one embodiment on the one hand (continue) that already in the DE 10 2007 059 478 B4 achieved explained advantages, in particular the alignment of a virtual model to a real object improved, in particular simplified and / or accelerated and / or performed more precisely, in particular, since the alignment of the virtual model and marker relative to each other regularly easier and / or more precise possible as the alignment of the virtual model and (image of) the real object relative to each other.

In addition, in one embodiment, on the other hand, the (first) real marker, whose position is detected during operation, and on the basis of which the virtual model is aligned with the real object, can be placed better, especially at positions that are good in operation, in particular by a Detection device of the system are always detectable, and / or at positions where it can not or only with difficulty be aligned exactly to the real object, especially at positions without edges, corners, flat surfaces or the like.

In particular, the invention can be advantageously used for augmented reality, which is composed of an image of a real environment with the real object and virtual information superimposed on this image, which contain the virtual model. In one embodiment, this is a representation of the real object. For this purpose, in a further development, at least substantially, they can have the same outer contour as the real object. Similarly, a virtual model that is a representation of the real object, such as an undeformed reference or a desired set configuration, kinematic variables, in particular speeds and / or accelerations, and / or coordinate systems of the real object and / or acting on the real object forces or the like, in particular consist thereof. The present invention is not limited to augmented reality applications, but can be used to advantage particularly in path planning or collision avoidance in robots or other applications in which a virtual model is to be aligned with a real object. Thus, a preferred application is in the modeling, simulation, planning, optimization and / or regulation of a robot or an automated production plant.

In one embodiment, the first real marker is a marker formed separately from the real object and attached thereto or integrated in the object. In a further development, it can be permanently attached to the object, in particular permanently, ie permanently or temporarily, in particular detachably. For this purpose, the real marker in one embodiment, for example, printed on the real object, sprayed on, glued, screwed, locked, clamped or even be placed only under its own weight.

In one embodiment, the second real marker is temporarily aligned relative to the object relative to the first real marker so that the position of an imaginary coordinate system defined by the second marker is fixed relative to the real object, in particular to an object-fixed coordinate system is.

In one embodiment, a position of the first real marker disposed on the object and / or the second marker temporarily aligned relative to the object are detected in a world coordinate system. In this case, a reference system for the real object, for example a reference system of the scene to be augmented or of a manipulator to be simulated, in particular an environment-fixed coordinate system is referred to as the world coordinate system.

In one embodiment, the virtual marker whose position relative to the virtual model relative to the position of the first real marker relative to the real object has the determined deviation and which is insofar (indirectly) associated with the first real marker, at least substantially the same appearance , in particular the same contour, texture and / or color as the second real marker.

In one embodiment, in the scene to be augmented, the virtual model is shifted and / or rotated, for example by means of a mouse or other input device, such that the position of the virtual marker relative to the virtual model equals the position of the second real marker relative to the real object or, by contrast, has a known (further) deviation. Additionally or alternatively, in one embodiment, the virtual marker, again for example with the help of a mouse or other input device, so shifted and / or rotated that the position of the virtual marker relative to the virtual model, this preferably in a clearly visible reference position is equal to the position of the second real marker relative to the real object or, on the other hand, has a known (further) deviation.

Subsequently, the virtual marker with the virtual model attached to it can be automatically aligned with the first real marker taking into account the determined deviation and possibly the known further deviation, whereby the model is automatically aligned as desired with the object.

Instead of manual positioning of the virtual marker or model by means of a mouse or another input device, semi-automatic or automatic positioning can also be carried out. For this purpose, for example, edges of the virtual marker, the virtual model and / or the image of the second real marker and / or the real object can be extracted semiautomatically or fully automatically and subsequently, user-assisted or fully automatic, one or more of the extracted edges can be selected. For example, automatic edge extraction on both the virtual model and the virtual marker can extract the bounding edges. The user can then, for example by means of a mouse, select one or more edges and specify that the match selected edges of the virtual model and marker to precisely align the virtual marker relative to the model.

The positions of the first and / or second real markers can in one embodiment be detected by an optical device, which preferably comprises one or more cameras, in particular CCD cameras, and an image processing device in order to further process an image recorded by the camera (s), in particular, to represent virtual models, information and the like in the image. In particular, the position of a real marker, i. e. its position and / or orientation relative to a world coordinate system.

In order to facilitate this detection and to improve the detection quality, the first and / or second real marker is / are preferably designed so that its position can be well detected by a camera and an image processing device. In particular, the real marker for this purpose can have one or more geometric figures, for example points, rectangles or the like, which have a predetermined arrangement relative to one another and thus determine the position of a coordinate system of the real marker. The first and / or second real marker preferably have / have strongly contrasting sections, in particular sections in complementary colors, light / dark or in black / white. Such geometric figures or other codings that can be detected by the image processing can advantageously also identify the respective real marker. This is particularly advantageous if, in a preferred embodiment of the present invention, a first real marker is arranged on a plurality of, in particular all real objects of a scene to be augmented, in order to align a virtual model with these objects.

In one embodiment, the virtual marker and the virtual model are oriented relative to one another such that the position of the virtual marker relative to the virtual model is equal to or opposite the position of the second real marker relative to the real object has further deviation, for example, by a predetermined angle, about 90 °, is rotated. In particular for this purpose it is advantageous to design the virtual marker in such a way that it represents a representation of the second real marker suitable for the augmented reality system, i. e. This is similar in particular to its contour, texturing and the like in order to facilitate the assignment of the individual pairs of virtual and second real markers to the user. This also increases the intuitive applicability of the method.

If a marker edge and an object or model edge are aligned with each other, the rotational degree of freedom around this edge remains between the marker and the object or model, and the translational degree of freedom along this edge. Therefore, in one embodiment, the second real or virtual marker is aligned with the real object or virtual model so that an edge and a corner of the marker align with an edge and a corner of the object or model.

In one embodiment, the position of the virtual marker relative to the virtual model is equal to or opposite to the position of the second real marker relative to the real object, which in turn is different from the position of the first real marker relative to the real object. the known further deviation. As a result, the orientation of the virtual marker and / or model can be improved, in particular simplified and / or performed rapidly and / or with greater precision.

In the present case, a position is understood to mean, in particular, one, in particular one, two or three dimensional, in particular Cartesian, position and / or orientation, in particular between corresponding coordinate systems, in particular a transformation between corresponding coordinate systems. Accordingly, in this case, the symbol "o" designates, in particular without limitation of generality, in general a combination of transformations, for example a multiplication of Denavit-Hartenberg matrices, a multiplication of rotation matrices and addition of translatory displacements or the like.

In one embodiment, the virtual model and the virtual marker are arranged on a common virtual plane, in particular laid on, and / or aligned relative to one another on a virtual edge. Additionally or alternatively, in one embodiment, the second real marker and the real object are arranged on a common real plane, in particular laid on, and / or aligned relative to one another on a real edge, wherein in a development the virtual plane represents a representation of the real plane and / or or the virtual edge is a representation of the real edge.

This is based on the knowledge that even real objects that have no flat surfaces can be regularly placed on a real plane and so a defined orientation of the second real marker or a coordinate system defined by this relative to the object or an object-fixed coordinate system especially if the second real marker is also on the common real plane and is aligned at the (common) real edge.

In one embodiment, the virtual model and / or the virtual marker is virtually rotated about the virtual edge until it or it rests on the common virtual plane. Additionally or alternatively, in one embodiment, a virtual contact point of the virtual model and / or the virtual marker is provided with the common virtual plane, which is spaced from the virtual edge.

This allows a rotational degree of freedom to be resolved around the edge.

In one embodiment, the virtual model corresponding to the real object, i. H. based on the detected position of the first real marker, the position of the virtual marker relative to the virtual model and the determined deviation, and optionally the known further deviation aligned. In particular, if in one embodiment the position of the virtual marker relative to the virtual model is equal to the position of the second real marker relative to the real object or if it has the known further deviation and that position of the second real marker relative to the real object itself Alignment of second real marker and real object is known relative to each other, from the detected position of the first real marker taking into account the deviation determined, the position of the virtual marker and from this then, possibly taking into account the known further deviation, the position of the virtual Model determined and this are thus aligned with the real object.

According to one embodiment of the present invention, a system, in particular an augmented reality system, for aligning a virtual model with a real object, in particular hardware and / or software, in particular program technology, for implementing a method described herein and / or has :
Detecting means for detecting a position of the first real marker; and alignment means for aligning the virtual model and the virtual marker relative to one another, ie based on, the position of the first real marker relative to the real object, the position of the virtual marker relative to the virtual model relative to the position of the first real marker relative has a deviation to the real object;
wherein the alignment means comprises:
Means for determining the position of the second real marker relative to the first real marker; and
Means for determining the deviation on the basis of the determined position of the second real marker relative to the first real marker, in particular such that the deviation is equal to the position of the second relative to the first real marker.

In one embodiment, the system has:
Means for arranging, in particular placing, the virtual model and the virtual marker on a common virtual plane and / or for aligning the virtual model and the virtual marker on a virtual edge relative to one another; and or
Means for virtually rotating the virtual model and / or the virtual marker around the virtual edge until it rests on the common virtual plane and / or for specifying a virtual contact point of the virtual model and / or the virtual marker with the common one virtual plane spaced from the virtual edge; and or
Means for aligning the virtual model with the real object corresponding to, ie based on, the detected position of the first real marker, the position of the virtual marker relative to the virtual model and the determined deviation, and possibly the known further deviation.

In this case, aligning a virtual model to a real object is understood in particular to mean that a position of the virtual model or model-fixed coordinate system, in particular in an augmented scene, with a position of the real object or object-fixed coordinate system, in particular an image of the real object in FIG the augmented scene, or the virtual model is positioned or moved in accordance with a position or movement of the real object (detected by means of the first real marker). This can be done in one embodiment by appropriate transformation of the virtual model and / or image of the real object in the augmented scene.

A means in the sense of the present invention may be designed in terms of hardware and / or software, in particular a data or signal-connected, preferably digital, processing, in particular microprocessor unit (CPU) and / or a memory and / or bus system or multiple programs or program modules. The CPU may be configured to execute instructions implemented as a program stored in a memory system, to capture input signals from a data bus, and / or to output signals to a data bus. A storage system may comprise one or more, in particular different, storage media, in particular optical, magnetic, solid state and / or other non-volatile media. The program may be such that it uses the methods described herein so that the CPU can perform the steps of such methods and, in particular, can align a virtual model with a real object.

In one embodiment, one or more, in particular all, steps of the method are performed manually, semi-automatically or-preferably optionally-automatically, in particular by the system or its means.

Further advantages and features emerge from the subclaims and the exemplary embodiments. This shows, partially schematized:

1 : a scene to be augmented with a real object and its not yet aligned virtual model; and

2 A method according to an embodiment of the present invention.

1 shows a scene to be augmented with a real object 1 and its not yet aligned virtual model 2 as well as a camera 5 and an alignment means in the form of a suitably programmed computer 10 to align the virtual model 2 on the real object 1 in the augmented scene according to an embodiment of the present invention.

In a step S10, on the real object 1 on one of the camera 5 In operation well detectable position a first real marker 3 arranged.

It does not have, as in 1 shown edges 3.1 . 3.2 the first real marker 3 with edges 1.1 . 1.2 of the real object 1 aligned. Rather, that can be done in 1 only schematically represented real object 1 at the position of the first real marker 3 be formed arbitrarily, for example, free-form surfaces or the like.

Before, subsequently or in parallel become the real object 1 and a second real marker 3 ' on a common real level 6 lying on a real edge 1.3 of the object 1 respectively. 3.3 the second marker 3 ' oriented relative to each other so that the two edges 1.3 . 3.3 aligned with each other and corners 1.4 of the object or 3.4 of the second real marker 3 ' to match.

In a step S20, the camera 5 a position r of the first marker 3 and a position r 'of the second marker 3 ' (relative to the camera or a world coordinate system I) detected and by the computer 10 a position a of the second marker 3 ' relative to the first marker 3 determined.

Previously, subsequently or in parallel, in a display device of the computer 10 the virtual model 2 and a virtual marker 4 ' shown.

By a user are by means of an input device of the computer 10 For example, a mouse, a space ball or the like, the virtual model 2 and the virtual marker 4 ' relative to each other aligned such that the position b of the virtual marker 4 ' or its coordinate system V 'relative to the virtual model 2 or its model-fixed coordinate system M equal to the position b of the second real marker 3 ' or its coordinate system R 'relative to the real object 1 or its object-fixed coordinate system O, in turn, from the position v of the first real marker 3 or its coordinate system R relative to the real object 1 or its object-fixed coordinate system O different, but due to the orientation of the second real marker 3 ' and object 1 is known to each other. In a modification, not shown, the position of the virtual marker 4 ' (or its coordinate system V ') relative to the virtual model 2 (or its model-fixed coordinate system M) relative to the position of the second real marker 3 ' (or its coordinate system R ') relative to the real object 1 (or the object-fixed coordinate system O) also have a known further deviation, for example, be rotated by 90 °.

For (virtual) alignment, first an edge 2.3 of the virtual model 2 and the virtual marker 4 ' on a common virtual level 7 arranged and virtual model 2 and virtual markers 4 ' on the edge 2.3 of the virtual model 2 respectively. 4.3 the virtual marker 4 ' oriented relative to each other so that the two edges 2.3 . 4.3 Aligned and a corner 2.4 of the model and a corner 4.4 the virtual marker 4 ' to match.

To the remaining rotational degree of freedom between model 2 and virtual marker 4 ' precise and in accordance with the position b of the second real marker 3 ' relative to the object 1 then set the model 2 virtually around its edge 2.3 Turned until it's on the common virtual level 7 rests.

Alternatively, model 2 and virtual markers 4 ' already in the in 1 shown orientation on the common virtual level 7 lying on the edge and resting 2.3 of the virtual model 2 respectively. 4.3 the virtual marker 4 ' aligned relative to each other and this a virtual point of contact 2.5 of the virtual model 2 with the common virtual layer 7 specified by the user by means of the input device, in particular clicked in the representation are. Also by this can the rotational degree of freedom between model 2 and virtual marker 4 ' precise and in accordance with the position b of the second real marker 3 ' relative to the object 1 be determined.

Alternatively or additionally, the virtual marker could also be used in the same way 4 ' be twisted until he is in the virtual plane 7 is, or a virtual contact point of the virtual marker 4 ' with the virtual level 7 given, in particular, clicked.

The computer 10 determines a deviation a of the position b of the virtual marker in a step S30 4 ' or its coordinate system V 'relative to the virtual model 2 or its model-fixed coordinate system M with respect to the position v of the first real marker 3 or its coordinate system R relative to the real object 1 or its object-fixed coordinate system O on the basis of the determined position a of the second real marker 3 ' or its coordinate system R 'relative to the first real marker 3 or its coordinate system R: b = vo a.

Thus, the virtual model 2 and the virtual marker 4 ' relative to one another, ie based on the position v of the first real marker 3 relative to the object 1 aligned, where the position b of the virtual marker 4 ' relative to the model 2 relative to this position v has the determined deviation a.

In the augmented scene can now the virtual model 2 automatically on the real object 1 be aligned by, so to speak, the virtual marker 4 ' with attached virtual model 2 with the real marker 3 - taking into account the above deviation a - is brought to cover. For this purpose, the position m of the model 2 or its model-fixed coordinate system M relative to the coordinate system 1 in which the position r of the first real marker 3 by means of the camera 5 determined in the augmented scene: m = boaor with the (camera) detected position r, the determined deviation a between v and b and the due to the orientation 1.3 ↔ 3.3 . 1.4 ↔ 3.4 known position b of the second real marker 3 ' relative to the object 1 ,

Although exemplary embodiments have been explained in the foregoing description, it should be understood that a variety of modifications are possible. It should also be noted that the exemplary embodiments are merely examples that are not intended to limit the scope, applications and construction in any way. Rather, the expert is given by the preceding description, a guide for the implementation of at least one exemplary embodiment, with various changes, in particular with regard to the function and arrangement of the components described, can be made without departing from the scope, as it turns out according to the claims and these equivalent combinations of features.

LIST OF REFERENCE NUMBERS

1

real object

1.1-1.3

edge

1.4

corner

2

virtual model

2.1-2.3

edge

2.4

corner

2.5

contact point

3

first real marker

3 '

second real marker

3.1-3.3

edge

3.4

corner

4; 4 '

virtual marker

4.1-4.3

edge

4.4

corner

5

camera

6

real level

7

virtual level

10

computer

a

determined deviation = position second relative to the first real marker

b

Position of virtual / second real marker to model / object

I

World coordinate system

m

Position model

M

model-fixed coordinate system

O

object-fixed coordinate system

r; r '

Position first / second real marker

R; R '

(Marker) coordinate system

v

Position of first real marker to object

V; V '

(Marker) coordinate system

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• DE 102007059478 B4 [0002, 0004, 0006, 0008, 0010, 0012]

Claims (11)

Method for aligning a virtual model ( 2 ) on a real object ( 1 ), in particular in an augmented reality, whereby a first real marker ( 3 ), comprising the steps of: detecting (S20) a position (r) of the first real marker; and aligning (S20) the virtual model and a virtual marker ( 4 ' relative to each other according to the position of the first real marker relative to the real object; wherein the position (b) of the virtual marker relative to the virtual model has a deviation (a) from the position (v) of the first real marker relative to the real object; characterized by the steps of: determining (S20) a position (a) of a second real marker ( 3 ' ) relative to the first real marker; and determining (S30) the deviation based on the determined position of the second real marker relative to the first real marker. A method according to claim 1, characterized in that the position of the virtual marker relative to the virtual model is equal to the position (b) of the second real marker relative to the real object, which is different from the position of the first real marker relative to the real object , is or has a known further deviation. Method according to one of the preceding claims, characterized in that the virtual model and the virtual marker are located on a common virtual level ( 7 ) and / or on a virtual edge ( 2.3 . 4.3 ) and / or the second real marker and the real object on a common real plane ( 6 ) and / or on a real edge ( 1.3 . 3.3 ) are aligned relative to each other. Method according to the preceding claim, characterized in that the virtual model and / or the virtual marker is rotated virtually around the virtual edge until it is arranged on the common virtual plane, and / or a virtual contact point ( 2.5 ) of the virtual model and / or virtual marker having the common virtual plane spaced from the virtual edge. Method according to one of the preceding claims, characterized in that the virtual model is aligned on the real object according to the detected position of the first real marker, the position of the virtual marker relative to the virtual model and the deviation. System, in particular Augmented Reality System, for aligning a virtual model ( 2 ) on a real object ( 1 ) is arranged to carry out a method according to one of the preceding claims and / or comprises: detection means ( 5 ) for detecting a position (r) of the first real marker; and alignment means ( 10 ) for aligning the virtual model and the virtual marker ( 4 ' ) relative to each other according to the position of the first real marker ( 3 ) relative to the real object ( 1 wherein the position (b) of the virtual marker relative to the virtual model has a deviation (a) from the position (v) of the first real marker relative to the real object; wherein the alignment means comprises: means for determining the position (a) of the second real marker relative to the first real marker; and means for determining the deviation based on the determined position of the second real marker relative to the first real marker. System according to the preceding claim, characterized by means for arranging the virtual model and the virtual marker on a common virtual level ( 7 ) and / or aligning the virtual model and the virtual marker with a virtual edge ( 2.3 . 4.3 ) relative to each other. System according to the preceding claim, characterized by means for virtual rotation of the virtual model and / or the virtual marker around the virtual edge until it is arranged on the common virtual plane and / or for specifying a virtual contact point ( 2.5 ) of the virtual model and / or virtual marker with the common virtual plane that is spaced from the virtual edge. System according to one of the preceding claims, characterized by means for aligning the virtual model on the real object according to the detected position of the first real marker, the position of the virtual marker relative to the virtual model and the deviation. A computer program that performs steps of a method according to any one of the preceding claims when stored on a computer ( 10 ) expires. Computer program product with a program code stored on one of a computer ( 10 ) readable medium, for performing a method according to any one of the preceding claims.

Images