WO2017198331A1

WO2017198331A1 - Method and system for aligning a virtual model with a real object

Info

Publication number: WO2017198331A1
Application number: PCT/EP2017/000589
Authority: WO
Inventors: Yevgen Kogan
Original assignee: Kuka Roboter Gmbh
Priority date: 2016-05-18
Filing date: 2017-05-15
Publication date: 2017-11-23
Also published as: DE102016006232A1; DE212017000139U1

Abstract

A method according to the invention for aligning a virtual model (2) with a real object (1), in particular in augmented reality, wherein a first real marker (3) is arranged on the real object, comprises the steps of: - detecting (S20) a position (r) of the first real marker; - and - aligning (S20) the virtual model and a virtual marker (4') with another in a manner corresponding to the position of the first real marker relative to the real object; wherein the position (b) of the virtual marker relative to the virtual model has a difference (a) in comparison with the position (v) of the first real marker relative to the real object; and the steps of: - determining (S20) a position (a) of a second real marker (3') relative to the first real marker; and - determining (S30) the difference on the basis of the determined position of the second real marker relative to the first real marker.

Description

description

Method and system for aligning a virtual model with a real one

object

The present invention relates to a method and a system for aligning a virtual model on a real object and a computer program or

Computer program product for carrying out the method.

DE 10 2007 059 478 B4 discloses a method for aligning a virtual model, which is a representation of a real object, with this real object. Referring to FIG. 1, a position 5 of a real marker 3, which is arranged on a real object 1, is detected by a camera 5.

DE 10 2007 059 478 B4 proposes a virtual model 2 which has a

Representation of the real object 1, and a virtual marker 4 relative to each other according to the position of the marker 3 relative to the real object 1 to align.

As a result, the orientation of the virtual model 2 on the real object 1 is facilitated, in particular the determination of a position m of a model-fixed

Coordinate system M in a world coordinate system /, in which a position r of the real marker 3 is detected, such that this position m coincides with the position of an object-fixed coordinate system O, in particular by linking the transformations vor = m with the known transformation v between a marker-fixed Coordinate system R and the object-fixed coordinate system O. In this way, the real object 1 can be deposited in an augmented reality with the virtual model 2. In DE 10 2007 059 478 B4, the position v of the virtual marker is preferred 4 relative to the virtual model 2 is equal to the position v of the real marker 4 relative to the real object 1, as indicated by dashed lines in FIG. 1 for the virtual marker 4. In this case, however, it may happen that the real marker 3 in such a position relative to the real object 1, which is favorable for aligning the virtual model 2 and virtual marker 4 relative to each other, poorly detectable during operation of the camera 5 and / or only poorly alignable on the real object 1,

For example, if in a modification of Fig. 1, no intersecting edges 1 .1, 1.2 or the like.

An object of the present invention is, in particular, to improve the method, system or computer program (product) known from DE 10 2007 059 478 B4. This object is achieved by a method having the features of claim 1, a system having the features of claim 6 and a computer program (product) according to claim 10 and 11, respectively. The subclaims relate to advantageous

Training.

According to one embodiment of the present invention, a method for

Alignment of a virtual model on a real object, in particular in an augmented reality or augmented reality, wherein a first real marker is arranged on the real object, the steps:

Detecting a position of the first real marker; and

Aligning the virtual model and a virtual marker relative to each other, i. based on the position of the first real marker relative to the real object, wherein the position of the virtual marker relative to the virtual model has a deviation from the position of the first real marker relative to the real object. In particular, in this respect, reference is additionally made to the aforementioned DE 10 2007 059 478 B4, the content of which is expressly the subject of the present application

Revelation is made.

According to an embodiment of the present invention, the method comprises the steps of:

Determining a position of a second real marker relative to the first real marker; and

Determining the deviation (the position of the virtual marker relative to the

virtual model relative to the position of the first real marker relative to the real object) on the basis of this determined position of the second real marker relative to the first real marker, in particular such that the deviation is equal to the position of the second relative to the first real marker.

As a result, in one embodiment, on the one hand, the advantages already explained in DE 10 2007 059 478 B4 can be achieved, in particular the alignment of a virtual model on a real object can be improved, in particular simplified and / or accelerated and / or performed more precisely, in particular because the alignment of virtual model and marker relative to each other regularly easier and / or more precise possible than the alignment of the virtual model and (image of) the real object relative to each other.

In addition, in one embodiment, on the other hand, the (first) real marker, whose position is detected during operation, and on the basis of which the virtual model is aligned with the real object, can be placed better, especially at positions that are good in operation, in particular by a Detection device of the system are always detectable, and / or at positions where it can not or only with difficulty be aligned exactly to the real object, especially at positions without edges, corners, flat surfaces or the like.

In particular, the invention can be advantageously used for augmented reality, which is composed of an image of a real environment with the virtual object superimposed on the real image and containing the virtual model, which in one embodiment is a Representation of the real object For this purpose, in a further development, at least in the

Essentially, have the same outer contour as the real object.

Similarly, a virtual model that is a representation of the real

Object is, for example, an undeformed reference or a desired

Target configuration, kinematic variables, in particular speeds and / or accelerations, and / or coordinate systems of the real object and / or acting on the real object forces or the like include, in particular consist thereof. The present invention is not limited to augmented reality applications, but may be advantageous in particular in web planning or

Collision avoidance are used in robots or other applications in which a virtual model is to be aligned with a real object. That's one preferred application in the modeling, simulation, planning, optimization and / or regulation of a robot or an automated production plant.

In one embodiment, the first real marker is a marker formed separately from the real object and attached thereto or integrated in the object. In a further development, it can be permanently attached to the object, in particular permanently, ie permanently or temporarily, in particular detachably. For this purpose, the real marker in one embodiment, for example, printed on the real object, sprayed on, glued, screwed, locked, clamped or even be placed only under its own weight. In one embodiment, the second real marker is temporarily aligned relative to the object relative to the first real marker so that the position of an imaginary coordinate system defined by the second marker is fixed relative to the real object, in particular to an object-fixed coordinate system is. In one embodiment, a position of the first real, on the object

arranged marker and / or the second, temporarily aligned relative to the object marker in a world coordinate system detected. With

World coordinate system becomes a frame of reference for the real object,

For example, a reference system of the scene to be augmented or of a manipulator to be simulated, in particular an environment-fixed coordinate system.

In one embodiment, the virtual marker whose position relative to the

virtual model relative to the position of the first real marker relative to the real object has the determined deviation and the extent (indirectly) associated with the first real marker, at least substantially the same appearance, in particular the same contour, texture and / or color as the second real marker.

In one embodiment, in the scene to be augmented, the virtual model is shifted and / or rotated, for example by means of a mouse or other input device, such that the position of the virtual marker relative to the virtual model equals the position of the second real marker relative to the real one Object is or in contrast has a known (further) deviation.

Additionally or alternatively, in one embodiment, the virtual marker, again for example with the help of a mouse or other input device, so shifted and / or rotated that the position of the virtual marker relative to the virtual model, this preferably in a clearly visible reference position is equal to the position of the second real marker relative to the real object or, on the other hand, has a known (further) deviation.

Subsequently, the virtual marker with the virtual model attached to it, taking into account the determined deviation and possibly the

known further deviation automatically to the first real marker

aligning the model automatically to the object as desired.

Instead of manual positioning of the virtual marker or model by means of a mouse or another input device, semi-automatic or automatic positioning can also be carried out. For this purpose, for example, edges of the virtual marker, the virtual model and / or the image of the second real marker and / or the real object semiautomatically or fully automatically extracted and then, user-based or fully automatic one or more of

extracted edges are selected. For example, automatic edge extraction on both the virtual model and the virtual marker can extract the bounding edges. The user can then, for example by means of a mouse, select one or more edges and specify that the selected edges of the virtual model and marker match and thus the virtual one

Align markers precisely relative to the model. The positions of the first and / or second real marker can be in one

Embodiment are detected by an optical device, preferably one or more cameras, in particular CCD cameras, and a

Image processing device comprises, to one of the or the cameras

to process the captured image, in particular virtual models,

To display information and the like in the image. By the

Image processing device can in particular the position of a real marker, ie its position and / or orientation relative to a world coordinate system.

In order to facilitate this detection and to improve the detection quality, the first and / or second real marker is / are preferably designed so that its position can be well detected by a camera and an image processing device. In particular, the real marker for this purpose may have one or more geometric figures, for example points, rectangles or the like, which have a predetermined arrangement relative to one another and thus the position of a

Set the coordinate system of the real marker. The first and / or second real marker preferably have / have strongly contrasting sections, in particular

Sections in complementary colors, light / dark or in black and white. Such geometric figures or others detectable by the image processing

Codings can advantageously also identify the respective real marker. This is particularly advantageous when in a preferred embodiment of

present invention is arranged on a plurality of, in particular all real objects of a scene to be augmented depending on a first real marker to align to each of these objects a virtual model.

In one embodiment, the virtual marker and the virtual model are oriented relative to one another such that the position of the virtual marker relative to the virtual model is equal to or opposite the position of the second real marker relative to the real object has further deviation, for example, by a predetermined angle, about 90 °, is rotated.

In particular, for this purpose it is advantageous to design the virtual marker in such a way that it represents a representation of the second real marker which is suitable for the augmented reality system, i.e. This is similar in particular to its contour, texturing and the like in order to facilitate the assignment of the individual pairs of virtual and second real markers to the user. This also increases the intuitive applicability of the method.

If a marker edge and an object or model edge are aligned with each other, the rotational degree of freedom around this edge remains between the marker and the object or model, and the translational degree of freedom along this edge. Therefore, in one embodiment, the second real or the virtual marker is so on the real object or virtual model so that one edge and one corner of the marker align with an edge and a corner of the object or model.

In one embodiment, the position of the virtual marker relative to the virtual model is equal to or opposite to the position of the second real marker relative to the real object, which in turn is different from the position of the first real marker relative to the real object. the known further deviation. As a result, the orientation of the virtual marker and / or model can be improved, in particular simplified and / or performed rapidly and / or with greater precision. In the present case, a position is understood to mean, in particular, one, in particular one, two or three dimensional, in particular Cartesian, position and / or orientation, in particular between corresponding coordinate systems, in particular a transformation between corresponding coordinate systems. Accordingly, in this case, the symbol "o" designates, in particular without limitation of generality, in general a combination of transformations, for example a multiplication of Denavit-Hartenberg matrices, a multiplication of

Rotation matrices and addition of translational displacements or

like.

In one embodiment, the virtual model and the virtual marker are arranged on a common virtual plane, in particular laid on, and / or aligned relative to one another on a virtual edge. Additionally or alternatively, in one embodiment, the second real marker and the real object are arranged on a common real plane, in particular laid on, and / or aligned relative to one another on a real edge, wherein in a development the virtual plane represents a representation of the real plane and / or or the virtual edge one

Representation of the real edge is.

This is based on the knowledge that even real objects that have no flat surfaces, can be regularly placed on a real plane and so a defined orientation of the second real marker or a coordinate system defined by this relative to the object or an object-fixed

Allow coordinate system, in particular, if the second real marker too lies on the common real level and is aligned to the (common) real edge.

In one embodiment, the virtual model and / or the virtual marker is virtually rotated about the virtual edge until it or it rests on the common virtual plane. Additionally or alternatively, in one embodiment, a virtual contact point of the virtual model and / or the virtual marker is provided with the common virtual plane, which is spaced from the virtual edge.

This allows a rotational degree of freedom to be resolved around the edge. In one embodiment, the virtual model corresponding to the real object, i. based on the detected position of the first real marker, the position of the virtual marker relative to the virtual model and the determined deviation, and optionally the known further deviation aligned.

In particular, if in one embodiment the position of the virtual marker relative to the virtual model is equal to the position of the second real marker relative to the real object or if it has the known further deviation and that position of the second real marker relative to the real object itself Alignment of second real marker and real object is known relative to each other, from the detected position of the first real marker taking into account the deviation determined, the position of the virtual marker and from this then, possibly taking into account the known further deviation, the position of the virtual Model determined and this are thus aligned with the real object.

According to one embodiment of the present invention, a system, in particular an augmented reality system, for aligning a virtual model with a real object, in particular hardware and / or software, in particular program technology, for implementing a method described herein and / or has :

Detecting means for detecting a position of the first real marker; and

Alignment means for aligning the virtual model and the virtual marker relative to one another, ie, based on the position of the first real one Marker relative to the real object, the position of the virtual marker relative to the virtual model having a deviation from the position of the first real marker relative to the real object;

wherein the alignment means comprises:

Means for determining the position of the second real marker relative to the first real marker; and

Means for determining the deviation on the basis of the determined position of the second real marker relative to the first real marker, in particular such that the deviation is equal to the position of the second relative to the first real marker. In one embodiment, the system has:

Means for arranging, in particular placing, the virtual model and the virtual marker on a common virtual plane and / or for aligning the virtual model and the virtual marker on a virtual edge relative to one another; and or

Means for virtually rotating the virtual model and / or the virtual marker around the virtual edge until it rests on the common virtual plane and / or for specifying a virtual contact point of the virtual model and / or the virtual marker with the common one virtual plane spaced from the virtual edge; and or

Means for aligning the virtual model with the real object, i. based on the detected position of the first real marker, the position of the virtual marker relative to the virtual model and the determined deviation, and optionally the known further deviation.

In this case, a realignment of a virtual model on a real object is understood in particular to mean that a position of the virtual model or model-fixed coordinate system, in particular in an augmented scene, with a position of the real object or object-fixed coordinate system,

in particular of an image of the real object in the augmented scene, or the virtual model is positioned according to a position or movement of the real object (detected by means of the first real marker) or

(with) is moved. This can be done in one embodiment by appropriate transformation of the virtual model and / or image of the real object in the augmented scene. A means in the sense of the present invention may be designed in terms of hardware and / or software, in particular a data or signal-connected, preferably digital, processing, in particular microprocessor unit (CPU) and / or a memory and / or bus system or multiple programs or program modules. The CPU may be configured to implement instructions implemented as a program stored in a memory system.

to process, capture input signals from a data bus and / or

Output signals to a data bus. A storage system may comprise one or more, in particular different, storage media, in particular optical, magnetic, solid state and / or other non-volatile media. The program may be such that it is capable of embodying or executing the methods described herein so that the CPU may perform the steps of such methods and, in particular, may align a virtual model with a real object. In one embodiment, one or more, in particular all, steps of

Method manually, semi-automatically or - preferably optionally - carried out automatically, in particular by the system or its means.

Further advantages and features emerge from the subclaims and the exemplary embodiments. 1 shows a scene to be augmented with a real object and its not yet aligned virtual model; and

2 shows a method according to an embodiment of the present invention.

Fig. 1 shows a scene to be augmented with a real object 1 and its not yet aligned virtual model 2 and a camera 5 and a

Alignment means in the form of a suitably programmed computer 10 for aligning the virtual model 2 on the real object 1 in the augmented scene according to an embodiment of the present invention.

In a step S10, a first real marker 3 is arranged on the real object 1 at a position which can be easily detected by the camera 5 during operation. In this case, as shown in FIG. 1, edges 3.1, 3.2 of the first real marker 3 do not have to be aligned with edges 1.1, 1.2 of the real object 1. Rather, the real object 1 shown only schematically in FIG. 1 can be arbitrarily shaped at the position of the first real marker 3, for example having free-form surfaces or the like. Previously, subsequently or in parallel, the real object 1 and a second real marker 3 'are arranged resting on a common real plane 6 and aligned relative to one another at a real edge 1.3 of the object 1 or 3.3 of the second marker 3', so that the two Edges 1.3, 3.3 are aligned with each other and corners 1 .4 of the object or 3.4 of the second real marker 3 'match. In a step S20, a position r of the first marker 3 and a position r 'of the second marker 3' (relative to the camera or a second marker 3 ') are determined by the camera 5

World coordinate system /) detected and determined by the computer 0 a position a of the second marker 3 'relative to the first marker 3.

Previously, subsequently or in parallel, the virtual model 2 and a virtual marker 4 'are displayed in a display device of the computer 10.

By means of an input device of the computer 10, for example a mouse, a spaceball or the like, the virtual model 2 and the virtual marker 4 'are aligned relative to one another such that the position b of the virtual marker 4' or its coordinate system V is relative or its model-fixed coordinate system M is equal to the position b of the second real marker 3 'or its coordinate system R' relative to the real object 1 or its object-fixed coordinate system O, in turn, from the position v of the first real Markers 3 and its coordinate system R relative to the real object 1 or its object-fixed coordinate system O different, but due to the alignment of the second real marker 3 'and object 1 is known to each other. In a modification not shown, the position of the virtual marker 4 '(or its coordinate system V) relative to the virtual model 2 (or its model-fixed coordinate system) compared to the position of the second real marker 3' (or its coordinate system R ¹ ) relative to the real object 1 (or the object-fixed coordinate system O) also have a known further deviation, for example, be rotated by 90 °. For (virtual) alignment, first an edge 2.3 of the virtual model 2 and the virtual marker 4 'are arranged on a common virtual plane 7 and virtual model 2 and virtual marker 4' are arranged on the edge 2.3 of the virtual model 2 or 4.3 of the virtual marker 4 'are aligned relative to each other, so that the two edges 2.3, 4.3 are aligned with each other and a corner 2.4 of the model and a corner 4.4 of the virtual marker 4' match.

To set the remaining rotational degree of freedom between model 2 and virtual marker 4 'precisely and in accordance with the position b of the second real marker 3' relative to the object 1, then the model 2 is virtually rotated about its edge 2.3 until it is on the common virtual Level 7 is up.

Alternatively, model 2 and virtual marker 4 'can already be arranged lying on the common virtual plane 7 in the orientation shown in FIG. 1 and aligned relative to each other at the edge 2.3 of the virtual model 2 or 4.3 of the virtual marker 4' and for this purpose a virtual one Contact point 2.5 of the virtual model 2 with the common virtual level 7 specified by the user by means of the input device, in particular clicked in the presentation, be. As a result, the rotational degree of freedom between model 2 and virtual marker 4 'can be determined precisely and in accordance with the position b of the second real marker 3' relative to the object 1. Alternatively or additionally, the virtual marker could also be used in the same way

4 'until it lies in the virtual level 7, or a virtual one

Contact point of the virtual marker 4 'with the virtual level 7 predetermined, in particular clicked, be.

The computer 10 determines in a step S30 a deviation a of the position b of the virtual marker 4 'or its coordinate system V relative to the virtual model 2 or its model-fixed coordinate system M with respect to the position v of the first real marker 3 or its coordinate system R relative to the real object 1 or its object-fixed coordinate system O on the basis of the determined position a of the second real marker 3 'or its coordinate system R' relative to the first real marker 3 or its coordinate system R. b = vo a. Thus, the virtual model 2 and the virtual marker 4 'are aligned relative to each other, ie, based on the position v of the first real marker 3 relative to the object 1, with the position b of the virtual marker 4' relative to the model 2 opposite to this Position v has the determined deviation a. In the augmented scene, the virtual model 2 can now be automatically aligned with the real object 1, so to speak, by placing the virtual marker 4 'with the virtual model 2 attached thereto with the real marker 3 - under

Consideration of above deviation a - is brought to coincidence. For this purpose, the position m of the model 2 or its model-fixed coordinate system M relative to the coordinate system /, in which the position r of the first real marker 3 is detected by the camera 5, determined in the augmented scene: m = boaor with the (camera) detected position r, the determined deviation a between v and b and due to the orientation 1 .3 <^ 3.3, 1 .4 <- »3.4 known position b of the second real marker 3 'relative to the object. 1 Although exemplary embodiments have been explained in the foregoing description, it should be understood that a variety of modifications are possible. It should also be noted that it is the exemplary

The explanations are merely examples of the scope of protection

Applications and the structure should in no way limit. Rather, the expert is by the preceding description a guide for the

Implementation of at least one exemplary embodiment given, wherein various changes, in particular with regard to the function and arrangement of the described components, can be made without departing from the scope, as it is apparent from the claims and these equivalent

Feature combinations results. LIST OF REFERENCE NUMBERS

1 real object

1 .1 - 1.3 edge

1.4 corner

2 virtual model

2.1 - 2.3 edge

2.4 Corner

2.5 contact point

3 first real marker

3 'second real marker

3.1 - 3.3 edge

3.4 Corner

4; 4 'virtual marker

4.1 - 4.3 edge

4.4 Corner

5 camera

6 real level

7 virtual level

10 Computer a detected deviation = Position second relative to the first real marker b Position of virtual / second real marker to model / object

/ World coordinate system

m position model

M model-fixed coordinate system

O object-fixed coordinate system

r; r 'position first / second real marker

R, R '(marker) coordinate system

v Position of first real marker to object

V, V (marker) coordinate system

Claims

Patent claims

Method for aligning a virtual model

(2) on a real object (1), in particular in an augmented reality (“augmented reality”), a first real marker (3) being arranged on the real object, with the steps:

Detecting (S20) a position (r) of the first real marker;

and

Aligning (S20) the virtual model and a virtual marker (4') relative to each other according to the position of the first real marker relative to the real object;

wherein the position (b) of the virtual marker relative to the virtual model has a deviation (a) compared to the position (v) of the first real marker relative to the real object;

characterized by the steps:

Determining (S20) a position (a) of a second real marker (3') relative to the first real marker; and

Determining (S30) the deviation based on the determined position of the second real marker relative to the first real marker.

Method according to claim 1, characterized in that the position of the virtual marker relative to the virtual model is equal to the position (b) of the second real marker relative to the real object, which is different from the position of the first real marker relative to the real object , or has a known further deviation.

Method according to one of the preceding claims, characterized in that the virtual model and the virtual marker are arranged on a common virtual plane (7) and/or on a virtual edge (2.

3,

4.3) are aligned relative to one another and/or the second real marker and the real object are arranged on a common real plane (6) and/or are aligned relative to one another on a real edge (1.3, 3.3).

Method according to the preceding claim, characterized in that the virtual model and/or the virtual marker is virtually rotated around the virtual edge until it rests on the common virtual plane is arranged, and / or a virtual contact point (2.5) of the virtual model and / or the virtual marker with the common virtual plane

is specified, which is spaced from the virtual edge.

5. Method according to one of the preceding claims, characterized in that the virtual model is attached to the real object in accordance with the detected

Position of the first real marker, the position of the virtual marker relative to the virtual model and the deviation is aligned.

6. System, in particular augmented reality system, for aligning a

virtual model (2) on a real object (1) to carry out a

Method according to one of the preceding claims is set up and/or has:

Detection means (5) for detecting a position (r) of the first real marker; and

Alignment means (10) for aligning the virtual model and the virtual marker (4') relative to each other according to the position of the first real one

Marker (3) relative to the real object (1), the position (b) of the virtual marker relative to the virtual model having a deviation (a) compared to the position (v) of the first real marker relative to the real object;

wherein the alignment means comprises:

means for determining the position (a) of the second real marker relative to the first real marker; and

Means for determining the deviation based on the determined position of the second real marker relative to the first real marker.

7. System according to the preceding claim, characterized by means for arranging the virtual model and the virtual marker on one

common virtual plane (7) and/or for aligning the virtual model and the virtual marker on a virtual edge (2.3, 4.3) relative to one another.

8. System according to the preceding claim, characterized by means for virtually rotating the virtual model and / or the virtual marker around the virtual edge until it is arranged resting on the common virtual plane, and / or for specifying a virtual contact point (2.5) of virtual model and/or the virtual marker with the common virtual plane spaced from the virtual edge.

9. System according to one of the preceding claims, characterized by means for aligning the virtual model with the real object according to the detected position of the first real marker, the position of the virtual marker relative to the virtual model and the deviation.

10. Computer program that contains steps of a method according to one of the

preceding claims when it runs on a computer (10).

1 1. Computer program product with a program code that is stored on a medium readable by a computer (10) for carrying out a

Method according to one of the preceding claims.