DE102007053008A1 - Target body and method for determining its spatial position - Google Patents

Abstract

A class of target bodies is described whose surface reflects or emits only two different types of light and is designed so that the boundaries between the two types of light, viewed from any spatial direction, appear as straight lines. Furthermore, a method is described which has knowledge of the light properties and the shape of the target body as well as the parameters of the camera used. The method processes the images taken by the camera and determines therefrom the relative spatial position of the target body relative to the imaging camera. A particularly suitable target body and the method transmitted to a computer are used to continuously determine the spatial position of a person's hand. Target body and method are further developed into a 6-degree of freedom mouse, that is to an input device for all six spatial degrees of freedom at the interface between human and computer.

Description

object

The The invention relates to a method and a device for determination and temporal tracking of the spatial location of a suitable designed target body by means of a single camera as well as his training to a computer mouse with six degrees of freedom.

Of the Term "spatial position" is within the scope of this patent for the six coordinate values - three translatory ones and three rotatory degrees of freedom - that are unique Description of the spatial position of a solid with respect to a given coordinate system.

field of use

3D programs are computer programs that are more spacious with models Things work. In engineering become spatial models in CAD systems, in surveying software or in programming systems used for processing machines or robots. Increasingly 3D programs are spreading in application areas such as chemistry or Medical technology, entertainment industry - examples: computer games and Film Production, Garment Industry, Operating System Surfaces, Internet services and much more. In contrast, there is the manual input interface of the computer for graphical and geometric content is still almost invariably two-dimensional Pointing devices such as a computer mouse, tablet, joystick or game console.

In In this environment, 6-degree-of-freedom mice could become fast Find applications, these should meet the requirements in terms of cost-effectiveness, accuracy, computing time requirements, meet universal usability and ease of use can.

State of the art

in the Commercially available quasi-6-degree-of-freedom mice are based on the position measurement of a handle knob relative to a Base. Because the relative freedom of movement of the knob is mechanical is strictly limited, the measured values of the stored driver software interpreted as translational and rotational velocities. This gives the computer user a desired spatial Location similar to a pilot moving in the moving frame of the aircraft and a fixed landing point controls. In contrast, the principle of the here computer user on solid ground, Holds his plane in his hand and leads this along a spatial trajectory to the finish.

to Determination and temporal tracking of the position in the space of objects By means of image processing were different arrangements and algorithms proposed, however, none of them has been considered spatial Computer mouse for the workstation computer can enforce.

The detection of spatial objects in two-dimensional images has been realized for the case of bodies that are characterized by straight edges more than twenty years ago [Lowe]. Since then, the process has been further developed into a pattern recognition process and patented [ US Pat. No. 6,711,293 B1 ]. Two-camera installations enable the real-time, spatial tracking of objects previously made known to the process using reference images [Skrypnyk, Lowe].

Procedures that rely on multiple cameras, see for example [ DE 10 2005 011 432 A1 ], always require careful calibration of camera positions. In addition to the associated reduced mobility and changeability of the workplace, these methods are in principle more computationally intensive than single-camera systems. Higher computation times, however, reduce the computation time available for the 3D program and thus its comfort or even its use for the computer user.

Spherical reflectors, which are often used in virtual reality systems, have the disadvantage in single-camera systems that the balls can obscure each other and then no exact position determination is possible, cf. WO 2004/114112 A1 ].

Often Color segmentation is used to create a target body in the camera image. For use as a 6D mouse This is disadvantageous in that disturbances from the light properties of the image background, shadow and streaks light arise can.

In the patent US6526166 ], the spatial position of a cube, whose six sides are covered with three different colors, is determined by means of a single camera. Three colors are required so that two different colors collide at each cube edge. At oblique viewing angles, three edges of the cube are visible. These form as color contrast along straight lines on the camera image. The orientation of the cube can be determined by means of the three obtained lines. If at least one line length is known, the position of the cube relative to the camera can also be calculated with known camera optics. If the camera captures only two or even one side of the cube, the position of the cube in space can be determined but can not be determined for sure.

Techniques from the measurement technique, which manage with only one camera are designed only for small deflections of the known target body of its desired position, see [ WO 2005/075936 A1 ]. In this method, points of the target body are identified in the image and used to calculate a spatial position.

On Marker-based methods circumvent the disadvantage of color segmentation by setting a background color and a marker color, in the simplest Case white and black, which together on a mostly flat Surface to be applied.

in the Project ARTools [MagicMouse] used square markers their spatial position by means of a single camera image can be determined. Through all-sided occupancy of a cube With these markers [Augmented Chemistry], the spatial Position of the cube from all directions without gaps be determined.

In the patents US 2005 201613 ] and [ WO 2007/04431 A2 ] small circular markers are used, which were applied to cubic or rod-shaped bodies.

D. Bradley and G. Roth [Sphere] also use small circular ones Markers, but they distribute them on a blue ball. By color segmentation the outer diameter of the ball is determined and the pattern the markers distributed thereon compared with a stored model. Development, like the present invention, is aimed at the field of application a spatial computer mouse for 3D programs from.

task

The Object of the present invention is the spatial Location of a suitably designed target body by means of from a single camera taken pictures and by means of a appropriate method of image processing and time to pursue.

Presentation of the invention

The The invention includes a method and an apparatus for determination the spatial position of a target body. Especially The invention includes a method and an apparatus for gapless, quasi-continuous and stable determination the location of the target body.

Even though the invention is not bound to fixed work spaces, it is particularly advantageous to one to a receiving unit - for example to use a camera - firm working space.

According to the invention presented a method that is particularly robust and in particular can work with difficult picture conditions. Especially is it possible, even with a visually difficult image background or at an unfavorable illumination a to achieve precise determination of the target body.

target body and methods are intended to enter into a device that spatial Motion of a human's hand captured and transmitted to a computer for processing.

solution elements

• a. ein von einer Kamera erzeugtes Bild geladen wird,
• b. eine Grauwertbild erzeugt wird, das die Lichtarten des Zielkörpers dem Minimal- bzw. Maximalwert zuweist,
• c. ein Kantendetektor zur Bildung verketteter Kantenpunkte herangezogen wird,
• d. gerade Linienstücke aus den Kantenpunktketten herausgeschnitten werden,
• e. die geraden Linienstücke zu Polygonen zusammengesetzt werden,
• f. eine Bewertung der Polygone erfolgt,
• g. die zu den Polygonen passenden Projektionen der Zielkörperflächen bestimmt werden, die jeweils einer räumlichen Lage des Zielkörpers entsprechen,
• h. eine Bewertung der räumlichen Lagen des Zielkörpers erfolgt,
• i. ein Best-Fit des akzeptierten Zielkörpers erfolgt und
• j. die ermittelte räumliche Lage übermittelt wird.

According to the invention, it is particularly expedient to use a device referred to below as a target body, the surface of which reflects or emits at least two types of light distinguishable by a camera, the boundaries between the surface portions of the two types of light being straight stretches, and methods for determining the spatial dimensions Location of the target relative to a camera by processing the images taken by this camera characterized in that

• a. an image generated by a camera is loaded,
• b. a gray value image is generated, which assigns the light types of the target body to the minimum or maximum value,
• c. an edge detector is used to form chained edge points,
• d. straight line pieces are cut out of the edge point chains,
• e. the straight line pieces are put together to polygons,
• f. an evaluation of the polygons takes place,
• G. the projections of the target body surfaces that match the polygons are determined, each of which corresponds to a spatial position of the target body,
• H. an evaluation of the spatial positions of the target body takes place,
• i. a best fit of the accepted target body takes place and
• j. the determined spatial position is transmitted.

In the method according to the invention, the object is achieved by selecting a target body, the surface of which emits or reflect only two types of light, the boundaries between surface portions of two different types of light along straight lines, where further an image evaluation the images of the target body taken by a camera after possible Projections of the surface portions and their Scans boundary lines, calculates the relative position of the target body to the camera and makes this location available to any application programs.

In Advantageous embodiment of the invention comprises the target body suitable light sources, in particular one in the target body built-in electric light source.

In Furthermore advantageous embodiment of the invention, the target body added to a conventional computer mouse, that the computer user this target body advantageously without significant change of his hand position on the housing Insert the computer mouse into the field of view of the camera and can move freely in it.

In Furthermore, advantageous embodiment of the invention is the order the target body complemented computer mouse by one Switch extended, when pressed the evaluation the camera images is switched on or interrupted.

The term "camera images" as well as the term "camera" in a general meaning to understand. The terms and other terms useful for understanding the invention are explained below.
The term "camera" stands for any image-giving device that can create two-dimensional projections of spatial objects according to well-known laws and forward them in digital form to a computer. A computer mouse based on image processing requires in particular those cameras whose image refresh rate is sufficiently high to make movements on the screen appear continuous.

The translated into three-dimensional space with its six degrees of freedom Principle of the computer mouse should be synonymous in the context of this patent with the terms "6-degrees of freedom mouse" or "spatial degrees Computer mouse "or shortened" 6D mouse ".

A Development of the device according to the invention and the method according to the invention characterized by the fact that the target body of the hand of a Person is led and the procedure thus the six spatial Degrees of freedom of the hand position determined.

A Development of the device according to the invention and the method according to the invention characterized by the fact that the determination of the spatial location The target body quasi-continuously takes place by each individual image captured by the camera processed by the method becomes.

A Development of the device according to the invention and the method according to the invention characterized by the fact that the types of light as color impressions be realized black and white.

A Development of the device according to the invention and the method according to the invention characterized by the fact that the intensity of one or both Types of light by a mounted in the target body light source preferably increased with electrical power supply and the target body made partially transparent becomes.

A Development of the device according to the invention and the method according to the invention characterized by the fact that the boundary lines between the types of light and the edges of the target body coincide.

A Development of the device according to the invention and the method according to the invention characterized by the fact that the target body in the form of a Kuboktaeders whose squares are of a kind of light and whose triangles are the be assigned to another type of light.

The The invention further includes a "6-degree mouse" Device for entering six degrees of freedom into a computer, which on the target body and method according to claim 1 and the determined spatial positions arbitrary Programs provides.

A Another embodiment of the invention includes a 6-degree freedom mouse, whose operation is switched on by means of at least one switch, can be interrupted or turned off.

Especially it is advantageous that the 6-degree of freedom mouse with a conventional, two-dimensional Computer mouse is connected and with the help of the switch optionally as a conventional 2D mouse or as spatial Computer mouse can be operated.

images

1 : Use of an embodiment of the device according to the invention at the man-machine interface of the computer workstation.

2 Embodiment of a target body according to the invention in the form of a Kuboktaeders, which is composed of black and transparent surface portions and provided with an inner light source.

3 : Alternative embodiments of the target body.

4 : Hardware and software architecture of the method according to the invention.

5 : Flow chart of the method according to the invention.

6 Cutting out straight lines from point chains and composing straight lines into triangular and quadrilateral polygons.

7 : Determining two positions in the space of a regular triangle for a given projection

8th Embodiment of a conventional computer mouse extended to a 6D mouse

design

1 shows an embodiment of the device according to the invention at the computer workstation. The computer workstation comprises at least one central unit 101 , a screen 102 , a keyboard 103 , a computer mouse 104 , and in particular a camera connected to the computer 105 and also in the hands of the computer user 107 guided target body 106 , The camera is positioned so that the target body is visible to the camera within as much of the motion space of the hand of a desk-top computer user.

2a shows the preferred embodiment of a target body according to the invention in the form of a Kuboktaeders 201 , The surface of the target body was chosen so that white light is either largely colorless reflected or colorless, whereby the color impressions white 202 and black 203 caused in the camera image.

2 B shows the preferred use of a light source 212 with electrical energy supply 213 inside the target body 211 which serves to enhance the contrast between the light types of the target body on the one hand and between the surface of the target body and the remaining content of the camera image on the other hand. Other light arrangements for improving the image quality according to the method of the invention could consist of an annular light source concentric with the camera on the one hand and highly reflective or highly absorbent surfaces of the target body on the other hand, in which case infrared light could be used in particular.

In the embodiment, the squares 203 of the cuboctahedron black and the triangles 202 executed white. However, regardless of this choice, the scope of the present invention extends to all combinations of illuminants that are distinguishable to cameras.

Advantageously, a convex shape was chosen for the target body, so that the surface elements, although distorted in perspective distorted on the camera image, but not affect each other in the image and disappear only when they are facing away from the viewing direction. Furthermore, in the cuboctahedron, as required, all boundaries between black and white areas occur along straight edges 204 ,

The present invention includes all bodies that are distinguished by having even boundaries between surface portions of two different types of light. 3 shows some simple examples of such target bodies.

3a shows a black and white occupied octahedron 310 , Like the cuboctahedron 330 he owns the property that in every corner 311 exactly four edges 312 meet each other. Thereby, any pair of two adjacent surfaces can be occupied by the two opposite types of light, so that the common edge 312 creates a borderline between the types of light.

3b puts a cube 320 There are always three edges on the cube 322 in a corner 321 collide with each other, for example, the front 323 and back with a pattern of four triangles 324 be occupied, so that all cube edges form boundary lines between the light types.

3c contains the cuboctahedron 330 , the opposite of the octahedron 310 has the advantage that in the minimum case still four surfaces are visible and thus also the four boundary lines lying between these four surfaces are visible. These boundary lines are always displayed in high contrast, as they are in contrast to the edges on the border 331 of the projected cuboctahedron do not contrast with the a priori unknown image background.

3d gives a rhombic cuboctahedron 340 and 3e an icosidodecahedron 350 Both have the disadvantage of shorter edges compared to the cuboctahedron 341 . 351 in proportion to their external dimensions. This makes it difficult to find straight lines in the camera image when the target body is far away from the camera.

4 schematically illustrates the arrangement of the inventive method within the hardware and software architecture of a computer with man-machine interface. The user of the computer 401 leads with his hand 402 the target body 403 in the viewing window 404 a camera 405 , The pictures taken by the camera of the target Body are in the form of digital data via data connection 406 to a suitable hardware component 407 forwarded. The operating system of the computer 408 is notified of the input of an image and provides mechanisms that allow other programs to directly or indirectly access the camera images in data form.

The inventive method - hereinafter "algorithm" 409 - uses these mechanisms to load the received image into an internal memory. The algorithm is realized as a program that works together with other programs by the operating system 408 on the computer hardware 407 can be executed. The algorithm, in turn, provides its results directly or indirectly to other programs.

An application program differs from other programs in that it is perceived and used consciously and in its interest by the computer user, by using this z. B. helps solve tasks or serves for entertainment. An application program 410 can affect the results of the algorithm 409 access and process these in turn. It could be z. B. use the determined spatial position of the target body to position a virtual object in space and directly or indirectly on the screen 411 graphically or to make a machine, z. As an industrial robot, or otherwise create a sensory impression that can be perceived by the computer user in response to the spatial movement of his hand.

5 shows the procedure of the inventive method for determining the relative spatial position of the target body with respect to a camera.

aim On the one hand, it is to determine whether the camera image is in the process a projection of the target body is located and on the other hand, if that is the case, the spatial location of the Target body with the greatest possible To determine accuracy from this projection. These goals will be according to the invention by repeated, alternating Compiling candidates with subsequent evaluation achieved, whereby the nature of the considered candidates changes gradually.

Once a new camera picture 500 is available, the inventive method engages in the step 501 on the digitized camera image and converts it into step 502 into a grayscale image such that the first of the two types of light of the target body is converted to a gray value near the minimum of the gray level scale and the second type of light is converted to a gray level near its maximum.

In the subsequent step 503 The algorithm performs a method known to those skilled in the art of image processing as the standard method of edge detection, which includes the following sub-steps: image noise smoothing, gradient height and direction calculation in the gray image, non-maximum deletion, edge point concatenation, and edge chain evaluation by thresholds. The method provides the progression of the boundary lines projected into the camera image between the light types of the target body and possibly further light edges which were not caused by the border lines of the target body.

In step 504 The algorithm cuts out all the straight sections from the acquired edge point chains, which are considered projections of linear boundary lines, and converts them into line segments.

In step 505 From the line pieces a list of candidates is set up, hereafter referred to as "polygons", which are considered projections of triangles or squares in the camera image in question.

step 506 evaluates the polygons, considering the area enclosed by the polygon, for their suitability as a candidate for the projection of a triangle or quadrangle of the target body.

In step 507 For each sufficiently promising polygon candidate, all spatial positions of the target body, in the exemplary embodiment of the cuboctahedron, are calculated relative to the camera whose projections on the camera image would contain precisely these polygon candidates. This creates a candidate list for the target body.

step 508 evaluates the target body candidates by adding more line pieces in the vicinity of the polygon and including the one in the step 7 determined spatial position.

By means of suitable thresholds step decides 509 of the method, whether at least one sufficiently promising target body candidate is present in the camera image. If this is not the case, the algorithm aborts and starts again at step 501 at.

Otherwise, the algorithm chooses in step 510 the best target body candidate, and calculated by means of a best-fit method known to those skilled in the spatial position of the target body whose projection on the camera image as best as possible with the target body associated line pieces.

In the last step 511 the algorithm sets the calculated spatial position of the target body 512 available for further processing by third party programs and returns to step 501 back.

6 shows by way of example the steps 503 to 505 of the algorithm.

6a gives a section of a camera image 611 of the cuboctahedron as well as one of the edge detector, step 503 , found edge point chains 612 that was projected onto the picture here.

6b shows the algorithm using tight straightness tolerances 622 found rectilinear sections in the edge point chain 621 as well as in this rectilinear sections inserted equalization line 623 ,

In 6c were first the equalizer 623 near the first 631 and last point 632 of the respective section of the edge point chain to a line piece 633 shortened. The directional arrow on the line section indicates which light type is in the original image 611 located on the right or left of the line piece. Furthermore, the picture shows the search for pairs from a 631 and an endpoint 632 two different line pieces 633 whose mutual distance is a threshold 634 below

In 6d the algorithm has the three pairs found 641 whose three start and three endpoints belong to the same three line-pieces, to a triangle 642 added, which approximates the white triangle in the middle of the 6a equivalent.

For Quadrilaterals, the procedure is analog, with the direction of rotation The arrows in both cases indicate which kind of light is must be located inside the polygon.

7 shows that in step 507 performed method, which is to assign each triangular triangle found in the camera image generally two equilateral triangles in space. The edge length of these equilateral triangles is equal to the edge length of the triangles of the target body. From the point of view of the camera, both equilateral triangles appear completely identical and they also coincide with the triangle found in the camera image.

For squares whose projections are generally parallelograms, a triangle is first drawn into the found image parallelogram, which has the same perspective distortion in relation to the equilateral triangle as the parallelogram in relation to the square. This drawn triangle will follow step 507 treated. The results are then transferred back to the parallelogram, yielding the two sought spatial locations of a square of known size.

The process is modeled on a camera orientation system 701 at the focal point of the camera lens and an image reference system 702 in an image plane that is perpendicular to the camera axis z. The pictorial rectangle is modeled in the image plane 703 whose center lies on the camera axis z. The distance of the image plane to the focal point is chosen so that the ratio of the image width to this distance corresponds to the horizontal angle of the camera. From the focal point of the camera go three rays 704 so in the room that each ray is a corner 705 of the triangle drawn in the picture plane 706 pierces. The method according to the invention places the equilateral triangles 707 and 708 so in the room that their three corners touch all three rays.

8th shows a further development of the present invention, a merely ergonomically modified, ordinary computer mouse 801 around a cuboctahedron 802 as well as a switch 803 was added. This arrangement realizes an input device that can be used both as an ordinary computer mouse and as a spatial computer mouse, that is, as an input device for all six spatial degrees of freedom.

In the spatial computer mouse according to the inventive method is a tracking mechanism held in the hand of the computer user input device 801 realized in that the position of the target body according to the invention, preferably a Kuboktaeders 802 , is determined within the field of view of a camera in each image transmitted by the camera.

By way of illustration, it should be understood that a common computer mouse controls the relative displacement of the pointer observable on the screen relative to any initial position of the pointer. The displacement of the pointer is visually controlled by the user and corrected, if necessary, to place the pointer at a desired point on the screen or to move along a path. If the range of motion of the hand guiding the mouse is no longer sufficient to perform a desired trajectory due to collisions or unfavorable posture, the user can lift the computer mouse, place it in a more favorable position and set it down again. Upon taking off, the user intentionally interrupts the tracking mechanism (bullet, laser, etc.) to bring the pointer to a standstill on the screen, while the computer mouse is being relocated. This process should be referred to as "delivery".

In order to be able to carry out a delivery even with a spatial computer mouse, it is natural that the third dimension taken up for withdrawal no longer be available. In the device according to the invention, the function of lifting is instead effected by means of a switch 803 realized. Advantageously, this is a switch attached to an ergonomically favorable location. Also advantageously, this switch can also be used to toggle the computer mouse from ordinary 2D mode to spatial mode of operation.

101

central processing unit of the computer

102

screen

103

keyboard

104

computer mouse

105

camera

106

target body

107

hand of the computer user

201

cuboctahedron

202

white triangle

203

black square

204

Just edge

211

target body

212

light source

213

electrical power supply

310

octahedron

311

corner of the octahedron

312

edge of the octahedron

320

cube

321

corner of the cube

322

edge of the cube

323

front of the cube

324

triangle on the cube front

330

cuboctahedron

331

border of the projected cuboctahedron

340

Rhombicuboctahedron

341

edge of the Rhombenkuboktaeders

350

icosidodecahedron

351

edge of the Ikosidodekaeders

401

computer users

402

hand of the computer user

403

target body

404

window the camera

405

camera

406

Data Connection

407

hardware of the computer

408

operating system of the computer

409

inventive algorithm

410

application program

411

screen

500

camera image

501 ... 511

steps 501 to 511 of the algorithm

512

spatial location

611

cuboctahedron

612

Edge point chain

621

Edge point chain

622

straightness

623

fit line

631

starting point a line piece

632

endpoint a line piece

633

line piece

634

distance threshold

641

At first endpoint pair

642

triangle

701

reference system the camera

702

reference system of the camera image

703

image rectangle

704

beam

705

corner of the triangle

706

triangle

707

first equilateral triangle

708

second simultaneous triangle

801

computer mouse

802

cuboctahedron

803

switch

QUOTES INCLUDE IN THE DESCRIPTION

This list The documents listed by the applicant have been automated generated and is solely for better information recorded by the reader. The list is not part of the German Patent or utility model application. The DPMA takes over no liability for any errors or omissions.

Cited patent literature

• - US 6711293 B1 [0007]
• - DE 102005011432 A1 [0008]
• WO 2004/114112 A1 [0009]
• US 6526166 [0011]
• WO 2005/075936 A1 [0012]
• US 2005201613 [0015]
• - WO 2007/04431 A2 [0015]

Claims (10)

A target body whose surface reflects or emits at least two types of light distinguishable by a camera, wherein the boundaries between the surface portions of the two types of light are designed as straight stretches, and methods for determining the spatial position of the target relative to a camera by processing them Camera recorded images characterized in that a. an image generated by a camera is loaded, b. a gray value image is generated, which the lights of the target body the c. Assign minimum or maximum value, d. an edge detector is used to form chained edge points, e. straight line pieces are cut out of the edge point chains, f. the straight line pieces are assembled into polygons, g. an evaluation of the polygons takes place, h. the projections of the target body surfaces matching the polygons i. are determined, each corresponding to a spatial position of the target body j. an evaluation of the spatial positions of the target body takes place, k. a best fit of the accepted target body takes place and l. the determined spatial position is transmitted. Method for using the target body according to claim 1, wherein the target body guided by the hand of a person and the method thus the six spatial degrees of freedom the hand position determined. The method of claim 2, wherein the determination of spatial position of the target body quasi-continuously is done by taking every single, captured by the camera image of the procedure is processed. Method according to claim 2 or 3, wherein the types of light realized as color impressions black and white become. Method according to one of claims 2 to 4, wherein the intensity of one or both types of light a mounted in the target body light source preferably increased electrical power and the target body partially transparent. Method according to one of claims 2 to 5, where the boundary lines between the types of light and the edges of the target body coincide. Method according to one of claims 2 to 6, wherein the target body takes the form of a Kuboktaeders, its squares of one kind of light and its triangles of the other kind of light be assigned to. Use of the device according to claim 1 for carrying out the method according to one of the claims 2 to 7. Use according to claim 8, wherein the operation of a "6-degree of freedom mouse" switched on by means of at least one switch, can be interrupted or turned off. Use according to claim 8 or 9, characterized that the 6-degree of freedom mouse optionally as conventional 2-D mouse or as a spatial computer mouse can be operated.