GB2451461A

GB2451461A - Camera based 3D user and wand tracking human-computer interaction system

Info

Publication number: GB2451461A
Application number: GB0714844A
Authority: GB
Inventors: Naveen Chawla
Original assignee: Individual
Current assignee: Individual
Priority date: 2007-07-28
Filing date: 2007-07-28
Publication date: 2009-02-04
Also published as: GB0714844D0

Abstract

A camera based tracking system is disclosed providing computer tracking of two handheld wands 1,2 by means of a pair of cameras 6,7, and tracking of the user's viewing position by means of a viewing position marker 3 which may be mounted on the users head, or by individually tracking the users eyes 9,10. The wands 1,2 may comprise a user operated trigger 15 to allow the user to pick up and manipulate objects inside a 3D virtual environment. The wands 1,2 may include a marker 4,5.

Description

Camera-V ision-Base'J 3D User-andwand..Track lug Stereoscopic Human-Computer Interaction System A camera-vision-based system, which includes the computer tracking of up to two handheld wands, as shown in Fig.2 and Fig.3, and denoted by I and 2 in Fig. 1, in addition to the tracking of the user's left and right eye viewing positions via a viewing position marker 3 in Fig. I which is mounted on their head, or by individually tracking the user's eyes denoted by 9 and 10 in Fig.!, is proposed as a crucially and uniquely unified system for the purpose of achieving complete stereoscopic 3D human-computer interaction. The unique completeness of interaction that is afforded by the system can only be achieved using the uniquely unified combination proposed. The "wands", as designed, would each have a user-operatej trigger 15 (Fig.l), and as such would allow the user to pick up and manipulate objects inside a computer 3D virtual world, and, depending how an application that uses this system is set up, would enable the user to control their movement in any way they like inside that 3D world, according to whatever parameters of movement the application programmer may set in the application. A given wand would use computer vision to determine its position in 3D space, using a marker on itsenddenoted by4 and Sin Fig.! andt oormo merasdenotec by 6 and 7 in Fig.l.

Determining the position of the wand in 3D space involves recognizing the fact that cameras have a diffuse and not a parallel pattern of vision which radiates from its optical centre as far as the angle of view. This means that if we want to determine the position in space we need to recognize that a single point as denoted by 1 in Fig.4 on a camera image as denoted by 2 in Fig.4 would represent a line of possible positions for that point as denoted by 3 in Fig.4 emanating from the optical centre of the camera as denoted by 4 in Fig.4. The camera in Fig. 4is as viewed from above. The 2D position of the point on the screen would be tracked by computer.

Using a second camera we can determine a second line of possible positions for the point, and use the intersection to determine the position in 3D space, as shown in Fig.5. This is regardless of the position of the cameras, as long as they are both detecting the point in question. The first camera is denoted by I in Fig.5, the second by 2 in Fig.5, .. the first line of possible positions for the point is denoted by 3 in Fig.5, the second line of :.:: possible positions for the point is denoted by 4 in Fig.5, and the 3D location of the point (x,y,z) is denoted by 5 in Fig.5. If the two lines are found to be skewed, the mid-point of the shortest joining line between those 3D lines should be used to determine a best estimate of the point's location.

A second wand is easily added for interaction with both hands simultaneously. The wand is designed as shown in the pictures, so that its trigger responds to a natural squeezing : action of the hand, in order to give the user a natural impression of grabbing an object, as opposed to the "thumb_middle..fingerplus..jfldexfig.,, action of the 3D mouse, which does not resemble the natural grabbing of an object. /

Four versions of the wand are shown in fig.2. Version 1 in fig.2 shows a thumb-operated button I on the wand so that the wand trigger is operated by squeezing between the thumb and other fingers. This is one of the natural grabbing actions of the hand, as mentioned. Version 2 in fig2 shows an index finger operated trigger button 2 so that the wand trigger is operated by squeezing between the index finger and the lower palm or the thumb, much like a gun. This is another natural grabbing action of the hand. Version 3 in fig.2 shows a trigger which can be activated anywhere along the length of the wand by squeezing between the two parts of the trigger 3 and 4. This can accommodate any natural grabbing action of the hand. Finally, version 4 in flg.2 shows a trigger 5 which is activated by squeezing anywhere on the wand. This also accormnodates all natural squeezing actions of the band.

One or more additional markers may also be included on the handheld device, either already attached or attachable by the user via an extension piece in each case, such as extension piece 11 in fig.! and 3 and 4 in flg.3, in order to allow the orientation of the handheld device to be additionally tracked. This is so that the handheld device may be used as a "sword", a "gun", an "object pointer" and so on.

Item I in flg.3 is an "object pointer" or "sword" version of the handheld device.

Item 2 in fig. 3 is a "gun" version.

In addition to this, what is proposed is again the use of computer vision from two or more cameras, which are either the same cameras or additional cameras, in order to determine the 3D coordinates of the user's left and right eye viewing positions, by tracking viewing position marker 3 in fig.! and estimating the displacement of the user's left and right eyes from the viewing position marker, or by individually tracking the position of the user's left eye 9 or right eye 10, or both, in fig. 1, in order that actual stereoscopic viewing realism of the computer's 3D world based on the 3D coordinates of the user's left and right eye viewing positions can be created, in other words, it would allow the user to move their head and view the side of objects stereoscopically, and when the user moves closer to the screen, denoted by 8 in fig. 1, the left and right eye images, viewed using stereoscopic apparatus, of the 3D virtual world would show the correct perspective of that 3D world for each eye in relation to the user's left and right eye viewing positions. It would do this by setting two virtual camera positions in the computer's 3D virtual world *. . to the 3D coordinates of the user's left and right eye viewing positions (or proportional :.:: equivalents), and setting the frustum of the virtual camera in each case in the computer's *.., virtual 3D world to match the 3D shape that the user's relevant eye viewing position makes with the corners of the image of that virtual 3D world on that screen at any given * . : time, typically the corners of the screen. The frustums would, thus, typically be * *" asymmetric. Examples of the frustums are shown by 12 and 13 in fig.l. The tracking of * the 3D coordinates of the user left and right eye viewing positions would use the same 3D line near-intersection method as described for tracking the 3D coordinates of the wand.

S... . . . . Thus, what is proposed is a system which uniquely creates total stereoscopic positional : interaction realism. With the system, the user can thus, uniquely, from any viewing position, use their natural depth perception to reach out for and interact with objects in the 3D virtual world, because of the fact that the system is stereoscopic.

Claims

I. A system which uses two cameras, or more, to detect the 3D coordinates of a person's left eye and right eye viewing positions relative to the screen they are looking at, which subsequently sets the frustums of two virtual cameras in the computer's virtual 3D world to match the 3D shapes that the user's left eye and right eye viewing positions make with the corners of the image of that virtual 3D world on that screen at any given time, in order to create a realistic stereoscopic picture of that virtual 3D world to the user, and which includes a handheld device which has a trigger which responds to a squeezing action between the thumb and one or more other fingers, whose signal is transmitted to the computer when the squeezing action is performed, which also uses the cameras to determine its position in 3D space, thereby allowing the user to interact stereoscopically with the 3D virtual world from the screen at any given time, while viewing that world stereoscopically using the two frustums as calculated.
2. A system which uses two cameras, or more, to detect the 3D coordinates of a person's left eye and right eye viewing positions relative to the screen they are looking at, which subsequently sets the fnistums of two virtual cameras in the computer's virtual 3D world to match the 3D shapes that the user's left eye and right eye viewing positions make with the corners of the image of that virtual 3D world on that screen at any given time, in order to create a realistic stereoscopic picture of that virtual 3D world to the user, and which includes a handheld device which has a trigger which responds to a squeezing action between one or more fingers and the palm, whose signal is transmitted to the computer when the squeezing action is performed, which also uses the cameras to determine its position in 3D space, thereby allowing the user to interact stereoscopically with the 3D virtual world fmm the screen at any given time, while viewing that world stereoscopically using the two frustums as calculated.
3. A system as claimed in Claim I or Claim 2 in which the position in 3D space of each article being tracked (the article being either the user's viewing position or the handheld device, where applicable) is calculated using the 2D position at which that article, or a marker on or in a fixed relative position near that article, appears on the image generated by each camera, to determine a straight 3D line of possible positions of that article or marker for each camera, and then the intersection of those 3D lines, or, if the 3D lines are found to be skewed, the mid-point of the shortest joining line between those 3D lines, to determine a single 3D position of location for that article or marker, and if that marker is displaced from the article itself by a fixed relative position, using a calculation or an estimation of that displacement to determine a single 3D position of location of the article itself.
4. A system as claimed in Claim I or Claim 2 or Claim 3 in which there is a marker placed on or in a fixed relative position near each article being tracked, which is a device which actively emits energy (for example an LED), which is capable of being detected by the cameras used.
5. A system as claimed in Claim 1 or Claim 2 or Claim 3 in which there is a marker placed on or in a fixed relative position near each article being tracked, which is a passive reflective marker which is capable of being detected by the cameras used.
6. A system as claimed in any preceding claim in which the cameras used comprise a combination of, or consist only of, infrared cameras, ordinary light cameras, ultrasound cameras or any other types of camera which are capable of detecting the markers or objects to be tracked.
7. A system as claimed in any preceding claim in which one or more additional markers are also included on the handheld device, either already attached or attachable by the user via an extension piece in each case, in order to allow the orientation of the handheld device to be additionally tracked.
8. A system as claimed in any preceding claim in which there are two of the handheld devices deployed, one for the left hand and one for the right hand.
9. A system substantially as herein described above and illustrated in the accompanying drawings.