GB2388418A - Input or pointing device with a camera - Google Patents

Input or pointing device with a camera Download PDF

Info

Publication number
GB2388418A
GB2388418A GB0207300A GB0207300A GB2388418A GB 2388418 A GB2388418 A GB 2388418A GB 0207300 A GB0207300 A GB 0207300A GB 0207300 A GB0207300 A GB 0207300A GB 2388418 A GB2388418 A GB 2388418A
Authority
GB
United Kingdom
Prior art keywords
shape form
device according
vector
visual
degrees
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
GB0207300A
Other versions
GB0207300D0 (en
Inventor
Marcus James Eales
Original Assignee
Marcus James Eales
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Marcus James Eales filed Critical Marcus James Eales
Priority to GB0207300A priority Critical patent/GB2388418A/en
Publication of GB0207300D0 publication Critical patent/GB0207300D0/en
Publication of GB2388418A publication Critical patent/GB2388418A/en
Application status is Withdrawn legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/0304Detection arrangements using opto-electronic means
    • G06F3/0325Detection arrangements using opto-electronic means using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/0346Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors

Abstract

A pointing device having a moveable 3-D shape form 1 with bold visual features 2 is imaged by a television camera 3 to obtain a 2-D electronic image. The electronic image is evaluated by means of computer software in order to extract the visual features and measure their spatial dispositions within the 2-D image. This information enables six degrees of movement of the device to be determined. The movement so determined is applied to a 2-D graphical representation of a virtual 3-D image displayed on a viewing screen of a computer monitor 5, or used to provide pointing information. The shape form 1 may be a cube, sphere (fig 4) or polyhedron (fig 3) with different coloured regions or LED's, or merely an assembly of LED's or the users hand; may be mounted on a base (figs 5-7); or enclosed by a transparent mouse type cover. A second camera (fig 8) may be provided for eclipsed or awkwardly lit features 2.

Description

238841 8

Multi-vector pointing device The present invention is an extension of conventional forms of mouse in common usage with personal computers (PCs) PC mice, trackballs and touch pads provide two degrees of freedom while some joysticks or 'spaceballs' provide three or more degrees by incorporating a rotation as well as linear action.

An objective of the present invention is to facilitate the representation of six degrees of freedom of movement of a 3-D object when displayed on a 2-D display screen.

Six degrees of freedom are provided by three (x, y and z axes) translation directions and three rotational movements about those axes.

The closest prior art are so called "spaceballs" (for example from IBM), which are

commercially available, and are able to provide up to six degrees of freedom. These are expensive and unwieldy to use. In use, it is difficult to maintain independent control of the degrees of freedom, using a "spaceball". It is an objective of the present invention to overcome these difficulties.

Conventional PC mice, while only providing two degrees of freedom, are also prior art for the present invention.

It is further an objective of the present invention to facilitate the representation of six degrees of freedom of movement of a 3-D object, when displayed on a 2-D display screen, using existing low cost hardware.

PC mice are used with solid computer modelling soft Nare to effect onscreen movement of 2-D pictorial representation of 3-D computer generated designs.

Visual 2-D representation of all six degrees of freedom is difficult to portray using a 2-D device such as a computer mouse.

It is an objective of the present invention to provide up to six substantially independent degrees of freedom, which can be translated into corresponding degrees of freedom in a 2-D representation of a 3-D image, in a readily useable ergonomic form.

In the present invention, a requisite number of independent degrees of freedom are derived from a 2-D optical image of a 3-D shape form. This is effected by converting the optical image into electronic form by existing known means, such as a charge coupled device (CCD) camera, and evaluating the electronic image in a way to determine movement in any of the six degrees of freedom. The present invention provides for any of the derived degrees of freedom to be appropriately applied to a virtual 3-D image displayed on a 2-D display, for example on a PC monitor. This enables an operator to manipulate the movements of a displayed image in a precise manner by virtue of the normal biofeedback loop translated through the operator.

The evaluation of the electronic image is performed by appropriate computer software, which, when written is inexpensive to replicate and may be installed on any PC.

According to the present invention, there is provided a multi-vector pointing device comprising a moveabJe shape form, a camera for producing a 2-D electronic image of the shape form in real time, characterized in providing pointing or viewing information of the object for software applications.

According to the present invention, there is provided a multi-vector pointing device comprising a moveable three dimensional (3-D) shape form, a camera for producing a two dimensional (2-D) electronic image of the 3D shape form in real time, characterized in providing evaluating means for determining up to six degrees of freedom of the 3-D shape form from the 2-D electronic image of the 3-D shape form and appropriately applying the degrees of freedom to a 2-D representation of a general virtual 3-D image displayed on a viewing screen.

Within the context of the present invention, the term "shape form" is used to describe an imagable muitivector pointing device represented in three spatial dimensions. The shape form may also have two dimensional form, such as when constructed from a planar material such as sheet material, but would still occupy three dimensional space.

Central to the invention is the use of a television camera, typically using CCD technology, for providing a 2-D image of a 3-D shape form, in electronic form. The 3-D shape form provides visual features whose positions are determined from the 2-

D electronic form of the image. mere are provided a sufficient number of visual features to determine the required number of degrees of freedom: for example three are require to produce six degrees of freedom, while two degrees of freedom may be determined from a single point feature.

Evaluation of the positions of visual features in a 2-D electronic form of the image is performed in a prescribed way according to rules contained in computer software.

This determines the directions and magnitudes of three orthogonal translations and three orthogonal rotations of the 3-D shape form.

The visual features provided on the 3-D shape form (forming part of the 3D shape form) can be made distinctive, for example with coloured spherical surfaces, positioned in a suitable geometrical arrangement. Coloured spherical surfaces may be positioned at the corners corresponding to the corners of a cube. It is within the scope of the present invention to provide a 3-D shape form having distinctive visual features disposed in any geometrical or non-geometrical 3-D configuration, suitable for facilitating their evaluation. Any particular configuration of visual features constitutes part of a 3-D shape form.

Any means of providing visually distinct features lies within the scope of the present invention. This includes: polished or matt balls (spherical surfaces) illuminated by external light source; and small emitting regions such as light emitting diodes (I.e.d.s); small white, coloured or reflective stickers placed on the surface of a shape

( form. The distinctive visual features will normally be attached to a featureless handheld rigid body, painted in one colour different from any of the visual features.

For example, the body may be matt white or black, or any colour which is conveniently separated by electronic means in the 2-D electronic image.

It is also within the scope of the present invention to include visual features on a 3-

D shape form that are present by virtue of the geometry of the shape form, and which are highlighted by any given mode of illumination, such as room lighting.

It is convenient that the distances separating pairs of visual features are pre-

determined for use by the evaluating algorithms in the software. This is especially appropriate when the visual features are disposed according to a distinctive geometrical shape such as the corners of a cube. This, however, is not a prerequisite, and the necessary spatial information about the visual features may be determined using software based teaching procedures known per se. This is especially useful when the 3-D shape form has arbitrary shape with arbitrary visual features identifiable from a 2-D image of the shape form.

While the 3-D shape form is primarily intended to be a handheld device, which replaces a computer mouse, it may be designed for use by disabled persons. It could for example be constructed from light weight materials, such as table tennis balls, which may be manipulated by a mouth held probe.

It is within the scope of the present invention to include a user's hand or clenched fist as a 3-D shape form. Paper stickers may be attached to an operator's hand to provide easily identifiable visual features.

It is also within the scope of the invention to provide evaluation software which will determine any of the six degrees of movement freedom of an arbitrary 3-D shape form (such as a clenched fist), by automatically identifying and tracking prominent visual features present in the shape form. This relies on measuring different amounts of displacement, between different visual feature pairs, detected in the 2-D

electronic image In this embodiment it is prudent to incorporate redundant information (redundant visual features) not only to improve accuracy, but also to cope with features coming in and out of prominence as the 3-D shape form rotates.

Standard imaging processing techniques, known in the art, may be used in identifying visual features for the purpose of evaluation. By way of example, one method of evaluation involves computing the 2-D coordinates of centroids of respective visual features. Dimensions between all pairs of respective centroids are calculated, and variations in these values are used to determine movement in all six degrees of freedom. Many image processing and pattern recognition algorithms are known in the art, for the purpose of feature extraction, many of which are applicable to the present invention. For example edge detection may be used to identify closed regions. For the purpose of this invention, a convenient number of closed regions may be selected and their centroids computed.

Tracking algorithms may be employed which correctly identify visual features as their shapes change with movement. Previously documented processing techniques may be applied in this invention for the purpose of achieving the desired result.

Movement in any of the six degrees of freedom of the 3-D shape form is computed solely from distance-vector measurements made between visual features in the 2-D image and rates at which these distance-vectors change. Computation is simplified if actual distances between features is known in advance, but this is not a prerequisite for successful operation.

It is also within the scope of the invention to provide more than one camera for viewing a shape form. This provides redundancy of information useful in circumstances when a feature is eclipsed from view by the shape form, or when lighting favours one viewing direction over another.

We invention is described in detail with reference to the following figures:

Figure shows salient features of a preferred embodiment of the invention.

Figures 2 shows a shape form in the shape of a cube.

Figure 3 shows a shape form in the shape of a regular solid polygon.

Figure 4 shows a shape form in the shape of a sub-divided sphere.

Figure 5 shows a shape form attached to a base via a ball joint.

Figure 6 shows a shape form attached to a base via coiled spring.

Figure 7 shows a shape form attached to a base via an expendable stalk and ball joint. Figure 8 shows an embodiment of the invention using two television cameras.

With reference to figure 1: shape form, i, supporting at least one visual feature, 2, is imaged by television camera, 3, to produce a 2-D electronic image. This electronic image passes into PC, 4, for evaluation by image evaluating soft Nare. The image passes into the PC by known means, for example via a universal serial bus (USB) port. The television camera may conveniently be a "webcam" type.

Monitor, 5, displays a 3-D representation of any obje*, 6, which has been created or forms part of any software application using 3-D representation.

Lyle 3-D representation of object, 6, is moved on the monitor screen in any of the 6 apparent degrees of freedom attributed to a three dimensional object, by imparting appropriate corresponding movement to shape form, 2. It will be appreciated that the 3-D representation of object 6, is actually a Z-D image on a flat screen.

Movement in any of six degrees of freedom are therefore apparent movement in the 3-D representation which enhance the 3-D perception of the object.

Figure 2 shows a shape form with bold visual features, 1, positioned at the comers of a cube. In this case the visual features are partial spherical surfaces which may be polished metal to reflect external light sources, or else coated black, white or different colours, in matt or any other suitable surface finish. The features,], positioned at the corners of cube, 2, may equally be emitting light sources such as light emitting diodes (l.e.d.s). Different coloured l.e.d.s can be used to facilitate feature detection and identification.

Figure 3 shows a regular 12 sided shape form having pentagonal faces. The faces themselves may be coloured to provide visual features or features may be attached to the corners as described above with reference to figure 2. The present invention is not restricted to this particular multifaceted shape form and any similar, not necessarily regular, multifaceted object may be used in accordance with the present invention. Figure 4 shows a spherical shape form, subdivided into distinct regions. The regions provide visual features, which may be coloured to facilitate identification.

Figure 5 shows a shape form, mounted to a base, 7, via a ballpoint, 8. This allows the shape form to move in two horizontal dimensions as well as rotation about three perpendicular axes. The shape form is nominally moved by hand, and a handle may be provided to assist hand movement.

Figure 6 shows a shape form with movement in all six degrees of freedom. The shape form, 2, is attached to a base, 7, via a loose coil, 9, which allows the shape form to be moved vertically as well as rotated about three orthogonal axes, relative to the base. The base is moveable in two horizontal axes; also the shape form is moveable in two horizontal axes relative to the base.

Figure 7 shows another way in which the shape form is moved in six degrees of freedom. The shape form is attached to a base, 7, by an extendable stalking, via pivots or a ball joint, 8. The base is moveable in two horizontal axes; also the shape form is moveable in two horizontal axes relative to the base.

Any shape form may be attached to a base in accordance with figures 5, 6, and 7 or by other means.

Figure 8 shows an embodiment of the invention incorporating more than one television camera. This provides redundant information, which is needed in cases where visual features become indistinct, such as when the shape form is an arbitrary shape. Any appropriate arbitrarily shaped object can be used as a shape form in accordance with the present invention. This includes an assembly of l.e.d.s without a regular geometrical relationship between them.

The present invention includes a visual feature set and/or shape form, enclosed by a transparent mouse, type, cover. This provides aesthetic appeal, and the cover may be fashioned as a handle for ease of use. The cover can, for example, be dome shaped moulded transparent plastic.

Included within the scope of the present invention is a shape form attached to a base via any semi-flexible coupling material, which allows complete freedom of movement of the shape form, but remains in a fixed position when released.

Those skilled in the art will appreciate that while the present invention can return information about six degrees of freedom it could also be used to return less information, and therefore replace joysticks (normally three degrees of freedom) and mice (normally two degrees of freedom)

Claims (1)

  1. iti-yector POinting de,jce CLAlMs 1 ACcording to the i there is Provided a
    multj vect deViCe C mPrising a mov ape fiorrn' a camera for p a 2electronic ima ape fo in real time Character' d g POinting r\,iewin9 jnfOrmatO f applications. ACCOrding to the i there is Provided a mult' ve de iCe comprisin9 a e three dimensional (3-D) she f a camera for prOduci inensional (2-D) electrOnjc im the 3-D shape form i time' characterised in prOvidin evaluatin9 mean ninin9 up to six degrees of f d the 3-D shape form f 2 D electronic image of the 3 h m and aPproprjatel P Y ng the deDrees of freedom to 2 representation g eral Virtual 3-D image display d VieU/in9 Screen e 3 mUlti- ector g deViCe according to claims 2 the moveable shape PPorts and Provides isual feat 4- m lti-vectOr 9 device accordin9 to claim 3 h number of su ed \tisual [eatures is at least th S. multi-vector g deViCe accordin9 to claims 1 herein the number P ted ViSual fieatures is sumcjent f thr of the vis res to be distinct in te 2 D el ge of the sh m at all times durjng p
    ( 6ulti-vector 9 eViCe according to claim 5 here' i features are re stinctive by virtue of visual features b i of different colours.
    7multi-vector 9 device according to claims 5 or 6 h Visual eature Posed in any suitable geometrical 3 D configuration. 8multi-vector 9 device according to any of the dai 5 7 wherein the visual {natures are rendered distinctive by virtue of being individual Spherical sur faces having polished or matt surfaces re illuminated by an external light 9A multivector 9 device according to any of the cl i 7 wherein the visual features are rendered distinctive by virtue of being individual light emitters 1 0.multi-vect p sting device according to claim 9 wh light emitters are light emitting di 1 1A multi-vector 9 device according to any one of d i 5 or 7 wherein v tures are white, coloured or refile ti placed on the surface of a strap f 12 A m lti-vector 9 device according to any of me claim wherein the shape form is a handheld rigid body.
    13 mUIti-VeCtOr pointing device according to claim 12 wherein the shape form is a 3-D polyhedron.
    14 ulti-vecto ng device according to claim 13 wh 3 D polyhedron are visual feat 1 ')
GB0207300A 2002-03-28 2002-03-28 Input or pointing device with a camera Withdrawn GB2388418A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
GB0207300A GB2388418A (en) 2002-03-28 2002-03-28 Input or pointing device with a camera

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
GB0207300A GB2388418A (en) 2002-03-28 2002-03-28 Input or pointing device with a camera

Publications (2)

Publication Number Publication Date
GB0207300D0 GB0207300D0 (en) 2002-05-08
GB2388418A true GB2388418A (en) 2003-11-12

Family

ID=9933872

Family Applications (1)

Application Number Title Priority Date Filing Date
GB0207300A Withdrawn GB2388418A (en) 2002-03-28 2002-03-28 Input or pointing device with a camera

Country Status (1)

Country Link
GB (1) GB2388418A (en)

Cited By (53)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2005116809A2 (en) 2004-05-24 2005-12-08 3D For All Számítástechnikai Fejlesztö Kft. System and method for operating in virtual 3d space and system for selecting an operation via a visualizing system
EP1552375B1 (en) * 2002-07-27 2006-09-20 Sony Computer Entertainment Inc. Man-machine interface using a deformable device
WO2007050885A2 (en) * 2005-10-26 2007-05-03 Sony Computer Entertainment America Inc. System and method for interfacing with a computer program
WO2009059716A1 (en) * 2007-11-05 2009-05-14 Sebastian Repetzki Pointing device and method for operating the pointing device
US20090298590A1 (en) * 2005-10-26 2009-12-03 Sony Computer Entertainment Inc. Expandable Control Device Via Hardware Attachment
US7646372B2 (en) 2003-09-15 2010-01-12 Sony Computer Entertainment Inc. Methods and systems for enabling direction detection when interfacing with a computer program
US7663689B2 (en) 2004-01-16 2010-02-16 Sony Computer Entertainment Inc. Method and apparatus for optimizing capture device settings through depth information
US7737944B2 (en) 2002-07-27 2010-06-15 Sony Computer Entertainment America Inc. Method and system for adding a new player to a game in response to controller activity
US7760248B2 (en) 2002-07-27 2010-07-20 Sony Computer Entertainment Inc. Selective sound source listening in conjunction with computer interactive processing
US7874917B2 (en) 2003-09-15 2011-01-25 Sony Computer Entertainment Inc. Methods and systems for enabling depth and direction detection when interfacing with a computer program
US7883415B2 (en) 2003-09-15 2011-02-08 Sony Computer Entertainment Inc. Method and apparatus for adjusting a view of a scene being displayed according to tracked head motion
US8035629B2 (en) 2002-07-18 2011-10-11 Sony Computer Entertainment Inc. Hand-held computer interactive device
US8072470B2 (en) 2003-05-29 2011-12-06 Sony Computer Entertainment Inc. System and method for providing a real-time three-dimensional interactive environment
US8089458B2 (en) 2000-02-22 2012-01-03 Creative Kingdoms, Llc Toy devices and methods for providing an interactive play experience
US8142288B2 (en) 2009-05-08 2012-03-27 Sony Computer Entertainment America Llc Base station movement detection and compensation
US8188968B2 (en) 2002-07-27 2012-05-29 Sony Computer Entertainment Inc. Methods for interfacing with a program using a light input device
US8210943B1 (en) 2006-05-06 2012-07-03 Sony Computer Entertainment America Llc Target interface
US8226493B2 (en) 2002-08-01 2012-07-24 Creative Kingdoms, Llc Interactive play devices for water play attractions
US8253801B2 (en) 2008-12-17 2012-08-28 Sony Computer Entertainment Inc. Correcting angle error in a tracking system
US8287373B2 (en) 2008-12-05 2012-10-16 Sony Computer Entertainment Inc. Control device for communicating visual information
US8310656B2 (en) 2006-09-28 2012-11-13 Sony Computer Entertainment America Llc Mapping movements of a hand-held controller to the two-dimensional image plane of a display screen
US8313380B2 (en) 2002-07-27 2012-11-20 Sony Computer Entertainment America Llc Scheme for translating movements of a hand-held controller into inputs for a system
US8323106B2 (en) 2008-05-30 2012-12-04 Sony Computer Entertainment America Llc Determination of controller three-dimensional location using image analysis and ultrasonic communication
US8342963B2 (en) 2009-04-10 2013-01-01 Sony Computer Entertainment America Inc. Methods and systems for enabling control of artificial intelligence game characters
US8342926B2 (en) 2008-07-13 2013-01-01 Sony Computer Entertainment America Llc Game aim assist
US8368753B2 (en) 2008-03-17 2013-02-05 Sony Computer Entertainment America Llc Controller with an integrated depth camera
US8393964B2 (en) 2009-05-08 2013-03-12 Sony Computer Entertainment America Llc Base station for position location
US8475275B2 (en) 2000-02-22 2013-07-02 Creative Kingdoms, Llc Interactive toys and games connecting physical and virtual play environments
US8527657B2 (en) 2009-03-20 2013-09-03 Sony Computer Entertainment America Llc Methods and systems for dynamically adjusting update rates in multi-player network gaming
US8542907B2 (en) 2007-12-17 2013-09-24 Sony Computer Entertainment America Llc Dynamic three-dimensional object mapping for user-defined control device
US8547401B2 (en) 2004-08-19 2013-10-01 Sony Computer Entertainment Inc. Portable augmented reality device and method
US8570378B2 (en) 2002-07-27 2013-10-29 Sony Computer Entertainment Inc. Method and apparatus for tracking three-dimensional movements of an object using a depth sensing camera
US8608535B2 (en) 2002-04-05 2013-12-17 Mq Gaming, Llc Systems and methods for providing an interactive game
US8686939B2 (en) 2002-07-27 2014-04-01 Sony Computer Entertainment Inc. System, method, and apparatus for three-dimensional input control
US8702515B2 (en) 2002-04-05 2014-04-22 Mq Gaming, Llc Multi-platform gaming system using RFID-tagged toys
US8708821B2 (en) 2000-02-22 2014-04-29 Creative Kingdoms, Llc Systems and methods for providing interactive game play
US8753165B2 (en) 2000-10-20 2014-06-17 Mq Gaming, Llc Wireless toy systems and methods for interactive entertainment
US8758136B2 (en) 1999-02-26 2014-06-24 Mq Gaming, Llc Multi-platform gaming systems and methods
US8761434B2 (en) 2008-12-17 2014-06-24 Sony Computer Entertainment Inc. Tracking system calibration by reconciling inertial data with computed acceleration of a tracked object in the three-dimensional coordinate system
US8781151B2 (en) 2006-09-28 2014-07-15 Sony Computer Entertainment Inc. Object detection using video input combined with tilt angle information
US8797260B2 (en) 2002-07-27 2014-08-05 Sony Computer Entertainment Inc. Inertially trackable hand-held controller
US8840470B2 (en) 2008-02-27 2014-09-23 Sony Computer Entertainment America Llc Methods for capturing depth data of a scene and applying computer actions
US20150029100A1 (en) * 2013-07-23 2015-01-29 University Of Kentucky Research Foundation 2d and 3d pointing device based on a passive lights detection operation method using one camera
US8961313B2 (en) 2009-05-29 2015-02-24 Sony Computer Entertainment America Llc Multi-positional three-dimensional controller
US8970707B2 (en) 2008-12-17 2015-03-03 Sony Computer Entertainment Inc. Compensating for blooming of a shape in an image
US9058063B2 (en) 2009-05-30 2015-06-16 Sony Computer Entertainment Inc. Tracking system calibration using object position and orientation
US9177387B2 (en) 2003-02-11 2015-11-03 Sony Computer Entertainment Inc. Method and apparatus for real time motion capture
US9393487B2 (en) 2002-07-27 2016-07-19 Sony Interactive Entertainment Inc. Method for mapping movements of a hand-held controller to game commands
US9446319B2 (en) 2003-03-25 2016-09-20 Mq Gaming, Llc Interactive gaming toy
US9474968B2 (en) 2002-07-27 2016-10-25 Sony Interactive Entertainment America Llc Method and system for applying gearing effects to visual tracking
US9526995B2 (en) 2006-11-22 2016-12-27 Sony Interactive Entertainment America Llc Video game recording and playback with visual display of game controller manipulation
US9682319B2 (en) 2002-07-31 2017-06-20 Sony Interactive Entertainment Inc. Combiner method for altering game gearing
US10279254B2 (en) 2005-10-26 2019-05-07 Sony Interactive Entertainment Inc. Controller having visually trackable object for interfacing with a gaming system

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0587138A2 (en) * 1992-09-08 1994-03-16 Kabushiki Kaisha Toshiba 3D Human interface apparatus using motion recognition based on dynamic image processing
US5297061A (en) * 1993-05-19 1994-03-22 University Of Maryland Three dimensional pointing device monitored by computer vision
US5454043A (en) * 1993-07-30 1995-09-26 Mitsubishi Electric Research Laboratories, Inc. Dynamic and static hand gesture recognition through low-level image analysis
US5616078A (en) * 1993-12-28 1997-04-01 Konami Co., Ltd. Motion-controlled video entertainment system
GB2345538A (en) * 1998-11-06 2000-07-12 David James Optical tracker
WO2002027453A2 (en) * 2000-09-26 2002-04-04 Eugenio Bustamante Providing input signals

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0587138A2 (en) * 1992-09-08 1994-03-16 Kabushiki Kaisha Toshiba 3D Human interface apparatus using motion recognition based on dynamic image processing
US5297061A (en) * 1993-05-19 1994-03-22 University Of Maryland Three dimensional pointing device monitored by computer vision
US5454043A (en) * 1993-07-30 1995-09-26 Mitsubishi Electric Research Laboratories, Inc. Dynamic and static hand gesture recognition through low-level image analysis
US5616078A (en) * 1993-12-28 1997-04-01 Konami Co., Ltd. Motion-controlled video entertainment system
GB2345538A (en) * 1998-11-06 2000-07-12 David James Optical tracker
WO2002027453A2 (en) * 2000-09-26 2002-04-04 Eugenio Bustamante Providing input signals

Cited By (123)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8758136B2 (en) 1999-02-26 2014-06-24 Mq Gaming, Llc Multi-platform gaming systems and methods
US8888576B2 (en) 1999-02-26 2014-11-18 Mq Gaming, Llc Multi-media interactive play system
US9861887B1 (en) 1999-02-26 2018-01-09 Mq Gaming, Llc Multi-platform gaming systems and methods
US9731194B2 (en) 1999-02-26 2017-08-15 Mq Gaming, Llc Multi-platform gaming systems and methods
US9186585B2 (en) 1999-02-26 2015-11-17 Mq Gaming, Llc Multi-platform gaming systems and methods
US10300374B2 (en) 1999-02-26 2019-05-28 Mq Gaming, Llc Multi-platform gaming systems and methods
US9468854B2 (en) 1999-02-26 2016-10-18 Mq Gaming, Llc Multi-platform gaming systems and methods
US8708821B2 (en) 2000-02-22 2014-04-29 Creative Kingdoms, Llc Systems and methods for providing interactive game play
US8089458B2 (en) 2000-02-22 2012-01-03 Creative Kingdoms, Llc Toy devices and methods for providing an interactive play experience
US9149717B2 (en) 2000-02-22 2015-10-06 Mq Gaming, Llc Dual-range wireless interactive entertainment device
US10188953B2 (en) 2000-02-22 2019-01-29 Mq Gaming, Llc Dual-range wireless interactive entertainment device
US8368648B2 (en) 2000-02-22 2013-02-05 Creative Kingdoms, Llc Portable interactive toy with radio frequency tracking device
US8790180B2 (en) 2000-02-22 2014-07-29 Creative Kingdoms, Llc Interactive game and associated wireless toy
US8686579B2 (en) 2000-02-22 2014-04-01 Creative Kingdoms, Llc Dual-range wireless controller
US9579568B2 (en) 2000-02-22 2017-02-28 Mq Gaming, Llc Dual-range wireless interactive entertainment device
US10307671B2 (en) 2000-02-22 2019-06-04 Mq Gaming, Llc Interactive entertainment system
US9713766B2 (en) 2000-02-22 2017-07-25 Mq Gaming, Llc Dual-range wireless interactive entertainment device
US9474962B2 (en) 2000-02-22 2016-10-25 Mq Gaming, Llc Interactive entertainment system
US8164567B1 (en) 2000-02-22 2012-04-24 Creative Kingdoms, Llc Motion-sensitive game controller with optional display screen
US8169406B2 (en) 2000-02-22 2012-05-01 Creative Kingdoms, Llc Motion-sensitive wand controller for a game
US8184097B1 (en) 2000-02-22 2012-05-22 Creative Kingdoms, Llc Interactive gaming system and method using motion-sensitive input device
US8475275B2 (en) 2000-02-22 2013-07-02 Creative Kingdoms, Llc Interactive toys and games connecting physical and virtual play environments
US8814688B2 (en) 2000-02-22 2014-08-26 Creative Kingdoms, Llc Customizable toy for playing a wireless interactive game having both physical and virtual elements
US9814973B2 (en) 2000-02-22 2017-11-14 Mq Gaming, Llc Interactive entertainment system
US8491389B2 (en) 2000-02-22 2013-07-23 Creative Kingdoms, Llc. Motion-sensitive input device and interactive gaming system
US8915785B2 (en) 2000-02-22 2014-12-23 Creative Kingdoms, Llc Interactive entertainment system
US9931578B2 (en) 2000-10-20 2018-04-03 Mq Gaming, Llc Toy incorporating RFID tag
US9320976B2 (en) 2000-10-20 2016-04-26 Mq Gaming, Llc Wireless toy systems and methods for interactive entertainment
US8753165B2 (en) 2000-10-20 2014-06-17 Mq Gaming, Llc Wireless toy systems and methods for interactive entertainment
US10307683B2 (en) 2000-10-20 2019-06-04 Mq Gaming, Llc Toy incorporating RFID tag
US9480929B2 (en) 2000-10-20 2016-11-01 Mq Gaming, Llc Toy incorporating RFID tag
US8961260B2 (en) 2000-10-20 2015-02-24 Mq Gaming, Llc Toy incorporating RFID tracking device
US9162148B2 (en) 2001-02-22 2015-10-20 Mq Gaming, Llc Wireless entertainment device, system, and method
US8711094B2 (en) 2001-02-22 2014-04-29 Creative Kingdoms, Llc Portable gaming device and gaming system combining both physical and virtual play elements
US8913011B2 (en) 2001-02-22 2014-12-16 Creative Kingdoms, Llc Wireless entertainment device, system, and method
US9737797B2 (en) 2001-02-22 2017-08-22 Mq Gaming, Llc Wireless entertainment device, system, and method
US10179283B2 (en) 2001-02-22 2019-01-15 Mq Gaming, Llc Wireless entertainment device, system, and method
US8384668B2 (en) 2001-02-22 2013-02-26 Creative Kingdoms, Llc Portable gaming device and gaming system combining both physical and virtual play elements
US9393491B2 (en) 2001-02-22 2016-07-19 Mq Gaming, Llc Wireless entertainment device, system, and method
US8248367B1 (en) 2001-02-22 2012-08-21 Creative Kingdoms, Llc Wireless gaming system combining both physical and virtual play elements
US8827810B2 (en) 2002-04-05 2014-09-09 Mq Gaming, Llc Methods for providing interactive entertainment
US10010790B2 (en) 2002-04-05 2018-07-03 Mq Gaming, Llc System and method for playing an interactive game
US9272206B2 (en) 2002-04-05 2016-03-01 Mq Gaming, Llc System and method for playing an interactive game
US9463380B2 (en) 2002-04-05 2016-10-11 Mq Gaming, Llc System and method for playing an interactive game
US9616334B2 (en) 2002-04-05 2017-04-11 Mq Gaming, Llc Multi-platform gaming system using RFID-tagged toys
US8608535B2 (en) 2002-04-05 2013-12-17 Mq Gaming, Llc Systems and methods for providing an interactive game
US8702515B2 (en) 2002-04-05 2014-04-22 Mq Gaming, Llc Multi-platform gaming system using RFID-tagged toys
US10478719B2 (en) 2002-04-05 2019-11-19 Mq Gaming, Llc Methods and systems for providing personalized interactive entertainment
US8035629B2 (en) 2002-07-18 2011-10-11 Sony Computer Entertainment Inc. Hand-held computer interactive device
US9682320B2 (en) 2002-07-22 2017-06-20 Sony Interactive Entertainment Inc. Inertially trackable hand-held controller
US8570378B2 (en) 2002-07-27 2013-10-29 Sony Computer Entertainment Inc. Method and apparatus for tracking three-dimensional movements of an object using a depth sensing camera
US7737944B2 (en) 2002-07-27 2010-06-15 Sony Computer Entertainment America Inc. Method and system for adding a new player to a game in response to controller activity
US7782297B2 (en) 2002-07-27 2010-08-24 Sony Computer Entertainment America Inc. Method and apparatus for use in determining an activity level of a user in relation to a system
US9381424B2 (en) 2002-07-27 2016-07-05 Sony Interactive Entertainment America Llc Scheme for translating movements of a hand-held controller into inputs for a system
US9474968B2 (en) 2002-07-27 2016-10-25 Sony Interactive Entertainment America Llc Method and system for applying gearing effects to visual tracking
US7760248B2 (en) 2002-07-27 2010-07-20 Sony Computer Entertainment Inc. Selective sound source listening in conjunction with computer interactive processing
US8313380B2 (en) 2002-07-27 2012-11-20 Sony Computer Entertainment America Llc Scheme for translating movements of a hand-held controller into inputs for a system
US8797260B2 (en) 2002-07-27 2014-08-05 Sony Computer Entertainment Inc. Inertially trackable hand-held controller
US10406433B2 (en) 2002-07-27 2019-09-10 Sony Interactive Entertainment America Llc Method and system for applying gearing effects to visual tracking
US8976265B2 (en) 2002-07-27 2015-03-10 Sony Computer Entertainment Inc. Apparatus for image and sound capture in a game environment
US8188968B2 (en) 2002-07-27 2012-05-29 Sony Computer Entertainment Inc. Methods for interfacing with a program using a light input device
US9393487B2 (en) 2002-07-27 2016-07-19 Sony Interactive Entertainment Inc. Method for mapping movements of a hand-held controller to game commands
EP1552375B1 (en) * 2002-07-27 2006-09-20 Sony Computer Entertainment Inc. Man-machine interface using a deformable device
US10099130B2 (en) 2002-07-27 2018-10-16 Sony Interactive Entertainment America Llc Method and system for applying gearing effects to visual tracking
US10220302B2 (en) 2002-07-27 2019-03-05 Sony Interactive Entertainment Inc. Method and apparatus for tracking three-dimensional movements of an object using a depth sensing camera
US8686939B2 (en) 2002-07-27 2014-04-01 Sony Computer Entertainment Inc. System, method, and apparatus for three-dimensional input control
US9682319B2 (en) 2002-07-31 2017-06-20 Sony Interactive Entertainment Inc. Combiner method for altering game gearing
US8226493B2 (en) 2002-08-01 2012-07-24 Creative Kingdoms, Llc Interactive play devices for water play attractions
US9177387B2 (en) 2003-02-11 2015-11-03 Sony Computer Entertainment Inc. Method and apparatus for real time motion capture
US10022624B2 (en) 2003-03-25 2018-07-17 Mq Gaming, Llc Wireless interactive game having both physical and virtual elements
US9993724B2 (en) 2003-03-25 2018-06-12 Mq Gaming, Llc Interactive gaming toy
US9039533B2 (en) 2003-03-25 2015-05-26 Creative Kingdoms, Llc Wireless interactive game having both physical and virtual elements
US9707478B2 (en) 2003-03-25 2017-07-18 Mq Gaming, Llc Motion-sensitive controller and associated gaming applications
US8961312B2 (en) 2003-03-25 2015-02-24 Creative Kingdoms, Llc Motion-sensitive controller and associated gaming applications
US9770652B2 (en) 2003-03-25 2017-09-26 Mq Gaming, Llc Wireless interactive game having both physical and virtual elements
US10369463B2 (en) 2003-03-25 2019-08-06 Mq Gaming, Llc Wireless interactive game having both physical and virtual elements
US9446319B2 (en) 2003-03-25 2016-09-20 Mq Gaming, Llc Interactive gaming toy
US9393500B2 (en) 2003-03-25 2016-07-19 Mq Gaming, Llc Wireless interactive game having both physical and virtual elements
US8373659B2 (en) 2003-03-25 2013-02-12 Creative Kingdoms, Llc Wirelessly-powered toy for gaming
US8072470B2 (en) 2003-05-29 2011-12-06 Sony Computer Entertainment Inc. System and method for providing a real-time three-dimensional interactive environment
US8251820B2 (en) 2003-09-15 2012-08-28 Sony Computer Entertainment Inc. Methods and systems for enabling depth and direction detection when interfacing with a computer program
US7646372B2 (en) 2003-09-15 2010-01-12 Sony Computer Entertainment Inc. Methods and systems for enabling direction detection when interfacing with a computer program
US8758132B2 (en) 2003-09-15 2014-06-24 Sony Computer Entertainment Inc. Methods and systems for enabling depth and direction detection when interfacing with a computer program
US7874917B2 (en) 2003-09-15 2011-01-25 Sony Computer Entertainment Inc. Methods and systems for enabling depth and direction detection when interfacing with a computer program
US7883415B2 (en) 2003-09-15 2011-02-08 Sony Computer Entertainment Inc. Method and apparatus for adjusting a view of a scene being displayed according to tracked head motion
US8303411B2 (en) 2003-09-15 2012-11-06 Sony Computer Entertainment Inc. Methods and systems for enabling depth and direction detection when interfacing with a computer program
US8062126B2 (en) 2004-01-16 2011-11-22 Sony Computer Entertainment Inc. System and method for interfacing with a computer program
US7663689B2 (en) 2004-01-16 2010-02-16 Sony Computer Entertainment Inc. Method and apparatus for optimizing capture device settings through depth information
WO2005116809A2 (en) 2004-05-24 2005-12-08 3D For All Számítástechnikai Fejlesztö Kft. System and method for operating in virtual 3d space and system for selecting an operation via a visualizing system
US10099147B2 (en) 2004-08-19 2018-10-16 Sony Interactive Entertainment Inc. Using a portable device to interface with a video game rendered on a main display
US8547401B2 (en) 2004-08-19 2013-10-01 Sony Computer Entertainment Inc. Portable augmented reality device and method
US9675878B2 (en) 2004-09-29 2017-06-13 Mq Gaming, Llc System and method for playing a virtual game by sensing physical movements
WO2007050885A2 (en) * 2005-10-26 2007-05-03 Sony Computer Entertainment America Inc. System and method for interfacing with a computer program
WO2007050885A3 (en) * 2005-10-26 2007-12-06 Sony Comp Entertainment Us System and method for interfacing with a computer program
US10279254B2 (en) 2005-10-26 2019-05-07 Sony Interactive Entertainment Inc. Controller having visually trackable object for interfacing with a gaming system
US20090298590A1 (en) * 2005-10-26 2009-12-03 Sony Computer Entertainment Inc. Expandable Control Device Via Hardware Attachment
US9573056B2 (en) * 2005-10-26 2017-02-21 Sony Interactive Entertainment Inc. Expandable control device via hardware attachment
US8210943B1 (en) 2006-05-06 2012-07-03 Sony Computer Entertainment America Llc Target interface
US8827804B2 (en) 2006-05-06 2014-09-09 Sony Computer Entertainment America Llc Target interface
US8310656B2 (en) 2006-09-28 2012-11-13 Sony Computer Entertainment America Llc Mapping movements of a hand-held controller to the two-dimensional image plane of a display screen
US8781151B2 (en) 2006-09-28 2014-07-15 Sony Computer Entertainment Inc. Object detection using video input combined with tilt angle information
US9526995B2 (en) 2006-11-22 2016-12-27 Sony Interactive Entertainment America Llc Video game recording and playback with visual display of game controller manipulation
US10099145B2 (en) 2006-11-22 2018-10-16 Sony Interactive Entertainment America Llc Video game recording and playback with visual display of game controller manipulation
WO2009059716A1 (en) * 2007-11-05 2009-05-14 Sebastian Repetzki Pointing device and method for operating the pointing device
US8542907B2 (en) 2007-12-17 2013-09-24 Sony Computer Entertainment America Llc Dynamic three-dimensional object mapping for user-defined control device
US8840470B2 (en) 2008-02-27 2014-09-23 Sony Computer Entertainment America Llc Methods for capturing depth data of a scene and applying computer actions
US8368753B2 (en) 2008-03-17 2013-02-05 Sony Computer Entertainment America Llc Controller with an integrated depth camera
US8323106B2 (en) 2008-05-30 2012-12-04 Sony Computer Entertainment America Llc Determination of controller three-dimensional location using image analysis and ultrasonic communication
US9295912B2 (en) 2008-07-13 2016-03-29 Sony Computer Entertainment America Llc Game aim assist
US8342926B2 (en) 2008-07-13 2013-01-01 Sony Computer Entertainment America Llc Game aim assist
US10035064B2 (en) 2008-07-13 2018-07-31 Sony Interactive Entertainment America Llc Game aim assist
US8287373B2 (en) 2008-12-05 2012-10-16 Sony Computer Entertainment Inc. Control device for communicating visual information
US8761434B2 (en) 2008-12-17 2014-06-24 Sony Computer Entertainment Inc. Tracking system calibration by reconciling inertial data with computed acceleration of a tracked object in the three-dimensional coordinate system
US8970707B2 (en) 2008-12-17 2015-03-03 Sony Computer Entertainment Inc. Compensating for blooming of a shape in an image
US8253801B2 (en) 2008-12-17 2012-08-28 Sony Computer Entertainment Inc. Correcting angle error in a tracking system
US8527657B2 (en) 2009-03-20 2013-09-03 Sony Computer Entertainment America Llc Methods and systems for dynamically adjusting update rates in multi-player network gaming
US8342963B2 (en) 2009-04-10 2013-01-01 Sony Computer Entertainment America Inc. Methods and systems for enabling control of artificial intelligence game characters
US8142288B2 (en) 2009-05-08 2012-03-27 Sony Computer Entertainment America Llc Base station movement detection and compensation
US8393964B2 (en) 2009-05-08 2013-03-12 Sony Computer Entertainment America Llc Base station for position location
US8961313B2 (en) 2009-05-29 2015-02-24 Sony Computer Entertainment America Llc Multi-positional three-dimensional controller
US9058063B2 (en) 2009-05-30 2015-06-16 Sony Computer Entertainment Inc. Tracking system calibration using object position and orientation
US9678583B2 (en) * 2013-07-23 2017-06-13 University Of Kentucky Research Foundation 2D and 3D pointing device based on a passive lights detection operation method using one camera
US20150029100A1 (en) * 2013-07-23 2015-01-29 University Of Kentucky Research Foundation 2d and 3d pointing device based on a passive lights detection operation method using one camera

Also Published As

Publication number Publication date
GB0207300D0 (en) 2002-05-08

Similar Documents

Publication Publication Date Title
Oh et al. Laser pointers as collaborative pointing devices
Harrington Computer graphics
Poupyrev et al. Developing a generic augmented-reality interface
US8013843B2 (en) Method for providing human input to a computer
Koike et al. Integrating paper and digital information on EnhancedDesk: a method for realtime finger tracking on an augmented desk system
US5982352A (en) Method for providing human input to a computer
US8395620B2 (en) Method and system for tracking of a subject
Tory et al. Visualization task performance with 2D, 3D, and combination displays
CN101963840B (en) System and method for remote, virtual on screen input
CN102915112B (en) For the system and method for closely motion tracking
US8368647B2 (en) Three-dimensional virtual input and simulation apparatus
US7239302B2 (en) Pointing device and scanner, robot, mobile communication device and electronic dictionary using the same
US8427449B2 (en) Method for providing human input to a computer
DE102009025236B4 (en) Interface system for gesture recognition with a light-scattering screen
Zeleznik et al. UniCam-2 D gestural camera controls for 3 D environments
Ware Using hand position for virtual object placement
CN1698357B (en) Method for displaying an output image on an object
US9477312B2 (en) Distance based modelling and manipulation methods for augmented reality systems using ultrasonic gloves
Burdea et al. Virtual reality technology
US7407106B2 (en) Method and system for hiding visible infrared markings
US20050264527A1 (en) Audio-visual three-dimensional input/output
KR101620777B1 (en) Enhanced virtual touchpad and touchscreen
Hancock et al. Shallow-depth 3d interaction: design and evaluation of one-, two-and three-touch techniques
US7460106B2 (en) Method and apparatus for computer input using six degrees of freedom
US8970625B2 (en) Three-dimensional tracking of a user control device in a volume

Legal Events

Date Code Title Description
WAP Application withdrawn, taken to be withdrawn or refused ** after publication under section 16(1)