US8180114B2 - Gesture recognition interface system with vertical display - Google Patents

Gesture recognition interface system with vertical display Download PDF

Info

Publication number
US8180114B2
US8180114B2 US12133836 US13383608A US8180114B2 US 8180114 B2 US8180114 B2 US 8180114B2 US 12133836 US12133836 US 12133836 US 13383608 A US13383608 A US 13383608A US 8180114 B2 US8180114 B2 US 8180114B2
Authority
US
Grant status
Grant
Patent type
Prior art keywords
input object
plurality
images
gesture recognition
based
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active, expires
Application number
US12133836
Other versions
US20080244468A1 (en )
Inventor
H. Keith Nishihara
Shi-Ping Hsu
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Northrop Grumman Systems Corp
Original Assignee
Northrop Grumman Systems Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Grant date

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/041Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
    • G06F3/0416Control and interface arrangements for touch screen
    • G06F3/0418Control and interface arrangements for touch screen for error correction or compensation, e.g. parallax, calibration, alignment
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017Gesture based interaction, e.g. based on a set of recognized hand gestures
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/041Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
    • G06F3/042Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means
    • G06F3/0421Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means by interrupting or reflecting a light beam, e.g. optical touch-screen
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/041Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
    • G06F3/042Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means
    • G06F3/0425Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/041Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
    • G06F3/042Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means
    • G06F3/0428Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means by sensing at the edges of the touch surface the interruption of optical paths, e.g. an illumination plane, parallel to the touch surface which may be virtual
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
    • G06F3/0488Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
    • G06F3/04883Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures for entering handwritten data, e.g. gestures, text
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06KRECOGNITION OF DATA; PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
    • G06K9/00Methods or arrangements for reading or recognising printed or written characters or for recognising patterns, e.g. fingerprints
    • G06K9/00335Recognising movements or behaviour, e.g. recognition of gestures, dynamic facial expressions; Lip-reading
    • G06K9/00355Recognition of hand or arm movements, e.g. recognition of deaf sign language
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/048Indexing scheme relating to G06F3/048
    • G06F2203/04808Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen

Abstract

One embodiment of the invention includes a gesture recognition interface system. The system may comprise a substantially vertical surface configured to define a gesture recognition environment based on physical space in a foreground of the substantially vertical surface. The system may also comprise at least one light source positioned to provide illumination of the gesture recognition environment. The system also comprises at least two cameras configured to generate a plurality of image sets based on the illumination being reflected from an input object in the gesture recognition environment. The system further comprises a controller configured to determine a given input gesture based on changes in relative locations of the input object in each of the plurality of image sets. The controller may further be configured to initiate a device input associated with the given input gesture.

Description

RELATED APPLICATIONS

This application is a continuation-in-part of U.S. patent application Ser. No. 11/485,788, filed Jul. 13, 2006. The subject matter of the aforementioned application is incorporated herein by reference in its entirety.

TECHNICAL FIELD

The present invention relates generally to interface systems, and specifically to a gesture recognition interface system with vertical display

BACKGROUND

As the range of activities accomplished with a computer increases, new and innovative ways to provide an interface with a computer are often developed to complement the changes in computer functionality and packaging. For example, touch sensitive screens can allow a user to provide inputs to a computer without a mouse and/or a keyboard, such that desk area is not needed to operate the computer. Examples of touch sensitive screens include pressure sensitive membranes, beam break techniques with circumferential light sources and sensors, and acoustic ranging techniques. However, these types of computer interfaces can only provide information to the computer regarding the touch event, itself, and thus can be limited in application. In addition, such types of interfaces can be limited in the number of touch events that can be handled over a given amount of time, and can be prone to interpret unintended contacts, such as from a shirt cuff or palm, as touch events. Furthermore, touch sensitive screens can be prohibitively expensive and impractical for very large display sizes, such as those used for presentations.

SUMMARY

One embodiment of the invention includes a gesture recognition interface system. The system may comprise a substantially vertical surface configured to define a gesture recognition environment based on physical space in a foreground of the substantially vertical surface. The system may also comprise at least one light source positioned to provide illumination of the gesture recognition environment. The system also comprises at least two cameras configured to generate a plurality of image sets based on the illumination being reflected from an input object in the gesture recognition environment. The system further comprises a controller configured to determine a given input gesture based on changes in relative locations of the input object in each of the plurality of image sets. The controller may further be configured to initiate a device input associated with the given input gesture.

Another embodiment of the invention includes a method of providing device inputs. The method comprises displaying visual content on a substantially vertical display surface, illuminating a gesture recognition environment that is defined by a physical space in a foreground of the substantially vertical display surface, and providing simulated inputs in the gesture recognition environment via gestures associated with an input object. The method also includes generating a first plurality of images and a second plurality of images associated with the input object substantially concurrently based on reflected light from the input object. The method also includes detecting features associated with the input object in each of corresponding pairs of the first and second plurality of images. The method also includes determining a plurality of three-dimensional physical locations of the input object based on a relative separation of the correlated points in the sequence of points of each of the corresponding pairs of the first and second plurality of images and determining if changes in the plurality of three-dimensional physical locations of the input object correspond to any of a plurality of pre-defined gestures. The method further comprises providing at least one device input based on a given one of the plurality of pre-defined gestures upon determining that the changes in the plurality of three-dimensional physical locations of the input object correspond to the given one of the plurality of pre-defined gestures.

Another embodiment of the invention includes a gesture recognition system. The system includes means for providing a brightness contrast between reflected light and non-reflected light in a gesture recognition environment that is defined by a physical space in front of a substantially vertical display surface. The reflected light can be reflected from an input object. The system also includes means for generating a first plurality of images of the input object based on the brightness contrast and means for generating a second plurality of images of the input object based on the brightness contrast. The first plurality of images and the second plurality of images can form a plurality of matched pairs of images of the input object. The system also includes means for generating a sequence of points associated with elongated portions of the input object in each of the first and second plurality of images and for correlating at least one pair of corresponding points in each of the plurality of matched pairs of images of the input object. The system also includes means for determining three-dimensional physical location information associated with features of the input object based on a relative separation of the correlated at least one pair of corresponding points in the plurality of matched pairs of images of the input object. The system further includes means for translating changes in the three-dimensional location information associated with the features of the input object to a given input gesture and means for providing device inputs based on matching the given input gesture with one of a plurality of pre-defined gestures.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example of a gesture recognition interface system in accordance with an aspect of the invention.

FIG. 2 illustrates an example of a camera system for a gesture recognition interface system in accordance with an aspect of the invention.

FIG. 3 illustrates an example of an illumination system for a gesture recognition interface system in accordance with an aspect of the invention.

FIG. 4 illustrates another example of a gesture recognition interface system in accordance with an aspect of the invention.

FIG. 5 illustrates an example of a diagram of hand image conversions based on an algorithm for use in a gesture recognition interface system in accordance with an aspect of the invention.

FIG. 6 illustrates an example of a diagram of parallax separation of converted hand images for use in a gesture recognition interface system in accordance with an aspect of the invention.

FIG. 7 illustrates an example of a gesture application in accordance with an aspect of the invention.

FIG. 8 illustrates an example of a method for providing inputs to a computer in accordance with an aspect of the invention.

DETAILED DESCRIPTION

The present invention relates generally to interface systems, and specifically to a gesture recognition interface system with vertical display. A user employs an input object to provide simulated inputs to a computer or other electronic device. It is to be understood that the simulated inputs are provided by gestures using the input object. For example, the user could provide gestures that include motion and/or contact with a vertical display surface using the input object. The input object could be, for example, the user's hand; a wand, stylus, pointing stick; or a variety of other devices with which the user can gesture. The simulated inputs could be, for example, simulated mouse inputs. One or more infrared (IR) light sources illuminate a gesture recognition environment that is defined by the area of physical space in front of the vertical display surface. A set of stereo cameras can each generate a plurality of images of the input object. The plurality of images can be, for example, based on a reflected light contrast of the IR light reflected back from the input object relative to substantially non-reflected light. The plurality of images of the input object from each camera could be, for example, a plurality of matched sets of images of the input object, such that each image in the matched set of images corresponds to the input object from a different perspective at substantially the same time. A given matched set of images can be employed to determine a location of the input object and the plurality of matched sets of images can be employed to determine physical motion of the input object.

A controller can be configured to receive the plurality of images to determine three-dimensional location information associated with the input object. For example, the controller could apply an algorithm to determine features of the input object, such as endpoints, length, and pitch of elongated portions of the input object in three-dimensional space. The controller could then translate the simulated inputs into device inputs based on the three-dimensional location information. For example, the controller could interpret gesture inputs based on motion associated with the input object and translate the gesture inputs into inputs to a computer or other device. The controller could also compare the motion associated with the one or more endpoints of the input object with a plurality of pre-defined gestures stored in a memory, such that a match with a given pre-defined gesture could correspond with a particular device input.

FIG. 1 illustrates an example of a gesture recognition interface system 10 in accordance with an aspect of the invention. The gesture recognition interface system 10 includes a first camera 12, a second camera 14, a first IR light source 16, and a second IR light source 18 mounted above a vertical display surface 20. As an example, the vertical display surface 20 can be a projection screen. The first camera 12 and the second camera 14 may each include an IR filter, such that the respective camera may only be able to receive IR light. The first IR light source 16 and the second IR light source 18 each illuminate a gesture recognition environment 22 that is defined as the three-dimensional physical space in the foreground of the vertical display surface 20 that is visible by the first and second cameras 12 and 14.

An input object 24 can provide simulated inputs over the vertical display surface 20. In the example of FIG. 1, the input object 24 is demonstrated as a user's hand, such that the simulated inputs can be provided through hand gestures. It is to be understood that the use of a hand to provide simulated inputs via hand gestures is but one example implementation of the gesture recognition interface system 10. Examples of other types of input objects could include a stylus, wand, pointing stick, or any of a variety of devices that could provide gestures to simulate inputs. It is to be further understood that the input object 24 need not be specially designed or suited for use in the gesture recognition interface system 10. For example, a user's naked hand could be used as the input object 24, and thus a user need not wear a glove that includes retroreflective material or one or more position sensors to provide gesture inputs to the gesture recognition interface system 10 in accordance with an aspect of the invention.

In the example of FIG. 1, the first camera 12 and the second camera 14 each receive separate images of the input object 24, where each of the separate images received, respectively, by the first camera 12 and the second camera 14 are a matched set (i.e., matched pair). As an example, each of the first camera 12 and the second camera 14 could rapidly take still photograph images at, for example, sixty times per second, such that each still photograph image taken by the first camera 12 is matched to a still photograph image taken by the second camera 14 at substantially the same time. The input object 24 can appear to be in a different location in each image of the matched set captured by each of the first camera 12 and the second camera 14, respectively, due to parallax caused by the different mounted locations of each of the first camera 12 and the second camera 14. In the example of FIG. 1, the first and second cameras 12 and 14 can each be positioned as angled toward the center of the vertical display surface 20, such as to provide for more accurate position determination of the input object 24.

The images received by each of the first and second cameras 12 and 14 can be based on IR light that is reflected from the input object relative to substantially non-reflected light in the gesture recognition environment 22. Specifically, an object is illuminated at a relative brightness intensity that is 1/D2, where D is the distance from the light source. Thus, an object that is twice as far away as another appears four times dimmer. Accordingly, although some of the IR light emitted from the first and second IR light sources 16 and 18 may be reflected from the floor 28 beneath the vertical display surface 20, the intensity of the reflected light may be significantly less than that reflected from the input object 24.

The first camera 12 and the second camera 14 can each provide their respective separate images of the input object 24 to a controller 26. The controller 26 could reside, for example, within a computer (not shown) for which the gesture recognition interface system 10 is designed to provide a gesture recognition interface. It is to be understood, however, that the hosting of a controller is not limited to a standalone computer, but could be included in embedded processors. The controller 26 can process the respective images associated with the input object 24 to generate three-dimensional location data associated with the input object 24.

For example, each of the first camera 12 and the second camera 14 could each be mounted at pre-determined angles relative to the floor 28 beneath the vertical display surface 20. For a given matched pair of images of the input object 24, if the pre-determined angles of each of the cameras 12 and 14 are equal, then each point of the input object 24 in two-dimensional space in a given image from the camera 12 is equidistant from a corresponding point of the input object 24 in the respective matched image from the camera 14. As such, the controller 26 could determine the three-dimensional physical location of the input object 24 based on a relative parallax separation of the matched set of images of the input object 24 at a given time. In addition, using a computer algorithm, the controller 26 could also determine the three-dimensional physical location of features associated with portions of the input object 24, such as fingers and fingertips.

The gesture recognition interface system 10 can also include a projector 30. The projector 30 can provide visual content with which the user can interact and provide inputs. In the example of FIG. 1, the projector 30 can project the visual content onto the vertical display surface 20. Because the IR light sources 16 and 18 do not illuminate visible light, the IR illumination may not interfere with the visual content projected from the projector 30. The user can thus employ the input object 24 in the gesture recognition environment 22 to simulate inputs in an interactive manner with the visual content. It is to be understood that, as described herein, interaction with the visual content via gestures is defined as providing the simulated inputs through gestures in response to the visual content or to generate the visual content, as described in greater detail in the example of FIG. 7 below.

As an example, the simulated inputs can be mouse inputs, such that the controller 26 could interpret two-dimensional motion of an end-point of the input object 24 across the vertical display surface 20 as a mouse cursor and a touch of the vertical display surface 20 by the end-point of the input object 24 as a left mouse-click. Accordingly, a user of the gesture recognition interface system 10 could navigate through a number of computer menus associated with a computer merely by moving his or her fingertip through the air above the vertical display surface 20 and by touching icons projected onto the vertical display surface 20. As another example, the vertical display surface 20 can act as a virtual canvas, such that the visual content is projected as virtual ink that is provided on the virtual canvas in response to touching the vertical display surface 20 with the input object 24. In addition, additional gestures can be implemented to erase the virtual ink or change its properties, such as color or pen-width. Accordingly, the gesture recognition interface system 10 can act as a virtual “dry-erase board” on which users can draw or write.

The controller 26 includes a calibration unit 32 for calibrating the gesture recognition interface system 10. The calibration unit 32 could be a separate system working in conjunction with the controller 26, or could merely be a software routine residing within the controller 26. As an example, the calibration unit 32 can implement a calibration routine, such as interacting with software of the controller 26, at power-up or setup of the gesture recognition interface system 10, or at periodic intervals. A calibration routine can be merely a prompt for a user to touch a specific point on the vertical display surface 20 with an endpoint of the input object 24. The specific point can be one or more points having predetermined coordinates, as identified by the controller 26 and/or the calibration unit 32, on the vertical display surface 20. As an example, the specific point can be a permanent visual mark on the vertical display surface 20 or can be projected onto the vertical display surface 20 by the projector 30. As a result, the calibration unit 32 can associate the images of the input object 24 touching the specific point with the predetermined coordinates of the specific point, thus calibrating the images to allow accurate gesture interpretation.

It is to be understood that the gesture recognition interface system 10 in the example of FIG. 1 is intended to represent but one example of a gesture recognition interface system. For example, the gesture recognition interface system 10 could include more than two cameras, in multiple locations above and/or below the vertical display surface 20 that each supply respective images of the input object 24 to the controller 26. As an example, a three-way stereo (i.e., trinocular) camera system can be employed, or multiple cameras can be implemented to increase visual coverage for a wide vertical display surface 20 or in situations where head-room above the vertical display surface 20 is insufficient. In addition, as another example, the projector 30 could project the visual content from behind the vertical display surface 20, or the vertical display surface 20 could be a liquid crystal display (LCD) or plasma monitor instead of a projection screen. As a further example, the IR light sources 16 and 18 may not illuminate in the IR spectrum, but could instead illuminate in a different spectrum, such as narrow frequency bands of visible light, with each of the respective cameras 12 and 14 having a corresponding spectrum filter. Accordingly, the gesture recognition interface system 10 can be configured in any of a variety of ways.

FIG. 2 illustrates an example of a camera system 50 for a gesture recognition interface system in accordance with an aspect of the invention. The camera system 50 can be implemented in the gesture recognition interface system 10 in the example of FIG. 1. Therefore, reference is to be made to the gesture recognition interface system 10 in the example of FIG. 1 in the following discussion of the example of FIG. 2.

In the example of FIG. 2, the camera system 50 is demonstrated in a side-view with respect to the vertical display surface 20. Thus, the camera system 50 demonstrates positioning of a camera 52, which can be representative of one of the cameras 12 and 14 in the example of FIG. 1. Specifically, as demonstrated in the example of FIG. 2, the camera 52 is positioned above the vertical display surface 20 such that a central lens axis 54 of the camera 52 has a glancing angle of θ with respect to a plane that is defined by the vertical display surface 20. As described herein, the central lens axis 54 is defined as an axis projecting from the center of the lens of the camera 52. As an example, the glancing angle θ can be a substantially small, non-zero angle. As another example, the glancing angle θ can be zero, such that the camera 52 can be positioned closer to the vertical display surface 20.

As a result of the positioning of the cameras 12 and 14 relative to the vertical display surface 20, as demonstrated by the camera 52 in the example of FIG. 2, both the vertical display surface 20 and the gesture recognition environment 22 in front of the vertical display surface 20 can be imaged by the cameras 12 and 14. Specifically, the position of the camera 52 in the example of FIG. 2 demonstrates that the cameras 12 and 14 are configured to image the vertical display screen 20 and a volume of physical space in a foreground of the vertical display surface 20 (e.g., approximately 10 to 20 inches of foreground) that is sufficient to capture input gestures with the input object 24 without necessarily requiring a touch of the vertical display screen 20. Therefore, based on the positioning of the camera 52, the controller 26 can calculate the three-dimensional position of the input object 24 from an analysis of the image positions and parallax between the cameras 12 and 14. Furthermore, the arrangement of the cameras 12 and 14 can allow for a higher precision measurement of the distance between the vertical display surface 20 and the input object 24 than is possible with the central lens axis 54 being substantially normal to the vertical display surface 20.

FIG. 3 illustrates an example of an illumination system 100 for a gesture recognition interface system in accordance with an aspect of the invention. The illumination system 100 can be implemented in the gesture recognition interface system 10 in the example of FIG. 1. Therefore, reference is to be made to the gesture recognition interface system 10 in the example of FIG. 1 in the following discussion of the example of FIG. 3.

The illumination system 100 includes a plurality of illuminators 102 that are positioned above the vertical display surface 20. As an example, the illuminators 102 can be IR illuminators. The illuminators 102 are divided into a first portion 104 and a second portion 106. Each of the first and second portions 104 and 106 can include one or more of the illuminators 102. As such, the illuminators 102 can be distributed across the top of the vertical display surface 20 to illuminate the gesture recognition environment 22 in a substantially uniform manner. In addition, the illuminators 102 can be positioned as having a large incidence angle (e.g., approximately 90°) with respect to a plane that is defined by the vertical display surface 20.

The illuminators 102 in the first and second portions 104 and 106 are each positioned with an angled central illumination axis 108. As described herein, the central illumination axis 108 is defined as an axis projecting from the center of the IR light source of the illuminator 102. Specifically, the illuminators 102 in the first portion 104 are positioned at an angle φ with respect to a normal axis 110 of the floor 28 (not shown in the example of FIG. 2) toward the center of the vertical display surface 20. Likewise, the illuminators 102 in the second portion 106 are positioned at an angle φ with respect to the normal axis 110 of the floor 28 toward the center of the vertical display surface 20, such that the first portion 104 and the second portion 106 of the illuminators 102 are mirror images.

Because of the angled positioning of the illuminators 102 with respect to the normal axis 110, direct illumination of the floor 28 beneath the vertical display surface 20 is mitigated. Specifically, the illuminators 102 illuminate the floor 28 at an angle of incidence of φ. As a result, significantly less illumination is reflected from the floor 28 and into the gesture recognition environment 22. Therefore, the brightness contrast between the light reflected from the input object 24 and the non-reflected light in the physical space of the gesture recognition environment is greater. Accordingly, edges of the input object 24 can be more distinctly identified in the images that are provided from the first and second cameras 12 and 14, such that features associated with the input object 24 can be detected more accurately by an algorithm that is implemented by the controller 26.

In addition, because the illuminators 102 are positioned as having a large angle of incidence with respect to the plane of the vertical display surface 20, the vertical display surface 20 can appear substantially dimmer to the cameras 12 and 14 relative to the input object 24 with respect to the reflected IR light. Furthermore, the vertical display surface 20 in the illumination system 100, and thus the gesture recognition interface system 10, can be configured as retroreflective. As such, light that is incident on the vertical display surface 20 is substantially reflected back to the illumination source. However, retroreflectivity may be substantially diminished at larger incidence angles. As a result of the retroreflectivity of the vertical display surface 20, IR light that is emitted from the illuminators 102, or that is reflected from the input object 24 onto the vertical display surface 20, is thus not reflected to the cameras 12 and 14. Accordingly, although the vertical display surface 20 is imaged by the cameras 12 and 14, as described above in the example of FIG. 2, the vertical display surface 20 can remain substantially dimmer to the cameras 12 and 14 relative to the input object 24 with respect to the reflected IR light.

FIG. 4 illustrates another example of a gesture recognition interface system 150 in accordance with an aspect of the invention. The gesture recognition interface system 150 can be substantially similar to the gesture recognition interface system 10 in the example of FIG. 1. As such, reference is to be made to the example of FIG. 1 in the following discussion of the example of FIG. 4.

The gesture recognition interface system 150 includes the controller 26, the first camera 12, and the second camera 14. The first and second cameras 12 and 14 each receive a plurality of images of the input object 24, such as a user's hand. The respective images of the input object 24 could be images based on a brightness contrast between light reflected from the input object 24 and non-reflected light in the gesture recognition environment 22.

The first and second cameras 12 and 14 each input their respective images of a matched pair of images into a digitizer 152. The digitizer 152 produces digitized versions of the images of the input object. The digitized images of the input object 24 are input to an image comparator 154. The image comparator 154 compares each of the digitized images of the input object to a previously stored digitized image of the input object to generate a binarized image of the input object 24. Such a comparison allows for an improved quality of the digitized images when the IR illumination of the gesture recognition environment 22 is not uniform across the vertical display surface 20. The previously stored digitized image could have been captured during a calibration operation and/or from repeatedly storing the digitized image in memory buffers.

As an example, a background model can be maintained for each of the cameras 12 and 14 without the input object 24 being present. The background model images can be used to decide at each pixel whether the images of the input object 24 correspond to a binary 1 or 0. For example, at each pixel location, if the image of the input object 24 has a value that is approximately greater than the corresponding background model image times a threshold scaling value of between 0 and 1, the output value will be a binary 1, thus denoting the presence of the input object 24. In this manner, the scaling value can be selected to provide an optimal balance between desirably detecting the input object 24 while being substantially insensitive to residual reflected illumination back to the first and second cameras 12 and 14. As a result, edges associated with the input object 24 can be detected in each of the images.

The contrast enhanced binarized images of the input object 24 are then each input to a skeletal representation object detection algorithm device 156. The object detection algorithm device 156 can be an integrated circuit (IC) or set of ICs within the controller 26, or could be a software routine residing in the controller 26. The object detection algorithm device 156 can be configured to detect features associated with the input object 24 in each of the binarized images. As an example, the object detection algorithm device 156 can detect length, inclination, and/or endpoints associated with elongated portions of the input object 24.

As an example, the object detection algorithm device 156 can be configured to determine a location of a sequence of mid-points associated with edges of the elongated portions of the input object 24. The mid-points can be symmetric points that are approximately midway between two boundary points at edges of the input object 24 in the image where there are no other boundary points present in a two-dimensional coordinate system. A linking algorithm can thus connect the symmetric points into chains, from which straight segments that correspond to the elongated portions can be parsed. The symmetric points can be annotated with a radial distance parameter which describes a ball centered on that point which fits within the boundaries of the image. The chains and segment structures can thus inherit the radial distance parameter to form the skeletal representation of the input object 24, from which the features of the elongated portions can be derived.

FIG. 5 illustrates an example of a diagram 200 of hand image conversions based on the skeletal representation object detection algorithm in accordance with an aspect of the invention. A first hand image 202 is demonstrated as having an extended index finger from a substantially closed fist. The object detection algorithm device 156 can thus convert the first hand image 202 into a skeletal representation 203. Specifically, the object detection algorithm device 156 can determine the sequence of mid-points along the elongated portions (i.e., arm and index finger), demonstrated in the example of FIG. 5 as dotted lines 204 and 206 corresponding to symmetric axes of the index finger and the arm, respectively. The dotted lines 204 and 206 are separated by a ball 208 that corresponds to the fist. As a result, the object detection algorithm device 156 can ascertain endpoints of the symmetric axes of the arm and the index finger, demonstrated respectively by the “X” 210 and the “X” 212, in two-dimensional space in each of the images.

A second hand image 214 is demonstrated as having all fingers and the thumb extended. The object detection algorithm device 156 can thus convert the second hand image 214 into a skeletal representation 215. Specifically, the object detection algorithm device 156 can determine the sequence of mid-points along the elongated portions (i.e., arm, thumb, and all fingers), demonstrated in the example of FIG. 5 as dotted lines 216 corresponding to symmetric axes of each of the arm, thumb, and fingers. The dotted lines 216 extend from overlapping balls 218 that correspond to the palm. As a result, the object detection algorithm device 156 can ascertain endpoints of the symmetric axes of the arm, thumbs, and fingers, demonstrated by an “X” 220 at the end of each, in two-dimensional space in each of the images.

Referring back to the example of FIG. 4, the skeletal representation images of the input object 24 are provided to a calibration data and location resolver 158. The calibration data and location resolver 158 determines a three-dimensional location of the input object 24, as well as all associated features, at a given time. As an example, the calibration data and location resolver 158 can be configured to compare the relative two-dimensional locations of the corresponding points in each of the skeletal representation images of the input object 24 and to interpolate a three-dimensional location of the points based on a parallax separation of the corresponding points.

FIG. 6 illustrates an example of a diagram 250 of parallax separation of converted hand images in accordance with an aspect of the invention. The diagram 250 includes a first converted image 252 and a second converted image 254 of the input object 24, demonstrated in the example of FIG. 6 as a user's hand. The converted images 252 and 254 are thus demonstrated as skeletal representations of the input object 24, such as described above in the example of FIG. 5. The first converted image 252 could have been received by the first camera 12 and the second converted image 254 could have been received by the second camera 14. The calibration data and location resolver 158 can thus receive both of the converted images 252 and 254 concurrently as a composite image superimposed together. Due to parallax caused by the separate locations of the cameras 12 and 14, an end-point 256 of the first converted image 252 and an end-point 258 of the second converted image 254 are demonstrated in the example of FIG. 6 as spaced apart from each other by a distance X. The distance X is thus demonstrative of a stereo disparity of the input object 24, such that the distance X represents a difference in position of the end-points 256 and 258 in the respective converted images 252 and 254.

Differing values of the stereo disparity X can denote changes in height associated with the user's fingertip, as represented by the end-points 256 and 258. For example, as X increases, the user's fingertip is moved vertically higher (e.g., closer to the cameras 12 and 14), and as X decreases, the user's fingertip is moved vertically lower (e.g., further from the cameras 12 and 14). Therefore, the calibration data and location resolver 68 interpolates the three-dimensional location of the endpoints of the input object based on parallax separation. The controller 26 can be calibrated to know which values of X correspond to the height of the user's fingertip relative to the point on the vertical display surface 20 having a predetermined location.

As an example, the corresponding points that are measured by the calibration data and location resolver 158 could be one or more of the end-points, such as the end-points 210, 212, and 220 in the example of FIG. 5. As another example, the corresponding points that are measured by the calibration data and location resolver 158 could be any of the mid-points along the elongated portions, such as mid-points 204, 206, and 216 in the example of FIG. 5. The gesture recognition interface system 150 can be calibrated to identify the amount of physical separation of the two-dimensional images that corresponds to a height, as well as lateral and proximal locations, of the features of the input object 24 relative to the vertical display surface 20. Accordingly, the calibration data and location resolver 158 can detect a two-dimensional position of the input object 24 with respect to the vertical display surface 20, as well as a touch of an endpoint of the input object 24 (e.g., the user's fingertip) to the vertical display surface 20.

The data output from the calibration data and location resolver 158 is input to a gesture recognition device 160. The gesture recognition device 160 interprets the three-dimensional location data associated with the one or more endpoints of the input object and translates changes in the location data into an input gesture. Because the gesture recognition device 160 implements the location data associated with the input object, it can be programmed to recognize any of a variety of gestures that utilize one or more fingertips of the user's hand. In this way, the gesture recognition interface system 150 has a much more versatile input capability than touch sensitive screens.

For example, gestures that use multiple fingertips, or even fingertips from both hands, can be interpreted as input gestures that simulate zoom commands, rotate or “twist” commands, or even environment adjustments, such as volume and brightness control, all of which can be programmed for interpretation by the gesture recognition device 160. The gesture recognition device 160 can also be programmed to recognize gestures from multiple users simultaneously. For example, the gesture recognition device 160 can provide multi-point control capability, such that coordinated actions between two hands and/or between multiple users can be implemented. Furthermore, the gesture recognition device 160 can work in conjunction with other computer input devices, such as a conventional mouse or keyboard, to provide additional types of gesture inputs. In addition, the simulated commands may not even require touching the vertical display surface. For example, the gesture recognition device 160 can evaluate not only changes in the three-dimensional location of the input object 24, but also a time threshold associated with its motion. Moreover, any of a variety of input gestures could be formed from six-degree of freedom motion based on changes in three-dimensional location and orientation of the input object 24 and any associated endpoints.

The controller 52 could also include a pre-defined gesture memory 162 coupled to the gesture recognition device 160. The pre-defined gesture memory 162 could include a plurality of pre-defined gestures, with each of the pre-defined gestures corresponding to a particular device input. For example, the pre-defined gesture memory 162 could include a database of specific arrangements and combinations of fingertip positions and motions that each correspond to a different computer input. The gesture recognition device 160, upon receiving the three-dimensional location data associated with the one or more endpoints of the input object over a given time, could poll the pre-defined gesture memory 162 to determine if the gesture input matches a pre-defined gesture. Upon determining a match, the gesture recognition device 160 could translate the gesture input into the device input that corresponds to the pre-defined gesture. The pre-defined gesture memory 162 could be pre-programmed with the appropriate pre-defined gesture inputs, or it could be dynamically programmable, such that new gestures can be added, along with the corresponding device inputs. For example, a user could activate a “begin gesture sample” operation, perform the new gesture, capture the appropriate images of the new gesture using the first camera 12 and the second camera 14, and input the appropriate device input for which the new gesture corresponds.

It is to be understood that a given gesture recognition interface system is not intended to be limited by the example of FIGS. 2-6. Other implementations are possible for providing inputs in accordance with an aspect of the invention. For example, one or more of the devices in the controller 26 could be integral with other devices, or could be separate from the controller 26. For example, the cameras 12 and 14 could each input their respective images to a common digitizer 152. As another example, similar to as described above regarding the example of FIG. 1, a given gesture recognition interface system is not intended to be limited to two cameras, but can include multiple cameras above and/or below the vertical display surface 20. Furthermore, the controller 26 is not limited to implementing the skeletal representation object detection algorithm to determine features associated with the input object. For example, other algorithms can be employed, such as a two-dimensional Laplacian of Gaussian convolution filtering operation to determine endpoints associated with the input object 24, such as fingertips. Accordingly, the gesture recognition interface system 150 can be configured in any of a variety of ways.

FIG. 7 illustrates an example of a gesture application 300 in accordance with an aspect of the invention. The gesture application 300 can be implemented on the gesture recognition interface system 10 described in the example of FIG. 1 above. As such, reference is to be made to the example of FIG. 1 in the following description of the example of FIG. 7.

The gesture application 300 is demonstrated as a virtual dry-erase board on the vertical display surface 20, similar to as described above in the example of FIG. 1. The gesture application 300 demonstrates a first gesture 302 that is formed by touching the vertical display surface 20 with an extended index finger from an otherwise closed fist, as demonstrated by the hand 304. The controller 26 can be configured (e.g., via the gesture recognition device 160) to recognize the first gesture 302 and to respond by providing the visual data, such as from the projector 30, as virtual ink 306 on the vertical display surface 20. As a result, the user can draw and/or write on the vertical display surface 20 with the virtual ink 306 by doing nothing more than touching and moving his or her finger across the vertical display surface 20, as demonstrated by the first gesture 302.

The gesture application 300 also demonstrates a second gesture 308, which is a composite gesture that is a variation of the first gesture 302. Specifically, the second gesture 308 is formed by touching the vertical display surface 20 with an extended index finger and an extended thumb from an otherwise closed fist, as demonstrated by the hand 310. Thus, the second gesture 308 is a composite gesture because it combines the first gesture 302 with an additional gesture (i.e., the extension of the thumb). The controller 26 can thus be configured to recognize the second gesture 308 as a combination of the first gesture 302 and a gesture that includes the extension of the thumb, such that the extended thumb provides a modification to the first gesture 302. In the example of the FIG. 7, the controller 26 responds to the second gesture 308 by providing the visual data as virtual ink 312, which is a variation of the virtual ink 306. Specifically, the virtual ink 312 is demonstrated as having a thicker pen-width and a different color relative to the virtual ink 306.

The second gesture 308 thus demonstrates that any of a variety of variations can be provided from the first gesture 302, and is not limited to merely adding the extension of the thumb. As an example, different pen-widths and/or colors can be implemented based on different combinations of extended fingers and/or the thumb of the hand 310. As another example, the visual data that is provided on the vertical display surface 20 can include a virtual color palette and/or a pen-width selector, such as in a corner of the vertical display surface 20. For example, the visual content can display just a currently selected color and/or pen-width, or can display all available colors and/or pen-widths with an arrow or outline that demonstrates the currently selected color and/or pen-width. Thus, the user can provide an additional gesture to toggle between or to select from the available color and/or pen-width. For example, the user can make a trigger gesture with the thumb to toggle between the selection options, or can implement gestures from the other hand that may or may not involve touching the screen while still performing write and/or draw gestures with the hand 310. As a result, any of a variety of composite gestures can be implemented in the example of FIG. 7.

The gesture application 300 also demonstrates a third gesture 314 that is formed by touching the vertical display surface 20 with an open palm, as demonstrated by the hand 316. The controller 26 can be configured to recognize the third gesture 314 and to respond by erasing virtual ink. In the example of FIG. 7, the third gesture 314 is demonstrated as the hand 316 moving through virtual ink 318, and erasing a portion of it that is covered by the hand 316, as demonstrated by the dotted lines 320. In addition, the third gesture 314 can likewise include composite gestures for an erasure of virtual ink. As an example, the controller 26 can be configured to respond to a gesture that includes the user placing both open palms on the vertical display surface 20 as an “erase all” command, such that all of the virtual ink is erased from the vertical display surface 20. As another example, portions of the virtual ink can be selected based on one command, such that a simple touch with a finger or open palm can correspond to erasure of the selected virtual ink.

It is to be understood that the gesture application 300 is not limited to the virtual dry-erase board application that is described in the example of FIG. 7. As an example, the controller 26 can be configured to recognize a large plurality of pre-programmed gestures, such that different gestures can manipulate drawn and/or written virtual ink in a variety of different ways. For example, the gesture application 300 can be configured substantially similar to any of a variety of commercially available computer drafting programs.

It is also to be understood that the virtual dry-erase board functionality can be extended to function as and/or can be combined with any of a variety of computer slide-show presentation software programs. As an example, some slide-show presentation software programs support drawing annotation on slides using a mouse. Therefore, as described herein, a slide-show presentation can be controlled using gestures for moving forward and backward through slides and for drawing on slides using natural finger drawing gestures, such as described in the example of FIG. 7. For example, one or more additional hand gestures can be employed to switch to a blank slide on which the virtual ink can be drawn and/or written to allow a more expansive presentation before, after, or during a presentation that includes pre-prepared slides. Accordingly, any of a variety of gesture applications can be implemented.

In view of the foregoing structural and functional features described above, a methodology in accordance with various aspects of the present invention will be better appreciated with reference to FIG. 8. While, for purposes of simplicity of explanation, the methodologies of FIG. 8 are shown and described as executing serially, it is to be understood and appreciated that the present invention is not limited by the illustrated order, as some aspects could, in accordance with the present invention, occur in different orders and/or concurrently with other aspects from that shown and described herein. Moreover, not all illustrated features may be required to implement a methodology in accordance with an aspect of the present invention.

FIG. 8 illustrates an example of a method 350 for providing inputs in accordance with an aspect of the invention. At 352, visual content is displayed on a vertical display surface. The vertical display surface could be a display screen on which visual content can be projected. At 354, a gesture recognition environment that is defined by a physical space in a foreground of a substantially vertical display surface is illuminated. The illumination could be provided by IR light sources. At 356, a user provides simulated inputs that are interactive with the visual content in the gesture recognition environment via gestures associated with an input object. The input object could be the user's hand, such that the user simulates the inputs via hand gestures. In addition, the simulated inputs could be simulated mouse inputs, zoom commands, or rotate commands. The simulated inputs could also be touches to the vertical display surface, such that the simulated inputs provide virtual ink with which to draw and/or write on the vertical display surface. As such, the gesture recognition system can be substantially configured as a virtual dry-erase board.

At 358, a first plurality and a second plurality of images of the input object are generated concurrently based on reflected light from the input object. The plurality of images could be a plurality of matched sets of images, such that each image of the matched set corresponds to the input object from a different perspective at substantially the same time. The images can be generated based on reflected light from the input object relative to non-reflected light in the gesture recognition environment. Therefore, the plurality of images could be images of the user controlled input object.

At 360, features associated with the input object are detected in each of corresponding sets of the first and second plurality of images. The features can be detected by generating a sequence of points along elongated portions of the input object in each of the first and second plurality of images. For example, the sequence of points can be mid-points that are symmetrically located between boundary points that define edges of the input object in the first and second images. Each point in the sequence of points of the first plurality of images can be correlated with each corresponding point in the sequence of points in the second plurality of images. The correlation of the points in each of the first and second images can be based on mutual end-points or any of the points along the elongated portions of the input object. At 362, a plurality of three-dimensional physical locations of the input object are determined based on a relative separation of the correlated points in the sequence of points of each of the sets of the first and second plurality of images. The relative separation can be based on a parallax separation between the first and second images. The three-dimensional location could be determined relative to the vertical display surface. Changes in location of the three-dimensional location of at least one end-point of the input object could be determinative of the physical motion of the input object. At 364, it is determined whether the physical motion associated with the input object corresponds to any of a plurality of pre-defined gestures. The pre-defined gestures could be stored in a memory. Each pre-defined gesture could be associated with a different device input. At 366, at least one device input is provided based on determining that the physical motion associated with the input object corresponds to a given one of the pre-defined gestures. Device inputs could be mouse inputs, such that two-dimensional motion across the vertical display surface could simulate motion of a mouse cursor, and a touch of the vertical display surface could simulate a mouse left-click. In addition, motion associated with multiple endpoints could provide different types of inputs, such as rotate and zoom commands.

What have been described above are examples of the present invention. It is, of course, not possible to describe every conceivable combination of components or methodologies for purposes of describing the present invention, but one of ordinary skill in the art will recognize that many further combinations and permutations of the present invention are possible. Accordingly, the present invention is intended to embrace all such alterations, modifications and variations that fall within the spirit and scope of the appended claims.

Claims (15)

1. A gesture recognition interface system comprising:
a substantially vertical surface in which three-dimensional physical space in a foreground of the substantially vertical surface defines a gesture recognition environment;
at least one light source positioned to provide illumination of the gesture recognition environment between the at least one light source and the substantially vertical surface;
at least two cameras configured to generate a plurality of image sets based on the illumination that is provided from the at least one light source being reflected from an input object positioned in the gesture recognition environment; and
a controller configured to implement an algorithm that detects features associated with the input object in each of the plurality of image sets based on determining a sequence of mid-points between opposing edges of elongated portions of the input object in a predetermined coordinate system in each of the plurality of image sets and based on correlating the sequence of mid-points of each of a first image with corresponding mid-points of a second image in each of the plurality of image sets to calculate endpoints, length, and pitch of the elongated portions of the input object and to determine a given input gesture performed in the gesture recognition environment based on changes in relative locations of the input object in each of the plurality of image sets, and being further configured to initiate a device input associated with the given input gesture.
2. The interface system of claim 1, wherein the at least two cameras are positioned such that a central lens axis of each of the at least two cameras is substantially parallel with respect to a plane that is defined by the substantially vertical surface to capture the substantially vertical surface and the gesture recognition environment in the plurality of image sets.
3. The interface system of claim 1, wherein the at least one light source is positioned such that a central illumination axis of the at least one light source has a non-zero angle of incidence with respect to a normal associated with a floor under the substantially vertical surface.
4. The interface system of claim 1, wherein the substantially vertical surface is configured to display visual content that is interactive with the input object.
5. The interface system of claim 4, wherein the controller is configured to provide the visual content as virtual ink on portions of the substantially vertical surface that are contacted by the input object.
6. The interface system of claim 5, wherein at least one of a color change, a pen-width change, and erasure of the virtual ink is controlled via a respective at least one additional predetermined gesture.
7. The interface system of claim 1, wherein the at least one light source comprises at least one infrared (IR) light source, and wherein the at least two cameras each comprise an IR filter.
8. The interface system of claim 7, wherein the controller comprises an automated calibration unit configured to calibrate the gesture recognition interface system based on an amount of parallax separation between matched sets of images of the input object that is placed in contact with at least one predetermined two-dimensional location on the substantially vertical surface.
9. The interface system of claim 1, wherein the substantially vertical surface is configured as a substantially vertical retroreflective surface to substantially mitigate the receipt of illumination reflected from the input object to the substantially vertical surface by at least one of the at least two cameras.
10. A method of providing device inputs, the method comprising:
displaying visual content on a substantially vertical display surface;
illuminating a gesture recognition environment that is defined by a three-dimensional physical space in a foreground of the substantially vertical display surface via at least one light source;
providing simulated inputs that are interactive with the visual content in the gesture recognition environment via gestures associated with an input object;
generating a first plurality of images and a second plurality of images associated with the input object substantially concurrently based on reflected light from the input object that is provided from the at least one light source;
detecting features associated with the input object in each of corresponding pairs of the first and second plurality of images based on determining edges of the input object in each of the first and second plurality of images based on the reflected light and based on generating a sequence of mid-points between opposing edges along the elongated portions of the input object in a predetermined coordinate system in each of the first and second plurality of images;
determining a plurality of three-dimensional physical locations of the input object based on a relative separation of the features associated with the input object in the corresponding pairs of the first and second plurality of images;
determining if changes in the plurality of three-dimensional physical locations of the input object correspond to any of a plurality of pre-defined gestures; and
providing at least one device input based on a given one of the plurality of pre-defined gestures upon determining that the changes in the plurality of three-dimensional physical locations of the input object correspond to the given one of the plurality of pre-defined gestures.
11. The method of claim 10, further comprising:
contacting at least one predetermined point in two-dimensional space on the substantially vertical display surface with an endpoint of the input object;
generating a first and second calibration image of the input object in contact with the at least one predetermined point; and
calibrating the three-dimensional physical locations of the input object based on an amount of parallax separation between the endpoint of the input object in each of the first and second calibration image.
12. A gesture recognition interface system comprising:
means for providing a brightness contrast between reflected light and non-reflected light in a gesture recognition environment that is defined by a three-dimensional physical space in front of a substantially vertical display surface, the reflected light being reflected from an input object in the gesture recognition environment;
means for generating a first plurality of images of the input object based on the brightness contrast;
means for generating a second plurality of images of the input object based on the brightness contrast, the first plurality of images and the second plurality of images forming a plurality of matched pairs of images of the input object;
means for generating a sequence of points associated with elongated portions of the input object in each of the first and second plurality of images and for correlating at least one pair of corresponding points in each of the plurality of matched pairs of images of the input object to calculate endpoints, length, and pitch of the elongated portions of the input object;
means for determining three-dimensional physical location information associated with features of the input object based on a relative separation of the correlated at least one pair of corresponding points in the plurality of matched pairs of images of the input object;
means for translating changes in the three-dimensional location information associated with the features of the input object to a given input gesture; and
means for providing device inputs based on matching the given input gesture with one of a plurality of pre-defined gestures.
13. The interface system of claim 12, wherein the means for generating the sequence of points is configured to determine edges of the input object in each of the first and second plurality of images based on the brightness contrast and to generate a sequence of mid-points between opposing edges of elongated portions of the input object in a predetermined coordinate system in each of the first and second plurality of images.
14. The interface system of claim 12, wherein the means for generating the first plurality of images and the means for generating the second plurality of images are configured to obtain the first and second plurality of images at a glancing angle that is substantially parallel with respect to a plane that is defined by the substantially vertical display surface to capture the substantially vertical display surface and the gesture recognition environment in each of the first and second plurality of images.
15. The interface system of claim 12, further comprising means for calibrating the means for determining three-dimensional location information associated with the input object based on an amount of parallax separation of a matched pair of images of an endpoint of the input object in contact with at least one predetermined two-dimensional physical location on the substantially vertical display surface.
US12133836 2006-07-13 2008-06-05 Gesture recognition interface system with vertical display Active 2029-04-22 US8180114B2 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US11485788 US8589824B2 (en) 2006-07-13 2006-07-13 Gesture recognition interface system
US12133836 US8180114B2 (en) 2006-07-13 2008-06-05 Gesture recognition interface system with vertical display

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US12133836 US8180114B2 (en) 2006-07-13 2008-06-05 Gesture recognition interface system with vertical display
GB0909597A GB2460542B (en) 2008-06-05 2009-06-04 Gesture recognition interface system with vertical display
DE200910023875 DE102009023875A1 (en) 2008-06-05 2009-06-04 Gestenerkennungsschittstellensystem vertical display area

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US11485788 Continuation-In-Part US8589824B2 (en) 2006-07-13 2006-07-13 Gesture recognition interface system

Publications (2)

Publication Number Publication Date
US20080244468A1 true US20080244468A1 (en) 2008-10-02
US8180114B2 true US8180114B2 (en) 2012-05-15

Family

ID=40902563

Family Applications (1)

Application Number Title Priority Date Filing Date
US12133836 Active 2029-04-22 US8180114B2 (en) 2006-07-13 2008-06-05 Gesture recognition interface system with vertical display

Country Status (3)

Country Link
US (1) US8180114B2 (en)
DE (1) DE102009023875A1 (en)
GB (1) GB2460542B (en)

Cited By (43)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110035952A1 (en) * 2008-04-21 2011-02-17 Carl Zeiss Industrielle Messtechnik Gmbh Display of results of a measurement of workpieces as a function of the detection of the gesture of a user
US20110205242A1 (en) * 2010-02-22 2011-08-25 Nike, Inc. Augmented Reality Design System
US20110280441A1 (en) * 2010-05-17 2011-11-17 Hon Hai Precision Industry Co., Ltd. Projector and projection control method
US20110289456A1 (en) * 2010-05-18 2011-11-24 Microsoft Corporation Gestures And Gesture Modifiers For Manipulating A User-Interface
US20110298798A1 (en) * 2005-10-21 2011-12-08 Apple Inc. Three-dimensional imaging and display system
US20120050535A1 (en) * 2010-08-31 2012-03-01 Gilray Densham System and method for tracking
US20120309516A1 (en) * 2011-05-31 2012-12-06 Microsoft Corporation Action trigger gesturing
US20120319945A1 (en) * 2010-05-21 2012-12-20 Mccarthy John System and method for reporting data in a computer vision system
US20130010068A1 (en) * 2011-04-12 2013-01-10 Radiation Monitoring Devices, Inc. Augmented reality system
US20130033571A1 (en) * 2011-08-03 2013-02-07 General Electric Company Method and system for cropping a 3-dimensional medical dataset
US8615108B1 (en) 2013-01-30 2013-12-24 Imimtek, Inc. Systems and methods for initializing motion tracking of human hands
US20140033141A1 (en) * 2011-04-13 2014-01-30 Nokia Corporation Method, apparatus and computer program for user control of a state of an apparatus
US8655021B2 (en) 2012-06-25 2014-02-18 Imimtek, Inc. Systems and methods for tracking human hands by performing parts based template matching using images from multiple viewpoints
US8686943B1 (en) 2011-05-13 2014-04-01 Imimtek, Inc. Two-dimensional method and system enabling three-dimensional user interaction with a device
WO2014070793A2 (en) * 2012-10-29 2014-05-08 Amazon Technologies, Inc. Gesture detection systems
US8723789B1 (en) 2011-02-11 2014-05-13 Imimtek, Inc. Two-dimensional method and system enabling three-dimensional user interaction with a device
US20140146016A1 (en) * 2012-11-29 2014-05-29 Pixart Imaging Inc. Positioning module, optical touch system and method of calculating a coordinate of a touch medium
US20140176559A1 (en) * 2012-12-21 2014-06-26 Samsung Display Co., Ltd. Image display system
US8773512B1 (en) * 2011-06-30 2014-07-08 Aquifi, Inc. Portable remote control device enabling three-dimensional user interaction with at least one appliance
US8830312B2 (en) 2012-06-25 2014-09-09 Aquifi, Inc. Systems and methods for tracking human hands using parts based template matching within bounded regions
US8840466B2 (en) 2011-04-25 2014-09-23 Aquifi, Inc. Method and system to create three-dimensional mapping in a two-dimensional game
US8854433B1 (en) 2012-02-03 2014-10-07 Aquifi, Inc. Method and system enabling natural user interface gestures with an electronic system
US8923562B2 (en) 2012-12-24 2014-12-30 Industrial Technology Research Institute Three-dimensional interactive device and operation method thereof
US9055226B2 (en) 2010-08-31 2015-06-09 Cast Group Of Companies Inc. System and method for controlling fixtures based on tracking data
US20150186039A1 (en) * 2012-08-27 2015-07-02 Citizen Holdings Co., Ltd. Information input device
US9092665B2 (en) 2013-01-30 2015-07-28 Aquifi, Inc Systems and methods for initializing motion tracking of human hands
US9158381B2 (en) 2013-02-25 2015-10-13 Honda Motor Co., Ltd. Multi-resolution gesture recognition
US9176588B2 (en) 2013-01-14 2015-11-03 Maxm Integrated Products, Inc. System and method for discerning complex gestures using an array of optical sensors
US9213448B2 (en) 2012-11-29 2015-12-15 Pixart Imaging Inc. Positioning module, optical touch system and method of calculating a coordinate of a touch medium
US20150375399A1 (en) * 2014-06-27 2015-12-31 Hansen Medical, Inc. User interface for medical robotics system
US9298266B2 (en) 2013-04-02 2016-03-29 Aquifi, Inc. Systems and methods for implementing three-dimensional (3D) gesture based graphical user interfaces (GUI) that incorporate gesture reactive interface objects
US9310891B2 (en) 2012-09-04 2016-04-12 Aquifi, Inc. Method and system enabling natural user interface gestures with user wearable glasses
US9350923B2 (en) 2010-08-31 2016-05-24 Cast Group Of Companies Inc. System and method for tracking
US9372541B2 (en) 2013-03-11 2016-06-21 Honeywell International Inc. Gesture recognition system operability verification
US9477319B1 (en) 2011-06-27 2016-10-25 Amazon Technologies, Inc. Camera based sensor for motion detection
US9501151B2 (en) 2013-02-13 2016-11-22 Google Inc. Simultaneous multi-user marking interactions
US9507417B2 (en) 2014-01-07 2016-11-29 Aquifi, Inc. Systems and methods for implementing head tracking based graphical user interfaces (GUI) that incorporate gesture reactive interface objects
US9549101B1 (en) * 2015-09-01 2017-01-17 International Business Machines Corporation Image capture enhancement using dynamic control image
US9612655B2 (en) 2012-10-31 2017-04-04 Audi Ag Method for inputting a control command for a component of a motor vehicle
US9619105B1 (en) 2014-01-30 2017-04-11 Aquifi, Inc. Systems and methods for gesture based interaction with viewpoint dependent user interfaces
US9798388B1 (en) 2013-07-31 2017-10-24 Aquifi, Inc. Vibrotactile system to augment 3D input systems
US9857868B2 (en) 2011-03-19 2018-01-02 The Board Of Trustees Of The Leland Stanford Junior University Method and system for ergonomic touch-free interface
US10057078B2 (en) 2015-08-21 2018-08-21 Samsung Electronics Company, Ltd. User-configurable interactive region monitoring

Families Citing this family (103)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7262785B2 (en) * 2003-08-21 2007-08-28 Microsoft Corporation Ink editing architecture
US9696808B2 (en) * 2006-07-13 2017-07-04 Northrop Grumman Systems Corporation Hand-gesture recognition method
US8589824B2 (en) * 2006-07-13 2013-11-19 Northrop Grumman Systems Corporation Gesture recognition interface system
US8180114B2 (en) 2006-07-13 2012-05-15 Northrop Grumman Systems Corporation Gesture recognition interface system with vertical display
US8234578B2 (en) * 2006-07-25 2012-07-31 Northrop Grumman Systems Corporatiom Networked gesture collaboration system
US8432448B2 (en) * 2006-08-10 2013-04-30 Northrop Grumman Systems Corporation Stereo camera intrusion detection system
US8139110B2 (en) * 2007-11-01 2012-03-20 Northrop Grumman Systems Corporation Calibration of a gesture recognition interface system
US9377874B2 (en) * 2007-11-02 2016-06-28 Northrop Grumman Systems Corporation Gesture recognition light and video image projector
US9171454B2 (en) 2007-11-14 2015-10-27 Microsoft Technology Licensing, Llc Magic wand
US20090219253A1 (en) * 2008-02-29 2009-09-03 Microsoft Corporation Interactive Surface Computer with Switchable Diffuser
US20110102570A1 (en) * 2008-04-14 2011-05-05 Saar Wilf Vision based pointing device emulation
US8952894B2 (en) * 2008-05-12 2015-02-10 Microsoft Technology Licensing, Llc Computer vision-based multi-touch sensing using infrared lasers
US8345920B2 (en) * 2008-06-20 2013-01-01 Northrop Grumman Systems Corporation Gesture recognition interface system with a light-diffusive screen
US8847739B2 (en) 2008-08-04 2014-09-30 Microsoft Corporation Fusing RFID and vision for surface object tracking
US20100031202A1 (en) * 2008-08-04 2010-02-04 Microsoft Corporation User-defined gesture set for surface computing
US8972902B2 (en) 2008-08-22 2015-03-03 Northrop Grumman Systems Corporation Compound gesture recognition
GB2468884A (en) * 2009-03-25 2010-09-29 Nec Corp User defined paths for control on a touch screen
KR101559178B1 (en) * 2009-04-08 2015-10-12 엘지전자 주식회사 Command input method and a mobile communication terminal applying this
US9498718B2 (en) * 2009-05-01 2016-11-22 Microsoft Technology Licensing, Llc Altering a view perspective within a display environment
US9898675B2 (en) 2009-05-01 2018-02-20 Microsoft Technology Licensing, Llc User movement tracking feedback to improve tracking
US9417700B2 (en) 2009-05-21 2016-08-16 Edge3 Technologies Gesture recognition systems and related methods
KR100936666B1 (en) * 2009-05-25 2010-01-13 전자부품연구원 Apparatus for touching reflection image using an infrared screen
GB2470653B (en) * 2009-05-26 2015-04-29 Zienon L L C Enabling data entry based on differentiated input objects
US8386963B2 (en) * 2009-05-28 2013-02-26 Microsoft Corporation Virtual inking using gesture recognition
GB2483168B (en) 2009-10-13 2013-06-12 Pointgrab Ltd Computer vision gesture based control of a device
KR101639383B1 (en) * 2009-11-12 2016-07-22 삼성전자주식회사 Apparatus for sensing proximity touch operation and method thereof
US20110117526A1 (en) * 2009-11-16 2011-05-19 Microsoft Corporation Teaching gesture initiation with registration posture guides
US8622742B2 (en) * 2009-11-16 2014-01-07 Microsoft Corporation Teaching gestures with offset contact silhouettes
KR101092722B1 (en) * 2009-12-02 2011-12-09 현대자동차주식회사 User interface device for controlling multimedia system of vehicle
KR100974894B1 (en) * 2009-12-22 2010-08-11 전자부품연구원 3d space touch apparatus using multi-infrared camera
US8396252B2 (en) 2010-05-20 2013-03-12 Edge 3 Technologies Systems and related methods for three dimensional gesture recognition in vehicles
US20110296352A1 (en) * 2010-05-27 2011-12-01 Microsoft Corporation Active calibration of a natural user interface
US9851829B2 (en) * 2010-08-27 2017-12-26 Apple Inc. Signal processing for touch and hover sensing display device
CN101943995A (en) * 2010-09-01 2011-01-12 惠州Tcl移动通信有限公司 Method and device for processing display information of mobile terminal and touch screen thereof
US8467599B2 (en) 2010-09-02 2013-06-18 Edge 3 Technologies, Inc. Method and apparatus for confusion learning
US8666144B2 (en) 2010-09-02 2014-03-04 Edge 3 Technologies, Inc. Method and apparatus for determining disparity of texture
US8655093B2 (en) 2010-09-02 2014-02-18 Edge 3 Technologies, Inc. Method and apparatus for performing segmentation of an image
EP2439700B1 (en) * 2010-10-06 2013-05-01 Alcatel Lucent Method and Arrangement for Identifying Virtual Visual Information in Images
US20130154913A1 (en) * 2010-12-16 2013-06-20 Siemens Corporation Systems and methods for a gaze and gesture interface
US8970589B2 (en) 2011-02-10 2015-03-03 Edge 3 Technologies, Inc. Near-touch interaction with a stereo camera grid structured tessellations
US8582866B2 (en) 2011-02-10 2013-11-12 Edge 3 Technologies, Inc. Method and apparatus for disparity computation in stereo images
US8845431B2 (en) * 2011-05-31 2014-09-30 Microsoft Corporation Shape trace gesturing
US9672609B1 (en) 2011-11-11 2017-06-06 Edge 3 Technologies, Inc. Method and apparatus for improved depth-map estimation
US8693731B2 (en) 2012-01-17 2014-04-08 Leap Motion, Inc. Enhanced contrast for object detection and characterization by optical imaging
US9679215B2 (en) 2012-01-17 2017-06-13 Leap Motion, Inc. Systems and methods for machine control
JP2015510169A (en) * 2012-01-17 2015-04-02 リープ モーション,インコーポレーテッドLeap Motion,Inc. Characterization by improving and optical imaging contrast for object detection
US8638989B2 (en) 2012-01-17 2014-01-28 Leap Motion, Inc. Systems and methods for capturing motion in three-dimensional space
DE102012206777A1 (en) * 2012-04-25 2013-10-31 Robert Bosch Gmbh A method for controlling an image representation of a projection apparatus
US8938124B2 (en) 2012-05-10 2015-01-20 Pointgrab Ltd. Computer vision based tracking of a hand
US9213436B2 (en) * 2012-06-20 2015-12-15 Amazon Technologies, Inc. Fingertip location for gesture input
US9400575B1 (en) 2012-06-20 2016-07-26 Amazon Technologies, Inc. Finger detection for element selection
US8970479B1 (en) * 2012-07-31 2015-03-03 Rawles Llc Hand gesture detection
CN103677217A (en) * 2012-08-30 2014-03-26 深圳市亿思达显示科技有限公司 3D space input system
KR20140046343A (en) 2012-10-10 2014-04-18 삼성전자주식회사 Multi display device and method for controlling thereof
KR20140046345A (en) 2012-10-10 2014-04-18 삼성전자주식회사 Multi display device and method for providing tool thereof
KR20140046323A (en) 2012-10-10 2014-04-18 삼성전자주식회사 Multi display apparatus and method for controlling display operation
KR20140046319A (en) 2012-10-10 2014-04-18 삼성전자주식회사 Multi display apparatus and multi display method
KR20140046344A (en) 2012-10-10 2014-04-18 삼성전자주식회사 Multi display device and method for photographing thereof
KR20140046346A (en) 2012-10-10 2014-04-18 삼성전자주식회사 Multi display apparatus and method for contorlling thereof
DE102012020607B4 (en) * 2012-10-19 2015-06-11 Audi Ag Motor vehicles with a gesture control device and method for controlling a selection element
US9285893B2 (en) 2012-11-08 2016-03-15 Leap Motion, Inc. Object detection and tracking with variable-field illumination devices
US9465461B2 (en) 2013-01-08 2016-10-11 Leap Motion, Inc. Object detection and tracking with audio and optical signals
KR20140096843A (en) * 2013-01-29 2014-08-06 삼성전자주식회사 Digital photographing apparatus and control method thereof
JP6111706B2 (en) * 2013-02-01 2017-04-12 セイコーエプソン株式会社 Position detecting device, adjustment method, and adjustment program
JP6144501B2 (en) * 2013-02-12 2017-06-07 富士通テン株式会社 Display device, and a display method
US10134267B2 (en) 2013-02-22 2018-11-20 Universal City Studios Llc System and method for tracking a passive wand and actuating an effect based on a detected wand path
US20140250398A1 (en) * 2013-03-03 2014-09-04 Microsoft Corporation Enhanced canvas environments
US9122916B2 (en) * 2013-03-14 2015-09-01 Honda Motor Co., Ltd. Three dimensional fingertip tracking
EP2778864A1 (en) * 2013-03-14 2014-09-17 BlackBerry Limited Method and apparatus pertaining to the display of a stylus-based control-input area
JP6146094B2 (en) * 2013-04-02 2017-06-14 富士通株式会社 Information operation display system, a display program, and a display method
JP6201379B2 (en) * 2013-04-02 2017-09-27 富士通株式会社 Position calculation system, the position calculation program, and a position calculating method
US10082935B2 (en) * 2013-04-15 2018-09-25 Carnegie Mellon University Virtual tools for use with touch-sensitive surfaces
WO2014194148A3 (en) * 2013-05-29 2015-02-26 Weijie Zhang Systems and methods involving gesture based user interaction, user interface and/or other features
US9483171B1 (en) * 2013-06-11 2016-11-01 Amazon Technologies, Inc. Low latency touch input rendering
US9529513B2 (en) * 2013-08-05 2016-12-27 Microsoft Technology Licensing, Llc Two-hand interaction with natural user interface
CN103605460B (en) * 2013-08-30 2017-01-25 华为技术有限公司 One kind of gesture recognition method and associated terminal
KR20150042039A (en) * 2013-10-10 2015-04-20 엘지전자 주식회사 Mobile terminal and operating method thereof
CN103530619B (en) * 2013-10-29 2016-08-31 北京交通大学 The gesture recognition method based on a small number of training data samples configured rgb-d
US9996638B1 (en) 2013-10-31 2018-06-12 Leap Motion, Inc. Predictive information for free space gesture control and communication
DE102013019197A1 (en) * 2013-11-15 2015-05-21 Audi Ag Automotive air conditioning with adaptive air vents
US9329727B2 (en) * 2013-12-11 2016-05-03 Microsoft Technology Licensing, Llc Object detection in optical sensor systems
US20150194187A1 (en) * 2014-01-09 2015-07-09 Microsoft Corporation Telestrator system
US9613262B2 (en) 2014-01-15 2017-04-04 Leap Motion, Inc. Object detection and tracking for providing a virtual device experience
US20150205358A1 (en) * 2014-01-20 2015-07-23 Philip Scott Lyren Electronic Device with Touchless User Interface
US9430095B2 (en) 2014-01-23 2016-08-30 Microsoft Technology Licensing, Llc Global and local light detection in optical sensor systems
CN104932797A (en) * 2014-03-17 2015-09-23 深圳富泰宏精密工业有限公司 Gesture unlocking method and system
JP6359862B2 (en) * 2014-04-17 2018-07-18 シャープ株式会社 Touch operation input device, a touch operation input method and a program
US9600999B2 (en) 2014-05-21 2017-03-21 Universal City Studios Llc Amusement park element tracking system
US9429398B2 (en) 2014-05-21 2016-08-30 Universal City Studios Llc Optical tracking for controlling pyrotechnic show elements
US10025990B2 (en) 2014-05-21 2018-07-17 Universal City Studios Llc System and method for tracking vehicles in parking structures and intersections
US9616350B2 (en) 2014-05-21 2017-04-11 Universal City Studios Llc Enhanced interactivity in an amusement park environment using passive tracking elements
US10061058B2 (en) 2014-05-21 2018-08-28 Universal City Studios Llc Tracking system and method for use in surveying amusement park equipment
US9433870B2 (en) 2014-05-21 2016-09-06 Universal City Studios Llc Ride vehicle tracking and control system using passive tracking elements
US20150378440A1 (en) * 2014-06-27 2015-12-31 Microsoft Technology Licensing, Llc Dynamically Directing Interpretation of Input Data Based on Contextual Information
JP6335695B2 (en) * 2014-07-09 2018-05-30 キヤノン株式会社 The information processing apparatus, a control method, program, and storage medium
CN104268507A (en) * 2014-09-15 2015-01-07 南京邮电大学 Manual alphabet identification method based on RGB-D image
JP2016186674A (en) * 2015-03-27 2016-10-27 セイコーエプソン株式会社 Interactive projector and interactive projection system
US9946298B2 (en) * 2015-04-08 2018-04-17 Mediatek Inc. Wearable device interactive system
US20170193288A1 (en) * 2015-12-31 2017-07-06 Microsoft Technology Licensing, Llc Detection of hand gestures using gesture language discrete values
US20170285742A1 (en) * 2016-03-29 2017-10-05 Google Inc. System and method for generating virtual marks based on gaze tracking
CN106078752A (en) * 2016-06-27 2016-11-09 西安电子科技大学 Kinect-based human behavior imitation method of humanoid robot
DE102016215746A1 (en) * 2016-08-23 2018-03-01 Robert Bosch Gmbh Projector with touch-free control
DE102017103825A1 (en) * 2017-02-24 2018-08-30 Dr. Schneider Kunststoffwerke Gmbh Apparatus for controlling an air vent

Citations (100)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4468694A (en) * 1980-12-30 1984-08-28 International Business Machines Corporation Apparatus and method for remote displaying and sensing of information using shadow parallax
JPS62264390A (en) 1986-05-12 1987-11-17 Omron Tateisi Electronics Co Visual sense recognizing device for supervisory robot
US4843568A (en) 1986-04-11 1989-06-27 Krueger Myron W Real time perception of and response to the actions of an unencumbered participant/user
US4924506A (en) 1986-07-22 1990-05-08 Schlumberger Systems & Services, Inc. Method for directly measuring area and volume using binocular stereo vision
JPH0431996A (en) 1990-05-28 1992-02-04 Matsushita Electric Works Ltd Image recognition type burgler sensor
JPH04271423A (en) 1991-02-27 1992-09-28 Nippon Telegr & Teleph Corp <Ntt> Information input method
US5220441A (en) 1990-09-28 1993-06-15 Eastman Kodak Company Mechanism for determining parallax between digital images
US5239373A (en) 1990-12-26 1993-08-24 Xerox Corporation Video computational shared drawing space
EP0571702A2 (en) 1992-05-26 1993-12-01 Takenaka Corporation Hand pointing type input unit and wall computer module
JPH06226439A (en) 1993-02-05 1994-08-16 Matsushita Electric Ind Co Ltd Heater
US5475422A (en) 1993-06-21 1995-12-12 Nippon Telegraph And Telephone Corporation Method and apparatus for reconstructing three-dimensional objects
US5483261A (en) 1992-02-14 1996-01-09 Itu Research, Inc. Graphical input controller and method with rear screen image detection
US5563988A (en) 1994-08-01 1996-10-08 Massachusetts Institute Of Technology Method and system for facilitating wireless, full-body, real-time user interaction with a digitally represented visual environment
WO1998013746A1 (en) 1996-09-27 1998-04-02 Samsung Electronics Co. Ltd. Method for feeding information into a computer
DE19739285C1 (en) 1997-09-08 1998-11-19 Siemens Nixdorf Inf Syst Data representation and virtual input device
EP0913790A1 (en) 1997-10-29 1999-05-06 Takenaka Corporation Hand pointing apparatus
US5913727A (en) 1995-06-02 1999-06-22 Ahdoot; Ned Interactive movement and contact simulation game
US5999185A (en) 1992-03-30 1999-12-07 Kabushiki Kaisha Toshiba Virtual reality control using image, model and control data to manipulate interactions
US6002808A (en) 1996-07-26 1999-12-14 Mitsubishi Electric Information Technology Center America, Inc. Hand gesture control system
WO2000002187A1 (en) 1998-07-01 2000-01-13 Deluca Michael J Stereoscopic user interface method and apparatus
WO2000021023A1 (en) 1998-10-07 2000-04-13 Intel Corporation Controlling a pointer using digital video
US6128003A (en) 1996-12-20 2000-10-03 Hitachi, Ltd. Hand gesture recognition system and method
US6147678A (en) * 1998-12-09 2000-11-14 Lucent Technologies Inc. Video hand image-three-dimensional computer interface with multiple degrees of freedom
US6195104B1 (en) 1997-12-23 2001-02-27 Philips Electronics North America Corp. System and method for permitting three-dimensional navigation through a virtual reality environment using camera-based gesture inputs
US6204852B1 (en) 1998-12-09 2001-03-20 Lucent Technologies Inc. Video hand image three-dimensional computer interface
US6222465B1 (en) 1998-12-09 2001-04-24 Lucent Technologies Inc. Gesture-based computer interface
US20010006426A1 (en) 1996-07-18 2001-07-05 Korea Institute Of Science And Technology Holographic projection screen for displaying a three-dimensional color images and optical display system using the holographic screen
US20010043719A1 (en) 1997-03-21 2001-11-22 Kenichi Harakawa Hand pointing device
US6327381B1 (en) 1994-12-29 2001-12-04 Worldscape, Llc Image transformation and synthesis methods
US6353428B1 (en) 1997-02-28 2002-03-05 Siemens Aktiengesellschaft Method and device for detecting an object in an area radiated by waves in the invisible spectral range
US6359612B1 (en) 1998-09-30 2002-03-19 Siemens Aktiengesellschaft Imaging system for displaying image information that has been acquired by means of a medical diagnostic imaging device
US20020090146A1 (en) 2001-01-09 2002-07-11 Siemens Aktiengesellschaft Hand recognition with position determination
EP1223537A2 (en) 2001-01-09 2002-07-17 Siemens Aktiengesellschaft Controllable device with user authentication
US20020093666A1 (en) * 2001-01-17 2002-07-18 Jonathan Foote System and method for determining the location of a target in a room or small area
US20020122113A1 (en) 1999-08-09 2002-09-05 Foote Jonathan T. Method and system for compensating for parallax in multiple camera systems
US20020126161A1 (en) * 1994-07-05 2002-09-12 Hitachi, Ltd. Information processing system
US20020186221A1 (en) 2001-06-05 2002-12-12 Reactrix Systems, Inc. Interactive video display system
US6512507B1 (en) 1998-03-31 2003-01-28 Seiko Epson Corporation Pointing position detection device, presentation system, and method, and computer-readable medium
US20030058341A1 (en) 2001-09-27 2003-03-27 Koninklijke Philips Electronics N.V. Video based detection of fall-down and other events
WO2003026299A1 (en) 2001-09-14 2003-03-27 Accenture Global Services Gmbh Lab window collaboration
US20030067537A1 (en) 2001-10-04 2003-04-10 Myers Kenneth J. System and method for three-dimensional data acquisition
US20030085866A1 (en) 2000-06-06 2003-05-08 Oliver Bimber Extended virtual table: an optical extension for table-like projection systems
US20030156756A1 (en) * 2002-02-15 2003-08-21 Gokturk Salih Burak Gesture recognition system using depth perceptive sensors
US6624833B1 (en) 2000-04-17 2003-09-23 Lucent Technologies Inc. Gesture-based input interface system with shadow detection
US20030218761A1 (en) 2002-05-22 2003-11-27 Carlo Tomasi Method and apparatus for approximating depth of an object's placement onto a monitored region with applications to virtual interface devices
US6681031B2 (en) 1998-08-10 2004-01-20 Cybernet Systems Corporation Gesture-controlled interfaces for self-service machines and other applications
US6695770B1 (en) 1999-04-01 2004-02-24 Dominic Kin Leung Choy Simulated human interaction systems
US20040046747A1 (en) 2000-09-26 2004-03-11 Eugenio Bustamante Providing input signals
US6714901B1 (en) 1997-11-19 2004-03-30 Inria Institut National De Recherche En Informatique Et En Automatique Electronic device for processing image-data, for simulating the behaviour of a deformable object
US6720949B1 (en) 1997-08-22 2004-04-13 Timothy R. Pryor Man machine interfaces and applications
US20040108990A1 (en) 2001-01-08 2004-06-10 Klony Lieberman Data input device
US20040113885A1 (en) 2001-05-31 2004-06-17 Yakup Genc New input devices for augmented reality applications
US20040125207A1 (en) 2002-08-01 2004-07-01 Anurag Mittal Robust stereo-driven video-based surveillance
US6788809B1 (en) 2000-06-30 2004-09-07 Intel Corporation System and method for gesture recognition in three dimensions using stereo imaging and color vision
US20040183775A1 (en) 2002-12-13 2004-09-23 Reactrix Systems Interactive directed light/sound system
US6796656B1 (en) 2003-06-14 2004-09-28 Imatte, Inc. Generating a matte signal from a retro reflective component of a front projection screen
US20040193413A1 (en) 2003-03-25 2004-09-30 Wilson Andrew D. Architecture for controlling a computer using hand gestures
US6806849B2 (en) 1998-04-20 2004-10-19 Lightspace Technologies Ab Multi-planar volumetric display system and method of operation using multi-planar interlacing
US20040239761A1 (en) 2003-05-26 2004-12-02 S1 Corporation Method of intruder detection and device thereof
US20050002074A1 (en) 2003-07-03 2005-01-06 Holotouch, Inc. Holographic human-machine interfaces
US20050012817A1 (en) 2003-07-15 2005-01-20 International Business Machines Corporation Selective surveillance system with active sensor management policies
US6857746B2 (en) 2002-07-01 2005-02-22 Io2 Technology, Llc Method and system for free-space imaging display and interface
US20050052714A1 (en) 2003-07-24 2005-03-10 Zebra Imaging, Inc. Enhanced environment visualization using holographic stereograms
US20050068537A1 (en) 2002-07-17 2005-03-31 New York University Method and apparatus for determining a bidirectional reflectance distribution function of a subject
US20050088714A1 (en) 1997-07-08 2005-04-28 Kremen Stanley H. Method for creating a holographic screen that reconstructs uniformly magnified three-dimensional images from projected integral photographs
US20050110964A1 (en) 2002-05-28 2005-05-26 Matthew Bell Interactive video window display system
US20050151850A1 (en) 2004-01-14 2005-07-14 Korea Institute Of Science And Technology Interactive presentation system
US20050166163A1 (en) 2004-01-23 2005-07-28 Chang Nelson L.A. Systems and methods of interfacing with a machine
US6950534B2 (en) 1998-08-10 2005-09-27 Cybernet Systems Corporation Gesture-controlled interfaces for self-service machines and other applications
US6956573B1 (en) 1996-11-15 2005-10-18 Sarnoff Corporation Method and apparatus for efficiently representing storing and accessing video information
US20050252714A1 (en) 2002-04-26 2005-11-17 Hirofumi Goda Sound absorbing body, sound absorbing structural body, and method of manufacturing these bodies
US20050275628A1 (en) 2002-01-25 2005-12-15 Alias Systems Corp. System for physical rotation of volumetric display enclosures to facilitate viewing
US20050285945A1 (en) 2004-06-25 2005-12-29 Hitachi, Ltd. Imaging apparatus
US20050286101A1 (en) 2004-04-13 2005-12-29 Board Of Regents, The University Of Texas System Holographic projector
US6983065B1 (en) 2001-12-28 2006-01-03 Cognex Technology And Investment Corporation Method for extracting features from an image using oriented filters
US20060010400A1 (en) 2004-06-28 2006-01-12 Microsoft Corporation Recognizing gestures and using gestures for interacting with software applications
US20060036944A1 (en) 2004-08-10 2006-02-16 Microsoft Corporation Surface UI for gesture-based interaction
US20060052953A1 (en) 2003-01-02 2006-03-09 Sociedad Espanola De Carburos Metalicos, S.A. Analyzing system for the detection of reducing and oxidizing gases in a carrier gas with a metal-oxide-semiconductor sensor arrangement
US20060092178A1 (en) 2004-10-29 2006-05-04 Tanguay Donald O Jr Method and system for communicating through shared media
US20060125799A1 (en) 2004-08-06 2006-06-15 Hillis W D Touch driven method and apparatus to integrate and display multiple image layers forming alternate depictions of same subject matter
EP1689172A1 (en) 2001-06-05 2006-08-09 Reactrix Systems, Inc. Interactive video display system
US20060187196A1 (en) 2005-02-08 2006-08-24 Underkoffler John S System and method for gesture based control system
US20060203363A1 (en) 2003-08-08 2006-09-14 Patrick Levy-Rosenthal Three-dimensional image display system
US20060209021A1 (en) 2005-03-19 2006-09-21 Jang Hee Yoo Virtual mouse driving apparatus and method using two-handed gestures
US7129927B2 (en) 2000-03-13 2006-10-31 Hans Arvid Mattson Gesture recognition system
US20070024590A1 (en) 2004-02-18 2007-02-01 Krepec Rafal J Camera assisted pen tablet
US20070064092A1 (en) 2005-09-09 2007-03-22 Sandbeg Roy B Mobile video teleconferencing system and control method
EP1879130A2 (en) 2006-07-13 2008-01-16 Northrop Grumman Corporation Gesture recognition interface system
EP1879129A1 (en) 2006-07-13 2008-01-16 Northrop Grumman Corporation Gesture recognition simultation system and method
US20080028325A1 (en) 2006-07-25 2008-01-31 Northrop Grumman Corporation Networked gesture collaboration system
US20080043106A1 (en) 2006-08-10 2008-02-21 Northrop Grumman Corporation Stereo camera intrusion detection system
WO2008001202A3 (en) 2006-06-28 2008-05-22 Vuokko T Lantz Touchless gesture based input
US20080150913A1 (en) 2002-05-28 2008-06-26 Matthew Bell Computer vision based touch screen
US20080244468A1 (en) 2006-07-13 2008-10-02 Nishihara H Keith Gesture Recognition Interface System with Vertical Display
US20090015791A1 (en) 2007-07-12 2009-01-15 Nelson Liang An Chang System and method for generating correspondence mappings using infrared patterns
US20090103780A1 (en) * 2006-07-13 2009-04-23 Nishihara H Keith Hand-Gesture Recognition Method
EP2056185A2 (en) 2007-11-02 2009-05-06 Northrop Grumman Space & Mission Systems Corporation Gesture recognition light and video image projector
US20090116742A1 (en) 2007-11-01 2009-05-07 H Keith Nishihara Calibration of a Gesture Recognition Interface System
GB2460937A (en) 2008-06-20 2009-12-23 Northrop Grumman Corp A gesture recognition system
US20100050133A1 (en) 2008-08-22 2010-02-25 Nishihara H Keith Compound Gesture Recognition

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1262366C (en) * 2000-08-11 2006-07-05 株式会社阿玛达 Bending device

Patent Citations (108)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4468694A (en) * 1980-12-30 1984-08-28 International Business Machines Corporation Apparatus and method for remote displaying and sensing of information using shadow parallax
US4843568A (en) 1986-04-11 1989-06-27 Krueger Myron W Real time perception of and response to the actions of an unencumbered participant/user
JPS62264390A (en) 1986-05-12 1987-11-17 Omron Tateisi Electronics Co Visual sense recognizing device for supervisory robot
US4924506A (en) 1986-07-22 1990-05-08 Schlumberger Systems & Services, Inc. Method for directly measuring area and volume using binocular stereo vision
JPH0431996A (en) 1990-05-28 1992-02-04 Matsushita Electric Works Ltd Image recognition type burgler sensor
US5220441A (en) 1990-09-28 1993-06-15 Eastman Kodak Company Mechanism for determining parallax between digital images
US5239373A (en) 1990-12-26 1993-08-24 Xerox Corporation Video computational shared drawing space
JPH04271423A (en) 1991-02-27 1992-09-28 Nippon Telegr & Teleph Corp <Ntt> Information input method
US5483261A (en) 1992-02-14 1996-01-09 Itu Research, Inc. Graphical input controller and method with rear screen image detection
US5999185A (en) 1992-03-30 1999-12-07 Kabushiki Kaisha Toshiba Virtual reality control using image, model and control data to manipulate interactions
EP0571702A2 (en) 1992-05-26 1993-12-01 Takenaka Corporation Hand pointing type input unit and wall computer module
JPH06226439A (en) 1993-02-05 1994-08-16 Matsushita Electric Ind Co Ltd Heater
US5475422A (en) 1993-06-21 1995-12-12 Nippon Telegraph And Telephone Corporation Method and apparatus for reconstructing three-dimensional objects
US20020126161A1 (en) * 1994-07-05 2002-09-12 Hitachi, Ltd. Information processing system
US5563988A (en) 1994-08-01 1996-10-08 Massachusetts Institute Of Technology Method and system for facilitating wireless, full-body, real-time user interaction with a digitally represented visual environment
US6327381B1 (en) 1994-12-29 2001-12-04 Worldscape, Llc Image transformation and synthesis methods
US5913727A (en) 1995-06-02 1999-06-22 Ahdoot; Ned Interactive movement and contact simulation game
US20010006426A1 (en) 1996-07-18 2001-07-05 Korea Institute Of Science And Technology Holographic projection screen for displaying a three-dimensional color images and optical display system using the holographic screen
US6002808A (en) 1996-07-26 1999-12-14 Mitsubishi Electric Information Technology Center America, Inc. Hand gesture control system
WO1998013746A1 (en) 1996-09-27 1998-04-02 Samsung Electronics Co. Ltd. Method for feeding information into a computer
US6956573B1 (en) 1996-11-15 2005-10-18 Sarnoff Corporation Method and apparatus for efficiently representing storing and accessing video information
US6128003A (en) 1996-12-20 2000-10-03 Hitachi, Ltd. Hand gesture recognition system and method
US6353428B1 (en) 1997-02-28 2002-03-05 Siemens Aktiengesellschaft Method and device for detecting an object in an area radiated by waves in the invisible spectral range
US20010043719A1 (en) 1997-03-21 2001-11-22 Kenichi Harakawa Hand pointing device
US20050088714A1 (en) 1997-07-08 2005-04-28 Kremen Stanley H. Method for creating a holographic screen that reconstructs uniformly magnified three-dimensional images from projected integral photographs
US7042440B2 (en) 1997-08-22 2006-05-09 Pryor Timothy R Man machine interfaces and applications
US6720949B1 (en) 1997-08-22 2004-04-13 Timothy R. Pryor Man machine interfaces and applications
DE19739285C1 (en) 1997-09-08 1998-11-19 Siemens Nixdorf Inf Syst Data representation and virtual input device
EP0913790A1 (en) 1997-10-29 1999-05-06 Takenaka Corporation Hand pointing apparatus
US6434255B1 (en) 1997-10-29 2002-08-13 Takenaka Corporation Hand pointing apparatus
US6714901B1 (en) 1997-11-19 2004-03-30 Inria Institut National De Recherche En Informatique Et En Automatique Electronic device for processing image-data, for simulating the behaviour of a deformable object
US6195104B1 (en) 1997-12-23 2001-02-27 Philips Electronics North America Corp. System and method for permitting three-dimensional navigation through a virtual reality environment using camera-based gesture inputs
US6512507B1 (en) 1998-03-31 2003-01-28 Seiko Epson Corporation Pointing position detection device, presentation system, and method, and computer-readable medium
US6806849B2 (en) 1998-04-20 2004-10-19 Lightspace Technologies Ab Multi-planar volumetric display system and method of operation using multi-planar interlacing
WO2000002187A1 (en) 1998-07-01 2000-01-13 Deluca Michael J Stereoscopic user interface method and apparatus
US6950534B2 (en) 1998-08-10 2005-09-27 Cybernet Systems Corporation Gesture-controlled interfaces for self-service machines and other applications
US6681031B2 (en) 1998-08-10 2004-01-20 Cybernet Systems Corporation Gesture-controlled interfaces for self-service machines and other applications
US6359612B1 (en) 1998-09-30 2002-03-19 Siemens Aktiengesellschaft Imaging system for displaying image information that has been acquired by means of a medical diagnostic imaging device
WO2000021023A1 (en) 1998-10-07 2000-04-13 Intel Corporation Controlling a pointer using digital video
US6222465B1 (en) 1998-12-09 2001-04-24 Lucent Technologies Inc. Gesture-based computer interface
US6147678A (en) * 1998-12-09 2000-11-14 Lucent Technologies Inc. Video hand image-three-dimensional computer interface with multiple degrees of freedom
US6204852B1 (en) 1998-12-09 2001-03-20 Lucent Technologies Inc. Video hand image three-dimensional computer interface
US6695770B1 (en) 1999-04-01 2004-02-24 Dominic Kin Leung Choy Simulated human interaction systems
US20020122113A1 (en) 1999-08-09 2002-09-05 Foote Jonathan T. Method and system for compensating for parallax in multiple camera systems
US7129927B2 (en) 2000-03-13 2006-10-31 Hans Arvid Mattson Gesture recognition system
US6624833B1 (en) 2000-04-17 2003-09-23 Lucent Technologies Inc. Gesture-based input interface system with shadow detection
US20030085866A1 (en) 2000-06-06 2003-05-08 Oliver Bimber Extended virtual table: an optical extension for table-like projection systems
US6788809B1 (en) 2000-06-30 2004-09-07 Intel Corporation System and method for gesture recognition in three dimensions using stereo imaging and color vision
US20040046747A1 (en) 2000-09-26 2004-03-11 Eugenio Bustamante Providing input signals
US20040108990A1 (en) 2001-01-08 2004-06-10 Klony Lieberman Data input device
EP1223537A2 (en) 2001-01-09 2002-07-17 Siemens Aktiengesellschaft Controllable device with user authentication
US20020090146A1 (en) 2001-01-09 2002-07-11 Siemens Aktiengesellschaft Hand recognition with position determination
US20020093666A1 (en) * 2001-01-17 2002-07-18 Jonathan Foote System and method for determining the location of a target in a room or small area
US20040113885A1 (en) 2001-05-31 2004-06-17 Yakup Genc New input devices for augmented reality applications
EP1689172A1 (en) 2001-06-05 2006-08-09 Reactrix Systems, Inc. Interactive video display system
US20020186221A1 (en) 2001-06-05 2002-12-12 Reactrix Systems, Inc. Interactive video display system
US7259747B2 (en) 2001-06-05 2007-08-21 Reactrix Systems, Inc. Interactive video display system
WO2003026299A1 (en) 2001-09-14 2003-03-27 Accenture Global Services Gmbh Lab window collaboration
US20030058341A1 (en) 2001-09-27 2003-03-27 Koninklijke Philips Electronics N.V. Video based detection of fall-down and other events
US20030067537A1 (en) 2001-10-04 2003-04-10 Myers Kenneth J. System and method for three-dimensional data acquisition
US6983065B1 (en) 2001-12-28 2006-01-03 Cognex Technology And Investment Corporation Method for extracting features from an image using oriented filters
US20050275628A1 (en) 2002-01-25 2005-12-15 Alias Systems Corp. System for physical rotation of volumetric display enclosures to facilitate viewing
US20030156756A1 (en) * 2002-02-15 2003-08-21 Gokturk Salih Burak Gesture recognition system using depth perceptive sensors
US20050252714A1 (en) 2002-04-26 2005-11-17 Hirofumi Goda Sound absorbing body, sound absorbing structural body, and method of manufacturing these bodies
US20030218761A1 (en) 2002-05-22 2003-11-27 Carlo Tomasi Method and apparatus for approximating depth of an object's placement onto a monitored region with applications to virtual interface devices
US20080150913A1 (en) 2002-05-28 2008-06-26 Matthew Bell Computer vision based touch screen
US20050110964A1 (en) 2002-05-28 2005-05-26 Matthew Bell Interactive video window display system
US6857746B2 (en) 2002-07-01 2005-02-22 Io2 Technology, Llc Method and system for free-space imaging display and interface
US20050068537A1 (en) 2002-07-17 2005-03-31 New York University Method and apparatus for determining a bidirectional reflectance distribution function of a subject
US20040125207A1 (en) 2002-08-01 2004-07-01 Anurag Mittal Robust stereo-driven video-based surveillance
US20040183775A1 (en) 2002-12-13 2004-09-23 Reactrix Systems Interactive directed light/sound system
US20060052953A1 (en) 2003-01-02 2006-03-09 Sociedad Espanola De Carburos Metalicos, S.A. Analyzing system for the detection of reducing and oxidizing gases in a carrier gas with a metal-oxide-semiconductor sensor arrangement
US20040193413A1 (en) 2003-03-25 2004-09-30 Wilson Andrew D. Architecture for controlling a computer using hand gestures
US20040239761A1 (en) 2003-05-26 2004-12-02 S1 Corporation Method of intruder detection and device thereof
US6796656B1 (en) 2003-06-14 2004-09-28 Imatte, Inc. Generating a matte signal from a retro reflective component of a front projection screen
US20050002074A1 (en) 2003-07-03 2005-01-06 Holotouch, Inc. Holographic human-machine interfaces
US20050012817A1 (en) 2003-07-15 2005-01-20 International Business Machines Corporation Selective surveillance system with active sensor management policies
US20050052714A1 (en) 2003-07-24 2005-03-10 Zebra Imaging, Inc. Enhanced environment visualization using holographic stereograms
US20060203363A1 (en) 2003-08-08 2006-09-14 Patrick Levy-Rosenthal Three-dimensional image display system
US20050151850A1 (en) 2004-01-14 2005-07-14 Korea Institute Of Science And Technology Interactive presentation system
US20050166163A1 (en) 2004-01-23 2005-07-28 Chang Nelson L.A. Systems and methods of interfacing with a machine
US20070024590A1 (en) 2004-02-18 2007-02-01 Krepec Rafal J Camera assisted pen tablet
US20050286101A1 (en) 2004-04-13 2005-12-29 Board Of Regents, The University Of Texas System Holographic projector
US20050285945A1 (en) 2004-06-25 2005-12-29 Hitachi, Ltd. Imaging apparatus
US20060010400A1 (en) 2004-06-28 2006-01-12 Microsoft Corporation Recognizing gestures and using gestures for interacting with software applications
US20060125799A1 (en) 2004-08-06 2006-06-15 Hillis W D Touch driven method and apparatus to integrate and display multiple image layers forming alternate depictions of same subject matter
US20060036944A1 (en) 2004-08-10 2006-02-16 Microsoft Corporation Surface UI for gesture-based interaction
US20060092178A1 (en) 2004-10-29 2006-05-04 Tanguay Donald O Jr Method and system for communicating through shared media
US20060187196A1 (en) 2005-02-08 2006-08-24 Underkoffler John S System and method for gesture based control system
US20060209021A1 (en) 2005-03-19 2006-09-21 Jang Hee Yoo Virtual mouse driving apparatus and method using two-handed gestures
US20070064092A1 (en) 2005-09-09 2007-03-22 Sandbeg Roy B Mobile video teleconferencing system and control method
WO2008001202A3 (en) 2006-06-28 2008-05-22 Vuokko T Lantz Touchless gesture based input
EP1879129A1 (en) 2006-07-13 2008-01-16 Northrop Grumman Corporation Gesture recognition simultation system and method
US20080013826A1 (en) 2006-07-13 2008-01-17 Northrop Grumman Corporation Gesture recognition interface system
EP1879130A2 (en) 2006-07-13 2008-01-16 Northrop Grumman Corporation Gesture recognition interface system
US20090103780A1 (en) * 2006-07-13 2009-04-23 Nishihara H Keith Hand-Gesture Recognition Method
US20080244468A1 (en) 2006-07-13 2008-10-02 Nishihara H Keith Gesture Recognition Interface System with Vertical Display
US7701439B2 (en) 2006-07-13 2010-04-20 Northrop Grumman Corporation Gesture recognition simulation system and method
US20080028325A1 (en) 2006-07-25 2008-01-31 Northrop Grumman Corporation Networked gesture collaboration system
US20080043106A1 (en) 2006-08-10 2008-02-21 Northrop Grumman Corporation Stereo camera intrusion detection system
US20090015791A1 (en) 2007-07-12 2009-01-15 Nelson Liang An Chang System and method for generating correspondence mappings using infrared patterns
US20090116742A1 (en) 2007-11-01 2009-05-07 H Keith Nishihara Calibration of a Gesture Recognition Interface System
EP2068230A2 (en) 2007-11-01 2009-06-10 Northrop Grumman Space & Mission Systems Corp. Calibration of a gesture recognition interface system
EP2056185A2 (en) 2007-11-02 2009-05-06 Northrop Grumman Space & Mission Systems Corporation Gesture recognition light and video image projector
US20090115721A1 (en) 2007-11-02 2009-05-07 Aull Kenneth W Gesture Recognition Light and Video Image Projector
GB2460937A (en) 2008-06-20 2009-12-23 Northrop Grumman Corp A gesture recognition system
US20090316952A1 (en) 2008-06-20 2009-12-24 Bran Ferren Gesture recognition interface system with a light-diffusive screen
US20100050133A1 (en) 2008-08-22 2010-02-25 Nishihara H Keith Compound Gesture Recognition

Non-Patent Citations (29)

* Cited by examiner, † Cited by third party
Title
Bretzner, et al.: "Hand Gesture Recognition Using Multi-Scale Colour Features, Hierarchical Models and Particle Filtering"; Automatic Face and Gesture Recognition, 2002, Proceedings. Fifth IEEE International Conference on, IEEE, Piscataway, NJ, USA, May 20, 2002, pp. 423-428, XP010949393, ISBN: 978-0-7695-1602-8, p. 2.
British Search Report for corresponding GB 0909597.7 completed Sep. 17, 2009.
British Search Report for corresponding GB0910067.8, completed Oct. 15, 2009.
DE Office Action for corresponding DE 10 2009 043 798.3, issued Nov. 10, 2010.
Dubois, et al.: "In Vivo Measurement of Surgical Gestures"; IEEE Transactions on Biochemical Engineering, vol. 49, No. 1, Jan. 2002, pp. 49-54.
EP Search Report for corresponding EP 07 25 2716, completed Jun. 4, 2010, The Hague.
EP Search Report for corresponding EP 07 25 2870 completed Aug. 16, 2010 by Suphi Umut Naci of the Hague.
European Search Report for corresponding EP 07 25 2717 completed Sep. 27, 2007 by Martin Müller of the EPO.
Fiorentino, et al.: "Spacedesign: A Mixed Reality Workspace for Aesthetic Industrial Design"; Mixed and Augmented Reality, 2002. ISMAR 2002. Proceedings. International Symposium on Sep. 30-Oct. 1, 2002, Piscataway, NJ, USA, IEEE, Sep. 30, 2002, pp. 86-318, XP010620945, ISBN: 0-7695-1781-1; Abstract, Figs. 1, 2; p. 86, left-hand col. ¶4; p. 87, left-hand col. ¶4-right-hand col.
German Office Action for corresponding DE 10 2009 034 413.6-53, issued Sep. 29, 2010.
Hartley, et al.: "Multiple View Geometry in Computer Vision, Structure Computation"; Jul. 31, 2000, Multiple View Geometry in Computer Vision, Cambridge University Press, GB, pp. 295-311, XP002521742, ISBN: 9780521623049, pp. 295-311, figures 11.1, 11.2 & 11.7.
Ishibuci, et al.: "Real Time Hand Gesture Recognition Using 3D Prediction Model"; Proceedings of the International Conference on Systems, Man and Cybernetics. Le Touquet, Oct. 17-20, 1993; New York, IEEE, US LNKD-DOI: 10.1109/ICSMC. 1993. 390870, vol. -, Oct. 17, 1993, pp. 324-328, XP010132504, ISBN: 978-0-7803-0911-1, pp. 325; figures 1-4.
Kjeldsen, et al.: "Toward the Use of Gesture in Traditional User Interfaces"; Automatic Face and Gesture Recognition, 1996, Proceedings of the Second International Conference on Killington, VT, USA 14-16 19961014' Los Alamitos, CA, USA, IEEE Comput. Soc., ISBN 978-0-8186-7713-7; whole document.
Korida, K et al: "An Interactive 3D Interface for a Virtual Ceramic Art Work Environment"; Virtual Systems and Multimedia, 1997. VSMM '97. Proceedings., International Conference on Geneva, Switzerland Sep. 10-12, 1997, Los Alamitos, CA, USA, IEEE Comput. Soc, US, Sep. 10, 1997, pp. 227-234, XP010245649, ISBN: 0-8186-8150-0; Abstract, Figs. 1, 2, 5, 7-11.
Leibe, et al.: "Toward Spontaneous Interaction with the Perceptive Workbench"; IEEE Computer Graphics and Applications; p. 54-65XP-000969594; Nov./Dec. 2000.
Mitchell: "Virtual Mouse"; IP.COM Inc, West Henrietta, NY, US, May 1, 1992 ISSN 1533-0001; whole document.
Office Action for corresponding DE 10 2009 025 236.3, issued May 2010.
Pajares, et al.: "Usability Analysis of a Pointing Gesture Interface"; Systems, Man and Cybernetic, 2004 IEEE International Conference on , Oct. 10, 2004, ISBN 978-0-7803-8566-5; see e.g. sections 2.1 and 4.
Pavlovic, et al.: "Visual Interpretation of Hand Gestures for Human-Computer Interaction: A Review"; Jul. 1, 1997, IEEE Transactions on Pattern Analysis and Machine Intelligence, IEEE Service Center, Los Alamitos, CA, US LNKD-DOI: 10.1109/34.598226, pp. 677-695, XP000698168, ISSN: 0162-8828, pp. 14-16, figure 5.
Plesniak, W et al.: "Spatial Interaction with Haptic Holograms"; Multimedia Computing and Systems, 1999, IEEE International Conference on Florence, Italy Jun. 7-11, 1999, Los Alamitos, CA USA, IEEE Comput. Soc. US, vol. 1, Jun. 7, 1999, pp. 413-426, XP010342817 ISBN: 0-7695-0253-9; Abstract, Figs. 7, 8.
Rehg: "Visual Analysis of High DOF Articulated Objects with Application to Hand Tracking"; [online] 1995, XP002585209, School of Computer Science Carnegie Mellon University, Retrieved from the internet: URL: http//www.dtoc/,o;/cgi-bin/GetTRDoc?AD=ADA306677&Location=U2&doc=GetRDoc.pdf> [retrieved on May 25, 2010], p. 1, 28, 31.
Sato, Y et al.: "Real-Time Input of 3D Pose and Gestures of a User's Hand and Its Applications for HCI"; Proceedings IEEE 2001 virtual Reality. (VR). Yokohama, Japan, Mar. 13, 2001, pp. 79-86, XP010535487; ISBN: 0-7695-0948-7; Abstract, Figs. 3, 4, 6, 8.
Search Report for corresponding British application No. GB0917797.3; completed Jan. 28, 2010 by Mr. Jeremy Cowen.
Search Report for corresponding GB 0715481.8, Date of Search: Nov. 27, 2007.
Search Report for corresponding GB 0913330.7; Completed Nov. 3, 2009 by Dr. Russell Maurice.
Sonka, et al.: "Image Processing, Analysis, and Machine Vision Second Edition"; Sep. 30, 1998, Thomson, XP002585208, ISBN: 053495393X, p. v-xii, p. 82-89.
Sutcliffe, et al.: "Presence, Memory and Interaction in Virtual Environments"; International Journal of Human-Computer Studies, 62 (2005), pp. 307-327.
Vámossy, et al.: "Virtual Hand-Hand Gesture Recognition System"; SISY 2007, 5th International Symposium on Intelligent Systems and Informatics, Aug. 24-25, 2007, Subolica, Serbia, IEEE, p. 97-102.
Vámossy, et al.: "Virtual Hand—Hand Gesture Recognition System"; SISY 2007, 5th International Symposium on Intelligent Systems and Informatics, Aug. 24-25, 2007, Subolica, Serbia, IEEE, p. 97-102.

Cited By (71)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9766716B2 (en) 2005-10-21 2017-09-19 Apple Inc. Three-dimensional imaging and display system
US8780332B2 (en) * 2005-10-21 2014-07-15 Apple Inc. Three-dimensional imaging and display system
US9958960B2 (en) 2005-10-21 2018-05-01 Apple Inc. Three-dimensional imaging and display system
US20110298798A1 (en) * 2005-10-21 2011-12-08 Apple Inc. Three-dimensional imaging and display system
US20110298704A1 (en) * 2005-10-21 2011-12-08 Apple Inc. Three-dimensional imaging and display system
US8743345B2 (en) * 2005-10-21 2014-06-03 Apple Inc. Three-dimensional imaging and display system
US8638984B2 (en) * 2008-04-21 2014-01-28 Carl Zeiss Industrielle Messtechnik Gmbh Display of results of a measurement of workpieces as a function of the detection of the gesture of a user
US20110035952A1 (en) * 2008-04-21 2011-02-17 Carl Zeiss Industrielle Messtechnik Gmbh Display of results of a measurement of workpieces as a function of the detection of the gesture of a user
US9384578B2 (en) 2010-02-22 2016-07-05 Nike, Inc. Augmented reality design system
US20110205242A1 (en) * 2010-02-22 2011-08-25 Nike, Inc. Augmented Reality Design System
US8947455B2 (en) * 2010-02-22 2015-02-03 Nike, Inc. Augmented reality design system
US9858724B2 (en) 2010-02-22 2018-01-02 Nike, Inc. Augmented reality design system
US8452057B2 (en) * 2010-05-17 2013-05-28 Hon Hai Precision Industry Co., Ltd. Projector and projection control method
US20110280441A1 (en) * 2010-05-17 2011-11-17 Hon Hai Precision Industry Co., Ltd. Projector and projection control method
US8457353B2 (en) * 2010-05-18 2013-06-04 Microsoft Corporation Gestures and gesture modifiers for manipulating a user-interface
US20110289456A1 (en) * 2010-05-18 2011-11-24 Microsoft Corporation Gestures And Gesture Modifiers For Manipulating A User-Interface
US20120319945A1 (en) * 2010-05-21 2012-12-20 Mccarthy John System and method for reporting data in a computer vision system
US9747697B2 (en) 2010-08-31 2017-08-29 Cast Group Of Companies Inc. System and method for tracking
US8854594B2 (en) * 2010-08-31 2014-10-07 Cast Group Of Companies Inc. System and method for tracking
US9350923B2 (en) 2010-08-31 2016-05-24 Cast Group Of Companies Inc. System and method for tracking
US20120050535A1 (en) * 2010-08-31 2012-03-01 Gilray Densham System and method for tracking
US9055226B2 (en) 2010-08-31 2015-06-09 Cast Group Of Companies Inc. System and method for controlling fixtures based on tracking data
US8723789B1 (en) 2011-02-11 2014-05-13 Imimtek, Inc. Two-dimensional method and system enabling three-dimensional user interaction with a device
US9857868B2 (en) 2011-03-19 2018-01-02 The Board Of Trustees Of The Leland Stanford Junior University Method and system for ergonomic touch-free interface
US20130010068A1 (en) * 2011-04-12 2013-01-10 Radiation Monitoring Devices, Inc. Augmented reality system
US20140033141A1 (en) * 2011-04-13 2014-01-30 Nokia Corporation Method, apparatus and computer program for user control of a state of an apparatus
US8840466B2 (en) 2011-04-25 2014-09-23 Aquifi, Inc. Method and system to create three-dimensional mapping in a two-dimensional game
US9504920B2 (en) 2011-04-25 2016-11-29 Aquifi, Inc. Method and system to create three-dimensional mapping in a two-dimensional game
US9207773B1 (en) 2011-05-13 2015-12-08 Aquifi, Inc. Two-dimensional method and system enabling three-dimensional user interaction with a device
US8686943B1 (en) 2011-05-13 2014-04-01 Imimtek, Inc. Two-dimensional method and system enabling three-dimensional user interaction with a device
US8740702B2 (en) * 2011-05-31 2014-06-03 Microsoft Corporation Action trigger gesturing
US20120309516A1 (en) * 2011-05-31 2012-12-06 Microsoft Corporation Action trigger gesturing
US9477319B1 (en) 2011-06-27 2016-10-25 Amazon Technologies, Inc. Camera based sensor for motion detection
US8773512B1 (en) * 2011-06-30 2014-07-08 Aquifi, Inc. Portable remote control device enabling three-dimensional user interaction with at least one appliance
US20130033571A1 (en) * 2011-08-03 2013-02-07 General Electric Company Method and system for cropping a 3-dimensional medical dataset
US8817076B2 (en) * 2011-08-03 2014-08-26 General Electric Company Method and system for cropping a 3-dimensional medical dataset
US8854433B1 (en) 2012-02-03 2014-10-07 Aquifi, Inc. Method and system enabling natural user interface gestures with an electronic system
US9600078B2 (en) 2012-02-03 2017-03-21 Aquifi, Inc. Method and system enabling natural user interface gestures with an electronic system
US9098739B2 (en) 2012-06-25 2015-08-04 Aquifi, Inc. Systems and methods for tracking human hands using parts based template matching
US8830312B2 (en) 2012-06-25 2014-09-09 Aquifi, Inc. Systems and methods for tracking human hands using parts based template matching within bounded regions
US8655021B2 (en) 2012-06-25 2014-02-18 Imimtek, Inc. Systems and methods for tracking human hands by performing parts based template matching using images from multiple viewpoints
US8934675B2 (en) 2012-06-25 2015-01-13 Aquifi, Inc. Systems and methods for tracking human hands by performing parts based template matching using images from multiple viewpoints
US9111135B2 (en) 2012-06-25 2015-08-18 Aquifi, Inc. Systems and methods for tracking human hands using parts based template matching using corresponding pixels in bounded regions of a sequence of frames that are a specified distance interval from a reference camera
US20150186039A1 (en) * 2012-08-27 2015-07-02 Citizen Holdings Co., Ltd. Information input device
US9310891B2 (en) 2012-09-04 2016-04-12 Aquifi, Inc. Method and system enabling natural user interface gestures with user wearable glasses
WO2014070793A3 (en) * 2012-10-29 2014-06-26 Amazon Technologies, Inc. Gesture detection systems
WO2014070793A2 (en) * 2012-10-29 2014-05-08 Amazon Technologies, Inc. Gesture detection systems
US9612655B2 (en) 2012-10-31 2017-04-04 Audi Ag Method for inputting a control command for a component of a motor vehicle
US9213448B2 (en) 2012-11-29 2015-12-15 Pixart Imaging Inc. Positioning module, optical touch system and method of calculating a coordinate of a touch medium
US20140146016A1 (en) * 2012-11-29 2014-05-29 Pixart Imaging Inc. Positioning module, optical touch system and method of calculating a coordinate of a touch medium
US9134855B2 (en) * 2012-11-29 2015-09-15 Pixart Imaging Inc. Positioning module, optical touch system and method of calculating a coordinate of a touch medium
US20140176559A1 (en) * 2012-12-21 2014-06-26 Samsung Display Co., Ltd. Image display system
US8923562B2 (en) 2012-12-24 2014-12-30 Industrial Technology Research Institute Three-dimensional interactive device and operation method thereof
US9176588B2 (en) 2013-01-14 2015-11-03 Maxm Integrated Products, Inc. System and method for discerning complex gestures using an array of optical sensors
US9507433B1 (en) 2013-01-14 2016-11-29 Maxim Integrated Products, Inc. System and method for discerning complex gestures using an array of optical sensors
US9129155B2 (en) 2013-01-30 2015-09-08 Aquifi, Inc. Systems and methods for initializing motion tracking of human hands using template matching within bounded regions determined using a depth map
US9092665B2 (en) 2013-01-30 2015-07-28 Aquifi, Inc Systems and methods for initializing motion tracking of human hands
US8615108B1 (en) 2013-01-30 2013-12-24 Imimtek, Inc. Systems and methods for initializing motion tracking of human hands
US9501151B2 (en) 2013-02-13 2016-11-22 Google Inc. Simultaneous multi-user marking interactions
US9158381B2 (en) 2013-02-25 2015-10-13 Honda Motor Co., Ltd. Multi-resolution gesture recognition
US9372541B2 (en) 2013-03-11 2016-06-21 Honeywell International Inc. Gesture recognition system operability verification
US9298266B2 (en) 2013-04-02 2016-03-29 Aquifi, Inc. Systems and methods for implementing three-dimensional (3D) gesture based graphical user interfaces (GUI) that incorporate gesture reactive interface objects
US9798388B1 (en) 2013-07-31 2017-10-24 Aquifi, Inc. Vibrotactile system to augment 3D input systems
US9507417B2 (en) 2014-01-07 2016-11-29 Aquifi, Inc. Systems and methods for implementing head tracking based graphical user interfaces (GUI) that incorporate gesture reactive interface objects
US9619105B1 (en) 2014-01-30 2017-04-11 Aquifi, Inc. Systems and methods for gesture based interaction with viewpoint dependent user interfaces
US20150375399A1 (en) * 2014-06-27 2015-12-31 Hansen Medical, Inc. User interface for medical robotics system
US10057078B2 (en) 2015-08-21 2018-08-21 Samsung Electronics Company, Ltd. User-configurable interactive region monitoring
US9594943B1 (en) * 2015-09-01 2017-03-14 International Busines Machines Corporation Image capture enhancement using dynamic control image
US20170085811A1 (en) * 2015-09-01 2017-03-23 International Business Machines Corporation Image capture enhancement using dynamic control image
US9549101B1 (en) * 2015-09-01 2017-01-17 International Business Machines Corporation Image capture enhancement using dynamic control image
US9888188B2 (en) * 2015-09-01 2018-02-06 International Business Machines Corporation Image capture enhancement using dynamic control image

Also Published As

Publication number Publication date Type
GB2460542A (en) 2009-12-09 application
US20080244468A1 (en) 2008-10-02 application
DE102009023875A1 (en) 2010-02-04 application
GB2460542B (en) 2010-12-01 grant
GB0909597D0 (en) 2009-07-15 application

Similar Documents

Publication Publication Date Title
US7598942B2 (en) System and method for gesture based control system
Wang et al. Detecting and leveraging finger orientation for interaction with direct-touch surfaces
Wilson PlayAnywhere: a compact interactive tabletop projection-vision system
US20140129990A1 (en) Interactive input system having a 3d input space
US20110141009A1 (en) Image recognition apparatus, and operation determination method and program therefor
US20110304632A1 (en) Interacting with user interface via avatar
US20110234492A1 (en) Gesture processing
US20140168062A1 (en) Systems and methods for triggering actions based on touch-free gesture detection
US20100026723A1 (en) Image magnification system for computer interface
US20120169671A1 (en) Multi-touch input apparatus and its interface method using data fusion of a single touch sensor pad and an imaging sensor
US20080013793A1 (en) Gesture recognition simulation system and method
US20110093821A1 (en) Displaying gui elements on natural user interfaces
US20100141589A1 (en) Touch input interpretation
US20100001963A1 (en) Multi-touch touchscreen incorporating pen tracking
US20100001962A1 (en) Multi-touch touchscreen incorporating pen tracking
US20110216007A1 (en) Keyboards and methods thereof
US8395600B2 (en) User interface device
US20110267265A1 (en) Spatial-input-based cursor projection systems and methods
US20120223882A1 (en) Three Dimensional User Interface Cursor Control
US20120327125A1 (en) System and method for close-range movement tracking
Agarwal et al. High precision multi-touch sensing on surfaces using overhead cameras
Wang et al. Empirical evaluation for finger input properties in multi-touch interaction
US20120206333A1 (en) Virtual touch apparatus and method without pointer on screen
US20120192119A1 (en) Usb hid device abstraction for hdtp user interfaces
Takeoka et al. Z-touch: an infrastructure for 3d gesture interaction in the proximity of tabletop surfaces

Legal Events

Date Code Title Description
AS Assignment

Owner name: NORTHROP GRUMMAN SPACE AND MISSIONS SYSTEMS CORP.,

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:NISHIHARA, H. KEITH;HSU, SHI-PING;REEL/FRAME:021054/0188;SIGNING DATES FROM 20080515 TO 20080603

Owner name: NORTHROP GRUMMAN SPACE AND MISSIONS SYSTEMS CORP.,

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:NISHIHARA, H. KEITH;HSU, SHI-PING;SIGNING DATES FROM 20080515 TO 20080603;REEL/FRAME:021054/0188

AS Assignment

Owner name: NORTHROP GRUMMAN SYSTEMS CORPORATION,CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:NORTHROP GRUMMAN SPACE & MISSION SYSTEMS CORP.;REEL/FRAME:023915/0446

Effective date: 20091210

Owner name: NORTHROP GRUMMAN SYSTEMS CORPORATION, CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:NORTHROP GRUMMAN SPACE & MISSION SYSTEMS CORP.;REEL/FRAME:023915/0446

Effective date: 20091210

FPAY Fee payment

Year of fee payment: 4