US20170024017A1 - Gesture processing - Google Patents

Gesture processing Download PDF

Info

Publication number
US20170024017A1
US20170024017A1 US15/285,944 US201615285944A US2017024017A1 US 20170024017 A1 US20170024017 A1 US 20170024017A1 US 201615285944 A US201615285944 A US 201615285944A US 2017024017 A1 US2017024017 A1 US 2017024017A1
Authority
US
United States
Prior art keywords
distance
input device
gesture
user
interaction
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US15/285,944
Inventor
Rahul AJMERA
Anbumani Subramanian
Sriganesh Madhvanath
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hewlett Packard Development Co LP
Original Assignee
Hewlett Packard Development Co LP
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority to IN846CH2010 priority Critical
Priority to IN846/CHE/2010 priority
Priority to US12/779,061 priority patent/US9477324B2/en
Application filed by Hewlett Packard Development Co LP filed Critical Hewlett Packard Development Co LP
Priority to US15/285,944 priority patent/US20170024017A1/en
Publication of US20170024017A1 publication Critical patent/US20170024017A1/en
Assigned to HEWLETT-PACKARD DEVELOPMENT COMPANY, L.P. reassignment HEWLETT-PACKARD DEVELOPMENT COMPANY, L.P. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: ANBUMANI, SUBRAMANIAN, MADHVANATH, SRIGANESH, AJMERA, RAHUL
Abandoned legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017Gesture based interaction, e.g. based on a set of recognized hand gestures
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/0304Detection arrangements using opto-electronic means
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
    • G06F3/0488Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
    • G06F3/0488Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
    • G06F3/04883Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures for entering handwritten data, e.g. gestures, text

Abstract

Presented is method and system for processing a gesture performed by a user of an input device. The method comprises detecting the gesture and determining a distance of the input device from a predetermined location. A user command is then determined based on the detected gesture and the determined distance.

Description

    RELATED APPLICATIONS
  • Benefit is claimed under 35 U.S.C. 119(a)-(d) to Foreign application Serial No. 846/CHE/2010 entitled “GESTURE PROCESSING” by Hewlett-Packard Development Company, L.P., filed on Mar. 29, 2010, which is herein incorporated in its entirety by reference for all purposes
  • BACKGROUND
  • Computing systems accept a variety of inputs. Some computer applications accept gestures provided by input devices to enable easier control and navigation of the applications.
  • Gestures are ways to invoke an action, similar to clicking a toolbar button or typing a keyboard shortcut. Gestures may be performed with a pointing device (including but not limited to a mouse, stylus, hand and/or finger). A gesture typically has a shape, pose or movement associated with it. Such a gesture may be as simple as a stationary pose or a straight line movement or as complicated as a series of movements or poses.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • For a better understanding, embodiments will now be described, purely by way of example, with reference to the accompanying drawings, in which:
  • FIG. 1 is a schematic diagram of the arrangement of apparatus according to an embodiment;
  • FIG. 2 shows detail of FIG. 1; and
  • FIG. 3 is a flowchart of a method according to an embodiment
  • FIG. 4 show an example of zones being provided around a user's body according to an embodiment; and
  • FIGS. 5a-5c show a user making a pointing gesture in the various zones of FIG. 4.
  • DETAILED DESCRIPTION
  • Embodiments provide a method of processing a gesture performed by a user of a first input device, the method comprising: detecting the gesture; detecting a distance of the first input device from a predetermined location, the detected distance being for disambiguating the gesture; and determining a user command based on the detected gesture and the detected distance. Accordingly, there is provided a natural and intuitive interface method by which to command an action using a gesture.
  • Embodiments comprise an architecture and related computational infrastructure such that the distance of a input device from a predetermined location (such as a display device or the user's upper torso) may be used so as to specify a gesture in more detail (in other words, disambiguate or qualify the gesture). Once determined, a gesture may be detected and combined with the distance to determine a command or action desired by the user. Thus, embodiments may employ hardware and software such that the distance of the input device from a predetermined location may be controlled by the user, as well as hardware and software such that a gesture can be input and detected. A variety of architectures may be used to enable such functions.
  • The distance may specify, for example, a desired command or a parameter for a command, such as the extent of a zoom-in command.
  • A natural and intuitive means of interaction is provided, enabling a user of such a system to feel as though he or she is physically interacting with the system, for example, by accurately moving an input device towards a display to select a data file. Thus, a unique and compelling gesture interface is hereby disclosed as a means of interacting with a graphical user interface (GUI).
  • Commands may be associated with the gesture. These operations may include navigation forward, backward, scrolling up or down, changing applications, and arbitrary application commands. Further, a gesture does not need to have a predefined meaning but rather may be customizable by a developer or user to perform an action or combination of actions so that a user may have quick access to keyboard shortcuts or macros, for example.
  • Different input devices may modify actions associated with gestures. For instance, a first set of actions may be associated with gestures when performed by a stylus. A second set of actions may be associated with gestures when performed by another pointing device. The number of sets of actions may be varied by the number of different input devices.
  • An embodiment, pictured in FIG. 1, provides apparatus for processing a gesture performed by a user 90 of an input device. The apparatus comprises a display surface 10; an input device 20, for performing a gesture; a range camera 30 for producing a depth-image; and a processor 50. The field of view of the range camera 30 includes the input device 20. The processor 50 is adapted to detect, from the depth-image (or otherwise), a gesture performed by a user of the input device 20. It is also adapted to determine the distance of the input device 20 from a predetermined location, such as the display surface 10 for example.
  • The processor then uses the determined distance to specify a detected gesture in more detail (in other words, disambiguate or qualify the gesture).
  • In the embodiment of FIG. 1, the input device 20 comprises part of the body of a user 90. In particular, the input device comprises the user's hand.
  • Using the input device 20, the user 90 can select, highlight, and/or modify items displayed on the display surface 10. The processor 50 interprets gestures made using the input device 20 in order to manipulate data, objects and/or execute conventional computer application tasks.
  • Other types of input devices, such as a mouse, stylus, trackball, or the like could be used. Additionally, a user's own hand or finger could be the input device 20 and used for selecting or indicating portions of a displayed image on a proximity-sensitive display. Consequently, the term “user input device”, as used herein, is intended to have a broad definition and encompasses many variations on well-known input devices.
  • The range camera 30 is sometimes also known as a depth camera. This is an imaging system which provides a two-dimensional array of depth values—that is, a depth image. Optionally, it may also produce a normal (grayscale or color) image in addition to the depth image. In the present example, the range camera is based on the time-of-flight principle: pulses of infra-red light are emitted to all objects in the field of view and the time of arrival of the reflected pulses is measured, to determine the distance from the sensor.
  • Note that range cameras of other types may also be used. The skilled person will be familiar with a variety of other potentially suitable distance-sensing technologies. These include stereo imaging, or stereo triangulation, in which two (or more) image sensors are used to determine a depth image by making disparity measurements. Another possibility is to illuminate a scene with so-called “structured light”, where a geometric pattern such as a checkerboard is projected, and depth/distance information is determined from the distortions observed when this known pattern falls on the objects in the scene.
  • In the arrangement of FIG. 1, the depth camera 30 is positioned to observe the display surface 10, from a relatively short distance of about 0.5 m to 1 m. The camera 30 is spatially positioned such that the display surface 10 is visible in the field-of-view of the camera. An interaction volume is defined as the region in front of the display where the user's hand is visible in the field of view during interaction.
  • A simple, one-time calibration procedure can be used to locate the four corners of the display surface. This may be either manual, whereby the user indicates the positions of the vertices, or could be automatic, by analysis of the image of the scene. To help with automatic or semi-automatic detection of the surface, its boundaries may be identified with markers of distinctive color or brightness. If calibration is manual, then the camera should be manually recalibrated if it is disturbed.
  • The range camera observes only one side of the hand; so the depth measurements of the side visible to the camera are inevitably biased slightly compared with the true central position of the hand 20. However, this slight bias can either be ignored or easily compensated for.
  • The distance from the hand to the display surface 10, for example, can be computed analytically to obtain an interaction distance. Thus, the distance from the display surface is determined. The distance from the user's hand to the display can be used as a measure to control of mode of interaction. By determining a distance from the input device 20 to the display surface 10, the processor 50 can control the user interaction differently according to this distance.
  • An example of this is illustrated in FIG. 2. If the distance D from the display is less than a small threshold distance, dth, a touch-like interaction happens. If the distance is greater than this threshold, a hover-like interaction occurs. A hover interaction may involve, for example, moving a pointer (similar to moving a conventional mouse). A touch interaction might involve, for example, a selection (like clicking a mouse button). One of the effects of this is to provide any display surface with the qualities of a touch-sensitive screen. That is, a surface that is not touch sensitive can be made to emulate a touch-sensitive surface. In the example of FIG. 2, the distance is measured perpendicular to the planar display surface. However, in other applications, it could be measured along a pointing axis (which may have a different orientation).
  • It will be understood that the location of the input device 20 may be made relative to other predetermined locations instead of the display surface 10. For example, the distance of a user's body may be determined from the depth image, thus enabling the distance of the input device 20 from the user's body to be determined and used to specify a detected gesture in more detail (in other words, disambiguate or qualify the gesture).
  • It will also be understood that alternative camera arrangements may be used such as a conventional arrangement of the camera in the same plane as the display and observing the user so that the display is not in the field of view of the camera.
  • The processor 50 can comprise hardware of various types. In this example, the processor is a central processing unit (CPU) of a personal computer (PC). Accordingly, the display surface 10 is the display of the PC, which is under the control of the CPU 50. The apparatus allows the user 90 to provide input to the PC by making hand gestures. The processor 50 acts on this user input by controlling an operation depending on the gesture(s) made and the location of the gesture relative to the display surface 10. The operation could be of almost any type: for example, the activation of a software application or the pressing of a button or selection of a menu item within an application. Of course, as will be readily apparent to those skilled in the art, the processor may be comprised in another hardware device, such as a set-top box (STB). The range of suitable operations which may be controlled will vary accordingly. With a STB, for example, the operation controlled may involve changing channels or browsing an electronic program guide (EPG).
  • In embodiments, the processor executes a method 100 of processing a gesture performed by a user of an input device, as illustrated in FIG. 3. The method 100 comprises: detecting 110 the gesture; capturing 120 a depth-image of the input device 20; determining 130, from the depth-image, the position of the input device 20; calculating 150, from the determined position, the distance of the first input device 20 from a predetermined location; and determining 160 a user command based on the detected gesture and the calculated distance.
  • The method can also comprise, before the step 150 of calculating the distance of the first input device 20 from a predetermined location: capturing a depth-image of a surface; and determining, from the depth-image, the location of the surface. In some embodiments, the depth image of the surface and the depth image of the input device will be the same image. The predetermined location from which the distance of the input device 20 is calculated can then be the surface. Also, in this case, the calibration or registration step of locating the surface can be done for each captured image frame (assuming it is automatic).
  • For a sequence of frames, such calibration and recalibration can include or be replaced by tracking of the position of the surface in the depth image. In other embodiments, the calibration step is carried out in advance, using a separate depth image from which the input device 20 is absent.
  • The user's hand is one simple and intuitive example of the input device 20.
  • However, other user devices may also be used. For example, the user may hold a wand or other pointer in his/her hand. This could be colored distinctively or exhibit characteristic markers, to aid detection in the image. Equally, another body part could be used, such as an extended finger or the head. In each case, the position of the input device can be calculated from a depth image.
  • In the example described above and shown in FIG. 1, the display surface is within the field-of-view of the range camera 30. However, in other embodiments, the surface may not be in the field of view. For example, the range camera could be rigidly fixed to the surface—in this case, the location of the surface relative to the camera coordinates will be fixed and known, so there is no need for the surface to appear in the scene of the depth image. The same applies if the relative arrangement of a predetermined location can be determined automatically by other means.
  • Embodiments can emulate touch control of a display surface (or any other surface) without the need for conventional touch-sensitive technology. They can also create a richer interaction experience with conventional graphic user interfaces because both touch and hover interactions may be catered for.
  • Also, the use of a range camera eliminates the need for the user to wear, carry or hold active pointing means, with inbuilt sensor or transmitter technology. Instead, the user's own hand, finger or forearm can be used as the pointing means. This can make the control of operations more intuitive and accessible. Embodiments provide an architecture and related computational infrastructure such that a location-based parameter may be provided by the user so as to specify a gesture in more detail. In the embodiment of FIG. 1, the apparatus comprises a depth camera for producing a depth-image. The depth camera is connected to a processor which is arranged to determine, using the depth image, the distance of the input device 20 from a predetermined location.
  • The determined distance can then be used as a parameter for specifying a gesture in more detail.
  • A gesture may therefore be combined with a distance parameter to determine a command or action desired by the user. Such a gesture which is combined with a parameter is hereinafter referred to as a distance-based (DB) gesture since a single gesture may be used for multiple modes of operation, the chosen mode being dependent on the distance parameter. A parameter may specify, for example, a target file location, target software program or desired command.
  • The distance-based gesture concept specifies a general pattern of interaction where there is a gesture command part and there is distance-based parameter part of an interaction. For example, a distance-based (DB) gesture according to an embodiment may be represented as follows:

  • DB Gesture=Gesture Command+Distance Parameter.
  • Thus, a DB gesture as an interaction consists of a user action performed at a particular location. When the user performs the pointing gesture, the location of the input device (relative to a predetermined location) is used as an extra parameter to specify the pointing gesture in more detail. Such a DB gesture may therefore be represented as follows:

  • DB Gesture=Pointing Gesture+Distance Parameter.
  • Considering now a DB gesture in more detail, two categories of operation can be identified: (i) Distance Measured Relative to the System or Sensor; and (ii) Distance Measured Relative to a User's Body.
  • (i) Distance Measured Relative to the System or Sensor
  • Some of the interactions that can be enabled here include the following examples.
  • Gesture Semantics Based on Proximity.
  • The concept here is that the same gesture can have different interpretations when made at different distances from the system. For example, when a user makes a pointing gesture from far away, the system can treat it as a large searchlight type of cursor. As the user comes closer to the system, the cursor reduces in size and becomes like a mouse pointer. Another example would be that as the user performs the same gesture from different distances, it operates on different levels of a multi-layered user interface.
  • Implementation: The action mapped to the same gesture (e.g. UI action) is varied based on the distance of the user from the system.
  • Control Assignment Based on Proximity
  • In a multi-user scenario, users closer to the system may be assigned more controls than users who are farther away. An exemplary scenario is a presentation where a person from the audience makes a gesture to skip to the next slide, versus the same gesture being made by the presenter. By only enabling the closest user to the presentation (i.e. the presenter) to perform a slide skipping action, the slide will only be skipped when the presenter makes the gesture. When someone in the audience makes the same gesture, the system could choose to ignore it, or treat it a request and seek confirmation from the presenter. The system could also aggregate audience requests and take appropriate action (e.g. skip the slide only more than 50% of the audience makes the gesture). On the other hand, the system may allow any one in the audience to point at an object of interest in the presentation.
  • In a variation, the assignment of control may be based on the relative distance of the users from the system, rather than the absolute distance.
  • These interactions are intuitive as most often presenters stand closer to the system whereas the audience is farther away.
  • Implementation: The gesture vocabulary may be designed such that the gestures allowed from a greater distance are a strict subset of, or different from, gestures from up close. Multiple such “tiers” may be defined. Alternatively, the gestures are the same, but the mapped action is different in the level of control or access implied (e.g. command vs. request).
  • Degree (Extent) of Command Based on Proximity
  • The degree or extent of impact resulting from a gesture (especially for manipulative gestures) may be varied based on the distance of the input device or user from the system. For example, a zoom-in gesture made from up close may result in a smaller amount of scaling that the same gesture made from a distance. Similarly for pointing gestures, the same movement of the pointing hand may translate to the larger cursor movement on the screen, if the user is farther away.
  • Implementation: The parameters of the mapped action such as “zoom-in image” or “move cursor” may be scaled or varied in proportion to the distance.
  • Type or Vocabulary of Gestures Based on Distance
  • The type of gesture supported may be made a function of distance. A specific instance of this is to vary the grossness of gestures made from different distances. The intuition here is that in human-to-human communication, people make gross gestures when gesturing from afar, and finer gestures when closer, to mean the same thing (e.g. moving the hand, versus the entire arm to wave goodbye). This can be used as follows. When a person walks towards the system, the kind of gestures he can produce become increasingly more granular and refined. For example, when the person is near the system, he could use small wrist-motion gestures to control a system and when far away, the person could use arm motion gestures for the same controls.
  • Implementation: The recognition system can automatically tune the vocabulary of gestures based on the distance and only allow the specific gesture set to be recognized. This tuning also reduces the processing load on the gesture recognition system as the search space is reduced.
  • The above ideas are clearly extensible to the multi-user situation wherein the distance of each user is known independently.
  • (ii) Distance Measured Relative to a User's Body
  • Some hand gestures could also be treated differently when made at different distances from the user's body. This can be used for example by creating virtual zones 100,200,300 around the user's body, as shown in FIG. 4.
  • The gesture, together with the zone that it was made in, can be mapped to an appropriate action. For example, as shown in FIG. 5a , when the user makes a pointing gesture very close to his/her body (i.e. within zone 100), the system will treat it as a large searchlight type of cursor. As the user stretches out his hand (as shown in FIG. 5b ), and enters the next zone, zone 200, the cursor reduces in size and becomes like a mouse pointer. As the user extends his arm further (as shown in FIG. 5c ), and it enters the zone, zone 300, furthest from his/her body, the system treats that as a mouse press.
  • It should be understood that the kinds of interactions defined earlier in the context of distance of the user from the system, can also be developed with respect to the distance from the body. In fact, it is also possible to imagine that in the most general case, various distances such as that of the user from the system, from other users, and the distance of the hand gesture from the body can be used together to determine the gesture vocabulary and/or interpretation.
  • Gesture Determination
  • Gestures may be performed in the natural mode without necessarily requiring the user to enter any special modes—although a mode requirement may used in alternative embodiments, for example, requiring the user to hold a button while performing a gesture. The occurrence of a gesture may be determined based on a profile of the physical or logical x and y co-ordinates charted against time.
  • A gesture may also be determined based upon timing information. Because a gesture of a human may be a quick gesture, one or more predefined thresholds can be chosen. A movement threshold may be, for example, greater than 1 cm and the time threshold greater than 0.2 milliseconds and less than 700 milliseconds. These values of course may be varied to accommodate all users. In some embodiments a threshold may be defined based upon the size of a screen and/or the distance of the graphical element from an edge of the screen.
  • In other embodiments, a velocity threshold may be used instead of or in addition to a speed threshold, wherein the velocity threshold defines a minimum velocity at which the user must move his/her finger or hand for it to qualify as a gesture.
  • Other aspects of a gesture may be compared against other thresholds. For instance, the system may calculate velocity, acceleration, curvature, lift, and the like and use these derived values or sets of values to determine if a user has performed a gesture.
  • While specific embodiments have been described herein for purposes of illustration, various other modifications will be apparent to a person skilled in the art and may be made without departing from the scope of the concepts disclosed.
  • For example, an embodiment may comprise a computing device having a processing unit, input device location detection means and data storage means (not visible). The data storage means stores one or more software programs for controlling the operation of the computing device. The software program includes routines for enabling multi-modal gestures to be used wherein a physical gesture (such as a flick) imparted by the user upon can be disambiguated or further defined by a distance-related parameter. These routines may be implemented in hardware and/or software and may be implemented in a variety of ways. In general, the routines are configured to determine when a user provides a gesture and to determine a distance of a user input device from a predetermined location.

Claims (21)

1-15. (canceled)
16. A method of processing a gesture performed by a user of an input device, the method comprising:
detecting the gesture;
determining a distance of the input device from a predetermined location; and
determining a user command based on the detected gesture and the determined distance,
the detected gesture at a first determined distance indicating a first command, and the detected gesture at a second determined distance indicating a second command different than the first command
17. The method of claim 16, wherein detecting the gesture comprises:
comparing the determined distance with a threshold distance,
the detected gesture at the determined distance less than the threshold distance indicating a first type of interaction, and the detected gesture at the determined distance greater than the threshold distance indicating a second type of interaction.
18. The method of claim 17, wherein the first type of interaction emulates a touch interaction and the second type of interaction emulates a hover interaction.
19. The method of claim 17, wherein the first type of interaction comprises a selection command and the second type of interaction comprises a movement command
20. The method of claim 16, wherein determining the distance of the input device from the predetermined location comprises generating a depth image and processing the depth image in accordance with a depth recognition process.
21. The method of claim 16, wherein determining the distance of the input device from the predetermined location comprises determining the distance of the input device relative to a display.
22. The method of claim 16, wherein determining the distance of the input device from the predetermined location comprises determining the distance of the input device relative to the user's body.
23. The method of claim 16, wherein the input device comprises a body part of the user.
24. The method of claim 16, wherein the input device comprises a device held or operated by the user.
25. A system for processing a gesture performed by a user of an input device, the system comprising:
means for detecting the gesture;
means for determining a distance of the input device from a predetermined location; and
means for determining a user command based on the detected gesture and the determined distance,
the detected gesture at a first determined distance indicating a first command, and the detected gesture at a second determined distance indicating a second command different than the first command.
26. The system of claim 25, the detected gesture at the determined distance less than a threshold distance indicating a first type of interaction, and the detected gesture at the determined distance greater than the threshold distance indicating a second type of interaction.
27. The system of claim 25, wherein means for determining the distance of the input device from the predetermined location is to process a depth image in accordance with a depth recognition process.
28. The system of claim 25, wherein means for determining the distance of the input device from the predetermined location is to determine the distance of the input device relative to a display.
29. The system of claim 25, wherein means for determining the distance of the input device from the predetermined location is to determine the distance of the input device relative to the user's body.
30. A system for processing a gesture performed by a user of an input device, the system comprising:
an imaging device to produce a depth image of the gesture performed by the user of the input device;
a processor to detect the gesture, determine a distance of the input device from a predetermined location based on the depth image, and determine a user command based on the detected gesture and the determined distance,
wherein a same detected gesture at different determined distances indicates different commands.
31. The system of claim 16, wherein the same detected gesture at a determined distance less than a threshold distance indicates a first type of interaction, and the same detected gesture at a determined distance greater than the threshold distance indicates a second type of interaction.
32. The system of claim 16, wherein the first type of interaction comprises a selection command and the second type of interaction comprises a movement command
33. The system of claim 16, wherein the distance of the input device from the predetermined location comprises a distance of the input device relative to at least one of a display and the user's body.
34. The system of claim 16, wherein the input device comprises at least one of a body part of the user and a device held or operated by the user.
35. The system of claim 16, wherein the imaging device comprises a range camera.
US15/285,944 2010-03-29 2016-10-05 Gesture processing Abandoned US20170024017A1 (en)

Priority Applications (4)

Application Number Priority Date Filing Date Title
IN846CH2010 2010-03-29
IN846/CHE/2010 2010-03-29
US12/779,061 US9477324B2 (en) 2010-03-29 2010-05-13 Gesture processing
US15/285,944 US20170024017A1 (en) 2010-03-29 2016-10-05 Gesture processing

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US15/285,944 US20170024017A1 (en) 2010-03-29 2016-10-05 Gesture processing

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US12/779,061 Continuation US9477324B2 (en) 2010-03-29 2010-05-13 Gesture processing

Publications (1)

Publication Number Publication Date
US20170024017A1 true US20170024017A1 (en) 2017-01-26

Family

ID=44655797

Family Applications (2)

Application Number Title Priority Date Filing Date
US12/779,061 Active 2035-05-23 US9477324B2 (en) 2010-03-29 2010-05-13 Gesture processing
US15/285,944 Abandoned US20170024017A1 (en) 2010-03-29 2016-10-05 Gesture processing

Family Applications Before (1)

Application Number Title Priority Date Filing Date
US12/779,061 Active 2035-05-23 US9477324B2 (en) 2010-03-29 2010-05-13 Gesture processing

Country Status (1)

Country Link
US (2) US9477324B2 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160124514A1 (en) * 2014-11-05 2016-05-05 Samsung Electronics Co., Ltd. Electronic device and method of controlling the same

Families Citing this family (41)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8296151B2 (en) * 2010-06-18 2012-10-23 Microsoft Corporation Compound gesture-speech commands
US20120016641A1 (en) * 2010-07-13 2012-01-19 Giuseppe Raffa Efficient gesture processing
US20130076909A1 (en) * 2011-09-27 2013-03-28 Stefan J. Marti System and method for editing electronic content using a handheld device
US9628843B2 (en) * 2011-11-21 2017-04-18 Microsoft Technology Licensing, Llc Methods for controlling electronic devices using gestures
EP2788838A4 (en) * 2011-12-09 2015-10-14 Nokia Technologies Oy Method and apparatus for identifying a gesture based upon fusion of multiple sensor signals
KR102084041B1 (en) * 2012-08-24 2020-03-04 삼성전자 주식회사 Operation Method And System for function of Stylus pen
US9423939B2 (en) 2012-11-12 2016-08-23 Microsoft Technology Licensing, Llc Dynamic adjustment of user interface
CN103809738B (en) * 2012-11-13 2017-03-29 联想(北京)有限公司 A kind of information collecting method and electronic equipment
CN103970260B (en) * 2013-01-31 2017-06-06 华为技术有限公司 A kind of non-contact gesture control method and electric terminal equipment
US9159116B2 (en) 2013-02-13 2015-10-13 Google Inc. Adaptive screen interfaces based on viewing distance
CN104969148B (en) * 2013-03-14 2018-05-29 英特尔公司 User interface gesture control based on depth
TWI505135B (en) * 2013-08-20 2015-10-21 Utechzone Co Ltd Control system for display screen, control apparatus and control method
US10152136B2 (en) * 2013-10-16 2018-12-11 Leap Motion, Inc. Velocity field interaction for free space gesture interface and control
EP3060941A1 (en) * 2013-10-23 2016-08-31 Oculus VR, LLC Three dimensional depth mapping using dynamic structured light
ES2726990T3 (en) * 2014-01-30 2019-10-11 Signify Holding Bv Control of a lighting system using a mobile terminal
CN106164995B (en) * 2014-01-30 2019-07-12 飞利浦灯具控股公司 Ability of posture control
US10436888B2 (en) * 2014-05-30 2019-10-08 Texas Tech University System Hybrid FMCW-interferometry radar for positioning and monitoring and methods of using same
US9575560B2 (en) 2014-06-03 2017-02-21 Google Inc. Radar-based gesture-recognition through a wearable device
US9921660B2 (en) 2014-08-07 2018-03-20 Google Llc Radar-based gesture recognition
US9811164B2 (en) 2014-08-07 2017-11-07 Google Inc. Radar-based gesture sensing and data transmission
US10268321B2 (en) 2014-08-15 2019-04-23 Google Llc Interactive textiles within hard objects
US9588625B2 (en) 2014-08-15 2017-03-07 Google Inc. Interactive textiles
US9778749B2 (en) 2014-08-22 2017-10-03 Google Inc. Occluded gesture recognition
US9600080B2 (en) 2014-10-02 2017-03-21 Google Inc. Non-line-of-sight radar-based gesture recognition
US10016162B1 (en) 2015-03-23 2018-07-10 Google Llc In-ear health monitoring
US9983747B2 (en) 2015-03-26 2018-05-29 Google Llc Two-layer interactive textiles
US9848780B1 (en) 2015-04-08 2017-12-26 Google Inc. Assessing cardiovascular function using an optical sensor
WO2016176574A1 (en) 2015-04-30 2016-11-03 Google Inc. Wide-field radar-based gesture recognition
EP3521853A3 (en) 2015-04-30 2019-09-25 Google LLC Rf-based micro-motion tracking for gesture tracking and recognition
EP3289433A1 (en) * 2015-04-30 2018-03-07 Google LLC Type-agnostic rf signal representations
US10080528B2 (en) 2015-05-19 2018-09-25 Google Llc Optical central venous pressure measurement
US9693592B2 (en) 2015-05-27 2017-07-04 Google Inc. Attaching electronic components to interactive textiles
US10088908B1 (en) 2015-05-27 2018-10-02 Google Llc Gesture detection and interactions
US10376195B1 (en) 2015-06-04 2019-08-13 Google Llc Automated nursing assessment
CN106484159A (en) * 2015-08-25 2017-03-08 中兴通讯股份有限公司 A kind of method of dynamic control electronic equipment and electronic equipment
US10379621B2 (en) 2015-10-06 2019-08-13 Google Llc Gesture component with gesture library
EP3371855A1 (en) 2015-11-04 2018-09-12 Google LLC Connectors for connecting electronics embedded in garments to external devices
WO2017192167A1 (en) 2016-05-03 2017-11-09 Google Llc Connecting an electronic component to an interactive textile
WO2017200571A1 (en) * 2016-05-16 2017-11-23 Google Llc Gesture-based control of a user interface
US10175781B2 (en) 2016-05-16 2019-01-08 Google Llc Interactive object with multiple electronics modules
US10579150B2 (en) 2016-12-05 2020-03-03 Google Llc Concurrent detection of absolute distance and relative movement for sensing action gestures

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040046736A1 (en) * 1997-08-22 2004-03-11 Pryor Timothy R. Novel man machine interfaces and applications
US20060139314A1 (en) * 2002-05-28 2006-06-29 Matthew Bell Interactive video display system
US20060252474A1 (en) * 2002-07-27 2006-11-09 Zalewski Gary M Method and system for applying gearing effects to acoustical tracking
US20090172606A1 (en) * 2007-12-31 2009-07-02 Motorola, Inc. Method and apparatus for two-handed computer user interface with gesture recognition
US20090183125A1 (en) * 2008-01-14 2009-07-16 Prime Sense Ltd. Three-dimensional user interface
US20090197615A1 (en) * 2008-02-01 2009-08-06 Kim Joo Min User interface for mobile devices
US20100299642A1 (en) * 2009-05-22 2010-11-25 Thomas Merrell Electronic Device with Sensing Assembly and Method for Detecting Basic Gestures
US20110193939A1 (en) * 2010-02-09 2011-08-11 Microsoft Corporation Physical interaction zone for gesture-based user interfaces
US20110219340A1 (en) * 2010-03-03 2011-09-08 Pathangay Vinod System and method for point, select and transfer hand gesture based user interface

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7414705B2 (en) * 2005-11-29 2008-08-19 Navisense Method and system for range measurement
US8904312B2 (en) * 2006-11-09 2014-12-02 Navisense Method and device for touchless signing and recognition
US8375336B2 (en) * 2008-05-23 2013-02-12 Microsoft Corporation Panning content utilizing a drag operation
US9652030B2 (en) * 2009-01-30 2017-05-16 Microsoft Technology Licensing, Llc Navigation of a virtual plane using a zone of restriction for canceling noise

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040046736A1 (en) * 1997-08-22 2004-03-11 Pryor Timothy R. Novel man machine interfaces and applications
US20060139314A1 (en) * 2002-05-28 2006-06-29 Matthew Bell Interactive video display system
US20060252474A1 (en) * 2002-07-27 2006-11-09 Zalewski Gary M Method and system for applying gearing effects to acoustical tracking
US20090172606A1 (en) * 2007-12-31 2009-07-02 Motorola, Inc. Method and apparatus for two-handed computer user interface with gesture recognition
US20090183125A1 (en) * 2008-01-14 2009-07-16 Prime Sense Ltd. Three-dimensional user interface
US20090197615A1 (en) * 2008-02-01 2009-08-06 Kim Joo Min User interface for mobile devices
US20100299642A1 (en) * 2009-05-22 2010-11-25 Thomas Merrell Electronic Device with Sensing Assembly and Method for Detecting Basic Gestures
US20110193939A1 (en) * 2010-02-09 2011-08-11 Microsoft Corporation Physical interaction zone for gesture-based user interfaces
US20110219340A1 (en) * 2010-03-03 2011-09-08 Pathangay Vinod System and method for point, select and transfer hand gesture based user interface

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160124514A1 (en) * 2014-11-05 2016-05-05 Samsung Electronics Co., Ltd. Electronic device and method of controlling the same

Also Published As

Publication number Publication date
US20110234492A1 (en) 2011-09-29
US9477324B2 (en) 2016-10-25

Similar Documents

Publication Publication Date Title
JP6364505B2 (en) Radar-based gesture recognition
US10203765B2 (en) Interactive input system and method
US10535323B2 (en) Display zoom controlled by proximity detection
US10564799B2 (en) Dynamic user interactions for display control and identifying dominant gestures
KR20170046624A (en) Apparatus and method for providing user interface, and computer-readable recording medium recording the same
EP2972727B1 (en) Non-occluded display for hover interactions
US10261594B2 (en) Systems and methods of creating a realistic displacement of a virtual object in virtual reality/augmented reality environments
US10162429B2 (en) Gesture enabled keyboard
US9600078B2 (en) Method and system enabling natural user interface gestures with an electronic system
JP6348211B2 (en) Remote control of computer equipment
US9746934B2 (en) Navigation approaches for multi-dimensional input
KR102061360B1 (en) User interface indirect interaction
Xiao et al. WorldKit: rapid and easy creation of ad-hoc interactive applications on everyday surfaces
KR101872426B1 (en) Depth-based user interface gesture control
JP5991041B2 (en) Virtual touch screen system and bidirectional mode automatic switching method
US9459694B2 (en) Cursor movement device
CN104956292B (en) The interaction of multiple perception sensing inputs
JP2019087279A (en) Systems and methods of direct pointing detection for interaction with digital device
TWI569171B (en) Gesture recognition
US9684439B2 (en) Motion control touch screen method and apparatus
US9298266B2 (en) Systems and methods for implementing three-dimensional (3D) gesture based graphical user interfaces (GUI) that incorporate gesture reactive interface objects
US20140002355A1 (en) Interface controlling apparatus and method using force
US8432362B2 (en) Keyboards and methods thereof
US9990062B2 (en) Apparatus and method for proximity based input
TWI398818B (en) Method and system for gesture recognition

Legal Events

Date Code Title Description
AS Assignment

Owner name: HEWLETT-PACKARD DEVELOPMENT COMPANY, L.P., TEXAS

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:AJMERA, RAHUL;ANBUMANI, SUBRAMANIAN;MADHVANATH, SRIGANESH;SIGNING DATES FROM 20100414 TO 20100416;REEL/FRAME:046369/0365

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STCB Information on status: application discontinuation

Free format text: FINAL REJECTION MAILED

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION