US20140300539A1 - Gesture recognition using depth images - Google Patents

Gesture recognition using depth images Download PDF

Info

Publication number
US20140300539A1
US20140300539A1 US14271191 US201414271191A US20140300539A1 US 20140300539 A1 US20140300539 A1 US 20140300539A1 US 14271191 US14271191 US 14271191 US 201414271191 A US201414271191 A US 201414271191A US 20140300539 A1 US20140300539 A1 US 20140300539A1
Authority
US
Grant status
Application
Patent type
Prior art keywords
measure
gesture
hand
depth images
thumb
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US14271191
Inventor
Xiaofeng Tong
Dayong Ding
Wenlong Li
Yimin Zhang
Original Assignee
Xiaofeng Tong
Dayong Ding
Wenlong Li
Yimin Zhang
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06KRECOGNITION OF DATA; PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
    • G06K9/00Methods or arrangements for reading or recognising printed or written characters or for recognising patterns, e.g. fingerprints
    • G06K9/00335Recognising movements or behaviour, e.g. recognition of gestures, dynamic facial expressions; Lip-reading
    • G06K9/00355Recognition of hand or arm movements, e.g. recognition of deaf sign language
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017Gesture based interaction, e.g. based on a set of recognized hand gestures
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06KRECOGNITION OF DATA; PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
    • G06K9/00Methods or arrangements for reading or recognising printed or written characters or for recognising patterns, e.g. fingerprints
    • G06K9/00221Acquiring or recognising human faces, facial parts, facial sketches, facial expressions
    • G06K9/00228Detection; Localisation; Normalisation
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06KRECOGNITION OF DATA; PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
    • G06K9/00Methods or arrangements for reading or recognising printed or written characters or for recognising patterns, e.g. fingerprints
    • G06K9/00221Acquiring or recognising human faces, facial parts, facial sketches, facial expressions
    • G06K9/00228Detection; Localisation; Normalisation
    • G06K9/00261Detection; Localisation; Normalisation using comparisons between temporally consecutive images

Abstract

Methods, apparatuses, and articles associated with gesture recognition using depth images are disclosed herein. In various embodiments, an apparatus may include a face detection engine configured to determine whether a face is present in one or more gray images of respective image frames generated by a depth camera, and a hand tracking engine configured to track a hand in one or more depth images generated by the depth camera. The apparatus may further include a feature extraction and gesture inference engine configured to extract features based on results of the tracking by the hand tracking engine, and infer a hand gesture based at least in part on the extracted features. Other embodiments may also be disclosed and claimed.

Description

    TECHNICAL FIELD
  • Embodiments of the present disclosure relate to the field of data processing, in particular, to methods, apparatuses and articles associated with gesture recognition.
  • BACKGROUND
  • Unless otherwise indicated herein, the materials described in this section are not prior art to the claims in this application and are not admitted to be prior art by inclusion in this section.
  • With the fast growth of smart TV, smart phones, and tablets, there is an urgent desire to create innovative features/usage models to provide customers with better personalized service, social network and added value service (AVS) based on higher computing power, wider bandwidth and richer content. User interaction plays an important part in the user's experience. Among existing user-interface techniques, keyboard, mouse, trackball, and other point and click devices are typically employed. Gesture is a natural method of interaction and has been widely utilized in daily life in reality. Most existing gesture recognition systems utilize common web-cameras, which depend on light, illumination, and other environment conditions. As a result, current gesture recognition systems are typically unstable or inefficient.
  • BRIEF DESCRIPTION OF THE DRAWING
  • Embodiments of the present disclosure will be presented by way of example embodiments, but not limitations, illustrated in the accompanying drawings in which like references denote similar elements, and in which:
  • FIG. 1 illustrates an overview of a gesture recognition arrangement, in accordance with various embodiments of the present disclosure;
  • FIG. 2 illustrates the gesture based input service of FIG. 1 in further detail, in accordance with various embodiments of the present disclosure;
  • FIG. 3 illustrates a gesture recognition method, in accordance with various embodiments of the present disclosure;
  • FIG. 4 illustrates an example depth image, in accordance with various embodiments of the present disclosure;
  • FIG. 5 illustrates an example computer system suitable for use to practice the computing apparatus of FIG. 1, in accordance with embodiments of the present disclosure; and
  • FIG. 6 illustrates an article of manufacture having programming instructions configured to cause an apparatus to practice some or all aspects of the method of FIG. 3, in accordance with embodiments of the present disclosure.
  • DETAILED DESCRIPTION
  • Methods, apparatuses, and articles associated with gesture recognition using depth images are disclosed herein. In various embodiments, an apparatus may include a face detection engine configured to determine whether a face is present in one or more gray images of respective image frames generated by a depth camera, and a hand tracking engine configured to track a hand in one or more depth images generated by the depth camera. The apparatus may further include a feature extraction and gesture inference engine configured to extract features based on results of the tracking by the hand tracking engine, and infer a hand gesture based at least in part on the extract features.
  • In various embodiments, the face detection engine may be configured to analyze the gray images, using e.g., a Haar-Cascade model, to determine whether a face is present in the one or more gray images. Further, either the face detection engine or the hand tracking engine may be configured to determine a measure of a distance between the face and the camera, using the one or more depth images.
  • In various embodiments, the hand tracking engine may be further configured to select respective regions of the depth images that are size-wise smaller than the one or more depth images, to detect and track the hand, based at least in part on the determined distance between the face and the camera. The hand tracking engine may be further configured to determine location measures of the hand. The hand tracking engine may be configured to determine the location measures in terms of a pair of (x, y) coordinates for a center of the hand for respective ones of the one or more depth images, using mean-shift filtering that uses gradients of probabilistic density.
  • In various embodiments, the feature extraction and gesture inferring engine may be configured to extract one or more of an eccentricity measure, a compactness measure, an orientation measure, a rectangularity measure, a horizontal center measure, a vertical center measure, a minimum bounding box angle measure, or a minimum bounding box width-to-height ratio measure, a difference between left-and-right measure, or a difference between up-and-down measure. Further, the feature extraction and gesture inferring engine may be configured to infer one of an open hand gesture, a closed hand fist gesture, a thumb up gesture, a thumb down gesture, a thumb left gesture or a thumb right gesture, based on the extracted features. Additionally, the feature extraction and gesture inference engine may be further configured to notify an application of the inferred hand gesture.
  • In various embodiments, the apparatus may further include the camera. The apparatus may be a selected one of a desktop computer, a laptop computer, a tablet computer, a server, a smart phone, a personal digital assistant, a game console, or a set-top box.
  • In various embodiments, a first method may include tracking, by a computing apparatus, a hand in selected respective regions of one or more depth images generated by a depth camera, wherein the selected respective regions are size-wise smaller than the respective one or more depth images. The first method may further include inferring a hand gesture, by the computing device, based at least in part on a result of the tracking.
  • In various embodiments, a second method may include extracting, by a computing apparatus, one or more features from respective regions of depth images of image frames generated by a depth camera. The second method may further include inferring a gesture, by the computing apparatus, based at least in part on the one or more features extracted from the depth images.
  • Various aspects of the illustrative embodiments will now be further described using terms commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art. However, it will be apparent to those skilled in the art that alternate embodiments may be practiced with only some of the described aspects. For purposes of explanation, specific numbers, materials, and configurations are set forth in order to provide a thorough understanding of the illustrative embodiments. However, it will be apparent to one skilled in the art that alternate embodiments may be practiced without the specific details. In other instances, well-known features are omitted or simplified in order not to obscure the illustrative embodiments.
  • Further, various operations will be described as multiple discrete operations, in turn, in a manner that is most helpful in understanding the illustrative embodiments; however, the order of description should not be construed as to imply that these operations are necessarily order dependent. In particular, these operations need not be performed in the order of presentation. A feature described in a parenthetical format denotes the feature as an optional feature, e.g., “an instance of (partitions of) an application service,” denotes the application service may or may not be “partitioned.”
  • The phrase “in one embodiment” is used repeatedly. The phrase generally does not refer to the same embodiment; however, it may. The terms “comprising,” “having,” and “including” are synonymous, unless the context dictates otherwise. The phrase “A/B” means “A or B”. The phrase “A and/or B” means “(A), (B), or (A and B)”. The phrase “at least one of A, B and C” means “(A), (B), (C), (A and B), (A and C), (B and C) or (A, B and C)”. The phrase “(A) B” means “(B) or (A B)”, that is, A is optional.
  • FIG. 1 illustrates an overview of a gesture recognition arrangement, in accordance with various embodiments of the present disclosure. As illustrated, for the embodiments, computing apparatus 102, incorporated with gesture based input service 114, may be coupled with camera 106 with depth feature, to enable person 104 to provide inputs to application 112 operating on apparatus 102, using gestures, in particular, hand gestures.
  • Computing apparatus 102, except for gesture based input service (hereinafter GBIS) 114, may be any one of a number of computing devices known in the art. Examples of such computing devices include, but are not limited to, desktop computers, laptop computers, tablet computers, smart phones, personal digital assistants, game console, and/or set-top boxes. Likewise, application 112, except for the fact it may receive gesture inputs, by virtue of the services provided by GBIS 114, may be any one of a number of applications that may benefit from gesture inputs; e.g., games, media players, video applications, and so forth. For the embodiments, in addition to application 112 and GBIS 114, apparatus 102 may further include system services 116 for providing a wide range of system services, including but not limited to, communication services for application 112 and GBIS 114, and peripheral services for camera 106. The coupling between computing apparatus 102 and camera 106 may be wired or wireless. For the embodiments, service services 116 may be further configured to support access to a network by computing apparatus 102. The network may be wired or wireless, local or wide area, private or public, including e.g., the Internet.
  • Camera 106 may be any depth camera known in the art configured to capture and output image frames, including determining distances of target objects from the camera, outputting depth images as well as gray images for the various image frames. An example of a suitable camera may be the DImager, available from Panasonic of Japan. DImager is configured to calculate the distance to a target object by measuring the “Time-of-Flight” of near-emitted infrared rays. DImager's horizontal field-of-view is about 60 degrees and its vertical field-of-view is about 44 degrees. The resolution is about 160×120 pixels, The operating range is from 1.2-9.0 meters. It outputs range (depth) image and infrared gray image simultaneously. The process speed can reach 30 fps. Other depth cameras with the ability to generate depth and gray images may also be employed to practice the present disclosure instead.
  • The gestures supported may be application dependent. In various embodiments, the gestures supported may include an open hand gesture, a closed hand fist gesture, a thumb up gesture, a thumb down gesture, a thumb left gesture or a thumb right gesture, based on the extracted features.
  • FIG. 2 illustrates the gesture based input service of FIG. 1 in further detail, in accordance with various embodiments of the present disclosure. As shown, for the embodiments, GBIS 114 may include face detection engine 202, hand tracking engine 204, feature extraction and gesture inference engine 206, and hand data queue 212, operatively coupled to each other. Face detection engine 202 may be configured to determine whether a face is present in the image frames, using the gray images of the image frames generated by camera 106. Hand tracking engine 204 may be configured to track a hand using the depth images of the image frames, and output results of hand tracking data to hand data queue 212. Feature extraction and gesture inference engine 206 may be configured to process the hand tracking data, extract features from the hand tracking data, and infer the hand gestures based at least in part on the extracted features.
  • In various embodiments, face detection engine 202 may be configured to determine whether a face is present in the image frames, using the gray images and various face detection techniques, e.g., a Haar-Cascade model. Further, for the embodiments, face detection engine 202 may be configured to determine a measure of a distance between the face and the camera, using the one or more depth images. In alternate embodiments, hand tracking engine 204 may be configured to determine the measure of distance between the face and the camera, using the one or more depth images, instead.
  • In various embodiments, hand tracking engine 204 may be configured to perform hand detection and tracking, only after face detection engine 202 has determined a face is present in the gray images of the image frames. In various embodiments, hand tracking engine 204 may be configured to perform hand detection while hand data queue 212 is empty, or every N frame. N may be equal to, e.g., 10. On detection of a hand, hand tracking engine 204 may track the hand and output hand tracking data into hand data queue 212, until the hand stops appearing for a predetermined number of image frames. Hand tracking data may include location measures, such as, position, size, and depth of a hand, described more fully below. The conditional configuration of detection and tracking may reduce the consumption of computation resources.
  • Further, in various embodiments, hand tracking engine 204 may be configured to select respective regions of the depth images that are size-wise smaller than the one or more depth images, to track a hand, based at least in part on the determined distance between the face and the camera. More specifically, hand tracking engine 204 may be configured to select respective regions of the depth images that represent an area capturing a part of the person (the hand) that is closer to the camera, than the body of the person. Generally, it can be expected a user typically makes hand gestures with the user's hands in front of the user's body, i.e., a distance or depth closer to the camera than the determined distance of the face.
  • In various embodiments, hand tracking engine 204 may be further configured to determine the location measures of the hand, within the tracking regions. Hand tracking engine 204 may be configured to determine position location measures in terms of a pair of (x, y) coordinates for a center of the hand for respective ones of the one or more depth images, using mean-shift filtering that uses gradients of probabilistic density. More specifically, hand tracking engine 204 may be configured to determine the (x, y) coordinates of the center using the equations:
  • x = ( x , y ) ROI p ( x , y ) x ( x , y ) ROI p ( x , y ) ( 1 ) y = ( x , y ) ROI p ( x , y ) y ( x , y ) ROI p ( x , y ) ( 2 )
  • where p(x, y) are probabilistic density in common, here in particular, the depth information; ROI means region of interest; and x′ and y′ are the estimated (x, y) coordinates of the center of the hand. As described earlier, hand tracking engine 204 may be configured to further determine a size measure for the hand.
  • In various embodiments, hand tracking engine 204 may be further configured to perform a verification to confirm whether the hand region is valid using depth, shape and size information. On determination that any hand tracking result may be invalid or unreliable, the hand tracking result or results may be deleted from the hand data queue 212.
  • In various embodiments, feature extraction and gesture inferring engine 206 may be configured to extract from the hand tracking data, for the various depth images, one or more of an eccentricity measure, a compactness measure, an orientation measure, a rectangularity measure, a horizontal center measure, a vertical center measure, a minimum bounding box angle measure, a minimum bounding box width-to-height ratio measure, a difference between left-and-right measure, or a difference between up-and-down measure. Further, feature extraction and gesture inferring engine 206 may be configured to infer one of an open hand gesture, a closed hand fist gesture, a thumb up gesture, a thumb down gesture, a thumb left gesture or a thumb right gesture, 214, based on the extracted features. Additionally, feature extraction and gesture inference engine 206 may be further configured to notify an application, e.g., application 112, of the inferred hand gestures, via e.g., system services 116.
  • In various embodiments, the inference logic may be implemented using a classifier with multi-layer perception (MLP). The MLP classifier uses linear computation operation to improve computation efficiency. Further, the classifier may be trained using known samples. In various embodiments, to smooth the inference results, a time sliding-window containing several frames elapsed before current timestamp may be employed to vote on the inference results, and the inference result with the most votes may be outputted.
  • FIG. 3 illustrates a gesture recognition method, in accordance with various embodiments of the present disclosure. As shown, for the embodiments, method 300 may begin at block 302, where image frames with depth information may be received. From block 302, the method may proceed to block 304, where a determination is made on whether a person, in particular, a face is present in the image frames, using e.g., face detection techniques, and the gray images.
  • From block 304, the method may proceed to block 306, where a hand of the person may be detected and tracked, using the depth images. From block 306, the method may proceed to block 308, where hand gestures may be inferred, based at least in part on the hand tracking data. From block 308, the method may proceed to block 310, where messages may be sent to a system service for routing to an application with interest, informing the application of the inferred hand gestures.
  • From block 310, the method may proceed to block 312, where the system service may route the messages to the application with interest, informing the application of the inferred hand gestures. From block 312, the method may proceed to block 314, where the application may respond to the receipt of the inferred hand gesture as input.
  • FIG. 4 illustrates an example depth image, in accordance with various embodiments of the present disclosure. The higher intensities (darker) of the grayscale represent further the distances the features are to the camera.
  • As shown, in various embodiments, for computational efficiency, application of face detection techniques to determine the presence of a person may be focused on the facial region 404 of depth image 402. Similarly, as described earlier, for computational efficiency, hand tracking may be focused on a size-wise smaller region where the object within the region is closer to the camera. Regions 404 and 406 are purposely kept smaller than the face and the hand respectively, for illustrative purpose only, because of the grayscale of the illustration. Regions 404 and 406, in practice, are typically larger than the illustrated regions, encompassing the face and the hand respectively, although the practice disclosure may be practiced with smaller regions as illustrated.
  • FIG. 5 illustrates an example computer system suitable for use to practice the computing apparatus of FIG. 1, in accordance with various embodiments of the present disclosure. As shown, computing system 500 may include a number of processors or processor cores 502, and system memory 504. For the purpose of this application, including the claims, the terms “processor” and “processor cores” may be considered synonymous, unless the context clearly requires otherwise. Additionally, computing system 500 may include mass storage devices 506 (such as diskette, hard drive, compact disc read only memory (CDROM) and so forth), input/output devices 508 (such as display, keyboard, cursor control and so forth), and communication interfaces 510 (such as network interface cards, modems and so forth). The elements may be coupled to each other via system bus 512, which represents one or more buses. In the case of multiple buses, they may be bridged by one or more bus bridges (not shown).
  • Each of these elements performs its conventional functions known in the art. In particular, system memory 504 and mass storage 506 may be employed to store a working copy and a permanent copy of the programming instructions implementing the GBIS 114 and related functions, herein collectively denoted as computing logic 522. The various components, modules, or routines of the functions may be implemented by assembler instructions supported by processor(s) 502 or high-level languages, such as, for example, C, that can be compiled into such instructions.
  • The permanent copy of the programming instructions may be placed into permanent storage 506 in the factory, or in the field, through, for example, a distribution medium (not shown), such as a compact disc (CD), or through communication interface 510 (from a distribution server (not shown)). That is, one or more distribution media having an implementation of the agent program may be employed to distribute the agent, and program various computing devices.
  • The constitution of these elements 502-512 are known, and accordingly will not be further described.
  • FIG. 6 illustrates an article of manufacture with programming instructions configured to enable an apparatus to practice the face recognition based control method and apparatus, in accordance with embodiments of the present disclosure. As shown, article of manufacture 600 may include computer-readable non-transitory storage medium 602. Storage medium 602 may include programming instructions 604 configured to implement the GBIS 114 of FIG. 1.
  • Storage medium 602 represents a broad range of persistent storage medium known in the art, including but not limited to flash memory, optical disks or magnetic disks. Programming instructions 604, in particular, may enable an apparatus, in response to their execution by the apparatus, to perform operations including:
      • tracking, by a computing apparatus, a hand in selected respective regions of one or more depth images generated by a depth camera, wherein the selected respective regions are size-wise smaller than the respective one or more depth images; and
      • inferring a hand gesture, by computing device, based at least in part on a result of the tracking.
  • In alternate embodiments, programming instructions 604, in particular, may enable an apparatus, in response to their execution by the apparatus, to perform operations including:
      • extracting, by a computing apparatus, one or more features from respective regions of depth images of image frames generated by a depth camera; and
      • inferring a gesture, by the computing apparatus, based at least in part on the one or more features extracted from the depth images.
  • Although specific embodiments have been illustrated and described herein, it will be appreciated by those of ordinary skill in the art that a wide variety of alternate and/or equivalent implementations may be substituted for the specific embodiments shown and described, without departing from the scope of the embodiments of the present invention. This application is intended to cover any adaptations or variations of the embodiments discussed herein. Therefore, it is manifestly intended that the embodiments of the present invention be limited only by the claims and the equivalents thereof.

Claims (38)

  1. 1. (canceled)
  2. 2. (canceled)
  3. 3. (canceled)
  4. 4. (canceled)
  5. 5. (canceled)
  6. 6. (canceled)
  7. 7. (canceled)
  8. 8. (canceled)
  9. 9. (canceled)
  10. 10. (canceled)
  11. 11. (canceled)
  12. 12. (canceled)
  13. 13. A method comprising:
    tracking, by a computing apparatus, a hand in selected respective regions of one or more depth images generated by a depth camera, wherein the selected respective regions are size-wise smaller than the respective one or more depth images; and
    inferring a hand gesture, by the computing device, based at least in part on a result of the tracking;
    wherein tracking comprises determining location measures of the hand for the depth images.
  14. 14. The method of claim 13, wherein determining location measures of the hand for the depth images comprises determining a pair of (x, y) coordinates for a center of the hand, using mean-shift filtering that uses gradients of probabilistic density.
  15. 15. The method of claim 13, wherein inferring comprises extracting one or more features for the selected respective regions, based at least in part on a result of the tracking, and inferring a hand gesture based at least in part on the extracted one or more features.
  16. 16. The method of claim 15, wherein extracting one or more features comprises extracting one or more of an eccentricity measure, a compactness measure, an orientation measure, a rectangularity measure, a horizontal center measure, a vertical center measure, a minimum bounding box angle measure, a minimum bounding box width-to-height ratio measure, a difference between left-and-right measure, or a difference between up-and-down measure.
  17. 17. The method of claim 13, wherein inferring a gesture comprises inferring one of an open gesture, a fist gesture, a thumb up gesture, a thumb down gesture, a thumb left gesture or a thumb right gesture.
  18. 18. (canceled)
  19. 19. A method comprising:
    extracting, by a computing apparatus, one or more features from respective regions of depth images of image frames generated by a depth camera; and
    inferring a gesture, by the computing apparatus, based at least in part on the one or more features extracted from the depth images;
    wherein extracting one or more features comprises extracting one or more of an eccentricity measure, a compactness measure, an orientation measure, a rectangularity measure, a horizontal center measure, a vertical center measure, a minimum bounding box angle measure, a minimum bounding box width-to-height ratio measure, a difference between left-and-right measure, or a difference between up-and-down measure.
  20. 20. The method of claim 19, wherein extracting one or more features from respective regions of depth images comprises extracting one or more features from respective regions of depth images denoted as containing a hand.
  21. 21. The method of claim 19 wherein inferring a gesture comprises inferring one of an open gesture, a fist gesture, a thumb up gesture, a thumb down gesture, a thumb left gesture or a thumb right gesture.
  22. 22. (canceled)
  23. 23. A computer-readable non-transitory storage medium, comprising
    a plurality of programming instructions stored in the storage medium, and configured to cause an apparatus, in response to execution of the programming instructions by the apparatus, to perform operations including:
    tracking a hand in selected respective regions of one or more depth images generated by a depth camera, wherein the selected respective regions are size-wise smaller than the respective one or more depth images; and
    inferring a hand gesture, based at least in part on a result of the tracking;
    wherein tracking comprises determining location measures of the hand for the depth images.
  24. 24. The storage medium of claim 23, wherein determining location measures of the hand for the depth images comprises determining a pair of (x, y) coordinates for a center of the hand, using mean-shift filtering that uses gradients of probabilistic density.
  25. 25. The storage medium of claim 23, wherein inferring comprises extracting one or more features for the selected respective regions, based at least in part on a result of the tracking, and inferring a hand gesture based at least in part on the extracted one or more features.
  26. 26. The storage medium of claim 25, wherein extracting one or more features comprises extracting one or more of an eccentricity measure, a compactness measure, an orientation measure, a rectangularity measure, a horizontal center measure, a vertical center measure, a minimum bounding box angle measure, a minimum bounding box width-to-height ratio measure, a difference between left-and-right measure, or a difference between up-and-down measure.
  27. 27. The storage medium of claim 23, wherein inferring a gesture comprises inferring one of an open gesture, a fist gesture, a thumb up gesture, a thumb down gesture, a thumb left gesture or a thumb right gesture.
  28. 28. The storage medium of claim 23, wherein the operations further comprise determining whether a face is present in the one or more depth images' corresponding one or more gray images of respective image frames generated by a depth camera.
  29. 29. The storage medium of claim 28, wherein determining whether a face is present comprises analyzing the one or more gray images using a Haar-Cascade model.
  30. 30. The storage medium of claim 28, wherein the operations further comprise determining a measure of a distance between the face and the camera, using the one or more depth images.
  31. 31. An apparatus, comprising:
    a tracking engine to track a hand in selected respective regions of one or more depth images generated by a depth camera, wherein the selected respective regions are size-wise smaller than the respective one or more depth images; and
    an inference engine coupled with the tracking engine to infer a hand gesture based at least in part on a result of the tracking;
    wherein to track a hand comprises to determine location measures of the hand for the depth images.
  32. 32. The apparatus of claim 31, wherein to determine location measures of the hand for the depth images comprises to determine a pair of (x, y) coordinates for a center of the hand, using mean-shift filtering that uses gradients of probabilistic density.
  33. 33. The apparatus of claim 31, wherein to infer comprises to extract one or more features for the selected respective regions, based at least in part on a result of the tracking, and to infer a hand gesture based at least in part on the extracted one or more features.
  34. 34. The apparatus of claim 33, wherein to extract one or more features comprises to extract one or more of an eccentricity measure, a compactness measure, an orientation measure, a rectangularity measure, a horizontal center measure, a vertical center measure, a minimum bounding box angle measure, a minimum bounding box width-to-height ratio measure, a difference between left-and-right measure, or a difference between up-and-down measure.
  35. 35. The apparatus of claim 31, wherein to infer a gesture comprises to infer one of an open gesture, a fist gesture, a thumb up gesture, a thumb down gesture, a thumb left gesture or a thumb right gesture.
  36. 36. An apparatus comprising:
    an extraction engine to extract one or more features from respective regions of depth images of image frames generated by a depth camera; and
    an inference engine coupled with the extraction engine to infer a gesture, based at least in part on the one or more features extracted from the depth images;
    wherein to extract one or more features comprises to extract one or more of an eccentricity measure, a compactness measure, an orientation measure, a rectangularity measure, a horizontal center measure, a vertical center measure, a minimum bounding box angle measure, a minimum bounding box width-to-height ratio measure, a difference between left-and-right measure, or a difference between up-and-down measure.
  37. 37. The apparatus of claim 36, wherein to extract one or more features from respective regions of depth images comprises to extract one or more features from respective regions of depth images denoted as containing a hand.
  38. 38. The apparatus of claim 36 wherein to infer a gesture comprises to infer one of an open gesture, a fist gesture, a thumb up gesture, a thumb down gesture, a thumb left gesture or a thumb right gesture.
US14271191 2011-04-11 2014-05-06 Gesture recognition using depth images Abandoned US20140300539A1 (en)

Priority Applications (3)

Application Number Priority Date Filing Date Title
PCT/CN2011/072581 WO2012139268A1 (en) 2011-04-11 2011-04-11 Gesture recognition using depth images
US201213387221 true 2012-01-26 2012-01-26
US14271191 US20140300539A1 (en) 2011-04-11 2014-05-06 Gesture recognition using depth images

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US14271191 US20140300539A1 (en) 2011-04-11 2014-05-06 Gesture recognition using depth images

Related Parent Applications (2)

Application Number Title Priority Date Filing Date
PCT/CN2011/072581 Continuation WO2012139268A1 (en) 2011-04-11 2011-04-11 Gesture recognition using depth images
US201213387221 Continuation 2012-01-26 2012-01-26

Publications (1)

Publication Number Publication Date
US20140300539A1 true true US20140300539A1 (en) 2014-10-09

Family

ID=47008773

Family Applications (2)

Application Number Title Priority Date Filing Date
US13387221 Active US8718327B2 (en) 2011-04-11 2011-04-11 Gesture recognition using depth images
US14271191 Abandoned US20140300539A1 (en) 2011-04-11 2014-05-06 Gesture recognition using depth images

Family Applications Before (1)

Application Number Title Priority Date Filing Date
US13387221 Active US8718327B2 (en) 2011-04-11 2011-04-11 Gesture recognition using depth images

Country Status (4)

Country Link
US (2) US8718327B2 (en)
EP (1) EP2697743A4 (en)
CN (1) CN103477352A (en)
WO (1) WO2012139268A1 (en)

Families Citing this family (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2012139241A1 (en) * 2011-04-11 2012-10-18 Intel Corporation Hand gesture recognition system
US8718327B2 (en) * 2011-04-11 2014-05-06 Intel Corporation Gesture recognition using depth images
US20130077820A1 (en) * 2011-09-26 2013-03-28 Microsoft Corporation Machine learning gesture detection
CN103376895B (en) * 2012-04-24 2016-08-03 纬创资通股份有限公司 Gesture control method and a gesture control device
US20140068740A1 (en) * 2012-09-05 2014-03-06 Element, Inc. System and method for biometric authentication in connection with camera equipped devices
US20140111617A1 (en) * 2012-10-23 2014-04-24 Lsi Corporation Optical source driver circuit for depth imager
CN103914133B (en) 2012-12-28 2017-04-26 纬创资通股份有限公司 Coordinate transformation method for interactive systems and computer systems
CN103914134B (en) 2012-12-28 2017-11-17 纬创资通股份有限公司 Gesture recognition and gesture recognition module
CN104182742B (en) * 2013-05-20 2018-03-13 比亚迪股份有限公司 Head pose recognition method and system for
CN103366188B (en) * 2013-07-08 2017-07-07 中科创达软件股份有限公司 Fist hand tracking method based on the detection information as auxiliary
WO2015175670A1 (en) 2014-05-13 2015-11-19 Element, Inc. System and method for electronic key provisioning and access management in connection with mobile devices
US9965728B2 (en) 2014-06-03 2018-05-08 Element, Inc. Attendance authentication and management in connection with mobile devices

Citations (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6144366A (en) * 1996-10-18 2000-11-07 Kabushiki Kaisha Toshiba Method and apparatus for generating information input using reflected light image of target object
US20030235327A1 (en) * 2002-06-20 2003-12-25 Narayan Srinivasa Method and apparatus for the surveillance of objects in images
US6788809B1 (en) * 2000-06-30 2004-09-07 Intel Corporation System and method for gesture recognition in three dimensions using stereo imaging and color vision
US20060083423A1 (en) * 2004-10-14 2006-04-20 International Business Machines Corporation Method and apparatus for object normalization using object classification
US7468742B2 (en) * 2004-01-14 2008-12-23 Korea Institute Of Science And Technology Interactive presentation system
US7668376B2 (en) * 2004-06-30 2010-02-23 National Instruments Corporation Shape feature extraction and classification
US20100166258A1 (en) * 2008-12-30 2010-07-01 Xiujuan Chai Method, apparatus and computer program product for providing hand segmentation for gesture analysis
US20110211754A1 (en) * 2010-03-01 2011-09-01 Primesense Ltd. Tracking body parts by combined color image and depth processing
US20120056995A1 (en) * 2010-08-31 2012-03-08 Texas Instruments Incorporated Method and Apparatus for Stereo-Based Proximity Warning System for Vehicle Safety
US20120268364A1 (en) * 2008-04-24 2012-10-25 Minnen David Fast fingertip detection for initializing a vision-based hand tracker
US20130273968A1 (en) * 2008-08-19 2013-10-17 Digimarc Corporation Methods and systems for content processing
US8693724B2 (en) * 2009-05-29 2014-04-08 Microsoft Corporation Method and system implementing user-centric gesture control
US8718327B2 (en) * 2011-04-11 2014-05-06 Intel Corporation Gesture recognition using depth images

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6363160B1 (en) * 1999-01-22 2002-03-26 Intel Corporation Interface using pattern recognition and tracking
JP4355341B2 (en) * 2003-05-29 2009-10-28 本田技研工業株式会社 Visual tracking using the depth data
US7308112B2 (en) * 2004-05-14 2007-12-11 Honda Motor Co., Ltd. Sign based human-machine interaction
CN101401124A (en) * 2006-03-20 2009-04-01 奥林巴斯株式会社 Video image information processing device, judging method, and computer program
JP4434236B2 (en) * 2007-06-11 2010-03-17 ソニー株式会社 Image processing apparatus, an image display device, an imaging device, these processing method and program
CN101388115B (en) * 2008-10-24 2011-07-27 北京航空航天大学 Depth image autoegistration method combined with texture information
WO2010144050A1 (en) * 2009-06-08 2010-12-16 Agency For Science, Technology And Research Method and system for gesture based manipulation of a 3-dimensional image of object
JP2011217209A (en) * 2010-03-31 2011-10-27 Sony Corp Electronic apparatus, content recommendation method, and program

Patent Citations (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6144366A (en) * 1996-10-18 2000-11-07 Kabushiki Kaisha Toshiba Method and apparatus for generating information input using reflected light image of target object
US6788809B1 (en) * 2000-06-30 2004-09-07 Intel Corporation System and method for gesture recognition in three dimensions using stereo imaging and color vision
US7409092B2 (en) * 2002-06-20 2008-08-05 Hrl Laboratories, Llc Method and apparatus for the surveillance of objects in images
US20030235327A1 (en) * 2002-06-20 2003-12-25 Narayan Srinivasa Method and apparatus for the surveillance of objects in images
US7468742B2 (en) * 2004-01-14 2008-12-23 Korea Institute Of Science And Technology Interactive presentation system
US7668376B2 (en) * 2004-06-30 2010-02-23 National Instruments Corporation Shape feature extraction and classification
US20060083423A1 (en) * 2004-10-14 2006-04-20 International Business Machines Corporation Method and apparatus for object normalization using object classification
US20120268364A1 (en) * 2008-04-24 2012-10-25 Minnen David Fast fingertip detection for initializing a vision-based hand tracker
US20130273968A1 (en) * 2008-08-19 2013-10-17 Digimarc Corporation Methods and systems for content processing
US20100166258A1 (en) * 2008-12-30 2010-07-01 Xiujuan Chai Method, apparatus and computer program product for providing hand segmentation for gesture analysis
US8693724B2 (en) * 2009-05-29 2014-04-08 Microsoft Corporation Method and system implementing user-centric gesture control
US20110211754A1 (en) * 2010-03-01 2011-09-01 Primesense Ltd. Tracking body parts by combined color image and depth processing
US20120056995A1 (en) * 2010-08-31 2012-03-08 Texas Instruments Incorporated Method and Apparatus for Stereo-Based Proximity Warning System for Vehicle Safety
US8718327B2 (en) * 2011-04-11 2014-05-06 Intel Corporation Gesture recognition using depth images

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
P. Serafinavicius, "Estimating Characteristic Points of Human Body for Automatic Hand Pointing Gesture Recognition", Electronics And Electrical Engineering Medicine Technology ISSN 1392-1215 No. 8(80) T 115 (2007). *
Sebastien Carbini, et al., "Pointing Gesture Visual Recognition for Large Display", Pointing'04 ICPR Workshop, Cambridge, United Kingdom, 22 August 2004. *

Also Published As

Publication number Publication date Type
US8718327B2 (en) 2014-05-06 grant
CN103477352A (en) 2013-12-25 application
EP2697743A1 (en) 2014-02-19 application
US20140037134A1 (en) 2014-02-06 application
EP2697743A4 (en) 2014-09-24 application
WO2012139268A1 (en) 2012-10-18 application

Similar Documents

Publication Publication Date Title
US9129157B2 (en) Method for image-based status determination
US9024972B1 (en) Augmented reality computing with inertial sensors
US20110242277A1 (en) Systems and methods for embedding a foreground video into a background feed based on a control input
US20110286631A1 (en) Real time tracking/detection of multiple targets
US20150172626A1 (en) Localisation and mapping
US20130114849A1 (en) Server-assisted object recognition and tracking for mobile devices
US20100166258A1 (en) Method, apparatus and computer program product for providing hand segmentation for gesture analysis
US20130121540A1 (en) Facial Recognition Using Social Networking Information
US20140125700A1 (en) Using a plurality of sensors for mapping and localization
Loy et al. An adaptive fusion architecture for target tracking
US20140139633A1 (en) Method and System for Counting People Using Depth Sensor
US20120272194A1 (en) Methods and apparatuses for facilitating gesture recognition
US20150077323A1 (en) Dynamic object tracking for user interfaces
US8306267B1 (en) Object tracking
US20150172539A1 (en) Distributing processing for imaging processing
US20140324888A1 (en) Method and Apparatus for Identifying a Gesture Based Upon Fusion of Multiple Sensor Signals
US20130238234A1 (en) Methods for determining a user's location using poi visibility inference
US9269011B1 (en) Graphical refinement for points of interest
US20130054377A1 (en) Person tracking and interactive advertising
Ji et al. Special issue: eye detection and tracking
US20140270528A1 (en) Local image enhancement for text recognition
US20110150275A1 (en) Model-based play field registration
US20110242326A1 (en) System and Method for Utilizing Motion Fields to Predict Evolution in Dynamic Scenes
US8472669B2 (en) Object localization using tracked object trajectories
US20140161354A1 (en) Method and apparatus for semantic extraction and video remix creation