WO2014106849A1 - Procédé d'identification d'un trajet de mouvement - Google Patents

Procédé d'identification d'un trajet de mouvement Download PDF

Info

Publication number
WO2014106849A1
WO2014106849A1 PCT/IL2014/050011 IL2014050011W WO2014106849A1 WO 2014106849 A1 WO2014106849 A1 WO 2014106849A1 IL 2014050011 W IL2014050011 W IL 2014050011W WO 2014106849 A1 WO2014106849 A1 WO 2014106849A1
Authority
WO
WIPO (PCT)
Prior art keywords
hand
motion path
user
processor
features
Prior art date
Application number
PCT/IL2014/050011
Other languages
English (en)
Inventor
Yonatan HYATT
Eran Eilat
Amir Kaplan
Original Assignee
Pointgrab Ltd.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Pointgrab Ltd. filed Critical Pointgrab Ltd.
Publication of WO2014106849A1 publication Critical patent/WO2014106849A1/fr

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47—End-user applications
    • H04N21/482—End-user interface for program selection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/0304—Detection arrangements using opto-electronic means
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
    • H04N21/4223—Cameras
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/44008—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/442—Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk
    • H04N21/44213—Monitoring of end-user related data
    • H04N21/44218—Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program

Definitions

  • the present invention relates to the field of gesture based control of electronic devices. Specifically, the invention relates to classifying types of gestures and to computer vision based identification of characters from user hand gestures.
  • Keyboards and pointing devices are input devices commonly used for interaction with computers and other electronic devices that are associated with electronic displays. Keyboards are usually used for inputting text and characters. Pointing devices and machine controlling mechanisms (e.g., an electronic mouse, a trackball, a pointing stick and touchpad, a touch screen and others) are usually used to control a location and/or movement of a cursor displayed on the associated electronic display. Pointing devices may also convey commands, e.g. location specific commands, by activating switches on the pointing device.
  • commands e.g. location specific commands
  • human gesturing such as hand gesturing
  • hand gesturing has been suggested as a user interface input tool, which can be used even at a distance from the controlled device.
  • a hand posture or gesture is detected by a camera and is translated into a specific command.
  • One method for touchless text entry uses an input zone as a gesture delimitation device for motion tracking sensors and gestures are interpreted for character recognition only when the user's hands are inside the input zone. This requires the user to position his hands according to requirements of the system rather than to use his hands freely and at his own convenience, making the use of this system awkward and cumbersome for the user.
  • Embodiments of the invention provide character input based control of a device, in which a user may easily move between gesture-controlling a device and controlling the device by inputting characters.
  • Methods of the invention ensure that unintended hand movements are not interpreted as character writing (or as other intended gestures).
  • a method which includes obtaining a sequence of images which include a user's hand; identifying a first posture of the user's hand; tracking the user's hand through the sequence of images; deriving a motion path of the user's hand; determining a character based on the derived motion path; and controlling a device based on the determined character and based on the identification of the first posture.
  • Other embodiments provide a method for differentiating between types of hand motion paths, to facilitate recognition of hand gestures, e.g., to facilitate recognition of characters written in the air and to differentiate between character writing and other hand gestures.
  • the method includes using databases of hand motion path features which correspond to different types of hand motion paths, to train a classifier so that the classifier may be then used to obtain an indication regarding the type of hand motion path of any inputted hand gestures.
  • hand motion paths may be classified as specific gestures or as random hand movements, enabling easier identification of specific hand gestures to control a device.
  • FIG. 1 schematically illustrates a method for computer vision based control of a device based on a determined character according to embodiments of the invention
  • FIG. 2 schematically illustrates a method for computer vision based control of a device based on a determined character and based on a first and second posture of the hand according to embodiments of the invention
  • FIG. 3 schematically illustrates a method for character recognition, according to embodiments of the invention
  • FIG. 4 schematically illustrates a method for computer vision based control of a device using motion path databases, according to an embodiment of the invention
  • FIG. 5 schematically illustrates a method which includes displaying a graphical representation of a motion path of the user's hand, according to embodiments of the invention
  • FIG. 6 schematically illustrates a TV set operable according to embodiments of the invention
  • FIG. 7 schematically illustrates a method for controlling a device using hand motion paths types, according to embodiments of the invention.
  • FIG 8 schematically illustrates a system according to embodiments of the invention.
  • a system for user-device interaction operable typically includes a device having a display and an image sensor which is in communication with the device and with a processor.
  • Methods according to embodiments of the invention may be implemented in such a user-device interaction system which includes a device to be operated by a user and an image sensor which is in communication with a processor.
  • the image sensor obtains image data (typically of the user) and sends it to the processor to perform image analysis and to generate user commands to the device based on the image analysis, thereby controlling the device based on computer vision.
  • FIG. 8 An exemplary system, according to one embodiment of the invention, is schematically described in Fig. 8 however, other systems may carry out embodiments of the present invention.
  • the system 800 may include an image sensor 803, typically associated with a processor 802, memory 82, and a device 801.
  • the image sensor 803 sends the processor 802 image data of field of view (FOV) 804 to be analyzed by processor 802.
  • FOV field of view
  • image signal processing algorithms and/or image acquisition algorithms may be run in processor 802.
  • a user command is generated by processor 802 or by another processor, based on the image analysis, and is sent to the device 801.
  • the image processing is performed by a first processor which then sends a signal to a second processor in which a user command is generated based on the signal from the first processor.
  • Processor 802 may include, for example, one or more processors and may be a central processing unit (CPU), a digital signal processor (DSP), a microprocessor, a controller, a chip, a microchip, an integrated circuit (IC), or any other suitable multi-purpose or specific processor or controller.
  • CPU central processing unit
  • DSP digital signal processor
  • microprocessor a controller
  • IC integrated circuit
  • processor 802 (or another processor) is in communication with the device 801 and may detect a user's hand and track the hand in a sequence of images; extract motion path features from the tracking of the hand; determine a motion path of the hand based on the motion path features using machine learning techniques trained on a database of motion path features; and control the device based on the determined motion path of the hand.
  • device 801 may include a display 81 and control of the device may include changes on the display 81.
  • a motion path of a hand may be determined to correspond to a character and detection of the character may cause that character to appear on the display 81 or for a screen to be changed (e.g., a TV screen may show another channel based on detection of a character).
  • Memory unit(s) 82 may include, for example, a random access memory (RAM), a dynamic RAM (DRAM), a flash memory, a volatile memory, a non-volatile memory, a cache memory, a buffer, a short term memory unit, a long term memory unit, or other suitable memory units or storage units.
  • RAM random access memory
  • DRAM dynamic RAM
  • flash memory a volatile memory
  • non-volatile memory a non-volatile memory
  • cache memory a buffer
  • a short term memory unit a long term memory unit
  • other suitable memory units or storage units or storage units.
  • the device 801 may be any electronic device that can accept user commands, e.g., TV, DVD player, PC, mobile phone, camera, set top box (STB) etc. According to one embodiment, device 801 is an electronic device available with an integrated standard 2D camera. The device 801 may include a display 81 or a display may be separate from but in communication with the device 801.
  • the processor 802 may be integral to the image sensor 803 or may be a separate unit. Alternatively, the processor 802 may be integrated within the device 801. According to other embodiments a first processor may be integrated within the image sensor and a second processor may be integrated within the device.
  • the communication between the image sensor 803 and processor 802 and/or between the processor 802 and the device 801 may be through a wired or wireless link, such as through infrared (IR) communication, radio transmission, Bluetooth technology and other suitable communication routes.
  • IR infrared
  • the image sensor 803 may include a CCD or CMOS or other appropriate chip.
  • the image sensor 803 may be included in a camera such as a forward facing camera, typically, a standard 2D camera such as a webcam or other standard video capture device, typically installed on PCs or other electronic devices.
  • a 3D camera or stereoscopic camera may also be used according to embodiments of the invention.
  • the image sensor 803 may obtain frames at varying frame rates. According to embodiments of the invention the image sensor 803 obtains image data of a user's hand 805 when the hand enters the field of view 804. Although the description refers mainly to identifying a user's hand, other user body parts and/or other objects may be identified according to embodiments of the invention and may be used to touchlessly control devices.
  • image data may be stored in processor 802, for example in a cache memory.
  • Processor 802 can apply image analysis algorithms, such as motion detection and shape recognition or detection algorithms to identify and further track the user's hand.
  • Processor 802 may perform methods according to embodiments discussed herein by for example executing software or instructions stored in memory 82.
  • shape detection algorithms may be stored in memory 82.
  • the shape of a hand may be searched, for example, by applying a shape recognition algorithm (for example, an algorithm which calculates Haar-like features in a Viola- Jones object detection framework).
  • a shape recognition algorithm for example, an algorithm which calculates Haar-like features in a Viola- Jones object detection framework.
  • the hand shape may be tracked through a series of images using known methods for tracking selected features, such as optical flow techniques.
  • a hand (or other object) shape may be searched in every image or at a different frequency (e.g., once every 5 images, once every 20 images or other appropriate frequencies) to update the location of the hand to avoid drifting of the tracking of the hand.
  • a user's hand may be tracked while detecting a predetermined shape throughout tracking.
  • a processor such as processor 802 which may carry out all or part of a method as discussed herein, may be configured to carry out the method by, for example, being associated with or connected to a memory such as memory 82 storing code or software which, when executed by the processor, carry out the method.
  • the system 800 may include an electronic display 81.
  • mouse emulation and/or control of a cursor on a display are based on computer visual identification and tracking of a user's hand, for example, as detailed above.
  • Embodiments of the invention may include an article such as a computer or processor readable non-transitory storage medium, such as for example a memory, a disk drive, or a USB flash memory encoding, including or storing instructions, e.g., computer-executable instructions, which when executed by a processor or controller, cause the processor or controller to carry out methods disclosed herein.
  • a computer or processor readable non-transitory storage medium such as for example a memory, a disk drive, or a USB flash memory encoding
  • instructions e.g., computer-executable instructions, which when executed by a processor or controller, cause the processor or controller to carry out methods disclosed herein.
  • a user may move his hand in a specific motion path (a motion path of a hand typically refers to the trajectory of the user's hand on a 2 dimensional plane within the FOV of a camera or image sensor associated with the system) to create a character (e.g., a letter, number or other symbol) that is recognized by the system.
  • a specific motion path typically refers to the trajectory of the user's hand on a 2 dimensional plane within the FOV of a camera or image sensor associated with the system
  • a character e.g., a letter, number or other symbol
  • Fig. 1 schematically illustrates a method for computer vision based control of a device, according to one embodiment of the invention.
  • the method includes the steps of obtaining a sequence of images which include a user's hand (102); identifying a first posture of the user's hand (104); tracking the user's hand through the sequence of images (106); deriving a motion path of the user's hand (108); determining a character based on the derived motion path (110); and controlling the device based on the determined character and based on the identification of the first posture (112).
  • a user operating a touchlessly controlled device having a camera such as described above may arrange his hand in a specific shape (posture) to indicate to the system that hand motion paths should be interpreted as characters.
  • the user may then draw a character in the air (in the FOV of the camera), for example, to indicate a number of a channel on TV or to input text for a text message (e.g., on a word processing application on a PC).
  • a user operating a touchlessly controlled device having a camera such as described above, may arrange his hand in a specific shape (posture) to indicate to the system that hand motion paths should be interpreted as characters.
  • the user may then draw a character in the air (in the FOV of the camera), for example, to indicate a number of a channel on TV or to input text for a text message (e.g., on a word processing application on a PC).
  • tracking of the user's hand will be utilized for determining the character being "drawn" or "written” by the user.
  • Specific postures may include, for example, a posture in which all fingers of the hand are brought together such that the finger tips are touching or almost touching, such as when the hand is closed in a fist or as if holding a bulb or a hand having all fingers folded except one finger that is extended or a hand in which a finger and the thumb are held together in a pinching-like posture or other postures.
  • the system utilizes tracking for determining a character only while the first posture is detected (namely, the user keeps his hand in the specific posture throughout the movement of the hand while intending to write a character).
  • tracking is utilized for determining a character regardless of the posture of the hand (namely, characters can be recognized by the system even if the user's hand is not kept in a specific posture). Identification of the first posture again or identification of a second specific posture may then indicate that the user is not interested in writing characters.
  • the device may be controlled based on the determined character. For example, a channel may be changed to the number drawn by the user, a sound track may be chosen, a web search may be performed, a text message may be written, etc.
  • one embodiment of the invention provides a method for changing channels on TV or on other devices.
  • the method includes obtaining a sequence of images, the images comprising a user's hand; tracking the user's hand through the sequence of images; deriving a motion path of the user's hand; determining a character based on the derived motion path; and controlling the device to play or run a channel based on the determined character.
  • the method includes detecting a shape of the user's hand and determining a character based on the shape of the hand and on the derived motion path.
  • the method includes the steps of obtaining a sequence of images which include a user's hand (202); identifying a posture of the user's hand (204); tracking the user's hand through the sequence of images (206); deriving a motion path of the user's hand (208); determining whether the identified posture is the first posture (210) and if the posture is identified as the first posture a character is determined based on the derived motion path (212) and the device may be controlled based on the determined character (214). If the posture identified (at step 210) is a second posture (e.g., other than the first posture), then based on the identification of the second posture, the step of determining a character based on the derived motion path (step 208) is discontinued. Instead, the system will proceed to identify a posture in a next image of the sequence of images.
  • a second posture e.g., other than the first posture
  • a user may easily move between gesture-controlling a device and controlling the device by inputting characters.
  • a user may switch between applications by gesturing (e.g., using a "swipe" gesture).
  • gesturing e.g., using a "swipe" gesture.
  • an audio application for example
  • the user may bring his hand into a predetermined posture (e.g. bring the fingers of the hand together or extend a pointing finger while all other fingers are folded) and may then write a number in the air to indicate which sound track should be played.
  • a predetermined posture e.g. bring the fingers of the hand together or extend a pointing finger while all other fingers are folded
  • the user may then again open his hand (or change the posture of his hand to a different posture) and future hand movements will not be interpreted by the system as characters.
  • characters are identified based on the motion path of the user's hand, whereas the user's hand may be in several possible shapes (postures) or even regardless of the shape of the hand.
  • deriving a motion path of the user's hand may be done by accumulating location parameters of the user's hand during tracking of the hand; and deriving a motion path from the accumulation of location parameters.
  • Location parameters may include coordinates and/or vectors related to the user's hand in each image.
  • a motion path of the user's hand may be compared to a database of motion paths to match a character in the database and thereby to determine the character drawn by the user.
  • motion path features are extracted from the tracking of the hand.
  • a motion path of the hand may then be determined based on the features, using machine learning techniques (such as by using a decision tree or neural network) trained on a database of motion path features.
  • the determined motion path of the hand may then be used to control the device.
  • a user's hand 301 (possibly, but not necessarily, in a predetermined posture) is tracked by the system and location parameters, such as coordinates 302 and 304 are accumulated, possibly stored, typically as vectors, in a suitable data structure 306.
  • location parameters such as coordinates 302 and 304 are accumulated, possibly stored, typically as vectors, in a suitable data structure 306.
  • features such as shapes or parts of shapes (circle, arc, line etc.), may be derived from the location parameters (which may be coordinates or different parameters such as information relating to start and end of motion, change of direction of motion and angles within the motion path).
  • the features include information relating to character formation.
  • the features include information relating to "swipe" formation.
  • the motion path features database may be constructed off-line, prior to a use of the system by a user. According to some embodiments the database may be updated on-line, motion path features being added or included in the calculations or decisions being carried out by the data structure 307, during use of the system.
  • the location parameters and/or the features derived or extracted from them and/or motion paths determined from these features are then compared to a database of motion paths 308, which may include a look-up table, to closely associate or match the accumulated location parameters and/or features to a set of known of characters (e.g., letters, number, geometrical shapes and other symbols).
  • a database of motion paths 308 which may include a look-up table, to closely associate or match the accumulated location parameters and/or features to a set of known of characters (e.g., letters, number, geometrical shapes and other symbols).
  • the location parameters or other parameters may be used to create an image of the motion path.
  • the created image may then be translated to specific characters using known algorithms for character recognition, such as Optical Character Recognition techniques.
  • characteristics of the user's hand's motion path are stored in a motion path features database and that database is then used for quicker, more accurate tracking and for quicker identification of specific motion paths.
  • the user's hand is tracked (402) and motion path features may be extracted. If the motion path of the tracked hand is determined to be a known specific motion path (e.g., by using the extracted motion path features in a machine learning process trained on a motion path features database (404)), then a motion path may be determined (406) from which a gesture or character may be derived (408). The motion path features of the determined motion path may then be stored in a motion path features database (410) (in the first database (404) or in a different database or need not be stored in a database but may be used to update processes in data structures) such that the information from a newly determined motion path may become available for updating the process of determining a motion path in the next occurrence.
  • a motion path features database (410) (in the first database (404) or in a different database or need not be stored in a database but may be used to update processes in data structures) such that the information from a newly determined motion path may become available for updating the process of determining a motion path in the next occurrence.
  • a device may be controlled based on the derived gesture or character (412), such as described above.
  • Characteristics of a motion path may include location parameters and features as described above (e.g., information relating to start and end of motion, change of direction of motion, angles within the motion path etc.). Other characteristics may also be stored in a motion path features database.
  • a motion path of the user's hand is determined based on the tracking of the user's hand and based on a motion path features database which can be continuously updated to provide progressively improving detection of a user's "writing" and/or other gestures.
  • a user may be notified by the system when he is in "writing mode" so that the user can plan his hand movements or gestures accordingly.
  • a user's hand may be tracked while being checked for specific postures. Alternatively, a posture of the user's hand may be determined prior to tracking the hand. If a first posture is identified then, according to some embodiments, a first indication is provided to the user to notify the user that the system is now in "writing mode". When the first posture is no longer detected (or when a specific second posture is detected) a second indication may be provided to the user to notify the user that the system is no longer in "writing mode".
  • the first indication and/or the second indication may include displaying a graphical indication to the user, such as displaying an icon of a hand in the first posture when the first posture is detected and displaying an icon of a hand in a second posture when the second posture is detected.
  • indications to the user may include sound or other alerts.
  • a graphical representation of the motion path of the user's hand is displayed to the user.
  • the user's hand 501 is tracked and a motion path of the hand 501 is derived. If the motion path of the user's hand 501 is determined to resemble the numeral 3 then the numeral 3 will be displayed on display 502 so that the user may be notified of the character determined by the system based on the tracking of his hand.
  • the user's hand 501 is in a pre-determined posture (for example, with all fingers folded except one extending or pointing finger), however, a character may be determined (and displayed to the user) also when the tracked hand is no longer in the pre-determined posture and even when no specific posture is detected.
  • deriving the motion path of the user's hand may be done by the system based on considerations, for example, based on the probability that the user intends to input characters. These considerations may be, for example, related to the distance of the user's hand from a display or related to the angle of the user's hand or finger relative to the display.
  • the distance of the user's hand from a display may be determined and the motion path of the hand may be determined only if the distance is below a predetermined threshold.
  • an angle of the user's hand or finger relative to a display may be determined and the motion path may be determined only if the angle is lower or higher than a predetermined value.
  • FIG. 6 A TV set operable according to embodiments of the invention is schematically illustrated in Fig. 6.
  • the TV set (600) includes a TV display (601) and a channel selector (602) for selecting a channel to be played on the TV display (601).
  • the TV set (600) also includes a camera (604) to obtain a sequence of images which include a user's hand (61).
  • the TV set (600) also includes a processor (603) to track the user's hand through the sequence of images and to determine a character based on the tracking of the user's hand.
  • the processor (603) is in communication with the channel selector (602) to select a channel based on the determined character.
  • a user may position his hand (61) within the FOV of the camera (604) and may then write a number (for example) in the air to indicate which channel should be played.
  • the camera (604) obtains a sequence of images, the images including the user's hand.
  • the processor (603) tracks the user's hand through the sequence of images and derives a motion path of the hand.
  • the processor (603) determines a character based on the derived motion path and communicates to the channel selector (602) the channel corresponding to the determined character.
  • the user's hand is typically detected based on its shape.
  • the user may bring his hand into a pre-determined posture (e.g., a posture in which all fingers of the hand are brought together such that the finger tips are touching or almost touching, such as when the hand is closed in a fist or as if holding a bulb or a hand having all fingers folded except one finger that is extended or a hand in which a finger and the thumb are held together in a pinching-like posture or other postures) for the processor (603) or a different processor to identify the shape of the user's hand (posture of the hand) and to determine a character based on the tracking of the hand (either tracking of the hand while in the first posture or tracking of the hand regardless of the posture of the hand during tracking).
  • the desired channel e.g., writing the desired number
  • the user may open his hand (or change the posture of his hand to a different posture) and future hand movements will not be interpreted by the system as characters.
  • Methods for selecting channels as described above may be used with other devices such as PCs, a set top box, or a video or audio output device.
  • the steps of the methods for selecting channels may include steps described above.
  • the system may differentiate between different types of hand motion paths, e.g., an intentional motion path and an unintentional motion path of the user's hand or character motion paths and non-character motion paths.
  • hand motion paths e.g., an intentional motion path and an unintentional motion path of the user's hand or character motion paths and non-character motion paths.
  • a method for controlling a device is schematically illustrated in Fig. 7.
  • the method may include receiving first and second sequences of images (which include a user's hand) (702) and using the first sequence of images to create a first database of hand motion path features which corresponds to a first type of hand motion paths and the second sequence of images to create a second database of hand motion path features which corresponds to a second type of hand motion paths (704).
  • the method further includes obtaining a first set of hand motion path features from the first sequence of images and a second set of hand motion path features from the second sequence of images (706).
  • a classifier is then trained, using the first and second sets of hand motion path features (708).
  • a user's hand may then be tracked in a third sequence of images, to obtain a third set of hand motion path features (710).
  • the third set of hand motion path features is input to the trained classifier (712) and an indication of the type of hand motion paths the third set of hand motion path features belongs to, may thus be obtained (714), dependent on classifier- based decisions.
  • This indication may be output to a processor (716) and the processor can control a device based on the indicated type (718).
  • each motion path feature is labeled.
  • Each feature may be differentially labeled according to the different type of motion path it represents (e.g., predefined hand gestures, unintended hand gestures, hand movement paths in a predetermined direction, arc-shaped paths and non-arc-shaped paths or linear paths).
  • a database of motion path features may include a plurality of features, each feature having a different label.
  • a user's hand may be tracked through a sequence of images during operation of a system, to derive a motion path of the user's hand and to determine if the derived motion path is a pre-determined, intentional, motion path or not.
  • an "intentional motion path" database may be created by using pre-defined hand gestures and an “unintentional motion path” database may be created by using gestures or hand movements other than the pre-defined hand gestures of the first database , e.g., by performing steps 702- 708 described above.
  • the user's hand is then tracked during operation of a system and the set of hand motion path features of the tracked hand are used to obtain an indication of the type of hand motion paths the user's hand motion path features belongs to (e.g., as described in steps 712- 714 above).
  • This indication may include a probability or a grade that relates to a probability of the set of user's hand motion path features belonging to either a first type of hand motion paths (e.g., intentional motion) or a second type (e.g., unintentional motion).
  • This indication is then used to control a device, such as described in steps 716-718 above.
  • a processor may carry out a user command based on the hand gesture. If the user's hand gesture is determined to have a probability of over 50% of being an unintentional motion (e.g., by using an unintentional database to train a classifier or by using information from an intentional database) then the processor will not carry out a user command.
  • this method may be used to control a display of a device.
  • a swipe gesture e.g., defined as being a left to right movement path of a hand
  • a first type of hand motion paths will include hand movement paths in a first direction
  • the second type of hand motion paths will include movement paths in a second direction.
  • a left to right movement of a user's hand will generate a "swipe command”
  • a movement of the hand in a different direction e.g., a reverse direction
  • a user may swipe and immediately move his hand back to swipe again without having the displayed content move back and forth on the display but only move according to the first (and intended) hand movement.
  • a first type of hand motion paths may include arc shaped paths whereas the second type of hand motion paths includes non-arc shaped paths (e.g., linear paths).
  • the first type of hand motion paths may include negative arcs (open downwards) and the second type of hand motion paths may include positive arcs (open upwards).
  • a first type of hand motion types may include character indicating gestures.
  • the system may be controlled based on identification of the character. If however no movement of the hand or other, non-character indicating, gestures are detected then the system will not be controlled based on identification of characters.
  • Methods as described above enable easy detection and classification of user hand gestures, providing smooth and user friendly computer vision based control of devices.

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Human Computer Interaction (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Social Psychology (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Databases & Information Systems (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

La présente invention concerne un procédé consistant à identifier la posture de la main d'un utilisateur; à poursuivre la main de l'utilisateur tout au long de la séquence d'images; à déduire un trajet de mouvement de la main de l'utilisateur; à déterminer un caractère sur la base du trajet de mouvement déduit; et à commander un dispositif sur la base du caractère déterminé et sur la base de l'identification de la posture. Un trajet de mouvement de la main de l'utilisateur peut être déterminé sur la base de techniques d'apprentissage automatique soumises à un apprentissage sur une base de données de traits caractéristiques de trajets de mouvement.
PCT/IL2014/050011 2013-01-06 2014-01-06 Procédé d'identification d'un trajet de mouvement WO2014106849A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201361749328P 2013-01-06 2013-01-06
US61/749,328 2013-01-06

Publications (1)

Publication Number Publication Date
WO2014106849A1 true WO2014106849A1 (fr) 2014-07-10

Family

ID=51062193

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IL2014/050011 WO2014106849A1 (fr) 2013-01-06 2014-01-06 Procédé d'identification d'un trajet de mouvement

Country Status (2)

Country Link
IL (1) IL230339A0 (fr)
WO (1) WO2014106849A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8938124B2 (en) 2012-05-10 2015-01-20 Pointgrab Ltd. Computer vision based tracking of a hand
US9665769B2 (en) 2015-08-18 2017-05-30 International Business Machines Corporation Handwriting recognition with natural user input on multitouch surfaces

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110110560A1 (en) * 2009-11-06 2011-05-12 Suranjit Adhikari Real Time Hand Tracking, Pose Classification and Interface Control
US20110221974A1 (en) * 2010-03-11 2011-09-15 Deutsche Telekom Ag System and method for hand gesture recognition for remote control of an internet protocol tv
US20110304541A1 (en) * 2010-06-11 2011-12-15 Navneet Dalal Method and system for detecting gestures

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110110560A1 (en) * 2009-11-06 2011-05-12 Suranjit Adhikari Real Time Hand Tracking, Pose Classification and Interface Control
US20110221974A1 (en) * 2010-03-11 2011-09-15 Deutsche Telekom Ag System and method for hand gesture recognition for remote control of an internet protocol tv
US20110304541A1 (en) * 2010-06-11 2011-12-15 Navneet Dalal Method and system for detecting gestures

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8938124B2 (en) 2012-05-10 2015-01-20 Pointgrab Ltd. Computer vision based tracking of a hand
US9665769B2 (en) 2015-08-18 2017-05-30 International Business Machines Corporation Handwriting recognition with natural user input on multitouch surfaces

Also Published As

Publication number Publication date
IL230339A0 (en) 2014-08-31

Similar Documents

Publication Publication Date Title
US20220261112A1 (en) Systems, devices, and methods for touch-free typing
US20180292907A1 (en) Gesture control system and method for smart home
US10120454B2 (en) Gesture recognition control device
US10126826B2 (en) System and method for interaction with digital devices
US8902198B1 (en) Feature tracking for device input
US10042438B2 (en) Systems and methods for text entry
US20150220149A1 (en) Systems and methods for a virtual grasping user interface
KR20130105725A (ko) 콘텐츠의 양 손 제어에 기반한 컴퓨터 비전
US8938124B2 (en) Computer vision based tracking of a hand
US20140118244A1 (en) Control of a device by movement path of a hand
US20170192519A1 (en) System and method for inputting gestures in 3d scene
CN103809733A (zh) 人机交互系统和方法
US20200142495A1 (en) Gesture recognition control device
US20140368434A1 (en) Generation of text by way of a touchless interface
CN109074224A (zh) 用于在字符串中插入字符的方法以及相应的数字设备
TW201543268A (zh) 用於使用手勢控制媒體播放之系統及方法
US20130285904A1 (en) Computer vision based control of an icon on a display
JP2015172887A (ja) ジェスチャ認識装置およびジェスチャ認識装置の制御方法
KR20140110356A (ko) 입력 장치, 디스플레이 장치 및 그 제어 방법
CN105242776A (zh) 一种智能眼镜的控制方法及智能眼镜
Wu et al. Beyond remote control: Exploring natural gesture inputs for smart TV systems
WO2014106849A1 (fr) Procédé d'identification d'un trajet de mouvement
KR101809278B1 (ko) 사용자의 움직임을 감지하여 문자를 입력하는 장치 및 방법
US20140301603A1 (en) System and method for computer vision control based on a combined shape
CN103809912A (zh) 基于多点触控触摸屏的平板电脑

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 14735310

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 14735310

Country of ref document: EP

Kind code of ref document: A1