JP5788853B2 - System and method for a gesture-based control system - Google Patents

Description

  This patent application is the priority of US Provisional Patent Application No. 60/651290, "Gesture-Based Control System", filed February 8, 2005, which is incorporated herein by reference in its entirety. It is what I insist.

  The present invention relates generally to the field of computer systems, and more particularly to systems and methods for gesture-based control systems.

  The user uses an input device such as a mouse, keyboard, joystick, or cross key to manipulate data or images in a window on the display, or by selecting an operation from a menu or associated program associated with the window Commands can be entered and interacted with the computer system. Such an input device can also operate as a position translation device that can be used to place a graphical on-screen pointer such as a cursor. The cursor functions, for example, to indicate a character to be corrected, or to indicate a position where data is input or an operation is performed. Some form or appearance of cursor is generally present on the computer display. The operation of the input device by the user results in a corresponding movement of the cursor. Thus, for example, when the mouse or other input device moves, the cursor moves in the same direction.

  The cursor may have a different appearance depending on its function and the state of the computer system. For example, when placed within a field of text on a computer display, the cursor may have the appearance of an “I-beam” or blinking vertical line. The position of the cursor in the text field indicates the position of the next character that is typically entered by the user via the keyboard. The cursor may have other appearances depending on its function. In a drawing or painting program, the cursor may be represented as a paint brush, pencil, eraser, bucket or other shape.

  The cursor can also be in the form of an arrow or pointer when placed over a user-selectable operation or used to select a graphic element such as a window. In order to select and activate the desired operation with the cursor, it can be placed on a graphic or textual representation of the operation. To perform an operation, a button located on the mouse input device may be pressed and / or released. The user is notified by visual feedback about the acceptance of an execution operation, usually in the form of some change in the image of the computer display. One or more of the programs in use typically generate this visual response. These programs generate a drawing command for updating the display image in response to the selected operation.

  A disadvantage of prior art systems is that the input device is often simply a device. The user is required to have a wired or wireless mouse, or other input device, and use that device to manage selection, repositioning, activation and other input functions. The use of these physical devices is often not natural or intuitive. Another drawback is the need to go through specific steps to change the context of the input device so that different functions can be performed.

As very large displays become widespread, further disadvantages of prior art input devices and systems become apparent. When using a mouse, for example, to move the position of the cursor across a large display, the user often lifts the mouse to allow the user to drag the cursor even though it is part of the large display, Must be placed on the mouse surface. This is a useless, unnatural exercise.

  Several prior art attempts have been made to provide solutions to these problems. One prior art solution is to use a glove in the user's hand. These gloves are intended to make the user's hand or hands the input device. In one embodiment, the input globe is hard-wired to the computer system. This solution has the disadvantage that it effectively ties the user to the location, is near the computer system, and requires a limited range of motion. In other cases, the globe is wireless. However, such wireless implementations require a separate power source for the globe. The glove cannot be used when the power supply needs to be charged.

  This system provides a gesture interface to a display screen or various visually presented elements presented on the screen.

  The operator of the system, in one embodiment, navigates and manipulates these elements by issuing a continuous stream of “gesture commands” using the operator's hand. In other embodiments, the user's head, feet, arms, legs, or the user's entire body may be used to provide navigation and control. Gesture vocabulary consists of “instant” commands that provide an immediate “one-off” action by forming an appropriate “pause” with one or both hands, and an on-screen element by an operator with a literal “pointing” (literal pointing) gesture. And a “spatial” command that performs navigation operations with relative or “offset” gestures. In addition to pointing gestures used for global or direct spatial gestures, the present invention can also recognize another category of relative spatial navigation gestures in XYZ space. This operational category is sometimes referred to as XYZ technology. By maintaining both high frame rates, guaranteeing almost undetectable delays in the interpretation of operator gestures, the system uses both carefully designed spatial metaphors and easily apparent “direct manipulation” mechanisms. Provides a clear “cognitive link” between the operator and the information & process represented. This system contemplates the ability to identify a user's hand. This identification system may be in the form of a glove, or a glove with a specific display on it, or any suitable means for providing a recognizable display on the user's hand, Also good. The camera system can detect the position, orientation and movement of the user's hand and translate that information into executable commands.

  A system and method for a gesture-based control system is described. In the following description, numerous features are set forth in detail in order to provide a more thorough understanding of the present invention. It will be apparent that the invention may be practiced without these specific details. In other cases, well-known features have not been described in detail.

System A block diagram of one embodiment of the present invention is shown in FIG. The user finds his hands 101 and 102 within the field of view of the cameras 104A-104D side by side. The camera detects the position, orientation and movement of the fingers and hands 101 and 102 and generates an output signal to the preprocessor 105. The preprocessor 105 converts the camera output into a gesture signal,
This gesture signal is supplied to the computer processing unit 107 of the system. Computer 107 uses the input information to generate commands that control one or more cursors on the screen and provides video output to display 103.

  Although this system is shown including a single user's hand as input, the present invention can also be implemented using multiple users. In addition, the system may track any part or parts of the user's body, including the head, feet, legs, arms, elbows, knees, etc., instead of or in addition to the hand.

  In the illustrated embodiment, four cameras are used to detect the position, orientation and movement of the user's hands 101 and 102. It should be understood that the present invention can be used with more or fewer cameras without departing from the scope or spirit of the present invention. Furthermore, although the cameras are arranged symmetrically in this exemplary embodiment, the present invention does not have such symmetry requirements. In the present invention, any number of cameras that allow the position, orientation and movement of the user's hand or camera positioning can be used.

  In one embodiment of the invention, the camera used is a motion capture camera capable of capturing grayscale images. In one embodiment, the camera used is from Vicon, such as a Vicon MX40 camera. This camera includes processing on the camera and can capture images at 1000 frames / second. The motion capture camera can detect the marker and locate the marker.

  In the described embodiment, the camera is used for light detection. In other embodiments, a camera or other detector may be used for electromagnetic, magnetostatic, RFID or any other suitable type of detection.

  The preprocessor 105 is used to generate 3D spatial point reconstruction and skeleton point labeling. The gesture converter 106 is used to convert 3D spatial information and marker motion information into a command language, which is interpreted by a computer processor to update the position, shape and movement of the cursor on the display. Can. In an alternative embodiment of the present invention, preprocessor 105 and gesture converter 106 can be combined into a single device.

  The computer 107 may be any general purpose computer such as those manufactured by Apple, Dell, or any other suitable manufacturer. Computer 107 executes the application and provides display output. Otherwise, cursor information coming from a mouse or other prior art input device now comes from a gesture system.

Marker Tag The present invention provides one or more fingers of a user so that the system can find the user's hand and identify whether it is looking at the left or right hand and which finger is visible. It is intended to use a marker tag on the top. This allows the system to detect the position, orientation and movement of the user's hand. This information allows multiple gestures to be recognized by the system and used as commands by the user.

The marker tag in one embodiment comprises a substrate (which in this embodiment is suitable for being placed at various locations on the human hand) and individual arrays arranged with a unique identification pattern on the surface of the substrate. A physical tag including a marker.

  The marker and associated external sensing system can operate in any region (optical, electromagnetic, magnetostatic, etc.) that allows accurate, precise, rapid and continuous acquisition of its three spatial positions. The marker itself can operate actively (for example by radiating structured electromagnetic pulses) or passively (for example by being optically retroreflective as in this embodiment). Sometimes.

  At each acquisition frame, the detection system collects a set of recovered three spatial positions that includes all markers from tags currently in the work area volume with the instrument (in the field of view of the camera or other detector). Receive the body's “cloud” (aggregate cloud). The markers on each tag are sufficient and are arranged in a unique pattern so that the detection system can perform the following tasks. (1) Segmentation where each recovered marker position is assigned to a unique subset of points forming a single tag, (2) Each segmented subset of points is identified as a particular tag Labeling, (3) positioning in which the 3-space position of the identified tag is restored, and (4) orientation determination in which the 3-space orientation of the identified tag is restored. Tasks (1) and (2) are enabled by the specific nature of the marker pattern, described below and as shown in one embodiment of FIG.

  Markers on the tag of one embodiment are attached to a subset of normal grid positions. As in this embodiment, this underlying grid may be of the conventional Cartesian type, or otherwise, some other normal surface mosaic (eg triangle / hexagon / Tile arrangement). Grid scales and spacing are established with respect to the known spatial resolution of the marker sensing system so that adjacent grid positions are less likely to be confused. The choice of marker pattern for all tags should meet the following constraints: The tag pattern matches that of any other tag pattern by any combination of rotation, translation, or mirroring is not. The diversity and arrangement of markers can further be selected to allow for the loss (or occlusion) of some number of component markers. After any arbitrary conversion, the likelihood of confusing a damaged module with any other module should still be low.

  Referring now to FIG. 2, a plurality of tags 201A-201E (left hand side) and 202A-202E (right hand) are shown. Each tag is rectangular and in this embodiment consists of a 5 × 7 grid array. The rectangular shape is selected to help determine tag orientation and to reduce the possibility of mirror replication. In the embodiment shown, there is a tag for each finger of each hand. In some embodiments, it may be sufficient to use one, two, three or four tags per hand. Each tag has a different grayscale or shade of color border. Within this boundary is a 3 × 5 grid array. Markers (represented by black dots in FIG. 2) are placed at specific points in the grid array to provide information.

Qualifying information may be encoded with the marker pattern of the tag by segmenting each pattern into “common” and “unique” sub-patterns. For example, this embodiment specifies the distribution of markers for two possible “boundary patterns”, rectangular boundaries. A “family” of tags is thus established, so the tag for the left hand uses the same boundary pattern, as shown in tags 201A-201E, and is attached to the finger of the right hand for tag 202A. Different patterns may be assigned, as shown at ~ 202E. This sub-pattern is chosen so that the left pattern can be distinguished from the right pattern in all orientations of the tag. In the example shown, the left-hand pattern includes a marker on each corner and on the second marker from the corner grid position. The right hand pattern has markers at only two corners and has two markers at non-corner grid locations. Inspection of the pattern reveals that the left-hand pattern can be clearly distinguished from the left-hand pattern as long as any three of the four markers are visible. In one embodiment, the border color or tint can also be used as an indication of the dominant hand.

  Each tag, of course, still must use a unique internal pattern, and the markers are distributed within the common boundaries of the family. In the illustrated embodiment, it has been found that two markers in the internal grid array are sufficient to uniquely identify each of the ten fingers without duplication due to finger rotation or orientation. Even if one of the markers is occluded, a unique identifier is provided by the combination of patterns and the dominant hand of the tag.

  In this embodiment, the grid location is visually present on the rigid substrate to aid in the (manual) task of attaching individual retroreflective markers at its intended location. These grids and intended marker positions are literally printed on a substrate by a color ink jet printer, which in this case is (initially) a sheet of flexible “shrink film”. Each module is cut from the sheet and then baked in a furnace, and during the heat treatment, each module undergoes accurate and repeatable shrinkage. During the short interval following this procedure, the cooling tag may be shaped slightly, for example to follow the vertical curve of the finger, after which the substrate will be properly hardened and the marker will be shown at the indicated grid point. Can be attached to.

  In one embodiment, the marker itself is three-dimensional, such as a small reflective sphere attached to the substrate by adhesive or some other suitable means. The three-dimensional nature of the marker can help detection and positioning compared to a two-dimensional marker. However, any can be used without departing from the spirit and scope of the present invention.

  Currently, tags are attached to the glove worn by the operator by Velcro or other suitable means, or alternatively directly attached to the operator's finger using soft double-sided tape. In a third embodiment, it is possible to dispense with all hard substrates and attach or “paint” individual markers directly on the operator's fingers and hands.

Gesture Vocabulary The present invention contemplates a gesture vocabulary consisting of hand poses, orientations, hand combinations, and a mixture of orientations. A notation language is also implemented to design and communicate poses and gestures in the gesture vocabulary of the present invention. The gesture vocabulary is a system for expressing a momentary “pose state” of kinematic coupling in a simple text form. The connection may be biological (eg, a human hand, or the entire human body, or a leg of a grasshopper, or a lemur's articulated spine), or otherwise, non-biological (eg, a robotic arm). ). In either case, the bond may be simple (spine) or branched (hand). The gesture vocabulary system of the present invention establishes a certain length sequence for any particular combination, and then the collection of specific ASCII characters that occupy the "character position" of the sequence is the instantaneous A unique description of a state or “pause”.

Hand Pose FIG. 3 shows a hand pose for one embodiment of a gesture vocabulary using the present invention. In the present invention, it is assumed that each of the five fingers of the hand is used. These fingers are codes as p-small finger, r-ring finger, m-middle finger, i-index finger and t-thumb. Multiple poses for fingers and thumbs are defined and shown in FIG. The gesture vocabulary string establishes a single character position for each expressible degree of freedom of the combination (in this case a finger). further,
Each such degree of freedom is discretized (or “quantized”) so that its full range of motion can be represented by assigning one of a finite number of standard ASCII characters at that string position. ). These degrees of freedom are expressed in terms of the body-specific origin and coordinate system (back of hand, locust body center, robot arm base, etc.). Thus, a small number of additional gesture vocabulary character positions are used to represent the position and orientation of the join “in general” in a more global coordinate system.

  Still referring to FIG. 3, multiple poses are defined and identified using ASCII characters. Some of the poses are split between the thumb and non-thumb. The present invention uses, in this embodiment, an encoding in which the ASCII character itself implies a pose. However, any character can be used to represent a pose, whether implicit or not. Further, in the present invention, there is no requirement to use ASCII characters for the notation sequence. Any suitable symbol, number or other expression may be used without departing from the scope and spirit of the invention. For example, the notation may use 2 bits per finger if desired, or any other number of bits as desired.

  A rounded finger is represented by the letter “^”, and a rounded thumb is represented by “>”. A straight finger or an upward thumb is indicated by “1”, and a tilted one is indicated by “\” or “/”. “-” Represents a thumb pointing straight sideways, and “x” represents a thumb facing the surface.

  Using these individual finger and thumb depictions, a substantial number of hand poses can be defined and written using the scheme of the present invention. Each pose is represented by five characters in the order p-r-m-t as described above. FIG. 3 illustrates a plurality of poses, where several poses are described for purposes of illustration and illustration. A hand held flat and parallel to the ground is represented by “11111”. A fist is represented by “^^^^>”. The “OK” sign is represented by “111 ^>”.

  Strings provide an easy and clear “human readable” opportunity when using suggestive characters. A set of possible strings indicating the respective degrees of freedom can generally be selected in view of quick recognition and obvious similarities. For example, a vertical line (“|”) probably means that the connecting element is “straight”, and an L-shape (“L”) can mean a 90 degree bend, and a mountain symbol (“^” ") May indicate a sharp bend. As described above, any character or encoding can be used as desired.

  Any system that uses the example gesture vocabulary as described herein will benefit from the high computational efficiency of column comparison, and any specified pose identification or search is effectively the desired pose. It becomes a “column comparison” between the column and the instantaneous actual column (eg, UNIX® “strcmp ()” function). In addition, the use of “wildcard characters” provides programmers or system designers with additional, well-known efficiency and effectiveness, with the degree of freedom that their instantaneous state is unrelated to matching is a question mark. ("?") And can be assigned additional wildcard meanings.

Orientation In addition to finger and thumb poses, hand orientation can represent information. Characters representing the orientation of the global space can also be selected transparently, and the characters “<”, “>”, “^” and “v” are left, right, Can be used to show the top and bottom ideas. FIG. 4 shows an example of hand orientation descriptors and encoding that combines pose and orientation. In one embodiment of the invention, the two character positions first indicate the direction of the palm and then the direction of the finger (if they are straight, regardless of the actual bending of the finger). The possible letters for these two positions represent the “body-centric” concept of orientation, where “−”, “+”, “x”, “*”, “^” and “v” are inside, The horizontal, forward (away from the body, forward), backward (away from the body, backward), head (upward), and tail (downward) directions are expressed.

  In the notation system of the present invention and its embodiments, a five finger pose representing a character is followed by a colon and then two orientation characters to define a complete command pose. In one embodiment, the starting position is referred to as an “xyz” pose, where the thumb points straight up, the index finger points forward, the middle finger is perpendicular to the index finger, and left when the pose is performed with the right hand Point to. This is represented by the sequence “^^ l −: − x”.

  “XYZ Hand” is a technique for utilizing the geometry of the human hand to enable full six-degree-of-freedom navigation of visually presented three-dimensional structures. This technique relies solely on a large amount of transformation and rotation of the operator's hand, so that the finger can in principle be held in any desired pose, but in this embodiment the forefinger points away from the body and the thumb A static configuration is preferred, with the pointing toward the ceiling and the middle finger pointing to the left and right. Thus, the three fingers (for a rough but quite obvious purpose) show the three-space coordinate system and hence the "XYZ Hands" orthogonal axes.

  The XYZ hand navigation then proceeds to the hand and fingers of the pose described above, held in front of the operator's body in a predetermined “neutral position”. Access to three transformations and three rotational degrees of freedom of a three-space object (or camera) is done in the following natural way: Left and right hand movements (relative to the body's natural coordinate system) result in movement along the X axis of the computational context, and vertical movements of the hand result in movement along the Y axis of the controlled context, Front and back hand movements (toward and away from the operator's body) result in z-axis movement in context. Similarly, the rotation of the operator's hand around the index finger leads to a “rotation” change in the orientation of the calculation context, and the changes in “pitch” and “sway” change the operator's hand around the middle finger and thumb, respectively. Is done in the same way.

  As used herein, “computational context” is used to refer to entities controlled by XYZ hand methods, and appears to suggest either a composite three-space object or a camera, but this technique Should be understood to be equally useful for real-world objects, control of various degrees of freedom, for example control of pan / tilt / rotation of video or video cameras with appropriate rotational actuators. Furthermore, some of the physical degrees of freedom given by the XYZ hand poses may not be mapped so literally within the virtual region. In this embodiment, the XYZ hand is also used to provide navigation access to a large panoramic display image, so that the left and right and up and down movements of the operator's hand are expected left and right or up and down around the image. "Panning" of the operator, but the front and back movements of the operator's hand map to the "zooming" control.

  In all cases, the coupling between hand movement and the resulting computational transformation / rotation is straightforward (ie, the operator's hand position or rotational offset is via some linear or non-linear function) Mapping one-to-one to the position or rotational offset of the object or camera in the computational context, or indirect (ie, the position or rotational offset of the operator's hand is via some linear or non-linear function, Mapping in the calculation context may be a one-to-one mapping to the first or higher order derivative of the position / orientation), and then the ongoing integration Change. This latter control means is similar to the use of an automobile “access pedal”, where a constant offset of the pedal results in a more or less constant vehicle speed.

  The “neutral position”, which serves as the origin of the local 6-degree-of-freedom coordinates of the XYZ hand in the real environment, is (1) the absolute position and orientation in space (for example relative to the surrounding room) Regardless of the overall position and “direction”, as a fixed position and orientation with respect to the operator himself (eg, 20.32 cm (8 inches) in front of the body, 25.4 cm (10 inches) under the chin), and Such as laterally along the shoulder surface), or (3) (eg, the operator's “other” indicating that the current position and orientation of the XYZ hand should be used as the origin of transformation and rotation from now on) It can be established interactively through the intentional secondary action of the operator (using gesture commands executed by one hand).

  It is further advantageous to provide a “detent” region (or “dead zone”) for the neutral position of the XYZ hand so that movements in this volume are not mapped to movements in the controlled context.

Other poses include:
[||||: VX] is a palm with the palm facing downward and the finger facing forward (thumb is parallel to the finger).
[||||: X ^] is a palm with the palm facing forward and the finger facing the ceiling.
[||| ::-X] is a palm with the palm facing the center of the body (right for the left hand, left for the right hand) and the finger facing forward.
[^^^^-:-X] is an upright thumb with a single hand (thumb points to the ceiling).
[^^^ |-:-X] is a forward gun gesture.

Combination of Both Hands The present invention contemplates a single hand command, pose, and command and pose using both hands. FIG. 5 shows an example of a combination of both hands and an associated notation according to an embodiment of the present invention. A closer look at the notation in the first example reveals that “complete stop” includes two closed fists. In the “snapshot” example, the thumb and index finger of each hand are stretched, the thumbs point towards each other, defining a frame in the form of a goal post. The “ladder and throttle start positions” are such that fingers and thumbs face up and the palm faces the screen.

Orientation Mixing FIG. 6 shows an example of orientation mixing in one embodiment of the present invention. In the example shown, blending is represented by enclosing pairs of orientation notations in parentheses after the finger pose row. For example, the first command shows the finger positions all pointing straight. The first pair of orientation commands causes the palm to flatten toward the display, and the second pair has hands rotating at a 45 degree pitch toward the screen. In this example, pairs of blends are shown, but any number of blends is contemplated by the present invention.

Exemplary Commands FIG. 8 illustrates a number of possible commands that may be used with the present invention. Part of the discussion here has been with cursor control on the display, but the invention is not limited to that activity. Indeed, the present invention has great applicability in manipulating any and all data on the screen, and portions of data, as well as the state of the display. For example, commands can be used to replace video controls during playback of video media. Commands can be used for pause, fast forward, rewind, etc. Further, the commands may be implemented for zooming in and out of the image, changing the orientation of the image, panning in any direction, and the like. The present invention can also be used in place of menu commands such as open, close and save. In other words, any command or activity that can be imagined can be performed with a hand gesture.

Operation FIG. 7 is a flow chart illustrating the operation of the present invention of one embodiment. In step 701, the detection system detects markers and tags. At decision block 702, it is determined whether a tag and marker have been detected. Otherwise, the system returns to step 701. If a tag and marker are detected at step 702, the system proceeds to step 703. In step 703, the system identifies hands, fingers and poses from the detected tags and markers. In step 704, the system identifies the orientation of the pose. In step 705, the system identifies the detected three-dimensional spatial position of the one or more hands. (Note that any or all of steps 703, 704, and 705 may be combined as a single step.)
At step 706, the information is converted to the gesture notation described above. At decision block 707, it is determined whether the pause is valid. This can be accomplished by a simple column comparison using the generated notation sequence. If the pause is not valid, the system returns to step 701. If the pause is valid, the system sends the notation and location information data to the computer at step 708. In step 709, the computer determines the appropriate action in response to the gesture and in step 710 updates the display accordingly.

  In one embodiment of the present invention, steps 701-705 are performed by a processor on the camera. In other embodiments, the processing can be performed by a system computer, if desired.

Analysis and Transformation This system “analyzes” and “transforms” the low-level gesture flow recovered by the underlying system and uses those analyzed and transformed gestures to control a wide range of computer applications and systems. Can be turned into command flow or event data that can be used to These techniques and algorithms may be implemented in a system consisting of an engine that implements these techniques and computer code that provides a platform for building computer applications that utilize the capabilities of the engine.

  One embodiment is focused on enabling the use of abundant gestures of the human hand in a computer interface, including but not limited to the body, torso, legs and head. Various types of both static and articulating hands, including, but not limited to, calipers, compass, flexible curve approximators and various forms of pointing devices It is also possible to recognize gestures made by physical tools that do not depend on. Markers and tags may be applied to items and tools that can be carried and used by an operator as desired.

  The system described herein incorporates multiple innovations that allow to build gesture systems that are rich in a wide range of recognized and actable gestures, while at the same time allowing easy integration into applications .

The gesture analysis and conversion system of one embodiment consists of the following.
1) A plurality of different aggregate level gestures: a. A single hand “pose” (composition and orientation of hand parts relative to each other), a single hand orientation and position in 3D space;
b. A two-handed combination for either hand that takes into account pose, position or both, c. Multiple people combination; this system can track more than two hands, so multiple people can work together (or compete in the case of gaming applications) to control the target system it can,
d. Continuous gestures in which poses are combined sequentially; these are called "animated" gestures,
e. “Grapheme” gesture, where the operator tracks the shape of the space,
A concise and efficient way to specify (encode for use in a computer program).
2) Programmatic techniques for registering specific gestures from each of the above categories associated with a given application context.
3) An algorithm for parsing the gesture stream so that registered gestures can be identified and events encapsulating those gestures can be passed to the appropriate application context.

The designation system (1) with components (1a) through (1f) provides the basis for utilizing the gesture analysis and transformation capabilities of the system described herein.
A single hand “pose”
i) represented as a sequence of relative orientations between the fingers and the back of the hand,
ii) Quantized to a small number of individual states.

  By using relative bond orientations, the system described herein can avoid the problems associated with various hand sizes and geometries. In this system, “operator calibration” is not necessary. Furthermore, by specifying poses as relative orientation columns or groups, more complex gesture specifications can be easily created by combining pose expressions with additional filters and specifications.

  Using a small number of individual states for pose designation, concise designation of poses, as well as various underlying tracking techniques (eg passive light tracking using cameras, active light tracking using illuminated dots and cameras, electromagnetic field tracking) Etc.) can be used to guarantee accurate pose recognition.

  Gestures within any category (1a) to (1f) can be partially (or minimally) specified so that non-critical data is ignored. For example, a gesture in which the position of two fingers is critical and the position of the other fingers is not important can be represented by a single designation, where the associated two-finger operating position In the same column, a “wild card” or generic “ignore these” indication is listed for the other fingers.

  All of the innovations described herein for gesture recognition, including but not limited to multilayer specification techniques, the use of relative orientation, data quantization, and tolerance for any level of partial or minimal specification are all manual. Beyond the gesture specification, it is generalized to the specification of gestures using other parts of the body and “manufactured” tools and objects.

  The programmatic technique (2) for “gesture registration” consists of a defined set of application programming interface calls that allow the programmer to determine which engine is responsible for the rest of the system being executed. It can be defined whether a gesture should be enabled.

These API routines are used during application setup and can create static interface definitions that are used throughout the lifetime of the running application. They can also be used during execution to allow interface characteristics to change on the fly. This real-time change of the interface
i) building a complex, conditional control state on the context;
ii) dynamically add hysteresis to the control environment, and iii) allow users to create applications that can change or extend the interface vocabulary of the execution system itself.

  Algorithm (3) for analyzing the flow of gestures compares the gestures specified as in (1) and registered as in (2) with incoming low-level gesture data. When a match is recognized for a registered gesture, event information representing the matched gesture is passed up the stack to the executing application.

In the design of this system, efficient real-time matching is desired, and the specified gesture is treated as a tree that can be processed as quickly as possible.
In addition, primitive comparison operators that are used internally to recognize specified gestures are also exposed for use by application programmers, resulting in further comparison (eg, flexibility in complex or complex gestures). State check) can occur even from within the application context.

The semantics of cognitive “locking” is the innovation of the system described herein. These semantics are suggested by the registration API (2) (and to a lesser extent, embedded within the specified vocabulary (1)). For registration API calls:
i) an “entry” status notifier and a “continue” status notifier;
ii) a gesture priority specifier.

  If a gesture is recognized, its “continuation” condition takes precedence over all “entry” conditions for the same or lower priority gestures. The distinction between entry and continuation state significantly increases the perceived system utility.

  The system described herein includes an algorithm for operating robustly in the face of real-world data errors and uncertainties. Data from low-level tracking systems may be incomplete (for a variety of reasons, including marker blockage in optical tracking, network loss or processing delays).

Missing data is marked by the analysis system and interpolated to a “last known” or “most likely” state, depending on the amount and context of the missing data.
If the data for a particular gesture component (eg, a particular joint orientation) is missing but the “last known” state of that particular component can be analyzed as physically possible, the system Uses this last known state in real-time matching.

  In contrast, if the “last known” state is analyzed as physically impossible, the system will return to the “best guess range” for that component and use this composite data in real-time matching. .

  The specification and analysis system described herein is carefully designed to support "handedness agnostic" for multiple hand gestures, so that either hand is allowed to meet the pose requirement. .

Matching virtual / display and physical space This system is such that the virtual space shown on one or more display devices (“screens”) matches the physical space where one or more operators of the system reside. Can be provided as an environment. One embodiment of such an environment is described herein. This current embodiment includes three projector-driven screens in a fixed position, driven by a single desktop computer, using the gesture vocabulary and interface system described herein. Be controlled. However, any number of screens can be supported by the described technology and they can be moved (rather than fixed) and the screens are driven simultaneously by many independent computers. Note that the entire system can be controlled by any input device or technique.

  The interface system described in this disclosure should have means for determining the size, orientation and position of the screen in physical space. Given this information, the system dynamically maps, as projections, the physical space in which these screens are located (and the presence of the system operator) to the virtual space of a computer application running on the system. be able to. As part of this automatic mapping, the system also converts two spatial measures, angles, depths, dimensions, and other spatial characteristics in various ways, depending on the needs of the application hosted by the system.

This continuous conversion between physical and virtual space makes it difficult to achieve on existing application platforms, or the consistency of multiple interface technologies that must be implemented piecewise for each application running on existing platforms Permeable use. These techniques include (but are not limited to):
1) The use of literal pointing as a penetrating and natural interface technology, using the hand in a gesture interface environment or using a physical pointing tool or device.
2) Automatic compensation for screen movement or repositioning.
3) Graphics rendering depending on operator position, simulating parallax shift, for example to improve depth perception.
4) Inclusion of physical objects in the screen display taking into account the position, orientation, state, etc. of the real environment. For example, an operator standing in front of a large translucent screen may see both application graphics and a representation of the true position of the scale model behind the screen (and maybe move or change orientation). )

  Note that literal pointing is different from abstract pointing used in mouse-based windowing interfaces and most other contemporary systems. In those systems, the operator must learn to manage the conversion between the virtual pointer and the physical pointing device and must recognize and map between the two.

  In contrast, in the system described in this disclosure, from an application or user perspective, there is no difference between virtual space and physical space (except that virtual space is more suitable for mathematical manipulation), and therefore the operator No recognition conversion is required.

The closest analog of literal pointing provided by the embodiments described herein is a touch-sensitive screen (eg, as found on many ATM machines). Touch-sensitive screens provide a one-to-one mapping between a two-dimensional display space on the screen and a two-dimensional input space on the screen surface. In a similar manner, the system described herein is flexible (although not necessarily one-to-one) between the virtual space displayed on one or more screens and the physical space in which the operator resides. Provide a simple mapping. While this analog is useful, it is worth understanding that it is important to extend this “mapping approach” to three dimensions, arbitrarily large architecture environments, and multiple screens.

  In addition to the components described here, this system is also capable of continuous system level mapping (possibly by rotation, transformation, scaling or other geometric transformations) between the physical space of the environment and the display space on each screen. It is also possible to implement an algorithm that implements

A rendering stack that takes computational objects and mappings and outputs a graphical representation of virtual space.
An input event processing stack that takes event data (both gesture data and pointing data from the system and mouse input in the current embodiment) from the control system and maps the spatial data from the input event to coordinates in virtual space. The converted event is then passed to the executing application.

An “adhesive layer” that allows the system to host applications that run across multiple computers on a local area network.
Thus, a gesture-based control system has been described.

It is a figure which shows one Embodiment of the system of this invention. It is a figure which shows one Embodiment of the marking tag of this invention. It is a figure which shows the pose of the gesture vocabulary of one Embodiment of this invention. It is a figure which shows the direction of the gesture vocabulary of one Embodiment of this invention. It is a figure which shows the combination of both hands of the gesture vocabulary of one Embodiment of this invention. It is a figure which shows mixing of the direction of the gesture vocabulary of one Embodiment of this invention. It is a flowchart which shows operation of one Embodiment of the system of this invention. It is a figure which shows the example of a command of one Embodiment of this invention.

Claims (45)

Automatically detecting body gestures from gesture data received via an optical detector, wherein the gesture data is absolute for the instantaneous state of the body in terms of time and space. Position data, receiving three spatial positions of the markers from the body tag, recovering the three spatial positions and orientations for each of the markers, and using the three spatial positions and orientations of the markers without using reference data Identifying the gesture using only, and
The gesture, a step of converting the gesture signal, the method comprising the step of converting converts the information of the gesture to the gesture notation, wherein the gesture notation represents a gesture vocabulary, the oxygenate scan Cha vocabulary, the Represent each gesture using a character string that describes the state of at least one kinematic coupling for the body, and each character of the character string is assigned to each of a plurality of positions in the at least one kinematic coupling. The gesture signal includes transmission of the gesture vocabulary; and
Controlling a component coupled to the computer in response to the gesture signal;
Including methods.   The method of claim 1, wherein the detecting step includes detecting the position of the body.   The method of claim 1, wherein the detecting step includes detecting the orientation of the body.   The method of claim 1, wherein the detecting step includes detecting the movement of the body.   The method of claim 1, wherein the detecting step includes identifying the gesture, wherein the identifying includes identifying a pose and orientation for the body part.   The method of claim 1, wherein the detecting step includes dynamically detecting a position of at least one tag. The method of claim 6 , wherein the detecting step includes detecting a position of a tag set coupled to the body part. Each tag in the tag set comprises a pattern, each pattern in each tag of the tag set is different from any pattern in any remaining tag of the plurality of the tags, as claimed in claim 7 Method. Each tag includes a first pattern and a second pattern, the first pattern is a common to all tags in the tag set, the second pattern, the tag sets at least two The method of claim 8 , wherein the method differs between tags. The method of claim 7 , wherein the tag set forms a plurality of patterns in the body. Wherein the step of detecting, using the set of tags coupled to each of a plurality of appendages of the body includes detecting a position of the appendages The method of claim 6. The first tag set is coupled to a second appendage, tag set the first comprises a first plurality of tags, each tag, the tag in the first plurality of tags The method of claim 11 , comprising: a first pattern common to each tag; and a second pattern that is unique to each tag in the first plurality of tags. The second tag set is coupled to a second appendage, tag set of the second comprises a second plurality of tags, each tag, the tag in the second plurality of tags The method of claim 12 , comprising: a third pattern common to each tag; and a fourth pattern unique to each tag in the second plurality of tags. The method of claim 6 , wherein the at least one tag comprises a tag set, the tag set comprising at least one tag type selected from the group consisting of active tags and passive tags.   The method of claim 1, wherein the detecting step comprises dynamically detecting and positioning a marker in the body. The method of claim 15 , wherein the detecting step includes detecting a position of a marker set coupled to the body part. The method of claim 16 , wherein the marker set forms a plurality of patterns in the body. 16. The method of claim 15 , wherein the detecting step includes assigning the position of each marker to a subset of markers that form a tag.   The method of claim 1, wherein the gesture vocabulary represents an instantaneous pose state of the kinematic coupling of the body in text form.   The method of claim 1, wherein the gesture vocabulary expresses the kinematic coupling direction of the body in text form.   The method of claim 1, wherein the gesture vocabulary represents a combination of body kinematic coupling orientations in text form. The kinematic coupling is at least one first appendage of the body The method of claim 1. Wherein each position of a character string includes the step of assigning a second appendage, appendages the second is coupled to the first appendage The method of claim 22. Assigning a character to a plurality of respective positions of the second appendage in a plurality of characters, the method according to claim 23. 25. The method of claim 24 , wherein the plurality of positions are defined with respect to a coordinate origin. Determining a coordinate origin using a position selected from a group consisting of an absolute position and orientation in space, the fixed position and orientation relative to the body being independent of the overall position and orientation of the body, 26. The method of claim 25 , wherein the method responds interactively to body movements. The characters in a plurality of characters, including a plurality of steps to be assigned to each direction of the first appendage The method of claim 24. Step of controlling the component, by mapping the first gesture appendages in three-dimensional space object, at the same time, comprising controlling said three-dimensional space object in six degrees of freedom, according to claim 22 the method of. 23. The method of claim 22 , wherein controlling the component comprises controlling a three-dimensional spatial object through three transformation degrees of freedom and three rotational degrees of freedom. 30. The method of claim 29 , comprising controlling movement of the three-dimensional spatial object by mapping a plurality of gestures of the first appendage to a plurality of object transformations of the three-dimensional spatial object. 32. The method of claim 30 , wherein the mapping includes direct mapping between the plurality of gestures and the plurality of object transformations. 32. The method of claim 30 , wherein the mapping includes an indirect mapping between the plurality of gestures and the plurality of object transformations. 32. The method of claim 30 , wherein the mapping includes associating a position offset of the plurality of gestures with a position offset of the object transformation in the three-dimensional space object. That said mapping, said first position offset appendages includes associating the conversion speed of the object conversion in the three-dimensional spatial objects The method of claim 30. The detecting step includes detecting when an estimated position of the three-dimensional space object intersects a virtual space, the virtual space comprising a space drawn on a display device coupled to the computer. Item 30. The method according to Item 29 . 36. The method of claim 35 , wherein controlling the component comprises controlling the virtual object in the virtual space when the estimated location intersects with the virtual object. 38. The method of claim 36 , wherein controlling the component comprises controlling the position of the virtual object in the virtual space in response to the estimated position in the virtual space. 38. The method of claim 36 , wherein controlling the component comprises controlling a posture of the virtual object in the virtual space in response to the gesture.   The method of claim 1, comprising identifying the gesture at a plurality of levels. 40. The method of claim 39 , wherein the plurality of levels comprises a first level comprising a pose of a first appendage of the body. The plurality of levels includes a second level comprising a combination of poses, the combination of poses comprising a first pose of a first appendage and a second pose of a second appendage in the body. 40. The method of claim 39 , comprising: The plurality of levels includes a third level comprising a combination of pose and position, wherein the combination of pose and position comprises a third pose of at least one appendage of the body; 40. The method of claim 39 , comprising a fourth pose of at least one appendage of the body. 40. The method of claim 39 , wherein the plurality of levels comprises a fourth level comprising at least one series of gestures. 40. The method of claim 39 , wherein the plurality of levels comprises a fifth level comprising grapheme gestures, the grapheme gestures comprising tracking free space shapes. 40. The method of claim 39 , comprising: generating a registration gesture by registering a gesture as associated with at least one application; and determining a priority of the registration gesture .