KR20130140772A - Method circuit and system for human to machine interfacing by hand gestures - Google Patents

Abstract

The present invention relates to systems, methods, and circuits for interfacing humans and machines using one or more 3D hand skeletal models, which include image acquisition circuitry for acquiring 2D images of the hands, and identifying hand features in the acquired images. A hand feature identification module and an image skeleton forming module for generating a 2D skeleton dataset of the hand using the identified features. The 3D skeletal shaping module is configured to generate the 3D hand skeletal dataset by determining the configuration of some or all of the one or more 3D hand skeletal models substantially corresponding to the 2D skeletal dataset from which the 2D projection was generated.

Description

System and Method Circuit for Interfacing Human-Machine by Hand Movement {METHOD CIRCUIT AND SYSTEM FOR HUMAN TO MACHINE INTERFACING BY HAND GESTURES}

The present invention relates generally to the field of human-machine interfacing and to the field of non-contact interfacing of human-machine. More specifically, the present invention relates to a machine for commanding using hand gestures performed at the front of the camera.

Conventional human-machine interfacing utilizes the use of several interfacing core technologies. First, and most often, physical interfacing units such as buttons, knobs, switches, etc., which are pressed, held, turned, moved, or pressed in order to make the corresponding response of the interface machine. Otherwise, the interfacing is designed to be handled differently by humans.

The second basic technique is to use a touch screen, in which the physical interfacing unit is replaced with the graphical image displayed on the screen, and the user touches or slides the interface with the finger / organ on the area of the graphical image, Give the interface machine the command he wants. As the graphical images can be varied, flexible interfaces corresponding to various types of uses or applications can be used to optimize the user's experience and interfacing efficiency.

Other methods include techniques such as human speech recognition to identify natural language commands provided by a user; And utilizing a device held or glued to the user, where the movement of the device is tracked and corresponds to machine instructions.

There is still a need for non-contact interfacing methods and systems in the field of human and machine interfacing, where such methods and systems provide for the optics of one or more of the natural human body or body part (s) / organs (s) (eg, human hands). An image may be translated into machine instructions corresponding to a particular movement and / or movement performed by the human parts / organs (s).

The present invention is a system and method circuit for the interfacing of a human and a machine by hand motion, which enables non-contact interfacing of a human and a machine.

According to some embodiments of the invention, interfacing with the machine may be accomplished by performing a hand gesture at the front of the camera.

According to some embodiments of the invention, the camera may be coupled to image processing logic configured to identify hand gestures. In some embodiments of the invention, the image processing logic may extract elements such as wrinkles, phalanges and fingertips from the hand features and / or hand images.

According to some embodiments of the invention, the image processing logic may create a 2D image skeleton based on hand features and / or identified elements. According to some embodiments of the invention, the image processing logic may produce a 3D skeleton, and the 2D projection of the 3D skeleton may be substantially similar to the 2D image skeleton and satisfy constraints. According to some embodiments of the invention, the constraints may be defined as a true hand pose.

According to some embodiments of the invention, the 3D skeleton may represent the pose of the interfacing hand and / or the position and / or distance from the camera. According to some embodiments of the invention, the image processing logic may initiate a system event depending on the pose of the hand gesture (represented by the 3D skeleton) and / or the position and / or distance from the camera. According to some embodiments of the invention, the image processing logic may translate the 3D skeletal motion into a command to be sent to a machine that the hands interface with.

According to some embodiments of the invention, a system for interfacing a human and a machine may use one or more 3D hand skeletal models, the system comprising an image acquisition circuit for obtaining a 2D image of the hand, the characteristics of the hand in the acquired image. A hand feature identification module for identifying an image, an image skeleton forming module for generating a 2D skeleton data set of a hand using the identified feature; And / or a 3D skeletal shaping module that generates a 3D hand skeletal data set by determining a configuration of some or all of the one or more 3D hand skeletal models, substantially corresponding to the 2D skeletal data set for which the 2D projection is generated. .

According to some embodiments of the invention, the image skeletal shaping module may be further configured to use anatomical relationships and constraints between elements of the human hand to generate a 2D skeletal data set.

According to some embodiments of the invention, the image skeleton forming module may be further configured to determine quality attributes for one or more 2D skeleton data set elements.

According to some embodiments of the invention, the human and machine interfacing system may comprise a skeletal database forming module for storing one or more previous 3D hand skeletal data sets, wherein the 3D skeletal forming apparatus is transferred by a database forming module. And generate a 3D hand skeleton data set based at least in part on one or more 3D hand skeleton data sets stored in.

According to some embodiments of the invention, a human-machine interfacing system may comprise a prediction module for predicting a 3D hand skeleton data set based on two or more previous 3D hand skeleton data sets to which the laws of motion dynamics physics are applied. And the 3D skeleton forming apparatus is further configured to generate the 3D hand skeleton data set based at least in part on the 3D skeleton data set as predicted by the prediction module.

According to some embodiments of the invention, a method for interfacing a human and a machine using one or more 3D hand skeletal models includes obtaining a 2D image of a hand, identifying one or more hand features in the acquired image, and identifying the identified Generating a 2D skeleton data set of the hand using the feature; And / or generating the 3D hand skeleton data set by determining a configuration of some or all of the one or more 3D hand skeleton models, substantially corresponding to the 2D skeleton data set for which the 2D projection was generated.

According to some embodiments of the invention, the method of interfacing humans and machines may further comprise generating a 2D skeletal data set based at least in part on anatomical relationships and constraints between elements of the human hand.

According to some embodiments of the invention, a method of interfacing a human and a machine may comprise generating a 2D skeletal data set based at least in part on the determination of a quality attribute for one or more 2D skeletal data set elements.

According to some embodiments of the invention, a method of interfacing a human and a machine comprises storing one or more 3D hand skeleton datasets and generating a 3D hand skeleton dataset based at least in part on previously stored 3D hand skeleton datasets. And generating a 3D hand skeleton data set based at least on the basis.

According to some embodiments of the present invention, a method of interfacing a human and a machine predicts a 3D hand skeleton data set by applying a motion dynamic physics law to two or more previously generated 3D hand skeleton data sets and predicts the predicted 3D hand skeleton data. Generating a 3D hand skeleton data set based at least in part on the set, comprising generating a 3D hand skeleton data set based at least in part.

According to some embodiments of the invention, circuitry for generating a 3D hand skeleton data set using one or more 3D hand skeleton models uses a hand feature identification module, identified features for identifying hand features in a 2D image of the hand. The 3D hand skeleton data set by determining an image skeleton forming module for generating a 2D skeleton data set of the hand, and / or configuration of one or more 3D hand skeletons substantially corresponding to the 2D skeleton data set for which the 2D projection was generated. It may include a 3D skeleton forming device to generate.

According to some embodiments of the invention, the image skeletal shaping module of the circuit for generating the 3D hand skeletal data set may generate the 2D skeletal data set using anatomical relationships and constraints between elements of the human hand.

According to some embodiments of the invention, the image skeleton forming module may determine quality attributes for one or more 2D skeleton data set elements.

According to some embodiments of the present invention, a circuit for generating a 3D hand skeleton data set stores one or more previous 3D hand skeleton data sets and stores the one or more 3D hand skeleton data sets previously stored by a database forming module. And a skeleton database formation module for generating a 3D hand skeleton data set based at least in part.

According to some embodiments of the invention, a circuit for generating a 3D hand skeleton data set is a prediction for predicting a 3D hand skeleton data set based on two or more previous 3D hand skeleton data sets to which the motion dynamic physics law is applied. And a module, wherein the 3D skeleton forming apparatus is configured to generate the 3D hand skeleton data set based at least in part on the 3D hand skeleton data set as predicted by the prediction module.

Materials considered to be the present invention are specifically presented and are explicitly claimed at the end of the present specification. However, all the methods and configurations of the present invention, together with the objects, features and effects thereof, can be best understood by reading the following detailed description in conjunction with the accompanying drawings.
1 is an illustration of a hand photographed by a camera of a communication / computer device, in accordance with some embodiments of the present invention, wherein the communication / computer device identifies an action made by the hand and optionally identified hand posture. And / or processing logic configured to generate an instruction or event based on the action.
2A-2I are photographic illustrations of a hand taking exemplary actions that may be recognized by a camera and correspond to commands and / or events of a computer device, in accordance with some embodiments of the present invention.
FIG. 3 shows exemplary features of the hand shown in FIGS. 2A-2I, in accordance with some embodiments of the present invention, which features may be extracted from an image of the hand by image processing logic included in a communication / computer device. have.
4 shows an exemplary structure of the hand skeleton, in accordance with some embodiments of the present invention.
5A-5C show an exemplary method for extracting the position of hand elements from visible and invisible hand features, in accordance with some embodiments of the present invention.
6A-6D show some examples of 2D image skeletons extracted from an interfacing hand, in accordance with some embodiments of the present invention.
7A-7C show substantially similar projections of some exemplary 2D image skeletons and 3D skeletons of different poses, in accordance with some embodiments of the present invention.
8A-8B show two projection views of a 3D skeleton that may be substantially similar to two 2D image skeletons of a hand located at two different distances from a capture camera, in accordance with some embodiments of the present invention.
9A and 9B are examples showing methods and mechanisms for enabling 3D skeletal bone length and quality to be updated in accordance with some embodiments of the present invention.
10 is a functional block diagram of an exemplary processing unit that includes a processing module configured to identify a motion of a hand based on one or more acquired images of the hand, in accordance with some embodiments of the present invention.
11 is a flow chart including the steps of an exemplary method for operating an image processing unit, in accordance with some embodiments of the present invention.
For simplicity and clarity of illustration, it will be appreciated that the elements shown in the drawings are not necessarily drawn to scale. For example, the dimensions of some elements may be exaggerated relative to others for clarity. Further, where appropriate, reference numerals may be repeated among the figures for corresponding or similar elements.

In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be understood by those skilled in the art that the present invention may be practiced without these specific details. As another example, well known methods, procedures, components and circuits have not been described in detail so as not to obscure the present invention.

Unless expressly indicated otherwise, discussions in the specification that use terms such as “processed”, “computer-used”, “computed”, “determined”, etc., throughout the specification, are indefinite from the discussion that follows. Or in the operation and / or processing of the computer system, or in the manipulation and / or conversion of data represented by a physical quantity in a record of the computer system and / or in a memory, recording device or other such storage, transfer or display device of the computer system. It relates to a similar electronic computer device, such as an electronic device, that stores in other data similarly represented in physical quantities.

Embodiments of the invention may include apparatus for performing the operations herein. The device may be specially manufactured for a given purpose, or the device may include a general purpose computer reconfigured or selectively activated by a computer program stored in the computer. Such computer programs may be stored in a computer readable storage medium, which may be a floppy disk, an optical disk, a CD-ROM, a magnetic optical disk, a read-only disk (ROM), a random extraction memory (RAM), or electrically Programmable read memory (EPROM), electrically erasable and programmable read-only memory (EEPROM), magnetic or optical cards, or any other type suitable for storing electronic instructions and capable of being coupled to a computer system bus Any type of disc that includes a medium, but is not limited to such.

The processes and displays presented herein are not inherently related to any particular computer or other device. Various general purpose systems may be used with the programs in accordance with the teachings herein, or may prove convenient to fabricate more specialized devices to perform certain methods. Certain structures for various such systems appear from the description below. In addition, embodiments of the present invention are not described with reference to any particular programming language. It will be appreciated that various programming languages may be used to practice the teachings of the invention described herein.

The present invention is a system and method circuit for interfacing a human and a machine via hand gestures and / or body contactless. The concepts and methods described herein are described with respect to hand gestures in order to keep the content simple, and the same concepts and methods can be applied to sensing whole body motion and / or motion of other body parts.

According to some embodiments of the invention, the condition of the hand or the human body can be determined. According to some embodiments of the invention, the condition may include hand / body distance from the camera, its spatial orientation, and / or its pose. According to some embodiments of the present invention, image processing logic may be provided for extracting a 3D skeleton that may represent the posture and / or orientation and / or distance of the hand, body or body part from the camera. According to some embodiments of the present invention, a method is provided for extracting a 3D skeleton that can represent a posture and / or orientation and / or orientation of a hand, body or body part from a camera. According to some embodiments of the invention, a method is provided for identifying a skeleton of a human body or part of a human body and for identifying a 3D skeleton of a hand within the human body.

According to some embodiments of the invention, a camera is provided, in front of which there may be an object such as a human hand capable of interfacing with a machine. According to some embodiments of the invention, a hand in front of the camera may interface with the machine by performing a hand gesture. According to some embodiments of the present invention, the camera may capture one or a series of images of the interfacing hands. According to some embodiments of the invention, the camera is configured to estimate the spatial location of one or more features, elements, phalanges and / or joints within an image of an interfaced object, such as a user's hand intentionally interfacing with the machine. Can be connected to.

According to some embodiments of the invention, the image processing logic may create a 3D skeleton that may be a skeleton representative of the interfacing hand. According to some embodiments of the invention, the dimensional ratios of the elements of the 3D skeleton may be substantially similar to the dimensional ratios of the elements of the interfacing hand. According to some embodiments of the invention, the dimensions or features of the elements of the 3D skeleton may be proportional to the length of one unit. According to some embodiments of the invention, the image processing logic may create a 2D image skeleton according to the features and / or elements identified in the image of the interfacing hand.

According to some embodiments of the invention, the image processing logic may make an optional 3D position of the 3D skeleton and apply constraints to the 3D skeleton to substantially match the 2D projection of the 3D skeleton to the 2D image skeleton.

In accordance with some embodiments of the present invention, motion dynamic physics may also be applied to two or more 3D skeletons to produce predictions for the next frame of the 3D position of the 3D skeleton.

1 is an exemplary schematic illustration of a system according to the invention. According to this example, there may be an interfacing hand 1 which makes a hand gesture in front of the camera 2 provided in the cellular phone 3. 2A-2F show exemplary images of interfacing hands in different postures.

According to some embodiments of the invention, the image processing logic may detect features of the interfacing hand such as fingers, fingertips, fingernails, finger prints, palm prints, finger joints, as well as other features.

3 shows example features of the example interfacing hands shown in FIGS. 2A-2F. In this figure, 4, 5, 6, 7 and 8 can show thumb, index finger, middle finger, ring finger and little finger respectively. In 9, 10, 11, 12 and 13, the fingertips of the thumb, index finger, middle finger, ring finger and little finger can be shown, respectively. 14, 15, 16, 17 and 18 can show the fingerprint of a finger. 19 and 20 can show fingerprints where the index finger and pinkie are connected to the palm. 21 and 22 may show a tangent to the phalanges of the index finger.

According to some embodiments of the invention, the image processing logic may comprise a memory. According to some embodiments of the invention, the image processing logic may store the general structure of the hand skeleton in a memory. According to some embodiments of the invention, the image processing logic may store the general structure of the human body skeleton in a memory.

According to some embodiments of the invention, the general structure of a 3D hand skeleton stored in memory can be described as a vector, where some primitives of the vector describe a parameter (eg, length) of the hand skeleton, and the vector Some of the primitives describe a parameter of the joint that represents a dynamic relationship (eg, angle) between the bone primitives.

4 shows an example of some structures of the hand skeleton in accordance with some embodiments of the present invention. According to this example, the hand skeleton is made of joints / fingertips A, B C ... U connected by b, c, d, e ... u.

The youngest child's hand is the smallest, the grown person's hand is the largest, and the hand that interfaces with the machine has the camera in such a way that the image of the hand becomes larger as the hand approaches the camera, and the image of the hand becomes smaller as the hand moves away from the camera. Since they may be at different distances from, the dimensions of the human hand are within any limits. Thus, the image of the interfacing hand may be of different sizes depending on the age of the person and the distance of the hand from the camera, but the dimensional ratio of the different features and / or elements of a given interfacing hand is constant regardless of the distance of the hand from the camera or its orientation. Is maintained. Therefore, the first feature of the hand and / or the given length of the element, all other features of the hand and / or the length of the element can be defined with respect to the first feature and / or element. For example, in Figure 4, the bone b may be defined as the first element, the length of the bone c may be k1 * length (b), the length of the bone d may be k2 * length (b) and the like.

According to some embodiments of the present invention, the interfacing hand may be sampled by a general structural 3D skeleton stored in memory. According to some embodiments of the invention, the length of a particular designated bone representing a hand object of the 3D skeleton (eg, the length of the bone of the index finger from the palm joint to the fingertip, the length of a particular bone, or the sum of the lengths of several bones) May be defined as one unit length. According to some embodiments of the invention, the length of the bone / hand-object of the 3D skeleton may be defined in proportion to the designated bone / hand-object.

According to some embodiments of the invention, the image processing logic may be estimated from the image of the hand, the relative length of the bone. According to some embodiments of the invention, the image processing logic may estimate bone length and / or hand distance from the camera. Because of various factors in the image, such as hand orientation and posture, hidden hand elements, and the like, the certain level of certainty of length may be less than the level of certainty of the length of elements that appear clearly in the image.

According to some embodiments of the invention, the 3D skeletal vector may have attributes associated with each bone and are stored in memory. According to some embodiments of the invention, the bone attribute may include the relative length of the bone. According to some embodiments of the invention, the bone attribute may comprise length quality. According to some embodiments of the invention, the bone attribute may include quality toughness / assurance. According to some embodiments of the present invention, bone attributes may include angular dependence on other bones. According to some embodiments of the present invention, the angle dependency may be hierarchical.

According to some embodiments of the invention, the image processing logic may store in the memory an image skeleton that may be a two-dimensional (2D) general structure of the hand skeleton. According to some embodiments of the invention, a 2D image skeleton stored in memory may be described as a vector consisting of primitives representing the hand bones and joints. According to some embodiments of the invention, the vector joint primitive may be described as a point on a plane, each point having two-dimensional coordinates (x, y). According to some embodiments of the invention, the vector primitives may include attributes representing primitive (joint) connections (bones) relative to other vector primitives (joints). According to some embodiments of the invention, an image 2D skeletal vector may have a set of 2D (x, y) coordinates associated with each primitive and stored in memory. According to some embodiments of the invention, primitive (x, y) coordinates may be extracted from the hand image.

According to some embodiments of the invention, the image processing logic may estimate the two-dimensional position of the joint, fingertip, and phalanges of the hand in the captured image. According to some embodiments of the invention, the position of the joints, fingertips and phalanges can be estimated by analyzing the detected hand features. According to some embodiments of the invention, the position of the joint may be estimated according to the position of the wrinkles of the fingers. According to some embodiments of the invention, the position of the joint may be estimated according to the intersection position of two lines which are tangent to two parts of the finger. According to some embodiments of the invention, the location of the phalanges can be estimated by connecting a straight line with two adjacent joints. According to some embodiments of the invention, the location of the phalanges can be estimated by identifying the phalanges in the image. According to some embodiments of the invention, the location of the phalanges can be estimated by identifying a portion of the phalanges in the image and interpolating and / or extrapolating the remaining phalanges.

5A-5C show some examples of images of hands. According to the example of FIG. 5A, the position of the joint J1 may be estimated according to the wrinkle C1. Lines L0 and L1 may be tangent to the phalanges P1, and line L2 may be intermediate between lines L0 and L1 and may represent the center of the phalanges P1. Lines L3 and L4 can be tangent to the phalanges P2, and line L5 can be intermediate between lines L3 and L4 and can represent the center of the phalanges P2. Joint J2 can be estimated according to the intersection of lines L2 and L5. In FIG. 5C, the position of the joint J6 is hidden behind the index finger and is not shown in the image, but can be estimated according to the intersection of the lines L6 and L7. In FIG. 5B, the location of the phalanges P3 can be estimated by interpolating the portions P3.1 P3.2.

According to some embodiments of the present invention, the position of the invisible hand elements (bones and joints) can be estimated according to the hierarchical relationship between the visible hand elements and the invisible elements.

According to some embodiments of the invention, the image processing logic may assign the estimated two-dimensional positions of the joints and fingertips to the corresponding primitives of the 2D image skeletal vector. According to some embodiments of the present invention, primitive coordinates may include primitive position quality attributes because the 2D position of some joints may be estimated more accurately and the position of other joints may be estimated less accurately. 6A-6D are some examples of image skeletons extracted from an interfacing hand.

According to some embodiments of the invention, the image processing logic may position the 3D skeleton in a spatial position substantially similar to the 2D image skeleton of the 2D image skeleton of the 3D skeleton. 7A-7C all show exemplary 2D image skeleton A, and several different poses B, C, and D of the 3D skeleton. The two-dimensional projection E of the 3D skeletal pose (B, C, D) may be substantially similar to the 2D skeletal A.

The fingers of a human hand can bend all three joints in only one direction into the inside of the palm. The finger cannot be bent laterally at the two joints adjacent to the fingertip, and the lateral movement at the joint connecting the finger with the palm is limited. Also, the fingers cannot be bent backwards in all three joints. The finger cannot be rotated around any of its joints. Each phalanx can have a specific angle that can be bent in each spatial direction relative to the phalanx adjacent to the wrist to which it is connected. According to some embodiments of the present invention, the joint properties may include anatomical constraints of the joints. According to some embodiments of the invention, the anatomical constraints of the joints may include angles in each spatial direction, wherein bones connected on the joints of the palms may be bent relative to bones adjacent to the palms. According to some embodiments of the invention, the angle may depend on the length of the bone and the angle at another joint adjacent the palm.

According to some embodiments of the invention, the image processing logic may apply joint constraints to the joints of the 3D skeleton. According to some embodiments of the invention, there may be a 3D skeleton in 3D spatial pose and position that may substantially meet the constraints. The spatial location of the 3D skeleton with the 2D projected skeleton substantially matching the 2D image skeleton may determine the relative distance of the hand from the camera.

8A-8B show two different 2D projected images of the 3D skeleton shown in FIG. 4. The skeleton of FIG. 8A is a projected image of the skeleton of FIG. 4 adjacent to the camera, and the skeleton of FIG. 8B is a projected image of the skeleton of FIG. 4 away from the camera.

The camera can capture a series of images of the operating interfacing hands. Each image may be somewhat different from the previous picture depending on the speed at which the image is captured, the speed at which the hand moves, the distance of the hand from the camera, and the resolution of the camera. Based on each captured image, the image processing logic can estimate the spatial position, orientation, distance and pose of the 3D skeleton. Each 3D skeleton may be somewhat different from the previous picture depending on the speed at which the image is captured, the speed at which the hand moves, the distance of the hand from the camera, and the resolution of the camera.

Two or more previous 3D skeletons derived from two or more previous images can be used to calculate and predict the current 3D skeleton based on the laws of motion dynamics physics. According to some embodiments of the invention, the image processing logic may predict the current 3D skeleton based on two or more previously 3D skeletons using motion dynamic physics laws. According to some embodiments of the invention, the predicted 3D skeleton can be spatially positioned and positioned in a manner that can substantially meet the constraints.

According to some embodiments of the invention, the current 3D skeleton can be predicted. The predicted current 3D skeleton may be spatially rearranged to create a skeleton that meets constraints and the 2D projection of the skeleton may be substantially the same as the 2D image skeleton. According to some embodiments of the present invention, the image processing logic may use two or more previous 3D skeletons to predict the current 3D skeleton based on the laws of motion dynamic physics. According to some embodiments of the present invention, the predicted 3D skeleton can be repositioned to a minimum in a manner that satisfies the constraints to create a 3D skeleton whose 2D projection can be substantially the same as the 2D image skeleton.

The data quality of the elements of the 2D image skeleton extracted from the hand image can be determined. For example: Elements that are partially hidden behind other elements may have lower quality scores than elements that are completely visible to the camera. Poorly identified elements will have lower quality scores than clearly identified elements. Elements pointing towards the camera may have lower quality scores than elements facing the camera. In the vector describing the 3D skeleton each bone has attributes related to things such as length; And a quality score attribute that can determine the data quality of the bone's length value. The length of the bone of the 3D skeleton can be determined from the length of the corresponding skeleton in the 2D image skeleton. The bone length quality of the 3D skeleton can be determined from the accuracy of the length of the bone corresponding to the 2D image skeleton.

According to some embodiments of the invention, the interfacing hand may move or take action. The camera can capture images of interfacing hands in different directions and postures, which allows to improve the data quality of the length of the 3D skeletal bones over time, for example, if the same bone lengths are interfacing hands. Determined from several orientations of, the quality score of that goal can be increased. If there is a difference in bone length between the 2D projection of the 3D skeleton matching the constraint and the 2D skeleton (2D image skeleton) extracted from the image of the interfacing hand, and / or the 2D skeleton extracted from the image (2D image skeleton) If there are bones in the 3D skeleton that may have a lower quality compared to the bones of, then the quality and / or bone length of the 3D skeleton with lower quality is the quality value of the corresponding bone of the 2D image skeleton with the higher quality score and / or It can be updated according to length.

9A shows an example of 3D bone length quality improvement. In the example of FIG. 9A, the bone 75 may be a 3D skeletal bone with an uncertain length. The bone 75 may begin at joint 70 and end at an uncertain length, anywhere between joints 71 and 72. The bone 76 may be a bone corresponding to the bone 75 in the 2D image skeleton. The bone 76 can have an uncertain length, which can start at joint 62 and end anywhere between joint 77 and joint 78. The valleys 75 can be located in multiple spatial orientations, and their 2D projections can match the valleys of the corresponding 2D image skeleton. In FIG. 9A, it is seen that valley 75 can have orientations 63, 64, 65, 66, and 81. The two-dimensional projection of valley 75 in all of the above orientations may be within the boundary of valley 76. Since the two-dimensional projection can be short compared to the valleys 76, the orientations 82 and 63 of the valleys 75 may not be effective orientations. As can be seen in FIG. 9A, in all valid orientations of valley 75 (63, 64, 65, 66, 81), valley 75 is within the uncertainty window boundary of valley 76 between lines 67 and 68, There is a need to be longer than the distance from joint 83 to arc 69, and the lower portion of arc 69 may never be included within the uncertainty boundary of valley 76. Therefore, the lower boundary of uncertainty of bone 75 length can be improved and updated, and joint 71 can be converted to 79. In the orientations 63 and 64 of the valley 75, the total length of the valley is included within the uncertainty boundary, so that the joint 72 may not be improved and may be left as the joint 80. The result is that the level of uncertainty of bone 75 length can be reduced from the distance between joints 71 and 72 to a shorter distance between joints 79 and 80.

Another example is shown in FIG. 9B, where the bone 75 of the 3D skeleton may have an uncertain length starting at joint 70 and ending at an uncertain position between joints 71 and 72. The valley 76 may be a corresponding valley in the image skeleton with respect to valley 75. The valley 76 can have an uncertain length, which can begin at joint 62 and end at any point between joint 77 and joint 78. The valley 75 may have constraints that can limit the spatial orientation with respect to the rest of the skeleton (not shown in the figure). In the example shown in FIG. 9B, the valley 75 can be positioned in multiple orientations between orientation 65 and orientation 66. As can be seen in the figure, the portion of the valley between arc 69 and arc 74 can be included within the uncertainty boundary of valley 76 between lines 67 and 68. The portion of the bone between the joints 83 and arc 69, and the portion of the bone above arc 69, is outside the bone 76 uncertainty boundary. Therefore, the uncertainty length of bone 75 can be reduced, joint 71 can be converted to 79, and joint 72 can be converted to 80. The result is that the level of uncertainty of bone 75 length can be reduced from the length between joints 71 and 72 to the shorter length between joints 79 and 80.

As the hand taking action moves in front of the camera, the camera can acquire images of the hand in different orientations and postures. According to some embodiments of the present invention, image processing logic can be updated and improve the quality of bone length.

Since the bone length of a human hand is proportional to each other, the hand may not have a longer index finger, for example, than the middle finger, and there may be a constraint that the length of the 3D skeletal bone is within certain proportional limits between the parts. have. Thus, for example, if the bone of the 3D skeleton corresponding to the middle finger has a length attribute and a high quality score (low uncertainty), the bone of the 3D skeleton corresponding to the index finger has a length attribute and a low quality score (high uncertainty). If the bone corresponding to the index finger can be longer than the bone corresponding to the middle finger, the quality attribute of the bone corresponding to the index finger is improved and the uncertainty is the bone corresponding to the middle finger. It can be reduced to a level not longer.

According to some embodiments of the invention, the image processing logic may update and improve the quality attributes of the 3D skeletal bone based on one or more constraints defining the quality of another bone of the 3D skeleton and the relationship between the two bones. .

10 is an exemplary block diagram of a system in accordance with some embodiments of the present invention. Image acquisition circuitry (e.g. cameras) may obtain 2D images of 3D hands that interface with the machine. The captured image can be stored in image memory.

According to some embodiments of the invention, the image processing logic / circuit may include some or all of the following modules: control unit, hand feature / element identification module, image skeletal shaping module, 3D skeletal shaping module, skeletal database shaping Module, skeletal prediction module, processed image memory module.

The hand feature / element identification module may receive a hand image and identify an element or feature of the hand, such as a fingertip, wrinkle of a finger joint, finger edge, angle between finger phalanges, and the like. The image skeleton forming module may receive data of the hand element from the hand element identification module and generate a 2D image skeleton of the hand. The image skeletal forming device may interpolate and / or extrapolate visible elements and / or use anatomical relationships between hand elements to fill in missing elements. The image skeleton forming apparatus may also determine the quality and / or uncertainty of the image skeleton elements produced by the apparatus.

The 3D skeletal shaping module may receive skeletal bone dimensions and attributes from the image skeletal shaping module, may receive previous 3D bones stored in memory, and / or receive 3D skeletal predictions from the skeletal prediction module. And a 2D image skeleton from the image skeleton forming module. Based on one or more of these inputs, the 3D skeletal forming module can produce a plurality of 3D skeletons, from which a particular skeleton can be selected.

The skeletal database forming module may receive a matching degree of skeleton from the 3D skeletal forming module; The information may include, but is not limited to, skeletal bone dimensions and / or skeletal orientation. The skeletal database forming apparatus may receive information regarding skeletal bone dimensions, skeletal orientation, skeletal bone dimension quality and uncertainty from the image skeletal forming apparatus. The skeletal database forming apparatus may then create and / or update and / or update / improve / improve the quality attributes of the length and / or length of one or more bones of the 3D skeleton.

The skeletal prediction module may receive a previous 3D skeleton stored in memory and may predict the next 3D skeletal position and / or pose based on the laws of motion dynamic physics. The 3D skeletal formation module can use predicted skeletons to create new 3D skeletons.

The processed image memory module may store captured images, 3D skeletons, image skeletons, one or more previous 3D skeletons, predicted skeletons, projected skeletons, attributes of different skeletons, anatomical constraints, and any other information that needs to be stored. .

11 is an exemplary schematic flowchart illustrating a method of some embodiments in accordance with the present invention. In step 101, an image of the interfacing hand may be captured by an image acquisition circuit (eg, a camera). The camera can obtain captured images at specific speeds per second. The captured one or more images may be stored in memory for further processing. At step 103, features of the hand such as wrinkles on the fingertips, fingers and palms can be identified. In step 104, the 2D image skeleton of the interfacing hand may be produced based on the identified hand feature. For example, finger wrinkles can be used to estimate the position of the finger joints. In step 105, the 3D skeleton may be positioned in a manner of 2D projection, which may be substantially similar to the image skeleton produced in step 104. In step 106, each candidate skeleton fabricated in each step 105 may be checked for constraints such as the allowed angle between the finger phalanges. The skeleton that satisfies the constraint can be a selected skeleton that is in a substantially similar posture to the interfacing hand-matched skeleton. In step 107, one or more previous skeletons may be used to predict the currently matching skeleton based on motion dynamic physics laws. In this step, the previous 3D movement of the skeleton is analyzed to extrapolate the current skeleton position. The predicted skeleton can be used in step 105 to make a candidate skeleton. In step 108, any ambiguity, inconsistency, uncertainty may be resolved based on input information collected from different skeletons. At this stage, the quality of the bone length can be improved. In step 109, information relating to the position and / or movement of the matching skeleton may be analyzed to output a sharp signal to the machine with which the hand interfaces.

While certain features of the invention have been illustrated and described herein, many modifications, substitution changes, and equivalents may occur to those skilled in the art. Therefore, it is to be understood that the appended claims are intended to cover all such modifications and variations as fall within the true spirit of the invention.

Claims (15)

In a system for interfacing a human and a machine using one or more 3D hand skeleton models, the system comprises:
Image acquisition circuitry for obtaining a 2D image of a hand;
A hand feature identification module for identifying hand features in the collected image;
An image skeletal shaping module, configured to generate a 2D skeletal dataset of the hand using the identified features; And
A 3D skeletal shaping module configured to generate a 3D hand skeletal dataset by determining a configuration of some or all of the one or more 3D hand skeletal models substantially corresponding to the generated 2D skeletal dataset of the 3D hand skeletal model.
≪ / RTI > The method of claim 1,
The image skeletal shaping module is further configured to use anatomical relationships and constraints between elements of the human hand to generate a 2D skeletal dataset. 3. The method of claim 2,
And the image skeletal shaping module is further configured to determine quality attributes for one or more 2D skeletal dataset elements. The method of claim 1,
Further comprising a skeleton database formation module configured to store one or more previous 3D hand skeleton datasets;
And the 3D skeleton forming apparatus is further configured to generate a 3D hand skeleton dataset based at least in part on the one or more 3D hand skeleton datasets previously stored by the database forming module. The method of claim 1,
A prediction module configured to predict the 3D hand skeleton dataset based on two or more previous 3D hand skeleton datasets to which motion dynamics physics rules apply;
And the 3D skeleton forming apparatus is further configured to generate a 3D hand skeleton dataset based at least in part on the 3D hand skeleton dataset predicted by the prediction module. In a method of interfacing a human and a machine using one or more 3D hand skeleton models, the method comprises:
Obtaining a 2D image of a hand;
Identifying one or more hand features in the acquired image;
Generating a 2D skeletal dataset of the hand using the identified features;
Generating a 3D hand skeleton dataset by determining a configuration of some or all of the one or more 3D hand skeleton models substantially corresponding to the generated 2D skeleton dataset of the 3D hand skeleton model
Human and machine interfacing method comprising a. The method according to claim 6,
Generating the 2D skeletal dataset further comprises using anatomical relationships and constraints between the elements of the human hand to generate the 2D skeletal dataset. The method of claim 7, wherein
Generating a 2D skeletal dataset further comprises determining a quality attribute for one or more 2D skeletal dataset elements. The method according to claim 6,
Generating a 3D hand skeleton dataset further comprises storing one or more 3D hand skeleton datasets, and generating a 3D hand skeleton dataset based at least in part on a previously stored 3D hand skeleton dataset. Human and machine interfacing method. The method according to claim 6,
Generating the 3D hand skeleton dataset comprises predicting the 3D hand skeleton dataset by applying motion dynamic physics laws to two or more previously generated 3D hand skeleton datasets, and generating the predicted 3D hand skeleton dataset. Generating a 3D hand skeletal dataset based at least in part on the human and machine interfacing method. A circuit for generating a 3D hand skeleton dataset using at least one 3D hand skeleton model, the circuit comprising:
A hand feature identification module for identifying hand features in a 2D image of the hand;
An image skeletal shaping module, configured to generate a 2D skeletal dataset of the hand using the identified features; And
3D skeletal forming apparatus configured to determine a configuration of at least one 3D hand skeleton model substantially corresponding to the 2D skeletal dataset from which the 3D hand skeleton model was generated
Circuit comprising a. The method of claim 11,
The image skeletal shaping module is further configured to use anatomical relationships and constraints between elements of the human hand to generate a 2D skeletal dataset. 13. The method of claim 12,
And the image skeletal shaping module is further configured to determine quality attributes for one or more 2D skeletal dataset elements. The method of claim 11,
Further comprising a skeleton database formation module configured to store one or more previous 3D hand skeleton datasets;
And the 3D skeleton forming apparatus is further configured to generate a 3D hand skeleton dataset based at least in part on one or more 3D hand skeleton datasets previously stored by the skeleton database forming module. The method of claim 11,
A prediction module configured to predict the 3D hand bone dataset based on two or more previous 3D hand bone datasets to which the motion dynamic physics law is applied;
And the 3D skeleton forming apparatus is further configured to generate a 3D hand skeleton dataset based at least in part on the 3D hand skeleton dataset predicted by the prediction module.

Images