CN102448561A - Gesture coach - Google Patents

Abstract

A capture device may capture a user's motion and a display device may display a model that maps to the user's motion, including gestures that are applicable for control. A user may be unfamiliar with a system that maps the user's motions or not know what gestures are applicable for an executing application. A user may not understand or know how to perform gestures that are applicable for the executing application. User motion data and/or outputs of filters corresponding to gestures may be analyzed to determine those cases where assistance to the user on performing the gesture is appropriate.

Description

The posture coach

Background technology

Use control to allow other aspects of user's direct game role or application such as many computing applications such as computer game, multimedia application, office application.Usually use that for example controller, remote controller, keyboard, mouse or the like are imported such control.Unfortunately, these controls possibly be difficult to learn, and have caused the obstacle between user and these recreation and the application thus.In addition, these controls maybe be different with actual play action or other using actions that these control is used for.For example, make game role brandish game control that baseball claps maybe be with to brandish the actual motion that baseball claps not corresponding.

Summary of the invention

In some system, display device can show the model that is mapped to the user movement of being caught by system.For example, this model can be shown as the incarnation on the screen, and wherein the motion of this incarnation can be through coming controlled to the motion of user in physical space the Motion mapping of this incarnation in the application space.The user possibly be unfamiliar with the system that user's motion is shone upon.For example, the user possibly not know that what posture is applicable to the application of carrying out.In some cases, the user does not understand or does not know how to carry out the posture that is applicable to the application of carrying out.Hand-written or picture description in the handbook possibly be not enough to instruct the user how correctly to make posture.

The system and method that is used for postural training disclosed herein.When the user attempts to make posture, can analyze the user data of being caught and from the output of the corresponding posture filter of this user data of catching, attempting but fail to carry out posture to confirm the user, and to offer help to the user be suitable.This help can comprise that the instruction user carries out the correct method of this posture.For example; Output to filter comprises the level of confidence of carrying out corresponding posture; And when this level of confidence is lower than recognition threshold, can confirm that this help is in the level of confidence of correspondence for the instruction user or the mode that surpasses this recognition threshold is carried out posture and is fit to.

Content of the present invention is provided so that be presented in some notions that further describe in the following specific embodiment with reduced form.Content of the present invention is not intended to identify the key feature or the essential feature of the protection theme that requires, and is not intended to be used to limit the scope of the protection theme that requires yet.In addition, theme required for protection is not limited to solve the realization of any or all shortcoming of in arbitrary part of the present invention, mentioning.

Description of drawings

Further describe the system that is used for postural training, method and computer-readable medium with reference to accompanying drawing according to this specification:

Figure 1A and 1B show the example embodiment of Target Recognition, analysis and the tracking system of playing games about the user.

Fig. 2 shows the example embodiment that can in Target Recognition, analysis and tracking system, use and comprise the capture device of postural training.

Fig. 3 shows another example embodiment of the computing environment that wherein can realize the technology that is used for postural training disclosed herein.

Fig. 4 shows another example embodiment of the computing environment that wherein can realize the technology that is used for postural training disclosed herein.

Fig. 5 A shows the skeleton mapping from the user of depth image generation.

Fig. 5 B shows the further details of gesture recognizers architecture shown in Figure 2.

Fig. 6 A-6D illustrates another example embodiment that user is wherein playing Target Recognition, analysis and the tracking of boxing game.

Fig. 7 A-7D illustrates the example that the visual representation vision side by side with user's posture helps and shows.

Fig. 8 A illustrates the example that the vision on the visual representation of the posture that is superimposed upon the user helps and shows.

The example that Fig. 8 B illustrates the vision help of the demonstration that comprises posture shows.

Fig. 9 illustrates and is used to get into the example demonstration of exercise mode with the option of reception vision help.

Figure 10 illustrates through the network connection and carries out mutual long-distance user, and the vision of one of them user's motion helps to be provided for second user.

Figure 11 A describes to be used for the exemplary operations process of postural training.

Figure 11 B and 11C have described the exemplary architecture that is used for postural training with gesture recognizers engine and application integration.

Figure 12 A and 12B have described example filter output, can confirm that from it postural training is suitable.

The specific embodiment

Like what will describe among this paper, the user can be controlled at the application of carrying out such as on the computing environment such as game console, computer through carrying out one or more postures.Herein disclosed is and be used for the demonstrate system and method for motion of required posture to the user.For example, this computing environment can be provided for training the user how to make the vision help of the appropriate exercise that is applicable to the application of carrying out.

In order to generate target or the model of object of expression in the physical space, capture device can be caught the depth image of this physical space and scanned each target in this scene.Target can comprise human or other objects in this scene.In one embodiment, capture device can confirm that one or more targets in the scene are whether corresponding to such as people's class targets such as users.In order whether to confirm target in the scene, can carry out that film color is filled and the pattern of itself and manikin is made comparisons each target corresponding to people's class targets.Can scan the target that is identified as the mankind and generate the skeleton pattern that is associated with it.Can this skeleton pattern be offered computing environment then follows the tracks of this skeleton pattern and presents the incarnation that is associated with this skeleton pattern.This computing environment can with the Motion mapping of user in physical space to the display device such as visual representations such as incarnation.To carry out which control command in the application that this computing environment can be confirmed on computer environment, to carry out based on the user's who for example identifies and be mapped to skeleton pattern posture.Therefore, can explicit user help, as via the incarnation on the screen, and the user can control the motion of this incarnation and carry out the control to operating system or the application carried out, for example through in physical space, making each posture.

In some cases, being provided for instructing the user how correctly to make posture, to control that the vision of the application of carrying out helps be desirable.For example, the user possibly not know and is applicable to the corresponding motion of given pose of the application of carrying out or do not know how to carry out this motion.This system can detect the mistake in user's the posture, correctly makes posture thereby indicate this user to practise.

In the functional unit of describing in this specification some are marked as module more specifically to stress their realization independence.For example, module can be implemented as hardware circuit, and this hardware circuit comprises self-defined VLSI circuit or gate array, such as stock semiconductors such as logic chip, transistors, or other discrete assemblies.The also available programmable hardware device of module realizes, like field programmable gate array, programmable logic array, PLD etc.

Module also can use the software of being carried out by various types of processors to realize.The module of the executable code that is identified for example comprises the one or more physics or the logical block of computer instruction, and this computer instruction for example can be organized into target, process or function.Yet the executable code of the module that is identified needn't be physically located at a place, but can comprise the different instruction that is stored in diverse location, the regulation purpose of when these instruct gang logically, forming this module and realizing this module.

System, method and the assembly that provides vision to practise help described herein can be implemented in the multimedia console such as game console; Or can in any other computing equipment that needs provide vision to help, implement; As an example but unexpected restriction, these other computing equipments comprise DVB, STB, electronic game machine, personal computer (PC), portable phone, PDA(Personal Digital Assistant) and other portable equipments.

Figure 1A and 1B illustrate the example embodiment of the configuration of Target Recognition, analysis and the tracking system 10 of following user 18 to play boxing game.In an example embodiment, system 10 can discern, analyzes and/or follow the tracks of the people's class targets such as user 18.System 10 can collect and the relevant information of the posture of user in physical space.

System 10 can provide the vision of the required posture of demonstration to help to the user.Can trigger with multiple mode provides what vision helped.For example, this system can detect wrong in the user movement or with the deviation of desired movement.Detect the vision help that such mistake or deviation can trigger the required posture of demonstration.In another example, the application of carrying out can be provided for practising vision purpose, the suitable controlled motion of demonstration and help.Help can be taked various forms, like sense of touch, the sense of hearing and vision.In one embodiment, help to comprise that audio frequency helps, vision helps, change the tracking pattern of certain combination of help of flash of light, these forms of the fading of lining, display element, the display element of color, the display element of display element.

Shown in Figure 1A, Target Recognition, analysis and tracking system 10 can comprise computing environment 12.Computing environment 12 can be computer, games system or console or the like.According to an example embodiment, computing environment 12 can comprise nextport hardware component NextPort and/or component software, makes that computing environment 12 can be used for carrying out such as application such as games application, non-games application.

Shown in Figure 1A, Target Recognition, analysis and tracking system 10 also can comprise capture device 20.Capture device 20 can be a camera for example; This camera is used in visually to be kept watch on such as one or more users such as users 18; So that can catch, analyze and follow the tracks of the performed posture of one or more users with one or more controls or action in the execution application, as will be described in greater detail below.

According to an embodiment, Target Recognition, analysis and tracking system 10 can be connected to can be to the audio-visual equipment 16 that recreation or application vision and/or audio frequency are provided such as users such as users 18, like television set, monitor, HDTV (HDTV) etc.For example, computing environment 12 can comprise that these adapters can provide the audio visual signal that is associated with games application, non-games application etc. such as video adapters such as graphics cards and/or such as audio frequency adapters such as sound cards.Audio-visual equipment 16 can be exported the recreation that is associated with this audio visual signal or use vision and/or audio frequency from computing environment 12 receiving said audiovisual signals then to user 18.According to an embodiment, audio-visual equipment 16 can be via for example, and S-vision cable, coaxial cable, HDMI cable, DVI cable, VGA cable etc. are connected to computing environment 12.

Shown in Figure 1A and 1B, Target Recognition, analysis and tracking system 10 can be used for discerning, analyzing and/or follow the tracks of the people's class targets such as user 18 etc.For example, can use capture device 20 to follow the tracks of user 18, so that can mobile being interpreted as of the user 18 be can be used for influencing the control by the application of computer environment 12 execution.Thereby according to an embodiment, user's 18 removable his or her healths are controlled application.

Shown in Figure 1A and 1B, in an example embodiment, the application of on computing environment 12, carrying out can be the boxing game that user 18 possibly play.For example, computing environment 12 can use audio-visual equipment 16 to come to provide to user 18 sparring partner 38 visual representation.Computing environment 12 also can use audio-visual equipment 16 to provide the user 18 can be through his or the visual representation of his moves player's incarnation 40 of controlling.For example, shown in Figure 1B, user 18 can shake one's fists in physical space and make player's incarnation 40 in gamespace, shake one's fists.Therefore; According to an example embodiment; The computing environment 12 of Target Recognition, analysis and tracking system 10 and capture device 20 can be used for discerning and analysis user 18 goes out fist in physical space, can be interpreted as the game control to the player's incarnation 40 in the gamespace thereby make this go out fist.

Other of user 18 move also can be interpreted as other controls or action, such as swing fast up and down, dodge, sliding steps, lattice retaining, straight punch or brandish the controls such as fist of various different dynamics.In addition, some moves that can be interpreted as can be corresponding to the control of the action except that control player incarnation 40.For example, the player can use to move and finish, suspends or preserve recreation, select rank, check high score, exchange with friend etc.

As described in greater detail below, system 10 can help to user 18 be provided for the demonstrating vision of the posture that is applicable to the application of carrying out.In an example embodiment, it is the pre-recorded content of forms such as skeleton representation, ghost image or player's incarnation that vision helps.In another example embodiment, can present live content to the user.

In each example embodiment, can hold an object such as user's 18 class targets such as people such as grade.In these embodiment, thereby the hand-holdable object of the user of electronic game can use the motion of player and object to adjust and/or control the parameter of recreation.For example, can follow the tracks of and utilize the motion of the hand-held racket of player to control racket on the screen in the electron motion game.In another example embodiment, can follow the tracks of and utilize the motion of the hand-held object of player to control weapon on the screen in the electronics FTG.System 10 can be provided for the demonstrating vision of the posture relevant with respect to the motion in physical space and/or in the application space of object that this player holds with the user helps.

According to other example embodiment, Target Recognition, analysis and tracking system 10 also can be used for target is moved operating system and/or the application controls that is interpreted as outside the field of play.For example, in fact any controlled aspect of operating system and/or application can be by controlling such as moving of target such as users 18.And system 10 can be provided for demonstrating and help with the vision of any can the control aspect relevant posture of operating system and/or application.

Fig. 2 illustrates the example embodiment of the capture device 20 that can in Target Recognition, analysis and tracking system 10, use.According to an example embodiment, capture device 20 can be configured to via any suitable technique, comprises that for example flight time, structured light, stereo-picture wait to catch the video that has depth information that comprises depth image, and this depth information can comprise depth value.According to an embodiment, capture device 20 can be organized as the depth information that is calculated " Z layer ", or the layer vertical with the Z axle that extends along its sight line from degree of depth camera.

As shown in Figure 2, capture device 20 can comprise image camera assembly 22.According to an example embodiment, image camera assembly 22 can be that the degree of depth camera that can catch the depth image of scene maybe can be caught the RGB camera 28 from the color of this scene.Depth image can comprise two dimension (2-D) pixel region of the scene that is captured, and wherein each pixel in the 2-D pixel region can (being unit with centimetre, millimeter or the like for example) be represented the length from the object in the scene that is captured of camera.

As shown in Figure 2, according to an example embodiment, image camera assembly 22 can comprise the IR optical assembly 24 of the depth image that can be used for catching scene, three-dimensional (3-D) camera 26 and RGB camera 28.For example; In ToF analysis; The IR optical assembly 24 of capture device 20 can be transmitted into infrared light on the scene, then, can use the sensor (not shown); With for example three-dimensional camera 26 and/or RGB camera 28, detect one or more targets and the backscattered light of object surfaces from scene.In certain embodiments, can use pulsed infrared light, make and to measure the time difference between outgoing light pulse and the corresponding incident light pulse and to use it for target or the physical distance of the ad-hoc location on the object confirming from capture device 20 to scene.In addition, in other exemplary embodiments, can the phase place of outgoing light wave and the phase place of incident light wave be compared to confirm phase shift.Can use this phase in-migration to confirm the physical distance of the ad-hoc location from the capture device to the target or on the object then.

According to another example embodiment; Can use ToF analysis, through via for example comprising that intensity that the various technology of fast gate-type light pulse in being imaged on to analyze in time folded light beam is to confirm from capture device 20 to target or the physical distance of the ad-hoc location on the object indirectly.

In another example embodiment, but capture device 20 utilization structure light are caught depth information.In this was analyzed, patterning light (that is, being shown as the light of the known pattern of lattice for example or candy strip) can be projected on the scene via for example IR optical assembly 24.In the time of on one or more targets in falling scene or the object surfaces, as response, the pattern deformable.This distortion of pattern can be caught by for example 3-D camera 26 and/or RGB camera 28, can be analyzed to confirm the physical distance of the ad-hoc location from the capture device to the target or on the object then.

According to another embodiment, capture device 20 can comprise the camera that two or more physically separate, and these cameras can check from different perspectives that scene obtains to be resolved to generate the vision stereo data of depth information.

Capture device 20 also can comprise microphone 30.Microphone 30 can comprise the transducer or the sensor that can receive sound and convert thereof into the signal of telecommunication.According to an embodiment, microphone 30 can be used to reduce capture device 20 and the help between the computing environment 12 in Target Recognition, analysis and the tracking system 10.In addition, microphone 30 can be used for receiving also can customer-furnished audio signal, with control can by computing environment 12 carry out such as application such as games application, non-games application.

In example embodiment, capture device 20 can also comprise and can carry out the exercisable processor of communicating by letter 32 with image camera assembly 22.Processor 32 can comprise the standard processor, application specific processor, microprocessor of executable instruction etc., and these instructions can comprise the instruction that is used for receiving depth image, be used for instruction or any other the suitable instruction confirming whether suitable target can be included in the instruction of depth image, be used for suitable Target Transformation is become the skeleton representation or the model of this target.

Capture device 20 can also comprise memory assembly 34, the image that this memory assembly 34 can store the instruction that can be carried out by processor 32, captured by 3-D camera or RGB camera or the frame of image or any other appropriate information, image or the like.According to an example embodiment, memory assembly 34 can comprise random-access memory (ram), read-only storage (ROM), high-speed cache, flash memory, hard disk or any other suitable storage assembly.As shown in Figure 2, in one embodiment, memory assembly 34 can be the independent assembly that communicates with image capture assemblies 22 and processor 32.According to another embodiment, memory assembly 34 can be integrated in processor 32 and/or the image capture assemblies 22.

As shown in Figure 2, capture device 20 can communicate via communication link 36 and computing environment 12.Communication link 36 can be to comprise the wired connection of for example USB connection, live wire connection, Ethernet cable connection etc. and/or the wireless connections that connect etc. such as wireless 802.11b, 802.11g, 802.11a or 802.11n.According to an embodiment, computing environment 12 can provide clock to capture device 20 via communication link 36, can use this clock to determine when and catch for example scene.

In addition, the image that capture device 20 can provide depth information and captured by for example 3-D camera 26 and/or RGB camera 28 to computing environment 12 through communication link 36, and the skeleton pattern that can generate by capture device 20.Computing environment 12 can be used the image of this skeleton pattern, depth information and seizure for example to discern user's posture then and control such as application such as recreation or word processing programs as response.For example, as shown in the figure, in Fig. 2, computing environment 12 can comprise gesture recognition engine 190 and the gesture library 190 that comprises one or more posture filters 192.

Each filter 191 can comprise definition posture and the parameter of this posture or the information of metadata.Can the data of being caught with skeleton pattern and mobile form associated therewith by camera 26,28 and equipment 20 be compared with the posture filter in the gesture library 192, when carry out one or more postures to identify (as represented) user by skeleton pattern.Can comprise such as joint data input, as the formed angle of bone that intersects at joint, from the rgb color data of scene and user's contents such as rate of change in a certain respect about user's joint position such as filter 191 filters such as grade.

For example, comprise a hand from health behind to the throwing of preaxial motion can be implemented as comprise represent the user a hand from the health behind to the posture filter of preaxial mobile information, this moves being caught by degree of depth camera.View data from this scene also can be caught from the RGB camera.Like what mentioned, can be posture parameter is set.In posture is under the situation of throwing, and distance that parameter can be this hand threshold velocity that must reach, this hand must be advanced (absolute, or with respect to user's whole size) and identifier engine are put the letter grading to what this posture taken place.These parameters that are used for posture can change between each context between each is used, in single application or in the context an application in time.

Output from filter 191 can comprise such as the confidence level of just making given posture, make the speed of posture motion and the contents such as time of making the posture motion.These postures can be associated with the various controls of using.Therefore, computing environment 12 can be used gesture recognizers engine 190 to explain moving of skeleton pattern and move based on this and control application.

In an embodiment, the posture filter comprises algorithm, and this algorithm is accepted one or more data about the user as input, and returns the one or more outputs about the posture of correspondence.For example, " user's height " posture filter algorithm can shine upon user's skeleton as input, handles this data, and returns the output of the user's height that is gone out by this algorithm computation.

Computing environment 12 can comprise processor 195, and processor 195 can be handled depth image and confirm that what target is arranged in the scene, like user in the room 18 or object.This can for example realize through the group pixels with share similar distance value in the depth image together.This image also can be resolved the skeleton representation that produces the user, wherein identifies such as the characteristics such as tissue between joint and the joint.Have the skeleton mapping techniques, it uses degree of depth camera to catch the individual, and therefrom confirms a plurality of points on this user's skeleton, the joint of hand, wrist, elbow, knee, nose, ankle, shoulder, and pelvis and the crossing part of vertebra.Other technologies comprise that the manikin that image transitions is behaved is represented and the grid model that image transitions is behaved are represented.

In one embodiment, processing is carried out 20 of capture devices on one's body, and the raw image data of the degree of depth and color (wherein capture device 20 comprises the 3-D camera) value is sent to computing environment 12 via link 36.In another embodiment, handle and carry out, be sent to computing environment 12 through the view data of resolving then by the processor that is coupled to camera 402 32.In another embodiment, the view data of raw image data and warp parsing is sent to computing environment 12.Computing environment 12 can receive through the image of resolving, but it still can receive initial data and carries out active procedure or application.For example, if the image of scene sends to another user through computer network, then computing environment 12 can be sent the initial data that supplies another computing environment to handle.

Computing environment 12 can be used gesture library 192 to explain moving of skeleton pattern and move based on this and control application.Computing environment 12 can be carried out modeling and demonstration to user representing, for example adopts incarnation or the form of pointer on the display such as display device 194.Display device 194 can comprise computer monitor, television screen or any suitable display device.For example, the computer system of camera control can be caught user image data, and on television screen, shows the user aid of the posture that is mapped to the user.User aid can be shown as the incarnation on the screen, shown in Figure 1A and 1B.

Vision helps storehouse 193 can comprise with the vision of one group of posture and helps relevant information.Vision helps storehouse 193 information in order to the visual representation that shows guiding gesture data to be provided to display device 194.For example, display device can show skeleton representation, ghost image or player's incarnation of the posture of can demonstrating.As describe in more detail hereinafter, Fig. 7 A-7D illustrates user aid and comprises that the vision of guiding gesture data helps both, and the two illustrates side by side.Fig. 8 A illustrates the example that the vision on the visual representation of the posture that is superimposed upon the user helps and shows.The example that Fig. 8 B illustrates the vision help of the demonstration that comprises posture shows.Fig. 9 illustrates and is used to get into the example demonstration of exercise mode with the option of reception vision help.Figure 10 illustrates through the network connection and carries out mutual long-distance user, and one of them user provides live vision to help to second user.

Variety of event can trigger the demonstration of the vision help that is used for the posture demonstration.This system can detect the mistake in the user movement, and user movement maybe be not corresponding with the posture of having discerned, and the user can ask exercise to learn given pose etc.Can trigger application to the analysis of user movement provides vision to help to the user, or the option of checking that vision helps is provided, and for example is used for the proper exercise of the throwing gesture that this application discerned with instruction.As stated, to help can be the form of skeleton representation of example such as correct overhand throwing gesture to vision.These parameters and the error identifier that are used for this posture can change between each context between each is used, in single application or in the context an application in time.

Can provide the vision help information to be used for practising, to proofread and correct, to revise or to instruct the user correctly to move with the triggering special exercise, or correctly move in application successful in physical space.For example, the user possibly play boxing game.Computing environment 12 can identify the posture that is applicable to the application of carrying out, and as above colludes fist, and guides display device 194 to provide and comprises that vision how to carry out the indication of colluding fist on this helps.It can be the incarnation that the demonstration on the screen is colluded the proper exercise of fist on this that vision helps.

When the posture filter 191 of discerning, follow the tracks of or identify posture in the gesture library 192 should provide vision to help if also can identifying.For example, when posture is an overhand when throwing, camera 26,28 and equipment 20 with the skeleton pattern be the form data of catching and be associated with it move and can compare with the posture filter in the gesture library 192.The output of filter can be to identify user's (as represented by skeleton pattern) to carry out the overhand throwing gesture.Comprise a hand from health behind to the overhand of preaxial motion throw can be implemented as comprise represent the user a hand from the health behind to the posture filter of preaxial mobile information, this moves being caught by degree of depth camera.

Can indicate the failure in the user's posture for the difference between the set parameter of overhand throwing gesture in user's posture and the filter, and trigger and get into exercise mode and instruct the correct motion of user.Wrong or the variation that detects in the posture that is identified based on filter parameter can trigger the demonstration that vision helps.Can be overhand and throw each parameter that the mistake in the help identifying user posture is set.For example, this system is carrying out the application of baseball.The space body that the overhand throwing gesture can should move user's arm therein is as parameter.If posture filter 191 identifies user's arms and shifts out outside this space body in the overhand throwing that is identified, then this can indicate the mistake in the user movement but not be converted to different gestures.This application maybe be at expectation overhand throwing gesture, because this recreation is in the moment of user to batter's pitching.Fail user's gesture recognition can be triggered the demonstration that vision helps for expection overhand throwing gesture.The posture filter possibly can be the overhand throwing gesture with this gesture recognition no longer, because the parameter that overhand is thrown is no longer satisfied in this motion, but changes the filter parameter that satisfies different gestures into, only takes on throwing like hand.Non-expection between the different gestures changes can trigger the demonstration that vision helps.

Like what mentioned, can be posture parameter is set.For example, be under the situation of throwing in posture, distance that parameter can be this hand threshold velocity that must reach, this hand must be advanced (absolute, or with respect to user's whole size) and identifier engine are put the letter grading to what this posture taken place.Each parameter that can be the mistake in the indication user comment is provided with various threshold values and scope.For example, depend on user's level of skill, the size of the space body that user's arm should pass through can change.

Can change based on user's performance, the application of carrying out, context, level of skill etc. with each posture corresponding parameter.For example, the parameter that the overhand rugby of " new hand " level of skill is thrown can comprise that hand can pass through than the large space body, make this system that overhand is thrown with this posture and carry out relatedly also correspondingly handling it.Through changing the special parameter that is associated with a posture, this system can adapt to the player with less experience.

Some possible postures can be corresponding to this user's posture.For example, the user's posture that in physical space, measures can satisfy the criterion of some filters, and each filter comprises the parameter of possibility posture.The executory failure of the posture that the difference between the data of the posture that measures of expression and the filter parameter of possible posture can indication measurement arrives.

If the data of expression user's posture are not corresponding with any filter parameter of possibility posture, then the user possibly correctly not carry out this posture, because it is relevant with the application of carrying out.If user's posture is not registered as one of each possibility posture, is desirable for this user triggers practice sessions then.

If user's posture is not corresponding with any filter data, then this system can predict the intention of this user's posture.This system can be based on what posture at that time will be applicable to carrying out should be used for prediction expection posture.Between the data of the posture that this prediction can measure based on expression and the filter parameter relatively and sign have the posture of the gesture data that coupling the most closely measures.

The executory failure of the posture that the difference between the data of the posture that measures of expression and the filter parameter of possible posture can indication measurement arrives.This difference can be accepted level with threshold value and compare, and the measures of dispersion that wherein is lower than this threshold value can trigger the vision that has guiding gesture data and help.Threshold value is accepted level can be relevant with the confidence level grading.As stated, the output from filter can comprise such as the things such as confidence level of making given posture.The low confidence grading can be the indication that the user does not correctly make posture.Threshold value is accepted level and can be graded based on confidence level and be provided with.For example, if posture is that overhand is thrown by the gesture recognizers engine identification, and the confidence level grading is low, and then this system can trigger the demonstration of the vision help with guiding gesture data.Perhaps, this system possibly need the high confidence level grading so that exist the user attempting the high confidence of given pose.When the data of the posture that expression measures and the difference between the filter parameter were lower than threshold value and accept level, vision helped and can trigger.

It can be the value that is provided with for the certain filter parameter that threshold value is accepted level.Each parameter of representing the filter of a posture can have threshold value and accept level.It can be single threshold value or acceptable value scope that threshold value is accepted level.If the measurement of user's the posture that measures is not satisfied threshold level or is not dropped in the tolerance interval, show that then it is desirable that the vision that comprises guiding gesture data helps.For example, the overhand threshold value of throwing is accepted level (when it is applicable to the throwing ball sport in the baseball) and can the speed of being set as be equaled 25mph.Therefore, throw if user's posture is identified as overhand, then the speed parameter of the filter thrown of overhand can compare with the speed of user's the posture that measures.If user's the speed that measures does not satisfy 25mph, then this can trigger the demonstration that vision helps, and how correctly to do motion with the instruction user and reaches correct speed.

Depend on context, user, the system based on posture, user's historical data, the historical data of application, improved identity etc., threshold value is accepted level and can be set up, revise or change.Threshold level can be with preferred parameter or with the acceptable measures of dispersion that is associated with range of parameter values that maybe posture.Threshold value is accepted the value of level can be based on user's performance, the application of carrying out, context, level of skill etc.Threshold value is accepted level can be based on single filter parameter or a plurality of filter parameter of given pose.Similarly, this system threshold value that can adapt to a posture is accepted level.For example, threshold level can be modified in time, is for example changed by user, this system or application.

Filter parameter and threshold value are accepted triggering that level can be configured such that vision helps can be not excessively or according to user preference.For example, some users possibly not want any exercise help or user possibly not want the application of interrupting carrying out because of practice sessions, but change the exercise mode of selecting to be used to instruct purpose into.Be not when each difference that exists with the filter data of posture, all to trigger vision to help, this system can determine whether to show guiding gesture data based on various triggerings.

Filter parameter is accepted level with the threshold value that is used for error identification and can be modified, and makes vision only help in useful time trigger.For example, before offering help, the posture that overhand is thrown possibly be identified X time.Therefore, the motion that this system can monitoring user, and do not provide help till specific times has been done in incorrect or vicissitudinous motion.Like this, if made the once throwing of error, it does not trigger.But, if this program identification goes out some things that the user can change, though then user's successful execution this posture, the option that this system also can be provided for practising.For example, under the situation that the overhand that has identified the user is thrown, the change that this system can identify user's low health can produce bigger speed, or the change that user's batting is accomplished in the motion can produce the curve ball motion.

Playing under the situation of throwing game unfamiliar user or novice users, the filter parameter of definition space body can be bigger, thereby before the vision that triggers this posture helps, allow bigger wrong surplus.When the user did not satisfy the certain skills level, this user also can ask to provide vision to help.The user can ask vision help to help the level of skill that reaches higher.The level of skill that depends on the user, the threshold value of parameter can change.For example, for the new hand player, can have bigger acceptance error surplus but go out the space body that fist or overhand throw.Therefore, depend on exercise type that the parameter of application, user-selected setting, this application are available etc., the triggering that gets into the triggering of exercise mode or provide vision to help can change.

Vision helps storehouse 193 to comprise to provide to the access right of storage and handles each module of the vision help information that is used for posture is demonstrated.Vision helps to be exclusively used in application, user or given pose.Some postures are applicable to application-specific, and wherein the same posture in the Another Application causes Different control.For example, in an application, waving can be the posture that expression is circled in the air; In Another Application, lifting both arms and swing back and forth lentamente can be the posture that expression is circled in the air.

Can take any suitable form with the vision relevant information of visual feedback in the storehouse 193 that helps.In an example embodiment, it is the form of pre-recorded content that vision helps, and promptly this content is a content recorded during using this content stage before.Pre-recorded content module 196 can write down or store the pre-recorded content of the demonstration of supporting the demonstration posture, like the prerecord audio or video relevant with a posture.Pre-recorded content module 196 can be provided for being connected to network and receive the technology of prerecord pose information from remote server or via on-line customer's computing environment.Pre-recorded content module 196 can be handled pre-recorded content provides the vision of this posture to help, like skeleton representation, ghost image or the player's incarnation through this posture of demonstration demonstration.

Pre-recorded content can be exclusively used in application, user, posture, or pre-recorded content goes for respectively using or being applicable to each user's combination.Pre-recorded content can be that the vision that is packaged in application-specific helps the pose information in the storehouse 193.For example, be that tennis game is used if use, then this application can comprise that the vision with the prerecord pose information that is used to show the relevant posture of tennis helps the storehouse.

Pre-recorded content can comprise the content of user record, and wherein user's posture of selecting to write down he or herself is checked or used after a while being used to.For example, the user possibly successfully make special exercise and write down it in tennis game, makes this user can visit this pre-recorded content after a while to check this prerecord posture.The user can check subsequently he or herself be used for the user in the past success the prerecord posture of demonstration of posture.The user can write down this system and can be used for the demonstrate posture content of posture to unfamiliar user.For example, father and mother's posture that can write down the success of he or herself is looked back for child.This child can check and use this prerecord posture exercise after a while, with the proper exercise of learning in physical space, to make.

In another example embodiment, it is the form of live content that vision helps, and promptly helps relevant any information with the real-time vision that a posture is provided.Show the demonstration of the visual representation that refers to posture or the demonstration that vision helps in real time, wherein this demonstration and the execution of this posture in physical space simultaneously or almost simultaneously show.Real-time quoting comprised execution, and wherein inessential processing delay causes minimum display delay or not visible as far as the user.Therefore, comprise in real time with by the required time delay of automatic data processing the relevant any inessential delay of promptness of data.

Live content module 197 can be provided for the technology that receives, handle and transmit live content.For example, live content module 197 can be provided for being connected to network and receive the technology of the live feed that comprises pose information from remote server or from on-line customer's computing environment.Live content module 197 can be handled the live feed that comprises pose information, with the demonstration of real-time demonstration posture.For example, long-distance user's posture of can demonstrating, wherein relevant with this long-distance user's posture information transmit through network and are received by local user's computing environment.Local computing environment can be handled should the live telecast feed, as via the live content module, and shows in real time that to the local user vision helps.Vision help can be the playback or the live expression of user's posture, representes via this user's visual representation.Live content module 197 can show in any suitable manner that vision helps, like skeleton representation, ghost image or the player's incarnation through this posture of demonstration demonstration.

Live content can be exclusively used in to be used or the user, or live content goes for respectively using or being applicable to each user's combination.For example, for application-specific, can support that wherein the live customer support provides the live feed of pose information from remote computation environment access live customer.Live content module 197 can receive live feed and when the user makes posture to user real time provide the vision of this posture of expression to help.

In another example that live telecast helps, the user can remotely connect or network, and makes a plurality of users to carry out alternately via their computing environment separately.Second user that first user can identify the distant place that is positioned at first user is carrying out posture improperly.First user can connect the demonstration that this posture is provided to this user via network.Second user's computing environment for example receives the information relevant with the posture of being demonstrated and provides vision to help to second user via live content module 197.Therefore, first user can provide the proper exercise of live pose information to help second user learning in physical space, to make to second user.

Gesture recognition engine 192, gesture library 190 and vision help storehouse 193 can use hardware, software or both combinations to realize.For example, gesture recognition engine 192, gesture library 190 and vision help storehouse 193 can be implemented as computing environment such as processors such as processors 195 on, on the processor 32 of capture device 20, on the processing unit 101 of Fig. 3 or the software of carrying out on the processing unit 259 of Fig. 4.

Be stressed that the block diagram of describing among Fig. 2-4 is exemplary, and be not intended to hint that one specifically realizes.Thus, the processing unit 259 of the processing unit 101 of the processor 195 or 32 of Fig. 1, Fig. 3 and Fig. 4 can be implemented as single processor or a plurality of processor.A plurality of processors can be located distributed or centralizedly.For example, gesture recognition engine 190 can be implemented as the software of on the processor 32 of capture device, carrying out, and gesture library and vision help storehouse 193 can be implemented as the software of carrying out on the processor 195 in computing environment.Conceived the combination in any that is applicable to the processor of carrying out technology disclosed herein.A plurality of processors can be wirelessly, make up via hardwired or with it and communicate by letter.

Fig. 2 has separately described capture device 20 and computing environment 12, can carry out function shown in Figure 2 but conceived the system that comprises any amount of equipment.For example, this computing environment can merge in the capture device 20, makes capture device can take on the individual unit with one or more processors.Therefore, although computing environment 12 is separated description with capture device 20 in this article, this is for purposes of illustration.Can use any suitable device, system or the equipment that to carry out disclosed technology and the combination of system.

Fig. 3 illustrates the example embodiment that the computing environment 12 that can be used for realizing Fig. 2 is come the computing environment of the one or more postures in objective of interpretation identification, analysis and the tracking system.As shown in Figure 3, this computing environment can be such as multimedia consoles such as game console 100.Also as shown in Figure 3, multimedia console 100 has the CPU (CPU) 101 that contains on-chip cache 102, second level cache 104 and flash rom (read-only storage) 106.On-chip cache 102 is with second level cache 104 temporary storaging datas and therefore reduce number of memory access cycles, improves processing speed and handling capacity thus.CPU 101 can be arranged to have more than one kernel, and additional firsts and seconds high-speed cache 102 and 104 thus.The executable code that loads during the starting stage of bootup process when flash rom 106 can be stored in multimedia console 100 energisings.

The Video processing streamline that GPU (GPU) 108 and video encoder/video codec (encoder/decoder) 114 are formed at a high speed and high graphics is handled.Transport data from GPU 108 to video encoder/video codec 114 via bus.The Video processing streamline is used to transfer to TV or other displays to A/V (audio/video) port one 40 output data.Memory Controller 110 is connected to GPU 108 making things convenient for the various types of memories 112 of processor access, such as but be not limited to RAM (random access memory).

Multimedia console 100 comprises preferably the I/O controller 120 on module 118, realized, System Management Controller 122, audio treatment unit 123, network interface controller 124, a USB master controller 126, the 2nd USB controller 128 and front panel I/O subassembly 130. USB controller 126 and 128 main frames as peripheral controllers 142 (1)-142 (2), wireless adapter 148 and external memory equipment 146 (for example flash memory, external CD/DVD ROM driver, removable medium etc.).Network interface 124 and/or wireless adapter 148 provide the visit of network (for example, internet, home network etc.) and can be comprise in the various wired or wireless adapter assembly of Ethernet card, modem, bluetooth module, cable modem etc. any.

Provide system storage 143 to be stored in the application data that loads during the bootup process.Media drive 144 is provided, and it can comprise DVD/CD driver, hard disk drive or other removable media drivers etc.Media drive 144 can be built-in or external to multimedia controller 100.Application data can be via media drive 144 visit, with by multimedia console 100 execution, playback etc.Media drive 144 is connected to I/O controller 120 via connect buses such as (for example IEEE 1394) at a high speed such as serial ATA bus or other.

System Management Controller 122 provides the various service functions that relate to the availability of guaranteeing multimedia console 100.Audio treatment unit 123 forms the corresponding audio with high fidelity and stereo processing with audio codec 132 and handles streamline.Voice data transmits between audio treatment unit 123 and audio codec 132 via communication link.The Audio Processing streamline outputs to A/V port one 40 with data and reproduces for external audio player or equipment with audio capability.

Front panel I/O subassembly 130 supports to be exposed to power knob 150 and the function of ejector button 152 and any LED (light emitting diode) or other indicators on the outer surface of multimedia console 100.System's supply module 136 is to the assembly power supply of multimedia console 100.Circuit in the fan 138 cooling multimedia consoles 100.

Each other assemblies in CPU 101, GPU 108, Memory Controller 110 and the multimedia console 100 are via one or more bus interconnection, comprise serial and parallel bus, memory bus, peripheral bus and use in the various bus architectures any processor or local bus.As an example, these frameworks can comprise peripheral component interconnect (pci) bus, PCI-Express bus etc.

When multimedia console 100 energisings, application data can be loaded into memory 112 and/or the high-speed cache 102,104 and at CPU 101 from system storage 143 and carry out.The graphic user interface that application can be presented on provides consistent when navigating to different media types available on the multimedia console 100 user experiences.In operation, the application that comprises in the media drive 144 and/or other medium can start or broadcast from media drive 144, to multimedia console 100 additional function to be provided.

Multimedia console 100 can be operated as autonomous system through this system is connected to television set or other displays simply.In this stand-alone mode, multimedia console 100 allows one or more users and this system interaction, sees a film or listen to the music.Yet, integrated along with the broadband connection that can use through network interface 124 or wireless adapter 148, multimedia console 100 also can be used as than the participant in the macroreticular community and operates.

When multimedia console 100 energisings, the hardware resource that can keep set amount is done system's use for multimedia console operating system.These resources can comprise memory reservation amount, and (for example, 16MB), CPU and GPU cycle reservation amount (for example, 5%), network bandwidth reservation amount are (for example, 8kbs) etc.Because keep when these resources guide in system, so institute's resources reserved says it is non-existent from application point of view.

Particularly, memory keeps preferably enough big, starts kernel, concurrent system application and driving to comprise.The CPU reservation is preferably constant, makes that then idle thread will consume any untapped cycle if the CPU consumption that is kept is not used by system applies.

Keep for GPU, interrupt dispatching code so that pop-up window is rendered as coverage diagram, show the lightweight messages (for example, pop-up window) that generates by system applies through using GPU.The required amount of memory of coverage diagram depends on overlay area size, and coverage diagram preferably with the proportional convergent-divergent of screen resolution.Use under the situation of using complete user interface the preferred resolution ratio that is independent of application resolution of using at concurrent system.Scaler can be used for being provided with this resolution ratio, thereby need not to change frequency and cause that TV is synchronous again.

After multimedia console 100 guiding and system resource are retained, systemic-function is provided with regard to the execution concurrence system applies.Systemic-function is encapsulated in the group system application of carrying out in the above-mentioned system resource that keeps.Operating system nucleus sign is system applies thread but not the thread of games application thread.System applies preferably is scheduled as at the fixed time and moves on CPU 101 with predetermined time interval, so that for using the system resource view that provides consistent.Dispatch is to be interrupted minimizing by the caused high-speed cache of the games application of on console, moving for handle.

When concurrent system application need audio frequency, then Audio Processing is dispatched to games application asynchronously owing to time sensitivity.Multimedia console application manager (being described below) is controlled the audio level (for example, quiet, decay) of games application when the system applies activity.

Input equipment (for example, controller 142 (1) and 142 (2)) is shared by games application and system applies.Input equipment is not a reservation of resource, but between system applies and games application, switches so that it has the focus of equipment separately.Application manager is preferably controlled the switching of inlet flow, and need not to know the knowledge of games application, and drives the status information of safeguarding that relevant focus is switched. Camera 26,28 and capture device 20 can be the extra input equipment of console 100 definition.

Fig. 4 shows another example embodiment of computing environment 220, and this computing environment can be used to realize the computing environment 12 of the one or more postures that are used for objective of interpretation identification, analysis and tracking system shown in Figure 1A-2.Computingasystem environment 220 is an example of suitable computing environment, and is not intended to the scope of application or the function of disclosed theme are proposed any restriction.Should computing environment 220 be interpreted as yet the arbitrary assembly shown in the exemplary operation environment 220 or its combination are had any dependence or requirement.In certain embodiments, the various calculating elements of being described can comprise the circuit that is configured to instantiation each concrete aspect of the present invention.For example, the terms circuit of using in the disclosure can comprise the specialized hardware components that is configured to carry out through firmware or switch function.In other examples, terms circuit can comprise by the General Porcess Unit of the software instruction configuration of the logic of implementing to can be used for to carry out function, memory etc.Comprise that at circuit in the example embodiment of combination of hardware and software, the implementer can write the source code that embodies logic, and source code can be compiled as the machine readable code that can be handled by General Porcess Unit.Because those skilled in the art can understand prior art and evolve between hardware, software or the hardware/software combination and almost do not have the stage of difference, thereby select hardware or software to realize that concrete function is the design alternative of leaving the implementor for.More specifically, those skilled in the art can understand that software process can be transformed into hardware configuration of equal value, and hardware configuration itself can be transformed into software process of equal value.Therefore, realize still being that the selection that realizes of software is design alternative and leaves the implementor for for hardware.

In Fig. 4, computing environment 220 comprises computer 241, and computer 241 generally includes various computer-readable mediums.Computer-readable medium can be can be by any usable medium of computer 241 visit, and comprises volatibility and non-volatile media, removable and removable medium not.System storage 222 comprises the computer-readable storage medium of volatibility and/or nonvolatile memory form, like read-only storage (ROM) 223 and random-access memory (ram) 260.Basic input/output 224 (BIOS) comprises the basic routine such as transmission information between the element that helps between the starting period in computer 241, and the common stored of basic input/output 224 (BIOS) is in ROM 223.But RAM 260 comprises processing unit 259 zero accesses usually and/or at present by the data and/or the program module of processing unit operation.And unrestricted, Fig. 4 shows operating system 225, application program 226, other program modules 227 and routine data 228 as an example.

Computer 241 also can comprise other removable/not removable, volatile/nonvolatile computer storage media.Only as an example; Fig. 4 shows and reads in never removable, the non-volatile magnetizing mediums or to its hard disk drive that writes 238; From removable, non-volatile magnetic disk 254, read or to its disc driver that writes 239, and from such as reading removable, the non-volatile CDs 253 such as CD ROM or other optical mediums or to its CD drive that writes 240.Other that can in the exemplary operation environment, use are removable/and not removable, volatile/nonvolatile computer storage media includes but not limited to cassette, flash card, digital versatile disc, digital recording band, solid-state RAM, solid-state ROM etc.Hard disk drive 238 usually by interface 234 grades for example not the removable memory interface be connected to system bus 221, and disc driver 239 is connected to system bus 221 by for example interface 235 interfaces such as removable memory such as grade usually with CD drive 240.

More than discuss and be that computer 241 provides the storage to computer-readable instruction, data structure, program module and other data at driver shown in Fig. 4 and the computer-readable storage medium that is associated thereof.In Fig. 4, for example, hard disk drive 238 is illustrated as storage operating system 258, application program 257, other program modules 256 and routine data 255.Notice that these assemblies can be identical with routine data 228 with operating system 225, application program 226, other program modules 227, also can be different with them.Be given different numberings at this operating system 258, application program 257, other program modules 256 and routine data 255 at this, they are different copies at least with explanation.The user can pass through input equipment, such as keyboard 251 and pointing device 252 (being commonly referred to as mouse, tracking ball or touch pads) to computer 241 input commands and information.Other input equipment (not shown) can comprise microphone, control stick, game paddle, satellite dish, scanner etc.These are connected to processing unit 259 through the user's input interface 236 that is coupled to system bus usually with other input equipments, but also can be by other interfaces and bus structures, and for example parallel port, game port or USB (USB) connect.Camera 26,28 and capture device 20 can be the extra input equipment of console 100 definition.The display device of monitor 242 or other types also is connected to system bus 221 through the interface such as video interface 232.Except that monitor, computer can also comprise can be through other peripheral output equipments such as loudspeaker 244 and printer 243 of output peripheral interface 233 connections.

The logic that computer 241 can use one or more remote computers (like remote computer 246) connects, in networked environment, to operate.Remote computer 246 can be personal computer, server, router, network PC, peer device or other common network nodes; And generally include many or all are above about computer 241 described elements, but in Fig. 4, only show memory storage device 247.Logic depicted in figure 2 connects and comprises Local Area Network 245 and wide area network (WAN) 249, but also can comprise other networks.These networked environments are common in office, enterprise-wide. computer networks, Intranet and internet.

When being used for the lan network environment, computer 241 is connected to LAN 245 through network interface or adapter 237.When in the WAN networked environment, using, computer 241 generally includes modem 250 or is used for through setting up other means of communication such as WAN such as internet 249.Modem 250 can be built-in or external, can be connected to system bus 221 via user's input interface 236 or other suitable mechanism.In networked environment, can be stored in the remote memory storage device with respect to computer 241 described program modules or its part.And unrestricted, Fig. 4 shows remote application 248 and resides on the memory devices 247 as an example.It is exemplary that network shown in should be appreciated that connects, and can use other means of between computer, setting up communication link.

Can store the people's who is used for scanning the scene of catching instruction on the aforementioned calculation machine readable storage medium storing program for executing.This computer executable instructions can comprise the instruction that is used to receive the depth image of physical space and presents the vision help of the guiding gesture data of representing a posture, and wherein this depth image comprises the data of representing this posture.Can store the instruction that is used to determine whether to provide guidance data on the aforementioned calculation machine readable storage medium storing program for executing.These instructions can comprise the view data (wherein this view data comprises the expression of this posture) that receives a scene; The data of this posture of expression and at least one output of posture filter are compared; Difference (wherein this difference is indicated the executory failure of this posture) between the data that detect this posture of expression and this at least one output of posture filter, and determine whether to provide guiding gesture data based on this difference.

The user's that Fig. 5 A has described to generate from capture device 20 example skeleton mapping.In this embodiment; Identify each joint and bone: the top 526 and the bottom 528 of each hand 502, each forearm 504, each elbow 506, each biceps 508, each shoulder 510, each hip 512, each thigh 514, each knee 516, each shank 518, each foot 520,522, trunk 524, vertebra, and waist 530.Following the tracks of under the situation of multiple spot more, can identify additional characteristic, such as the bone and the joint of finger or toe, or each characteristic of face, like nose and eye.

The user can create posture through the health that moves him.Posture comprises user's motion or attitude, and it can be captured as view data and resolve its meaning.Posture can be dynamic, comprises motion, like the imitation pitching.Posture can be a static attitude, intersects like trunk 524 fronts a people and holds his forearm 504.Posture can be single moving (for example, jumping) or continuous posture (for example, driving), and on the duration, can lack and can grow (for example, driving 20 minutes).Posture also can combine stage property, as through brandishing imitated sword.Posture can comprise more than a body part, as clapping both hands 502, or more small motion, as sticked up a people's lip.

User's posture can be used as the input in the general computational context.For example, the various motions of hand 502 or other body parts can be corresponding to common system-level task, as in hierarchical lists, navigate up or down, open file, close file and preservation file.For example, the user can upwards refer to and the centre of the palm makes his hand maintenance motionless to capture device 20 with finger.He can draw finger in then and form fist towards palm, and this can be that indication should pent posture based on the focus window in the user interface computing environment of window.Posture also can depend on to play in the video-game specific context to be used.For example, for driving recreation, the various motions of hand 502 and pin 520 can be corresponding to operating and controlling vehicle, gearshift, acceleration and brake on a direction.Thus, posture can indicate be mapped to that the user that shown representes, such as the various motions in the wide variety of applications such as video-game, text editor, word processing, data management.

The user can walk in the original place or run and generate the posture corresponding to walking or running through oneself.For example, the user can alternatively mention and put down each leg 512-520 and comes to plan to implement away at the situation counterdie that does not move.System can resolve this posture with each thigh 514 through analyzing each hip 512.When a hip-thigh angle (like what measure with respect to vertical line, the leg of wherein standing has hip-thigh angle of 0 °, and the leg of horizontal stretching has hip-thigh angle of 90 ° forward) surpasses the specific threshold with respect to another thigh, can discern a step.Walk or run and after the continuous step of a certain quantity that replaces leg, to be identified.Time between two nearest steps can be considered to one-period.Do not satisfying after threshold angle reaches the cycle of a certain quantity, system can confirm that the walking or the posture of running stop.

Given " walk or run " posture is used and can be the pre-set parameter that is associated with this posture.These parameters can comprise that the periodicity that the step does not take place and definite posture of the required step number of above-mentioned threshold angle, initiation walking or the posture of running, recovery are walking or the threshold period of running.Fast period can be corresponding to running, because the user will move his leg apace, and the slower cycle can be corresponding to walking.

Posture can be associated with one group of default parameters at first, uses available its oneself parameter and covers this group default parameters.In this scene, not forcing to use provides parameter, can change the default parameters that uses a group to allow identification posture under the situation of the parameter that does not have application definition into but use.The information relevant with posture can be stored the purpose of the posture animation that is used for prerecording.

The various outputs that can be associated with posture are arranged.Can be relevant for posture occurent baseline " be or not " whether.Level of confidence can also be arranged, and it is corresponding to the possibility corresponding to posture that moves of usertracking.This can be that scope is the lineal scale of the floating number that (comprises end points) between 0 and 1.Can not accept in the false application as input certainly in the application that receives this pose information, it can only use has high confidence level, the posture of having discerned as at least 0.95.Under the situation of using each instance that must the identification posture, even be cost certainly with vacation, it can use the posture that has much lower level of confidence at least, as only greater than those postures of 0.2.Posture can have two outputs of the time between the step recently, and under the situation of only having registered the first step, this can be set as retention, like-1 (because the time between any two steps just is necessary for).Posture also can have the output about the highest thigh angle that during a nearest step, reaches.

Another exemplary posture is " heel is mentioned jumping ".In this posture, the user can be through lifting from ground with his heel, creates this posture but keep his toe to land.Alternatively, the user can jump in the air, wherein his pin 520 complete built on stilts.Whether this system can resolve the skeleton of this posture through the angular relationship of analyzing shoulder 510, hip 512 and knee 516, are the aligned positions that equal upright to check them.Then, can keep watch on these points with higher 526 and low 528 vertebras point find any upwards acceleration.Enough acceleration combinations can trigger the jump posture.Enough combinations of acceleration and a certain posture can be satisfied the parameter of transition point.

Given " heel the is mentioned jumping " posture of being somebody's turn to do is used and can be the pre-set parameter that is associated with this posture.Parameter can comprise above-mentioned acceleration threshold value, and it confirms certain combination of user's shoulder 510, hip 512 and knee 516 must move up how soon trigger this posture; And the maximum alignment angle of takeing on 510, still can trigger jump between hip 512 and the knee 516.Output can comprise level of confidence, and the body angle of user when jumping.

Based on the details of the application that will receive posture come for the posture setup parameter for identifying posture exactly, be important.The intention that correctly identifies posture and user greatly helps to create positive user and experiences.

Application can identify the point of the animation that uses prerecording for the parameter setting values that is associated with various transition points.Transition point can be defined by various parameters, like angle or its any combination of sign, speed, target or the object of a certain posture.If transition point is defined by the sign of a certain posture at least in part, then correctly identify posture and help to improve the confidence level that the parameter of transition point has been satisfied.

Another parameter for posture can be the distance that moves.Under the situation of the action of the incarnation in user's ability of posture control virtual environment, this incarnation can be the length that arm leaves ball.If the user hopes mutual and catch it with this ball, then this arm 502-510 that can require the user to stretch him makes grip posture simultaneously to total length.In this situation, the similar grip posture that the user only partly stretches his arm 502-510 possibly can't reach the result mutual with ball.Similarly, the parameter of transition point can be the sign to grip posture, if wherein the user only partly stretches his arm 502-510, thereby does not have to realize and the mutual result of ball that then user's posture will not satisfy the parameter of transition point.

The space body that posture or its part can must it take place therein is as parameter.Comprise that in posture this space body can be expressed with respect to health usually under the situation that health moves.For example, for the rugby throwing gesture of right-handed user can be only be not less than right shoulder 510a and with the space body of throwing arm 502a-310a in 522 same side in discern.All borders that maybe unnecessary definition space body as for this throwing gesture, are wherein kept from the outside border of health and are not defined, and this space body ad infinitum stretches out, and perhaps extend to the edge of the scene of just being kept watch on.

Fig. 5 B provides the further details of an exemplary embodiment of the gesture recognizers engine 190 of Fig. 2.As shown in the figure, gesture recognizers engine 190 can comprise at least one filter 518 that is used for confirming one or more postures.Filter 518 comprises the information of definition posture 526 (below be called " posture "), and can comprise at least one parameter 528 or the metadata that is used for this posture.For example, comprise that throwing that a hand is crossed preaxial motion behind from health can be implemented as and comprise that a hand representing the user crosses the posture 526 of preaxial mobile information behind from health that this moves being caught by degree of depth camera.Can be this posture 528 setup parameters 526 then.In posture 526 is under the situation of throwing, and distance that parameter 528 can be this hand threshold velocity that must reach, this hand must be advanced (absolute, or with respect to user's whole size) and identifier engine are to the confidence level grading of posture takes place.These parameters 526 of posture 528 can change along with the time between each context between each application, in single application or in the context an application.

Filter can be modular or interchangeable.In one embodiment, filter has a plurality of inputs and a plurality of output, and each in these inputs has a type, and each in these outputs has a type.In this situation, first filter can be with having other aspects of replacing with second filter of the input and output of the first filter equal number and type and not changing the identifier engine architecture.For example, possibly have first filter that will drive, this first filter with skeleton data as input, and the output occurent confidence level of posture and the steering angle that are associated with this filter.Drive filter in hope with second and replace this and first drive under the situation of filter (this possibly be because second to drive filter more efficient and need processing resource still less); Can come to do like this through replacing first filter with second filter simply, as long as second filter has same input and output---input of skeleton data type and two outputs of confidence level type and angular type.

Filter need not have parameter.For example, " user height " filter that returns user's height possibly not allow any parameter that can be conditioned." user's height " filter that substitutes can have customized parameter, such as the footwear, hair style, headwear and the figure that when confirming user's height, whether consider the user.

Can comprise such as joint data, as the formed angle of bone that intersects at joint, from the rgb color data of scene and user's contents such as rate of change in a certain respect the input of filter about user's joint position.Output from filter can comprise such as the confidence level of just making given posture, make the speed of posture motion and the contents such as time of making the posture motion.

Context can be a cultural context, and can be environmental context.Cultural context refers to the user's of using system culture.Different cultural can use similar posture to give significantly different implication.For example, hope to make another user " see " or the U.S. user of " using his eyes " can be placed on his forefinger on his head far-end near his eyes.Yet as far as Italian user, this posture can be interpreted as quoting Mafia.

Similarly, among the varying environment of single application, has different contexts.To relate to the first person shooting game of operating motorcycle is example.When the user is walking, will point and clench fist towards ground and can express the fist posture forward and from the protruding fist of health.When the user was driving in the context, identical motion can be represented " gearshift " posture.Possibly also have one or more menu environment, wherein the user can preserve his recreation, between his personage's equipment, select or carry out similarly not comprise the action of directly playing games.In this environment, this recreation posture can have the 3rd implication, as selects some thing or advance to another screen.

Filter can move side by side, and a plurality of filter can be sought identical things but different execution.Thereby, depending on the arrangement of the motion of definable posture, the quantity of filter can increase.For example, curve ball can be the overhand posture, but the player prefers pitching underhand (when they do under the very successful situation like this) subsequently.

Gesture recognizers engine 190 can have the basic identifier engine 518 that function is provided to posture filter 516.In one embodiment; The function that identifier engine 516 is realized comprises that the input in time (input-over-time) of posture that tracking has been discerned and other inputs is filed, hidden Markov model realize (wherein modeling system be assumed that have unknown parameter Markov process-wherein current state has encapsulated the required any past state information of definite state in future; Therefore needn't safeguard the process of any other past state information for this purpose, but and hiding parameter confirm from observed data) and other required functions of particular instance of finding the solution gesture recognition.

Filter 518 loads on basic identifier engine 516 and realizes, and engine capable of using 516 offers the service of all filters 518.In one embodiment, the received data of basic identifier engine 516 processing confirm whether it satisfies the requirement of any filter 518.Because these wait the service that provided to provide but not provided by each filter 518 by basic identifier engine 516 is disposable such as resolving input; Therefore this service only need be processed once in a period of time rather than this time period is handled once each filter 518, has reduced the required processing of definite posture thus.

The filter 518 that application can use gesture recognizers engine 190 to be provided, perhaps it can provide its oneself filter 518, and this filter is inserted in the basic identifier engine 516.In one embodiment, all filters 518 have the general-purpose interface of launching this insertion characteristic.In addition, therefore all filter 518 parameters 528 capable of using can use the single posture instrument that is described below to diagnose and regulate whole filter system 518.

These parameters 528 can be regulated for the context of using or use by posture instrument 520.In one embodiment, posture instrument 520 comprises that the figure of a plurality of slide blocks 522 and health 524 representes, each slide block 522 is corresponding to a parameter 528.When adjusting parameter 528 with corresponding slide block 522, health 524 can be showed and will be identified as the action of posture and use these parameters 528 will not be identified as the action of posture with these parameters 528, like what identified.The visual effective means that provides debugging and refinement to regulate posture of this of the parameter 528 of posture.

The example of the application of on the computing environment shown in Fig. 6 A-6C, carrying out is the boxing game that the user is playing.Fig. 6 A-6C representes that the user goes out fist posture 62a, 62b, 62c in physical space, and wherein user's visual representation 64a, 64b, 64c are mapped to user's in the physical space motion.Among Fig. 6 A-6C each is illustrated in the user make out the fist posture during the user at position 62a, 62b, the 62c of three discrete time points in physical space and example 64a, 64b, the 64c that is displayed on the user's in the application space visual representation.The speed of seizure and display image data frame is confirmed the continuity level of the motion that is shown of visual representation.Though can catch and show additional image data frame, the frame of describing among Fig. 6 A-6C is that the purpose of property is presented for purpose of illustration selected.

The motion that the user makes can be caught, analyzes and followed the tracks of to capture device 20 in physical space, like user's shake one's fists posture 62a, 62b, 62c.According to an example embodiment, capture device 20 can be configured to catch the video that has depth information, comprises the depth image of scene.Capture device 20 can provide the depth information and the image of being caught and can be shown for audiovisual assembly 16 by the skeleton pattern that capture device 20 generates.The user can check this image or user's visual representation 64a on the audio-visual equipment such as television screen 16 shown in Fig. 6 D.

Audio-visual equipment 16 can provide the user can use his or her visual representation 64a, 64b, the 64c that moves player's incarnation of controlling.User's motion can be mapped to the visual representation in the application space, in using, to carry out one or more controls or action.For example, can use capture device 20 to follow the tracks of user 18, so that can user 18 posture be interpreted as the control that can influence the application of carrying out by this computing environment.In boxing game was used, the posture that the posture 62a of the shake one's fists forms of motion of user in physical space, 62b, 62c control visual representation 64a, 64b, 64c have done made in gamespace, to go out fist.Therefore; According to an example embodiment; The computing environment of Target Recognition, analysis and tracking system 10 and capture device 20 can be used for discerning and analysis user 18 goes out fist in physical space, can be interpreted as the game control to the player's incarnation 40 in the gamespace thereby make this go out fist.Visual representation can be mapped to user's posture and come to show in real time with respect to the execution of this posture in physical space.

In some cases, the vision help that provides expression to be used to instruct the user how correctly to make the guiding gesture data of posture is desirable.The directiveness gesture data can be instructed the user how correctly to make posture and controlled the application of carrying out.In an example embodiment, wrong in can the posture of identifying user of computing environment or the application carried out also provides guiding gesture data give prominence to these mistakes of demonstration.In another example embodiment, the user selects practice sessions or gets into exercise mode to learn how correctly to make posture.

Fig. 7 A-C describes to be used for to come the posture of playback user and show the example split screen screenshot capture of guiding gesture data side by side with this incarnation as incarnation.The left side of each split screen is presented on visual representation 64a, 64b, the 64c of the user's that the user shakes one's fists during the posture at three discrete time points posture, shown in Fig. 6 A-6C.The right side of each split screen presents the example embodiment that the vision of expression and the corresponding guiding gesture data 74a of each 64a, 64b, 64c of image data frame, 74b, 74c helps.Fig. 7 D describes the display part from the system 10 of Figure 1A and 1B, and it illustrates the example embodiment that is used for split-screen display.In this example, the left side of each among Fig. 7 A-7C is corresponding to the snapshot or the image data frame that are captured by degree of depth camera, parallel RGB camera, or corresponding to the image from these two kinds of cameras combinations.On the right side of split screen, this system shows that the vision of the mistake in the posture of giving prominence to explicit user helps.

It can be the demonstration of proper exercise that vision helps.For example, user practice sessions and this system that can select to be used to fist can start the specific interactive practice sessions that goes out the proper exercise of fist of this user of instruction.This system can help the motion (live or playback) of explicit user with vision, thus any mistake in the motion of outstanding explicit user.Mistake in the posture of the outstanding explicit user of various Display Techniques.This help can be described user's physical location and the increment between the ideal position.For example, the part of the location of expression user in physical space of the health 64a of the arrow points incarnation among Fig. 7 B and the 7C, 64b, 64c, thus be illustrated in this moment user's location and desirable postural position 74a, 74b, the difference between the 74c.

The vision that this system can be provided for practising purpose helps, and is used to control this system or the appropriate exercise of the application carried out with demonstration.Exercise can be the part of user's exercise mode of initiatively selecting to get into.The user also can ask exercise, when using as carrying out first the user.Variety of event can trigger the demonstration of the vision help that is used for the posture demonstration.This system can detect wrong in the user movement or with the deviation of desired movement.This system can identify can be modified in this posture and in the application of carrying out, reach higher successful user's locating area.User's posture is not and is discerned the corresponding posture of posture, thereby indicates this user to be unfamiliar with suitable posture.The measurable user's of this system expection posture, and be provided for training this user to come correctly to make the option of this posture.This system can discern unfamiliar user and exercise is provided.In these incidents any can trigger the demonstration of the vision help that comprises guiding gesture data.

This system or application can comprise exercise mode and execution pattern.Result as such as triggerings such as above-mentioned triggerings can get into exercise mode automatically.For example, if user 18 shoots in physical space not and discerns that posture is corresponding to go out fist, then system 10 can interrupt this application, gets into exercise mode and provide the vision of the suitable posture of demonstration to help to the user.

This system can come the mistake in the identifying user motion through the position of assesses user in single Frame of catching or in series of frames.For example, the fist motion is colluded on can being with user's gesture recognition by this system, by identifying with respect to moving in user's the particular space body.This system also can be identified in the model image Frame, and the user does not correctly move their arm or can in boxing game, provide more ten-strike to the change of user movement in physical space.

Whether this system can identify mistake based on some parameter of user's the posture that measures, as dropping on outside the expection space body with the deviation of ideal velocity scope or user's the posture that measures.For example, baseball is used and can be instructed the user to make the overhand throwing to come the pitching to the batter.Degree of depth camera or RGB camera can the motion of identifying user in physical space each side, and posture filter 191 can be designated this motion and is in the throwing gesture class.Based on the space body of being kept watch on around user's head, the parameter of all kinds of throwing gestures can be different.It can be that user's head the place ahead is with the rear but the space body on user's throwing is takeed on that overhand is thrown.Hand is only takeed on throwing and can be defined by the space body at user's waist the place ahead and rear, and this space body is between shoulder and user's waist.The posture filter can identify overhand based on each parameter of the motion of user in physical space and throw.Because the time point in application, carried out (for example, the user is in baseball this user in using and is instructed the time point to batter's pitching), the posture filter possibly expect that overhand throws (or the overhand of attempting is thrown).With the deviation of filter parameter or fail to satisfy threshold value and accept level and can trigger the demonstration that vision helps.

The demonstration that comprises the vision help of guiding gesture data can be taked any suitable form.In this embodiment, guiding gesture data 74a, 74b, 74c are shown as stick figure and represent, can take any suitable expression but conceived visual representation.For example, the posture that the part of the outstanding demonstration of arrow or user's visual representation can indicating correct is shown in Fig. 7 A-7D.Shown in Fig. 7 A-7D, this demonstration can be the expression side by side of user's posture and desirable posture equally.Vision help can be the form (to catch speed, to slow down speed, fast velocity etc.) of the motion of playback or shown in real time.Shown in Fig. 8 A, represent that the image of guiding gesture data can ghost image or be superimposed upon on the visual representation.

Snapshot or image data frame that the left side of each among Fig. 7 A-7C can be directly captures corresponding to degree of depth camera and/or RGB camera.Selected each image data frame can be with any speed playback, and can checked or check as the screen capture that separates with coming one time one frame with the continuous playback of motion.Image data frame can be to reset with the corresponding speed of per second frame per second of catching.Guiding gesture data such as guiding gesture data such as illustrating on each the right-hand side in Fig. 7 A-7C can be corresponding with the image data frame that on the left side, illustrates.For any given posture, can catch any amount of image data frame and can generate the guiding gesture data frame of any amount of correspondence.Thereby additional image data frame possibly be available with corresponding guiding gesture data.The user can suspend the demonstration that vision helps, rolls, checks, convergent-divergent etc.

The snapshot of the view data on the left side of Fig. 7 B is depicted in the user's at the second time point place posture 64b.On the right side of split screen, this system shows that the vision of the mistake in the posture of giving prominence to explicit user helps.With the guiding gesture data of mistake in the posture of outstanding explicit user abreast the visual representation of the posture of explicit user can help the user to proofread and correct he or her motion in physical space.For example, Fig. 7 B gives prominence to the mistake in the explicit user posture through the position of using arrow points user arm.In this example, vision helps the more good position of this some place user arm of sensing in this posture how correctly to make with the instruction user and colludes fist motion or more successful in boxing game.Similarly, Fig. 7 C describes the snapshot of user's view data, points out this some place of colluding the fist posture last, accomplishes this motion if the user is at the arm that makes him under the situation of higher position, and then he or she will reach better success.Describe to demonstrate user's physical location and the increment between the ideal position side by side.

This system can help the motion of explicit user to give prominence to the mistake in the motion of explicit user with live telecast or real-time vision.Thereby the snapshot of the view data among Fig. 7 A-7C is not the playback of user's posture, and this user can make posture and check that the live vision on the right side of each split screen helps in physical space.For example, guiding gesture data can be provided in interactive practice sessions.The correct location that practice sessions can be demonstrated and colluded each discrete time point place in the fist posture last.At each discrete some place, the user can make posture in physical space, and helps to observe in real time when comparing this posture in this user's posture with the vision that correct postural position is shown.The user can be circulated between each time point, thereby study and the real-time vision carrying out the correct body position at every bit place and receive the ideal position that should be at this moment about the user help.The user can proofread and correct his or her motion by frame.

Although Fig. 7 A-7C describes user's visual representation and the split screen of guiding gesture data shows that having conceived guiding gesture data can provide with any suitable form.For example, be not split screen, the left side that can only show the split screen among Fig. 7 A-7C is with the mistake in the outstanding explicit user posture.In this example, mistake is by the outstanding arrow that is shown as the deviation of directed towards user posture.In another example, or combined with the guiding gesture data of another form, help can be this user wrong or possible correction such as sense of hearings such as offscreen voice with the speech performance.

The directiveness gesture data can be indicated the correct aligning of user in physical space, in the correct visual field of notifying the seizure the visual field how user to move to capture device.For example; If the given pose filter fail because the motion of the limbs that capture of capture device is shifted out outside the visual field in the seizure visual field of capture device for a posture provide consistent result, then guiding gesture data to comprise to notify the user they need in physical space, move sector data or the help that could aim at capture device better.For example; Offscreen voice we can say " please to moving to left "; Or the user can only illustrate a part of representing on the screen at the visual representation on the screen (visual representation shown in 64a, 64b, 64c), thereby the indication user need aim at he or herself again in physical space.

Fig. 8 A and 8B describe to provide to the user additional example of guiding gesture data.In Fig. 8 A, the expression 804 (shown in stick figure is represented) of calibrated posture animation superposes or covers on user's the visual representation.Therefore, the indication of proper exercise covers on user's the view data.Can reset his or her motion and check the guiding gesture data on the visual representation of the posture that covers the user of user.The user can observe user's physical location and the increment between the ideal position subsequently.Arrow just is used to provide an example of the visual representation of user's arms position and the vision increment between helping.Outstanding this increment that shows allows the user to confirm that what modification in the physical space can improve this posture.

Use (such as ongoing recreation) the term of execution, the exercise data can be used as to cover and show, shown in Fig. 8 A.Therefore, interrupting this recreation possibly be unnecessary to get into independent exercise mode.The directiveness gesture data can comprise prompting or little suggestion to user movement, and this prompting can be in recreation express as covering, and making that recreation continues and do not interrupt when guiding gesture data is provided.

The example that vision among Fig. 8 B helps shows it is the visual representation that comprises the guiding gesture data of the demonstration through posture.This demonstration can be broken down into each stage or continuous video montage.The visual representation of directiveness gesture data can be any suitable form, like the form of skeleton representation, ghost image or player's incarnation.

Select exercise mode, detect the user's posture mistake, start the result who uses etc. as the user, this application can trigger the entering exercise mode.For example, during wrong in detecting user's posture, can be provided to chosen wantonly the inlet of exercise mode to the user.This system possibly can not discern the posture that the user makes, and based on the prediction to user's desired movement the exercise suggestion is provided.This system can identify this posture and the suggestion about the better posture that is applicable to this application at this moment is provided, or is provided for practising the suggestion of making this posture of being made by the user better with more successful in this application.

Fig. 9 describes the example of this system can be shown to the user in boxing game option 902a, 902b, 902c, 902d.Display device 16 can show the option of predicting posture, and the user can select option to receive exercise correctly to carry out this posture or to improve the execution of this posture.Possible posture can be any posture that is applicable to system, the application of carrying out based on posture, has the plug-in unit etc. of additional posture.Option 902a shown in Fig. 9,902b, 902c, 902d are applicable to the application that this is being carried out---boxing game.

In this example, the posture of user in physical space is identified as out the fist posture.As stated, gesture recognition engine 190 can comprise the set of posture filter 191.Each filter 191 can comprise definition posture and the parameter of this posture or the information of metadata.Can data of being caught with skeleton pattern and mobile form associated therewith by camera 26,28 and equipment 20 and the posture filter in the gesture library 190 be compared, when carry out one or more postures with identifying user.

In an example embodiment, this systematic collection is also stored specific user's historical data, as with history data store in user profiles.This system can adapt to filter parameter and this user's threshold level based on this historical data.For example, according to research or emulated data, the filter parameter that is used to identify this posture and is the space body that overhand throws can be set as default value.Yet, and generally to compare, user's hand maybe be often far away apart from user's first watch, and therefore exceed for overhand and throw outside the border of the space body that is provided with.At the beginning, this difference can be indicated the executory failure of this posture, is used to get into how exercise mode carries out this posture with study demonstration or option thereby trigger.This system can collect about this user's data in time and revise filter parameter to adapt to this user's tendency.For example, the parameter of definition space body can be modified space body is moved to such an extent that more closely aim at user's body position.Threshold value is accepted level and also can correspondingly be changed.

Practice sessions can be interactively, so that such as coming with the mode of in Fig. 7 A-7C, describing demonstration, correction etc. are assessed, given prominence in user's actual motion.It can be the demonstration of the correct body position that is applicable to that the user of application can implement in physical space that vision helps, and does not have the visual representation of user movement.Fig. 8 B describes the example of the demonstration of correct body position via each motion frame that separates.This demonstration can be given prominence to explicit user should place the position that his or her health correctly carries out this posture.This demonstration can be regardless of user's posture.Perhaps, it can be the result to the assessment of user's posture that vision helps, wherein vision help to point out to be used for this user special-purpose improved specific region.

Be used to show that the other technologies that vision helps are possible.For example, the visual representation of guiding gesture data can be on the part of display space, and wherein this part is less than total display space.Provide the vision of guiding gesture data to help to comprise the coach, like the coach of user's establishment, human coach, prerecord coach etc.The coach can be used as the video feed and ejects.The coach can provide and the corresponding phonetic order of posture of being demonstrated.The user can learn a posture through the visit exerciser, the caber ball such as how.Basically, coach's each posture that the user is rehearsed form to throw.The user can select specific coach, as from the set of the incarnation of representing true or virtual portrait, selecting.Can use different personages, like famous baseball player.Coach's expression can be the pre-recorded data that shows with the posture of being demonstrated.Perhaps, vision help can be and the live mankind coach who shows in another user real time ground.It can be the form of offscreen voice or the live telecast coach who provides posture to demonstrate via network that vision helps.

Figure 10 describes long-range connection and uses the example carry out two mutual users through boxing game.Long-range connection for example can be passed through network.This network can and can be based on the system of subscription by Host Administration.Such connection allows long-range connection of user or networking, so that a plurality of user can carry out via their computing environment separately alternately.

For example, consider that first and second long-distance users are carrying out games application, resist other users as team through long-range connection.First user recognizes that second user is carrying out a certain motion improperly and thereby causing this team in recreation, to fail.First user can provide the demonstration 1002a of posture in real time, makes second user can on the display in second user's the computing environment, observe this vision and helps 1002b.Second user's computing environment can receive the information relevant with the posture of being demonstrated and provide vision to help to second user.Vision helps to offer second user lively, has only postponed to transmit the time that is spent from first user, and has handled and show this information to second user.Can realize suitable technique so that the transmission and the processing time be fast, thereby demonstrate to minimal visual to help second user's delay based on first user's live posture.

The visual representation of user #1 is shown to the result that user #2 can be a plurality of triggerings in real time.For example, user #1 can be from user #2 request indication, and user #2 can identify the mistake in the posture of user #1, and these two users can come mutual etc. from the purpose of learning tactics each other.The user can suspend application and carry out interactive practice sessions.The computing environment of user #1 can explicit user #2 visual representation, thereby the simulated field practice sessions.User #2 can exchange, show motion etc. similarly with user #2.

Figure 11 A describes to be used for posture coach's exemplary operational process.

Operation 1102 has described to receive the data of being caught by capture device 20, the posture that these data are carried out corresponding to the user.Capture device at a distance can be caught the scene that comprises whole users, such as from ground to the ceiling and to the wall of every side in user room of living in.Capture device also can be caught the scene of the part that only comprises the user, the user when being sitting in limit such as he or she more than the belly.Capture device can also be caught the object by user control, is held in the stage property camera in he or her hand such as the user.

Operation 1104 described to analyze data with produce with data whether corresponding to the corresponding output of the posture of system identification.In an embodiment, this analysis can be carried out with the filter that is applied to data by gesture recognizers engine 190 518.

Operation 1106 has been described to confirm that from this output the user has unlikely correctly carried out the posture of system identification.In an embodiment, the posture of system identification comprises and the corresponding posture of filter.

In an embodiment, output comprises level of confidence.In an embodiment, when level of confidence was lower than threshold value, the user had unlikely correctly carried out and the corresponding posture of filter.This threshold value can be the level that the user possibly correctly carry out posture.The user unlikely carried out between the threshold level of posture and possibly have difference when the user possibly correctly carry out the threshold level of posture and be lower than threshold level when being higher than threshold level; Level between those two threshold values neither is considered to possible, also is not considered to unlikely.

Comprise among the embodiment of a plurality of outputs that at the posture filter unlikely correctly carry out posture when corresponding when at least one output with the user, the user unlikely correctly carries out posture.For example, " car steering " filter can comprise the distance that is used between user's both hands, user hand with respect to the position of his other parts of health and the output of the angle that hand tilts.The user makes his hand separate 12-18 ", the front and every the hand that are positioned at him shows with identical driving angle is corresponding when leaving vertical rotation angles and spending, the user might be at the execution vehicle driving posture.The user is his hand and be placed on his front, outwards rotate both hands separately correctly; Make his left hand indication drive left and the indication of his right hand is driven to the right, short of of satisfying in these parameters just is enough to confirm that the user unlikely correctly carries out posture.

Operation 1108 confirms that from exporting the user might want the posture of executive system identification.

Must keep among the embodiment of posture the user; Keep when driving steering position to reach the time period of prolongation through the user; Can confirm that the user might want to carry out posture; Wherein output is corresponding with the posture that just is being performed at least occasionally, but not corresponding with the posture that just is performed the time period that reaches prolongation.

Comprise among the embodiment of a plurality of outputs that at the posture filter carry out posture when corresponding when output with the user, the user might want to carry out this posture.For example; Given above " car steering " filter of discussing might the user have correct hand at interval, and his corresponding hand-screw gyration is corresponding to identical driving direction; But he places the intimate position of loosening of his both sides with both hands, rather than before his body, holds his hand.In this example, two (hand intervals and hand-screw gyration) in the output show that the user might want to make vehicle driving posture, but the 3rd output (hand position) can confirm that the user wants to carry out this posture when not showing.

Operation 1110 has described to provide the help of the posture that relevant user carries out.

In an embodiment, offering help comprises adjustment output, and will send to the corresponding application of carrying out with the user of posture through the output of adjustment.This can be similar to parameter or the tolerance of loosening application.When definite user wants to carry out posture, the output that is used for corresponding posture filter can be changed for the corresponding output of the posture that possibly just be performed, and maintenance customer's intention still.For example, the user shows as and wants steering vehicle rapidly left, but when having incorrect hand position, can be adjusted with the corresponding output of hand position, and still corresponding to vehicle drive (rather than slowly turn left, or turn right) rapidly left.

In an embodiment, adjustment output comprises the response that improves filter.For example, the user can make might be corresponding little mobile with user view, but fail to carry out posture.For vehicle driving posture, no matter how rapidly this can comprise the less amount of hand of when he wants to turn, only rotating him.These move and can be exaggerated.The user rotates his maximum 20 ° of hand, and 90 ° of rotations are during corresponding to the most rapidly turning, and user's actual rotation can be multiply by the factor 4.5, and making his 20 ° of rotations be taken as it is that 90 ° of rotations are treated.

Expect that from actual moving to this mapping of moving can be linear or nonlinear.Might the user approximately correctly carry out very trickle driving campaign, and failed to carry out turning more rapidly.In this case, the response of trickle driving campaign can only slightly be improved, and does not perhaps improve, and wants to pass on the response of the motion of driving rapidly greatly to be improved.

In an embodiment, only definite filter be can controlled filter after, just adjustment output.Some filter can be benefited from adjusting.For example, the user who makes " weapon of opening fire " posture badly and miss his re-set target continually possibly feel depressed to control, and has bad user and experience.Thus, adjustment " weapon of opening fire " filter possibly be useful with the aiming that helps the user, and " weapon of opening fire " filter can be can controlled filter.Yet for " car steering " posture, this can not be true.On traditional sense, the user possibly drive very badly---and he collides or disorbit continually.Yet the certain user likes ignoring the declared target of scene, and intentionally bad the execution.In this situation, in fact adjustment " car steering " filter possibly damage user's experience, because this stops him to do that he wants to do.Here, " car steering " posture can be can not controlled posture.

Can confirm whether filter can be adjusted, for example, through can be by boolean's output of the entity filter that read of adjustment filter, perhaps through with the corresponding data structure of filter that can read by the entity of adjustment filter in Boolean set.

In an embodiment, offer help and comprise and use second filter to replace filter.For example, the user possibly be difficult to carry out the posture that is associated with " expert's driving " filter, and this filter has the lower tolerance to the variation of exemplary motion.This can be determined, and " expert's driving " filter can be replaced by " beginner's driving " filter, and the latter compares " expert's driving " filter variation is had higher tolerance.For example, it will use the output of the output of this new filter filter before replacing the application that can be associated through indication, perhaps places new filter in the position through the filter before removing from the gesture recognition engine and its and replaces filter.

In an embodiment, offer help and comprise and hang up the corresponding application of carrying out with the user of posture.For example, confirm to provide when instructing the user how correctly to carry out the help of posture, the user will be difficult to learn this posture and still mutual with this application.Therefore, this application can be suspended or hung up, and provides and helps to reach certain hour, and such as can as one man carrying out posture up to the user is verified, and after helping session, continue should application.

In an embodiment, offer help to be included on the display device and show output to the user.For example, when output comprises level of confidence, when the user attempts to carry out posture, can make the curve map of level of confidence and time.When he correctly carries out posture, he will see that level of confidence correspondingly improves, and can those moved and correctly carry out posture and be associated.This demonstration can also comprise when to this output be acceptable indication, such as via the flash of light on color change, warning tones or the screen.Output hand distance and the time is made curve map and the hand distance must be positioned at 12 " with 18 " between the time, this curve map can be positioned at 12 in user's hand distance " and 18 " between the time be green, and be red at whole other times.

In an embodiment, the offer help expression of the posture that comprises that explicit user is carried out and the demonstration of posture.This can comprise and shows two postures side by side, makes the user visually identify his and is moving part improperly.In addition, when the user correctly carries out the part of posture and carries out posture a part of improperly, can there be those indications constantly that the user is correctly being carried out, such as through playing sound.In addition, help can comprise the direct guidance about the difference between two postures, is " your both hands must separate 12-18 " such as content.Your both hands show to such an extent that too separate.Try them near some." display on text.

In an embodiment, offer help difference between the demonstration that comprises posture that explicit user is carried out and posture.This can comprise the top that demonstration is superimposed upon the posture of user's execution, makes that difference is tangible.This can comprise the outstanding zone that shows the health that difference exists.In an embodiment, the demonstration of the posture carried out of different ground explicit user and posture so that they identified more independently.For example, one the posture that the user carries out can be shown as user's video or incarnation to be represented when being superimposed on another, and demonstration can be shown as the wire frame incarnation, and vice versa.

In an embodiment, help to be derived from another user.In an embodiment; The help that is derived from is from the set at least one, this set comprise be positioned at or near the help at ranking list top, by the help of highly grading, by its help, second user who has identified as the user who is suitable for helping creating has been designated the help that is suitable for the user, from the user's of the identical culture of user help, from the user's of user's same-language help, from the user's at similar age of user help and from the user's of the similar position of user help.

In an embodiment, offer help and comprise and loosen the tolerance levels that is associated with filter.

Figure 11 B and 11C have described the exemplary architecture that be used for postural training 1150 integrated with gesture recognizers engine 190 and application 1 152.

In Figure 11 B, posture coach 1150 and application 1 152 the two output from each filter 518 each filter 518 of reception.This architecture allows posture to train the output of 1150 monitor filters to confirm whether help is fit to, and exports and allow application 1 152 to receive simultaneously.Alternatively, posture coach 1150 can communicate by letter with application 1 152, such as sending the modified output of being used by application 1 152.

In Figure 11 C, posture coach 1150 receives the output of each filter 518 from those filters 518, and subsequently output is passed to application 1 152.In this architecture, posture coach 1150 can revise any output that has received before output being sent to application 1 152.

Figure 12 A and 12B have described can confirm that from it postural training is suitable example filter output.Each curve map charts with the time to user's hand distance, like the output from " car steering " posture filter.

Figure 12 A describes the exemplary hand distance in unskilled user's the certain hour, this unskilled user the posture that will be performed needed keep roughly unified hand apart from aspect have any problem.Can confirm to help to be fit to from this output.Equally, when hand distance must be positioned at given range, if the user keeps roughly unified hand distance, and if this roughly unified distance is positioned on the given range or under, then still can confirm to help to be fit to.

Figure 12 B describes the exemplary hand distance in the certain hour of skilled user, and this skilled user can be kept roughly unified hand distance as the posture that will be performed is needed.Although the hand distance is not constant, filter can allow this variation, as long as it is enough little.

Conclusion

Although combined each preferred aspect shown in the drawings to describe the present invention, be appreciated that and use other similar aspect or can make amendment or add said aspect and carry out identical function of the present invention and do not break away from the present invention.Therefore, the present invention should not only limit to any single aspect, but should in according to the range of appended claims and scope, explain.For example, various process available hardware described herein or software or both combinations realize.Therefore, the method and apparatus of disclosed each embodiment or its some aspect or part can adopt the form that is included in such as the program code in the tangible mediums such as floppy disk, CD-ROM, hard disk drive or any other machinable medium (that is instruction).When program code is loaded into when carrying out such as machines such as computers and by it, this machine becomes the device that is configured to implement disclosed each embodiment.Except the concrete realization of here clearly setting forth, consider specification disclosed herein, others and realization are with apparent to one skilled in the art.Specification with shown in realize being intended to only be considered to example.

Claims (15)

1. method that is used to provide the help of the posture of carrying out about the user comprises:

The data that reception is caught by capture device (20), the posture (518) (1102) that said data are carried out corresponding to the user;

Analyze said data with produce with said data whether corresponding to the corresponding output of the posture of system identification (1104);

Confirm that from said output said user unlikely correctly carries out the posture (1106) of said system identification;

Confirm that from said output said user might want to carry out the posture of said system identification (1108); And

The help (1110) of the posture of relevant said user's execution is provided.

2. the method for claim 1 is characterized in that, offering help comprises:

Adjust said output; And

To send to and the corresponding application of the posture of said system identification through the output of adjustment.

3. method as claimed in claim 2 is characterized in that, adjusts said output and comprises:

Improve output and correctly carry out the corresponding possibility of posture that said system carries out with the user.

4. method as claimed in claim 2 is characterized in that, also comprises:

Before the said output of adjustment, confirm that said output can be adjusted.

5. the method for claim 1 is characterized in that, said output comprises level of confidence.

6. method as claimed in claim 5 is characterized in that, when said level of confidence was lower than threshold value, the user unlikely correctly carried out the posture of said system identification.

7. whether the method for claim 1 is characterized in that, analyze said data and also comprise corresponding to the corresponding output of the posture of said system identification to produce with data:

Whether also comprise to produce by the said data of filter analysis with said data corresponding to the corresponding output of the posture of said system identification:

Offer help and comprise:

Replace said filter with second filter.

8. the method for claim 1 is characterized in that, offering help comprises:

Hang up the corresponding application of carrying out with said user of posture.

9. the method for claim 1 is characterized in that, offering help comprises:

Show the demonstration of posture of expression and the said system identification of the posture that said user carries out.

10. method as claimed in claim 9 is characterized in that, also comprises:

Show the difference between the demonstration of posture of posture that said user carries out and said system identification.

11. a system that is used to provide the help of the posture of carrying out about the user comprises:

Processor (259);

Reception is by the assembly of the data of camera seizure, the posture (1102) that said data are carried out corresponding to the user;

Analyze said data with produce with said data whether corresponding to the assembly (1104) of the corresponding output of posture of system identification;

Confirm that from said output said user unlikely correctly carries out the assembly (1106) of the posture of said system identification;

Confirm that from said output said user might want to carry out the assembly (1108) of the posture of said system identification; And

The assembly (1110) of the help of the posture of carrying out about said user is provided.

12. system as claimed in claim 11 is characterized in that, the assembly of offering help also comprises:

Loosen the tolerance levels that is associated with the said assembly of analyzing said data.

13. system as claimed in claim 11 is characterized in that, the assembly of offering help also comprises:

Adjust the assembly of said output; And

To send to the assembly of the corresponding application of carrying out with said user of posture through the output of adjustment.

14. system as claimed in claim 13 is characterized in that, the assembly of adjusting said output comprises:

Improve the assembly of the response of the assembly of analyzing said data.

15. system as claimed in claim 11 is characterized in that, said output comprises level of confidence.

Images