CN114905514B

CN114905514B - Human skill learning method and system for outer limb grasping control

Info

Publication number: CN114905514B
Application number: CN202210582105.7A
Authority: CN
Inventors: 李可; 胡元栋; 李光林; 魏娜
Original assignee: Shandong University
Current assignee: Shandong University
Priority date: 2022-05-26
Filing date: 2022-05-26
Publication date: 2023-07-28
Anticipated expiration: 2042-05-26
Also published as: CN114905514A

Abstract

The invention provides a human skill learning method and system for outer limb grasping control, comprising the following steps: acquiring gesture information and motion information in the process of grasping the human arms and generating a motion track; based on teaching learning, a control model of human arm movement and outer limb movement is established, and the outer limb simulates the movement track of human arm through the control model; the method comprises the steps of obtaining touch information in the process of grasping an outer limb and feeding back the touch information to a human arm, wherein the human arm adjusts grasping postures according to the touch information fed back by the outer limb; and learning a plurality of motion tracks generated in the process of grasping the human arms to obtain learning tracks, and transmitting the learning tracks and corresponding posture information to the outer limbs so that the outer limbs can adapt to environment transformation.

Description

Human skill learning method and system for outer limb grasping control

Technical Field

The invention belongs to the technical field related to virtual reality, and particularly relates to a human skill learning method and system for outer limb grasping control.

Background

The statements in this section merely provide background information related to the present disclosure and may not necessarily constitute prior art.

The outer limb robot is novel wearable human body auxiliary equipment, and can improve the capabilities of human body activity, perception, operation and the like through mutual assistance with a human body. The device integrates the advantages of a wearable robot and a cooperative robot, can greatly enhance the movement range and the capability of a human body by depending on the structure independent of the limbs of the human body, breaks through the limitation of the space of the movement range and can move along with the human body. The use of outer limb robots to replace some of the simple, repetitive work in human production and life has become increasingly widespread in recent years, such as automobile assembly, food packaging, and medical assistance, among others.

Teaching learning (Learning from Demonstration, lfD) is an effective way to simplify robotics learning strategies that are currently in widespread use, especially for users whose computer programming capabilities are not very strong. The teaching learning can greatly simplify the complexity of programming, so that the robot can well obtain the expected task performance in the process of simulating human behaviors, and the key is to process the human motion trail, find a proper method to migrate to the robot system and establish a good control model.

How to effectively obtain the motion trail of the human body is a primary task of teaching and learning, and the following methods are mainly adopted at present: inertial measurement units that measure linear acceleration and angular velocity based on optical detection of passive, active marker and marker-less systems rely on mechanical devices to measure the relative joint angle of the extremities, and use of magnetic and acoustic sensors, etc. Compared with optical detection, the inertial measurement unit is low in cost, has no shielding problem, and does not need to build complex basic equipment before use; compared with mechanical equipment, the device is not constrained by the rigidity of the equipment; compared with sensitive elements such as a magnetic sensor, the device has low complexity and is not interfered by external environments such as temperature and humidity. Inertial measurement units are therefore often chosen as the means of recording the motion trajectories of the human body, but most of them are used in a way that each joint is measured and processed individually, and we wish to find a way to simplify the use. Meanwhile, the current teaching learning paradigm is mostly dominated by subjective feelings of people, feedback interacting with the environment is lacked, and proper feedback adjustment means are found to enable people to conduct skill demonstration better.

Disclosure of Invention

In order to solve the above problems, the present invention provides a human skill learning method and system for controlling grasping of an outer limb, which allows a user to learn teaching by using data gloves and transfer human grasping skills to an outer limb robot. Data are acquired by using various sensors, and the tactile information is utilized to combine with vibration feedback to adjust, so that more personified grasping skill learning is realized

In order to achieve the above object, a first aspect of the present invention provides the following technical solutions: a human skill learning method for outer limb grip control, comprising:

s1: acquiring gesture information and motion information in the process of grasping the human arms and generating a motion track;

s2: based on teaching learning, a control model of human arm movement and outer limb movement is established, and the outer limb simulates the movement of human arms through the control model;

s3: the method comprises the steps of obtaining touch information in the process of grasping an outer limb, feeding back the touch information to a human arm, and adjusting the gesture of the human arm according to the touch information fed back by the outer limb;

s4: repeating S1-S3 until the outer limb completes stable grasping;

s5: and learning a plurality of motion tracks generated in the process of grasping the human arms to obtain learning tracks, and transmitting the learning tracks and corresponding posture information to the outer limbs so that the outer limbs can adapt to environment transformation.

Further, in the step S1, the step of obtaining, by the data glove, motion information of the human arm during the grasping process specifically includes:

the bending sensor is arranged on the bending side of the five fingers of the data glove and is used for acquiring bending information in the movement of the arms of the human body;

and the inertial measurement unit is arranged on the back of the data glove and used for acquiring the space motion information in the motion of the human arm.

Further, the data glove further comprises a vibration motor disposed at the five fingertips of the data glove for generating haptic feedback of the outer limb during grasping.

Further, in the step S2, according to the interval of mapping all measured values including the maximum and minimum curvatures acquired by the bending sensor to [0,1], the finger joint position at the tail end of the outer limb is kept vertical and the completely bent posture is mapped to the interval of [0,1], the corresponding relationship between the curvature of the arm of the human body and the curvature of the outer limb is established based on the expression of the same state, and the outer limb simulates the curvature of the arm of the human body according to the corresponding relationship;

and establishing a mapping relation between the spatial motion data measured by the inertial measurement unit and the spatial motion of the tail end of the outer limb, regulating the mapping proportion by a learning proportion regulating mechanism, and simulating the spatial motion of the arm of the human body by the outer limb through the mapping relation regulated by the learning proportion.

Further, taking the time of keeping the wearing data glove horizontal stable as the starting time of motion demonstration, integrating acceleration of the x, y and z axes of the position coordinates in the same time interval from the starting time to obtain displacement variation delta x, and integrating angular velocities of the phi, theta and gamma axes of the direction coordinates to obtain angle variation delta a;

the delta x and delta a respectively correspond to the position change delta x of the coordinates in the Cartesian space within the equal time interval of the outer limb terminal _r Amount of change in direction Δa _r The learning proportion is adjusted as follows:

Δx _r ＝αΔx

Δa _r ＝βΔa

wherein alpha, beta E [0, n ], n is a constant.

Further, the step S3 specifically includes:

establishing a corresponding relation based on the contact force measured by the tail end of the outer limb and the vibration degree of the vibration motor of the data glove fingertip;

respectively collecting the contact force of five fingers when the human arms grasp an object, and taking the contact force as an outer limb contact force threshold value;

judging whether the contact force measured by the outer limb tail end exceeds a threshold value or not;

if the contact force measured by the outer limb tail end exceeds a threshold value, controlling the vibration degree of the vibration motor according to the established corresponding relation;

the human arm adjusts the finger gesture according to the vibration degree of the vibration motor.

Further, in the step S5, learning a plurality of motion trajectories generated during the process of grasping the arm of the human body based on the probabilistic motion primitive to obtain a learning trajectory, and transmitting the learning trajectory to the outer limb, specifically including:

a linear model of the demonstration track is represented by a basis function matrix and a weight vector;

calculating a weight vector by using a least square method according to the linear model;

obtaining track probability distribution according to weight vector calculation;

combining variable information in an external environment to obtain the joint distribution of environment variables and weights;

calculating probability distribution of the environment variable based on joint distribution of the environment variable and the weight, and obtaining an expected track according to probability analysis of the environment variable;

and comparing the obtained expected track with the demonstration track, wherein the expected track with the maximum similarity is the learning track of the outer limb.

A second aspect of the present invention provides a human skill learning system for external limb grip control, comprising:

the data glove is used for acquiring the motion information of the human arms;

the outer limb is provided with a manipulator with multiple degrees of freedom, and the tail ends of the five fingers of the manipulator are respectively provided with a sensor for detecting touch information;

the feedback module is used for feeding back the tactile information of the manipulator of the outer limb to the human arm, and the human arm adjusts the posture of the human arm according to the tactile feedback information of the outer limb;

the control module is used for establishing a control model of the human arm and the outer limb based on teaching learning, and enabling the outer limb to simulate the movement and the gesture of the human arm through the control model;

the learning module is used for learning a plurality of motion tracks generated in the process of grasping the human arms to obtain learning tracks, and transmitting the learning tracks to the outer limbs so that the outer limbs can adapt to environment transformation.

A third aspect of the invention provides a computer readable storage medium storing computer instructions which, when executed by a processor, perform the steps of the method described above.

A fourth aspect of the invention provides an electronic device comprising a memory and a processor and computer instructions stored on the memory and running on the processor, which when executed by the processor, perform the steps of the method described above.

The beneficial effects of the invention are as follows:

according to the method, the user uses the data glove to conduct teaching learning, the human grasping skills are transferred to the outer limb robot, various sensors are used for collecting data, tactile information is used for adjusting in combination with vibration feedback, more anthropomorphic grasping skill learning is achieved, and a better suggestion is provided for the development of a later robot learning strategy.

Drawings

The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the invention.

FIG. 1 is a schematic diagram of a learning strategy for the outer limb as a whole;

FIG. 2 is a schematic illustration of a teaching flow;

FIG. 3 is a schematic illustration of an outer limb control strategy flow;

FIG. 4 (a) is a schematic back view of a data glove;

FIG. 4 (b) is a schematic front view of a data glove;

FIG. 4 (c) is a haptic value versus time schematic and a vibration level versus time schematic, respectively;

fig. 5 is a schematic view of the environment of the human arm and outer limb of the present invention.

Detailed Description

The invention will be further described with reference to the drawings and examples.

It should be noted that the following detailed description is illustrative and is intended to provide further explanation of the invention. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

The invention has the overall conception that:

firstly, a user wears the data glove, starts to collect data after the data glove is kept horizontal and stable, and sends the data to the outer limb, when a human hand moves in space, the outer limb also moves along with the movement, when the tail end of the outer limb contacts an object, the touch information is fed back, the vibrating motor of the finger tip of the human hand generates corresponding vibration according to the touch feedback of the outer limb, so that the human hand makes corresponding posture adjustment according to the vibration feedback, and finally, the whole process that the outer limb imitates the human body to finish the grasping is realized.

Example 1

As shown in fig. 1 to 5, the present embodiment provides a human skill learning method for outer limb grip control, comprising the steps of:

s4: repeating S1-S3 until the outer limb completes stable grasping;

In the step S1, in order to effectively obtain a motion track of human body grasping, a data glove meeting requirements is designed, as shown in fig. 4 (a) -4 (b), wherein a is a vibrating motor posted by fingertips, b is a bending sensor posted by five fingers, c is an IMU placed on the back, d is an outer limb mechanical arm, and e is a five finger hand at the tail end of the mechanical arm; 1 represents Cartesian coordinate information of the tail end of the outer limb mechanical arm, 2 represents tactile information of the tail end of the five-finger hand, and 3 represents posture information of the five-finger hand.

In this embodiment, bending sensors are respectively posted on the five-finger bending sides of the data glove to measure bending information of a human hand during movement.

A vibrating motor is posted at the positions of the fingertips of the five fingers of the data glove, and vibration is generated through the magnitude of contact force in the process of outer limb movement, so that a demonstrator can make corresponding posture change according to the vibration degree.

An inertial measurement unit (Inertial measurement unit, IMU) is arranged on the back of the data glove and is used for measuring and recording the spatial movement information of the human hand in the movement process, wherein the spatial movement information comprises acceleration, angular velocity and angle information.

In the step S2, the outer limb learns the motion trail of the human body, and the data measured by the data glove is used as the input of learning, specifically including learning of the bending information of the human hand and learning of the spatial motion information of the human arm.

For the study of the bending of the human hand, the measured value of the bending sensor including the maximum value and the minimum value is mapped to the [0,1] interval, the same gesture data for keeping the finger joint position at the tail end of the outer limb vertical and completely bending is mapped to the [0,1] interval, and after the bending degree of the human hand and the bending degree of the outer limb are expressed in the same state, the bending state of the human hand can be mapped to the tail end of the outer limb.

For learning of human arm space motion information, it is assumed that human hand motions in a proper range can be summarized as all motion spaces for outer limbs to complete grasping learning:

firstly, taking the moment of keeping the level stable after the hand wears the data glove as the starting moment of motion demonstration, integrating the acceleration of the position coordinates x, y and z in the same time interval from the starting moment to obtain displacement variation delta x, and integrating the angular velocity of the direction coordinates phi, theta and gamma to obtain angle variation delta a. Δx and Δa respectively correspond to the position change Δx of the coordinates in the Cartesian space in the equal time interval of the outer limb end _r Amount of change in direction Δa _r 。

Adjusting the position change delta x of the outer limb by using the learning proportion _r Amount of change in direction Δa _r And (3) performing proportion adjustment:

Δx _r ＝αΔx (1)

Δa _r ＝βΔa (2)

wherein alpha, beta epsilon [0, n ], n is a constant, and the specific numerical value is selected by the user according to the actual situation.

Different values for alpha and beta can be selected for different requirements: in the middle and front period of the exercise, the hand is required to move in a proper range, the outer limb moves in a larger range, and alpha and beta can both be larger values; in the later period of movement, the outer limb is required to move in a smaller range, and alpha and beta can both be smaller values; in the movement process, the hand movement may reach a position where the user continues to move unchanged, if the horizontal extension is inconvenient, α can be increased, if the horizontal extension is inconvenient to incline to one side, β can be increased, and the like.

As shown in fig. 2 and fig. 4 (c), in S3, the grasping demonstration of the outer limb is performed by obtaining the haptic feedback, and by finding the appropriate correspondence between the magnitude of the contact force of the outer limb end and the vibration intensity of the motor of the data glove, the hand of the user can adjust the arm posture in real time according to the vibration feedback, so that the outer limb can be grasped more stably.

In this embodiment, first, the contact forces of the five fingers when the human hand grips the object are collected and recordedWherein i is E [1,5 ]]Representing 5 fingers, then taking the fingers as a threshold value of external limb contact, and establishing a corresponding relation between a force value of contact force detected by the external limb and the vibration degree of a vibration motor of the glove fingertip, wherein the relation is as shown in the following formula:

wherein,,for detecting the contact force value of the outer limb, alpha is a proportional gain term, and the value is constant, V _i Is the degree of vibration.

After the contact force is generated, vibration starts, when the deviation between the detected force value and the threshold value is large, the vibration degree felt by the human hand is increased, then the human hand makes corresponding adjustment of the gesture to reduce the vibration, when the vibration degree is tiny and even 0, stable grabbing is finished, single teaching is finished, and then all the taught tracks are learned and reproduced according to the selected environment variables.

As shown in fig. 3 and 5, 4 is an outer limb manipulator of 7DO-F, 5 is a data glove of design, and 6 is a cylindrical object to be grasped. In S5, the recorded multiple trajectories of the human hand are learned by using the ProMPs probability motion primitive, and the time step of each trajectory is denoted by T and is denoted as the motion time.

S5-1: the linear model of the presentation trajectory τ is represented by a basis function matrix Φ and a weight vector ω:

τ＝Φω+ε (4)

wherein epsilon is track noise, epsilon is the mean value of 0 and the variance is delta ² Is a gaussian noise of (c).

The basis function Φ consists of M basis functions, using a normalized gaussian function as the basis function, the i-th basis function is represented by the center and variance of the gaussian function as follows:

wherein b _i For a defined function, t is the time step, c _i Is the center of the gaussian function, h is the variance of the gaussian function.

S5-2: the weight vector ω is calculated using the least square method:

ω＝(Φ ^T Φ) ^-1 Φ ^T τ (7)

the probability distribution p (ω) over the weight vector ω is also a gaussian distribution:

p(ω)＝N(μ _ω ,Σ _w ) (8)

wherein mu _ω Is the average value of Σ _ω Is the variance.

S5-3: the trajectory distribution p (τ) is calculated by marginalizing the weight vector ω:

p(τ)＝fp(τ|ω)p(ω)dω (9)

because p (ω) is a gaussian distribution, p (τ) can be reduced to the following formula:

p(τ)＝N(μ _τ ,Σ _τ ) (10)

wherein mu _τ ＝Φμ _ω ，Σ _τ ＝δ ² +ΦΣ _ω Φ ^T 。

In this embodiment, it is desirable that the learning result can adapt to the task that the external environment is a variable, such that the environment variable may be a plurality of attributes of the object, such as the weight of the object, the position of the object, etc., and the method used is to learn the joint distribution p (c, ω) of the environment and the weight, c is defined as the environment variable:

p(c,w)＝N(μ _n ,Σ _n ) (11)

wherein,,

μ _c is the average value of the Gaussian distribution of the environmental variable, Σ _n For variance of joint distribution of environment and weight, wherein Σ _ωω 、Σ _ωc 、Σ _cω 、Σ _cc The variances of the multiplication matrices indicated by the subscripts, respectively.

For a certain environment variable c, the probability distribution can be calculated as:

p(ω|c)＝N(μ _ω|c ,Σ _ω|c ) (12)

wherein,,

the following equation is then solved to calculate the conditional probability distribution:

p(τ|c)＝N(μ _τ|c ,Σ _τ|c ) (13)

wherein mu _τ|c ＝Φμ _ω|c

Σ _τ|c ＝δ ² +ΦΣ _ω|c Φ ^T

Φ ^T Transpose of the matrix of basis functions phi, the mean value mu thus obtained _τ|c The value of (2) is the desired trajectory.

Through the scheme, the optimal learning track can be obtained from a plurality of demonstration tracks of the same environment, and the method can also adapt to the track which can be used for a new environment when the environment information changes.

Optimal track mu obtained by learning grabbing gesture of hand _τ|c Comparing with the demonstration track tau in similarity, searching the demonstration gesture of the track with the maximum similarity as the corresponding grabbing gesture of the learning track, wherein the comparison method comprises the following steps:

wherein n is the track length, alpha _k 、β _k Mu respectively _τ|c And points on τ.

Taking the gripping gesture corresponding to the obtained most similar track as the gripping gesture of the new track, and finally taking the track mu _τ|c And grip posture x _g And sending the artificial limb to the outer limb so that the outer limb can finish the artificial grasping.

The invention enables a user to learn teaching by using the data glove and transfer human grasping skills to the outer limb robot. The data are collected by using various sensors, and the tactile information is combined with vibration feedback to adjust, so that more personified grasping skill learning is realized.

Example two

The present embodiment provides a human skill learning system for outer limb grip control, comprising:

the data glove is used for acquiring the motion information of the human arms;

Example III

The present embodiment provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps in a human skill learning method for outer limb grip control as described in embodiment one.

Example IV

The present embodiment provides a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of a human skill learning for external limb gripping control as described in embodiment one when the program is executed.

It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, magnetic disk storage, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The above description is only of the preferred embodiments of the present invention and is not intended to limit the present invention, but various modifications and variations can be made to the present invention by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A human skill learning method for outer limb grip control, comprising:

s2: based on teaching learning, a control model of human arm movement and outer limb movement is established, and the outer limb simulates the movement track of human arm through the control model;

s3: the method comprises the steps of obtaining touch information in the process of grasping an outer limb and feeding back the touch information to a human arm, wherein the human arm adjusts grasping postures according to the touch information fed back by the outer limb;

s4: repeating S1-S3 until the outer limb completes stable grasping;

s5: learning a plurality of motion tracks generated in the process of grasping the human arms to obtain learning tracks, and transmitting the learning tracks and corresponding posture information to the outer limbs so that the outer limbs can adapt to environment transformation;

in the step S2, the time when the wearing data glove keeps horizontal stable is taken as the starting time of motion demonstration, and the positions in the same time interval are seated from the starting timeThe acceleration of the x, y and z axes is integrated to obtain the displacement variationDirection coordinate->、/>、/>The angular velocity of the shaft is integrated to obtain the angular variation +.>；

Will be、/>The position change of the coordinates in the Cartesian space respectively corresponding to the outer limb terminal in equal time intervalsDirection change amount->The learning proportion is adjusted as follows:

wherein,,，/>is a constant; the outer limb simulates the spatial movement of the human arm through learning the mapping relation of proportion adjustment;

in the S5, learning a plurality of motion trajectories generated during the process of grasping the arm of the human body based on the probabilistic motion primitive to obtain a learning trajectory, and transmitting the learning trajectory to the outer limb, specifically including:

2. A human skill learning method for external limb gripping control as claimed in claim 1, comprising: in the step S1, motion information of a human arm in a grasping process is obtained through a data glove, which specifically includes:

3. A human skill learning method for external limb gripping control according to claim 2, wherein the data glove further comprises a vibration motor disposed at the fingertips of the data glove, the vibration motor being adapted to generate haptic feedback of the external limb during gripping.

4. The human skill learning method for external limb grasping control according to claim 2, wherein in the S2, according to all measured values including maximum and minimum curvatures acquired by the bending sensor mapped to the interval of [0,1], the posture of the external limb end finger joint position kept vertical and completely bent is all mapped to the interval of [0,1], a correspondence of the curvature of the human arm and the curvature of the external limb is established based on the expression of the same state, and the external limb mimics the curvature of the human arm according to the correspondence;

5. A human skill learning method for external limb gripping control according to claim 1, wherein in S3 specifically comprises:

6. A human skill learning system for outer limb grip control, comprising:

the data glove is used for acquiring the motion information of the human arms;

the control module is used for establishing a control model of the human arm and the outer limb based on teaching learning, and enabling the outer limb to simulate the movement and the gesture of the human arm through the control model; in the step S2, the time when the wearing data glove keeps horizontal stable is taken as the starting time of motion demonstration, and the acceleration of the position coordinates x, y and z in the same time interval is integrated from the starting time to obtain the displacement variationDirection coordinate->、/>、/>The angular velocity of the shaft is integrated to obtain the angular variation +.>；

the learning module is used for learning a plurality of motion trajectories generated in the process of grasping the human arms to obtain a learning trajectory, and transmitting the learning trajectory to the outer limb, so that the outer limb can adapt to environmental transformation, and the learning module learns the plurality of motion trajectories generated in the process of grasping the human arms based on the probability motion primitives to obtain the learning trajectory, and transmitting the learning trajectory to the outer limb, and specifically comprises the following steps:

7. A computer readable storage medium having stored thereon a computer program, which when executed by a processor performs the steps of a human skill learning method for outer limb grip control as claimed in any of claims 1-5.

8. A processing device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor performs the steps in learning human skills for external limb gripping control as claimed in any of claims 1 to 5 when the program is executed.