JP2557789B2 - Human interface device - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a human interface device, for example, a virtual arm which is superior in operability and naturalness in place of a mouse and a trackball which are widely used at present in a general use environment of a computer. The present invention relates to a human interface device that realizes a virtual body that functions as an alter ego of the body in a computer environment such as a virtual hand.

[0002]

2. Description of the Related Art A conventional human interface device between a computer and an operator is a mouse, a joystick, a three-dimensional mouse using a trackball or a cable, a three-dimensional magnetic position measuring device, a goniometer, a master arm, a data glove, a data suit, etc. As represented,
Physical changes that occur as a result of movements of hands, arms, trunks, etc., which are part or whole of the body, are measured by various means as positions and forces, and they are measured in various ways, such as the Casa, hands and arms in the computer world. Is reflected in the movement of.

A simple two-dimensional position change measuring device such as a mouse provides a human interface which is much more natural than a conventional keyboard, but an interface device only for measuring a position has data glove and data. Even if you increase the degree of freedom like a suit, the feeling of use is unnatural, force feedback is not applied, and you cannot control the speed, acceleration, and force, so you will feel a lot of fatigue during use.

On the other hand, a master robot arm and a goniometer of a tele-operation system capable of measuring force are very large and expensive as mechanical devices, and are extremely dangerous at the time of runaway. It is difficult to be established as a general-purpose human interface device because of its safety, convenience, price, space utility, etc.

[0005]

An interface device with a computer using a myoelectric signal has been proposed in Japanese Patent Laid-Open No. 172322/1988, for example. This is to compare the myoelectric signal pattern obtained during any movement of the human body with a plurality of reference patterns having preset specific information, and input the data of the information set to the closest reference pattern. Allows desired data input without operating the keyboard. However, with such a classification method by pattern recognition, a limited number of discrete data such as alphabets and decimal motions can be input, but a large number of degrees of freedom can be provided instead of a mouse or data glove. There is a drawback that it is not possible to construct a human interface device that continuously changes the vector input value that it has.

Therefore, the main object of the present invention is to measure the activity of the brain and the nervous system, which is the cause of the movement, by some non-invasive method, and based on the measurement, a part of the body or The objective is to provide a human interface device that naturally controls the virtual body that is the entire model as if it were its own alternator, and can freely control not only position but also force, velocity, and acceleration.

Another object of the present invention is safety, convenience,
It is to provide a human interface device that is similar to conventional mice and data gloves in terms of price and space utility and is far superior to mechanical force measurement interface devices.

[0008]

The invention according to claim 1 is
A human interface device for inputting necessary information to a computer by an operator's operation, which includes a surface electrode worn on the operator's body and a surface electrode for detecting a signal according to the activities of a plurality of muscles of the operator. Depending on the output of this nonlinear dynamics model and the nonlinear dynamics model that converts each detection output of the electrode into a signal representing pseudo tension corresponding to the tension generated by each muscle and estimates the motion trajectory and force trajectory based on that signal And a virtual body operating on the estimated motion trajectory or force trajectory.

[0009]

In the invention according to claim 2, the nonlinear dynamics model according to claim 1 includes a neural circuit, and this neural circuit filters signals corresponding to the activities of a plurality of muscles inputted from the surface electrodes, A preceding circuit that outputs a signal representing a pseudo tension corresponding to the tension generated by the muscle,
It includes a latter-stage circuit composed of a plurality of layers for receiving signals from the former-stage circuit and estimating pseudo tension, torque from the position and velocity, force exerted on the outside and acceleration signals.

In the invention according to claim 3, the neural circuit according to claim 2 learns by using a predetermined learning rule for determining a parameter.

In the invention according to claim 4, the neural circuit according to claim 3 substitutes the estimated torque into a non-linear equation for describing the dynamics of the body and performs numerical integration to estimate the trajectory, or the trajectory is estimated. The acceleration is numerically integrated by cyclic coupling to estimate the trajectories and control the virtual body.

In the invention according to claim 5, the neural circuit according to claim 4 is divided into portions for estimating acceleration, torque, force, etc. during rest and movement from position and velocity.

[0014]

The human interface device according to the present invention detects a signal corresponding to the activity of a plurality of muscles of the operator by the surface electrode and converts the detected signal into a signal representing a pseudo tension corresponding to the tension generated by each muscle. After that, the motion trajectory and the force trajectory are estimated by the nonlinear dynamics model, and the virtual body is operated on the estimated motion trajectory and the force trajectory.

[0015]

1 is a schematic block diagram of an embodiment of the present invention. In FIG. 1, the surface electrode 1 detects a signal corresponding to the muscle activity of the operator, for example, a myoelectric signal, and is attached to the operator's body. The detected myoelectric signal is given to the myoelectric signal amplifier 2, amplified, and given to the filter 3. The filter 3 filters the amplified myoelectric signal, converts it into a signal representing pseudo tension, and supplies the signal to the body dynamics model 4. The body dynamics model 4 uses, for example, a non-linear neural circuit, estimates a motion trajectory and a force trajectory based on a signal representing a pseudo tension, and gives them to the computer 5. The computer 5 controls the virtual body 6 according to the pattern of the motion trajectory and the force trajectory given from the body dynamics model 4 based on a built-in software program.

FIG. 2 is a flow chart for explaining the operation of one embodiment of the present invention. Next, the operation of FIG. 1 will be described with reference to FIG. When the surface electrode 1 is attached to the user's body and the user moves the body or generates a force, a myoelectric signal is given to the myoelectric signal amplifier 2,
Amplified. The amplified myoelectric signal is applied to the filter 3 and the pseudo tension is applied to the body dynamics model 4. The body dynamics model estimates the trajectory of force and the force trajectory based on the pseudo tension. This estimated movement pattern is given to the computer 5, and based on a program stored therein, a movement simulation is performed based on the estimated trajectory of the virtual body, and the result is displayed on the display. Then, if the use has not been completed, the operation returns to the initial state and the above operation is repeated.

FIG. 3 is a diagram showing the rough positions of ten muscles for measuring myoelectric signals and surface electrodes. In one embodiment of the invention, myoelectric signals are measured from the ten muscles of the arm. That is, as shown in FIG. 3, the extensor muscle of the shoulder joint, the anterior deltoid muscle DLS as the flexor muscle, the upper delta muscle DLA, the posterior deltoid muscle DLC, the pectoralis major muscle PMJ, the great oval muscle TEM, and the two joint muscles of the shoulder and elbow. Biceps long head BIL, triceps long head TR as
L, extensor muscle of elbow joint, brachial muscle BRC as flexor muscle, triceps medial temporal muscle TRM, and triceps lateral temporal muscle TRA. FIG.
The surface electrode shown in FIG. 2 uses, for example, a pair of silver-silver chloride surface electrodes to induce the surface myoelectric potential in a bipolar manner, and the myoelectric signal amplifier
After being differentially amplified by, the myoelectric signal is sampled by 12 bits by a sampling clock of 2 kHz. The surface electrode 1 has a diameter of about 10 mm and is attached to the body with a distance between the electrodes of 15 mm along the muscle fiber. After this signal is full-wave rectified, the average (EMGave) for every 10 points is taken, and the moving average for every 5 points is taken as in the equation (1) to smooth the signal. This signal is the average myoelectric signal (EMGm
Call a).

[0018]

[Equation 1]

Therefore, 200 Hz sampling is performed. In voluntary movements, control signals from the upper center are transmitted to each muscle via α motor neurons of the spinal cord, causing the muscles to contract, producing torque in each joint, and the desired movement occurs. When a person tries to measure the control signal from the upper center, the non-invasive surface electromyographic signal is considered. The signal obtained by measuring the excitatory potential of the membrane during muscle contraction on the skin is the surface myoelectric signal. The surface myoelectric signal is not only an optional signal from the central nervous system but also a signal transmitted to the muscle including peripheral feedback, which is temporally and spatially superimposed.

A signal transmitted to muscle including not only an optional signal from the central nervous system but also peripheral feedback is called a motion command. Surface EMG signals are suitable for analyzing body movements because they record muscle activity in a unit. The tension generated by a muscle increases as the muscle length increases even with the same motion command. The manner of this change is non-linear and is called the length-tension curve. Also,
Even with the same motion command, the tension decreases as the muscle shortening speed increases. The manner of this change is also non-linear, and is called a shortening speed-tension curve.

In the case of humans, it is not possible to directly take out the muscle and measure the tension. Therefore, it has been attempted to estimate the tension generated by the muscle from the myoelectric signal. As signal processing of the myoelectric signal, there are a method of averaging for a fixed time after full-wave rectification, a method of passing through a low-pass filter, a method of integrating, and the like. The method using a low-pass filter often estimates the muscle tension when the muscle contracts slowly. However, in the case of fast expansion and contraction, it cannot be accurately estimated, and the nonlinear characteristics of muscles such as the length-tension curve cannot be considered. However, the output signal that has passed through the low-pass filter is called pseudo tension, and α
Since it is expected to reflect the firing frequency of motor neurons, it is considered to be close to the true tension. It is known that this low-pass filter is sufficient for a secondary system. That is, the transfer function is expressed by the following equation (2).

[0022]

[Equation 2]

This impulse response can be expressed by the equation (3), and when the input u and the output y are realized by the FIR filter, the equation (4) is obtained.

[0024]

(Equation 3)

This coefficient can be obtained by optimization by estimating the torque from the surface myoelectric signal in isometric contraction.

FIG. 4 shows a four-layer perceptron showing an example of the body dynamics model 4 shown in FIG. 1, and FIG. 5 shows a two-layer perceptron having a module structure for improving the accuracy of the basic circuit of FIG. FIG. 6 is a block diagram up to layers, and FIG. 6 is a diagram showing a nonlinear dynamical system model in which the network of FIG. 5 is recurrently connected to infer a motion trajectory from an myoelectric signal.

Next, the body dynamics model 4 shown in FIG. 1 will be specifically described with reference to FIGS. 4 to 6.
In FIG. 4, the body dynamics model 4 is the first layer 4
1, the second layer 42, the third layer 43, and the fourth layer 44, and the first layer 41 includes the myoelectric signal EMG from the past to the present.
ma (n) ... EMGma (n−N + 1) is input.
The FIR filter 3 described above is realized by the conversion from the first layer 41 to the second layer 42, and the pseudo tension 11 is applied to the second layer 42 as the output of the filter 3. Further, in the second layer 42, the shoulder joint angle θ _s1 , the elbow joint angle θ _e1 ,
The shoulder joint angular velocity θ _s2 and the elbow joint angular velocity θ _e2 are input. The second layer 42 and the third layer 43 are connected by a synapse connection 45, and the third layer 43 and the fourth layer 44 are connected by a synapse connection 46.

The body dynamics model 4 shown in FIG. 4 estimates a motion trajectory and a force trajectory according to the following principle.
That is, since the muscle only generates tension, it is necessary to arrange a pair of muscles that act antagonistically on both sides of the joint in order to change the angle of the joint. The muscles that extend the joints are called extensors and the muscles that bend are called flexors. The distance between the axis of rotation of the joint and the line of action of the muscle is the moment arm. When you bend and stretch your joints,
Since muscles are bent by the skin and bones, this distance is not constant and the relationship between torque and tension depends non-linearly on the joint angle. The joint torque is generated by the difference between the torques generated by the extensor and flexor muscles, and depends on the tension and the moment arm. This joint torque is expressed by the following equation (5).

[0029]

[Equation 4]

Where τ _s is the shoulder torque, τ _e is the elbow torque, θ _s1 is the shoulder joint angle, and θ _e1 is the elbow joint angle. α _is (θ), α _ie (θ) are the moment arms of the i-th muscle (0 for joints that are not involved in a single joint muscle), T _i is the muscle tension of the i-th muscle, and T _i ′ is the i-th muscle Represents the pseudo tension of a muscle.

The relationship between the joint torque and the trajectory in the two-joint motion within the plane follows the motion equation shown in the following equation (7).

[0032]

(Equation 5)

Here, τ _i , θ _i1 , θ _i2 , and θ _i3 represent driving torque, joint angle, joint angular velocity, and joint angular acceleration of each joint, and M _i , L _i , l _gi , I _i , and b _i. Represents the mass of each link, the length, the length from the joint to the center of mass, the moment of inertia around the joint, and the coefficient of viscous resistance.

It is considered that the angular acceleration of each joint is estimated from the surface myoelectric signal using a neural circuit model. The surface myoelectric signal is a signal in which the membrane excitation potential is superimposed temporally and spatially. As shown in FIG. 4, the pseudo tension can be calculated from the surface myoelectric signal by passing a second-order low-pass filter. The torque generated in each joint when a person is exercising cannot be measured in a natural state. Therefore, as described above, the torque is estimated from the trajectory by the inverse dynamics calculation using the equation of motion. However, it is necessary to estimate the required parameter values.

On the other hand, if the joint angular acceleration can be estimated directly from the pseudo tension, the trajectory can be generated without explicitly calculating the tension or torque. Therefore, in one embodiment of the present invention, the angular acceleration of each joint is estimated from the myoelectric signal, the joint angle, and the joint angular velocity using the network of the first layer 41 to the fourth layer 44. Since the pseudo tension does not include the non-linear property of the muscle, the non-linear property of the musculoskeletal system and the dynamics of the arm are acquired by learning between the second layer 42 and the fourth layer 44 of the network.

Even when the arm is stationary at a certain position,
There are various cases in which you can relax, stiffen the arm by activating the extensor and flexor muscles at the same time, and compare the tension generated by the muscles. However, the acceleration is 0 when the arm is stationary, and the difference cannot be understood by looking only at the acceleration. At this time, since the joint angle does not change, the shortening speed-tension curve is not affected.

On the contrary, when exercising, the motion command
Is affected by local feedback such as stretch reflex.
You. One network is stationary and moving
When estimating the acceleration at the time of
In other words, when it is almost stopped, it is moving
Since the value of acceleration is smaller than that when
It will be evaluated small. Therefore, the recurrent net
Accumulation of errors occurs when generating trajectories using a workpiece.
Therefore, the correct trajectory may not be generated. Consideration above
Incorporate the module structure shown in FIG.
Solve the problem by. That is, as shown in FIG.
2 expert networks 2
1, 22 divided into these expert networks
The outputs of 21, 22 are cut off by the gate network 23.
Can be replaced. Entering expert networks 21 and 22
The force includes muscle tension T and shoulder joint angle θ._s1， Elbow joint angle
Degree θ _e1And shoulder joint angular velocity θ_s2And elbow joint angular velocity θ_e2Is in
Applied to the gate network 23, the muscle tension
Force T and shoulder joint angle θ_s1And elbow joint angle θ_e1And shoulder joint
Angular velocity θ_s2Absolute value of, elbow joint angular velocity θ_e2The absolute value of
Shoulder joint angular acceleration θ_s3Absolute value of and elbow joint angular acceleration θ
_e3The absolute value of and is input.

As shown in FIG. 5, when the learning is successfully performed by using the two expert networks 21 and 22, the gate network 23 will select one of the expert networks 21 and 22 according to the input signal. . In the present case, the correct acceleration can be estimated when one of the two expert networks 21, 22 is stationary and the other is moving. In this way, since the gate network 23 determines how much one of the expert networks stochastically outputs the correct output depending on the input signal, all the outputs of the gate network 23 are positive and the sum is 1. is necessary. Therefore, as shown in the equation (7), the output g _j of the gate network 23 corresponding to the j-th expert network is normalized.

[0039]

(Equation 6)

Here, x _i is a value determined by the input signal of the gate network 23, and N is the number of outputs of the gate network 23. And the final output is
As shown in Expression (8), the output θ _i3 ′ of each expert network is multiplied by the output of the gate network 23 to obtain the sum.

[0041]

(Equation 7)

The individual expert networks perform learning by using the error back propagation of the equation (9) so as to maximize the log likelihood lnL.

[0043]

(Equation 8)

Since the acceleration estimation network can estimate the change (acceleration) in velocity after a unit time from the current myoelectric signal, joint angle and angular velocity, the joint after a unit time from the input joint angle and angular velocity and output angular velocity. Angle and angular velocity can be calculated. By repeating the calculation of the joint angle and angular velocity after the next unit time from the joint angle and angular velocity and the myoelectric signal at the next time, the myoelectric signals of the initial joint angle, angular velocity and total exercise time are repeated. The trajectory can be generated from only. The neural circuit model in this case is a recurrent type as shown in FIG.

In FIG. 6, the myoelectric signal EMG is model 3
Input to 1. The model 31 outputs an acceleration signal based on the myoelectric signal. This acceleration signal is added by the adder 32 to the output of the delay element 33 for one unit time, and the speed signal is output. This speed signal is input to the one-unit-time delay elements 33 and 34, and the output of the one-unit-time delay element 33 is given to the above-mentioned adder 32 and the model 3
It is fed back to 1. One unit time delay element 34
Of the output of the delay element 35 of one unit time by the adder 36.
Is added to the output of. The output of the adder 36 becomes a position signal, which is given to the delay element 35 for one unit time, and the output of the delay element 35 for one unit time is fed back to the model 31.

FIG. 7 shows the state of the trajectory when the virtual arm is moved from a certain point to a certain point using the model shown in FIG. FIG. 7 (a) shows the change over time in the position of the shoulder,
(B) shows the time change of the shoulder speed, and (c) shows the time change of the shoulder acceleration. However, the dotted line shows the actual arm movement, and the solid line shows the virtual arm movement. Figure 7
(D) shows the output of the two gate networks for the shoulders, the dotted line is in motion and the solid line is in stationary. Further, FIG. 7E shows a time change of the elbow position, FIG. 7F shows a time change of the elbow speed, and FIG. 7G shows a time change of the elbow acceleration. Even in this case,
The dotted line shows the actual arm movement, and the solid line shows the virtual arm movement. FIG. 7 (h) shows the output of the elbow gate network 23, where the dotted line is the motion and the solid line is the output when the robot is stationary.

As is apparent from FIGS. 7D and 7H,
Expert network 21 when stopped,
22 outputs with almost the same probability, and the expert network 22 mainly outputs when exercising. Also, there is no delay even when the gate network 23 is switched at the beginning and end of the exercise, and the change in acceleration is well estimated.

[0048]

As described above, according to the present invention, a signal corresponding to the activity of a plurality of muscles of an operator is detected, and the detected signal is converted into a signal representing a pseudo tension corresponding to the tension generated by each muscle. It transforms and estimates the motion trajectory and force trajectory based on the signal to control the virtual body, so the virtual body, which is a model of part or all of the body in the computer world, is as if It is possible to realize a human interface device that can be naturally controlled as if it were present and can freely manipulate not only position but also force, velocity, and acceleration. Moreover, in terms of safety, convenience, price, space utility, etc., it is possible to realize a device that is equivalent to the conventional mouse and data glove and is far superior to the mechanical force measurement interface device.

[Brief description of drawings]

FIG. 1 is a schematic block diagram of an embodiment of the present invention.

FIG. 2 is a flowchart for explaining the operation of FIG.

FIG. 3 is a diagram showing the rough positions of ten muscles and surface electrodes for measuring myoelectric signals.

FIG. 4 is a diagram showing a basic four-layer perceptron for estimating joint angular acceleration from myoelectric signals, position, and velocity.

5 is a block diagram of a network of 2 to 4 layers in which a module structure is introduced to improve the accuracy of the basic circuit shown in FIG.

6 is a diagram showing a non-linear dynamical system model in which the network of FIG. 5 is recurrently connected to estimate a motion trajectory from an myoelectric signal.

FIG. 7 is a diagram showing a state of a trajectory when the virtual arm is moved from a certain point to a certain point using the model of FIG.

[Explanation of symbols]

 1 surface electrode 2 myoelectric signal amplifier 3 filter 4 body dynamics model 5 computer 41 1st layer 42 2nd layer 43 3rd layer 44 4th layer 21,22 expert network 23 gate network 31 model 32,36 adder 33,34 , 35 One unit time delay factor

Claims (5)

(57) [Claims] 1. A human interface device for inputting necessary information to a computer by an operator's operation, which is worn on the operator's body to detect signals corresponding to activities of a plurality of muscles of the operator. A surface electrode, a non-linear dynamics model for converting each detection output of the surface electrode into a signal representing a pseudo tension corresponding to the tension generated by each muscle, and estimating a motion trajectory and a force trajectory based on the signal, and A human interface device comprising a virtual body operating on the estimated motion trajectory or force trajectory in accordance with the output of a nonlinear dynamics model. 2. The non-linear dynamics model includes a neural circuit, and the neural circuit filters a signal according to the activities of a plurality of muscles input from the surface electrode and corresponds to a tension generated by each muscle. A front-stage circuit that outputs a signal representing the pseudo tension, and a rear-stage circuit that includes a plurality of layers that receives the signal from the front-stage circuit and estimates the pseudo tension, the torque from the position and speed, the force exerted on the outside world, and the acceleration signal. The human interface device of claim 1, comprising: 3. The human interface device according to claim 2, wherein the neural circuit learns using a predetermined learning rule to determine a parameter. 4. The neural circuit substitutes the estimated torque into a non-linear equation that describes the dynamics of the body and performs numerical integration to estimate a trajectory, or numerically integrates the estimated acceleration by cyclic coupling. The human interface device according to claim 3, wherein the trajectory is estimated to control the virtual body. 5. The human interface device according to claim 4, wherein the neural circuit is divided into a portion for estimating acceleration, torque, force, etc. during rest and movement from the position and velocity.