DE102017208451A1 - Human-machine interface and method for calibrating such - Google Patents

Abstract

The invention relates to a method for calibrating a human-machine interface (10), wherein input commands are generated by a plurality of myographic sensors (12) placed on the skin surface (14) of a user based on the muscular movement of a user by the user in which the mygraphic sensors (12) are held in a defined relative position relative to one another on the skin surface (14) of the user by a holding device (16), wherein the human-machine interface (10) is calibrated upon depositing and reappearing Holding device (16) is carried out as follows: a) detecting a first reference muscle movement of the user while the holding device (16) is in a first position, b) detecting a second reference muscle movement of the user after the holding device (16) stored and reapplied again was so that she is in a second position, extending from the first butt c) determining a plurality of images, by means of which the individual positions of the myographic sensors (12) in the second position of the holding device (16) are imaged onto the individual positions of the myographic sensors (12) in the first position of the holding device (16) , d) determining a measure of error for each mapping by determining how similar the sensor signals are to the myographical sensors (12) in the second position of the fixture (16) in performing the second reference muscle motion to the sensor signals of the myographic sensors (12) in the first position the holding device (16) in performing the first reference muscle movement, e) selecting that image having the least error measure as the correct image.

Description

The invention relates to a human-machine interface and a method for calibrating such.

Various possibilities are known from the prior art for designing a human-machine interface in order to enable a human user to generate input commands for a device, for example a robot. For example, in prosthodontics electromyography is used in which a plurality of electrodes are applied to the surface of the skin of the user, by which muscular movements of the user are recognized. Due to this detection, input commands are generated, for example, for a movement of the prosthesis or a robot.

The problem here is that after the electrodes have been removed and reapplied, the system must be re-trained, since the electrodes will never occupy the same position as before and their signals will thus differ from the previous signals after re-application. The more complex the control commands, the more electrodes are needed, and thus the longer the inevitable training phase of the pattern recognizing algorithm, which translates the specific electrode activation pattern into a corresponding command. The problem here is that the specific pattern of electrode activation depends very much on the individual electrodes on the skin, so that even the smallest displacements of the electrodes lead to other electrode patterns. As a result, the pattern-recognizing algorithms do not recognize the altered patterns, or even worse, generate a wrong motion command based on a supposedly recognized pattern.

• - US 2010/0069736 A1
• - D. Prutch, „A High-Resolution Large Array (HLRA) EMG System“, 1995, Med. Eng. Phys., Bd. 17, 442-454
• - US 8447704 B2
• - Sebastian Bitzer, Patrick von der Smagt, 2006, „Learning EMG-control of a robotic hand: Towards Active Prostheses“, ICRA 2006, S. 4175-4186
• - DE 10 2012 211 799 B4

Information on the use of electromyographic electrode arrays can be found in the following publications:

• - US 2010/0069736 A1
• - D. Prutch, "A High-Resolution Large Array (HLRA) EMG System", 1995, Med. Eng. Phys., Vol. 17, 442-454
• - US 8447704 B2
• - Sebastian Bitzer, Patrick von der Smagt, 2006, Learning EMG-control of a robotic hand: Towards Active Prostheses, ICRA 2006, pp. 4175-4186
• - DE 10 2012 211 799 B4

The object of the invention is to provide a man-machine interface that can be calibrated particularly easily after dropping and re-attaching. Furthermore, a corresponding method for calibrating a human-machine interface is to be provided.

The object is achieved according to the invention by the features 1 and 12.

In the method according to the invention for calibrating a human-machine interface, a multiplicity of myographic sensors are placed on the skin surface of a user. These may be, for example, electromyographic sensors but also mechanomyographic sensors, that is to say pressure sensors. Based on the user's muscular movements, input commands intended by the user are generated. These may, for example, be transferred to a prosthesis or a robot in order to control the prosthesis or the robot.

In addition to prostheses or robots, the method according to the invention can also be used for inputting gestures into a computer for controlling the same. Furthermore, it can be used to control a robotic exoskeleton.

The myographic sensors are held by a holding device in a defined relative position to each other on the skin surface of the user. The holding device may be any suitable device by which neighborhood relationships between the individual myographic sensors can be maintained within certain tolerances. In other words, the holding device should ensure that the distance and the relative position of the individual myographic sensors are not changed to one another or only within the tolerance measure. For example, if the myographic sensors are mounted on the forearm of the user, an overcoat of a stretchable material such as a silicone or a thermoplastic could be used. Also, a device rigidly adapted to the anatomy of the user may be used, for example, made of hard plastic. Woven or fabric-like structures may prove to be advantageous due to their good extensibility and ease of processing.

1. a) Es wird eine erste Referenzmuskelbewegung des Nutzers erfasst, während sich die Haltevorrichtung in einer ersten Position befindet. Befindet sich beispielsweise die Haltevorrichtung am Arm des Nutzers, kann als Referenzbewegung eine kraftvolle Abduktion der ganzen Hand durchgeführt werden. Alternativ ist es möglich, den Zeigefinger und den Daumen aufeinander zu pressen. Für diese Referenzmuskelbewegung ist die Soll-Aktivierung bei der ursprünglichen Elektrodenplatzierung bekannt. Natürlich sind auch andere markante Bewegungen benutzbar.
2. b) Es wird eine zweite Referenzmuskelbewegung des Nutzers erfasst, nachdem die Haltevorrichtung abgelegt und erneut wieder angelegt wurde, so dass sich nunmehr die Haltevorrichtung in einer zweiten Position befindet, die sich von der ersten Position unterscheidet. Die zweite Referenzmuskelbewegung muss hierbei mit der ersten Referenzmuskelbewegung identisch sein. Beispielsweise kann erneut eine kraftvolle Abduktion der ganzen Hand durchgeführt werden, oder aber es werden Zeigefinger und Daumen aufeinander gepresst. Grundsätzlich kann als Referenzmuskelbewegung jede Bewegung verwendet werden, bei der eine Vielzahl von mygraphischen Sensoren notwendig ist, um diese Bewegung korrekt zu erfassen. Im vorliegenden Verfahrensschritt werden somit die Sensorsignale der myographischen Sensoren, die sich in der zweiten Position befinden, bei Durchführen der zweiten Referenzmuskelbewegung erfasst. Da sich die zweite Position der Haltevorrichtung von der ersten Position unterscheidet, werden sich auch diese Signale von den gemäß Verfahrensschritt a) erfassten Sensorsignalen unterscheiden.
3. c) Es wird eine Vielzahl von Abbildungen ermittelt, durch die die Einzelpositionen der myographischen Sensoren in der zweiten Position der Haltevorrichtung auf die Einzelpositionen der myographischen Sensoren in der ersten Position der Haltevorrichtung abgebildet werden können. Hierbei können verschiedenste Arten von Abbildungen verwendet werden, die im weiteren Verlauf der vorliegenden Anmeldung näher beschrieben werden.
4. d) Für jede Abbildung wird ein Fehlermaß ermittelt, indem ermittelt wird, wie ähnlich die Sensorsignale der myographischen Sensoren in der zweiten Position der Haltevorrichtung beim Durchführen der zweiten Referenzmuskelbewegung den Sensorsignalen der myographischen Sensoren in der ersten Position der Haltevorrichtung beim Durchführen der ersten Referenzmuskelbewegung sind. Je besser hierbei die gewählte Abbildung ist, desto geringer wird das Fehlermaß sein. Dies liegt darin begründet, dass eine bessere Abbildung die Sensorsignale der mygraphischen Sensoren in der zweiten Position der Haltevorrichtung beim Durchführen der zweiten Referenzmuskelbewegung präziser auf die Sensorsignale der myographischen Sensoren in der ersten Position der Haltevorrichtung beim Durchführen der ersten Referenzmuskelbewegung abbildet.
5. e) Diejenige Abbildung, die das geringste Fehlermaß aufweist, wird als korrekte Abbildung ausgewählt. Diese ausgewählte Abbildung kann nunmehr verwendet werden, um die Sensorsignale den myographischen Sensoren in der zweiten Position der Haltevorrichtung beim Durchführen verschiedener Muskelbewegungen in Sensorsignale umzurechnen, die erzeugt würden, wenn sich die Haltevorrichtung noch in der ersten Position befände. Selbstverständlich ist es notwendig, dass das Mensch-Maschine-Interface in der ersten Position der Haltevorrichtung derart trainiert wurde, dass es in präziser Weise eine Vielzahl von Eingabebefehlen erkennen kann.

The calibration of the human-machine interface when placing and re-applying the holding device is performed as follows.

1. a) A first reference muscle movement of the user is detected while the holding device is in a first position. If, for example, the holding device is on the user's arm, then Reference movement performed a powerful abduction of the whole hand. Alternatively, it is possible to press the index finger and the thumb against each other. For this reference muscle movement, the target activation at the original electrode placement is known. Of course, other striking movements are usable.
2. b) A second reference muscle movement of the user is detected after the holding device has been deposited and re-applied, so that now the holding device is in a second position, which differs from the first position. The second reference muscle movement must be identical to the first reference muscle movement. For example, a powerful abduction of the whole hand can be performed again, or index fingers and thumbs are pressed together. In principle, any movement in which a multiplicity of mygraphic sensors is necessary in order to detect this movement correctly can be used as reference muscle movement. In the present method step, the sensor signals of the myographic sensors, which are in the second position, are therefore detected when the second reference muscle movement is carried out. Since the second position of the holding device differs from the first position, these signals will also differ from the sensor signals detected according to method step a).
3. c) A plurality of images is determined by which the individual positions of the myographic sensors in the second position of the holding device can be imaged onto the individual positions of the myographic sensors in the first position of the holding device. In this case, the most varied types of images can be used, which are described in more detail in the further course of the present application.
4. d) For each mapping, a measure of error is determined by determining how similar the sensor signals of the myographic sensors in the second position of the holding device when performing the second reference muscle movement are the sensor signals of the myographic sensors in the first position of the holding device when performing the first reference muscle movement. The better this is the chosen image, the lower will be the error measure. This is due to the fact that better imaging maps the sensor signals of the myrographic sensors in the second position of the holding device when performing the second reference muscle movement more precisely to the sensor signals of the myographic sensors in the first position of the holding device when performing the first reference muscle movement.
5. e) The image showing the least error is selected as the correct image. This selected image can now be used to convert the sensor signals to the myographic sensors in the second position of the fixture when performing various muscle movements into sensor signals that would be generated if the fixture were still in the first position. Of course, it is necessary that the human-machine interface in the first position of the holder be trained so that it can accurately recognize a plurality of input commands.

The method according to the invention thus makes it possible to continue to operate the man-machine interface even after the holding device has been deposited and re-applied, without having to undertake a complex new teaching procedure. Instead, the map described is used to convert the sensor signals and continue to use the previous algorithm of the original position of the fixture for which teaching has already taken place.

In a preferred embodiment, the mapping is determined by means of a discrete value shift.

It is also possible to determine the mapping by means of a linear interpolation.

The mapping can also be determined by means of nonlinear interpolation over several neighboring sensors. In particular, spline interpolation, Bezier curves or kernel functions can be used.

It is preferred that the distance between two adjacent sensors in the holder is ≦ 2 cm. This is particularly preferred in an embodiment in which the holding device is attached to the arm of the user. Experiments by the Applicant have shown that if the distance between adjacent sensors is too great, the proximity relationships of the sensors in the holding device can not be sufficiently taken into account so that the method according to the invention can no longer be carried out.

In a preferred embodiment, a calculation of the root mean square, the L1 standard and / or an infinite standard is performed to determine the error measure according to method step d).

Furthermore, it is preferred that sensors which have a poor contact with the skin surface are not taken into account in the imaging and, in particular, their sensor values are replaced by an interpolation or averaging of the values of adjacent sensors. In electromyography, poorly connected electrodes are relatively easy to detect by impedance measurements, but also the observation of the amplitude can give an indication. This amplitude observation is also suitable for mechanomyographic methods in which an impedance measurement is naturally not possible.

It is furthermore preferred that sensor values of sensors located in a safety area at the edges of the holding device are discarded, the width of the safety area being determined as a function of the expected positioning inaccuracy when the holding device is re-applied.

It is further preferred that in the calculation of the error measure according to method step d), the neighborhood relationships of the myographic sensors in the holding device are taken into account by calculating only the error measure for possible mappings. Figures that are not possible due to the considered neighborhood relationships of the myographic sensors are not considered. For example, a non-possible neighborhood relationship would be one that would only be possible if the arrangement of the myographic sensors in the holding device were resolved or changed beyond the permissible degree of tolerance. A further example is explained in the description of the figures of the present application.

It is further preferred that a plurality of different first and second reference muscle movements are detected, wherein in addition to the changing position of the holding device, the change of further parameters is determined. In principle, in all embodiments of the invention, a change in position means both a translatory shift and a rotational test. Other parameters which can be determined by a larger number of reference muscle movements would be, for example, the reduction of the distance between the individual myographic sensors, for example, if the holding device is flexible. When attaching the holding device on the forearm of a user, this can for example be caused by a changing forearm circumference. As a result of the stretching or shearing of the carrier material for the myographic sensors, a changed distance of the sensors from one another can thus arise. The more indeterminate parameters need to be determined, the more different reference muscle movements must be performed.

The invention further relates to a human-machine interface with a holding device having a plurality of myographic sensors. The device according to the invention can have all the features that have been presented in connection with the method according to the invention and vice versa.

The myographic sensors are held by a holding device in a defined relative position to each other on the skin surface of a user.

The human-machine interface according to the invention has a control device for carrying out the method described so far.

In the following, preferred embodiments of the invention will be explained with reference to a figure.

The figure shows the attachment of an embodiment of the human-machine interface according to the invention on the forearm of a user.

Here are a variety of electromyographic sensors 12 on a flexible holding device 16 attached by them on the skin surface 14 held by the user. When the user activates the muscles in his forearm for a particular desired movement, the individual muscle movements are controlled by the electromyographic sensors 12 detected and converted into an input command for a device, not shown, for example, a prosthesis. Here, an algorithm is used, which must first be trained by the user by performing a variety of movements.

When storing and re-applying the holding device can be avoided by the inventive method re-learning by an image is determined by which the detected sensor signals of the electromyographic sensors in the second position can be mapped to the sensor signals in the first position.

For example, an electrode array with many electrodes, for example, 128 electrodes in 8 rings each 16 Electrodes are attached to the forearm of the user. For this electrode configuration, a classifier trained, for example, to be able to detect both grip forces and various hand poses from the EMG data of the forearm, thereby enabling control of any device.

One way to find an image is to shift the values electrode-wise, that is, discretely. In a ring of 4 electrodes e_1, e_2, e_3, and e_4, the mapping looks like this when shifted by one: a_1 = e_2; a_2 = E_3; a_3 = e_4; a_4 = e_1. It is obvious that this very simple method can only be successful with very narrow-meshed electrode arrays (ie with many electrodes). To find the image, the reference electrode values of a reference movement are compared with the current electrode values when the same reference movement is applied. The comparison is based on a norm, eg the L2-norm, which is also called RMS (root mean square). This works, starting from the above 4-electrode example as follows: Given the 4 reference values of the 4 electrodes in the original position: r_1, ..., r_4; and the pictures a_1, ..., a_4. The error measure according to the L2 standard is then calculated as follows: sqrt ((r_1-a_1) ^ 2 + (r_2-a_2) ^ 2 + (r_3-a_3) ^ 2 + (r_3-a_3) ^ 2). For reasons of efficiency, dragging the root at the end can also be omitted, since this will not change the order of the errors (-> strictly continuous function). The error measure is now calculated for all possible mappings, taking into account the neighborhood relations, ie an impossible mapping would be the following: a_1 = e_1; a_2 = E_3; a_3 = e_2; a_4 = e_4, because here the electrodes e_2 and e_3 would have changed places. At the end, the image is selected as the final, which has the lowest error measure.

The other possible method is based on a linear interpolation between the neighboring electrodes. Through a linear interpolation, it is possible to determine even non-discrete shifts. In the example above, the shift by "one-quarter electrode" would look like this: a_1 = .75 * e_1 + .25 * e_2; a_2 = .75 * .25 * + e_2 E_3; a_3 = .75 * .25 * E_3 + e4; a_4 = .75 * + .25 * e_4 e_1. Again, to find the best map, use a measure of error, such as that according to the above L2 standard. Now the error measure is minimized, preferably by means of a "gradient descent", for which purpose the partial derivation of the error function with respect to the possible shifts is formed. Since the gradient descent works best in the "proximity" of the correct mapping, the best possible starting position can be sought first according to the discrete method described above, which is then further optimized by means of gradient descent. Other optimization methods not based on gradients, e.g. Genetic algorithms or Swarm optimizers are also conceivable. The mapping from this is much better than the discrete method even for less densely populated arrays.

Of course, the above method of linear interpolation can be extended to non-linear interpolation. Instead of a linear interpolation of the neighborhoods, a non-linear function over several neighbors, and neighbors of neighbors, etc. is used. Particularly suitable for this purpose are b-splines, Bezier curves or else kernel-based methods such as e.g. Gaussian kernel. Unlike the linear interpolation, it is particularly important in the nonlinear that the starting point for fine tuning by means of e.g. Gradient descent is close to the expected optimum, since in nonlinear optimizations often only a local optimum is found. Thus, it is particularly important to first select the best possible starting position according to the discrete method.

After an initial optimization, it is also possible to further optimize in terms of keeping pace, provided that the system can determine the quality of the classification at runtime.

If this is possible, the "figure of merit" from the map determining system can be used to further improve the mapping with the principles of Reinforcement Learning.

Another application is the computational replacement of poorly contacted EMG electrodes. It often happens that with the large number of electrodes, individuals have very bad skin contact. This can easily be detected by an impedance measurement of each individual electrode. Now, if a poorly contacted electrode has been detected, it can be computationally removed and replaced by a linear or non-linear interpolation of its neighbors (and possibly more distant neighbors). In the case of a simple linear interpolation, and of four electrodes in ring form, as described above, the following figure would result for the identical mapping, but omitting the electrode e_2: a_1 = e_1; a_2 = .5 * e_1 + 0.5 * E_3; a_3 = E_3; a_4 = e_4.

The method according to the invention can be applied wherever it is necessary to provide a complex man-machine interface for controlling various devices, for example prostheses, robotic exoskeletons, human-computer interfaces. The latter devices can be used for example in office use or for consumer electronics.

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• US 2010/0069736 A1 [0004]
• US 8447704 B2 [0004]
• DE 102012211799 B4 [0004]

Cited non-patent literature

• D. Prutch, "A High-Resolution Large Array (HLRA) EMG System", 1995, Med. Eng. Phys., Vol. 17, 442-454 [0004]
• Sebastian Bitzer, Patrick von der Smagt, 2006, Learning EMG Control of a Robotic Hand: Towards Active Prostheses, ICRA 2006, pp. 4175-4186 [0004]

Claims (12)

Method for calibrating a human-machine interface (10), wherein input commands are generated by a plurality of myographic sensors (12) placed on the skin surface (14) of a user based on the muscular movement of a user by the user; wherein the mygraphic sensors (12) are held in a defined relative position to each other on the skin surface (14) of the user by a holding device (16), wherein the calibration of the man-machine interface (10) is carried out when placing and re-applying the holding device (16) as follows: a) detecting a first reference muscle movement of the user while the holding device (16) is in a first position, b) detecting a second reference muscle movement of the user after the holding device (16) has been deposited and reapplied so that it is in a second position that is different from the first position, c) determining a plurality of images by which the individual positions of the myographical sensors (12) in the second position of the holding device (16) are imaged onto the individual positions of the myographic sensors (12) in the first position of the holding device (16), d) determining a measure of error for each mapping by determining how similar the sensor signals are to the myographic sensors (12) in the second position of the fixture (16) performing the second reference muscle motion to the sensor signals of the myographic sensors (12) in the first position of the Holding device (16) when performing the first reference muscle movement, e) Select the image that has the least error measure as the correct image. Method according to Claim 1 , characterized in that the mapping is determined by means of discrete value shift. Method according to Claim 1 , characterized in that the mapping is determined by means of linear interpolation. Method according to Claim 1 , characterized in that the mapping is determined by means of non-linear interpolation over a plurality of adjacent sensors (12), in particular using spline interpolation, Bezier curves or kernel functions. Method according to one of Claims 1 to 4 , characterized in that the distance between two adjacent sensors (12) is ≤ 2 cm. Method according to one of Claims 1 to 5 , characterized in that for the determination of the error measure according to method step d), a calculation of the root mean square, the L1 standard and / or an infinite standard is performed. Method according to one of Claims 1 to 6 , characterized in that sensors (12) which have a poor contact with the skin surface are not taken into account in the image and in particular their sensor values are replaced by an interpolation or averaging of the values of adjacent sensors (12). Method according to one of Claims 1 to 7 , Characterized in that sensor values of sensors (12) located in a secure area B at edges of the holding device (16) are to be discarded, wherein the width of the safety area B depending on the anticipated positioning inaccuracy in the re-application of the holding device (16) is determined. Method according to one of Claims 1 to 8th , characterized in that in the calculation of the error measure according to method step d), the neighborhood relationships of the myographic sensors (12) in the holding device (16) are taken into account by calculating only the error measure for possible mappings and mappings based on the considered neighborhood relationships of the myographic sensors (12) in the holding device (16) are not possible to be considered. Method according to one of Claims 1 to 9 , characterized in that a plurality of different first and second reference muscle movements are detected, wherein in addition to the changing position of the holding device (16) and the change of further parameters is determined. Method according to one of Claims 1 to 10 , characterized in that the myographic sensors (12) are electromyographic sensors and / or mechanomyographic sensors. A human-machine interface comprising a holding device (16) having a plurality of myographic sensors (12), the myographic sensors (12) held by the holding device (16) in a defined relative position to each other on the skin surface (14) of a user become, wherein the man-machine interface (10) comprises a control device for carrying out the method according to any one of Claims 1 to 11 having.

Images