US20140198034A1

US20140198034A1 - Muscle interface device and method for interacting with content displayed on wearable head mounted displays

Info

Publication number: US20140198034A1
Application number: US14/155,087
Authority: US
Inventors: Matthew Bailey; Aaron Grant; Stephen Lake
Original assignee: Thalmic Labs Inc
Current assignee: Meta Platforms Technologies LLC
Priority date: 2013-01-14
Filing date: 2014-01-14
Publication date: 2014-07-17
Also published as: US11009951B2; US20140198035A1; US10528135B2; US20200159325A1

Abstract

There is disclosed a muscle interface device and method for interacting with content displayed on wearable head mounted displays. In an embodiment, the muscle interface device comprises a sensor worn on the forearm of a user, and the sensor is adapted to recognize a plurality of gestures made by a user's hand and or wrist to interact with content displayed on the wearable head mounted display. The muscle interface device utilizes a plurality of sensors, including one or more of capacitive EMG, MMG, and accelerometer sensors, to detect gestures made by a user. The detected user gestures from the sensors are processed into a control signal for allowing the user to interact with content displayed on the wearable head mounted display in a discreet manner.

Description

FIELD OF THE INVENTION

The present disclosure relates generally to muscle interface devices, and more specifically to a muscle interface device and method for interacting with content displayed on wearable head mounted displays.

BACKGROUND OF THE INVENTION

In recent years, wearable head mounted displays have begun to gain wider acceptance, with a number of recently introduced wearable head mounted display devices having the potential for widespread adoption by consumers.
One such device disclosed in U.S. Pat. No. 8,203,502 issued to Chi et al. utilizes a finger operable input device such as a touch pad built into the wearable head mounted display (e.g. built into a side-arm of a pair of glasses, with one of the lenses functioning as a display screen) such that a user can interact with and control content appearing on the display screen with positioning and movement of a finger along a planar direction relative to a surface of the input device. A potential drawback of this approach is that a user is required to conspicuously raise his or her hand to touch the input device each time the user wants to interact with content displayed on the screen.
Another such device is disclosed in US 2012/0293548 (Perez et al.) in which a head mounted display provides users with supplemental information on a display screen provided in at least one of the lenses of a pair of glasses. A processing unit may be connected to the head mounted display to provide the computing power necessary for its operation. However, the method of user interaction with the display is not specified.
Yet another example of such a device is disclosed in U.S. Pat. No. 8,212,159 issued to Tang et al. in which a source image is projected onto screens built into head mounted displays worn by a user. Tang et al. focuses on the method and system for projection, and does not specify the manner of user interaction with the head-mounted display device.
Therefore, what is needed is an effective user interface device for wearable head mounted displays which can be utilized in an inconspicuous manner.

SUMMARY OF THE INVENTION

The present disclosure relates to a muscle interface device and method for interacting with content displayed on wearable head mounted displays.
More generally, the muscle interface device comprises a sensor worn on the forearm of a user, and the sensor is adapted to recognize a plurality of gestures made by a user's hand and or wrist to interact with content displayed on the wearable head mounted display.
In an embodiment, the muscle interface device utilizes a plurality of electromyographic (EMG) sensors to detect electrical activity produced by muscles during contraction, and convert the electrical signals for processing. The electrical signals detected from the muscles are interpreted as gestures (e.g. a combination of hand, wrist and arm movements) made by a user which provide a control input to a wearable head mounted display. The control input is preferably provided wirelessly via a near field communication protocol, such as Bluetooth™, for example.
In another embodiment, various types of sensors may be used alone or in lieu of or in combination with EMG sensors to detect gestures made by a user, for processing as a control input for interacting with a wearable head mounted display. This may be one or more mechanomyographic (MMG) sensors to detect vibrations made by muscles during contraction, or one or more accelerometer sensors to detect larger movements.
In another embodiment, the muscle interface device includes a calibration module with a routine for calibrating the muscle interface device for use with the wearable head mounted display.
Other features and advantages of the present invention will become apparent from the following detailed description and accompanying drawings. It should be understood, however, that the detailed description and specific examples are given by way of illustration and not limitation. Many modifications and changes within the scope of the present invention may be made without departing from the spirit thereof, and the invention includes all such modifications.

DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a user wearing a head mounted display and a muscle interface device in accordance with an embodiment.

FIG. 2A illustrates a detailed view of a muscle interface device in accordance with an embodiment.

FIG. 2B illustrates a data graph corresponding to an electrical signal detected by an EMG sensor.

FIG. 3 illustrates wireless near field communication between a head mounted display and a muscle interface device in accordance with an embodiment of the invention.

FIG. 4 illustrates a user's hand and wrist gesture processed as a control signal by the muscle interface device for interacting with content displayed on the head mounted display.

FIG. 5 illustrates a schematic system architecture of a muscle interface device in accordance with an embodiment.

FIG. 6 illustrates a schematic flow chart of a method in accordance with an embodiment.

In the drawings, embodiments of the invention are illustrated by way of example. It is to be expressly understood that the description and drawings are only for the purpose of illustration and as an aid to understanding, and are not intended as a definition of the limits of the invention.

DETAILED DESCRIPTION

The present disclosure relates to a muscle interface device and method for interacting with content displayed on wearable head mounted displays.
In an aspect, the muscle interface device comprises a sensor worn on the forearm of a user, and the sensor is adapted to recognize gestures made by a user to interact with content displayed on the wearable head mounted display. In another aspect, the user interface method comprises processing a control signal from gestures made by a user's hand, wrist and arm movements to interact with content displayed on the wearable head mounted display.
In a preferred embodiment, the muscle interface device utilizes a plurality of sensors to detect electrical signals and/or vibrations produced by muscles during contraction, and movement of the arm, and converts the electrical signals or vibrations to a digital signal for processing. The electrical signals and/or vibrations detected from the muscles are interpreted as gestures made by a user which provide a control input to a wearable head mounted display.
In an embodiment, the control input is preferably provided wirelessly via a near field communication protocol, such as Bluetooth™, for example. However, it will be appreciated that other types of near field communications may be used, including any NFC protocol developed for smart phones and similar devices.
In addition to capacitive EMG, MMG, and accelerometer sensors, various other types of sensors may be used to detect gestures made by a user. However, some of these sensors may have drawbacks if used alone.
For example, surface electromyographic (sEMG) sensors may be used to measure forearm muscle activity. An sEMG sensor typically requires direct contact with the skin of the user in order to measure the electrical activity conducted from the underlying muscles, through the fat and skin. There are some inherent limitations with sEMG, as the quality of the acquired signal is directly related to the skin impedance, which varies according to the user's skin perspiration, amount of arm hair, fat content, and a number of other attributes. This may necessitate the use of moisturizing and conductive gels, shaving the skin, or other skin preparation practices to get a reliable and repeatable signal from this type of sensor. In contrast, the capacitive EMG sensors of the present invention do not require direct surface contact or skin preparations.
As another example, one or more accelerometer sensors may be used to measure larger gestures made by a user, for example involving the elbow or even the shoulders of a user. When used together with EMG and/or MMG sensors for detecting more limited gestures (e.g. made by the hand and/or wrist for example), the accelerometer sensors may be utilized to increase the range of control inputs that may be generated for interaction with a wearable head mounted display.
An illustrative embodiment will now be described with reference to the drawings.
Shown in FIG. 1 is an illustrative user 100 wearing a head mounted display 310 with display control 300, and a muscle interface device 200 in accordance with an embodiment. In this illustrative example, muscle interface device 200 is a flexible, stretchable band that may be worn on the forearm of user 100 as shown.
FIG. 2A illustrates a detailed view of the muscle interface device 200 of FIG. 1 in accordance with an embodiment. As shown, muscle interface device 200 may comprise a central processing unit 210, and one or more batteries 220, which may be rechargeable, and which may be utilized concurrently or sequentially in conventional manner. As shown, muscle interface device 200 includes a plurality of sensors 230 which may be positioned radially around the circumference of the band, such that the sensors 230 can detect gestures made by user 100. Muscle interface device 200 may further include a feedback mechanism, such as a vibratory motor 240 to provide haptic feedback as described further below.
In an embodiment, the muscle interface device 200 is calibrated when first worn, prior to operation, such that the positioning of the sensors 230 does not need to depend on the location of particular muscles in the forearm.
In an embodiment, sensors 230 include at least one or more capacitive EMG sensor adapted to detect electrical signals in the forearm of user 100 for generating a control signal. Capacitive EMG does not require direct contact with the skin as with sEMG sensors described earlier. Rather, capacitive EMG sensors are capacitively coupled to the electrical signals generated by contracting muscles, and may operate at a distance of up to 3 mm from the skin. By way of example, the capacitive EMG signal can be an oscillating waveform that varies in both frequency and amplitude, and the majority of signal information is contained within the 5 Hz to 250 Hz frequency band. An illustrative example of an EMG signal is shown in FIG. 2B.
In another embodiment, one or more MMG sensors comprising piezoelectric sensors may be used to measure the vibrations at the surface of the skin produced by the underlying muscles when contracted. By way of example, the MMG signal generated may be an oscillating waveform that varies in both frequency and amplitude, and a majority of signal information is contained within the 5 Hz to 250 Hz frequency band. Because the MMG signal is acquired via mechanical means, electrical variations like skin impedance do not have an effect on the signal. The MMG signal is very similar to the illustrative example of the EMG signal shown in FIG. 2B.
Thus, capacitive EMG or MMG both provide a reliable control signal that can be obtained over the duration of a full day, as skin perspiration and moisturization changes do not affect the signal.
In another embodiment, sensors 230 may include one or more accelerometer sensors for detecting additional aspects of gestures made by user 100 in three degrees of freedom. The accelerometer signal consists of a three digital channels of data, each representing the acceleration in either the x, y, or z direction. The signal is subject to all of the accelerations that the users arm is subject to, and may further incorporate motion of the body as a whole.
Now referring to FIG. 3, shown is an illustration of wireless near field communication (NFC) between muscle interface device 200 and the head mounted display 310 and display control 300. This wireless NFC is utilized to transmit the control signal from muscle interface device 200 to display control 300.
This is illustrated by way of example in FIG. 4, in which user's hand and wrist gesture is detected and processed as a control signal by the muscle interface device 200 for interacting with content displayed on the head mounted display 310.
In this particular example, a gesture 410 made by the user extending an index finger, and making a wrist flexion motion 420 is detected by the sensors 230 of muscle interface device 200, and processed by CPU 210 (FIG. 2) as a control signal for causing a menu appearing on display 310 to scroll downwards.
As another example, a similar gesture in which user 100 extends the index finger and makes a wrist extension motion is detected by sensors 230 of muscle interface device 200 and processed by CPU 210 (FIG. 2) as a control signal for causing a menu appearing on display 310 to scroll upwards.
As yet another example, a gesture in which user 100 extends the index finger and makes a poking motion involving a slight movement of the elbow and shoulder may be detected by sensors 230 of muscle interface device 200 and processed by CPU 210 (FIG. 2) as a control signal for causing a highlighted menu item appearing on display 310 to be selected.
If the user extends a different finger other than the index finger, sensors 230 will detect this and may cause a different control signal to be generated. For example, extending the little finger or “pinky” finger instead of the index finger may cause muscle interface device 200 to interpret the user's gestures with functions analogous to clicking a right mouse button rather than a left mouse button in a conventional mouse user interface. Extending both the index and pinky fingers at the same time may cause muscle interface device 200 to interpret the user's gestures with yet other functions analogous to clicking a third mouse button in a conventional mouse user interface.
Thus, muscle interface device 200 may be adapted to be calibrated to recognize a wide range of user gestures made by a user, based on measurements from a plurality of sensors in the muscle interface device 200.
In an embodiment, muscle interface device 200 may itself be adapted to interpret the gestures from the detected signals as described.
However, in an alternative embodiment, the detected signals may be transmitted to the head mounted display 310 and display control 300 to be interpreted as gestures at the display control 300. Whether the detected signals are interpreted at the device 200 or at the display control 300, the detected signal is first interpreted as a recognized gesture in order to interact with content displayed on the display 310.
In another embodiment, upon interpretation of a gesture, muscle interface device 200 may include a haptic feedback module to provide feedback that a gesture has been recognized. This haptic feedback provides a user with confirmation that the user's gesture has been recognized, and successfully converted to a control signal to interact with content displayed on display 310. The haptic feedback module may comprise, for example, a vibrating mechanism such as a vibratory motor 240 built into the muscle interface device.
Alternatively, rather than haptic feedback provided by the muscle interface device 200, confirmation of recognition of a gesture may be provided by auditory feedback, either generated by a speaker on the muscle interface device, or operatively connected to the head mounted display 310.
In still another embodiment, confirmation of recognition of a gesture may also be provided visually on the display 310 itself. If there is more than one possible gesture that may be interpreted from the detected signals, rather than providing resulting in an error, the muscle interface device 200 and/or the display control 300 may provide a selection of two or more possible gestures as possible interpretation, and the user may be prompted to select from one of them to confirm the intended gesture and corresponding control.
Now referring to FIG. 5, shown is an illustrative schematic system architecture 500 of a muscle interface device in accordance with an embodiment. As shown, system architecture 500 includes a CPU 502, memory 504, system clock 506, an NFC module 508 (e.g. Bluetooth™), and a direct memory access (DMA) controller 510. As shown, DMA controller 510 is adapted to receive inputs from various sensors including one or more EMG sensors 520, MMG sensors 530 and accelerometer sensors 540.
In an embodiment, detected signals from one or more EMG sensors 520 are processed through signal filter 532 and converted from analog to digital signals by ADC 534. If one or more MMG sensors 530 are also used, then the detected signals from the sEMG sensors 520 are processed through signal filter 522 and converted from analog to digital signals by ADC 524. Digital signals from one or more accelerometer sensors 540 may also be processed through signal filter 542 and received by DMA controller 510.
In an illustrative embodiment the data from the various types of sensors 520, 530, 540 is acquired through an analog filtering chain. The data is bandpassed through filters 522, 532 between 10 Hz to 500 Hz, and amplified (e.g. by a total of 1000 times). This filtering and amplification can be altered to whatever is required to be within software parameters. A notch filter at 60 Hz, or at any other relevant frequency, may also be used to remove powerline noise.
In an embodiment, data from the sensors is converted to 12 bit digital data by ADCs 524, 534, and then clocked into onboard memory using clock 506 by the DMA controller 510 to be processed by the CPU 502.
Now referring to FIG. 6, shown is a schematic flow chart of a method in accordance with an embodiment. As shown, method 600 begins at block 602, where method 600 pairs a muscle interface device with a wearable head mounted display. Method 600 then proceeds to block 604, where content and/or user interface (UI) is displayed on the wearable head mounted display.
Method 600 then proceeds to block 606, where method 600 determines if the displayed content and/or UI is navigable. If no, method 600 returns to block 604. If yes, method 600 proceeds to block 608, where method 600 detects user gestures utilizing the muscle interface device 200, and wirelessly transmits the identified gesture to the display control 300 (FIG. 1).
Method 600 then proceeds to block 610, where display control 400 receives the detected gestures from the muscle interface device 200, and translates the detected gestures to a control signal to interact with navigable content displayed on the wearable head mounted display.

Use Cases

As will be appreciated, the muscle interface device and method of the present disclosure may be used for interaction with a wearable head mounted display in a wide range of applications, in virtually any application in which wearable head mounted displays are contemplated. By providing a discrete method of interacting with a wearable head mounted display, a user is able to interact with such a display in any operating environment, including situations where overt gesturing (e.g. raising the hand to touch an input device provided on the wearable head mounted display itself) is not desirable.
While various embodiments and illustrative examples have been described above, it will be appreciated that these embodiments and illustrative examples are not limiting, and the scope of the invention is defined by the following claims.

Claims

1. A muscle interface device for interacting with content displayed on a wearable head mounted display, comprising:

a plurality of sensors that in use are worn on a portion of an arm of a user to detect signals generated by at least one muscle of the arm of the user; and

a transmitter communicatively coupled to the plurality of sensors and which in use transmits communication signals indicative of muscle activity detected by the plurality of sensors to a receiver on the wearable head mounted display;

wherein the communication signals transmitted to the receiver on the wearable head mounted display cause one or more interactions with content displayed on the wearable head mounted display.

2. The muscle interface device of claim 1, further comprising:

a processor communicatively coupled to the plurality of sensors and the transmitter, and which interprets the signals detected by the plurality of sensors as a gesture and translates the signals detected by the plurality of sensors to a control signal to interact with content displayed on the wearable head mounted display, wherein the control signal is transmitted by the transmitter at least as part of the communications signals.

3. The muscle interface device of claim 1 wherein the detected signals are transmitted to the receiver on the wearable head mounted display to be interpreted as a gesture by the wearable head mounted display.

4. The muscle interface device of claim 1, further comprising a haptic feedback module for confirming the interpretation of a gesture.

5. The muscle interface device of claim 1 wherein the plurality of sensors includes at least one sensor selected from the group consisting of: an electromyographic (EMG) sensor and a mechanomyographic (MMG) sensor.

6. The muscle interface device of claim 1 wherein the transmitter includes a wireless transmitter to wirelessly transmit wireless communications signals to the receiver on the wearable head mounted display.

7. The muscle interface device of claim 1, further comprising:

an accelerometer to detect an acceleration of the arm of the user and to provide a signal representing the acceleration.

8. A method for interacting with content displayed on a wearable head mounted display, the method comprising:

displaying content on the wearable head mounted display;

detecting a user gesture by a muscle interface device, wherein the muscle interface device includes a plurality of sensors responsive to signals generated by muscles of a user and a transmitter, and wherein detecting the user gesture includes detecting signals generated by muscles of the user by the plurality of sensors of the muscle interface device;

identifying the user gesture by at least one circuit of the muscle interface device;

transmitting the identified gesture to the wearable head mounted display by the transmitter of the muscle interface device;

receiving the identified gesture by the wearable head mounted display; and

translating the identified gesture to a control signal by the wearable head mounted display to interact with content displayed thereon.

9. The method of claim 8 wherein the transmitter of the muscle interface device is a wireless transmitter, and wherein:

transmitting the identified gesture to the wearable head mounted display by the transmitter of the muscle interface device includes wirelessly transmitting the identified gesture to the wearable head mounted display by the wireless transmitter of the muscle interface device;

and receiving the identified gesture by the wearable head mounted display includes wirelessly receiving the identified gesture by the wearable head mounted display.

10. The method of claim 8, further comprising:

pairing the muscle interface device with the wearable head mounted display.

11. The method of claim 8 wherein the plurality of sensors includes one or more electromyography (EMG) sensors, and wherein detecting a user gesture by a muscle interface device includes detecting an electrical signal produced by a muscle of the user by the one or more EMG sensors of the muscle interface device.

12. The method of claim 8 wherein the plurality of sensors includes one or more mechanomyography (MMG) sensors, and wherein detecting a user gesture by a muscle interface device includes detecting a vibration produced by a muscle of the user by the one or more MMG sensors of the muscle interface device.

13. The method of claim 8 wherein the muscle interface device further comprises an accelerometer, and wherein detecting a user gesture by the muscle interface device includes detecting acceleration by the accelerometer.