WO2016115976A1

WO2016115976A1 - Smart wearable input apparatus

Info

Publication number: WO2016115976A1
Application number: PCT/CN2016/070067
Authority: WO
Inventors: Liang KONG
Original assignee: Kong Liang
Priority date: 2015-01-21
Filing date: 2016-01-04
Publication date: 2016-07-28

Abstract

A text and pointer input system for a computing device comprises: a wearable sensor device (12a, 12b) for measuring movement of at least one hand (100a, 100b) of a user, the sensor (12a, 12b) being capable of measuring movement of at least one individual finger with respect to the hand (100a, 100b) as well as movement of the hand (100a, 100b) as a whole with respect to its surroundings, and the sensor device (12a, 12b) being arranged to transmit movement data to a controller (14); a display screen (16) arranged to receive display information from the controller (14); and a controller (14) adapted to display an image of a virtual keyboard (20) on the display screen (16), and to display an image of a virtual hand (18a, 18b) on the display screen (16), the virtual hand (18a, 18b) moving on the display screen (16) in response to input from the sensor device (12a, 12b), the controller (14) being further adapted to identify a key press when the at least one finger is moved when the corresponding finger on the virtual hand (18a, 18b) is in a position corresponding with a key on the virtual keyboard (20), and to pass the key press as text input to the computer.

Description

SMART WEARABLE INPUT APPARATUS

Field of the Patent Application

The present invention relates to input apparatus for controlling a computing device, and particularly to wearable input apparatus which allows text input.

Background to the Invention

The most common means of providing text input to a computer system is via a standard keyboard. The keyboard allows for high-speed text input, but does have a number of drawbacks. The use of keyboards is associated with the development of repetitive strain injury and other pains in the palms, wrists, hands, shoulders, neck, and back.

Keyboards are not especially portable, and therefore are not generally used with tablets or mobile telephones. On these devices, the most common mode of text input is to provide a ‘virtual’ keyboard displayed on a touch-sensitive screen. Text may be input by ‘typing’ on the touch-screen keyboard. However, this in itself has a number of problems–for example, it can be difficult to find a good position to place the tablet where the user can easily look at the screen and also easily type on the screen. In the case of a mobile phone, the screen will almost certainly be too small to allow anything other than ‘two finger’ typing, and therefore a high typing speed cannot be achieved.

Motion tracking and gesture recognition devices and systems are also known, for example the Xbox (RTM) Kinect games system. However, these systems are generally not suitable for high-speed text input. Text input is possible in essentially the same way as an on-screen keyboard is used on a touch-screen device, but the typing speed is generally slow.

It is an object of the invention to reduce or substantially obviate the above mentioned problems.

Statement of Invention

According to the present invention, there is provided a text input system for a computer, the system comprising:

a wearable sensor device for measuring movement of at least one hand of a user, the sensor being capable of measuring movement of at least one individual finger with respect to the hand as well as movement of the hand as a whole with respect to its surroundings, and the sensor device being arranged to transmit movement data to a controller；

a display screen arranged to receive display information from the controller； and

a controller adapted to display an image of a virtual keyboard on the display screen, and to display an image of a virtual hand on the display screen, the virtual hand moving on the display screen in response to input from the sensor device, the controller being further adapted to identify a key press when the at least one finger is moved when the corresponding finger on the virtual hand is in a position corresponding with a key on the virtual keyboard, and to pass the key press as text input to the computer.

The text input system is advantageous, because it allows high-speed text input to a computer by movement of the hands in exactly the same way as text input is achieved with a standard keyboard. Because a standard keyboard is likely to be familiar to the user, relatively little learning will be required to use the device. At the same time, the user is not constrained to have his hands in a certain position, as with a regular physical keyboard. Since the device is wearable, the user can input text to a computer with his hands in more or less any position, for example on the arms of a chair or by his sides while lying in bed. This may be a particular advantage to users having certain physical disabilities, who may not be able to position their hands correctly to use a standard keyboard.

Preferably, the controller is implemented as software running on the computer to which text input is being provided.

The text input system is suitable for use with desktop or laptop computers, and also with more portable devices such as tablets or smartphones. In the latter case, the text input system of the invention allows for high-speed text input which is simply not possible using the touch screen keyboard which is typically provided with these devices.

Preferably, two wearable sensor devices are provided, so that the user can use one sensor device to measure movement in each of his two hands. Correspondingly, two virtual hands may be displayed on the screen and keys on the virtual keyboard may be pressed by a finger on either hand. In this case, the position of each virtual hand with respect to the real hand may be calibrated independently, allowing the physical hands to be separated by some distance, for example at either side of the user’s body, whilst the virtual hands are displayed relatively close together, both over the virtual keyboard.

The ratio of the distance moved by the or each real hand to the distance moved by the or each virtual hand may be set by the user. This allows the perceived ‘responsiveness’ of the virtual hands to be set to a level which is found to be most natural for the user.

Preferably, the or each sensor device may be capable of measuring movement individually in each of the five fingers of a user’s hand. With two such sensor devices, the user is able to use all his fingers to achieve a high typing speed, in exactly the same way as with a standard physical keyboard.

The controller may be adapted to provide for pointer input, as well as text input, to the computer. In this case, when any virtual hand is not positioned over the virtual keyboard, that hand can be used to point and ‘click’ items on the display screen. In this way, the system can replace both keyboard and mouse input, again allowing each hand to remain in a natural position whilst providing for full use of the computer. The controller may also be adapted to recognise certain gestures other than ‘typing’ on the virtual keyboard. For example, gestures for the up, down, left and right keys may be provided to make those functions accessible without the need to move the virtual hands over the corresponding keys on the virtual keyboard. The combination of typing on the virtual keyboard, pointing and clicking with the virtual hand, and gesture control to access common functions makes it possible to operate a computer very quickly, as compared with standard known input devices.

The size of the virtual keyboard on the display screen may be adjustable by the user. In particular, the size of the keyboard may be adjusted when a ‘pinching’ motion between two fingers is measured by the sensor device when the virtual hand is positioned near an edge of the virtual keyboard. In this way, the user can pinch or grab one or more edges of the keyboard, and move the edge (s) to adjust the size of the keyboard.

The size of the virtual hand (s) on the display screen may also be adjustable by the user. In some embodiments, the size of the virtual hand (s) may be adjusted automatically to match the size of the virtual keyboard when the size of the virtual keyboard is adjusted.

Different visual, audio, or any other feedback effects may be provided when a key on the virtual keyboard is pressed by the virtual hand. In some embodiments, the wearable sensor device may incorporate force feedback components, so that the user can ‘feel’ when he has pressed a key.

The text input system of the invention provides for high-speed text input on a computer with any size of screen, whilst allowing the hands of the user to remain in substantially any position. In addition, the system is highly flexible when compared to a standard keyboard. As described above, the size of the virtual keyboard may be adjusted. In addition, the virtual keyboard may be moved to different points on the display screen, and the transparency, design and colour of the virtual hands and keyboard can be set at the user’s discretion. Even the shape of the keys on the virtual keyboard, the spacing between keys, and the arrangement of keys may be changed to suit the individual user. This level of flexibility is impossible with a standard physical keyboard.

Description of the Drawings

For a better understanding of the present invention, and to show more clearly how it may be carried into effect, reference will now be made by way of example only to the accompanying drawings, in which:

Figure 1 is a schematic illustration of a text input system according to the invention；

Figure 2a shows the size of the virtual keyboard being adjusted in the text input system of Figure 1；

Figure 2b shows the position of the virtual keyboard being adjusted in the text input system of Figure 1；

Figure 2c shows the virtual keyboard being temporarily hidden or switched off in the text input system of Figure 1；

Figure 3 shows the arrangement of keys on the virtual keyboard being adjusted in the text input system of Figure 1；

Figure 4 shows example gestures which may be used to access common functions in the text input system of Figure 1；

Figures 5a and 5b show an example gesture which may be used to emulate a mouse click in the text input system of Figure 1；

Figure 6 shows how the system of Figure 1 may be used to scroll up and down a document；

Figure 7 shows how the system of Figure 1 may be used to select items on a screen； and

Figure 8 shows how the system of Figure 1 may be used to select text on a screen.

Description of the Preferred Embodiment (s)

Referring firstly to Figure 1, an embodiment of a text input system according to the invention is indicated generally at 10. The text input system 10 includes a first wearable sensor device 12a and a second wearable sensor device 12b. Each

sensor device

12a, 12b is attached to one of the user’s

wrists

100a, 100b. In this embodiment, the sensor devices are worn like a bracelet, but it is envisaged that other forms of sensor device may be used, for example sensors in the form of gloves.

Each

sensor device

12a, 12b tracks the movement of the

corresponding hand

100a, 100b. In particular, the overall speed and direction of movement of each hand with respect to its surroundings is measured, as well as the movement of each finger on each hand. The measurements from the

sensor devices

12a, 12b are transmitted to a controller 14, which in this embodiment is implemented as software running on the computing device which is being controlled.

The system 10 further includes a display screen 16, which is the display screen which is already used by the computing device for output. The controller 14 is adapted to display items on the display screen 16, superimposed over other items which are being displayed by the computing device. In particular, the controller 14 displays a virtual hand 18a and a virtual hand 18b. The controller moves the

virtual hands

18a, 18b around the screen in response to movement of the corresponding

real hands

100a, 100b with respect to their surroundings, as measured by the

sensor devices

12a, 12b. The controller 14 also moves individual fingers on the

virtual hands

18a, 18b in response to movement of the corresponding fingers on the

real hands

100a, 100b, the movement of the fingers being measured by the

sensor devices

12a, 12b.

The controller also displays a virtual keyboard 20 on the display screen 16. By moving his

hands

100a, 100b, the user can control the movement of the

virtual hands

18a, 18b and therefore use all ten fingers to ‘type’ on the virtual keyboard 20.

When not typing, the user can also use his

hands

100a, 100b to control the

virtual hands

18a, 18b outside of the virtual keyboard 20, to access functions other than text input. For example, as shown in Figure 2a, the user can ‘pinch’ with his thumb and forefinger. The

virtual hands

18a, 18b will replicate this motion, and the controller (14) can interpret this to allow access to a re-sizing function. In the Figure, it is the virtual keyboard 20 which is being re-sized, by ‘pinching’ a t the corners and ‘stretching’ the virtual keyboard 20. However, it will be understood that other on-screen windows or other elements may be re-sized in this way.

In Figure 2b, the virtual keyboard 20 is being moved using virtual hand 18b. To move the virtual keyboard (or again, any other movable on-screen item) , the user moves his hand 100b so that the virtual hand 18b is at a corner of the item to be moved. He then makes a ‘grasping’ gesture, i.e. a clenched fist. The controller (14) interprets this to allow access to a moving function. The item is then moved to its new location by moving the hand 100b, before releasing the clenched fist to ‘drop’ the item in place.

As well as moving and resizing the virtual keyboard 20, the virtual keyboard can conveniently be minimized, or temporarily switched off, when text input is not required. This allows more space on the display screen for other items. As shown in Figure 2c, this may very simply be achieved by providing an ‘off’ key 22 on the virtual keyboard 20. When the ‘off’ key is pressed, the keyboard shrinks to leave only one key–an ‘on’ key 24. The ‘on’ key can be pressed to restore the full-size keyboard when required.

As shown in Figure 3, the layout of the keys on the virtual keyboard 20 may be adjusted to suit the user’s preferences. A simple and common adjustment may be to adjust the spacing between keys or the size of the keys, but full flexibility is available if required. In Figure 3 the user is completely rearranging the keys on the virtual keyboard 20.

As already described with reference to Figures 2a and 2b, the

virtual hands

18a, 18b (as controlled by

real hands

100a, 100b) may be used to access functions other than text input. For high-speed control of a computer, it is useful to combine text input with the virtual keyboard as described with ‘point and click’ input, and other gestures to quickly allow access to common functions. Figure 4 shows an example set of gestures which may be used to provide the functions of the arrow keys (up, down, left and right) without having to move the virtual hands to any particular position on the screen. On the far left, the up function is shown as a movement of the left middle finger away from the palm. In the next illustration, the down function is shown as a movement of the left middle finger towards the palm. The left arrow function is accessed by stretching out the left ring finger, and the right arrow function is accessed by stretching out the left index finger. These gestures are of course only examples, and mapping of gestures to common functions may be fully configurable by the user.

Figures 5a and 5b show how the input system may be used in place of a standard mouse. The user’s right hand 100b can be moved to control movement of virtual hand 18b around the screen and, for example, a pressing motion with the index finger can be configured to correspond with a left mouse-click, and a pressing motion with the ring finger can be configured to correspond with a right mouse-click.

Figure 6 shows how the input system may be used for scrolling up and down a document. In one scrolling mode (shown on the far left of the Figure) , an outstretched index finger gesture of the hand 100b causes scrolling in a ‘hand tracking’ mode, where the hand is simply moved to scroll the page up and down. Alternatively, as shown in the centre of the Figure, the virtual hand 18b can be used to operate a scroll bar 50, in the same way as the scroll bar 50 can be operated with a mouse pointer. On the right of the Figure, a third scrolling method is shown, in which the virtual hand 18b is manipulated to scroll just as a real hand would on a touch-sensitive screen.

In Figure 7, the input system 10 is being used to select items on a screen. The virtual hand 18b is used in exactly the same way as a conventional mouse to draw a box 70 around

items

52, 54, 56, 58, 60, 62, to select those items.

Items

64, 66, 68 are not selected in this example. Virtual hand 18b is moved to the top left corner of the area to be selected, and held there for a predetermined period (for example, two seconds) . The virtual hand 18b is then dragged to the bottom right corner of the area to be selected, to create a box 70 and select the items within the box 70.

The input system is extremely flexible, and allows text input at a similar speed to a standard keyboard, without associated constraints on posture. In addition, the system can provide high-speed text input in situations where a standard keyboard is not available, such as on a mobile telephone.

In Figure 8, like Figure 7, a selection is being made using the input system 10. In this case, a portion of text 70 (the first two lines in the figure) are being selected.

The embodiments described above are provided by way of example only, and various changes and modifications will be apparent to persons skilled in the art without departing from the scope of the present invention as defined by the appended claims.

Claims

A text and pointer input system for a computing device, the system comprising:

a wearable sensor device for measuring movement of at least one hand of a user, the sensor being capable of measuring movement of at least one individual finger with respect to the hand as well as movement of the hand as a whole with respect to its surroundings, and the sensor device being arranged to transmit movement data to a controller；

a display screen arranged to receive display information from the controller； and

a controller adapted to display an image of a virtual keyboard on the display screen, and to display an image of a virtual hand on the display screen, the virtual hand moving on the display screen in response to input from the sensor device, the controller being further adapted to identify a key press when the at least one finger is moved when the corresponding finger on the virtual hand is in a position corresponding with a key on the virtual keyboard, and to pass the key press as text input to the computer,

the controller being further adapted to emulate mouse or other pointer input when the virtual hands are not in a position corresponding with the virtual keyboard.
The text input system as claimed in claim 1, wherein two wearable sensor devices are provided for measuring movement of two hands.
The text input system as claimed in claim 2, wherein two virtual hands are displayed on the screen, each virtual hand moving in response to input from one of the two sensor devices.
The text input system as claimed in any of the preceding claims, wherein the ratio of the moved distance measured by the or each sensor to the distance moved by the or each virtual hand is adjustable.
The text input system as claimed in any of the preceding claims, wherein the or each sensor device is capable of measuring movement of each of the five fingers on the user’s hand individually.
A text input system as claimed in any of the preceding claims, wherein the controller is adapted to recognise gestures other than key-press gestures, and to activate functions dependent on recognised gestures.
The text input system as claimed in any of the preceding claims, wherein the size of the virtual keyboard on the display screen is adjustable.
The text input system as claimed in any of the preceding claims, wherein the position of the virtual keyboard on the display screen is adjustable.
The text input system as claimed in any of the preceding claims, wherein the size of the or each virtual hand is adjustable.
The text input system as claimed in claim 9, when dependent on claim 7, wherein the size of the or each virtual hand is automatically adjusted when the size of the virtual keyboard is adjusted.
The text input system as claimed in any of the preceding claims, wherein the wearable sensor device (s) include output means for providing feedback when a key is pressed.
The text input system as claimed in claim 11, wherein the output means on the wearable sensor device include a force-feedback output device.
The text input system as claimed in claim 12, wherein the force-feedback output device includes a vibrator.
The text input system as claimed in any of the preceding claims, wherein the sensor device is in the form of a band.
The text input system as claimed in any of the preceding claims, wherein the sensor device is in the form of a glove.
The text input system as claimed in any of the preceding claims, wherein the virtual hand may be used to scroll a page on the screen.
The text input system as claimed in any of the preceding claims, wherein the virtual hand may be used to select from a set of items on the screen.
The text input system as claimed in any of the preceding claims, wherein the virtual hand may be used to select a portion of text on the screen.
The text input system substantially as described herein, with reference to and as illustrated in Figures 1 to 5 of the accompanying drawings.