WO2004055660A2

WO2004055660A2 - Data processing system with input device

Info

Publication number: WO2004055660A2
Application number: PCT/IB2003/050027
Authority: WO
Inventors: Daniel W. E. Schobben
Original assignee: Koninklijke Philips Electronics N.V.
Priority date: 2002-12-18
Filing date: 2003-11-28
Publication date: 2004-07-01
Also published as: WO2004055660A3; AU2003286385A8; AU2003286385A1

Abstract

The invention relates to a data processing system for executing an interactive application. The user may interact with the system by means of an input device (109, 110) for generating signals in response to user operation of the input device. The input device is provided with a sensor (111) for sensing the nearness of a user's hand to the input device. The system further comprises an activity monitor (112) for determining a user's activity with respect to the input device and alert means (113) for generating at least one user-perceptible signal in response to the sensor sensing the nearness of the user's hand during a period of inactivity.

Description

Data processing system with input device

FIELD OF THE INVENTION

The invention relates to a data processing system comprising a data processing apparatus for executing an interactive application, and an input device for generating signals in response to user operation of the input device and transferring the signals to the data processing apparatus,

The invention further relates to an input device for generating signals in response to user operation of the input device and transferring the signals to a data processing apparatus. The invention further relates to a data processing apparatus for executing an interactive application. The invention further relates to a method of enabling a user to interact with a data processing apparatus, the method comprising the steps of generating signals in response to user operation of an input device and transferring the signals to the data processing apparatus.

BACKGROUND OF THE INVENTION A well known example of such a system is a personal computer which enables a user to interact with software applications by means of a keyboard and a mouse, or any other suitable input device. Another example is a consumer electronics system which can be controlled by the user by means of a remote control, or a game computer equipped with one or more game controllers for controlling game characters. Intensive use of these systems may cause a series of health problems which are classified under the name repetitive strain injury (RSI). RSI accounts for a large portion of work-related illnesses and the incidence of RSI is expected to grow as the number of people operating input devices such as keyboards and mice increases. RSI is not caused by tlie input devices, but rather by the user's behavior. Intense typing, that is, typing for long periods without a break, slowly damages the soft tissues of a person's hands, wrists, and arms. To avoid such problems, a known system, disclosed in US 6,065,138, monitors the user activity over some time period. The known system functions by first initializing an activity status indicator in the memory. Next, the activity of the operator's input device is measured over a time period to determine an activity rate. The activity rate is then compared with a work limit. If the activity rate is greater than the work limit, an alert is generated to urge the user to take a short break.

A problem of the known system is that the workload is often a given fact which the user cannot always change. Another problem is that activity is mainly measured by monitoring operation of the input device, which method does not reveal the activities of reading and contemplating information read from screen or paper, or perceived in other ways.

OBJECT AND SUMMARY OF THE INVENTION

It is an object of the invention to provide an improved system and method of the type defined in the opening paragraph. To this end, the system according to the invention comprises a sensor for sensing the nearness of a user's hand to the input device, an activity monitor for determining a user's activity with respect to the input device, and - alert means for generating at least one user perceptible signal in response to the sensor sensing the nearness of the user's hand during a period of inactivity.

Recent studies have shown that an important cause for RSI is formed by constant muscle tension, e.g. when someone continuously holds the computer's mouse, even if it is not actually being used. The inventor recognized that this problem is not adequately tackled by prior-art systems. The user's actual activity cannot be measured reliably because his tasks may not only involve operation of input devices, but all kinds of activities including reading, thinking, making phone calls, etc. During all these activities, the user may keep holding his hands on or above the most recently used input devices, just by sheer habit. As the recent studies have shown, this behavior forms a high risk to develop RSI problems, because the constant muscle tension is maintained. It is an achievement of the present invention, that the user is warned if he unnecessarily holds his hand in the nearness of the input device, particularly keeping it in touch with the input device, while he is not actually operating the input device. The user's activity with respect to the input device, as opposed to the user's overall activity, can be measured reliably, because the former is only correlated with the signals generated by the input device. The system need only monitor the generation of these input signals, and interpret the absence of such signals during a certain time period, or possibly the absence of signals exceeding some threshold value to filter out e.g. unintentional mouse movements, as a period of inactivity with respect to the input device. If, during such a period of inactivity, the user still holds his hand near the input device, for example in touch with the input device, without actually operating the input device, a warning is generated to urge the user to take his hand back so that the muscle strain is removed. The alert preferably involves generating a user-perceptible signal, for example a visual, auditory or tactile signal, or any combination of them. Also, the user-perceptible signal could cause the input device to be deactivated temporarily.

In an embodiment of the system according to the invention, at least one characteristic of the at least one user-perceptible signal depends on the duration of sensing the nearness of tlie user's hand during the period of inactivity. Such a dependency may have various characteristics. Preferably, the alert is generated only after a predetermined delay during which no activity with respect to the input device is detected. The reason for this is that it is quite natural to pause a short time just to plan a successive data input. Preferably, the alert signal starts in a non-obtrusive way, for example, by a delicately glowing light behind the keys of a keyboard or a mouse, just to warn the user that he is not actually operating the input device. If the state continues, said dependency may be such that the intensity of the alert signal gradually increases, until a predetermined maximum intensity is reached. Other characteristics of the user-perceptible signal may be the color of the visual signal, for example, gradually changing from green to red, the frequency or timbre of an auditory signal, for example, gradually changing from a low frequency to a high frequency. This dependency is preferably user-adjustable, e.g. the user may be allowed to set the initial delay time, or the frequency/color range.

In an alternative embodiment of the system according to the invention, said dependence of said characteristic is a function of the number of times the user's hand is sensed during periods of inactivity. For example, an experienced user of the inventive system who has already learned to remove his hand from the input device could be alerted after a shorter delay than an initial user whose behavior still has to be modified. Generating the alert signal for the initial user too early would likely cause irritation.

In an embodiment of the system according to the invention, the alert means are arranged to generate multiple user-perceptible signals, of which the at least one characteristic depends in different ways on the duration of the period of inactivity. It is thus achieved that two or more alert signals can be generated during the same period of inactivity with different dependencies on the duration of the period of inactivity. For example, after three seconds of inactivity with respect to the input device, a visual signal may be generated as a first signal, and if the user persists in holding his hand near the input device, an auditory signal may be generated as a more insistent alert to urge the user to withdraw his hand. In an embodiment of the system according to the invention, the input device is arranged to command the data processing apparatus to generate the user-perceptible signal. It is not necessary that the alert means are located in the input device itself. Instead, the input device may send a signal to the data processing apparatus as a request to generate a user- perceptible signal, such as an auditory signal via a loudspeaker or some graphical indication on a display screen.

The invention is particularly suitable for computer input devices such as a keyboard, a mouse, a graphical stylus and/or tablet, a touch pad, etc. The invention can also be suitably applied in input devices for dedicated apparatuses such as game computers, remote control units for consumer electronics devices, or (electronic) musical instruments. The invention may also be applied to a stylus for operating a pen-based computer, digitizer or PDA. For example, a sensor may be built in the stylus for detecting the user holding the stylus while he is not actually using it. Alternatively, the data processing apparatus may be adapted to sense not only the user's hand but also any normal or specially adapted stylus. If the stylus is held in a position near the pen-based apparatus while no substantial input is received, the user is urged to put the stylus away. The invention may also be applied for touch screens which can be fingertip-operated. If the user's finger is held in a position near the touch-based apparatus while no substantial input is received, the user is urged to remove his finger from the touch screen. The sensor data may also be used to switch the input device to a power saving mode, i.e. if a computer mouse according to the invention does not sense the nearness of the user's hand, it may deactivate its optical movement detection.

BRIEF DESCRIPTION OF THE DRAWINGS These and other aspects of the invention are apparent from and will be elucidated, by way of a non-limitative example, with reference to the embodiment(s) described hereinafter. In the drawings,

Figure 1 shows a diagram of a personal computer system as a data processing system embodying the invention, Figure 2 schematically shows a computer mouse as an input device embodying the invention,

Figure 3 schematically shows a computer keyboard as an input device embodying the invention, Figure 4 shows a flow chart of a possible implementation of the method according to the invention.

DESCRIPTION OF EMBODIMENTS Throughout the Figures, the same reference numerals indicate similar or corresponding features. Some of the features indicated in the drawings are typically implemented in software, and as such represent software entities, such as software modules or objects.

Figure 1 shows a diagram of a personal computer system as a data processing system embodying the invention. A personal computer 101 comprises a central processing unit (CPU) 102, which can transfer data to and from a permanent storage 103 and a main memory 104. The pennanent storage 103 may be used for storing application programs and data files. The application programs and data files are partially or entirely loaded into main memory 104 in order to be executed. The CPU 102 is further connected to a video processor 105 for generating video data to be displayed on a display screen 106, and to an audio processor 107 for generating audio data to be reproduced via a loudspeaker 108. A user can interact with the application programs by means of a keyboard 109 and a computer mouse 110. Both the keyboard 109 and the mouse 110 comprise an optical sensor for detecting (the shadow of) the user's hand hovering above or touching the respective input device. The input devices 109 and 110 are connected via an activity monitor 112 to the CPU 102, to enable the user to provide input data to the application programs executed by the CPU 102. The activity monitor 112 is arranged to measure the time that no (substantial) input data has been received from input device 109 or 110. The activity monitor 112 may be further arranged to generate various kinds of statistical information about the user's activity. The information generated by the activity monitor 112 is passed on to an alert module 113, which uses the received information to decide whether the user should be warned that he uses the input device in the wrong manner. If that is the case, the alert module 113 sends a signal to the CPU 102 to generate a user-perceptible signal via either or both the display screen 106 and the loudspeaker 108. Preferably, the activity monitor 112 and the alert module 113 are arranged to separately monitor the keyboard 109 and the mouse 110, and the alert module 113 is similarly arranged to generate an alert pertaining to the proper input device. For example, if the user keeps his left hand resting on the keyboard 109 while he is only using the mouse 110, an alert is generated which is specifically directed to removal of the user's left hand from the keyboard 109. This can be communicated to the user by using a proper message text on the display screen 106, or by displaying a graphical icon which clearly indicates the keyboard 109, possibly in combination with an auditory message, e.g. a voice message. In an alternative embodiment, the alert message is passed on to the proper input device which itself has means for generating a user-perceptible signal, e.g. a beeper, one or more LEDs, or a vibrating member for providing tactile feedback.

Figure 2 schematically shows a computer mouse as an input device embodying the invention. The mouse 201 comprises the usual left and right buttons 202 and 203, which are normally used for selection and context menus, respectively. A scroll wheel 204 is provided for easy scrolling of documents. The mouse 201 comprises an optical sensor 205 which in its simplest form can distinguish between dark and light, or possibly some shades in between. In a more advanced form, the optical sensor 205 may be able to distinguish between various colors and/or textures or even recognize certain shapes. In yet another embodiment, the optical sensor 205 comprises an infrared transmitter and receiver for measuring the amount of reflected IR light (possibly after subtracting the amount of IR light when the transmitter is switched off to take ambient light into account). The purpose of the optical sensor 205 is to determine whether the user's hand is hovering above or resting on the mouse 201, and preferably distinguish the user's hand from a piece of paper which is unintentionally covering the mouse. In an alternative embodiment, the optical sensor 205 is replaced by a touch sensor which determines whether the user is holding the mouse in his hand. The touch sensor may be simply detecting a touch of an arbitrary object, or it may be capable of distinguishing between various types of objects, for example, by measuring warmth or conductivity. Other methods of sensing a user's hand, e.g. based on induction or capacitance, are also applicable.

Incorporated in the mouse 201 are an activity monitor 206, and alert module 207 and a vibration member 208. The activity monitor 206 monitors the user's activity with respect to the mouse 201, i.e. how many times the mouse is actually used as a positioning device, using a movement detector at the bottom of the mouse (not shown) or as a selection/scrolling device, using any of the buttons 203, 204 or the scroll- wheel 205. This information is fed to the alert module 207, which also receives data from the optical sensor 205. Using this information, the alert module 207 can detect a state wherein no substantial activity is measured (e.g. no or only very small movements of the mouse 201 or the scroll- wheel 205, and no operation of the buttons 203 and 204) and yet the user's hand is resting on or hovering above the mouse 201. If that situation is detected, the alert module 207 sends a control signal to the vibration member 208 so as to provide a tactile signal as feedback to the user. The vibration will urge the user to remove his hand as long he does not intend to actually use the mouse. Instead of the vibration member 208, for example, a beeper (not shown) may be used to alert the user. The user may be enabled to adjust the delay time with which the alert signal is generated in response to detecting inactivity. A possible way to do this is adding some hardware controls for increasing and/or decreasing the delay time.

Alternatively, some unusual operation of the mouse, e.g. pressing both buttons 203 and 204 simultaneously for a predetermined time period, may invoke a setup mode, which is indicated to the user by some signal generated by the alert module 207. Subsequently, the buttons 203 and 204, and/or the scroll-wheel 205, may be redefined so as to enable the user to alter the delay time, and possibly other mouse parameters such as the sensitivity, etc. Feedback on tlie present and or new value may be given through the alert means, for example, a beep or vibration whose duration is proportional to the delay period or other mouse parameter. The advantage of this method is that no additional hardware is required. After some time-out period after the last change of the delay period or other mouse parameter, the mouse may return to its normal mode.

In an advanced embodiment, the regular movement sensors of the mouse according to the invention may also be used as sensors for sensing the presence of the user's hand by distinguishing between substantial movements and non-substantial movements. In the latter case, the user is probably holding his hand on the mouse without actually using it, causing very small and unintended movements, either of the mouse itself or of the scroll- wheel. A further confirmation of this state might be derived from the absence of operation of the buttons. An advantage of this embodiment is that a separate sensor can be dispensed with.

Figure 3 schematically shows a computer keyboard 301 as an input device embodying the invention. In this embodiment, the sensors 302 are located between some of the keys in the center of the keyboard where the user would normally position his hands when he is touch-typing. Dependent on the size of the keyboard, even more sensors may be provided in order to reliably determine whether the user is holding his hand(s) on or above the keyboard. Similarly to the computer mouse 201 described above, the optical sensors can be replaced or complemented by other types of sensors, for example, for measuring temperature, physical contact, capacitance, etc. One or more of the regular keys on the keyboard may be replaced by 'smart' keys comprising a sensor for detecting contact with or nearness of a user's finger or hand.

Instead of, or in addition to, a beeper or a vibration mechanism, the user may be alerted by means of a glowing light 303 behind and shining through the various parts of the keyboard. This very effectively draws the user's attention to his wrong behavior in anon- intrusive manner. Additionally, after a few more seconds, a beeper (not shown) may provide a more insistent signal.

Figure 4 shows a flow chart of a possible implementation of the method according to the invention. In step 401, a timer is reset and started. In step 402, it is determined whether substantial input from the input device is received. If such input is received, the timer is reset again in step 401. If no input is received, it is determined whether the user's hand is detected near the input device. If the user's hand is detected, the timer is reset again in step 401. If no user's hand is detected, it is determined whether a predetermined time period has elapsed wherein no substantial input was detected and the user's hand was continuously sensed near the input device. If this predetermined time period has not elapsed yet, the process returns to step 402, otherwise characteristics of an alert signal, e.g. color or sound level, are initialized in step 405. Subsequently, a user-perceptible alert signal with the initial characteristics as set in step 405 is generated in step 406. Steps 407 and 408 are similar to steps 402 and 403, i.e. therein it is checked whether substantial user input is detected or whether the user's hand has been removed from the input device. If either of these conditions is true, the flow returns to step 401 wherein the time is reset, otherwise the characteristics of the alert signal are changed in a predetennined way in step 409, for example, the alert signal is gradually intensified until a maximum value is reached, and steps 407 and 408 are performed again. Consequently, if the user's hand is sensed near the input device while no substantial operation of the input device is detected for some predetermined period of time, e.g. 3 seconds, an alert signal with initial characteristics is produced. If the same situation persists, the alert signal is continuously produced with gradually changing characteristics, e.g. with gradually increasing sound level, until the user removes his hand or actually operates the input device.

While the invention has been described in connection with preferred embodiments, it will be understood that modifications thereof within the principles outlined above will be evident to those skilled in the art, and thus the invention is not limited to the preferred embodiments but is intended to encompass such modifications. The invention resides in each and every novel characteristic feature and each and every combination of characteristic features. Reference numerals in the claims do not limit their protective scope. Use of the verb "to comprise" and its conjugations does not exclude the presence of elements other than those stated in the claims. Use of the article "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. 'Means', as will be apparent to a person skilled in the art, are meant to include any hardware (such as separate or integrated circuits or electronic elements) or software (such as programs or parts of programs) which perform in operation or are designed to perform a specified function, be it solely or in conjunction with other functions, be it in isolation or in co-operation with other elements. A 'computer program' is to be understood to mean any software product stored on a computer- readable medium, such as a floppy disk, downloadable via a network, such as the Internet, or marketable in any other manner.

Claims

CLAIMS:

1. A data processing system comprising: a data processing apparatus (101) for executing an interactive application, an input device (109, 110) for generating signals in response to user operation of the input device and transferring the signals to the data processing apparatus, a sensor (111) for sensing the nearness of a user's hand to the input device, an activity monitor (112) for determining a user's activity with respect to the input device, and alert means (113) for generating at least one user-perceptible signal in response to the sensor sensing the nearness of the user's hand during a period of inactivity.

2. A system as claimed in claim 1, wherein the user-perceptible signal is at least one of an auditory, a visual or a tactile signal.

3. A system as claimed in claim 1, wherein at least one characteristic of the at least one user-perceptible signal depends on the duration of sensing the nearness of the user's hand during the period of inactivity.

4. A system as claimed in claim 3, wherein said dependence of said characteristic is user-adjustable.

5. A system as claimed in claim 3, wherein said dependence of said characteristic is a function of the number of times the user's hand is sensed during periods of inactivity.

6. A system as claimed in claim 3, wherein tlie alert means (113) are arranged to generate multiple user-perceptible signals, of which the at least one characteristic depends in different ways on the duration of the period of inactivity.

7. A system as claimed in claim 1, wherein the input device (109, 110) is arranged to command the data processing apparatus (101) to generate the user-perceptible signal.

8. A system as claimed in claim 1, wherein the sensor (111) is a touch sensor.

9. An input device for generating signals in response to user operation of the input device and transferring the signals to a data processing apparatus comprising: a sensor for sensing the nearness of a user's hand to the input device, - an activity monitor for determining the user's activity with respect to the input device, and alert means for generating at least one user-perceptible signal in response to the sensor sensing the nearness of the user's hand during a period of inactivity.

10. A data processing apparatus for executing an interactive application, the data processing apparatus comprising: an input for receiving input signals from an input device, said input signals comprising sensor data signaling the nearness of a user's hand to the input device, an activity monitor for determining a user's activity with respect to the input device, and alert means for generating at least one user perceptible signal in response to the sensor sensing the nearness of the user's hand during a period of inactivity.

11. A method of enabling a user to interact with a data processing apparatus, the method comprising the steps of: generating signals in response to user operation of an input device and transferring the signals to the data processing apparatus, sensing the nearness of a user's hand to the input device, determining a user's activity with respect to the input device, and - generating at least one user-perceptible signal in response to the sensor sensing the nearness of the user's hand during a period of inactivity.

12. A computer program product enabling a programmable device when executing said computer program product to function as a system as defined in any one of claims 1 to 8.