WO2004082248A1

WO2004082248A1 - Configurable control of a mobile device by means of movement patterns

Info

Publication number: WO2004082248A1
Application number: PCT/IB2004/050186
Authority: WO
Inventors: Hauke Schramm
Original assignee: Philips Intellectual Property & Standards Gmbh; Koninklijke Philips Electronics N.V.
Priority date: 2003-03-11
Filing date: 2004-03-02
Publication date: 2004-09-23

Abstract

A mobile device and a method for control of a mobile device are presented. The device comprises at least one sensor for detection of a movement, rotation and / or acceleration of the device and for delivering a sensor signal. In a recording mode, a movement pattern is recorded and a device function is defined by the user, which is assigned to the movement pattern. The sensor signal (S (t)) is continuously assessed and compared with the movement pattern in an operating mode, where the defined device function is triggered on a predefined match.

Description

Configurable control of a mobile device by means of movement patterns

The invention relates to a mobile device and a method to control a mobile device.

A large number of electronic mobile devices is known, among inter alia, mobile telephones, organizers, palmtops as well as audio and video playback devices. These devices are mobile, i.e. they can be easily carried around by a user in one hand and are not cable-bound.

The operation of these units is through an operating interface, which usually includes optical indicators such as LEDs, LCD displays and/or a screen as well as input means such as keys, touch screen or an input area, which reacts to touches, for example by using a pen. Speech control is also known now to a limited extent.

JP 2000-312255 describes a mobile telephone which has an acceleration sensor. When a mobile telephone moves, the acceleration sensor delivers a signal and a recognition unit recognizes a corresponding acceleration pattern. The acceleration pattern is compared with a previously stored acceleration pattern. On matching, a speech, recognition unit in the mobile telephone is activated.

It is an object of the invention to propose a mobile device and a method for controlling a mobile device, which will make easy and individual operation of a mobile device possible.

This objective is achieved by a mobile device according to claim 1 and a method according to claim 9. Dependent claims relate to advantageous embodiments of the invention.

According to the invention, the mobile device has a device operating unit, at least one sensor, a recording unit, an assigning unit and a comparing unit. These units are first to be rega.rded as purely functional, i.e. they need not necessarily be separate sub- assemblies, but the function of several of said units can be realized within one or more common sub-assemblies. Preferably, the electronic device has at least one microprocessor, on which at least parts of the functions of the device operating unit as well as the functions of recording, assigning and comparing unit are realized in the form of a program running on it. The device functional unit represents the actual device function: This may be, for example, the operation of the mobile device as a mobile telephone. In this case, the device operating unit includes all the functions of a known mobile telephone as well as the sub- assemblies required for it. Similarly, the device operating unit can also include the functions of a playback device for compressed digital music files (for example MP3) or of a CD player. The device operating unit can also include the functions of a palmtop computer or organizer. Many other functions can also be considered.

Furthermore, the mobile device has at least one sensor for detecting the movement of the device. A suitable sensor can detect, for example, a rotation, linear movement and/or acceleration of the device. Preferably, several sensors are provided which produce sensor signals corresponding to the acceleration in different spatial directions or the rotation around different axes. The signal delivered by the sensors reflects the movement of the device.

According to the invention, the device can be controlled through movements. To do this, the user moves the mobile device by hand. The movement is recognized by the sensor. Different movements, such as circling, shaking, turning to and fro can be differentiated with the help of the resultant sensor signal. The temporal sequence of the movements is designated here as a movement pattern.

The device can be brought into a recording mode by recording a movement pattern. The recording mode can be activated for example by pressing a key as a start signal and similarly ended by pressing a key as an end signal. There may also be provided that the recording mode terminates automatically after a predefined time span. It is also possible that several sensor signals are assessed for recording a movement pattern, i.e. the respective movement of the device is carried out several times by the user and the corresponding sensor signals are processed to take certain deviations in the repeated execution of the same movement into consideration. In the recording mode the signal of the sensor(s) is assessed. A representation of the signal is stored in a memory. This may be for example sample values of the signal so that the time of the signal is mostly completely stored. Preferably, the sensor signal is subdivided however into semi-stationary sections (called "windows") for which a spectrum is computed. The intensity of the signal in different spectral areas is stored in the form of a vector (called 'feature vector') which represents the signal in the pertinent window. The sequence of all feature vectors can be used directly as representation of the total signal. It can, however, also be used for setting (called "training") of the parameters of a statistical model (for example, called Hidden Markov Model, HMM). In this case, the sensor signal would be represented by the statistical model and its parameters. The use of statistical models for signal representation is particularly advantageous, if more than one recording of the movement pattern is available.

According to the invention, the user can define an assignment between a movement pattern and a device operation to be triggered by it. The assigning unit is used for this purpose. The definition is stored. The defining is done preferably in the form of a dialog, where the user selects one from a number of offered device operations and links it to a stored movement pattern. The assignment of several different movement patterns to one operation is also possible.

In an operating mode of the device, the sensor signal is assessed continuously. A comparing unit serves for continuous comparison of the current sensor signal with the recorded movement pattern or the stored representation. If it is established that the current sensor signal corresponds to the recorded movement pattern or the representation exactly or with a defined deviation, then the device operation assigned to the movement pattern is triggered. For the comparison between sensor signal and recorded movement pattern, it is possible to use various known methods for example, use of the neural networks or Hidden Markov Models as described in the book "Pattern Classification" by Duda, Hart, Stork, Second Edition, Wiley-Interscience Publication. Preferred are known pattern comparison methods, as they are used in speech recognition.

The mobile device can very individually be adapted for operation by the user in the described manner. For a certain device operation often used by the user, for example, querying the voice mailbox, the user can define along its longitudinal axis, for example, a brief shaking of the telephone with to and fro movements. If the user wishes to query his voice mailbox in the operating mode, i.e. form the normal stand-by mode of the mobile telephone, he makes the appropriate movement with the mobile telephone. The movement is recorded by the sensors and recognized as matching with the recorded movement pattern. This automatically triggers the device operation i.e. calling the voice mailbox.

Preferably not only a movement pattern is recorded, but several different movement patterns can also be trained i.e. recorded. Each of these movement patterns is assigned by the user to a desired device operation. As a result, a very individual operation is achieved, easy for the user, by means of which often-needed operations can be triggered without the user having to direct his attention to the keyboard. The user can also achieve easily an intuitive operation of the device in such a manner that, for example, a strong shake of an audio playback device can lead to interruption of the current playback and start of a playback of a new audio file selected randomly from the memory. Particularly for young people, it could be interesting to configure their device by assigning a personalized movement pattern vocabulary such that operating them is practically possible only for them. The control method as invented has major advantages even for people who are not in a position to use the device in any other form, say due to a handicap.

These and other aspects of the invention are apparent from and will be elucidated by way of non-limiting example, with reference to the embodiments described hereinafter. In the drawings: Fig. 1 shows a mobile device in a three-dimensional coordinates system,

Fig. 2 shows a schematic representation of sub-assemblies of the mobile device of Fig. 1,

Fig. 3 shows time sequences for sensor signals, Figs. 4 to Fig.6 show displays of a mobile device.

Fig. 1 shows a mobile telephone as an example of a mobile device. A coordinate system with X, Y and Z axes is shown in Fig. 1 in relation to device 10, where the Y axis runs in the longitudinal direction of the oblong casing of the mobile telephone 10. The X, Y and Z axes are mutually orthogonal. The angles of rotation ψx, ψγ_; ψz are defined around the coordinates X, Y and Z respectively.

Acceleration sensors are arranged in the mobile telephone 10, which detect positive and negative accelerations a_x, ay, az in the direction of the X, Y, Z axes and rotational accelerations ψx, ψγ₅ ψz around these axes. Such sensors are known as such and therefore not explained any further here.

Fig. 2 shows the functional units of the mobile telephone 10 in a schematic manner. The mobile telephone 10 has a display 12 and a keyboard 14. The actual device functions of the mobile telephone 10 are brought together in the device operating unit 16. The details of the mobile telephone operating unit 16 are sufficiently known as such and therefore not explained any further here.

The sensors, here generally called S, a recording and comparing unit 18 and a memory 20 are provided for the special control of the device operating unit 16 by means of movement patterns. The sensors S deliver the respective time-dependent sensor signals (t), ay (t), az, ψx (t), ψy (t), ψz (t) to the unit 18, which processes these signals as shown below. Fig. 3 shows by way of example a possible time sequence of these signals as obtained on a respective movement of the mobile telephone 10 in the coordinate system from Fig. 1. The signal of the sensors S is combined in a vector S (t). Differentiation is made between recording mode and operating mode for the operation of the mobile telephone 10 by movement patterns. The recording mode does the recording of a movement pattern which is assigned to a device function. The operating mode corresponds to the usual operation of the device operating unit 16, where the sensor signal S (t) is continuously observed and compared to the stored movement pattern. In the illustrated example, the mobile telephone is placed in the recording mode by keying in on the keyboard 14. Fig. 4 shows how a respective message on the screen 12 prompts the user to activate the recording mode. The recording is started by pressing a key 22. The sensor signal S (t) is recorded.

As shown in Fig. 5, the user is prompted to end the recording by pressing a key 24.

The recording of the signal S (t) takes place first by sampling the signal and storing the sampling value. After terminating the recording, characteristic features of the recorded sensor signals are stored in a memory as shown below.

The sensor signals (a(t), ψ(t)) are sub-divided into quasi-stationary sections ("windows") lasting for example 10 ms. For each signal and each window, a spectrum is computed by using known methods from digital signal processing (DFT). The intensity of the signals in different spectral areas (filter bank) is recorded in the form of a vector. The vectors of the different sensor signals pertaining to a window thus obtained, when written one below the other, form a new vector, the what is called feature vector. This describes the movement of the device in a small tune window. The sequence of all the feature vectors pertaining to the respective windows represents the recorded movement and is used for training the parameters of a Hidden-Markov-Model (HMM). This training is carried out with known methods from speech recognition (Baum-Welch algorithm). The only difference with speech recognition is that the sequence of the feature vectors represents not a voice signal but the movement of the device.

The thus obtained characteristic features (parameters of the HMM) as representation of the recorded movement pattern are stored in the memory 20. The corresponding movement pattern can be recorded by the user through only a one-time training. In general, however, better results are obtained if the same movement pattern is executed multiple times in the recording mode and the corresponding sensor signal S (t) is recorded and used for training the pertinent HMM.

After recording the movement pattern appears the prompt for the user as shown in Fig. 6, to assign the just recorded movement pattern to one of the device operations offered in the list. A respective function can be selected by pressing the key 22 or 24.

In this manner, a number of different movement patterns can be trained and assigned to respective device operations. The number of trainable movement patterns is restricted only by the memory space required for them. The operation of the mobile telephone, when the above-described recording does not take place, is designated as operating mode. This comprises the stand-by mode in the mobile telephone 10, but may also include the operating mode in which a telephone conversation is conducted.

The possibly pre-processed (i.e. extraction of the above-described feature vectors) sensor signal S (t) is reviewed continuously in the operating mode, if it matches one of the recorded movement patterns. The checking for a match of the movement patterns is similarly carried out with known techniques. When the HMM-Technology is used, the probability of the respective model generating the respective signal section is determined, for example, at every moment for each of the previously trained movement pattern-HMMs and each possible signal section ending at the time t. The model with the highest probability is selected ("recognized"), or none in case this probability lies below one of the previously settable sensitivity thresholds. If none of the models shows the minimum probability required, a new attempt is made at time t+1. Otherwise, the history of the signal is erased (because there cannot be any overlapping patterns) and the search for other patterns is started anew at t+1. The selection of a model means that a movement pattern was recognized. In this case, the recording and comparing unit 18 control the operating unit 16 appropriately, such that the device function linked with the recognized movement pattern is executed. Whereas the invention was explained at the outset with a mobile telephone as an example, an expert can easily recognize that the invention can also be applied to other mobile devices.

Claims

CLAIMS:

1. A mobile device equipped with at least one device operating unit (16) and at least one sensor (S) for detecting a movement of the device (10) and for delivering a sensor signal (S (t)), and a recording unit (18) with a memory (20) for recording a movement pattern in a recording mode by assessing the sensor signal (S (t)) and by storing a representation of it, and an assigning unit for inputting a device function, which is assigned to the movement pattern, and a comparing unit (18) for continuous assessment of the sensor signal in an operating mode, for comparison of the sensor signal (S (t)) with the recorded movement pattern, and for control of the device operating unit (16) for execution of the assigned device operation for a defined match.

2. A device as claimed in claim 1, in which - the device (10) has a screen (12) and a keyboard (14), where the assigning unit processes inputs from the keyboard (14) in a dialog for inputting a device function and displays feedback on the screen (12).

3. A device as claimed in any one of the preceding claims in which - the recording unit (18) processes a key depression as a start signal for the beginning of the recording.

4. A device as claimed in claim 3, in which the recording unit (18) processes a key depression as an end signal for the end of the recording.

5. A device as claimed in claim 3, hi which the recording unit (18) has a time measuring facility, and terminates the recording after a predefined time span elapses.

6. A device as claimed in any one of the preceding claims in which - the recording unit (18) assesses sensor signals (S (t)) several times for the recording of a movement pattern, and stores the common characteristic features of the sensor signals (S (t)).

7. A device as claimed in any one of the preceding claims in which - the device operating unit (16) is designed for recording and/or reproduction/playback of image, sound and / or data signals.

8. A device as claimed in any one of the preceding claims in which several sensors (S) are provided for the detection of movements, rotations and / or acceleration of the device (10) in different directions or around different axes (X, Y,

Z), and several sensor signals (ax, a_γ, a_z, ψx, ψy, ψz) can be delivered and assessed on recording the movement pattern.

9. A method for control of a mobile device (10) with at least one sensor (S) for detection of a movement of the device (10) and for delivering a sensor signal (S (t)), where the user activates a recording mode, in which a movement pattern is recorded, where the sensor signal (S (t)) is assessed and a representation of it is stored in a memory, - and the user defines a device function, which is assigned to the movement pattern, and the sensor signal (S (f)) is continuously assessed and compared with the movement pattern in an operating mode, where the defined device function is triggered on a match.

10. A method as claimed in claim 9, in which several movement patterns are recorded and are assigned to one or more device functions.