WO2009071128A1

WO2009071128A1 - Control device with a heart rate sensor and a motion sensor

Info

Publication number: WO2009071128A1
Application number: PCT/EP2007/063469
Authority: WO
Inventors: Mario El Khoury; Rolf Vetter
Original assignee: Csem Centre Suisse D'electronique Et De Microtechnique Sa - Recherche Et Developpement
Priority date: 2007-12-06
Filing date: 2007-12-06
Publication date: 2009-06-11
Also published as: EP2217338A1

Abstract

A system comprising a heart rate sensor (3) for delivering a sensor signal; a signal processing device (5) for correcting artifacts due to user's motion in said sensor signal; and a motion sensor (4) delivering a motion reference signal, the motion sensor (4) being characterized by having a main function for controlling a process in said system and an auxiliary function for controlling said signal processing device (5).

Description

Control device with a heart rate sensor and a motion sensor

Field of the invention

The present invention concerns a control device such as a handheld controller or footboard comprising a heart rate sensor and a motion sensor, able to provide a heart rate signal corrected for the user's motion artifacts and a motion signal to control an electronic game or fitness activity process.

Description of related art

There is an interest in controlling an electronic game or fitness activity where both, the heart rate and the movement of a user are used as parameters in order to control the game or fitness activity.

An interactive game apparatus comprising a heart rate measurement device worn by the user and which pulse signal is transmitted to the game apparatus as a parameter for effecting an interactive game between user and game machine is known.

For example, EP1275420 discloses an interactive game apparatus using the heart rate signal measured by a heart rate detection means worn by the user to interact with the game parameter such as the game time period and speed.

In EP674927, the level of difficulty of an electronic game is adjusted in response to a signal that measures the galvanic skin resistance between two points on the player's skin.

In EP890338, an electrocardiogram (ECG) sensor is attached to a player's body and produces a heart rate signal used to adjust the play of a video game, either independently or in conjunction with other game controls such as control signals from a joystick. A drawback with heart rate measurements on a user in movement is, indeed, the corruption of the heart rate signal by user movements, so-called motion artifacts. These artifacts lead to erroneous interpretation of heart rate signals and degrade the accuracy and reliability of the estimation of cardiovascular parameters.

In order to solve this problem, the present applicant has proposed in US7018338 to add a motion sensor, a three-dimensional accelerometer, to filter out motion artifacts from a heart rate signal determined with an optical method using an infrared light source.

WO2007053146 claims a method and portable device for generating a heart rate signal corrected for the user's motion contribution where the device comprises a pressure or piezoelectric sensor or a force transducer for measuring the heart rate and an accelerometer or a pressure sensor to detect the user's motion.

In WO2007072239, a watch-like device for monitoring a user's heart rate comprises two electrodes placed along the surface of a wrist strap, in contact with the user's skin, in order to measure changes in an electrocardiogram (ECG) signal. The device further comprises an accelerator for determining an acceleration of the user's arm and compensate for the arm movement in the heart rate signal. The compensated ECG signal may be used as a parameter in computer games. The accelerator signal itself, however, is not used as a game parameter.

The use of a motion sensor to control the process of a game or a fitness activity and to filter out the motion artifacts from a heart rate signal is, however, not provided in the above mentioned documents.

Brief summary of the invention

An object of the invention is therefore to propose a new system and method which overcomes at least some limitations of the prior art. According to the invention, these objectives are achieved by means of a system and method comprising the features of the independent claims, preferred embodiments being indicated in the dependent claims and in the description.

Those objectives are also achieved by a control device comprising a heart rate sensor for delivering a sensor signal; a signal processing device for correcting artifacts due to user's motion in said sensor signal; and a motion sensor delivering a motion reference signal.

The motion sensor is characterized by having a main function for providing a motion reference signal used to control a system such as an electronic game or fitness activity process and an auxiliary function where the motion reference signal is used in a signal processing device to determine from the sensor signal, a heart rate signal corrected for movement artifacts.

In an embodiment of the invention, the control device is a handheld controller and the heart rate signal and motion reference signal are used in combination to control an electronic game level of difficulty, speed, duration, complexity, etc.

In another embodiment, the control device is used for a fitness activity where the motion reference signal is used to determine the user's movement amplitude and frequency while the heart rate signal controls for example, the exercise duration and intensity. Here, the control device can be a footboard.

Brief Description of the Drawings

The invention will be better understood with the aid of the description of an embodiment given by way of example and illustrated by the figures where:

Fig. 1 shows a schematic view of the system;

Fig.2 represents a possible setting of the control device. Detailed Description of possible embodiments of the Invention

Figure 1 illustrates schematically one embodiment of the invention where a system contains a control device 1 comprising a housing 2, a heart rate sensor 3, a motion sensor 4 and a signal processing device 5. The system also comprises a transfer device 6 for transferring the signals produced by the motion sensor 4 and the heart rate sensor 3 to an external console 7.

The control device 1 is, for example, a handheld controller which may contain operating buttons and keys for interacting with the entertainment activity process. Such handheld controller can be held in one hand as, for example, the Wii remote controller or the Nunchuk, produced by Nintendo or possibly held in both hands such as, for example, the Nintendo N64 controller.

The heart rate sensor 3 is preferably based on an electro-optic technique, such as photoplethysmography (PPG) described in documents US7018338 and US717560. In the PPG technique, the heart pulse wave caused by the periodic pulsations of arterial blood volume is measured by the changing optical absorption of radiant energy. The PPG based heart rate sensor 3 comprises an emitter, preferably an infrared light emitting device (LED), and a receiver, preferably a photodiode, for detecting the intensity of the radiant energy after propagation through the user's skin.

In an exemplary embodiment shown in Fig. 2, the heart rate sensor 3 is placed on a security armband 8 which is part of the control device. Here, the emitter 9 and the receiver 10 can be placed along the security armband 8, on the side which is in contact with the user's skin. Preferably, the receiver 10 comprises a pair light detectors 11 disposed on each side of the emitter 9, formed by one light source, at a determined distance from the light source of the emitter 9, for example, about 10 mm. Other possible arrangements of the emitter and receiver can comprise one or several light sources and light detectors respectively. The spatial arrangement of the light detectors should be selected in a way to provide sufficient spatial diversity in order to remove artifacts due to tissue inhomogeneities.

The security armband 8 of Fig. 2 further comprises a signal acquisition module 12, into which the sensor signal is transferred, comprising, for example, an analog to digital (A/D) converter and signal filtering and shaping means. The acquired signal outputted from the acquisition module 12 is transferred to the control device 1, more particularly to the signal processing device 5 through a cable 13. The security armband 8 can be powered by a battery (not shown) contained in the control device 1. The cable 13 can be connected to the control device 1 through a USB connector or any other type of connector.

In another embodiment, the acquisition module 12 can be placed in the control device 1 or, alternatively, the functionalities of the acquisition module 12 can be performed by the signal processing device 5 located within the control device 1.

In another embodiment, the sensor or acquired signal, outputted from the heart rate sensor 3 or from the acquisition module 12 respectively, is transferred to the signal processing device 5 in a wireless fashion. In this case, the security armband 8 is powered with a battery contained in the security armband 8.

In another embodiment, the emitter 9 and the receiver 10 are disposed on the housing 2 of the control device 1 in the parts where the user's hands are placed.

In another embodiment of the invention (not represented), the heart rate sensor 3 is based on an ECG technique using at least two electric potential electrodes. Here, one electrode can be placed on the security armband 8 worn on one user's wrist and the other electrode can be disposed on another strap (not shown) worn, for example, on the other user's wrist, both electrodes being in contact with the user's skin. In another embodiment, both electrodes can be placed on the security armband 8, in a similar fashion to the apparatus described in WO2007/072239. Both electrodes can also be placed on a chest belt. Optionally, the electrodes can be located on the housing of the handheld controller, preferably, on areas of the housing 2 that are grabbed by the right and left hands respectively. Flexible electrodes, for example such as described in US20060142654, can also be integrated into the security armband and/or the strap tissue or into a garment fabric. The electrodes can be connected to a comparator (not shown) for determining a difference value between the measured electric potentials at the different electrode locations. In addition, the heart rate sensor 3 may comprise a signal acquisition module 12 outputting an acquired signal that is transferred to the signal processing device 5 through a cable 13 or wirelessly.

In another embodiment, the heart rate sensor 3 can be based on an impedance cardiography (ICG) technique or a combination of at least two of the electro-optic technique, ECG or ICG techniques.

In another embodiment, the heart rate sensor 3 is placed on a chest belt and the sensor signal outputted by the heart rate sensor 3 is transferred wirelessly to a wristwatch, similarly to some of the heart monitoring systems proposed by Polar Electro Oy. Here, the wristwatch may comprise a visual display for displaying a heart rate and a motion sensor 4 for evaluating the user's position, speed, etc. The motion sensor 4 may alternatively be placed in the user's shoes.

The motion sensor 4 is preferably a MEMS-based three dimensional accelerometer composed of three accelerometers disposed along three orthogonal measurement axis and providing three dimensional acceleration data representative of the acceleration to which the device is subjected. For example, the accelerometer may be of the type manufactured by the company Colibrys S. A. under reference MS 6100. It will however be appreciated that other types of accelerometers or motion detecting devices can be used provided they deliver a reliable measure of motion. For example, the motion sensor 4 could be a gyro-sensor of any suitable technology incorporating a one or two dimensional accelerometer, or a rotating or vibrating element.

The main function of the motion sensor 4 is to control a process in the system, here, for example, an electronic game or a fitness activity. In the case the motion sensor 4 is a three dimensional accelerometer it will detect acceleration signals along the three orthogonal measurement axis. These acceleration signals can be processed in the signal processing device 5 in order to generate a reliable motion reference signal representative of the movement, position and/or attitude of the control device and/or the user. For example, by detecting static linear acceleration (i.e., gravity), the linear acceleration output of the motion sensor 4 can be used to infer tilt of the object relative to the gravity vector by correlating tilt angles with detected linear acceleration.

Alternatively, the motion sensor 4 may include an embedded signal processor (not shown) or other type of dedicated processors for performing any desired processing of the acceleration signals output from the accelerometers therein prior to outputting signals to the signal processing device 5.

The signal processing device 5 is preferably housed within the control device 1 and can be an adequately programmed digital signal processor or DSP or a general purpose microcontroller that also controls the control device 1 . Alternatively, an external signal processing unit 5 can be an external signal processing unit 14 located, for example, within the console 7, and linked to the control device 1 by means of a direct or wireless connection.

The acquired signal outputted from the acquisition module 12 is affected by movement artifacts which may prevent the direct extraction of heart rate information from this acquired signal. The acquired signal is then further processed to correct it for movement artifacts. In order to obtain a robust heart rate signal in a large variety of conditions, a signal enhancement method, exploiting the information contained in the motion reference signal is used. The method assumes that under rhythmical movements, artifacts on the acquired signal appear as harmonics of the movement frequency. The method comprises the steps below.

In a first step, the fundamental movement frequency is extracted from the motion reference signal through one of the following technique: zero-crossing, parametric or non-parametric spectral estimation, autocorrelation, recurrence plots.

In a second step, the fundamental movement frequency extracted in the first step is used to enhance the acquired signal. Here, for example, a multiple notch filter can be employed to attenuate harmonics of movement contributions in the acquired signal. Alternatively, non- parametric enhancement can be used where the movement contribution is cancelled, for example, by putting high resolution restricted discrete Fourier transform estimations at zero from nearest lower minimum to nearest upper minimum at each harmonic of movement frequency.

In a third step, heart rate signals are extracted from the enhanced signal using a zero-crossing, parametric or non-parametric spectral estimation, autocorrelation or recurrence plot method.

In a fourth step, the reliability of the heart rate signals is assessed in order to merge and/or discard heart rate signals based on a reliability indicator and provide a reliable heart rate signal corrected for motion artifacts.

In some conditions, the user's fundamental movement frequency may become synchronized with the user's heart rate. This may happen, for example, when the frequency of the steps of a jogger approaches its heart rate. In this case, the fundamental movement frequency of the motion reference signal overlaps completely the heart rate component of the sensor signal and the extracted fundamental movement frequency can be used instead of the enhanced signal to estimate the user's heart rate signal. As soon as the user's movement is no more in phase with its heart rate, the user's heart rate signal can be again estimated using the enhanced signal as in the method described above.

In a first embodiment, the processing method described above is performed completely in the signal processing device 5. Preferably, the heart rate signal is transferred to the console 7 through a wireless transfer device 6 such as Bluetooth, 802.11 UWB (ultra wide band), infrared, and the like, or through a wired connection. The transfer device 6 or the wired connection can also be used to transfer the motion reference signal determined from the motion sensor 4. One advantage of processing the acquired signal completely on the signal processing device 5 is that a simple signal, such as a pulse/min value, is transmitted between the control device 1 and console 7.

In a second embodiment, the processing of the acquired signal is completely or partially performed by the external signal processing unit 14 located within the console 7. This embodiment has the advantage of offering more calculation power since a more powerful processor may be used in the console 7. However, a more complex signal is transmitted between the control device 1 and console 7 with a transfer rate possibly slower than when a simple signal is transferred.

In accordance with an aspect of the invention, there is provided a computer program product configured to be operable on the signal processing device 5 or the external signal processing unit 14 located within the console 7, in order to carry out the processing of the acquired signal to correct the acquired signal for motion artifacts. The processing is performed according to the method described above when the program is executed by said signal processing device 5 or said external signal processing unit 14. The software product can be downloaded in a memory 15 associated with the signal processing device 5 or associated with the external signal processing unit 14. The downloading operation can be performed using a storage reader device (not shown), such as a CD or DVD reader, etc., integrated on the console 7, or an removable storage reader device (not shown), such as an external CD or DVD reader, a USB memory key or a flash memory, connected to the console 7 or the control device 1 through a USB connector or any other type of connector. The downloading operation can also be performed in a wireless fashion.

The control device 1 may further comprise an output device 16 for outputting an indication of the detected pulse rate in the form of an optical, audible signal, or other sensorial signal. Such means could be a LED, a display, a buzzer, a vibrating device or any other suitable device adapted for transmitting information representative of the pulse rate measurement to the user. Additionally, the output device 16 may also comprise alarm means for generating an alarm when the heart rate signal reaches a determined threshold, which could be either a low or high threshold or both.

The motion reference signal and heart rate signal can be used in combination with an electronic game or fitness activity process. For example, a motion reference signal representative of the position and/or attitude of the controller device 1 can be used as a pointer able to interact with an object or a region on a screen of the game console 7. The heart rate signal can then be used as a supplementary parameter, for example, indicating the level of excitement of the user or the level of difficulty of the activity. For example, the speed and/or duration of the activity, the complexity of the activity tasks, the type and/or number of obstacles or challenges, etc., can be adjusted in response to the heart pulse signal.

The motion reference signal can also be used to translate the user's movements, such as the number of steps, jumps, jogging, biking or any other movements, in a parameter in an activity involving physical exercising. Here, for example, the user would be invited to produce a series of jumps or any other movements until his heart rate reaches a given level. The motion sensor 4 would then be used to count the number of jumps, or movements, and deliver a motion signal used to obtain heart rate signal free from motion artifacts. In another exemplary embodiment, the heart rate signal and the motion reference signal can be used as parameters in a fitness activity in order to obtain optimal benefit of the fitness activity, for example, allowing the user to define his objectives such as losing weight or improving fitness.

More particularly, in the case the user whishes to lose weight, requirements in movement amplitude and frequency would be adjusted in a way to ensure the user's heart rate to stay in the range of, say, 60% to 75% of his maximal heart rate during the whole fitness activity period. Conversely, if the user's goal is improving fitness, requirements in movement amplitude and frequency would be adjusted to cause the user's heart rate to fall in different level ranges for different activity periods such as, for example :

• 65% to 80% of his maximal heart for 70% of the activity period; • 80% to 90% of his maximal heart for 25% of the activity period;

• 90% to 100% of his maximal heart for 5% of the activity period.

Here, the change from an activity period to another can be triggered by a combination of the motion reference signal and the user's heart rate signal. These signals can be transferred through the transfer device 6 to the console 7 where, for example, the console screen would produce visual effects representative of the fitness activity level.

In the example above, the control device 1 can be a handheld device or a portable device fixed on the user's body (waist, arm, leg, etc.). In order to achieve an activity with high intensity, the handheld or portable control device 1 can be replaced by a footboard such as, for example, a running belt or the Wii balance board produced by Nintendo. Here, the motion sensor 6 located within the running belt or footboard should be able to respond adequately to the impulsions produced by the user's feet, for example, by using a pressure sensor. Alternatively, the motion sensor 6 could be placed in a wrist strap connected through a wire or in a wireless fashion to the footboard or, eventually, to the console. It is understood that the present invention is not limited to the exemplary embodiments described above and other examples of implementations are also possible within the scope of the patent claims.

For example, the control device 1 can be a steering wheel with the heart sensor 3 installed on areas of the steering wheel that is grabbed by the right hand and/or left hand, such as described in EP1749477. The motion sensor 4, placed within the steering wheel would then sense the rotational and lateral movements of the steering wheel.

Reference numbers

1 Control device

2 Housing

3 Heart rate sensor

4 Motion sensor

5 Signal processing device

6 Transfer device

7 Console

8 Security armband

9 Emitter

10 Receiver

1 1 Light detectors

12 Acquisition module

13 Cable

14 External signal processing unit

15 Memory

16 Output device

Claims

1. A system comprising a heart rate sensor (3) for delivering a sensor signal; a signal processing device (5) for correcting artifacts due to user's motion in said sensor signal; and a motion sensor (4) delivering a motion reference signal, the motion sensor (4) being characterized by having :

• a main function for controlling a process in said system and,

• an auxiliary function for controlling said signal processing device (5).

2. The system of claim 1, where the motion reference signal delivered by the motion sensor (4) is exploited in the signal processing device (5) to determine a heart rate signal corrected for the movement artifacts from the sensor signal.

3. The system of claims 1 and 2, where said process controlled by the motion sensor (4) corresponds to an entertainment game process or a fitness activity process.

4. The system according to any of the claims 1 to 3, where the heart rate signal is used in combination with the motion reference signal to control said process, said process being an entertainment game process or a fitness activity process.

5. The system according to any of the claims 1 to 4, where the signal processing device (5) and the motion sensor (4) are placed in a control device (1).

6. The system according to any of the claims 1 to 5, where the control device (1) is a handheld controller.

7. The system according to any of the claims 1 to 5, where the control device (1) is a footboard.

8. The system according to any of the claims 1 to 7, where the heart rate sensor (3) is based on one the techniques comprising: electrocardiogram (ECG), impedance cardiography (ICG) or an electro-optic technique or a combination of at least two of these techniques.

9. The system according to any of the claims 1 to 8, where the heart rate sensor (3) is placed on a security armband (8) of the control device (1).

10. The system of claim 9, where the security armband (8) is linked to the control device (1) or a console (7) by means of a wired or wireless connection.

1 1. The system of claims 9 or 10, where the security armband (8) comprises an acquisition module (12) for acquiring and filtering the sensor signal.

12. The system according to any of the claims 1 to 8, where the heart rate sensor (3) is disposed on the housing (2) of the control device (1).

13. The system according to any of the claims 1 to 12, where the heart rate sensor (3) comprises a light source as emitter (9) and one pair of light detectors (1 1) as receiver (10).

14. The system according to any of the claims 1 and 13, where the heart rate sensor (3) comprises two electric potential electrodes.

15. The system according to claim 14 where one of the two electrodes is placed on the security armband (8) worn on one user's wrist and the other electrode is placed on another strap worn on the other user's wrist.

16. The system according to claim 14, where the two electrodes are placed on the security armband (8).

17. The system according to any of the claims 14 to 16, where the two electric potential electrodes are flexible electrodes.

18. The system according to claim 17, where the two flexible electrodes are integrated into a garment fabric.

19. The system according to any of the claims 1 to 18, where the motion sensor (4) is a three dimensional accelerometer.

20. The system of claim 19, where the acceleration signals outputted by the motion sensor (4) are processed in the signal processing device (5) in order to provide a motion reference signal representative of the movement, position and/or attitude of the control device and/or the user.

21. The system according to any of the claims 1 to 20, where the heart rate signal and the motion reference signal are transferred to a console (7) through a wireless transfer device (6) or through a wired connection.

22. The system according to any of the claims 1 to 21, where the motion reference signal delivered by the motion sensor (4) is exploited in an external signal processing unit (14) located within a console (7) to determine a heart rate signal corrected for the movement artifacts from the sensor signal.

23. Method for processing a sensor signal using a motion sensor (4) available in a control device (1) and having a main function for controlling an entertainment game process or fitness activity process and an auxiliary function where a motion reference signal provided by the motion sensor (4) is used in a signal processing device (5) to determine a heart rate signal corrected for the movement artifacts from the sensor signal.

24. Method according to claim 23, where the processing of the sensor signal is aimed at correcting the sensor signal for motion artifacts and comprises the steps of: a) detecting the sensor signal using the heart rate sensor (3); b) providing a motion reference signal using a motion sensor (4) during the sensor signal detection period; characterized by: c) extracting a fundamental movement frequency from the motion reference signal; d) enhancing the detected sensor signal with the fundamental movement frequency using linear or non-linear, model-based noise cancelling techniques; e) extracting heart rate signals from the enhanced sensor signal; f) assessing of reliability of extracted heart rate signals ; g) merging and/or discarding heart rate signals based on the reliability assessed in the previous step in order to obtain a reliable heart rate signal corrected for motion artifacts.

25. Method according to claims 23 or 24, where the fundamental movement frequency is extracted using one of the techniques comprising: zero-crossing, parametric spectral estimation, non-parametric spectral estimation, autocorrelation, recurrence plots.

26. Method according to any of the claims 23 to 25, where the measured sensor signal is enhanced using one of the techniques comprising: a model-based signal enhancement or a non-parametric enhancement such as discrete Fourier transform.

27. Method according to any of the claims 23 to 26, where the heart rate signal is determined using the fundamental movement frequency extracted according to step c) for claim 24, if the fundamental movement frequency approaches the user's heart rate.

28. Computer carrier comprising program code portions to be executed by a signal processing device (5) in order to carry out the method of one of the claims 23 to 27 when said program is executed by said signal processing device (5).

29. Computer carrier according to claim 28, where the program code portions are executed by an external signal processing unit (14) located within a console (7) in order to carry out the method of one of the claims 23 to 27 when said program is executed by said external signal processing unit (14).