WO2010109383A1

WO2010109383A1 - Adapting interactive programs based on a physiological characteristic of a user performing physical activity

Info

Publication number: WO2010109383A1
Application number: PCT/IB2010/051178
Authority: WO
Inventors: Harm J. Buisman; Robert Pinter
Original assignee: Koninklijke Philips Electronics N.V.; Philips Intellectual Property & Standards Gmbh
Priority date: 2009-03-27
Filing date: 2010-03-18
Publication date: 2010-09-30

Abstract

There is provided a method comprising monitoring a physiological characteristic of a user performing physical activity in connection with an interactive program; and adjusting one or more parameters of the interactive program based on the monitored physiological characteristic.

Description

ADAPTING INTERACTIVE PROGRAMS BASED ON A PHYSIOLOGICAL CHARACTERISTIC OF A USER PERFORMING PHYSICAL ACTIVITY

TECHNICAL FIELD OF THE INVENTION

The invention relates to a method and system for adapting interactive programs.

BACKGROUND TO THE INVENTION

Interactive programs, including computer and video games, are known in which a user follows a series of on-screen or audio instructions or controls an on-screen character to complete certain tasks (the on-screen character can be, for example, an avatar of the user or an object or process that can be influenced by the user). Conventionally, the user provides input to the interactive program using buttons or control sticks on a hand held controller, buttons on a keyboard or via a mouse.

Recently, however, interactive programs have been developed that utilize motion sensors, such as accelero meters, for detecting the motion of the user or part of their body, and using this detected motion as an input. In this way, the user can control an on- screen character by performing the physical actions that they would like the character to perform, the user can perform actions to manipulate the on-screen object or process or the user can perform a physical action in accordance with an instruction from the interactive program. For example, the user can swing the handheld controller like a tennis racket or golf club, and this can be used to control an on-screen character that is playing tennis or golf. One example of a computer games console that implements this functionality is the Nintendo Wii™.

In this way, interactive programs have been made more immersive and more enjoyable for many people. However, it is desirable to further improve the experience of a user using such interactive programs.

SUMMARY OF THE INVENTION

In accordance with a first aspect of the invention, there is provided a method comprising monitoring a physiological characteristic of a user performing physical activity in connection with an interactive program; analyzing the monitored physiological characteristic to determine if the user should change the level of their physical activity and adjusting one or more parameters of the interactive program so as to cause the user to adjust the level of their physical activity.

A second aspect of the invention provides a system comprising a processor configured to receive measurements of a physiological characteristic of a user performing a physical activity in connection with an interactive program being executed by the system, analyze the monitored physiological characteristic to determine if the user should change the level of their physical activity and adjust one or more parameters of the interactive program so as to cause the user to adjust the level of their physical activity. A third aspect of the invention provides a computer program product comprising computer program code that, when executed on a computer or processor, is configured to perform the method described above.

BRIEF DESCRIPTION OF THE DRAWINGS Embodiments of the invention will now be described, by way of example only, with reference to the following drawings, in which:

Fig. 1 is a block diagram of a user and a system running an interactive program;

Fig. 2 is a block diagram of the system in accordance with the invention; Fig. 3 is a flow chart illustrating a method in accordance with the invention;

Fig. 4 is a flow chart illustrating a specific embodiment of the invention used in cardiac rehabilitation;

Figs. 5(a) and (b) are graphs illustrating the power spectral densities of a heart beat interval time series for a user that is asleep and during physical activity; and Fig. 6 is a flow chart illustrating a specific embodiment of the invention used in improving the enjoyment level of a user.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Although the invention will be described below with reference to an interactive program being executed on a computer games console, it will be appreciated that the invention can be applied to an interactive program executing on a general purpose computer, a set-top box, a mobile telephone, a PDA, a wristwatch, a laptop, or any other suitable portable or fixed device. Fig. 1 shows a user 1 interacting with a computer games console 2 that is executing an interactive program. The visual and audio output of the interactive program is provided to the user 1 by a display unit 4, such as a television or computer monitor, which is connected to the computer games console 2 via an audio -video cable 6. The computer games console 2 has at least one controller 8 for allowing the user 1 to provide specific inputs to the interactive program. The controller 8 includes a plurality of buttons and/or controls, and, in a preferred embodiment, one or more motion sensors for sensing the motion of the controller 8 by the user 1. In this illustrated embodiment, the controller 8 is connected wirelessly to the computer games console 2 (for example using Bluetooth or WiFi) but in other embodiments, the controller 8 can be connected through a wired connection to the console 2.

In accordance with embodiments of the invention, a sensor 10 is provided for measuring and monitoring one or more physiological characteristics of the user 1. In this embodiment, the sensor 10 is attached to the chest of the user 1 using a chest belt 12 or similar. In this embodiment, the sensor 10 is connected wirelessly to the computer games console 2 (again using Bluetooth or WiFi, for example), so that the measurements of the physiological characteristics are provided to the computer games console 2. The sensor 10 may also be capable of measuring the motion of the user 1, in addition to, or instead of, the controller 8 having a motion sensor. In some embodiments, the sensor 10 can be integrated into the controller 8, and the chest belt 12 can be omitted.

Fig. 2 shows a more detailed block diagram of the system in accordance with the invention. In this illustrated embodiment, the controller 8 comprises the motion sensor 14 and buttons and/or controls 16. The computer game console 2 comprises a processor 18 that receives signals from the controller 8 representing the pressing of buttons, operating of controls and/or motion of the controller 8, and signals from the physiological sensor 10 representing the monitored physiological characteristic(s). The computer game console 2 also comprises a medium 20 on which the interactive program is stored. The medium 20 may be fixed in the console 2, or may be removable from the console 2 so that the console 2 can execute different programs. The medium may be a hard disk, an optical disk, a solid-state memory or any other suitable type of storage medium.

The sensor 10 can measure and monitor any desired physiological characteristic or characteristics of the user 1, for example a heart rate, a heart rate variability (HRV), an electrocardiogram (ECG) signal, a breathing rate, skin temperature or skin resistivity. By pressing the buttons or controls on the controller 8 and moving either the controller 8 (if the controller 8 includes a motion sensor) or their whole body (if the sensor 10 can also sense motion) around, the user 1 can provide specific inputs to the interactive program executing on the computer games console 2. The user 1 can provide these inputs as a direct result of prompts from the interactive program (such as "press button A", "move controller clockwise", "jump up and down"), or they can be provided in order to control or influence some aspect of the interactive program, such as the motion of a character shown on the display unit 4.

The interactive program includes a number of parameters that can be varied to change the experience of the interactive program for the user 1. A common parameter provided in an interactive program is a difficulty level, and this indicates, generally, the level of skill and/or the level of physical activity required by the user 1 in order to progress through the program. The difficulty level can also specify how accurate particular movements need to be to a sample movement in order to correctly register in the program. Other common parameters, related to the difficulty level, can be the pace or speed the interactive program is presenting new information to the user 1 (for example how often new physical activity instructions are displayed for the user 1 , or how fast the character controlled by the user 1 can move). Further parameters can relate to the audio provided by the interactive program, messages displayed on the display unit 4 or the design of the environment depicted on the display unit 4 or otherwise provided to the user. The environment can include anything that can be sensed or perceived by the user (for example the design of the environment can relate to the visual or aural elements and/or tactile sensations through the controller 8).

In accordance with the invention, a parameter or parameters of the interactive program are adjusted based on the measurements of the physiological characteristic of the user 1 so as to cause the user to adjust the level of their physical activity. It should be noted that "level of physical activity" is intended to cover any suitable measure of physical activity of the user and includes, for example, the intensity of physical activity being undertaken, a total number of repetitions of an exercise that should be completed by the user or the number of repetitions required per unit time.

Fig. 3 illustrates a method in accordance with the invention. In step 101, a physiological characteristic of the user 1 is measured, and in step 103, a parameter or parameters of the interactive program are adjusted based on the measurement of the physiological characteristic. After step 103, the method returns to step 101 and repeats. The adjustment of the parameter or parameters, and the specific measurements of the physiological characteristic(s) that trigger these adjustments, will depend on the particular application of the invention. For example, as described in more detail below, the parameter or parameters can be adjusted in order to prevent the user 1 from over-exerting themselves while interacting with the program, to encourage the user 1 to use the interactive program for longer or to improve the enjoyment of the interactive program by the user 1. Further applications of the invention will be apparent to a person skilled in the art.

Cardiac Rehabilitation

Rehabilitation after a cardiac event, such as a heart attack or stroke, consists of, among other aspects, exercise, and this can be facilitated by a system as described above. The nature of the conditions affecting these users means that their heart rate or ECG signal needs to be monitored during exercise, in order to evaluate whether continuing the exercise is safe.

In hospitals and rehab clinics, phase I and phase II rehab take place respectively, where the patient is monitored by a health professional on location. However, in phase III rehabilitation, which occurs in the user's home, no health professional is supervising the patient. Although there are risks related to exercising, the user usually benefits from exercise in the long term, as this can prevent future cardiac events. However, without the health professional observing the user, the user will not know when to stop, or to lower their workload. This embodiment of the invention addresses this problem. This embodiment of the invention can also be used to provide clinical decision support for a health professional and/or for unqualified or junior health professionals. In this embodiment, which is described with reference to the flow chart in Fig.

4, the sensor 10 measures physiological characteristics of the heart of the user 1 (step 111). The sensor 10 can measure, for example, the heart rate, the heart rate variability or an ECG signal. The processor 18 analyses the measurements and determines whether the user 1 should lower their workload (steps 113 and 115). This determination can be carried out by comparing the measurements to a threshold or series of thresholds, or by inputting the measurements into an appropriate algorithm.

In an exemplary algorithm, the processor 18 determines that the user 1 should lower their workload if (heart rate - constant_l)/physical activity is greater than another constant. In an alternative algorithm, data is collected from a user performing a physical activity under observation by a physician, and this heart rate and activity data can be stored and used as an input to the algorithm. For example, the processor 18 can determine that the user 1 should lower their workload if the current heart rate/activity level is greater than the stored heart rate/activity level.

In a further alternative algorithm, where the sensor 10 measures an ECG signal, the processor 18 can determine that the user 1 should lower their workload if abnormalities, particularly related to the regularity of heart beats, are detected in the ECG signal. In a yet further alternative algorithm, the processor 18 can monitor particular frequency intervals in a heart rate variability power spectral density. Fig. 5 shows the power spectral density of a heart beat interval time series from a subject that is asleep (Fig. 5(a)) and a subject undertaking physical activity (Fig. 5(b)). Thus, it can be seen that under exercise conditions the high-frequency components disappear. In this algorithm, the different conditions of the user are identified by defining a series of frequency intervals, and examining the power that is contained in those frequency intervals. The following intervals can be used:

Total HRV power: 0 - 0.5 Hz

Ultra-low frequency (ULF) power: 0 - 0.0033 Hz Very low frequency (VLF) power: 0.0033 - 0.04 Hz

Low frequency (LF) power: 0.04 - 0.15 Hz

High frequency (HF) power: 0.15 - 0.4 Hz

Very high frequency (VHF) power: 0.4 - 0.5 Hz

This algorithm can be further modified by calculating the ratios between powers contained in different intervals, for example the LF/HF ratio. Clearly, this ratio would be small in Fig. 5(a), but large in Fig. 5(b).

If the user 1 does not need to lower their workload, the algorithm returns to step 111, and repeats until the user 1 finishes the interactive program or the interactive program is otherwise ended. If it is determined that the user 1 should lower their workload (for example because the heart rate is too high, or because the heart rate variability indicates that the heart is under too much strain), the processor 18 can adapt the parameters of the interactive program so that a warning message is displayed for the user 1 on the display unit 4, indicating that they should lower their workload (for example by slowing down their movements). In addition, or alternatively, an audible warning can be given to the user 1 that they should lower their workload.

It will be appreciated by a person skilled in the art how the parameter or parameters of the interactive program can be adapted by the processor 18 to provide the warning to the user 1. For example, a parameter within the program relating to the provision of warnings can be changed by the processor 18 from an "off state to an "on" state to provide the warning.

In step 119, the processor 18 analyses subsequent measurements of the physiological characteristics of the heart to determine if the characteristics have improved (i.e. has the heart rate reduced, or the heart rate variability reduced). If the characteristics have improved, the algorithm returns to step 111. If the characteristics have not improved (either because the user 1 has not lowered their workload enough, or has missed or ignored the warning), the processor 18 can adjust further parameters of the interactive program in order to get the user 1 to lower their workload (step 121). In particular, the processor 18 can change a difficulty level of the program, so that, for example, the user 1 does not have to move so quickly to complete a certain instruction (like running on the spot), or is not presented with new instructions as frequently. Those skilled in the art will be aware of other parameters of an interactive program that can be adjusted in order to reduce the workload of the user 1. In step 125, the processor 18 analyses subsequent measurements of the physiological characteristics to determine if the characteristics have improved. If the characteristics have improved, the algorithm returns to step 111. If the characteristics have still not improved, the processor 18 can terminate the program or current exercise routine (step 125). It will be appreciated, however, that many variations of this exemplary embodiment are possible, without departing from the scope of the invention.

In particular, as indicated by the dashed arrow 127 in Fig. 4, after the processor 18 determines in step 115 that the user 1 should lower their workload, the algorithm could move straight to step 121 in which further parameters of the interactive program are adjusted (such as the difficulty level). In this way, the processor 18 can dynamically adapt the interactive program without the user 1 necessarily being made aware of the changes. In addition, or alternatively, path 127 may be used if the measured physiological characteristic exceeds a permitted level by a significant amount, and it is necessary to quickly reduce the workload of the user 1. In a similar fashion, the processor 18 can terminate the program or exercise routine as soon as it is determined that the user should lower their workload in step 115 (as shown by path 129).

Furthermore, it is possible for step 121 to be repeated more than once (i.e. the parameter or parameters can be further adjusted) before moving to step 125 if the measured physiological characteristics do not improve, or do not improve significantly enough.

As an extension to the above, or as an alternative embodiment, it is possible for the processor 18 to adjust a parameter or parameters of the interactive program (such as a difficulty level) in order to increase the workload of the user 1 in the event that the analysis of the measured physiological characteristics shows that the user 1 is not working hard enough (i.e. they are not getting the full benefit of the exercise).

In a yet further embodiment, the algorithm executed by the processor 18 can be extended by providing a self-learning feature. In particular, the processor 18 can store data relating to each session of the user's activity with the interactive program (including the particular values of the parameters used, the physiological characteristic measurements, the progress of the user 1 through the program, etc.) and can use this data in the assessment carried out in step 115 of Fig. 4. For example, the processor 18 can use this data to identify a point or exercise in the interactive program that the user 1 has difficulty with (i.e. that causes undesirable changes in the measured physiological characteristics) and provide an appropriate advance warning to the user 1 , or, alternatively, the processor 18 can dynamically adjust the parameters of the program in order to try and prevent these undesirable changes occurring.

Moreover, the self- learning feature can also use the stored data relating to each session of the user's activity in order to optimize the interactive program to the user 1. For example, if it is desired for the user 1 to achieve a certain activity level while using the interactive program, the processor 18 can use the stored data to present the user 1 with the tasks or activities most likely to help the user 1 achieve those activity levels.

The specific thresholds or algorithm used by the processor 18 to determine whether the user 1 should lower their workload can be set or influenced by a healthcare provider, based on the healthcare provider's assessment of the user 1, which means that the invention can be adapted to the particular needs and requirements of a specific user 1.

If the system is connected to a remote monitoring station (for example at a clinic or hospital), the system can upload data relating to the performance of the activities by the user to a healthcare provider for review. In this way, it is possible for the healthcare provider to provide or download new settings to the system in order to adapt the specific thresholds or algorithm used by the processor 18, or to specify particular parameters for the interactive program (such as a starting difficulty level).

It will be further appreciated that this embodiment of the invention can be applied to interactive programs in general, not just interactive programs that are for use in cardiac rehabilitation.

Enhancing enjoyment and/or motivation

It has been found that providing the user 1 with a motion sensor 14 for interacting with the program improves the engagement and motivation of the user 1 with the program. However, the motion sensor 14 cannot provide any indication of the enjoyment level of the user 1, which is a key factor for keeping the user 1 motivated to follow the exercise regularly.

It has been shown in the past that a measure of heart rate variability, and in particular the power spectral density of an RR series in an ECG signal, can be used as an indicator of a person's state of mind.

Heart rate variability (HRV) refers to the beat-to-beat alterations in heart rate.

Under resting conditions, the heart rate of healthy individuals exhibits a periodic variation.

This rhythmic phenomenon, known as respiratory sinus arrhythmia (RSA), fluctuates with the phase of respiration: cardio-acceleration during inspiration and cardio-deceleration during expiration. Under certain conditions, the heart rate tends to synchronize with the person's breathing activity. Furthermore, the heart rate can also be correlated with the individual's current sensations and emotions.

The ECG signal is the electrical excitation of the heart muscle. The prominent peaks in the ECG signal are called the "R peaks", and every R peak indicates a heart beat. The timely differences between successive R peaks can be written down as a sequence of numbers, which is called the "RR series" of the ECG. The power spectral density can be calculated by applying the Fourier Transform to the autocorrelation function of the RR series. The calculation of the power density spectrum (PSD) of the autocorrelation function of the RR series with the help of the Fourier Transform is an established method for assessing a person's state of mind. In particular, it has been found that when a person is in a state of appreciation, there is an almost sinusoidal shape of the heart rate curve synchronized with the respiratory cycles, and the power spectral density shows a rather pronounced peak, whereas when the person is in a state of frustration, there is no regular heart rate pattern and the power spectral density is spread over a wide bandwidth. Therefore, in this embodiment of the invention, the measurements from the physiological sensor 12 are used to assess a stress level of the user 1, and the parameters of the interactive program are adjusted on the basis of the assessment to improve the enjoyment level of the user 1. In a preferred embodiment, the sensor 12 measures an ECG signal of the user

1. In addition, or alternatively, the sensor 12 can measure electrical skin resistance or a peripheral skin temperature. In particular, if the user 1 is relaxed and comfortable, sweat production is reduced, which increases the electrical skin resistance. Furthermore, the blood vessels widen, which means that more blood streams into the extremities (hands, feet etc.) and the temperature of the extremities increases. In these latter embodiments, it is preferable for the sensor 12 to be integrated into a finger clip.

An example of the algorithm that can be executed by the processor 18 is shown in Fig. 6. In step 131, a physiological characteristic is measured and in step 133 the measurement is analyzed to determine a stress and/or enjoyment level of the user 1. If the user 1 is stressed, or not enjoying the interactive program (at step 135), the processor 18 can adjust a parameter or parameters of the interactive program with the aim of improving the stress or enjoyment level of the user 1. The adjusted parameter could be, for example, a difficulty level of the interactive program, particularly if the processor 18 determines that the user 1 has made numerous attempts to clear the same part of the interactive program. Alternatively, the parameter could be related to a design of the program environment shown on the display unit 4, such as the color or complexity of a background.

The algorithm then returns to step 131. If, at step 135, the user 1 is not determined to be stressed and/or is determined to be enjoying the program, the parameters of the program can be left unchanged, and the algorithm returns to step 131. This embodiment of the invention can be applied to many different types of interactive programs and different scenarios, and can, for example, be combined with the cardiac rehabilitation embodiment described above in order to help motivate the user 1 to perform the rehabilitation exercises.

While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive; the invention is not limited to the disclosed embodiments.

Variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. A computer program may be stored/distributed on a suitable medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems. Any reference signs in the claims should not be construed as limiting the scope.

Claims

CLAIMS:

1. A method, comprising: monitoring a physiological characteristic of a user performing physical activity in connection with an interactive program; analyzing the monitored physiological characteristic to determine if the user should change the level of their physical activity; and adjusting one or more parameters of the interactive program so as to cause the user to adjust the level of their physical activity.

2. A method as claimed in claim 1, further comprising the step of: measuring the motion of the user or a part of the body of the user, and using the measurement of the motion as an input to the interactive program.

3. A method as claimed in claim 1 or 2, wherein the one or more parameters comprise a difficulty level of the interactive program, whether a message should be displayed or presented to the user, whether the interactive program should be terminated and/or the design of the environment provided to the user by the interactive program.

4. A method as claimed in claim 1, 2 or 3, wherein the physiological characteristic comprises a heart rate, a heart rate variability, an ECG signal, a breathing rate, skin temperature and/or skin resistivity.

5. A method as claimed in any of claims 1 to 4, wherein the step of adjusting comprises displaying or otherwise presenting a warning to the user indicating that they should adjust the level of their physical activity.

6. A method as claimed in claim 5, further comprising the step of analyzing subsequent measurements of the physiological characteristic to determine if the user has adjusted the level of their physical activity, and, in the event that the user has not adequately adjusted their level of physical activity, the method further comprises adjusting a difficulty level of the interactive program or terminating the interactive program.

7. A method as claimed in any of claims 1, 2, 3 or 4, wherein the step of adjusting comprises adjusting a difficulty level of the interactive program.

8. A method as claimed in any of claims 5, 6 or 7, further comprising the steps of: storing information relating to one or more events in which a parameter of the interactive program is adjusted; and providing a warning to the user that they should adjust the level of their physical activity if one of said events reoccurs.

9. A method as claimed in any of claims 5, 6 or 7, further comprising the steps of: storing information relating to one or more events in which a parameter of the interactive program is adjusted; and adjusting a difficulty level of the interactive program if one of said events reoccurs.

10. A method as claimed in any of claims 1 to 4, further comprising the step of analyzing the monitored physiological characteristic to determine if the user is stressed while using the interactive program, and the step of adjusting comprises adjusting the one or more parameters of the interactive program to reduce the stress of the user.

11. A system, comprising: a processor configured to: receive measurements of a physiological characteristic of a user performing a physical activity in connection with an interactive program being executed by the system analyze the monitored physiological characteristic to determine if the user should change the level of their physical activity and adjust one or more parameters of the interactive program so as to cause the user to adjust the level of their physical activity.

12. A system as claimed in claim 11, further comprising: a physiological characteristic sensor coupled to the processor, the sensor being suitable for attachment to a user and for measuring a physiological characteristic of the user.

13. A system as claimed in claim 11 or 12, wherein the system is one of a computer games console, a desktop computer, a laptop computer, a set-top box, a mobile telephone, a PDA or a wristwatch.

14. A computer program product, comprising computer program code that, when executed on a computer or processor, is configured to perform the method of any of claims 1 to 10.