WO2017216043A1

WO2017216043A1 - A movement rehabilitation system and method

Info

Publication number: WO2017216043A1
Application number: PCT/EP2017/064052
Authority: WO
Inventors: Sheng Jin; Jin Wang
Original assignee: Koninklijke Philips N.V.
Priority date: 2016-06-16
Filing date: 2017-06-09
Publication date: 2017-12-21
Also published as: CN109310912A

Abstract

A movement rehabilitation system provides a representation of a target movement to a user, together with a permitted range of deviation from the target movement. An image is displayed showing the performance of the user of the system in attempting to perform the target movement. A score is generated in respect of the ability of the user of the system to perform the target movement, and this score is worsened (e.g. reduced) in real time in response to deviations beyond the permitted range of deviation.

Description

A movement rehabilitation system and method

FIELD OF THE INVENTION

This invention relates to a movement rehabilitation system and method.

BACKGROUND OF THE INVENTION

Stroke is one of the leading causes of death in the world and a vast majority of survivors from stroke suffer from a number of dysfunctions. The most common one is motor dysfunction.

Long term rehabilitation is needed to help patients improve motor ability. However, after suffering from a stroke, patients can only receive rehabilitation therapy with the guidance and help of a therapist in a hospital for a short period of time.

For example, patients may be hospitalized for about one month and then discharged to home due to the finite number of therapists and rehabilitation centers. To help those patients who do not receive enough rehabilitation therapy to reach their maximum possible level of recovery, unsupervised rehabilitation systems have been developed. With the help of an unsupervised rehabilitation system, patients can practice a scheduled rehabilitation program outside of hospital.

A stroke rehabilitation system is used to guide and monitor post-stroke patients to do rehabilitation movement exercises and assess the performance and quality of training by tracking the patient's movement. For motivating patients and evaluating patient progress, a scoring mechanism is important for this kind of system.

An effective method for motion capture is a visual based tracking system which utilizes visual markers and a camera. The combination of an infra-red camera and retro -reflective markers is currently used to monitor patients' movements during

rehabilitation exercises. A camera continuously captures images and markers, for example placed at the shoulder, the elbow, and the hand. These markers can be detected in the captured images because the reflective marker materials make them brighter than other parts of the body and background environment. Using image processing techniques, the upper limb motion can be tracked. A method to evaluate the training performance hence to derive a training score for stroke rehabilitation is to track the trajectory of patient limb movement and compare it to a standard trajectory. For example, for a particular exercise the hand movement will generate an associated trajectory and there is also a standard trajectory assuming the movement is perfect. By comparing these two trajectories, a score can be derived based on similarity. A method called dynamic time warping can deal with this matching by stretching time and calculating the distance of corresponding points on both trajectories, and generate a whole trajectory matching score from 0 to 100 (full score 100).

However, a matching score like 86 or 64 is less meaningful for patients since they don't know exactly how these scores are obtained. It is hard to correlate this kind of score with patient's movement. This is demotivating for the patient, since it is not clear how they need to change their movements in order to increase their score and the score is obtained only when the whole exercise is completed.

There is therefore a need for a rehabilitation system which can track the position of the fixed points of the patient, and provide feedback about the performance of the patient in a simple but meaningful way for the patient.

SUMMARY OF THE INVENTION

The invention is defined by the claims.

According to examples in accordance with an aspect of the invention, there is provided a movement rehabilitation system, comprising:

an image capture system for capturing an image of a user of the system;

a display device; and

a controller,

wherein the controller is adapted to control the display device to:

output a representation of a target movement;

output in graphical form a permitted range of deviation from the target movement;

output a representation of the performance of the user of the system in attempting to perform the target movement;

output a score in respect of the ability of the user of the system to perform the target movement,

wherein the controller is adapted to worsen the score in real time in response to deviations beyond the permitted range of deviation; wherein the controller is adapted to implement the output of an alarm each time the score is worsened, the alarm indicating a change of the score.

This system provides an intuitive way for a user to know how well they are able to perform target movements instantaneously, for example as part of a rehabilitation program. The user is able to visually identify the causes of a reduced score, which stem from their movements falling outside a permitted range. This permitted range of deviation forms an allowed area within which the patient movement can take place. The permitted range may be different for different parts of the movement or it may be based on a single margin of permitted error. In this way, the user has a better understanding of their final score and a better understanding of how they can improve their movement during the exercise in order to obtain an improved score at the end of the exercise. As the instantaneous and detailed feedback, which may includes the change of the score, is provided to the user once the score is worsened, the user will be stimulated to perform better in next action in the exercise to gain back some points the user has already lost. This feedback makes the movement exercise more engaging, so the user of the system is more proactive in the exercise and more likely to continue to follow the exercise program.

The change of the score is associated to the deviation level or the body part that deviates. Generally the score is worsened accordingly as the movement is deviated from the target movement. However, such worsening may not be based on a linear relationship between the score change and deviation, but based on a predetermined rules/guideline.

Weights may be applied to the score change according to the different deviation amount. Furthermore, the score change may also be associated to the body part that deviates. The deviation of a target body part, for example a hand for rehabilitation, may cause a greater score deduction than a non-target body part, for example a shoulder.

The target movement is a standard trajectory template, which depends on the exercise, as well as the range of movements which the patient is hoping to achieve.

The controller may be adapted to implement the output of an alarm each time the score is worsened.

This makes sure the user is aware of the incidents that have contributed to a lowering of their performance score. Thus, if the trajectory followed by the patient is out of the permitted region, an alarm is generated and the score is altered (reduced) by way of punishment.

The representation of the target movement may be static. An area may be displayed, for example representing the path that needs to be followed by the hand or elbow. This path is then visible throughout the exercise, and the patient tries to follow a trajectory (for example of the hand) within the displayed area.

Alternatively, the representation of the target movement may be dynamic.

In this way, the representation of the target movement also guides the user in real time. For example, there may be an arrow indicating the specific point along the path that the user should be aiming for at that particular time. Feedback may be used to ensure that the representation of the target movement advances at the same pace as the movement performed by the user.

The controller is for example adapted to implement a calibration phase which involves input of the body dimensions of the user of the system which are relevant to the target movement.

By taking account of the physical size and shape of the user, the representation of the target movement is a closer fit to the actual movement which can be achieved by the user. The target movements may be standard exercises for given classes of patient, but they may also be tailored to specific individuals.

The controller may be adapted to tailor the target movement and optionally also the permitted range of deviation to the user of the system based on the calibration.

The range of deviation may be large or small depending on the progress made by the user of the system. This can be considered as different levels of difficulty to keep the user engaged.

The system for example comprises a set of markers for application to body parts of the user of the system.

These body markers can be tracked, and they may for example be provided on joints such as the elbow, shoulder, wrist etc. This simplifies the image processing needed to capture the relevant posture and limb positions of the user of the system.

The controller may be adapted to control the display device to output an initial maximum score and to deduct a penalty each time there is a deviation or predetermined set of deviations beyond the permitted range of deviation.

This emulates a game scenario, with the user trying to maintain a maximum score, such as 100.

The system may comprise a database which stores a set of standard target motions and associated permitted ranges of deviation.

The system may thus be used for many different rehabilitation exercises and may also store user-specific information for a set of users. Examples in accordance with another aspect of the invention provide a movement rehabilitation method, comprising:

capturing an image of a user of the system;

controlling a display device to:

output a representation of a target movement;

output a representation of the performance of the user of the system in attempting to perform the target movement; and

wherein the score is worsened in real time in response to deviations beyond the permitted range of deviation.

The method may comprise outputting an alarm each time the score is worsened. This makes the user aware of the incidents that have resulted in a lowered score.

The representation of the target movement may be static or dynamic.

The method may comprise performing a calibration phase which involves input of the body dimensions of the user of the system which are relevant to the target movement, wherein the method comprises tailoring the target movement and optionally also the permitted range of deviation to the user of the system based on the calibration.

An initial maximum score may be displayed, and a penalty may be deducted each time there is a deviation or predetermined set of deviations beyond the permitted range of deviation.

The invention may be implemented at least in part using computer software.

BRIEF DESCRIPTION OF THE DRAWINGS

Examples of the invention will now be described in detail with reference to the accompanying drawings, in which:

Figure 1 shows in schematic form a patient with one arm for which movement rehabilitation is being carried out lowered by their side;

Figure 2 shows a movement rehabilitation system;

Figure 3 shows a first example of an output provided by the system;

Figure 4 shows a second example of an output provided by the system;

Figure 5 shows a third example of an output provided by the system;. Figure 6 shows a fourth example of an output provided by the system; and Figure 7 illustrates an example of a computer for implementing the controller of the system.

DETAILED DESCRIPTION OF THE EMBODIMENTS

The invention provides a movement rehabilitation system in which a representation of a target movement is provide to a user, together with a permitted range of deviation from the target movement. An image is displayed showing the performance of the user of the system in attempting to perform the target movement. A score is generated in respect of the ability of the user of the system to perform the target movement, and this score is worsened (e.g. reduced) in real time in response to deviations beyond the permitted range of deviation.

Figure 1 shows in schematic form a patient with one arm (for which movement rehabilitation is being carried out) lowered by their side. The patient wears a hand marker 10, an elbow marker 12 and a shoulder marker 14. An image processing system is used to identify the positions of the markers, and thereby create a model of the arm position of the user. The arm is only one example. The system may be used for leg movement exercises or indeed other movements.

Figure 2 shows a movement rehabilitation system. It comprises the set of markers comprising the shoulder marker 14, elbow marker 12 and hand marker 10 and an image capture device or system 20 for capturing an image including the markers.

A controller 22 includes an image processing system which determines the arm position and movement from the markers. The marker positions detected (possibly also as well as interpolated during occlusion) are presented on a display device 24.

Figure 3 shows a first example of an output provided by the display device.

The display device provides a representation 30 of the target movement. This is for example a trajectory, e.g. a straight or curved path, which is to be followed by a particular part of the body. Typically, it is the path to be followed by the hand or foot since this is most easily controlled by the user.

In addition, a permitted range of deviation from the target movement is displayed. This may comprise an identified region 32 around the target movement 30.

Based on the image processing, a representation 34 of the performance of the user of the system in attempting to perform the target movement is also shown. This comprises the actual path followed by the body part (e.g. hand). A score 36 is presented in respect of the adherence of the user of the system to perform the target movement. Each time the user fails to stay within the region 32, the score is worsened, e.g. reduced from an initial maximum score of 100 as shown in Figure 3.

The representation of the performance of the user may include an image 38 of the user, either based on a captured actual image of the user, or else based a graphical representation derived from the tracked markers.

If the user moves relative to the image capture system while performing the task, the representation 30 of the target movement and identified region 32 move

accordingly, so that the target movement is spatially coupled to the representation of the user. This provides the most intuitive feedback.

The representation 34 of the performance of the user and the representation 38 of the user, may comprise a reflection of an image of the user rather than a computer- generated image. By way of example, a system which combines a reflection of a real image with a displayed computer-generated overlay image is disclosed in US 8 328 691.

The representation 38 may be a schematic representation such as a stick-man, or a generic body image or an accurate computer-generated image of the actual patient. The representation of the user and their performance needs to be sufficiently clear that the user can intuitively link the representation to the control of their body, and thus try to follow the target movement as presented on the display.

The representation 30 of the target movement needs to take account of the physical size of the user, and optionally also the state of their injury or disability.

For this purpose, the controller 22 comprises a database 23 which stored patient-specific information.

During system start up, a calibration process is followed to determine patient- specific parameters several such as upper arm length, lower arm length, elbow, shoulder and hand starting positions, etc. Depending on the disability being treated, the upper and lower leg length may also be input. Information about the nature of the disability may also be stored, such as the patient's current range of movement, and an ultimate target range of movement.

According to the identification of the user and their patient-specific

parameters, for a current exercise to be practiced, a standard trajectory template is generated in order to set the target movement, and thereby generate the representation 30 of the target movement. According to a predefined distance threshold, which may be patient-specific or may be generic, the identified region 32 around the target trajectory is generated. As explained above, the target trajectory and the region 32 of interest are displayed, together with a representation of the whole patient, or the relevant limb of the patient, or even only the joint and/or limb end (such as the hand) which needs to follow the target trajectory.

The patient then attempts to move the relevant limb so that the relevant part of the limb (e.g. the hand) follows the trajectory. If the trajectory followed by the patient ventures outside the identified region 32 of interest, a real-time feedback alarm is generated and the score is reduced. The feedback alarm may comprise the information about how much the score is worsened.

The score may be reduced each time the patient trajectory is outside the region 32, or it may require a set number of events before the score is reduced. Alternatively, the patient trajectory may need to be outside the region of interest for a certain time, or for a certain path length, for the alarm and score reduction to be effected.

Figure 3 shows the identified region 32 as having a fixed distance from the target movement in the form of a line path. In this case, it is possible to determine that the patient trajectory is outside the region 32 whenever the shortest distance between the patient trajectory and the target trajectory exceeds a threshold.

After the exercise training, in addition to showing the score, the full patient trajectory may be shown together with an indication of the events that led to a score reduction.

In the example of Figure 3, the whole target trajectory is shown in static form. An icon such as an arrow 40 may be used to make clear to the patient how to move along the target trajectory.

The target trajectory may instead by dynamic. By this is meant that it the representation evolves over time as the patient performs the exercise. This enables the patient to focus on the particular part of the exercise being conducted.

Figure 4 shows a display output where the identified region 32a (i.e. the permitted range of deviation) is only shown near the current user position, based on feedback from the image capture system. In this way, the permitted range is dynamically shown. It may have the same deviation distance from the target as it evolves dynamically, or it may change. The change may be based on different parts of the exercise being more difficult, or it may be based on analysis of the user's previous performance. In this way, the permitted deviation from the target path may evolve over time so that the patient is gradually trained to perform better and better.

Figure 4 shows that a part of the path 34 has left the region 32a. This generates a warning 42 which may be visual and/or audible. It also means the score 36 has been reduced in value.

Figure 5 shows that different parts of the target trajectory may have different permitted deviations so that there are multiple sections 32b, 32c to the identified region. These may be displayed simultaneously as in Figure 3 or sequentially as in Figure 4.

Figure 6 shows that the permitted range of deviation may be more complex, with more sections 32d, 32e, 32f. Some sections are either inherently more difficult or else more difficult as a result of the circumstances of a particular patient.

The system described above makes use of a controller for processing image data and driving the display.

Figure 7 illustrates an example of a computer 70 for implementing the controller or processor described above.

The computer 70 includes, but is not limited to, PCs, workstations, laptops, PDAs, palm devices, servers, storages, and the like. Generally, in terms of hardware architecture, the computer 70 may include one or more processors 71, memory 72, and one or more I/O devices 73 that are communicatively coupled via a local interface (not shown). The local interface can be, for example but not limited to, one or more buses or other wired or wireless connections, as is known in the art. The local interface may have additional elements, such as controllers, buffers (caches), drivers, repeaters, and receivers, to enable communications. Further, the local interface may include address, control, and/or data connections to enable appropriate communications among the aforementioned components.

The processor 71 is a hardware device for executing software that can be stored in the memory 72. The processor 71 can be virtually any custom made or

commercially available processor, a central processing unit (CPU), a digital signal processor (DSP), or an auxiliary processor among several processors associated with the computer 70, and the processor 71 may be a semiconductor based microprocessor (in the form of a microchip) or a microprocessor.

The memory 72 can include any one or combination of volatile memory elements (e.g., random access memory (RAM), such as dynamic random access memory (DRAM), static random access memory (SRAM), etc.) and non-volatile memory elements (e.g., ROM, erasable programmable read only memory (EPROM), electronically erasable programmable read only memory (EEPROM), programmable read only memory (PROM), tape, compact disc read only memory (CD-ROM), disk, diskette, cartridge, cassette or the like, etc.). Moreover, the memory 72 may incorporate electronic, magnetic, optical, and/or other types of storage media. Note that the memory 72 can have a distributed architecture, where various components are situated remote from one another, but can be accessed by the processor 71.

The software in the memory 72 may include one or more separate programs, each of which comprises an ordered listing of executable instructions for implementing logical functions. The software in the memory 72 includes a suitable operating system (O/S) 74, compiler 75, source code 76, and one or more applications 77 in accordance with exemplary embodiments.

The application 77 comprises numerous functional components such as computational units, logic, functional units, processes, operations, virtual entities, and/or modules.

The operating system 74 controls the execution of computer programs, and provides scheduling, input-output control, file and data management, memory management, and communication control and related services.

Application 77 may be a source program, executable program (object code), script, or any other entity comprising a set of instructions to be performed. When a source program, then the program is usually translated via a compiler (such as the compiler 75), assembler, interpreter, or the like, which may or may not be included within the memory 72, so as to operate properly in connection with the operating system 74. Furthermore, the application 77 can be written as an object oriented programming language, which has classes of data and methods, or a procedure programming language, which has routines, subroutines, and/or functions, for example but not limited to, C, C++, C#, Pascal, BASIC, API calls, HTML, XHTML, XML, ASP scripts, JavaScript, FORTRAN, COBOL, Perl, Java,

ADA, .NET, and the like.

The I/O devices 73 may include input devices such as, for example but not limited to, a mouse, keyboard, scanner, microphone, camera, etc. Furthermore, the I/O devices 73 may also include output devices, for example but not limited to a printer, display, etc. Finally, the I/O devices 73 may further include devices that communicate both inputs and outputs, for instance but not limited to, a network interface controller (NIC) or

modulator/demodulator (for accessing remote devices, other files, devices, systems, or a network), a radio frequency (RF) or other transceiver, a telephonic interface, a bridge, a router, etc. The I/O devices 73 also include components for communicating over various networks, such as the Internet or intranet.

When the computer 70 is in operation, the processor 71 is configured to execute software stored within the memory 72, to communicate data to and from the memory 72, and to generally control operations of the computer 70 pursuant to the software. The application 77 and the operating system 74 are read, in whole or in part, by the processor 71, perhaps buffered within the processor 71, and then executed.

When the application 77 is implemented in software it should be noted that the application 77 can be stored on virtually any computer readable medium for use by or in connection with any computer related system or method. In the context of this document, a computer readable medium may be an electronic, magnetic, optical, or other physical device or means that can contain or store a computer program for use by or in connection with a computer related system or method.

As mentioned above, the image capture system may use reflective markers. These may be at the joints but they may also be between the joints. The system may use multiple markers at the different joints. If markers are occluded, interpolation may be used to provide a continuous representation of the movement. The system may instead operate using image analysis without needing the patient to wear markers.

In the example above, the score decreases from a maximum. However, it may increase from a best value of zero. Furthermore, the scope does not need to be numeric. It may be descriptive (for example with a finite number of different descriptive levels, such as excellent, good, average etc.). It may also be graphical, showing different images or image colors for different performance levels. The score may include or be supplemented by an indication of the change in performance compared to previous exercises. The nature of the output may be configurable by the user, to provide the type of output that they find most encouraging.

The example above is for monitoring one movement. However, multiple synchronized movements may be monitored, so that both coordination and range of movement are exercised. Thus, there may be simultaneous movement tasks for both arms, or both legs, or one arm and one leg. The range of permitted deviations may then take account of the required synchronization between target movements e.g. of the two hands.

The system is not limited to limb movements. It may be used for whole body movements, such as bending back and forth or side to side. The system may adapt the target movements based on the previous performances of the user, or else they may be set by a healthcare professional at periodic intervals. The user may also have the option to adjust the range of permitted deviations, for example if they wish to advance more quickly.

The output is described above as a display. Of course, the output may be supplemented with sound, for example including speech generation to provide instructions, performance information or motivational encouragement.

Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope.

Claims

CLAIMS:

1. A movement rehabilitation system, comprising:

an image capture device (20) for capturing an image of a user of the movement rehabilitation system;

a display device (24); and

a controller (22) coupled to the image capture device (20),

wherein the controller is adapted to control the display device to:

output a representation of a target movement (30);

output in graphical form a permitted range (32) of deviation from the target movement;

output a representation (38) of the performance of the user of the system in attempting to perform the target movement based on the captured image;

output a score (36) in respect of the ability of the user of the system to perform the target movement,

wherein the controller is adapted to worsen the score in real time in response to deviations from the performance of the user beyond the permitted range of deviation;

wherein the controller (22) is adapted to implement the output of an alarm each time the score is worsened, the alarm indicating a change of the score.

2. A system as claimed in claim 1, wherein the change of the score is associated to the deviation level or the body part that deviates.

3. A system as claimed in claim 1, wherein the representation (30) of the target movement is static.

4. A system as claimed in claim 1, wherein the representation (30) of the target movement is dynamic.

5. A system as claimed in claim 1, wherein the controller (22) is adapted to implement a calibration phase which involves input of the body dimensions of the user of the system which are relevant to the target movement.

6. A system as claimed in claim 5, wherein the controller (22) is adapted to tailor the target movement and optionally also the permitted range of deviation to the user of the system based on the calibration.

7. A system as claimed in claim 1, comprising a set of markers (10, 12, 14) for application to body parts of the user of the system.

8. A system as claimed in claim 1, wherein the controller (22) is adapted to control the display device (24) to output an initial maximum score and to deduct a penalty each time there is a deviation or predetermined set of deviations from the performance of the user beyond the permitted range of deviation.

9. A system as claimed in any preceding claim, comprising a database (23) which stores a set of standard target motions and associated permitted range of deviation.

10. A movement rehabilitation method, comprising:

capturing an image of a user of the system;

controlling a display device (24) to:

output a representation (30) of a target movement;

output a representation (38) of the performance of the user of the system in attempting to perform the target movement based on the captured images; and output a score (36) in respect of the ability of the user of the system to perform the target movement,

wherein the score is worsened in real time in response to deviations from the performance of the user beyond the permitted range of deviation; outputting an alarm each time the score is worsened, the alarm indicating a change of the score.

11. A method as claimed in claim 10, wherein the change of the score is associated to the deviation level or the body part that deviates.

12. A method as claimed in claim 10, further comprising performing a calibration phase which involves input of the body dimensions of the user of the system which are relevant to the target movement, wherein the method comprises tailoring the target movement and optionally also the permitted range of deviation to the user of the system based on the calibration.

13. A method as claimed in claim 10, comprising controlling the display device to output an initial maximum score and to deduct a penalty each time there is a deviation or predetermined set of deviations beyond the permitted range of deviation.

14. A computer program comprising code means which is adapted, when said program is run on a computer, to perform the method of any one of claims 10 to 13.