JP2010536459A - System and method for displaying anonymized annotated physical exercise data - Google Patents

Abstract

  The present invention relates to a method for displaying anonymously annotated physical exercise data to a person performing the exercise. Based on the physical exercise data, the physical exercise data is annotated in physically separated annotation units. At the place of the person doing the exercise, a visual record of the person doing the exercise is displayed to the person, along with information on the synchronized annotations. A system for carrying out the method comprises a physical data processing unit 1, a display device 2, at least one attitude recording device 3, 3 ′, a visual recording device 4, a data storage device 5, A physically separated annotation unit 6 connected to a physical data processing unit 1 via an interconnected computer network 7.

Description

  The present invention relates to a system and method for displaying anonymously annotated physical exercise data to a person performing the exercise.

  Home rehabilitation exercises for people suffering from medical conditions such as strokes, or home training exercises for people who want to improve body movements such as golf swings are sensors Can be recorded via. Exercises can also be evaluated by professionals such as physiotherapists or golf instructors to direct direct feedback.

  During the exercise, if there is no specialist to check, a video camera record can be sent to him. These records can be checked intuitively by the expert, and the commented records can be intuitively understood by the person performing the exercise. However, these records may violate human privacy, especially when sent to specialists who are far away. Furthermore, fully automated processing of such recorded images to provide meaningful feedback is a laborious task.

  Alternatively, transmitting data from the sensor exclusively does not violate human privacy. In this regard, US Pat. No. 6,817,979 B2 relates to a system and method for providing information exchange with a user's virtual physiological model using a mobile communication device. Physiological data about the user is obtained from the person. The physiological data is preferably transmitted to the mobile communication device using a wireless communication protocol. This procedure further requires the use of a mobile communication device for communicating physiological data with the network server. Physiological data is accumulated into the user's virtual physiological model. The user can access the data and a virtual model depicted for the user made from physiological data.

  In an exemplary aspect, a user can create an avatar that represents the person's current physical condition. The user can modify the avatar to change the appearance of the avatar to a more desired appearance. For example, the anatomical dimensions of the avatar can be varied to reflect the desired waist, chest, upper arm, and thigh dimensions. Given the differences between desired avatar characteristics and current avatar characteristics, various training, diet, and related fitness recommendations help users achieve their desired fitness objectives. Can be made to build the most suitable training regimen. Physiological data is then acquired and applied to the user's avatar. The physiological data is compared with the desired avatar data to determine if the training regime is effective in achieving the desired fitness objective.

  However, in general, how to interpret front-end sensor signals is difficult for some users. It is difficult to relate (understand) the abstract representation of artificial screen characters.

  Thus, despite efforts for this association, there is a need for systems and methods in the art to display anonymously annotated physical exercise data to the person performing the exercise. .

To achieve this and other objectives, the present invention is directed to a method for displaying anonymously annotated exercise data to the person performing the exercise. The method is
a) collecting physical exercise data from the person performing the exercise;
b) gathering visual records about the person doing the exercise synchronously;
c) sending physical exercise data to physically separated annotation units;
d) annotating the physical exercise data with physically separated annotation units based on the physical exercise data;
e) sending annotation information to the display and the data processing unit for the person performing the exercise to check;
f) displaying a visual record of the person performing the exercise, along with synchronized annotation information, to the person.

  Before the present invention is described in detail, it is understood that the present invention is not limited to the specific components of the apparatus described, nor is it limited to the process steps of the described method. Should. This is because such devices and methods can vary. It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the specification and the appended claims, the singular forms “a”, “an”, and “the” refer to single and / or multiple indications unless the text clearly indicates otherwise. It should be noted that

  In the context of the present invention, the term anonymous annotated data means data for which the annotating third party has no knowledge of the identity of the person he is annotating. In particular, the data does not make it possible to identify a person. One aspect of realizing anonymization is assigning an identification number to the data. Physical exercise data is data about a person's movements or other exercises of a person.

  The first two steps of the method explain how two different sets of information about a person's exercise can be collected. First, physical exercise data is collected, for example, by continuously monitoring sensor signals from a person. At the same time, visual recordings are collected, for example using a digital video camera. Collecting this data in synchronization ensures that a certain portion of the video stream can later be determined to be that of a certain portion of the sensor signal stream, or vice versa.

  Since the visual recording and the physical exercise data are separate entities, the physical exercise data can be sent to physically separated annotation units. The fact that the annotation units are physically separated provides data anonymization. In the annotation unit, the physical exercise data can be data processed into an exercise display for review by a third party. The physical exercise data can then be annotated. Annotation includes automatic processing of data, for example, by detecting deviations from a motion template. Furthermore, third parties can include comments and suggestions to provide useful feedback to the person performing the exercise. Thereafter, the annotation information is transmitted to the display unit and the data processing unit at the site of the person performing the exercise. Here, the annotation information is combined with a visual record. The record of the person doing the exercise is displayed to the person along with the information of the next synchronized annotation. Synchronization provides for displaying annotations at the correct time so that the person can directly understand what attracted the attention of the person checking or the automated check system.

  In summary, with the method according to the invention, a person's exercise can be checked anonymously and feedback can be given to the person. Anonymization allows sharing of expert resources and makes the checking process more efficient. At the same time, when a person receives feedback, he is very clearly shown through visual records which part of the exercise prompted the feedback.

  In one embodiment of the invention, the avatar is calculated based on the physical exercise data in the physically separated annotation unit of step d). For the purposes of the present invention, the term “avatar” means a computer generated abstract drawing that represents a person's posture or movement. In a simple case, the avatar may be a stick line drawing. In more advanced cases, the avatar may represent additional information such as pulse rate, sweating, muscle fatigue, and the like. The advantage of using an avatar display is that the avatar can rotate on the screen of the annotation unit while displaying the exercise. This allows the checker to choose the best viewing angle to assess the exercise.

In another embodiment of the invention, step f) comprises
-Calculating the avatar;
-Additionally including the step of showing the avatar to the person in sync with the visual record and annotation. In summary, after this, a person will see visual records, annotations, and avatars about his exercise. This is an advantage because an avatar can more clearly represent a person's movement if the person's movement is not properly recorded by the camera, or if the movement is hidden in the visual recording by the loose clothing. is there. Again, the avatar can be rotated to achieve the best field of view. Another option is to serve one or more avatars at multiple viewing angles.

  In another embodiment of the present invention, the step c) transmitting physical exercise data, and the step e) transmitting annotation information include via an interconnected computer network, preferably the Internet. Done. This allows a remotely located person to check and annotate. Suitable protocols can include the TCP / IP protocol.

  In another embodiment of the present invention, physical exercise data from a person includes movement data, posture data, muscle exercise electronic data, pulse rate, blood pressure, blood oxygen content, blood sugar content, sweating volume, And / or selected from the group consisting of respiratory rates. These types of data are either related to the exercise itself, as in the case of motion data and posture data. Other types of data relate to a person's overall condition or physical fitness. Knowledge of this data may provide valuable insights into the effects of rehabilitation or training tools. For example, it can be inferred whether a person is in a super recovery phase after a training stimulus.

  In another embodiment of the invention, the note information is selected from the group comprising visual information, audio signals, and / or speech records. Visual information may be in the form of markings such as arrows that are inserted into the image of the avatar to point to a particular problem. In addition, a short video clip can be inserted to show the correct execution of the exercise. Other visual information is written comments or graphs showing data statistics such as electronic recording data of muscle movement, pulse rate, blood pressure, blood oxygen content, blood glucose content, sweating rate, and / or respiratory rate There is. This data makes it possible to assess at a glance the situation of the person performing the exercise. The audio signal can be a simple warning sound when the movement is not performed correctly. Comments from recorded speech can be added by the checker if this is the simplest way to describe the exercise.

The present invention is further directed to a system for displaying anonymously annotated physical exercise data to the person performing the exercise. The system
-A physical data processing unit;
A display device in communication with the physical data processing unit;
-At least one attitude recording device assigned to the person performing the exercise and communicating with the physical data processing unit;
-A visual recording device in communication with said physical data processing unit;
A data storage device for storing and extracting data from the physical data processing unit and the visual recording device and communicating with the physical data processing unit;
A physically separated annotation unit connected to a physical data processing unit via an interconnected computer network.

  In one embodiment of the present invention, the at least one posture recording device comprises a motion sensor for the person performing the exercise. The sensor is selected from the group comprising acceleration sensors, inertial force sensors, and / or gravity sensors. The motion sensor can be applied to a selected location, such as the upper arm, lower arm, upper leg, lower leg, or torso of a human torso. These sensors can be commercially available highly integrated solid state sensors. The transmission of the sensor signal to the posture evaluation unit can be performed via wired / wireless or in an area network of the human body using human skin conductivity. After calculating a person's posture, the result can be shown in the form of an avatar.

  In another embodiment of the present invention, the at least one posture recording device includes an optical mark that is attached to the person performing the exercise. The attitude recorder uses an optical tracking system for tracking at least one optical mark at the same time. Based on the signal of the optical tracking system, a representation of the person's posture is then calculated. The optical mark can be placed on a selected location such as the upper arm, lower arm, upper leg, lower leg, or torso of a human torso. Mark tracking can be done by a single camera or multiple cameras. When a stereoscopic camera is used, three-dimensional posture data and motion data are generated. After image processing and computation on the person's posture, the result can be shown in the form of an avatar.

  It is also possible to combine a plurality of attitude monitoring principles. For example, a combination of motion sensors and optical tracking can provide complementary data to better calculate a person's posture.

  A further aspect of the present invention is the use of the system according to the claims of the present invention for displaying anonymously annotated physical exercise data to the person performing the exercise.

  The present invention will be more readily understood with reference to the following figures.

1 shows a system according to the invention. Shows visual recordings and avatars showing physical exercise data in a synchronized top-down state 2 shows a flowchart of a method according to the invention. Fig. 3 shows a module for carrying out the method according to the invention.

  FIG. 1 illustrates a system according to the present invention for displaying annotated annotated physical exercise data to the person performing the exercise. As a posture recording device, a person has a motion sensor 3 located in his thigh and his ankle joint. Furthermore, an optical mark 3 'is attached to the wrist and the trunk. When there is physical exercise data, the signal from the motion sensor 3 is wirelessly transmitted to the physical data processing unit 1, where the raw sensor signal is processed into motion data and posture data. A video camera 4 records the movement of a person. Furthermore, the physical data processing unit 1 performs an optical tracking operation on the video stream of the camera 4 in order to identify the position and movement of the optical mark 3 ′. The result of the optical tracking is also processed into motion data and attitude data, and supplements the data obtained from the motion sensor 3.

  The raw signal or data processed sensor signal from the optical mark 3 ′ and the position information are stored in the data storage unit 5. Furthermore, the video stream of the person performing the exercise is also stored in the unit. Data in the data storage unit 5 is stored together with information relating to the recording time. This makes it possible to correlate or synchronize information such as knowing which position indicated by the posture recorder 3, 3 ′ corresponds to which frame of the person's video clip the person doing the exercise. To.

  Using an interconnected computer network such as the Internet 7, the physical data processing unit 1 is physically separated from the data processed sensor 3 signal and the position information from the optical mark 3 '. To the annotation unit 6. Temporary information is also sent. The annotation unit then calculates a visual display such as an avatar from the received physical data. The physiotherapist sees the movement of the visual display on his terminal 8, comments on the sequence of movement, and thus annotates it. The annotation is sent back to the physical data processing unit 1 at the location of the person performing the exercise together with the time information of the exercise when the annotation was made. Again, transmission is through an interconnected computer network such as the Internet 7.

  The physical data processing unit 1 then accesses the data storage unit 5 to retrieve the recorded data and video clips from the particular exercise annotated. A sequence of movies is generated for viewing by a person and displayed on the display 2. In this case, the video stream of the person and the avatar calculated from the recorded data are shown at the same time. At the appropriate time, the physical therapist's comments are also displayed or spoken to the person.

  FIG. 2 shows a state in which a visual record and an avatar representing physical exercise data are superimposed in synchronization. The person was exercising. Physical data representing his movement was recorded and used to calculate the avatar display. The avatar's movement was temporally resolved and divided into a stream of individual frames 20. Similarly, human movements were recorded by a video camera. This sequence of video images was also temporally decomposed and divided into streams consisting of individual frames 21. Physical exercise data and visual records can be collected simultaneously and a common time axis can be assigned to them. The time axis in Figure 2 below the frame stream optionally starts at 4 minutes 16 seconds and ends at 4 minutes 21 seconds.

  In Figure 2, a person begins exercising with his arms extended and lowered. In the image, the left arm is kept stretched and raised along the frontal plane until the hand is above the person's head. The left arm is kept in this position while the same movement is performed by the right arm. At 4 minutes and 20 seconds, a person cannot keep his right arm in a horizontal position. The right arm is bent at the elbow. This posture makes it very easy to raise the arm, so there is no therapeutic benefit in this regard. A physiotherapist remotely commenting on the avatar frame 20 can now select a 4 minute 20 second frame and add a visual or audio comment. This comment is sent to the person for future comments, with information to be shown at the exercise 4 minutes 20 seconds. Annotations are combined with visual records 21 wherever you are so that a person can associate more directly with the exercise and consider his wrong movements when performing the exercise be able to.

FIG. 3 shows a flowchart of the method according to the invention. The first step 30 is a step of visually recording an exercise performed by a person using a camera and using a sensor through a posture data. The visual record is stored (step 31) and the posture record is sent to the annotation system (step 32). Using the annotation system, the commenter checks the posture record and adds his comment and his sign (ID) (step 33). These annotations are sent back to the patient's system (step 34).
Here, "patient" means a person who is exercising. On the patient side, the stored visual record is retrieved (step 35) and combined with the annotation (step 36) to give the patient extensive feedback that still does not jeopardize his anonymity.

  FIG. 4 shows modules for performing the method according to the invention to supplement the depiction of the system of FIG. The sensor receiver 40 receives information from optical mark tracking or a signal from a motion sensor. The sensor receiver 40 communicates the above data with the motion transmission module 41. In synchronization with the sensor receiver 40, the camera 42 takes a video sequence of the person performing the exercise. These video sequences are stored in the storage device 43. The motion transmission module 41 transmits the video data to a motion receiver 45 located at a remote location. This is symbolized by a barrier 44 separating the two subgroups of modules.

  The motion receiver module 45 passes the data to a motion annotator 46 where the data is converted to data-processable data and annotated by the checking person. The annotation is delivered to the annotation sending module 47 along with the temporary location information of the exercise when the annotation was made. The annotation sending module 47 described above sends information to the annotation receiver 48 located in the subgroup, where there is a module assigned to the person performing the exercise. This annotation information reaches a data processing / superposition module 49 that accesses the video sequence from the storage module 43 and combines the sequence with the annotation so that the annotation is shown at the appropriate time of the video sequence. Finally, the superimposed video sequence is displayed to the person performing the exercise through the drawing module 50.

  In order to provide a broad disclosure without unduly lengthening the specification, Applicants incorporate each of the patents and patent applications indicated above in this application.

  In the detailed embodiments described above, the particular components and feature combinations are merely exemplary, and are interchanged with and replaced by other teachings of these teachings herein. Patents / applications are also clearly anticipated. Those skilled in the art will appreciate that variations, modifications, and other implementations of the embodiments described herein can occur to those skilled in the art without departing from the scope and spirit of the claimed invention. Accordingly, the foregoing description is by way of example only and is not intended as limiting. The scope of the invention is defined by the following claims and synonyms for the claims. Furthermore, reference signs used in the description and claims do not limit the scope of the claimed invention.

Claims (10)

A method of displaying physical exercise data with anonymous notes to the person performing the exercise,
a) collecting physical exercise data from the person performing the exercise;
b) synchronously collecting visual records of the person performing the exercise;
c) transmitting the physical exercise data to physically separated annotation units;
d) annotating the physical exercise data with the physically separated annotation unit based on the physical exercise data;
e) sending the annotation information to a display and data processing unit for review by the person performing the exercise;
f) displaying the visual record of the person performing the exercise to the person along with the synchronized note information;
Including methods.   The method of claim 1, wherein in step d) an avatar is calculated based on the physical exercise data at the physically separated annotation units. Calculating an avatar;
Displaying the avatar to the person in synchronization with the visual record and the annotation;
The method according to claim 1 or 2, wherein said step f) additionally comprises:   Transmission of the physical exercise data in step c) and transmission of the annotation information in step e) takes place via an interconnected computer network, preferably the Internet. The method as described in any one of thru | or 3.   The physical exercise data from the person includes movement data, posture data, electromyography data, pulse rate, blood pressure, blood oxygen content, blood sugar content, sweating, and / or respiratory rate. 5. A method according to any one of claims 1 to 4 selected from the group comprising.   6. A method according to any one of the preceding claims, wherein the annotation information is selected from the group consisting of visual information, audio signals, and / or speech recordings. A system for displaying physical exercise data with anonymous annotations to the person performing the exercise,
-A physical data processing unit;
A display device in communication with the physical data processing unit;
-At least one attitude recording device assigned to the person performing the exercise and communicating with the physical data processing unit;
-A visual recording device in communication with said physical data processing unit;
-A data storage device for storing and extracting data from said physical data processing unit and said visual recording device and communicating with said physical data processing unit;
A physically separated annotation unit connected to the physical data processing unit via an interconnected computer network;
Having a system.   The at least one posture recorder comprises a motion sensor selected from the group comprising an acceleration sensor, an inertial force sensor, and / or a gravity sensor attached to the person performing the exercise. The described system.   The system of claim 7, wherein the at least one attitude recording device includes an optical mark attached to a person performing the exercise.   10. Use of a system according to any one of claims 7 to 9 for displaying physical exercise data with anonymous annotations to the person performing the exercise.

Images