EP3908969A1

EP3908969A1 - Method and system for capturing the sequence of movement of a person

Info

Publication number: EP3908969A1
Application number: EP20700648.7A
Authority: EP
Inventors: Horst-Michael Groß; Andrea Scheidig; Thanh Quang Trinh; Benjamin Schütz; Alexander Vorndran; Andreas Bley; Anke Mayfarth; Robert Arenknecht; Johannes Trabert; Christian Martin; Christian STERNITZKE
Original assignee: Metralabs Neue Technologien und Systeme GmbH; Technische Universitaet Ilmenau
Current assignee: Tediro GmbH
Priority date: 2019-01-07
Filing date: 2020-01-07
Publication date: 2021-11-17
Also published as: WO2020144175A1; CN113490945A; US20220108561A1; DE112020000351A5

Abstract

The invention relates to a system and method for capturing a sequence of movement of a person. The method according to the invention comprises capturing, by means of a contactless sensor, a plurality of images of the person during a sequence of movement, wherein the plurality of images shows the movements of the body elements of the person, creating at least one skeleton model having limb positions for at least some of the plurality of images, and calculating the sequence of movement from the movements of the body elements of the person by comparing the changes in limb positions in the at least one created skeleton model.

Description

description

Title:

Method and system for recording a person's movement.

Field of the Invention

The invention comprises a method and system for detecting the movement sequence of a person.

Background of the Invention

The health system suffers from a considerable shortage of skilled workers. This means that therapy and care can increasingly only be provided to an insufficient extent, with considerable effects on health costs and on the value added of the economy. For patients, this may also mean prolonged suffering or even secondary illnesses, which can arise, for example, from poor posture during rehabilitation measures, provided that patients are not adequately instructed. These effects go hand in hand with the fact that the need to document the patient's condition is increasing so that, from a clinic perspective, it may be necessary to defend yourself against claims for damages that can be attributed to inadequate therapies. In some cases, this can lead to a self-reinforcing effect.

The system described in this document addresses this problem by the primary monitoring of rehabilitation measures, in particular those of the posture and gait during a movement, is provided by, for example, a service robot. This service robot is also able to document the completed exercises precisely, which means that the healthcare facility in question is able to meet its compliance obligations in this regard, without having to second staff separately. Another effect is that the use of such a system standardizes the assessment of the success of therapy, because at present the assessment of whether an exercise is performed correctly is subject to the assessment of a therapist, who in turn differs from other therapists through individual experiences. Thus, therapists' assessments for the same exercise are different Assessments possible, while using the system or the service robot there is a uniform assessment.

Furthermore, the use of the service robot, for example in the areas of gait and stair training, provides a significant relief for the therapeutic staff: in neurological, geriatric and internal medicine facilities, the service robot can take care of patients with poor orientation as well as training the

Sequence of movements of patients who would improve their participation in life significantly by exercising walking frequently. In orthopedic facilities, for example, gait training with and without forearm crutches (UAGS), armpits or other aids, and climbing stairs after surgical interventions on the lower ones

Extremities or on the spine represent an increased time factor for the sparse therapeutic staff. This expenditure of time can be achieved by using the mobile

Service robots can be reduced, for example, in patients suitable for walking and stair training.

The service robot helps to maintain the result of the operation and to avoid incorrect movement patterns in the movement sequence. Therapeutic training after an operation attempts to correct the wrong movement, e.g. an incorrect gait, which the patient may experience due to pain or

Has become accustomed to restricted movement before the operation (e.g. protective postures and / or evasive movement) to train again or to get used to. This is the more successful the more often the correct movement is trained (repetitive training). To have

Therapists during rehabilitation less time for therapeutic training, the service robot is a good alternative to recognize and correct errors in the movement process in good time.

State of the art

Various types of system, including service robots in the health or geriatric field, are known to the person skilled in the art. This is how CN108422427 describes one

Rehabilitation robot that is able to serve food on trays. CN206833244, in which a service robot distributes materials in the hospital, is similarly stored. Chinese patent applications CN107518989 and CN101862245, which include a service robot that transports patients, also operate in the hospital environment. similar to a wheelchair. CN205950753 describes a service robot that detects patients using sensors and guides them through a hospital. CN203338133 describes a service robot to support the nursing staff, who accompanies patients in the hospital with everyday things. In contrast, CN203527474 refers to a service robot that supports the elderly with its arm.

CN108073104 refers to a nursing robot that cares for infected patients by providing these patients with medication or administering these medications, massaging the patient, eating enough, communicating with the patient, etc. The nursing robot thereby reduces the risk of infection for medical staff by reducing the number of patient contacts between staff. A service robot to accompany older people can be found in CN107598943. This service robot has some monitoring functions, but above all a function for floor cleaning.

CN106671105 is a mobile service robot for the care of the elderly. The service robot uses sensors to monitor parameters of the body, such as temperature, but also facial expressions. He also recognizes whether the person has fallen and can alarm help accordingly via a network.

CN104889994 and CN204772554, in which a service robot from the medical field detects the heart rate, supplies patients with oxygen and has voice recognition and a multimedia module for entertainment purposes, are similarly mounted. The blood oxygen is also determined in CN105082149. CN105078445 refers to a service robot that makes it possible to record an electrocardiogram and measure the oxygen content in the blood, especially in older people. CN105078450 with an electroencephalogram measurement is similar.

Some of the health robots explicitly relate to performing exercises with patients or assessments. In CN108053889, a system is described in a relatively abstract manner, which carries out exercises with a patient based on stored information. CN108039193 describes a system for the automatic generation of health reports that is used in a service robot. The recording of movements / fitness exercises using a service robot, the recording and

Storage of the same data for evaluation purposes and transmission to external Systems are described in CN107544266. At the same time, this service robot is able to monitor the intake of medication via various sensors.

CN106709254 describes a service robot for medical diagnosis of a patient, who at the same time, based on the diagnosis, also creates a plan for treatment. For this purpose, the service robot evaluates voice and image information and compares it with information stored in memories. A neural network is used here.

[0011] CN106407715 describes a service robot which carries out the patient's medical history by means of speech processing and image recognition. In addition to querying via voice input and output devices via a touchpad, a photo of the tongue, which is taken by a camera of the service robot, is also used for the medical history.

CN105078449 describes a service robot with a tablet computer as a communication unit, via which u.a. a cognitive functional training or a cognitive psychological assessment takes place to discover Alzheimer's in patients. For this purpose, the tablet records a telephone call between the patient and a child that follows a certain process and derives from the course of the conversation whether the patient has Alzheimer's.

Jaeschke et al. 2018 validates whether a gait evaluation using Microsoft Kinect on a service robot can provide valid results in comparison to established stationary gait evaluation systems (the gold standard) when it comes to determining the position of the joints and limbs, i.e. whether the parameters relevant for the evaluation of gait training can be recorded in this way. Steps per minute,

Stride speed, stride length, time on one or both feet are mentioned as relevant parameters as well as extension and flexion of the ankles, knees and hips as well as the inclination of the pelvis and the forward or backward leaning of the trunk. When evaluating the static comparison between the two systems, it was found that the recording of the relevant parameters for clinical use using the Kinect is sufficient and does not differ significantly from those of the (stationary) gold standard. Trinh et al. 2018 set out how a seated person can be recognized by a service robot and how a service robot can interact with this person. It is also described that people with walking aids are identified via a 2D laser scanner.

Vorndran et al. 2018 illustrate how a user who completes gait training is tracked by a service robot driving in front of the user. A camera that can be controlled in its orientation is also used for this purpose, which enables better tracking of the user. People tracking takes place using LID AR and RGB camera. The service robot also uses both a laser scanner and an RGB-D camera (3D camera) to determine (future) users. This forecast of user behavior is used both to control the service robot and to track the RGB camera using a PID controller.

Summary of the invention

The invention comprises a method and a system e.g. a service robot to support the therapy of a patient, in particular the therapy of the

Movement. The service robot has sensor devices to detect the

To record movements of the body elements in the course of movement of the patient and to compare the detected movements with those movements that occur in one

Storage unit are stored in the service robot or in a cloud. Based on any deviations in the movements, the service robot can provide the patient with tips on how to improve his movements. The exercises completed and the data stored here can then be evaluated by a therapist.

[0017] In one aspect, a computer-implemented method for recording a movement sequence of a person is described in this document, the

Sequence of movements includes movements of body elements of the person. The method comprises a detection by a non-contact sensor of a plurality of images of the person during a movement sequence (e.g. a gait sequence), the plurality of images representing the movements of the body elements of the person, creation of at least one skeleton model with limb positions for at least some of the Variety of images, and a calculation of the movement from the movements of the

Body elements of the person by comparing the changes in limb positions in the at least one created skeleton model. As already explained above, the method can also include a comparison of the calculated movement sequence with a predetermined movement sequence, which is stored in a memory.

The calculation of the movement sequence includes the evaluation of movements of the body elements over at least one complete gait cycle, so that the system receives a complete picture of the movement sequence.

In a further aspect, the method also includes recognition of at least one walking aid in the plurality of images by comparison with walking aid patterns. The method can carry out a coherent evaluation of the at least one walking aid and at least one ankle point obtained from the skeleton model, the evaluation determining the difference between the at least one ankle point and a floor-side end point that comprises at least one walking aid. This difference is e.g. determined in the Sagittal plane and the evaluation is carried out e.g. at a point of contact with the ground.

The method may further include a notification if the movements in the detected movement sequence deviate from the movements in the predetermined movement sequence in order to inform the person about incorrect movements in the movement sequence. The number of messages output depends on the number and type of the detected

Deviations of the movements depend.

The system for detecting a movement sequence of a person comprises at least one sensor for contactless detection of a plurality of images of the person during a movement sequence (for example a course of gait), the plurality of images being the

Represents movements of the body elements of the person and an evaluation unit for creating skeleton models with positions of the body elements for at least some of the plurality of images and for calculating movements of the body elements of the person by comparing the created skeleton models.

The evaluation unit can have a memory with predetermined values for the positions of the body elements in the case of an intended movement sequence and, in operation, compares the predetermined values with the movements of the body elements. During operation, the evaluation unit evaluates the positions of the body elements with the aid of walking aids over at least one gait cycle and / or a symmetry of the movement of the body elements.

The system also has an output unit for outputting messages when determining deviations between the movements of body elements and the predetermined movement sequence.

In a further aspect, the system comprises a segmentation unit for recognizing objects e.g. Mates or other items in the variety of images.

Drawings:

The invention will now be described in more detail with the aid of the drawings. Show it:

Fig. 1: Exemplary system architecture

Fig. 2: Top view of the wheels of the service robot

Fig. 3: Management system of the service robot

Fig. 4: exemplary exercise plan

Fig. 5: Configuration GUI exercise plan

6: Data exchange between patient administration module, navigation module,

Transponders and service robots

Fig. 7: Course of the exercises

Fig. 8: 3D data acquisition and evaluation

9: Decision matrix based on movement sequence classifications for the output of

Instructions or feedback to patients

Fig. 10: Self-learning procedure for adapting the exercise plan

Fig. 11: Evaluation matrix of the patient's movement data for the therapist

Fig. 12: Method for improving the motion sequence correction and

Movement sequence evaluation based on labeled video sequences

Fig. 13: automated improvement of the movement sequence correction and

Movement sequence evaluation

Fig. 14: Sequence of the lean of trunk, hip flexion and knee flexion of a patient with a prosthesis (TEP) in the right hip.

Fig. 15: Use of props over time Fig. 16: Sequence of standing times of a patient with a prosthesis in the right hip

Fig. 17: Histogram of the error classes: symmetry of the gait sequences

Fig. 18: Exercise interruption

Fig. 19 Angle of the limbs and the upper body

Fig. 20 skeleton model

Fig. 21 Supplementary determination of ankle points

Fig. 22 three-point gear

Fig. 23 two-point gear

Description of the invention

A service robot 17 is shown in FIG. 3 and can be designed in different hardware-software configurations that include different components and / or modules. This service robot 17 is an example of a system for detecting the

Movement sequence of a person e.g. of a patient and is not limitative of the invention.

[0026] An exemplary system architecture is shown in FIG. 1. As described by way of example at other points in this document, alternative aspects are also possible in which individual components and / or modules are added and / or omitted. In principle, a service robot 17 has at least one processor (in PC or ARM architecture) and at least one memory connected to the processor.

In the aspect of FIG. 1, the system architecture comprises four levels, including three software levels (an application level 2010, a behavior level 2020, and a level 2030 of service robot capabilities), and a hardware level 2080. The levels 2010, 2020 and 2030 mainly depict modules, which expressis verbis is not shown in Fig. 1 for reasons of clarity and also does not have to be shown in all places in the text expressis verbis. At level 2030, the service robot capabilities are mapped, which in turn form the basis for behavior level 2020, which depicts the behavior of service robot 17, while application level 2010 covers the application.

In this application level 2010 there is, for example, a gait training application in a movement training module 2011, in which instructions etc. for a patient are stored. However, the training module can also do other training instructions include that are not necessarily movement-related, e.g. instructions for training the memory, etc. In addition, there is a exercise plan or instructions such as speech and / or display output to implement the exercise plan 2012, the evaluation of the exercises 2013 in the exercise plan, and ultimately ( optional) patient data such as age, comorbidities, room numbers of the patient, etc. 2014.

The four levels each build on one another. For example, the application motion training certain robot skills, which in turn determined certain

Require hardware components.

At the behavior level 2020 there is a module for user guidance 2021 and a movement sequence correction in the movement correction module 2022. Furthermore, there is a module which depicts how the service robot 17 approaches the patient 2023, i.e. also how the service robot 17 communicates with the patient. Another behavior that is mapped on this behavior level in 2020 is driving to the 2024 goal and that

Taking a waiting position 2025.

At the level of service robot capabilities 2030, there are several modules that contain different, important sub-modules. First there is a person recognition module 2040 for person recognition. The personal recognition module 2040 includes one

Personal identification module 2041 for personal identification, a first one

Person tracking module 2042 for visual person tracking primarily via 2D camera, a second person tracking module for LIDAR-based person tracking 2043. There is also a sub-module as a re-identification module 2044 for person re-identification, which is used when a person (patient) uses a tracking area has left one

Submodule as a seat recognition module 2045 for the seat recognition 2045, which is used to recognize people (patients) sitting on a chair and one

Submodule as skeletal recognition module 2046 for 3D skeletal recognition. This can be done using a 2D or 3D camera.

Another module at the level of service robot capabilities 2030 is one

Motion evaluation module 2050, which as a submodule

Movement sequence extraction module 2051 for feature extraction of the movement sequence and includes a submodule as a movement sequence evaluation module 2052 for recording and evaluating the movement sequence of the patient.

In the navigation module 2060 there is a submodule for the 2D / 3D acquisition 2061, a mapping module 2061a for mapping its surroundings, a map module 2061b with a map of the surroundings in which the service robot 17 moves. Furthermore, the navigation module 2060 has a submodule for self-localization 2062, for example within a mapped environment. In addition, the navigation module 2060 has a submodule in order to keep the service robot 17 always in view of the people to be tracked 2063. A path planning module 2064 for metric path planning ensures that the service robot 17 can efficiently calculate its own route to be covered. A motion planning module 2065 for motion planning, in one aspect using evolutionary algorithms, uses i.a. the results of the metric path planning from the path planning module 2064 and calculates an optimal path for the

Service robot 17 taking into account various target functions, including the

Determination of the patient's probable route. Rules are stored in the submodule for the user address 2066 as to how the service robot 17 navigates in order to e.g. to address the patient. In contrast, the submodule 2067 ensures that a distance to the user (e.g. a patient, a therapist, a caregiver, or another person) is maintained, which reflects both security requirements and the personal, culturally shaped, interpersonal distance that the service robot 17 when interacting with people. To prevent the service robot 17 from navigating, i.e. Assumes positions in space from which the service robot 17 can no longer free itself with the conventional control algorithms, the service robot 17 has a mechanism which detects this self-blocking 2068 and can also release it again. A module for determining waiting positions 2069 ensures that the service robot takes 17 waiting positions where it does not bother anyone. A 2060e submodule for the

Energy supply ensures that the service robot 17 automatically searches for a charging station when it is low on energy, docks there and charges its battery.

The level for service robot capabilities 2030 also holds a module that is dedicated to human-service robot interaction 2070. A submodule covers the graphical user interface (GUI) from 2071, another submodule provides eye contact between the patient and the patient Service robot 17 from 2072 (if the service robot 17 has a head with eyes 2094) and two further submodules use speech synthesis 2073 and speech recognition 2074.

Various components are implemented at hardware level 2080.

Below is an odometry module 2081, i.e. a measuring and control unit for the odometry function, which is connected to the navigation module 2060 via an interface. Pressure sensitive bumpers 2082 are located several centimeters above the ground and allow collision detection. If a collision in

Navigation module 2060 is detected, an immediate stop of the differential drive 2090 is triggered. This differential drive 2090 otherwise ensures the general locomotion of the service robot 17. A charging port with associated charging electronics 2091 makes it possible to recharge the integrated battery and to be supplied with the appropriate energy by an external charging device. Alternative energy sources such as a fuel cell, including a direct methanol or a solid oxide fuel cell, are also possible for the power supply of the service robot 17.

The service robot 17 has a LID AR 2083 and a panorama camera (2D, RGB) 2084. There is also an RGB-D camera (3D) 2085 on

Service robot 17, which has a zoom function and can be tracked 2086. Two

Wireless interfaces, once a WLAN module 2088 and once for an RFID transponder 2089, allow the electronic exchange of data.

The service robot 17 has a touch display 2087. At least one

Loudspeaker 2092 enables, for example, the output of speech synthesis 2073, at least one microphone 2093 the recording of speech signals, for example with the aim of

Speech recognition 2074 using natural language processing. Finally, a head with controllable eyes 2094 (with 6 degrees of freedom) ensures improved human-machine communication on an emotional level. The components 2087, 2092-2094 serve primarily the human-s ervi cerob oter interaction.

The display 2087 can i.a. can be used for the following purposes within the 2021 user guide:

• Identification of the patient / therapist using a password • Conducting a dialogue with the patient / therapist, for example querying / entering information

• The transmission of instructions to the patient, for example as feedback on the exercises completed and / or as an invitation to follow the service robot 17

• Simulation of exercises that the patient should complete. A stored avatar can be used for this, but also a video that exemplifies an exercise.

• Play recordings of completed exercises

• Representations of the evaluation of the exercises, for example for a therapist

In one aspect, the service robot 17 can also have lighting elements in order to give instructions to the patient, for example to signal that the patient should turn into a certain aisle. Such lighting elements are located, for example, in the upper area of the service robot 17 and comprise, for example, LEDs.

In one aspect, the service robot 17 has two drive wheels 6, which are centered and arranged parallel to one another (see FIG. 2). Around it, for example on a circular path, there are two or more support wheels 5. This arrangement of the support wheels 5 allows the service robot 17 to be rotated on the spot by driving the drive wheels 6 in opposite directions. For this purpose, the horizontal axis of the support wheels 5 is mounted such that the axis can rotate 360 degrees around the vertical axis. When using two support wheels 5, the distance between the drive wheels 6 is greater than that shown in FIG. 2, so that the service robot 17 is prevented from tilting too easily.

3 illustrates that the service robot 17 is connected to the cloud 18 via a wireless network connection such as the WLAN module 2088. This cloud 18 can be both a public and private cloud (“on premise”). A therapist has that

Possibility of using a terminal 13 on a computer of the

Access patient administration module 161, which in turn is connected to a memory of the patient administration module 162. The calculator of the

The patient administration module 161 and the memory of the patient administration module 162 are collectively referred to as the patient administration module 160. The therapist can store patient data in the memory of the patient administration module 162 or, in one aspect, this patient data from at least one other via an interface Import patient data management system 170, which in turn via a computer of the patient data management system 171 and a memory of the

Patient data management system 172 has. These other systems include hospital management systems, hospital information systems (HIS) and / or patient data management systems, as are usually used in clinics or

Rehabilitation facilities can be used. Furthermore, the therapist can

Patient administration module 160 assign the exercise plan to the patient

Modify the exercise plan, for example, over time and view the evaluations of the exercises in the exercise plan that the patient has carried out with the service robot 17 and which were transferred from the service robot 17 into the memory of the patient administration module 162 via an interface. The patient administration module 160 documents the treatment progress of the patient in that the patient administration module 160 receives evaluations carried out by the service robot 17, and can transfer the treatment progress to external patient data management systems 170 such as hospital management systems. Furthermore, a navigation system 180 is located in the cloud 18, which contains navigation information and displays it via a computer of the navigation system 181 and a memory of the navigation system 182. This navigation information is connected via an interface to the navigation module 2060 of the service robot 17, in particular to a room plan module 2060r. In this room plan module 2060r, the coordinates of the rooms in which the service robot 17 moves and which have also been mapped by the service robot 17 are assigned, for example, room numbers which are in the

Navigation system 180 and / or in the space planning module 2060r.

The cloud 18 is connected to a set of rules 150, which has a computer of the set of rules 151 and a memory of the set of rules 152. Central to this are primarily those algorithms that are primarily used at the application level 2010, behavior level 2020 and service robot capabilities 2030 of the service robot 17 in FIG. 1, but also those that are used in the

Patient administration module 160 can be used. Algorithms that are used to evaluate the movement sequence in the movement sequence evaluation module 2052 may be mentioned as examples. This also means that, depending on the aspect, individual modules from FIG. 1 can only be kept in the cloud 18, provided the service robot 17 has an online connection to the cloud 18, in particular also when it is for navigation purposes. Other algorithms that are in rule set 150 may make therapist suggestions for exercise plan adjustments.

Furthermore, there is a learning module 190 in the cloud 18 with at least one learning module computer 191 and at least one learning module memory 192. Historical data that the therapist has recorded, which the therapist has created, for example, are stored here has generated a exercise plan in the patient administration module 160 and / or which come from an external patient data management system 170 and, for example, was previously transferred to the patient administration module 160 and / or directly from the service robot 17. If this historical data relates to the patient, this historical data is anonymized in front. This historical data is accessible via a terminal 12 (see FIG. 3) and can be labeled, for example. As will be described in more detail below, this historical data is used to improve the algorithms in the rule set 150.

The rule set 150 is configured such that the algorithms installed locally on the service robot 17 can be updated, for example, via a wireless interface such as the WLAN module 2088, with algorithms being transferred from the memory of the rule set 152 here. In addition, algorithms in the memory of the

Patient administration module 162 can be updated via the cloud 18.

The service robot 17 shown as an example itself has a computer 9 and a memory 10, at least one sensor 3, at least one support wheel 5 and at least one drive wheel 6 and an energy source 8.

Sensors

The service robot 17 can have alternative and / or additional sensors. These include ultrasonic sensors and / or radar sensors that are pressure sensitive

Replace bumper 2082. In addition, the service robot 17 can have one or more magnetic sensors which are arranged in such a way that the magnetic sensors detect magnets on the floor with the purpose of limiting the spatial dimensions of the

To implement a radius of action, for example in front of stairs for security reasons. As an alternative and / or in addition, the service robot 17 can also have infrared sensors, ultrasound and / or radar sensors which are directed towards the floor and in one aspect in this way are configured so that the infrared sensors, ultrasound and / or radar sensors can detect steps, for example. The information from these infrared sensors, ultrasound and / or radar sensors can be found, for example, in the mapping module 2061 in the maps created.

The at least 3D camera (either designed as an RGB-D or as a pure depth camera) can be used not only for functions of the person recognition module 2040, but also for three-dimensional mapping in the mapping module 2061. The at least one 2D-RGB camera can be used With the help of appropriate

Frameworks such as Open pose (Cao et al. 2017) can also be used for 3D person tracking 2042, for which the corresponding framework is used in the case of a Kinect, in the case of an Astra Orbbec, for example, NUITrack. In one aspect, the 3D camera can be replaced.

In 3D cameras, both time-of-flight (ToF) technology can be used, such as in the Microsoft Kinect, or a speckle sensor as in the Astra Orbbec. For both technologies there are corresponding ones in the prior art

Evaluation algorithms. Depending on the aspect, one or more 3D cameras can also be used as a replacement for the LID AR 2083.

The person recognition in the person recognition module 2040 takes place on the basis of at least one optical sensor such as the LID AR 2083, a 2D camera 2084 and / or a 3D camera (either configured as an RGB-3D camera 2085 or as a pure depth camera). The LID AR 2083, especially in the case of a 2D LIDAR, is also not used alone, but in combination with at least one 2D camera 2084 and / or 3D camera. The distance between the patient and the service robot 17 is determined via the LID AR 2083 and a detection of their movements and poses via the 2D camera 2084 or the 3D camera, the latter also being able to determine the data for determining the distance between patient and service robot 17. Alternatively, a 2D-RGB camera and a separate 3D depth camera can also be used, which means an additional effort in signal processing, in particular synchronization, compared to a (3D) RGB-D camera. The term “poses” is understood to mean the orientation of a person in the room including their limbs / body parts (body elements) as well as the orientation of the service robot 17 in the room. Establish exercise plans

The starting point, for example, for therapeutic gait training is the exercise plan, the aim of which is to show the physical abilities of the patient and thus his or her over time

To gradually improve the movement. The service robot 17 is now configured in such a way that the described sensor system supports the therapeutic one

Gait training is possible. For example, after the implantation of a total hip endoprosthesis (hip TEP), the patient must now take care of relieving the operated area with the help of forearm crutches (UAGS) and try to relieve the normal or physiological movement as close as possible get. This sequence of movements is clearly defined and, for example, detailed in Chapter 2 by Götz-Neumann, "Understanding Walking", Thieme-Verlag, 2016 and in Chapter 6 by Smolenski et al, "Janda, Manual Muscle Function Diagnostics (Elsevier-Verlag, 2016). Due to pain, the patient "trained" a gait pattern (i.e. movement sequence) until the time of the operation, which the patient now does after the

Surgery for a good surgical and rehabilitation result should take place if possible. This trained gait pattern is also referred to below as "deviating gait sequence" or "deviating movement sequence", corresponding features of the

Movement sequence, such as a step length influenced by pain and therefore a different stride length than in the physiological movement sequence, is called a different stride length.

This is done, for example, through targeted gait training (initially also with UAGS / walking aids), whereby special attention must be paid to the interaction of a large number of other body parts or elements. Starting with the top, this affects the posture of the head, but also the inclination of the trunk, the position of the upper and lower arms, the pelvis and the legs / feet.

In the context of the use of UAGS, two gaits have emerged which are used in rehabilitation: the three-point gait and the two-point gait. In the three-point gait, the load on the operated / affected leg is reduced by simultaneously placing both UAGS forward with this (operated / affected) leg. The foot of the operated leg is in contact with the floor. Then the healthy leg is put forward. This type of (three-point gait) supported walking enables a higher weight relief for the operated leg, since the UAGS takes over part of the support. The two-point course represents an increase in the course of healing The two-point gait partially relieves one or both legs. The support and contralateral leg are placed forward at the same time or promptly, followed by the opposite side (right forearm support - left leg, left forearm support - right leg; this corresponds to a reciprocal sequence of movements). As soon as the patient no longer needs this relief, consult the doctor and the doctor

Therapists omitted the UAGS. The transition from three-point gait to two-point gait takes place when the patient has a fluid movement, characterized by standing time, playing leg phase, stride length and / or increasing symmetry between both legs and an increase in walking speed. The criteria for the transition to the two-point gait or omission of the UAGS are met if the service robot 17 makes fewer corrections to the patient than a threshold value and the walking speed determined by the service robot 17 is above a threshold value, which in turn can depend on various influencing parameters such as the initial walking speed, comorbidities, etc. determined during the first training session

The transition can also be made if previously determined

Exercise parameters and the exercise progress derived from these parameters show a high degree of agreement with interventions in the exercise plan that a therapist carries out and gives the patient approval for the two-point course. Exercise progress is shown as the difference in the movement sequence parameters in the movement sequence

Speed, standing time, playing leg phases, stride lengths etc. as well as symmetry values determined in the movement sequence. The service robot 17 can either make the change from three-point to two-point gear independently in the exercise plan or suggest it to the therapist, as illustrated, for example, in FIG. 10 (explained in more detail later) with a view to various other parameters.

Figure 4 illustrates such an exemplary exercise plan that the patient is completing. While the lines indicate the days, the third column shows which exercises the patient is carrying out with the service robot 17. In this example, this is not an exercise on the first day, on the second day, for example, the three-point gait is practiced for 5-10 minutes, with the patient covering approx. -20-200m. In the second column are the tasks for the

Physiotherapist deposited. On the second day, these include 20 minutes of physiotherapy and evaluation of exercise data from service robot 17. The tasks and procedures on the other days are shown in the table. With a view of the service robot 17 such exercise plans are in the

Patient administration module 160 created. For this purpose, the therapist can, for example, access a graphical user interface (GUI) (FIG. 5), via which, for example, the exercise plans are configured for each day that the patient spends in the clinic or rehabilitation facility. These exercise plans include a patient ID and, for example, the date of the operation, the side of the operation (in the case of knee and / or hip operations), a possible release of certain gaits such as the two and / or three-point gait from a certain date, a possible training release for exercises on a staircase, as well as the dosage of the exercise (frequency, duration and route length per day or exercise). Here, in

Service robots 17 clinic- / facility-specific exercise plans are stored, which correspond to the requirements of the respective surgeons. The aim is, for example, after an operation on the hip / knee / ankle to learn the correct sequence with the UAGS depending on the respective degree of stress and healing and on the other hand the respective movements in the joints. These exercise plans adapt to the defined / agreed scheme of the surgeons. However, they can be actively changed or adapted by the therapist, as explained below. If, for example, the therapist is shown too many deviations in the process with the UAGS via the evaluation module (possibly because the patient has difficulty in implementing the two-point gait, which can be learned from a specific day according to the exercise plan), the therapist can change the gait in the exercise plan "Reset" three point gait and let the patient learn the course of the two point gait at a later time. In addition, the therapist can see in one aspect, and as described in more detail elsewhere, on the basis of the evaluation of historical data, for example in the GUI, which adjustments to the exercise plan are to be made, primarily dynamically, i.e. for example on day three after the operation based on the training success of the previous days. As described elsewhere in this document, are also automatic

Corrections to the training plans possible.

Transfer of the exercise plans to the service robot

After the exercise plan has been created or adapted, this exercise plan or the instructions associated therewith are transmitted to the service robot 17. There are different approaches depending on the aspects: 6 describes the data exchange between the patient administration module 160, the navigation system 180, which contains room information (alternatively, the room plan module 2060r, which also contains this room information - optional),

Storage medium (e.g. transponder) and service robot 17. In the first step 605, patient data for a patient are created in the patient administration module 160. These patient data include the name, an ID, the diagnosis (hip operation if necessary),

Comorbidities, patient age, etc., a room number (where the patient can be found - optional) and the exercise plan. The patient data can also be obtained, for example, via an external system such as the hospital information system via an interface. On the basis of the room number, the coordinates of the room can be taken from the room number module 2060r in the navigation module 2060 of the service robot 17, alternatively from one

complementary module in the memory of the cloud-based navigation module 160 are transmitted via an interface in step 610, which in one aspect of the method can be used to pick up patients in their room. Alternatively and / or additionally, the area in which the service robot 17 meets the patient can be defined.

There are various ways in which the data relevant for performing the exercise are transferred from the patient administration module 160 to the service robot 17. In a variant, the exercise plan (and possibly patient data) are transferred 610 to a transponder or storage medium (such as a USB stick). This is given to the patient. The patient transfers the data to the service robot 17 by holding the transponder to an RFTD reading (and possibly writing) device 2089 of the service robot 17, the service robot 17 recognizes 612 this transponder and reads it 615 accordingly. Instead of a contactless RFTD In the case of a memory (for example a USB memory card), the reading and / or writing device 2089 can use a contact-based interface such as a USB port for the USB memory.

In a further variant, only a patient ID from the patient administration module 160 is transmitted 620 to the transponder. Alternatively and / or additionally, an ID of the transponder in the patient administration module 160 is associated with the patient and is thus (at least for data transfer via transponder or possibly also memory) for the patient ID. The transponder (possibly also the memory) is on the service robot 17 on the Reading and / or (writing) device 2089 is recognized 612 and the patient ID is thus read 625, which is also done by the patient handling the transponder / memory. The service robot 17 then downloads, via an interface from the patient administration module 160 located in the cloud 18, 660 the data necessary for carrying out the exercise, such as the exercise plan, the patient ID obtained being used for this, the relevant data record in the memory 162 of

Identify patient administration module 160. As an alternative to using a

A transponder and / or memory can also be generated, which contains the patient ID 630 and which is given to the patient. This is recognized 632 by the service robot 17 when the barcode is held in front of at least one of the 2D cameras 2084 or 3D cameras. The barcode or the patient ID are read 635. Based on this, the service robot 17 downloads 660 the data necessary for the execution of the exercise, such as the exercise plan, from the patient administration module 160 located in the cloud 18. The patient ID 630 obtained is used to identify the relevant data record in the database 162 of the patient administration module 160. As an alternative to the identification methods mentioned so far on

The service robot 17 can also receive a login for the patient, which is associated 640 with a patient ID in the patient administration module 160. If the login is entered on the service robot 17 in step 645, the service robot 17 loads the login with this login (and thus the patient ID) associated and relevant for the execution of the exercise from the patient administration module 160 in step 660. As an alternative to a login, biometric data of the patient can also be used, such as an iris scan, a fingerprint or a scan of the face in step 650, which in the

Patient administration module 160 are associated with the patient. The patient can then identify himself appropriately on the service robot 17, a corresponding reading device having to be installed in the case of an iris scan or a fingerprint scan. The 3D camera of the service robot 17 can also be configured accordingly for a scan of the face, for example the RGB-D camera 2085. After these steps 645 and 655, the service robot 17 loads the data associated with the patient ID for carrying out the Exercises down.

The service robot 17 then carries out the exercises in step 665, records the results of the exercises in step 670 and analyzes the results with a view to exercise plan 675. These three steps 665, 670, and 675 are discussed in more detail elsewhere. After completion of the exercises, the data recorded by the service robot 17, in particular the evaluations of the service robot 17, possibly also video recordings of the exercises, raw data of the skeletal recognition (which will be described in more detail below), are transmitted again to the patient administration module 160. In the first case described (steps 610-612-615), this can be done by transferring this data to the transponder / memory in step 680 and this

Transponder / memory is transferred from the patient to the therapist who

Then reads the transponder / memory at a terminal into the patient administration module 160 in step 685. In the other procedures described (steps 620, 630, 640 and 650), data can be transferred from the service robot 17 to the patient administration module 160 in the cloud 18 in step 690 via an interface, the data being stored there in accordance with the patient ID. Such transmission can take place at the end of the exercises or during the execution of the exercises, in real time or at intervals.

One aspect of the data transmission relies on anonymity in the data exchange, for which no data are transmitted to the service robot 17, which allow the patient to be identified (without the association of person ID with, for example, names), and also no data are stored on the service robot 17 that would allow identification of the patient. If the service robot creates 17 video recordings of the patient, these video recordings, as will be described elsewhere, are

anonymized.

Basic exercise sequence

7 shows the basic exercise sequence. In one aspect, the

Service robot 17 configured so that the service robot 17 picks up a patient at one location, accompanies the patient to an exercise area, does the exercises there and, if necessary, brings the patient back again. These steps are shown in dashed boxes in FIG. 7, since these steps are optional, because the service robot 17 can also wait for the patient at one location. In optional step 405, the patient signals the service robot 17 that he wishes to exercise. Alternatively, a scheduled exercise can be scheduled for the patient (date, time defined, for example, in the exercise schedule) and / or a complementary schedule. As the next step 410 (optional), the service robot 17 visits the patient, the information on premises (such as the patient's room number) coming from the navigation system 180 or the room plan module 2060r, in

Patient administration module 160 are stored and transmitted together with the exercise plan to the service robot 17, as was described in FIG. 6. Of the

Service robot 17 can use the room plan module 2060r and the map created in the mapping module 2061 to locate the room and navigate there, for which the service robot 17 uses its route guidance module 2024 (in example 7, the mapping procedure is discussed in more detail). If the service robot 17 arrives at the patient, this patient can identify himself at the service robot 17 in step 415, as has already been explained in detail in FIG. 6. As an alternative to this, the patient can also go to a place where the service robot 17 is waiting. Based on this identification in step 415, the

Service robot 17 its sensor system, comprising a LID AR 2083, the 2D camera 2084 and / or the 3D camera (either configured as an RGB-D camera 2085 or as a pure depth camera) in order to recognize the patient in step 420, what can be done by means of the

Person recognition module 2040 happens. First, there is one

Personal identification takes place in the personal identification module 2041. The patient is then tracked in step 425, which is done by means of the first

Person tracking module 2042 and / or the second person tracking module 2043 can happen. In the event of a disturbance in the personal tracking, which leads to the service robot 17 losing the patient “out of sight”, the patient is re-identified with the aid of the re-identification module. Example 4 shows a possible embodiment in more detail.

[0062] Via a dialog in step 430, the patient can transmit information about training to the service robot 17, for example an individual training request. As an alternative or in addition, the patient can also select exercises that the patient would like to complete, in contrast to the exercise plan, which contains predefined exercises. In one aspect, the service robot 17 checks, in the case of an exercise request chosen by the patient himself, whether it corresponds to the therapist's specifications. Because gait training is usually one

Progress path corresponds (e.g. three-point to two point), the patient cannot skip essential therapy sections freely. As an alternative to the therapist's specification, automated exercise plan adjustments and / or releases can also be taken into account, e.g. the automated determination of the transition from three-point to two-point gait based on historical data and / or on the basis of the corrections made to the movement sequence, which other aspects of

Gait trainings relate to the distance to be covered, etc. With regard to a patient, this means that patients can choose exercise plan configurations that are based on an automated release for certain exercises. Alternatively and / or additionally, the number and / or type of corrections made in the movement sequence can also be used for the automated release.

The service robot 17 can also query information in dialog 430 in step 430 and / or simulate exercises to be completed. The dialog can be done via a graphical user interface 2071 via a screen (e.g. a touch display or

touch-sensitive display 2087) and / or voice input and output using loudspeaker 2092 and voice synthesis 2073 or microphone 2093 and voice recognition 2074. Depending on the aspect, the service robot 17 can maintain an optimal distance 2067 from the user. Thereafter (optional), the service robot 17 navigates to the exercise area 435 with the aid of algorithms from the route guidance module 2024 “drive to the destination”.

The service robot 17 asks the patient to follow him via the described output units, LEDs, etc. or gives him information on where the patient should move. The patient can move in front of the service robot 17 or follow the service robot 17, both during navigation to the exercise area and during the exercises themselves. The service robot 17 also moves primarily at a constant distance from the patient, which improves sensor detection of the exercises to be completed.

The service robot 17 calculates a route that the patient and the service robot 17 should or must complete. For this purpose, the mapping module 2061 in connection with the metric path planning from the path planning module 2064 and the

Motion planning module 2065 can be used in one aspect using evolutionary algorithms. As soon as the practice real is reached, the exercises begin. The gives Service robot 17 Instructions for exercises / corrections in step 440, which are based on the movement correction module 2022. In the further course the

Motion evaluation module 2050 recorded the execution of the exercises (in

Movement sequence extraction module 2051) and evaluated in the movement sequence evaluation module 2052. If deviations are detected (step 445 movement sequence evaluation) or the correction recommendations (step 450 output movement sequence correction) are not yet implemented correctly, step 440 can be generated again. The instructions for correcting the movement sequence, which are generated, for example, as speech output using speech synthesis (and / or are also output on a display), can provide information on positioning the UAGS, straightening the upper body, etc.

include. Below this can be both hints for direct correction and praise.

Steps 440-450 are an iterative process, since several corrections and possibly several sub-elements of an exercise can be carried out over the course of the exercise. At the end of the process there is the evaluation of the exercises in step 455 and in step 460 the data transfer to the patient administration module, as has already been described elsewhere (see FIG. 6). In an optional aspect, the service robot 17 can also (again) accompany the patient to a location in step 465 (back), for example his room. Of course, the patient is generally tracked continuously during navigation. If tracking is interrupted during this time, re-identification must take place.

Data acquisition and evaluation especially during the exercises

The sequence of the acquisition of sensor data and the evaluation of the acquired

Sensor data is exemplarily illustrated in FIG. 8. In one aspect, movements of the patient are detected by the sensor, such as a Kinect 2 or an Astra Orbbec, which is a 2085 RGB-D camera. The depth image generated by the 3D depth camera in step 710 from the sensor data is then transformed in step 715 into data for representing a 3D point cloud, in which each pixel of the 3D camera is assigned to a spatial coordinate. This creates a 3D representation of the

Realized environmental information.

This environmental information is next evaluated in step 720 (feature extraction) to create a skeletal model of the patient (step 725

Skeleton modeling). Third-party software programs, e.g. NUITrack or that KinectSDK use the data from the 3D point cloud and derive signals for the recorded color, spatial depth and skeleton information in step 725. Among other things, these signals contain information about, for example, joint points of the respective skeleton model, which, for example, describe the patient's knee joint or hip joint. FIG. 20 exemplifies a skeleton model, with the body 1703, the articulation points 1701 and the connections between articulation points 1702, which, if directed, are given as

Direction vectors can be output. If frameworks such as OpenPose are used, a 2D camera such as an ordinary RGB camera can also be used instead of a 3D camera.

If there are several people in the image (i.e. not only the patient but also e.g. the therapist), these people are represented by the software using their own skeleton model. The number of recognized articulation points depends on the one hand on whether the complete person can be seen in the picture and on the other hand on the software used. Joint points can also be recognized if the person is not fully visible.

In one aspect, the sensor data of the 3D camera are evaluated in such a way that, seen from the sensor, the distance to each detected object or area of an object is determined, this evaluation being based on the resolution of the 3D camera and the distance of the objects is dependent. Via the viewing angle to the 3D camera and the distance from the 3D camera, spatial coordinates can be assigned to the sensor data from the 3D camera. These spatial coordinates are then in turn assigned to the articulation points. In this way, mathematical vector operations can be used to define the direction vectors between articulation points in direction and length (which also corresponds to the distance), as well as calculate angles between them, etc.

Next in step 730 there is a joint selection, i.e. only the articulation points necessary for the calculations to be carried out subsequently

processed further. Then, in step 735, angle calculations are carried out, for example for the angle between the lower and upper thigh, or the angle by which the thigh deviates from the vertical, for which purpose a joint point is defined as the base and the orientation of the limbs / trunk and / or e.g. the plumb as Direction vectors represent the basis for an angle calculation (see also explanations Fig. 19). Furthermore, distance determinations are carried out in step 760, for example

So-called time-distance parameters. This category includes, for example, the stride length, the service life, the track width and the flexion and extension of the hip and knee joints (depending on the therapy to be carried out) over time. They are usually determined over the course of a double step. An example of this can be found in Example 1. For example, the stride length can be a Euclidean distance between the

Ankle points 1950 can be determined within the sagittal plane, e.g. each at the time when the feet come into contact with the floor (to be recognized, for example, by the minima of the height of the ankle points 1950 above the floor), which indicates the stride length.

In addition to this skeleton-based processing of the data, a recognition of the forearm crutches is used in step 740, also referred to as UAGS recognition (which in one aspect can also support armpits and / or can include other types of canes / walking aids). The UAGS recognition thus allows the gait pattern to be assessed at a later time in conjunction with gait aids. The 3D point cloud of the built-in depth camera recorded in step 710 also serves as the starting point for this detection. In step 755, the UAGS are shown in the point cloud by a

real-time capable and fault-tolerant segmentation algorithm in one

Segmentation unit found. Knowledge of the patient's skeleton is included to preselect suitable candidate regions 745, i.e. Areas in the room where the UAGS are likely to be located. It can be assumed that these represent an extension of the arms downwards, making this area, for example, the candidate region for the detection of UAGS. These candidate regions are then checked for agreement with model assumptions about the shape of typical UAGS (elongated, narrow) in step 750 and selected if necessary. The position of the UAGS, for example the end points 1970 which touch the ground, are in turn used as time-distance parameters within the feature extraction 720, for example within the

Distance determination 760 evaluated.

This is followed by the feature classification (or feature evaluation) in step 765, which can take place according to various criteria. Here, for example, a previously determined and possibly patient-specific threshold value comes into question, the over- or is undercut, the extracted feature is evaluated primarily over time. For example, with a patient size of 1.75 m and a step length of 65 cm deemed to be normal, a step length of 30 cm can be determined by the service robot 17. The maximum allowed deviation of the recorded stride length according to the

Classification is 20cm. Since 65 cm standard stride length minus 30 cm measured by the service robot 17 stride length> 20 deviation value from the classification, the stride length of 30 cm would be assessed as too short.

It is also possible, for example, to compare and classify the symmetry of the measured values for the right and left side of the body. For this purpose, successive values, for example of the right and left stride length, are saved. The difference between the two measured values is then determined and, if necessary, standardized with respect to a reference variable, for example on the basis of the measured value of the operated side as a reference variable. The resulting differences between the two sides of the body (and thus deviations in symmetry) can be further evaluated using one or more threshold values. To stay with the example of the stride length, you can

Step length deviations (spoken in absolute values) of 20 cm serve as a threshold value or, for example, 20%, based on the physiological step length (i.e. that of the leg not operated on) - in contrast to the deviating step length, which relates to the operated leg and from the physiological step the operation deviates conditionally.

After the feature classification 765 has been carried out, the classified features are evaluated in context (referred to as movement sequence classification in step 770), i.e. not with a view to individual characteristics, e.g. Stride length or lane width, but taking into account combinations of features that are expressed, for example, in body poses, in one aspect taking walking aids into account. This can be used to make statements about the use of the prescribed movement type (two-point gait, three-point gait). This happens, like the feature extraction 720 or the feature classification 765 in the motion sequence evaluation module 2052. The goal is the output of the

Movement sequence correction (step 450), i.e. an issue of instructions to the patient to ask him to adapt his movements in his own movement sequence so that they correspond to or at least come close to the physiological movement sequence. The service robot 17 provides feedback to the patient (indication of errors,

Correction request, confirmation in behavior / praise). For this the Movement sequence classification assigned a set of rules that can be represented in one aspect as a decision matrix in which for certain poses (e.g.

comprehensive combinations of different feature classifications that

Show deviations / errors in the movement sequence) corresponding instructions for corrections of this movement sequence are stored. This motion sequence correction is shown as step 775 decision classification in FIG. 9.

For the correction of the error "healthy leg is not passed by the operated leg" in the three-point gait, e.g. the detection values for the stride length of the healthy leg are "too short" (ie below a certain threshold) and at the same time the distance between the UAGS and the healthy leg will be assessed as "too short forward" (ie above a certain threshold) ). If these two and only these two feature classifications apply in this combination, the decision matrix makes the correction: "Put the healthy leg past the operated leg, from the over the imaginary connecting line between the supports."

Triggered user feedback. Fig. 9 shows this schematically. For example, as the

Exercise instruction / correction to the above-mentioned example only if errors lund 3 (for example defined as deviations from a threshold value) were recognized. The

Correction announcement 2 in turn is triggered, for example, when errors 1, 2 and 4 in

Combination occur, etc. Since there are initially several deviations from the patient

Movement sequence can be made and not all errors can be corrected at the same time, a prioritization of the therapist determined by therapists

Movement correction can be made (what in the next section

is received). I.e. not every error detection leads to the output of a

Exercise instruction / correction by the service robot 17, since there is usually a minimum time and distance between two corrections. Detected errors, i.e.

Movement sequence classifications 770, which e.g. a correction has been assigned, but can still be saved and made available to the therapist, which is described in more detail in the following section.

The matrix in FIG. 9 does not have to be deterministic, ie it is not necessary to store a correction output for each detected deviation of the movement sequence from the physiological movement sequence, which is output in the case of this deviation. In one aspect, the output can also be dynamic. The correction editions can, for example, also be prioritized. Different priority scores are assigned to individual deviations. The service robot can then carry out a certain number of outputs per time interval, with only the highest priority outputs, ie those with the highest priority scores. For this purpose, defined delay periods can be stored in one aspect after a detected deviation, after which the movement sequence correction 450 is output, for example, within these delay periods of the correction output with the highest priority.

In one aspect, therapists can influence the set of rules for the movement sequence correction, for example the decision matrix for the movement sequence classification 770, by making settings in the patient administration module in such a way that certain poses / instructions are prioritized, others are possibly ignored, with which the

Movement sequence correction is adaptive. This information can be transmitted to the service robot 17 together with the exercise plan and is therefore in the

Movement training module 2011. Such settings can be learned together with the exercise plan settings via the suggestion function in the learning module 190 and proposed to the therapist, in a further aspect they can also be made automatically by the service robot 17, as will be explained in more detail below.

Model for the improvement of the suggestion algorithms of the patient administration module, especially with a view to the exercise plans, and implementation of suggestions

As described above, therapists can, after viewing the evaluations of the movement training that the service robot 17 has carried out with the patient, modify the exercise plan for the patient in order to improve the success of the treatment. In one aspect, the system illustrated in FIG. 3 is capable of providing the therapist

Propose exercise plan adjustments based on historical data. For example, the rules implemented here can make suggestions to switch back from two to three points if the patient makes too many mistakes in the course of two points, is too slow, the operated leg is still too heavily loaded, etc. The therapist can make such suggestions accept or reject. In an alternative and / or supplementary aspect, the service robot 17 even carries out these past-based exercise plan adjustments automatically. The basis for this ability of the system is a self-learning system, which is shown in FIG. 10 accordingly. This self-learning system increases iteratively Quality of therapy success, which mainly occurs in two ways: a) recording situations that have not been described before because they may rarely occur, and b) increasing the number of cases. Both leads to the fact that more precise weight determinations of the node weight can be carried out within the framework of machine Lem models and / or neural networks, which the effectiveness of the service robot 17

improve the therapies performed.

The patient administration module 160 has the function of submitting treatment suggestions to the therapist 1330. In a first stage that gets

Patient administration module 160 patient data about the patient, either through input by the therapist and / or via the interface to an external system such as a hospital information system (HIS) / patient data management system 170.

Here, factors are recorded that have an influence on the design of the therapy, such as parameters that affect the general mobility or agility of the patient (degree of independence, possible paralysis of the extremities, aids to be used) 1310, comorbidities (heart failure , Heart attack, dizziness, diabetes, diseases that are associated with an increased risk of falling, such as Parkinson's) 1315, but above all the reason for completing the exercises (such as a hip TEP on the right, due to osteoarthritis) 1320. Here, in One aspect also includes the type of surgery (direct anterior anterior access, lateral access to the hip, etc.) 1325, which in

to different degrees affect the muscles during the operation and therefore may also require different postoperative therapies. As a rule, the therapist 1330 defines an exercise plan 1335 for each patient.

[0082] In the memory of the rule set 152, standardized exercise plans 1335, which are transmitted to the patient administration module 160 via an interface (not shown in FIG. 10 for reasons of simplification), which are proposed to the therapist 1330 in one embodiment variant, and are selected automatically in another embodiment variant. For example, agility 1305, location of the OP 1320 (such as knees, hips) and / or type of OP 1325 are taken into account accordingly, i.e. Depending on these parameters, there are different training plans, which are in one

Embodiment variants can also be clinic-specific and in this case can be configured as such, for example, via the patient administration module 160. Transferred to the GUI 5 it may be that the therapist 1330 is already shown a pre-selection of options, such as the three-point gait 1345 from the second postoperative day, the transition from three-point gait to two-point gait 1340 from day 4 postoperatively, the release for climbing stairs three days after completed operation 1350 and a maximum distance of 300m, with two exercises per day. The configuration of exercise plan 1355 is thus a function of, for example, the starting day, the duration of the exercises, the distance, the

Frequency of the exercises, the intensity of the exercises, etc. The set of parameters either defined by the therapist, which are already predefined and / or by the

Rules 150 are automatically specified, represent the exercise plan 1335.

[0083] This exercise plan 1335 is, in one aspect, also transmitted to the service robot 17 via an interface 1360 together with data of the patient, such as, for example, the body size and possibly also comorbidities, the type of operation (OP), etc. stored 1405. The service robot 17, as shown in FIG

Gait training and, if necessary, make automatic adjustments to the exercise plan before 1335. The service robot 17 then evaluates exercises 2013 (gait training) with a view to the sizes that are shown, for example, in the GUI representation in FIG. 11, which is shown in FIG Module for evaluating the exercises in 2013 can be implemented. This data is transmitted to the cloud via an interface 1360 and flows into the learning module 190. As an alternative and / or in addition, the data recorded by the service robot 17 are not yet processed with respect to the exercise plan evaluation as shown in FIG. 11, but only the raw data (such as the measured step length) are sent to the

Transfer patient administration module 160 and / or the set of rules 150, where the evaluation then takes place and the data are prepared as shown in FIG. 11 (in the case of preparation in the set of rules 150, these are then added to the

Transfer patient administration module 160). There, for example, those of the measured different stride lengths would be set in relation to the normal (physiological) stride length. Depending on the aspect, you can do this in one or both cases

processed data are then also transmitted back to the service robot 17 via the interface 1360.

In the context of the evaluation 1406, it is shown (FIG. 11) how long a training lasted per day, how long the distance and the speed of the patient was, whereby the progress from day to day is also determined. It also shows which Concrete deviations from the physiological movement sequence occur, in particular with regard to key parameters, including the sequence of the use of the support determined by the service robot 17, the standing time, leg phase, upper body / view, stride length and / or track width. The progress of the therapy is also shown, as well as in one aspect which corrective measures have been initiated in the meantime. Furthermore, in one aspect, the therapist 1330 can receive information for the individual therapy to be carried out in order to target individual deviations from the desired one

To go into the sequence of movements or specifically control the course.

[0085] Elm therapists 1330 in order to be able to give suggestions for adapting the exercise plan or in order to be able to carry out an automated exercise plan adaptation, the data of the evaluation of the gait training are transmitted to the lem module 190 in the cloud over time 1406. As explained above, this transmitted data also includes data from

Service robot 17 outputs the movement sequence corrections 450 or deviations of the movement sequence associated therewith. This data is stored in

Lem module memory 192 of the lem module 190 is stored and, if already available, supplements historical data 1505 that originate from previous exercises, wherein this historical data 1505 could and should also originate from more than one patient and more than one therapist. The exercise plans 1335 defined before the exercise and the plan adjustments that were made, be it by a therapist or by the previously created rules, also flow into this database

Training plan adjustment 1515.

Then using machine Lem models and / or neural networks

Weight determinations for node weights in the nodes of the neural networks and / or the Lem models for an exercise plan adaptation 1510 were carried out. In this case, the evaluation of the patient performed by the service robot 17, possibly in conjunction with the set of rules 150 and / or patient administration module 160, are used as input variables

Movement training over time 1406 taken at a first point in time t as well as patient data such as age of the patient, his height, person weight, comorbidities, type of operation, etc. (1310-1325) and as a starting point the settings of the exercise plan for the exercise later time t + 1 or t. Once such node weights have been determined, they can be used to determine the patient's weight based on evaluations of the service robot 17 (possibly in conjunction with the set of rules 150 and / or patient administration module 160) Movement training forecasts are made about which settings the therapist would or should make, for example, these settings both

Exercise plan adjustments can also be to leave an already defined exercise plan on standard settings. Ultimately, you can also do that

Define default settings.

In a next step, the determined node weights are transmitted to the rule set 150, where any node weights from previous determinations are updated. Based on the node weights, rules derived from the node weights can then be updated in the next step (for example, if a certain speed is present on a certain day and taking other parameters into account, the duration of the training is extended).

These rules of the exercise plan adjustment 1530 are then transferred in step 1535 to the patient administration module 160, where the rules 1530 are used surgically, i.e. based on a standard exercise plan that may be clinic or

is therapy-specific, based on those determined by the service robot 17

Exercise results and, if applicable, the progress of the exercise, as well as parameters of the patient such as the age of the patient, his height, person weight, comorbidities, type of operation, etc., the therapist is made suggestions for therapy and / or a therapy adjustment, which the therapist confirms can reject or modify. Furthermore, in one aspect, these rules 1530 can also be transmitted directly to the service robot 17 so that this service robot 17 can carry out an autonomous adaptation of the exercise plans.

The system outlined in FIG. 10 can either recalculate the node weights after the completion of each exercise or can only do this recalculation at certain intervals, in which case the data relevant for the new weighting of the node weights are temporarily stored. Without recalculating the node weights, the rules stored in the rules 150 or the rules stored in corresponding modules of the service robot (for example 2012) are used. For example, the automated adaptation of the exercise plan 2012 can also be carried out in the patient administration module, ie the proposals for the 1535 plan adaptation are not implemented as suggestions, but instead an automated plan adaptation takes place that does not require any therapist intervention. Improvement of movement sequence evaluation and movement sequence correction

The movement sequence evaluation in the movement sequence evaluation module 2052 in a first stage and movement sequence correction in the movement correction module 2022 in a second stage (implemented in the service robot 17) essentially determine how well errors in the movement sequence are recognized and then also corrected. Together they play

Movement sequence evaluation and the movement sequence correction play a role for the

Therapy success. The algorithms implemented here can be continuously improved, as described below. There are two main ways of improving this: a) recording situations that have not yet been described because they may be rare, and b) increasing the number of cases. The recording of situations not previously described and the increase in the number of cases lead to the more precise one

Evaluations are possible, in the case of an evaluation using machine learning and / or neural networks, for example, more precise weight determinations of the node weights can also be carried out.

Fig. 12 describes the underlying system and the process. In the first step, the service robot 17 carries out the movement training with a patient. In the motion sequence extraction module 2051, a feature extraction of the motion sequence takes place and in the motion correction module 2022 the output of the motion sequence correction 450 takes place, which is carried out by the processes feature extraction 720, feature classification 765, motion sequence classification 770, motion sequence correction like that

Decision classification 775 and output of motion sequence correction 450 is characterized.

In order to carry out the method, the service robot 17 here records the data on the movement sequence of the person (i.e. the patient) and / or stores this recorded data in a step 1535a. The recorded data include the recorded data

Movement sequence parameters such as the movements of the patient, including the movement sequence classification 770, the feature classification 765, and / or the decision classification 775 and / or the output of the movement sequence correction 450, and / or video sequences 1425, which document the movements of the patient and from an RGB-2D Camera 2084 and / or RGB-D-3D camera 2085. The

Motion sequence parameters also include in one execution

Movement sequence classification 770 and feature classifications 765. This data are transmitted to the lem module 190 via the interface 1360 and stored there 1535a.

The video data with the video sequences are previously anonymized to the extent that facial features are pixelated so that the identity of the person cannot be recognized when the videos are viewed. Solutions of this type are known in the prior art, are used for the automated blurring (pixelation or blackening) of picture elements such as faces or vehicle license plates and are available, for example, as product solutions from 3DIS GmbH or Guardian Project.

Therapists can access this via a terminal, view and record the recorded video data and various aspects of the movement sequence. The therapist respects e.g. on personal parameters such as the stride length, the track width and / or the posture of the upper body including shoulder area of the patient. The therapist enters these personal parameters (stride length, track width, posture, etc.) into the motion sequence assessment module 2052 together with the associated ones

Video sequences and provides this information with a so-called label 1535b, i.e. the therapist marks the recorded movement sequences and differentiates the movement sequence into deviating and physiological for the movement sequence evaluation in the movement sequence evaluation module 2052. The therapist differentiates between normal = physiological stride lengths, track widths and postures vs.

illness-related = deviating. Here, the labeling by the therapist represents a movement sequence and / or feature classification. The movement sequence and / or feature classification is continuously re-evaluated in steps 1540 and 1542 after an initial recording and labeling, and there may also be a re-evaluation of the

Decision classification 1545, which is practically an adjustment of rules in the

Decision matrix of FIG. 9 equals.

This adaptation can also be a manual control adaptation of the

Movement sequence classification 1570, the characteristic classification 1572 and / or the

1575 decision classification. This can mean, for example, that with a certain movement sequence classification (e.g. a combination of movements of the UAGS and legs, trunk, etc.) the movement sequence correction (output of instructions based on speech synthesis and / or display on the display) is redefined or defined and / or a threshold value must be adjusted in the feature classification (e.g. stride length). These rule adaptations are ultimately transferred to the rule set 150 and there the rules for the movement sequence evaluation 1590,

Characteristic classification 1592 and / or decision classification 1595 updated. These updates to the rule set 150 are (via an interface, not shown) to the motion sequence extraction module 2051 for the feature extraction of the

Movement sequence or the movement correction module 2022 transmitted in the service robot 17, whereby the process chain feature classification 765,

Movement sequence classification 770 and decision classification 775 and possibly output of the movement sequence correction 450 is updated. A reassessment of the

In one aspect, the motion sequence can also result in the feature extraction of the motion sequence in the motion sequence extraction module 2051 also having to be adapted.

[0096] Alternatively and / or additionally, the reevaluation of the movement sequence 1540 and / or the reevaluation of the decision classification 1545 can take place automatically. FIG. 13 shows this method in which a model is trained and then used to evaluate the movement sequence and the movement sequence correction. The service robot 17 carries out the movement training with a patient. It takes place in

Movement sequence extraction module 2051 a feature extraction of the movement sequence movement sequence and via the movement correction module 2022 this takes place

Exercise instruction / correction. The service robot 17 captures data and / or stores this captured data in step 1410. This includes the movement sequence parameters, i.e. the patient's movements, in one aspect the ones that have occurred

Movement sequence classifications 770, feature classifications 765 and / or the decision classifications 775 made by him. Furthermore, those also count

Evaluations of the movement training (as shown, for example, in FIG. 11) 1406. These evaluations of the movement training are transmitted via the interface 1360 to the learning module 190 and stored there 1535a. In one aspect, the data is saved in a database and thereby an addition to existing, historical ones

Service robots 17 recorded data of the data with a view to the movement sequence classification 1550 and feature extraction 1552 and / or historical data of the

Decision classification 1555, which are supplemented by the newly stored data. Taking into account the evaluation of the movement training over time 1406, as shown, for example, in FIG. 11, a weight determination of the node weights of the

Movement sequence classification 1560, the characteristic classification 1562 and / or the

Decision classification 1565 can be made in a Lem model or a neural network. For this purpose, comparable to the weight determination of the node weights when adapting the exercise plan, machine learning algorithms such as

Clustering processes, support vector machines as well as regression processes and / or neural networks such as convolutional neural networks are used. Here, the evaluations of the movement training, such as the distance covered, standing time, leg phase etc., serve as direct values and / or calculated as an improvement over the previous period as output variables.

The ascertained features of the movement sequence, including in one aspect the movement sequence classifications 770 and / or feature classifications 765 and / or decision classification 775 and / or output of the

Movement sequence corrections 450 and the patient's personal parameters such as age of the patient, his height, person weight, comorbidities, type of operation, etc. (not shown in FIG. 13) as input variables. As a result of these methods, new node weights are generated and transmitted to rule set 150, one

Weight update of the node weights on the part of the movement sequence classification 1580, feature classification 1582 and / or decision classification 1585 result. These weight updates to the node weights in turn lead to

Rule updates of the movement sequence classification 1590, feature classification 1592 and / or the decision classification 1595. These updates of the rule system 150 are (via an interface, not shown directly) to the

Movement sequence extraction module 2051 for the feature extraction of the movement sequence or the movement correction module 2022 transmitted in the service robot 17.

The two described methods of determining the weight of the node weights by means of machine learning and / or neural networks enable, for example, only those movement sequence corrections to be carried out which show actual success in terms of the progress of the therapy. This can also lead to fewer corrections being made by the service robot 17, which leads to savings in the

Energy consumption and processing time of the service robot 17 can result. With view on Features of the movement sequence can be shortened, for example, with fewer features to be recognized, on the one hand, and those with the calculations on the other

associated energy consumption. In contrast to video-based labeling, therapists can use smaller bandwidths in data transmission between the service robot and the cloud.

In one aspect, a reassessment of the movement process can also result in the feature extraction 720 in the movement process extraction module 2051 having to be adapted for the feature extraction of the movement process.

[0101] In one aspect, the methods for manually improving the

Movement sequence evaluation and the manual improvement of the movement sequence correction as well as which are combined based on the approaches of machine learning and / or neural networks.

Examples of use

Example 1: Detection and evaluation of the three-point gear

For the evaluation of the three-point walk by the service robot 17, in particular the movement sequence, some definitions are initially relevant. A person's stride length is defined as the Euclidean distance between the ankles, i.e. the dated

Skeleton model recognized ankle points in 1950, during a gait cycle. A

Gait cycle consists of a swing phase and a stance phase for each leg. The swing phase begins when the foot is lifted off the ground and continues as long as the foot is in the air and the leg is brought forward. As soon as the foot, ideally the heel, touches the ground (initial contact), the stance phase begins. The standing phase of a leg is defined as the period in which the foot is on the floor, as is also evident from the skeleton model, which also identifies the level that corresponds to the floor. During a gait cycle, there are one for the right and left leg

Stride length that always refers to the leg that has initial ground contact after completing its swing phase. In contrast to the stride length, the track width is defined as the distance between the two heel centers and is in the range of 5-13 cm, also ascertainable via the distance between the identified ankles 1950 within the frontal plane. As stated above, in step 725 the positions of joints of the

Patients in the skeleton model in the room are output as hinge points. Although the Kinect 2 used in this example shows no poses of body parts in the skeleton model, these poses can be modeled via the connection of adjacent articulation points that the Kinect recognizes, which is implemented as part of the feature extraction 720 in step 735.

Figure 19 illustrates this modeling. Some recognized articulation points are also shown here as filled circles. Using a mathematical operation, direction vectors between recognized articulation points are calculated, e.g. by creating a vector between the 3D coordinates of adjacent hinge points. In Fig. 19 they are drawn as dashed arrows between the joint points lying next to each other. For example, as shown in FIG. 19 a), the angle a of the knee joint point 1930 can be defined via the two direction vectors 1910 and 1920, which correspond to the course of the thigh and lower leg. The first direction vector 1910 from the knee to the hip skeleton point and the second direction vector 1920 from the knee to the ankle skeleton point (or ankle-skeletal point) are calculated, namely by determining a connecting line between a knee joint point 1930 and the

Hip joint point 1940 or between the hip joint point 1940 and the

Ankle joint point 1960. Analogous calculations are also possible for the arms, for example. In Fig. 19 a) the specified angle a shows the knee flexion, for example. The determination can, for example, take place during the standing leg phase, i.e. in the phase from the first contact of the heel to the transition in the direction of weight gain through the other leg, which also initiates the so-called leg phase in the leg under consideration.

In order to obtain the values for the flexion and extension of both hip joints, the corresponding shoulder and knee points are used. The angle is determined using two direction vectors, one from the hip joint point 1940 to the

Knee joint point 1930 extends as well as the other from the hip joint point 1940 to

Shoulder pivot 1960 _(j in each case on either side, that is, for example. Right

Hip joint point 1940r to right shoulder joint point 1960r). One speaks of flexion, especially hip flexion, when a leg is oriented forward from the vertical, ie the flexion angle is generally defined via the direction vector 1910 (shown in reverse orientation) when the leg extends beyond the vertical in the walking direction located in front of it and the direction vector between the hip joint point 1940 and the shoulder joint point 1960 (see FIG. 19 b) with the angle β i). Extension, on the other hand, is defined as the backward orientation of the leg, ie the extension angle is defined via the direction vector towards the shoulder joint point 1960 and the direction vector 1910 (shown in reverse orientation) between the hip joint point 1940 and

Knee joint point 1930, when the leg is beyond the vertical in the walking direction behind it (see Fig. 19 c) with the angle ßi). In one aspect, the angles can be determined on both sides.

The flexion and extension angle of the hip in turn is influenced, for example, by the forward inclination of the upper body, which has an effect on the course of movement. For this reason, the angle of the front tilt, for example, is also recorded, which, for example, is determined by the direction vector from the middle hip joint point 1940 to the middle point

Shoulder joint point 1960 and the plumb line that runs through the middle hip joint point 1940 results (see FIG. 19 d) with the angle of inclination g.

These calculations result in the sagittal plane, but are also carried out in a similar form in the frontal plane, which is shown in FIG. 19 e) by way of example on the lateral upper body inclination. Here, a direction vector parallel to the plumb line and a direction vector parallel to the spine are used, between which a lateral one

Inclination angle e is spanned.

As an alternative, calculations are also possible which relate to complement, supplement and / or secondary angle and / or which include the solder, for example, for determining the extension and / or flexion of the knee, hip or other limbs. This is shown by way of example in FIGS. 19 f) and g), in which the hip joint extension can be seen with the hip extension angle di in g) and the flexion with the hip flexion angle yi in f). In addition, the angle can be used, for example, together with the angle of the upper body inclination g in order to arrive at the angle β, etc.

19 h) shows an example of a skeleton model in which the end points 1970 of UAGS 1970 are shown, which are used, for example, elsewhere for determining the distance to the ankle points 1950. The UAGS are indicated by dashed lines 1980, because they are not part of a framework like OpenPose etc.

Skeleton model.

14 illustrates the further evaluation over time on the right side of the patient. The plotted curves illustrate the trunk inclination (top, lean of trunk), hip (center) and knee flexion (bottom) over a period of 23s for a patient with TEP (total endoprosthesis) in the right hip. For the left knee, the flexion is significantly more pronounced (i.e. there are stronger amplitude fluctuations), which is due to the asymmetrical postoperative movement, which among other things. expressed in a larger stride on the non-operated side.

The gait training, which the service robot 17 accompanies, takes place shortly after the operation and, as described, the patients must first complete the three-point gait, then the two-point gait. The three-point gait and the two-point gait include the use of forearm crutches (UAGS) to reduce the stress on the operated joint. Before a patient can start with robot-assisted gait training, the patient is instructed by a therapist in how to use the service robot 17. In addition, the therapist must explain to the patient how to use the UAGS when sitting down and getting up, when turning around and opening the door, and the procedure of the

Setting the column during the three-point gait. The therapist then releases the training with the service robot 17. As soon as the therapist receives feedback from the service robot 17 via the patient administration module that the switch to the two-point gear could take place, the therapist also shows the patient the correct sequence of setting the support here for the first time and, if necessary, checks the suggestion of the service robot 17 before he or she does so Gait "releases".

For this reason, the UAGS must also be recognized. The service robot 17 uses depth data from the Kinect2 3D sensor for this. In the first step 710, the depth image is converted into a point cloud, for which the point cloud library described in the prior art is used. This point cloud will become smaller in the next step 745

Point clouds segmented based on the patient's skeleton model. Here, the assumption is made use of that the UAGS must be close to the forearms and hands and approximately parallel to the legs, ie a selection is made Candidate regions. Standard segmentation and fitting algorithms can thus be used efficiently to evaluate one of the smaller point clouds near the forearms / hands / legs in step 755. This is helped by the fact that certain model assumptions can also be made in step 750 which take into account the shape of the UAGS, i.e. it is taken into account in the segmentation algorithm or in the generation of the 3D data that the UAGS is quite thin in relation to the limbs of the Are patients. The RANSAC framework can be used, for example, to

to segment corresponding point clouds with high accuracy and robustness in real-time (30 fps). As an alternative and / or in addition, classification rules can also be used which are created by recording UAGS, for example from different

Perspectives that have been created and that make it possible to recognize UAGS on a picture.

On the basis of the identified UAGS, it is possible to further evaluate these identified UAGS, in particular in relation to the skeleton model, for which the

Movement sequence classification is used in step 770. The position of the two feet is essentially evaluated in relation to the determined position of the UAGS. If the three-point movement is correct, the patient simultaneously moves the leg of the operated side forward with both UAGS in order to achieve optimal relief of the operated joint. The UAGS relieve the operated leg throughout

Stance phase and are only relocated at the transition to the swing phase. Since the movement of the UAGS largely occurs simultaneously to the operated leg, a straight line is defined between the two UAGS end points in 1970 and the distance between the base points and this straight line is evaluated. In an alternative or additional aspect, the direction of movement of the patient is determined, ie the orientation of his sagittal plane, which can be done, for example, by the position and / or direction of movement of the arms, legs and / or the orientation of the pelvis and / or shoulders. Alternatively and / or in addition, the patient is tracked over a defined period of time and the orientation results from the movement of the patient over time. In a next step, a line can be determined, orthogonal to the direction of movement or orientation of the patient, which runs through the UAGS and to which the distance of the base points is evaluated. This makes it possible to identify typical errors in the three-point gait, such as putting the UAGS on too early or too late and relieving the pressure on the wrong leg, by statistically evaluating the distance curve. A Deviation from the arranged gait, for example towards an unordered / incorrect sequence, can be recognized by evaluating the UAGS position relative to the patient's body. Based on the definition of

Three-point gait, both UAGS are at approximately the same level in the patient's sagittal plane when the movement is correct. Deviations in this position that exceed a corresponding threshold value can then be recognized as errors in the process.

Fig. 15 shows the use of props over time. The upper time-course diagram shows the height of the props above the ground (left prop: higher amplitudes on average, i.e. the solid line; right prop: smaller on average)

Amplitudes, i.e. the dashed line). This information, i.e. especially the

Contact with the ground is used to assess the distance between the supports and the ankles, as illustrated in the lower time course diagram. Here, over time, the operated leg is placed forward at the level of the so-called (invisible / imaginary)

Connection line between the UAGS shown. An ideal curve would have its minimum close to the minimum of the upper curve (i.e. on the vertical dashed lines). From the example shown in FIG. 15, it is therefore possible to conclude an undesired sequence of setting the supports. This patient first places the supports, then the operated leg and then the healthy leg. The output of the movement sequence correction 450 now informs the patient that the patient should bring the UAGS forward with the operated leg as simultaneously as possible. This means that the patient's movement is more fluid and less checked. Further parameters can be derived from this time curve. These are "support too short / too wide", which leads to unequal stride lengths, and "support too late" (foot of the operated leg touches the ground rather than the UAGS), which would lead to an undesirably high load in the operated leg .

[0115] In addition to the feature extraction in step 760, the service robot 17 must distinguish different motion sequence features from physiological motion sequence features. This classification must be carried out in real time while the patient is moving behind a service robot 17. The classification whether stride length, standing time, stride width, trunk inclinations or joint rashes / movements are in the physiologically typical areas is carried out by physiotherapists. This involves movements of healthy people and physically impaired people recorded by means of the Kinect device and a 2D camera 2084 and the movements of the people, broken down to each individual movement, are labeled, which in each case takes advantage of time stamps in the labeling and in the recordings of the Kinect device. In this case, features are labeled that include movement errors as well as a correct movement sequence. The time stamps are used for the synchronization of labeled data and the 3D recordings of the 3D camera.

[0116] Since the physiological deviations of the movement features can have many different facets, an evaluation based on absolute value deviations from threshold values is not particularly efficient. It is more efficient to evaluate the similarities of the movements of the two legs over a gait cycle (i.e. double step). The similarity is determined by the ratio of the stride lengths, the standing time, step angle between leg with and without prosthesis, etc. 16 shows the sequence of standing times of a patient with a prosthesis in the right hip a few days after his operation over the course of four gait cycles. The width of the vertical lines indicates the duration of one leg, whereby the narrower intervals (duration on the right) refer to the operated side. The patient of FIG. 16 consequently stressed his operated right leg for less than the healthy leg in order to minimize the pain during exercise. The diagram also shows the different stride lengths. The curve shown is determined from the distance between the ankles (sagittal plane). The minima result from the moment when the ankles are at the same height. The maxima represent the maximum distance between the ankle of the forward leg and the ankle of the supporting leg. The diagram shows that the left, non-operated leg takes a significantly shorter step compared to the right leg. It would be optimal for the gradual increase in stress in the operated leg and for the maintenance of a fluid movement sequence to take the same big steps. The relationship between the stride length and the standing time of both legs can be regarded as a suitable instrument for characterizing a patient's gait as physiological or deviating, a classification as “deviating” resulting in the output of a movement sequence correction 450.

In order to differentiate between physiological and deviating movement sequences, the Fl score is evaluated, which divides both classes (errors / deviations vs. correct ones Movement execution) differentiated, which can be calculated for different threshold values. The Fl score is generally defined as

F = 2 * (precision * recall) / (precision + recall).

The best threshold corresponds to the highest Fl score. Fig. 17 shows the

Distribution of the deviating and physiological step length symmetries as a histogram, the first representing the error class, the latter the non-error class. A symmetry of 0.0 means perfect symmetry between the left and right legs, with symmetries of ± 1.0 meaning that one leg is more than twice the stride length than the other. By evaluating the Fl Score, a threshold value is found that best separates the two classes. In this case the best symmetry threshold is - 0.17 (precision: 0.85, recall: 0.87), which means that stride lengths are less than -0.17

Deviation from the normal movement sequence can be classified and thus initiate a correction by the service robot 17.

[0118] In summary, the evaluation of the three-point gear is shown in FIG. 22. In step 1805, information about the leg to be spared (operated side) is first used (which is obtained, for example, as in the explanation for FIG. 5). In the context of feature extraction 720, the articulation points, the direction vectors between

Articulation points and / or the UAGS recorded as space-time parameters in step 1810. The position of the ankle points 1950 on contact with the ground is recorded

Time course in step 1815. Based on this, the track width is determined, for example, by calculating the distance between the ankle points 1950 in the frontal plane 1825. For example, within the scope of the feature classification 765, the track width is assessed in step 1865, i.e. a determination of the distance between the ankle points 1950.

Furthermore, as a result of the detection of the position of the ankle points 1950 in contact with the ground over time in step 1815, for example, a determination of the stride lengths (i.e. the measured distance of the ankle points 1950 in the sagittal plane at

successive contact with the ground) in step 1830, with, for example, an assessment of the stride length ratio in the gait cycle 1870 in the context of the Feature classification 765. Furthermore, for example as a result of step in step 1815, the service life in step is recorded by a time measurement and evaluated in step 1875. In addition, the inclination of the upper body can be determined in step 1840 with a subsequent evaluation in step 1890 within the feature classification 765, for example also a detection of the flexion and / or extension 1845, for example of hip and / or knee joints, with a subsequent evaluation in 1895 as part of the 765 feature classification.

[0120] The position of the UAGS endpoints 1970 is also recorded in the

Course of time in step 1820. Based on this, a measurement of the distance between the UAGS end points 1970 for ground contact in the frontal plane can be carried out in step 1851, with an assessment of the distance between the UAGS end points 1970 for ground contact (which corresponds to the UAGS distance) in Step 1884 within the feature classification 765. Here it is determined, for example, whether the UAGS are too far apart. As an alternative and / or in addition, the distance between the UAGS end points 1970 in contact with the floor and the ankle points 1950 in the sagittal and / or frontal plane 1863 can be determined, with subsequent evaluation of the distance between the UAGS end points 1970 and ankle points 1950 in contact with the floor in step 1885. Here it is determined whether the UAGS are positioned too closely or too far away from the body with reference to the ankle points 1950, in the sagittal plane and / or the frontal plane. Threshold values can be taken into account in each case. Exceeding the threshold in the

For example, the frontal level would indicate that the UAGS was set too broad, a drop below a threshold value would set the UAGS too narrow, while a threshold value in the sagittal plane would indicate the UAGS set too far forward.

In addition, based on steps 1815 and 1820, an assessment of the

UAGS and ankle points in 1950 are placed on the floor in step 1880 (for example, to evaluate the correct timing of the leg being operated on, which should if possible only be set up after the UAGS has come into contact with the floor).

To this extent, these aspects assess the movement sequence of a person, for example when walking at UAGS. For the detection of the three-point course, the position of the UAGS end points 1970 on contact with the ground is recorded in step 1850 and a determination is made as to whether the position of the contact points of the UAGS on the ground is approximately parallel to the The frontal plane of the person is 1852. In step 1854, it is also determined whether the leg to be relieved or the foot of this leg is located between the UAGS. This can be implemented in at least two alternative and / or additional ways. On the one hand, a connecting line between the positions of the UAGS on the floor can be determined in step 1855 and then the distance from the ankle point 1950 of the leg to be protected to the connecting line in step 1857

Feature classification 765, this distance can be evaluated in step 1887. On the other hand, for the UAGS endpoints 1970, a perpendicular to the sagittal plane as well as a perpendicular to the ankle point 1950 of the leg to be protected in the

Sagittal plane are formed (step 1860) and then in step 1862 the distance of these perpendicular to each other are determined in the sagittal plane, followed by an evaluation of these distances in step 1889.

[0122] Finally, a movement sequence classification 770 takes place, in which the individual feature classification is evaluated in a coherent manner. If there are any deviations from the rules stored in the movement sequence classification, which are based, for example, on defined errors (for example, the setting of the UAGS is assigned to the error "supports too far forward" while the upper body is inclined too much), and one occurs Output of a motion sequence correction 450 (for example, the instruction not to set the UAGS so far forward) on the basis of a decision matrix (FIG. 9).

Example 2: Classification based on machine lemens / neural networks

The feature classification 765 and the movement sequence classification 770 are determined deterministically in the sequence described, i.e. based on

Expert assessments defined. In alternative aspects, methods of machine learning and / or neural networks can also be used for this. In the case of support vector machines, the direction vectors mentioned would be derived from the

Feature extraction form classes that define themselves from vector spaces (i.e. certain vectors with a certain similarity would fall into the classes, dissimilar vectors accordingly do not). Thus, for example, a feature in a certain form represents a space that is spanned by the vectors. It is the same with the

Classification of movements. Other methods like k-NN, K-Means, general Clustering or neural networks work similarly. In one aspect, the sequence of movements is the course of the gait.

Example 3: Anomaly detection in movement behavior and implications

In one aspect of the service robot, 17 rules for anomaly detection are stored in the movement sequence evaluation module 2052 for evaluating the movement sequence and / or in the movement correction module 2022 for movement movement correction. An anomaly is understood to mean a deviation in the movement behavior that deviates from the "normal" (ie the physiological gait behavior. This can mean, for example, that the healthy leg has a stride length of 65 cm, the operated leg a stride length of only 30 cm. A deviation from this "usual deviating movement sequence" can again be present, for example, if the service robot 17 measures only 10 cm instead of the 30 cm step length. As an alternative and / or in addition, the anomalies can also be detected over time from curve profiles of individual joint points of the skeleton model , in which

Amplitude height, the position of minima, maxima and / or of turning points over time can characterize an anomaly. This type of detection runs in the

Movement sequence evaluation module 2052 for evaluating the movement sequence, specifically the feature classification 765 and movement sequence classification 770 are used for this.

[0125] If one or more of these anomalies are detected, rules stored in the modules mentioned trigger events. An event can be a notification of a therapist, for example by first transmitting information from the service robot 17 via the wireless interface to the patient administration module 160, which in turn can inform a patient, e.g. via a notification via a wireless network such as GSM, LTE, WLAN, which can be received via a therapist’s mobile device. This information can include, for example, a video sequence that the

Anomaly in recorded form. In one aspect, this video sequence can also be stored in the patient administration module 160 in such a way that on the basis of the

Notification of the therapist via his mobile device to the im

Patient administration module 160 can access stored video sequence.

[0126] In an alternative and / or supplementary aspect, the anomaly is provided with a time stamp and in the memory of the lem module 190 corresponding to Information that the video sequence transmitted to the learning module 190 contains the anomaly. In a GUI, via which the therapist can label the video sequences transmitted to the lem module 190, sequences with anomalies are given higher priority, this prioritization, for example, within a

Database is done. In this way, efficient labeling can be implemented in order to

Further improve algorithms of motion sequence evaluation and motion sequence correction. In one aspect, the movements recorded, classified, evaluated and / or corrected are the gait sequence.

Example 4: Person identification, visual person tracking and person re-identification

The person identification in person identification module 2041 can take place by means of skeleton model recognition, as is made possible by evaluation frameworks of 2D and 3D cameras and / or frameworks such as OpenPose, OpenCV etc. A

Synchronization of the recordings of the sensors that implement the skeleton model recognition and an RGB recording make it possible to assign body regions of the tracked person, colors and / or color patterns and / or textures of clothes that originate, for example, from the clothes of the person. Based on the parameters color or color pattern per

Body region and / or size parameters of the patient (body height, arm length, leg length) can be tracked and recognized again over time. Alternatively and / or in addition, movement sequence patterns, such as Gait pattern can be used. Tools such as OpenPTrack can be used here.

In addition and / or alternatively, face recognition can also be used.

In another aspect, markers can be used for person identification, tracking and also re-identification. They can be positioned on the patient or on the walking aids to identify the patient. Such markers can be color-based patterns or light sources with a certain frequency. Thus, in one aspect, the patient can wear a vest, on the surface of which barcodes are visible, visible to the service robot 17. The RGB-2D camera 2084 can identify the patient using these barcodes. Barcode information can be provided in a further optional aspect in accordance with the Patients are assigned, ie stored in the patient administration system 160 or in the storage medium or transponder.

Example 5: Adaptation of the exercise plan based on a multinomial logistic regression A multinomial logistic regression is shown as an example for the machine-lemen approach. Such an estimation model makes it possible to estimate the probability for different, not necessarily directly connected output variables based on a series of input variables. For example, the selection options of the exercise plan, which result from the GUI in FIG

Three-point gear, a period of e.g. 10 minutes and a distance of 100m, for example, can be estimated at the same time. The input variables are, for example, those previously determined by the service robot 17. Selection options of the exercise plan in question, but also patient comorbidities such as a general restriction of movement, if necessary

Impairment of mental ability, his height, the type of surgery performed (which can affect different muscle groups and therefore

varying degrees of impairment in the postoperative phase), etc. Implementation of the procedure is possible, for example, using the "LogicRegression ()" command in the "SCIKIT-LEARN" toolkit for Python, for example with the additional command mutliclass = "multinomial" between the clinging.

Within the framework of multinomial logistic regression, a linear

Predictor function is used which determines a score using a set of linearly combined node weights, which are multiplied by the input variables as a scalar product: score (Xi, k) = ß _k X _t , where X _{t is} a vector of the input variables for the

Observation i, ß _k is a weight vector or regression coefficient with a view to the respective output variable k. The score can turn directly into one

Probability value are converted that the observation i of the input variables leads to the respective output variable k. The input variables are also referred to as independent variables, the output variables as dependent variables.

The nature of the operation, the physical condition, the length of time since the operation, the distance traveled, the type of gait, etc., which can have different characteristics, can serve as independent variables. Settings or combinations of settings that the Therapist can undertake in the exercise plan, these also being different

Can have expressions. Specifically, for example, a patient in normal physical condition, with a left hip TEP performed as a surgical procedure with lateral access to the hip, three days after the operation the day before, as of

Service robot 17 determined, covered a distance of 250m within 5 min in three-point gait (this would be the characteristics of the input variables) and a therapist would adjust the exercise plan at this point in such a way that the patient can walk 10 min during the next exercise, the three-point gear would be maintained (these would be characteristics of the output variables). A multinomial logistic model can be estimated on the basis of the characteristics of the individual variables. The regression coefficients determined on the basis of this model formation phase (weights for the individual input variables such as physical condition, access to the hip, time period after the

Operation, distance, duration of the exercise, three-point gait) can be used to suggest to the therapist which exercise plan configuration he should undertake when recording additional exercises (primarily by other patients). If, for example, the setting described above is made several times by therapists, if the specified values of the input variables provide templates, this indicates that it is sensible to make these settings, which, in other words, is expressed in significant values for the node weights. It is then possible to propose to a therapist exactly those measures that have already been described above, namely in normal physical condition, with a left-sided hip TEP carried out as a surgical procedure with a lateral access to the hip, three days after the previous day's surgery, and one covered Distance of 250m within 5 minutes in three-point gait then as a setting to be made by the therapist 10 minutes in three-point gait. This suggestion is displayed in the GUI in FIG. 5, for example by means of a colored highlighting (not shown). Since it is a statistical model, i.e. the score only gives a probability that a therapist will have certain prescriptions

Characterization of input variables makes modifications in the exercise plan

Threshold values are stored in the set of rules 150, which, for example, from a probability of 80% for this combination of measures consisting of three-point gait and 10 min gait duration, also suggests these to the therapist. [0133] Alternative or additional methods for determining the weight of the node weights are naive bayes, decision trees or else neural networks such as long short-term memory recurrent neural networks.

Example 6: Signaling exercise request

This signaling can be transmitted to the patient administration module 160 and / or the service robot 17 via a mobile terminal device that the patient uses, which in turn wirelessly instructs the service robot 17 in this regard. can send. Alternatively, in one aspect a house call system can also be used, as is standard in clinics, or a permanently installed terminal that is in communication with the patient administration module 160 and / or the service robot 17. Such a request for exercise can initially also be entered in a database via

Mechanisms to plan the order of exercises. In this context, rules can be stored that, for example, give priority to certain patients.

Example 7: Mapping

If the service robot 17 is active for the first time in an area such as a clinic building, the service robot 17 uses the submodule 2D / 3D recognition and the

Mapping module 2061 by mapping its environment. All of them

Spaces are traversed and the surroundings are recorded using 2D and / or 3D sensors. Sensors that are suitable for this are at least the LID AR 2083, at least the RGB-D 3-D camera 2085, at least the RGB-2-D camera 2084 and / or ultrasound and / or radar sensors. Combinations of these sensors can also be used. The (2D) RGB camera 2084 is sometimes only used as an aid to evaluate colors, for example. Rules can be stored in the mapping module 2061, in which areas the service robot 17 may move and, if necessary, for what purpose. These include, for example, areas in which the service robot 17 is not allowed to move, areas in which it is allowed to pass, and still others in which it is allowed to complete a training, such as movement training.

[0136] In one aspect, the module for 2D / 3D recognition and the mapping module 2061 has an interface to the movement training module 2011 for the exchange of Localization data. For example, a room number can be stored in the movement training module 2011, while the information assigned to this number can be found in the

Mapping module 2061. The service robot 17 can thus recognize the location at which the service robot 17 can meet the patient.

Example 8: Map enrichment using CAD data

The cards that the service robot 17 picks up from the building in which the

Service robot 17 is located, can be enriched in one aspect by CAD data of the building, which flow into the 2D / 3D acquisition and mapping module 2061 via an interface. Alternatively, they can be included in a separate module that is connected to the mapping module 2061 via an interface. CAD data is understood to mean, on the one hand, site plans in 2D or 3D, which come from software programs such as those used for building planning. However, it can also

Image data (e.g. PNG, JPEG, PDF) can be used, from which the system derives appropriate information regarding the structural arrangement. The consideration of these site plans supports the service robot 17, for example, in recognizing passages and door openings, which can be noted accordingly by translating the drawing symbols in the mapping module 2061. Furthermore, the site plans can be used to identify temporary or quasi-stationary obstacles in the building, i.e. Objects that are not part of the building and may change their position over time or disappear entirely.

Example 9: Mapping and self-localization using electromagnetic waves.

[0138] In one aspect, the service robot 17 can also take into account electromagnetic waves such as light and / or for improved navigation within the building

Radio signals, for example from WLAN access points. In addition, the light intensity (e.g.

Solar radiation at the time of day) and / or the angle of incidence of the light

Photodetectors are detected on the service robot 17, both during the mapping process and during general navigation in the building. At the

Comparison of the detected light radiation with the mapped may take into account the time of day and the season, as well as the geographical latitude and longitude, in order to be natural

Fluctuations in the angle of sunlight and its intensity must be taken into account. Alternatively and / or in addition, artificial light can also be recorded in the building, both with regard to the intensity and the light spectrum.

As an alternative and / or in addition, WLAN signals from a plurality of routers, the location of which is known in the building and is noted in the map of the service robot 17, can be recognized by the WLAN module 2088. Via triangulation is therefore a better one

Position determination in the building possible. During the detection of the incidence of light, the speed, the distance traveled, the orientation of the service robot 17 in space etc. are recorded and stored and compared with stored values. These methods described in this example can be combined with other methods described in this document.

Example 10: Measuring the distance covered by the patient using odometry and patient tracking

In one aspect, the distance traveled by the patient is determined using the odometry module 2081 together with an optical sensor such as the LID AR 2083 and / or the 3D camera. Here, at least one optical sensor detects the position of the patient in relation to movement training

Service robot 17, the path of which is in turn determined by the odometry module 2081. Depending on aspects, this can have magnetic sensors that determine the rotation of the wheels 6 and take into account the distance traveled over the radius. The e.g. Inaccuracies resulting from slip can be corrected by combining them with suitable inertial sensors such as acceleration sensors. The basis for the position of the patient, who is identified by the service robot 17 in the person identification module 2041 and tracked with the aid of the person tracking module 2043 or the module for 3D person tracking 2046, can be the center point between the detected legs (in the case of detection by the LID AR 2083) and / or the midpoint between the detected hip joints, the spine, the ankles 1950, etc. in the case of

Patient acquisition with a sensor, whose data are evaluated as a skeleton model.

With regard to the ankles 1950, an approach analogous to that of the LID AR can be used. The distance that the patient has traveled is determined relative to the distance of the patient. [0141] In one aspect, the position of the ankle point 1950 can only be inadequately extracted from the skeleton model, ie the position is detected, for example, with an increased blur compared to other joint points. In order to still increase the detection accuracy, the ankle point 1950, as shown in FIG. 21, alternatively and / or additionally via the position of the knee joint point 1930, a direction vector which is oriented parallel to the lower leg from the knee joint point, and the height of the

Determine knee joint point 1930 above the floor in the event that the direction vector passes through the plumb line (FIG. 21 a), the height of the knee joint point 1930 above the floor in the case of the direction vector passing through the plumb line indicating the distance at which the ankle 1950 extends from the Knee joint point seen from 1930 (see Fig. 21 b). In one aspect, the segmentation methods used for UAGS detection can be used to determine the direction vector parallel to the lower leg, for example in order to detect a detected point cloud in connection with the skeleton model as the lower leg.

Example 11: Measuring the distance covered by the patient using

Sum up the recorded stride length

[0142] In one aspect, the distance traveled by the patient is determined by

Totaling the stride length determined, which is recorded as part of the movement training. The basis for this can be the distance between the ankles, which can be recognized by 3D cameras such as a Microsoft Kinect or an Astra Orbbec and the associated evaluation frameworks. Alternatively and / or additionally, the position of the feet can also be determined using the LIDAR.

Example 12: Measuring the distance covered by the patient using

Coordinate system s

[0143] In one aspect, the distance traveled by the patient over the

Totaling Euclidean distances between coordinate points that the patient passes. The service robot 17 previously loads in the context of the mapping module 2061

Coordinate system of its environment. By self-localization 2062, the service robot 17 determines its position in space, the position in turn being assigned corresponding coordinates. An optical sensor such as the LID AR 2083 and / or the 2D camera 2084 or the 3D camera detect the patient as part of the exercises and determine its position relative to the service robot 17. This position is also assigned a spatial coordinate. A chain of spatial coordinates is thus determined by tracking the patient over time. The Euclidean distance can be determined between each coordinate point, which are successively added up. The basis for the center of the patient can be the center between the detected legs (in the case of detection by the LID AR 2083) and / or the center between the detected hip joints, the spine, etc. in the case of patient detection with a sensor whose data be evaluated as a skeleton model. The position of the patient in the room can also be determined on the basis of the patient coordinates, without necessarily taking into account the position of the service robot.

Example 13: Determining and outputting the route to be completed

The service robot 17 can compare the distance traveled by the patient with the route planned according to the exercise plan and give the patient information via loudspeaker 2092 and speech synthesis 2073 as to how long the remaining distance to be completed according to the exercise plan is. Alternatively and / or in addition, the output can also be shown on the display 2087.

Example 14: Determining the patient's speed

The service robot 17 records the time of the exercises and, in parallel, the distance traveled. In this way, the service robot 17 can determine the speed of the patient and store it in the motion evaluation module 2050. The patient's speed can be compared with a previously stored speed, a difference is calculated and the service robot 17 can provide information on the extent to which the patient deviates from the historical value during the exercise via the display and / or the loudspeaker 2092 and speech synthesis 2073.

Example 15: Pilot of the patient

The service robot 17 navigates in front of or follows the patient during the actual exercise. In both cases, the motion planning module 2065 is used, in one aspect using evolutionary algorithms, to estimate the path of the patient. In both cases, the service robot 17 navigates the patient through the exercise area. If the service robot 17 picks up the patient in his room in advance navigation is initially carried out through an area in which the patient has not yet completed any exercises before reaching the exercise area, which is primarily characterized by a few obstacles (including other people who move there). The service robot 17 guides the patient by means of acoustic and or visual / optical cues, which are given via the loudspeakers 2092 or the display 2087, in one aspect also via signal lights of the service robot 17. Here, an acoustic procedure is chosen especially when the service robot 17 is located behind the patient. The patient is tracked and the position of the patient is compared within the building. In the event of deviations from the planned route, the service robot 17 signals a directional correction, for example advising the patient to turn into a certain aisle, and possibly also to reverse, if the patient should not have turned.

Example 16: Movement planner

[0147] In order to efficiently calculate the distances to be covered by the service robot 17, various goals are simultaneously optimized as part of the movement planning in the movement planning module 2065. First, the service robot 17 calculates the optimal route in the path planning module 2064. In the event of a change in its environment, in particular on the previously calculated path, the service robot 17 does not completely recalculate the route to the destination, but only for the section on the path that is dynamic has changed. Secondly, the dynamic movement planning also takes into account the orientation of the service robot 17, for example shortly before the target, when the service robot 17 moves directly towards it. Thirdly, it is taken into account that the service robot 17, for example, can only drive backwards when directed more slowly than, for example, when it is directed forwards. Fourth, this optimization also takes into account the keeping of safety distances from static and dynamic obstacles. And fifth, the service robot 17 takes into account that the

Service robot 17 maintains a certain distance from a tracked person, in this case the patient. These target variables are considered together as a cost function and the individual target variables are presented as a weighted sum in order to undertake a global optimization of the route. The approach is based on the dynamic window approach known in the prior art (Fox et al 1997). In one aspect, evolutionary algorithms that optimize the acceleration of individual path sections can be used to select individual path sections. Using this movement planning, the camera can be adjusted, for example, taking into account the path of the service robot 17 and the estimated direction of movement of the patient, such that the patient can be captured centrally by the camera, for example. A PID controller can be used for this, which uses the integrator clamping known in the prior art (ie causes an upper and lower limit of the results) and adjusts the horizontal camera angle. Alternatively and / or in addition, the angle determined by the PID controller can also be used to correct horizontal angles in the skeleton model, which are caused by the rotation of the

Service robot 17 occur compared to the orthogonal camera perspective.

Example 17: Practice areas

[0149] During the exercises, a certain distance is usually covered in the aisles of a clinic or in another area that was stored as such in the room plan module 2060r. In particular, those areas should be selected that have few obstacles that the patient and / or service robot 17 must avoid. In one aspect, these spaces have a width which corresponds to at least three times the width of the service robot 17. Obstacles are also understood to mean people who, apart from patients and service robots, 7 also move in this area, including other patients from the clinic, medical staff and, for example, beds, trolleys, seating, etc., which can be found in everyday clinical practice. Fewer such obstacles not only allow a more fluid movement training of the patient, depending on the state of health, also less difficult, but also a better detection of the patient by the service robot 17. There are fewer cornering movements, thus shortening the time

The distance traveled and the storage capacity of the battery carried are protected. Furthermore, there are fewer incidents in which people move between the patient and the service robot 17. This case in turn could lead to tracking problems, but also to the critical situation that the service robot 17 incorrectly recognizes another person as a patient. Avoiding this situation thus reduces the cases of person re-identification, which would then have to be carried out with the 2044 re-identification module.

In one aspect, the service robot 17 is also able to independently

Navigating in pre-defined areas (e.g. a clinic) to recognize such subareas which are suitable for exercises. To do this, he records the time, for example the daily and weekly schedule, the number and type of obstacles, their dimensions (absolute and / or relative to the aisle width), the density of these obstacles over the entire area as a function of time. These values are stored in the memory of the service robot 17, for example within the room plan module 2060r. This data acquisition over time can be recorded both in the context of “empty trips” of the service robot 17 (ie trips without parallel movement training with a patient) as well as during exercises with a patient. The recorded data on the obstacles can be processed within the service robot 17 or within a cloud, into which the data are transmitted.

If an exercise is now pending with a patient, the service robot can determine and / or forecast which areas have had the lowest density in the past during the exercise time (for example Friday 13: 00-13: 15) by accessing the historical data had obstacles. The density determination can include clustering methods. In one aspect, the service robot 17 also takes into account the distance to be traveled

Distance, therefore makes route choice decisions depending on the density. In this case, the service robot 17 can select a gear of 50 m for an envisaged route length of, for example, 100 m, which is run back and forth, or the service robot 17 recognizes on the basis of the historical data that a section of the aisle with a length of 25 m had a density of obstacles at the time in the past (either a minimum density, e.g. calculated as the lowest 90%, or a density below a certain relative or absolute threshold value). In such a case, the service robot 17, for example, independently selects the area with a length of 25 m which the patient is to walk 4x.

Example 18: Arrangement of the cameras and distances to the patient

In order to capture a patient as best as possible, the 2D camera 2084 and / or the 3D camera is mounted in such a way that the 2D camera 2084 and / or the 3D camera can record the patient as centrally as possible. Taking an average

For patient sizes of 175-180 cm, the camera is therefore mounted at a height of 80-90 cm. Depending on the technology used for the 3D camera (ToF for a Kinect, speckle sensor for an Astra Orbbec), there are different implications for the optimal distance to the patient. On the one hand, this distance is determined by the angle of the 3D camera and the resulting possibility of capturing the patient with the whole body in the best case. A Kinect, for example, has a range of up to 4m, which means that a patient can be detected without any problems, ie his body parts are also recognized at this distance. The situation is different when, for example, the UAGS are to be tracked and, above all, a 3D camera with speckle technology is used. Guidi et al. 2016 that a Kinect2 has a significantly higher measurement accuracy than a number of speckle cameras. If you take a standard Astra Orbbec 3D camera, this 3D camera has a horizontal exit opening of 60 ° and a vertical one of 50 ° with a resolution of 640x480. At a distance of 2m from the 3D camera, an image with a horizontal width of approx. 231cm is taken. If the 3D camera is attached at a height of 92 cm, the maximum height of the captured image is approx. 184 cm. Larger people cannot be recorded with it. In one aspect, therefore, it is tracked at a greater distance from the patient what can be at the expense of UAGS detection, depending on what is used for a camera model. Therefore, in one aspect, UAGS identification and UAGS tracking is carried out via the LID AR 2083 and / or an ultrasound or radar sensor, for which the signals of the at least one camera system and possibly LIDARs 2083, ultrasound or radar sensor are used to record the UAGS synchronized and merged accordingly. In one aspect, the camera and LID AR 2083, ultrasound or radar can be used together to detect and / or track the UAGS.

Example 19: Exercise interruption

As shown in FIG. 18, the patient can interrupt the exercise plan at any time by taking an optional break in step 510. Alternatively, the patient can cancel the exercise in step 515. To do this, the patient must

Log out service robot 17 in step 520. If the patient forgets to log off or the patient is inactive for a defined period of time due to the break, the system is automatically logged off in step 525. In order to continue or restart the exercises, you have to log on to the system again in step 530. Example 20: Documentation of therapy

The system with the patient administration module 160 and the service robot 17 documents the therapy. The documentation itself can be transferred to external systems such as a hospital information system or a patient data management system 170 via an interface. On the one hand, the documentation includes personal data such as name, age, gender, etc., the anamnesis (e.g. osteoarthritis in the hip in the

advanced stage), the treatments performed (total hip replacement) and the operated side. Such information is primarily in the memory of the

Patient data management system 172 maintained. In the patient administration module 160 itself, the exercise plans according to Figs. 4 and 5 as well as the results of the exercises completed, which result, for example, from FIG. 11. This also includes any

Interventions in the exercise plan, be they initiated by the therapist or by the

Service robot 17 based on established and / or learned rules.

Example 21: Therapist notification

[0156] In one aspect, a server that interfaces with the

Patient administration module 160 is connected, transfer data to a mobile device that is available to the therapist, for example. As a trigger for this

Notifications can in turn serve to deviate from threshold values in the evaluation of the movement sequence, for example a stride length that deviates from the “typical” deviating stride length, or a combination of movements of different ones

Limbs that do not correspond to the physiological movement sequence and that do not correspond to the "typical" deviating movement sequence. In the former case, for example, a step length could trigger a trigger event that only makes up 10% of the physiological step length. In the second case, a step length of possibly 20cm, with an inclination of the upper body of more than 30 ° from the vertical could trigger a trigger. Furthermore, a trigger can also trigger a deviation in the exercise plan that a certain one

Threshold falls below or exceeds, for example, a two-point gait instead of a three-point gait, without the therapist having released it or without this being due to the

Patient data and the exercise results previously recorded by the service robot 17, which were recorded in a database with historical values, occur in this phase of the exercise cycle with a frequency that is above a defined threshold value, the threshold value being derived in one aspect from the evaluation of the historical data results. Of the Exercise cycle includes the typical sequence of components of the exercise plan, calculated from the first exercise that the patient completes to the last.

Example 22: Cloud-based speech recognition

[0157] Speech recognition 2074 may be implemented in one aspect via third-party cloud-based services that the service robot 7 accesses wirelessly via an API. Here, for example, a speech-to-text API such as Google Speech or Amazon Transcribe can be considered in the first step. In a second step, the text data generated in this way can then be evaluated using APIs such as Amazon Comprehend and the results, for example using commands, which can also be implemented alternatively in the form of a screen-based menu input, can be converted into answers or commands for the service robot 17. Combinations of these services are also possible via a single API.

Example 23: Anonymization of data transfer between the cloud and the service robot

The therapist can assign the patient a mobile storage unit such as a transponder, i.e. the therapist hands the patient a transponder and assigns this transponder to the patient in the patient administration module 160. The transponder contains the patient ID and / or a further token ID which is assigned to the patient or his patient ID. The patient can identify himself on the service robot 17 with this transponder or the serial number and / or patient ID. The service robot 17 now downloads from the cloud 18 in accordance with the exercise plan stored by the therapist - but without his personal data - via an interface - the assignment is made via the patient ID. After completing the exercises, the service robot 17 loads the data recorded by the service robot 17 during the exercises into the patient administration module 160 in encrypted form - the assignment is made via the patient ID. The data is only decrypted in the patient administration module 160. In this way, no data are transmitted that contain conclusions about the patient's name or address.

[0159] In another aspect, the therapist transfers the exercise plan to one

Storage medium (eg transponder in the form of an RFID tag, USB stick), which the patient receives in order to identify himself at the service robot 17. The data are transferred from the storage medium to the service robot 17 including the patient ID, which is from the Patient administration module 160 was specified. After completing the exercise, the service robot 17 transfers the recorded data of the exercises back to the

Storage medium so that the therapist can transfer the data into the patient administration module 160 when reading out the storage medium.

[0160] There are also combinations of the approach described above and the

Data exchange via storage medium (e.g. transponder) possible.

In another aspect, the patient can identify himself via a login and / or password. These serve as a patient ID or are associated with a patient ID, so that the service robot 17 can use this information to download further data such as exercise plans from the cloud. In a further aspect, biometric features such as a fingerprint scan, face scan or iris scan can be used for patient identification.

Example 24: Detection of leg stress using 2D / 3D sensors

[0162] In some cases, patients may have limited leg load. Although it is difficult to implement exact load detection optically, indications can be derived from some skeletal parameters that indicate the leg load. The following parameters are used for this: a) angle between the forearm and upper arm when walking, and / or b) angle of the lower leg to the thigh and / or extension and

Flexion angle, and / or c) posture of the upper body. Relief occurs above all if i) the angle between the forearm and upper arm is less than 170 ° or less than 160 °, ii) the angle of the leg that can only be loaded to a limited extent is less than 172 °, iii) the upper body around is inclined more than 5 ° forwards, and / or iv) the upper body is inclined more than 5 ° away from the affected side. The more pronounced the characteristics i-iiii), the greater the relief of the leg. This defines that an outstretched arm has an angle of 180 °. The person is in the three-point gear, which is explained in more detail in FIG. 22. In addition to the evaluation mentioned there, the arms are recorded and evaluated, while the other aspects are part of the

Characteristic classification 765 and movement sequence classification 770 can be evaluated.

The poses determined are accordingly classified and are stored in the rules 150, both in the rules 150 and locally on the service robot 17 in the frame the evaluation of the movement sequence in the movement sequence evaluation module 2052. In the event of a load restriction which is stored in the patient administration module 160 and which is also incorporated into the exercise plan, the load on the patient's leg is continuously monitored by the processor of the service robot 17 - in parallel with the evaluation of the other poses and the patient receives instructions optically and / or acoustically if the leg load exceeds or falls below a certain threshold.

Example 25: Detection of leg stress using external sensors

[0164] The intensity of a person's leg load can also be determined by means of external sensors, which are wirelessly coupled to the service robot 17, for example. In orthopedics there are a number of products for measuring the pressure on the sole of the foot. For example, insoles are known that are placed in the patient's shoes and measure the pressure on the shoe sole in different spatial resolutions. The actual pressure sensor system can be capacitive, resistive (e.g. based on

Strain gauges), inductive or piezoelectric pressure sensors. The sensor signals are amplified via a bridge circuit and processed by means of an analog-digital converter in such a way that transmission via commercially available radio standards such as Bluetooth, WLAN etc. is possible.

The service robot 17 also has a software unit that the

Measurement results of the different sensors added up over time to determine the total load on the leg. These historical values are continuously compared (by a comparison unit) with the load parameters that are stored in the exercise plan. If a load deviation is determined which is above or below a defined threshold value, the patient receives instructions optically and / or acoustically to reduce or increase the load.

Example 26: Radio location of patients

The service robot 17 has at least one radio interface such as.

Bluetooth, which enables wireless communication with nearby IoT devices. Here, the UAGS have at least one sensor that is wirelessly connected to the service robot 7 via this interface. The sensor effect can only result from the fact that its position is recognized. This can be done through active or passive location and associated triangulation of the position. Passive location means a backscattering of radio signals primarily emitted by the service robot 17, the sensor not having its own power source. Such methods are sufficiently described in the prior art in RFID technology. Active location means a transmitter with its own power supply. The service robot 17 detects the emitted signals and uses triangulation to determine the position of the transmitter. Such an implementation is also superior to visual identification of people when the patient and service robot 17 move in an environment in which there are many people. As a result, the direct line of sight service robot - patient is interrupted more often, so that re-identification often has to take place.

Example 27: Signal button on walking aids

In another aspect of the walking aid e.g. the UAGS have a button that transmits a signal wirelessly to the service robot 17 when pressed. The button is attached in such a way that it can be reached by the patient without any problems while doing the exercises, in particular without having to change the load on the legs. If the walking aid is, for example, a UAGS, the button can be located on the distal end of the T-shaped angled handle, which is enclosed by the patient's hands when walking. The sensor device on the walking aids is configured so that a different number of button presses or also the

Pressing frequency different commands can be triggered by the service robot 17. Pressing once can signal the service robot 17 that the patient wants to sit down. The exercise is interrupted. Pressing twice can signal that the patient has recognized that the service robot 17 is following another person instead of the patient (for example, indicates that a person has moved between the patient and the service robot 17) and / or a re-identification (in the re -Identification module 2044) was not successful. After sending such a signal, the service robot 17 interrupts the training and continues this training only after the patient has logged on again in person. This prevents the personalized training from being taken over by other people and unauthorized access to the data of the already registered person.

Example 28: external sensors on the patient Furthermore, the patient, in particular the limbs, the trunk and / or head, but also the UAGS sensors (in particular acceleration sensors), which are connected to the service robot 17 via at least one interface, can also be attached. These sensors allow the movement of patients to be recorded and

evaluate. It is advantageous here to attach sensors to at least as many elements of the limbs, such as, for example, also using 3D skeletal detection and further processing, in particular as part of the feature extraction 720. This would mean, for example, on each thigh, on the trunk, etc one each

Attach accelerometer. The data recorded here can be used alternatively and / or in addition to 3D detection by means of optical sensors (LID AR 2083 2D camera 2084, and / or 3D camera). The evaluation of this data can in turn

Angle information is provided in the same way as the 3D point clouds, which can be used to build on the algorithms of the 3D sensor technology.

Example 29: Fall detection using optical sensors

To ensure the safety of the patient, the service robot 17 has a fall detection. On the one hand, this can be done via the integrated optical sensors, i.e. based on a skeleton model. The system recognizes from the angle of the upper body, the position or the angle of the head, shoulders, but also the legs relative to the plumb line that the patient is in a fall position, lying on the floor or on his knees has gone. The distance from articulation points identified in the skeleton model to the floor can serve as an alternative and / or supplementary evaluation, i.e. As soon as these fall below a threshold value (e.g. in the knee with a distance of 25cm), a fall is recognized.

Example 30: Fall detection using inertial sensors

Alternatively, an acceleration sensor / inertial sensor, possibly with

Magnetometer, which the patient is wearing or which is integrated into the patient's walking aids. Inertial sensor-based detection of falls or generally “falling to the floor” / falling objects is described, for example, in US7450024 and US8279060. The sensors used here can either use a wireless

Transmission technology such as RFID or Bluetooth transmit the determined acceleration to the service robot 17, which then determines whether the determined acceleration is one Has exceeded the threshold. Alternatively and / or in addition, the

Threshold value determination takes place directly in the sensor system, which merely transmits the information as to whether a threshold value has been exceeded. The threshold value of the acceleration can relate to the maximum of the acceleration, to the angle of the acceleration and / or to a combination of these values.

Example 31: automated movement sequence classification

[0171] It is shown as an example how a movement sequence classification 770 can be carried out on the basis of the data of an exercise recorded and evaluated by the service robot. These data are, for example, on the one hand time-variant data such as distance traveled, speed, standing time, leg phase, as well as the exercises to be completed according to the exercise plan and on the other hand time-variant data, patient data such as age, height, possibly person weight, type of operation , Day of the operation performed, etc. (shown in FIG. 11). This data is all collected and fed into the lem module memory 192 of the lem module 190, where it accumulates over time, a historical one

Form data base and be evaluated from time to time. The evaluation assumes that over the period of a patient's stay in the clinic (from the time of the operation to discharge), the movement sequence, which initially has many disorders, should continue to adapt to the physiological movement sequence, i.e. ideally, the exercise should bring about an improvement, which is reflected in normalizing evaluation parameters. Specifically, the values should increase significantly with distance traveled, speed and duration of the exercise towards discharge and the frequency of corrections should decrease. Depending on the affected / operated side, the values from standing time and leg phase must adjust and normalize in the course. This in turn depends on the exercises that the patient is performing

Exercise plan completed, but also patient data such as his age, comorbidities, etc.

The human sequence of movements () is very different individually. However, there are various standard values described in the literature such as torques, angles and accelerations as well as loads and essential characteristics. These form the basis for standardizing, for example, gait training in the form described. Walking at UAGS or other walking aids is in itself a disturbance or deviation of the so-called "normal sequence of movements", but the sequence of setting the supports should, in the manner already detailed, especially in the

orthopedic / surgical area. This sequence of movements, but also the sequence of movements when using other walking aids, can be standardized well and is defined by the interaction of various movement parameters or features over time, e.g. setting the UAGS or walking aids relative to the feet, the respective sequence of steps, step length, etc. These and other features are now recorded for each gait cycle, which characterize the movement sequence for themselves or walking on UAGS / walking aids, and, in one aspect, the average value over the gait cycles per exercise is formed; it is known whether it is the patient's first exercise, the second, the third, etc. So these constellations are on

Movement flow characteristics are those that are classified and that act as dependent variables, while the results of the exercise (e.g. Fig. 11) following the recorded movement sequence characteristics, the patient data and the exercise plans act as the influencing variables (independent variables) that determine the classification .

Such a procedure requires multi-target or multi-output regressions (for an overview of different approaches see Borchani et al, WIRE's Data Mining Knowl Discov 2015, DOF 10.1002 / widm.1157). Since the constellations of

Movement features are metrically available, this approach is also called

called multidimensional classification. There are two basic ones

Procedures: a) transform the problem by estimating the dependent variables independently of each other, and b) adapt the algorithms in such a way that a simultaneous estimation is necessary. The latter approach takes into account the interdependencies of the dependent variables, which better reflects the reality of the movement. In the latter procedure, decision trees or GLM estimates (general linear models) can be used, which are implemented, for example, in software packages such as glmnet for R. An alternative approach is to use multivariate regression trees (known by the acronym MRT), implemented in the statistical software R in the mvpart package. Alternatively or in addition, the CLUS software package can also be used, which uses decision trees in the context of predictive clustering. The classes resulting from these procedures result in the updated

Classification of movements. Example 32: Detection and expansion of the two-point course

The determination of the two-point course is carried out in many steps in analogy to

Three-point gear. There are some differences in feature extraction 720,

Feature classification 765, movement sequence classification 770, the

Movement sequence correction such as decision classification 775 and the output of movement sequence correction 450. The two-point gait is used either immediately after the operation (depending on the instructions of the surgeon) or after the three-point gait if the patient has recovered further. Ideally, the two-point gear takes place

symmetrical gait behavior of the two legs with the UAGS. Specifically, one leg and the corresponding contralateral support are placed forward, for example, simultaneously and at approximately the same height.

23 illustrates the two-point gait. To a large extent, there are analog ones

Evaluations on the three-point gear take place. The main difference is that information about the leg 1805 to be protected is not necessarily required. As a result of the detection of the position of the UAGS end points 1970 in contact with the ground 1850, it is first detected in step 1856 whether the UAGS and a contralateral leg are in the front in the sagittal plane. A perpendicular of the UAGS end point 1970 touching the ground and of the contralateral ankle joint 1950 to the sagittal plane is formed 1858, and the distance of the perpendicular to one another in the sagittal plane is determined 1859. Alternatively or additionally, the minimum of the distance between the UAGS can be determined and the contralateral base point are evaluated. This distance is in the

Within the scope of the feature classification in step 1886, the extent to which the deviation is or whether the course of the two-point course is carried out correctly (step 1888). For this purpose, for example, threshold values for the determined distances can be evaluated. This data then flows into the movement sequence classification 770 and instructions are assigned to detected deviations in the movements, for example in the decision classification 775. Here, in some cases, other threshold values and

Combinations of classified features as the basis for the three-point gear, in order to trigger, for example, the appropriate output of the movement sequence correction 450, which are stored, for example, in the correction matrix (see FIG. 9). Example 33: Projection of instructions by the service robot

In one aspect, the service robot 17 is equipped with a projection unit by means of which the service robot 17 can project instructions to the patient, for example on the floor. The instructions may include information on speed, posture, distance (turning, turning, etc.) and may be textual in nature and / or based on pictograms, including traffic signs (such as stop,

Turn arrows, etc.). For this purpose, the service robot has a projection unit, for example a commercially available projector. This has a defined projection surface, i.e. the projected area is, for example, within defined areas on the floor, for example at a distance of 2 to 3 m from the service robot and with a width of 1.2 m. If the distance between the service robot and an obstacle, such as a wall, is less than the distance over which the service robot projects from the projection onto the floor, the projection would fall onto the obstacle at least in part. However, this can affect the legibility of the projected content. The service robot therefore carries out a comparison with its sensor system for obstacle detection. In one aspect, this can be carried out on the basis of at least one map which the robot generates dynamically from its surroundings and which contains fixed and possibly movable obstacles. Alternatively and / or additionally, the sensor data can also be evaluated directly in such a way that no obstacles may be detected for the projection area. The sensors used here can be a LID AR, a camera, ultrasound, infrared and / or radar sensors. The service robot is also configured so that it can adjust the type of output based on the type of obstacles found. If output is necessary, but the projection surface is at least partially covered by an obstacle, the service robot can instead select, for example, the display and / or the voice output. Such a projection can replace or complement many of the display and / or voice output mentioned elsewhere in this document.

Example 34: Two- and three-point gear detection

[0177] This example comprises a computer-implemented method for detecting a movement sequence of a person, comprising a contactless detection of a person over time, creation of a skeleton model of the person and evaluation of the articulation points and / or direction vectors of the limbs extracted from the skeleton model Classification of the person's movement. Alternatively and / or additionally, the movement sequence is classified into two-point or three-point. In one aspect, instructions are given to the person to issue a

Movement sequence correction 450, which takes place, for example, as an output via a voice output, projection and / or a display. In one aspect, the

Motion sequence correction 450 depending on a combination of detected

Deviations in the course of movement, these deviations being defined, for example, by rules stored in a memory. In one aspect, these deviations are determined as deviations in the symmetry of the movement sequence. There is a minimum period of time between individual editions of the instruction about the movement sequence correction, which is stored in a memory and in one aspect of the number

previous corrections during the exercise. As part of the computer-implemented method, forearm crutches (UAGS) are identified in one aspect and evaluated in terms of time and space, in particular the end points of UAGS 1970 that touch the ground. In one aspect, at least one end point of the UAGS 1970 is evaluated in relation to at least one ankle point 1950 in the sagittal plane. For example, the distance between the ankle point 1950 and the end point of the UAGS 1970 is evaluated and compared with the threshold value stored in a memory. In an alternative and / or supplementary aspect, it is recorded whether the UAGS are set approximately parallel to the frontal plane of the person being recorded. A line can be determined between the UAGS and the distance of this line from an ankle point 1950, the ankle point 1950 being, for example, the leg joint point of a leg which is to be protected.

In an alternative and / or supplementary aspect, the position of a

Ankle joint point 1950 to endpoint 1970 of a contralateral UAGS in the sagittal plane was evaluated. This is done, for example, for the leg and the UAGS, which are in the direction of walking of the person in front. This evaluation between the ankle point 1950 and the UAGS takes place, for example, when the foot in question and the UAGS touches the ground.

In an optionally supplementary aspect, the method determines person-related parameters, for example the stride length, standing time, track width, the distance between the UAGS in the frontal plane, the trunk inclination, head inclination and / or joint flexion and / or joint extension and classifies these in one aspect. The process captures this personal parameters in one aspect within at least one gait cycle of the detected person. In one aspect, deviations from the symmetry of the movement sequence are determined and evaluated within the gait cycle. The procedure differentiates between physiological and deviating movements. In one aspect, this can be done using the F1 score defined above. In one aspect there is one

Feature extraction of parameters from a skeleton model, a feature classification 765, a movement sequence classification 770 and, for example, a decision classification 775 with a suitable output of the movement sequence correction 450 which may result therefrom.

[0180] The method may further determine the duration of an exercise that

speed of the person recorded and / or the type of

Include support insert / walking aid of the person captured.

Example 35: Method for detection of a forearm crutch

[0181] This example includes a computer-implemented method for evaluating the position of a forearm crutch (UAGS) relative to the position of at least one ankle point 1950 of a person. In one aspect, the person is recorded over time, for example without contact. A skeleton model of the person is created 725. The

Sensor data are evaluated in the spatial environment of at least one wrist point for the presence of UAGS. For this purpose, for example, a candidate region is selected 745. This can, for example, essentially extend downward from the wrist points. Then, for example, point clouds in the vicinity of the

Wrist points recorded and compared with data stored in a memory. In one aspect, the orientation of the forearms can be used, which is, for example, approximately parallel to the orientation of the UAGS. For example, model assumptions can also be taken into account which relate, for example, to the shape 750. A fault-tolerant segmentation algorithm 755 can be used to detect the UAGS. The data for the point clouds are recorded by at least one sensor that the person can contact without contact, for example a 2D camera 2084, a 3D camera, a LID AR 2083, a radar and / or an ultrasound sensor. If more than one sensor is used, the data is synchronized and merged in time. In one aspect, the position of the UAGS at the point where the UAGS touches the ground in the sense of the end point is evaluated.

This location is then evaluated, for example, relative to the position of an ankle point 1950, for example, at the time when the foot touches the floor. The evaluation takes place, for example, in the Sagittal plane, in one aspect the distance between the ankle point 1950 and the end point of the UAGS 1970 is determined. In one aspect, the two-point gait of the person or the three-point gait of the person is then evaluated.

Example 36: Robot system

[0182] This example comprises a service robot with at least one computer, at least one memory 10, at least one camera, at least one device for detecting obstacles in the close range, an odometry unit 2081 and one

Display, e.g. a touch display 2087, a module for 2D and / or 3D detection of the environment 2061, a map module 2061b, a self-localization module 2062, a module for controlling and / or initiating the automatic charging of a

Energy storage and a module for movement planning 2065. This service robot can alternatively and / or additionally, a mapping / mapping module 2061a, a module for metric path planning 2064, a module for detecting a self-blockade 2068, a module for addressing users 2066, a module for determining an optimal distance to a user 2067, a spatial planning module 2060r, a module for determining a waiting position 2069, a module for laser-based personal tracking 2043, a module for person re-identification 2044, a module for motion evaluation 2050

Movement sequence extraction module 2051, a movement sequence evaluation module 2052, a module for movement correction 2022, a module for storing patient parameters 2014, a module for storing a exercise plan 2012, and / or a module for

Evaluation of exercises 2013, which are recorded by the sensors of the service robot, include. In one aspect, the service robot can have an RFID interface for data exchange 2089. In one aspect, the motion planning module 2065 uses information from the metric path planning 2064, for example to determine an optimal path of the robot, taking into account various target or cost functions. For example, the module for motion planning 2065 can be the expected one

Determine the patient's distance. A person 2044 can be re-identified, for example, by means of color patterns, size parameters of a person and / or a classification of the gait pattern of the person. In one aspect, the movements recorded, evaluated and corrected can be gait sequences. Example 37: Anomaly detection in gait behavior

[0183] This example includes a computer-implemented method for anomaly detection in a person's motion sequence, and has a feature extraction 720, a feature classification 765, and a motion sequence classification 770, wherein the feature extraction 720 is based on a person's skeleton model. The

Characteristic classification 765 is based on the evaluation of space-time parameters for articulation points of the person and / or direction vectors between articulation points. Anomalies are characterized, for example, by the amplitude height, positions of minima and / or maxima and / or turning points, which are determined on the basis of the observation of the articulation points over time. In one aspect, anomalies about the evaluation of a gait cycle are recorded. For example, the stride length, service life, track width, the trunk inclination, head inclination and / or joint flexion are recorded and evaluated. In one aspect, the position of the ankle points 1950 is additionally evaluated relative to detected walking aids, for example forearm crutches. The evaluation can be carried out, for example, by determining the distance in the sagittal plane. The movement sequence classification 770 comprises the evaluation of at least two parameters such as stride length, standing time, track width, trunk or head inclination, joint flexion and / or the ratio of the position of ankle points 1950 and walking aids. An anomaly is identified by the deviation of determined parameters from classification parameters, such as defined ones

Thresholds. The threshold values can be, for example, values that were generated by evaluating previously recorded movement sequences. In one aspect, classification can be done through machine learning techniques. It takes place in an optional, additional aspect in the event of a deviation from the classified

Parameters of threshold values a note direction of a terminal.

In one aspect, the data from which the skeleton model is created is recorded, for example as a video sequence. The sequences are provided with a time stamp, for example when an anomaly is detected. For example, a transmission of the

Video sequence to a memory. This memory can be located in a lem module 190. Video sequences can be prioritized in the memory. In one aspect, this prioritization of video sequences depends on the type and / or number of anomalies detected. In one aspect, an anomaly can be a deviation of the recorded, completed exercises from an exercise plan, such as the duration of an exercise, the speed traveled, the type of use of the supports / walking aids, etc.

Example 38: Automated exercise plan adjustment to determine the transition from three-point to two-point gear

This example comprises a system for the automated determination of the transition from three-point to two-point gear, with a processor and a memory, the processor and the memory via an interface, for example an interface to

Data transmission, connected to a sensor for contactless detection of a person. In an alternative and / or supplementary aspect, it is a computer-implemented method for adapting exercise plans based on processed data from a sensor for contactless detection of a person, in one aspect the exercise plan adaptation defining the transition from the three-point to the two-point path represents. In one aspect, after the transition from the three-point to the two-point gear has been determined, an automated intervention takes place, for example an automated adaptation of a exercise plan and / or the transmission of a message via an interface. The system or computer-implemented method compares, for example, recorded values by

Further processing of data (which was determined by the sensor for contactless detection of the person) with corresponding reference values. The values can be person-related parameters such as speed, standing time, playing leg phase, stride length, etc. of a person, which were detected, for example, by means of a contactless sensor. In one aspect, the personal parameters are a combination of speed, standing time, leg phase, and / or stride length. The system or computer-implemented method can, for example, the symmetry between the

Determine gait movements of a person's left and right leg, e.g. via the gait cycle, and use deviations from symmetry to determine the transition from three-point to two-point gait. In one aspect, the detected deviations from the physiological movement sequence are recorded and evaluated in terms of their type and frequency. In one aspect, correction information stored in the memory is also taken into account, which during the recording of the exercise of the person (for example by the system or a system connected via an interface with at least one sensor) contactless detection of a person), for example the type and frequency of these correction notices.

The values were recorded in one aspect over time, for example over several exercises. The system or the computer-implemented method can, for example, carry out a difference analysis over several exercises in order to

To determine the person's exercise progress. In one aspect, the determined

Exercise progress compared with previously determined exercise progress of other people and / or reference data stored in a memory.

Example 39: Method for independently choosing a training plan configuration taking historical data into account

[0188] This example includes a computer-implemented method for a person to independently choose exercise plan configurations. The exercise plan configuration desired by the person is compared with releases within a exercise plan and can only be selected if there is an approval. In addition, the release can take place on the basis of an automated training plan adjustment. In one aspect, data previously made by the person and recorded and evaluated by means of a contactless sensor are compared with historical data. For example, a release for a training plan configuration can be released if there is a high degree of agreement between the recorded data and the historical data. The exercise plan can, for example

Get gait training. Furthermore, for example, the type and / or number of outputs of movement sequence corrections 450, for example, gait corrections, or the same

underlying movement deviations can be used to determine an automated release. In one aspect, the computer-implemented method is implemented in a service robot, for example one for the acquisition and evaluation of gait training.

Example 40: System and / or computer-implemented method for the automated adaptation of a training plan

This example comprises a system for the automated adaptation of a training plan by means of a wireless interface, at least one processor and at least one memory, which contains at least one database in which a training plan is stored. In addition, the system can receive data via the wireless interface from a second system, which is equipped with a sensor for contactless detection of a person, e.g. a camera, records and evaluates exercises stored in the exercise plan and makes these recorded exercises available to the first system poses.

Furthermore, the system can receive information from a third system, in which

Exercise schedule adjustments are made. As an alternative and / or in addition to these systems, a computer-implemented method can be used for an exercise plan

Receive records of the exercises of a person completing the exercise plan. The exercise plan may include information about the person, including their agility, type of surgery, location of surgery, height, and / or person weight. The adaptation of the exercise plan can include the distance to be covered, the type of use of the supports / walking aids etc. The recorded data are

personal parameters and can be saved, for example, over time in the system. The personal parameters can include Service life, track width,

Game leg phase, upper body, line of sight and / or the stride length of a person include, for example, symmetry values of the stride length. Alternatively and / or in addition, the recorded data can include the temporal and / or local sequence of the use of walking aids such as UAGS. This recorded data with the personal parameters can supplement previously obtained historical data within a database. This recorded data and, for example, the data transmitted by the second system (with the camera for recording a person) and from the third system (for adapting the exercise plan) can be evaluated by means of machine learning and / or neural networks. Personal data such as the age of the patient, his height, person weight, comorbidities, type of operation, etc., as well as the data recorded during the recording of the exercises, can be used as the input variable for these evaluations. For example, the settings of the exercise plan for the exercise at time t or at later time t + 1 can be determined as the starting variable. The one with these

Knot weights determined by calculations can be transmitted to the second system (with the camera for recording a person) and / or the third system (for adapting the exercise plan). These node weights can, for example, be used to automatically adapt the exercise plan. In one aspect, the second system is a service robot that performs gait training, for example. Example 41: System for automated training plan adjustment

This example comprises a first system for the automated adaptation of a exercise plan, which has a patient administration module 160, a wireless interface to a second system with at least one camera, a processor and a memory, and an interface to a learning module 190. A second system can be a service robot 17, for example. In addition, the first system can be connected to a set of rules 150 via an interface. A exercise plan is stored in the patient administration module 190 and can be configured via a terminal 12. The exercise plan can take into account, for example, a person's agility, the type of operation performed and / or the location of the operation (such as region and side, e.g. left knee). Agility includes a person's general condition and / or their comorbidities. The exercise plan can include exercises based on the two-point walk, the three-point walk and / or climbing stairs. As a result of the exercises addressed by the exercise plan, the second system records the duration of the exercises, the distance covered, the frequency of the exercises and / or their intensity, etc. The exercise plan can be transmitted to the second system via an interface and stored there. The second system can be, for example, an exercise with a person, such as gait training in one aspect, and / or an output of one

Motion sequence correction 450, for example triggering a gear correction. The recorded data are saved and evaluated over time. The data can, for example, be transmitted to a learning module 190 via an interface. The learning module 190 has historical data in a lem module memory, which are supplemented by the evaluated data and transmitted via the interface. For example, the historical data are supplemented by settings made in the exercise plan. With those made

Settings can be e.g. is about the exercise plan and / or exercise plan adjustments that were made on the basis of the recorded data. By means of a

The machine learning method and / or neural networks can be used, for example, to determine the node weight for the exercise plan. The evaluation of the exercise and optionally also the components of the exercise plan are used as the input variable for determining the knot weight, and configurations of the exercise parameters are used as the starting parameter

Exercise plan and / or exercise plan changes. In one aspect, the node weights determined here are transferred to a set of rules. There you can e.g. Replace already stored node weights for the training plan adjustment. Furthermore, a

Regular updates to the exercise plan can be made. Based on the rules 150, for example, suggestions can be made for a training plan adjustment, for example based on the evaluation of historical data. In one aspect, these suggestions can be made available via a terminal. In an alternative and / or supplementary aspect, the suggestions are made on the basis of data recorded by the system with at least one camera, a processor and a memory and / or the configuration of a training plan. The exercise plan, in turn, can include exercises for gait training.

Example 42: Computer-implemented procedure for motion sequence evaluation

[0191] The example comprises a computer-implemented method for

Movement sequence evaluation of a person, the procedure being the recording of the

Movements of a person, a feature extraction 720 of the recorded movements, for example the movement sequence; a classification of the recorded movements, for example a movement sequence classification 770; a feature classification 765 of the recorded movements; an assignment of instructions to detected

Deviations of the movements, for example a decision classification 775 and an output of the movement sequence correction 450 of the movements, for example an output of a gait correction. The feature classification 765 includes, for example, the evaluation of personal parameters such as stride length, leg phase, standing time, etc. of the person being recorded. The movement sequence classification 770 comprises, for example, a combined consideration of different limbs and / or the use of forearm crutches. The output of the movement sequence correction (for example an output of a gait correction) comprises the output of instructions to the person, for example via a loudspeaker, a projection and / or a display. The recorded movements can be exercises as part of gait training, for example. The expenditure of

Movement sequence correction 450 can also take place in a prioritized manner, in one aspect different deviations which have different priority scores are compared within a time interval, and only those deviations which have the highest priority score are output in each case.

[0192] In one aspect, the data of the movement sequence classification 770, the feature classification 765 and / or the movement sequence correction (for example a decision classification 775) as well as the evaluations of the recorded ones Movements, such as gait training, can be saved with corresponding historical data.

In an alternative and / or supplementary aspect, the recorded and evaluated data, i.e. For example, the results of the movement sequence classification 770, the movement sequence correction such as the decision classification 775 and / or the raw data of the recordings are transmitted to a learning module 190. The method also includes access to the data in the learning module, for example via a terminal, and a re-evaluation of the movement sequence classification, for example one

Gait sequence classification, a re-evaluation of the feature classification and / or a re-evaluation of the movement sequence correction, e.g. the decision classification, e.g. with regard to the output of a movement sequence correction 450. This reassessment can be done manually, for example. In an alternative and / or supplementary aspect, the at least one manual rule adaptation is transferred to a rule set 150. In the rule set 150, for example, an update of the movement sequence classification, for example the gait sequence classification, the feature classification and / or

Movement sequence correction such as the decision classification, for example a movement sequence correction. The updated rules of the

Movement sequence classification, for example the gait sequence classification, the

Characteristic classification and / or movement sequence correction, for example the

Decision classification, transmitted to the system which records the movements of the person. In the event that the raw data of the motion recordings are video recordings, these motion recordings can be anonymized in an optional step, for example by pixelating the faces. In the lem module 190, for example, the video sequences can be tagged, in particular with regard to a deviating and / or physiological sequence of the movement sequence. Time stamps can be assigned, for example.

In addition, for example, an automated reassessment of the movement sequence and / or an automated reassessment of the movement sequence correction, such as the decision classification, can be carried out. This automated reassessment takes place, for example, by means of machine learning and / or neural networks. The evaluations of the movements recorded, for example an exercise such as a gait training, and serve as output variables the movement sequence classification, for example a gait sequence classification, the feature classification and / or the

Movement correction, for example the decision classification. The node weights determined in the automated reassessment for the feature classification, the movement sequence classification, for example a gait sequence classification and / or the movement sequence correction, for example a decision classification, can be transmitted to the rule set 150, for example. This means that 150 the node weights of the movement sequence classifications stored in the rules, for example a gait sequence classifications, characteristic classifications and / or

Movement sequence corrections, for example the decision classification, are updated and, for example, also the corresponding rules of the movement sequence classification, for example the gait sequence classification, the feature classification and / or the

Movement correction, for example the decision classification. In a further step e.g. the updated node weights and / or rules are transmitted to the system that detects the person.

Example 43: Service robot with the ability to select one independently

Practice areas

[0195] This example shows a service robot 17 with at least one optical one

Sensor unit, a radar sensor and / or an ultrasonic sensor and at least one memory that contains a map of the area surrounding the service robot, wherein the service robot 17 detects and evaluates the number and / or type of obstacles in its area in order to identify such subareas within the map that have a low density of obstacles. Areas, for example, are stored in the map in which the service robot 17 mainly moves. The obstacles are recorded over time, for example in the daily routine, weekly routine and / or monthly routine. The obstacles can be dynamic and / or static obstacles, for example. The dimensions of the obstacles can also be recorded, for example. The dimension can be determined, for example, absolutely and / or relative to the width of an aisle in which the service robot moves. The obstacles are recorded during an exercise during which the service robot 17 moves in the subareas. In one aspect, the service robot, in an alternative aspect an external system into which the data has been transferred, determines the hindrance density over time. The data from the external system then becomes the service robot again 17 transferred. The evaluation of the data includes, for example, clustering. For example, on the basis of the recorded data, time intervals can be defined in which certain subareas have a low Hindemis density. In this way, for example, forecasts can be made about a future density of obstacles. As a result, the service robot 17 can, for example, perform tasks to be completed especially in those subareas which have a low density of obstacles. The tasks to be performed can depend on density threshold values. The service robot 17 can, for example, make route selection decisions based on the density determined.

Example 44: Computer-implemented method for recognizing the load on a leg This example comprises a computer-implemented method for recognizing the load on a leg of a person and has the contactless detection of the movements of the person. The contactless detection can be carried out using a camera. In one aspect, a skeleton model is created and joint points and / or direction vectors between the joint points are evaluated in terms of time and space. Here, e.g. a feature extraction 720, a feature classification 765 and / or a movement sequence classification 770 can be carried out. In one aspect, walking aids are also recorded and the course of the walking aids is evaluated relative to the person's ankle points 1950. In one aspect, a leg to be loaded is placed in the sagittal plane near at least one forearm crutch (UAGS). In one aspect, the leg to be relieved is close to the line that connects the UAGS. In an alternative and / or supplementary aspect, the gait sequence of the person who is supposed to relieve the leg is the three-point gait. E.g. the recorded movements are compared with stored rules. There is, for example, decision classification 775 for evaluating which of the detected deviations should result in a correction output and output of a movement sequence correction 450, for example via the route of a voice output of instructions to the person.

To identify the load on the leg, the angle between the forearm and upper arm, extension and / or flexion angle of the hip and knee and / or the inclination of the upper body (in the frontal and sagittal planes) are evaluated. Relief of a leg is determined by the angle between the forearm and upper arm being less than 170 °, For example, is less than 160 °, the knee angle of the leg, which can only be loaded to a limited extent, for example.

is smaller than 172 ° and / or the upper body is inclined forward by more than 5 °.

Example 45: Walking aid with communication device

[0198] The following describes a walking aid that is equipped with a power source, a control unit, wireless interface and a button that is pressed when pressing the

Wireless interface transmits a signal, is equipped. At the recipient of the

transmitted signal can be, for example, a service robot. In one aspect, the walking aid is a forearm crutch. The button is located at the distal end of the T-shaped handle that is gripped by the patient's hands.

The control unit is configured in such a way that different signals can be transmitted by pressing the button. The pressure frequency and / or the number of pressure events can represent different signals, for example. In one aspect, triggering the button triggers a re-identification of the patient by the service robot.

Example 46: Method for evaluating a three-point gear

[0199] The example comprises a computer-implemented method for evaluating a three-point gait with the following steps:

Direction vectors between articulation points and / or forearm crutches (UAGS) as space-time parameters; recording the position of the UAGS endpoints 1970 over time; and the detection of the position of the UAGS endpoints in 1970 on contact with the ground.

In addition, the determination is made, for example, of whether the position of the contact points of the UAGS on the floor is approximately parallel to the frontal plane of the person. The frontal plane can be determined, for example, by the direction of movement of the person being detected. It follows, for example, whether the foot of the leg to be protected is positioned between the UAGS.

[0200] In one aspect, a connection line is determined between the

Positions of the UAGS on the floor, a determination of the distance of the ankle point 1950 of the leg to be protected from the connecting line and an evaluation of the distance of the

1950 joint point of the leg to be protected to the connecting line. Alternatively and / or in addition, the position of the UAGS end points 1970 and the ankle point 1950 of the leg to be spared in the sagittal plane is evaluated with respect to one another and the distances between these points are evaluated. [0201] In addition, at least one of the following aspects can first be ascertained and then evaluated, wherein in the case of more than one aspect, the aspects can be taken into account individually and / or in combination: the inclination of the upper body; the extension and / or flexion, for example of the knee and / or hip joint; the track width, for example by determining the distance of the ankle points 1950 in the frontal plane; the stride lengths, for example by determining the distance of the ankle points 1950 in the sagittal plane, for example subsequently evaluated with regard to the length of the individual steps in the gait cycle; the

Service life; the time of putting on the UAGS and / or feet or ankle points 1950; the distance between UAGS endpoints 1970 in contact with the ground in the frontal plane; the distance between UAGS endpoints 1970 in contact with the ground to ankle points 1950 in sagittal and / or frontal plane; the last two aspects are used to evaluate the UAGS use relative to the body position, e.g. whether the supports are set too close or too far away from the body (sideways or forward).

[0202] In addition, for example, a movement sequence classification 770 can take place, in which at least two features are evaluated in combination and compared with stored rules. If deviations in the movement sequence are recognized, the decision matrix for the detected deviation is used to assess whether an output of

Instructions are given to the person.

Example 47: Evaluation of a two-point gait

[0203] The example discloses a computer-implemented method for evaluating a three-point gait, comprising the detection of articulation points, direction vectors between articulation points and / or forearm crutches (UAGS) as space-time parameters; the

Recording the position of the UAGS endpoints 1970 over time; and the detection of the position of the UAGS endpoints in 1970 on contact with the ground. In addition, the UAGS and the leg that is in the front in the sagittal direction are detected; followed, for example, by forming the perpendicular of the UAGS end point in contact with the ground in 1970 and the perpendicular of the contralateral ankle point in 1950 to the sagittal plane, followed by determining the distance between the two perpendiculars or, for example, determining the UAGS end point in contact with the ground in 1970 and the ankle point in 1950 in of the

Sagittal plane and a distance determination of the UAGS endpoint 1970 and the Ankle point 1950 in the Sagittal plain. In one aspect, the distance of the perpendicular to one another in the sagittal plane or the distance between the two points in the sagittal plane can be evaluated and / or an assessment can be made as to whether there is a contralateral use of UAGS and leg. This evaluation is done, for example, by comparing

Thresholds. In addition, the further aspects from the previous example (last two paragraphs) can be used to evaluate the two-point course as further steps.

Swell:

[0204] Jaeschke, B., Vomdran, A., Trinh, TQ, Scheidig, A., Gross, H.-M., Sander, K., Layher, F. Making Gait Training Mobile - A Feasibility Analysis in IEEE Int . Conf. on Biomedical Robotics and Biomechatronics (Biorob), Enschede, The Netherlands, pp. ..., IEEE 2018

[0205] Trinh, TQ, Wengefeld, T., Mueller, St., Vorndran, A., Volkhardt, M., Scheidig, A., Gross, H.-M. “Take a seat, please”: Approaching and Recognition of Seated Persons by a Mobile Robot in: Int. Symposium on Robotics (ISR), Munich, Germany, pp. 240-247, VDE Verlag 2018.

Vorndran, A., Trinh, T.Q., Mueller, St., M., Scheidig, A., Gross, H.-M. How to Always Keep an Eye on the User with a Mobile Robot? in: Int. Symposium on Robotics (ISR), Munich, Germany, pp. 219-225, VDE Verlag 2018

[0207] Zhe Cao, Tornas Simon, Shih-En Wei, Yaser Sheikh. OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields; The IEEE Conference on Computer Vision and Pattem Recognition (CVPR), 2017, pp. 7291-7299

[0208] G. Guidi, S. Gonizzi, L. Mico, 3D CAPTURING PERFORMANCES OF LOW-COST RANGE SENSORS FOR MASS-MARKET APPLICATIONS, The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLI-B5, 2016 XXIII ISPRS Congress, July 12-19, 2016, Prague, Czech Republic.

D. Fox, W. Burgard and S. Thrun, "The dynamic window approach to collision avoidance," in IEEE Robotics & Automation Magazine, vol. 4, no. 1, pp. 23-33, March 1997

Müller S., Trinh T.Q., Gross HM. (2017) Local Real-Time Motion Planning Using Evolutionary Optimization. In: Gao Y., Fallah S., Jin Y., Lekakou C. (eds) Towards

Autonomous robotic systems. TAROS 2017. Lecture Notes in Computer Science, vol 10454. Springer, Cham. R. Philippsen and R. Siegwart, "An Interpolated Dynamic Navigation Function,"

Proceedings of the 2005 IEEE International Conference on Robotics and Automation, Barcelona, Spain, 2005, pp. 3782-3789.

[0212] Borchani, H., Varando, G., Bielza, C. and Larranaga, P. (2015), A survey on multi-output regression. WIRE's Data Mining and Knowledge Discovery, 5: 216-233. DOI:

10.1002 / wid. 1157

Reference numerals

3 sensor

5 support wheel rotatable about the vertical axis

6 drive wheel

8 energy source

9 computer unit

10 memories

12 Terminal for system administration

13 terminal

17 service robots

18 cloud

150 rules

151 Calculator of the regulations

152 memory of the regulations

160 patient administration module

161 Patient Administration Module Calculator

162 Patient Administration Module Memory

170 patient data management system

171 Computer of the patient data management system

172 Memory of the patient data management system

180 navigation system

181 Calculator of the navigation system

182 Navigation system memory

190 Lem module

191 Lemmodul calculator

192 Lemmodul memory

1305 agility

1310 -General Constitution 1315 comorbidities

1320 place of operation

1325 type of operation

1330 therapist

1335 exercise schedule

1340 two-point gear

1345 three-point gear

1350 stairs

1355 configuration exercise plan

1360 interface

1505 historical data

1701 pivot point

1702 Connection between articulation points

1703 body of a person

1910 Direction vector between knee joint point and hip joint point

1920 Direction vector between knee joint and ankle point

1930 knee joint point

1940 hip joint point

1940r right hip joint point

19401 left ankle point

1950 ankle point

1960 shoulder joint point

1960r right shoulder joint point

19601 left shoulder joint point

1970 end point of the UAGS

1980 stylized UAGS

2010 application level

2011 movement training module

2012 exercise schedule 2013 module for the extension of the exercises

2020 behavioral level

2022 motion correction module

2024 route guidance module

2030 service robot capabilities

2040 person recognition module

2041 personal identification module

2042 First personal tracking module

2043 Second person tracking module

2044 re-identification module

2045 seat detection module

2046 3D person tracking

2050 motion evaluation module

2051 motion sequence extraction module

2052 motion sequence assessment module

2060 navigation module

2061 mapping module

2060r room plan module

2064 path planning module

2065 motion planning module

2071 graphical user interface (GUI)

2073 speech synthesis module

2074 speech recognition module

2081 odometry module

2082 pressure sensitive bumper

2083 LIDAR

2084 2D camera

2085 RGB 3D camera

2086 camera zoom and tilt function 2087 touch display

2088 WLAN module

2089 RFID reader and writer

2090 differential drive

2091 charging port and electronics

2092 speakers

2093 microphone

2094 head with eyes

Claims

Expectations

1. A computer-implemented method for detecting a movement sequence of a person, the movement sequence comprising movements of body elements of the person, the method comprising:

Detection by a non-contact sensor of a plurality of images of the person during a movement sequence, the plurality of images being the

Represents movements of the body elements of the person;

Creation of at least one skeleton model with link positions for at least some of the plurality of images; and

Calculation of the movement sequence from the movements of the body elements of the person by comparing the changes in the limb positions in the at least one created skeleton model.

2. The method of claim 1, further comprising comparing the calculated

Movement sequence with a predetermined movement sequence, which is stored in a memory.

3. The method according to claim 1 or 2, wherein the sequence of movements is a course of gears.

4. The method according to any one of the preceding claims, wherein the calculation of the movement sequence comprises the evaluation of movements of the body elements over at least one complete gait cycle.

5. The method according to any one of the preceding claims, further comprising a

Recognition of at least one walking aid in the plurality of images

Comparison with assistant samples.

6. The method according to claim 4, further comprising a coherent evaluation of the at least one walking aid and at least one ankle point obtained from the skeleton model.

7. The method according to claim 5, wherein the evaluation comprises determining the difference between the at least one ankle point and a bottom end point of the at least one walking aid.

8. The method according to claim 6, wherein the difference in the sagittal plane is determined.

9. The method according to any one of claims 5-7, wherein the evaluation to a

It is placed on the ground.

10. The method according to any one of claims 2-8, further comprising a message

Deviation of the movements in the recorded movement sequence from the

Movements in the predetermined sequence of movements.

11. The method according to claim 10, wherein the number of messages issued depends on the number and type of the detected deviations of the movements.

12. Device for performing the method according to one of claims 1 to 11.

13. System for detecting a movement sequence of a person, the

Sequence of movements Movements of body elements of the person includes, the system comprising:

at least one sensor for contactless detection of a plurality of images of the person during a movement sequence, the plurality of images representing the movements of the body elements of the person;

an evaluation unit for creating skeleton models with positions of the body elements for at least some of the plurality of images and

Calculation of movements of the body elements of the person by comparing the created skeleton models.

14. The system of claim 13, wherein the movement is a

Gear sequence is.

15. The system of claim 13, wherein the evaluation unit has a memory with predetermined values about the positions of the body elements in a

has intended movement sequence, and compares the predetermined values with the movements of the body elements in operation.

16. The system of claim 13-15, wherein the evaluation unit evaluates the positions of the body elements with the help of walking aids over at least one gait cycle.

17. The system according to any one of claims 13-16, wherein the evaluation unit evaluates a symmetry of the movement of the body elements in operation.

18. System according to any one of claims 13-17 further comprising an output unit for outputting messages when determining deviations between the movements of body elements and the predetermined movement sequence.

19. System according to any one of claims 13-18 further comprising

Segmentation unit for recognizing objects in the large number of images.

20. System according to one of claims 13 - 19, wherein the sensor at least one 2D camera, a depth camera, an ultrasound sensor, a radar sensor or a lidar

Sensor.