EP3485463A1

EP3485463A1 - Method and system for locating and reconstructing in real time the posture of a moving object using embedded sensors

Info

Publication number: EP3485463A1
Application number: EP17748526.5A
Authority: EP
Inventors: Sylvie NAUDET-COLLETTE; Vincent WEISTROFFER; Saifeddine ALOUI; Claude Andriot; Cyril Condemine; Romain DUPONT; Jean-Philippe Gros
Original assignee: Commissariat a lEnergie Atomique CEA; Commissariat a lEnergie Atomique et aux Energies Alternatives CEA
Current assignee: Commissariat a lEnergie Atomique et aux Energies Alternatives CEA
Priority date: 2016-07-13
Filing date: 2017-07-10
Publication date: 2019-05-22
Also published as: WO2018011498A1; FR3054061A1; FR3054061B1

Abstract

The invention relates to a system for reproducing the movement of an object by means of an articulated structure, which comprises an optical sensor (C0) and relative movement sensors (C1-C4) attached to the object, and a unit (20) for processing measurements made by the sensors (C0, C1-C4). The unit comprises a module (21) giving the absolute location of each segment of the structure in a reference frame, said module utilising the absolute location ({R ^a ₀,p ^a ₀},{R ^a _i ,p ^a _i}) of each sensor. The unit further comprises: - an image processing module (22) for determining the absolute location ({R ^a ₀,p ^a ₀}) of the optical sensor (C0) in the reference frame from a sequence of images (D0) acquired by the optical sensor, - a module (24) for determining the relative location ({R⁰ _i,p ⁰ _i) of each relative movement sensor (C1-C4) in relation to the optical sensor, from the measurements made by the sensors, - a module (23) for determining the absolute location ({R ^a _i ,p ^a _i}) of each relative movement sensor (C1-C4), from the absolute location of the optical sensor and the relative location of each relative movement sensor.

Description

METHOD AND SYSTEM FOR REAL-TIME LOCALIZATION AND RECONSTRUCTION OF THE POSTURE OF A MOVING OBJECT USING ONBOARD SENSORS

DESCRIPTION

TECHNICAL AREA

The field of the invention is that of the capture of the movement of an object, and the real-time transfer of this movement to a (poly) articulated mass structure modeling the object.

The invention finds application to the capture of the movement of a human body via sensors arranged on a person, and the representation of this movement via the animation of an avatar in a numerical simulation. STATE OF THE PRIOR ART

In a number of applications, we seek to represent the movements of a person in a numerical simulation. In the majority of cases, the person carries a set of sensors whose data are used to animate his digital avatar in the simulation.

The main needs of a motion capture are, on the one hand, to locate the person in the environment, and, on the other hand, to estimate the complete posture of the person.

To meet both these needs, it is now necessary to instrument the environment. Cameras are for example placed in the environment and observe the person (or sensors carried by it) to detect its movements.

The instrumentation of the environment, however, imposes several constraints: it is costly in time, it limits the capture area, and it is incompatible with external environments or a priori unknown. This is why the use of sensors completely embedded on the person is a recurring need. But few solutions exist today to respond jointly to the two needs mentioned above by means of embedded sensors only, without instrumentation of the environment. However, the article by T. Shiratori et al. entitled "Motion capture from body-mounted cameras", ACM Trans. Graph. 30, 4, Article 31 (July 2011) which describes a solution based on several cameras placed on the torso and the limbs of the person. For each camera, a Structure from Motion (SfM) algorithm is applied to calculate the camera's successive positions. Then, a global optimization makes it possible to find the location and the posture of the person over time. This overall optimization consists in solving a beam adjustment with additional constraints related to the movement of the skeleton, for example the fluidity of the movements.

This solution however has a number of disadvantages: It is expensive in computing time (more than 10 hours of calculation according to the article). It is sensitive to the quality of the images (ex: blurred images related to the movement, occultation of the sensor by a part of the body, etc.).

It is sensitive to scale factor drift, and reference images are needed to solve this problem

The data stream to be transferred is very large.

DISCLOSURE OF THE INVENTION The object of the invention is to propose a technique for locating and reconstructing the posture of a moving object in real time using onboard sensors only, which is at least partially free of the disadvantages above.

For this purpose, the invention proposes a system for reproducing the movement of an object by an articulated structure comprising a plurality of segments, which comprises:

sensors fixed, in use, on the object,

a unit for processing measurements made by the sensors fixed on the object, comprising a module for determining a command to be applied to the articulated structure so as to locate it in a reference frame by making it adopt a posture reproducing that of the object, said module outputting the absolute location of each of the segments of the structure in the reference frame, said module using the absolute location of each of the sensors attached to the object in the reference frame.

The sensors fixed in use on the object comprise an optical sensor and relative motion sensors and the processing unit further comprises:

an image processing module configured to determine an absolute location of the optical sensor in the reference frame from a sequence of images acquired by the optical sensor,

a module for determining the relative location of each of the relative motion sensors with respect to the optical sensor, based on measurements made by the sensors fixed on the object,

an absolute location determination module of each of the relative motion sensors in the reference frame, from the absolute location of the optical sensor and the relative location of each of the relative motion sensors.

BRIEF DESCRIPTION OF THE DRAWINGS

Other aspects, objects, advantages and characteristics of the invention will appear better on reading the following detailed description of preferred embodiments thereof, given by way of non-limiting example, and with reference to the accompanying drawings. on which ones :

FIG. 1 represents an example of a poly-articulated mass structure used as a physical model (in the meaning of the laws of physics) of a human body;

FIG. 2 is a diagram of a system according to the invention for tracking the movement of an object on which sensors are fixed; FIG. 3 illustrates the calculation of the location of the optical sensor with respect to the reference mark;

FIG. 4 represents the acquisition of the movement data of the limbs in a current coordinate system associated with the optical sensor;

- Figure 5 is a diagram showing the steps of a method according to the invention.

DETAILED PRESENTATION OF PARTICULAR EMBODIMENTS

The invention relates to the control of an articulated mass structure from data representative of the movement of an object. Motion sensors are attached to the object and it is desired to enslave the articulated mass structure to reproduce the movements of the object.

The object is typically a human body or the body of an animal, or even a single part of such a body. The invention is however not limited to these examples, and thus extends to monitoring the movement of any moving object that can be represented by an articulated structure.

The articulated mass structure is composed of segments connected by at least one joint. It is used as a physical model of the object whose movement we follow. It is typically a poly-articulated structure.

The articulated mass structure can be a numerical model of the object. The objective can then be the representation of the movements of the object in a numerical simulation, by means of the animation of the mass structure articulated in the numerical simulation according to a command determined by the invention.

FIG. 1 gives an example of a poly-articulated mass structure that can be used as a physical model of the human body in an implementation of the invention. The poly-articulated mass structure is composed of segments connected by joints. A segment refers to a rigid or supposedly rigid object that is defined by a geometry (that is, a well-defined volume shape), mass, and inertia. An articulation denotes a connection between two segments. This link defines the relative configuration that a segment can have with respect to a segment to which it is bound. An articulation is defined by one or more representative degrees of freedom, in particular the characteristics of centers, axes or stops.

In the example of Figure 1, the poly-articulated structure 10 is composed of twenty-one segments connected by twenty joints. Thus, for example, the hand is modeled by a first segment 13 connected by a hinge 16 to a second segment 12 corresponding to the forearm. The articulation 16 has for example three degrees of freedom. The segment 12 corresponding to the forearm is connected by a hinge 15 with two degrees of freedom to the segment 11 corresponding to the arm. The segment 11 is connected to the clavicle by a hinge 14 with three degrees of freedom.

In addition to the definition of the articulated structure (the geometries of each segment, the masses and the inertia of each segment and the parameters of the joints: degrees of freedom, center, axes, value of the stops), other parameters necessary for the implementation process of the invention may comprise:

The definition of gravity (direction and norm of the gravitational field), whose effect (weight) is represented by external forces applied on the center of mass of the segments of the articulated structure.

The definition of the soil. This definition of the environment includes the geometry of the ground (reliefs, stairs etc.), and a reference reference attached to the ground. In particular, it makes it possible to detect potential collisions between the articulated structure and the environment.

The definition of friction behavior of the articulated structure on the environment. For example, if this behavior is modeled by a Coulomb law, the coefficient of adhesion is given.

Referring now to Figure 2, the invention provides a system for reproducing the movement of an object by an articulated structure. This system exploits the data from a set of sensors CO, C1-C4 fixed on the object, and does not require instrumentation of the environment.

The sensors attached to the object include a central element having an optical sensor, and relative motion sensors. The optical sensor and the module 22 described below make it possible to calculate the position / orientation of the object in a reference reference linked to the environment. The movements of the object, typically the movements of the limbs of a person, are obtained by means of the relative motion sensors and the module 24 described below by which one can have a location, in position and orientation, in a reference relative to the central element.

In the example shown in FIG. 2, the relative motion sensors are four in number, with a relative movement sensor C1, C2 on each hand of the person and a relative motion sensor C3, C4 on each foot of the nobody. The optical sensor C0 is placed on the head of the person. This number of four relative motion sensors is given by way of example only, the invention extending to any number of relative motion sensors.

The optical sensor C0 provides a sequence of images DO, while the relative motion sensors provide motion information Di in a reference relative to the optical sensor, i.e. a current marker corresponding to the current position of the optical sensor in the environment.

These data DO, Di are supplied to a computer processing unit 20. This unit 20 receives the measurements made by the sensors fixed on the object, and is configured, by means of different modules described below, to implement steps at each time step of calculating the configuration to be given to the poly-articulated structure to reproduce the movements of the object.

The processing unit 20 is in particular provided with a module 21 for determining a command to be applied to the articulated structure so that it reproduces the movement of the object. This command more precisely locates the articulated structure in a reference frame ¾ _a , for example a marker attached to the ground, making him adopt a posture reproducing that of the object. The module 21 for determining the command to be applied to the articulated structure uses the absolute location of each of the sensors attached to the object in the reference frame. The following notations are adopted:

- {Rs, Ps) denotes all the orientations and positions of the segments s of the articulated structure, expressed in the reference reference ¾ _a . This set constitutes the outputs of the module 21 for determining the control to be applied to the articulated structure, namely the location and the complete posture of the articulated structure in the reference frame ¾ _a .

- {Rf, pf} denotes all the orientations and positions of the relative sensors Ci (i = 1 ... n) at times t, expressed in ns the reference reference ¾ _a . These orientations and positions are provided to the module 21 for determining the control to be applied to the articulated structure.

- {R _Q , PO} denotes the set of orientations and positions of the current coordinate system ¾ ₀ corresponding to the current position, at times t of the central element da ns the environment, expressed in the reference reference ¾ _a . These orientations and positions, representative of the location of the object, are supplied to the module 21 for determining the command to be applied to the articulated structure.

The module 21 for determining the command to be applied to the articulated structure takes into account the locations in the reference reference frame of the sensors C0, C1-Cn. It is also able, in the context of a physical simulation of an articulated structure, to take into account the effect of gravity, to detect collisions between the geometries of the segments of the articulated structure and the geometry of the structure. environment and resolve the contacts by generating efforts opposing penetrations between the segments of the articulated structure and the environment.

In this respect, the collisions between the geometries of the segments of the articulated structure and the geometry of the environment are detected at each time step for a configuration of the given structure. This detection can be based on the penetration detection of the geometries as on the detection of proximity according to a given threshold. The collision information obtained is the point and the normal of each contact. Collision information is used to generate efforts to prevent penetrations between segment geometries and the environment. A modeling of the friction (for example by a law of Coulomb) allows, moreover, to simulate the adhesion of the segment on the environment and constrain sliding. The resulting efforts are called contact efforts.

The geometry of the environment can be obtained either by a priori knowledge of the environment, for example it is assumed that the ground is flat, or is thanks to the mapping of the environment provided by the SLAM.

The module 21 for determining the control to be applied to the articulated structure can thus exploit a real-time physical simulation engine, such as the XDE engine (for the "eXtended Dynamic Engine") of the Applicant, which makes it possible to perform simulations in a virtual environment by assigning physical constraints to an articulated structure, detecting collisions and managing contacts.

In one possible embodiment, the central element and the relative motion sensors C1-Cn each comprise means for measuring the relative distances between the latter, for example an Ultra Wide Band module. In this embodiment, the module 21 for determining the command to be applied to the articulated structure can implement an estimation algorithm (exploiting for example a Kalman filter). The use of an analytical method, for example of the MDS ("Multi Dimentional Scaling") type, makes it possible to recover the initial posture by merging the distance measurements provided by the central element and the relative movement sensors Cl-Cn and a priori knowledge of a reference posture serving as an initialization posture. Once initialized, the estimation algorithm merges the distance data provided by the central element and the relative motion sensors to determine the position of the joints as a function of time. Ideally, the use of biomechanical constraints (knowledge a priori of the model of the human body, number of segments and their sizes that can be measured during the initialization phase) makes the estimation of the position of the joints more robust. .

The command determined by the module 21 can be used to perform a motion analysis. Indeed, from the knowledge of the type of movement, specific information can be extracted for particular application needs. This command can also be used to perform a complete reconstruction of the movement for display on a screen 30, for example that of a computer or a virtual reality headset. The movement can be modified or amplified to achieve the desired effects.

In order to enable the module 21 for determining the control to be applied to the articulated structure to be located in the reference reference frame of the sensors fixed on the object CO, C1-Cn, the processing unit 20 comprises a module an image processing unit 22 and a relative location determination module 24 of the relative motion sensors.

The image processing module 22 is configured to determine, in real time, an absolute location of the optical sensor C0 in the reference frame ¾ _a from a sequence of images D0 acquired by the optical sensor. This module thus provides the set {R, p £} at times t ^. It can notably implement a visual SLAM type algorithm (for Simultaneous Localization And Mapping designating simultaneous mapping and localization) or SfM. Such an algorithm makes it possible to iteratively reconstruct a map of 3D primitives (generally points) and to locate the optical sensor C0 in this map from the video stream D0. The reference mark is then fixed either on the first position of the sensor or on the reconstructed map.

FIG. 3 shows the calculation of the orientation and the position of the central element with respect to the reference mark, with R and o the orientation and the position of the current mark ¾ ₀ relative to the reference mark ¾ _has at successive times t ^, t ^ ί ¹ and ^2.

The relative location determination module 24 of the relative motion sensors is in turn configured to determine, in real time, the relative location of each of the relative movement sensors C1-Cn, from the measurements made by the sensors fixed on the 'object. This module thus provides the set {Ri, Pi) which designates the orientations and positions of the relative sensors i (i = 1 ... n) at times t, expressed in the current frame ¾ ₀ .

FIG. 4 shows the acquisition of the movement data of the limbs in the current coordinate system ¾ ₀ fixed on the central element, with R and pf the orientation and the position of each relative motion sensor i with respect to the current coordinate system ¾ ₀ at different successive instants t, t ^ ⁺¹ and t ^{+ 2} ·

In a possible embodiment, the central element provided with the optical sensor may be associated with means for completing the measurements from the sensors fixed on the object. This is particularly the case when these measurements are insufficient to provide position information of the relative motion sensors with respect to the optical sensor, for example because they only make it possible to determine a relative distance. This is the case, for example, when the relative motion sensors each comprise a UWB transmitter ("Ultra Wide Band") to enable the calculation of a relative distance from a receiver. In such a case, the central element is provided not with a UWB receiver but with three UWB receivers positioned in a predetermined configuration, each of which makes it possible to recover the relative distance of each UWB transmitter. The relative location determination module 24 can then implement a triangulation to determine the exact position of each motion sensor with respect to the optical sensor.

The processing unit further comprises an absolute location determination module 23 of each of the relative motion sensors in the reference frame. This module 23 exploits the absolute location of the optical sensor {R _Q , PO} ^AND the relative location of each of the relative motion sensors {R. pf} relative to the optical sensor to make a reference change and replace each of the positions and orientations of the relative motion sensors calculated in the current coordinate system at each instant t in the reference frame. This module thus provides the absolute locations {Rf, tf}. In the case where the two inputs of the module 23 do not arrive synchronously (t ^ ≠ t), an additional synchronization step is performed by the module 23 to obtain data synchronized at the output. For example, if the outputs of the module 24 (relative sensors) arrive at a frequency F _C and if the outputs of the module 22 (optical sensor) arrive at a frequency F ₀ <F _C , the module 23 can use the frequency F _C as output frequency and interpolate the missing locations of the optical sensor. An exemplary embodiment of the invention exploits relative motion sensors which are each composed of an inertial unit which provides information on the orientation of the sensor, and a UWB (Ultra-Large Band) transmitter which makes it possible to calculate a relative distance from receivers. The person is equipped with at least 4 relative sensors fixed rigidly on his body, including a sensor on each hand and a sensor on each foot.

The central element comprises a camera, for example a 360 ° camera, as well as three UWB receivers positioned in a predetermined configuration, each of which makes it possible to recover the relative distance of each UWB transmitter.

In a preferred embodiment, the optical sensor comprises at least two cameras rigidly connected and calibrated between them, whose fields of view are overlapped. This is for example a stereoscopic sensor. Alternatively, you can use a RGB-D sensor that provides an RGB image and a depth map of the image scene. The use of such optical sensors makes it possible to obtain a real scale location and to minimize the drift inherent in any monocular SLAM or SfM solution.

Indeed, in the case of a monocular SLAM, the reconstruction and the location are obtained at an arbitrary scale factor. This solution may therefore not be satisfactory for merging the relative movements of the limbs and the displacement of the person. In the case of a stereo SLAM, the scale is provided thanks to the knowledge of the rigid transformation connecting the two cameras. This makes it possible to obtain a reconstruction and a location on a real scale.

In an alternative embodiment of the invention, the optical sensor is composed of a single camera, and to obtain a scale reconstruction it is associated with:

a previously reconstructed 3D point map, or

reference images previously located as in the article by T. Shiratori et al. mentioned in the introduction, or

- a known object in the scene. The invention is not limited to the system as previously described, but also extends to the method of reproducing the movement of an object by an articulated structure implemented by such a system. The invention also relates to a computer program product comprising program code instructions for executing the method according to the invention when said program is executed on a computer.

With reference to FIG. 5, this method comprises a prior "INST" step of placing the relative motion sensors on the object. When it is a human body, it is typically placed relative motion sensors on each hand and on each foot of the person. The optical sensor is also installed, for example on the person's head.

The method includes the acquisition "ACQ. Measurements made by the various sensors fixed on the object (video flow delivered by the optical sensor and measurements of the relative motion sensors).

The image sequence acquired by the optical sensor is subjected to "SLAM" processing to determine the absolute location of the optical sensor in the reference frame.

The measurements made by the sensors fixed on the object are used during a "LOC-R" step to determine the relative locations of each of the relative motion sensors relative to the optical sensor.

Then, during a "LOC-A" step, the absolute location of each of the relative motion sensors in the reference frame is determined from the absolute location of the optical sensor and the relative location of each of the motion sensors. related. If necessary, a resynchronization of the outputs between them is performed during this step "LOC-A".

Once the absolute localizations determined, one comes during a step "DET-CDE" determine the command to be applied to the articulated structure so as to locate it in the benchmark by making him adopt a posture reproducing that of the object . This command can be used to perform a complete reconstruction of the movement for display on a screen, for example that of a computer or a virtual reality headset. The method then comprises an animation "DISP" step, according to the determined command, of the articulated structure in a 3D digital simulation.

The invention allows the capture of movement in real time, without instrumenting the environment, by locating the person in his environment in an absolute way while estimating his complete posture.

The advantages of the invention over the solution proposed in the article by T. Shiratori et al. are the following. The solution is less expensive in computing time. It uses a single optical sensor and therefore offers a cost reduction. The data stream to be transmitted is lighter, and the use of embedded relative sensors (compared to cameras on the limbs) is more robust to abrupt limb movements and occultations. Finally, the drift of the system is reduced by the choice of the optical sensor placed on the central element.

Claims

A system for reproducing the movement of an object by an articulated structure comprising a plurality of segments, comprising:

sensors (CO, C1-C4) fixed, in use, on the object,

a processing unit (20) for measurements made by the sensors (CO, C1-C4) fixed on the object, comprising a module (21) for determining a command to be applied to the articulated structure so as to locate it in a reference frame (¾ _a ) by having it adopt a posture reproducing that of the object, said module providing at the output the absolute location ({R, p _Σ }) of each of the segments of the structure da ns the reference of reference, said module exploiting the absolute location ({R _Q , PO}, {R? _> Pf}) of each of the sensors fixed on the object in the reference frame,

characterized in that the sensors fixed in use on the object comprise an optical sensor (CO) and relative motion sensors (C1-C4) and in that the processing unit (20) further comprises:

an image processing module (22) configured to determine an absolute location ({R, p £}) of the optical sensor (C0) in the reference frame from a sequence of images (D0) acquired by the optical sensor, a relative location determination module (24) {{R. pf}) of each of the relative motion sensors (C1-C4) with respect to the optical sensor, from measurements made by the sensors fixed on the object,

a module (23) for determining the absolute location {{Rf, pf}) of each of the relative motion sensors (C1-C4) in the reference frame, based on the absolute location of the optical sensor and the relative location of the each of the relative motion sensors.

2. The system of claim 1 wherein the optical sensor comprises a stereoscopic sensor.

The system of claim 1, wherein the optical sensor comprises a RGB-D sensor.

4. System according to one of claims 1 to 3, in which each relative motion sensor comprises an inertial unit and an Ultra-Large transmitter

Strip, and wherein the optical sensor comprises three Ultra-Wide Band receivers positioned in a predetermined configuration.

5. A method of reproducing the motion of an object by an articulated structure, comprising a determination step (DET-CDE), from the absolute location ({RQ, Po}, {R † _> p †}), in a reference mark, sensors fixed on the object, a command to be applied to the articulated structure so as to locate it in the reference frame by making it adopt a posture reproducing that of the object, characterized in that that, the sensors fixed on the object comprising an optical sensor (CO) and relative motion sensors (C1-C4), it further comprises the steps of:

processing (SLAM) an image sequence (DO) acquired by the optical sensor to determine an absolute location ({RQ, PO}) of the optical sensor (CO) in the reference frame,

determination of the relative location (LOC-R, {R, pf}) of each of the relative motion sensors (C1-Cn) by reference to the optical sensor, based on measurements made by the sensors fixed on the object,

determining absolute location (LOC-A, {Rf, pf}) of each of the relative motion sensors (C1-C4) in the reference frame, from the absolute location of the optical sensor and the relative location of each of the relative motion sensors.

The method of claim 5, wherein the processing step (SLAM) of the image sequence acquired by the optical sensor comprises implementing a simultaneous mapping and locating algorithm.

7. Method according to one of claims 5 and 6, wherein the articulated structure is a numerical model of the object and further comprising a representation step (DISP) of the movements of the object in a numerical simulation by means of the animation of the articulated structure in the numerical simulation according to the determined command.

A computer program product having program code instructions for executing the method according to one of claims 5 to 7 when said program is executed on a computer.