WO2004051391A2

WO2004051391A2 - Method and device for capturing the movement of a solid using at least one camera and one angular sensor

Info

Publication number: WO2004051391A2
Application number: PCT/FR2003/003484
Authority: WO
Inventors: Dominique David
Original assignee: Commissariat A L'energie Atomique
Priority date: 2002-11-27
Filing date: 2003-11-25
Publication date: 2004-06-17
Also published as: FR2847689A1; AU2003294084A1; FR2847689B1; WO2004051391A3; AU2003294084A8

Abstract

The invention concerns a method and a device for capturing the movement of a solid using at least one camera and one angular sensor. The invention, which is applicable in particular to video games, is characterized in that it consists in: equipping the solid (2) with at least one first sensor (4) for observing a scene and providing an image thereof and at least a second sensor (6) for measuring a rotation; acquiring, by means of the first sensor, images at successive times, which delimit time intervals for each of which the rotation of the first sensor is determined by means of the second sensor and the translation of said solid from said images.

Description

METHOD AND DEVICE FOR CAPTURING THE MOTION OF A SOLID, USING AT LEAST ONE CAMERA AND ONE SENSOR

ANGULAR

DESCRIPTION

TECHNICAL AREA

The present invention relates to a method and a device making it possible to capture (“sensing”) - it is also said to measure - the movement of an object or, more precisely, of a solid.

The invention applies to any field where there is a need to capture a movement, from the simple attitude station to miniature capture systems with six degrees of freedom. The invention applies for example to the following fields:

- multimedia navigation,

- video games, computer aided design and industrial design,

- computer peripherals,

- communicating objects, and

- health (in particular surgery and home care).

STATE OF THE PRIOR ART

So far, no one has been able to design a motion capture system that is both miniature and versatile. We know that professionals use magnetic or optical capture techniques. As for the techniques reserved for the general public, they must be inexpensive and currently lead either to accelerometry or to three-dimensional measurement from video sequences.

None of these methods is satisfactory: they are all difficult to master from a technical point of view and video requires great computing power and complex implementation.

More precisely, in the field of motion capture with six degrees of freedom, the most efficient professional systems call on active methods, requiring equipping the measurement volume, for example a room, or with cameras (for active optical methods) or magnetic field generators. These techniques are expensive, incompatible with a democratization of the capture of the movement, whose applications are however numerous. In addition, they require experts for their implementation. A lot of research is underway to remedy these drawbacks and to propose systems that are less expensive, reliable and easy to use.

For the moment, the products that result from this research are rare. Nonetheless, one can cite, among these products, the Company's system

Intersense, called "Inertiacube" (registered trademark). This The system uses nine sensors, namely three accelerometers, three magnetometers and three gyrometers.

However, this system is not satisfactory because it works on a principle of signal integration. However, we know that integration methods are extremely difficult to master because they are classically too sensitive to measurement noise. In use, the Inertiacube system (registered trademark) shows this handicap through a significant drift in the measurement, corrected by readjusting approximately every second. This system is therefore difficult to use because it is not reliable enough.

We also refer to the following document:

[1] EP 1 089 215 A, “Optical sensing and control of movements using multiple passive sensors”. This document describes a technique that uses only cameras to reconstruct the six parameters of a movement.

This technique requires complex algorithms to correct a systematic drift during the measurements.

STATEMENT OF THE INVENTION

The object of the present invention is to remedy the above drawbacks. According to one aspect of the invention, translation measurements without integration, carried out, are combined. with passive optical means, and rotation measurements also without integration, carried out with appropriate sensors. No integration is thus necessary. According to another aspect of the invention, one or more video cameras and one or more sensors, such as accelerometers or magnetometers, are associated, the measurements made with these sensors are used as corrective measurements and the latter are applied. to the measurements made with the cameras.

It should be noted that a multimodal system, comprising accelerometers and / or magnetometers, is particularly well suited to restoring the angles of rotation of a solid around its center of gravity. However, it is more difficult to implement such a system with the intention of restoring the translations of this solid.

On the contrary, a video-based system is capable of easily estimating translational movements, at least qualitatively even quantitatively.

The present invention takes advantage of these complementarities to propose a device making it possible to capture the rotations and translations of a solid, with a view to restoring these rotations and translations.

It is advisable to distinguish the device object of the invention from a known device, whose constitution resembles its own but whose objectives are totally different. This known device implements a technique making it possible to restore the third dimension in a scene filmed by a video camera. We use a single camera that we move to generate a succession of different points of view of the same scene.

There are software packages containing algorithms which take into account these different points of view to restore this third dimension in whole or in part. For example, the

Realviz company markets such software.

In some embodiments, the camera can be equipped with a motion capture system. This gives a device which resembles that of the present invention but, in no case, the image output of the camera is combined with the output of the capture system, unlike the present invention.

The output of this system is used, in the algorithms, to restore the third dimension in the scene seen by the camera and not to restore the six degrees of freedom of movement of the camera itself, the content of the image in this case being also unimportant.

The objectives of this known device are therefore completely different from those of the invention.

Specifically, the present invention relates to a method for measuring the movement of a solid, this method being characterized in that:

- at least one first sensor capable of observing a scene is made rigidly integral with the solid and to provide an image thereof and at least one second sensor capable of measuring a rotation,

- using the first sensor, images are acquired at successive times, which delimit time intervals, these time intervals being less than a predetermined value, chosen so that two consecutive images have a common scene part and , for each time interval, we determine, using at least the second sensor, the rotation of the first sensor, and therefore of the solid, between the first and second instants which delimit this time interval, and we determine the translation of the solid between these first and second instants, from the first and second images respectively acquired at the first and second instants.

According to a particular embodiment of the invention, the rotation of the first sensor is determined using the second sensor and the first sensor.

According to a first particular mode of implementation of the method which is the subject of the invention,

- before determining the translation, the rotation is applied to the first image, to obtain a first transformed image, or the reverse rotation to the second image, to obtain a second transformed image, and

- The translation of the solid is then determined according to the first transformed image and the second image or according to the first image and the second transformed image. According to a second particular embodiment of the method which is the subject of the invention, the predetermined value is also chosen so as to be able to neglect the rotation of the solid between the first and second instants and the translation of the solid is determined as a function of the first and second images, neglecting this rotation to do this.

According to a preferred embodiment of the process which is the subject of the invention, several first sensors and several second sensors are made rigidly integral with the solid.

The present invention also relates to a device for measuring the movement of a solid, this device being characterized in that it comprises: - at least a first sensor, capable of observing a scene and providing an image of it, and at least a second sensor capable of measuring a rotation, these first and second sensors being made rigidly integral with the solid, with a view to acquiring, using the first sensor, images at successive instants, which delimit time intervals , these time intervals being less than a predetermined value, chosen so that two consecutive images have a common scene part.

The device which is the subject of the invention may further comprise means for processing the signals supplied by the first and second sensors, these processing means being capable of determining, for each time interval, using the signals supplied by the second sensor, the rotation of the first sensor, and therefore of the solid, between the first and second instants which delimit this time interval, and the translation of the solid between these first and second instants, from the first and second images acquired respectively at the first and second instants.

According to a first particular embodiment of the device which is the subject of the invention, before determining the translation, the processing means are able to: apply the rotation to the first image, to obtain a first transformed image, or the reverse rotation to the second image, to obtain a second transformed image, and - then determine the translation of the solid as a function of the first transformed image and of the second image or as a function of the first image and of the second transformed image.

According to a second particular embodiment of the device which is the subject of the invention, the predetermined value is also chosen so as to be able to neglect the rotation of the solid between the first and second instants and the processing means are capable of determining the translation of the solid from first and second images, neglecting this rotation to do this.

The device which is the subject of the invention may comprise several first sensors and several second sensors. Preferably, each first sensor is a camera. In addition, each second sensor is preferably chosen from magnetometers and accelerometers.

It should be noted that a device according to the invention can be designed, provided with video cameras and angular sensors in sufficient number so that the device can continue to operate even if one of its sensors is defective or if the one of its cameras is obstructed.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be better understood on reading the description of exemplary embodiments given below, by way of purely indicative and in no way limiting, with reference to the appended drawings, in which:

- Figure 1 is a schematic perspective view of a first particular embodiment of the device object of the invention, and - Figure 2 is a schematic perspective view of a second particular embodiment of this device.

DETAILED PRESENTATION OF PARTICULAR EMBODIMENTS

In examples of the invention, one or more video cameras are used which operate in the visible or infrared range. The number of pixels in each camera can be very small. It is worth for example 32x32. Furthermore, in these examples, one or more angular position sensors, or angular sensors, are used, each of these sensors being capable of providing information on inclination with respect to at least one reference axis.

Each angular sensor is for example an accelerometer or a magnetometer, or even a gyrometer. The reference axis is for example a vertical axis, in the case of an accelerometer, or the direction of magnetic north in the case of a magnetometer.

This angular sensor, which is made rigidly integral with the video camera, therefore makes it possible to know the orientation of the latter relative to this axis. The determination of the rotation can also be carried out by using, in addition to information coming from the angular sensor, that coming from the video camera.

If you use several angular sensors with a single sensitivity axis (single-axis sensors) or an angular sensor with several sensitivity axes (multi-axis sensor), you have several sensitivity axes and you can then find out the respective orientations of the various axes of sensitivity with respect to the reference axis mentioned above.

If we use a sensor (accelerometer or magnetometer) with three orthogonal sensitivity axes (tri-axis sensor), we are then able to know the three Euler angles associated with the solid carrying the video camera. It should be noted that in certain applications, it is not necessary to know all the degrees of freedom: some of them can be fixed by assumption or included / understood in a reduced range of values. The invention can then be simplified by using a limited number of sensors.

One can implement a calculation based on the correlation which makes it possible to calculate the direction of translation of the solid (to which are fixed the angular sensor (s) and the video camera (s)), or even the value of this translation in certain cases (namely in a calibrated environment, for example by adding sights to the scene observed). This calculation is then adapted, as will be seen below, to take into account the additional data constituted by the knowledge of the orientation or orientations of the video camera.

For example, the image supplied by this video camera can be subjected to a prior orientation, the value of which is deduced from the rotation indicated by the angular sensor, before this image is subjected to the correlation mentioned above.

In an example of the invention, capable of providing a large number of pieces of information, up to three translation values and three rotation values, we use:

- a tri-axis accelerometer,

- a tri-axis magnetometer and - from one to six video cameras whose respective optical axes are orthogonal. In this case, the three axes of rotation of the camera (or of each camera) are taken into account in the correlation algorithm between successive images. Quantitative information is also available in the case of this example.

Let us specify what is meant by "orthogonal optical axes".

This notion only has meaning if we use at least two video cameras whose optical axes, denoted X and Y, are then orthogonal.

If we add a third video camera, we arrange it so that its optical axis Z is orthogonal to X and Y. If we also use a fourth video camera, we arrange it so that its optical axis is parallel to X.

If a fifth video camera is also provided, it is arranged so that its optical axis is parallel to Y.

If a sixth video camera is also used, it is arranged so that its optical axis is parallel to Z.

Figure 1 is a schematic perspective view of an example of the device object of one invention.

The device of FIG. 1 is intended to measure, or capture, the movement of a solid 2.

The device comprises a video camera 4, an angular sensor 6, which is an accelerometer or a magnetometer, and electronic means 8 provided for memorize and process the information or signals supplied by the video camera 4 and by the angular sensor 6 and to memorize the results of the treatment. This video camera and this angular sensor are fixed to the solid 2 whose movement is to be measured. The electronic means 8 may or may not be integral with this solid 2.

The video camera, the axis of which is denoted X in FIG. 1, operates in the visible range or in the infrared range. At a given instant, it provides an image of a scene 10, that is to say of a part of the environment of the solid 2, which is in the field of this video camera 4. With the aid of the latter, images are recorded at successive instants. It is assumed that these instants are close enough that part of the scene recorded in one image is also found in the following image. More precisely, the successive instants delimit time intervals and the duration of each of these is less than a predetermined value, chosen so that two consecutive images have a common scene part. This choice depends on the maximum speed that the solid 2 is capable of reaching, the depth of field of the scene observed and the optical characteristics of the camera.

As a purely indicative and in no way limitative, a value of 20 ms or 40 ms is chosen, corresponding to the standard rate of acquisition of a camera, for a maximum speed of lm / s, for a depth of field of a few meters and for a focal length of a few millimeters.

Are known, in particular from document [1], techniques for recovering the movement of the video camera. When this movement includes a rotation, these techniques become much more complex from the computational point of view and simultaneously less "robust", that is to say less resistant to measurement noise.

The present invention overcomes these drawbacks as will be understood in the light of the following.

With the device of FIG. 1, the movement of the solid is measured in the following manner.

For each time interval [tl; t2] delimited by instants tl and t2, the video camera 4 provides at the instant tl (respectively t2) an image il (respectively i2) and the angular sensor 6 provides an angle al (respectively a2).

This angle is the angle formed, at this instant, between the axis of sensitivity of the sensor and a reference direction, namely the vertical direction, in the case where the sensor is an accelerometer, and the direction of magnetic north when it it is a magnetometer.

We know that between instants tl and t2 the movement of solid 2 decomposes into a translation and a rotation. According to the invention, this movement is determined in the following way by the electronic means 8. In a first step, these electronic means 8 determine the rotation of the video camera 4, and therefore that of the solid 2, between the instants tl and t2, from the measurements provided by the angular sensor 6: it is a rotation whose angle is a2 - al.

In a second step, these means 8 determine a transformed image, resulting from the angle rotation a2 - al of the image il, so that this transformed image is "aligned" with the image i2, that is to say - say made parallel to the latter.

As a variant, the means 8 determine a transformed image by applying in this case a rotation of angle al - a2 to the image i2 to further obtain an alignment of the image il and of this transformed image.

In a third step, from the images thus aligned, the electronic means 8 carry out a conventional correlation to determine the translation of the solid 2 between the instants t1 and t2.

We thus know the movement (translation and rotation) of the solid 2 between these instants.

By repeating the steps mentioned above, it is possible to determine and store the movement of the solid 2 during each time interval considered and therefore the movement of this solid 2 between the initial instant and the final instant of its displacement.

The advantages of the method and of the device which have been described with reference to FIG. 1 are indicated below. On the one hand, the algorithms are simplified and consume less computing power than in the prior art.

On the other hand, the device uses only components which are available at low cost.

In addition, the method does not require any mathematical integration, that is to say that it does not work from the first or second derivatives of the trajectory of the solid, derivatives which must then be integrated, which is a source known to drift.

This process is therefore very resistant to measurement noises, which is a decisive advantage compared to known techniques.

In a variant of the method for measuring the movement of a solid, which has been explained above, it is possible to use an even faster correlation to recover the translation of the solid between the times t1 and t2.

To do this, we replace the conventional two-dimensional correlation, which is used to determine the translation of the solid, in a plane perpendicular to the optical axis X of the camera, by two one-dimensional correlations of the projections of the two images il and i2 first on an axis XI then on an axis X2, XI and X2 being two axes perpendicular to the plane mentioned above (which is perpendicular to the axis X).

The device of FIG. 1 allows the quantitative determination, in any time interval considered, of the translation of the solid, parallel to a plane which is perpendicular to the optical axis of the camera 4, and of the rotation of this solid around this axis.

This device comprises only one video camera so that only two components of the translation vector can be easily measured in the time interval considered, namely the projections of this vector in the image plane of the video camera. .

The third component is theoretically also accessible with this same camera but this requires signal processing which is complex and not very robust.

To completely and robustly measure the translation of the solid, it is necessary to fix on the latter, in addition to the video camera 4, at least one or even two other video cameras 4a and 4b whose respective optical axes Y and Z define a trirectangle trihedron with the X axis, in order to obtain the three components of the translation vector in the time interval considered.

Likewise, the angular sensor of the device in FIG. 1 only makes it possible to determine two angles of rotation of the optical axis X of the video camera 4 (uni-axis sensor), whence only a partial determination of the rotation of this camera, and therefore of solid 2, during the time interval considered.

To fully measure the rotation of the solid, you must use at least one tri-axis angular sensor 7 (or at least three uni-axis angular sensors), instead of the uni-axis sensor 6, in order to obtain three angles of rotation solid 2 during the time considered and be able to rotate the image mentioned above for each video camera used.

The movement of the solid 2 is then measured in the manner described below. For each time interval [tl; t2] and each of the three video cameras, the image il (respectively t2) is acquired at the instant tl (respectively i2) and the angle al or bl or cl (respectively a2 or b2 or c2) between the sensitivity axis associated with this camera and the reference axis.

We then determine the three angles of rotation of this camera (first step), we successively apply the three rotations corresponding to the image it or the inverse rotations to the image i2 (second step) and, these images then being

"Aligned -", a conventional correlation is carried out to determine the component of the translation vector, which is associated with the camera considered

(third step) . One thus determines qualitatively the translation (ie the three components of the vector of translation) and quantitatively the rotation of the solid 2 for each time interval and therefore for the meeting of these intervals. The three video cameras mentioned above can be fixed respectively on the three faces of a trihedron, in the case where the solid comprises such a trihedron.

In the example of FIG. 2, the solid 12 has the shape of a cube and six video cameras 18a, 18b, 20a, 20b, 22a and 22b) are respectively fixed on the six faces of this cube.

We denote x, y and z axes of the cube, such that each of these axes passes through the centers of two opposite faces of the cube and is perpendicular to these two faces.

The latter carry two cameras whose optical axes coincide with the x or y or z axis associated with these faces. As can be seen, the x, y and z axes are respectively associated with the pairs 18a-18b, 20a-20b and 22a-22b of video cameras.

In addition, several angular sensors are fixed on the solid 12, for example an accelerometer 24 having three axes of sensitivity and a magnetometer 26 also having three axes of sensitivity.

The advantage of using both this accelerometer and this magnetometer is as follows: it is thus possible to access the measurement of the three degrees of freedom of rotation of the solid in a robust and inexpensive manner.

Similarly, the advantage of using more than three video cameras, namely six cameras in the example of FIG. 2, is as follows: the system then becomes redundant and, therefore, even more robust.

It should be noted that the device in FIG. 2 also comprises electronic storage and processing means 28, capable of storing and processing signals supplied by the video cameras and the angular sensors and of storing the results of the processing. Now consider another example of the invention.

Suppose that the solid 12 (figure 2), whose movement we want to restore, is affected by a rotational movement around the x axis (see figure

2), this rotational movement being superimposed on a translation along the y axis (Figure 2).

The angular sensors (that is to say the tri-axis accelerometer 24 and the tri-axis magnetometer 26) indicate the rotation around the x axis.

If this rotation is low - which is always possible by choosing a sufficiently small time interval between two consecutive measurements - one of the video cameras whose optical axis coincides with the z axis (Figure 2) essentially sees the imposed translation solid 12.

It is then no longer even necessary to rotate the image observed along this optical axis (second step in the algorithm mentioned above). The images provided by this camera make it possible to directly obtain the corresponding translation component.

This treatment can of course be applied (neglect of rotation) for all the cameras with which the solid 12 is provided.

In the example of FIG. 2, the angular sensors 24 and 26 are fixed on the same face of the cube. However, these angular sensors, which are mechanically attached to the video cameras, could be distributed over the cube. The device of Figure 2 has various advantages.

Compared to a device on which only video cameras are fixed, the device in FIG. 2 is more robust, that is to say better withstands measurement noise.

Indeed, a rotation for example around the x axis results in apparent translations on the cameras whose optical axes coincide respectively with the y and z axes. We must therefore be able to distinguish whether it is a real translation or a rotation. This results in an algorithmic complexification, while any of the angular sensors fixed to the solid 12 provides this information.

In addition, the redundancy of the information provided by several sensors of different natures results in increased robustness. This point is crucial for the creation of a device intended for the general public, which must operate in very variable environments.

Claims

1. Method for measuring the movement of a solid (2, 12), this method being characterized in that:

- at least one first sensor (4, 4a, 4b, 18a, 18b, 20a, 20b, 22a, 22b) is rigidly secured to the solid, capable of observing a scene and providing an image of it and at least a second sensor (6, 7, 24, 26) capable of measuring a rotation,

2. Method according to claim 1, in which the rotation of the first sensor is determined using the second sensor and the first sensor.

3. Method according to any one of claims 1 and 2, wherein - before determining the translation, the rotation is applied to the first image, to obtain a first transformed image, or the reverse rotation to the second image, to obtain a second transformed image, and

- The translation of the solid (2, 12) is then determined according to the first transformed image and the second image or according to the first image and the second transformed image.

4. Method according to any one of claims 1 and 2, wherein the predetermined value is also chosen to be able to neglect the rotation of the solid (2, 12) between the first and second instants and the translation of the solid is determined. depending on the first and second images, neglecting this rotation to do this.

5. Method according to any one of claims 1 to 4, in which several first sensors (4, 4a, 4b, 18a, 18b, 20a, 20b, 22a, 22b) are rigidly secured to the solid (2, 12) and several second sensors (7, 24, 26).

6. Device for measuring the movement of a solid (2, 12), this device being characterized in that it comprises:

- at least a first sensor (4, 4a, 4b, 18a, 18b, 20a, 20b, 22a, 22b), capable of observing a scene and providing an image of it, and - At least one second sensor (6, 7, 24, 26) capable of measuring a rotation, these first and second sensors being made rigidly secured to the solid, with a view to acquiring, using the first sensor, images to successive instants, which delimit time intervals, these time intervals being less than a predetermined value, chosen so that two consecutive images have a common scene part.

7. Device according to claim 6, further comprising means for processing (8, 28) the signals supplied by the first and second sensors, these processing means being able to determine, for each time interval, using the signals provided by the second sensor, the rotation of the first sensor, and therefore of the solid (2, 12), between the first and second instants which delimit this time interval, and the translation of the solid between these first and second instants, from first and second images respectively acquired at the first and second instants.

8. Device according to claim 7, in which, before determining the translation, the processing means (8, 28) are able to: apply the rotation to the first image, to obtain a first transformed image, or the reverse rotation to the second image, to obtain a second transformed image, and - then determine the translation of the solid as a function of the first transformed image and of the second image or as a function of the first image and the second transformed image.

9. Device according to claim 7, in which the predetermined value is also chosen so as to be able to neglect the rotation of the solid (2, 12) between the first and second instants and the processing means are capable of determining the translation of the solid from first and second images, neglecting this rotation to do this.

10. Device according to any one of claims 6 to 9, comprising several first sensors (4, 4a, 4b, 18a, 18b, 20a, 20b, 22a, 22b) and several second sensors (7, 24, 26).

11. Device according to any one of claims 6 to 10, in which each first sensor (4, 4a, 4b, 18a, 18b, 20a, 20b, 22a, 22b) is a camera.

12. Device according to any one of claims 6 to 11, in which each second sensor (6, 7, 24, 26) is chosen from magnetometers and accelerometers.