WO2022117535A1

WO2022117535A1 - Device for compensating for movement of an event sensor, and associated systems and methods

Info

Publication number: WO2022117535A1
Application number: PCT/EP2021/083486
Authority: WO
Inventors: Maxence Bouvier; Alexis BIGE
Original assignee: Commissariat à l'énergie atomique et aux énergies alternatives
Priority date: 2020-12-02
Filing date: 2021-11-30
Publication date: 2022-06-09
Also published as: FR3116977A1; FR3116977B1

Abstract

The invention relates to a device (16) for compensating for movement of an event sensor (12) in an event stream generated by the event sensor (12) during observation of an environment in a time interval, the compensating device (16) comprising: - a compensating unit (34), the compensating unit (34) being able to receive data relating to the movement of the event sensor (12) during the time interval and to apply a compensating technique to the event stream generated by the event sensor (12) depending on the received data so as to obtain, in the time interval, a compensated event stream; and - a control unit (40) for controlling the compensating unit (34), the control unit (40) being able to control the compensating unit (34) via transmission of data relating to the movement of the event sensor (12).

Description

TITLE: Event sensor motion compensation device, associated systems and processes

The present invention relates to a device for compensating the movement of an event sensor. The present invention also relates to an observation system as well as a system for evaluating the position of the event sensor comprising the above compensation device. The present invention also relates to a corresponding compensation and evaluation method.

In the field of onboard video surveillance, a difficulty is to analyze a large volume of images in which many images are not relevant. Indeed, this implies significant material resources and therefore a high energy consumption, which is incompatible with the constraints of an embedded system, namely a limited weight, size and energy power.

A promising way to meet such a challenge is to use an event sensor.

A DVS sensor or an ATIS sensor are two examples of such a sensor. The abbreviation DVS refers to the English name of "Dynamic Vision Sensor" which literally means dynamic vision sensor while the acronym ATIS refers to the English name of "Asynchronous Time-based Image Sensor" which literally means asynchronous image sensor time based.

Traditional imagers supply images, namely a succession of matrices which encode the light intensity values measured by a grid of pixels at a regular frequency. Instead, an event sensor generates an asynchronous and sparse stream of events since a pixel generates an event only when an intensity gradient on the pixel exceeds a certain threshold.

An event sensor therefore allows no data to be transmitted when nothing is happening in front of the event sensor, which greatly limits the number of data to be processed.

In addition, due to the asynchronous operation, such sensors also make it possible to benefit from a high dynamic range and acquisition frequency. In particular, for some sensors, the rate of events that can be generated can be as high as 10 GeV/s (GeV/s means "Giga Events per second" and represents the number of billions of events per second contained in a stream of events).

However, such an acquisition frequency implies in return a significant computing power to be able to process the events of the stream of events. To this difficulty is added the fact that the computational load is, by nature, unpredictable so that it is difficult to process the data with maximum efficiency (which is often obtained when the processing is implemented with maximum load).

Furthermore, due to its intrinsic noise, an event sensor generates spurious events, which further unnecessarily increases the computational load.

Moreover, when the event sensor moves, different pixels emit pulses even in the presence of a stationary object. This results in spatial redundancy, again involving many useless calculations.

There is therefore a need for a device for compensating for the faults introduced by an event sensor in a flow of events generated during an observation of an environment which reduces the computing capacity required to allow a physical implementation in an embedded system while retaining the useful information captured by the event sensor.

To this end, the description describes a device for compensating the movement of an event sensor, in a flow of events generated by the event sensor during an observation of an environment in a time interval, the compensation device comprising a compensation unit, the compensation unit being adapted to receive data relating to the movement of the event sensor during the time interval and adapted to apply a compensation technique to the stream of events generated by the event sensor according to the data received to obtain a stream of events compensated in the time interval, and a unit for controlling the compensation unit, the control unit being able to control the compensation unit by sending data relating to the movement of the event sensor.

According to particular embodiments, the compensation device has one or more of the following characteristics, taken separately or according to all the technically possible combinations:

- the control unit is able to send to the compensation unit information on the absence of available data as data relating to the movement of the event sensor, the compensation unit delivering a stream of compensated events identical to the stream of events generated by the event sensor.

- the control unit comprises a processing sub-unit, the control unit being capable of receiving measurements of the movement of the event sensor, the processing sub-unit being capable of implementing processing on the measurement data received to obtain processed data, the processing being, in particular, a temporal integration of at least some of the measurements received, the unit of control also being able to send the processed data to the compensation unit.

- the control unit is able to receive positions evaluated from the event sensor, the control unit being able to send the positions evaluated to the compensation unit.

- the control unit further comprises a determination sub-unit suitable for determining data relating to the movement of the event sensor to be sent, the determination sub-unit determining the data to be sent according to the data received by the control unit.

The description also describes a system for observing an environment, the observation system comprising an event sensor generating a stream of events during an observation of an environment of the event sensor, a measurement unit capable of measuring the movement of the event sensor during a time interval, and a compensation device as previously described.

According to particular embodiments, the observation system has one or more of the following characteristics, taken in isolation or according to all the technically possible combinations:

- the system further comprises a unit for evaluating the position of the event sensor and a unit for reconstructing frames, the reconstruction unit being able to generate corrected frames from the flow of events compensated in the time interval, the control unit being separate from the evaluation unit, the evaluation unit being able to receive measurements of the movement of the event sensor, the evaluation unit being able to apply a technique of evaluation to the reconstructed frames according to the received measurements to obtain evaluated positions of the event sensor in the time interval.

- the assessment technique is a visual odometry technique or a simultaneous localization and mapping technique.

- the event sensor and the compensation device are part of the same component comprising a stack of at least three layers, the first layer of the stack comprising the event sensor, the second layer of the stack comprising the control and the third layer comprising the compensation unit and the evaluation unit.

The description also relates to a system for evaluating the position of an event sensor, the evaluation system comprising an event sensor generating a stream of events during an observation of an environment of the event sensor, a unit of measure suitable for measuring the movement of the sensor event during a time interval, and a compensation device as previously described.

According to particular embodiments, the evaluation system has one or more of the following characteristics, taken in isolation or in all technically possible combinations:

- the system further comprises a unit for obtaining, the unit for obtaining being able to obtain, for each event of the flow of events generated, at least one characteristic relating to the nature of the movement of an associated object to the event, the object being the object of the environment having caused the generation of the event by the event sensor, the compensation device also comprising a determination sub-unit capable of determining data relating to the movement of the event sensor to be sent, the determination sub-unit determining the data to be sent according to the data received by the control unit and also according to the at least one characteristic relating to the nature of the movement

- the compensation unit is made on another component, the component and the compensation unit being attached to a substrate comprising electrical connections, the substrate being, for example, an interposer.

- the compensation unit, the reconstruction unit, the evaluation unit and the control unit are made on the same integrated circuit. The present description also proposes a method of compensating for the movement of an event sensor in a stream of events generated by the event sensor during an observation of an environment in a time interval, the compensation method being implemented by a compensation device, the compensation device comprising a compensation unit and a compensation unit control unit, the compensation method comprising a step of sending data relating to the movement of the event sensor by the control intended for the compensation unit, a step of receiving data relating to the movement of the event sensor by the compensation unit, a step of application by the compensation unit of a compensation technique to the flow of events generated by the event sensor according to the received data to obtain a stream of events compensated in the time interval, and a generation step by the unit for reconstructing corrected frames from the stream of events compensated in the time interval.

The description also describes a method for evaluating the position of an event sensor, the event sensor generating a stream of events in a time interval, the evaluation method being implemented by a system for evaluating the position of an event sensor, the evaluation system comprising a compensation unit, a frame reconstruction unit, an event sensor position evaluation unit and a compensation unit control unit, the control unit being separate from the evaluation unit, the evaluation method comprising a step of sending data relating to the movement of the event sensor by the control unit to the compensation unit, a step of receiving data relating to the movement of the event sensor by the compensation unit, a step of applying by the compensation unit a compensation technique to the flow of events generated by the event sensor in fo nction of the data received to obtain a stream of events compensated in the time interval, a step of generation by the reconstruction unit of corrected frames from the stream of events compensated in the time interval, a step of reception by the evaluation unit of measurements of the movement of the event sensor, and a step of application by the evaluation unit of an evaluation technique to the frames reconstructed according to the measurements received to obtain evaluated positions of the event sensor in the time interval.

Characteristics and advantages of the invention will appear on reading the following description, given solely by way of non-limiting example, and made with reference to the appended drawings, in which:

- Figure 1 is a schematic view of an example of an observation system, - Figure 2 is a schematic representation of an example of physical implementation of an observation system according to Figure 1,

- Figure 3 is a schematic representation of another example of physical implementation of an observation system according to Figure 1,

- Figure 4 is a schematic representation of yet another example of physical implementation of an observation system according to Figure 1, and

- Figure 5 is a schematic view of another example of an observation system.

An observation system 10 is schematically illustrated in Figure 1.

The representation is schematic insofar as it is a representation of the functional type by block allowing a good understanding of the operation of the observation system 10.

The observation system 10 is suitable for observing an environment.

The observation system 10 comprises an event sensor 12, a measurement unit 14, a compensation device 16 and an evaluation unit 17.

The event sensor 12 is able to generate a stream of events by observing the environment in a time interval, called observation time interval.

The stream of events generated is generally a sparse stream.

As indicated above, the generated stream is asynchronous, which allows the event sensor 12 to operate at a high frequency.

More specifically, event sensor 12 comprises a set of pixels 20 arranged in a matrix of pixels 22, collection optics 23 and a reading system 24.

Each pixel 20 is capable of generating an event in the form of a pulse. Such a pulse is often usually referred to as a “spike” with reference to the corresponding English terminology.

To generate an event, each pixel 20 continuously measures the incident light intensity using a photodiode and compares at any time the relative difference between the light intensity I _curr measured at a time t and the light intensity l _prev measured at the immediately preceding instant at a contrast threshold C _th according to the following formula: icurr iprev „ j — th. iprev

When the previous condition is fulfilled, the pixel 20 generates a pulse.

Alternatively, other conditions may be used. For example, the condition is that the measured intensity is greater than or equal to a threshold or that the time taken to reach a predetermined intensity is less than or equal to a threshold.

Nevertheless, in each of the cases, the pulse generation only takes place if the condition is met to guarantee high-rate operation of the event sensor 12.

Such a pulse is often expressed according to the AER protocol. The acronym AER refers to the English name of “Address Event Representation” which literally means representation of the address of the event.

However, other representations such as analog representations (for example, by the emission of several pulses to encode information) are also possible.

The collection optics 23 makes it possible to collect the incident light and to guide it towards the matrix of pixels 22.

According to the example described, the collection optics 23 is a matrix of microlenses, each microlens of which is associated with a single pixel 20.

For example, each microlens of the collection optic 23 is a hypergone optic.

Such optics are more often called fisheye optics (which literally means fish eye) in reference to their very large field of vision.

This very large field implies that the 23 collection lens introduces a large distortion that should be compensated for.

Other geometric aberrations can also be introduced by the 23 collection optics such as vignetting.

The reading system 24 is an electronic circuit assembly generating information representing each initial event in the form of a first plurality of information fields in a first space.

With such a formalism, for the example described, the impulse is a triplet with three elements A1, A2 and A3.

The first information field A1 is the address of pixel 20 which generated the pulse.

The address of pixel 20 is, for example, encoded by giving the line number and the column number of the matrix of pixels 22 where pixel 20 is located.

As a variant, a coding of the type y*x _max + x or x*y _max + y can be used. In the previous formula, x designates the column number of pixel 20, y the row number of pixel 20, x _max the number of columns and y _max the number of rows of the pixel matrix 22. The second information field A2 is the instant of generation of the pulse by the pixel 20 which generated the pulse.

This implies that the event sensor 12 is able to timestamp the generation of pulses with good precision to facilitate subsequent processing of the generated event stream.

The third information field A3 is a value relating to the pulse.

In the following, as an example, the third information field A3 is the polarity of the pulse.

The polarity of a pulse is defined as the sign of the intensity gradient measured by pixel 20 at the instant of generation of the pulse.

According to other embodiments, the third information field A3 is the light intensity value at the instant of generation of the pulse, the depth observed if the purpose of the event sensor 12 is to measure the depth, or again the precise value of the measured intensity gradient.

Alternatively, the plurality of information fields in the first space includes only the first information field A1 and the second information field A2.

The reading system 24 is suitable for routing the flow of events generated to the compensation device 16. This routing is represented schematically by the arrow 26 in FIG.

Measurement unit 14 is a motion measurement unit.

The measurement unit 14 is suitable for measuring the movement of the event sensor 12.

According to the proposed example, the measurement unit 14 is an inertial unit.

Such an inertial unit is sometimes called an inertial unit or more often under the abbreviation IMU which refers to the English name of "Inertial Measurement Unit".

The measuring unit 14 thus comprises gyrometers 28 and accelerometers 30 making it possible to measure the movements in rotation and in translation of the event sensor 12.

Depending on the case, the output data of the movement measurement unit 14 can be raw or integrated data.

For example, the integrated data is expressed in the form of a rotation matrix R corresponding to the rotational movements of the event sensor 12 or a translation matrix T corresponding to the translation movements of the event sensor 12. As a variant, the data relating to the rotation are supplied using a quaternion which is generally a vector with four values, one value of which represents the norm, the other values being normalized and characterizing the rotation in space.

The compensation device 16 is a device suitable for compensating the movements of the event sensor 12 in the flow of events generated.

In this sense, the compensation device 16 is a device configured to implement a method of compensation for the motion of the event sensor 12 in the generated event stream.

The compensation device 16 of FIG. 1 comprises a compensation unit 34, a reconstruction unit 36 and a control unit 40.

The compensation unit 34 is a motion compensation unit of the event camera 12 in the generated event stream.

In this sense, the compensation unit 34 is configured to implement a step of the compensation method, namely a step of compensation for the movement of the event camera 12 in the generated event stream.

Therefore, the compensation unit 34 can be described as an EMC unit, the acronym EMC referring to the English name of "Ego-Motion Compensation" or "Ego-Motion Correction" which literally means compensation of the proper movement or correction of the proper movement.

The compensation unit 34 takes as input the stream of events generated, each event of which is a pulse characterized by a triplet (A1, A2, A3).

The compensation unit 34 is able to receive measurements of the movement of the event sensor 12 during the observation time interval.

More specifically, the compensation unit 34 receives the data relating to the movement of the event sensor 12 from the control unit 40 which will be described later.

The compensation unit 34 is also capable of applying a compensation technique to the event stream generated according to the data received to obtain a compensated event stream in the observation time interval.

The reconstruction unit 36 is a frame reconstruction unit.

The reconstruction unit 36 is able to generate corrected frames from the stream of events compensated in the time interval.

Such a reconstruction unit 36 can be called EFR unit, the acronym EFG referring to the English denomination of “Event-Frame Generation” which literally means generation of event frames. The reconstruction unit 36 takes as input the stream of compensated events coming from the compensation unit 34 as indicated by the arrow 44 in FIG. 1 and makes it possible to obtain corrected frames as output, as illustrated diagrammatically by the arrow 45 in figure 1.

According to the example described, the reconstruction unit 36 leaves the value of the third information field A3 unchanged for each event of the frame to be reconstructed.

As a variant, the reconstruction unit 36 is also capable of also modifying the value of the third information field A3 of each event of the frame to be reconstructed.

The corrected frames are sent to evaluation unit 17 as shown by arrow 46 and/or used as output for another application such as display.

The evaluation unit 17 is a unit for evaluating the position of the event sensor.

The evaluation unit 17 is able to receive measurements of the movement of the event sensor 12 as shown by the arrow 48

More specifically, the evaluation unit 17 receives the data relating to the movement of the event sensor 12 of the movement measurement unit 14 which are, in the example described, the rotation matrix R and the translation matrix T.

The evaluation unit 17 is also capable of applying an evaluation technique to the frames reconstructed according to the measurements received to obtain evaluated positions of the event sensor 12 in the time interval.

In one example, the evaluation technique is a visual inertial odometry technique.

The visual inertial odometry technique is more often referred to by the acronym VIO which refers to the name "Visual Inertial Odometry".

By definition, such a technique is a technique for merging data from a visual sensor and an inertial unit. Here, the technique is a technique of merging the data of the event sensor 12 and the measurement unit 14.

According to another example, the evaluation technique is a simultaneous localization and mapping technique.

The simultaneous localization and mapping technique is more often referred to by the acronym SLAM, which refers to the name "Simultaneous Localization and Mapping".

The SLAM technique is a VIO technique further comprising a step of mathematical optimization of all the positions evaluated over time. In each case, an evaluation of the position of the event camera 12 is obtained at the output of the evaluation unit 17 as shown by the arrow 50.

Control unit 40 is a control unit of compensation unit 34.

The control unit 40 is distinct from the evaluation unit 17.

The control unit 40 is able to control the compensation unit 34 by sending data relating to the movement of the event sensor 12.

More precisely, the control unit 40 is capable of modifying the output 44 of the compensation unit 34 by sending to the compensation unit 34 data relating to the movement of the event sensor 12 which are different.

This means that according to the data sent by the control unit 40, for the same stream of events generated, the compensation unit 34 will send to the reconstruction unit 36 a different stream of compensated events.

In the remainder of the description, to clearly distinguish these different outputs, the term "operating mode" of the compensation unit 34 will be used.

Each operating mode is associated with a type of data relating to the movement of the respective event sensor 12.

Data are of different types if their content is different or, if their content is identical, the associated expected precision is very different.

According to this terminology, the control unit 40 controls the mode of operation of the compensation unit 34 by transmitting the data relating to the movements.

According to the example of FIG. 1, the control unit 40 is able to send three different types of data relating to the movement of the event sensor 12, which means that the compensation unit 34 is able to operate according to three modes of operations.

Furthermore, the control unit 40 comprises several sub-units, namely a reception sub-unit 52, a processing sub-unit 54, a transmission sub-unit 56 and a determination sub-unit 58.

In the first mode of operation, the control unit 40 is able to send information on the absence of data relating to the movement of the event sensor 12.

This means that the transmission subunit 56 does not send any encrypted data relating to the movement of the event sensor 12.

In such a mode of operation, the compensation unit 34 outputs a stream of corrected events identical to the stream of events generated by the event sensor 12. The compensation unit 34 thus does not introduce any compensation into the generated event stream.

In the first operating mode, the compensation unit 34 is thus inhibited. The first mode of operation is thus an inhibition mode in the example of figure 1 .

In the second mode of operation, the control unit 40 is able to receive measurements of the movement of the event sensor 12.

In this case, the receiving subunit 52 is linked to the measuring unit 14 as shown by the arrow 63.

The processing subunit 54 is then able to implement a processing on the measurement data received to obtain processed data.

For example, the processing is a temporal integration of at least some of the measurements received.

By way of illustration, time integration is a Madgwick filter, for example applied to angular velocity. This is very effective when the movement of the event sensor 12 is mainly rotary.

The control unit 40 is then able to send the processed data to the compensation unit 34. More precisely, it is the transmission subunit 56 which sends the processed data to the compensation unit 34.

These processed data are the values of the rotation R and translation T matrices.

In this second case, the translation matrix T corresponds to the null matrix (in this case, the translation matrix T is a vector of 3 null values).

The second mode of operation and the first mode of operation therefore differ from each other by the nature of the data relating to the movement of the event sensor 12 provided. Indeed, in one case, no data is transmitted while in the other case, these are measurements (processed here) coming from the measurement unit 14.

The operation of the compensation unit 34 in the second mode is thus an operation involving the application of the compensation technique and therefore a modification of the flow of events generated.

According to the example in Figure 1, the compensation technique includes an operation to cancel the distortion introduced by the collection optics 23 followed by an operation to compensate for the movement of the event sensor 12.

During the cancellation operation, the first information field A2 relating to the position of a pixel is modified by taking the distortion into account. It should be noted that the cancellation operation can be replaced or supplemented by an operation of partial compensation of the optical aberrations introduced by the collection optics 23.

The compensation operation corrects the position of the pulses corrected by the cancellation operation according to the movements of the event sensor 12.

The compensation operation makes it possible to reduce the number of pulses emitted to a minimum.

Indeed, with the movement of the event sensor 12, different pixels 20 generate pulses even in the presence of a stationary object. The compensation operation ensures that these different pulses are not repeated and are assigned to the same 20 pixel (or failing that to the same set of 20 pixels if the object is extended).

Thus, the quantity of pulses emitted by the event sensor 12 is greatly reduced thanks to the compensation unit 34.

For example, the motion compensation operation of the event sensor 12 includes the implementation of two successive sub-operations for each pulse.

During the first sub-operation, the value of the rotation matrix R and of the translation matrix T at the instant of generation of the pulse is determined. Such a determination is, for example, implemented by an interpolation, in particular between the rotation matrices R and the translation matrices T closest to the moment of generation of the pulse.

The second sub-operation then consists in multiplying the coordinates obtained at the output of the first operation with the rotation matrix R and then adding the translation matrix T to obtain the coordinates of the pulse after taking into account the proper motion of the event sensor 12.

According to another embodiment, the compensation technique is a machine learning algorithm.

Such an algorithm is more often referred to as "machine learning" because of the corresponding English terminology.

For example, the algorithm is a neural network.

In each case, a stream of events compensated with the measurements coming from the measurement unit 14 is obtained.

In the third mode of operation, the control unit 40 is able to receive the positions evaluated from the event sensor 12. These evaluated positions come from the evaluation unit 17 as shown by the arrow 68. In this case, the receiving subunit 52 is linked to the evaluation unit 17.

The control unit 40 is then able to send the positions evaluated to the compensation unit 34. More precisely, it is the transmission sub-unit 56 which sends the positions evaluated to the compensation unit 34.

As in the case of the second mode of operation, the third mode of operation and the first mode of operation therefore differ from each other in the nature of the data relating to the movement of the sensor of the event sensor 12 supplied. Indeed, in one case, no data is transmitted while in the other case, these are evaluated positions coming from the evaluation unit 17.

However, even if the content of the data sent in the second operating mode and the third operating mode are identical (it is indeed data relating to the movement of the event sensor), the associated expected precision is very different, which justifies that their type differs. Indeed, the data sent in the second mode do not take into account the flow of events generated by the event sensor 12, unlike the data sent in the third mode which are obtained by the evaluation unit 17 from the reconstructed frames. , itself obtained using the compensated event stream.

According to the example described, it is the determination subunit 58 which is responsible for determining the data relating to the movement of the event sensor 12 to be sent, and thereby the mode of operation of the compensation unit 34.

The determination subunit 58 determines the data to be sent based on the data received by the control unit 40.

For example, the determination subunit 58 sends to the compensation unit 34 the data relating to the movement of the event sensor 12 which is the most precise.

Thus, in the absence of data received by the reception sub-unit 52, the determination sub-unit 58 imposes an operation of the compensation unit 34 according to the first mode of operation.

If the receiving subunit 52 only receives data corresponding to the second mode of operation or the third mode of operation, the determination subunit 58 chooses to send the received data which is more precise than no data and therefore d impose an operation of the compensation unit 34 according to the mode of operation of the data received by the reception sub-unit 52.

When the receiving subunit 52 receives both data corresponding to the second mode of operation and data corresponding to the third mode of operation, the determination subunit 58 chooses to send the data corresponding to the third mode of operation which are normally the most accurate. The determination sub-unit 58 thus imposes an operation of the compensation unit 34 according to the third mode of operation.

Of course, the example that has just been described can be generalized to any number of data types. In particular, it is particularly advantageous for the determination sub-unit 58 to be able to operate according to five distinct operating modes, namely no data, data resulting from integration in rotation only of the data from the measurement unit 14, data resulting from a “total” integration (rotation and translation) of the data of the measuring unit 14, of the data resulting from an application of a VIO technique and of the data resulting from an application of a SLAM technique.

It can now be noted that the origin of these data is irrelevant for the determination sub-unit 58, it being able to come from the processing sub-unit 54 as well as from the evaluation unit 17 or any other unit in communication with the control unit 40.

It emerges from the preceding description that the determination sub-unit 58 can be seen as a data control sub-unit relating to the movement of the event sensor 12.

The determination sub-unit 58 thus confers on the control unit 40 a function of switching between the different possible operating modes of the compensation unit 34.

The control unit 40 makes it possible to elegantly solve the problem of initializing the compensation device 16.

Indeed, to operate the compensation unit 34 needs the output of the evaluation unit 17 which itself needs the output of the compensation unit 34 which is used by the reconstruction unit 36 .

Control unit 40 effectively initializes compensation device 16 by allowing evaluation unit 17 to operate without motion compensation.

The compensation device 16 is thus compatible with any type of evaluation technique, including evaluation techniques having no initialization protocol and even non-inertial visual odometry techniques. Non-inertial visual odometry techniques are more often referred to by the acronym VO referring to the corresponding English name of “Visual Odometry”.

As explained previously, the control unit 40 also makes it possible to impose an operating mode adapted to the data relating to the movement which is available. Furthermore, the addition of the control unit 40 performs very little calculation to implement such adaptability.

The control unit 40 thus has very little impact on the weight, size and energy power of the compensation device 16.

The observation system 10, and in particular the compensation device 16, is thus able to implement a method making it possible to make any spatial localization algorithm compatible with an event sensor 12.

In addition, the method that the observation system 10 is capable of implementing also makes it possible to adapt the processing to the scene so as to improve the computational efficiency and/or the processing quality.

Thanks to the aforementioned advantages, such an observation system 10 is compatible with an on-board physical implementation.

An example of such an implementation is now described with reference to Figure 2.

In the example illustrated, the observation system 10 is a stack 78 of two layers 80 and 82 along a stacking direction.

The first layer 80 and the second layer 82 are superimposed.

The event sensor 12 is fabricated in the first layer 80.

For this, a BSI technique is, for example, used.

The acronym BSI refers to the English terminology of "Backside Illumination" literally meaning illumination from behind and designates a technique for manufacturing a sensor in which the photodiodes of the pixels 20 are positioned in direct contact with the collection optics 23 .

In the second layer 82, the compensation device 16 is made under the matrix of pixels 22.

This makes it possible to limit the reading system 24 to simple connections since parallel access to each pixel 20 is permitted.

The second layer 82 is connected to the first layer 80 by three-dimensional copper-copper bonds 84. Such a type of bond 84 is more often referred to as 3D bonding with reference to the corresponding English terminology.

When it is not possible to physically implement the control unit 40 and the compensation unit 34, the reconstruction unit 36 and the evaluation unit 17 on the same layer 82 for reasons of bulk, a third layer 86 is used.

The third layer 86 is part of the stack 78 and is superimposed with the first layer 80 and the second layer 82. In such a configuration illustrated schematically in Figure 3, the second layer 82 comprises the control unit 40 while the third layer 86 comprises the compensation unit 34, the reconstruction unit 36 and the evaluation unit 17 .

To ensure communication between the second layer 82 and the third layer 86, the second layer 82 is provided with through vias 88.

A through-via 88 is more often designated according to the English terminology of "through-silicon via" and designates an electrical contact extending along the stacking direction and emerging, that is to say extending from one side of the second layer 82 to the other side of the second layer 82.

Such an implementation allows parallel type communication between the second layer 82 and the third layer 86.

Alternatively, as seen in Figure 4, the communication between the second layer 82 and the third layer 86 is ensured by a serial interconnection 90 involving the use of a data serialization unit (not shown in Figure 4) at the output of the control unit 40.

Such an implementation is indicated when the use of through vias 88 prevents the physical implementation of the control unit 40. Indeed, each through via 88 reduces the useful space, that is to say the space in which the control unit 40 can be manufactured, which may make it impossible to physically implement the control unit 40 for lack of space. In the implementation with a 90 series interconnection, the useful space is on the contrary very little reduced as illustrated by the comparison between figures 3 and 4.

In each of the cases proposed in FIGS. 2 to 4, the event sensor 12 and the compensation device 16 form part of the same stack 78 of at least two layers 80, 82 and 86, the first layer 80 of the stack 78 comprising the event sensor 12, the at least one other layer 82 and possibly 86 of the stack 78 comprising the projection unit 34 and the compensation unit 34).

The observation system 10 thus physically implemented has the advantage of being a compact on-board system.

Other embodiments of the observation system 10 are still possible.

For example, the compensation unit 34 is made on another component, the component and the compensation unit 34 being attached to a substrate comprising electrical connections.

According to one embodiment, the substrate is an interposer.

As a variant or in addition, the observation system 10 comprises additional filtering which is implemented at the level of the event sensor 12. The filtering is, for example, a filtering by group of pixels (typically 4). When a single pixel of a group of pixels generates an event without correlation with its neighbors, this event is considered as noise and therefore eliminated.

To improve such filtering, the group of pixels can, in certain cases, be programmable according to rules.

According to another embodiment, the flow of events is represented not in the form of a non-continuous and asynchronous flow of pulses but as a succession of sparse matrices, that is to say mainly empty matrices.

According to a variant or in addition, the observation system 10 further comprises a unit for obtaining.

The obtaining unit is suitable for determining at least one characteristic relating to the nature of the movement of the object.

In particular, the obtaining unit can determine whether the nature of the movement is mainly rotational.

In such a case, the determination sub-unit 58 is able to send the data coming from the processing sub-unit 54 and not the data coming from the evaluation unit 17.

With reference to FIG. 5 described below, it is also possible to implement this sending by controlling the operation of the evaluation unit 17.

In a way, this amounts to inhibiting the evaluation unit 17, its calculation precision not being useful to compensate for the movement of the event sensor 12.

As a variant, the nature can also indicate the speed of the movement of the event sensor 12, for example, a slow speed (less than or equal to a first threshold) and a fast speed (greater than or equal to a second threshold, the second threshold being higher or equal to the first threshold).

The obtaining unit is linked with the receiving subunit 52 of the control unit 40 so that the receiving subunit 52 receives the data relating to the nature of the movement.

The determination subunit 58 then determines the data to be sent according to the data received by the control unit 40 and also according to the nature of the movement.

This makes it possible to obtain a compensated event stream in which the movement of the event sensor 12 is better compensated during the observation time interval.

It should be noted that nothing prevents the determination unit 90 and a modification unit from being physically implemented close to the event sensor 12, in particular in the third layer 86. Another embodiment is presented with reference to Figure 5.

In this embodiment, the processing subunit 54 is part of the evaluation unit 17 which further comprises an evaluation subunit 100 capable of implementing a VIO or SLAM technique.

The determination sub-unit 58 is then also able to control the mode of operation of the evaluation unit 17 which, depending on the case, will supply data coming from the processing sub-unit 54 or from the evaluation sub-unit 100.

Such a control is materialized in Figure 5 by the arrow 102.

This embodiment corresponds to a case where the data relating to the movement of the event sensor 12 all come from an element external to the control unit 40.

Such an embodiment is in particular advantageous for benefiting from the unit capable of supplying the best data to the compensation device 16 and thereby improving the precision of the compensation.

According to other embodiments corresponding in particular to applications in which the hardware implementation is less constrained, the physical implementation of the compensation device 16 is, for example, a computer implementation.

By way of non-limiting example, an example of such an implementation is now described with reference to a computer.

The interaction of a computer program product with a computer makes it possible to implement the compensation process. The evaluation process is thus a computer-implemented process.

More generally, the computer is an electronic computer suitable for manipulating and/or transforming data represented as electronic or physical quantities in registers of the computer and/or memories into other similar data corresponding to physical data in memories. , registers or other types of display, transmission or storage devices.

It should be noted that, in the present description, the expression “specific to” means “suitable for”, “suitable for” or “configured for” without distinction.

The computer has a processor comprising a data processing unit, memories and an information carrier drive. The computer alternatively and additionally includes a keyboard and a display unit.

The computer program product comprises a readable carrier of information.

A readable information carrier is a carrier readable by the computer, usually by the reader. The readable information carrier is a medium suitable for storing electronic instructions and capable of being coupled to a bus of a computer system. By way of example, the readable information medium is a floppy disk or floppy disk, an optical disk, a CD-ROM, a magneto-optical disk, a ROM memory, a RAM memory, an EPROM memory, an EEPROM memory, a magnetic card or an optical card.

On the readable information carrier is stored a computer program comprising program instructions.

The computer program can be loaded onto the data processing unit and is suitable for driving the implementation of the observation method.

In each of the embodiments presented previously and which can be combined with each other to form new embodiments when technically possible, there is proposed a device or a method making it possible to compensate for the movement of the event sensor in a flow of events generated during an observation of an environment which reduces the computing capacity required to allow a physical implementation in an embedded system while retaining the useful information captured by the event sensor 12.

Such a device or method is therefore particularly suitable for any application linked to on-board vision. Among these applications, non-exhaustively, mention may be made of surveillance, augmented reality, virtual reality or vision systems for autonomous vehicles or drones.

It should be noted that such a device can also be used not to obtain compensated frames but to evaluate the position of the event sensor 12.

Indeed, for multiple applications and in particular in the field of autonomous vehicles (car, drone or others), knowing the position of the vehicle is crucial.

Such knowledge is to be obtained under several constraints: good precision (at least centimetric, even millimetric), good speed and independence from external conditions and in particular light conditions.

For this, it is known to implement a sensor fusion technique. One such technique is to collect data from several different sensors of the same vehicle and combine them to obtain the position of the vehicle.

Such fusion techniques are used in particular using data from a camera, an inertial unit and possibly a radar or a lidar.

However, none of the aforementioned sensors makes it possible to comply with the three aforementioned constraints. In particular, a camera is not independent of the external light conditions and the other sensors do not make it possible to obtain centimetric precision at all times. As a result, after implementation of the merging technique, the estimated position does not comply at best with at least one of the aforementioned conditions. The evaluation system described makes it possible to remedy all of these problems since it makes it possible to quickly obtain good precision of the position of the sensor and this, in all possible light conditions.

Claims

22 CLAIMS

1. Compensation device (16) for the movement of an event sensor (12) in a stream of events generated by the event sensor (12) during an observation of an environment in a time interval, the device for compensation (16) comprising:

- a compensation unit (34), the compensation unit (34) being capable of receiving data relating to the movement of the event sensor (12) during the time interval and capable of applying a compensation technique to the flow of events generated by the event sensor (12) according to the received data to obtain a stream of events compensated in the time interval, and

- a control unit (40) of the compensation unit (34), the control unit (40) being able to control the compensation unit (34) by sending data relating to the movement of the event sensor (12 ).

2. Compensation device according to claim 1, in which the control unit (40) is capable of sending to the compensation unit (34) information on the absence of data available as data relating to the movement of the sensor. (12), the compensation unit (34) delivering a stream of compensated events identical to the stream of events generated by the event sensor (12).

3. Compensation device according to claim 1 or 2, in which the control unit (40) comprises a processing subunit (54), the control unit (40) being able to receive measurements of the movement of the event sensor (12), the processing sub-unit (54) being able to implement a processing on the measurement data received to obtain processed data, the processing being, in particular, a temporal integration of at least some of the measurements received, the control unit (40) also being able to send the processed data to the compensation unit (34).

4. Compensation device according to any one of claims 1 to 3, in which the control unit (40) is suitable for receiving positions evaluated from the event sensor (12), the control unit (40) being suitable sending the evaluated positions to the compensation unit (34).

5. Compensation device according to any one of claims 1 to 4, wherein the control unit (40) further comprises a determination subunit (58) capable of determining data relating to the movement of the sensor (12) to be sent, the determination subunit (58) determining the data to be sent according to the data received by the control unit (40).

6. Observation system (10) of an environment, the observation system (10) comprising:

- an event sensor (12) generating a stream of events during an observation of an environment of the event sensor (12),

- a measuring unit (14) able to measure the movement of the event sensor (12) during a time interval, and

- a compensation device (16) according to any one of claims 1 to 5.

7. System for evaluating the position of an event sensor (12), the evaluation system comprising:

- a compensation device (16) according to any one of claims 1 to 5.

8. System according to claim 6 or 7, in which the system (10) further comprises a unit for evaluating (17) the position of the event sensor (12) and a reconstruction unit (36) of frames, the reconstruction unit (36) being adapted to generate corrected frames from the stream of events compensated in the time interval, the control unit (40) being distinct from the evaluation unit (17) , the evaluation unit (17) being adapted to receive measurements of the movement of the event sensor (12), the evaluation unit (17) being adapted to apply an evaluation technique to the frames reconstructed according to the measurements received to obtain estimated positions of the event sensor (12) in the time interval.

9. The system of claim 8, wherein the evaluation technique is a visual odometry technique or a simultaneous location and mapping technique.

10. System according to claim 8 or 9, in which the event sensor (12) and the compensation device (16) are part of the same component comprising a stack (78) of at least three layers (80, 82, 86), the first layer (80) of the stack (78) comprising the event sensor (12), the second layer (82) of the stack (78) comprising the control unit (40) and the third layer (86) comprising the compensation unit (34) and the evaluation unit (17).

11. System according to any one of claims 7 to 10, the system (10) further comprising an obtaining unit, the obtaining unit being capable of obtaining, for each event of the flow of events generated , at least one characteristic relating to the nature of the movement of an object associated with the event, the object being the object of the environment having caused the generation of the event by the event sensor (12), the device compensation (16) also comprising a determination sub-unit (58) capable of determining data relating to the movement of the event sensor (12) to be sent, the determination sub-unit (58) determining the data to be sent as a function of the data received by the control unit (40) and also depending on the at least one characteristic relating to the nature of the movement.

12. Evaluation system according to any one of claims 8 to 11, wherein the compensation unit (34) is made on another component, the component and the compensation unit (34) being attached to a substrate comprising electrical connections, the substrate being, for example, an interposer.

13. Evaluation system according to any one of claims 8 to 11, in which the compensation unit (34), the reconstruction unit (36), the evaluation unit (17) and the control (40) are made on the same integrated circuit.

14. A method of compensating for the movement of an event sensor (12) in a stream of events generated by the event sensor (12) during an observation of an environment in a time interval, the compensation method being put implemented by a compensation device (16), the compensation device (16) comprising a compensation unit (34) and a control unit (40) of the compensation unit (34), the compensation method comprising a step of:

- sending of data relating to the movement of the event sensor (12) by the control unit (40) to the compensation unit (34), 25

- reception of data relating to the movement of the event sensor (12) by the compensation unit (34),

- application by the compensation unit (34) of a compensation technique to the stream of events generated by the event sensor (12) according to the data received to obtain a stream of events compensated in the time interval, and

- generation by the reconstruction unit (36) of corrected frames from the stream of events compensated in the time interval.

15. A method of evaluating the position of an event sensor (12), the event sensor (12) generating a stream of events in a time interval, the evaluation method being implemented by a system of position evaluation of an event sensor (12), the evaluation system comprising a compensation unit (34), a frame reconstruction unit (36), a sensor position evaluation unit (17) (12) and a control unit (40) of the compensation unit (34), the control unit (40) being separate from the evaluation unit (17), the evaluation method comprising a step of:

- sending of data relating to the movement of the event sensor (12) by the control unit (40) to the compensation unit (34),

- application by the compensation unit (34) of a compensation technique to the stream of events generated by the event sensor (12) according to the data received to obtain a stream of events compensated in the time interval,

- generation by the reconstruction unit (36) of corrected frames from the stream of events compensated in the time interval,

- reception by the evaluation unit (17) of measurements of the movement of the event sensor (12), and

- application by the evaluation unit (17) of an evaluation technique to the reconstructed frames according to the measurements received to obtain evaluated positions of the event sensor (12) in the time interval.