EP4268445A1

EP4268445A1 - Apparatus, dynamic vision sensor, computer program, and method for processing measurement data of a dynamic vision sensor

Info

Publication number: EP4268445A1
Application number: EP21839590.3A
Authority: EP
Inventors: Diederik Paul MOEYS
Original assignee: Sony Europe BV United Kingdom Branch; Sony Semiconductor Solutions Corp
Current assignee: Sony Europe BV United Kingdom Branch; Sony Semiconductor Solutions Corp
Priority date: 2020-12-22
Filing date: 2021-12-22
Publication date: 2023-11-01
Also published as: WO2022136514A1

Abstract

Embodiments of the present disclosure relate to an apparatus, a dynamic vision sensor, a computer program, and a method for processing measurement data of a dynamic vision sensor. The method comprises receiving measurement data of the DVS. The measurement data is indicative of a plurality of events detected by pixels of the DVS and of a plurality of timestamps indicative of recording times at which the events were successively recorded by a read out circuit of the DVS after detection by the pixels. The timestamps of the events differ by a delay caused by recording the events successively. The method also comprises determining simultaneously detected events from the plurality of events based on deviations of the timestamps to each other. Further, the method comprises approximating timestamps of simultaneously detected events for compensating at least partly the delay between the simultaneously detected events.

Description

Apparatus, dynamic vision sensor, computer program, and method for processing measurement data of a dynamic vision sensor

Field

Embodiments of the present disclosure relate to an apparatus, a dynamic vision sensor, a computer program, and a method for processing measurement data of a dynamic vision sensor. In particular but not exclusively, the present disclosure relates to a concept for compensating at least partly a delay between detecting and recording events in dynamic vision sensors.

Background

Dynamic vision sensors (DVS) are used in various technical applications, e.g., surveillance and automotive applications.

According to its basic principle, a DVS detects so-called “events” when an illumination and, thus, a light stimulus of a pixel of the DVS changes. In response to such events, pixels then request to be read out/recorded. For multiple events detected simultaneously or shortly after each other, the events are queued to be successively read out/recorded and assigned to timestamps indicative of a time when a respective event is recorded. Thus, a (variable) delay (“jitter”) occurs between a moment/time when an event is detected (“true event time”) and when a respective timestamp indicative of a time when the event is recorded (“recording time”) is assigned to this event. This delay, e.g., depends on how many events are queued and may impair a temporal resolution of the events. Thus, the temporal resolution may be insufficient for some applications (e.g. in high-speed imaging devices/high-speed cameras)

Hence, there may be a demand for an improved concept for dynamic vision sensors.

Summary

This demand may be satisfied by the appended independent and dependent claims. Embodiments of the present disclosure provide a method for processing measurement data of a dynamic vision sensor (DVS). The method comprises receiving measurement data of the DVS. The measurement data is indicative of a plurality of events detected by pixels of the DVS and of a plurality of timestamps indicative of recording times at which the events were successively recorded by a read out circuit of the DVS after detection by the pixels. The timestamps of the events differ by a delay caused by recording the events successively. The method also comprises determining simultaneously detected events from the plurality of events based on deviations of the timestamps to each other. Further, the method comprises approximating timestamps of simultaneously detected events for compensating at least partly the delay between the simultaneously detected events.

The DVS, e.g., comprises a pixel array which comprises a plurality of pixels which detected the so-called “events” in response to changes in their illumination. It is noted that, the DVS can be understood as any sensor for event-based imaging, in the sense of an (spatially resolving) imaging concept which provides for imaging changes in the illumination. The DVS may be also referred to as silicon retina or event (-based) vision sensor (EVS). The DVS may be comprised of or correspond to an event camera or neuromorphic camera.

After the events have been detected, they may be signaled to the read out circuit to be recorded. In practice, movements of an object in a field of view (FOV) of the DVS lead to a plurality of events detected (in theory) contemporaneously or at least within a time interval below a temporal resolution of the read out circuit, i.e. a processing time which the read out circuit takes to record one event. Thus, such events are queued to be successively recorded by the read out circuit. For a skilled person, such events are therefore deemed to be detected simultaneously (disregarding deviations below the temporal resolution of the read out circuit). Also, such events may be deemed to be detected “together” (in time). Due to its technical design, the read out circuit may record the simultaneously detected events one after another and assign the timestamps indicative of their recording times (i.e. times when the events are recorded). Therefore, also timestamps of simultaneously detected events differ by (variable) delays through recording the events one after another.

In particular, timestamps of simultaneously events may differ less than successively detected events (i.e. events which are detected within a time interval equal or larger than the temporal resolution of the read out circuit). Thus, the deviations of the timestamps indicate whether and/or which of the events were detected simultaneously. Determining simultaneously detected events may comprise determining one or more groups of simultaneously detected events. In particular, the one or more groups may include two or more events. For approximating timestamps of the simultaneously detected events, e.g., one or more of those timestamps are modified and, thereby, approximated to one or more of other timestamps of simultaneously detected events. In this way, delays caused in recording the events are at least partly compensated for an enhanced temporal resolution of the events. Preferably, the timestamps of the simultaneously detected events may be aligned with each other for a further enhanced temporal resolution.

The measurement data may be further indicative of positions of the pixels. In this case, determining simultaneously detected events may comprise determining simultaneously detected events from the plurality of events based on the positions. In practice, simultaneously detected events resulting from movements of an object occur in practice (spatially) close to each other, e.g. at neighboring pixels. Hence, the simultaneously detected events may be distinguished more reliably from other (e.g. noise events and/or successively detected) events based on the positions.

In some embodiments, determining the simultaneously detected events based on the deviations of the timestamps and positions of the events comprises detecting an edge including the simultaneously detected events in a representation of the events over time and position based on the timestamps of the events and the positions in favor of an enhanced efficiency and speed in determining the simultaneously detected events.

Further embodiments provide a computer program comprising instructions, which, when the computer program is executed by a processor, cause the processor to carry out the method proposed herein.

Further embodiments provide an apparatus for processing measurement data of a dynamic vision sensor (DVS). The apparatus comprises one or more interfaces for communication and a data processing circuit configured to control the one or more interfaces and to execute one of the methods proposed herein. Some embodiments provide a dynamic vision sensor comprising the apparatus proposed herein.

Brief description of the Figures

Some examples of apparatuses and/or methods will be described in the following by way of example only, and with reference to the accompanying figures, in which

Fig. 1 illustrates a flow chart schematically illustrating an embodiment of a method for processing measurement data of a dynamic vision sensor;

Fig. 2 schematically illustrates a block diagram schematically illustrating an embodiment of an apparatus for processing measurement data of a dynamic vision sensor; and

Fig. 3 schematically illustrates an application of the proposed concept.

Detailed Description

Some examples are now described in more detail with reference to the enclosed figures. However, other possible examples are not limited to the features of these embodiments described in detail. Other examples may include modifications of the features as well as equivalents and alternatives to the features. Furthermore, the terminology used herein to describe certain examples should not be restrictive of further possible examples.

Throughout the description of the figures same or similar reference numerals refer to same or similar elements and/or features, which may be identical or implemented in a modified form while providing the same or a similar function. The thickness of lines, layers and/or areas in the figures may also be exaggerated for clarification.

When two elements A and B are combined using an 'or', this is to be understood as disclosing all possible combinations, i.e. only A, only B as well as A and B, unless expressly defined otherwise in the individual case. As an alternative wording for the same combinations, "at least one of A and B" or "A and/or B" may be used. This applies equivalently to combinations of more than two elements. If a singular form, such as “a”, “an” and “the” is used and the use of only a single element is not defined as mandatory either explicitly or implicitly, further examples may also use several elements to implement the same function. If a function is described below as implemented using multiple elements, further examples may implement the same function using a single element or a single processing entity. It is further understood that the terms "include", "including", "comprise" and/or "comprising", when used, describe the presence of the specified features, integers, steps, operations, processes, elements, components and/or a group thereof, but do not exclude the presence or addition of one or more other features, integers, steps, operations, processes, elements, components and/or a group thereof.

Fig. 1 illustrates a flow chart schematically illustrating an embodiment of a method 100 for processing measurement data of a dynamic vision sensor (DVS).

As can be seen from the flow chart, method 100 comprises receiving 110 the measurement data of the DVS. The measurement data is indicative of a plurality of events detected by pixels of the DVS. Further, the measurement data is indicative of a plurality of timestamps indicative of recording times at which the events were successively recorded by a read out circuit of the DVS after detection by the pixels. The measurement data, e.g., comprises for the plurality of events a respective record including their timestamp which indicates their recording. As mentioned above, the timestamps in practice differ from each other by a delay caused by recording the events successively.

Also, method 100 comprises determining 120 simultaneously detected events from the plurality of events based on deviations of the timestamps to each other. In order to do so, clusters or groups of events (in the time domain) may be identified whose timestamps have a smaller deviation from each other than from events outside a respective group. For this, cluster analysis may be applied to the records and/or timestamps. In some applications, the cluster analysis provides for edge detection in a representation of the events over time and space, as laid out in more detail later.

Optionally, the simultaneously detected events are determined based on a threshold comparison applied to deviations of their timestamps. For example, events having timestamps with deviations lower than or equal to a predefined threshold are considered as simultaneously detected and events having timestamps with deviations above the predefined threshold are deemed to be successively detected.

It is noted that also other techniques may be used to determine simultaneously detected events based on deviations of their timestamps to each other.

Further, method 100 comprises approximating 130 timestamps of simultaneously detected events for compensating at least partly the delay between the simultaneously detected events. For this, e.g., one or more of the timestamps of the simultaneously detected events are modified such that their deviation to timestamps of one or more other simultaneously detected events decreases. In this way, delays resulting from successively recording the events successively and worsening the temporal resolution of the events may be compensated at least partially.

In some embodiments, approximating 130 the timestamps comprises aligning one or more timestamps of the simultaneously detected events. For example, one or more timestamps of the simultaneously detected events (e.g., simultaneously detected events of a respective clus- ter/group) are aligned to a predefined time or an average of their timestamps. In some embodiments, all timestamps of the simultaneously detected events, e.g. of a respective cluster/group of simultaneously detected events, are aligned.

Optionally, approximating 130 the timestamps comprises aligning one or more timestamps of the simultaneously detected events to the latest of the timestamps (i.e. the “youngest” timestamp indicative of the latest recording time) of the simultaneously detected events. For example, one, multiple, or all timestamps of simultaneously detected events of a respective cluster/group are aligned. This principle can be also applied to multiple clusters/groups of simultaneously detected events such that in multiple or each cluster/group the timestamps of simultaneously detected events of this cluster/group are aligned to the latest timestamp of the simultaneously detected events of this cluster/group.

In addition to the timestamps, the measurement data may be further indicative of positions of the pixels. The records of the events, e.g., comprise the positions or at least a pixel addresses indicative of the position of the pixels. In particular, the positions may be positions of the pixels in a pixel array of the DVS. In practice, simultaneously detected events are detected by neighboring pixels and/or pixels which are close together. Therefore, the positions can indicate whether events where detected simultaneously.

For the case that the measurement data is further indicative of the positions of the pixels, determining simultaneously detected events may comprise determining simultaneously detected events from the plurality of events based on the positions and on the deviations of the timestamps to each other. In addition to the timestamps, the positions are an additional indicator for simultaneously detected events and, thus, allow a more reliable determination of simultaneously detected events.

Also the positions allow to differentiate between clusters/groups of true/actual simultaneously detected events in the space domain, which result from incidents (e.g. movements of an object) in the environment, and noise events outside such clusters/groups in the space domain, i.e. noise events which occur at the pixel array and outside such clusters/groups. Such noise events, e.g., result from thermal leakage and/or parasitic photocurrents. Also, this allows to differentiate between clusters/groups of simultaneously detected events in the space domain which, e.g., result from movements of different objects, and to separately approximate timestamps of simultaneously detected events of each cluster/group for compensating at least partly the delay between the simultaneously detected events of the clusters/groups. Thus, separate incidents (e.g. movements of different objects) in the environment can be resolved in time, i.e. in temporal terms.

Also, the measurement data may be further indicative of polarities of the events. In context of the present disclosure, the polarity of an event denotes whether an increase or decrease in illumination caused this event. Events resulting from an increase in the illumination can be referred to as “ON-events” and events resulting from a decrease in the illumination as “OFF- events”.

For the case that the measurement data is indicative of the polarities of the events, determining simultaneously detected events may comprise determining simultaneously detected events based on the polarities. In practice, separate incidents in the environment may cause events having different polarities. E.g. a first incident causes a number of ON-events and a second incident a number of OFF-events. Accordingly, determining simultaneously detected events based on the polarities may allow to differentiate between a first cluster/group of simultaneously detected ON-events resulting from the first incident in the environment and a second cluster/group of simultaneously detected OFF-events resulting from the second incident in the environment. Further, this allows to separately approximate timestamps of simultaneously detected events of the separate clusters/group for compensating at least partly the delay between the simultaneously detected events of the clusters/groups. Thus, separate incidents (e.g. movements of different objects) in the environment can be resolved in time, i.e. in temporal terms.

The skilled person will appreciate that for the case that the measurement data is indicative of the positions and the polarities, the simultaneously detected events can be determined based on the positions and on the polarities for a more accurate, reliable and/or more trustworthy differentiation of clusters/groups of simultaneously detected events resulting from different incidents in the environment.

For the case that the measurement data is indicative of positions of the pixels, determining 120 simultaneously detected events may comprise obtaining a representation of the events over time and position based on the timestamps of the events and positions of the pixels. The representation, e.g., is a coded or visual representation (e.g. a diagram) which represents the events over time and position and in accordance with their timestamps and positions of the pixels which detected the events.

Also, determining 120 simultaneously detected events may comprise detecting in the representation at least one edge including the simultaneously detected events based on the on deviations of the timestamps to each other and the positions. In order to do so, edge detection may be applied. For example, the representation may be used as input of an edge detection framework or edge detection algorithm (e.g. Gabor filter kernel, Canny edge detector, Deriche edge detector, etc.).

In some applications of the proposed concept, the measurement data is indicative of a plurality of first events detected by pixels of the DVS and of a plurality of first timestamps indicative of recording times at which the first events were successively recorded by the read out circuit, and of a plurality of subsequent second events detected by one or more of the pixels and of a plurality of second timestamps indicative of recording times at which the second events were successively recorded by the read out circuit. For example, one or more of the pixels which detected the first events also detected subsequent second events after they detected the first events.

Also, determining the simultaneously detected events may comprise determining simultaneously detected events from the first and the second events based on the first and the second timestamps.

For example, first events comprise event A, event B and event C. Event A is detected by a first pixel, event B is detected by a second pixel, and event C by a third pixel. The first events A, B, and C may be assigned to first timestamps TA, TB and TC, respectively. The second events, e.g., comprise an event D detected by the first pixel after event A, and an event E detected by the second pixel after event B. The second events D and E may be assigned to second timestamps TD and TE, respectively. Events, which are detected simultaneously with event A or B, are in practice recorded with less delay to TA and TB than the events D and E, respectively. Therefore, time intervals between TA and TD, and between TB and TE can be understood as a maximum delay (“maximum jitter”) in timestamps of events detected simultaneously with event A and B, respectively. The maximum delay for event A and B may be different. Regarding the present example, e.g., it can be excluded that event C was detected simultaneously with event A if timestamp TC differs by more than the maximum delay from TA, e.g., if TC is more recent than TD, i.e. indicative of a time after TD. Otherwise, if TC differs less than the maximum delay from TA, event C may be considered as detected simultaneously with event A. This principle optionally is applied to multiple and/or other events. For example, maximum delays are specified for multiple events and only those events may be considered as detected simultaneously whose timestamps differ less than by their maximum delays.

As described below with reference to Fig. 2, the concept proposed herein may be also implemented in an apparatus.

Fig. 2 schematically illustrates a block diagram schematically illustrating an embodiment of an apparatus 200 for processing measurement data of a dynamic vision sensor (DVS) 2000.

As indicated by dashed lines in Fig. 2, the apparatus 200 may be implemented in the DVS 2000. Optionally, the apparatus 200 is implemented separated from the DVS 2000. As can be seen from the block diagram, the apparatus comprises one or more interfaces 212 for communication and a data processing circuit 214 configured to control the one or more interfaces and to execute one of the methods proposed herein.

In embodiments, the one or more interfaces 212 may comprise or correspond to any means for obtaining, receiving, transmitting or providing analog or digital signals or information, e.g. any connector, contact, pin, register, input port, output port, conductor, lane, etc. which allows providing or obtaining a signal or information. An interface may be wireless or wire- line and it may be configured to communicate, i.e. transmit or receive signals, information with further internal or external components. The one or more interfaces 212 may include transceiver (transmitter and/or receiver) components, such as one or more Low-Noise Amplifiers (LNAs), one or more Power-Amplifiers (PAs), one or more duplexers, one or more diplexers, one or more filters or filter circuitry, one or more converters, one or more mixers, etc. In some embodiments, the one or more interfaces 212 comprise means to receive the measurement data directly from the DVS or means to read the measurement data from a data medium or data storage storing the measurement data. In particular, the one or more interfaces 212 can comprise any means enabling the data processing circuit 214 to receive the measurement data of the DVS 2000.

As shown in the block diagram the one or more interfaces 212 are coupled to the data processing circuit 214 of the apparatus 200. In embodiments data processing circuit 214 may be implemented using one or more processing units, one or more processing devices, any means for processing, such as a processor, a computer or a programmable hardware component being operable with accordingly adapted software. In other words, the described functions of the data processing circuit 214 may as well be implemented in software, which is then executed on one or more programmable hardware components. Such hardware components may comprise a general-purpose processor, a Digital Signal Processor (DSP), a micro-controller, and the like. In particular, the data processing circuit 214 may comprise any means enabling the apparatus 200 to receive the measurement data (via the one or more interfaces 212), determine the simultaneously detected events, and approximate their timestamps in the manner of the proposed concept. As described below with reference to Fig. 3, the proposed concept, e.g., is used in applications for event-based imaging/vision.

Fig. 3 schematically illustrates an application of the proposed concept for processing measurement data of a DVS.

Fig. 3 schematically illustrates a DVS comprising a pixel array 302 and a read out circuit 308. The measurement data, e.g., is indicative of events 310 detected by the pixel array 302 in response to changes in illumination of pixels of the pixel array.

In particular, the measurement data is indicative of a plurality of first events detected by pixels of the DVS and of a plurality of first timestamps indicative of recording times at which the first events were successively recorded by the read out circuit 308, and of a plurality of subsequent second events detected by one or more of the pixels and of a plurality of second timestamps indicative of recording times at which the second events were successively recorded by the read out circuit 308.

Accordingly, the events 310 comprise first and second events. Event 310A detected by a first pixel, event 310B detected by a second pixel, and event 310C detected by a third pixel, e.g., are first events. Event 310D detected by the first pixel after event 310A and an event 310E detected by the second pixel after event 310B, e.g., are second events. As indicated by Fig. 3, the events 310 are signaled to the read out circuit 308 and the read out circuit 308 may in a first step 304 determine a position of the events 310, e.g., based on spatial coordinates of the pixels.

In a second step 306, the read out circuit 308 successively assigns timestamps to the events 310. In practice, this leads to the above described delay in the timestamps. Also timestamps of simultaneously detected events differ by at least this delay. Hence, the measurement data may be referred to as “perturbed measurement data”.

Also, the read out circuit 308 may determine a polarity of the events 310.

For recording the events 310, the read out circuit 308, e.g., enters the timestamp, the polarity, and the position of the events 310 in respective records E(x, y, t, p) of the events 310 in the measurement data, wherein x and y denote the spatial coordinates of the respective pixels and events 310, t denotes the timestamp, and p denotes the polarity of the respective events.

The measurement data including the records, e.g., are made available for method 100 and/or the apparatus 200 via direct communication or on a data storage/medium. Method 100 and/or apparatus 200 optionally are implemented in the DVS or in a separate device (e.g. a separate data processing circuit).

In the present application, method 100, e.g., provides for obtaining a representation of the events 310 over time and position based on the timestamps and positions of the pixels. As can be seen from Fig. 3, the events 310, e.g., are mapped over time and position, such as in diagram 320. The abscissa of diagram 320, e.g., indicates the time, and the ordinate of diagram 320 indicates a position of pixels of the pixel array 302. It is noted that, for easier visualization, the position is schematically indicated by (only) one axis, i.e. the ordinate. In practice, the position may be indicated in two dimensions of a three-dimensional representation.

The events 310 are mapped/plotted in the diagram over time and position in accordance with their timestamps and the spatial coordinates of the respective pixels. Accordingly, records of event 310A and 310D are plotted at the same coordinates relative to the ordinate and at different times relative to the abscissa. As can be seen from diagram 320, the timestamp of event 310A, e.g., is indicative of time t and the timestamp of event 310D is indicative of time tiast- According to this principle, the other events (310B, 310E, 310C) are also plotted in diagram 320.

Determining 120 the simultaneously detected events, e.g., comprises detecting in the representation of the events, e.g. in diagram 320, one or more edges (e.g. edge 309a and 309b) including simultaneously detected events based on the positions and the first and second timestamps. For example, an edge detection algorithm or edge detection operator determines the edges 309a and 309b based on the positions, polarities, and/or deviations of the timestamps as input to the edge detection algorithm or edge detection operator. The edge detection algorithm or edge detection operator, e.g., is a Gabor filter kernel, a Canny edge detector, a Deriche edge detector, or the like. The edges 309a and 309b, e.g., comprise (a group or a cluster of) events which have the same polarity, and/or whose timestamps and positions differ less from each other than from timestamps and/or positions of events off the edges 309a and 309b or less than a predefined threshold for the deviations of the timestamps and/or positions.

In practice events which are detected simultaneously with the first events are recorded before the subsequent second events. Hence, the second timestamps can be also used to determine simultaneously detected events from the events 310, e.g. to determine edge 309a and/or 309b. For example, in practice, events which are detected simultaneously with event 310A are recorded before event 310D. In other words, delays in timestamps of events which are simultaneously detected with event 310A differ less from t than tiast. Hence, a time interval Jmax between t and tiast specifies a maximum delay (“maximum jitter”) for timestamps of events which are simultaneously detected with event 310A. Event 310C, e.g., is recorded after tiast and it is therefore excluded that event 310C was detected simultaneously with event 310A. Event 310B, e.g., is recorded between t and tiast. Accordingly, its timestamp differs less than Jmax. Therefore, it is at least not excluded that event 310B was detected with event 310A. It is noted that Jmax further allows to check for other events if they could be detected simultaneously with event 310A. Jmax may be also defined for other first events, e.g. for 310B. For event 310B, Jmax is a time interval between the timestamp of event 310B and the timestamp of event 310E. This allows to determine simultaneously detected events based on the Jmax of the first events, e.g., such that the timestamps of events which are considered to be detected simultaneously differ less than Jmax from each other. In other words, Jmax is specified for multiple events and only those events may be considered as detected simultaneously whose timestamps differ less than by their Jmax. This principle may be applied to edge detection. For this, Jmax, e.g., is used as input for edge detection. For example, this principle is applied for determining edge 309a in such a way that edge 309a includes only events whose timestamps differ by less than their Jmax. In the present example, edge 309a, e.g., includes event 310A, 310B. Further, this principle may be applied for detecting edge 309b and/or further edges.

In a subsequent step, the timestamps of simultaneously detected events are approximated for compensating at least partially the delay between the simultaneously detected events. In the present example, approximating 130 the timestamps, e.g., provides for aligning/matching the timestamps of events of edge 309a and aligning the timestamps of events of edge 309b. Diagram 330 schematically illustrates the alignment of the events. In particular, diagram 330 illustrates the events 310 over time and position/space and in accordance with the position and the aligned timestamps. The abscissa of diagram 330 indicates the time and the ordinate indicates the position of the events 310.

In practice, the edges 309a and 309b represent moving solid objects with defined structure and the events of 309a and 309b result from the same “moving front” of such objects. For a more plausible representation of the object the “first leading event” (e.g. the “youngest event”) representing the moving front of the object is supposed to align in time with other events representing this moving front. To this end, the events of each edge, e.g., are aligned with the “youngest” timestamp of the respective edge. Among the events of edge 309a, e.g., event 310B has the youngest timestamp. As can be seen from diagram 330, the timestamp of event 310B is indicative of time tc. Accordingly, each timestamp of events of edge 309a are aligned with tc. According to the same principle, the events of edge 309b are aligned. E.g., the events of edges 309b are aligned with the timestamp of 310E. Thus, delays in the timestamps of simultaneously events are at least partially compensated. In particular, the aligned timestamps better reflect the actual timing of the events than the perturbed measurement data and provide an enhanced temporal resolution of the events 310. Hence, method 100 can be also understood as a concept for “timing correction”.

Method 100 optionally is implemented in a machine-learning based edge detection framework. For example, detecting the at least one edge comprises using a machine-learning based edge detection framework for detecting the at least one edge in the representation. The machine-learning based edge detection framework, e.g., is implemented in a neural network. The machine-learning based edge detection framework, e.g., is configured to determine, based on the measurement data (positions, timestamps, polarity) and Jmax of one or multiple events, one or more edges including simultaneously detected events.

For example, a so-called “event stream” comprising the measurement data from the DVS, and Jmax are fed into a shallow or other convolutional neural network for highly spatially localized computation. Such a network, e.g., is trained to detect edges in a representation over time and space/position (e.g. in diagram 320) using Gabor filter kernels, and to correct the timing of records in the measurement data/event stream in the above manner so as to re-align simultaneously detected events in time with their “leading edge event” (i.e. the event having the youngest timestamp of a respective edge) within a jitter window defined by Jmax of events on the respective edge.

The output of the neural network can be a negative regression value for the timestamps of simultaneously detected events to cause timestamps of the simultaneously detected event with their youngest timestamp.

The machine-learning based edge detection framework or the neural network can be trained on perturbed measurement data and related (ideally) unperturbed measurement data. Due to the aforementioned way of recording events, dynamic vision sensors may not be able to produce unperturbed measurement data. For this reason, the unperturbed and perturbed measurement data (training data) for training the machine-learning based edge detection framework or neural network, e.g., is simulated.

The machine-learning based edge detection framework, e.g., provides a higher reliability and efficiency in determining the simultaneously detected events compared to other approaches and/or frameworks for edge detection.

Further embodiments pertain to:

(1) A method for processing measurement data of a dynamic vision sensor, DVS, the method comprising: receiving measurement data of the DVS, wherein the measurement data is indicative of a plurality of events detected by pixels of the DVS and of a plurality of timestamps indicative of recording times at which the events were successively recorded by a read out circuit of the DVS after detection by the pixels, wherein the timestamps of the events differ by a delay caused by recording the events successively; determining simultaneously detected events from the plurality of events based on deviations of the timestamps to each other; and approximating timestamps of simultaneously detected events for compensating at least partly the delay between the simultaneously detected events. (2) The method of (1), wherein the measurement data is further indicative of positions of the pixels, and wherein determining simultaneously detected events comprises determining simultaneously detected events from the plurality of events based on the positions.

(3) The method of (1) or (2), wherein the measurement data is further indicative of polarities of the events, and wherein determining simultaneously detected events comprises determining simultaneously detected events based on the polarities.

(4) The method of any one of (1) to (3), wherein approximating the timestamps comprises aligning the timestamps of the simultaneously detected events.

(5) The method of any one of (1) to (4), wherein approximating the timestamps comprises aligning the timestamps of the simultaneously detected events to the latest of the timestamps of the simultaneously detected events.

(6) The method of (2), wherein determining simultaneously detected events from the plurality of events comprises: obtaining a representation of the events over time and position based on the timestamps of the events and positions of the pixels; and detecting in the representation at least one edge including the simultaneously detected events based on the on deviations of the timestamps to each other and the positions.

(7) The method of any one of (1) to (6), wherein the measurement data is indicative of a plurality of first events detected by pixels of the DVS and of a plurality of first timestamps indicative of recording times at which the first events were successively recorded by the read out circuit, and of a plurality of subsequent second events detected by one or more of the pixels and of a plurality of second timestamps indicative of recording times at which the second events were successively recorded by the read out circuit; and wherein determining the simultaneously detected events comprises determining simultaneously detected events from the events based on the first and the second timestamps.

(8) The method of (7), wherein determining simultaneously detected events from the first and the second events comprises: obtaining a representation of the first and the second events over time and position based on the first and the second timestamps and positions of the pixels; and detecting in the representation of the first and the second events one or more edges including simultaneously detected events from the events based on the positions and the first and second timestamps.

(9) The method of (7) or (8), wherein detecting the at least one edge comprises using a machine-learning based edge detection framework for detecting the at least one edge in the representation.

(10) A computer program comprising instructions, which, when the computer program is executed by a processor, cause the processor to carry out the method of any one of (1) to (9).

(11) An apparatus comprising: one or more interfaces for communication; and a data processing circuit configured to control the one or more interfaces, wherein the data processing circuit is further configured to execute one of the methods of any one of (1) to (9).

(12) A dynamic vision sensor comprising the apparatus of (11). The aspects and features described in relation to a particular one of the previous examples may also be combined with one or more of the further examples to replace an identical or similar feature of that further example or to additionally introduce the features into the further example.

Examples may further be or relate to a (computer) program including a program code to execute one or more of the above methods when the program is executed on a computer, processor or other programmable hardware component. Thus, steps, operations or processes of different ones of the methods described above may also be executed by programmed computers, processors or other programmable hardware components. Examples may also cover program storage devices, such as digital data storage media, which are machine-, processor- or computer-readable and encode and/or contain machine-executable, processor-executable or computer-executable programs and instructions. Program storage devices may include or be digital storage devices, magnetic storage media such as magnetic disks and magnetic tapes, hard disk drives, or optically readable digital data storage media, for example. Other examples may also include computers, processors, control units, (field) programmable logic arrays ((F)PLAs), (field) programmable gate arrays ((F)PGAs), graphics processor units (GPU), application-specific integrated circuits (ASICs), integrated circuits (ICs) or system-on-a-chip (SoCs) systems programmed to execute the steps of the methods described above.

It is further understood that the disclosure of several steps, processes, operations or functions disclosed in the description or claims shall not be construed to imply that these operations are necessarily dependent on the order described, unless explicitly stated in the individual case or necessary for technical reasons. Therefore, the previous description does not limit the execution of several steps or functions to a certain order. Furthermore, in further examples, a single step, function, process or operation may include and/or be broken up into several sub-steps, - functions, -processes or -operations.

If some aspects have been described in relation to a device or system, these aspects should also be understood as a description of the corresponding method. For example, a block, device or functional aspect of the device or system may correspond to a feature, such as a method step, of the corresponding method. Accordingly, aspects described in relation to a method shall also be understood as a description of a corresponding block, a corresponding element, a property or a functional feature of a corresponding device or a corresponding system. The following claims are hereby incorporated in the detailed description, wherein each claim may stand on its own as a separate example. It should also be noted that although in the claims a dependent claim refers to a particular combination with one or more other claims, other examples may also include a combination of the dependent claim with the subject matter of any other dependent or independent claim. Such combinations are hereby explicitly proposed, unless it is stated in the individual case that a particular combination is not intended. Furthermore, features of a claim should also be included for any other independent claim, even if that claim is not directly defined as dependent on that other independent claim.

Claims

Claims A method for processing measurement data of a dynamic vision sensor, DVS, the method comprising: receiving measurement data of the DVS, wherein the measurement data is indicative of a plurality of events detected by pixels of the DVS and of a plurality of timestamps indicative of recording times at which the events were successively recorded by a read out circuit of the DVS after detection by the pixels, wherein the timestamps of the events differ by a delay caused by recording the events successively; determining simultaneously detected events from the plurality of events based on deviations of the timestamps to each other; and approximating timestamps of simultaneously detected events for compensating at least partly the delay between the simultaneously detected events. The method of claim 1 , wherein the measurement data is further indicative of positions of the pixels, and wherein determining simultaneously detected events comprises determining simultaneously detected events from the plurality of events based on the positions. The method of claim 1, wherein the measurement data is further indicative of polarities of the events, and wherein determining simultaneously detected events comprises determining simultaneously detected events based on the polarities. The method of claim 1, wherein approximating the timestamps comprises aligning the timestamps of the simultaneously detected events. The method of claim 1, wherein approximating the timestamps comprises aligning the timestamps of the simultaneously detected events to the latest of the timestamps of the simultaneously detected events.

6. The method of claim 2, wherein determining simultaneously detected events from the plurality of events comprises: obtaining a representation of the events over time and position based on the timestamps of the events and positions of the pixels; and detecting in the representation at least one edge including the simultaneously detected events based on the on deviations of the timestamps to each other and the positions.

7. The method of claim 1, wherein the measurement data is indicative of a plurality of first events detected by pixels of the DVS and of a plurality of first timestamps indicative of recording times at which the first events were successively recorded by the read out circuit, and of a plurality of subsequent second events detected by one or more of the pixels and of a plurality of second timestamps indicative of recording times at which the second events were successively recorded by the read out circuit; and wherein determining the simultaneously detected events comprises determining simultaneously detected events from the events based on the first and the second timestamps.

8. The method of claim 7, wherein determining simultaneously detected events from the first and the second events comprises: obtaining a representation of the first and the second events over time and position based on the first and the second timestamps and positions of the pixels; and detecting in the representation of the first and the second events one or more edges including simultaneously detected events from the events based on the positions and the first and second timestamps.

9. The method of any one of claim 7, wherein detecting the at least one edge comprises using a machine-learning based edge detection framework for detecting the at least one edge in the representation. 10. A computer program comprising instructions, which, when the computer program is executed by a processor, cause the processor to carry out the method of claim 1.

11. An apparatus comprising: one or more interfaces for communication; and a data processing circuit configured to control the one or more interfaces, wherein the data processing circuit is further configured to execute the method of the claim 1. 12. A dynamic vision sensor comprising the apparatus of claim 11.