EP4315274A1

EP4315274A1 - Circuitry and method for encoding events

Info

Publication number: EP4315274A1
Application number: EP22713303.0A
Authority: EP
Inventors: Mikhail Makarov; Marc Osswald
Original assignee: Sony Advanced Visual Sensing AG; Sony Semiconductor Solutions Corp
Current assignee: Sony Advanced Visual Sensing AG; Sony Semiconductor Solutions Corp
Priority date: 2021-03-26
Filing date: 2022-02-25
Publication date: 2024-02-07
Also published as: WO2022199976A1

Abstract

A circuitry for encoding events, including a plurality of event image pixels, wherein the circuitry is configured to: generate events based on the plurality of event image pixels; determine, for each event, a product of a polarity of the event and dictionary data associated with the event image pixel that generated the event; and generate encoded event image data by summation of the products.

Description

CIRCUITRY AND METHOD FOR ENCODING EVENTS

TECHNICAL FIELD

The present disclosure generally pertains to a circuitry and a method for encoding events generated based on event image pixels of an event image sensor.

TECHNICAL BACKGROUND

Generally, event cameras are known. An overview is given, for instance, in: G. Gallego et al., "Event-based Vision: A Survey," in IEEE Transactions on Pattern Analysis and Machine Intelligence, doi: 10.1109/TPAMI.2020.3008413.

Event image sensors used in event cameras typically differ from conventional image sensors in that each event image pixel asynchronously and independently detect changes in the amount of light incident onto the event image pixel such that the dynamic of a scene is captured rather than a frame- based synchronous sequence of pictures. This may result in a high temporal resolution, low latency, high dynamic range, and low power consumption of event cameras.

As generally known, an event image sensor may generate an event based on an electric signal from an event image pixel (which may also be referred to as event-based vision sensor pixel) in response to a detected change of the amount of light exceeding a threshold. Such an event may identify an event image pixel that generated the event (e.g., column and row index in an event image pixel array), a time when the event was generated, and a polarity indicating whether the change is an increase or decrease of the amount of light.

Different representations or formats (encodings) for generated events are known which are used when the event information is transmitted from the event image sensor (which may also be referred to as event-based vision sensor (EVS)) to a receiver (e.g., a processor), however, such known formats may have characteristics unsuitable for further processing.

Although there exist techniques for encoding events, it is generally desirable to improve the existing techniques.

SUMMARY

According to a first aspect the disclosure provides a circuitry for encoding events, comprising a plurality of event image pixels, the circuitry being configured to: generate events based on the plurality of event image pixels; determine, for each event, a product of a polarity of the event and dictionary data associated with the event image pixel that generated the event; and generate encoded event image data by summation of the products.

According to a second aspect the disclosure provides a method for encoding events, the method comprising: determining, for each event generated based on a plurality of event image pixels, a product of a polarity of the event and dictionary data associated with the event image pixel that generated the event; and generating encoded event image data by summation of the products.

According to a third aspect the present disclosure provides a circuitry for event image processing, the circuitry being configured to: receive encoded event image data; input the encoded event image data into a neural network, wherein the neural network includes a layer which adapts a size of the encoded event image data to an event matrix size; and perform event image processing based on the neural network.

According to a fourth aspect the disclosure provides a method for event image processing, the method comprising: receiving encoded event image data; inputting the encoded event image data into a neural network, wherein the neural network includes a layer which adapts a size of the encoded event image data to an event matrix size; and performing event image processing based on the neural network.

Further aspects are set forth in the dependent claims, the following description and the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments are explained byway of example with respect to the accompanying drawings, in which:

Fig. 1 schematically illustrates in a block diagram an embodiment of an event image sensor and a receiver;

Fig. 2 schematically illustrates an embodiment of an event matrix, dictionary data and encoded event image data;

Fig. 3 schematically illustrates in a flow diagram a first embodiment of a method for encoding events;

Fig. 4 schematically illustrates in a block diagram a first embodiment of a data transfer control;

Fig. 5 schematically illustrates in a block diagram a second embodiment of a data transfer control; Fig. 6 schematically illustrates in a block diagram a third embodiment of a data transfer control;

Fig. 7 schematically illustrates in a block diagram an embodiment of a processing unit; and

Fig. 8 schematically illustrates in a flow diagram a second embodiment of a method for encoding events.

DETAILED DESCRIPTION OF EMBODIMENTS

Before a detailed description of the embodiments under reference of Fig. 1 is given, general explanations are made.

As mentioned in the outset, generally, event cameras are known. In some embodiments, each event image pixel of an event image sensor is configured to detect a change in the amount of light incident onto the respective event image pixel and to generate an electric signal when the detected change exceeded a threshold.

An event is generated based on the electric signal from the event image pixel which, in some embodiments, includes event information that identifies the event image pixel that generated the event (e.g., column and row index in an event image pixel array), a time when the event was generated by the event image pixel, and a polarity indicating whether the change is an increase or decrease of the amount of light (“raw” event format).

As further mentioned in the outset, different representations or formats (encodings) for generated events are known which are used when the event information is transmitted from the event image sensor to a receiver (e.g., a processor), however, such known formats may have characteristics unsuitable for further processing (e.g., storage, further encoding, statistical analysis without decoding, data transfer).

For example, using a “raw” event format, as described above, may have characteristics unsuitable for further encoding or statistical analysis without decoding. Moreover, a data size to be transferred to the receiver may vary as the number of events varies in time.

Moreover, e.g., using an event matrix format similar to conventional image formats may have characteristics which may require increased memory for storage on the event image sensor (and the receiver), may lead to a bottleneck in the data transfer and may not be easily combined with standard video encoders due to the resulting large and sparse event matrices.

It has been recognized that, in some instances, events should be encoded with a format having a constant size for simplifying data transfer, since existing technologies may be optimized for fixed size data transfer. It has further been recognized that, in some instances, events should be encoded with a format which allows further compression using existing methods such as entropy encoding (e.g., Huffman encoding, arithmetic coding).

Hence, some embodiments pertain to a circuitry for encoding events, including a plurality of event image pixels, wherein the circuitry is configured to: generate events based on the plurality of event image pixels; determine, for each event, a product of a polarity of the event and dictionary data associated with the event image pixel that generated the event; and generate encoded event image data by summation of the products.

The circuitry may be based on or may include or may be implemented by typical electronic components configured to achieve the functionality as described herein.

The circuitry may be based on or may include or may be implemented as integrated circuity logic and the functionality may be implemented by software executed by a processor or the like. The circuitry may be based on or may include or may be implemented by a CPU (central processing unit), a microcontroller, an FPGA (field programmable gate array), an ASIC (application specific integrated circuit), a GPU (graphical processing unit), a DSP (digital signal processor) or the like.

The circuitry may be based on or may include or may be implemented in parts by typical electronic components and integrated circuitry logic and in parts by software.

The circuitry may include storage capabilities such as magnetic storage, semiconductor storage, etc. The circuitry may include a data bus for transmitting and receiving data and may implement corresponding communication protocols.

The circuitry includes a plurality of event image pixels. Different types of event image sensors including, e.g., an event image pixel array are known, for example, Dynamic Vision Sensor (DVS), Asynchronous Time Based Image Sensor (ATIS) or Dynamic and Active Pixel Vision Sensor (DAVIS), which may be used in some embodiments. Moreover, such known event image sensors may combine event detection and conventional frame-based image capturing.

In some embodiments, an event image pixel generates an electric signal when a detected change of an amount of light incident onto the pixel exceeds a threshold and stores a representation of the amount of light at the time the event is generated.

In some embodiments, the electric signal is used by the circuitry to generate an event and, thus, the circuitry is configured to generate events based on the plurality of event image pixels. As mentioned above, in some embodiments, an event includes event information which identifies the event image pixel that generated the event (e.g., column and row index in an event image pixel array of the event image sensor), a time when the event was generated by the event image pixel, and a polarity indicating whether the change is an increase or decrease of the amount of light incident onto the event image pixel.

Typically, as mentioned above, the events are generated asynchronously such that, when a number of events are accumulated in the “raw” event format before transfer to the receiver, the data size to be transferred may vary, however, existing technologies may be optimized for fixed size data transfer.

Furthermore, as mentioned above, while generation of an event matrix (having a size corresponding to the number of rows and columns of an event image pixel array) allows to associate events generated in a predetermined time interval with an event image similar to conventional image formats, this format may require increased memory for storage on the event image sensor (and the receiver), which may lead to a bottleneck in the data transfer and may not be easily combined with standard video encoders due to the resulting large and sparse event matrices.

However, it has been recognized that it may be useful to generate event matrices representing an event image, since the event image provides information about a scene complementary to a conventional image (for example, the predetermined time interval, as mentioned above, may be synchronized with a frame rate for capturing conventional images of the scene). This is because the event image provides information about the dynamics in the scene and the conventional image provides information about the statics of the scene.

Thus, it has been further recognized that, in some instances, the encoding of events should provide a format which, on the one hand, allows to generate an approximated reconstruction of an event matrix representing an event image and, on the other hand, requires less memory for storage (compressed representation of an event matrix), provides a fixed data size for data transfer and allows to further encode (e.g., entropy encoding) the encoded events.

Hence, the circuitry is configured to accumulate events by determining, for each event, a product of a polarity of the event and dictionary data associated with the event image pixel that generated the event, and by generating encoded event image data by summation of the products. Thereby, events are encoded in a format suitable for further processing, as will be discussed in more detail below.

The dictionary data represent a vector or matrix or the like having a fixed size (and the same fixed size for each pixel) smaller than the size of the event matrix (which has a size corresponding to the number of rows and columns of an event image pixel array). In some embodiments, the dictionary data include pseudo-randomly generated values +1 and -1. In some embodiments, the pseudo- randomly generated values +1 and -1 are generated based on a uniform probability distribution such that each value +1 and -1 has a probability of 0.5 to be generated.

Hence, a product of the polarity and the dictionary has the size of the dictionary data since polarity may be represented by an Integer such as +1 and -1.

Thus, the encoded event image data have the same size (and representation, e.g., vector or matrix or the like) as the dictionary data and include the sum of the products.

Thereby, the encoded event image data provide a compressed and dense representation of a large and sparse event matrix (representing an event image).

In some embodiments, the dictionary data, for each event image pixel, may be stored in a memory and obtained when required. Dictionary data of an event image pixel is required when the event image pixel generated an event and the product for that event is to be determined. In some embodiments, the circuitry is further configured to generate the dictionary data when the dictionary data is required in the determination of the products (“on-the-fly”) .

As, in some embodiments, the dictionary includes pseudo-randomly generated values +1 and -1, the summation of the products results in encoded event image data including Integer values with approximately Gaussian (normal) distribution (or the like). In such embodiments, as the encoded event image data is dense and includes Integer values with approximately Gaussian (normal) distribution, entropy encoding allows further compression, since it may be more efficient in such cases.

Hence, in some embodiments, the circuitry is further configured to entropy encode the encoded event image data.

As mentioned above, the encoded event image data basically correspond to a dense and compressed representation of an event matrix (event image).

Thus, in order to approximately reconstruct (decode), in some embodiments, the event matrix from the encoded event image data, the sparsity of the event matrix (degrees of freedom) should be less or equal than the size of the encoded event image data (number of values in the encoded event image data). This constraint represents a compression limit. The sparsity of an event matrix (degrees of freedom) may be estimated by the number of events generated within a predetermined time interval times two related to the row and column indices for each event (however, the exact number can be determined by running encoding-decoding numerical experiments). The approximated reconstruction of the event matrix can be performed, in some embodiments, by compressed sensing decoding algorithms such as FISTA (“Fast Iterative Shrinkage -Thresholding Algorithm”). The FISTA is discussed, for instance, in: Amir Beck and Marc Teboulle, “A Fast Iterative Shrinkage -Thresholding Algorithm for Linear Inverse Problems”, SIAM Journal on Imaging Sciences 2009, Vol. 2, No. 1, pp. 183-202.

Such compressed sensing decoding algorithms typically solve an underdetermined linear system of equations under the constraint of sparsity of the solution such that from the infinite number of possible solutions only a few sparse solutions remain.

Generally, the event image sensor or the circuitry may operate or output encoded event image data asynchronously or synchronously (“pseudo -frame based”), which will be discussed in more detail in the following. In some embodiments, the event image sensor or the circuitry may switch between asynchronous and synchronous operation or output.

Hence, in some embodiments, the circuitry is further configured to output a stream of encoded event image data.

The stream of encoded event image data basically refers to a regular output of encoded event image data, wherein each encoded event image data represents an event matrix (event image). For illustration, first encoded event image data are output at a first time point, second encoded event image data are output at a second time point, and so on.

In some embodiments, events are accumulated in encoded event image data until transmission of the encoded event image data is triggered.

In some embodiments, the circuitry is further configured to output encoded event image data when a predetermined time elapsed since a last output of encoded event image data.

For example, encoded event image may be output every millisecond, every ten milliseconds, etc. Thereby, a constant data transfer rate may be achieved due to periodic output triggering.

However, in such embodiments, a compression may not be optimized. For example, in a case of a low event generation rate, the number of accumulated events may be low and well below the compression limit. For example, in a case of a high event generation rate, the number of accumulated events may be well above the compression limit such that event image reconstruction may be less accurate.

In some embodiments, the circuitry is further configured to output encoded event image data when a number of generated events is within a predetermined range since a last output of encoded event image data. For example, encoded event image data may be output when the number of events is close to the compression limit. Thereby, compression may be optimized due to compression limit-based triggering.

However, in such embodiments, a data transfer rate may vary. For example, when the event generation rate varies due to frequent changes in the scene. However, a data transfer size may still be fixed.

Hence, in some embodiments, the circuitry is further configured to output encoded event image data when a predetermined time elapsed since a last output of encoded event image data or when a number of generated events is within a predetermined range since a last output of encoded event image data. Thereby, a combination of the trigger mechanisms discussed above is provided.

In a case of synchronous operation or output, in some embodiments, time is divided into a sequence of (substantially) equal predetermined time intervals. In such embodiments, each predetermined time interval represents a pseudo-frame and the number of predetermined time intervals per unit time represents a pseudo-frame rate, since, as discussed, event generation is actually asynchronous. However, it has been recognized that it may be useful to provide a synchronized stream of encoded image data each representing an event matrix (event image) based on a sequence of predetermined time intervals.

The division into the sequence of equal predetermined time intervals may be synchronized with a frame rate of conventional image data acquisition (for example, the event image sensor may provide both functionalities). The division into the sequence of equal predetermined time intervals may be based on an event generation rate, technical constraints of the event image sensor (e.g., memory constraints) or an application (e.g., automotive applications may require higher pseudo-frame rates than surveillance applications) or the like.

Hence, in a case of synchronous operation or output, in some embodiments, events that are generated within a predetermined time interval are accumulated in encoded event image data associated with the predetermined time interval.

In some embodiments, when an average event generation rate is above a predetermined rate threshold, each encoded event image data in the stream are based on events generated within an associated predetermined time interval. A corresponding embodiment will be discussed under reference of Fig. 4. In such embodiments, each encoded event image data represents one event image (event matrix) .

For example, when the average event generation rate (number of events generated in a predetermined time interval in average over a predetermined number of time intervals) is such that the average number of events generated in a predetermined time interval is more than half the compression limit, the compression achieved in the encoded event image data and data transfer efficiency may be optimized.

In some embodiments, when an average event generation rate is above a predetermined rate threshold, at least two of the encoded event image data in the stream are based on events generated within an associated predetermined time interval.

For example, the circuitry may have a limited storage capacity and the encoded event image data stored on the circuitry may be of limited size accordingly such that a compression limit may be reached already well before the end of an associated predetermined time interval due to a high event generation rate. Thus, the circuitry may output first encoded event image data already when the compression limit is reached and may not wait until the end of the associated predetermined time interval. Hence, the rest of the events arriving between output of the first encoded event image data and the end of the associated predetermined time interval may be accumulated in second encoded event image data which may be output at the end of the associated predetermined time interval. Accordingly, at least two encoded event image data in the stream are based on events generated within the associated predetermined time interval.

In some embodiments, when an average event generation rate is below a predetermined rate threshold, one or more of the encoded event image data in the stream are based on a combination of first events generated within a first predetermined time interval and of second events generated within a consecutive second predetermined time interval. A corresponding embodiment will be discussed under reference of Fig. 5.

For example, when the average event generation rate (number of events generated in a predetermined time interval in average over a predetermined number of time intervals) is such that the average number of events generated in a predetermined time interval is less than half the compression limit, the compression achieved in the encoded event image data and data transfer efficiency may be not optimized. Hence, instead of outputting two separate encoded event image data, events of both associated predetermined time intervals should be combined for improving compression and data transfer efficiency.

Of course, generally, such a combination is not limited to two predetermined time intervals and may be extended to more than two predetermined time intervals, which may be based on the average event generation rate, technical constraints (e.g., memory, speed, power, etc.) or the like.

The events may be combined in the generation process of the encoded event image data or the corresponding event image data may be combined afterwards. Moreover, in some embodiments, the combination is based on using, when determining the products, a first portion of each dictionary data used for the first events and a second portion, different from the first portion, of each dictionary data used for the second events. A corresponding embodiment will be discussed under reference of Fig. 6.

For example, when dictionary data is represented by a matrix, the first portion may be the upper half of a matrix and the second portion may be the lower half of a matrix.

In some embodiments, the circuitry is further configured to select a number of bits used in the encoded event image data. This may allow further configuration of compression and accuracy of event image (event matrix) reconstruction (decoding), storage and data transfer optimization.

For example, the circuitry may have a limited storage capacity such that a number of bits for storing the encoded event image data may be selected dynamically. For example, the number of bits may be selected dynamically according to the size of the encoded event image data or the average event generation rate. For example, in a case of a low event generation rate, the number of bits may be selected to be lower than in a case of a high event generation rate, for instance, for storing more than one encoded event image data (e.g., matrix) for combination of the more than one encoded event image data before transmission.

Generally, in some cases, it may happen that the number of accumulated events is above the compression limit. In such cases, the circuitry may generate an error message indicating that compression limit is exceeded and may send the error message to a control unit of the event image sensor. The control unit may decrease the length of the predetermined time intervals or may adjust the threshold (threshold which a change in the amount of light must exceed for generating an event) for event generation of the event image pixels in order to decrease a sensitivity for event generation.

Some embodiments pertain to a method for encoding events, wherein the method includes: determining, as described herein, for each event generated based on a plurality of event image pixels, a product of a polarity of the event and dictionary data associated with the event image pixel that generated the event; and generating, as described herein, encoded event image data by summation of the products.

The method may be performed by the circuitry described herein. The method may be performed by a CPU (central processing unit), a microcontroller, an FPGA (field programmable gate array), an ASIC (application specific integrated circuit) or the like.

Some embodiments pertain to a circuitry for event image processing, wherein the circuitry is configured to: receive encoded event image data; input the encoded event image data into a neural network, wherein the neural network includes a layer which adapts a size of the encoded event image data to an event matrix size; and perform event image processing based on the neural network.

The circuitry may be based on or may include or may be implemented as integrated circuity logic and the functionality may be implemented by software executed by a processor or the like. The circuitry may be based on or may include or may be implemented by a CPU (central processing unit), a microcontroller, an FPGA (field programmable gate array), an ASIC (application specific integrated circuit), a GPU (graphical processing unit), a DSP (digital signal processor), an event- based vision processor, a neural network processor, an artificial intelligence accelerator or the like.

The circuitry may include storage capabilities such as magnetic storage, semiconductor storage, memristive storage, etc. The circuitry may include a data bus for transmitting and receiving data and may implement corresponding communication protocols.

Some embodiments pertain to a method for event image processing, the method including: receiving encoded event image data; inputting the encoded event image data into a neural network, wherein the neural network includes a layer which adapts a size of the encoded event image data to an event matrix size; and performing event image processing based on the neural network.

The method may be performed by the circuitry for event image processing. The method may be performed by a CPU (central processing unit), a microcontroller, an FPGA (field programmable gate array), an ASIC (application specific integrated circuit), a GPU (graphical processing unit), a DSP (digital signal processor) or the like.

Event image processing may be or may include object/pattern classification, gesture recognition, hand tracking, visual odometry, object tracking, anomaly detection in production lines, image enhancement such as deblurring or slow motion video (for example, for enhancing image quality of a conventional image by motion blur reduction or extra slow motion utilizing the high temporal dynamics of event-based vision sensors).

The methods as described herein are also implemented in some embodiments as a computer program causing a computer and/ or a processor to perform the method, when being carried out on the computer and/or processor. In some embodiments, also a non-transitory computer-readable recording medium is provided that stores therein a computer program product, which, when executed by a processor, such as the processor described above, causes the methods described herein to be performed.

Returning to Fig. 1, there is schematically illustrated in a block diagram an embodiment of an event image sensor 1 and a receiver 11, which will be discussed in the following under reference of Fig. 1.

The event image sensor 1 includes a circuitry 2 for encoding events and a control unit 5.

The circuitry 2 includes an event image pixel array 3 which includes a plurality of event image pixels 4 arranged in rows and columns (the dotted lines indicate further event image pixels between the shown four event image pixels). Each of the plurality of event image pixels 4 is configured to generate an electric signal when a change of an amount of light incident onto the respective event image pixel exceeds a threshold (“event detection”).

The control unit 5 controls the overall operation of the event image sensor 1 such as timing and readout control. For example, when an event image pixel of the plurality of event image pixels 4 detects an event, the control unit 5 may instruct readout of the electric signals of the row or column or multiple rows or columns including the event image pixel that detected an event.

The circuitry 2 further includes a processing unit 6. The processing unit 6 includes an event generation unit 7, an encoding unit 8 and an output unit 9.

The event generation unit 7 generates events based on the electric signals of the plurality of event image pixels 4.

The encoding unit 8 determines, for each event, a product of a polarity of the event and dictionary data associated with the event image pixel that generated the event.

The encoding unit 8 further generates encoded event image 10 data by summation of the products.

The output unit 9 outputs the encoded event image data 10 to the receiver 11 when an output trigger occurred. The output unit 9 includes a data communication interface (e.g., a data bus interface for outputting the encoded event image data 10) .

The receiver 11 includes a data communication interface 12 (e.g., a data bus interface) for receiving the encoded event image data 10.

The receiver 11 further includes a decoding unit 13, a computing unit 14, a statistics computing unit 15 and a neural network unit 16.

The encoded event image data 10 is input to the decoding unit 13 which is configured to decode the encoded event image data 10 based on a compressed sensing decoding algorithm (e.g., FISTA). The decoded encoded event image data 10 represents an approximate reconstruction of an event matrix (event image).

The decoded encoded event image data 10 is input to the computing unit 14 which is configured to perform event image processing (which may be application specific).

The encoded event image data 10 is input to the statistics computing unit 15 which is configured to perform statistical analysis on the encoded event image data 10 without decoding it, since the encoded event image data 10 preserves statistical properties, such as variance, of the event matrix it represents in an encoded and compressed way.

The encoded event image data 10 is input to the neural network unit 16 which is configured to implement a neural network for event image processing. The neural network for event image processing includes an additional first layer which adapts the size of the encoded event image data to an event matrix size used as input for the neural network. The neural network is trained to perform event image processing (which may be application specific). The neural network unit 16 is an example of a circuitry for event image processing.

As mentioned above, event image processing may be or may include object/pattern classification, gesture recognition, hand tracking, visual odometry, object tracking, anomaly detection in production lines, image enhancement such as deblurring or slow motion video or the like.

Fig. 2 schematically illustrates an embodiment of an event matrix 17, dictionary data 18 and encoded event image data 10 which may be used in the encoding unit 8 of Fig. 1.

The encoding unit 8 obtains a set of events {^ei> _> ^eM}, wherein e_k = (¾< k’ Pk) for k = 1, ... , M, wherein M is the number of generated events, x_k is the row index of the event image pixel that generated the event k, y_k is the column index of the event image pixel that generated the event k, and p_k is the polarity of the event k, wherein the polarity p_k is +1 or -1.

The set of events {^ei_{> >} ^eM} represents the event matrix 17 (E) which includes, for example,

M = 11 events (of course, the number of events is not limited to any specific number of events). Other entries of the event matrix 17 are zero (not shown for illustration purposes only).

Of course, the size of the event matrix 17, which may correspond to the size of the event image pixel array 3 of Fig. 1, is only for illustration purposes and the present disclosure is not limited to any specific size of the event matrix 17.

The dictionary data example for dictionary data of one event image pixel that generated the event k. The dictionary data 18 has a size smaller than the event matrix 17 such that a dense and compressed representation of the event matrix 17 is generated. The encoded event image data 10 (C) has the same size as the dictionary data 18 and is generated according to:

The encoded event image data 10 (C) has a maximum possible value of positive eleven and a minimum possible value of minus eleven, since the exemplarily number of events is M = 11.

Thereby, a dense and compressed representation (encoded event image data 10 (C)) of the event matrix 17 is generated.

Fig. 3 schematically illustrates in a flow diagram a first embodiment of a method 20 for encoding events.

The method may be performed by the circuitry 2 of Fig. 1.

At 21, a stream of events 21 is generated based on a plurality of event image pixels.

At 22, the events in the stream of events are encoded by determining, for each event, a product of a polarity of the event and dictionary data associated with the event image pixel that generated the event and by generating encoded event image data by summation of the products.

At 23, it is checked whether a trigger condition for output of the encoded event image data is met. The trigger condition may be met when a number of generated events is within a predetermined range since a last output of encoded event image data or when a predetermined time elapsed since a last output of encoded event image data or a combination thereof.

When the trigger condition is met, the encoded event image data is output at 24 and decoded at 25 at a receiver.

When the trigger condition is not met, further events are accumulated in the encoded event image data until the trigger condition is met.

Optionally, the encoded event image data may be entropy encoded before output.

Fig. 4 schematically illustrates in a block diagram a first embodiment of a data transfer control.

The circuitry 2 (of Fig. 1) generates first events in a first predetermined time interval representing a first event matrix El, second events in a consecutive second predetermined time interval representing a second event matrix E2, and third events in a consecutive third predetermined time interval representing a third event matrix E3.

The first, second and third event matrix are shown in the circuitry 2 for illustration purposes only, since they are typically not held in the memory of the circuitry 2 due to limited storage capacities. In this embodiment, the average event generation rate is above a predetermined rate threshold such that the number of first events, second events and third events is above half of the compression limit.

The circuitry 2 accumulates the first events in first encoded event image data Cl, the second events in second encoded event image data C2, and the third events in third encoded event image data C3.

In some embodiments, the circuitry 2 may hold the first encoded event image data Cl, the second encoded event image data C2 and the third encoded event image data C3 in a buffer. In other embodiments, the circuitry may output the first encoded event image data Cl, the second encoded event image data C2 and the third encoded event image data C3 once they are generated (e.g., due to limited storage capacities).

The circuitry 2 outputs a stream of encoded event image data including the first encoded event image data Cl, the second encoded event image data C2, and the third encoded event image data C3, wherein the first encoded event image data Cl is output first, then the second encoded event image data C2, and then the third encoded event image data C3.

Hence, each encoded event image data (Cl, C2, C3) represent one event matrix (El, E2, E3) corresponding to one pseudo-frame.

As the number of accumulated events in each encoded event image data (Cl, C2, C3) is more than half the compression limit, compression and data transfer efficiency are optimized.

Moreover, as each encoded event image data (Cl, C2, C3) represent one event matrix (El, E2, E3), each event matrix (El, E2, E3) can be approximately reconstructed by decoding the respective encoded event image data (Cl, C2, C3).

Fig. 5 schematically illustrates in a block diagram a second embodiment of a data transfer control.

The circuitry 2 (of Fig. 1) generates first events in a first predetermined time interval representing a first event matrix E4, second events in a consecutive second predetermined time interval representing a second event matrix E5, and third events in a consecutive third predetermined time interval representing a third event matrix E6.

The first, second and third event matrix are shown in the circuitry 2 for illustration purposes only, since they are typically not held in the memory of the circuitry 2 due to limited storage capacities.

In this embodiment, the average event generation rate is below a predetermined rate threshold such that the number of first events, second events and third events is below half of the compression limit. The circuitry 2 accumulates the first events in first encoded event image data C4, the second events in second encoded event image data C5, and the third events in third encoded event image data C6.

However, as the number of accumulated events in each encoded event image data (Cl, C2, C3) is less than half the compression limit, compression and data transfer efficiency may not be optimized.

Hence, the circuitry 2 combines the second encoded event image data C5 and the third encoded event image data C6 by summation. Of course, the present disclosure is not limited to the combination of two event images and multiple event images may be combined, for example, three or more (which may be based on the event generation rate).

In other embodiments, the second and third events are combined in the generation process of the combined encoded event image data C+, rather than combining the already generated encoded event image data C5 and C6.

In some embodiments, the circuitry 2 may hold the first encoded event image data C4, the second encoded event image data C5 and the third encoded event image data C6 (and/ or the combined encoded event image data C+ instead of the second encoded event image data C5 and the third encoded event image data C6) in a buffer. In other embodiments, the circuitry may output the first encoded event image data C4 and the combined encoded event image data C+ once they are generated (e.g., due to limited storage capacities).

Thus, the combination may improve compression and data transfer efficiency.

The circuitry 2 outputs a stream of encoded event image data including only the first encoded event image data C4 and combined encoded event image data C+ (sum of C5 and C6), wherein the first encoded event image data C4 is output first and then the combined encoded event image data C+.

However, for the combined encoded event image data C+, only the sum of the event matrices E5 and E6 can be approximately reconstructed. Effectively, in such embodiments, a temporal resolution is lowered, since every event image (event matrix) has a timestamp and by adding two of them together, the event image sampling rate (pseudo frame rate) is basically doubled.

Hence, there is a trade-off between compression and data transfer efficiency and event matrix reconstruction.

Fig. 6 schematically illustrates in a block diagram a third embodiment of a data transfer control.

This embodiment discusses an alternative solution to Fig. 5 for the combination of two sets of events generated in consecutive predetermined time intervals.

The first, second and third event are shown in the circuitry 2 for illustration purposes only, since they are typically not held in the memory of the circuitry 2 due to limited storage capacities.

In this embodiment, the average event generation rate is below a predetermined rate threshold such that the number of first events, second events and third events is below half of the compression limit.

The circuitry 2 accumulates the first events in first encoded event image data C4, the second events in second encoded event image data C5, and the third events in third encoded event image data C6.

Hence, the circuitry 2 combines a first portion of the second encoded event image data C5 and a second portion, different than the first portion, of the third encoded event image data C6 by stacking. For example, by stacking an upper half (first portion) of the second encoded image data C5 and a lower half (second portion) of the third encoded image data C6.

In other embodiments, the second and third events are stacked in the generation process of the stacked encoded event image data CS, rather than stacking the already generated encoded event image data C5 and C6.

In such embodiments, when determining the products, a first portion of each dictionary data used for the first events is used and a second portion, different from the first portion, of each dictionary data used for the second events is used.

Of course, the present disclosure is not limited to stacking of two encoded event image data (C5, C6), since more than two may be stacked. For example, four may be stacked by using different quarters of encoded event image data or dictionary data.

Thus, compression and data transfer efficiency may be improved. Moreover, as individual properties of the second and third encoded image data C5 and C6 are maintained by the stacking, both individual event matrices E5 and E6 can be approximately reconstructed.

In some embodiments, the circuitry 2 may hold the first encoded event image data C4, the second encoded event image data C5 and the third encoded event image data C6 (and/ or the stacked encoded event image data CS instead of the second encoded event image data C5 and the third encoded event image data C6) in a buffer. In other embodiments, the circuitry may output the first encoded event image data C4 and the stacked encoded event image data CS once they are generated (e.g., due to limited storage capacities).

The circuitry 2 outputs a stream of encoded event image data including only the first encoded event image data C4 and stacked encoded event image data CS (stacking of C5 and C6), wherein the first encoded event image data C4 is output first and then the stacked encoded event image data CS.

Fig. 7 schematically illustrates in a flow diagram an embodiment of a processing unit such as the processing unit 6 of Fig. 1.

The circuitry 2 generates an event stream based on the plurality of event image pixels 4.

The events are input to a dictionary data generator 31 (D generator). The events may be input iteratively such as individually, as a row, part of a row, multiple rows or columns etc., which may be based on, for example, a trade-off between memory, speed and power.

The dictionary data generator 31, determines, for each event, a product of a polarity of the event and dictionary data associated with the event image pixel that generated the event.

For the determination of the products, the dictionary data generator 13 obtains parameters for the pseudo-random generation of the values +1 and -1 of the dictionary data from a look-up table 32. Flence, in this embodiment, the circuitry 2 (the processing unit 6) is configured to generate the dictionary data when the dictionary data is required in the determination of the products (“on the fly”). In other embodiments, the dictionary data is precomputed and stored in the look-up table 32.

The dictionary data is generated based on a seed specific for an event image pixel that generated the event (however, other initialization strategies such as discrete Fourier coefficients may be used alternatively). Moreover, the dictionary data generator 31 selects a number of bits used in the dictionary data and, thus, in encoded event image data (C).

The dictionary data generator 31 outputs the products which are summed up iteratively by a summation unit 33 to generate the encoded event image data.

An encoded event image data memory 34 initializes encoded event image data with zeros when encoded event image data is output.

The summation unit 33 obtains the initialized encoded event image data and iteratively sums up the products in the encoded event image data.

The transfer control unit 35 has an event counter such that encoded event image data is output when the compression limit is reached. Moreover, the transfer control unit 35 may have a timer such that encoded event image data is output after a predetermined time interval. Hence, the transfer control unit 35 triggers output of encoded event image data. Furthermore, the transfer control unit 35 may send stacking parameters to the dictionary data generator 31 for stacking of encoded event image data.

Optionally, the summation unit 33 provides the encoded event image data to an entropy encoder 36 which is configured to entropy encode the encoded event image data. The entropy encoder obtains parameters for entropy encoding from a look-up table 38.

Once a trigger is output by the transfer control unit 35, a data transfer unit 38 outputs the encoded event image data (which may optionally be entropy encoded) .

The procedure is repeated continuously such that a stream of encoded event image data is output.

Fig. 8 schematically illustrates in a flow diagram a second embodiment of a method 100 for encoding events.

The method may be performed by the circuitry 2 of Fig. 1.

At 101, for each event generated based on a plurality of event image pixels, a product of a polarity of the event and dictionary data associated with the event image pixel that generated the event is determined, as discussed herein.

At 102, encoded event image data is generated by summation of the products, as discussed herein.

At 103, the dictionary data is generated when the dictionary data is required in the determination of the product, as discussed herein.

At 104, optionally, a number of bits used in the encoded event image data is selected, as discussed herein.

At 105, the encoded event image data is entropy encoded, as discussed herein.

At 106, a stream of encoded event image data is output, as discussed herein.

At 107, encoded event image data is output when a number of generated events is within a predetermined range since a last output of encoded event image data, as discussed herein.

At 108, encoded event image data is output when a predetermined time elapsed since a last output of encoded event image data, as discussed herein.

It should be recognized that the embodiments describe methods with an exemplary ordering of method steps. The specific ordering of method steps is however given for illustrative purposes only and should not be construed as binding.

Please note that the division of the circuitry 2 into units 6 to 9 is only made for illustration purposes and that the present disclosure is not limited to any specific division of functions in specific units. For instance, the circuitry 2 could be implemented by a respective programmed processor, field programmable gate array (FPGA) and the like.

All units and entities described in this specification and claimed in the appended claims can, if not stated otherwise, be implemented as integrated circuit logic, for example on a chip, and functionality provided by such units and entities can, if not stated otherwise, be implemented by software.

In so far as the embodiments of the disclosure described above are implemented, at least in part, using software-controlled data processing apparatus, it will be appreciated that a computer program providing such software control and a transmission, storage or other medium by which such a computer program is provided are envisaged as aspects of the present disclosure.

Note that the present technology can also be configured as described below.

(1) A circuitry for encoding events, including a plurality of event image pixels, wherein the circuitry is configured to: generate events based on the plurality of event image pixels; determine, for each event, a product of a polarity of the event and dictionary data associated with the event image pixel that generated the event; and generate encoded event image data by summation of the products.

(2) The circuitry of (1), wherein the circuitry is further configured to output a stream of encoded event image data.

(3) The circuitry of (2), wherein, when an average event generation rate is above a predetermined rate threshold, each encoded event image data in the stream are based on events generated within an associated predetermined time interval.

(4) The circuitry of (2) or (3), wherein, when an average event generation rate is below a predetermined rate threshold, one or more of the encoded event image data in the stream are based on a combination of first events generated within a first predetermined time interval and of second events generated within a consecutive second predetermined time interval.

(5) The circuitry of (4), wherein the combination is based on using, when determining the products, a first portion of each dictionary data used for the first events and a second portion, different from the first portion, of each dictionary data used for the second events.

(6) The circuitry of anyone of (1) to (5), wherein the circuitry is further configured to output encoded event image data when a number of generated events is within a predetermined range since a last output of encoded event image data. (7) The circuitry of anyone of (1) to (6), wherein the circuitry is further configured to output encoded event image data when a predetermined time elapsed since a last output of encoded event image data.

(8) The circuitry of anyone of (1) to (7), wherein the circuitry is further configured to entropy encode the encoded event image data.

(9) The circuitry of anyone of (1) to (8), wherein the dictionary data include pseudo-randomly generated values +1 and -1 and/or wherein the circuitry is further configured to generate the dictionary data when the dictionary data is required in the determination of the products.

(10) A circuitry for event image processing, wherein the circuitry is configured to: receive encoded event image data; input the encoded event image data into a neural network, wherein the neural network includes a layer which adapts a size of the encoded event image data to an event matrix size; and perform event image processing based on the neural network.

(11) A method for encoding events, the method including: determining, for each event generated based on a plurality of event image pixels, a product of a polarity of the event and dictionary data associated with the event image pixel that generated the event; and generating encoded event image data by summation of the products.

(12) The method of (11), further including outputting a stream of encoded event image data. (13) The method of (12), wherein, when an average event generation rate is above a predetermined rate threshold, each encoded event image data in the stream are based on events generated within an associated predetermined time interval.

(14) The method of (12) or (13), wherein, when an average event generation rate is below a predetermined rate threshold, one or more of the encoded event image data in the stream are based on a combination of first events generated within a first predetermined time interval and of second events generated within a consecutive second predetermined time interval.

(15) The method of (14), wherein the combination is based on using, when determining the products, a first portion of each dictionary data used for the first events and a second portion, different from the first portion, of each dictionary data used for the second events. (16) The method of anyone of (11) to (15), further including outputting encoded event image data when a number of generated events is within a predetermined range since a last output of encoded event image data. (17) The method of anyone of (11) to (16), further including outputting encoded event image data when a predetermined time elapsed since a last output of encoded event image data.

(18) The method of anyone of (11) to (17), further including entropy encoding the encoded event image data. (19) The method of anyone of (11) to (18), wherein the dictionary data include pseudo -randomly generated values +1 and -1 and/ or further including generating the dictionary data when the dictionary data is required in the determination of the product.

(20) A method for event image processing, the method including: receiving encoded event image data; inputting the encoded event image data into a neural network, wherein the neural network includes a layer which adapts a size of the encoded event image data to an event matrix size; and performing event image processing based on the neural network.

(21) A computer program comprising program code causing a computer to perform the method according to anyone of (11) to (19), when being carried out on a computer. (22) A non-transitory computer-readable recording medium that stores therein a computer program product, which, when executed by a processor, causes the method according to anyone of (11) to (19) to be performed.

(23) A computer program comprising program code causing a computer to perform the method according to (20), when being carried out on a computer. (24) A non-transitory computer-readable recording medium that stores therein a computer program product, which, when executed by a processor, causes the method according to (20) to be performed.

Claims

1. A circuitry for encoding events, comprising a plurality of event image pixels, the circuitry being configured to: generate events based on the plurality of event image pixels; determine, for each event, a product of a polarity of the event and dictionary data associated with the event image pixel that generated the event; and generate encoded event image data by summation of the products.

2. The circuitry according to claim 1, wherein the circuitry is further configured to output a stream of encoded event image data.

3. The circuitry according to claim 2, wherein, when an average event generation rate is above a predetermined rate threshold, each encoded event image data in the stream are based on events generated within an associated predetermined time interval.

4. The circuitry according to claim 2, wherein, when an average event generation rate is below a predetermined rate threshold, one or more of the encoded event image data in the stream are based on a combination of first events generated within a first predetermined time interval and of second events generated within a consecutive second predetermined time interval.

5. The circuitry according to claim 4, wherein the combination is based on using, when determining the products, a first portion of each dictionary data used for the first events and a second portion, different from the first portion, of each dictionary data used for the second events.

6. The circuitry according to claim 1, wherein the circuitry is further configured to output encoded event image data when a number of generated events is within a predetermined range since a last output of encoded event image data.

7. The circuitry according to claim 1, wherein the circuitry is further configured to output encoded event image data when a predetermined time elapsed since a last output of encoded event image data.

8. The circuitry according to claim 1, wherein the circuitry is further configured to entropy encode the encoded event image data.

9. The circuitry according to claim 1, wherein the dictionary data include pseudo-randomly generated values +1 and -1.

10. A circuitry for event image processing, the circuitry being configured to: receive encoded event image data; input the encoded event image data into a neural network, wherein the neural network includes a layer which adapts a size of the encoded event image data to an event matrix size; and perform event image processing based on the neural network.

11. A method for encoding events, the method comprising: determining, for each event generated based on a plurality of event image pixels, a product of a polarity of the event and dictionary data associated with the event image pixel that generated the event; and generating encoded event image data by summation of the products.

12. The method according to claim 11, further comprising outputting a stream of encoded event image data.

13. The method according to claim 12, wherein, when an average event generation rate is above a predetermined rate threshold, each encoded event image data in the stream are based on events generated within an associated predetermined time interval.

14. The method according to claim 12, wherein, when an average event generation rate is below a predetermined rate threshold, one or more of the encoded event image data in the stream are based on a combination of first events generated within a first predetermined time interval and of second events generated within a consecutive second predetermined time interval.

15. The method according to claim 14, wherein the combination is based on using, when determining the products, a first portion of each dictionary data used for the first events and a second portion, different from the first portion, of each dictionary data used for the second events.

16. The method according to claim 11, further comprising outputting encoded event image data when a number of generated events is within a predetermined range since a last output of encoded event image data.

17. The method according to claim 11, further comprising outputting encoded event image data when a predetermined time elapsed since a last output of encoded event image data.

18. The method according to claim 11, further comprising entropy encoding the encoded event image data.

19. The method according to claim 11, wherein the dictionary data include pseudo -randomly generated values +1 and -1.

20. A method for event image processing, the method comprising: receiving encoded event image data; inputting the encoded event image data into a neural network, wherein the neural network includes a layer which adapts a size of the encoded event image data to an event matrix size; and performing event image processing based on the neural network.