WO2022264405A1 - Signal processing device, signal processing method, and program - Google Patents

Abstract

Provided is a signal processing device comprising a data correction unit for correcting at least one event signal among a series of event signals outputted from an event-driven vision sensor that includes a plurality of sensors constituting a sensor array and including the position information of each sensor in the sensor array, the polarity of a light intensity change, and a time stamp, on the basis of at least one of a temporal pattern specified by the time stamp of the series of event signals and a spatial pattern specified by the position information of the series of event signals.

Description

SIGNAL PROCESSING DEVICE, SIGNAL PROCESSING METHOD, AND PROGRAM

The present invention relates to a signal processing device, signal processing method and program.

An event-driven vision sensor is known, in which pixels that detect changes in the intensity of incident light generate signals asynchronously with time. An event-driven vision sensor is advantageous in that it can operate at low power and at high speed compared to a frame-type vision sensor that scans all pixels at predetermined intervals, specifically image sensors such as CCD and CMOS. is. Techniques related to such an event-driven vision sensor are described in Patent Document 1 and Patent Document 2, for example.

Japanese Patent Publication No. 2014-535098 JP 2018-85725 A

Here, with event-driven vision sensors, for example, due to differences in the characteristics of each sensor, afterimages and noise that do not occur in actual light intensity changes may occur. However, no technology has yet been proposed for correcting such afterimages and noise.

Therefore, an object of the present invention is to provide a signal processing device, a signal processing method, and a program capable of effectively correcting afterimages, noise, etc. in the output of an event-driven vision sensor.

According to one aspect of the invention, the output from an event-driven vision sensor comprising a plurality of sensors forming a sensor array includes the position information of each sensor within the sensor array, the polarity of the light intensity change and a time stamp. At least one event signal in the series of event signals is in at least one of a temporal pattern identified by time stamps of the series of event signals and/or a spatial pattern identified by location information of the series of event signals. A signal processing apparatus is provided that includes a data corrector that corrects based on the data.

According to another aspect of the present invention, output from an event-driven vision sensor including a plurality of sensors constituting a sensor array, position information of each sensor in the sensor array, polarity of light intensity change, and time stamp. At least one event signal in the series of event signals comprising at least one of a temporal pattern identified by time stamps of the series of event signals and/or a spatial pattern identified by location information of the series of event signals. A signal processing method is provided that includes correcting based on .

According to yet another aspect of the present invention, output from an event-driven vision sensor including a plurality of sensors forming a sensor array, position information of each sensor in the sensor array, polarity of light intensity change and time stamp at least one event signal in the sequence of event signals comprising at least one of a temporal pattern identified by time stamps of the sequence of event signals and/or a spatial pattern identified by location information of the sequence of event signals A program is provided for causing a computer to implement a function of correcting based on the above.

FIG. 1 is a diagram showing a schematic configuration of a system according to one embodiment of the present invention. FIG. 4 is a diagram showing an example of a correction value table in one embodiment of the present invention; FIG. 4 is a flow chart showing an example of processing for correcting an event signal in one embodiment of the present invention; It is a figure which shows the example of the preparation method of the correction value table in one Embodiment of this invention. 6 is a flowchart showing an example of processing for creating a correction value table in one embodiment of the present invention; FIG. 4 is a diagram for explaining a specific example of a method of correcting an event signal; FIG. FIG. 4 is a diagram for explaining a specific example of a method of correcting an event signal; FIG.

Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. In the present specification and drawings, constituent elements having substantially the same functional configuration are denoted by the same reference numerals, thereby omitting redundant description.

FIG. 1 is a diagram showing a schematic configuration of a system according to one embodiment of the present invention. In the illustrated example, the system 10 includes a vision sensor 100 which is an Event Driven Sensor (EDS) and a signal processor 200 .

Vision sensor 100 includes a sensor array 120 made up of a plurality of sensors 110 corresponding to pixels of an image, and processing circuitry 130 coupled to sensor array 120 . The sensor 110 includes a light-receiving element and generates an event signal upon detecting a change in the intensity of incident light, more specifically a change in luminance. In sensor array 120, sensors 110 are arranged in two orthogonal directions (illustrated as x-direction and y-direction). For example, event signals are read out from sensor 110 according to addresses generated by an address generator included in processing circuitry 130 for the x and y directions, respectively. Here, if the sensor 110 at a certain address does not detect an event, the event signal is not read even if reading is executed, and reading of the next address is executed. Therefore, the event signals output from the vision sensor 100 are time asynchronous. The event signal (X, Y, P, t) output from the vision sensor 100 includes the positional information X, Y of the sensor 110 in the sensor array 120 and the polarity P (p: positive or n: negative) of the luminance change. , and timestamp t.

The signal processing device 200 includes a communication interface 210, a buffer memory 220, a processing circuit 230, and a storage section 240. Communication interface 210 receives event signals from processing circuitry 130 of vision sensor 100 . The received event signal is temporarily stored in buffer memory 220 . The processing circuit 230 operates, for example, according to a program stored in the storage section 240 and processes event signals read from the buffer memory 220 . In this embodiment, the processing circuitry 230 includes a data corrector 231 that corrects the event signal as described below. Based on the event signal corrected by the data correction unit 231, the processing circuit 230 generates, in time series, an image mapping the position where the luminance change occurs, and temporarily or permanently stores it in the storage unit 240. Alternatively, processing circuitry 230 may transmit the event signal to yet another device via communication interface 210 . Also, the processing circuit 230 may further include an illumination intensity detector 232 . The storage unit 240 stores a correction value table 241 referenced by the data correction unit 231 . The configuration of each unit will be further described below.

The data correction unit 231 corrects the event signal temporarily stored in the buffer memory 220 based on at least one of the temporal pattern and the spatial pattern of the event signal defined in the correction value table 241. do. More specifically, the data correction unit 231 corrects a series of event signals in which at least one of position information and time stamps satisfies a predetermined relationship, a temporal pattern specified by the time stamps of the series of event signals, Alternatively, when the spatial pattern specified by the position information of the series of event signals fits the pattern defined in the correction value table 241, at least one of the series of event signals is corrected.

For example, the data correcting unit 231 may be configured based on a temporal pattern specified by time stamps of a series of event signals having position information that satisfies a predetermined relationship with the position information of at least one event signal to be processed. , at least one event signal may be corrected. More specifically, when there are a plurality of event signals output from the sensors 110 with the same positional information X and Y, the data correction unit 231 corrects the temporally specified by the time stamp t and the polarity P of these event signals. corrects the event signal based on the pattern. If the temporal pattern specified by the time stamp t matches the pattern defined in the correction value table 241, the data correction unit 231 may disable the output of the specific sensor 110 in the event signal for a predetermined period of time. good. The disabled output event signal is not displayed, for example, in the image mapping the location where the luminance change occurred, nor is it transmitted to another device.

In addition, for example, the data correction unit 231 converts the spatial pattern specified by the position information of a series of event signals having time stamps that satisfy a predetermined relationship to the time stamps of at least one event signal to be processed into a spatial pattern. At least one event signal may be corrected based on. More specifically, when there are a plurality of event signals having time stamps t close to each other, the data correction unit 231 corrects the spatial pattern specified by the position information X, Y and the polarity P of the event signals. Correct the event signal based on When the spatial pattern specified by the position information X and Y matches the pattern defined in the correction value table 241, the data correction unit 231 converts the output of the specific sensor 110 in the event signal to the adjacent sensor 110. can be synchronized with the output of Here, synchronizing the outputs of a plurality of sensors 110 means, for example, changing the polarity of an event signal output from one sensor 110 to match the polarity of an event signal output from another sensor 110. Alternatively, the time stamp of the event signal output from one sensor 110 may be changed to match the time stamp of the event signal output from the other sensor 110 .

FIG. 2 is a diagram showing an example of a correction value table in one embodiment of the invention. In this embodiment, in the correction value table 241, temporal patterns and spatial patterns are defined for each position information X, Y of the sensor 110 and for each event signal polarity P(p, n). In this case, the temporal pattern and the spatial pattern differ respectively according to the position information and the polarity of the at least one event signal to be corrected. In the illustrated example, the records of the correction value table 241 are stored using the position information X, Y and the polarity P of the sensor 110 as keys. For example, for the sensor 110 with position information (X, Y)=(x ₁ , y ₁ ), patterns are defined for the polarity P=p and for the polarity P=n. Each sensor 110 constituting the sensor array 120 has different characteristics, and the characteristics may differ even when the polarities of p and n are detected. It is possible to correct the event signal more appropriately according to the difference in characteristics of each. As shown, multiple patterns (pattern 1 through pattern n) may be defined for the same position information and polarity.

Specifically, for example, in the record whose key is (x ₁ , y ₁ , p), “repeated output with a period T less than ₁ ” is defined as “pattern 1”, and “first Invalidate other than p output" is defined. According to this definition, the data correction unit 231, for example, the event signal (x ₁ , y ₁ , p, t ₁ ), that is, the sensor 110 of the position information (X, Y)=(x ₁ , y ₁ ) at the time t ₁ Event signals (x ₁ , y ₁ , n, t ₂ ) and event signals (x ₁ , y ₁ , p, t ₃ ) are acquired after the event signal indicating that a luminance change with polarity P=p was detected. If t ₃ −t ₁ <T ₁ , it is determined that a pattern of “repeated output with less than period T ₁ ” occurs, and event signals at times t ₂ and t ₃ are invalidated. Such correction, as in an example described later, prevents afterimages caused by the sensor 110 repeatedly detecting luminance changes multiple times when there is actually only one luminance change. can be effective for The period T1 functions as _a threshold value for determining whether the repeated output is due to the characteristics of the sensor or whether luminance changes actually occur multiple times.

For a record whose key is (x ₁ , y ₁ , n), that is, the same position information as the first record but a different polarity, "pattern 1" is defined as "repeated output with period T less than ₂ ", and "correction "Method 1" is defined as "invalidating all but the first n outputs". If the sensor 110 with the same position information (X, Y) produces repeated outputs for both polarities P=p, n, patterns are thus defined for each of the polarities P=p, n. The periods T ₁ and T ₂ may have the same value (T ₁ =T ₂ ), or may have different values (T ₁ ≠T ₂ ) based on actual measurement results as described later.

On the other hand, in the record whose key is (x ₂ , y ₂ , p), "p output at only 8 pixels out of 9 adjacent pixels" is defined as "pattern 1", are all p-outputs" is defined. According to this definition, the data correction unit 231 selects eight sensors 110 out of nine adjacent pixels (constituting a 3×3 area) sensors 110 including position information (X, Y)=(x ₂ , y ₂ ). , the output of the remaining sensor 110 is synchronized with the outputs of the eight sensors 110 when the output of the remaining sensor 110 is not synchronized while the event signal of the same polarity P=p is output. Specifically, when the remaining one sensor 110 does not output an event signal or outputs an event signal with polarity P=n, the data correction unit 231 changes the output of the corresponding sensor 110 to the polarity P=p event signal. Alternatively, if an event signal with polarity P=p is output at a later time stamp, the time stamp of that event signal is rewritten according to the time stamp of the event signal of the other sensor 110 . Such correction, as in an example described later, actually causes a change in brightness, and the output of the sensor 110 that does not detect a change in brightness even though a change in brightness is detected by an adjacent sensor is the output of the vision sensor. It can be useful to prevent going into noise in the 100 output.

It should be noted that in the example shown in FIG. 2, for sensors 110 with position information (X, Y)=(x ₃ , y ₃ ) and (X, Y)=(x ₄ , y ₄ ), polarity P=p , n, no pattern requiring correction occurs, so no record is recorded. The data correction unit 231 skips correction processing for these event signals from the sensors 110 because there is no key in the correction value table 241 . The format of the correction value table 241 shown in FIG. 2 is an example, and the data recording format is not particularly limited as long as the definition of the temporal or spatial pattern of the event signal is associated with the correction method.

Referring to FIG. 1 again, the illumination intensity detection unit 232 detects illumination intensity in the environment of the sensor array 120 using the illumination sensor 140 arranged near the sensor array 120 . If the illumination intensity detection section 232 is present, the data correction section 231 corrects the output value of the vision sensor 100 based on the correction value table 241 defined for each illumination intensity detected by the illumination intensity detection section 232 . Specifically, for example, a plurality of correction value tables 241 may be prepared for each illumination intensity range, and the records of the correction value table 241 as shown in FIG. (X, Y, P, L) to which the illumination intensity range L is added may be recorded as a key.

FIG. 3 is a flow chart showing an example of processing for correcting an event signal in one embodiment of the present invention. In the illustrated example, when an event signal is received by the communication interface 210 (step S101), the data correction unit 231 creates a correction value table using position information (X, Y) and polarity P of the received event signal as keys. 241 is searched (step S102). If there is a record in the correction value table 241 that matches the key, that is, a pattern definition for the acquired event signal position information (X, Y) and the polarity P (step S103), the data correction unit 231 receives For the event signal, it is determined whether or not correction is necessary based on the pattern.

Specifically, the data correction unit 231 reads from the buffer memory an event signal related to the pattern defined in the correction value table 241 (step S104). For example, if a temporal pattern is defined for the position information (X, Y) and the polarity P of the event signal, the data correction unit 231 detects the time-series event signals obtained by the sensors with the same position information (X, Y). is read from the buffer memory. Further, when a spatial pattern is defined for the position information (X, Y) and the polarity P of the event signal, the data correction unit 231 has a predetermined relationship with the position information (X, Y), for example, adjacent positions The event signal acquired by the information sensor is read out from the buffer memory.

Further, the data correction unit 231 determines whether or not the event signal received in step S101 and the event signal additionally read in step S104 apply to the pattern defined in the correction value table 241 ( step S105). If the event signal fits the defined pattern, the data correction unit 231 corrects the event signal according to the correction method associated with the pattern in the correction value table 241 (step S106). Specifically, for example, the data correction unit 231 performs correction by invalidating a specific event signal or rewriting the event signal as described above.

FIG. 4 is a diagram showing an example of a method of creating a correction value table according to one embodiment of the present invention. The correction value table 241 described in the above example is created as part of the calibration procedure of the vision sensor 100, for example. In the illustrated example, the vision sensor 100 is attached to a two-axis camera platform 501, and the vision sensor 100 is moved using the camera platform 501 with the test pattern 502 at least partially arranged within the angle of view of the vision sensor 100. move. At this time, the reflectance in each area of the test pattern 502, the positional relationship between the vision sensor 100 and the test pattern 502, and the movement speed of the vision sensor 100 by the platform 501 are known, and the vision sensor calculated based on these is known. 100 ideal output values 521 are stored in the storage unit 520 of the computing device 500 . The correction value determiner 510 of the computing device 500 compares the event signal received from the vision sensor 100 with the ideal output value 521 to respond to the temporal or spatial pattern of the event signal and the pattern. A correction method is specified and stored in the storage unit 520 as a correction value table 241 .

As described above, when the correction value table 241 is defined for each illumination intensity, the illumination intensity-adjustable illumination 503 is arranged when the correction value table 241 is created, and the illuminance sensor 140 similar to that shown in FIG. By detecting the illumination intensity in the environment of the sensor array 120 at , a correction value table 241 that differs according to the illumination intensity is created.

FIG. 5 is a flowchart showing an example of processing for creating a correction value table in one embodiment of the present invention. In the illustrated example, the correction value determining unit 510 of the arithmetic device 500 compares the ideal output value and the actually generated event signal for each of the sensors 110 that make up the sensor array 120 of the vision sensor 100. (Step S201), it is determined whether or not correction is necessary (Step S202). For example, if the detection of the luminance change is delayed for a predetermined time or more from the ideal timing, if the luminance change that should have been detected is not detected, or if the luminance change that should not have been detected is detected, correction is not necessary. It is determined that there is If correction is necessary, the correction value determination unit 510 recognizes the temporal or spatial pattern of the target event signal (step S203). If a temporal or spatial pattern can be recognized for the event signal requiring correction (step S204), the correction value determination unit 510 converts the recognized pattern and event signal into an ideal output value. A correction method for making the distance closer is recorded in the correction value table 241 (step S205).

6 and 7 are diagrams for explaining a specific example of the event signal correction method. In the example shown in FIG. 6, position information (X, Y) ₌ ( x ₁ , y ₁ ) sensor (sensor A) and position information (X, Y)=(x ₂ , y ₂ ) sensor (sensor B) event signals are shown. Sensor A outputs an event signal (x ₁ , y ₁ , p, t ₁ ) at time t ₁ after time t ₀ and has not output an event signal thereafter. On the other hand, sensor B outputs event signals (x ₁ , y ₁ , p, t ₁ ) at time t ₁ and then outputs event signals (x ₁ , y ₁ , n, t ₂ ) at time t ₂ . Furthermore, at time t3, event signals (x ₁ , y ₁ , p _{, t 3 )} are output. In this case, for the sensor B, a repetitive output in which the polarity P is output in the order p, n, p is identified as the temporal pattern to be corrected. Specifically, for example, a cycle T ₁ is set such that t ₃ −t ₁ <T ₁ , and “repeated output less than cycle T1” is set as a temporal pattern in the case of sensor B with polarity P=p. may be defined, and the correction method may be defined as "invalidating all but the first p outputs".

On the other hand, in the example shown in FIG. 7, 9 adjacent pixels (constituting a 3×3 area) included in the area on the sensor array 120 where the luminance change occurs due to the movement of the test pattern 502 shown in FIG. The event signal of sensor 110 is shown. In this example, the sensor D at the right end of the region does not output an event signal, although the event signal with the polarity P=p is output by the sensors of eight pixels out of the nine adjacent pixels. In this case, for the sensor E adjacent to the sensor D, eight of the nine adjacent pixels output an event signal with a polarity of P=p, and the remaining one sensor outputs an event signal with a polarity of P=p. are identified as spatial patterns to be corrected. Specifically, for example, when the sensor E has the polarity P=p, the spatial pattern is defined as ``p output from only 8 pixels out of 9 adjacent pixels'', and the correction method is ``p output for all adjacent 9 pixels. may be defined as "to".

According to one embodiment of the present invention as described above, in the event signal, a temporal or spatial pattern is defined for each sensor position information and polarity, and when there is an output that fits the pattern, Event signals are corrected. This allows, for example, to disable repetitive outputs with polarities that differ from the actual luminance changes in the event signal, or to synchronize the output of a sensor whose output is out of sync with adjacent sensors to an adjacent sensor. For example, by defining a pattern for each sensor position information and polarity by the actual measurement procedure as shown in FIG. , it is possible to effectively correct afterimages, noise, etc. in the output of the vision sensor.

Although the preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, the present invention is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field to which the present invention belongs can conceive of various modifications or modifications within the scope of the technical idea described in the claims. It is understood that these also belong to the technical scope of the present invention.

DESCRIPTION OF SYMBOLS 10... System, 100... Vision sensor, 110... Sensor, 120... Sensor array, 130... Processing circuit, 140... Illuminance sensor, 200... Signal processing apparatus, 210... Communication interface, 220... Buffer memory, 230... Processing circuit, 231 ... data correction unit, 232 ... illumination intensity detection unit, 240 ... storage unit, 241 ... correction value table.

Claims (12)

1. At least one of a series of event signals output from an event-driven vision sensor including a plurality of sensors forming a sensor array and including position information of each sensor in the sensor array, polarity of light intensity change, and time stamp. correcting one event signal based on at least one of a temporal pattern identified by the time stamps of the sequence of event signals and/or a spatial pattern identified by the location information of the sequence of event signals. A signal processing device comprising a data corrector.
2. said sequence of event signals having said location information satisfying a predetermined relationship to said location information of said at least one event signal;
   2. The signal processing device according to claim 1, wherein said data correction unit corrects said at least one event signal based on said temporal pattern specified by said time stamps of said series of event signals.
3. said sequence of event signals having said time stamps satisfying a predetermined relationship to said time stamps of said at least one event signal;
   2. The signal processing device according to claim 1, wherein said data correction unit corrects said at least one event signal based on said spatial pattern specified by said position information of said series of event signals.
4. The signal processing device according to any one of claims 1 to 3, wherein at least one of said temporal pattern or said spatial pattern differs according to said position information of said at least one event signal.
5. The signal processing device according to any one of claims 1 to 4, wherein at least one of said temporal pattern or said spatial pattern differs according to said polarity of said at least one event signal.
6. further comprising an illumination intensity detection unit that detects illumination intensity in the environment of the sensor array;
   6. The signal processing device according to any one of claims 1 to 5, wherein at least one of said temporal pattern and said spatial pattern is defined for each illumination intensity.
7. The signal processing device according to any one of claims 1 to 6, wherein the temporal pattern includes repeated outputs of the polarity.
8. The signal processing device according to claim 7, wherein the data correction unit invalidates the repeated output of the polarity.
9. 9. The spatial pattern of any one of claims 1 to 8, wherein the spatial pattern includes a first sensor output being out of sync with a second sensor output adjacent to the first sensor. A signal processor as described.
10. The signal processing device according to claim 9, wherein the data correction unit synchronizes the output of the first sensor with the output of the second sensor.
11. At least one of a series of event signals output from an event-driven vision sensor including a plurality of sensors forming a sensor array and including position information of each sensor in the sensor array, polarity of light intensity change, and time stamp. correcting one event signal based on at least one of a temporal pattern identified by the time stamps of the sequence of event signals and/or a spatial pattern identified by the location information of the sequence of event signals. A signal processing method that includes steps.
12. At least one of a series of event signals output from an event-driven vision sensor including a plurality of sensors forming a sensor array and including position information of each sensor in the sensor array, polarity of light intensity change, and time stamp. correcting one event signal based on at least one of a temporal pattern identified by the time stamps of the sequence of event signals and/or a spatial pattern identified by the location information of the sequence of event signals. A program that makes a computer implement a function.
    

Images