CN113269683B - Local space-time event stream filtering method and system based on self-adaptive threshold - Google Patents
Local space-time event stream filtering method and system based on self-adaptive threshold Download PDFInfo
- Publication number
- CN113269683B CN113269683B CN202110435263.5A CN202110435263A CN113269683B CN 113269683 B CN113269683 B CN 113269683B CN 202110435263 A CN202110435263 A CN 202110435263A CN 113269683 B CN113269683 B CN 113269683B
- Authority
- CN
- China
- Prior art keywords
- noise
- event stream
- event
- window
- value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 73
- 238000001914 filtration Methods 0.000 title claims abstract description 41
- 230000000694 effects Effects 0.000 claims abstract description 36
- 238000010276 construction Methods 0.000 claims abstract description 18
- 238000012216 screening Methods 0.000 claims abstract description 18
- 230000003044 adaptive effect Effects 0.000 claims abstract description 17
- 230000000007 visual effect Effects 0.000 claims description 5
- 238000004364 calculation method Methods 0.000 claims description 4
- 230000036961 partial effect Effects 0.000 claims description 4
- 238000002360 preparation method Methods 0.000 claims 1
- 238000005516 engineering process Methods 0.000 abstract description 4
- 230000014759 maintenance of location Effects 0.000 description 6
- 238000005286 illumination Methods 0.000 description 4
- 230000000670 limiting effect Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000009825 accumulation Methods 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000002427 irreversible effect Effects 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000001303 quality assessment method Methods 0.000 description 1
- 230000002829 reductive effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- G06T5/70—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20024—Filtering details
Landscapes
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The present application relates to the field of filtering technologies, and in particular, to a local spatiotemporal event stream filtering method and system based on adaptive thresholds. The method comprises the following steps: a step of reading in blocks, which is to receive asynchronous event streams and read event stream blocks in a fixed time iteration mode; a space-time plane construction step, namely receiving the event stream block, accumulating the number of events in a time dimension by taking a pixel as a unit, and calculating a pixel activity value to form a two-dimensional space-time plane; a local sliding window step, namely receiving the two-dimensional space-time plane, setting a local window with a fixed size on the two-dimensional space-time plane, and calculating a window mark value of each sliding window; a noise candidate domain selecting step of receiving the window flag value, calculating a noise candidate threshold value, and selecting a noise candidate domain according to the noise candidate threshold value; and a noise screening step, namely marking the noise event, removing the noise event from the event stream block, and outputting the filtered asynchronous event stream. The method and system can effectively remove noise events in asynchronous event streams.
Description
Technical Field
The present application relates to the field of filtering technologies, and more particularly, to a local spatiotemporal event stream filtering method and system based on adaptive thresholds.
Background
The conventional camera has a problem of motion blur when capturing an object moving at a high speed. Meanwhile, the dynamic range of the traditional camera is small, and the acquisition and presentation information is limited when the traditional camera works under an extreme illumination condition. The dynamic vision sensor makes up the disadvantages of the traditional camera. The dynamic vision sensor has the advantages of high time resolution, high dynamic range, low delay, low power consumption and the like, and provides a new solution for computer vision tasks which are challenged by some traditional cameras. However, in practical scenarios, the application performance is severely affected by the large amount of noise generated by the dynamic vision sensor. Meanwhile, a developed standard image denoising algorithm cannot be directly applied to an unstructured asynchronous event stream, and therefore a special denoising method needs to be designed for an event.
With the continuous development of the hardware technology of the dynamic vision sensor, the research of the vision technology based on the dynamic vision sensor is receiving wide attention. Currently, researchers have proposed a variety of event stream filtering methods, which mainly include: the event stream filtering method based on the traditional image denoising, the event stream filtering method based on the space and the event stream filtering method based on the time.
An event stream filtering method based on traditional image denoising is commonly implemented as follows: and mapping the 3D event stream into a 2D image frame, and then carrying out denoising treatment by using a traditional image denoising method. However, the problem with these methods is that the mapping from the event stream to the frame image is irreversible, i.e. the 3D denoised event stream cannot be recovered from the denoised 2D frame image. This causes the denoised event stream to lose features, limiting subsequent applications of the event camera.
The principle of the spatial-based event stream filtering method is as follows: spatial redundancy of events is exploited to remove spatially independent event points. The common practice is as follows: for each newly arrived event, it is checked whether there is an event in the past T microseconds in 8 adjacent pixels. If not, it is determined as noise and discarded. In addition, it can be generalized to a plurality of extensions such as distinguishing polarities, distinguishing ranges, defining the number of adjacent events, and the like. The method has high running speed and good denoising effect on simple scenes, but is easy to filter out the event errors generated by small, slow-moving or small-volume objects, and meanwhile, the noise accumulation is easy to cause because the number of reference points is too small.
The principle of the event stream filtering method based on time is as follows: events that are redundant in time are removed. It is common practice for an arrival event to be retained only if the difference between the timestamp and the timestamp of the most recent event for that pixel is greater than a threshold. The method can quickly extract object features, but most events in an event stream are usually filtered, so that scene information is reduced sharply.
The present application therefore proposes an improved method and system to at least partially solve the above technical problem.
Disclosure of Invention
The technical scheme of the invention is as follows: integrating local event stream information, analyzing the distribution characteristics of events from space and time dimensions in sequence, and judging the effectiveness of the events; meanwhile, an adaptive threshold mechanism is applied to adapt to different environments, so that the filter is widely applied. Finally, on the basis of maintaining the characteristics of the asynchronous event stream data, the method not only effectively removes noisy noise information in a scene, but also retains effective events for describing object lines, achieves the purpose of improving the quality of the asynchronous event stream, and provides powerful support for the visual task based on the event stream.
In order to achieve the technical purpose, the application provides a local spatiotemporal event stream filtering method based on an adaptive threshold, which comprises the following steps:
a step of reading in blocks, which is to receive asynchronous event streams and read event stream blocks in a fixed time iteration mode;
a space-time plane construction step, namely receiving the event stream block, accumulating the number of events in a time dimension by taking a pixel as a unit, and calculating a pixel activity value to form a two-dimensional space-time plane;
a local window sliding step, namely receiving the two-dimensional space-time plane, setting a local window with a fixed size on the two-dimensional space-time plane, traversing the whole two-dimensional plane by sliding the window with a fixed step length, and calculating a window mark value of each sliding window;
a noise candidate domain selecting step of receiving the window flag value, calculating a noise candidate threshold value, and selecting a noise candidate domain according to the noise candidate threshold value;
and a noise screening step, namely receiving the event stream block, the two-dimensional space-time plane and the noise candidate domain, calculating a noise threshold value, marking a noise event, removing the noise event from the event stream block, and outputting the filtered asynchronous event stream.
Preferably, the asynchronous event stream is obtained by a dynamic visual sensor.
Specifically, the block reading step specifically includes:
s1.1, initializing an event stream block index value i =0, initializing an input event index value j =0, initializing a block internal time flag value T =0, and initializing a block reproduction flag bit flag = True;
s1.2, reading data, if the read data is empty within 20ms, recording as no event input, and switching to S1.5, otherwise, receiving an event, recording as e j ＝[x j ，y j ，t j ，p j ]Wherein x is j ，y j As an event e j Pixel coordinate of (d), t j As an event e j Time stamp of p j As an event e j Polarity of (c);
s1.3, judging whether the current block remaking zone bit is in a remaking state, if flag = True, creating a new event stream block for the current event and switching to S1.4, otherwise, judging the current event e j Whether it belongs to the event stream block E i If the current event timestamp t j If the sum of the time mark value T in the block and the fixed time interval delta T is smaller than the sum, the step is switched to S1.4 if the event belongs to the event stream block, otherwise, the step is switched to S1.5 if the event belongs to the next event stream block;
s1.4, setting the current event e j Adding to the Current event stream Block E i Increasing the event index value j by one, and turning to the step S1.3 to receive the next event;
s1.5, current event stream block E i And sending out, increasing the index value i of the event stream block by one, and turning to S1.3 to read the next event stream block, wherein the block reproduction flag bit flag = True.
The method for creating the new event stream block for the current event comprises the following steps: initializing an event stream Block E i Let us orderIntra block timestamp value T = T j And a block reproduction flag bit flag = False.
Specifically, the spatio-temporal plane construction step specifically includes:
initializing a two-dimensional spatio-temporal plane N i A size of WXH, where W and H represent the dynamic vision sensor length and width, respectively, from which the data was generated;
receiving an event stream block E i Counting the event stream block E for each pixel position i The number of events occurring at the pixel position is recorded as pixel activity value, and the event stream block E i All pixel activity values in the space-time plane form a two-dimensional space-time plane, and the pixel activity value corresponding to each pixel position (x, y) is
Wherein P is the polarity.
Specifically, the step of locally sliding the window specifically includes:
s3.1, initializing global pixel flag bit P with size of W × H i Wherein W and H represent the dynamic vision sensor length and width, respectively, of the generated data, initializing the window flag value
Wherein m and n respectively represent the abscissa and ordinate values of the upper left corner of the current local window;
s3.2, receiving the two-dimensional space-time plane N i ；
S3.3, for x is more than or equal to 0 and less than or equal to W and y is more than or equal to 0 and less than or equal to H, making
S3.4, initializing the upper left corner of a local window to be marked as (0, 0), wherein the window comprises L multiplied by L pixel units in total and is positioned on a two-dimensional space-time plane N i Upper, partial window H i Sliding from top to bottom and from left to right by step length S until traversing a complete two-dimensional space-time plane, and in the process of sliding a local window, if boundary coordinates of a certain local window are out of range, considering the local window as an invalid local windowA window, not including it in the calculation, otherwise, calculating a window flag value by calculating a window flag value
Specifically, the noise candidate domain selecting step specifically includes:
initializing a dense candidate domain threshold hyperparameter α =0.9, and a sparse candidate domain threshold hyperparameter β =0.9;
receiving the window flag value
Computing a dense noise candidate domain threshold
Wherein L is the side length of the local window;
computing sparse noise candidate domain thresholds
Wherein it is present>
Marking values by all windows
Sorting from large to small to remove zero value, K 1 Candidate domain threshold in sequence for sparse noise>
By an index subscript of->
The length of the sparse candidate domain is multiplied by the threshold value over-parameter of the sparse candidate domain and rounded upwards;
according to the threshold Th of the dense noise candidate domain dense And the sparse noise candidate domain threshold Th sparse Selecting a noise candidate domain, and for a local window with (m, n) coordinates at the upper left corner, if the window mark value of the window is
Greater than threshold Th of dense noise candidate domain dense Or less than the sparse noise candidate domain threshold Th sparse Adding the coordinates (M, n) of the upper left corner into the set M of coordinates of the upper left corner of the noise candidate domain i In (1).
Further, the noise screening step specifically comprises:
s5.1, initializing a noise threshold over-parameter gamma =0.9;
s5.2, receiving event stream block E i Receiving a two-dimensional space-time plane N i Receiving a set M of upper left corner coordinates of a noise candidate field i ；
S5.3, calculating a noise threshold value
Wherein it is present>
From a two-dimensional space-time plane N i All pixel activity value N i (x, y) is obtained by sorting from large to small and removing zero values, K 2 For noise threshold in the sequence->
Index subscript of (1), by sequence>
The length of the noise threshold value is multiplied by a noise threshold value over parameter gamma;
s5.4, according to the noise threshold Th noise Screening noise event, marking it, if it is corresponding pixel activity value N for certain pixel position (x, y) i (x, y) is less than a noise threshold Th noise The event contained in the pixel is considered as a noise event, which is added to the noise set F i Performing the following steps;
s5.5, from event stream block E i Mid-erasure noise set F i Obtaining a filtered asynchronous event stream D i ；
S5.6, filtering the asynchronous event stream D i And (6) outputting.
A second aspect of the present invention provides an adaptive threshold-based local spatiotemporal event stream filtering system, comprising:
a block reading module for executing the block reading step, receiving the asynchronous event stream and iteratively reading the event stream block at a fixed time;
the space-time plane construction module is used for executing the space-time plane construction step, receiving the event stream block, accumulating the number of events in a time dimension by taking a pixel as a unit, and calculating a pixel activity value to form a two-dimensional space-time plane;
a local sliding window module for executing the local sliding window step, receiving the two-dimensional space-time plane and setting a local window with a fixed size on the two-dimensional space-time plane, traversing the whole two-dimensional plane by a fixed step length sliding window, and calculating a window mark value of each sliding window;
a noise candidate domain selection module for executing a noise candidate domain selection step, receiving the window flag value, calculating a noise candidate threshold value, and selecting a noise candidate domain according to the noise candidate threshold value;
and the noise screening module is used for executing a noise screening step, receiving the event stream block, the two-dimensional space-time plane and the noise candidate domain, calculating a noise threshold value, marking a noise event, removing the noise event from the event stream block, and outputting the filtered asynchronous event stream.
Preferably, the asynchronous event stream is input by a dynamic visual sensor.
The beneficial effect of this application does:
the application provides a local space-time event stream filtering method and system based on an adaptive threshold, which can effectively remove noise events which can not describe the characteristics of an object in an asynchronous event stream, can keep real events which describe lines of the object, obtain a clean, clear and complete scene description from an original event stream, and realize a relatively accurate and rapid denoising effect. Moreover, the method can greatly improve the application capability of the event camera in the actual task, improve the problem of serious noise influence of the event camera in the actual scene, and enable the event camera to be better applied to challenging environments such as high-speed motion, high dynamic range and the like, provides a new idea for solving challenging computer vision problems, and has good application potential in practical problems such as automatic driving and the like.
Drawings
FIG. 1 shows a schematic flow chart of the method of embodiment 1 of the present application;
FIG. 2 is a graph showing the comparison result of the method in example 1 of the present application and other methods in a single scene;
FIG. 3 is a graph showing the average comparison result of the method in example 1 of the present application and other methods in all scenes;
FIG. 4 is a graph showing the comparison of information retention capacity between the method of example 1 and other methods;
fig. 5 shows a system configuration diagram of embodiment 2 of the present application.
Detailed Description
Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. It should be understood that the description is intended to be exemplary only, and is not intended to limit the scope of the present application. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present application. It will be apparent to one skilled in the art that the present application may be practiced without one or more of these details. In other instances, well-known features of the art have not been described in order to avoid obscuring the present application.
It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments according to the present application. As used herein, the singular is intended to include the plural unless the context clearly dictates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
Exemplary embodiments according to the present application will now be described in more detail with reference to the accompanying drawings. These exemplary embodiments may, however, be embodied in many different forms and should not be construed as limited to only the embodiments set forth herein. The figures are not drawn to scale, wherein certain details may be exaggerated and some details may be omitted for clarity of presentation. The shapes of various regions, layers, and relative sizes and positional relationships therebetween shown in the drawings are merely exemplary, and deviations may occur in practice due to manufacturing tolerances or technical limitations, and a person skilled in the art may additionally design regions/layers having different shapes, sizes, relative positions, as actually required.
Example 1:
the embodiment implements a local spatiotemporal event stream filtering method based on adaptive threshold, as shown in fig. 1, comprising the following steps:
s1, a block reading step, namely receiving an asynchronous event stream and iteratively reading an event stream block at fixed time;
s2, a space-time plane construction step, namely receiving the event stream block, accumulating the number of events in a time dimension by taking a pixel as a unit, and calculating a pixel activity value to form a two-dimensional space-time plane;
s3, a local window sliding step, namely receiving the two-dimensional space-time plane, setting a local window with a fixed size on the two-dimensional space-time plane, traversing the whole two-dimensional plane by sliding windows with fixed step length, and calculating a window mark value of each sliding window;
s4, a noise candidate domain selecting step, namely receiving the window sign value, calculating a noise candidate threshold value, and selecting a noise candidate domain according to the noise candidate threshold value;
s5, a noise screening step, namely receiving the event stream block, the two-dimensional space-time plane and the noise candidate domain, calculating a noise threshold value, marking a noise event, removing the noise event from the event stream block, and outputting the filtered asynchronous event stream.
Wherein the asynchronous event stream is obtained by a dynamic vision sensor.
In the step of reading in blocks, reading events one by one, and setting a timestamp of a first event in a block as a time mark value; and for each event input later, if the difference value between the event timestamp and the time mark value is smaller than a preset fixed time interval, dividing the event into the event stream block, otherwise, dividing the event stream block into the next event stream block. In each event stream block, the first event timestamp divided into the block is set to a new flag value.
The block reading step of S1 specifically comprises the following steps:
s1.1, initializing an event stream block index value i =0, initializing an input event index value j =0, initializing an intra-block time flag value T =0, and initializing an intra-block reproduction flag bit flag = True;
s1.2, reading data, if the read data is empty within 20ms, recording as no event input and transferring to S1.5, otherwise, receiving an event, recording as e j ＝[x j ，y j ，t j ，p j ]Wherein x is j ，y j As an event e j Pixel coordinate of (d), t j As an event e j Time stamp of p j As an event e j Polarity of (c);
s1.3, judging whether the current block remaking zone bit is in a remaking state, if flag = True, creating a new event stream block for the current event and switching to S1.4, otherwise, judging the current event e j Whether it belongs to this event stream block E i If the current event timestamp t j If the sum of the time mark value T in the block and the fixed time interval delta T is smaller than the sum, the step is switched to S1.4 if the event belongs to the event stream block, otherwise, the step is switched to S1.5 if the event belongs to the next event stream block;
s1.4, setting the current event e j Adding to the Current event stream Block E i Increasing the event index value j by one, and turning to the step S1.3 to receive the next event;
s1.5, current event stream block E i And sending out, increasing the index value i of the event stream block by one, and turning to S1.3 to read the next event stream block, wherein the block reproduction flag bit flag = True.
The method for creating the new event stream block for the current event comprises the following steps: initializing an event stream Block E i Let the intra-block timestamp value T = T j And a block reproduction flag bit flag = False.
The construction step of the space-time plane of S2 specifically comprises the following steps:
initializing a two-dimensional spatio-temporal plane N i A size of WXH, where W and H represent the dynamic vision sensor length and width, respectively, from which the data was generated;
receiving an event stream block E i Counting the event stream block E for each pixel position i The number of events occurring at the pixel position is recorded as pixel activity value, and the event stream block E i All pixel activity values in the space-time plane form a two-dimensional space-time plane, and the pixel activity value corresponding to each pixel position (x, y) is
Wherein P is the polarity, and the value is 1 or-1.
The step of S3 of sliding the window locally specifically includes:
s3.1, initializing global pixel flag bit P with size of W × H i Wherein W and H represent the dynamic vision sensor length and width, respectively, of the generated data, initializing the window flag value
Wherein m and n respectively represent the abscissa and ordinate values of the upper left corner of the current local window;
s3.2, receiving the two-dimensional space-time plane N i ；
S3.3, aiming at a two-dimensional space-time plane N i If it corresponds to a pixel activity value of N i If (x, y) is greater than 0, marking the pixel corresponding to the position with a bit P i (x, y) is set to a value of 1, indicating that an event has occurred at the pixel location; otherwise, corresponding the positionPixel flag bit P i (x, y) is set to 0, which means no event occurs at the pixel position, i.e. for x ≦ 0 ≦ W and y ≦ 0 ≦ H, let
S3.4, initializing the upper left corner of a local window to be marked as (0, 0), wherein the window comprises L multiplied by L pixel units in total and is positioned on a two-dimensional space-time plane N i Upper, partial window H i Sliding from top to bottom and from left to right by step length S until a complete two-dimensional space-time plane is traversed, in the process of sliding a local window, if boundary coordinates of a certain local window are out of range, the local window is regarded as an invalid local window and is not included in calculation, otherwise, a window mark value is calculated, and the method for calculating the window mark value comprises the following steps of
The noise candidate domain selecting step of S4 specifically includes:
initializing a dense candidate domain threshold hyperparameter α =0.9, and a sparse candidate domain threshold hyperparameter β =0.9;
receiving the window flag value
Computing a dense noise candidate domain threshold
Wherein L is the side length of the local window;
computing sparse noise candidate domain thresholds
Wherein it is present>
Marking values by all windows
Sorting from large to small and removing zero values, wherein K1 is obtained by sorting the sparse noise candidate domain threshold value in the sequence->
By an index subscript of->
The length of the sparse candidate domain is multiplied by the threshold value over-parameter of the sparse candidate domain and rounded upwards;
according to the threshold Th of the dense noise candidate domain dense And the sparse noise candidate domain threshold Th sparse Selecting noise candidate domain, and for the local window with (m, n) coordinates at upper left corner, if the window mark value of the window
Greater than threshold Th of dense noise candidate domain dense Or less than the sparse noise candidate domain threshold Th sparse Adding the coordinates (M, n) of the upper left corner into the set M of coordinates of the upper left corner of the noise candidate domain i I.e. is->
The noise screening step of S5 specifically comprises the following steps:
s5.1, initializing a noise threshold over-parameter gamma =0.9;
s5.2, receiving event stream block E i Receiving a two-dimensional space-time plane N i Receiving a set M of upper left corner coordinates of a noise candidate field i ；
S5.3, calculating a noise threshold value
Wherein +>
From a two-dimensional space-time plane N i All pixel activity value N i (x, y) is arranged from large to smallSequence removal of zero value to obtain, K 2 For noise threshold in the sequence->
Index subscript of (1), by sequence>
The length of the noise threshold value is multiplied by a noise threshold value over parameter gamma;
s5.4, according to the noise threshold Th noise Screening noise event, marking it, if it is corresponding pixel activity value N for certain pixel position (x, y) i (x, y) is less than a noise threshold Th noise Then the event contained in the pixel is considered as a noise event, which is added to the noise set F i In (1),
F i
＝{e(t，x，y，p)|(m，n)∈M i ，N i (x，y)＜Th noise ，m≤x≤m+L，n≤y≤n+L}；
s5.5, from event stream block E i Mid-erasure noise set F i Obtaining a filtered asynchronous event stream D i I.e. D i ＝E i -F i ；
S5.6, filtering the asynchronous event stream D i And (6) outputting.
According to the local space-time event stream filtering method based on the adaptive threshold, 15 scenes including a normal illumination scene, a low illumination scene, an overexposure illumination scene and a high-speed motion scene are selected from 4 open event camera data sets to be tested, and the performance of the filtering method in different scenes is tested.
Testing the denoising capability of 4 different event stream filtering methods under the 15 scenes, reconstructing the event streams before and after denoising into intensity frames respectively, and evaluating the quality of the reconstructed frames before and after denoising by using a spatial quality assessment score (BRISQE for short) without reference images. For a single image, the lower the BRISQUE score is, the better the image quality is, i.e., the better the denoising effect of the corresponding event stream filtering method is. The comparison of different event stream filtering methods in a single scene is shown in fig. 2. As can be seen from fig. 2, in 15 scenes, the method in this embodiment has better denoising effect than the BAF method in all scenes, better denoising effect than the IE algorithm in 10 scenes, and better denoising effect than the IE + TE algorithm in over 86% of sequences. Fig. 3 shows average comparison results of different event stream filtering methods in all scenes, and it can be seen from fig. 3 that, by comprehensively considering a plurality of scenes, the method implemented in the embodiment is superior to other algorithms in terms of the effectiveness of event stream denoising, and more effectively removes noise information in the scenes.
And then comparing the information retention capacities of different event stream filtering methods, reconstructing the event streams before and after denoising into intensity frames, and evaluating the information retention capacity of the event stream filtering method by using the number percentage of the reconstructed frames, namely the ratio (RFP for short) of the number of the reconstructed frames before and after denoising. For a single scene, the higher the RFP score of the filtering method, indicating that it retains more information in the denoising process. The information retention capability comparison results of the different event stream filtering methods are shown in fig. 4. As can be seen from fig. 4, the information retention capability of the method implemented in this embodiment is much higher than that of the IE and IE + TE methods, in this figure, the BAF method has the highest RFP score, and in combination with BRISQUE index analysis of this method, this method erroneously retains much noise during the denoising process.
In summary, the embodiment can implement noise filtering based on event streams, and compared with other methods, the embodiment has the advantages of optimal denoising capability and optimal effect, and has better information retention capability while effectively removing noise events.
Example 2:
the embodiment implements an adaptive threshold-based local spatiotemporal event stream filtering system, as shown in fig. 5, comprising: the system comprises a block reading module, a space-time plane construction module, a local sliding window module, a noise candidate domain selection module and a noise screening module, wherein a dynamic vision sensor inputs an asynchronous event stream to the block reading module for processing.
The block reading-in module receives asynchronous event stream data from the dynamic vision sensor, reads independent event stream blocks according to a fixed time period, and transmits the independent event stream blocks to the space-time plane construction module and the noise screening module. The specific method for dividing the event stream blocks is as follows: reading events one by one, and setting a timestamp of a first event in a block as a time mark value; and for each event input later, if the current event timestamp is less than the sum of the time mark value in the event stream block and the fixed time interval, dividing the current event timestamp into the event stream block, and otherwise, dividing the current event timestamp into the next event stream block.
The space-time plane construction module receives the event stream blocks from the block reading-in module, accumulates the number of events in a time dimension by taking pixels as units, calculates pixel activity values to form a two-dimensional space-time plane, and sends the two-dimensional space-time plane to the local sliding window module and the noise screening module. The spatial-temporal plane intuitively summarizes the correlation between the local spatial continuity and the time of the event stream block, is the basis for analyzing the local spatial-temporal information of the event stream, and the specific construction method comprises the following steps: and accumulating the number of events in the block in a time dimension by taking the pixel as a unit to obtain a pixel activity value, wherein the pixel activity values of all the pixels form a two-dimensional space-time plane.
The local sliding window module receives a two-dimensional space-time plane from the space-time plane construction module, sets a local window with a fixed size on the space-time plane, traverses the whole two-dimensional plane through the sliding windows with fixed step length, calculates the window mark value of each sliding window, and sends all the window mark values contained in the two-dimensional space-time plane to the noise candidate domain selection module. The local window helps to count the spatial distribution condition of the local event, and the specific method for calculating the window mark value is as follows: for a single sliding window, the total number of pixels in which an event occurs is calculated, i.e. the window flag value of the window.
The noise candidate domain selection module receives a window sign value contained in a two-dimensional space-time plane from the local sliding window module, calculates a noise candidate domain threshold value by combining the size of a local window and the sign value, selects sparse and dense noise candidate domains in the space-time plane according to the threshold value, and sends a left upper corner coordinate set of the noise candidate domain to the noise screening module. The noise candidate domain comprises a dense noise candidate domain and a sparse noise candidate domain, and can lock noise events in a local range to help to carry out final noise judgment. The specific method for selecting the noise candidate domain is as follows: if the window mark value of a certain local window is greater than the threshold value of the dense noise candidate domain, the window mark value is regarded as the dense noise candidate domain; if the window mark value is smaller than the sparse noise candidate domain threshold value, the window mark value is regarded as a sparse noise candidate domain; otherwise, it is considered as a valid event domain. Wherein, the threshold value of the dense noise candidate domain is selected in a self-adaptive way according to the size of a local window; the sparse noise candidate domain threshold is adaptively selected by analyzing the global spatial distribution of events within the event stream block.
The noise screening module receives the event stream block from the block reading-in module, receives a two-dimensional space-time plane from the space-time plane construction module, receives a coordinate set at the upper left corner of a noise candidate domain in the current event stream block from the noise candidate domain selection module, calculates a noise threshold value based on the pixel activity value of the space-time plane, judges whether an event triggered at a certain pixel position is a noise event according to the noise threshold value in the noise candidate domain, marks the noise event, discards the noise event from the event stream block, and outputs a filtered asynchronous event stream. The specific method for judging the noise event comprises the following steps: if the number of events occurring at a pixel in a single event stream block is less than the noise threshold, the event at the pixel position is regarded as noise and is subjected to noise labeling.
The above description is only for the preferred embodiment of the present application, but the scope of the present application is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present application should be covered within the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (8)
1. A local spatiotemporal event stream filtering method based on adaptive thresholds, characterized by comprising the following steps:
a step of reading in blocks, which is to receive asynchronous event streams and read event stream blocks in a fixed time iteration mode;
a space-time plane construction step, namely receiving the event stream block, accumulating the number of events in a time dimension by taking a pixel as a unit, and calculating a pixel activity value to form a two-dimensional space-time plane;
a local sliding window step, namely receiving the two-dimensional space-time plane, setting a local window with a fixed size on the two-dimensional space-time plane, traversing the whole two-dimensional space-time plane by a fixed step length sliding window, and calculating a window mark value of each sliding window;
a noise candidate domain selecting step of receiving the window flag value, calculating a noise candidate threshold value, and selecting a noise candidate domain according to the noise candidate threshold value;
a noise screening step, namely receiving the event stream block, the two-dimensional space-time plane and the noise candidate domain, calculating a noise threshold value, marking a noise event, removing the noise event from the event stream block, and outputting the filtered asynchronous event stream;
the space-time plane construction step specifically comprises:
initializing a two-dimensional spatio-temporal plane N i W is multiplied by H, wherein W and H respectively represent the length and the width of a dynamic vision sensor for generating data, and the value of i is 1, 2, 3, 8230a, 8230z, wherein z is a positive integer more than or equal to 1;
receiving an event stream block E i Counting the event stream block E for each pixel position i The number of events occurring at the pixel position is recorded as pixel activity value, and the event stream block E i All pixel activity values in the space-time plane form a two-dimensional space-time plane, and the pixel activity value corresponding to each pixel position (x, y) is
Wherein P is the polarity;
the step of partially sliding the window specifically includes:
s3.1, initializing global pixel flag bit P with size of W × H i Wherein W and H represent the dynamic vision sensor length and width, respectively, of the generated data, initializing the window flag value
Wherein m and n respectively represent the abscissa and ordinate values of the upper left corner of the current local window;
s3.2, receiving the two-dimensional space-time plane N i ；
S3.3, for x is more than or equal to 0 and less than or equal to W and y is more than or equal to 0 and less than or equal to H, making
S3.4, initializing the upper left corner of a local window to be marked as (0, 0), wherein the window comprises L multiplied by L pixel units in total and is positioned on a two-dimensional space-time plane N i Upper, partial window H i Sliding from top to bottom and from left to right by step length S until a complete two-dimensional space-time plane is traversed, in the process of sliding a local window, if boundary coordinates of a certain local window are out of range, the local window is regarded as an invalid local window and is not included in calculation, otherwise, a window mark value is calculated, and the method for calculating the window mark value comprises the following steps of
Wherein, P i And (x, y) is the pixel flag bit corresponding to the pixel.
2. The adaptive threshold-based local spatiotemporal event stream filtering method according to claim 1, characterized in that the asynchronous event stream is obtained by a dynamic visual sensor.
3. The adaptive threshold-based local spatiotemporal event stream filtering method according to claim 2, characterized in that said block reading step comprises in particular:
s1.1, initializing an event stream block index value i =0, initializing an input event index value j =0, initializing an intra-block time flag value T =0, and initializing an intra-block reproduction flag bit flag = True;
s1.2, reading data, if the read data is empty within 20ms, recording as no event input and transferring to S1.5, otherwise, receiving an event, recording as e j ＝[x j ，y j ，t j ，p j ]Wherein x is j ，y j As an event e j Pixel coordinate of (a), t j As an event e j Time ofStamp, p j As an event e j The polarity of (1);
s1.3, judging whether the current block remaking zone bit is in a remaking state, if flag = True, creating a new event stream block for the current event and switching to S1.4, otherwise, judging the current event e j Whether it belongs to this event stream block E i If the current event timestamp t j If the sum of the time mark value T in the block and the fixed time interval delta T is smaller, the step is switched to S1.4 if the event belongs to the event stream block, otherwise, the step is switched to S1.5 if the sum of the time mark value T in the block and the fixed time interval delta T is recorded as the event belongs to the next event stream block;
s1.4, setting the current event e j Adding to the Current event stream Block E i Increasing the event index value j by one, and turning to the step S1.3 to receive the next event;
s1.5, current event stream block E i And sending out, increasing the index value i of the event stream block by one, and turning to S1.3 to read the next event stream block, wherein the block reproduction flag bit is = True.
4. The adaptive threshold-based local spatiotemporal event stream filtering method according to claim 3, characterized in that the method of creating a new event stream block for the current event is: initializing an event stream Block E i Let the intra block timestamp value T = T j And a block reproduction flag bit flag = False.
5. The adaptive threshold-based local spatiotemporal event stream filtering method as defined in claim 2, wherein said noise candidate domain selection step specifically comprises:
initializing a dense candidate domain threshold hyperparameter alpha =0.9, and a sparse candidate domain threshold hyperparameter beta =0.9;
receiving the window flag value
Computing a dense noise candidate domain threshold
Wherein L isThe side length of the local window;
computing sparse noise candidate domain thresholds
Wherein +>
By all window flag values->
Sorting from large to small to remove zero value, K 1 Candidate domain threshold in sequence for sparse noise>
Index subscript of (1), by the sequence
The length of the sparse candidate domain is multiplied by the threshold value hyperparameter of the sparse candidate domain and rounded upwards;
according to the threshold Th of the dense noise candidate domain dense And the sparse noise candidate domain threshold Th sparse Selecting a noise candidate domain, and for a local window with (m, n) coordinates at the upper left corner, if the window mark value of the window is
Greater than threshold Th of dense noise candidate domain dense Or less than the sparse noise candidate domain threshold Th sparse Adding the coordinates (M, n) of the upper left corner into the set M of coordinates of the upper left corner of the noise candidate domain i In (1).
6. The adaptive threshold-based local spatiotemporal event stream filtering method according to claim 2, characterized in that the noise filtering step specifically comprises:
s5.1, initializing a noise threshold over-parameter gamma =0.9;
s5.2, receiving event stream block E i Receiving twoDimensional space-time plane N i Receiving a set M of upper left corner coordinates of a noise candidate field i ；
S5.3, calculating a noise threshold value
Wherein it is present>
From a two-dimensional space-time plane N i All pixel activity value N i (x, y) is obtained by sorting from large to small and removing zero values, K 2 For noise thresholds in the sequence>
Index subscript of (1), by sequence>
The length of the noise threshold value is multiplied by a noise threshold value over parameter gamma;
s5.4, according to the noise threshold Th oise Screening noise event, marking it, if it is corresponding pixel activity value N for certain pixel position (x, y) i (x, y) is less than a noise threshold Th noise The event contained in the pixel is considered as a noise event, which is added to the noise set F i The preparation method comprises the following steps of (1) performing;
s5.5, from event stream block E i Mid-erasure noise set F i To obtain a filtered asynchronous event stream D i ；
S5.6, filtering the asynchronous event stream D i And (6) outputting.
7. An adaptive threshold based local spatiotemporal event stream filtering system, comprising:
a block reading module for executing the block reading step, receiving the asynchronous event stream, and iteratively reading the event stream block at a fixed time;
the space-time plane construction module is used for executing the space-time plane construction step, receiving the event stream block, accumulating the number of events in a time dimension by taking a pixel as a unit, and calculating a pixel activity value to form a two-dimensional space-time plane;
a local sliding window module for executing the local sliding window step, receiving the two-dimensional space-time plane and setting a local window with a fixed size on the two-dimensional space-time plane, traversing the whole two-dimensional space-time plane by a fixed step length sliding window, and calculating a window mark value of each sliding window;
a noise candidate domain selection module for executing a noise candidate domain selection step, receiving the window flag value, calculating a noise candidate threshold value, and selecting a noise candidate domain according to the noise candidate threshold value;
a noise screening module for performing a noise screening step, receiving the event stream block, the two-dimensional spatio-temporal plane and the noise candidate domain and calculating a noise threshold, marking and eliminating a noise event from the event stream block, and outputting a filtered asynchronous event stream;
the spatio-temporal plane construction module specifically executes the following steps:
initializing a two-dimensional spatio-temporal plane N i W x H, wherein W and H respectively represent the length and width of the dynamic vision sensor generating data, and the value of i is 1, 2, 3 \8230, z is a positive integer more than or equal to 1;
receiving an event stream block E i Counting the event stream block E for each pixel position i The number of events occurring at the pixel position is recorded as pixel activity value, and the event stream block E i All pixel activity values in the space-time plane form a two-dimensional space-time plane, and the pixel activity value corresponding to each pixel position (x, y) is
Wherein P is the polarity;
the local sliding window module specifically executes the following steps:
s3.1, initializing a global pixel flag bit P with the size of W multiplied by H i Wherein W and H represent the dynamic vision sensor length and width, respectively, of the generated data, initializing the window flag value
Wherein m and n respectively represent the abscissa and ordinate values of the upper left corner of the current local window;
s3.2, receiving the two-dimensional space-time plane N t ；
S3.3, for x is more than or equal to 0 and less than or equal to W and y is more than or equal to 0 and less than or equal to H, making
S3.4, initializing the upper left corner of a local window, marking as (0, 0), wherein the window totally comprises L multiplied by L pixel units and is positioned on a two-dimensional space-time plane N i Upper, partial window H i Sliding from top to bottom and from left to right by step length S until a complete two-dimensional space-time plane is traversed, in the process of sliding a local window, if boundary coordinates of a certain local window are out of range, the local window is regarded as an invalid local window and is not included in calculation, otherwise, a window mark value is calculated, and the method for calculating the window mark value comprises the following steps of
Wherein, P i And (x, y) is the pixel flag bit corresponding to the pixel.
8. The adaptive threshold-based local spatiotemporal event stream filtering system according to claim 7, characterized in that the asynchronous event stream is input by a dynamic visual sensor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110435263.5A CN113269683B (en) | 2021-04-22 | 2021-04-22 | Local space-time event stream filtering method and system based on self-adaptive threshold |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110435263.5A CN113269683B (en) | 2021-04-22 | 2021-04-22 | Local space-time event stream filtering method and system based on self-adaptive threshold |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113269683A CN113269683A (en) | 2021-08-17 |
| CN113269683B true CN113269683B (en) | 2023-04-18 |
Family
ID=77229097
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110435263.5A Active CN113269683B (en) | 2021-04-22 | 2021-04-22 | Local space-time event stream filtering method and system based on self-adaptive threshold |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113269683B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113923319B (en) * | 2021-12-14 | 2022-03-08 | 成都时识科技有限公司 | Noise reduction device, noise reduction method, chip, event imaging device and electronic equipment |
| CN114285962B (en) * | 2021-12-14 | 2023-04-07 | 成都时识科技有限公司 | Noise processing device, method, chip, event imaging device and electronic equipment |
| CN114885074B (en) * | 2022-05-06 | 2023-07-04 | 中国科学院光电技术研究所 | Event camera denoising method based on space-time density |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111931752A (en) * | 2020-10-13 | 2020-11-13 | 中航金城无人系统有限公司 | Dynamic target detection method based on event camera |
| CN112184572A (en) * | 2020-09-14 | 2021-01-05 | 中山大学 | Novel rain removing method and system for dynamic vision sensor event stream |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120242864A1 (en) * | 2010-09-20 | 2012-09-27 | Pauli Myron R | Flash detection and clutter rejection processor |
| CN110176028B (en) * | 2019-06-05 | 2020-12-15 | 中国人民解放军国防科技大学 | Asynchronous corner detection method based on event camera |
| CN111340699B (en) * | 2020-02-17 | 2022-02-18 | 西北大学 | Magnetic resonance image denoising method and device based on non-local prior and sparse representation |
| CN111770290A (en) * | 2020-07-29 | 2020-10-13 | 中国科学院长春光学精密机械与物理研究所 | Noise reduction method for dynamic vision sensor output event stream |
| CN112307078B (en) * | 2020-09-29 | 2022-04-15 | 安徽工业大学 | Data stream differential privacy histogram publishing method based on sliding window |
-
2021
- 2021-04-22 CN CN202110435263.5A patent/CN113269683B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112184572A (en) * | 2020-09-14 | 2021-01-05 | 中山大学 | Novel rain removing method and system for dynamic vision sensor event stream |
| CN111931752A (en) * | 2020-10-13 | 2020-11-13 | 中航金城无人系统有限公司 | Dynamic target detection method based on event camera |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113269683A (en) | 2021-08-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113269683B (en) | Local space-time event stream filtering method and system based on self-adaptive threshold | |
| Irani et al. | Detecting and tracking multiple moving objects using temporal integration | |
| JP6616521B2 (en) | Image processing device | |
| WO2006022493A1 (en) | Method for removing noise in image and system thereof | |
| Chen et al. | Video Carving. | |
| CN112232356A (en) | Event camera denoising method based on cluster degree and boundary characteristics | |
| CN108270945B (en) | Motion compensation denoising method and device | |
| Chen et al. | A residual learning approach to deblur and generate high frame rate video with an event camera | |
| CN106934819A (en) | A kind of method of moving object segmentation precision in raising image | |
| Malik et al. | Object detection and tracking using background subtraction and connected component labeling | |
| Fang et al. | AEDNet: Asynchronous event denoising with Spatial-Temporal correlation among irregular data | |
| TWI381735B (en) | Image processing system and method for automatic adjustment of image resolution for image surveillance apparatus | |
| Mahayuddin et al. | Efficiency measurement of various denoise techniques for moving object detection using aerial images | |
| Nie et al. | High frame rate video reconstruction and deblurring based on dynamic and active pixel vision image sensor | |
| Moura et al. | A spatiotemporal motion-vector filter for object tracking on compressed video | |
| CN111754477B (en) | Railway perimeter foreign matter intrusion detection method based on dynamic candidate area multi-scale image | |
| Chae et al. | Siamevent: Event-based object tracking via edge-aware similarity learning with siamese networks | |
| CN112184572B (en) | Novel rain removing method and system for dynamic vision sensor event stream | |
| CN110097523B (en) | Video image fog concentration classification and self-adaptive defogging method | |
| Li et al. | High-speed large-scale imaging using frame decomposition from intrinsic multiplexing of motion | |
| Geng et al. | Dense Face Detection via High-level Context Mining | |
| She et al. | SpikeFormer: Image Reconstruction from the Sequence of Spike Camera Based on Transformer | |
| Abdulghafoor et al. | Real-Time Object Detection with Simultaneous Denoising using Low-Rank and Total Variation Models | |
| Kaur et al. | An improved weighted median filter for the image processing application | |
| Latzel et al. | People tracking using robust motion detection and estimation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |