CN113269683A - Local space-time event stream filtering method and system based on self-adaptive threshold - Google Patents
Local space-time event stream filtering method and system based on self-adaptive threshold Download PDFInfo
- Publication number
- CN113269683A CN113269683A CN202110435263.5A CN202110435263A CN113269683A CN 113269683 A CN113269683 A CN 113269683A CN 202110435263 A CN202110435263 A CN 202110435263A CN 113269683 A CN113269683 A CN 113269683A
- Authority
- CN
- China
- Prior art keywords
- noise
- event
- event stream
- window
- block
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 76
- 238000001914 filtration Methods 0.000 title claims abstract description 43
- 230000000694 effects Effects 0.000 claims abstract description 33
- 230000003044 adaptive effect Effects 0.000 claims abstract description 19
- 238000012216 screening Methods 0.000 claims abstract description 18
- 238000010276 construction Methods 0.000 claims abstract description 16
- 230000000007 visual effect Effects 0.000 claims description 5
- 238000004364 calculation method Methods 0.000 claims description 3
- 230000036961 partial effect Effects 0.000 claims description 3
- 238000005516 engineering process Methods 0.000 abstract description 4
- 230000014759 maintenance of location Effects 0.000 description 6
- 238000005286 illumination Methods 0.000 description 4
- 230000000670 limiting effect Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000009825 accumulation Methods 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000002427 irreversible effect Effects 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000001303 quality assessment method Methods 0.000 description 1
- 230000002829 reductive effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20024—Filtering details
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The present application relates to the field of filtering technologies, and in particular, to a local spatiotemporal event stream filtering method and system based on adaptive thresholds. The method comprises the following steps: a step of reading in blocks, which is to receive asynchronous event streams and read event stream blocks in a fixed time iteration mode; a space-time plane construction step, namely receiving the event stream block, accumulating the number of events in a time dimension by taking a pixel as a unit, and calculating a pixel activity value to form a two-dimensional space-time plane; a local sliding window step, receiving the two-dimensional space-time plane, setting a local window with a fixed size on the two-dimensional space-time plane, and calculating a window mark value of each sliding window; a noise candidate domain selecting step of receiving the window flag value, calculating a noise candidate threshold value, and selecting a noise candidate domain according to the noise candidate threshold value; and a noise screening step, namely marking the noise event, removing the noise event from the event stream block, and outputting the filtered asynchronous event stream. The method and system can effectively remove noise events in asynchronous event streams.
Description
Technical Field
The present application relates to the field of filtering technologies, and more particularly, to a local spatiotemporal event stream filtering method and system based on adaptive thresholds.
Background
The conventional camera has a problem of motion blur when capturing an object moving at a high speed. Meanwhile, the dynamic range of the traditional camera is small, and the acquisition and presentation information is limited when the traditional camera works under an extreme illumination condition. The dynamic vision sensor makes up the disadvantages of the traditional camera. The dynamic vision sensor has the advantages of high time resolution, high dynamic range, low delay, low power consumption and the like, and provides a new solution for computer vision tasks which are challenged by some traditional cameras. However, in practical scenarios, the application performance is severely affected by the large amount of noise generated by the dynamic vision sensor. Meanwhile, a developed standard image denoising algorithm cannot be directly applied to an unstructured asynchronous event stream, and therefore a special denoising method needs to be designed for an event.
With the continuous development of the hardware technology of the dynamic vision sensor, the research of the vision technology based on the dynamic vision sensor is receiving wide attention. Currently, researchers have proposed a variety of event stream filtering methods, which mainly include: the event stream filtering method based on the traditional image denoising, the event stream filtering method based on the space and the event stream filtering method based on the time.
An event stream filtering method based on traditional image denoising is commonly implemented as follows: and mapping the 3D event stream into a 2D image frame, and then carrying out denoising treatment by using a traditional image denoising method. However, the problem with these methods is that the mapping from the event stream to the frame image is irreversible, i.e. the 3D denoised event stream cannot be recovered from the denoised 2D frame image. This causes the denoised event stream to lose features, limiting subsequent applications of the event camera.
The principle of the event stream filtering method based on the space is as follows: spatial redundancy of events is exploited to remove spatially independent event points. The common method is as follows: for each newly arrived event, it is checked whether there is an event in the past T microseconds in 8 adjacent pixels. If not, it is judged as noise and discarded. In addition, it can be generalized to a plurality of extensions such as distinguishing polarities, distinguishing ranges, defining the number of adjacent events, and the like. The method has high running speed and good denoising effect on simple scenes, but is easy to filter out the event errors generated by small, slow-moving or small-volume objects, and meanwhile, the noise accumulation is easy to cause because the number of reference points is too small.
The principle of the event stream filtering method based on time is as follows: events that are redundant in time are removed. It is common practice for an arrival event to be retained only if the timestamp differs from the timestamp of the most recent event for that pixel by more than a threshold value. The method can quickly extract object features, but most events in an event stream are usually filtered, so that scene information is reduced sharply.
The present application therefore proposes an improved method and system to at least partially solve the above technical problem.
Disclosure of Invention
The technical scheme of the invention is as follows: integrating local event stream information, analyzing event distribution characteristics from spatial and time dimensions in sequence, and judging the effectiveness of an event; meanwhile, an adaptive threshold mechanism is applied to adapt to different environments, so that the filter is widely applied. Finally, on the basis of maintaining the characteristics of the asynchronous event stream data, the method not only effectively removes noisy noise information in a scene, but also retains effective events for describing object lines, achieves the purpose of improving the quality of the asynchronous event stream, and provides powerful support for the visual task based on the event stream.
In order to achieve the technical purpose, the application provides a local spatiotemporal event stream filtering method based on an adaptive threshold, which comprises the following steps:
a step of reading in blocks, which is to receive asynchronous event streams and read event stream blocks in a fixed time iteration mode;
a space-time plane construction step, namely receiving the event stream block, accumulating the number of events in a time dimension by taking a pixel as a unit, and calculating a pixel activity value to form a two-dimensional space-time plane;
a local sliding window step, namely receiving the two-dimensional space-time plane, setting a local window with a fixed size on the two-dimensional space-time plane, traversing the whole two-dimensional plane by a fixed step length sliding window, and calculating a window mark value of each sliding window;
a noise candidate domain selecting step of receiving the window flag value, calculating a noise candidate threshold value, and selecting a noise candidate domain according to the noise candidate threshold value;
and a noise screening step, namely receiving the event stream block, the two-dimensional space-time plane and the noise candidate domain, calculating a noise threshold value, marking a noise event, removing the noise event from the event stream block, and outputting the filtered asynchronous event stream.
Preferably, the asynchronous event stream is obtained by a dynamic visual sensor.
Specifically, the block reading step specifically includes:
s1.1, initializing an event stream block index value i equal to 0, initializing an input event index value j equal to 0, initializing an intra-block time flag value T equal to 0, and initializing an block reproduction flag bit flag equal to True;
s1.2, reading data, if the read data is empty within 20ms, recording as no event input, and switching to S1.5, otherwise, receiving an event, recording as ej＝[xj，yj，tj，pj]Wherein x isj，yjAs an event ejPixel coordinate of (d), tjAs an event ejTime stamp of pjAs an event ejThe polarity of (1);
s1.3, judging whether the current block remaking flag bit is in a remaking state or not, and if the flag is equal to True, creating new events for the current eventsThe stream block is transferred to S1.4, otherwise, the current event e is judgedjWhether it belongs to the event stream block EiIf the current event timestamp tjIf the sum of the time mark value T in the block and the fixed time interval delta T is smaller than the sum, the step is switched to S1.4 if the event belongs to the event stream block, otherwise, the step is switched to S1.5 if the event belongs to the next event stream block;
s1.4, setting the current event ejAdding to the Current event stream Block EiIncreasing the event index value j by one, and turning to the step S1.3 to receive the next event;
s1.5, current event stream block EiAnd sending out, increasing the index value i of the event stream block by one, turning to S1.3 to read the next event stream block, wherein the flag bit flag of the block reproduction is True.
The method for creating a new event stream block for the current event comprises the following steps: initializing an event stream Block EiLet the time mark value T in the block be TjThe block copy flag is False.
Specifically, the spatio-temporal plane construction step specifically includes:
initializing a two-dimensional spatio-temporal plane NiA size of WXH, where W and H represent the dynamic vision sensor length and width, respectively, from which the data was generated;
receiving an event stream block EiCounting the event stream block E for each pixel positioniThe number of events occurring at the pixel position is recorded as pixel activity value, and the event stream block EiAll pixel activity values in the space-time plane form a two-dimensional space-time plane, and the pixel activity value corresponding to each pixel position (x, y) is
Wherein P is the polarity.
Specifically, the step of sliding the window locally specifically includes:
s3.1, initializing global pixel flag bit P with size of W × HiWherein W and H represent the dynamic vision sensor length and width, respectively, of the generated data, initializing the window flag value
Wherein m and n respectively represent the abscissa and ordinate values of the upper left corner of the current local window;
s3.2, receiving the two-dimensional space-time plane Ni；
S3.3, for x is more than or equal to 0 and less than or equal to W and y is more than or equal to 0 and less than or equal to H, making
S3.4, initializing the upper left corner of a local window to be marked as (0, 0), wherein the window comprises L multiplied by L pixel units in total and is positioned on a two-dimensional space-time plane NiUpper, partial window HiSliding from top to bottom and from left to right by step length S until a complete two-dimensional space-time plane is traversed, in the process of sliding a local window, if boundary coordinates of a certain local window are out of range, the local window is regarded as an invalid local window and is not included in calculation, otherwise, a window mark value is calculated, and the method for calculating the window mark value comprises the following steps of
Specifically, the noise candidate domain selecting step specifically includes:
initializing a dense candidate domain threshold value hyperparameter alpha which is 0.9, and initializing a sparse candidate domain threshold value hyperparameter beta which is 0.9;
receiving the window flag value
Computing a dense noise candidate domain threshold
Wherein L is the side length of the local window;
computing sparse noise candidate domain thresholds
Wherein,
marking values by all windows
Sorting from large to small to remove zero value, K1Candidate domain thresholding in sequence for sparse noise
Index subscript of (1), by the sequence
The length of the sparse candidate domain is multiplied by the threshold value over-parameter of the sparse candidate domain and rounded upwards;
according to the threshold Th of the dense noise candidate domaindenseAnd the sparse noise candidate domain threshold ThsparseSelecting a noise candidate domain, and for a local window with (m, n) coordinates at the upper left corner, if the window mark value of the window is
Greater than threshold Th of dense noise candidate domaindenseOr less than the sparse noise candidate domain threshold ThsparseAdding the coordinates (M, n) of the upper left corner into the set M of coordinates of the upper left corner of the noise candidate domainiIn (1).
Further, the noise screening step specifically comprises:
s5.1, setting an initialization noise threshold value hyperparameter gamma to be 0.9;
s5.2, receiving event stream block EiReceiving a two-dimensional space-time plane NiReceiving a set M of upper left corner coordinates of a noise candidate fieldi；
S5.3, calculating a noise threshold value
Wherein,
from a two-dimensional space-time plane NiAll thereinPixel activity value N ofi(x, y) is obtained by sorting from large to small and removing zero values, K2For noise in the sequence
Index subscript of (1), by sequence
The length of the noise threshold value is multiplied by a noise threshold value over parameter gamma;
s5.4, according to the noise threshold ThnoiseScreening noise event, marking it, if it is corresponding pixel activity value N for certain pixel position (x, y)i(x, y) is less than a noise threshold ThnoiseThe event contained in the pixel is considered as a noise event, which is added to the noise set FiPerforming the following steps;
s5.5, from event stream block EiMid-erasure noise set FiObtaining a filtered asynchronous event stream Di；
S5.6, filtering the asynchronous event stream DiAnd (6) outputting.
A second aspect of the present invention provides an adaptive threshold-based local spatiotemporal event stream filtering system, comprising:
a block reading module for executing the block reading step, receiving the asynchronous event stream and iteratively reading the event stream block at a fixed time;
the space-time plane construction module is used for executing the space-time plane construction step, receiving the event stream block, accumulating the number of events in a time dimension by taking a pixel as a unit, and calculating a pixel activity value to form a two-dimensional space-time plane;
a local sliding window module for executing the local sliding window step, receiving the two-dimensional space-time plane and setting a local window with a fixed size on the two-dimensional space-time plane, traversing the whole two-dimensional plane by a fixed step length sliding window, and calculating a window mark value of each sliding window;
a noise candidate domain selection module for executing a noise candidate domain selection step, receiving the window flag value, calculating a noise candidate threshold value, and selecting a noise candidate domain according to the noise candidate threshold value;
and the noise screening module is used for executing a noise screening step, receiving the event stream block, the two-dimensional space-time plane and the noise candidate domain, calculating a noise threshold value, marking a noise event, removing the noise event from the event stream block, and outputting the filtered asynchronous event stream.
Preferably, the asynchronous event stream is input by a dynamic visual sensor.
The beneficial effect of this application does:
the application provides a local space-time event stream filtering method and system based on an adaptive threshold, which can effectively remove noise events which can not describe the characteristics of an object in an asynchronous event stream, can keep real events which describe lines of the object, obtain a clean, clear and complete scene description from an original event stream, and realize a relatively accurate and rapid denoising effect. Moreover, the method can greatly improve the application capability of the event camera in the actual task, improve the problem of serious noise influence of the event camera in the actual scene, and enable the event camera to be better applied to challenging environments such as high-speed motion, high dynamic range and the like, provides a new idea for solving challenging computer vision problems, and has good application potential in practical problems such as automatic driving and the like.
Drawings
FIG. 1 shows a schematic flow chart of the method of embodiment 1 of the present application;
FIG. 2 is a graph showing the comparison result between the method in example 1 and other methods in a single scene;
FIG. 3 is a graph showing the average comparison result of the method in example 1 of the present application and other methods in all scenes;
FIG. 4 is a graph showing the comparison of information retention capacity between the method of example 1 and other methods;
fig. 5 shows a system configuration diagram of embodiment 2 of the present application.
Detailed Description
Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. It should be understood that the description is intended to be exemplary only, and is not intended to limit the scope of the present application. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present application. It will be apparent to one skilled in the art that the present application may be practiced without one or more of these details. In other instances, well-known features of the art have not been described in order to avoid obscuring the present application.
It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments in accordance with the application. As used herein, the singular is intended to include the plural unless the context clearly dictates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
Exemplary embodiments according to the present application will now be described in more detail with reference to the accompanying drawings. These exemplary embodiments may, however, be embodied in many different forms and should not be construed as limited to only the embodiments set forth herein. The figures are not drawn to scale, wherein certain details may be exaggerated and omitted for clarity. The shapes of various regions, layers, and relative sizes and positional relationships therebetween shown in the drawings are merely exemplary, and deviations may occur in practice due to manufacturing tolerances or technical limitations, and a person skilled in the art may additionally design regions/layers having different shapes, sizes, relative positions, as actually required.
Example 1:
the embodiment implements a local spatiotemporal event stream filtering method based on adaptive threshold, as shown in fig. 1, comprising the following steps:
s1, a block reading step, namely receiving an asynchronous event stream and iteratively reading an event stream block at a fixed time;
s2, a space-time plane construction step, namely receiving the event stream block, accumulating the number of events in a time dimension by taking a pixel as a unit, and calculating a pixel activity value to form a two-dimensional space-time plane;
s3, a local window sliding step, namely, receiving the two-dimensional space-time plane, setting a local window with a fixed size on the two-dimensional space-time plane, traversing the whole two-dimensional plane by sliding windows with fixed step length, and calculating a window mark value of each sliding window;
s4, a noise candidate domain selecting step, wherein the noise candidate domain selecting step is used for receiving the window sign value, calculating a noise candidate threshold value and selecting a noise candidate domain according to the noise candidate threshold value;
s5, a noise screening step, namely receiving the event stream block, the two-dimensional space-time plane and the noise candidate domain, calculating a noise threshold value, marking a noise event, removing the noise event from the event stream block, and outputting the filtered asynchronous event stream.
Wherein the asynchronous event stream is obtained by a dynamic vision sensor.
In the step of reading in blocks, reading events one by one, and setting a timestamp of a first event in a block as a time mark value; and for each event input later, if the difference value between the event timestamp and the time mark value is smaller than a preset fixed time interval, dividing the event into the event stream block, otherwise, dividing the event stream block into the next event stream block. In each event stream block, the first event timestamp divided into the block is set to a new flag value.
The block reading step of S1 specifically includes:
s1.1, initializing an event stream block index value i equal to 0, initializing an input event index value j equal to 0, initializing an intra-block time flag value T equal to 0, and initializing an block reproduction flag bit flag equal to True;
s1.2, reading data, if the read data is empty within 20ms, recording as no event input, and switching to S1.5, otherwise, receiving an event, recording as ej＝[xj，yj，tj，pj]Wherein x isj，yjAs an event ejPixel coordinate of (d), tjAs an event ejTime stamp of pjAs an event ejThe polarity of (1);
s1.3, judging whether the current block remaking flag bit is in a remaking state, if the flag is equal to True, establishing a new event stream block for the current event and switching to S1.4, otherwise, judging the current event ejWhether it belongs to the event stream block EiIf the current event timestamp tjIf the sum of the time mark value T in the block and the fixed time interval delta T is smaller than the sum, the step is switched to S1.4 if the event belongs to the event stream block, otherwise, the step is switched to S1.5 if the event belongs to the next event stream block;
s1.4, setting the current event ejAdding to the Current event stream Block EiIncreasing the event index value j by one, and turning to the step S1.3 to receive the next event;
s1.5, current event stream block EiAnd sending out, increasing the index value i of the event stream block by one, turning to S1.3 to read the next event stream block, wherein the flag bit flag of the block reproduction is True.
The method for creating the new event stream block for the current event comprises the following steps: initializing an event stream Block EiLet the time mark value T in the block be TjThe block copy flag is False.
The space-time plane construction step of S2 specifically includes:
initializing a two-dimensional spatio-temporal plane NiA size of WXH, where W and H represent the dynamic vision sensor length and width, respectively, from which the data was generated;
receiving an event stream block EiCounting the event stream block E for each pixel positioniThe number of events occurring at the pixel position is recorded as pixel activity value, and the event stream block EiAll pixel activity values in the space-time plane form a two-dimensional space-time plane, and the pixel activity value corresponding to each pixel position (x, y) is
Wherein P is the polarity, and the value is 1 or-1.
The step of S3 of locally sliding the window specifically includes:
s3.1, initializing global pixel flag bit P with size of W × HiWherein W and H represent the dynamic vision sensor length and width, respectively, of the generated data, initializing the window flag value
Wherein m and n respectively represent the abscissa and ordinate values of the upper left corner of the current local window;
s3.2, receiving the two-dimensional space-time plane Ni；
S3.3, aiming at a two-dimensional space-time plane NiIf it corresponds to a pixel activity value of NiIf (x, y) is greater than 0, marking the pixel corresponding to the position with a bit Pi(x, y) is set to a value of 1, indicating that an event has occurred at the pixel location; otherwise, marking the pixel flag bit P corresponding to the positioni(x, y) is set to 0 value, which indicates that no event occurs at the pixel position, i.e. for x is more than or equal to 0 and less than or equal to W and y is more than or equal to 0 and less than or equal to H, let
S3.4, initializing the upper left corner of a local window to be marked as (0, 0), wherein the window comprises L multiplied by L pixel units in total and is positioned on a two-dimensional space-time plane NiUpper, partial window HiSliding from top to bottom and from left to right by step length S until a complete two-dimensional space-time plane is traversed, in the process of sliding a local window, if boundary coordinates of a certain local window are out of range, the local window is regarded as an invalid local window and is not included in calculation, otherwise, a window mark value is calculated, and the method for calculating the window mark value comprises the following steps of
The noise candidate domain selecting step of S4 specifically includes:
initializing a dense candidate domain threshold value hyperparameter alpha which is 0.9, and initializing a sparse candidate domain threshold value hyperparameter beta which is 0.9;
receiving the window flag value
Computing a dense noise candidate domain threshold
Wherein L is the side length of the local window;
computing sparse noise candidate domain thresholds
Wherein,
marking values by all windows
Sorting from big to small and removing zero value, K1 is sparse noise candidate domain threshold value in sequence
Index subscript of (1), by the sequence
The length of the sparse candidate domain is multiplied by the threshold value over-parameter of the sparse candidate domain and rounded upwards;
according to the threshold Th of the dense noise candidate domaindenseAnd the sparse noise candidate domain threshold ThsparseSelecting a noise candidate domain, and for a local window with (m, n) coordinates at the upper left corner, if the window mark value of the window is
Greater than threshold Th of dense noise candidate domaindenseOr less than the sparse noise candidate domain threshold ThsparseAdding the coordinates (M, n) of the upper left corner into the set M of coordinates of the upper left corner of the noise candidate domainiI.e. by
The noise screening step of S5 specifically comprises the following steps:
s5.1, setting an initialization noise threshold value hyperparameter gamma to be 0.9;
s5.2, receiving event stream block EiReceiving a two-dimensional space-time plane NiReceiving a set M of upper left corner coordinates of a noise candidate fieldi；
S5.3, calculating a noise threshold value
Wherein,
from a two-dimensional space-time plane NiAll pixel activity value Ni(x, y) is obtained by sorting from large to small and removing zero values, K2For noise in the sequence
Index subscript of (1), by sequence
The length of the noise threshold value is multiplied by a noise threshold value over parameter gamma;
s5.4, according to the noise threshold ThnoiseScreening noise event, marking it, if it is corresponding pixel activity value N for certain pixel position (x, y)i(x, y) is less than a noise threshold ThnoiseThe event contained in the pixel is considered as a noise event, which is added to the noise set FiIn (1),
Fi
＝{e(t，x，y，p)|(m，n)∈Mi，Ni(x，y)＜Thnoise，m≤x≤m+L，n≤y≤n+L}；
s5.5, from event stream block EiMid-erasure noise set FiObtaining a filtered asynchronous event stream DiI.e. Di＝Ei-Fi；
S5.6, filtering the asynchronous event stream DiAnd (6) outputting.
According to the local space-time event stream filtering method based on the adaptive threshold, 15 scenes including a normal illumination scene, a low illumination scene, an overexposure illumination scene and a high-speed motion scene are selected from 4 open event camera data sets to be tested, and the performance of the filtering method in different scenes is tested.
Testing the denoising capability of 4 different event stream filtering methods under the 15 scenes, reconstructing the event streams before and after denoising into intensity frames, and evaluating the quality of the reconstructed frames before and after denoising by using a spatial quality assessment score (BRISQE for short) without reference images. For a single image, the lower the BRISQUE score is, the better the image quality is, i.e. the better the denoising effect of the corresponding event stream filtering method is. The comparison of different event stream filtering methods in a single scene is shown in fig. 2. As can be seen from fig. 2, in 15 scenes, the method in this embodiment has better denoising effect than the BAF method in all scenes, better denoising effect than the IE algorithm in 10 scenes, and better denoising effect than the IE + TE algorithm in over 86% of sequences. Fig. 3 shows average comparison results of different event stream filtering methods in all scenes, and it can be seen from fig. 3 that, by comprehensively considering a plurality of scenes, the method implemented in the embodiment is superior to other algorithms in terms of the effectiveness of event stream denoising, and more effectively removes noise information in the scenes.
And then comparing the information retention capacities of different event stream filtering methods, reconstructing the event streams before and after denoising into intensity frames, and evaluating the information retention capacity of the event stream filtering method by using the number percentage of the reconstructed frames, namely the ratio (RFP for short) of the number of the reconstructed frames before denoising and the number of the reconstructed frames after denoising. For a single scene, the higher the RFP score of the filtering method, indicating that it retains more information in the denoising process. The information retention capability comparison results of the different event stream filtering methods are shown in fig. 4. As can be seen from fig. 4, the information retention capability of the method implemented in this embodiment is much higher than that of the IE and IE + TE methods, in this figure, the BAF method has the highest RFP score, and in combination with BRISQUE index analysis of this method, this method erroneously retains much noise during the denoising process.
In summary, the embodiment can implement noise filtering based on event streams, and compared with other methods, the embodiment has the advantages of optimal denoising capability and optimal effect, and has better information retention capability while effectively removing noise events.
Example 2:
the embodiment implements an adaptive threshold-based local spatiotemporal event stream filtering system, as shown in fig. 5, comprising: the system comprises a block reading module, a space-time plane construction module, a local sliding window module, a noise candidate domain selection module and a noise screening module, wherein a dynamic vision sensor inputs an asynchronous event stream to the block reading module for processing.
The block reading-in module receives asynchronous event stream data from the dynamic vision sensor, reads independent event stream blocks according to a fixed time period, and transmits the independent event stream blocks to the space-time plane construction module and the noise screening module. The specific method for dividing the event stream blocks is as follows: reading the events one by one, and setting a timestamp of a first event in the block as a time mark value; and for each event input later, if the current event timestamp is less than the sum of the time mark value in the event stream block and the fixed time interval, dividing the current event timestamp into the event stream block, and otherwise, dividing the current event timestamp into the next event stream block.
The space-time plane construction module receives the event stream blocks from the block reading-in module, accumulates the number of events in a time dimension by taking pixels as units, calculates pixel activity values to form a two-dimensional space-time plane, and sends the two-dimensional space-time plane to the local sliding window module and the noise screening module. The spatial-temporal plane intuitively summarizes the correlation between the local spatial continuity and the time of the event stream block, is the basis for analyzing the local spatial-temporal information of the event stream, and is constructed by the following specific method: and accumulating the number of events in the block in a time dimension by taking the pixel as a unit to obtain a pixel activity value, wherein the pixel activity values of all the pixels form a two-dimensional space-time plane.
The local sliding window module receives the two-dimensional space-time plane from the space-time plane construction module, sets a local window with a fixed size on the space-time plane, traverses the whole two-dimensional plane by sliding windows with fixed step lengths, calculates the window sign value of each sliding window, and sends all the window sign values contained in the two-dimensional space-time plane to the noise candidate domain selection module. The local window helps to count the spatial distribution condition of the local event, and the specific method for calculating the window mark value is as follows: for a single sliding window, the total number of pixels in which an event occurs is calculated, i.e. the window flag value of the window.
The noise candidate domain selection module receives a window sign value contained in a two-dimensional space-time plane from the local sliding window module, calculates a noise candidate domain threshold value by combining the size of a local window and the sign value, selects sparse and dense noise candidate domains in the space-time plane according to the threshold value, and sends a left upper corner coordinate set of the noise candidate domain to the noise screening module. The noise candidate domain comprises a dense noise candidate domain and a sparse noise candidate domain, and can lock noise events in a local range to help to carry out final noise judgment. The specific method for selecting the noise candidate domain is as follows: if the window mark value of a certain local window is larger than the threshold value of the dense noise candidate domain, the local window is regarded as the dense noise candidate domain; if the window mark value is smaller than the sparse noise candidate domain threshold value, the window mark value is regarded as a sparse noise candidate domain; otherwise, it is considered as a valid event domain. Wherein, the threshold value of the dense noise candidate domain is selected in a self-adaptive way according to the size of a local window; the sparse noise candidate domain threshold is adaptively selected by analyzing the global spatial distribution of events within the event stream block.
The noise screening module receives the event stream block from the block reading-in module, receives a two-dimensional space-time plane from the space-time plane construction module, receives a coordinate set at the upper left corner of a noise candidate domain in the current event stream block from the noise candidate domain selection module, calculates a noise threshold value based on the pixel activity value of the space-time plane, judges whether an event triggered at a certain pixel position is a noise event according to the noise threshold value in the noise candidate domain, marks the noise event, discards the noise event from the event stream block, and outputs the filtered asynchronous event stream. The specific method for judging the noise event comprises the following steps: if the number of events occurring at a pixel in a single event stream block is less than the noise threshold, the event at the pixel position is regarded as noise and is subjected to noise labeling.
The above description is only for the preferred embodiment of the present application, but the scope of the present application is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present application should be covered within the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (10)
1. A local spatiotemporal event stream filtering method based on adaptive thresholds, characterized by comprising the following steps:
a step of reading in blocks, which is to receive asynchronous event streams and read event stream blocks in a fixed time iteration mode;
a space-time plane construction step, namely receiving the event stream block, accumulating the number of events in a time dimension by taking a pixel as a unit, and calculating a pixel activity value to form a two-dimensional space-time plane;
a local sliding window step, namely receiving the two-dimensional space-time plane, setting a local window with a fixed size on the two-dimensional space-time plane, traversing the whole two-dimensional plane by a fixed step length sliding window, and calculating a window mark value of each sliding window;
a noise candidate domain selecting step of receiving the window flag value, calculating a noise candidate threshold value, and selecting a noise candidate domain according to the noise candidate threshold value;
and a noise screening step, namely receiving the event stream block, the two-dimensional space-time plane and the noise candidate domain, calculating a noise threshold value, marking a noise event, removing the noise event from the event stream block, and outputting the filtered asynchronous event stream.
2. The adaptive threshold-based local spatiotemporal event stream filtering method according to claim 1, characterized in that the asynchronous event stream is obtained by a dynamic visual sensor.
3. The adaptive threshold-based local spatiotemporal event stream filtering method according to claim 2, characterized in that said block reading step comprises in particular:
s1.1, initializing an event stream block index value i equal to 0, initializing an input event index value j equal to 0, initializing an intra-block time flag value T equal to 0, and initializing an block reproduction flag bit flag equal to True;
s1.2, reading data, if the read data is empty within 20ms, recording as no event input, and switching to S1.5, otherwise, receiving an event, recording as ej＝[xj，yj，tj，pj]Wherein x isj，yjAs an event ejPixel coordinate of (d), tjAs an event ejTime stamp of pjAs an event ejThe polarity of (1);
s1.3, judging whether the current block remaking flag bit is in a remaking state, if the flag is equal to True, establishing a new event stream block for the current event and switching to S1.4, otherwise, judging the current event ejWhether it belongs to the event stream block EiIf the current event timestamp tjIf the sum of the time mark value T in the block and the fixed time interval delta T is smaller than the sum, the step is switched to S1.4 if the event belongs to the event stream block, otherwise, the step is switched to S1.5 if the event belongs to the next event stream block;
s1.4, setting the current event ejAdding to the Current event stream Block EiIncreasing the event index value j by one, and turning to the step S1.3 to receive the next event;
s1.5, current event stream block EiAnd sending out, increasing the index value i of the event stream block by one, turning to S1.3 to read the next event stream block, wherein the flag bit flag of the block reproduction is True.
4. The adaptive threshold-based local spatiotemporal event stream filtering method according to claim 3, characterized in that the method of creating a new event stream block for the current event is: initializing an event stream Block EiLet the time mark value T in the block be TjThe block copy flag is False.
5. The adaptive threshold-based local spatiotemporal event stream filtering method according to claim 2, characterized in that said spatiotemporal plane construction step comprises in particular:
initializing a two-dimensional spatio-temporal plane NiW is sizeH, wherein W and H represent the dynamic vision sensor length and width, respectively, from which the data was generated;
receiving an event stream block EiCounting the event stream block E for each pixel positioniThe number of events occurring at the pixel position is recorded as pixel activity value, and the event stream block EiAll pixel activity values in the space-time plane form a two-dimensional space-time plane, and the pixel activity value corresponding to each pixel position (x, y) is
Wherein P is the polarity.
6. The adaptive threshold-based local spatiotemporal event stream filtering method according to claim 2, characterized in that said local sliding window step comprises in particular:
s3.1, initializing global pixel flag bit P with size of W × HiWherein W and H represent the dynamic vision sensor length and width, respectively, of the generated data, initializing the window flag value
Wherein m and n respectively represent the abscissa and ordinate values of the upper left corner of the current local window;
s3.2, receiving the two-dimensional space-time plane Ni；
S3.3, for x is more than or equal to 0 and less than or equal to W and y is more than or equal to 0 and less than or equal to H, making
S3.4, initializing the upper left corner of a local window to be marked as (0, 0), wherein the window comprises L multiplied by L pixel units in total and is positioned on a two-dimensional space-time plane NiUpper, partial window HiSliding from top to bottom and from left to right by step length S until traversing a complete two-dimensional space-time plane, in the process of sliding a local window, if boundary coordinates of a certain local window are out of range, the local window is regarded as an invalid local window and is not included in calculation, otherwise, window mark values are calculated, and window mark values are calculatedThe method comprises
7. The adaptive threshold-based local spatiotemporal event stream filtering method according to claim 2, characterized in that said noise candidate domain selection step comprises in particular:
initializing a dense candidate domain threshold value hyperparameter alpha which is 0.9, and initializing a sparse candidate domain threshold value hyperparameter beta which is 0.9;
receiving the window flag value
Computing a dense noise candidate domain threshold
Wherein L is the side length of the local window;
computing sparse noise candidate domain thresholds
Wherein,
marking values by all windows
Sorting from large to small to remove zero value, K1Candidate domain thresholding in sequence for sparse noise
Index subscript of (1), by the sequence
The length of the sparse candidate domain is multiplied by the threshold value over-parameter of the sparse candidate domain and rounded upwards;
according to the threshold Th of the dense noise candidate domaindenseAnd the sparse noise candidate domain threshold ThsparseSelecting a noise candidate domain, and for a local window with (m, n) coordinates at the upper left corner, if the window mark value of the window is
Greater than threshold Th of dense noise candidate domaindenseOr less than the sparse noise candidate domain threshold ThsparseAdding the coordinates (M, n) of the upper left corner into the set M of coordinates of the upper left corner of the noise candidate domainiIn (1).
8. The adaptive threshold-based local spatiotemporal event stream filtering method according to claim 2, characterized in that the noise filtering step specifically comprises:
s5.1, setting an initialization noise threshold value hyperparameter gamma to be 0.9;
s5.2, receiving event stream block EiReceiving a two-dimensional space-time plane NiReceiving a set M of upper left corner coordinates of a noise candidate fieldi；
S5.3, calculating a noise threshold value
Wherein,
from a two-dimensional space-time plane NiAll pixel activity value Ni(x, y) is obtained by sorting from large to small and removing zero values, K2For noise in the sequence
Index subscript of (1), by sequence
Length and noise threshold value of (2) over-parameter gammaMultiplying to obtain;
s5.4, according to the noise threshold ThnoiseScreening noise event, marking it, if it is corresponding pixel activity value N for certain pixel position (x, y)i(x, y) is less than a noise threshold ThnoiseThe event contained in the pixel is considered as a noise event, which is added to the noise set FiPerforming the following steps;
s5.5, from event stream block EiMid-erasure noise set FiObtaining a filtered asynchronous event stream Di；
S5.6, filtering the asynchronous event stream DiAnd (6) outputting.
9. An adaptive threshold-based local spatiotemporal event stream filtering system, comprising:
a block reading module for executing the block reading step, receiving the asynchronous event stream and iteratively reading the event stream block at a fixed time;
the space-time plane construction module is used for executing the space-time plane construction step, receiving the event stream block, accumulating the number of events in a time dimension by taking a pixel as a unit, and calculating a pixel activity value to form a two-dimensional space-time plane;
a local sliding window module for executing the local sliding window step, receiving the two-dimensional space-time plane and setting a local window with a fixed size on the two-dimensional space-time plane, traversing the whole two-dimensional plane by a fixed step length sliding window, and calculating a window mark value of each sliding window;
a noise candidate domain selection module for executing a noise candidate domain selection step, receiving the window flag value, calculating a noise candidate threshold value, and selecting a noise candidate domain according to the noise candidate threshold value;
and the noise screening module is used for executing a noise screening step, receiving the event stream block, the two-dimensional space-time plane and the noise candidate domain, calculating a noise threshold value, marking a noise event, removing the noise event from the event stream block, and outputting the filtered asynchronous event stream.
10. The adaptive threshold-based local spatiotemporal event stream filtering system according to claim 9, characterized in that the asynchronous event stream is input by a dynamic visual sensor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110435263.5A CN113269683B (en) | 2021-04-22 | 2021-04-22 | Local space-time event stream filtering method and system based on self-adaptive threshold |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110435263.5A CN113269683B (en) | 2021-04-22 | 2021-04-22 | Local space-time event stream filtering method and system based on self-adaptive threshold |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113269683A true CN113269683A (en) | 2021-08-17 |
| CN113269683B CN113269683B (en) | 2023-04-18 |
Family
ID=77229097
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110435263.5A Active CN113269683B (en) | 2021-04-22 | 2021-04-22 | Local space-time event stream filtering method and system based on self-adaptive threshold |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113269683B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113923319A (en) * | 2021-12-14 | 2022-01-11 | 成都时识科技有限公司 | Noise reduction device, noise reduction method, chip, event imaging device and electronic equipment |
| CN114285962A (en) * | 2021-12-14 | 2022-04-05 | 成都时识科技有限公司 | Noise processing device, method, chip, event imaging device and electronic equipment |
| CN114885074A (en) * | 2022-05-06 | 2022-08-09 | 中国科学院光电技术研究所 | Event camera denoising method based on space-time density |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2012040157A1 (en) * | 2010-09-20 | 2012-03-29 | The Government Of The United States Of America, As Represented By The Secretary Of The Navy | Flash detection and clutter rejection processor |
| CN110176028A (en) * | 2019-06-05 | 2019-08-27 | 中国人民解放军国防科技大学 | Asynchronous corner detection method based on event camera |
| CN111340699A (en) * | 2020-02-17 | 2020-06-26 | 西北大学 | Magnetic resonance image denoising method and device based on non-local prior and sparse representation |
| CN111770290A (en) * | 2020-07-29 | 2020-10-13 | 中国科学院长春光学精密机械与物理研究所 | Noise reduction method for dynamic vision sensor output event stream |
| CN111931752A (en) * | 2020-10-13 | 2020-11-13 | 中航金城无人系统有限公司 | Dynamic target detection method based on event camera |
| CN112184572A (en) * | 2020-09-14 | 2021-01-05 | 中山大学 | Novel rain removing method and system for dynamic vision sensor event stream |
| CN112307078A (en) * | 2020-09-29 | 2021-02-02 | 安徽工业大学 | Data stream differential privacy histogram publishing method based on sliding window |
-
2021
- 2021-04-22 CN CN202110435263.5A patent/CN113269683B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2012040157A1 (en) * | 2010-09-20 | 2012-03-29 | The Government Of The United States Of America, As Represented By The Secretary Of The Navy | Flash detection and clutter rejection processor |
| CN110176028A (en) * | 2019-06-05 | 2019-08-27 | 中国人民解放军国防科技大学 | Asynchronous corner detection method based on event camera |
| CN111340699A (en) * | 2020-02-17 | 2020-06-26 | 西北大学 | Magnetic resonance image denoising method and device based on non-local prior and sparse representation |
| CN111770290A (en) * | 2020-07-29 | 2020-10-13 | 中国科学院长春光学精密机械与物理研究所 | Noise reduction method for dynamic vision sensor output event stream |
| CN112184572A (en) * | 2020-09-14 | 2021-01-05 | 中山大学 | Novel rain removing method and system for dynamic vision sensor event stream |
| CN112307078A (en) * | 2020-09-29 | 2021-02-02 | 安徽工业大学 | Data stream differential privacy histogram publishing method based on sliding window |
| CN111931752A (en) * | 2020-10-13 | 2020-11-13 | 中航金城无人系统有限公司 | Dynamic target detection method based on event camera |
Non-Patent Citations (3)
| Title |
|---|
| SHASHA GUO ET AL.: "A noise filter for dynamic vision sensors using self-adjusting threshold", 《ARXIV:2004.04079V2》 * |
| YANG FENG ET AL.: "Event Density Based Denoising Method for Dynamic Vision Sensor", 《APPLIED SCIENCE》 * |
| 闫昌达等: "基于事件相机的可视化及降噪算法", 《北京航空航天大学学报》 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113923319A (en) * | 2021-12-14 | 2022-01-11 | 成都时识科技有限公司 | Noise reduction device, noise reduction method, chip, event imaging device and electronic equipment |
| CN114285962A (en) * | 2021-12-14 | 2022-04-05 | 成都时识科技有限公司 | Noise processing device, method, chip, event imaging device and electronic equipment |
| CN114285962B (en) * | 2021-12-14 | 2023-04-07 | 成都时识科技有限公司 | Noise processing device, method, chip, event imaging device and electronic equipment |
| CN114885074A (en) * | 2022-05-06 | 2022-08-09 | 中国科学院光电技术研究所 | Event camera denoising method based on space-time density |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113269683B (en) | 2023-04-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113269683B (en) | Local space-time event stream filtering method and system based on self-adaptive threshold | |
| Irani et al. | Detecting and tracking multiple moving objects using temporal integration | |
| JP6616521B2 (en) | Image processing device | |
| US7983501B2 (en) | Noise detection and estimation techniques for picture enhancement | |
| US20070147697A1 (en) | Method for removing noise in image and system thereof | |
| KR20070040026A (en) | A method for removing noise in image using statistical information and a system thereof | |
| Chen et al. | Video Carving. | |
| JP2011130203A (en) | Video information processing method and apparatus therefor | |
| Chen et al. | A residual learning approach to deblur and generate high frame rate video with an event camera | |
| CN116385316B (en) | Multi-target image dynamic capturing method and related device | |
| CN106934819A (en) | A kind of method of moving object segmentation precision in raising image | |
| CN108270945B (en) | Motion compensation denoising method and device | |
| Aloraini et al. | Statistical sequential analysis for object-based video forgery detection | |
| Fang et al. | AEDNet: Asynchronous event denoising with Spatial-Temporal correlation among irregular data | |
| She et al. | SpikeFormer: Image Reconstruction from the Sequence of Spike Camera Based on Transformer | |
| TWI381735B (en) | Image processing system and method for automatic adjustment of image resolution for image surveillance apparatus | |
| Chae et al. | Siamevent: Event-based object tracking via edge-aware similarity learning with siamese networks | |
| Mahayuddin et al. | Efficiency measurement of various denoise techniques for moving object detection using aerial images | |
| Xu et al. | Features based spatial and temporal blotch detection for archive video restoration | |
| Moura et al. | A spatiotemporal motion-vector filter for object tracking on compressed video | |
| Antic et al. | Efficient wavelet based detection of moving objects | |
| CN111754477B (en) | Railway perimeter foreign matter intrusion detection method based on dynamic candidate area multi-scale image | |
| CN112184572A (en) | Novel rain removing method and system for dynamic vision sensor event stream | |
| Gao et al. | People counting using combined feature | |
| Kumar et al. | Cut scene change detection using spatio temporal video frame |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |