CN116546339B - Noise reduction device and method for distinguishing event polarity, chip and electronic equipment - Google Patents
Noise reduction device and method for distinguishing event polarity, chip and electronic equipment Download PDFInfo
- Publication number
- CN116546339B CN116546339B CN202310826935.4A CN202310826935A CN116546339B CN 116546339 B CN116546339 B CN 116546339B CN 202310826935 A CN202310826935 A CN 202310826935A CN 116546339 B CN116546339 B CN 116546339B
- Authority
- CN
- China
- Prior art keywords
- event
- polarity
- value
- storage space
- pixel
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 37
- 230000009467 reduction Effects 0.000 title claims abstract description 28
- 238000003860 storage Methods 0.000 claims abstract description 115
- 238000001914 filtration Methods 0.000 claims abstract description 46
- 238000003384 imaging method Methods 0.000 claims description 25
- 239000003638 chemical reducing agent Substances 0.000 claims description 16
- 238000004364 calculation method Methods 0.000 claims description 7
- 238000013528 artificial neural network Methods 0.000 claims description 3
- 230000000694 effects Effects 0.000 abstract description 14
- 238000010586 diagram Methods 0.000 description 13
- 230000008569 process Effects 0.000 description 8
- 230000004048 modification Effects 0.000 description 6
- 238000012986 modification Methods 0.000 description 6
- 208000014674 injury Diseases 0.000 description 5
- 230000000630 rising effect Effects 0.000 description 5
- 230000008901 benefit Effects 0.000 description 4
- 230000008859 change Effects 0.000 description 4
- 238000006467 substitution reaction Methods 0.000 description 4
- 208000012260 Accidental injury Diseases 0.000 description 3
- 230000004397 blinking Effects 0.000 description 3
- 238000005259 measurement Methods 0.000 description 3
- XUIMIQQOPSSXEZ-UHFFFAOYSA-N Silicon Chemical compound [Si] XUIMIQQOPSSXEZ-UHFFFAOYSA-N 0.000 description 2
- 208000027418 Wounds and injury Diseases 0.000 description 2
- 230000006378 damage Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000008520 organization Effects 0.000 description 2
- 229910052710 silicon Inorganic materials 0.000 description 2
- 239000010703 silicon Substances 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 230000003213 activating effect Effects 0.000 description 1
- 238000007792 addition Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000001747 exhibiting effect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 239000011664 nicotinic acid Substances 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000000844 transformation Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/60—Noise processing, e.g. detecting, correcting, reducing or removing noise
- H04N25/61—Noise processing, e.g. detecting, correcting, reducing or removing noise the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4"
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/47—Image sensors with pixel address output; Event-driven image sensors; Selection of pixels to be read out based on image data
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/70—SSIS architectures; Circuits associated therewith
- H04N25/703—SSIS architectures incorporating pixels for producing signals other than image signals
- H04N25/707—Pixels for event detection
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Picture Signal Circuits (AREA)
- Studio Devices (AREA)
- Transforming Light Signals Into Electric Signals (AREA)
Abstract
The application discloses a noise reduction device and method for distinguishing event polarity, a chip and electronic equipment. In order to solve the difficult problems that flicker noise is difficult to filter and the actual application is difficult to influence, the method for distinguishing the event polarity is used for classifying and processing pulse events with different polarities, and different storage spaces are arranged for the pulse events, so that the method is used for filtering the flicker noise caused by continuous flicker of alternating current. According to the polarity classification processing of the newly generated event, the application obtains a better flicker noise reduction effect. The application is suitable for the fields of event cameras and neuromorphic chips.
Description
Technical Field
The present application relates to a noise reduction device and method, a chip and an electronic device for distinguishing event polarity, and in particular, to a noise reduction device and method, a chip and an electronic device for distinguishing event polarity.
Background
An event camera (event camera), which is good at capturing moving objects in a field of view, has the characteristics of low power consumption and low delay, is a novel bionic vision sensor, but still faces a plurality of difficult challenges in the application process of the technology. For example, alternating current illumination is commonly used in the environment, and this can lead to the light source it drives constantly blinking (flicker), causing event cameras to constantly generate events (referred to as flicker noise or flicker noise events) that create undesirable false "motion" scenes or result in very poor imaging quality, thereby affecting subsequent information processing difficulty and capability.
Flicker noise is very troublesome, it is caused by the optical characteristics of the event camera, and is often difficult to avoid due to the high optical sensitivity of the event camera, and the flicker light source causes all actions, such as body movements or gestures, to be submerged in the flickering light rays, as shown in fig. 1. Part (a) of fig. 1 is an event camera imaging effect under normal lighting conditions, and part (b) of fig. 1 is an event camera imaging effect under strobe lighting conditions.
Currently, some schemes for eliminating flicker noise exist, but most of the schemes are not event driven, are complex in calculation, and have disadvantages in terms of cost/area, power consumption, delay and the like. Applicant previously proposed an event-driven flicker noise filtering method (prior art 1: cn115412688 b), which is centered on obtaining a confidence (degree) count (i.e. a first value) by setting an appropriate time interval threshold (first threshold), so that flicker noise is filtered well.
However, in the practical application process, the applicant finds that the fluctuation speed of the light intensity change is uneven due to various reasons such as unstable alternating current, so that the rising and falling speeds of the light intensity change waveform are greatly different, and the random occurrence of the flicker noise caused by the rising and falling speeds is caused. Actually measuring intensity variation waveforms (abbreviated as light waves) of the light source under various environments, wherein the waveforms show asymmetric characteristics, as shown in fig. 2; the time interval at which the pixel points generate flicker noise events is also uneven, as shown in fig. 3.
Based on at least this, it becomes difficult or desirable to select an appropriate time interval as the first threshold for the actual application scenario in the prior art 1, and an improper first threshold may cause an unclean noise filtering or a false injury effective event, thereby affecting the actual application.
In particular, when the event camera faces the light source, the appropriate first threshold is particularly important, and fig. 4 shows the influence of the relative position of the event camera and the light source on flicker noise, when the event camera faces away from the flicker light source, the flicker noise is small, and when the event camera faces (or faces) the light source, the flicker noise is almost full-screen, and is most influenced by the flicker noise.
For 50Hz ac, the flicker frequency of the light source intensity is 100Hz (period is 10 ms), and ideally, the interval between adjacent flicker noise is 5ms uniformly.
If the scheme in the prior art 1 is adopted, the actual measurement result (the main parameter is that the first threshold is set to 25ms and the values near the first threshold, and all events above 40Hz are filtered) shows that although all flicker noise can be filtered, a plurality of effective events are accidentally injured, and the imaging quality still has a gap from the actual requirement. In addition, if the first threshold is 5ms or 7ms and the values near it, the flicker noise cannot be completely eliminated, and the flicker noise with polarity 1 is not successfully filtered out as shown in fig. 5.
Based on this, for the scheme closer to practical application, how to effectively eliminate flicker noise and avoid false injury to effective events, i.e. as many events for retaining and reflecting real motion information as possible so as to effectively respond to dynamic information, is a technical problem to be solved in the event camera commercialized application process.
Disclosure of Invention
In order to solve or alleviate some or all of the above technical problems, the present application is implemented by the following technical solutions:
the noise reduction device for distinguishing the event polarity is used for filtering flicker noise and comprises a storage module, a calculation module and a filtering module;
the storage module at least comprises a first storage space, a second storage space and a third storage space, wherein the first storage space, the second storage space and the third storage space comprise a plurality of storage units, and the storage units in each storage space have a one-to-one correspondence with pixels in a pixel array of the event imaging device;
the first storage space stores a first value, the second storage space stores a time stamp of a first polarity event, and the third storage space stores a time stamp of a second polarity event;
the computing module acquires a timestamp of a first event newly generated by a first pixel in the pixel array of the event imaging device and the polarity of the timestamp, acquires a second storage unit corresponding to the first pixel in a second storage space or acquires a timestamp stored in a third storage unit corresponding to the first pixel in a third storage space according to the acquired polarity, and acquires a time interval according to the difference between the timestamp of the first event and the stored timestamp;
the filtering module is used for judging whether the time interval is matched with the flicker frequency of the light source or not, and updating a first numerical value stored in a first storage unit corresponding to the first pixel in the first storage space according to a matching result;
the filtering module filters or releases the first event according to the first value before or after updating.
In some embodiments, if the time interval matches the flicker frequency, the first value stored in the first memory location in the first memory space is incremented by a first non-zero constant, and otherwise, the second non-zero constant is subtracted to update the first value.
In some types of embodiments, the first value is a trust count value or a number of matches;
if the first numerical value is a trust count value, filtering the first event when the trust count value is greater than or equal to a first threshold value;
if the first value is the continuous matching times, filtering the first event when the continuous matching times are greater than or equal to M times, wherein M is a positive integer.
In some class of embodiments, the first value is a trust count value or a number of consecutive mismatches;
if the first value is a trust count value, releasing the first event when the trust count value is smaller than or equal to a second threshold value;
and if the first numerical value is the continuous unmatched times, releasing the first event when the continuous unmatched times are greater than or equal to N times, wherein N is a positive integer.
In some types of embodiments, the first non-zero constant is less than the second non-zero constant.
In certain classes of embodiments, the first non-zero constant is equal to the second non-zero constant; alternatively, N is greater than or equal to 2.
In some class of embodiments, the time stamp in the second or third storage unit is updated in an overlapping manner to the time stamp of the first event, depending on the polarity of the first event.
In certain types of embodiments, the second storage unit or/and the third storage unit store a lower accuracy of the time stamp than when the event imaging device generated the event.
In a class of embodiments, the event imaging device is an event camera.
A noise reduction method for differentiating event polarity for filtering flicker noise, comprising:
acquiring a timestamp and polarity of a first event newly generated by a first pixel in a pixel array of the event imaging device;
according to the polarity of the first event, acquiring a time stamp of a previous event which is generated by the first pixel and has the same polarity as the first event and is adjacent to the first event, and calculating the difference between the time stamp of the first event and the time stamp of the previous event to obtain a time interval;
comparing whether the time interval is matched with the flicker frequency of the light source, updating a first numerical value stored in a first storage unit corresponding to the first pixel in the first storage space according to the matching result, and filtering or releasing a first event according to the first numerical value before or after updating.
In certain classes of embodiments, the polarity of the first event includes a first polarity and a second polarity.
In certain embodiments, a second storage unit of the second storage space corresponding to the first pixel stores a timestamp of the first polarity event; a third storage unit of the third storage space corresponding to the first pixel stores a timestamp of the second polarity event;
and updating the time stamp of the previous event stored in a second storage unit corresponding to the first pixel in the second storage space or a third storage unit corresponding to the first pixel in the third storage space into the time stamp of the first event in a coverage mode according to the polarity of the first event.
In some class of embodiments, the second storage unit or/and the third storage unit stores a lower accuracy of the time stamp than when the event imaging device generated the event.
In some embodiments, if the time interval matches the flicker frequency, the first value stored in the first memory location in the first memory space is incremented by a first non-zero constant, and otherwise, the second non-zero constant is subtracted to update the first value.
In some types of embodiments, the first value is a trust count value or a number of matches;
if the first numerical value is a trust count value, filtering the first event when the trust count value is greater than or equal to a first threshold value;
if the first value is the continuous matching times, filtering the first event when the continuous matching times are greater than or equal to M times, wherein M is a positive integer.
In some class of embodiments, the first value is a trust count value or a number of consecutive mismatches;
if the first value is a trust count value, releasing the first event when the trust count value is smaller than or equal to a second threshold value;
and if the first numerical value is the continuous unmatched times, releasing the first event when the continuous unmatched times are greater than or equal to N times, wherein N is a positive integer.
In some types of embodiments, the first non-zero constant is less than the second non-zero constant.
In certain classes of embodiments, the first non-zero constant is equal to the second non-zero constant; alternatively, N is greater than or equal to 2.
In a class of embodiments, the event imaging device is an event camera.
A chip comprising a noise reduction device for distinguishing event polarity as described above or filtering flicker noise using a noise reduction method for distinguishing event polarity as described above.
In certain classes of embodiments, the chip is a vision sensor chip, or a neural network processor chip, or a sense-all-in-one chip that integrates a vision sensor with a processor at the same time.
An electronic device configured with a chip as described above.
Some or all embodiments of the present application have the following beneficial technical effects:
1) By distinguishing the event polarity, the application avoids the influence of asymmetric light waves in an actual scene, can more effectively eliminate flicker noise and avoids the accidental injury of effective events.
2) The application takes advantage of the trust counting scheme in the prior art 1 as a reference, has event driving characteristics, directly processes the event stream, and has the characteristics of ultra-high event resolution and ultra-low power consumption of the event camera without sacrificing. And from newly activating the pixel when the count value is below a certain value, the pixel is prevented from being blocked all the time.
3) The application can be realized through simple addition, subtraction and comparison calculation, has simple scheme and less calculation amount, and has advantages in the aspects of cost/area, power consumption, delay and the like.
4) The application can effectively filter flicker noise and retain a large number of effective events in various actual measurement environments such as the event camera facing the light source, the backlight source and the like, the data quality is greatly improved, and the imaging quality is good.
5) The method can be suitable for complex actual scenes, and has good robustness in the environment with asymmetric light waves caused by random reasons such as bandwidth, light intensity variation fluctuation and the like.
Further advantageous effects will be further described in the preferred embodiments.
The above-described technical solutions/features are intended to summarize the technical solutions and technical features described in the detailed description section, and thus the ranges described may not be exactly the same. However, these new solutions disclosed in this section are also part of the numerous solutions disclosed in this document, and the technical features disclosed in this section and the technical features disclosed in the following detailed description section, and some contents in the drawings not explicitly described in the specification disclose more solutions in a reasonable combination with each other.
The technical scheme combined by all the technical features disclosed in any position of the application is used for supporting the generalization of the technical scheme, the modification of the patent document and the disclosure of the technical scheme.
Drawings
FIG. 1 is an exemplary diagram of the effect of flicker noise on event camera imaging;
FIG. 2 is a schematic diagram of a light source waveform in a practical application environment;
FIG. 3 is a schematic diagram of a distribution of flicker noise events generated at a pixel point of an event camera;
FIG. 4 is a timing diagram of the present application for differentiating polarity of flicker noise events in a practical application environment;
FIG. 5 is an exemplary flicker noise filtering effect diagram;
FIG. 6 is a timing diagram for differentiating polarity of flicker noise events in a practical application environment;
FIG. 7 is a schematic diagram of a flicker noise filtering scheme to distinguish event polarities;
FIG. 8 is a flow chart of a flicker noise filtering scheme to distinguish event polarities;
FIG. 9 is a flicker noise filtering effect of the present application on a chip to distinguish event polarity;
FIG. 10 is a comparison of the flicker noise filtering effect of the present application with prior art 1;
FIG. 11 is a schematic diagram of the generation of false flicker noise in a certain class of application environments;
FIG. 12 is a schematic illustration of an event flow associated with a blinking light source;
fig. 13 is a schematic diagram of a noise reducer that distinguishes between event polarities.
Detailed Description
Since various alternatives are not exhaustive, the gist of the technical solution in the embodiment of the present application will be clearly and completely described below with reference to the drawings in the embodiment of the present application. Other technical solutions and details not disclosed in detail below, which generally belong to technical objects or technical features that can be achieved by conventional means in the art, are limited in space and the present application is not described in detail.
Except where division is used, any position "/" in this disclosure means a logical "or". The ordinal numbers "first", "second", etc., in any position of the present application are used merely for distinguishing between the labels in the description and do not imply an absolute order in time or space, nor do they imply that the terms preceded by such ordinal numbers are necessarily different from the same terms preceded by other ordinal terms.
The present application will be described in terms of various elements for use in various combinations of embodiments, which elements are to be combined in various methods, products. In the present application, even if only the gist described in introducing a method/product scheme means that the corresponding product/method scheme explicitly includes the technical feature.
The description of a step, module, or feature in any location in the disclosure does not imply that the step, module, or feature is the only step or feature present, but that other embodiments may be implemented by those skilled in the art with the aid of other technical means according to the disclosed technical solutions. The embodiments of the present application are generally disclosed for the purpose of disclosing preferred embodiments, but it is not meant to imply that the contrary embodiments of the preferred embodiments are not intended to cover all embodiments of the application as long as such contrary embodiments are at least one technical problem addressed by the present application. Based on the gist of the specific embodiments of the present application, a person skilled in the art can apply means of substitution, deletion, addition, combination, exchange of sequences, etc. to certain technical features, so as to obtain a technical solution still following the inventive concept. Such solutions without departing from the technical idea of the application are also within the scope of protection of the application.
The expression greater than, less than, etc. in the present application is essentially a logical comparison, and the boundary value may be modified slightly to obtain the same logical comparison result, but this is merely an equivalent conventional alternative means in the art, such as ". Gtoreq.2" and "> 1", and the comparison logical result is equivalent in some cases. These basic logical transformations or boundary value modifications, etc., may generally be logically altered, replaced by those skilled in the art, without departing from the basic concepts of the present application as such, and remain within the scope of the present application.
The present application incorporates by reference the entire contents of prior art 1 as part of the present disclosure and uses its terminology.
Event imaging: the essence is an event driven image sensor, such as an event camera, also known as a Dynamic Vision Sensor (DVS). Based on this principle, there are some solutions to fuse it with the pixels of the conventional frame image, and the obtained sensor can output both an event and a pixel brightness, gray level or other values, such as a DAVIS sensor, an ATIS sensor, and these event-based sensors (EBSs) are collectively referred to as event imaging devices in the present application, which belong to one of the sensors. The event camera is taken as an example, but not limited thereto. Noise events, valid events, are events or impulse events.
Flicker noise: alternating current causes the light source to continually blink, exciting the event camera to continually produce undesirable events (also known as pulses or pulsed events), resulting in very poor imaging quality.
As shown in fig. 1, in real-world situations, when an object is moving, such as waving a hand, pixels are excited within the field of view of the event camera to generate an effective event, but they are almost submerged in flicker noise. How to completely retain the effective events from the pulse train received in real-world situations, filter out flicker noise, and obtain ideal results is a difficult challenge.
FIG. 6 is a timing diagram of differentiating polarity of flicker noise events in a practical application environment according to the present application. When the light wave rises, the pixel point generates a flicker noise event with the polarity of 0 (ON event); when the light wave drops, the pixel point generates a flicker noise event with polarity of 1 (OFF event), and 0/1 is just one marking method of the light intensity change trend. The inventors of the present application found that by differentiating the polarities of the flicker noise, the flicker noise of the same polarity is more nearly uniformly distributed, and the time intervals between adjacent flicker noise of the same polarity are all about 10ms, which is much more stable in anti-flicker noise performance than in the case of the scheme without differentiating the polarities.
FIG. 7 shows the present applicationA flicker noise filtering scheme diagram clearly distinguishing event polarities. Without loss of generality, for any pixel included in the event camera, such as the first pixel (which means that the operation of the first pixel herein applies to other pixels of the event camera as well), for example, its coordinates in the event camera are (x, y), and for a certain period of time, the pixel outputs an event e with polarity 0 0 Event e 0 Previous event "e 0 -1"; event e with polarity 1 1 Event e 1 Previous event "e 1 -1”。e 0 ±i、e 1 I is any positive integer, and i is two different polarity events generated by the same pixel.
For the pulse events generated by the same pixel, the time stamps of the same-polarity events are stored in the same storage unit according to the sequence of the generation time according to the difference of the polarities of the pulse events. Preferably, the time stamps of the same polarity events are stored by overriding. As shown in fig. 7, in an exemplary embodiment, three storage spaces, that is, a first storage space, a second storage space, and a third storage space, are provided, each storage space includes a plurality of storage units, and the storage units in the storage spaces have a one-to-one correspondence with pixels in the event camera pixel array. For example, one simple mapping logic is: the pixel generating the first event e has coordinates (x, y) corresponding to the (x, y) memory location in each of the three memory spaces. This logical mapping may be in any reasonable manner, and is not limited in this regard by the present application.
The first memory space is at least used for storing a first value (which may be a trust count value, or a continuous matching number or/and a continuous unmatching number), the second memory space is at least used for storing a time stamp of an event with polarity 0, and the third memory space is at least used for storing a time stamp of an event with polarity 1. For a pulse event newly generated by the first pixel of the event camera, its timestamp is stored in the second storage unit of the second storage space (when p=0) or in the third storage unit of the third storage space (when p=1) according to its polarity (p=0 or p=1).
Preferably, the second memory location or the third memory location is stored in an overwrite (i.e., the previous value is replaced) while being written. This helps save memory space and reduce die silicon cost overhead.
Preferably, the time stamp stored in the memory unit has a lower precision than the time stamp at the time of the generation of the pulse event. This also helps to save memory space and reduce die silicon cost overhead.
FIG. 8 is a flow chart of a flicker noise filtering scheme for differentiating event polarities in accordance with certain embodiments of the present application. The pulse event polarity generated by the event camera can be divided into case 1 and case 2.
The generation time interval delta of events which are generated by the first pixel and have the same polarity and are adjacent in generation time is calculated. For example, for event e with polarity 0 0 The previous event "e" whose polarity is the same as that of the previous event should be calculated 0 -time interval between 1"; for event e with polarity 1 1 The previous event "e" whose polarity is the same as that of the previous event should be calculated 1 -time interval between 1 ". The time interval may be obtained by calculating the difference between the time stamps of the events.
If the first pixel of the event camera continues to generate a new pulse event, e.g. e 0 +1, the difference between the corresponding previous event time stamps is taken out, and the corresponding previous event time stamp is covered, which should be equal to e 0 The associated timestamp inherits the previous example as a second storage unit stored in a second storage space. If it is e 1 +1, then it should be e of the third memory cell stored in the third memory space 1 A time stamp.
It is determined whether the time interval delta is equal to the flicker frequency of the light source (e.g.,) Matching. If the time interval is matched with the flicker period, the first value stored in the first storage unit in the first storage space is increased by a first nonzero constant, and otherwise, the second nonzero constant is subtracted to update the first value. The matching here may be, for example, the flicker noise of the time interval delta theoryThe period delta is within a reasonable error interval.
In certain embodiments, the first event is filtered out or released based on the first value before or after the update. If the first event is filtered or released by using the first value before updating, it is more advantageous to select various thresholds (M, N, first/second thresholds, etc.) with better noise reduction effect for the embodiment that only uses the 3-bit first memory cell to store the first value. Fig. 8 shows the use of the updated first value to filter or release the first event.
By approximately equal, it is meant: the time interval delta is within a reasonably narrow range (delta) around the flicker noise period delta 1 ，δ 2 ) Within, the time interval can be considered to be matched to the flicker frequency of the light source, wherein delta 1 <δ<δ 2 Exemplary delta 1 =9.8<Δ<10.2=δ 2 The time interval is considered to be within a reasonably narrow range of the flicker period.
Theoretically, for a frequency of use ofIs the flicker noise period caused by the light source of the alternating current +.>. However, the pulse events in real-world situations, such as unsteady changes in light intensity and bandwidth resource limitations, tend to be much more complex than this, which can result in flicker noise events exhibiting various random characteristics.
If the first numerical value is the continuous matching times, filtering the first event when the continuous matching times are more than or equal to M times; and if the first numerical value is the trust count value, filtering the first event when the trust count value is greater than or equal to a first threshold value. In this case, the first pixel is suppressed, and an event generated by the first pixel is blocked (block) or filtered (flicker noise is determined).
If the first value is a trust count value, releasing the first event when the trust count value is smaller than or equal to a second threshold value; and if the first value is the number of continuous mismatch times, releasing the first event when the number of continuous mismatch times is greater than or equal to N (such as greater than or equal to 2). In this case, the pixels are activated, allowing events generated by the first pixel to pass (determined to be non-flicker noise) the filtering module. Wherein M, N is a positive integer and includes the same case.
In a preferred embodiment, the first non-zero constant value is smaller than the second non-zero constant value, for example, the first non-zero constant value is 1, and the second non-zero constant value is 3, which has the advantages of enabling the pixels to be activated quickly, reducing the duration that the pixels are blocked, enabling the perception of the effective target to be restored quickly, filtering out flicker noise, and avoiding the accidental injury of the effective event effectively. In yet another class of embodiments, the first non-zero constant is equal to the second non-zero constant, which helps filter out spurious events.
It should be noted that the first non-zero constant and the second non-zero constant may also be negative numbers in the present application, which thus results in a logical flip of the size decision, which is a common alternative in the art, and is also within the inventive concept. The signal-to-noise count value in the application is a credibility representing a pixel output pulse event. If the above-mentioned match determination result is a match, a first value as a confidence count value is incremented by a non-zero constant (e.g., +1), in which case the confidence count value is an inverted indicator, the larger the value is, the more unreliable, so that the pixel should be suppressed when the confidence count value is greater than or equal to the first threshold.
Alternatively, the time stamp stored in the storage unit corresponding to the pixel with coordinates (x, y) in the second storage space may be updated in any reasonable step. The time stamp as stored in the storage unit is formed by ts (e 0 -1) update to ts (e) 0 ) Or from ts (e) 1 -1) update to ts (e) 1 ). The corresponding time stamp is updated, illustratively, at the same time as the time interval is calculated, or at some step after the time interval is calculated. For example, to perform the time stamp difference, the time stamp is extracted after ts (e 0 -1) or ts (e) 1 -1) immediately updating the value of the second memory location to ts (e) 0 ) Or the value in the third memory cell is ts (e 1 )。
FIG. 9 illustrates the flicker noise filtering effect of the present application for differentiating event polarity on a chip. The part (a) of fig. 9 is data before noise reduction, and the part (b) of fig. 9 is data after noise reduction (both are effects shown after frame compression). Compared with the prior art, the flicker noise filtering scheme for distinguishing the event polarity has the remarkable advantage that the effective event can be completely reserved while the flicker noise is completely filtered, so that the accuracy of motion recognition is improved.
Fig. 10 is a comparison of the flicker noise filtering effect of the present application with that of prior art 1. From the change of the number of events before and after filtering and the proportion of the filtered events, the number of flicker noise is far higher than that of effective events, and the effective events are sparsely interspersed (or exist) in the event stream output by the event camera, so that when filtering the flicker noise, how to avoid the accidental injury of the effective events is a difficult challenge. The application can completely retain the effective events while effectively filtering the flicker noise.
In addition to the uneven fluctuation speed of the light intensity variation (inconsistent rising and falling speeds), the amplitude or depth of the light intensity fluctuation also affects the effect of the existing noise reduction scheme.
Ideally, the event camera alternately generates the events with the polarity of 0 and the polarity of 1, however, in the actual measurement process, it is found that, although the interval between the 0 event and the 1 event and the 0 event generated by the light wave flicker is 7ms and 3ms respectively, the event camera is very sensitive and has extremely high response speed, and can generate the events within a few milliseconds, if the rising speed of the light wave fluctuates too much, two (or more) flicker noise events may be generated in the rising process, and at this time, the latter flicker noise event may be called as a "false event" or a "false event".
FIG. 11 illustrates a schematic diagram of the generation of false flicker noise in a certain class of application environments. Due to some randomness, false events are not generated around every 0 event, so that the time interval between flicker noise with polarity of 0 is not unique, and is not only 10ms, for example, the time interval between a false event and the previous 0 event is 3ms, and the time interval between the false event and the next adjacent 0 event is 7ms.
As an example, fig. 12 shows a schematic view of an event flow associated with a blinking light source, labeled according to polarity: 0101001010010010101 … …. In an example, if the result of the foregoing determination is a match, the count value is trusted +1, otherwise-2.
Before the event camera, in particular the light source is strobed, if no moving object passes, the first value representing the trust count is pulled up to a maximum value (when the maximum value is reached, no longer +1 to prevent overflow). If some false events occur within a certain period of time, the trust count value is repeatedly pulled down and pulled up, and if the second threshold is-3, 3 false events only result in 1 pulse event being passed, which effectively masks part of the false events.
On the other hand, if there is an object moving, a plurality of events similar to the false events in the graph are generated, and the scheme can quickly reduce the trust count value, enable the trust count value not to be higher than the second threshold value in a shorter time, and enable the pulse event to be released. And when no moving object exists in the field of view, the trust count value is quickly pulled up to the maximum value due to high-frequency flickering light, and the pulse event output by the pixel is restrained.
To eliminate spurious events, in another preferred embodiment, the second threshold (release threshold) is suitably lowered to prevent spurious events coming suddenly. In other alternative embodiments, the false event may also be eliminated by reducing the second non-zero constant, e.g., by reducing the second non-zero constant from 3 to 2 or 1, which would otherwise be directly released by reducing 3, and if the release condition is not met by reducing 1, then the event is not released, and the pixel is activated to release the subsequent event only if there is a subsequent event that further reduces the trust count value, which is equivalent to this embodiment suppressing the possibility of false event output by the back-end system.
In some class of embodiments, the second threshold is selected such that false events of two or three or four or more consecutive times, and beyond, are released.
FIG. 13 is a schematic diagram of a noise reduction device for distinguishing event polarity according to the present application, which includes a calculation module, a storage module, and a filtering module.
The storage module at least comprises a first storage space, a second storage space and a third storage space, wherein the first storage space, the second storage space and the third storage space comprise a plurality of storage units, and the storage units in each storage space are in one-to-one correspondence with pixels in a pixel array of the event imaging device.
The first memory space stores a first value, the second memory space stores a timestamp of a first polarity event, and the third memory space stores a timestamp of a second polarity event.
The computing module acquires a timestamp of a first event newly generated by a first pixel in the pixel array of the event imaging device and the polarity of the timestamp, acquires a second storage unit corresponding to the first pixel in a second storage space or acquires a timestamp stored in a third storage unit corresponding to the first pixel in a third storage space according to the acquired polarity, and acquires a time interval according to the difference between the timestamp of the first event and the stored timestamp.
The filtering module is used for judging whether the time interval is matched with the flicker frequency of the light source or not, and updating a first numerical value stored in a first storage unit corresponding to a first pixel in the first storage space according to a matching result; the filtering module filters or releases the first event according to the first value before or after updating.
If the time interval is matched with the flicker period, the first value stored in the first storage unit in the first storage space is increased by a first nonzero constant, and otherwise, the second nonzero constant is subtracted to update the first value.
In some embodiments, the first value is a trust count value or a number of matches. If the first numerical value is a trust count value, filtering the first event when the trust count value is greater than or equal to a first threshold value; if the first value is the continuous matching times, filtering the first event when the continuous matching times are greater than or equal to M times, wherein M is a positive integer.
In some embodiments, the first value is a trust count value or a continuous mismatch number; if the first value is a trust count value, releasing the first event when the trust count value is smaller than or equal to a second threshold value; and if the first numerical value is the continuous unmatched times, releasing the first event when the continuous unmatched times are greater than or equal to N times, wherein N is a positive integer. Wherein M and N may be the same.
Further, the second threshold may be lowered to avoid spurious events being released.
Optionally, the first non-zero constant is less than the second non-zero constant to avoid a false positive event.
Optionally, the first non-zero constant is equal to the second non-zero constant to avoid spurious events being released.
In certain embodiments, the noise reduction chip for distinguishing the event polarity is a vision sensor chip, or a neural network processor chip, or a sense and calculate integrated chip for integrating the vision sensor and the processor at the same time.
The present application relates to an electronic device provided with a noise reduction chip as described above that distinguishes between event polarities.
For the remainder of the noise reducer embodiments, the same methods as described above are incorporated herein by reference in their entirety for the avoidance of redundancy.
Although the present application has been described with reference to specific features and embodiments thereof, various modifications, combinations, substitutions can be made thereto without departing from the application. The scope of the present application is not intended to be limited to the particular embodiments of the process, machine, manufacture, composition of matter, means, methods and steps described in the specification, but rather, the methods and modules may be practiced in one or more products, methods, and systems of the associated, interdependent, inter-working, pre/post stages.
The specification and drawings are, accordingly, to be regarded in an abbreviated manner as an introduction to some embodiments of the technical solutions defined by the appended claims and are thus to be construed in accordance with the doctrine of greatest reasonable interpretation and are intended to cover as much as possible all modifications, changes, combinations or equivalents within the scope of the disclosure of the application while also avoiding unreasonable interpretation.
Further improvements in the technical solutions may be made by those skilled in the art on the basis of the present application in order to achieve better technical results or for the needs of certain applications. However, even if the partial improvement/design has creative or/and progressive characteristics, the technical idea of the present application is relied on to cover the technical features defined in the claims, and the technical scheme shall fall within the protection scope of the present application.
The features recited in the appended claims may be presented in the form of alternative features or in the order of some of the technical processes or the sequence of organization of materials may be combined. Those skilled in the art will readily recognize that such modifications, changes, and substitutions can be made herein after with the understanding of the present application, by changing the sequence of the process steps and the organization of the materials, and then by employing substantially the same means to solve substantially the same technical problem and achieve substantially the same technical result, and therefore such modifications, changes, and substitutions should be made herein by the equivalency of the claims even though they are specifically defined in the appended claims.
The steps and components of the embodiments have been described generally in terms of functions in the foregoing description to clearly illustrate this interchangeability of hardware and software, and in terms of various steps or modules described in connection with the embodiments disclosed herein, may be implemented in hardware, software, or a combination of both. Whether such functionality is implemented as hardware or software depends upon the particular application or design constraints imposed on the solution. Those of ordinary skill in the art may implement the described functionality using different approaches for each particular application, but such implementation is not intended to be beyond the scope of the claimed application.
Claims (21)
1. The utility model provides a distinguish noise reduction device of event polarity for filtering flicker noise, its characterized in that:
the system comprises a storage module, a calculation module and a filtering module;
the storage module at least comprises a first storage space, a second storage space and a third storage space, wherein the first storage space, the second storage space and the third storage space comprise a plurality of storage units, and the storage units in each storage space have a one-to-one correspondence with pixels in a pixel array of the event imaging device;
the first storage space stores a first value, the second storage space stores a time stamp of a first polarity event, and the third storage space stores a time stamp of a second polarity event;
the computing module acquires a timestamp of a first event newly generated by a first pixel in the pixel array of the event imaging device and the polarity of the timestamp, acquires a second storage unit corresponding to the first pixel in a second storage space or acquires a timestamp stored in a third storage unit corresponding to the first pixel in a third storage space according to the acquired polarity, and acquires a time interval according to the difference between the timestamp of the first event and the stored timestamp;
the filtering module is used for judging whether the time interval is matched with the flicker frequency of the light source or not, and updating a first numerical value stored in a first storage unit corresponding to the first pixel in the first storage space according to a matching result;
the filtering module filters or releases the first event according to the first numerical value before or after updating;
and updating the time stamp in the second storage unit or the third storage unit into the time stamp of the first event in an overlapping mode according to the polarity of the first event.
2. The noise reducer of claim 1, wherein the noise reducer is configured to distinguish event polarity from event polarity.
If the time interval is matched with the flicker frequency, the first value stored in the first storage unit in the first storage space is increased by a first nonzero constant, and otherwise, the second nonzero constant is subtracted to update the first value.
3. The noise reducer of claim 2, wherein the noise reducer is configured to distinguish between event polarities:
the first numerical value is a trust count value or continuous matching times;
if the first numerical value is a trust count value, filtering the first event when the trust count value is greater than or equal to a first threshold value;
if the first value is the continuous matching times, filtering the first event when the continuous matching times are greater than or equal to M times, wherein M is a positive integer.
4. The noise reducer of claim 2, wherein the noise reducer is configured to distinguish between event polarities:
the first numerical value is a trust count value or continuous unmatched times;
if the first value is a trust count value, releasing the first event when the trust count value is smaller than or equal to a second threshold value;
and if the first numerical value is the continuous unmatched times, releasing the first event when the continuous unmatched times are greater than or equal to N times, wherein N is a positive integer.
5. The noise reducer of any of claims 2-4, wherein the noise reducer is configured to distinguish between event polarities:
the first non-zero constant is less than the second non-zero constant.
6. The noise reducer of claim 4, wherein the noise reducer is configured to distinguish event polarity from event polarity.
The first non-zero constant is equal to the second non-zero constant; alternatively, N is greater than or equal to 2.
7. Noise reducer for distinguishing event polarity according to any one of claims 1-4 and 6, characterized in that:
whether the time interval matches the flicker frequency of the light source is determined based on whether the time interval is within an interval in which the flicker period of the light source is located.
8. The noise reducer of claim 7, wherein the noise reducer is configured to distinguish event polarity from event polarity.
The second storage unit or/and the third storage unit stores a lower accuracy of the time stamp than when the event imaging apparatus generates the event.
9. Noise reducer for distinguishing event polarity according to any one of claims 1-4 and 6, characterized in that:
the event imaging device is an event camera.
10. A noise reduction method for distinguishing event polarity is used for filtering flicker noise, and is characterized in that:
acquiring a timestamp and polarity of a first event newly generated by a first pixel in a pixel array of the event imaging device;
according to the polarity of the first event, acquiring a time stamp of a previous event which is generated by the first pixel and has the same polarity as the first event and is adjacent to the first event, and calculating the difference between the time stamp of the first event and the time stamp of the previous event to obtain a time interval;
comparing whether the time interval is matched with the flicker frequency of the light source, updating a first numerical value stored in a first storage unit corresponding to the first pixel in a first storage space according to a matching result, and filtering or releasing a first event according to the first numerical value before or after updating;
a second storage unit corresponding to the first pixel in the second storage space stores a time stamp of the first polarity event;
a third storage unit corresponding to the first pixel in the third storage space stores a time stamp of the second polarity event;
and updating the time stamp of the previous event stored in a second storage unit corresponding to the first pixel in the second storage space or a third storage unit corresponding to the first pixel in the third storage space into the time stamp of the first event in a coverage mode according to the polarity of the first event.
11. The noise reduction method according to claim 10, characterized in that:
whether the time interval matches the flicker frequency of the light source is determined based on whether the time interval is within an interval in which the flicker period of the light source is located.
12. The noise reduction method according to claim 11, wherein:
the second storage unit or/and the third storage unit stores a lower time stamp accuracy than when the event imaging device generates an event.
13. The noise reduction method according to any one of claims 10 to 12, characterized in that:
if the time interval is matched with the flicker frequency, the first value stored in the first storage unit in the first storage space is increased by a first nonzero constant, and otherwise, the second nonzero constant is subtracted to update the first value.
14. The method of noise reduction to distinguish event polarity according to claim 13, wherein:
the first numerical value is a trust count value or continuous matching times;
if the first numerical value is a trust count value, filtering the first event when the trust count value is greater than or equal to a first threshold value;
if the first value is the continuous matching times, filtering the first event when the continuous matching times are greater than or equal to M times, wherein M is a positive integer.
15. The method of noise reduction to distinguish event polarity according to claim 13, wherein:
the first numerical value is a trust count value or continuous unmatched times;
if the first value is a trust count value, releasing the first event when the trust count value is smaller than or equal to a second threshold value;
and if the first numerical value is the continuous unmatched times, releasing the first event when the continuous unmatched times are greater than or equal to N times, wherein N is a positive integer.
16. The noise reduction method for distinguishing event polarities according to claim 14 or 15, wherein:
the first non-zero constant is less than the second non-zero constant.
17. The method of noise reduction to distinguish event polarity according to claim 15, wherein:
the first non-zero constant is equal to the second non-zero constant; alternatively, N is greater than or equal to 2.
18. The noise reduction method for distinguishing event polarities according to any of claims 10-12, 14-15, wherein:
the event imaging device is an event camera.
19. A chip, characterized in that:
noise reduction device comprising a distinguishing event polarity according to any of claims 1 to 9.
20. The chip of claim 19, wherein:
the chip is a vision sensor chip, or a neural network processor chip, or a sensing and calculation integrated chip integrating the vision sensor and the processor.
21. An electronic device, characterized in that:
the electronic device is configured with a chip as claimed in claim 19 or 20.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310826935.4A CN116546339B (en) | 2023-07-07 | 2023-07-07 | Noise reduction device and method for distinguishing event polarity, chip and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310826935.4A CN116546339B (en) | 2023-07-07 | 2023-07-07 | Noise reduction device and method for distinguishing event polarity, chip and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116546339A CN116546339A (en) | 2023-08-04 |
| CN116546339B true CN116546339B (en) | 2023-09-08 |
Family
ID=87452869
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310826935.4A Active CN116546339B (en) | 2023-07-07 | 2023-07-07 | Noise reduction device and method for distinguishing event polarity, chip and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116546339B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110530513A (en) * | 2018-05-24 | 2019-12-03 | 三星电子株式会社 | The sensor based on event that flashing is filtered |
| CN115412688A (en) * | 2022-10-31 | 2022-11-29 | 深圳时识科技有限公司 | Flicker noise filtering method and device, sensor, chip and electronic equipment |
| CN116381334A (en) * | 2023-03-14 | 2023-07-04 | 武汉大学 | Event camera-based power grid frequency estimation method and system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP7352637B2 (en) * | 2019-08-28 | 2023-09-28 | 株式会社ソニー・インタラクティブエンタテインメント | Sensor module, sensor system, image processing device, image processing method and program |
-
2023
- 2023-07-07 CN CN202310826935.4A patent/CN116546339B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110530513A (en) * | 2018-05-24 | 2019-12-03 | 三星电子株式会社 | The sensor based on event that flashing is filtered |
| CN115412688A (en) * | 2022-10-31 | 2022-11-29 | 深圳时识科技有限公司 | Flicker noise filtering method and device, sensor, chip and electronic equipment |
| CN116233628A (en) * | 2022-10-31 | 2023-06-06 | 深圳时识科技有限公司 | Flicker noise filtering system |
| CN116381334A (en) * | 2023-03-14 | 2023-07-04 | 武汉大学 | Event camera-based power grid frequency estimation method and system |
Non-Patent Citations (1)
| Title |
|---|
| 一种基于地址-事件表达的实时视觉传感器实现方法;于璐 等;光学学报;第33卷(第01期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116546339A (en) | 2023-08-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111919434B (en) | Method and apparatus for processing signals from event-based sensors | |
| KR102423565B1 (en) | System and method for event camera data processing | |
| US20200372254A1 (en) | Method for outputting a signal from an event-based sensor, and event-based sensor using such method | |
| CN109803096B (en) | Display method and system based on pulse signals | |
| CN111709967B (en) | Target detection method, target tracking device and readable storage medium | |
| EP3913907A1 (en) | Pixel acquisition circuit and image sensor | |
| US20240064422A1 (en) | Noise filtering for dynamic vision sensor | |
| Stoffregen et al. | Event-based kilohertz eye tracking using coded differential lighting | |
| US20230125031A1 (en) | Event filtering method for operating an event-based image sensor | |
| CN116546339B (en) | Noise reduction device and method for distinguishing event polarity, chip and electronic equipment | |
| CN111131718B (en) | Multi-exposure image fusion method and system with LED flicker compensation function | |
| CN115412688B (en) | Flicker noise filtering method and device, sensor, chip and electronic equipment | |
| US10331244B2 (en) | Navagation device with fast frame rate upshift and operating method thereof | |
| CN115412686B (en) | Fusion noise reduction method and device, sensor, chip and electronic equipment | |
| CN107103602A (en) | A kind of intelligent material method of counting based on computer vision technique | |
| US11375109B2 (en) | Maintaining peak intensity frames as preview frames in slow motion mode of electronic device | |
| CN117223051A (en) | Ambient illuminance sensor system | |
| Bialik et al. | Fast-moving object counting with an event camera | |
| CN116757968B (en) | Noise reduction method and device, chip, event imaging device and electronic equipment | |
| CN115412687B (en) | Active and passive combined noise reduction device and method, visual sensor and chip | |
| WO2024026585A1 (en) | An event based vision sensor for flicker environment detection and dtecting method thereof | |
| WO2023100092A1 (en) | High accuracy representation of events generated by an event-based vision sensor | |
| CN118301490A (en) | Scintillation noise filtering method and device based on trust counting and dense detection | |
| CN116703971A (en) | Target tracking device, method, chip and electronic product | |
| CN116757256A (en) | Band-pass filtering method and device, chip and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |