CN116347257A - Fusion noise reduction system - Google Patents
Fusion noise reduction system Download PDFInfo
- Publication number
- CN116347257A CN116347257A CN202310003190.1A CN202310003190A CN116347257A CN 116347257 A CN116347257 A CN 116347257A CN 202310003190 A CN202310003190 A CN 202310003190A CN 116347257 A CN116347257 A CN 116347257A
- Authority
- CN
- China
- Prior art keywords
- event
- noise
- storage space
- noise reduction
- events
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
The invention discloses a fusion noise reduction system. In order to obtain the best noise reduction performance by using the least hardware resources, the fusion noise reduction method of the invention utilizes a credit count and a shift register to filter flicker noise and non-flicker noise respectively. In addition, the invention can also fuse the pixel hardware noise reduction scheme to eliminate black noise from the source. The invention solves the problem of low imaging quality of the event camera at the cost of extremely low hardware resources and power consumption, obtains the technical effect of high-speed real-time and accurate filtering aiming at flicker noise and various conventional noises, and improves the industrial application value of the event camera. The method is suitable for the fields of event cameras and brain-like calculation.
Description
The invention relates to a Chinese patent application (2022, 10, 31, a fusion noise reduction method and device, a sensor, a chip and an electronic device) with the application number of 202211345463.2. All technical solutions described in this application are incorporated herein by reference.
Technical Field
The invention relates to a fusion noise reduction system, in particular to a method and a device for fusion noise reduction of events by utilizing multiple noise reduction modes, a sensor, a chip and electronic equipment.
Background
The event imaging device can be used in the fields of tracking, VR (eye tracking), obstacle avoidance, optical flow estimation, driver state detection, etc., which is good at capturing moving objects in the field of view, while stationary objects are not imaged in the field of view, so that imaging is entirely event driven.
Since the event imaging device senses an event by setting a threshold value, each pixel (event imaging unit) senses a light intensity change, and generates an event or pulse when the light intensity change exceeds the threshold value. Ac lighting is commonly used in, for example, home environments, and this can result in ac-driven light sources constantly blinking, which changes in light intensity (both the light source itself and the object reflected light) cause event imaging devices to constantly generate events (known as flicker noise) that produce undesirable false "motions".
Meanwhile, because all events occur asynchronously, the time stamps of the events are different, the events generated by the movement of the target object have continuity in space and time, the reasons for generating the conventional noise have randomness, the number and the position of the conventional noise are random, and the frequency of the conventional noise is random. Taking Dynamic Vision Sensors (DVS) as an example, conventional noise sources are diverse, including background activity noise, thermal pixel (hot-pixel) noise, and the like.
Most of the existing noise reduction schemes are developed around the randomness and isolation of noise, and most of the schemes stay in the software or algorithm level, the schemes are easy to occupy hardware resources such as larger storage space, and although some schemes can occupy fewer hardware resources to realize noise reduction, the schemes are easy to accidentally injure effective events, and the fewer hardware resources are difficult to adopt to filter flicker noise and filter various conventional noise.
Based on this, there is a need in the art for a fusion noise reduction method: the method can accurately filter flicker noise and various conventional noises in real time at high speed, and meanwhile, the method has the advantages of less consumption of computing resources and storage resources, low power consumption and easiness in implementation in hardware.
Disclosure of Invention
In order to solve or alleviate some or all of the above technical problems, the present invention is implemented by the following technical solutions:
a fusion noise reduction method for noise reduction of an output event set output from an event imaging device, the output event set including events to be discriminated, each event in the output event set including at least coordinate information and timestamp information, the fusion noise reduction method comprising the steps of: obtaining the time difference between the event to be distinguished and the event before the event to be distinguished according to the time stamp information of the event to be distinguished and the time stamp information of the event before the event to be distinguished; if the time difference is smaller than a first threshold value, subtracting a non-zero constant from a first value in a second storage unit corresponding to an event to be distinguished in a second storage space, otherwise, adding a non-zero constant to the first value to obtain an updated first value; judging whether the event to be judged is a noise event or not according to the magnitude relation between the updated first numerical value and the second threshold value; if the event to be discriminated is not a noise event, then: according to the sequence of event generation or acquisition, storing the coordinate information and the time stamp information of the output event set into a first storage space in a first-in first-out mode; determining events with the difference between the time stamps of the events to be distinguished within a preset range according to the time stamp information of each event stored in the first storage space, and forming a first event set; determining whether the first event set has the events reaching the preset number within the preset distance range of the event coordinates to be distinguished according to the coordinate information of the events in the first event set and the coordinate information of the events to be distinguished; and judging whether the event to be judged is a noise event again according to whether the event reaching the preset number exists in the first event set.
In certain classes of embodiments, the event imaging apparatus includes a pixel circuit for generating a pulsed event based on a change in light, the pixel circuit including a light receiver and a switch module; the optical receiver is coupled with the switch module; and a dark current enhancement module for increasing a dark current through both ends of the light receiver or a total current through the switching module; alternatively, the dark current through both ends of the light receiver is increased by a process.
In certain embodiments, the fusion noise reduction method further comprises the steps of: and acquiring the time stamp information of the second event included in the first storage space and the time stamp information of the latest stored event in the first storage space or the time stamp information of the new event to be moved into the first storage space, calculating the time stamp difference between the two, and if the time stamp difference is smaller than a preset black noise threshold value, judging that the event generated or acquired later than the second event is at least a noise event.
In some embodiments, if the non-zero constant is positive, if the updated first value is smaller than a second threshold, determining that the event to be determined is a flicker noise event; or if the non-zero constant is a negative number, if the updated first value is greater than a second threshold, determining that the event to be determined is a flicker noise event; or if the first event set does not reach the preset number of events, judging that the event to be judged is a noise event.
A fusion noise reduction method for noise reduction of an output event set output from an event imaging device, the output event set including events to be discriminated, each event in the output event set including at least coordinate information and timestamp information, the fusion noise reduction method comprising the steps of: obtaining the time difference between the event to be distinguished and the event before the event to be distinguished according to the time stamp information of the event to be distinguished and the time stamp information of the event before the event to be distinguished; judging the size relation between the time difference and the first threshold value, and writing the result representing the size relation into a first position in a first list corresponding to the event to be judged in a second storage space; judging whether the event to be judged is a noise event or not according to a plurality of results which are stored in a plurality of positions corresponding to a first list of the event to be judged and are stored in a second storage space and represent the size relationship; if the event to be judged is not judged as the noise event, the event to be judged is: according to the sequence of event generation or acquisition, storing the coordinate information and the time stamp information of the output event set into a first storage space in a first-in first-out mode; determining events with the difference between the time stamps of the events to be distinguished within a preset range according to the time stamp information of each event stored in the first storage space, and forming a first event set; determining whether the first event set has the events reaching the preset number within the preset distance range of the event coordinates to be distinguished according to the coordinate information of the events in the first event set and the coordinate information of the events to be distinguished; and judging whether the event to be judged is a noise event again according to whether the event reaching the preset number exists in the first event set.
In some class of embodiments, if there are only the first locations in the first list: if the result of the representation of the magnitude relation is that the time difference is smaller than a first threshold value, judging that the event to be judged is a flicker noise event, otherwise, judging that the event to be judged is not the flicker noise event; alternatively, if there are at least two positions of the first list: judging whether the event to be judged is a flicker noise event according to the result of the size relation stored in the first position and the number of the results representing the same type of size relation, which are stored in the other positions of the first list and are stored recently; and if the first event set does not reach the preset number of events, judging that the event to be judged is a noise event, otherwise, judging that the event to be judged is not the noise event.
A fusion noise reduction device comprising at least a first storage space and a second storage space, and performing noise filtering on an event to be discriminated according to the first storage space, the second storage space and the fusion noise reduction method as described in any one of the preceding claims.
A sensor, which is an event imaging device, the event imaging device comprising a pixel array, a first storage space and a second storage space, wherein the pixel array comprises a first pixel for generating an event to be distinguished and generating a previous event of the event to be distinguished, and noise filtering is performed on the event to be distinguished according to the first storage space and the second storage space and the fusion noise reduction method according to any one of the previous claims.
A chip comprising an event imaging device and a processor, a first storage space and a second storage space, wherein the fusion noise reduction method according to the first storage space and the second storage space and any one of claims 1-6 at least carries out noise filtration on events to be distinguished generated by the event imaging device; and the processor processes the event generated by the event imaging device according to the event to be distinguished which is at least subjected to the noise filtering.
An electronic device having a chip as described above disposed thereon.
Some or all embodiments of the present invention have the following beneficial technical effects:
1) The hardware resource consumption is low and does not increase significantly with an increase in the resolution of the event imaging device. Further, this will reduce the silicon area/silicon cost of the chip, reducing the static power consumption of the chip.
2) The real-time performance is high. Because parallel computing is supported, the time complexity is extremely low, supporting real-time processing of high-speed event streams. This is important to determine whether the solution can be put into practical use.
3) The noise reduction effect is outstanding. The invention can accurately filter noise events, retain effective events and reduce accidental injury to the effective events.
4) Filtering of flicker noise and a variety of conventional noise is supported. In addition to conventional noise, the history shift register scheme can simply and conveniently filter black (block) noise.
5) The scheme is easy to realize in hardware. The invention does not involve iterative operation and complex operation consuming a large amount of computation resources, thus being easy for hardware realization.
Further advantageous effects will be further described in the preferred embodiments.
The above-described technical solutions/features are intended to summarize the technical solutions and technical features described in the detailed description section, and thus the ranges described may not be exactly the same. However, these new solutions disclosed in this section are also part of the numerous solutions disclosed in this document, and the technical features disclosed in this section and the technical features disclosed in the following detailed description section, and some contents in the drawings not explicitly described in the specification disclose more solutions in a reasonable combination with each other.
The technical scheme combined by all the technical features disclosed in any position of the invention is used for supporting the generalization of the technical scheme, the modification of the patent document and the disclosure of the technical scheme.
Drawings
FIG. 1 is a schematic overall view of an embodiment of the present invention
FIG. 2 is a flow chart of a flicker noise filtering scheme in the fusion noise reduction scheme of the present invention;
FIG. 3 is a schematic representation of temporal filtering in a fused noise reduction scheme of the present invention;
FIG. 4 is a schematic representation of spatial filtering for a first set of events;
FIG. 5 is a schematic diagram of determining a distance relation value according to an embodiment;
FIG. 6 is another type of spatial filtering schematic for a first set of events;
FIG. 7 is a schematic diagram of a pixel circuit based on a dark current enhancement module according to the present invention;
fig. 8 is a schematic diagram showing an effect of the pixel circuit for suppressing black noise or black block noise according to the present invention;
FIG. 9 is a schematic diagram of the present invention for a black noise or black block noise filtering scheme;
fig. 10 is a schematic structural view of an embodiment of the present invention.
Detailed Description
Since various alternatives are not exhaustive, the gist of the technical solution in the embodiment of the present invention will be clearly and completely described below with reference to the drawings in the embodiment of the present invention. Other technical solutions and details not disclosed in detail below, which generally belong to technical objects or technical features that can be achieved by conventional means in the art, are limited in space and the present invention is not described in detail.
Except where division is used, any position "/" in this disclosure means a logical "or". The ordinal numbers "first", "second", etc., in any position of the present invention are used merely for distinguishing between the labels in the description and do not imply an absolute order in time or space, nor do they imply that the terms preceded by such ordinal numbers are necessarily different from the same terms preceded by other ordinal terms.
The present invention will be described in terms of various elements for use in various combinations of embodiments, which elements are to be combined in various methods, products. In the present invention, even if only the gist described in introducing a method/product scheme means that the corresponding product/method scheme explicitly includes the technical feature.
The description of a step, module, or feature in any location in the disclosure does not imply that the step, module, or feature is the only step or feature present, but that other embodiments may be implemented by those skilled in the art with the aid of other technical means according to the disclosed technical solutions. The embodiments of the present invention are generally disclosed for the purpose of disclosing preferred embodiments, but it is not meant to imply that the contrary embodiments of the preferred embodiments are not intended to cover all embodiments of the invention as long as such contrary embodiments are at least one technical problem addressed by the present invention. Based on the gist of the specific embodiments of the present invention, a person skilled in the art can apply means of substitution, deletion, addition, combination, exchange of sequences, etc. to certain technical features, so as to obtain a technical solution still following the inventive concept. Such solutions without departing from the technical idea of the invention are also within the scope of protection of the invention.
Part of the important terminology and symbol interpretation:
e i : events labeled i, where i is a positive integer.
diff(e 0 ，e i ): the difference in horizontal and vertical coordinates between the new event numbered 0 and the event numbered i is a vector such as (1, 3).
d(e 0 ，e i ): abbreviated as d (0, i), new event e 0 Coordinate information (x (e) 0 )，y(e 0 ) And event e) i Coordinate information (x (e) i )，y(e i ) A distance measure, which may be defined as a scalar under certain embodiments).
R(e 0 ，e i ): abbreviated as R (0, i), representing a new event e 0 And event e i Distance relation value between the two. Rs is the sum of several distance relationship values, called the relationship value sum.
x(e i )，y(e i )，t(e i ): event e, respectively numbered i i Abscissa, ordinate, timestamp.
Event cameras, which are essentially event driven image sensors, are also known as Dynamic Vision Sensors (DVS). Based on this principle, there are some solutions to fuse it with the pixels of the conventional frame image, and the obtained sensor can output both the event and the pixel brightness, such as a DAVIS sensor and an ATIS sensor, and these event-based sensors (EBS) are collectively referred to as event imaging devices in the present invention, which belong to one of the sensors. The present invention discloses a scheme for filtering flicker noise and various conventional noises, taking an event camera (event camera) as an example.
Event cameras, such as DVS, are affected by noise interference and/or moving objects, generating sequences of pulsed events (events for short), these output event sets typically comprising coordinate information (or position information), such as abscissa, ordinate, of the imaging unit generating the event in the event camera; timestamp information at the time of event generation; as well as other information, such as polarity information.
The first memory space follows a first-in first-out memory principle. For example, e will be stored n Information in the storage unit of the coordinate information and the time stamp information is removed, and then the event e n-1 Storing e before writing coordinate information and time stamp information of (a) n Storage unit for coordinate information and time stamp information, and event e is pushed by the storage unit 1 Storing e before writing coordinate information and time stamp information of (a) 2 Storage unit for coordinate information and time stamp information, new event e 0 Storing e before writing coordinate information and time stamp information of (a) 1 And a storage unit for coordinate information and time stamp information. In other words, the coordinate information and the time stamp information of the event are sequentially stored in the respective storage units of the first storage space in the order of the generation of the event (time stamp) or the order of the acquisition of the first storage space, and the information stored in the respective storage units may be not limited to the coordinate information and the time stamp information.
New event e 0 (event to be discriminated) is an event which is currently newly generated by the event imaging apparatus, and its coordinates are (x (e) 0 )，y(e 0 ) With a timestamp t (e) 0 ). The first memory space comprises a number of memory cells and is used for storing at least coordinate information and time stamp information of the events. For example, the first memory space has n memory cells, which store event e 1 ～e n Each storage unit stores the coordinate information and the time stamp information of one event, wherein n is a positive integer. For example, event e n The information stored in the first storage space is (x (e) n )，y(e n )，t(e n ) And (c), wherein (x (e) n )，y(e n ) Is the coordinate information of the event, t (e) n ) The time stamp information specifically corresponds to the nth memory cell. Although the first storage space is only used to store at least the time stamp information and the coordinate information of the event, for simplicity of description, the first storage space can be stored as corresponding to the time stamp informationEvents of information and coordinate information. For example, the new event is an event to be moved into the first storage space.
Coupling: electrical connections may be established between electrical components, including direct connections and connections by means of other circuit modules.
Referring to fig. 1, for an event e to be discriminated generated by an event camera 0 Without loss of generality, it may be any event generated by the event camera, from the first pixel. At event e 0 The event previously generated and generated by the same pixel (first pixel) is denoted as "e 0 -1", which is event e 0 The previous event (abbreviated as previous event) of which the source pixels are the same and the coordinates are the same but the time stamps are different, and the previous event e 0 -1 with a time stamp t (e 0 -1). The first pixel generates the previous event e 0 -1, after which event e is successively generated 0 . The information of the event may further include the direction of the change in brightness of the pixel's light sensitivity, and is referred to as polarity.
In addition, there are two storage spaces, respectively called a third storage space and a second storage space, and data called an activity Map and a boolean Map are stored, respectively. There is a one-to-one correspondence between any memory cell corresponding to the activity map and the boolean map and any pixel of the event camera pixel array. In other words, each pixel has a corresponding memory cell in the third memory space and the second memory space, respectively. Without loss of generality, for any event e 0 The first memory cell and the second memory cell are referred to as a third memory cell and a second memory cell, respectively. To simplify the description, any event e 0 The coordinates of the generated pixels of (c) are (x (e 0 )，y(e 0 ) The coordinates of the third memory cell and the second memory cell corresponding to each other in the activity diagram and the boolean diagram are logically (x (e) 0 )，y(e 0 )). This logical mapping may be in any reasonable manner, and is not limited in this regard by the present invention.
The storage unit in the corresponding third storage space stores the timestamp of the latest event generated by the corresponding pixel in the activity diagram. In other words, the data in the activity map is the time stamp of the last issued event for each pixel of the entire pixel array. Preferably, the memory unit may be a 16-bit memory length. It is worth mentioning that the shorter the storage length, the smaller the storage space required. On the premise of meeting the precision requirement, the minimum storage length is selected to be beneficial to reducing the storage space/chip area. Therefore, the time stamp may be a time stamp obtained by decreasing the precision (original time stamp precision) of the pixel time stamp generated in the event camera, but it is needless to say that the time stamp may be stored with the original time stamp precision.
The corresponding storage unit in the second storage space stores a count, and the count reflects the credibility of the corresponding pixel. Event e 0 The greater the corresponding value of the count, event e 0 The less likely it is to be caused by a flickering light source. Preferably, the memory unit may be 3-bit memory length, and the 1-bit memory length is a very special case, and may also be 4-bit or 5-bit. If it is 3 bits, in one embodiment it stores a value between-4 and 3, with an initial value of 0.
The third memory space and the second memory space are part of the chip memory area. The pixel array of the event camera is the photosensitive portion of the chip and the pixel circuit may include a photodiode therein. The pixel array and the third memory space as well as the second memory space may constitute a chip which is merely an event imaging device, i.e. a sensor. The sensor can also be connected with the processor through the adapter plate to form a chip with integrated sensing and calculation, and the third storage space and the second storage space can be regarded as part of an interface circuit as a circuit/module for constructing noise reduction.
The fusion noise reduction scheme comprises filtering of flicker noise and conventional noise, and respectively judges the flicker noise and the conventional noise of the event to be judged, wherein the judgment can be carried out simultaneously or sequentially, the judgment sequence of the flicker noise and the conventional noise is not limited, and the event to be judged is filtered when the event to be judged is the flicker noise or the conventional noise.
Referring to FIG. 2, the means for generating an event herein may be anyAn event imaging apparatus is provided. Event e to be discriminated 0 Its time stamp t (e 0 ) And the previous event e 0 Timestamp t (e) of-1 0 -1) difference with a first threshold value θ t Comparing if the difference between the two time stamps is at the first threshold value theta t Within (i.e., less than the first threshold, the boolean outcome is true), the first value bootmap stored in the second memory location in the boolean graph (x (e) 0 )，y(e 0 ) (essentially the confidence count value) is subtracted by a non-zero constant, otherwise (false result), the first value bolmap (x (e) 0 )，y(e 0 ) Performing an increment of a non-zero constant. The non-zero constant here may be either positive or negative, and if negative, means that the first value is compared with a second threshold value θ c After the magnitude relation, judging the event e 0 The logic of (2) will be reversed. The aforementioned non-zero constant is preferably 1.
As a further preferred embodiment of the foregoing embodiment, the result of the foregoing difference is further compared with a third threshold value θ '' t (e.g., second threshold/2) comparison, if: if the difference between the two time stamps is at the second threshold value theta t Within and greater than a third threshold value θ' t (the boolean result is true at this time), the first value bootmap stored in the second memory cell in the boolean graph is divided into two pairs (x (e) 0 )，y(e 0 ) Performing a subtraction of a non-zero constant, otherwise increasing a non-zero constant.
And then based on the first numeric value bootmap (x (e) 0 )，y(e 0 ) And a second threshold value theta c Comparing if the first value is smaller than the second threshold value theta c Then consider event e 0 Is a flicker noise event caused by a flicker light source; otherwise consider event e 0 Not flicker noise.
Event e, whether or not it is determined as flicker noise 0 Time stamp t (e) 0 ) The coordinates stored in the third storage space are (x (e 0 )，y(e 0 ) In the third memory cell, its value is denoted as activityMap (x (e) 0 )，y(e 0 )). The storing operation may be performed after obtaining a time difference between the event to be discriminated and the event preceding the event, or may be performed after judging whether the event to be discriminated is flicker noise. As a subsequent step, when the coordinates in the pixel array are (x (e 0 )，y(e 0 ) A new event e) is issued again by the pixel 0 After +1, the activityMap (x (e) 0 )，y(e 0 ) A) the value is read and taken as the aforementioned new event e 0 +1 previous event e 0 Time stamp t (e) 0 ). In other words, in the course of updating the activity map, event e 0 Previous event e of (2) 0 Timestamp t (e) of-1 0 -1) event e 0 Time stamp t (e) 0 ) Flushing away, and therefore, is why the data in the activity map is the last time each pixel of the entire pixel array issued an event.
The method for filtering the conventional noise in the fusion noise reduction method comprises a time domain cluster filtering part and a space domain cluster filtering part. A time domain clustering filtering part: according to a first time threshold T θ A first set of events for the event to be discriminated is determined. Specifically, the time stamp information (t (e) 1 ),t(e 2 ),t(e 3 ),…,t(e n ) (ii) and timestamp information of the new event t (e) 0 ) The magnitude relation between the difference value and the first time threshold value, and the timestamp information t (e) 0 ) Events with a difference less than a first time threshold (e.g., 20 milliseconds). Referring to FIG. 3, event e in this example 6 Is an event meeting the condition, and the first set of events is then determined to be e 1 ～e 6 。
There may be a variety of ways to determine the first set of events. For example, according to event e 1 ～e n Sequentially calculating the time stamp of the event and t (e) 0 ) And comparing the difference value with the event threshold value to be judged, and finding the position where the size relation is turned over for the first time to determine a first event set. Of course, the determination may be performed in reverse order, such as in event e 1 Order of occurrence of en.
With reference to figure 4 of the drawings,the first event set (instead of all events in the first storage space) is then used as a time domain clustering result to participate in a subsequent spatial domain cluster filtering part. In some class of embodiments, based on new event e 0 Coordinate information of (a) and each event (e) 1 ～e m ) Completing airspace clustering filtering by the coordinate information of (2) and finally judging a new event e 0 Whether it is a regular noise event.
Referring to FIG. 5, new event e 0 Is set to (x (e) 0 )，y(e 0 ) From a spatial perspective, if the new event is within a certain range around, e.g. (x (e) 0 )±d,y(e 0 ) Events occurring within the range of d) are of the same cluster event as the new event, e.g. logically considered as belonging to a change of the event imaging unit within a certain area triggered by a certain target object, where d is a real number. Of course, the certain range defined herein may be set in various manners according to practical application requirements, such as square, rectangle with unfixed side length, and the invention is not limited thereto.
With continued reference to FIG. 3, in order to accomplish spatial cluster determination, in certain embodiments of the present invention, a distance relationship value calculation module may employ a method based on a distance relationship value R (e 0 ,e i ) To reduce the dimension of the information. Illustratively, in a preferred embodiment, the mathematical description is as follows:
Wherein R (e) 0 ,e i ) (abbreviated as R (0, i)) represents a new event e 0 And event e i Distance relation value between the new event e 0 Coordinate information (x (e) 0 )，y(e 0 ) And event e) i Coordinate information (x (e) i )，y(e i ) Distance d (e) 0 ,e i ) (abbreviated as d (0, i)) is within a certain set distance d, the distance relation value is 1, otherwise the distance relation value is 0, where i is a positive integer and d is a first distance threshold, such as d=8.
It is obvious that the definition of the distance relation value may be various, for example, the values may not be 1 and 0, but may be 2 and 0,1 and 0.01, -1 and 0,1 and-1, etc., and these definitions may result in logic adjustment or even logic inversion in determining whether it is a conventional noise event, but these logic adaptations are obvious to those skilled in the art, and the invention is not limited thereto.
Furthermore, it is apparent that the definition may also be varied, for example, in certain preferred embodiments it may be defined as: or is the same. In fact, in principle any new event e can be characterized 0 And event e i The quantification modes of the distance can be used as a new event e 0 The distance relationship to the event ei is not limited by these examples.
Referring to fig. 4, the relationship value summing module sums the distance relationship values between all events in the first event set and the new event to obtain a relationship value sum Rs. Based on the comparison result of the relation value sum Rs and the fourth threshold value, the noise judgment module judges whether the new event is a noise event or a valid event. For example, in the foregoing embodiments where the set distance relationship values are 1 and 0, if the relationship value sum Rs is less than (or less than) the fourth threshold value, then the new event is considered a noise event, otherwise the new event is considered a valid event. In other words, the event is considered to be a valid event if it is determined that the new event forms a valid cluster with an event in the first event set, and is otherwise a noisy event.
Referring to fig. 4, if the number of events in the first event set is an integer m, in some embodiments, the abscissa distance and the ordinate distance between the new event and each event in the first event set may be calculated, and if the abscissa distance and the ordinate distance are equal, the event is an event issued by the same event imaging unit. Of course, the coordinate values of the events may be directly compared and filtered, and a fifth threshold may be set, so that the events with the same number of events being greater than the fifth threshold are deduplicated. Let the number of filtered events be the integer m ', where m' is less than or equal to m.
Referring to fig. 6, another class of embodiments of spatial cluster filtering for a first set of events is presented. Unlike the embodiment represented in FIG. 3, this type of embodiment does not immediately determine a new event e 0 Whether it is a normal noise event or not,but rather, when an event, such as event e 4 After moving into the first storage space to a certain depth, the event e 4 The determination of whether the event to be determined is a noise event is performed in the same manner as in the above embodiments, and the determination logic is incorporated herein by reference to the above description, and will not be repeated here. When determining the first event set, determining that an event with a time difference (such as a difference value of time stamps) between the event and the event to be discriminated is an element in the first event set. The events in the first event set may occur before the event to be discriminated (e 5 ～e m ) Or may occur later than the event to be discriminated (e 1 ～e 3 )。
After moving in a certain depth, the method for judging whether the event is a noise event has the following advantages: the new event just generated by the edge of the moving object may be determined as a noise event in the foregoing embodiment because of insufficient event accumulated in the adjacent space region, which affects the imaging effect, and after moving into a certain depth, the subsequent event is allowed to have a chance to form an effective cluster with the new event, so that although a certain delay is introduced, misjudgment of the event can be effectively avoided, and the accuracy is improved.
In the present invention, at event e 0 If not any one of the flicker noise event and the regular noise event, then event e 0 Is a valid event, and performs post-processing.
Moreover, the expression greater than, less than, etc. in the present invention is essentially a logical comparison, and the boundary values may be modified slightly to obtain the same logical comparison result, but this is merely an equivalent conventional alternative means in the art, such as ". Gtoreq.2" is equivalent to "> 1", and the comparison logical result in some cases is equivalent. These basic logical transformations or boundary value modifications, etc., may generally be logically altered, replaced by those skilled in the art, without departing from the basic concepts of the present invention as such, and remain within the scope of the present invention.
Preferably, referring to fig. 7, the event imaging apparatus of the present invention includes a pixel circuit for generating a pulse event according to a light change. The pixel circuit includes a light receiver, a switching module, and a dark current enhancement module. The light receiver is coupled with the switch module, and dark current between two ends of the light receiver is increased by coupling the dark current enhancement module at two ends of the light receiver. The embodiment of the dark current enhancement module may be any manner that can increase the dark current between two nodes, and the invention is not limited in any particular way.
By way of example, the present invention also provides the following four types of dark current enhancement module implementations:
1) Switching tubes. For example, two ends of the light receiver are coupled by at least one MOS transistor to increase the dark current 5 flow across the light receiver.
2) And (3) resistors. For example, a resistor circuit module having an equivalent resistance, such as a resistor, is coupled across the light receiver, for example by low doped silicon, to increase the dark current across the light receiver.
3) Tunneling current (tunnel current) class. I.e. the dark current is increased by the electron tunneling current in the dielectric layer.
For example, a capacitor module with a certain equivalent capacitance value is coupled to two ends of the optical receiver. The dark current across the 0 optical receiver is increased by the tunneled current across the capacitor.
Preferably, the gate is coupled to one end of the optical receiver and the drain and source are coupled to a bias voltage or the other end of the optical receiver (e.g., commonly grounded) via at least one switching tube (NMOS or/and PMOS).
4) Junction current (junction current) class. For example, by connecting one reverse diode in parallel, the dark current across the light receiver is increased by the reverse bias current in the reverse diode.
5 in another class of embodiments, the dark current across the light receiver (not shown) is increased by a schottky diode. Diode, pin here
The diode belongs to the dark current generation based on the junction current device.
In addition, the invention also discloses a process scheme. The dark current of the light receiver can be improved through doping, surface treatment, heterogeneous materials and other processes. Generally, the more advanced the process, the lower the dark current of the light receiver, but through doping or other process means, the dark current of the light receiver is raised, and noise events under low illumination can be significantly suppressed in the present invention.
In this class 0 embodiment, rather than achieving dark current enhancement by a separate device, it is internalized inside the light receiver by light reception
The device is improved to obtain a larger dark current.
The above ways are examples of how to increase the dark current between two nodes, and the present invention is not repeated because there are more ways to increase the dark current. The invention is not limited to the foregoing examples.
For example, below different illumination conditions (e.g., less than 10 Lux), the sum of the photocurrent and the increased dark current is greater than the circuit current 5 noise, e.g., the former is 10 times and above the latter.
As shown in fig. 8, by the above-mentioned scheme, after increasing the dark current, the frequency of noise events is very effectively suppressed under low illumination, and the applicable scene of the visual sensor such as the event camera is effectively improved.
For example, as an illustration of the principles of the present invention, the dark current I of the light receiver dc Typically much less than 1fA, and circuit current noise i noise (containing
Optical noise, electrical noise, etc., typically characterized by a broad spectrum, a time average value of 0, etc.) is several fA or less. For photocurrent I ph :0 when I ph >At 10fA, i noise <<I ph +I dc Thus, there are few noise events;
when I ph <1fA, i noise >>I ph +I dc Thus, noise events are very numerous.
For noise reduction, according to an embodiment of the present invention, the dark current I is increased by fusing pixel circuits in the noise reduction method dc To 10 fA), then: i.e noise <<I ph +I dc Will always be true, thisThe reason why the present invention can effectively suppress noise event (black noise or black block noise) under dim light, and then obtain event e 0 If not any one of the flicker noise event and the regular noise event, then event e 0 Is a valid event and event e 0 And (5) delivering to a post-stage treatment. The embodiment can restrain black noise and filter flicker noise and various conventional noise under the condition of low hardware resource consumption, and has outstanding noise reduction effect.
Referring to fig. 9, a filtering scheme to increase black noise or black block noise is shown. The black noise or black block noise is a noise with a particularly high frequency, and is generated by event imaging devices for objects with all or part of black blocks or frequent false triggering under low light (or no light) conditions, and is characterized in that noise events are output continuously almost without intervals, the triggering frequency is far higher than that of normal events, the triggering period is usually less than 10 microseconds, and the frequency is above 10 ten thousand hertz.
In the foregoing embodiments of the present invention, it is very convenient to implement a black noise or black block noise discrimination scheme. Since a large number of events are stored in the first storage space in the order of time stamps, a certain integer p is set, and a second event e p And the latest stored event e in the first storage space 0 The time difference between them is t (e p )-t(e 0 ) If t (e) p )-t(e 0 )<T b Then the second event e can be directly determined p And event e 0 The events in between are either black noise or black block noise (high frequency noise), otherwise not black noise or black block noise, where t (e p ) And t (e) 0 ) Respectively the second event e p And event e 0 Time stamp, T of b Is a black noise threshold. For example, assuming that an event with a firing frequency of not less than 3 μs is black noise, p=100, then T b May be set to 300 microseconds.
Alternatively, the boolean graph stored in the second memory space may store the aforementioned boolean results of a plurality of events generated successively corresponding to the same pixel via a set of data. In other words, the foregoing embodiment is based only on the latest event e 0 Is the aforementioned Boolean of (1)Results for the first value boost map (x (e 0 )，y(e 0 ) While alternative embodiments keep a history of the boolean results before and after (in principle it is also possible to keep a time stamp of events before and after each pixel, but there is a greater need for memory space), it is evident that from a richer history the corresponding first value, and its correlation with the second threshold value theta, is easy to calculate c Is a size relationship of (a). This solution can solve the flicker noise problem, but has the disadvantage that a larger second storage space is required. Such specific or alternative embodiments are within the scope of the disclosed technology.
In other words, the fusion noise reduction method disclosed herein includes the steps of: the fusion noise reduction method comprises the following steps of: according to the sequence of event generation or acquisition, storing the coordinate information and the time stamp information of the output event set into a first storage space in a first-in first-out mode; obtaining the time difference between the event to be discriminated and the event before the event to be discriminated according to the time stamp information of the event to be discriminated and the time stamp information of the event before the event to be discriminated; judging the size relation between the time difference and the first threshold value, and writing the result representing the size relation into a first position in a first list corresponding to the event to be judged in a second storage space; or determining the event with the difference between the time stamp of the event to be distinguished and the time stamp of the event to be distinguished within a preset range according to the time stamp information of each event stored in the first storage space to form a first event set; determining whether the first event set has the events reaching the preset number within the preset distance range of the event coordinates to be distinguished according to the coordinate information of the events in the first event set and the coordinate information of the events to be distinguished; and judging whether the event to be judged is a noise event according to a plurality of results which are stored in a plurality of positions corresponding to the first list of the event to be judged and are stored in the second storage space and are used for indicating the magnitude relation, and whether the event reaching the preset number exists in the first event set.
Preferably, if the first list has only the first position (the first position may be a temporary position storing the result of the foregoing size relationship): and if the result of the representation of the magnitude relation is that the time difference is smaller than a first threshold value, judging that the event to be judged is flicker noise, otherwise, judging that the event to be judged is not flicker noise. This embodiment corresponds to the specific case described above.
Preferably, if there are at least two positions of the first list: and judging whether the event to be judged is flicker noise or not according to the result of the size relation stored in the first position and the number of the results representing the same type of size relation, which are stored in the other positions of the first list and are stored recently. Here, the most recently stored source of results representing size relationships for several other locations is selected
Because the same pixel is triggered the size relationship recently, the event e to be distinguished can be reflected 0 Whether it is flicker noise. For example, 5 values representing less than the relationship and 2 values representing greater than the relationship are stored in the first list 5, which is equivalent to the aforementioned case where the first value is equal to 3.
It should be noted that this alternative embodiment or specific example, other features, which are the same as those described hereinabove (without significant departure from logic), are hereby incorporated by reference.
Referring to fig. 10, a noise reduction process in a preferred embodiment of the present invention is shown. Firstly, eliminating black noise through a thousand-speed pixel circuit layer;
flicker noise is then removed by the trust count scheme represented in fig. 2, and finally conventional noise (including hot-pixel noise, background random noise, etc.) is removed by 0 again using the history shift register scheme represented in fig. 4 or 6.
The above process of fusion noise reduction method can be realized by designing a corresponding integrated circuit to realize a corresponding information processing process so as to form a final event imaging device or chip, and can also be realized by an FPGA or software method, and the physical carrier for realizing the above method is the fusion noise reduction device. It should be noted that, parameters such as various thresholds in the invention are configurable, so that the invention is convenient to adjust according to different scenes. In addition, the storage units in the first storage space and the second storage space can be centralized or distributed.
5 applying the fusion noise reduction method or the event imaging device comprising the fusion noise reduction device, the device can have better flicker noise resistance
Can be used. The electronic device using the flicker noise filtering method or the electronic device comprising the noise filtering device and the event imaging device can work under the light source driven by the alternating current power supply relatively easily.
In other words, disclosed herein are: a fusion noise reduction device at least comprises a first storage space and a second storage space,
and the fusion noise reduction device performs 0 noise filtration on the event to be judged according to the first storage space, the second storage space and the fusion noise reduction method according to any one of the previous items.
A sensor which is an event imaging device comprising a pixel array, a first storage space and a second storage space, wherein the pixel array comprises a first pixel for generating an event to be distinguished and generating a previous event of the event to be distinguished, and noise filtering is carried out on the event to be distinguished according to the first storage space and the second storage space and a fusion noise reduction method of any one of the previous claims.
A chip comprising an event imaging device and a processor, a first memory space and a second memory space, wherein the event imaging device generates at least noise for the event to be distinguished according to the first memory 5 space and the second memory space and the fusion noise reduction method according to any one of the previous claims
Acoustic filtering; and the processor processes the event generated by the event imaging device according to the event to be distinguished which is at least subjected to the noise filtering.
An electronic device is deployed with a chip as described above and is used to process environmental signals.
Although the present invention has been described with reference to specific features and embodiments thereof, it may be practiced without departing from the invention
Various modifications, combinations, substitutions are made. The scope of the present application is not intended to be limited to the particular embodiments of the process, machine, manufacture, composition of matter, means, methods and steps described in the specification, and the methods, modules may be practiced in the context of the associated, inter-dependent, inter-matched processes
Combining, front/back-end one or more products, methods.
The specification and drawings are, accordingly, to be regarded in an abbreviated manner as an introduction to some embodiments of the technical solutions defined by the appended claims and are thus to be construed in accordance with the doctrine of greatest reasonable interpretation and are intended to cover as much as possible all modifications, changes, combinations or equivalents within the scope of the disclosure of the invention while also avoiding unreasonable interpretation.
Further improvements in the technical solutions may be made by those skilled in the art on the basis of the present invention in order to achieve better technical results or for the needs of certain applications. However, even if the partial improvement/design has creative or/and progressive characteristics, the technical idea of the present invention is relied on to cover the technical features defined in the claims, and the technical scheme shall fall within the protection scope of the present invention.
The several features mentioned in the attached claims may be present as alternative features or the order of some technical flows, the order of 5 substance organization may be reorganized. Those skilled in the art will readily recognize such alternatives, or change the technical flow, after appreciating the present invention
The sequence of processes, the organization of materials, and then the use of substantially the same means solves substantially the same technical problem to achieve substantially the same technical effect, so that even if the means and/or sequence are defined in the claims, such modifications, changes, substitutions shall fall within the scope of protection of the claims according to the equivalent principles.
The various method steps or modules described in connection with the embodiments disclosed herein may be embodied in hardware, software, or a combination thereof, 0 in order to clearly illustrate the interchangeability of hardware and software, various steps and modules of the embodiments have been described above generally in terms of functionality
Composition is prepared. Whether such functionality is implemented as hardware or software depends upon the particular application or design constraints imposed on the solution. Those of ordinary skill in the art may implement the described functionality using different approaches for each particular application, but such implementation is not intended to be beyond the scope of the claimed invention.
Claims (10)
1. A fusion noise reduction method for reducing noise of an output event set output from an event imaging device, the output event set including events to be discriminated, each event in the output event set including at least coordinate information and timestamp information, the fusion noise reduction method comprising the steps of:
obtaining the time difference between the event to be distinguished and the event before the event to be distinguished according to the time stamp information of the event to be distinguished and the time stamp information of the event before the event to be distinguished;
judging the size relation between the time difference and the first threshold value, and writing the result representing the size relation into a first position in a first list corresponding to the event to be judged in a second storage space;
judging whether the event to be judged is a noise event or not according to a plurality of results which are stored in a plurality of positions corresponding to a first list of the event to be judged and are stored in a second storage space and represent the size relationship;
if the event to be judged is not judged as the noise event, the event to be judged is: according to the sequence of event generation or acquisition, storing the coordinate information and the time stamp information of the output event set into a first storage space in a first-in first-out mode; determining events with the difference between the time stamps of the events to be distinguished within a preset range according to the time stamp information of each event stored in the first storage space, and forming a first event set; determining whether the first event set has the events reaching the preset number within the preset distance range of the event coordinates to be distinguished according to the coordinate information of the events in the first event set and the coordinate information of the events to be distinguished; and judging whether the event to be judged is a noise event again according to whether the event reaching the preset number exists in the first event set.
2. The fusion noise reduction method according to claim 1, characterized in that:
if there are only the first locations in the first list: if the result of the representation of the magnitude relation is that the time difference is smaller than a first threshold value, judging that the event to be judged is a flicker noise event, otherwise, judging that the event to be judged is not the flicker noise event; or alternatively, the process may be performed,
if there are at least two positions of the first list: judging whether the event to be judged is a flicker noise event according to the result of the size relation stored in the first position and the number of the results representing the same type of size relation, which are stored in the other positions of the first list and are stored recently; the method comprises the steps of,
and if the first event set does not reach the preset number of events, judging that the event to be judged is a noise event, otherwise, judging that the event to be judged is not the noise event.
3. The fusion noise reduction method according to claim 1, characterized in that:
the event to be distinguished and the event before the event to be distinguished are both from the first pixel of the event imaging device.
4. A fusion noise reduction method according to claim 3, characterized in that:
The event imaging device comprises a pixel circuit, a pulse generation circuit and a pulse generation circuit, wherein the pixel circuit is used for generating a pulse event according to light ray change;
each pixel of the event imaging device has a corresponding storage unit in a third storage space;
each storage unit in the third storage space stores a time stamp of the last issue event of the corresponding pixel.
5. The fusion noise reduction method according to claim 4, wherein:
the time stamp accuracy of the event stored in the third storage space is lower than the time stamp accuracy when the event is generated in the event imaging apparatus.
6. The fusion noise reduction method according to any one of claims 1-5, wherein the event imaging device comprises a pixel circuit for generating pulse events from light variations, characterized in that:
the pixel circuit comprises a light receiver and a switch module;
the optical receiver is coupled with the switch module; the method comprises the steps of,
a dark current enhancement module used to increase the dark current through both sides of the light receiver or the total current through the switching module; or alternatively, the process may be performed,
the dark current through the light receiver is increased by a process.
7. The utility model provides a fuse device of making an uproar falls which characterized in that:
The fusion noise reduction device at least comprises a first storage space and a second storage space, and performs noise filtration on an event to be discriminated according to the first storage space, the second storage space and the fusion noise reduction method as claimed in any one of claims 1 to 6.
8. A sensor, the sensor being an event imaging device, the event imaging device comprising a pixel array, a first storage space and a second storage space, the pixel array comprising a first pixel for generating an event to be distinguished and generating a previous event to the event to be distinguished, the sensor being characterized in that:
noise filtering is performed on the event to be discriminated according to the first storage space and the second storage space and the fusion noise reduction method according to any one of claims 1 to 6.
9. A chip comprising an event imaging device and a processor, a first memory space and a second memory space thereon, characterized in that:
noise filtering at least events to be discriminated generated by the event imaging device according to the first storage space and the second storage space and the fusion noise reduction method of any one of claims 1 to 6;
and the processor processes the event generated by the event imaging device according to the event to be distinguished which is at least subjected to the noise filtering.
10. An electronic device, characterized in that: the electronic device being deployed with a chip as claimed in claim 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310003190.1A CN116347257A (en) | 2022-10-31 | 2022-10-31 | Fusion noise reduction system |
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211345463.2A CN115412686B (en) | 2022-10-31 | 2022-10-31 | Fusion noise reduction method and device, sensor, chip and electronic equipment |
| CN202310003190.1A CN116347257A (en) | 2022-10-31 | 2022-10-31 | Fusion noise reduction system |
Related Parent Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211345463.2A Division CN115412686B (en) | 2022-10-31 | 2022-10-31 | Fusion noise reduction method and device, sensor, chip and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116347257A true CN116347257A (en) | 2023-06-27 |
Family
ID=84168293
Family Applications (2)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310003190.1A Pending CN116347257A (en) | 2022-10-31 | 2022-10-31 | Fusion noise reduction system |
| CN202211345463.2A Active CN115412686B (en) | 2022-10-31 | 2022-10-31 | Fusion noise reduction method and device, sensor, chip and electronic equipment |
Family Applications After (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211345463.2A Active CN115412686B (en) | 2022-10-31 | 2022-10-31 | Fusion noise reduction method and device, sensor, chip and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (2) | CN116347257A (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116757968B (en) * | 2023-08-18 | 2023-11-24 | 深圳时识科技有限公司 | Noise reduction method and device, chip, event imaging device and electronic equipment |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20180014992A (en) * | 2016-08-02 | 2018-02-12 | 삼성전자주식회사 | Event signal processing method and apparatus |
| CN111770290A (en) * | 2020-07-29 | 2020-10-13 | 中国科学院长春光学精密机械与物理研究所 | Noise reduction method for dynamic vision sensor output event stream |
| WO2022188120A1 (en) * | 2021-03-12 | 2022-09-15 | Huawei Technologies Co., Ltd. | Event-based vision sensor and method of event filtering |
| CN113923319B (en) * | 2021-12-14 | 2022-03-08 | 成都时识科技有限公司 | Noise reduction device, noise reduction method, chip, event imaging device and electronic equipment |
| CN114285962B (en) * | 2021-12-14 | 2023-04-07 | 成都时识科技有限公司 | Noise processing device, method, chip, event imaging device and electronic equipment |
| CN114885074B (en) * | 2022-05-06 | 2023-07-04 | 中国科学院光电技术研究所 | Event camera denoising method based on space-time density |
-
2022
- 2022-10-31 CN CN202310003190.1A patent/CN116347257A/en active Pending
- 2022-10-31 CN CN202211345463.2A patent/CN115412686B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN115412686B (en) | 2023-03-24 |
| CN115412686A (en) | 2022-11-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103348475B (en) | Sensor or the improvement that relates to sensor | |
| CN109100737B (en) | Photoelectric sensor and method for measuring distance to object | |
| CN106597463B (en) | Photo-electric proximity sensor and detection method based on dynamic visual sensor chip | |
| CN113923319B (en) | Noise reduction device, noise reduction method, chip, event imaging device and electronic equipment | |
| CN109842768B (en) | Pixel acquisition circuit and image sensor | |
| CN111770290A (en) | Noise reduction method for dynamic vision sensor output event stream | |
| CN116347257A (en) | Fusion noise reduction system | |
| JP2010238032A (en) | Smoke detection device | |
| CN110619651B (en) | Driving road segmentation method based on monitoring video | |
| KR20210129043A (en) | How to process information from event-based sensors | |
| CN110674672B (en) | Multi-scene people counting method based on tof camera | |
| CN114885074B (en) | Event camera denoising method based on space-time density | |
| US20220205835A1 (en) | Systems and methods for imaging and sensing vibrations | |
| CN115412688B (en) | Flicker noise filtering method and device, sensor, chip and electronic equipment | |
| KR20230078991A (en) | Noise reduction device, cluster noise reduction device and chip | |
| CN114078100A (en) | Clustering noise reduction device, method, chip, event imaging device and electronic equipment | |
| CN115442544A (en) | Dynamic visual event stream noise reduction method based on hot pixels and enhanced space-time correlation | |
| CN117115451B (en) | Adaptive threshold event camera denoising method based on space-time content correlation | |
| CN110895336A (en) | Object detection device based on avalanche diode | |
| CN115412687B (en) | Active and passive combined noise reduction device and method, visual sensor and chip | |
| Brajovic et al. | Temporal photoreception for adaptive dynamic range image sensing and encoding | |
| Habibi et al. | Asynchronous demodulation technique for use in vision sensor image classification and segmentation | |
| CN219555081U (en) | Acquisition circuit of optical signal in image sensor and electronic equipment | |
| JP2022121991A (en) | Sensing system, signal processor | |
| JP3546296B2 (en) | Motion detection device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |