CN116757968A - Noise reduction method and device, chip, event imaging device and electronic equipment - Google Patents

Abstract

The invention discloses a noise reduction method and device, a chip, an event imaging device and electronic equipment. In order to solve the defect that the prior scheme for filtering background noise in the event imaging process is easy to filter the effective events together by mistake, the invention preliminarily screens the effective events based on a time domain processing method for dividing time intervals. The invention further judges whether the event to be judged is likely to belong to noise or not according to the fluctuation of the event quantity generated by the pixel generating the event to be judged and the neighborhood pixel thereof between the first time interval and the time interval before the first time interval, thereby realizing the technical effect of noise event preliminary screening. The invention combines time domain and space domain to process noise, and obtains the technical effects of accurately, effectively and rapidly filtering background noise. The invention is suitable for the fields of event cameras and neuromorphic chips.

Description

Noise reduction method and device, chip, event imaging device and electronic equipment

Technical field

The present invention relates to a noise reduction method and device, a chip, an event imaging device and an electronic device, and in particular to a noise reduction method and device for a background noise event, a chip, an event imaging device and an electronic device.

Background technique

The background noise events of neuromorphic sensors are randomly generated. Some existing noise reduction technologies consume a lot of computing resources to filter out background noise. Some existing technologies do not consider the position information of pixels and treat some valid events as background noise. Filtering, noise reduction accuracy is low.

A common type of noise reduction scheme is correlation filtering, which determines whether other events occur within a certain period of time and within a range when the current event occurs. If so, it is considered a valid event. In some early noise reduction schemes, the timestamp of a pixel is stored in the timestamp storage units corresponding to the surrounding eight neighboring pixels, and then it is determined whether the difference between the current timestamp and the previous timestamp corresponding to the pixel is greater than the threshold. , for time domain support. However, although this type of method has a low computational cost, the filtering effect is not ideal and can only filter out low-frequency background noise. In addition, this type of method makes incomplete use of spatial information and will lose spatial information.

Prior art 1: CN116170702A. The event imaging device in the prior art 1 senses the event by setting a threshold, and determines whether the change in light intensity of the pixel unit reaches the set threshold. An event is generated only when it reaches the set threshold, otherwise it is filtered out as background noise. The threshold is usually set very small to ensure high sensitivity, but at the same time, it is easy to capture small external interference as noise events, so it is extremely sensitive to noise. Increasing the threshold will reduce the number of effective events captured when the object is moving. affect the imaging effect. The noise reduction accuracy of this solution is low, and it is easy to misjudge valid events as background noise.

Based on this, this field is in urgent need of a noise reduction method that can accurately and efficiently filter out background noise and reduce the probability of accidental damage to effective events.

Contents of the invention

In order to solve or alleviate some or all of the above technical problems, the present invention is implemented through the following technical solutions:

A denoising method for denoising an event stream output by an event imaging device. The event stream includes events to be distinguished generated by a first pixel. Each event in the event stream at least includes coordinate information and time. Stamp information, the noise reduction method includes the following steps: determine whether the event to be determined belongs to the first time interval according to the timestamp information of the event to be determined; if the event to be determined does not belong to the first time interval, determine whether the event to be determined belongs to the first time interval according to the time stamp of the first pixel. The fluctuation between the statistical number of events generated in a time interval and the statistical number of events generated by the first pixel in the previous time interval of the first time interval is used to determine whether the event to be determined is blocked.

In certain embodiments, for unblocked events to be determined, whether the unblocked events to be determined should be released is determined based on whether the timestamps of spatial neighbor pixels around the first pixel can provide sufficient support.

In certain embodiments, if the event to be determined belongs to the first time interval, then add one to the statistics of events generated by the first pixel in the first time interval.

In certain embodiments, if the fluctuation between the statistical number of events generated by the first pixel in the first time interval and the statistical number of events generated by the first pixel in the previous time interval of the first time interval is greater than or equal to the first threshold, then the event to be determined is allowed to enter the next-level system.

In certain embodiments, if the fluctuation between the statistical number of events generated by the first pixel in the first time interval and the statistical number of events generated by the first pixel in the previous time interval of the first time interval is less than The first threshold is: determine whether there are more than a preset number of pixels among the spatial neighbor pixels of the first pixel. The fluctuation between the statistics of events generated in the first time interval and the previous time interval of the first time interval. is greater than or equal to the first threshold, if so, the event to be determined is allowed to enter the next level system, otherwise the event to be determined is blocked.

In certain embodiments, if the statistics of events generated by the first pixel in the first time interval are zero, then the state value corresponding to the first pixel in the state matrix is updated to the first state value; if the first pixel is in The fluctuation between the statistical number of events generated in the first time interval and the statistical number of events generated by the first pixel in the previous time interval of the first time interval is less than the first threshold, then the corresponding number in the state matrix will be The status value of a pixel is updated to the second status value; if the statistical number of events generated by the first pixel in the first time interval is different from the statistical number of events generated by the first pixel in the previous time interval of the first time interval If the fluctuation between them is greater than or equal to the first threshold, then the state value corresponding to the first pixel in the state matrix is updated to the third state value.

In certain embodiments, if there are more than a preset number of pixels among the spatial neighbor pixels of the first pixel, the fluctuation between the statistics of events generated in the first time interval and the previous time interval of the first time interval is greater than or equal to the first threshold, then the state value corresponding to the first pixel in the state matrix is updated to the third state value.

In certain embodiments, if the state value corresponding to the first pixel in the state matrix is the third state value, then the mark value corresponding to the first pixel in the mark matrix is updated to the first mark value;

If the state value corresponding to the first pixel in the state matrix is the second state value, then the mark value corresponding to the first pixel in the mark matrix is updated to the second mark value.

In certain embodiments, if the mark value of the first pixel corresponding to the first pixel in the mark matrix is the first mark value, the event to be determined is allowed to enter the next level system. If the first pixel corresponds to If the mark value of the first pixel is the second mark value, the event to be determined is blocked.

In certain embodiments, the next-level system is a noise reduction system, which is based on the timestamp of the spatial neighbor pixels corresponding to the first pixel in the timestamp matrix and the time of the corresponding first pixel in the timestamp matrix. The time difference between stamps is counted, and the number of spatial neighbor pixels whose time difference is less than or equal to the predetermined threshold is counted. If it exceeds the second threshold, the event to be determined is released, otherwise the event to be determined is blocked.

In certain embodiments, if the corresponding state value of the first pixel in the state matrix is the third state value, then the event to be determined is released to enter the next level system. If the corresponding state value of the first pixel in the state matrix is is the second status value, then the event to be determined is blocked.

In certain embodiments, after adding one to the statistics of events generated by the first pixel in the first time interval, the corresponding tag value of the first pixel in the tag matrix or the number of the first pixel in the state matrix is The corresponding status value is used to determine whether to release or block the pending event.

In certain embodiments, the length of the first time interval is the same as the length of the previous time interval of the first time interval, and the length of the first time interval is the same as the length of the subsequent time interval of the first time interval.

A noise reduction device, which uses the previous noise reduction method.

An event imaging device includes the noise reduction device as described above.

A chip is provided with an event imaging device as described above.

An electronic device is provided with a chip as described above, an event imaging device as described above, or a noise reduction device as described above.

In certain embodiments, the electronic device further includes a computing module, and one or more of the following modules: an energy collection module, a control module, and a communication module.

Some or all embodiments of the present invention have the following beneficial technical effects:

1) The present invention discovers the distribution characteristics of random noise through testing and proposes a new noise reduction method that can accurately filter out background noise and greatly reduce the probability of accidental damage to effective events, thus improving the accuracy of event imaging devices such as DVS.

2) The noise reduction method of the present invention can efficiently filter out background noise. The effect of ordinary spatial filtering on filtering out background noise is only about 10 times. After testing, the time domain and spatial domain filtering effect of the present invention can reach 80 times or more, and the noise reduction effect is only about 10 times. The effect is outstanding.

More beneficial effects will be further introduced in the preferred embodiments.

The technical solutions/features disclosed above are intended to summarize the technical solutions and technical features described in the specific embodiments section, so the scope of the description may not be exactly the same. However, these new technical solutions disclosed in this section are also part of the many technical solutions disclosed in the present invention document. The technical features disclosed in this section are the same as the technical features disclosed in the subsequent specific embodiments section and the drawings that are not explicitly described in the specification. Part of the content in the document discloses more technical solutions in a reasonable combination with each other.

The technical solution composed of all the technical features disclosed anywhere in the present invention is used to support the summary of the technical solution, the modification of the patent document, and the disclosure of the technical solution.

Description of the drawings

Figure 1 is a schematic diagram of the time domain processing part of the noise reduction method of the present invention.

Figure 2 is a schematic diagram of some specific embodiments of the time domain processing part of the noise reduction method of the present invention.

Figure 3 is a schematic diagram of a specific embodiment of a next-level system.

Figure 4 is a schematic diagram of a specific embodiment of the noise reduction method of the present invention applied to a certain device or chip.

Detailed ways

Since various alternatives cannot be described exhaustively, the key points of the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Other technical solutions and details not disclosed in detail below generally belong to technical goals or technical features that can be achieved by conventional means in this field. Due to space limitations, the present invention will not introduce them in detail.

Unless it means division, "/" at any position in the present invention means logical "or". Serial numbers such as "first" and "second" in any position in the present invention are only used as descriptive distinction marks and do not imply an absolute sequence in time or space, nor do they imply that terms with such serial numbers are different from those with The same terms in other attributive terms must have different referents.

The present invention will describe various elements for combination into various specific embodiments, and these elements will be combined into various methods and products. In the present invention, even if only the main points are described when introducing the method/product solution, it means that the corresponding product/method solution also clearly includes the technical feature.

When describing the existence or inclusion of a certain step, module, or feature anywhere in the present invention, it does not imply that such existence is exclusive and unique. Those skilled in the art can completely supplement it with other methods based on the technical solutions disclosed in the present invention. Other embodiments can be obtained using technical means. The disclosed embodiments of the present invention are generally for the purpose of disclosing preferred embodiments, but this does not imply that the opposite embodiments of the preferred embodiments are excluded/excluded by the present invention, as long as such opposite embodiments at least solve the problem of Certain technical problems of the present invention are all expected to be covered by the present invention. Based on the key points described in the specific embodiments of the present invention, those skilled in the art can completely replace, delete, add, combine, change the order and other means to certain technical features to obtain a technical solution that still adheres to the concept of the present invention. These solutions that do not depart from the technical concept of the present invention are also within the protection scope of the present invention.

Explanation of some important terms:

An event imaging device is a sensor that senses changes in light and responds to these changes in the form of events. Any form of sensor including pixels that can generate spiking events can be regarded as a visual sensor as defined in the present invention, including but not limited to event cameras/DVS, DAVIS sensors, and fusion of event imaging and RGB imaging, Fusion sensors that combine event imaging and infrared fusion.

The inventor found through testing that the time interval of random noise approximates a Poisson distribution, that is, within a time interval of the same length, the same pixel points generally have a similar number of events. And if a pixel has more events within a certain time interval, the events are likely to be generated by motion rather than background noise. Based on the inventor's test results, analysis, and experiments, the present invention proposes an efficient and accurate method for filtering out background noise. It determines whether an event is noise based on the timestamp information of the event to be identified and the position information of the pixels that generated the event to be identified.

In the present invention, without loss of generality, a certain event to be determined contained in the event stream generated by the event imaging device is recorded as generated by the first pixel (also called A pixel), and its position coordinates (x, y ) and timestamp t information are recorded as A (x, y, t), the time interval [t ₀ , t ₁ ] is named the first time interval/current time interval, and the previous time interval is the previous time interval of the first time interval. A time interval (also called a second time interval).

The core of the present invention lies in the time domain processing part. Preferably, in addition to the time domain processing part, the present invention also includes a spatial domain processing part (the essence is still a solution that relies on space-time domain support).

A storage unit in the storage space stores the statistics of events generated by a corresponding pixel in the pixel array within the current time interval, and the set of statistics corresponding to the entire pixel array is called an event quantity matrix. Similarly, the matrix of the aforementioned type corresponding to the second time interval is called the second event quantity matrix.

A storage unit in another storage space stores the state value of a corresponding pixel in the pixel array in the current time interval, and the set of state values corresponding to the entire pixel array is called an event state matrix.

After the time domain processing of the present invention, a storage unit in another storage space stores the mark of the pulse event generated by a corresponding pixel in the pixel array, and the set of marks corresponding to the entire pixel array is called a mark matrix.

Preferably, each time interval has the same length, for example, the first time interval and the second time interval have the same length. For example, the first time interval t ₀ to t ₁ in Figure 2 has a total length of 10 milliseconds.

Referring to Figure 1, a noise reduction method for the time domain processing part is used to reduce noise on an event stream output by an event imaging device. The event stream includes an event to be discriminated generated by a first pixel. Each event stream in the event stream Each event at least includes coordinate information and timestamp information. The noise reduction method includes the following steps: judging whether the event to be identified belongs to the first time interval according to the timestamp information of the event to be identified; if the event to be identified does not belong to the first time interval interval, based on the fluctuation between the statistical number of events generated by the first pixel in the first time interval and the statistical number of events generated by the first pixel in the previous time interval of the first time interval, determine whether to block the Events to be determined.

More specifically, corresponding to a specific embodiment of the above technical solution, the following steps are included (as shown in Figure 2):

Step S1: Determine whether the timestamp (information) t of the event to be determined generated by pixel A belongs to the first time interval [t ₀ , t ₁ ]. If so, add 1 to the corresponding statistical number of pixel A in the event quantity matrix. Go to step S6; otherwise, go to step S2.

Step S2: Query the statistics corresponding to the current pixel A in the event quantity matrix. If the statistics are 0, update the state value of the corresponding pixel A in the state matrix to the first state value (0 in the example), and execute the steps S5; otherwise, execute step S3.

Step S3: Calculate the difference between the statistics of the first time interval corresponding to pixel A in the event quantity matrix and the statistics of the second time interval corresponding to pixel A in the second event quantity matrix (i.e., the first pixel in the first time period interval and the statistics of events generated in the previous time interval of the first time interval), if the absolute value of the difference between the statistics is greater than or equal to 3 (example), pixel A is placed in the corresponding state matrix The status value is updated to the third status value (example is 2), otherwise it is updated to the second status value (example is 1). Then, proceed to step S4.

Step S4: When the corresponding state value of pixel A in the state matrix is 1, check the corresponding state values of the spatial neighbor pixels (referred to as neighbor pixels) in the neighborhood of pixel A in the state matrix, and based on these state values, decide whether to The state value corresponding to pixel A is updated to the third state value. For example, if the sum of these status values is greater than the preset value, the aforementioned update will be performed.

For example, in the 3*3 neighborhood range, if the sum of the corresponding state values of pixel A's 8 neighbor pixels in the state matrix is greater than 3 (the example in which the second state value is 1 is followed), then pixel A will be The corresponding state value in the state matrix is updated to the third state value (2 in the example), otherwise it is not updated. For another example, by counting the number of pixels whose state value is the third state value among the neighboring pixels around pixel A, if the number of pixels in this state exceeds the preset value (3 in the example), pixel A will also be in the state. The corresponding state value in the matrix is updated to the third state value (2 in the example), otherwise it is not updated. Then execute step S5.

The purpose of step S4 is that when pixel A generates an event to be determined, if its neighbor pixels also generate a certain number (for example: 1, 2, 3) of pulse events, then the corresponding state of pixel A in the state matrix is The value is corrected (modified from the second state value to the third state value), giving it the opportunity to make further judgments in the next-level system (ie, the airspace processing part). This step advantageously avoids the accidental killing of some valid events.

Step S5: Query the corresponding state value of pixel A in the state matrix. If the value is the third state value, update the corresponding label value of pixel A in the label matrix to the first label value (0 in the example); If the state value is the second state value, then the corresponding mark value of pixel A in the mark matrix is updated to the second mark value (1 in the example).

Step S6: Query the corresponding tag value of pixel A in the tag matrix. If the value is the first tag value, the event to be determined is allowed to enter the next-level system. If the value is the second tag value, the event to be determined is blocked.

For steps S5 and S6, alternatively, the step of converting the state matrix into a mark matrix can be omitted, and the release/blocking judgment can be made directly based on the state value.

In certain embodiments, the next-level system that the event to be determined enters is preferably the aforementioned airspace processing part, please refer to Figure 3 .

A storage unit in a certain storage space stores the timestamp of the last generated event of a corresponding pixel in the pixel array, and the set of timestamps corresponding to the entire pixel array is called a timestamp matrix.

Events to be determined that are released (not blocked) by the time domain processing part are sent to the air domain processing part. Preferably, the spatial processing part of the present invention is to use the time stamps of spatial neighbor pixels around the first pixel to determine whether they can provide sufficient support to allow the event to be determined to be released. For example, the specific process of airspace processing is as follows:

The event stream to be determined that can enter the next-level system is used as a preliminary filtered event stream. The initially filtered event stream enters the next-level system. Still taking pixel A as an example, the timestamp corresponding to pixel A in the timestamp matrix (before the update, the storage unit corresponding to pixel A stores the event before the event to be determined) timestamp) is updated to the timestamp t of the event to be identified, and the timestamps corresponding to several neighbor pixels of pixel A stored in the timestamp matrix are recorded as t _n (n=1, 2, 3, 4,... ).

Calculate the time difference between timestamp t and t _n , compare the difference with a set threshold (such as 10 milliseconds), and count the number of neighbor pixels whose difference is less than or equal to the set threshold. If the statistical number is greater than the second threshold, the event to be determined is released, otherwise the event to be determined is blocked.

For example, within the 5*5 neighborhood, calculate the time difference between timestamp t and t _n (n=1, 2, 3, 4,..., 24), and set the threshold to 10 milliseconds. If the difference If the value is greater than 10 milliseconds, the timestamp corresponding to the nth neighbor pixel in the timestamp matrix is updated to 0. And count the pixels with timestamps other than 0 in the neighborhood. If the sum of the values is greater than 4, the event to be determined is released, otherwise the event to be determined is blocked. In this example, the operation of setting the timestamp to zero does not destroy the properties of the timestamp matrix, and has the advantage of saving storage space/hardware overhead.

The advantages of this type of airspace processing part are fast speed, event-driven processing, and high quality noise reduction.

In certain embodiments, the noise reduction method of the present invention is applied to a noise reduction device, the noise reduction device is located in an event imaging device, and the event imaging device can be deployed in a chip. A device, such as an electronic device, includes the aforementioned chip, event imaging device or noise reduction device, as shown in Figure 4 .

In the device, an energy collection module, a sensing module, a computing module, a control module, and a communication module are deployed. The energy collection module can receive environmental energy, such as various known electricity-generating modules that use light energy, temperature difference energy, thermal energy, wave energy, friction energy, mechanical energy, and piezoelectric materials to generate electricity, such as nanogenerators.

The sensing module can sense environmental signals and transmit the environmental signals to the computing module for processing. For example, it can be an event imaging device, an IMU, a microphone, etc. The sensed signal is preferably inferred through a spiking neural network to obtain a processed result.

The processed results are respectively transmitted to the control module or/and communication module. The energy collection module connects the sensing module, computing module, control module, and communication module to provide energy for them. The communication module can connect to the cloud or other nodes for necessary information exchange, such as forming a self-organizing network with other nodes. The control module is connected to the external device to control the external device to make corresponding responses; or the control module is deployed in the external device and outputs control instructions based on the communication module to control the external device to make corresponding responses. For these, according to specific application needs The changes made are not limited by the present invention.

In alternative embodiments, the device selectively includes one or more of the following modules: an energy harvesting module, a control module, a communication module.

Preferably, the electronic device is an Internet of Things device/edge computing device/battery-driven device.

Although the invention has been described with reference to its specific features and embodiments, various modifications, combinations and substitutions are possible without departing from the invention. The scope of the present invention is not intended to be limited to the specific embodiments of the process, machine, manufacture, composition of matter, means, methods and steps described in the specification, and these methods, modules may also be implemented in associated, interdependent, cooperative , among one or more products and methods of the front/end stage.

Therefore, the description and drawings should be regarded simply as an introduction to some embodiments of the technical solution defined by the appended claims, and therefore the appended claims should be interpreted in accordance with the principle of maximum reasonable interpretation, and are intended to cover the disclosure of the invention as much as possible. All modifications, changes, combinations or equivalents within the scope of this Agreement, and unreasonable interpretations should be avoided.

In order to achieve better technical effects or meet the needs of certain applications, those skilled in the art may make further improvements to the technical solutions based on the present invention. However, even if some of the improvements/designs are creative and/or progressive, as long as they rely on the technical concept of the present invention and cover the technical features defined in the claims, the technical solution should also fall within the protection scope of the present invention.

There may be alternative technical features for some of the technical features mentioned in the appended claims, or the order of certain technical processes and the order of material organization may be reorganized. After those of ordinary skill in the art know the present invention, they can easily think of these alternative means, or change the sequence of technical processes and the order of material organization, and then adopt basically the same means to solve basically the same technical problems and achieve basically the same technical effects. Therefore, even if the above-mentioned means or/and sequence are clearly defined in the claims, such modifications, changes, and substitutions shall fall within the protection scope of the claims based on the principle of equivalents.

Each method step or module described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the above description has been based on functions. The steps and components of each embodiment are generally described. Whether these functions are performed in hardware or software depends on the specific application or design constraints of the technical solution. Persons of ordinary skill in the art may use different methods to implement the described functions for each specific application, but such implementations should not be considered to be outside the scope of the invention.

Claims (10)

1. A denoising method for denoising an event stream output by an event imaging device, the event stream including an event to be identified generated by a first pixel, and each event in the event stream at least including coordinate information. and timestamp information, characterized in that the noise reduction method includes the following steps: Determine whether it belongs to the first time interval according to the timestamp information of the event to be determined; If the event to be determined does not belong to the first time interval, according to the statistics of events generated by the first pixel in the first time interval and the statistics of events generated by the first pixel in the previous time interval of the first time interval. The fluctuation between the numbers determines whether to block the event to be determined. 2. The noise reduction method according to claim 1, characterized in that: If the fluctuation between the statistical number of events generated by the first pixel in the first time interval and the statistical number of events generated by the first pixel in the previous time interval of the first time interval is less than the first threshold, then: Determine whether there are more than a preset number of pixels among the spatial neighbor pixels of the first pixel, and the fluctuation between the statistics of events generated in the first time interval and the previous time interval of the first time interval is greater than or equal to the first threshold , if so, the event to be determined is allowed to enter the next level system, otherwise the event to be determined is blocked. 3. The noise reduction method according to claim 2, characterized in that: If the statistics of events generated by the first pixel in the first time interval are zero, update the state value corresponding to the first pixel in the state matrix to the first state value; If the fluctuation between the statistical number of events generated by the first pixel in the first time interval and the statistical number of events generated by the first pixel in the previous time interval of the first time interval is less than the first threshold, then the The state value corresponding to the first pixel in the state matrix is updated to the second state value; If the fluctuation between the statistical number of events generated by the first pixel in the first time interval and the statistical number of events generated by the first pixel in the previous time interval of the first time interval is greater than or equal to the first threshold, then Update the state value corresponding to the first pixel in the state matrix to the third state value. 4. The noise reduction method according to claim 3, characterized in that: If there are more than a preset number of pixels among the spatial neighbor pixels of the first pixel, and the fluctuation between the statistics of events generated in the first time interval and the previous time interval of the first time interval is greater than or equal to the first threshold, Then the state value corresponding to the first pixel in the state matrix is updated to the third state value. 5. The noise reduction method according to claim 3 or 4, characterized in that: If the corresponding state value of the first pixel in the state matrix is the third state value, update the mark value corresponding to the first pixel in the mark matrix to the first mark value; If the state value corresponding to the first pixel in the state matrix is the second state value, then the mark value corresponding to the first pixel in the mark matrix is updated to the second mark value. 6. The noise reduction method according to claim 5, characterized in that: If the tag value of the first pixel corresponding to the first pixel in the tag matrix is the first tag value, then the event to be determined is allowed to enter the next level system. If the tag value of the first pixel corresponding to the first pixel in the tag matrix is The second mark value blocks the event to be determined. 7. A noise reduction device, characterized in that the noise reduction device uses any one of the noise reduction methods according to claims 1-6. 8. An event imaging device, characterized in that: the event imaging device includes the noise reduction device according to claim 7. 9. A chip, characterized in that: the chip is equipped with the event imaging device according to claim 8. 10. An electronic device, characterized in that: the electronic device is equipped with the chip according to claim 9, or the event imaging device according to claim 8, or the noise reduction device according to claim 7.